Keywords

1 Introduction

The 5G ESSENCE project [1] addresses the paradigms of Edge Cloud computing and Small Cell as-a-Service (SCaaS) by fuelling the drivers and removing the barriers in the Small Cell (SC) market, forecasted to grow at an impressive pace up to 2020 and beyond, and to play a “key role” in the 5G ecosystem. 5G ESSENCE provides a highly flexible and scalable platform, able to support new business models and revenue streams by creating a neutral host market and reducing operational costs, by providing new opportunities for ownership, deployment, operation and amortization.

The 5G ESSENCE accommodates a range of use cases, in terms of reduced latency, increased network resilience, and less service creation time. One of its major innovations is the provision of E2E network and cloud infrastructure slices over the same physical infrastructure, so that to fulfil vertical-specific requirements as well as mobile broadband services. In this paper, we investigate the core architectural components, as well as the cloud testbed for the deployment of our use case.

2 Use Case Description

Broadcasters are looking for new ways to cover events, to offer exciting point-of-view perspectives to viewers on one hand, and to reduce production and delivery costs on the other. At the same time, network operators “target” to increase the usage of their networks, while stadium owners have a strong interest in making the visitors’ experience as pleasant as possible and look for ways to increase side-revenues apart from ticket sales. The reception of live content from cameras located in the playing field, replays and additional contextual information on mobile devices, becomes a strong use case for all the above actors. The main challenges even with 4G networks are the delay in the delivery of such content in the range of milliseconds rather than seconds and the considerable strain imposed on the backhaul network. By deploying a 5G ESSENCE-based network these challenges can be efficiently “addressed”.

The 5G ESSENCE delivers benefits to media producers and mobile operators as it enables them to offer a highly interactive fan experience and optimizes operations by deploying “key functionalities” at the edge, i.e., evolved Multimedia Broadcast Multicast Services (eMBMS) or local network services like real-time analytics together with multitenancy support by small cells. By leveraging the benefits of SC virtualization and radio resource abstraction, as well as by optimizing network embedded cloud, it becomes possible to ease the coverage and capacity pressure on the multimedia infrastructure and also to increase security, since content will remain locally. Furthermore, additional benefits for the operators and the venue owners arise such as: (i) lower latency, due to shortening the data transmission path, and; (ii) maintained backhaul capacity, due to playing out the live feeds and replays locally that puts no additional strain on the backhaul network and upstream core network components. Figure 1 depicts the use case “5G edge network acceleration at stadium”.

Fig. 1.
figure 1

5G edge network acceleration at stadium

Full size image

The scenario provides the logic for distributing the live video feeds received from the local production room to local spectators in a highly efficient manner. Different scale facilities can be used for the validation of the deployment topology. First, a small-scale facility as the municipal open swimming pool with a capacity of 500 spectators; secondly a medium-scale facility like the municipal indoor stadium (which is used for basketball, volleyball and handball games) with a capacity of 2,000 spectators, and; thirdly a large-scale facility as the municipal football stadium “Stavros Mavrothalasitis” which is located at the centre of Egaleo town in Athens and it is an open stadium with capacity of about 8,000 spectators.

Considering a proper selection from the aforementioned facilities, the selected facility will be covered with a cluster of eMBMS enabled CESCs (Cloud Enabled SCs) and, together with the Main DC (Data Centre), can be connected to the core networks of multiple telecom operators. The video content from cameras will be sent for processing locally at the Edge (similar to the proposed use case by ETSI MEC [2,3,4,5,6,7,8,9]). Then the video streams will be broadcasted locally using the 3GPP eMBMS functionality. Spectators will be able to dynamically select among different offered broadcast streams. In this and in similar “big event” scenarios, massive data traffic will not affect -nor overload- the backhaul connection as it will be produced, processed and consumed just locally (i.e. a case of a 5G MEC scenario, aligned to the 5G 3GPP specifications [10,11,12,13,14,15,16,17]).

3 Architectural Components

The stadium use case shows a typical MEC configuration that keeps traffic as close as possible to the users to allow them to consume digital media with high quality and low latency. It shows that integrating network functionalities at the edge allows to fully benefit from advanced applications and services, including also broadcast or multicast-based technologies.

Within the 5G ESSENCE infrastructure, the stadium use case will leverage the main DC to realize its services running as VNFs (Virtual Network Functions). The actual distribution of the service across the data centres depends on the actual load required by the system. The Edge Video Orchestrator – Master Orchestrator (EVO-MO) service, eMBMS and EPC (Evolved Packet Core) functionalities actually run as VNFs in the DCs. In case where a cluster of CESCs is to be used to enable larger deployments, the aforementioned architecture can be further extended. The Edge Video Orchestrator (EVO), eMBMS and EPC core services can be deployed on each CESC’s light DC, while the EVO instances being managed by an EVO-MO located on a central location. In order to achieve multi-tenancy, the EVO, eMBMS and EPC core services could be instantiated multiple times, depending on whether the services needed are also “multi-tenancy capable”. The eMBMS functionality will be triggered by the EVO-MO communicating to a Broadcast Multicast Service Centre (BMSC), which is running locally in the stadium premises. If a camera’s stream is to be transmitted via eMBMS (controlled by a management application), then the EVO-MO will request the BMSC to set up an eMBMS channel. Once this is set up, the BMSC acts as a proxy for the data and forwards it to respective eMBMS channels.

All devices use unicast transmissions for connecting to the EVO network service, which subsequently manages data transmission. Camera/Encoders use unicast transmission to transmit video and audio to the EVO service. Clients use eMBMS multicast reception to receive video and audio transmissions. The EVO services can deliver data to the 5G ESSENCE telemetry and analytics systems that can feed, for example the NFV Orchestrator (NFVO). Typical values that can be sent (other than platform related values such as CPU (Central Processing Unit), occupancy, network traffic, etc.) are application specific information used for orchestration like the number of connected users. More information regarding the 5G ESSENCE architecture is provided in [18] and [19].

The solution platform can be deployed in another different context, as it can be event-specific (stadium, concert, etc.), venue-specific (mall, museum, university, airport, etc.) or area-specific (neighborhood, city and region). In the following sections, the main components of the solution architecture are presented.

3.1 Hotspot LTE eMBMS

The EPC and eMBMS VNFs build the so-called hotspot LTE (Long Term Evolution) eMBMS. In practice, the two VNFs, EPC and eMBMS, are the “core” network components allowing the provision of LTE unicast communications and broadcast communications in the stadium. The two VNFs are also a MEC solution to offload traffic in a certain edge area without affecting the national backbone, as the local traffic stays local.

With the advent of the LTE technology, eMBMS has become an “attractive” solution for network operators who desire to increase bandwidth capacity and improve service quality, while maintaining low-cost investments for upgrading the network architecture. The eMBMS is an optimized broadcast/multicast service, which leverages on a point‐to‐multipoint link to transmit control/data information from the base station (eNB) to a group of users.

One of the key hurdles of eMBMS deployment for business/mission critical applications is that an eMBMS “island” must be integrated within the existing service provider and/or private national mobile network. Our solution solves this problem by providing partial functions of the EPC integrated with the broadcast functions to: (i) eliminate dependencies on a national/centralized mobile core network while being able to interface with it via existing interfaces; (ii) rapid low latency signalling; (iii) optionally eliminate failures due to backhaul scarcity, latency and failures, and; (iv) flexibly deploy virtualized functions at the network edge.

The eMBMS solution can be fully integrated with an existing service provider mobile core network. Moreover, depending on the customer’s needs, the eMBMS can be deployed centralized or distributed to the edge of the network. The latter represents an important step towards making eMBMS efficient in terms of resource usage and delivery delay, while avoiding impacts on the national core network. The eMBMS can also interoperate with an external EPC. The SGi interface of the network operator (MNO)’s P-GW (Packet Data Network Gateway) is used to deliver a media (provided by the MNO) as IP multicast through the BMSC and MBMS-GW.

3.2 Edge Video Orchestrator and Distributor

According to [20], the EVO solution typically consists of the following components: Master Orchestrator (MO), Distributor, encoder and corresponding applications. Additionally, in case of LTE broadcast distribution, the eMBMS component is necessary. The core part of EVO is represented by the MO. The MO is provided as a virtual machine image using CentOS Linux as a base system. The main purpose of MO is to: (i) receive the encoded video streams from one or more connected encoders; (ii) forward the streams to one or more distributors (for unicast distribution); (iii) forward the streams to the eMBMS component (for broadcast distribution), and; (iv) encrypt video streams.

In each practical deployment there is always one MO and one or more Distributors. The encoders are always connected to the MO only. For unicast distribution and multicast, the system does not require any extra hardware components. Configuration of the multicast operation can be done by the EVO interface. Once the parameters are configured the EVO can be used to “tell” specific cameras attached to it to deliver their data to the according multicast channel.

Opposite to that, for LTE multicast, an additional eMBMS component provides HTTPS control interface and dynamic UDP (User Datagram Protocol) ports for video and audio data. The HTTPS control interface provides an API (Application Programming Interface) for managing broadcast sessions. This includes creating and starting broadcast sessions, querying the currently active sessions, and stopping a session. Each created session is identified by the Temporary Mobile Group Identifier (TMGI) and it contains one or more streams (e.g. HD (High Definition)/SD (Standard Definition) streams and audio stream). The number and type of the required streams is specified by the MO when the session is created and cannot be changed afterwards. The process is as follows:

  1. 1.

    MO creates a session and specifies the number and type of channels/streams to allocate.

  2. 2.

    The eMBMS component creates the session and returns a pair of UDP ports for each allocated channel and the TMGI of the created session.

  3. 3.

    The MO starts the session on the eMBMS component and starts pushing the associated video and audio streams to the eMBMS component.

  4. 4.

    The eMBMS component broadcasts them to the UEs (User Equipments).

To connect to the session and open the stream on the UE side, the following information is needed: (i) TMGI of the session, (ii) Multicast address which is used to broadcast the data within the session, (iii) UDP ports used for video and audio data, and; (iv) Key for decrypting the data.

The application retrieves this information from the MO and Distributor upon startup, using a dedicated unicast connection. This ensures that only authenticated users can receive and decrypt the broadcasted streams. After that, the application uses the Qualcomm Broadcast SDK (Software Development Kit) [21] to enable the session using the TMGI and receives the UDP data afterwards, which it can then decode and display.

When using a multicast delivery means, either eMBMS or Wi-Fi, the packet loss probability increases due to the missing MAC HARQ mechanisms. In order to mitigate this problem, it is possible to add redundancy to the user plane meaning that the same data is repeated in a special way in the user plane. If some data is lost, the original data might still be able to be recovered from the received subset. The mechanism is similar as the well-known FEC (Forward Error Correction) mechanisms employed by 3GPP on the radio link. The benefit of additional redundancy in the data path must be traded against higher data volume on the link. Therefore, it is possible to configure the amount of data plane redundancy provided by the system. Further, the system needs a distributor component in case the EVO service should be offered in multiple locations or if one MO instance cannot handle all the load and the installation must be scaled up to multiple physical machines. Basically, the distributor is the same component as the MO, just with a different configuration.

To be completely functional, the system also needs encoders. They receive the captured video and audio data using HDMI (High-Definition Multimedia Interface) or HD-SDI (High-Definition – Serial Digital Interface) and convert them to one or several IP streams. Thus, MO manages these provided streams. The advantage of such approach is that any professional or consumer camera that provides the necessary signal can be used. The encoders are specially designed and manufactured. Its software is based on OpenWrt and extended in such a way that it can directly connect to the MO via the web API.

3.3 Radio Access and Small Cells

The deployment architecture of the stadium use case also includes the small cells that provide the radio interface to support the video transmission to the UE. The small cells will form a zone-based architecture comprised of multiple low power access points coordinated by a localized zone controller, thus enabling the operators to offer high user experience while offloading unwanted data traffic from their macro and core networks. With the deployment of LTE SC technologies, mobile operators (and/or virtual mobile operators) are enabled to service a large number of mobile users, as well as provide users with the bandwidth they need to utilize mobile devices to enhance their productivity. Additionally, since poor 3G coverage remains an issue, mobile operators are enabled to deploy dedicated indoor coverage for voice and data in public indoor and enterprise locations.

Using SCs provides seamless mobility and enhanced user experience in enterprise and public locations by improving the coverage/capacity of the network. Moreover, these SCs provide LTE and Wi-Fi offloading capabilities. The multi-support of both LTE/3G and Wi-Fi access, integrated within a single compact form factor allows for reduced number of boxes and shared services (i.e., power, backhaul, etc.).

3.4 User Equipment

To enable the eMBMS reception in the end devices (UEs), special requirements have to be fulfilled. First, the modem part of the UE hardware and firmware must be eMBMS enabled, meaning the eMBMS technology must be implemented in physical and MAC/L3 layers of the modem chip. To make this functionality usable by an application running under the control of the UE’s operating system (iOS, Android), this eMBMS functionality should be controlled from the UE’s application processor (typically different from the modem). Thus, a so-called eMBMS middleware must be present on the phone to form the interface between the application receiving video and audio data and the low-level functions on the modem. Currently, there are a limited set of middleware SW products allowing this, but are not yet recognized and spread in the market. Hence, special UEs have to be used for this use case. A further challenge for the UEs, in order to use low latency video transmission, is that chunk-based transport cannot be used. The edge video orchestration service uses the RTP (Real-Time Transport Protocol) and eMBMS functions such as service announcements are not used. The EVO product uses own control channels to convey the information. Correspondingly, the eMBMS middleware of the UE must allow it to only use a subset of the standardised eMBMS functionality.

As described in [20], the corresponding client application is available for Android and iOS systems. The client application shows the available cameras. One of the options that can be chosen is eMBMS. If on, then the client will display the video received via eMBMS. Only eMBMS enabled camera feeds can be shown via eMBMS. Users can adjust the number of viewable streams. The client enables up to 9 streams to be shown simultaneously.

3.5 Main Data Center

The 5G ESSENCE platform has been designed to take advantage of MEC concepts to improve user experience, bandwidth efficiency and help mobile operators lighten the service delivery process. The two-tier virtualised execution environment makes a distinction between a very light computational environment, directly attached to the Small Cells (Light DC) – typically used to support the virtualisation of the SC and the execution of “light” service VNFs – and the Main DC, where the most computationally demanding VNFs can run and where implementing all the processes can take advantage of a centralised view of the underlying infrastructure.

Depending on the deployed scenario, the border between Light DC and Main DC can be more or less evident with the possibility, in specific circumstances, to collapse it in a unique physical layer. For small or medium size deployments, the Light DC makes room for the Main DC as the only tier of the Edge DC. The choice of edge architecture is mainly based on the size of covered area and on the number of SCs. In a small deployment scenario, the hardware of the Main DC can vary from one to a few nodes (servers), clustered together through a typical Ethernet infrastructure. In this use case we have no constraints related to physical dimensions or power consumption, so there are many possibilities in the choice of computing resources. If transcoding applications would be needed, the platform could be equipped with one or more GPUs (Graphics Processing Units). This kind of resources could be also useful for applications requiring Machine Learning (ML) algorithms.

Apart from that, this use case does not provide demanding applications in terms of computing resources. On top of the HW Infrastructure there is the virtualization layer that will make use of a standard KVM Hypervisor. This layer will make available the Virtual Machines for hosting the required VNFs (vEPC, eMBMS, EVO-MO).

4 Cloud Testbed

The cloud testbed that will be used for the duration of the 5G ESSENCE, to enable this use case, consists of three different site deployments: (i) The campus of NCSR “Demokritos” (NCSRD), in north-east Athens, is a 150-acre area, combining indoor and outdoor environments, covered by five software-driven 5G wireless nodes and supported by an optical backbone. (ii) The OTE building (OTE Academy), to the north of the city, is a multi-functional complex, combining various indoor and outdoor usage scenarios, and; (iii) The Egaleo Stadium, in west Athens, is going to be an actual “field” testbed, supporting a wide variety of real-life scenarios, ranging from massive MTC (Machine-Type Communications) to flash crowd events. The interconnected sites are located approximately 10–15 km apart and their interconnection is based on microwave links. All three sites use a common base infrastructure, but each one has different features.

4.1 NCSR “Demokritos” (NCSRD) Campus

The experimental setup at the campus of NCSR “Demokritos” comprises of five main parts: the core network, the backhaul network, the access network, mobile terminals connecting to the access network and distributed cloud deployments with computing, storage and networking resources. Topology and infrastructure are discussed as follows:

Topology:

The related infrastructure consists of a macro-cell located at a high mast, covering a wider area, within the campus and three SCs serving focused coverage areas. Both the macro-cell and SCs are purely software-defined, in the sense that: (i) they are driven by fully reconfigurable and open SDR equipment, and; (ii) they also feature edge computing equipment (“light Data Centres”) for hosting virtual appliances. All radio access points are connected to a core facility (currently based on LTE EPC) through dedicated fibre links.

Infrastructure:

The radio infrastructure for the front-end is as follows:

  • For the macro cell: SDR (Software-defined Radio) platform running a full software-based eNodeB implementation, driving a 5 W HPA (High-Power Amplifier) and a sector antenna (outdoor, installed on a 70 m-high mast).

  • For the small cells: SDR platform running a full software-based eNodeB implementation, with indoor low-power antennas.

The testbed also features micro-servers integrated in each macro-/small cell, according to the CESC concept, which has been adopted in 5G SESAME project [22] and the 5G ESSENCE project [1] for mobile edge services. These micro-servers create a distributed Network Function Virtualisation Infrastructure (NFVI) based on OpenStack (supporting VMs or Containers) and OpenDaylight. In this manner, CESCs are both edge computing-capable and multitenant, enabling a small cell infrastructure to be shared among multiple tenants. Edge services deployed in the CESCs include several VNFs (vDPI, Security Appliance, Network Monitoring) as well as of virtualized appliances (Transcoder, Video analysis for object recognition).

The core functions (EPC) are also located on site. EPCs from different vendors have been tested. The eNodeBs (SCs and Macro Cell) are connected to the EPC over different types of backhaul links (fibre, RF, copper). UEs of various types have been tested (mobile phones by various vendors, laptops equipped with appropriate dongles, software-defined UEs based on SDN platforms). All types of UEs have embedded USIMs, appropriate for authentication to the EPC.

The testbed also hosts a core Cloud/SDN experimental platform for the experimentation needs of the 5G ESSENCE project. The cloud infrastructure is based on the OpenStack Cloud Operating System, comprising of 18 compute nodes (180 CPU cores, 1.5 TB RAM) and storage system supporting SAS (Serial Attached SCSI) and iSCSI (Internet Small Computer System Interface). NCSRD’s testbed also features SDN-enabled network devices that support OpenFlow protocol (i.e., HP 3500yl-24G, Dell PowerConnect 5524), as well as legacy devices including firewalling and routing equipment (i.e., a Cisco 2921 Integrated Router, a Cisco ASA 5510 Firewall). Finally, the NCSRD site connects to the Internet via the national NREN (GRNET), using a dedicated hardware firewall whose rules are fully customizable. VPN access is possible, via SSL (Secure Socket Layer) and IPSec (IP Security) connections (Fig. 2).

Fig. 2.
figure 2

(left) High (70 m) mast within the campus of NCSRD, hosting the macro-cell base station; (right) Cloud/SDN infrastructure in NCSRD

Full size image

4.2 OTE Testbed

OTE’s testbed topology is as illustrated in Fig. 3, below.

Fig. 3.
figure 3

Topology of the testbed in OTE premises

Full size image

The OTE’s testbed includes the following equipment:

  1. (i)

    An OpenStack-based cloud infrastructure (>220 CPU cores, >30 TB HDD, >340 GB RAM), consisting of 1 gateway, 5 controllers, 4 x86 + 2 ARM-based compute nodes, a VPN Server, a CISCO PIX FWFootnote 1, switches/routers while being interconnected to OTE’s Labs thus providing additional capabilities for testing new technologies either for proof-of-concept (PoC) or for field trials.

  2. (ii)

    Eight Nokia 4G/4G+/Wi-Fi Small Cells distributed in two floors.

  3. (iii)

    A flexible and scalable E2E (end-to-end) IoT (Internet of Things) platform, developed from scratch exclusively by OTE, including: (a) A wide range of end-devices/sensors such as, air-quality, temperature, humidity, pressure, activity, luminance, fire as well as power/energy ones, communicate with the backend (cloud) infrastructure over a wide range of short/long range technologies (Ethernet, Wi-Fi, z-waveFootnote 2, BLEFootnote 3, LoRaWANFootnote 4, NB-IoTFootnote 5); (b) IoT hubs/gateways (local and remote - based on LoRaWAN) for facility automation and energy management/control (based on events/rules) supporting multiple HAN (Home Area Network)/BAN (Body Area Network)/LAN (Local Area Network)/WAN (Wide Area Network) technologies/interfaces; over 150 Techs/protocols are currently supported, and; (c) A (common) backend infrastructure (including storage, monitoring/data visualization, command exchange, etc.).

  4. (iv)

    A broadband connection over GRNETFootnote 6, serving as backhaul link.

4.3 Egaleo Stadium

The Egaleo municipal stadium “Stavros Mavrothalassitis” is a football ground located in Egaleo, in the west of Athens. The stadium was built in 1966 and totally renovated in 2006 (Fig. 4). The stadium belongs to the Municipality of Egaleo (MoE) and its capacity is 8.217 (seated) spectators. The stadium is used by the football team of Egaleo (Egaleo FC). In addition to the football matches, the stadium is used for various events, such as music concerts. In general, the Egaleo municipal stadium is a modern stadium and has all the appropriate facilities and infrastructures. The infrastructure at the Egaleo municipal stadium (Fig. 4 - left) aims at showcasing the value of 5G in big sports and/or cultural events. The access part of the infrastructure is based on three low-power cells, covering the entire stadium (Fig. 4 - right), integrated with local virtualization-capable IT resources, thus forming CESCs as described in the previous section. The capability of the CESCs to be multi-tenant and to host edge services enable a new business model, under which the CESC infrastructure is owned by the stadium operator (rather than by MNOs), who, in turn, leases slices to MNOs, MVNOs and enterprise users. The stadium is connected to the main site of the testbed (NCSR “Demokritos”), where the core functions are expected to reside, with a backhaul point-to-point microwave link. A VPN (Virtual Private Network) over the Internet also exists, yet only as a fall-back solution.

Fig. 4.
figure 4

(left) Egaleo municipal stadium; (right) Placement and coverage of radio cells

Full size image

5 Discussion

5G ESSENCE delivers benefits to media producers and mobile operators as it enables them offering a highly interactive fan experience and optimizes operations by deploying key functionalities at the edge (i.e., eMBMS or local network services like real-time analytics together with multitenancy support by small cells). In this paper, we investigated the core architectural components of the aforementioned use case, as well as the cloud testbed for the respective deployment.

A large-scale facility as the municipal football stadium “Stavros Mavrothalasitis” will be used for the validation of the respective 5G ESSENCE use case. The coverage in this facility will be provided by a cluster of multitenant, eMBMS-enabled SC and a main DC connected to the core networks of multiple telecom operators. The Edge DC will be processing video content from cameras deployed on-site, which will be broadcast locally without affecting the backhaul. Our future work will be focused on validation and testing in order to evaluate the proposed 5G ESSENCE architecture.