SVG Tech Insight: Network and Resource Orchestration as the Key Enabler of Live Sports Production

This fall, SVG will be presenting a series of White Papers covering the latest advancements and trends in sports-production technology. The full series of SVG’s Tech Insight White Papers can be found in the SVG Fall SportsTech Journal HERE.

Nothing beats the excitement of live sports, but broadcast production of these events can be a major technical and financial headache. At all levels of sports and esports, production teams need to balance the viewers’ expectation of the highest production values, with the cost of production.

This dilemma has been the driving force behind the move to remote and even distributed production.

Designed to bring greater flexibility, quality, and cost-effectiveness to production, these models enable the very best resources (or at least the most appropriate ones) to be used — wherever they are located.

The 3Ps

These production resources can be described as the 3Ps: places, processing, and people.

Places encompass studios, control rooms, and outside broadcast production trucks.

Processing represents all the technical capabilities that are needed to carry out the events, including video and audio mixers, record and playback, and up/down/cross conversion for video, as well as audio embedding/de-embedding, shuffle, gain, and switching. Historically, this processing has been done on-prem, but many functions are now increasingly available in the cloud.

People are the production staff representing the most important resource in a production — not least because of the uniqueness of their skills.

Figure 1: Infrastructure building blocks for new workflows

The Required Infrastructure

To enable those 3Ps resources to be pooled and shared in live production, the right infrastructure needs to be in place. There are essentially three main aspects to consider: hybrid processing, media transport, and orchestration.

The main focus of this paper is the orchestration layer. However, it is also worth considering briefly the other two factors.

Hybrid processing simply describes the fact that live processing is evolving to a combination of on-prem and cloud operations, as noted above. Cloud technology is obviously getting a lot of traction right now, with a recent survey by Nevion finding that 89% of broadcasters intend to adopt cloud technologies in the next year for some applications. However, cloud adoption will vary: some organizations may go all-cloud, while others will stay firmly on the ground, with probably the majority opting for a combination of on-prem and cloud processes.

Realizing a distributed production requires the accommodation of existing processing (i.e. essentially equipment) in the new workflows. It also involves the progressive adoption of new distributed and easily scalable processing capabilities on the ground, and as well as cloud-agnostic processing (to avoid a provider’s lock-in). At the same time, it also requires a geographical separation between the generic A/V/M processing backend, and the control functions end-to-end (keyboards, panels, and control surfaces), so that the production staff can effectively work from virtually anywhere, regardless of the location where the actual processing takes place.

Media transport is obviously the piece that connects all 3Ps production resources. There has been a lot of focus in this area in recent years, as the industry has moved to IP technology. For example, standards such as SMPTE ST 2110 and NMOS, have ensured equipment interoperability among various manufacturers. Despite this, the broadcast industry still often seems to underestimate the challenges of scaling, resilience, and security.

Orchestration

This leads us to the orchestration part of the infrastructure. Orchestration is often considered to be an afterthought in workflow transformation projects — despite being fundamental to their success. Creating a distributed production requires the orchestrating of video, audio, and data flows within and between multiple locations. Almost without exception this involves “stitching together” very diverse environments: SDI, IP, IGMP-networks, SDN (software defined networks), LANs, and WANs, and now communication to/from 5G and cloud. So the orchestration needs to be able to handle all these technologies seamlessly, hiding the complexity from the users.

Given that most production environments will have equipment and network switches from multiple vendors, the orchestration system also needs to be vendor independent. A lot of progress has been made in the development and the adoption of NMOS as a standardized way to control equipment, but many older pieces of equipment don’t support the standard, and some specific operations are also not yet supported by it. In such cases, specific equipment drivers are required from the orchestration system. Similarly, network switches from different vendors have different interfaces that the orchestration system must support.

When it comes to controlling the network, the debate still rages on in the industry between the supporters of IGMP and SDN. It is now extending into what exactly qualifies as SDN.

At the core of the issue is how IP networks can cope with the very unique challenge of live media; that is, needing to handle both very high volumes of data and extremely low latency at the same time. Most other industries using IP networks require either one or the other, but not both.

As IP media networks become more widely deployed, the evidence is becoming increasingly stronger that, while IGMP works perfectly well for smaller set-ups (e.g. OB production trucks), SDN is a better approach when media networks are more substantial.

There are of course different flavors of SDN. Some even claim that dynamically changing the class of service in IP switches is SDN. That functionality is in fact a really long way from what proper SDN control can and should be.

Take, for example, the ability of the SDN controller to be service aware so that extra audio streams can be squeezed in the same bandwidth as much larger 4K video streams, to optimize bandwidth usage. Or the capability to include planned usage (e.g. a production taking place at a set date and time) together with ad-hoc usage (e.g. breaking news).

In short, the orchestration needs to offer advanced SDN control — as well as being able to handle IGMP networks, as noted earlier.

Figure 2: Autonomous systems collaborating to share, control, and connect

Federation

Distributed production also brings a new set of challenges when it comes to orchestration.

The first is obviously scalability. The number of devices and media flows that a system needs to orchestrate is huge and likely to grow. This can put an enormous computational strain on the system, and therefore on performance.

A second aspect is security. Connecting facilities together or with external sports venues effectively opens up secured production networks to the outside, creating a risk. Properly built networks can overcome this, for example with firewalls. However, what secures the orchestration system? It needs to control all the resources everywhere to effectively connect them, but this is potentially open to abuse.

The third challenge is resilience. Each facility needs to be able to function independently of the others — broadcasters need to produce shows on the main campus, even if another facility or a venue is unreachable for some reason.

A new development in orchestration provides the answer to those challenges: federation.

With federation, individual systems — for example, at each site — can now collaborate with other systems to share, control, and connect resources across locations securely.

As each system is autonomous and in charge of its own resources, it continues to function and collaborate, even if problems occur in other parts of the federation. The federation capability also enables orchestration systems to reach new heights in scalability.

While remote and distributed production are obvious applications for the federation functionality, the capability can also be used to compartmentalize networks within facilities, for example between ingest, production, and playout.

Federation is also a great opportunity for telecom service providers. It allows them to provide a WAN orchestration that can operate seamlessly with broadcasters’ orchestration, to bring together the customers’ facilities.

Conclusion

Sports production is clearly moving increasingly to remote and distributed production models, in which places, processing and people can contribute to the production regardless of their geographical location.

The building blocks for these new production models include the ability to connect processing in the cloud and on-prem through media network technology. However, orchestration plays the most central role in enabling the new production workflows.

Such systems are now facing unprecedented demands though, in terms or scalability, resilience, and security. However, advanced capabilities, most notably the new federation concept, can deliver on the promise of greater flexibility and cost-effectiveness in sports production.