Live From FIFA World Cup: Inside Look at Fox Sports’ IBC Operations
In its first men’s World Cup, the broadcaster has deployed its largest production effort ever
The 2018 World Cup Knockout Stage is under way and, with it, a chance for the team at Fox Sports to reflect on a busy period of preparation and execution for an event that the entire Fox Sports family has not only been looking forward to but has been working on.
Fox Sports’ Kevin Callahan says the World Cup operations have been going smoothly.
The core World Cup efforts begin in Moscow, with Fox Sports teams at the International Broadcast Center (IBC) and the main studio at Red Square. But that’s just the beginning: there are talent and technicians working at each of the venues, as well as a large team back home in Los Angeles handling match control, editing, and other duties. The key is leveraging the work done by FIFA and HBS to provide comprehensive match coverage, technical infrastructure, and production services.
“At the end of the day, I think the product Fox Sports has put on with the help of HBS has been fantastic and second to none,” says Kevin Callahan, VP, World Cup operations, Fox Sports. “And that’s what it should be.”
The effort has been years in the making, but, in the past eight weeks, the Fox Sports team in Russia shifted into overdrive. At one point, about 30 days of work had been lost to team members’ suffering from illness, but a lot of the prefabrication work done by the Fox team in Charlotte, NC, and the NEP team in the UK helped save more than a week of integration work.
While the NEP team was busy handling the integration work at the IBC and for the remote flight kits at each stadium, the Bexel teams in the U.S. was busy integrating two temporary control rooms in conference rooms in Fox Sports’ Pico facility. In the match-control rooms, the match output from Moscow is integrated with graphics and a final audio mix, a move that Callahan says gives the coverage more consistency. Bexel was brought in to build two temporary control rooms in Pico.
Inside the IBC
The Fox Sports World Cup production-control room at the IBC in Moscow
At the center of the IBC operations is a production-control room that is focused on match coverage and preps the signal before it is sent to Los Angeles, where the master-control team readies it for delivery to viewers. An important part of the ecosystem is 10-Gbps circuits between Moscow, Frankfurt, and Los Angeles as well as 1-Gbps circuits between the venues. Those circuits allow the Fox Sports team to embrace remote production and home-run productions, such as having the nighttime studio show from Red Square cut from Pico.
With much of the creative team for World Cup Tonight in Los Angeles, the challenge for the technical team was to make the workflows as seamless as possible.
“We didn’t want to hamper the creativity of what the team can be doing in Los Angeles,” says Callahan. “We wanted to give them everything.”
That meant changing some workflows that had been used at the 2015 Women’s World Cup and the 2018 Confederations Cup. At those events, ingest and live-playback operations were combined. For the World Cup Russia, those groups operate independently, and the ingest team has its own area within the IBC.
“At the Women’s World Cup and Confederations Cup,” notes Callahan, “the team was pulling double duty, and that didn’t serve either function well.”
From left: Chris Swan, John Daigle, Mike Carnow, and Brandon Potter inside the Fox World Cup ingest area in Moscow
There are only two edit bays at the IBC with between nine and 17 bays in Los Angeles concurrently running Adobe After Effects and Premiere. The Telestream Lightspeed Live, Vantage media-processing platform, and Telestream cloud services are in use and allow up to 60 simultaneous record channels in the IBC, with more than 25-55 TB of material uploaded every day to Amazon Web Services in Frankfurt via a 10-Gbps circuit. Once on the AWS service, the content is available to the team in Los Angeles. Also, three of the core feeds from FIFA are sent directly to Los Angeles via 220-Mbps streams. Brandon Potter, director, post production, Fox Sports, credits Dave Norman, principal sales engineer, Telestream, and Mike Flathers, CTO, Aspera, with creating the integrated Telestream/Aspera workflow.
“We threw a number of curveballs at them, and they hit home runs every time,” says Potter. “We can do live-record subclipping with the tight integration between Telestream and Aspera FASP Stream protocol to write three high-priority feeds directly to storage in Pico. There is only a 10-second delay, and then the Premiere editors have access to edit with the growing files.”
By the end of the World Cup, the system is expected to have done 6,500 full-length records at 1080p accessible via Levels Beyond Reach Engine media-asset–management system in Pico with a petabyte of content then moved to the cloud.
“We cast a pretty wide net and want the team in Pico to have access to all of the feeds HBS provides,” adds Potter.
One of the resources FIFA makes available to rightsholders is FIFA Max Server, which offers a wealth of content. The clips also are available to the team in Pico via Levels Beyond Reach Engine. That system allows the team in Los Angeles to look at low-resolution proxy video and select a clip that can then be sent over at full resolution.
The ingest team’s work also extends to the venues. Creative Mobile Solutions (CMSI) has provided remote–point-of-capture (RPOC) systems located within the Fox Sports remote kits. Those systems allow any of the six ENG crews that Fox has roaming around Russia to plug a card into the system, call the team at the IBC, and have the team transfer the content from the card to the servers in the IBC. CMSI is also overseeing the Fox Sports Assignment Desk’s use of LiveU bonded-cellular transmission systems.
Callahan’s preparations for the World Cup involved meetings with the production team to make sure that as many creature comforts as possible were available. That even extended down to the two onsite conference rooms that are replicas of the production team’s favorite conference rooms in Pico.
“The distances separating the talent and production staff is a challenge, but we have been working through that pretty well with the use of the Zoom teleconferencing system,” says Callahan. “It’s one of the best I have ever seen, with a remote-control camera and Logitech speaker that sounds great and has very few drops.”
One other area where the production has shifted home is for translation services for postmatch interviews. The original plans called for interpreters to be onsite at the IBC, but, instead, they are in Los Angeles, interpreting interviews and press conferences off-tube (which they also would have been doing if they were in Moscow).
“We realized it was just as easy for the whisper rooms for interpretation to be done from Pico,” Callahan explains. “The roundtrip delay is only about a quarter of a second.”
Fox Sports also has deployed 10 RTS intercom frames running OMNEO: one at the IBC, one at Red Square, one in Los Angeles, one at match control, and one each in the six remote kits. The six are coupled with the RTS ROAMEO wireless intercom system for more flexibility, and, if necessary, the system can always revert to analog for backup.
Backups and redundancy are playing a bit part at the IBC. CenturyLink is on hand managing redundant fiber paths, and there is also a backup that has a similar look to the main studio at Red Square. The hope is that it is never seen on-air, but, in the case of an issue at Red Square, it’s the kind of thing the team would wish it had.
“This is the largest event in the 24-year history of Fox Sports,” notes Callahan, “so we want to do everything we can to stay on-air.”
Different Workflows
Like the Olympics, the World Cup is one of those events where the rightsholder has a lot less control over the match coverage than viewers at home might expect. That lack of control also is a challenge for production professionals accustomed to putting their own stamp on coverage.
“It can be difficult to get some people into the mindset [that they don’t have full control], but, when you put it in perspective, each match has a minimum of 38 cameras, which is the equivalent of an NFC Championship game for us,” says Callahan. “They are bringing a level of production to every match that allows us to focus on the storytelling and what we want to do at halftime.”
He notes that the relationship with FIFA and HBS has been evolving since Fox Sports won the rights to the World Cup.
“It has paid dividends here,” says Callahan. “We have been in lockstep with all of their technology initiatives. We were excited that they had 1080p because it helps with our 720p output, and then the Gigabit services allow us to move a lot of things on the network — like audio, video, data — and push files out to those who are at the matches.”