Live From FIFA World Cup: Kevin Callahan of Fox Sports on the IP Transition and Return of Match-Day Control Rooms
Matches are composited into programs at Fox’s facilities in Doha and sent to Tempe and L.A.
The 2022 FIFA World Cup has shifted from the grind of the Group Stage, with four matches a day, to the Knockout Stage. For the Fox Sports production team, getting to this point has taken years of planning and months of onsite activity to ensure that new workflows and old workflows mesh and help deliver a top-level product for fans back home. And Fox Sports VP, Field Operations and Engineering, Kevin Callahan is the first to acknowledge that knowing that fans are tuning in at a record pace makes a difference.
The Fox Sports crew at the IBC gets a welcome break later this week.
“It helps, and it’s one reason we like to let everybody know the figures and viewership that we’re getting,” says Callahan. “It’s great to see the work is being appreciated back home and is being consumed. Being 8,000 miles away, it would be more difficult if we weren’t getting that feedback from friends and family. It helps give you that extra energy on Day 20.”
The IBC facility is 30 x 38 meters and houses the central operations for all Fox Qatar operations. All match feeds from HBS and all unilateral feeds from the venues or studios and stages hit this plant before heading to the U.S. and the Fox facilities in Tempe, AZ, and Los Angeles.
The biggest change for this World Cup compared with the previous few is that all matches are being put together in one of two match-control rooms, each comprising an audio room and a video-mixing room.
“We insert the clock and score as well as do transitions to and from the studio show,” says Callahan. “The audio is here as well, taking our announcers from the stadium and incorporating them with the effects that come from HBS. We haven’t had those match-control rooms in-country since 2015, when we did the Women’s World Cup in Canada.”
The reason for the change? There is an additional commercial break between the national anthems and kickoff.
“Because of the tightness of that break and some elements we are required to get in as a condition of that,” says Callahan, “we wanted to have the match-control rooms here in-country to give us the most flexibility to work with the studio show.” Ross Carbonite production switchers and Calrec audio consoles are key components in those rooms.
Fox Sports Kevin Callahan: “We’ve always talked about converged networks and things coming together, and we truly are there now with a singular network.”
The advantages of having match control onsite is that it reduces the number of signals Fox needs to send between facilities.
“We’re able to deliver our final composited program from here in Doha; that cuts down latency with return feeds back to the venue and it makes for a smoother integration with the studio,” he says. “The downside is, we need to travel more people and it’s taking up real estate in the IBC.”
The IBC is also where Fox Sports drives its media-management and central-ingest operations.
“The Google IAS MAM system makes our content instantly available wherever somebody needs it,” Callahan says. “We are writing that content to storage located here at the IBC and sending the growing files up to the cloud. We’re also doing some live streaming to our storage in Los Angeles direct from here. The latter is for post and quick-turnaround stuff.”
In terms of innovations, the biggest advance has been the use of the EVS Strata media-infrastructure products.
“That’s the core behind everything we’re doing here,” he says. “Their Neuron Compress, Neuron Convert, and Neuron Bridge are at the core of our signal management and routing. They’re extremely flexible: [for example,] Convert allows for up-, down-, cross-, and tone-mapping natively in the ST 2110 environment. We don’t have to go through a gateway.”
Callahan notes that it has flexibility in terms of LUTs (Look Up Tables) that can be loaded in for the UHD/HDR side of things.
“The FIFA LUT is slightly different from the ones we’ve used in our other broadcasts,” he adds. “And the Neuron Shuffle gives us greater flexibility with audio and having different group sizes. With that, we can take in what comes from the host, use it in our own production, but then archive something different, and create the different versions that are needed throughout Fox.”
FIFA and HBS are offering a choice of feed formats, and Fox is taking both the 12-Gbps 4K HDR feed and then the 3G HDR feed with the latter being the base level.
“Everything in this facility is operating at a minimum of 1080p50 HDR,” says Callahan. The studio operations are limited to 1080p50, primarily because of the current limitations of augmented-reality graphics to be fully 4K HDR-capable. The 4K HDR feed is being sent to a control center in Tempe, AZ, where ad insertion is completed and the signal distributed to MVPDs and the FOX Sports App.
“It’s the same source material the linear broadcast would have,” he says.
More than 100 feeds are being sent to the U.S., and latency is only 128 ms one way. That does open up the type of remote workflows that signal a future where everything can be done at home. But there is still very much a need to be close to where the signals originate because problem solving and triage is much easier when they can be done onsite.
“That’s one of the reasons we do all our frame-rate conversion here,” says Callahan. “We are sending product back that is ready for air, and we can compare it to the original and say, Okay, we need to change the setting on the Alchemist or we need to do this or do that.”
That said, there is plenty of reliance on the cloud.
“We have our graphic engines running in the cloud,” he explains, “and we’re using NDI to come out of the cloud, convert back to SDI, and output to the LED displays at the studio. The lower-thirds and inserts are all essentially Viz Trios that are running in the cloud. The graphics operators and replay operators are in Los Angeles sending commands to the equipment here.”
Latency, again, is the key driver for that workflow.
“There is less latency on a graphics reveal,” he adds. “You don’t need to send fill and key across 8,000 miles, when the likelihood of them falling out of time from each other increases tremendously. If we can output them both from here and just have the operator remotely controlling them, it is similar to what we do on the NFL, NASCAR, and college football from the Vault [Fox Sports’ graphics/replay facility in Los Angeles].”
The Evolution of ST 2110
Callahan says that the team and its vendor partners continue to learn about operations in an ST 2110 IP world. Although the technical facilities were able to be fully connected and powered up in the shortest time ever — six hours from loading racks off the flatbeds to turned on — configuration of the IP network took some additional time.
“It always comes down to interoperability and making sure everything is talking to each other,” he explains. “In the IP world, you can have anything you want, but you have to define the rules and define the workflows and make sure that everyone is talking the same protocols. There are different manufacturers, different pieces of equipment, and sometimes it’s even different pieces of equipment from the same manufacturer that are not playing well with others. Some members of the team, Mark Girard and Aavery Mundt in particular, spent countless hours working with vendors to get everything talking. The good news is, all the manufacturers have pledged their support to everyone dealing with this challenge.”
Some of the issues included lack of NMOS support, as well as equipment that wasn’t self-registering on the network and required that each item be defined. The Fox team and the technical teams from key vendors — Riedel, EVS, Calrec, TagVS, Grass Valley — were able to work together and overcome the issues to get operations up and running.
As with many events, big and small, there is less SDI than there used to be.
“Everything here is virtual for the most part,” notes Callahan. “You can’t just put a patch cord in and see video like you could in an SDI truck. There is very little SDI in this place.”
That said, an SDI router is set up and ready to go in case something were to happen to the ST 2110 network.
Greater Teamwork Than Ever
Another key advance this year, Callahan says, is that the Fox broadcast and digital teams are working more closely than ever. The digital team, for example, has its own stage next to the main Fox stage, a move that gives the main stage some additional exposure as a backdrop (reminding viewers to tune in to the main broadcasts and streams) and also gives the digital team greater access to all the content.
“The Twitter Now show is a completely different production that’s originating at the stage but is taking all the host-broadcast resources provided by HBS so they can composite their show together,” says Callahan. “And then, for the first five minutes of the match, they’re taking our program and sending it out on their line to Twitter to help drive tune-in to linear or digital streaming.”
This is also the first World Cup where the Fox Media and the Broadcast Networking Team have worked together to create an IP architecture offering maximum network security.
“It gives us the protection to save ourselves from ourselves,” says Callahan. “It has been a robust system. In this 2110 world, being able to work with your corporate teams to create a network that works for both broadcast and corporate is important. We’ve always talked about converged networks and things coming together, and we truly are there now with a singular network.”
Another essential part of the technical infrastructure has been intercoms, and Callahan is enthusiastic about the Riedel operation.
“There is an extremely large deployment of Riedel across 12 venues here in Qatar, between the stadiums, the Fan Fest, the IBC, and stage locations,” he points out. “It has worked seamlessly, and they’ve been a great partner to make sure that everything we’ve asked for has been delivered. That also goes for integration back to Los Angeles and Tempe.”