Live From FIFA World Cup: Inside Fox Sports' Remote Production Ops at Pico Blvd. Facility in L.A.

'The Vault' houses remote graphics, replay ops while World Cup Tonight is also produced as an HRP in L.A.

While Fox Sports has rolled a robust presence in Doha for the 2022 FIFA World Cup, it’s Pico Blvd. broadcast facility back home in Los Angeles and FOX Media Center in Tempe, AZ are also playing a sizeable roll in the network’s sprawling production. In addition to having replay and graphics operators located at The Vault in Pico, Fox is producing World Cup Tonight as a Home Run Production (HRP) out of PCR 205 in LA. Tempe, meanwhile, is handling the bulk of the origination, affiliate coordination, and commercial insertion for broadcasts on Fox and FS1.

Inside Fox Sports’ FIFA World Cup remote production operating out of The Vault at its Pico Blvd. facility in L.A.

“I’ve worked a lot of World Cups at this point in my career,” says Gabe Nucci, engineering lead for Fox’s World Cup coverage, who came out of retirement to work on the event. “And what makes this event so special is the technical challenges and the people I get to work with. To be reunited with a group that I’ve worked with on so many World Cups in the past is what it’s all about. It’s an amazing group.”

VP of production and engineering Rob Rees seconds that sentiment, “The World Cup is not a set shoot strike show. It’s a massive, long tournament. So, sure, the days can be a bit of a grind, but having the people to the left of you and the right of you to solve these challenges, it’s hard not to be proud.”

Inside The Vault: Replay, Graphics, and LED Studio Playout Ops Back at Pico

Fox has five EVS replay operator positions in The Vault that remotely control server located in Doha for the studio shows. These stations are staffed by 10 rotating operators and teamed up with another 10 tape producers.

Additionally, Fox has got five graphics stations in LA, including two Vizrt Trio systems running in the cloud for lower thirds and full screen graphics for the studio shows. An Unreal Engine station is also in LA powering all the AR and virtual graphics for the studio shows, as well as two additional Viz systems remotely driving the cavalcade of LED screens inside Fox’s Doha studio. Two Fox Box operators are also on hand at Pico to provide the scorebug graphic for Fox’s live match coverage.

“We have our graphic engines running in the cloud,” explains Fox Sports VP, Field Operations and Engineering, Kevin Callahan, “and we’re using NDI to come out of the cloud, convert back to SDI, and output to the LED displays at the studio. The lower-thirds and inserts are all essentially Viz Trios that are running in the cloud. The graphics operators and replay operators are in Los Angeles sending commands to the equipment here.”

World Cup Tonight Gets the HRP Treatment

World Cup Tonight, one of two daily shows being hosted from Fox’s Doha studio, is being produced as a HRP out of PCR 205 at the Pico facility. Eight camera feeds (three of which have AR integration) and their associated audio feed are coming back to LA, where the show is cut. These camera feeds are converted from 50 fps to 59.94 (all 1080p HDR) in Doha before being shipped back to the U.S.

“All the production integration is here [in L.A.],” says Nucci. “We do everything locally here in 59.94 1080p and the graphics are locally generated here for [World Cup Tonight]. And the show’s buttoned up and delivered from here that way.”

Transmission Scheme: Redundant Pipelines and Origination Out of Tempe

The Tempe facility serves as the origination point for all of Fox’s domestic transmission. While LA and Tempe parallel each other to provide redundancy, Tempe serves as the origination point for all domestic transmission (with the exception of the Opening Ceremonies, which utilized the L.A. facility for origination).

“We have a network between us so that anything coming in here also goes to Tempe and vice versa,” says Nucci. “Because of how busy it gets with [NFL and college] football operations here on the weekends, we decided it was best to have Tempe handle all of that. And then we protect each other just in case.”

“Everything is designed for redundancy so that both [facilities] are backing each other up,” adds Rees. “So if at any moment if Tempe goes down, Pico can take over. Or if in a situation something changes in this distribution or contribution schedule, Pico can take over. So we’ve got essentially everything built so that any match room or studio show can come into either the facility, but Tempe is the originating facility.”

All match feeds from host broadcaster HBS and all unilateral feeds from the venues and studios are sent from Fox’s IBC setup and then onto Tempe and L.A. FIFA and HBS are offering a choice of feed formats, and Fox is taking both the 12-Gbps 4K HDR feed and then the 3G HDR feed with the latter being the base level. More than 100 feeds are being sent to the U.S., and latency is only 128 ms one way.

Fox has established two diverse red and blue 100 Gbps fiber circuits that go through Europe to bring all these feeds back (provided by Lumen and VIDI). In addition, Fox has two paths going through Singapore to serves as failovers for the Europe-based paths.

“For our studio programs, there are actually four distinct program feeds coming back: a primary and backup for both Pico and Tempe, and they’re travelling over four discreet paths,” says Nucci. “SO, if Tempe was on the air and their backup was to go away, we would still have two copies of that coming here and they could take either one of those as their backup. So we’re crisscrossed every which way to [ensure] everything is protected.”

Fox’s file-transfer operation also run on these same pipelines. For the 2018 World Cup, Fox was sending all its content to the cloud, but this year files are being delivered directly to Pico. As a result, all content is instantly available in Fox’s in-house Google Cloud-based IAS media-asset management system across all of Fox’s facilities in LA, Tempe, Doha, and Charlotte.

“A lot of our workflows are centralized around being able to leverage direct connectivity into Google and in-country in Doha, which allows us then to seamlessly access all of our Fox Sports library and content that has already been ingested prior to the start of World Cup,” say Rees. “Now, we can grab both archival footage and streamline our media management workflows by sending [files] either through 100-gig circuits back into Pico or archive it directly into Google. We are also leveraging a complement of storage on onsite in Doha for means of quick turnaround editing and ingest, and it can also be funneled directly into the Google IAS for access to anybody with a Fox Sports login.”

The UHD and HDR Factors

Fox is has also undertaken the massive challenge of UHD HDR with the 4K HDR feed being sent Tempe, where ad insertion is completed and the signal distributed to MVPDs and the FOX Sports App.

“That’s being handled between TOC here and Tempe and we’ve gotten great support from them,” says Nucci. “There haven’t been a lot of folks who have dived into the UHD world for playout and true UHD distribution, so that’s a big challenge that we’ve worked together on. But I couldn’t be more pleased and impressed by how they’ve handled it.”

In terms of the HDR component, Fox is relying heavily on Telestream’s Vantage software for up/down conversion between archived SDR content and HDR content from this year’s tournament.

“Wherever we’ve got historic footage from previous tournaments or anything that’s not originally HDR, it goes through the LUTs [Look Up Tables] in the Vantage workflow, which then delivers that [content] to our EVS’s and to our craft edit [suites],” says Nucci. “It’s really more about people understanding the workflow than the technology itself so we spent a lot of educating our staff and crew. The technology works, it’s just that you if we’re not vigilant it’s easy let something sneak by.”

Rees says his team’s production of HDR broadcasts as HRPs during the USFL season earlier this year provided valuable experience leading up to the World Cup.

“That was one of our major steps, at least for our facility into leveraging broadcast in true HDR,” he says. “Fortunately for a lot of our group here in LA, we’ve had enough experience going through an entire USFL season, dealing with Vantage transcodes, LUTs, and the different color space, so we knew how to deal with the challenges that have come up during the World Cup.

“One of the biggest challenges is just preparing and educating your staff and working with the vendors,” he adds. “People think HDR just means everything looks brighter and better, and that’s not always the case. So having that experience with USFL allowed us to get our feet wet. So going into World Cup tonight and dealing with LUTing and dealing with different color spaces, it was a little bit easier to muster.”