Live From 2019 FIFA Women’s World Cup: Kevin Callahan Reflects on Fox Sports’ Solid Planning, Execution

The broadcaster basks in the glow of another run by the USWNT

The Fox Sports technical and production plan for the 2019 FIFA Women’s World Cup took years to pull together and months to make a reality. Later this afternoon, it comes to an end as the U.S. and Netherlands face off in the final. And, although many of the Fox Sports team have had their hearts and minds focused on their hometown of Los Angeles with the recent earthquakes, today they will put all their energy into making sure soccer fans at home get all the information they need to cheer on Team USA. Kevin Callahan, vice president, technical operations, World Cup, Fox Sports, sat down with SVG’s Ken Kerschbaumer to discuss this year’s effort, the lessons, and the successes.

Kevin Callahan (left) and Gabe Nucci, engineering manager, World Cup IBC, Fox Sports, inside the Fox quality-control room at the IBC

How have things been going here in France?
I’d say everything has gone according to plan and the team has done a fantastic job. We’ve used more host services than we have in the past, which has been refreshing. I think that we picked the right commentators to do that — [they] are used to [calling a match off-tube] — and I would say that, as a result, there’s no degradation in quality of the product. We’ve had matches where we haven’t had venue kits out, and it changes our communications a little bit, but it goes to the flexibility we’ve built into the system so that we can make changes at a moment’s notice.

How have a couple of the big changes and improvements worked out? What are some highlights?
A big change that we did this year was, we’ve moved to [hitless redundancy] on our transmission paths. That’s been a game-changer for us. We haven’t been without our fiber cuts, and I love playing the game of, “Where is today’s latest back-home incident?” but it’s been seamless on-air. It’s been seamless to us. The biggest effect that we’ve seen is that, depending on where the fiber has been cut, we’ll lose our IPTV contribution from Los Angeles, and that’s because of the way we decided to route certain things. It’s easier not to let that traffic go on the backup.

Fiber connectivity plays a role in everything from hitless transmission paths to video- and audio-signal transport and even file transfers.

I guess that was a result of conversations with CenturyLink?
Yeah. We talked with CenturyLink about it. It was really a design consideration that, the way that the building is set up in L.A., we were always feeding two different sources in. Last year, we had a couple of different times when we were down to our last backup. We weren’t using a hitless-merge situation; we were using dynamic routing. This year, everything is statically routed. We have a north path and a south path, and it stays that way. The backup for the hitless merge is on the other path.

.How are signals being sent to L.A.?
We have basically four copies of everything getting into L.A., and then two copies come out and go into the system. It’s constantly going through, essentially, a voting process as to where it’s getting each packet from: if it’s coming from the north path or the south path. If a packet is missing from the south path, it’ll grab it from the north path.

That fiber connectivity is obviously the key to things like at-home or production anywhere. How does that change the conversation with a backhaul-services provider?
It’s part of the reason we went with all protected services this year. The video itself is hitless merge, and the true data is protected the same way, with a primary and a backup and a north and south path. We’re at the point where we need that old AT&T, Bell Labs, five-nines of reliability. Everything, our whole show, essentially, is now on that. Our entire tape room is in Los Angeles. It would be like if you cut the fiber to the B unit on a traditional mobile unit, cut the connection to it. We expect that type of reliability.

As for the conversations with vendors, we now have them as part of our design process, and they have a more active role. We also now have a network engineer here from the broadcast IT network side of things full-time. And we have a third-party company, Beagle Networks, managing the network and the transport. And then we have a member of our IT security team here full-time, making sure that none of our links get compromised and, if we do see something weird, that they start investigating right away.

I know one of the things you were looking forward to the most was instant archiving, where the feeds are being archived as they occur so that, at the end of the match, you don’t have to do any ingest. What kind of a difference has that made?
It’s been fantastic: it allows for much quicker turnaround of pieces, and it gets things directly into edit systems without having to go through download. And, if you know exactly what it is that you’re looking for on those feeds, it truly makes it available instantaneously.The Aspera FASPStream-based auto-archive that we’ve been doing from the Live Edit streams has saved five hours a day here of archiving. It’s auto-archiving everything as the match is going on. As it’s being written here, it’s also being written into AWS, Last year, because of that compressed timeline, it would write to the storage here, and then, overnight, we would go ahead and move everything up to AWS.

One of the big reasons we have to do that is, we’re operating here at 1080p50. Our edit base in Los Angeles is operating at 1080p50. All of the circuits that go into [the Pico facility] go into the 101 building, the main network center, are 720p. They don’t have the capability of ingesting a feed there in 1080p. By doing this remote ingest, they’re able to use that highest-quality video quickly. Honestly, they’re able to turn it around quicker than if we’d recorded in 1080p in the building in Los Angeles.

One of the technical racks at the heart of Fox Sports IBC operations with equipment from Levels Beyond, Aspera, Studio Technologies, and Lawo

Telestream, Aspera, and Levels Beyond have worked very closely with the retention folks and our media-management team to get all of the little tweaks that were needed to optimize bandwidth. [For more on the project, click here.] It’s funny, when I sit there and I look at our bandwidth utilization, I’m like, “Wow, I could have done with 5 gig instead of 10 gig, apparently.” It’s tremendous, because it shows how well it’s working, and also, now we know we can do this for additional feeds. We’re at the point where we can do this for UHD. We’re actually going to be use Aspera FASP for the UHD feed of the final, which we haven’t done previously. All of that is hopefully to get us out of here that much quicker.

Like I said, it’s worked out fantastic. We’ll be moving a lot of these workflows to our NFL product this year as well.

You’ve also had a number of vendors who have helped in the effort. How have they made a difference?
The NEP family of companies is our equipment provider for this. We have NAB UK equipment over at our set location with the Aurora mobile unit. We have Creative Technologies doing all of our RF, getting all of those shots of the Trocadéro, the fountains themselves, and bringing that home to the viewer. Bexel has provided all the equipment for us over in Pico, all of the additional equipment, building out the two trailers. One is operating in 50 Hz; one is operating in 60 Hz. The 60 Hz is our UHD match control rooms. The 50 Hz is essentially the B unit for everything that’s going on here in Paris. Graphics, replay, social-media integrations, tag board, the Vizrt touchscreen [are] all being done out of Los Angeles but delivered via data by IP here in Paris. We also have a Bexel flypack out for the final just doing a little two-camera insert stage for us.

CMSI has also been involved with our media-management stations here, as all the edit systems we are using in Pico are in-house systems. Noah Gusdorff, [founder/CEO], CMSI, was in Los Angeles when we were getting everything up and going; having him there was a huge help. CMSI has been involved in all four of our World Cup events.

And Canon has also been a great partner. They came out [to ensure] all the lenses were up to spec. It’s the same type of big-event support we expect from Canon and Director, Canon Europe, Hank Yoshida came out to visit the set location on one of the dark days to go through all the lenses.

Kevin Callahan (right) with the team at the IBC Remote Operations Center, where transmission, venue engineering, ingest, and media management have been merged

A new vendor on this is VIDI, and it is our first real experience with them. In the past, HBS used VIDI to provide the connectivity and the transport to all the different venues, which we’ve ordered through HBS. Here, this year, we’re using VIDI and their equipment for the first mile for the connectivity out of the IBC, where it gets handed off to CenturyLink. Robert Oszvald, [director, event services, VIDI,] and his team have been fantastic partners to work with.

We’re also using the brand-new EVS XStore, part of the MediaGrid family. This year, we swapped out what we used last year for newer hardware from EVS, which, looking at the post-tournament time, allows us to break apart the different chassis and use them for NFL. We have 1.3 PB of usable storage here from EVS, and that’s the intermediary step, and, quite honestly, the backup for all of the workflows that we’re doing up to AWS.

And, of course, FIFA and HBS have been wonderful. All of the content they’ve provided this year has been fantastic, and it has been at the level that you would expect from them. They’re really doing this tournament properly.

The front-bench area inside NEP’s Aurora truck, which today will be at the center of studio production for the 2019 Women’s World Cup Final

The biggest part of the operation here in France is the studio show, which is produced completely here in Paris, except for graphics and replay, which are produced in Pico. Was there any thought to just home-run all the cameras for the studio show to Los Angeles?
There was, but, for production reasons, we decided to bring the production staff and the control room here to Paris but keep it as minimal as we could. The feeling is that the interaction the producers are able to have with the presenters being right there, being able to walk up and talk to them in person during the matches, right after halftime, is invaluable. That personal interaction is why we still have that control room here, and, quite honestly, part of the reason the Fox Soccer Tonight show is being completely done out of Pico, where the talent and the production team are together.

It would seem that, from a storytelling standpoint, you can’t have that energy and feedback loop if you are working apart.
There’s no substitute for being able to be in the bar at the end of the day and talk to somebody or being able to just walk up and have a conversation in person and not over an intercom, not over an IFB. That’s something that I think is going to remain.

Look for part two of our interview with Callahan tomorrow as he discusses the evolution of Fox Sports’ World Cup operations over the past five years and early thoughts regarding the 2022 World Cup in Qatar.