EA Gets Apex Legends Esports Back On-Air With Cloud-Based Remote-Production Workflow

Grabyo sits at the core of the production model

As shelter-in-place orders went into effect across the Bay Area in mid March, EA’s esports-production team in Silicon Valley’s Redwood City suddenly found itself banished from its own studios and unable to produce the live esports content it had ramped up dramatically since opening the EA Broadcast Center exactly one year prior.

Rather than admitting defeat, however, the team snapped into action and created a remote-production workflow from scratch in just 10 days, and it has allowed the team to resume live-streaming Apex Legends Global Series online tournaments — from the comfort of their respective homes.

The EA Esports broadcast team created a new remote-production workflow that has allowed the team to resume live-streaming Apex Legends Global Series online tournaments – from their own homes.

“At first,” says Joe Lynch, head of broadcast, EA, “we considered using the control room with a skeleton crew, but we quickly decided that wasn’t an option. Then the question became, How can we pull off our already scheduled event on March 23 without a studio? That’s when our team started evaluating a ton of different cloud-based options. Once we decided we were going to be totally cloud-based, we had only about five days to build and rehearse the entire [workflow], but we were able to go live on-air on the 23rd, and the show was phenomenal.”

While EA’s FIFA 20 Global Series and Madden NFL 20 Championship Series events are still on hiatus, the EA broadcast team successfully live-streamed an Apex Legends Global Series online tournaments on March 21 and 23 and April 4 and 6 and three more planned through the end of May.

Inside the Production Workflow: Grabyo Sits at the Core
The R&D process for this new production model began when Senior Technical Production Manager Brian Bredenbeck sat down with Coordinating Producers Simona Reed (Apex Legends), Sean LaRrett (FIFA 20), and Drew Ohlmeyer (Madden NFL 20) to address the primary goals and challenges.

EA Apex Legends caster Bravo’s setup for calling the action at home

“First and foremost,” says Bredenbeck, “we were looking for solutions that were truly cloud-based because our main objective was locating everyone at their home. We also wanted to have the highest bitrate possible and the highest amount of sources we could ingest at the same time. A lot of the cloud-based products in the market can be very limited when you start piling in all these 1080p60 signals along with all the graphics and the audio, so we needed to find a solution where that wouldn’t be an issue.”

That solution proved to be Grabyo’s cloud-based live-video-production platform, which has allowed EA to deploy a host of feeds for the Apex productions, including four observer (in-game camera operator) feeds and low-latency feeds for talent and guests to interact. Because of its need to incorporate so many feeds, EA nested three layers of Grabyo: one for the observers to cut the game action via OBS (Open Broadcast Software), a quad-split virtual multiviewer for a producer to monitor the observer feeds, and another instance for the main program feed streamed online.

The Grabyo Editor tool also allows EA to iso-record all observer feeds, so that tape-room editors can pull content, clip highlights, and edit packages and then push that content to the producer in the virtual control room for playback.

EA Apex Legends caster Fallout’s greenscreen setup at home

“Grabyo was our home base for all the main production elements — like audio mixing, video playback, replays, and editing,” says Reed, “but we had other issues: like how to allow our casters to see what we were cutting so we came up with some creative solutions. For example, all of the details inside of Apex are so important, and the casters have to be able to see them completely clearly, so you can’t have any pixelation whatsoever. We just dropped the RTMP [Real-Time Messaging Protocol] stream into VLC [Media Player], and it gave us true 1080p quality. The casters were able to see the small finite points that they needed to.”

For graphics, EA used a mix of Ross Video Xpression, its standard graphics platform, and Singular, which integrates directly into Grabyo.

Comms and Audio: Thinking Outside the Broadcast-Tech Box
The 26-person crew, which grew to 30 this weekend, took a page from the gaming world for comms, deploying the popular Discord videogaming VoIP software to stay connected.

EA has announced Apex Legends Global Series online tournaments through the end of May.

“We didn’t go the traditional broadcast-minded route for a lot of this because we were able to use a lot of commercially available products,” says Reed. “For comms, we were presented with the option of using a more traditional virtual comms system, but we decided to go with Discord instead because of the flexibility it provides. Not only did the crew talk to each other on a Discord, but the talent did as well because that was the only way they were able to hear each other.”

Since Grabyo doesn’t feature a native audio desk, the A1 mixed the audio in a roundabout fashion.

“The audio [production] was very unconventional,” Reed explains. “The A1 had the talent open and close their mics themselves. Audio also had to work in tandem with our TD, who was mixing the visuals. We had about eight seats that were able to go live to air if they wanted. It was a whole different kind of workflow: everyone had to trust everyone else not to accidentally go live with [an audio or video element]. At any given moment, somebody could have clicked something, and it could have gone live.”

An Evolving Workflow: Making Enhancements in Week 2
After a successful show on March 21 and 23, EA continued to evolve the workflow for its second Apex Legends online tournament on Saturday and today. In Week 2, Grabyo boosted EA’s dedicated server from 8 Mbps to 12 Mbps (it has since boosted to 20 Mbps), revised the color profile to make the live stream crisper, and released a performance upgrade for the system. Grabyo has also incorporated a mix-minus comms system so that casters can hear each other through the platform (Discord will still be used for the production team to talk to talent and one another) and is actively looking to improve the Grabyo Editor system.

The opening montage for EA’s Apex Legends Global Series live stream on March 23

“It was the first time we had done it,” Lynch notes. “There were definitely some bumps that we are looking to fix next time out. We have spent a lot of time with the Grabyo team, working on things like signal degradation and color issues, and I think we’ve done a really good job addressing those issues. We’re also evolving our personnel and staffing. Obviously, no one had ever done this before so we were figuring it out as we go and shuffling positions in order to maximize what we can do.”

EA has upgraded its hardware complement for the second go-round, the entire crew having worked on a variety of their own equipment at home for the first online tournament. EA has purchased 22 high-end PC gaming systems for the crew, upgraded the crew’s internet connection to business-grade broadband, and deployed custom caster kits featuring high-end web camera, microphone, headset, lighting, and virtual-green-screen equipment.

“A lot of the team was limited by their PCs at home,” says Bredenbeck. “Everyone having different equipment made it difficult to troubleshoot, so we bought high-end gaming systems that will up the quality of hardware and rendering required to do what we’re doing here. And having the [caster kits] will standardize those talent locations and make it easier for us to troubleshoot. To top it off, we launched a business-grade internet at all of these locations, which will be a huge boost.”

Beyond Apex: What’s Next for Madden and FIFA Titles?
Although Apex Legends esports are back in action, EA has yet to announce online tournaments for its two other primary esports titles Madden NFL 20 and FIFA 20. However, Lynch says EA’s esports teams for both titles have been watching the Apex Legends production closely to help develop best practices for when they relaunch their own competitions.

“We have two other [game] titles that have [esports] projects to work on, so we are also using this experience to figure out how they are going to do their shows,” he adds. “We had people watching a screenshare on Zoom and listening in on Discord to get up to speed as to how this process works so that they could start working on their titles. The team did an amazing job just figuring out how to actually pull off a broadcast, and now our workflow is going to continue to evolve, get better and more efficient.”

Lynch notes that, since Apex Legends is a battle royale format with up to 100 teams in a single tournament, 1-vs.-1 games like Madden and FIFA should benefit mightily from the production infrastructure being created now.

“I am truly in awe of how this team came together during such a tough time,” he says. “To watch this team dive in and be so engaged in solving this problem with everything else that’s going on was really inspiring to watch. It was good to be able to dig into something and to try to solve a problem that we could actually have control over during such an unknown time. We’re just excited about being back on-air, and, especially in this time, we’re very happy to be able to create entertainment for our fans. There’s a lot of hard work ahead, but we’re all really invested, and we’re excited for what’s ahead.”