Esports Production Forum: At-Home Workflow Helps Deal With Explosive Growth
Vendor, producer execs explore major issues
In esports productions, the sheer scale of events played in far-flung locations poses big challenges for the production teams. The biggest is figuring out whether connectivity will allow an at-home production, whether those circuits are reliable, and whether the scale of the show is suitable for at-home production. That topic was the subject of a panel discussion at SVG’s inaugural Esports Production Forum.
Mitch Rosenthal, head of production and operations, Riot Games, said that at-home productions help make the production efforts required around the globe more sustainable. At the recent League of Legends World Championship (commonly referred to as “Worlds”) in South Korea, the Riot Games team expanded on an at-home workflow that it had relied on in May during the Mid-Season Invitational in Paris.
“We wanted to push the envelope,” said Rosenthal. “We did a proof of concept from Oakland to Los Angeles before using it for 22 days to do live production in Korea.”
Surprisingly, he said, the challenges were not as significant as they had been in Paris because, by Worlds, the talent and production team had more experience. The main challenge was the increased number of checklists the team had to run through.
“You have to consistently look at every piece of gear and how it is working over the IP circuit,” he explained. “But, in terms of workflow efficiency, it was pretty immense. At the last Worlds, we had six different production teams, and the core engineering and production team had to handle the broadcast, the trucks, and five other countries that were producing onsite.”
For the most recent edition, the world feed was centralized, and functions like camera shading and audio were handled in Los Angeles.
“I can’t say enough about the team and the infrastructure,” said Rosenthal.
The at-home approach allows staff and crew to work in a production environment that they have called home for the entire season, to sleep in their own beds. It also allows financial resources to be used for more-important production elements, such as augmented reality or moveable stages.
“It’s all about reinvestment,” he said.
Linda Hannan, director, sales, The Switch, cited BlizzCon, which finished up at the Anaheim Convention Center last week and demonstrated how major esports events require large signal-transport pipes. The Switch, for example, provided 800 Mbps of connectivity from the convention center to AWS in supporting Blizzard Entertainment, which had 21 stages running and was pumping out 211 concurrent streams. On top of that, there was ESPN’s need for a 50-Mbps link to handle content creation for Twitter out of Bristol, CT, and even Disney XD delivered a 720p version of the action.
“It’s a brave new world with all of this,” she said. “How do we maximize the value of the content and take different competitions from different venues all over the world? There is both a domestic and international audience, and the international audience is huge.”
One of the gating factors, however, is reliability and the cost of fiber connectivity. At certain places where the events are held, connectivity options are very expensive and, in the end, make it more cost-effective to be onsite.
Many U.S. arenas and stadiums have connectivity that allows a very sleek at-home model to exist, Hannan noted. “Sleek REMI packages are onsite and can be rolled out when you need them. Univision loves the REMI model because they can save so much money, and crews can work out of a centralized facility. The same thing can go for esports,” she added, “although there are different challenges at international venues.”
Robert Erickson, advanced technology director, Grass Valley, pointed out that new workflows are still to be developed for esports and at-home productions.
The embrace of at-home production is occurring as other technologies, such as IP standards and cloud-based tools, continue to evolve. It provides a larger set of tools that can be applied in new ways. Erickson noted that Grass Valley’s use of SMPTE 2110 IP-based connections between camera and base stations allowed camera signals, tally, audio, and return video to be sent via IP, solving some major problems.
“The new codecs are what we are focused on,” he said. For example, something as simple as using a J2K codec for production in a cloud environment requires a lot of computer processing for encoding. Also content can’t be distributed on the cloud using J2K.
With respect to codecs, Ray Panahon, technical lead, esports, Riot Games, said that his company has undergone a paradigm shift in the way it does production.
“A lot of people have frowned that we moved to MPEG and H.264, a smaller codec, but it fundamentally worked and gave us a workflow that was extensible,” he said. That is important because the team produces events from so many countries around the world.
“We’re trying to open eyes to do things differently, push the craft, and get that skill set out to other regions,” added Panahon. “When we look at emerging regions like Vietnam, the biggest challenge at a simple level was that we could not find a Ross or Vizrt operator, and that massively impacts the way we tell a story.”
SMPTE ST 2110 is also playing a role. According to Panahon, a 4K studio that has been commissioned in South Korea will take advantage of the IP standard and open new bandwidth opportunities. “We are excited about the efficiency gains.”
Erickson added that the move to ST 2110 eliminates the need for quad-link SDI, which brings much more efficiency to the production.
Latency is also a consistent problem that is always being addressed, and it impacts both the players and teams playing the matches as well as the production crew. Riot has its own private LAN infrastructure, Riot Direct, which Panahon says gives the company more control over the experience the players have.
“One day,” he said, “there would be 160 ms of delay, and, the next, it would be 50 ms. We found out it was ISPs’ getting circuits at the lowest cost or using the flavor of the week. And it impacted gameplay.”
Riot Direct provides a peering relationship and mitigates risk by providing a significant amount of bandwidth that can be relied on.
“Delay is the biggest fear everyone has, how it will affect the production flow,” said Hannah. “We have been working with Net Insight to push the environment, and I am impressed by the esports world and how efficient they are at doing a lot of content over a short amount of time. That also makes it a really good model for at-home production,” she continued. “They can mirror control rooms and leverage staff. The question is, how protected are the paths, how reliant are they, and how flexible are they? And then how can you make that seamless in a way that lets you sleep at night?”
Erickson said that Grass Valley is also working with Net Insight on minimizing delays. “We do what we can to save 5 ms here and there, because, after the signal has been through 20 pieces of hardware, it adds up.”
Panahon noted that latency is significantly more than two frames and can vary depending on where the event is being produced. Erickson added that, because most human beings cannot notice delay of less than 100 ms, there is some wiggle room with respect to the hunt for zero latency.
“If you can absorb another 120 ms of delay and save $200,000, would people be happy with that?” he said. “I think most would say yes. It’s well within human tolerance. You take a hit but have a massive amount of savings.”