Live From PyeongChang: NBC Olympics Digital Workflow Group Finds Its Groove
Pretesting, preconfiguration smooth the way for massive infrastructure
Darryl Jefferson, VP, postproduction and digital workflow, NBC Sports, his team, and more than 600 people at NBC Olympics split between PyeongChang and Stamford, CT, find themselves in a good spot at the 2018 Olympics. A wide range of testing and preconfiguration allowed a massive digital infrastructure to be up and running in the days prior to the Games without many of the difficulties experienced during Rio 2016.
“For me, the story of our group has been all the work we did beforehand,” says Jefferson. “We have been sprinting since last spring, and that goes for [the work we did on] every subsystem.”
Those subsystems include a storage- and asset-management system used by NBC staffers at the venues, in the IBC, in Stamford, and even beyond. Last spring, the subsystems that would be used at the IBC in PyeongChang were set up in the basement level of NBC Sports’ Broadcast Center in Stamford, and tests were run to optimize the performance under real-world conditions.
“Avid built legions of testing algorithms to hammer on the system and make sure we did not have failures [when hundreds of users access the system],” he explains. “Every system and subsystem was stood up during the tests and preconfigured, so we were a lot more prepared than we were in the previous Games.”
Matthew Green, senior digital workflow engineer, NBC Sports and NBC Olympics, notes that there were five phases to the tests.
“The initial round of tests was to find the failure point and then identify the hardware that needed to be replaced and the software optimizations that Avid needed to make,” he explains. “Then, we would run another round, identify new places to improve, and test again. At the end, it was less about stability and more about usability and making sure search and playback was faster and better. We wanted to end up with a much smoother experience for everyone.”
According to Jim Miles, director of digital workflow systems, NBC Sports and Olympics, tests in Stamford raised the baseline for readiness once the equipment was up and running in the IBC. That rock-solid baseline allowed the digital-workflow team to focus more on smaller issues and final tweaks.
The Digital Workflow
The digital system put together for the Olympics is used by hundreds of staffers not only to deliver the live content to viewers but also to edit highlight packages, feature stories, and graphics; create social-media content; and, ultimately, get that content ready for delivery via NBC networks, apps, social-media platforms, and connected-TV devices.
At a basic level, the system begins with content coming in from the venues as feeds or, if cut up and assembled at a venue or elsewhere on EVS replay servers or Avid editing systems, as files. Depending on what those files will be used for, they will land at one or more of 100 targets, such as storage systems at the IBC, Stamford, or even a production team at a venue.
“If it is for fast turnaround,” says Miles, “it will be right next to you on the EVS server. If it was a host feed, it will be here at the IBC, and, if it was an off-air signal, it would be recorded in Stamford.”
The SAN at the IBC is just under a petabyte of MediaGrid, about 72 TB of Isilon, and more than 700 TB of Avid storage. The total number of people who can touch the content in the asset-management system is close to 300 at the IBC and 300 in Stamford, and the system is built to handle upwards of 500 concurrent users. There usually will not be more than 150 users at one time, and that potential for a heavy load is one of the reasons the tests were crucial.
One of the key goals prior to the Games, Green says, was to reduce the choices facing users as they went about their tasks.
“We wanted to reduce the ways you could [find and move media] because the users would wonder which way was best to do the work they needed to do,” he explains. The inability for the user to easily find the most efficient way to do something [like send a file from a venue to Stamford or search for content] also meant that the support team had to be ready to solve more types of problems. By streamlining operations, the users can work faster, and support can more quickly respond to any issues.
Jefferson says the team also worked with Avid on the behavior of the player used to find content in the system. The player has been improved so that video-search results start up more quickly and the set of results is easier to manage. “Having it return every gold medal when you just care about the last one means you have too much content to sort through.”
Harmonic is playing a key role in storage and ingest alongside Avid editing and storage systems used for both asset management and editing. EVS systems are at every venue and play an important part in the replay operation at the venues and in the control rooms.
“We have great long-running relationships with those companies,” says Jefferson. “It was nice we didn’t have to reinvent the wheel in such a short turnaround.”
One important part of the process is a File Intake Room at the IBC, which is overseen by Kamal Bhangle, manager, remote engineering and technology, NBC Sports and Olympics.
“It is the only room where you can bring content in from the outside and get it into the system,” she says. “Everything is scanned on a separate machine before it is plugged into the main system. Yes, it takes longer, and people have to wait, but everyone knows how serious [virus] protection is now.”
Bhangle notes that, although there were a couple of issues early on with respect to viruses, the intake process has been clean ever since. It is an important step that is coupled with the inability of any critical computers to access the open internet and even some more stringent separation of work groups so that viruses cannot spread like wildfire in the event something occurs.
Miles adds that one interesting evolution of the workflow is that editors at the IBC or Stamford are not only pulling content from the venues but are also now pushing it to the venues.
“Graphics and edited pieces are being sent from home to the venues so that a finished segment can then be sent back to Stamford for playback to air,” he says. “The location is becoming far less important.”
The second part of the digital workflow is readying the feed for delivery to a wide range of devices, from cable and satellite set-top boxes to streaming platforms and handheld devices.
Rob McKnight, director of media operations, NBC Sports and NBC Olympics, says that the team in Stamford tackles transcoding of the multi-bitrate mezzanine-level file delivered to Akamai.
The next Olympics, the 2020 Tokyo Summer Games, is about 30 months away, and, although that may sound like a long time, the time will go quickly, especially given that IBC planning begins in earnest this summer, venue construction is already under way, and any big changes in terms of workflow will need to be locked down in 2019. Some evaluating is already taking part at the 2018 Games.
“We’ve been kicking the tires on Avid’s Nexis storage system, and that has been pretty good,” says Jefferson. “On the EVS front, we are playing around with the 4X and 6X cameras and how they provide a challenge with the ingest channels.”
With respect to future changes to the system, the goals are clear: give the production teams tools to create content more quickly, more securely, and more cost-effectively. And, with more than two years of technology development related to distribution of video via social-media platforms, connected-TV devices, streaming, and more, one of the challenges for the digital-workflow team will be simply keeping up with a client base that gets wider and wider. Every new group of clients or an expanding client like the social-media team adds a new dimension to the workflow.
“The social team is just getting bigger and bigger,” says Bhangle. And those social-media platforms will demand higher-bitrate content to deliver a better experience.
At the top of Jefferson’s wish list is a faster and more human way for users to find the video content they want. Search functionality that is frustrating or confusing is a year-round complaint from many who have to hunt for specific clips in a sea of content.
And interoperability is more important than ever.
“As you try new stuff,” he says, “you care less about what is in the box and more about how it will interact with neighboring equipment. If it’s difficult to get it to interact well, it’s worthless to us.”
Cloud-based workflows, more remote production, increased reliance on virtual machines, and more could be part of the plan, but there is no going backwards.
“Our asset-management system has been running on virtual machines since 2014 in Sochi,” notes Jefferson. “We are looking into making that system a little more flexible and seeing where it makes sense to use a VM infrastructure. You can use that infrastructure for more systems, like doubling the amount of transcoding on the fly without having to need more physical space for more hardware.”
Miles says that the VM side on the computer-processing end could extend out to storage with buckets of specialized storage systems replaced by commoditized storage options. “We can’t change the service level or have clients working slower. If we move workflows into the cloud, it has to work as good as it is now or better.”
The good news, says Green, is the technology refresh cycle is faster, so the Olympics, which used to be where new technologies were tested, could benefit from similar efforts at other NBC Sports events. “It is much more of a two-way street and easier to try something at a small scale and then see how it ramps up here. And then we can scale it back down for other events.”
Although the team in Stamford and the team at the IBC are separated by thousands of miles, they do their best to work closely together and make sure that the digital systems are running properly and the end users are working efficiently.
“I think the team overall has been doing really well on both sides of the equation,” says Jefferson. “Our staffers have been great, the freelancers that we have known for years have also been great, and the folks we have selected from the vendor groups have been great. I feel really good about the team. At home in Stamford, we have a full team that wakes up when we go to bed, and we would not survive a day here without them.”