Stamford Spotlight, Part 1: Inside NBC Olympics’ Largest ‘At-Home’ Production Ever
1,000+ support Olympic efforts back at home
This is Part 1 of SVG’s four-part series on NBC Olympics’ at-home operation in Stamford, CT. Make sure to check out Part 2, a look at the studios and control rooms; Part 3, on the “Off-Tube Factory” commentary operation; and Part 4, on the Streaming Factory setup.
The Rio 2016 Games mark NBC Olympics’ largest “at-home” operation to date, with more than 1,100 people supporting the production from its Sports Production Operations Center (SPOC) in Stamford, CT. Although the Peacock used an at-home production model at 30 Rock in New York for the 2008 Beijing and 2012 London Olympics (and, to lesser extent, for the 1996 Atlanta Games), the Rio 2016 operation is roughly 50% larger than those 30 Rock efforts and is the first based in NBC Sports’ state-of-the-art Stamford, CT, facility, which opened in summer 2013.
“I think what the team has built here is truly a sustainable model for future Olympics to go on through 2032,” says Tim Canary, VP, engineering, NBC Olympics. “A big part of all this is projecting what the needs will be in the future and trying to get ahead of it, because you can’t just put technology in overnight and then rely on it for the Olympics. But we’re pretty good at projecting and communicating what we need to do. But, honestly, it’s all about the team here: they’ve done a masterful job in taking what they normally do day to day and expanding it to an Olympic setting.”
Located nearly 5,000 miles from NBC’s operation at the International Broadcast Center (IBC) in Rio de Janeiro, the SPOC houses seven control rooms and NEP’s NCPII mobile unit parked at the truck bay, five studios, more than 40 edit rooms, three Avid Pro Tools audio suites, the Broadcast Operations Center (BOC), and Transmission, along with the Highlights Factory, Streaming Factory, and Off Tube Factory (featuring 18 announce booths).
“It’s just been an ultimate team effort, and you need that with a project of this size,” says Ken Goss, SVP, remote operations and production planning. “[NBC Olympics SVP/CTO] Dave Mazza and Tim [Canary] continue to look at the most efficient workflows, and that’s the key: leveraging what we have built here and creating a better Olympics product. These guys have been working for the last two years, so why not take advantage of the amazing campus that they have built here.”
Central Ingest at the Center of It All
As the main recording area where OBS host and NBC feeds are recorded and logged for later commentary, editing, and playback, Central Ingest is at the core of every piece of content produced by NBC Olympics for every platform. It serves as the content hub for NBC’s massive Stamford operation, a corresponding Central Ingest area is located at the IBC in Rio, and both locations can see the other’s ingests at all times.
“The is very different from the London and Beijing Games we ran from 30 Rock,” says Coordinating Producer Karre Numme, who oversees NBC’s entire at-home effort. “This time, most of the content is live, whereas, in the past, it was all mostly on tape. We do have a ton of live [content] for this Olympics, but roughly 40% of our sports are going to be on tape. So it’s definitely a different vibe.”
At the core of Central Ingest is a command station where production supervisors select feeds from Rio to be ingested, allot an ingest fiber path for each, and monitor the feeds to ensure that they are fed to proper Off Tube announce booths for commentary. Feeds begin coming in around 7 a.m. ET each day and continue through midnight, making Central Ingest a nearly round-the-clock operation.
“We are much more on our toes because there is so much live [content] this year and live sports can obviously end early or go long,” says Numme. “We’re adjusting our schedule throughout the course of the day with more or less taped content to bridge those gaps. It’s a constant juggling act all day.”
As was the case for the 2014 Sochi Games, the majority of NBC Olympics’ loggers are located in Stamford. These operators are responsible for inputting logging information to the media-asset–management (MAM) system, which allows searches of the recorded material. Each log executed in Stamford is immediately visible to users in both Rio and Stamford.
Says Numme, “We have this long row of workstations with two people working together: a logger working on sport and an AD working with the sport producer in the announce booth to create segment sheets and take in the segments, making sure everything is good from an audio and visual standpoint.”
In the back of Central Ingest is a series of EVS playout rooms outfitted with EVS XT3 servers, which supply playback elements to the respective control rooms in Stamford. The Central Ingest area also features nine Avid Media Composer edit suites, which handle segments that require extensive postproduction work.
“[For Beijing and London] we didn’t have any Avid [edit suites]; we were doing everything off of EVS,” Numme points out. “That has been a big [enhancement] for segments that need to be highly touched up. Weightlifting, for example, is not as easy to call from start to finish as a handball match would be.”
Comms Go IP With Dante, RTS OMNEO
NBC’s comms infrastructure for the Rio Games has been largely revamped from previous Games, with NBC transitioning from 99% analog audio to 99% digital audio with almost no analog audio for comms.
Most notable at both Stamford and the Rio IBC are increased use of MADI digital audio, the addition of RTS OMNEO IP networking architecture (using Dante for media transport), and the extension of trunking from all A-, B-, and C-level venues all the way back to the SPOC (as well as to the Rio IBC). Although trunking was used extensively between the IBC and Stamford or 30 Rock for previous Games, this marks the first time Stamford has direct access to resources within any of the other trunked intercoms at the major venues at the Games.
“We are trunking all the major venues through our trunk master, which is a huge benefit,” says Canary. “You can pull up the major venues right from your panel. We simply could not do that [at 30 Rock] because of the complexity of trunking in New York, and we didn’t have enough expandability. It’s an absolute game-changer, because we can talk directly to [the venues] much more easily and reliably.”
Highlights Factory Is Bigger Than Ever
As NBC delivers more VOD content with each passing Games, the Highlights Factory (HLF) located in Stamford has become even more integral to the NBC Olympics operation. The roughly 80-person operation comprises roughly 80 workers (plus 70 for the Streaming Factory), is built around an Avid Interplay MAM infrastructure, and is responsible for creating hundreds of clips a day (largely highlights but also B-roll, behind-the-scenes footage, and social-media material).
The HLF depends on 4G connection between Rio and Stamford, with a Harmonic Omneon MediaGrid shared-storage system in both locations, which actively replicates proxies to both systems. Each file is replicated via FileCatalyst so that HLF users, as well as users in Rio, can see a proxy seconds after it begins growing on the Rio MediaGrid. Hi-res replication occurs when the user sends an EDL to a target destination.
“[Highlights Factory] is a bit different from [that deployed for the two previous Games] because we always try to make it better every time,” says Jim Miles, director. digital workflows, MAM/HLF/EVS. “We’ve once again taken our core media-management system here in Stamford and grown it out to a remote site so that it’s one big system with hi-res recordings on both sides and replicated proxies; we always have a local copy of the proxy media. This is FileCatalyst’s second Games with us for file acceleration, and they’ve worked quite well.”
New this year for users in Rio is remote access to archival footage in Stamford and remote send-to-archive functionality to Stamford. In parallel, use of a cloud-based editing platform has allowed quicker turnaround of content for the web and mobile.
“For the most part, if you need content from anywhere across the different systems, you can reach in and pull it,” says Miles. “If you were at a remote venue like swimming and you need something from the London Games, you can reach into the archive system, pull that shot you need, and [the system] will grab that piece of hi-res and move it all the way out to the venue for you.”
After the Games, NBC will write a hi-res LTO archive copy on each side, bring the tapes home, and load them into the Spectra Logic/SGL tape robot in Stamford.
“We’re stretching this facility beyond its capacity, and we’ve doubled our staff that’s normally here in the building,” says Miles. “But it’s nice to have our own space that’s more dedicated to the Games [compared with 30 Rock].”
CLICK HERE for SVG’s in-depth report on Highlights Factory.
The Connectivity Backbone: Four 10-Gig Pipes
NBC’s Rio and Stamford operations are connected via four 10-Gbps fiber circuits (10G essentially has a full backup, making 20G of usable bandwidth) and a 45-Mbps back satellite circuit via NSS 806 (Intelsat 806). Between the MPEG, J2K, and IP HD video feeds, more than 130 HD paths are leaving Rio, most landing in Stamford. More than 30 paths are leaving Stamford, along with dozens more paths sent to iStreamPlanet for streaming.
“We are about double what we were in Sochi, which was about the same as London. So, from size, it’s definitely gone up,” says Chris Connolly, director, transmission engineering. “I think, with the advances in IP, things have gone smoother. We’ve learned a lot in the past four years about how IP flows with video in reliability, and we’ve definitely been more reliable than ever this time around.”
NBC Olympics is also using Haivision Makito encoder/decoder systems for non-contribution quality video/audio feeds to and from the venues that allow Stamford control rooms to view feeds directly from Rio venues. These units deliver secure, low latency, HD video over NBC’s venue data networks at extremely low bitrates for the multi-viewer return of sources available at venue.
In addition, program feeds are delivered to commentators, so they can respond to graphics inserted from the control rooms. The engineering team installed a frame of encoder blades in the Rio and Stamford IBCs, which were fed with the Program Output feed of the local control rooms. The encoders are connected to the NBC IT network for distribution to the venues. At the venues, a decoder appliance is connected to the network and is configured to decode the correct control room feed and deliver the HD/SDI feed to the venue distribution system.
“In certain venues we have access to multiple feeds but because of costs we don’t want to use a large amount of resources to bring all those feeds back,” says Connolly. “So we put a Blackmagic [Design] switcher out in Rio that we will then take the output of that switcher and control it from here and that’s what gives us our feed. We use Haivision [encoder/decoders] to bring back a multiview of all those feeds. It gives us the ability to look at it in almost real time. Because of the speed of the Haivision, it’s not drastically behind anything else we have with the J2K [compression]. And then we also are using Haivision to bring back a couple feeds that we’re streaming online.”
Other paths land directly at 30 Rock; at Golf Channel in Orlando; at Telemundo in Hialeah, FL; and at NBC’s Dry Creek facility in Centennial, CO. As always, NBC’s “shortest path to air” philosophy means that on-air transmission paths do not daisy-chain through multiple NBC facilities, except in the case of emergency circuit restoration.