Live From Tokyo Olympics: NBC, Ross Video Take Primetime AR to New Level

The NBC Olympics efforts around HDR are taking augmented reality to a new level. NBC’s graphics-design team and Ross Video’s Rocket Surgery Virtual Productions team have worked closely to bring new realism to AR elements integrated into the primetime set on the terrace of the Hilton Tokyo Odaiba.

NBC and Ross Video are making sure AR elements pop for HDR viewers.

“One of the coolest things we’ve done is getting the AR to look so good in HDR,” says Michael Drazin, NBC Sports, director, production, engineering, and technology. “We can fully integrate AR and HDR. Ross is supporting the NBC LUT [look-up table], and they’ve been working with AR lighting in the scene.”

According to Bo Cordle, Ross Video, senior designer, virtual production, it took a good bit of development effort by NBC’s graphics team and Ross as they dialed in a way to correctly pass NBC’s HDR camera feeds and properly represent the AR graphics in both HDR and SDR.

The Voyager engine, which is Ross Video’s build of the Unreal Engine, plays a key role in bringing features like glass simulation/refraction, reflections, ray-tracing, and real-time shadows generated by accurately geo-located simulated sunlight. Ross’s Lucid interface enables the operators to quickly and efficiently call up and adjust the AR elements at a moment’s notice. Stype RedSpy camera tracking is configured on a Technocrane and Steadicam on the set.

Ross Video’s Chris Brown (left) and Bo Cordle manage AR elements at the primetime set from the IBC.

Cordle says the two-rack flypack was built specifically for the Olympics. It houses three Voyagers, a couple of Xpressions, a Carbonite, an Ultrix, and Opengear. He adds that the kit provides plenty of flexibility to do whatever a graphics team dreams up.

The racks are located at the IBC, about a mile from the hotel, and a protected data path ensures that the AR data is secure and reliable.

“It’s not normally the case where you have a tracked camera so far away from the rendering engines,” says Cordle. “We’re doing all the rendering here at the IBC.”

A return path allows the camera operator to see where the AR elements are placed in their shot.

“The camera guys are incredible and can bring creativity into compositions,” says Chris Brown, Ross Video, technical designer. “That is what I am most excited about. We’re creating a more flexible tool so we can be responsive to the needs of the producer and director.”

Shadows bring new realism to AR graphics on the NBC Olympics primetime set.

He notes that tools built into Voyager and the Unreal Engine allow lighting and shadows to reflect the location of the sun. The system can also simulate cloud cover and even the shadows cast by clouds passing overhead.

“It knows where the set is in terms of longitude, latitude, the day of the year,” Brown explains. “It knows exactly where the sun is in the sky in real life. When we’re lighting these elements, we can ensure that the shadows match up with the real shadows that [host] Mike Tirico is casting.”

The work gives the AR elements more realism, which also means they look more natural and are less jarring.

Cordle says the NBC design team aimed for a tasteful note with the AR graphics. “They’re believable, and they maintain the prestige of the Olympics as they’re tastefully done.”

Besides working with NBC on workflow, the Ross Video team also worked on improving the realism. Brown says the progress in the last three months took things like refractions to a new level.

“Three months ago,” he says, “we were kind of faking refraction. But, in the last month, we’ve been able to dial that in, and there is just a tiny bit of refraction like you would have with a real glass panel.”

Also vital to the development process was the use of a to-scale model of the set, which was brought into Voyager to allow the team to preview the graphics inside a virtual environment months before the actual set was built.

Says Brown, “We were able to figure out how we would eventually use these graphics before we even got here, which was enormously beneficial.”