Super Bowl LVI: Ross Video’s Rocket Surgery Takes AR Graphics to the Next Level for NBC Sports, Van Wagner

The creative-services supplier was also involved in the broadcaster’s Olympics coverage

In the seven years since Ross Video’s Rocket Surgery Creative & Professional Services Division began working on NBC’s Sunday Night Football package, the broadcaster’s use of augmented-reality graphics has grown significantly — in terms of both frequency in the broadcast and quality of the graphics. Last month at Super Bowl LVI, Rocket Surgery cranked up its effort yet another notch, providing AR elements not only for NBC’s broadcast but also for the in-venue show produced by Van Wagner Sport & Entertainment.

Rocket Surgery provided all AR elements for NBC’s broadcast of Super Bowl LVI.

Given all the irons in the fire, this is without a doubt the biggest [show we’ve ever done],” says Rocket Surgery Director, Creative Services, Jim Doyle. “Not only did we provide all the augmented-reality graphics for the NBC show itself — like we always do — but we also debuted a new look for the show and, for the first time, we actually led the creative [process] on this one. Plus, we were also doing the AR and PIERO graphics for the in-venue show with Van Wagner. So there were a lot of different areas where we played a role.”

At SoFi: Big Onsite Presence for Rocket Surgery’s Biggest Show Ever

With Rocket Surgery playing so many roles for the big game, the team opted to roll in a mobile unit (NEP Dakota) at SoFi Stadium to serve as its operations hub. Normally, Rocket Surgery is housed in NBC’s SNF trucks, but, with 14 people onsite (plus three designers/technical artists located remotely) simultaneously producing content for NBC and Van Wagner — as well as for NBC’s Olympics coverage —a dedicated mobile unit was brought in.

“There was a need for us to have all of our people in one place to make all that happen, as opposed to being spread out all over the facility as we typically are,” Doyle explains. “That was not only for communication purposes but also to make setup and teardown easier.”

Dakota housed NBC’s pregame show, Van Wagner operators, and designer stations, while Rocket Surgery’s normal space in NBC’s NEP ND1 D unit was home to the primary in-game AR operations.

The Super Bowl LVI pregame show featured AR graphics that used a Technocrane on the Lakeside set outside SoFi Stadium.

Rocket Surgery relied on Ross Video’s Voyager virtual-production platform powered by Unreal Engine — along with some Ross Video XPression graphics — to drive the AR graphics for Super Bowl LVI. In addition, Ross Video gear/software used for the production comprised Lucid, PIERO, OpenGear, Ultrix, Carbonite Ultra, Gator, DashBoard, Furio VR100, TouchDrive, and Ultritouch.

As is the case on all SNF broadcasts, AR graphics were available on both goalpost cameras (13 and 14). However, instead of using the low Skycam (Camera 40), Rocket Surgery deployed AR on the high Skycam (Camera 30) using Trackmen VioTrack technology. In addition, Rocket Surgery worked with Ncam, Stype, and AVS to implement AR tracking on the on-field Steadicam wirelessly via RF.

A full arsenal of Ross Video gear was on-hand in L.A. to power Rocket Surgery’s AR graphics operation for Super Bowl LVI.

“We didn’t radically change the workflow because we wanted to make sure it was reliable and still comfortable for the director and the production team,” says Daily. “But we did add a lot to that more cameras and more tracking. All the [additional cameras] were added into our normal workflow, and we still triggered everything the same way we do [typically] with DashBoard. It was just a lot more in terms of volume.”

Taking the Creative Reins: Rocket Surgery Spearheads the Process

Prior to Super Bowl LVI, the creative process for Sunday Night Football AR graphics was led by NBC Sports Art Director John Schleef and his team. However, with the NBC graphics team’s docket crammed with the concurrent Super Bowl and 2022 Beijing Olympics productions, the Rocket Surgery team was enlisted to lead the design process for the Super Bowl LVI AR-graphics package.

https://www.youtube.com/watch?v=pFd5CtyGnjc

“[Previously,] John and teams would give us the building blocks and the main creative direction,” says Terry Daily, creative manager, Rocket Surgery Virtual Creative Services. “We would implement and optimize that for playback in our engine. This time was really the first time we transitioned to complete design-from-scratch service.

Daily and his creative team pitched all the concepts to Schleef from the beginning and received approval, but the Rocket Surgery team handled the design and construction of all elements. Schleef and company provided early looks at the insert package they were designing for Super Bowl LVI, and the Rocket Surgery team drew its initial inspiration from that.

“Ultimately,” says Daily, “John [Schleef] oversaw the main creative vision and direction, but it was unique for us this time around in that we actually created the looks in-house completely with our own design team. We got to dream up how we wanted these graphics to look and how we were going to execute shots, especially in terms of how we could play off the stadium design with the Infinity Screen. We let that dictate how we designed.”

Besides Super Bowl LVI, Rocket Surgery was also providing AR graphics for NBC’s Olympics coverage. Announcer Mike Tirico at the L.A. Lakeside set also did double duty.

SoFi Stadium is heavy on Ross Video technology, including the state-of-the-art Infinity Screen, which is driven by Ross Video’s Xpression Tessera platform. Daily says this provided his team with a distinct edge in integrating the AR package into the real-world setting at SoFi.

“That gave us a major advantage because we could understand how things could look on camera,” he explains. “One of the key things we did was [previsualize] very meticulously with the same camera lenses we intended to shoot with. We had the reference geometry of the space since we knew the stadium and its architecture so well. Since we had the reference geometry of the screen, we could place the concepts of the artwork in space ahead of time.

Ross Video’s Rocket Surgery rolled out a dedicated NEP mobile unit at SoFi Stadium to serve as its home base for Super Bowl LVI.

“That way,” Daily continues, “we could see exactly what it would look like on the Steadicam, Skycam, and other [cameras]. It was a lot less of a guessing game or a shoot-from-the-hip approach. It was much more focused and targeted so we would know exactly how something was going to work when we saw it in that space.”

Beyond the Game: AR Graphics Play Key Role in Pregame, Olympics Coverage

On the Super Bowl pregame show, Rocket Surgery worked with NBC to deploy a 22-ft. telescoping TechnoCrane with built-in Stype encoder kit to enable live AR elements on the broadcaster’s primary set.

“During the pregame show, [NBC Sports Super Bowl LVI Pre-Game Show director Pierre Moossa and producer Tommy Roy] used some really cool stuff using that [TechnoCrane],” says Daily. “We did a shot with real-time water simulation where we covered the lake with simulated fluid in Unreal Engine and brought an AR object out of the water. It interacted with the rippling water and came right out of the lake. That was definitely one of the more exciting [graphics] we’ve ever gotten on-air.

The Steadicam on the field was equipped with a tracking system to enable AR graphics throughout the broadcast.

“We also did a really fun effect for the sponsors,” he continues. “We had an animated camera within a virtual-set world that was all real time. And then, as the animation ended at the end of the tunnel, it blended into the [real-world] tracked-crane camera. It blended the two worlds [of virtual and real world]. It was a subtle effect but ended up being one of the coolest things we did all weekend that made air.”

On top of all its efforts at the Super Bowl in L.A., Rocket Surgery was also providing AR graphics for NBC’s Olympics coverage via dedicated engines using the same cameras as the pregame show for primetime and primetime+ programming from Sofi throughout Super Bowl weekend.

Throughout the Olympics, Rocket Surgery was also using Stype-configured cameras located at the IBC studio in Beijing and transporting video and tracking data to NBC’s Sports Production Operations Center (SPOC) in Stamford, CT, where the render engines and operators were located. NBC was also using two Stype-configured cameras powered by Rocket Surgery for AR in Studio 1 in Stamford.

Looking Back: ‘Like Night and Day’ Compared With Seven Years Ago

In the seven years since Ross/Rocket Surgery began working with NBC Sports on AR graphics, the technology and workflows have evolved at lightning-fast speed. With the company’s biggest show to date under its belt, Doyle takes a moment to reflect on just how far Rocket Surgery has come.

Ross Video’s Rocket Surgery had more than a dozen crew members onsite for the Super Bowl LVI production.

“When we started working with NBC seven years ago,” he says, “we were doing all this on XPression and working within the XPression UX environment. Compared with what we’re able to accomplish today using Voyager on Lucid, and DashBoard, it’s like night and day. The realism and the quality of the graphics are far and above what we were able to produce previously.

“It’s also now possible for us to get more [camera] tallies than ever before,” Daily continues, “because we can be far more responsive in the live environment than we were seven or eight years ago. Instead of getting three or four graphics on the air, we’ve had as many as 18 get on the air this season. It has made a big difference in our ability to satisfy the client and get more graphics on the air.”