Canon Free Viewpoint Tech Used for ESPN+ Full NBA Game Production

Virtual camera system provides views from nearly any angle, three-second delay

ESPN last week rolled out a volumetric video version of NBA CourtView for an ESPN+ broadcast of a game between the Brooklyn Nets and Dallas Mavericks. The production team relied almost 100% on the Canon Free Viewpoint virtual camera system, which has been installed at the Barclays Center. The result was a viewer experience that was like watching NBA2K on steroids, with sweeping camera views from nearly any angle and with only a three-second delay from the happenings on the court.

The Canon Free Viewpoint system gives the production team ultimate freedom in terms of angles.

“We did the entire game as if it was a full six-camera production,” says Ed Placey, senior coordinating producer, ESPN. “We had six camera stations, with four of them designated for the live pictures and two of them had specialty opportunities with replays. We also had other traditional replay record devices at [ESPN facilities in] Bristol, [CT,] where we produced and directed it; the announcers were also in Bristol. We were dealing with the three seconds of latency, so the announcers needed to be downstream of the latency rather than courtside.”

The Canon Free Viewpoint system at the Barclays Center comprises more than 100 Canon Cinema EOS-based cameras and lenses. The video information they capture is processed and turned into data on a proprietary Canon image-processing box behind each camera. The data from all the cameras is sent to a server system that creates a point-cloud 3D model of the players and other objects on the court. A similar system was used to create highlights last month at the NBA All-Star Game in Cleveland.

“We’ve seen technology like this over five to eight years in different forms,” says Placey. “It’s not that it wasn’t great technology and something to keep an eye on, but it didn’t have the tools necessary to do much with it from a production standpoint that makes it interesting and engaging time and time again. It has to more than just a clip two or three times.”

Images are built using voxels, which are units of graphic information that define a point in three-dimensional space (basically, the 3D equivalent of a pixel) combined with RGB values. An operator with a joystick akin to a cabled camera system “flies” the virtual camera within the virtual environment and builds the clips in a process similar to rendering an animation.

Placey thinks of Canon Free Viewpoint as a virtual version of a cabled camera system. A physical cabled camera system in an arena historically has been difficult to install and operate: the pilot needs to avoid things like hanging scoreboards and backboards, and the device needs to be operated safely relative to the fans in the stands.

“The lengths to which directors work to make sure the Skycams don’t appear in their other shots is important,” he says. “In basketball and hockey, the only way they’ve been able to do that is by keeping it high and out of the way.”

The Canon Free Viewpoint system is operated via joystick control.

ESPN brought in drone pilots in to operate the system, which has a joystick that is similar to those of robotic and cabled camera systems. “We brought drone operators to [Barclays] for about four games so they could get comfortable with flying it,” Placey says. “Over the course of those four games, they got better and better.”

ESPN’s team went in eyes wide open, knowing that the system still has inherent imperfections  due to the amount of processing needed, he adds. “It’s impressive to this point, but it’s really only in its infancy. But this gives people a glimpse of the type of things you can do. If you can do it at a higher quality and higher resolution, there are real opportunities you could create as opposed to a clip here or there within a game.”

One potential? The ability to get the ultimate look at fouls and other events. Says Placey, “We could see what the referee saw, look over their shoulder, and see the player’s head snap back.”

The production team learned a lot from the first full game. he adds. Replay operations can be tightened up to provide a more optimal viewpoint, and the addition of preset flight paths that could be available at the push of a button would allow the system to take another step forward.

A virtual centerhung scoreboard allowed real video to be a part of the virtual volumetric experience.

Although most of the show existed in the virtual world, there was a way to get real video into the show. A virtual centerhung scoreboard allowed shots like cutaways of coaches or players on the bench, which were not rendered in the virtual environment.

Says Placey, “We had the ability to put our own cameras from the main production into the jumbotron so we could always get replays or cutaways.”