Live From the Rugby World Cup: Canon Free Viewpoint System Offers Near Real-Time Volumetric Capture

Virtual camera flies through the action to create 1080p replays from any angle

The Rugby World Cup is about to enter the most exciting portion of the competition, and the remaining matches at Yokohama Stadium will feature what has been the most exciting broadcast innovation at this year’s event: the Canon Free Viewpoint system. The volumetric capture system synthesizes image data from 125 cameras located around the stadium in near real time and allows an operator to fly a virtual camera through the action on the field and create 1080p replays from any angle.

The Canon Free Viewpoint system deploys 125 Canon cameras and lenses to capture volumetric video at the Rugby World Cup.

Tsuyoshi Wakazono, manager, Image Solutions Business Operations, Canon SV Business Planning Department, and Atsushi Date, senior principal architect, Canon, discussed the technology with SVG early in the tournament.

“We have received a lot of good feedback, and it makes us very proud and happy that it is going well,” says Wakazono. To see the clips that have been created, click here.

The Canon team has been hard at work managing the system, for which 125 specially modified Canon 4K Cinema EOS cameras and lenses operating in 60 fps mode have been mounted on 60 columns around the stadium, a process that took approximately three days. The cameras have proprietary sensors customized for the application, and the data is transferred to proprietary video servers in a small cabin located outside of the stadium.

A team of 10 people are located in the cabin, with five overseeing the computer-control system. System software extracts balls, players, and other objects of interest from the scene, and 3D computer-graphics software processes them from multiple angles. A pilot uses controls similar to those of a cable camera system to virtually fly within the environment, and another team of five people oversee the image processing of the desired replay.

“We believe we can get the number of people who can control the system down to two or three people,” notes Date.

The process occurs in near real time, a major improvement over a system that, only two years ago, would have taken more than a day. The team is currently creating a minimum of six clips for each match, making them available to rightsholders, who can integrate them into their postmatch analysis.

“We are working on a real-time system and hope to make it available in the near future,” says Wakazono.

Depending on the sport being covered and the size of the venue, fewer cameras can be used, which would increase processing speed (likewise, adding more cameras or higher-resolution cameras may improve the replay quality, but the tradeoff would be longer processing time).

Deployment of the system at the Rugby World Cup is an exciting real-world application of a volumetric capture system that Canon hopes to commercialize in 2021. Permanent installations are a goal, but the system has also been optimized for quick setup for temporary events. For example, the core equipment is located in two broadcast vans, which can roll up to a stadium and not affect control-room operations inside the venue. Date says the team also continues to improve resolution, processing speed, and quality.

Adds Wakazono, “We are very happy with the results but want to find a good partner in the future who sees the important of real-time instant replay and can help us reach that goal.”

Discussions are currently ongoing with potential customers of the technology, and there are also potential applications beyond broadcast replays. Postmatch analysis for coaches and players, for example, could take on a completely new level of usefulness, and the technology also could be used to help with VAR and other video-assisted replay systems because it can offer a definitive look at whether a ball crossed a line.