How Mixed and Augmented Reality Are Becoming Viable In-Venue Options

Baltimore Ravens, Carolina Panthers, and Houston Texans are producing MR activations

Trends in in-venue production constantly ebb and flow. In recent years, there was the arms race to see who could install the biggest videoboard. Then it was who could implement HDR and/or IP the quickest. Now mixed-reality (MR) activations are starting to appear in the National Football League, and both production teams and manufacturers expect them to become more prevalent in the future.

The Beginnings: Baltimore Raven Flies Around M&T Bank Stadium in 2019

The path toward mainstream use was paved two years ago. On Dec. 19, 2019, with the crowd packed tightly on a cold Thursday night in Baltimore, the Ravens were looking to lock up consecutive AFC North Division titles and their first back-to-back crowns since 2011-12. The tension was palpable for their game against the New York Jets, but, shortly before kickoff, fans were treated to something that few had seen inside a professional stadium. Thanks to the Ravens in-venue team, led by VP, Broadcasting and Gameday Productions, Jay O’Brien, and new technology from The Famous Group (TFG) and other partners, a mixed-reality raven flew above the spectators at M&T Bank Stadium.

The Baltimore Ravens first showcased their MR raven in 2019.

The activation caught the attention of millions on social media, television programs like SportsCenter With Scott Van Pelt and Pardon the Interruption, and around the league.

“We’re always thinking, ‘What can we do that’s never been done before that would be awesome for the in-venue fan?’ and ‘What can we do to have people talk about our game-day experience the next day?,'” says O’Brien. “Our team is trying to win the Super Bowl, but we want to be the best as well. We want our stadium to be the best place in all of sports to watch a sporting event.”

At the time, it was one of the only deployments of large-scale MR intended for an in-person audience, but it spurred the technology’s deployment in sports venues.

“Jay deserves a huge shoutout for being a pioneer in this,” says Andrew Isaacson, EVP, The Famous Group. “We’ve continued to up the ante but, at the same time, evolved the technology to do things that are riskier than what we did with the raven.”

Incremental Progress: Broadcasters Lay Framework for In-Venue MR

A few months later, The Famous Group technology was front and center during the pregame for Super Bowl LIV at Miami’s Hard Rock Stadium. In this case, the venue was used as a backdrop for the unveiling of the NFL 100 All-Time Team. People who caught this six-minute presentation witnessed a technology that had the momentum to be featured on one of the most-watched games on the professional-sports calendar.

The MR panther took center stage at Bank of America Stadium in NFL Weeks 1 and 2.

“The league came to us with a unique challenge,” said Isaacson in February 2020, “and we were extremely grateful for the opportunity. As premium-content creators and storytellers, we see mixed reality as the latest tool in our toolbox to create things that have never been seen before. The NFL 100 [project] took a tremendous amount of technical and creative work, but we could not have been happier with the results.”

Originally a tool driven by innovation, mixed reality became a tool of necessity at the onset of the COVID-19 pandemic in March 2020. Venues around the country began playing their games with empty crowds, and, for at-home fans watching the action from their couches, the sight of empty stands was a bit jarring. Piped-in crowd noise was the common solution, but Fox Sports added to the production of its nationally televised slate of MLB games. Teaming with SMT, Pixotope, and Silver Spoon, the broadcaster placed digital fans in the seats to replicate a “live” environment for the natural eye. It wasn’t a grand showing of augmented or mixed reality, but the overlay gradually pushed the technology forward to its current state.

“The idea was to augment the broadcast, not take away from it,” said Dan Pack, managing director/partner, Silver Spoon, in September 2020. “We wanted to show broadcasters that there is a lot of value in using AR.”

Multiple Deployments: Carolina Panthers Create a Home-Field Tradition

The Houston Texans are integrating sponsors into MR elements, including this fan-driven race presented by Kroger

Fast forward to earlier this month, and MR is grabbing the sports-video–production headlines once again. In the past, activations like the one in Baltimore or at Super Bowl LIV were one-time events. The Carolina Panthers’ activation, which uses Epic Games’ Unreal Engine to create an MR panther prowling around Bank of America Stadium, was deployed on Sept. 12 and 19, demonstrating the improvements in MR as well as the ability for a franchise to deploy it whenever it wants. There are still some challenges in the execution, which are being fine-tuned, but the Panthers’ production crew, led by Director/Executive Producer, Game Presentation and Production, Mike Bonner, is taking this technology and running with it.

In addition, simplifying the product is erasing logistical hassles experienced by the in-venue production staff and allowing it to use the technology in more scenarios. For example, Bonner and his crew has played the panther before kickoff, but it has become a prompt used before the defense gets onto the field and could potentially be used on third down to replace the traditional noise of a siren.

“We want this building to be known for its noise,” Bonner explains. “When our offense is on the field, we want this place to be a library, but, when our defense is playing, we want the fans to be loud.”

Other Uses: Texans Tap Technology for Real-Time Sponsorship Opportunities

In a stroke of coincidence, the Panthers recently squared off against another NFL team deploying mixed reality: the Houston Texans. Before the first whistle of the Thursday Night Football contest on Sept. 23, The Famous Group was once again involved in a mixed-reality display at NRG Stadium. Led by Houston Texans Executive Producer/Senior Director, Live Video Productions, Gavin Gehrt, the production team used its three-camera MR system to re-create a landscape that featured robotic bulls dodging a laser beam from above. The intro fired up the crowds during the game, and Gehrt and his crew are integrating sponsorship into the in-game elements. Most notably, on the same day Carolina debuted the panther, the Texans produced a fan-driven racing game sponsored by Kroger on a mixed-reality track.

“We love the ability that our mixed reality setup has to take traditional elements such as opens, races, shuffles, prompts, and on-field presentations, and make them seem like their own live event each week,” says Gehrt. “Being able to use the technology throughout your entire show is the key to success, and as a live producer, it’s been a ton of fun figuring out ways to incorporate [MR] into our show. We want to keep pushing the limits on where this can go in the future.”

The Ravens’ O’Brien believes that MR can help sponsor-centric content gain traction and appeal. The technique could also help sponsorship messaging become more congruent with other high-quality aspects of the overall show.

“Our in-venue show relies on new and creative unique moments,” he adds. “We have a talented team of animators, editors, and videographers to always keep content fresh. It’s only a matter of time before teams deliver those same ‘Did you see that?’ moments with partner executions, and I’m excited to see what opportunities are out there to generate revenue.”

What’s Next: MR Moves From Broadcast-Only to In-Venue Entertainment

The popularity of mixed reality in a live, in-game environment may have taken lots of baby steps into the limelight, but it’s there, and professional teams and venues will run with the concept for as long as possible. The days of one-time use may be behind us for many reasons, including justification of the technology and the myriad ways of using it to energize or captivate patrons on game day.

On Sept. 23, the Houston Texans debuted these robotic bulls before their game against the Panthers.

“The challenge for teams and even for companies like TFG is that the creative is as beautiful as it is time-consuming — and thus is a significant investment,” says O’Brien. “With only 10 home games, we’d ideally want the MR content to evolve each game, but development is a bit more complicated.”

Despite the creative challenges, opening up this door can only benefit the community by pushing others to adopt these workflows.

“We all want to be innovative,” adds Bonner. “I think the sky’s the limit. I hope that my colleagues, not only in football but in different sports, say, Hey, that’d be really cool if we could do this.”