Massive Effort Gets Nashville’s Bridgestone Arena Quickly Back in Operation After Flood

Broadcast infrastructure faces a year’s worth of restoration after water-main break

Black Friday lived up to its moniker this year for the NHL’s Nashville Predators. About 5 a.m. on Nov. 25, an estimated 2 million-plus gallons of water poured into the production and event levels of the team’s Bridgestone Arena home, the result of a water-main break. The disaster inundated the venue’s broadcast-control room and TOC spaces, damaging broadcast and live AV systems and equipment.

One of the main points of water intrusion was directly above the DiGiCo S31 console used to mix broadcast audio and live sound. Water sat 3 ft. deep in some spots and caused some melt damage to the ice.

The arena had already experienced the effects of a 100-year catastrophic flood in Nashville: a 19-in. rainfall over two days in May 2010 brought the Cumberland River literally inside.

“My chief engineer, Stephen Hart, was on his way in when [the main] broke,” recalls Jacob Lutz, director, technical operations, Nashville Predators. “He called me to say we had 3-6 ft. of standing water in the control room and operations center. I knew we were in trouble.”

The day after Thanksgiving became a mad scramble to save what broadcast- and event-production components could be salvaged and immediately returned to use and to find substitutes and alternatives for everything that couldn’t be saved at a venue that Pollstar described as the busiest arena in the world. Fortunately, Nashville’s experience with dealing efficiently with disasters from floods to tornados and the fact that it takes the Tennessee state motto — The Volunteer State — seriously gave the Preds more than a fighting chance.

Nashville Predators’ Jacob Lutz: “My chief engineer called me to say we had 3-6 ft. of standing water in the control room and operations center. I knew we were in trouble after that.”

An exhaustive and collaborative effort by arena staff, more than 30 public-works agencies, and vendors and other private companies brought 300 people to the arena within hours of the break. As a result, only two games, against Colorado and Columbus on that Friday and Saturday, respectively, were postponed, and the crew was ready for the Tuesday, Nov. 29 match with the Anaheim Ducks.

“It would have been easy to shut the building down for 20, 30, 40, 50 days,” Nashville Predators CEO Sean Henry told the Tennessean. “Relocate the games; work a deal with another city; temporarily cancel or postpone concerts. But we sat together on Friday night, Saturday, and Sunday and said, ‘How do we host all those events?’ because it’s an important thing that we do.”

Much of the immediate restoration work was brute force. For instance, more than 1,000 box fans were brought into the building to vent the moisture, along with temporary duct work put into place. Meanwhile, Lutz’s teams had begun to protect what it could from additional water damage, such as sourcing tarps and covering the Forecast workstations that make up many of the work surfaces in the control room and TOC.

“It’s a hockey arena. We don’t have a lot of tarps and things lying around,” says Lutz. “We’re just grabbing whatever we could grasp. We’re going with the mindset that we need to save everything we can because this tech center is probably going down.”

Virtual Cavalry Arrives Remotely

It may have been Thanksgiving in the U.S. but not in Canada, where Evertz, one of the Predators’ main technology vendors, is located. According to Lutz, Evertz assembled a reaction team that remoted in within 45 minutes, assessing damage to routers and other platforms and confirming data backups, the latter especially important as the broadcast teams transitioned to external mobile units for production, enabling the team to quickly upload router and other settings.

“It was important for us,” he says, “to have companies like Evertz, Ross, Anthony James Partners, our integrator CTI [the last two having designed and built, respectively, the venue’s control rooms], and others able to pick up the phone at that moment and help us out, to be able to get in remotely while the equipment was on because we didn’t have any time to waste.” Good backup practices, he notes, helped immeasurably in getting production back on track as quickly as it did.

The situation was not completely dire. The venue still had a working video system, and the backup systems for OES scoreboard management and shot clock were undamaged.

With the control room and TOC out of commission, production migrated to a production truck. Live Mobile Group (LMG) sent over locally based TNDV Elevation to handle the first two games post flood. Elevation served as the production-control room for the facility and served as the distribution point for feeds, per NHL requirements.

Thanks to the work of arena staff, public agencies, and vendors, Nashville’s Bridgestone Arena postponed only two events after a water-main break flooded the production-control room and TOC.

The venue’s Ross Xpression graphics system was ported over to TNDV Elevation, and the show files were loaded into Elevation’s onboard platforms. Working directly with Ross and Anthony James Partners, the Predators were able to get the existing Ross Tessera LED system operable for the first game. Although the system was unstable, having a team of engineers on standby and onsite, with multiple redundant backups in place throughout the entire signal flow, allowed the use of all LEDs for clock, score, and in-bowl advertising.

It was a good short-term fix. By Tuesday evening, the crew was ready to produce a game.

“We didn’t have the full complement of graphics and resources that we’re used to,” notes Lutz, “but, outside of lip sync and a few other things, the game went really, really well. [The production] was compelling, exciting, enticing — and we won the game that night! A win makes everything better.”

Sound Issues

In terms of audio, the damaged DiGiCo S31 used for routing has been already replaced with an S21 desk provided by Nashville’s Spectrum Sound, which had installed the venue’s PA system. The audio is interfacing with the production truck over a combination of an Optocore network, Dante, MADI, and analog. The venue’s ice-effects and referee microphones remain in place. These and other audio sources, such as announcers, are routed through the S21 in the truck and back to the SD12 front-of-house console in the venue.

“We’re able to provide certain feeds to Bally Sports in the truck, such as intercoms and other transmission feeds,” Lutz explains. “In return, Bally helps us out by distributing NHL and camera feeds back through the temporary truck because we don’t have the usual number of sources, such as cameras. Much of what we’re able to do is because of a collaboration with Bally and the engineers of the Mobile TV Group.”

Immediate Future

At the same time, LMG and the Predators are working to retrofit Lyon-17 from Lyon Video, a Live Media Group Holdings Company, to better fit the short-term needs of the Predators, starting with their game on Dec. 13. For instance, a Carbonite switcher similar to the Ross Acuity damaged by water has been installed in Lyon-17, allowing the crew to restart its previous workflow. The Riedel intercom system has been similarly replicated.

“While it’s still not the full technology complement we enjoyed in the actual control room,” Lutz explains, “LMG was willing to assist with retrofitting Lyon-17 and its technology complement with something that is more consistent with the size and specs of what was in the Bridgestone Arena. It will allow the show to go on without losing substantial capabilities and minimize the learning curve for our staff as the onsite control room and TOC are rebuilt. Once Lyon’s truck is in place long-term with the retrofits, the show will be modified to fit Lyon-17.

“One of the major hurdles to overcome was resolutions,” he continues. “Bridgestone Arena is native-1080p HDR. It was not possible to get a truck that had those capabilities on such short notice. Because of this, a substantial amount of frame syncs were utilized to get signals converted and locked for the facility.”

While the control room and TOC are being rebuilt, the Predators Tech Ops staff is working to build a temporary TOC using rental, loaned, and replacement equipment in a less affected part of the building, using temporary fiber to connect to the truck and providing LED-processing support for the facility.

“The Tech Ops staff’s main goal is to keep the same fun, exciting, and exhilarating fan experience that Smashville is famous for,” says Lutz. “We feel this will allow us to do that long-term during the rebuild.

Long Way To Go

The cause of the leak remains unknown, but Nashville is no stranger to them. According to the city’s Metro Water Services agency, such breaks are common along the 3,000 miles of pipe and more than 200,000 connections throughout the rapidly expanding city. Also, the area experienced a drought in November, which can cause ground shifts that can lead to main breaks and may have played a role in the break.

The team and the venue have a long road ahead. Full restoration is estimated to take six to 12 months, with ongoing supply-chain delays expected to continue to make sourcing some equipment a challenge.

“There was essentially a lake underneath all the equipment in the racks,” Lutz points out. “You have the fans pull in air from the front and shoot out warm air from the back. With 100% humidity and moisture in the room, it’s kind of scary because you don’t know what equipment did or didn’t get moisture in it. It’s going to be a while before we know what fails and what doesn’t. That’s where we are now; we’ve got to come up with redundant solutions.”

The intermediate plan, he says, is to build a temporary TOC in the venue across the hall from the damaged one, deploying more than 200 strands of fiber and dropping temporary power into that space. The Mitsubishi NovaStar Tech processors that drive the control-room operating systems are being replaced by manufacturer’s technicians. Once all the rescued and replaced equipment is moved to the temporary TOC, the work on rebuilding the original will begin in earnest.

Until then, he says, “it’s a weird world of rentals and procurement. The control room will go through an RFP process and insurance, but it’s up to us to be good stewards of what we have and what we’re able to rebuild. Thanks to my staff and our game-presentation staff and their quick thinking and ability to get tarps out there,” he adds, “we have still a decent amount to work with.”

One other circumstance has worked in the venue’s favor: after the Nov. 29 game, the Predators didn’t have another home game scheduled until Dec. 10, against Ottawa. And the recovery was far enough along by Dec. 2 to allow a scheduled Maren Morris concert to take place.

SVG will be providing updates as the restoration work on the Bridgestone Arena’s systems progresses throughout 2023.