World Series Power Outage: Teamwork, Quick Decision-Making Get Fox Back On-Air

When the twin-pack diesel power generator supplying juice to the Fox Sports World Series compound (as well as that of NHK and Televisa) failed during Game 1, it provided the kind of drama that no one working in sports-TV production wants.

What happened on Tuesday night definitely does not happen frequently, but it does happen and at major events, such as the 2006 World Cup and the 2012 Super Bowl. In each instance, it poses a mystery as to exactly what happened, gets many people in the industry (not just those involved) back to the drawing board to plan additional redundancies, and offers an opportunity for disparate entities within a TV compound to come together, work together, and quickly solve daunting problems.

“The team worked together, and everybody — from competitors to vendors — onsite was reaching out a helping hand to make things better and get it right,” says Mike Davies, VP, remote operations, Fox Sports. “At the top of the list is MLB International. They knew immediately what was happening and worked with our engineers to get us back up.”

CLICK HERE for SVG’s coverage of the MLB side of the World Series power outage.

Fox Sports headed into the World Series with the plan it had used during the League Championship Series games played in Kansas City. During the World Series, other entities roll in, including MLB Network and MLB International. The MLB entities brought in a separate Caterpillar generator (via CAT Entertainment Services), and that decision proved to be a difference-maker.

“In a Dark Truck’
“Obviously, there was the sheer terror of being in a dark truck, and the first thing you want to do is make sure nothing horrible is happening and everyone is safe,” says Davies. “Within seconds, we realized it was only our trucks that were affected and that MLBI appeared to be up.”

Helping out during the crisis was that Fox Sports and MLB International have worked side by side at major MLB events for nearly 20 years and are well-versed in each other’s workflows and production needs. The All-Star Game and the World Series provide a chance to make bonds that come in handy during a crisis.

The first task was to get the MLB world feed (produced and distributed by MLB Network and MLB International) to the Fox Broadcast Center in Los Angeles. The Fox team transferred the MLB world feed into the Vyvx circuit to Los Angeles, and, once Los Angeles had a signal back on-air, the production team focused on how to get the Fox announce team set up in the MLB world-feed announce booth.

“Our friends at MLB were fantastic,” says Davies. “All they wanted to do was help, and they made their resources and personnel available to us for the taking. It was an entire compound that came together in crisis, and I was pleased with how everyone responded.”

Getting the Trucks Back Up
Having a signal back on-air allowed the technical team to slow down a bit and become more methodical in tackling the challenge of getting power back to the three production trucks that constitute the NEP EN2 production unit and then powering it back up.

First up was getting the trucks connected to the MLB generator. Then came the big challenge: rebooting a production unit that housed hundreds of computers, all improperly powered down.

Fox Sports’ World Series compound in Kansas City

“Even with a desktop computer at home, those things are a concern,” says Davies. “Ninety-nine times out of 100, it will be fine, but then it will come up with an error. We were fortunate that everything came up somewhat happily and was networking properly. It was miraculous and a testament to how the trucks were built.”

With the facilities up, the next decision was figuring out how to coordinate the return to the Fox feed. Once that changeover was done, it was back to baseball and, arguably, one of the more exciting and most watched first games of a World Series in years.

Best-Laid Plans
As for the power plan, Davies notes, “We had parallel and redundant generators so that, if one goes down, the other one kicks in. But, for some reason, both engines were killed and powered down, and we still don’t know what caused it.”

Caterpillar provided the twin-pack generator and has even taken fuel and oil samples to the lab to see whether something wrong with those elements could have led to the failure. At this point, though, any thoughts on what happened are strictly conjecture.

Should the World Series return to Kansas City for Games Six and Seven, Fox Sports will add some diversity to the power plan. The main truck (NEP EN2) will run off the house power with a CAT Entertainment generator as backup, and the pregame truck (NCP 14) will be powered off a generator and have house power available as backup.

As the World Series shifts to the New York Mets’ home, Citi Field, the plan is to have the three Game Creek Dynasty trucks (which will handle game coverage) run off an uninterruptible power supply (UPS) unit and the Game Creek Freedom truck (handling pregame) powered from parallel generators. A third generator on hand in case the UPS needs it.

“We used a UPS for an entire month at the Women’s World Cup. It was a feat of necessity because we could not keep a generator running all month,” says Davies. “UPS is tempting to think about. You see more and more of them out there, and we are looking at a UPS array for the U.S. Open golf championship.”

He adds that the support during Game 1 came from well beyond those on the compound.

“Everybody who works in our position knows that nightmare scenario, and it has happened to everybody,” he says. “We knew we were definitely not alone as we had texts and e-mails of support coming in. You just need to make sure you are able to respond the right way, and our team worked so quickly to get us up and running.”