Pac-12 Networks Continues To Blaze Remote-Production Trail With New Software-Defined-Production Model

Software-defined workflow leverages NDI and vMix, builds on ‘Multicam’ model

Upon launching in 2012, Pac-12 Networks instantly established itself as a pioneer in remote-production techniques. Its groundbreaking Multicam production model was a precursor of the remote-production revolution the industry has experienced in recent years, and the centralized-production model has allowed Pac-12 Networks to produce thousands of live events from its San Francisco headquarters for a fraction of the cost of traditional production. Now the conference-owned network is taking it to the next level with its new Software Defined Production (SDP) model.

“Pac-12 Networks has been exploring new, efficient ways to streamline the entire production package since its inception with the Multicam model [pioneered] by Leon Schweir [the late SVP, production and digital content, Pac-12 Networks],” says Don McKinney, senior director, production operations, Pac-12 Networks. “SDP is essentially the next iteration of this [concept] but streamlined even further with the idea of having just the transmission stack onsite and no truck. And it’s a truly software-defined approach because it is separated from control surfaces like switchers and audio boards.”

This SDP prototype room became Pac-12 Networks’ Production-Control Room X at Pac-12 Networks headquarters in San Francisco.

The SDP ecosystem is built on vMix live-production software and tied together via extensive NDI IP connectivity. The model is already in full operation, and Pac-12 Networks is on pace for 225 productions this year and looking to expand to 400+ productions next season.

Early Development: Searching for Multicam 2.0

Although COVID drastically accelerated its development over the past two years, SDP was conceived prior to the pandemic. McKinney and Senior Director, Remote Operations, Sarah Backerman were brainstorming on how to streamline the Multicam model by eliminating the production truck and finding another solution using Internet2 (I2) connectivity (which links more than 140 Pac-12 venues across the country) and campus resources. They began searching for a production stack that would allow multiple cameras and mics on campuses to be connected to a remote-control room without a truck onsite.

“Essentially, we were looking for Multicam 2.0,” McKinney explains. “We needed a way to reduce the cost point, not only for our own productions but also for our universities’ productions. We went out to look for a system that would accomplish all of our goals. Unfortunately, it didn’t exist.

David Hasselfeld is Pac-12 Networks’ first SDP operator.

“We evaluated a variety of software platforms.” he continues, “but none of them could do exactly what we wanted end to end. And most of them ended up costing more than our existing Multicams anyway. So the only solution for us was to find a several individual products and build it ourselves.”

Despite initial development hurdles, the idea of a Software Defined Production model that was separated from control surfaces like switchers and audio boards promised a powerful and flexible approach. So McKinney persevered, and, eventually, his team discovered vMix, a live-production solution offering an open structure that allowed unlimited customization at a low price point. The vMix-based model was presented to senior leadership, and the project was approved for testing.

With vMix in place, McKinney and company began building a transmission stack that was portable, easily moved, feature-rich, stable, and flexible for multiple sports. The NDI video-over-IP transport protocol played an integral role in development of this stack and the overall SDP ecosystem.

Located at Pac 12 Networks HQ, Jackson Haselnus handles live production of a University of Utah volleyball game.

“NDI is not something that is necessarily used very much in broadcast, but [SDP] would not have been possible without it,” says McKinney. “It eliminated a lot of the baseband-video issues, and it was very complementary to the software platforms that we’re using and the I2 network we use to transfer everything around. I think one of the big hurdles in the initial phases was that NDI is focused on local networks and we needed to trick it into being a network running over much larger distances. That was a lot of our early experimentation.”

With the SDP model still in its infancy, soccer was selected as the template for developing the system initially, and a three-game test series (later expanded to 14 games) was marked on the calendar for early 2020.

Pandemic Effect: SDP Model Gets Fast-Tracked

However, in the midst of testing, the pandemic arrived, and what was previously an evolutionary project defined to succeed the Multicam model suddenly became a top priority since the SDP model could dramatically reduce the number of people needed both onsite and in San Francisco for live game productions. SDP development was fast-tracked, and, following an initial demo on April 1, 2020, McKinney’s team completed its first soccer demo on April 22 and its first full live-game test on Aug. 4.

Pac-12 Network’s development of SDP was accelerated by the COVID-19 pandemic.

What was initially a small series of tests was quickly expanded to a goal of 250+ live game productions with plans for a 2021 Q1 debut. To get it all done in time, Pac-12 Networks had to work quickly to set up the necessary equipment and assets at all Pac-12 universities — constructing three control rooms, assembling all 12 schools’ remote kits, and training all operators in just 64 days.

“We began training operators like crazy to get them up to speed on this new system,” recalls McKinney, “while we cleared out an entire studio to have space to build the remote kits for 12 institutions and two spares.”

Using SDP, David Saunders at Pac-12 Network headquarters handles a water polo match from USC.

The first non-televised SDP event took place on Feb. 12; the first televised event, on March 14. However, SDP’s true coming-out party wasn’t until May 9 for the Pac-12 Lacrosse Championships at Stanford. The initial show preset allowed four cameras, two channels of graphics with scorebug, full replay ability for all cameras and program, an onsite talent booth for color and play-by-play commentators, and at least two nat mics, comms, talkback, tally, and a talent-program return.

“The Pac-12 Lacrosse Championships were a complete success and really vindicated all of the work and preparation we had done on a system we believed in,” says McKinney.

Continued Evolution: More Sports, More Features, More Tech Firepower

Since then, the SDP model has been dramatically expanded and is now able to support eight cameras with full replay for lacrosse, soccer, water polo, field hockey, wrestling, baseball, and softball. An enhanced kit in development for men’s and women’s basketball will debut next season.

Pac-12 Networks assembled and shipped SDP remote-production kits to the conference’s universities.

In addition, Pac-12 Networks is working on a school-based version of the system. The universities would be able to produce their own shows at a lower cost but at a higher production-quality level, making them more suitable for broadcast on Pac-12 Networks’ linear, on-demand, streaming, and free ad-supported television (FAST) platforms.

Pac-12 Networks also envisions three distinct versions of SDP productions: an on-premises one based in San Francisco, a completely on-campus version produced from the event venue, and a cloud-based version leveraging AWS. The conference’s four California-based schools are using the on-premises version based in San Francisco, while campuses farther from Pac-12 Networks HQ are using the on-campus version to avoid latency issues. In the on-campus scenario, all the NDI sources go directly into the transmission kit at the venue, and Pac-12 Networks can control the box via Parsec from San Francisco.

“It’s the same [SDP kit], no matter how you use it,” McKinney explains. “Depending on where you locate it, that gives you a lot of different advantages. But, even when the box is on campus for each one of these locations, the control rooms are always in San Francisco.”

Mike Fitzgerald at Pac-12’s San Francisco headquarters handles a UCLA volleyball game via SDP.

As for the cloud-based version, McKinney’s team is currently in the midst of further tests, and he expects the AWS-powered scenario to be available for both Pac-12 Networks-produced shows and university-produced shows by next season.

“We believe that this will give us a powerful platform to be able to provide the universities with the same high level of capabilities in terms of replay, graphics, animations, and tools,” he says. “All the things we consider important to our shows [at Pac-12 Networks] will now be available to them too in ways that were not available with their legacy equipment.”

Here To Stay: SDP Model Will Work in Conjunction With Multicam, Not Replace It

Although the SDP model represents a dramatic leap forward in production efficiency, McKinney says, the existing Multicam-production model (which features a small truck onsite to send the feeds) isn’t going anywhere. The SDP model will function alongside Multicam — rather than replace it — allowing Pac-12 Networks to produce more events at a higher production quality than ever before.

Jeff Backerman produces Stanford University volleyball from Pac-12 Networks HQ.

“This new system allows us to radically increase our production volumes because the SDP production model uses different resources than our Multicam system,” he says. “They do not compete with each other, allowing us to do much more with the same Internet 2 bandwidth. In addition, we have been able to add in nearly every element in use in our Multicam productions to the point that the viewer — and even people in our Production Operations department — can’t tell which productions are Multicams and which are SDP/vMix productions.”

After more than two years of development, SDP is now live and in action on an almost daily basis as part of Pac-12’s operation. McKinney attributes the project’s success to the hard work of the production, ops, and engineering teams.

“This project has been overwhelmingly successful,” he says, adding that it “is the basis of much of our production planning going forward, allowing us to cover more sports while maintaining the exact same quality at half the cost.

“Building all this in the middle of COVID was unbelievably challenging,” he continues. “There were so many things that could have stopped the project in its tracks, but, thanks to the amazingly smart people here and their hard work, we were able to overcome all that and produce something that I think is unique.”