Pac-12 Networks’ New Production Headquarters Is a Significant Step Forward in SMPTE ST 2110

The facility in San Ramon, CA, is a REMI workhorse featuring nine control rooms

Flexibility. Investing in the future. The ability to quickly pivot in an always evolving media landscape.

These are the bullet points regularly cited to justify investment in broadcast-technology that may be generally seen to be on the bleeding — or, at least, cutting — edge. Betting on yourself in a time of uncertainty. But, really, when is anything ever certain?

Nowhere is that fact truer in sports today than at Pac-12 Networks. Consistently one of the more innovative media organizations in terms of broadcast technology, the collegiate-conference–owned broadcaster recently opened a brand-new production facility in San Ramon, CA (about 15 miles west of Oakland).

Boasting nine control rooms, large bullpen areas for graphics and replay, a full studio, and much more, the facility is the new home of a broadcaster that produces well north of 800 live game broadcasts per year. It also happens to be significant in the maturation of IP and the SMPTE ST 2110 IP standard.

In August, Pac-12 Networks opened a brand-new production facility San Ramon, CA. Its facility in downtown San Francisco had served as its headquarters for its first 11 years. (Photo: Pac-12 Networks)

After 11 years in the San Francisco facility where it was born, Pac-12 Networks has moved to this brand-new home, providing a valuable asset and laying a critical broadcast foundation for a league with, to put it simply, an uncertain future.

“This was, honestly, an amazing chance to reflect on what we had built in San Francisco many years ago with great success,” says Ryan Currier, SVP, engineering and products, Pac-12 Networks. “Even though the core of that facility was 10+ years old at that point, we were still doing a lot of innovation. This, though, was the chance to start fresh in terms of the underlying fundamental technology and technical stack.”

The REMI Pioneers Soldier On

Since its first days in 2012, Pac-12 Networks has been a leader in at-home (or REMI) production, handling larger REMI-style shows than anyone in the U.S. then or since. This academic season, the broadcaster will produce live coverage of more than 850 events as well as a robust lineup of live studio programming, and nearly all of it will run through the San Ramon facility as REMI shows.

The new building is loaded, highlighted by nine control rooms — four of them full production-control rooms for REMI/at-home game production built around Grass Valley K-Frame XP production switchers. Each of the four control rooms, as well as a studio-control room, has an accompanying audio room with a Calrec audio console at the center.

“We’ve always felt that we’ve been a pioneer of the REMI style of production,” says Larry Meyers, EVP, content, Pac-12 Networks. “We wanted to take that even a step further. Even when we do remote studio programming, it’s a REMI-style production: camera and audio [feeds] are brought into the facility in an encoded fashion, and we [produce] everything from the facility. We think we’ve taken that a step further with this facility by integrating into an IP environment.”

San Ramon also has two large bullpen areas: one housing 26 Ross Xpression graphics stations; the other, 14 Evertz DreamCatcher replay stations. There’s also a highlights center built around a Dalet asset-management system.

Pac-12 Networks’ new production facility is built to the IP standards SMPTE ST-2110. It has a single studio space for anchoring linear television coverage. (Photo: Advanced Systems Group)

Additionally, there are four software-defined (vMix) control rooms for smaller REMI/at-home game productions. The game broadcasts can be produced by as few as two operators. According to Meyers, one person at the front of the room serves as producer, director, replay operator, audio mixer, and technical director, and another person behind them works graphics. Onsite at this level of production, there are typically two or three camera operators, the game announcers, and a small contingent of support personnel.

Pac-12 Networks’ new facility features two large bullpen areas dedicated to graphics (Ross Xpression) and replay (Evertz Dreamcatcher) for REMI live game production. (Photo: Advanced Systems Group)

“We think we have the lowest-latency REMI-style production facility anywhere,” says Meyers. “The fact that we’re still using legacy J2K encoding and decoding equipment and integrating it into an ST 2110 environment is challenging. It’s easier to build a facility from scratch when you’re not integrating legacy equipment or legacy signal types into it, not trying to put the old and the new together. That said, we did do that. At the same time, if we find ourselves in the fall with a requirement to deliver in 1080p or 1080p HDR, that might require changing our encoding equipment. But it would lead us to make choices that would allow us to stay in this super-low-latency environment where we can have announcers onsite and everything works — whether that’s tally lights, comms systems, IFBs, talkback, or announcers’ being able to look at replays.”

A Full Commitment to IP and SMPTE ST 2110

All these tools, toys, and spaces are built on and linked by a cutting-edge SMPTE ST 2110 infrastructure, an Arista IP switching core, an Imagine Communications routing and edge-processing core, and 2,088 fiber cables.

With this set of SMPTE IP standards still in relative infancy, it was a bold move for Pac-12 Networks.

“We were looking for flexibility for our productions,” explains Hieu Ho, lead broadcast technology engineer, Pac-12 Networks. “I think 2110 is getting mature, and it has proven to be ready for our use case.”

WATCH: Pac-12 Networks Lead Broadcast Technology Engineer Hieu Ho spoke about the network’s commitment to building an all-IP facility at the SVG Summit in December during an Imagine Communications–sponsored case study titled “The Fast Track to ST 2110 with Pac-12 Networks.”

Says Currier, “I think the timing of it was incredible because we had the benefit of knowing exactly what we needed to build. We knew what the scale of our operation needed to be. SMPTE ST 2110 was at a point where we were on the leading edge but not the bleeding edge. It was out; it had been implemented by a bunch of vendors. Other facilities with other folks we knew in the industry had implemented it. That gave us enough confidence, but we also knew it wasn’t so mature in terms of some of the technologies that we knew we needed to bring to bear.”

Meyers laughs, “Did we have trepidation as we were starting to turn everything on? Sure. ST 2110 is still young enough and new enough that you never really know what’s going to happen when you turn it on.”

Ultimately, working closely with vendors proved vital. John Mailhot, CTO, networking and infrastructure, Imagine Communications, worked with Pac-12 Networks on the facility and also happens to be a SMPTE Fellow who helped edit the 2110 documents. As a standard, he believes, it’s a finished work.

“I feel like we’re in the position now,” Mailhot said in a case study presented at the 2023 SVG Summit, “that most new projects of any significant scale or size can look to the 2110 specs and the many products in the industry that support those specs and confidently put a system together.”

It also simply came down to timing for Pac-12 Networks. With much uncertainty in the conference’s future, flexibility seemed essential for a building that might have very different demands in the near future.

“If we were to build it to 1080, it would be obsolete fast,” says Meyers. “We built it knowing that we were going to need to be flexible in terms of what the future was going to bring us in delivery requirements by distributors. We wanted to be able to meet the needs of potential new distributors that we didn’t even have identified at the time.”

On Jan. 21, Stanford University’s Tara VanDerveer became the winningest basketball coach in NCAA history. That game aired on Pac-12 Networks and was produced from one of nine control rooms designed for live REMI-style production at the San Ramon facility. Front row, from left: Associate Director Christine Assily, Producer Zohra Ziani, and Director Lori Brooks. Back row, near to far: Senior Graphics Operator Keith Campbell, Graphics Coordinator David Saldana-Montgomery, and Scorebug Operator Jacob Violante. (Photo: Pac-12 Networks)

It was also a steadfast commitment to the 2110 backbone that ultimately saw the effort through. That and the help of Advanced Systems Group (ASG), which served as systems integrator on the project.

“The client was very committed to an ST 2110 infrastructure, and, quite frankly, very few clients are today,” says ASG President Dave Van Hoy. “It is a real cost-benefit decision for a client. They had absolute clarity and knew what their potential benefits were. Most important, they had a network team that was 100% aligned with their broadcast-engineering team. That’s what made it possible.”

Smiles Currier, “It required a little bit of blind faith but also a lot of confidence in the team. I think how the facility has been functioning this season and the timeline in which we stood it up have exceeded all our expectations. It’s something we’re super-proud of.”

Navigating a Tight Timeline

Perhaps the most impressive aspect of this entire process is the timeline on which this was all achieved and how Pac-12 Networks navigated that timeline while staying committed to an all-IP build.

Regardless of the state of the conference and its membership, Pac-12 Networks was nearing end of its initial 12-year distribution deal and, more pressingly, the 11-year lease on its San Francisco space. Moves needed to be made without the broadcaster’s knowing what the future holds.

WATCH: Pac-12 Networks EVP, Content, Larry Meyers spoke about the network’s new facility at the SVG Remote Production Forum in September during a panel titled “Perspectives on New Remote Production Workflows.”

There was even a debate on what made sense to do geographically. Stay in the expensive greater limits of San Francisco? Find another space in the Bay Area? Leave California entirely? Eventually, the behind-the-scenes talent that the broadcaster had recruited from the Bay Area region became too difficult to pass up, and so the space in San Ramon was agreed on.

“We have so many people with so much institutional knowledge on production techniques unique to us that we decided to stay in the Bay Area,” Meyers explains. “Our engineering team, which has led the way from the very beginning in terms of technical architecture, decided early on that they wanted the facility to be ST 2110, and this was before we had a building, before we had an integrator, before we had an architect or a contractor.”

The decision to go fully IP was ultimately settled when the engineering team signed on for SMPTE ST 2110 at NAB 2022. It was there — following meetings with Arista, Imagine Communications, and others — that the broadcaster began placing orders on key pieces of equipment and infrastructure. That’s certainly not the norm, considering that the broadcaster hadn’t even contracted ASG as its systems integrator yet. However, with the state of the supply chain at the time and the rapidly approaching lease deadline, calls needed to be made.

“Some of the fundamental questions — what are our formats that we’re going to be using? what is our transport? what is our network architecture? — we were able to answer early on in that process,” says Currier. “When it got to the point of going through an RFP process and picking an integrator, we had a lot of our base requirements well understood in terms of the scope of the facility, the timeline, and some of those core technical issues. We had a pretty clear vision from our team.”

By the time Pac-12 Networks did partner up with ASG, the project was just over a year away from the facility’s scheduled debut.

“There’s nothing better than an experienced client who knows what matters to them and what doesn’t,” says Van Hoy. “It helps with clarity in a project. Often, we have clients who might be less experienced and feel they have great clarity, and we question it. But, with [Pac-12 Networks], there was no debating that they knew exactly what they were doing and what the mission was. It was impressive.”

Four centralized control rooms are built around vMix, a software-based vision mixer. Shows run from these rooms can be produced by as few as two operators. (Photo: Pac-12 Networks)

For ASG, it was a unique project. Not only had Pac-12 Networks already begun investing in the new infrastructure, but the San Ramon building needed to be built (giving ASG a little more than two months to implement the commission and complete the system), and the client was requesting a more transparent budgeting process along the way.

To land the deal, ASG VP, Systems Integration, Michele Ferreira suggested a cost-plus model and a time-and-materials model. That would allow Pac-12 Networks to manage the costs of what it wanted ASG to do and not do. As a result, ASG could be transparent about the costs, the broadcaster could make decisions within its budget, and ASG could advise along the way.

“We settled on that kind of model, which, for this size project, is very unusual within the system-integration world,” says Van Hoy. “It is quite common in the construction world; that’s how a lot of very large construction projects are run. It led to a very different business relationship that allowed it to be very collaborative in ways that often aren’t the case.”

Pre-Commissioning Lab Tests Out the Technology

One of the most critical ways Pac-12 Networks was able to meet its August 2023 deadline for opening the new facility was a result of a strategy proposed by ASG: a testing lab.

Since Pac-12 Networks would be leaving the San Francisco office in May 2023 and needed to be up and running for the next athletic season the following August, there wouldn’t be time to comfortably test all that material. ASG proposed setting up essentially a pre-commissioning lab in the San Francisco facility to test out the ST 2110 system on a small scale, simulating how Pac-12 Networks planned to use it at the San Ramon building.

“Because of that, we were able to just crank through individual problems,” says Currier. “Vendors would do software patches just for us, and we’d be able to test that in real time. We had vendors and partners sitting side-by-side with our engineering team in this pre-commissioning lab just watching stuff go. I think getting as close as possible to the engineers at the vendors [was important].

“We may have benefited from some of the stuff we were trying to do,” he continues. “I won’t say it had never been done before, but it hadn’t been done very often, or it might not have been done in our exact way. I think we benefited from a lot of interest from the vendors because it was a great way for them to validate and harden their software and their technology as well. I think there was a lot of win-win there in terms of our being able to achieve our goals but also in helping the vendors and then, ideally, hopefully, the overall industry moves forward in terms of vetting out a lot of this interoperability.”

Five of the facility’s production-control rooms are accompanied by a dedicated audio control room anchored by Calrec audio mixing consoles. (Photo: Advanced Systems Group)

Says Nik Kumar, technology specialist, systems integration, ASG, “We had the opportunity to evaluate and gauge not only the system’s conformance to the 2110 set of standards but also how well they would comply within the confines of Pac-12 Networks’ operational workflows. Through the lessons we learned from that exercise, we were able to ensure that there weren’t any major obstacles in the critical path of the project. One of the big things was, we also wanted to get Pac-12 Networks more comfortable with the technology and more comfortable with the user interfaces because, ultimately, they’re the end users.”

Mailhot says, “I have to commend that. We get involved in projects [that start] with an RFP of a bunch of ‘system shall do X, Y, Z’ and it’s 50 pages. You eventually put the real end user in front of it, and they didn’t want half those things and wanted things that weren’t on that list, and you’re in a big rework. That aspect of going through it at a small scale with the end customer was critical.”

By August, Pac-12 Networks was able to start testing the system in the San Ramon building. The team worked with mobile-facilities vendor Gravity Media to park a full truck at UCLA and hook it up to existing connectivity to carry signals to the new building. Meyers says that, by a month out from the first live game broadcast of the 2023-24 academic year, his team had a working system.

“[ASG] worked their butts off to get this thing done,” he says. “We have a great relationship with them. We felt it was a true collaboration, which doesn’t always happen with integrators.”

Remaining Focused in Turbulent Times

As the winter sports season rolls on, the spectacular facility is producing hundreds of live events in its unique model. The facility, though, much like its tenant, remains a work in progress.

As it stands, two members remain in the Pac-12 Conference (Oregon State and Washington State), and it’s unclear whether the Pac-12 will rebuild itself, merge with another conference, or dissolve entirely. Although it was a challenge, at times, for team members working on the San Ramon facility to stay motivated in the face of major shifts occurring outside their control, the work was done on time and at an elite level.

“Everyone who works here knew that the Pac-12 Networks as currently constructed was going to end in June 2024 and the new world would look different,” says Currier. “We used that as inspiration. This was a cool opportunity to create a foundation that can be flexible even though we don’t know how things are going to be different.”

From a broadcast standpoint, will a new media-rights partner want HDR or even 4K? What will connectivity to a new collection of schools look like? All significant questions for another day, but, for now at least, those who call San Ramon home continue to drive the live-content train that is Pac-12 Networks in a building they’re proud to work out of each day.

“Ultimately,” says Meyer, “we wanted to build a place that was a pleasant work environment and bring in the freelancers. Our goal was to do just that: build a flexible, state-of-the-art, scalable facility that would foster good storytelling. That’s the bottom line.”