Pac-12 Networks’ IP-Production Workflow, Internet2 Infrastructure Continue To Grow

A dedicated network aids in the move away from fiber/satellite transport

Since launching in 2012, Pac-12 Networks has continued to cultivate the IP-enabled at-home-production model that it helped to pioneer. For nearly a half a decade now, Pac-12 Networks has used its multi-campus Internet2 network to send camera and audio feeds from remote productions to its Network Operations Center (NOC) in San Francisco, where centralized control rooms are used to integrate the telecast. This practice has grown since Pac-12 Network’s birth, with more than half of Pac-12 Networks’ 850 live-game productions last year deploying the at-home production model.

Now Pac-12 Networks has taken its efforts to the next level by enhancing its existing Internet2 network, which already boasted connectivity between the NOC and more than 150 venues across Pac-12 campuses. After months of planning and development, last December, Pac-12 Networks erected a 10-Gbps IP transport network dedicated solely to Pac-12 traffic. As a result of this IP network, Pac-12 Networks now has its own IP video-contribution network and plans to reduce its reliance on traditional fiber/satellite video-transmission services to almost zero.

Pac-12 Network Sports Report. Photo Credit: Kelley L Cox – KLC fotos for Pac-12 Network

“[Since we launched], we’ve not only increased our total events from 500 to 850, the workflows have become far more sophisticated as well,” says Michael Harbin, VP, technology, engineering, and media management, Pac-12 Networks. “We went from, in year one, using a lot of satellite trucks for our games to, today, not using video [circuits] at all and relying entirely on data pipes. We are even looking at dropping our [CDN] lines at our basketball arenas and just using video [circuits] as a backup for our football games. So the transition is pretty much complete.”

Upping the Internet2 Ante: Pac-12’s New IP Network
Since Pac-12 Network is owned by its 12 member institutions, the broadcaster has been able to leverage the Internet2 Network, a not-for-profit IP network platform used for research and education, for several years. Last year, Pac-12 began exploring expansion of its Internet2 infrastructure, and, in December, launched a dedicated a 10-Gbps IP transport network with three optical rings to provide redundancy and connect its three regions: Northwest (Northern California/Oregon/Washington), Central (Southern California), and South (Arizona/Colorado/Utah).

“Our goal with Internet2 was to architect a network that would provide us more stability and more capacity, and that’s exactly what we got,” says Howie Chung, network engineering manager, Pac-12 Networks. “We’ve seen significant improvement in stability of transmission, and we were able to increase the bitrate on all of our video paths back to San Francisco higher than we’ve ever done – all with extremely clear pictures.”

An Internet2 Coming-Out Party at Pac-12 Hoops Tourneys
With this new Internet2 network established across member campuses, Pac-12 Networks looked to take it one step further by bringing off-site venues online for its Women’s Basketball Tournament (at KeyArena in Seattle) and Men’s Basketball Tournament (at T-Mobile Arena in Las Vegas) this month.

“We’ve brought off-site venues online before but only with private point-to-point video circuits,” says Chung. “We engaged different entities in Seattle and Las Vegas to provide us with dark fiber back to our Internet2 POP [point of presence] where we landed on our optical ring.”

As a result, Pac-12 Networks was able to use a 10-GBps data pipe (with traditional satellite or CDN) to bring eight individual paths from the Women’s Tournament in Seattle to San Francisco. Then, at the Men’s Tournament in Las Vegas, Pac-12 Networks deployed two 10-GBps circuits for diversity.

“A [traditional] multimedia circuit, which probably offers a 1.5-gig [path] of uncompressed HD video and1-gig of data, would be roughly twice as expensive as running our two 10-gig data-only paths out of the T-Mobile Arena,” says Harbin. “So this offers us a significant reduction in price, depending on the venue, with a massive increase in bandwidth.”

In addition, Pac-12 Networks was able to assist ESPN, which owns rights to three of the 10 Men’s Basketball Tournament games, onsite in Las Vegas by transmitting ESPN’s network return feed on the Internet2 network.

“We had more than enough bandwidth and paths to even help out our broadcast partner, which is pretty amazing,” says Harbin. “This network is truly robust, reliable, and stable, with low latency and high bandwidth.”

ICE ICE Baby: Taking Content Exchange to the Next Level
Although the tournament games were not true at-home productions (Pac-12 Networks deployed NEP’s MIRA M14 mobile unit and a full crew), the Men’s and Women’s Tournaments did provide a unique opportunity to connect the remote production with the NOC in San Francisco. Pac-12 Networks implemented a brand-new EVS Instant Content Exchange (ICE) system, allowing the broadcaster to exchange content between Seattle/Vegas and the NOC without any file-acceleration technology needed.

“The [Internet2] network is the foundational aspect of it, and, once that was built, we’re finding new ways to use it,” says Harbin. “On top of this big network, our team devised an Instant Content Exchange system to exchange highlights in real time between the studio and the venue and the venue back to the studio. So we were able to [produce] breakdowns, highlights packages, teases, and bumps, as well as post content to the web and social media with multiple angles included — all in real time.”

For the Men’s Tournament, MIRA M-14 was equipped with seven networked EVS XT3 replay servers, an XFile3 live file-archive/transfer system, a networked switch, and Pac-12 Networks’ shared server to send the content to San Francisco. The ICE system used a dedicated 1-GBps path to send content to the NOC, where three dedicated editors (for social-media posting, event teases, re-teases, bumps, recaps, and other specialty packages), two dedicated ChyronHego Paint virtual telestration operators (for studio and remote game analysis), and one replay operator (for the studio) were churning out a wealth of content. In addition, the truck’s PL comms system was connected directly to edit bays in San Francisco, so both the remote and home teams can communicate easily.

“We were able to exchange content back and forth between the field and studio seamlessly,” says Erwin Tugadi, senior project manager, production, Pac-12 Networks. “We integrated content directly into the studio, used it for digital content and social-media posting, and supplemented the truck production — all very quickly. And one key that differentiates this file-exchange model with a lot of the other networks that are doing it is that we don’t use any file acceleration on the exchanges. It’s just an extremely fast 1-GBps dedicated path.”

At the end of just the first day at the Men’s Tournament, the ICE system sent more than 1 TB of content to San Francisco. The system also negates any need for a post-game melt, since the content is being sent as the game is happening.

The hoops tournaments served as a proof of concept for the ICE system, and Pac-12 Networks is planning a mass implementation of the system for all live event productions moving forward.

“We had to think creatively how we could supplement these productions at a low cost while still bringing as much quality value as possible,” adds Tugadi. “And we’re able to do that remotely with our Instant Content Exchange [system].”