NEP Leverages Telestream SPGs to Align Timing Protocols for Mixed IP-SDI Productions

Live Broadcasting With the New Generation of IP-Based Trucks – It’s All in the Timing

This year’s major sporting events were unique in many ways, not the least of which was a newly built golf compound which housed more IP-based outside broadcast (OB) trucks than ever before. Since IP-based trucks would be responsible for creating the primary program feed, NEP, a privately owned international production company, together with their clients, made the decision to synchronize the trucks using the IEEE PTP (Precision Time Protocol) standard. This would ensure that, throughout the events, all remote production gear would remain precisely timed. The challenge was that while some vehicles had IP-infrastructure, several trucks still used SDI and black and burst or tri-level synchronization methods. Therefore, a hybrid approach was needed to sync all trucks successfully.

Dan Turk, chief engineer and vice president of NEP, recalls the challenges of migration to IP. “For years we’ve been building large compounds with multiple vendors, and originally had a couple of IP trucks that were the early proprietary versions of today’s SMPTE 2022 technology, which included a four-line delay. We had to deal with the Encap/Decap (encapsulation/decapsulation) process delays when converting signals between worlds and tie the IP video timing into the baseband world.”

For this year’s events, about half the trucks used the SMPTE 2110 standard for IP-video. The remainder used traditional SDI video with black burst or tri-level sync. Each of the NEP IP trucks utilized two Telestream Tektronix SPG8000A Sync Pulse Generators (SPGs) with an ECO8000 automatic changeover unit for redundancy. The Grandmaster reference for all the trucks was supplied by two SPG8000A’s locked to GPS in a custom made flypack. In addition to acting as a timing reference, the SPGs also generate a wide array of video test patterns and audio signals.

“If you keep all the IP trucks in an island at the top of the chain, then you can time all the SDI trucks and facilities to that,” says Turk. “Each truck signal must be in precisely timed because of shared cameras and feeds. We use Telestream’s PRISM Waveform monitors to measure the timing for each truck.”

Unlike a fixed facility where timing need only be completed once (and then periodically re-checked), broadcast trucks must be setup from scratch and timed out at each new event. The technical director and the full crew must be on hand during setup. The longer setup takes, the more it costs. Everything must be precisely timed, including the DR (Disaster Recovery) systems that are running in parallel. Setup typically occurs a couple days before an event when all the trucks arrive, and the big job of cabling everything together and testing signals begins. Delays between trucks can only go so far before the timing slips out of the switch’s window of tolerance. In previous years, testing and adjusting the timing could take four hours or more. By using PTP and the Telestream SPG8000A SPGs for both grandmaster and internal timing reference in the trucks, the setup has been reduced to just one hour. “Time-wise, we saved about a half a day by using the SPG8000A locked to GPS,” comments Turk. “We had a small learning curve on the way in, but it all worked perfectly, and when we actually did the test, everything was correctly lined up and easier than ever.” Turk and the NEP team now use this configuration for every major event at large compounds with multiple trucks.

Should an SPG8000A lose its GPS or external reference, the patented Stay Genlock and Holdover Recovery algorithm can continue to synchronize the IP-based trucks locally until the reference is restored, while the Telestream ECO8000 Electronic Changeover unit can automatically switch references for the more sensitive analog based systems using black burst and tri-level sync. “One of the reasons NEP buys the SPG8000A over other options is because of the holdover feature. It’s something that we check as part of our DR workflow at large shows.” The Holdover Recovery feature has been put to the test in the past and everything stayed perfectly locked until reference was restored.

“With a mixed compound of SDI and IP trucks, there’s still a need to be able to swing in traditional SDI-based trucks. This is done by providing an individually adjustable input to the SPGs in the SDI trucks from the main system. Doing this from the sync generator allows each individual truck to be adjusted to match the entire compound.”

For redundancy, Turk said that two SPGs are connected to main and redundant Cisco switches. Each truck is a leaf node in a spine-leaf network architecture. “We populate things out so the primary side and the redundant side both have network connectivity to each SPG should one fail. This gave us full connectivity, network and SPG redundancy,” says Turk. “These TV compounds are temporary constructions, so you never know when they’re going to add or change something. Trailers and crews are coming in and out based on the shows so redundancy is a necessity.”

The NEP team has relied on Tektronix scopes and SPGs for many years, so the equipment was familiar. This was the first time year the company used PTP as the backbone to implement the sophisticated timing requirements that result from working in a hybrid environment with 2110 and SDI trucks.

“Telestream has been a great partner through the whole migration, and we’ve all learned throughout the process. We’ve had good luck, because in some ways we’ve been creating best practices as we go. We can now say we’ve tamed the PTP beast that so many people find intimidating,” concludes Turk. “This will be the plan for all future compounds.”