SVG Digital Chat: Limelight Networks’ Rob Colantuoni and Micah Vivion
Content may be king, but when it comes to live-streaming a sports event, that king requires plenty of support to get it where it needs to go. Live streaming takes plenty of planning, and choosing the right content-delivery network (CDN) must be an integral part of that plan.
As a CDN service provider, Limelight Networks works with clients to deliver faster Websites, responsive applications, and high-quality video to any device. SVG sat down with Limelight’s Rob Colantuoni, advanced services architect, and Micah Vivion, senior product manager, for some technical tips on how to plan and deliver a successful live-streamed event, how to monitor and secure valuable content, and what market segments show the most growth.
How do you work with clients and, specifically, sports clients to help them plan to live-stream their content?
Micah Vivion: We start by sitting down with the customer and understanding what the workflow is. We ask how they publish events; what services, equipment, and players they use; what bitrates they want to stream to; who the audience is; and the devices they’re most likely to consume it on. That’s a very important part to really understand their entire workflow process. It’s important to understand the type of event and the quality they want and then, from there, fine-tune your player or mobile application or desktop device for it. There are many different technical options to consider. You can configure your encoder to best optimize the quality of the stream depending on what type of content it is and the encoder the customer has.
Colantuoni: The first step begins with choosing a content-delivery network. Part of your decision should include taking into account your ingest location (where you’re streaming from) and whether the CDN that you select has ingest points close to that physical location. Confirm that the CDN can support primary and backup ingest into separate physical locations for redundancy and that their delivery policy supports failover in the event of an outage.
A significant element impacting a live-streaming event is the amount of time the traffic spends on the public Internet. For example, if you are streaming an event in New York, you can publish to a Limelight point of presence [POP] in New York. At that point, the stream is carried on our dedicated backbone until it exits closer to the client. When transiting the Internet, you’re ultimately going to run into contention with other public traffic, which will impact stream quality. Limelight Networks has a global private network, so we can drastically reduce the amount of time spent transiting the public Internet, thus improving the user experience.
Another way to ensure viewers receive the best possible viewing experience is with a multi-copy policy. Oftentimes, you may end up getting hot spots on a CDN, especially if you’re offering a single live stream to thousands of consumers and certain locations serve more content than others. With Limelight, in each of our POPs, we’ll have multiple servers spreading out that load. Ultimately, this reduces contention on the CDN edge servers and allows for an overall better user experience.
You hit on an issue many sports-content providers face: scaling up for big events and thousands of consumers. How do you work with clients to anticipate audience, and how do you manage bandwidth needs?
Colantuoni: If it’s a brand-new customer, we’ll create an estimate based on a number of factors, such as expected viewership, desired bitrate, and format. If they’re an existing customer, we often review prior events with similar viewership and use that in our estimates. Then, we’ll configure our tuning and multi-copy policy to match expected load. We recognize there’s no second chance on streaming a live event and take extra precautions to ensure success. For example, during delivery, we often have an advanced service architect on the bridge with the client, closely watching the traffic numbers in real time so we can adjust tuning to be more aggressive if there are more viewers than anticipated. Usually, our ballpark figures are high, and we might over-provision, but that’s never a bad thing.
Vivion: We have analytics available that report traffic numbers, viewership, device breakdown, formats. Also, in certain cases, depending on the customer’s needs, we can help them integrate with third-party analytics as well as help monitor, maintain, and leverage for planning future capacity needs.
Rob, you mentioned delivery. What are some things clients can be doing to make sure that they deliver an exceptional live-streaming experience?
Colantuoni: Have metrics, prepare for worst-case scenario, and understand your escalation workflow. Having metrics available during the event will help you spot issues before they become user-affecting. You need both a primary and backup ingest and a documented set of escalations to deal with things if they fail. If you’re running on a mobile platform and the metrics point to a problem with the app, then having an escalation process to the developers is key. If your encoder is failing, then have a process for escalation to your broadcast engineer. With live events, there is very little room for error, and these make a big difference.
Vivion: You have a very fine line to work with when you’re dealing with live events. You want to make sure that you’re delivering — on the quality side — the formats that the mobile or desktop clients need: HLS, HDS, DASH, MSS. You want to make sure that your ingest stream is a high enough quality so that, when we transrate it — we take it out and break it into different bitrates — there is enough resolution to support the type of quality that you want. And that actually differs with the type of events. If you’re, for example, streaming news, you might not need as high a bitrate source stream as if you’re streaming a live sporting event: people don’t hang out and watch a news program on a huge flatscreen TV with all their friends. For sporting events, you want a high-quality ingest stream so that, when we transrate it down for mobile clients and viewing on smaller devices, we can maintain that quality.
And then there are some small details you want to confirm on your side. Make sure that your encoders are time-synced with each other. The time sync is very important in the failover and backup phase of streaming. If your clients are supporting adaptive bitrate, which a lot of them do, make sure all your settings that deal with adaptive bitrate — such as your key frames — are set properly. Typically, your key frame is where a client is going to either uprate or downrate in response to changing network connectivity, so a client might start off with a medium bitrate — a medium-quality stream — and, as they see their local network can handle a higher-quality stream, they’ll uprate. But, if you set your key frames too far apart, there will be a delay in uprating to that higher-quality stream. You want to start with a medium setting and tune it as you go with different events.
Colantuoni: Exactly. And, once you’ve gotten your key frame and incoming bitrates configured, you’re going to want to run a pre-event test with the expected clients and formats. Confirm that your clients properly uprate/downrate as expected and that the quality is to your standards. Test your failover and confirm that the user doesn’t experience a visible artifact when failing over; the timecoding that Micah mentioned will help with that.
How important is player choice?
Vivion: The player that you choose for your legacy site and your desktop site must understand your platform capabilities. Can the device you’re watching streaming on support HLS, HDF? That player should then adapt to each device’s capabilities. That’ll increase the number of people who can view your content.
How do you work with clients to monitor the live stream and, if there is an issue, report that back to you?
Vivion: We have war rooms with dedicated staff who monitor the events, the streams, the ingest, etc., and we also have advanced service architects [ASAs] who handle higher-level expectations. For major events, we’ll have a very large war room with 20, 30 people on a bridge all communicating and passing back what they see in that instant to the ASA.
Colantuoni: We’ll monitor the stream to confirm that everything is performing as expected and that we’re not overrunning any of our tuning measures. Having a partner that looks at internal metrics from the CDN side and then ties that with your client-side metrics is very valuable. Typically, customers give us a view into their real-user monitoring [RUM] data so we can correlate the RUM data and our internal metrics side by side and tune as needed.
Vivion: Don’t underestimate the customer-service aspect of it. Don’t underestimate having an advanced services architect around to help with the planning and delivery. We do this every day and see every possible variation on delivery, so, although you may be very experienced in live-streaming events, you can really benefit from a CDN that will give you excellent customer service and will be there to walk you through the process, especially the transport and delivery piece.
Colantuoni: At the core, it’s about preparation. If you sit down and work with experts in live streaming, such as Limelight, it’s easy to optimize and create a workflow for the day of the event that will address any issues that could occur.
How do you handle security of the live stream?
Colantuoni: Our MediaVault solution allows clients to secure their stream and offers them the ability to customize access. We also support content that has had DRM pre-applied, or we can add PlayReady or WideVine using our media translation layer. Specifically, in sporting events, there are often geographical restrictions that need to be honored. For example, there could be an event where users can view the stream in Brazil but users from the United States can’t view the stream. That functionality must be available in your CDN, or you won’t meet the business requirements of the event.
Vivion: Additionally, some customers want to limit just where the URLs will play — meaning you want to stop someone from stealing your content and putting it on their site — and our MediaVault will help enforce that. We can also help enforce blackout areas and add time-based security, meaning someone can’t record the stream and play it back later. They can only consume it and watch it at that time. And we also work with a variety of prepackaged DRM solutions and have some DRM solutions ourselves.
Where do you see the most demand for live streaming? What market segments show the most growth?
Colantuoni: I see it across the board. For us, it’s one of the most exciting growth areas. We are constantly expanding our infrastructure to support the increasing demand, and the clients are finding new and inventive ways to use live streaming to reach consumers.
Vivion: I agree, it’s everywhere. People’s use of traditional TV is down, but their consumption of content on connected devices, mobile devices, connected-TV viewing is higher. We’re seeing businesses, sporting events, and companies trying to leverage where people are viewing content, and that means the Internet. We’re seeing, across the board, businesses want to do daily or weekly presentations or events with customers. We see sporting events where people don’t want to have to use the large peering relationship with someone else. They want to do the event themselves, so they use Limelight live services. I can’t really say it’s one place; it’s everywhere.
Colantuoni: Over the past few years, our peak live traffic has frequently come from live sporting events and live concert events. World Cup Soccer, for us, was a huge offering; other events — such as football, baseball, and wrestling — are as well. Categorically, our live streaming of sporting and concert events have increased rapidly in the past few years. All of these events have given us the experience and expertise in guiding clients to success with streaming live sporting events, and we look forward to the future.