Live From the US Open: IBM Watson Media Opens Up AI Possibilities for USTA

IBM launches suite of solutions for Cognitive Highlights, Closed Captioning, and more

IBM has long been a major tech player for the USTA at the US Open, providing backend technology ranging from the US Open app to the SlamTracker point-by-point analysis tool. This year, however, IBM has taken it up a notch, using the Open as its launch platform for IBM Watson Media, a suite of AI-powered solutions on the IBM Cloud that analyze images, video, language, sentiment, and tone. Throughout the tournament, the USTA has used IBM Watson Media’s Cognitive Highlights solution to rapidly create and share highlight videos of matches across multiple platforms.

“This allows the USTA to produce more highlights at scale,” explains John Kent, program manager, sports and entertainment partnerships, IBM. “If you asked someone to [produce] highlights for every player of every match on the seven [TV] courts and then push them out to the mobile and social immediately, it would be a monumental task. That’s what Watson does as an advisor. There’s still a great editorial team and video team here that do a wonderful job creating highlights, but this allows them to produce that content at scale.”

Cognitive Highlights Service Makes US Open Debut
Developed at IBM Research with IBM iX, Cognitive Highlights identifies a match’s most important moments by analyzing the statistical tennis data, sounds from the crowd, and the reactions of a player, using both action and facial-expression recognition. The system ranks the shots from the seven TV courts at the USTA Billie Jean King National Tennis Center (NTC) and auto-curates the highlights, simplifying video production and ultimately positioning the USTA team to scale and accelerate creation of cognitive-highlight packages.

The Cognitive Highlights platform identifies important moments in a match by analyzing data, crowd sounds, and player reactions.

IBM first piloted the Cognitive Highlights platform at The Masters golf tournament in April and then supplied it as part of the content-production system at Wimbledon two months later. According to IBM, although many editors may fear that AI technology could replace them, the users at these events and the Open embraced Cognitive Highlights once they saw its capabilities first-hand.

“As soon as [the edit team] gets their hands on the technology, they [see] it as a tool that can help them be more effective,” says Dave Kulczar, senior offering manager, Watson Media. “Bringing insights into their existing editorial tools is a big focus for us as well because it allows them to take information and clip it, cut it, repackage it, or put together in a reel. Those are the things that you’re never going to automate; those are things that editors are going to do forever. We’re just making their job easier.”

The finished highlight videos are available in four ways. First, the USTA posts a Highlight of the Day, as ranked by Watson, on its Facebook page each day. Second, fans that “favorite” players on the US Open apps automatically receive real-time push notifications about those players’ highlights (and iOS 10 users can play the highlights within the lock screen). Third, video highlights are available on player bio pages across all USTA digital platforms. And, fourth, highlights are available onsite in the players lounge and in the fan-facing IBM Watson Experience at the NTC.

“The USTA is always looking for opportunities to enhance that fan experience, which is where the concept of cognitive highlights come into play,” says Kent. “On the seven courts being monitored, we are identifying those individual shots that we would call highlight-worthy, and we are coming up with an excitement level based on the crowd cheering, player gestures, and, for the first time here, facial expressions. So we’re able to truly understand the emotion of that player and factor that in, as well as the high fives and other gestures.”

IBM Watson Media Goes Live With New Products
Although IBM has been piloting Watson-powered AI tools at sports events for several years, the launch of IBM Watson Media signals the company’s productization of these solutions.

“With IBM Watson Media, we are packaging up real, scalable products that we can build upon and customize for individual media and entertainment clients,” says Kulczar. “The technology being used here at the US Open is now part of the IBM Watson Media portfolio. We plan on scaling that technology for other environments, other customers, and other sports.”

In addition to Cognitive Highlights, IBM Watson Media offers a variety of other AI solutions: Content Search and Discovery, Intelligent Closed Captioning, Recommendation Uplift, and Spotlighting.

Editors are finding that, instead of automating them out of a job, AI solutions like Cognitive Highlights are just another tool.

The Content Search and Discovery solution creates refined, detailed searches of content down to the scene level, based on rich metadata identifying items and context. “The idea,” says Kulczar, “is to annotate both live and VOD content and be able to annotate that with extremely deep metadata in order to more easily search and discover the most relevant highlight clips.”

Automated speech-to-text for closed captioning is also key to the Watson Media offering, transcribing audio and improving by learning topic-specific terminology. In addition to Cognitive Highlights, the USTA is using this Intelligent Closed Captioning technology at the Open this year.

“[Watson] is listening to the audio and transcribing for closed captioning,” Kulczar explains. “For each sentence, Watson gives you a confidence level in terms of its ability to translate that [segment]. That’s the starting point, but the editors can easily override it. Then, if they make an editorial adjustment, we note those changes and feed them back into Watson so it can learn from that and, the next time, have a stronger understanding.”

IBM Watson Media also promises to improve content recommendations for viewers, providing more-informed suggestions of new content for viewers based on a deeper understanding of video content.

The Spotlighting functionality identifies content — violence, nudity, adult language — that may require further screening and labeling.

Available this month through the IBM Cloud on the IBM Marketplace, IBM Watson Media will provide a packaged product that can be scaled and used across a company’s video assets.

“The essence of Watson Media is bundling up these solutions into scalable products that we can provide to media and entertainment with three [goals] in mind,” says Kulczar. “First is automating your workflow to cut back on the manual editorial efforts. Second is driving revenue through search and discovery and recommendations. And third is pulling the insights out so that you can quickly monetize content that you wouldn’t have been able to monetize otherwise.”

Also at the Open: The App, Cognitive Concierge, and More
Although the launch of Watson Media has been the headline, IBM has plenty more going on at the US Open this year, including once again designing, developing, and hosting the tournament’s MobileWeb for iPhone and Android platforms.

In addition, The Cognitive Concierge, a pilot project last year, has been incorporated into the US Open app’s Guest Info feature, which answers a variety of questions from fans attending the tournament. Enabled by Watson APIs, the app provides information about scoring, schedules, transportation, and dining options via a natural-language interface, which means that attendees can just type their questions and Watson will respond. Depending on the question, Watson sometimes offers an interactive map of the venue, indicating an attendee’s current location.

SlamTracker, which is back in action processing player and ball position data for analysis of pace of play and proximity to the baseline, is now using Watson Machine Learning to train the player-style models used as part of the Keys to the Match analysis.

IBM Cloud is also helping the USTA scale up in any or all public-cloud locations and/or private-cloud locations. In addition, given the multi-active cloud architecture, if an entire cloud location were to be taken offline for maintenance, operation of USOpen.org and associated digital platforms would be unaffected.