Professors Say Public Needs to Know Sports Dispute Technology Not Perfect

By Andrew Lippe and Ken Kerschbaumer

Tennis fans are debating whether this weekend’s epic victory by Rafael Nadal over Roger Federer is the greatest Wimbledon Gentlemen’s Final ever played but another tennis debate has been roused by a research paper written by Cardiff University professors: whether or not the public needs to be informed that instant replay technologies like Hawk-Eye are not 100% accurate.

“We think [Hawk-Eye] is a great idea and a great machine and they should use it but it should be [clear to fans that it makes errors,” says co-author and Professor Harry Collins, the director of the Centre for the Study of Knowledge Expertise and Science at Cardiff University’s School of Social Sciences in the UK.

The research paper, entitled “You cannot be serious! Public Understanding of Technology with special reference to Hawk-Eye” was posted in the journal, The Public Understanding of Science by Cardiff professors Dr. Collins and Dr. Robert Evans. The paper deals with the larger issue of relying on technology as the end-all-and-be-all for on-the-field sports disputes and bad calls due to human error. In fact, the real point is not to attack Hawk-Eye but to ensure society has a more nuanced and widespread understanding of the statistics of uncertainty.

“Virtual replays of the flight and impact of the ball as used in tennis give the impression that Hawk-Eye provides a reconstruction of what actually happened rather than a representation of what was most likely to have happened,” the professors state.

The public, however, should use an understanding of how replay systems work as a metaphor for uncertainty. “This potential for a more widespread understanding of the domain of statistical uncertainty should be actualized through the presentation of measurement errors and confidence intervals in the outputs of automated sports decision aids,” say the authors. “As the most casual observations make evident, committed sports fans watching TV act as quasi participants in the decisions made by umpires and refs.”

The professor’s issue with Hawk-Eye begins with the statement on the company’s own Web site that the mean error in the position of the tennis ball as measured by its system is 3.6 mm (which is within the 5mm required by the International Tennis Federation).

“Because in a normal distribution 95% of the points lie within approximately two standard deviations of the mean and 99% lie within about 2.6 standard deviations in 5% of Hawk-Eye predictions the error could be greater than about 9mm and in 1% could be greater than 11.7mm,” state the authors.

However the issue is complicated by the fact that even if the professors computations are correct it does not necessarily mean that Hawk-Eye got the call wrong if it makes a mistake. Why? Because if, for example, the ball is actually in by 2.5mm and Hawk-Eye’s error makes the ball appear to be 6mm further in then its error will not impact play. But if the error swings the other way and it appears that the ball is 1mm out then the Hawk-Eye error is impacting the match.

To Collins the solution is simple: make clear to tennis fans that the Hawk-Eye representation is a simulation of calculations, not actual reality.

Stuart Miller, Head of the International Tennis Federation science and technical department, says there is no device that is not subject to measurement error. “It might be small it might be big, but they are all subject to some error and line calling systems are no different,” says Miller. “We test using high speed video cameras and we compare our measurements to that of the Hawk-Eye system.”

The ITF says Hawk-Eye was only approved after strict testing. First, it had to make every in and out line decision correct and if it made a wrong decision the difference between the high-speed measurements and the Hawk-Eye system could be no more then 5mm. Second, the average difference between measurements at any position on the court could also be no more then 5mm. Finally no single impact difference between ITF cameras and Hawk-Eye could exceed 10mm. Hawk-Eye passed the ITF criteria and should any other system meet those standards it too would be deemed suitable.

“This technology is there to provide players with the best opportunity to get the right call in a tennis match where they feel they may have not gotten the right call,” says Miller. “While educating the public on scientific issues is not a bad thing, and I fully support that, I am not convinced that Hawk-Eye technology should be used this way.”

Adds Dr. Paul Hawkins, Hawk-Eye Innovations managing director: “We are not saying we get every single on right but we get a very high percentage of them right.”

Hawkins notes that the technology has been tested thousands of times and that the ITF camera is set up “on the very back edge of the line” for the best accuracy. Hawkins said that “it is difficult to compensate for the varying distance the ball is away from the camera, and it is slightly subjective to determine the exact earliest point of contact. Therefore, no system could achieve 0 mm.”

Dr. Collins gathered most of his information for the project from the Internet from published articles, and Hawkins patent. “With regards to questioning the accuracy of Hawk-Eye, this is simply poor journalism,” adds Hawkins.

The professors, in fact, are the first to admit that open questions and caveats make it clear that the paper is far from complete.

“In our view, a sports decision aid should be as transparent as possible in the presentation of its potential errors,” say the professors. “Automated sports decisions, if their capabilities were presented in a transparent way, could add more to the enjoyment of sport and, in addition, to a better understanding of the limits and possibilities of technology.