We are developing the social individualist meta-context for the future. From the very serious to the extremely frivolous... lets see what is on the mind of the Samizdata people.

Samizdata, derived from Samizdat /n. - a system of clandestine publication of banned literature in the USSR [Russ.,= self-publishing house]

Of cricket and climate

An unfolding saga in the game of cricket in recent years has been the question of whether technology should be used to aid umpires in the case of close or potentially controversial decisions. Like many things in life, the question of whether to do this has turned out to be more complex than it may at first have appeared to be. There have been situations in which the on field umpire has asked for a replay, and the replay has been unclear but has none the less been used to overrule an on field umpire who probably saw more. There have been situations in which the players have appealed to a video replay that didn’t show anything, when they likely knew themselves what had happened and the situation would previously have been resolved with a gentlemanly code of conduct. There have been situations when the television company did not manage to produce the appropriate replay in time, and the umpire then made a decision that was revealed to be incorrect five minutes later. Many decisions depend on whether the batsman hit the ball, and a mixture of sound and picture is used to make these decisions, and determining which items of the bat, ball, ground, clothing, safety equipment etc came in contact with each other is often no clearer using the technology than not.

Fans of other sports are no doubt nodding at this point, as similar issues have come up in most sports that have attempted similar things. Cricket has one further issue, not unique to it but relatively central to it, which is the question of how technology should be used in the interpretation of the leg before wicket rule (LBW).

One of the principal ways of getting a batsman out is for the bowler to bowl at the batsman, for the batsman to miss the ball, and for the ball to then strike the wooden stumps behind him. It would be possible for a batsman to avoid getting out this way by his simply standing in front of the stumps at all times. In order to avoid this, the rules of cricket allow for a batsman to be out if he is standing directly in front of the stumps, the ball hits his leg, and the ball would have gone on to hit the stumps. (The rule is actually more complex than this, but the complexities are not relevant to the point I am making). The umpire stands in a good position from which to judge whether the ball will hit the stumps, and traditionally the umpire’s judgment has been used to decide whether the ball would have hit the stumps.

Umpires inevitably make mistakes, and there have been many accusations of umpire bias over the years. For a long time people have been watching replays in slow motion on television in order to second guess umpires, but these have never been conclusive. Occasionally an umpire will make an obviously wrong decision, but most of the time there is as much of an element of doubt watching at home as there is for the umpire. Or perhaps more: the umpire is in a better viewing position than the TV cameras.

In recent years, however, things have changed. The “Hawk-eye” system was initially used by television companies, and there was then pressure for it to be used in assisting umpires as well. Basically, this system looks at a number of video replays, and from them constructs a three dimensional model of the ball, the pitch, the bat, etc. From the this model, the path of the ball is extrapolated going forward. Television viewers see a computer graphic image of the ball hitting (or not) a computer graphic image of the stumps, and are told whether the ball would have hit the stumps and whether the batsman was or was not out.

Every since this system has been in place as part of television coverage, there has been pressure for it to be used in umpiring decisions. When people have asked me about this, I have stated my position with unexpected vehemence, particularly given that I am generally in favour of using video replays as part of the adjudication process. For I am, at present, unequivocally opposed to the use of Hawk-Eye and similar decisions in umpiring decisions.

My reason for this is as follows. Hawk-Eye is a system featuring a lot of complex computer code. The code is proprietary, so what follows is largely reasonably well informed speculation. Although we do know the laws of physics with respect to motion of cricket balls, air resistance, the effect of gravity, bounce when the ball hits the pitch, linear and angular momentum, etc etc etc, the complexity of a even a relatively simple system such as a cricket ball moving in a cricket game is such that it is difficult to impossible to develop a useful model directly from the physics. In addition there are margins of error when triangulating the motion of the ball from video imagery. What this means is that Hawk-Eye’s models are not really physical models per se. What they have likely done is a more simple matter of trial and error followed by extrapolation. The ball has been measured going through the air, perhaps half way down the pitch. Various ways of further extrapolating the position of the ball have been tried, and they have been compared with the actual motion of the ball further down the pitch. Trial and error has continued until prediction and reality have become close, and the resulting (statistically derived) algorithm has been used to predict the motion of the ball in cases where the ball has not actually traveled the full distance (because, perhaps the batsman’s leg was in the way).

This is all actually fine, except that there are circumstances in which the system can break down. Weather conditions that were never encountered in the development phase might be one. Balls made by different manufacturers might behave differently. Pitches in different places may be made of different kinds of grass. In cricket, local conditions matter a lot, and pitches in different countries are known to have different characteristics and favour different types of bowlers and bowling. (The greatest players will succeed in any conditions, but some players will only be effective in certain kinds of conditions).

These sorts of empirical models are not always terribly robust: the further you are from looking it the actual physics the more likely that your software will do a poor job when you depart from the exact conditions in which you did your calibration. A model developed in English conditions by Englishmen may not work as well when used in India. It may contain biases, either accidentally or deliberately.

However, there is a dangerous tendency of people to fall in love with the technology: in cases where the technology disagrees with reality, then it is reality that is at fault. “Science” has given you a definitive answer, which is obviously right. However, if you dive into the science, you will discover it has been developed using a methodology involving a lot of ad hoc steps and statistical approximations and errors.

It may well be that Hawk-Eye, as it exists, is better than the unaided umpire’s eye. However, arbitrary choices have gone into its design. Human biases have gone into its design. Different, equally skilled programmers may have made different choices. Hawk-Eye is not definitive.

In tennis, Hawk-Eye is used in umpiring decisions. Players have the right to challenge line calls. When they do so, a simulation of the ball hitting the line on the court is shown, and the computer rules the ball in or out.

And yet, when this decision is made, all the umpire sees is the simulation. A simulation of the ball intersects with a simulation of the line. There is no attempt to superimpose the simulated ball on the real ball, or the simulated line on the real line. It is all technology. Players have from time to time complained that the computer’s decision is wrong. And yet, on television there is no way of telling that.

Given that Hawk-Eye is not definitive, another nasty possibility rears its head. Like baseball, cricket is a game greatly suited to gambling. There are a huge number of statistics that followers of the game are interested in, and it is possible to bet on the outcomes of most of them. Cricket has had a significant number of corruption scandals in recent decades. Most famously, a captain of the South African national team took bribes from Indian bookmakers to lose matches. (He later died in a mysterious plane crash). Many other players took money from bookmakers in return for various favours, more of which probably influenced specific match statistics rather than actual results of matches. The manager of the Pakistan team may or may not have been murdered by gangsters for reasons related to gambling during the last world cup.

Imagine then a situation where statisticians and programmers are running a system of computer code that nobody understands (and which, in fact, they are extremely secretive about the workings of, on the basis that it is “proprietary intellectual property” and a valuable trade secret) which has the ability to overrule umpires decisions. Nobody outside the firm they work for (and indeed few people inside the firm) knows exactly who the people are who run this stuff – certainly not the official administrators of the game. The potential for corruption is obvious.

Far fetched? Well, the best way of running such corruption would be to not be obvious about it. Decisions should still appear approximately right. You do not change the result of matches, but apply a systematic bias of 5% in the direction that improves your profitability. As it happens, I am not a bookmaker, but I am a financial analyst. Give me a 5% systematic bias in the financial markets that I can control, and I will shortly be a very rich man.

You don’t believe that bookmakers could figure out who was in charge of this code, who was vulnerable to bribery, and get to them? Well, bookmakers have got to a lot of people in this game. Players, officials, and coaches. And as far as understanding technologists and statisticians, bookmakers are way ahead of just about everyone. Bookmakers these days employ huge numbers of programmers and statisticians. If you think about their business for even a moment, it is obvious why. In fact, it is likely that statisticians and programmers in both these places know one another already, as they were likely recruited from the same pools of people – mathematically adept obsessives who love sport.

Which is why I have been opposed to the use of Hawk-Eye or similar systems to assist umpires in cricket. The potential for corruption is too great.

Except, when people have pressed me further, I have qualified this a little, and I have actually said that Hawk-Eye and similar systems should not be used unless the code and the data are open. If people at home are able to go through the code line by line, see what it does, and then run it themselves, and duplicate it themselves, using the same video footage being used by the actual system assisting the umpires, then any corruption will be visible instantly, and the game is safe.

As it happens, I suspect that the administrators of cricket and the television networks broadcasting it consider the footage (ie the data) to be too valuable to make it available to everyone. I suspect also that the developers of Hawk-Eye consider their intellectual property to be too valuable to make the code available for technically minded cricket fans to run at home. If so, then fine. LBW decisions should be made by the on field umpires. They may be corruptible, but at least they are the people responsible for their actions, rather than some unidentifiable programmer. Hawk-Eye is a wonderful tool for observation and commentary on the game, but involving it in the administration of the game raises issues that its developers may not want to deal with.

This story, of course, is very similar to the stories of many other fields of endeavour, particularly scientific endeavour. A huge amount of modern life, and a huge amount of modern science, involves computer simulations of statistical systems, either in the foreground or the background. The level of complexity and the level of obscurity of these systems is such that (intentionally or unintentionally) such systems are very vulnerable to becoming sloppy, biased, or corrupt.

This, of course, is why, if it deals with anything important, such code must be open. And data must be available. Outside eyes must be able to investigate such biases, and the possibility of the presence of outside eyes must be there in order to discourage corruption and simply to discourage sloppiness and detect genuine errors. And the data must be available, so that other people with different biases can construct competing models, as, always, little is ever settled and improvement comes through competition.

It is when results are robust to such biases that interesting things are discovered.

Tweet about this on TwitterShare on FacebookShare on LinkedInShare on TumblrShare on RedditShare on Google+Share on VKEmail this to someone

15 comments to Of cricket and climate

  • I agree in pronciple to what you are saying regarding have the code open, but in order to convine them to do it, you would have to provide safeguards for them to protect their investment.
    As you have stated, it’s a wonderful piece of technology that they have invested time, resources and energy into, and they deserve to reap the financial rewards from it.
    Do you have any suggestions as to how you could satisfy this. I have been trying to come up with some but so far am stumped.

  • Alice

    >”it’s a wonderful piece of technology that they have invested time, resources and energy into, and they deserve to reap the financial rewards from it.”

    Isn’t this a classic example of what the military call “Mission Creep”?

    The developers of the Hawk-Eye system made those investments to get a return from TV companies who in turn charge the people who choose to pay to watch a cricket match. (Presumably, there are such people?). Fair enough. Let the developers keep their Intellectual Property and the revenue stream they drive from it.

    The problem arises when that same system is proposed to be used to control the game, not merely to analyze it.

    Mr Jennings has it right. Either leave Hawk-Eye as merely an interesting analysis tool paying profits to its developers, or bring it into the game control loop as open source software.

  • In tennis, Hawk-Eye is used in umpiring decisions. Players have the right to challenge line calls. When they do so, a simulation of the ball hitting the line on the court is shown, and the computer rules the ball in or out.

    And yet, when this decision is made, all the umpire sees is the simulation. A simulation of the ball intersects with a simulation of the line. There is no attempt to superimpose the simulated ball on the real ball, or the simulated line on the real line. It is all technology. Players have from time to time complained that the computer’s decision is wrong. And yet, on television there is no way of telling that.

    It’s worse than this. There was a case at the US Open a few years back where a player challenged, and the pretty picture showed the ball on the line. The computer verdict, however, rended in big bold letters, read OUT. (I believe this is the video.) If the software can’t get the simple part of drawing the pretty picture right, what hope does it have of doing the more complex math of determining where the ball actually landed.

    There’s also the fact that the call is taken as gospel truth by imbecilic announcers, when of course there’s a margin of error. If 1% of the pretty picture shows the ball on the line, they’ll say that of course the ball was in by a millimeter — and the line judge’s out call will get overturned — when in fact the margin of error really means there’s inconclusive evidence.

    As for gospel truth and imbecilic announcers, consider that in tennis, we’re not getting to the clay court season. If you’ve watched any clay court tennis, you’ll know that Hawk-Eye isn’t used there, because the ball leaves a mark when it hits the clay. It happens several times a match that, on a close call, a player will circle a mark, and the umpire will come down from his chair to inspect the mark and call the ball in or out appropriately. 99% of the time, this mollifies both players. However, when Hawk-Eye has been used solely as an extra for the broadcasters, there have been cases where the umpire rules one way based on the mark, and the line-guessing computer simulation yields a different result. I’ve heard announcers (Pat McEnroe in particular) claim that this obviously meant the umpire called the wrong mark. And yet, the announcers still treat Hawk-Eye as infallible. It’s maddening.

  • Tex

    Paging Scott Wickstein to this thread

  • M4-10

    As I often do after reading a long post here, I scrolled back up to see who wrote it. And noticed the title: “Of cricket and climate”. I’m going to assume that Michael considered the title to be the only connection necessary to make a parallel between Hawk-Eye’s modeling and proprietary code to climate science, and not that he forgot to write that part.

    As a colonial cricket non-fan: Good post!

  • Hawkeye is very easy to test for cricket – bowl a few overs at the wicket without a batsman, give the software the video images up to where the ball reaches the area where it might normally hit the pads, and see if it can accurately tell whether the ball did or did not go on to hit the wicket.

    There is no reason for the authorities to be in any doubt about how accurate hawkeye is.

  • I’m going to assume that Michael considered the title to be the only connection necessary to make a parallel between Hawk-Eye’s modeling and proprietary code to climate science, and not that he forgot to write that part.

    Indeed. To me it’s fairly obvious he’s saying the climate modeling software may not be entirely cricket.

  • Kevin B

    To me it’s fairly obvious he’s saying the climate modeling software may not be entirely cricket.

    Hawkeye has a big advantage over climate modelling software in that it can test its own predictions immediately and correct for variations by adjusting the parameters in its algorhythm.

    I don’t know for sure, but from personal experience of track prediction software, (many moons ago), I would guess that Hawkeye comes equipped with a standard model of a cricket ball hitting a pitch which it uses to make its first prediction of where the ball will go. It will use any error to modify its parameters and then make it’s next prediction and so on. By the time the first ball hits the pad it will have a pretty accurate model for the prevailing conditions.

  • Robbo

    It does not matter if Hawkeye is inaccurate – the issue is that a decision must be made, and it is better if the decision, right or wrong, is impartial. So is a piece of software more likely to make an impartial decision than a human being ?

  • Eric

    Even if the code were to be made “open”, it wouldn’t be impossible to load and run a modified version during the game, or portions of it.

  • This is why it is important to have the data open as well as the code. In the end, it is open data that is the important part, as code with equivalent functionality can be written elsewhere.

  • Dyspeptic Curmudgeon

    Not being a cricket player, and having only just read the exposition of ‘LWB’ on wikipedia, surely a ‘better’ technological response would be to have the outside stumps each contain an infra-red (or ultra-blue) laser projecting in a plane aimed through the opposing stump. If the laser intersects the batsman, and contact occurs inside the wicket lane, he is LBW (unless the contact is outside the off-side wicket plane). AIUI, contact outside the wicket line is not LBW, although it is unclear to me, whether a batsman can be LBW when the ball is curving/bouncing into the wicket, although all contact occurs outside the wicket lane. I could be wrong about that, but from my quick reading, it seems wrong to conclude that a ball bowled at say 45 degrees could result in an LBW when the batsman is not in the wicket lane.
    At least this technology does not require extrapolative software: the replay shows whether or not the batsman was in the lane, AND NOTHING MORE. (And it works from both directions, in the event that the bowler has himself blocked the wicket line on his end).
    Comments?

  • Owinok

    I come late so its nothing new to mention that this is a most informative for me. It is a nice reminder that models too work within margins of error. If the problem is defined as the need to reduce errors in calls by unpires, then the “Hawk-Eye” software is just one approach. It is possible to have the software audited by some professional firms in the instances where its owners are reluctant to make it available in open format.

  • Laird

    I see nothing wrong with computer models, notwithstanding the inherent “margin of error”, as long as the model is fairly and equally applied to all contestants. With a fair model, over time the errors will cancel each other out. Certainly the margin of error is no worse than the human error of the umpires. Hawkeye in tennis works well enough, and even if it’s not perfect (what human invention is?) it’s good enough and serves its purpose of providing a neutral means of appealing what are perceived as errors by the linesmen. The key is some means of ensuring that there are no distortions intentionally built into the software to benefit one side or the other. And the only means of ensuring that is open source code (as well as open source data). If the supplier won’t open his code to independent review it shouldn’t be accepted by the sport’s regulatory body.

  • D.P.: No to the lasers. As Michael said, LBW occurs if “the ball would have gone on to hit the stumps.” It is perfectly possible – and more usual on some types of ground than others, e.g. dry, dusty Indian wickets – for the ball to have bounced somewhere way outside the direct line but be swinging back in towards the stumps. I assume Hawkeye caters for simple cases of this perfectly adequately; but, as Michael says, perhaps not for the peculiarities of every bowler’s spin technique on every surface in every atmospheric condition.