[This is the second part of my review of the 2010 MIT Sloan Sports Analytics Conference.]

In the afternoon, I attended the main forum of the day, which was provocatively titled, "What Geeks Don't Get: The Limits of Moneyball".  Like the baseball, basketball, and emerging analytics sessions, this one was also standing room only.  That it happened in this session was most impressive because the hall had seating capacity for just over 1000 people.  Of course, all of the panelists had considerable drawing power.

On the stage were some of the highest-profile names of the conference: Mark Cuban, owner of the NBA's Dallas Mavericks, Jonathan Kraft, general manager of the New England Patriots (and part of the Kraft Group which owns the New England Revolution), Bill Polian, general manager of the NFL's Indianapolis Colts, and Bill Simmons, the principal Page 2 columnist for ESPN.com (and a super-fan of Boston sports teams).  Moderating the session was the man who wrote the book that spawned the sports analytics industry: Michael Lewis, author of "Moneyball", "The Blind Side", and "Liar's Poker", among other best-sellers. 

Lewis came out of the box with an intentionally provocative question to the panel (I am paraphrasing here): You see this tremendous growth in the use of and interest in sports analytics — is there something about it that bothers you, and where do you think it's gone too far?  The question had the desired effect among the audience for sure.  The panelists, in response, didn't see the growth of analytics as a problem; in fact, they saw it as very useful for their evaluation and selection of player personnel. But as they elaborated their points, they started to mention areas where the analysts do in fact face challenges. 

Bill Simmons mentioned one of these challenges, which was the ability of the analysts to communicate analytics to lay people who didn't have familiarity with advanced mathematics or statistics.  This is a common complaint in all technical fields, and a constant challenge to those who work in highly technical fields that interact with the public. It's something that I recognize that I need to improve on in my day job and in this blog.  It's also a difficult task because most of the time I'm too busy getting algorithms and formulas to work and not developing a complete understanding of what has been developed. (In that vein I remember a statement by Nobel-winning physicist Richard Feynman who said that if he couldn't deliver a freshman physics lecture on some new development, it meant that he didn't truly understand the topic.)  Simmons' point was backed up by Bill Polian, who emphasized that these metrics must be made practical, useful, and understandable to decisionmakers. 

Another limit to analytics was mentioned by Jonathan Kraft, which is that they are limited by a small set of available data — the problem of small sample spaces.  This is very true in the playoffs which is much shorter yet much more valued than the regular season (which has a randomizing effect), but also can occur during the regular season itself.  Often there are constraints on sports teams by the league that the average fan is not aware of, and perhaps not even an analyst who has closely followed the league for years. The end result could be something that makes little sense to a decision maker or is merely irrelevant to the matter at hand because it does not take into account other constraints.  Once again, this is a common issue in the engineering fields.  I see a lot of publications on certain theories in my field that are interesting from a mathematical standpoint, yet utterly useless for my day-to-day work because they make assumptions that just aren't true in the practical world.  It's wise for analysts to avoid that trap.

A third limit was the inability to make real-time decision making based on analytics.  Here the Patriots-Colts game received a lot of attention — naturally, with officials from both teams on the panel!  The interesting thing about that incident is that Polian thought that Belichick's fourth-down call was the right one to make under the circumstances and taking into account the state of both teams. Polian emphasized an analyst could present lots of statistical analysis showing that the chances of winning go up when certain plays are made from certain points in the field, but in the end, the situation on the field dictates the call.  There are simply too many variables to consider.  On that point, I wholly agree, and I would go on to say that anyone who thinks that analytics are going to replace coaches on the field or in the box are 100% delusional.  They're not supposed to be a replacement, they're supposed to be tools that are part of the decision-making of a team.  I believe they are most useful before the season, in the middle of season during off-days, and after the season.  But during the game itself, the human element takes over and the players and coaches have to perform.

Lewis asked if the panel thought there would ever be a point where sports analytics would cease to provide an advantage to teams.  Mark Cuban gave an emphatic no and said that there are several areas where there exist inefficiencies in the system that could be exploited.  In line with the principles of market equilibrium, those inefficiencies won't exist for very long but they will continue to exist. Cuban, Polian, and Kraft said that just a difference of 2-3% in performance means the difference between a top seed in the playoffs and missing them entirely.  Even a percentage of less than one percent is important, and that's not surprising in a highly competitive and elite field like professional sport.  Lewis asked Cuban if he could give an example of one inefficiency in the NBA and he responded succinctly, "Referees!"  After the audience laughed for 30 seconds, he gave a very detailed and very perceptive analysis supporting his point — not naming anyone of course, but demonstrating that certain referee crews do make certain calls more than others, and thus make significant impacts in the outcome of games.  (All of which shows why gambling scandals involving referees are much more serious than those involving players, especially now that players make so much money the incentives to throw a game are minimal.)  There are also imbalances in intelligence, of course, and certain teams will make different and often baffling decisions even if everyone had the same information.  Polian felt that this is why it was important to hire front office people from a broad spectrum of careers rather than someone who just played football for most of his life. There is so much money in professional sports that it's important to get the decisions right, and that requires hiring the best people.

This session did not disappoint in the slightest, and they could have gone on for another hour with ease.  All of the members of the panel made very perceptive comments, and Michael Lewis made very thought-provoking questions.  I have to make a special comment about Mark Cuban.  Mark Cuban has a certain media persona that comes across as rather obnoxious, and I came to the conference wondering if that was really true.  If there is anyone who does not match the media-generated persona about him, it is definitely Mark Cuban.  He is direct in his opinions and he can be a little loud at times, but he is a very smart and very perceptive man. Every comment he made was useful; you might not agree with everything, but it's clear that he's given them some thought.  So it was a pleasure to have heard you speak, Mr. Cuban.

It was also nice to meet Bill Polian in the hallways as well and I thanked him for the comments he made about analysts needing to communicate better with decision-makers.  He was very kind and said that the decision-makers often need to communicate football-specific terms to the analysts better!  I'm a Miami Dolphins fan, but I do respect the Colts' front-office staff and their decision-making ability.

This session was a useful corrective to the world of sports analytics, but that's not all bad.  I think it's useful to find out what the problems and limits are in order to figure out which ones to address first.