I need to make an admission: I dread doing team and player performance analysis for the European Championships.

Not only that, but I also dread doing team and player analysis for the World Cup, or the Copa América, or any other continental tournament.

Admittedly, that’s a dangerous thing to say, especially for someone who performs match analysis in soccer.  I’ve presented analyses of this year’s African Cup of Nations using match data, and I will present analyses for UEFA Euro 2012.  But month-long national team tournaments like the World Cup or the continental championships present a unique situation of a very limited number of matches and a large amount of public and club interest.

It’s not unusual for certain players to burst into the public consciousness after a successful World Cup, and there are several examples of players having earned lucrative contracts on the back of a strong performance (Pável Pardo to Stuttgart in 2006 is one example, Clint Dempsey to Fulham is another).  More clever clubs have identified talent before the World Cup, who later went on to have a successful tournament, like Javier Hernández to Manchester United in 2010.

Yet there are several other examples of players who were acquired after distinguishing themselves in a tournament but failed to achieve the same performance with their new clubs. They illustrate very well the problem of small-sample statistical analysis.

We — the sport analytics community — seek to extract meaningful information from a limited data set, primarily through estimating a mean value or assessing the significance of a statistic.  Naturally, we can obtain more precise estimates from a large sample of data than a smaller one (central limit theorem).  It is possible to develop an estimate of performance with a small sample, but the confidence interval of that estimate becomes very large.  If we knew things such as variance of goals scored, or cards received, or whatever figure within the sample of players in the competition, we might be able to tighten up that confidence interval, but it will still be very large.  Testing for statistical significance is even more challenging; depending on the application, the amount of data will yield an assessment that is marginally better than random.  Incidentally, I have yet to see such limitations mentioned in the Euro analysis, so perhaps I should start a trend.  I suppose one other solution is to embrace Bayesian techniques, which is a lot more work but appears to deal with the statistical uncertainties that are inherent in small-sample analysis (and large samples, for that matter).

It’s natural for fans, media, and decision-makers in the football industry to fixate on players who have distinguished themselves at the Euro.  It’s also natural to weigh the most recent observations on a player or team more heavily than previous knowledge, and the danger ensues when people act as if those observations indicate future performance.  The remedy is to be aware of these tendencies and insist on making judgments of players and teams using as much data as possible.