Using Data to Make World Cup Quarterfinals Predictions: A Cautionary Tale

Thinking ahead to the last three rounds of World Cup play about to begin on Friday, I was reminded of a former colleague who used to say that “Without data, you’re just another guy with an opinion.” In fact, he loved the quote so much that he put it in his email signature. Talk about faith in data!
You’d think that a website devoted to understanding the beautiful game with numbers would embrace the sentiment. You’d be wrong - at least when it comes to saying something new or useful about who will advance in the tournament beyond the group stage and eventually win this thing.

Here’s why: it is very difficult to use numbers (statistics) to make very meaningful predictions at this late stage of the tournament. Part of it has to do with the nature of the data themselves. By the time you get to the quarterfinals in a tournament of 32 teams that emerged from years of qualifying, there is very little variation left among the teams still competing in the tournament. They're all great teams, compared to the almost 200 that didn't make it: they are obviously in very good form, usually have experience playing in World Cups, and they tend to be big and rich countries. So if we want to apply any kind of prediction model to these kinds of teams, the teams just aren't different enough from one another to say anything with certainty about who will win.

Take the Soccernomics model, for example, which tells us that wealth, population, experience and home advantage matter. At this point in the tournament, we’re talking about countries with a very reduced range on these major factors (when compared to all countries or even all countries in this year's World Cup). And from a statistical vantage point, you can explain outcomes that vary among teams (wins, points, goals, etc.) only with things that vary themselves. Or, to use statistics-talk, you can’t explain a variable with a constant (by definition something that doesn't vary). Practically speaking, this means that using the kinds of variables that are typically used to make predictions about the quarterfinals or who will eventually win produces the obvious set of countries like Brazil, Germany, or the Netherlands.

But what about Paraguay or Ghana, you may ask. Good question. They obviously are smaller and poorer than the other major soccer powers that typically make it to the quarterfinals and beyond. But this is exactly why they are long shots – something we already know. But aside from this, they are not all that different in the scheme of things. For example, Ghana's and Uruguay's FIFA rankings of 30 and 32 before the tournament puts them in the top 15% of all countries ranked on the FIFA index (there are 202 countries in all). They are hardly Papua New Guinea (with apologies to Papua New Guinea).

So when you’re faced with very little variation (in the grand scheme of things) on the factors we think matter to international soccer success, we are left with making obvious predictions based on things like rankings. And these predictions will vary ever so slightly, depending on the assumptions we make going into the statistical analysis. Now, I wouldn’t label well-founded assumptions mere “opinions”, but it is important to remember that what comes out of the model is affected by what you put in.

Let me give you a couple of examples. In an earlier post, I made predictions about who will make it to the quarterfinals, based on FIFA ranking and past World Cup experience. We can use the same model to make predictions about who will make it to the semifinals. Using these semifinals predictions to make a forecast about the quarterfinal matches to be played on Friday and Saturday gives the following results:

According to these odds, Uruguay, Brazil, and Spain go through, and the Germany-Argentina match is anyone’s guess.

But what if we based the predictions model only on teams that made it to the second round in 1998, 2002, and 2006? This will give you the following predictions:

The substantive predictions are pretty much the same, with the exception that Germany is now tipped to beat Argentina. But the odds have changed quite a bit. Now, Spain's are about three times Paraguay's, whereas before the difference was much smaller. And overall, the differences in odds among all the teams have increased slightly, so we seem to be more certain of our predictions of winners and losers.

Now, what if we simply use the pre-tournament FIFA rankings to predict semifinalists? This will give you the following odds, by team and match:

Turns out, this is pretty similar to the other predictions, with the Germany-Argentina match moving to the toss-up category again. So, depending on which model you trust, Germany will win, or we're not sure.

Guess what? These sets of predictions are similar to what forecasting guru Nate Silver of 538.com predicts or that most people would predict, quite frankly, looking at fan discussions. So at this point in the tournament, the numbers game becomes a contest of predictions models, like the models used by political scientists and economists to forecast presidential (and other kinds of) elections.

More importantly, these results are what most people without numbers would predict. It’s good to have data to support your opinion, I suppose, but at this point, your guess is as good as mine (and they’re likely to be the same).

So how can statistics be helpful aside from producing predictions the guy at the corner pub will be able to make? Turns out, statistics do offer a clue why the predictions game is so much fun this late in the tournament: First, if you think about it, the odds we are talking about aren’t all that high in absolute terms – 35% (in the case of Brazil, Germany, or Argentina, depending on the model), or even Nate Silver’s 60-some % for Brazil and Spain. These predictions are a long way from 100%. So keep in mind that, overall, you’re actually more likely to be wrong than right betting on these horses.

Second, the predictions tell us the average outcome we would expect, based on all kinds of assumptions about the data distribution (they are so-called point predictions), but not specific events. And these forecasts come with a so-calledconfidence interval, which tells you that the true number would fall within a certain range most of the time (usually, we use 95% of the time) were we to collect ever more data.

For World Cup predictions, these confidence intervals are large – and therefore, our predictions are uncertain. For example, the prediction that Brazil will reach the semis (based on FIFA rankings alone) is .304. But the 95% confidence interval for this particular prediction ranges from as low as .096 to as high as .511. This means that we are 95% certain Brazil’s odds of reaching the semis fall somewhere in this range. But is this really saying much? (The same is true for virtually any predictions I have been able to generate.)

So, keep in mind what these numbers really can and cannot do for you. (A great read on this, especially as it relates to soccer is Zach Slaton’s “Statistics Are Just Numbers” on his A Beautiful Numbers Game blog). For the purposes of World Cup predictions this late in the tournament, I’d paraphrase my colleague’s take on data: “Sometimes even with data, you’re just another guy with an opinion.” Or: there’s nothing like watching teams beat the odds. And my money is still on Brazil, and my heart is rooting for Germany.

NYT/MatchAnalysis World Cup Rankings

Here's another very cool little gadget from the folks at my favorite newspaper and Match Analysis.

Set Them Free: Democracy and the 2010 World Cup

The only real consolation I have after yesterday’s defeat of the U.S. team by Ghana in the Round of 16 is that a democracy won. I know; this isn't saying much. After all, it was bound to happen since political scientists classify both Ghana and the U.S. as democratic states. And as a fan of Germany, the fact that a kid from Berlin (Kevin-Prince Boateng) who didn’t even make the German squad scored a goal for Ghana hopefully bodes well for Germany’s match against England.

But back to democracy.

Round 2 Matches and 2010 World Cup Quarterfinal Predictions

So far, 7 of my 8 predicted Quarterfinalists are still in the tournament (Italy being the exception). So here's my preview for the Round 2 matches starting tomorrow, based on teams' predicted odds of making the Elite Eight (recall that the prediction models are based on FIFA ranking and World Cup experience, and that World Cup experience was an especially powerful predictor of making it to the quarterfinals).

We'll have to see whether the real world behaves as these numbers would predict. But I'm keeping my fingers crossed!

World Cup Draws: Final Numbers Update

So, it turns out that 2010 was a very normal year as far as overall numbers of draws are concerned. Take a look:

\

After an initial burst of draws in first matches mentioned in an earlier post (which can be seen in the far right columns below), second matches ended in fewer draws than previous tournaments in the 32-team era, while the incidence of third match draws was pretty much the same as before.

So, in the end, lots of first match draws, followed by fewer second match draws, resulting in an overall percentage of matches ending in draws that was almost identical to previous years (around 30%). While each World Cup has its own rhythm, some of the results follow a familiar beat.

Hugging It Out At the World Cup: Teams That Touch More Win More

A couple of days after Robert Green’s goaltending mistake in the England-USA match, I was listening to BBC Radio 5’s iconic 606 football phone-in show. Fans’ reactions were pretty negative – ok, they were ranting – pointing out the various mistakes players and the coach had made before and during the match.

The one comment that stood out for me, though, was made by a guy from Liverpool (let’s name him Gary), who didn’t have much of a problem with how the team played or who was playing. Gary thought the draw was fair, all things considered. But he faulted the team for one specific incident that he thought didn’t bode well for England. He noted that, right after Green made the game-changing mistake, not a single member of the English team came over to him to tell him it was ok, put his arm around him, or to pat him on his behind in a show of encouragement and shared pain. “They just left him out there”, Gary said (I’m paraphrasing). At least the captain, but any of his teammates, should have come over, Gary thought, to communicate that we all make mistakes, and that they win and lose as a team.

Crime and Punishment: Red Cards at the World Cup

After watching the confetti of cards in the Germany-Serbia match and Harry Kewell’s sending off in the Australia-Ghana match, I started to have this feeling that things didn’t use to be this strict. Maybe I’m getting old (ok, not maybe, but still), but I remember many times when someone would foul a striker looking to score or commit a handball in the box without getting sent off. What happened?

Klose and Kewell of course weren’t the first ones to be sent off; they are just the latest in a long line of players (142 before this year’s tournament, to be exact), starting with Peru’s Mario de Las Casas against Romania in 1930. But watching this year’s tournament, I have had this nagging sense that there’s a strictness to the game that seems more rigid and unforgiving than before.

So, to indulge my “zero tolerance” hypothesis, I took a quick look at discipline statistics (red cards only) to see how many cards have been given in any one tournament, and how many were given, on average, per match played.

Here’s what you see. The first graph is the total number of cards, and the second graph calculates the average number of cards per match played (and thus adjusts for the fact that the number of matches played in the tournament has expanded over the decades).

What's With All the Draws?

My wife tolerates the soccer obsessed family (of one big and two little boys) we have turned into this month by finding her own ways of enjoying the World Cup. I think her favorite moments in any match come right before and after play. She enjoys the close-ups of players singing their national anthem (I think she has a secret rating system for varying levels of handsomeness). Gratuitous shots of David Beckham are appreciated, too. And, of course, after the match, when the players take off their jerseys, she makes sure she knows who is who on the field.

So given her relative lack of close attention to actual play, I was surprised to hear her ask last night after the England-Algeria match: what's with all the draws? To answer the question with numbers, I've updated the numbers/percentages of draws as a proportion of first round matches so far (as of this morning before the Netherlands-Japan match). Take a look.

I think she's on to something. About a third of all matches have resulted in draws - and low-scoring ones, typically. So far, the 2010 World Cup is the tournament with the highest percentage of ties in the 32-team tournament era. And, my spouse agreed with Capello that David James is preferable to Robert Green and she, too, was completely unimpressed with Rooney. I'm just not sure if she was watching before, during, or after the match.

Do Democracies Win More?

The fact that teams that qualify for the World Cup typically are more democratic than the average country around the globe suggests that democracies might be better at soccer. And, wouldn’t you know it, the two most democratic of the six African teams in this year’s World Cup (Ghana and South Africa) won or drew their first round matches, while the undemocratic ones lost three (Nigeria, Cameroon, Algeria) and tied in one (Ivory Coast). Is this is just a fluke or is there something more systematic going on in the data?

Let’s take a look. 

The United Nations of Germany

“This is not your grandfather’s Germany”, I heard the (soccer) journalist Franklin Foer say this morning (June 14, 2010) on MSNBC’s Morning Joe show when asked about the Germany-Australia match the night before. The Germans had played a great match and played with the kind of flair not typically associated with German World Cup play. According to Foer and other commentators, some of this has to do with players like Mezut Özil and Lukas Podolski.

In the U.S., we are used to people with global ethnic lineages. Nobody bats an eye at Jozy Altidore's Haitian background or the fact that Stuart Holden was born in Aberdeen (Scotland, that is). But to the dyed-in-the-wool fan of Germany’s soccer heydays, last night’s lineup is nothing short of astonishing. And, I would add, really fun to watch. No one is surprised to see names like Müller, Schweinsteiger, Badstuber, or Friedrich in a German lineup. But Özil, Podolski, Gomez, Khedira, Cacau?

Of the 14 players on the pitch last night against Australia, 7 (or a whopping 50%) had a non-German connection, either through parents who are not ethnic Germans or because they weren’t even born in the country and are naturalized German citizens. Of the squad as a whole 11 of 23 (or almost 50%) have a non-German connection.

Day 1: More Draws Yet To Come?

After today’s 1-1 draw between South Africa and Mexico, my colleague Allen stopped by to chat about the tournament. And wouldn’t you know, Allen had picked a draw between these two. Naturally, I envied him for his foresight (actually, I cursed him under my breath - in the most collegial way of course - since I had picked Mexico to win). He also casually mentioned - what you might call an instinctive theory - that the number of draws should be smaller in the first matches and most common in the third (and last) matches of the first round (remember, each team plays three matches in group play). The idea would be that teams play to win in the first match, and might be more willing to be strategic toward the end and take a draw if it takes them securely into the second round. Given his excellent pick in the South Africa-Mexico match, I thought Allen might be on to something.

So when I watched France drawing Uruguay 0-0, I really started to wonder: how common are draws at different stages of group play? Well, it’s easy enough to count and calculate percentages of draws during group play. So I did and here they are, for the 1998-2006 World Cups.

The first graph shows the actual number of draws in the 32-tournament era; the second graph calculates the number of draws as a percentage of all draws in first round play.

Turns out, if anything, more draws happen in the middle match of the first round. Between 1998 and 2006, there were 13 draws in the first match, 16 in the second, and 11 in the third match of the first round. And each year, the smallest number of draws happened in the last match of the first round.

As the percentage of all draws in the first round (remember, there are no draws in the second round), 40% of all draws happened in the middle match, followed by about a third of draws in the first match, and only 27.5% of all draws happened in the third match. But these differences seem marginal (and remember, they're based on data from only 3 tournaments). So, draws seem to be part of the dynamics of the first round every step of the way. Given that the number of first match draws has hovered around 4, this means that we should see (roughly) 2 more draws in the first matches to be played this weekend. Let's see if this tournament "behaves" as others have. After the USA-England 1-1 draw today, I'm starting to wonder ...

A Picture's Worth a 1000 Words: The Soccernomics Model in Color

Courtesy of the nifty A Beautiful Numbers Game blog, here's the Kuper and Szymanski Soccernomics prediction model, explained and applied to this year's World Cup. This beautiful graphic was produced by Section Design and published, with data from Kuper and Szymanski, in the June 2010 issue of Wired UK magazine. Zach Slaton from A Beautiful Numbers Blog has applied the model to the England-USA match on Saturday. Take a look.

And in case you can't see it, here's another version of the graphic. Who said only the game was beautiful?

PS: The predictions shown in this model are NOT my predictions; they're simply an application of the Soccernomics model to this year's field.

Democracy and the World Cup

When South Korea kicks off against Greece on June 12, 2010, I’ll be as excited about the match as I’ll be about the symbolism of the venue: Nelson Mandela Bay Stadium. Bringing the tournament to Africa may be a rational calculation on the part of FIFA to fill an obvious hole on the World Cup marketing map. But its symbolic value is hard to overlook: the World Cup is held in a country that has transformed itself from a pariah of the international community into a functioning democracy – surely one with lots of problems, but a democracy nonetheless, and that is something South Africans can rightly be proud of.

South Africa’s transformation from an apartheid state to a functioning democracy is one of the great stories of the last century. In this, it has been part of the world’s continuing transformation toward a more democratic planet.

As the first graph below (courtesy of political scientist Carles Boix) shows, democracy has been on the rise over the last two centuries. But that doesn’t mean its rise has been smooth. While there may be more democracies today than ever before, the rise in democracy as a way of running countries was particularly steep between 1985 and 1920, and then again since the 1960s, while the period between 1920 and 1945 saw a significant decline in the number and proportion of democracies around the globe.

Incidentally, the first significant wave of democratic development in the late 1800s coincided with association football (soccer) taking root, mainly in Europe and Latin America.

Is this a fluke, or does the World Cup reflect what is happening in the world – that is, the level of democracy or changes in democracy?

The Best World Cup Calendar So Far

This has nothing to do with numbers, but if you're trying to keep track of the World Cup schedule, this cool little gadget is worth checking out:


http://www.marca.com/deporte/futbol/mundial/sudafrica-2010/calendario-english.html

World Cup History By The Numbers

Click here for a "best of" compilation of some of my recent World Cup statistics calculations, courtesy of the New York Times Goal blog.

History and the Technology of Playing World Cups

The past has a powerful hold on countries’ collective memories, and the history of the World Cup is no exception. Who could forget the “Hand of God”; penalties missed, saved, and not given; or the goal that wasn’t (1966 anyone?!)? Many teams have proud World Cup traditions, while some are in South Africa for the first time to make their own history.

But having a sense that the past matters to people and to the players doesn’t mean that we (a) know for sure it does, or (b) exactly how it matters. Social scientists call these empirical and theoretical questions. The empirical question is: Does the past affect what happens today? The theoretical question is: why would it?

Let’s talk empirics first. Looking at the history of World Cup competitions, it is clear that some of the same teams consistently come out ahead. Since 1930, about 80 countries (give or take, given the vagaries of 20th century global politics and shifting boundaries) have competed in the tournament. Of these app. 80 countries, only 7 have actually won the World Cup (Argentina, Brazil, England, France, Germany, Italy, and Uruguay). And it doesn't seem to matter whether we look at all World Cups or more recent history. For example, in the 10 World Cups held since 1970, only 5 countries have: Argentina (2), Brazil (3), France (1), Germany (2), and Italy (2).


On the flipside, futility is pretty common. Of the roughly 75 countries that have qualified for the World Cup since 1970, about 30 made it to the final tournament only once, and another 10 have participated a total of two times (unsurprisingly, these are also the teams that do not make it to the World Cup final.)

So does it help teams if their country has played in previous World Cups? 
On its face, it is not obvious that it would. After all, teams and coaches change, and the team that is playing in this year's tournament is not the same team that played in, say, 1998. So the fact that a country has played in the World Cup before can’t help players if they haven’t been there.

So maybe the fact that having participated in the past is a good statistical predictor of this year’s success is just coincidence (some of these kinds of correlations are also called "spurious": both are affected by the same underlying factors). Or it could be that part participation and today's success measure the same underlying thing: team quality. Maybe these countries are just better at playing the game.

Finally, one common argument I've heard is that, somehow, a country’s history in the World Cup becomes a motivating factor for players to perform up to expectations, overcome past failures – what have you. I doubt that this is particularly salient; today’s players are highly skilled professionals who are trained to be adept at adjusting to a variety of environments and play at the highest level regardless of what happened yesterday.

I suspect that history could be about something else: maybe these countries – that is, their respective national soccer associations – know something about what I would call the “technology” of playing tournaments and achieving peak performance at just the right time that other countries do not.

Think about it. Participating in a World Cup is not just about finding 11 great players, ship them off to a foreign country, and watch how things play out. Instead, think about teams as formal organizations akin to any other organization. It just so happens that these organizations consist of players, managers, coaches, doctors, logistics and equipment people, security personnel, cooks, etc. (World Cup teams are quite a traveling circus; I was reminded of this when I recently read a  story about the German team’s hotel woes and the extensive amount of planning and personnel involved in making the World Cup a success.)

So why would the experience of having played in 2002 in Japan/Korea help a team today? It might help because the technology of playing international tournaments is quite specific and is acquired through a process of learning in the form of trial and error. Sure, you can play lots of matches against high-level competition, and you can do so in faraway lands. But assembling a team of star players who do not usually play together to succeed in matches in short succession under high pressure with different first and second-round tournament formats in unfamiliar environments against unfamiliar opponents while being cooped up in a hotel is not something you can really read about in books or learn by playing qualifying matches in your geographic region. Instead, it helps to have done it before.

There are two kinds of knowledge that any organization needs to succeed. These are things you “just know” – things that aren’t written down, but you know them and can rely on them (students of organizations call this  “tacit knowledge”). And then there are things that are written down and disseminated among the organization (so called “explicit knowledge”). Organizations like national soccer teams that have participated in a World Cup (or two or three or four ...) undoubtedly have vast amounts of both tacit knowledge and explicit knowledge to handle the multitude of moving parts that lead to tournament success.

Every four years, commentators and fans look into the past to make prognostications about the future. But learning the right lesson from history is important: for a national soccer association, having played in a World Cup means more than the heartbreak of defeat and the joy of victory. That is, history isn’t necessarily about what is spectacular and memorable to fans. Instead, it could be about the mundane to soccer organizations: which hotel to pick, how to form a team on short notice, how to cope with the pressures of unfamiliar environments, and so on. As the results from statistical analyses show, past experience may matter more in the second round of the tournament. So, perhaps the lesson from history should be this. Those who simply count on history to repeat itself would be well advised to cheer their team from the word go. Experience clearly helps, but only if you make it to the second round.

Does Group Strength Matter?

How does playing in a tough group affect a country's chances of success in the World Cup? Well, if you're North Korea and you're paired with Brazil and Portugal, it's going to be a short excursion to South Africa. But does it matter if North Korea plays Brazil or Argentina or Germany? Probably not. I just don't see how they can advance to the second round. Weaker teams will struggle with better teams.

So, does the overall quality of a group matter for a team's survival in the tournament in any other way? If we already know how good individual teams are or what experience countries have in the World Cup, does it help to know how strong a group is if we want to predict that country's success in the World Cup?

Measuring Group Strength: What You See Depends On Where You Look

There is no Group of Death. Ok, perhaps that’s a bit strong, but I wanted to grab your attention. But really: it’s not clear that there’s a group of death. Or to put it more neutrally: the quality of groups and their competitiveness are highly similar across the board.

In eager anticipation of the first round matches, a lot of discussion has focused on groups that are seen as particularly tough. Terms like “group of death” suggest that some groups are difficult to survive in the first round. In my subversive mind, this raises a question: Are groups all that different – really?

What does it mean to say there’s a group of death or that groups differ in strength? To me, there are two relevant dimensions: first, the average level of team quality in a group – or: are there groups with better teams? But since 50% of all teams will make it out of the group into the second round, thanks to the design of the tournament, perhaps the more important question is how evenly matched groups are, making it less predictable who will emerge from the group in stage 2 of the tournament.

World Cup Team Histories: Wins and Goals Per Match

To compare how this year's teams are matching up in terms of their World Cup histories, I've put together a simple table of wins per match (WPM) and goals per match (GPM), along with a record of how many World Cups countries have participated in. Wins and goals per match measure how many wins or goals, on average, a team has produced per match. For example, if a team scores 2 goals in 4 matches, its goals per match ratio is 2 divided by 4 or 0.5. If it scored 4 goals in 4 matches, its goal ratio would be 1 (4 divided by 4).

There are two sets of calculations: World Cup history since 1930 for all teams participating in 2010; and records since 1990 (the past two decades). Twenty years is admittedly arbitrary, but I figured it would produce evidence of success in the most recent era of World Cup play. Enjoy!

Predictions III: Quarterfinals

In previous posts (here and here), I have estimated the effect of current form (with the help of FIFA rankings) on the odds of succeeding in the World Cup (in the 32-team tournament era since 1998). I found that current ranking was a good indicator of succeeding in the World Cup. But one question is whether this effect remains once we account for the fact that teams have varying levels of World Cup experience. Does past participation matter? And if it does, is it a better predictor of World Cup success than current form, or do they cancel each other out? Put simply: who wins out – the present or the past?

To answer this question, I estimated models of first round success and making it to the quarterfinals, using data for the 1998, 2002, and 2006 World Cups (for more information on these, check my previous post). But this time I also wanted to see if the number of times a country has participated in the World Cup in the previous 20 years mattered (I know; 20 years is an arbitrary number, but I chose it under the assumption that the use of knowledge decays over time).