Monday, February 13, 2012

How Efficient Are Player Salaries in Major League Soccer? Data From 2011

By Benjamin Leinwand and Chris Anderson

When it comes to pay, not all positions are created equal. In fact, we have long known that strikers command a premium for their services. So it should come as no surprise that Major League Soccer is no exception. In 2011, average earnings were distinctly tilted toward the offensive side of the pitch. Consider this: the average MLS forward earned $183,060, while midfielders made $141,594, defenders $118,558, and goalkeepers $86,208 – less than half of what strikers took home.

Of course we can argue these numbers – does it make sense to compare goalkeepers, most of whom ride the bench – to forwards – since a lower percentage of goalkeepers played real minutes compared to other positions? Others could point out that the averages may be skewed, given that every designated player in the MLS is a midfielder or a forward. But even if we leave goalkeepers to the side and even if we look at median rather than average pay (to eliminate some of the distortions produced at the very high and low ends), the overall pattern holds up. The table below shows some of the details about which positions played and which positions were paid in Major League Soccer.

The data show significant skew by position. Taking median salaries, midfielders earn roughly a $6,000 premium over defenders, and forwards a roughly $8,000 premium over midfielders, and these numbers are significantly higher when we consider averages. Any way we slice it, forwards are the most valued position by compensation, followed, with pretty significant gaps, by midfielders, defenders, and goalkeepers. While this is not surprising, it is made more notable by the fact that forwards played on average the fewest minutes of any position.

Is this market efficient? As Billy Beane showed the baseball world, sometimes the players who can create the most wins aren’t paid the most. In fact, the inefficiencies discovered by Beane are what made him and the A’s so successful and his story so interesting. By overpaying for some skills and undervaluing others, is it possible that general managers in the MLS are making the same mistake general managers in baseball were making?  

We thought we’d take a look. 

A few days ago, we reported what the Castrol Index – a metric developed to measure individual player performance – could show us about how MLS teams were playing on the whole. The Castrol Index proved to be a very good tool for sorting out the better and worse teams in Major League Soccer. When added up for a team as a whole, teams that performed better on the Castrol Index also produced more points and wins over the course of a season.

Today, we take the next step and examine how the performance of different positions as measured by the Castrol Index affects the performance of teams on a season-long basis.

To start, here is another table showing the summary statistics of how teams performed across different positions in the Castrol Index. Using a similar methodology as in the last post, we took each player’s Castrol Index Score (only for those players above the minimum of 3.89) and multiplied it by the number of minutes played to get a player’s Total Castrol contribution. To determine the average team values for every position, we broke down each team's Total Castrol Contribution by the 4 positions, and divided it by each team’s total minutes played by position. The average performance values for each position are shown in the table below, along with the standard deviation, a measure of how dispersed performances were.

The data show that defenders had the highest average Castrol ratings last year. According to the creators of the Castrol Index, this means that defenders were more likely to act in a way that led to a goal for their team or that prevented a goal for the other team than any other position, while goalkeepers – the position with the lowest Castrol rating – were least likely to do so. Keepers also had the least variation over this metric, indicating that most keepers performed very similarly. Leaving goalkeepers out for the time being (and for future analysis) because of potential problems with properly measuring their performance, what stands out to us is that forwards have lower performance ratings than both midfielders and defenders.

As we know, some teams clearly had better forward play than others last year, but did better performance at the forward position produce more points for the team? That is, if we want to value performance appropriately, we would want to see that the highest paid positions also will produce more value on the pitch. Translated for our purposes here, we want to know if performance by forwards does a better job predicting league points than performance by players in other positions, or do they all contribute equally?

Let’s start to answer this question with the most expensive and most variable position, forwards. When we calculated correlations between a team’s average forward play performance and points, we found a positive but modest correlation of .258, and a .254 correlation with team wins. Furthermore, when we regressed team points or wins on team forward Castrol Indices, the results were actually insignificant statistically speaking, strongly suggesting that data do not allow us to conclude whether better forward performance was actually associated with better standings! 

That may be hard to believe for some. In case you’re curious where teams line up on forward performance and team success, below is a chart plotting forward Castrol ratings by teams against points, along with the league averages along both dimensions (shown by the vertical and horizontal lines). And the pattern is not particularly pronounced; some teams had excellent play by their forwards – RSL and the Red Bulls, for example – but were far from topping the league on points. In contrast, some teams’ forwards performed woefully – Colorado and Philly come to mind – yet did just or about as well.

So what exactly can explain the overall performance correlation we documented for team performance and points? Let’s see if midfielders’ performance ratings can account for this. 

When we repeated the analysis for midfielders, we found that midfield performance was a much better predictor of team points. The correlation was a healthy .64 with points and .62 with wins, and these correlations were statistically highly significant. Better midfield contributed significantly more to more points in the league table than forward play. The results suggest that teams that manage a 1 point increase in Castrol Index in midfield performance should be expected to win 17 extra points. One point in the Castrol Index is a sizable jump, though, but even midfielders performing one standard deviation better than average should help produce about 6.8 extra points, or more than two additional wins per season. Below is a chart displaying the results, along with a regression line in green.

But even this picture isn’t entirely clean. A number of clubs bunch in the upper right hand corner, with Toronto, Vancouver, and New England somewhat lonely dots in the lower left. As importantly, clubs like LA Galaxy and Seattle over performed relative to the average club and what the Castrol ratings would have predicted, while San Jose, Chivas, or New England underperformed.

So now that we know midfielders are more important than forwards – statistically speaking – let’s take a look at defenders. And this is where we see the most interesting pattern. The data show that defender performance is even more predictive of league success than midfielder performance. The correlation between a team’s defensive Castrol Index and points is .74, nearly three times that of forwards. Similarly, the correlation between defender play and wins is also very high at .72. This link is statistically highly significant, translating into 1 extra Castrol point expecting to produce an extra 13.9 extra points a season. At first blush, this looks lower than the contribution by midfielders; but in fact, since the deviation in Castrol performance among defenses is greater than that of midfielders, playing one standard deviation better on defense will lead a team to expect an extra 7.8 points, compared to 6.8 for midfield.  Here is the graph of Castrol performance by points for defenses, so far the most important position.

The relationship is more straightforward than for either of the other positions. Teams line up very nicely from lower left to upper right, revealing that better defenses are associated with more points. What is more, teams’ deviations from the regression line are generally small. But what we see again is that the best teams in the league points wise – LA and Seattle – managed to outperform what we would expect of them based on the Castrol index, while the worst – New England and Vancouver – underperformed somewhat. And Toronto – with the worst defensive performance in the league on Castrol points – managed to punch above their weight and defy what the Castrol Index predicted. 

The regressions reported so far were conducted with points as the dependent variable and each team’s average Castrol score at each position as the lone independent variable. The real world isn’t quite as neat, in part because some players were listed at more than one position – defensive midfielders who also play defense or offensive midfielders who also are listed as strikers – so including all 4 positions in a single regression means counting some players twice in their different roles. But when we combine the performance metrics of each position type in one multivariate regression, we find that performance by a team’s defenders was the only variable that had a statistically significant impact on season performance, contributing 11.2 extra points for 1 extra Castrol point, or 6.3 extra points for playing one standard deviation better than the average defense. This provides further evidence that something quite curious may be happening in MLS or perhaps in football generally.

Despite what the salaries might say – on the whole and over the long run of a season – better forwards don’t produce points or wins. Instead, better defenders are the easiest and surest way to win more games in MLS, followed by better midfielders. Put simply, the joint performance of players in the highest paid position, forward, doesn’t matter, statistically speaking. Major League Soccer general managers may take note; they may have been paying the wrong players if what they are after is success on the pitch rather than shirt sales. If they want to win, they may want to invest in some great defenders and save themselves some cash in the process.

This analysis might surprise you, and should be treated with the necessary amount of caution. To put the results in perspective, several caveats are in order. First, there is the small sample size of only 18 teams based on only one season of play. Second, the two positions that have a significant impact also happen to be the two positions that play the most minutes. Third, some players are listed at two positions, and we are unsure how the Castrol Index accounts for that. This points to a related issue about our measure of performance. And finally, perhaps MLS is unusual - given its somewhat different demographic structure or the fact that it has a salary cap, designed to ensure competitive balance.

We also have assumed for the purposes of this analysis that the Castrol Index is indeed a good measure of how effective each player is on the pitch. This assumption may be reasonable (based on our prior analysis) or prove false, or there may be bias in the rating and this bias may be associated with player position. For these reasons, one could just as easily argue that the Castrol index proves that general managers are incompetent because pay doesn’t go with performance, or that the market shows that the Castrol Index is obviously wrong because it doesn’t capture what the market seems to value. 

At the moment, we are not completely convinced by either claim. Statistical revolutions in other sports have clearly shown that team decision makers routinely pay too much for the wrong kinds of players and perhaps this is one such example. But we also know by now that it has been more difficult to analyze what happens on a soccer field than a baseball field. Because “perfect stats” don't exist, it is probably best to maintain some skepticism and use all the information available to make the best decision. Of course having better players at any position is helpful, and saying anything else would be, well, lunacy. Nonetheless, if we were in a position to pick between a forward and a defender who are both equally talented, the results show that the defender is probably the better value in MLS, given the salary differentials. Whether fans in the stands feel the same way is another question for another day.