Why Statistical Analysis Makes Sports Fans Unhappy

Imagine you and your friend sat down to discuss a simple basketball question:

Who is better—Carmelo Anthony or LeBron James?

Basketball fans could spend hours debating a question like this. But let’s say you invited me to come along for the discussion. At first, this might seem like a good idea. My co-authors and I have published dozens of academic papers studying the economics of sports, many of which are focused specifically on professional basketball. Studies examining basketball often require measures of player performance, and such measures might seem fairly helpful if one wished to evaluate the merits of Melo and LeBron.

However, inviting me to the conversation creates a very big problem.

But before we get to the details of this problem, let’s briefly discuss what is meant by “measuring player performance.”

Players take various actions on the court (scoring, rebounding, etc.) For our research, my co-authors and I needed a method that could connect these actions to team wins. Existing measures—like Player Efficiency Rating, Player Impact Estimate (found at NBA.com), or Win Shares—fail to empirically justify the weights employed in their statistical evaluations of players. The basic question—what is the value of a rebound? (or steal, assist, etc.)—is not actually answered by these measures. Consequently, you can’t claim that these metrics actually measure a player’s productivity.

The failure of these box-score measures has led some people to turn to plus-minus models. However, these models are often based on suspect empirical methods, or offer suspect empirical findings. Also, they are also not capable of telling us why a player is productive. So these measures don’t work very well, either.

Given the problems with existing measures, I developed the Wins Produced model. This model uses standard econometric techniques to measure how many wins the statistics tracked for the individual player are worth.The purpose of this approach is to address such topics as the efficiency of decision-making, the impact of contract status on productivity, the role managers play in the performance of workers, and the existence and extent of racial discrimination. But the model can also be used to determine how Carmelo Anthony compares to LeBron James.

***

So let’s say you and I decided to discuss this issue. And let’s imagine you were a huge Melo fan. You might start the conversation by talking about Melo’s ability to score, and/or the failures of his teammates.

And then I would say something like this:

“Prior to this season, LeBron James had produced 182.2 wins in his career and his career Wins Produced per 48 minutes (WP48) was 0.265. In contrast, Carmelo Anthony — before this season — had only produced 37.1 wins with a career WP48 of 0.062 (average WP48 in the league is 0.100). Therefore, LeBron is immensely more productive.”

We could go on for a bit. I could note—as I did at the Atlantic last May—that Melo’s teammates have actually produced more wins than LeBron’s teammates. And I could add that scoring totals are not useful in measuring a player’s impact on wins, since a relatively inefficient scorer can score more points by simply taking more shots. Since inefficient shooting does not help a team win games, inefficient scorers are not that valuable.

You could respond by saying that you don’t like how I am measuring a player’s contribution to wins. You could say I don’t take into account interactions between teammates (although I do). Or that I don’t consider coaching (which I have). Or I ignore team defense (which I don’t). But if you take this approach, you are no longer discussing the merits of two players. You are now debating research methodology. And very few people actually want to spend hours in such a discussion.

This is why a model like Player Efficiency Rating never disappears. As I noted years ago, PER is simply not a very good measure of player performance in the NBA. And I have repeated and expanded upon this observation over the years. But explaining to most people why this model has problems is not a useful exercise (and again, I have tried for years). Most people have no idea which research methods are good or bad. Furthermore—and this is the most important factor—they really don’t care.

What they really want to do is debate sports. And statistical analysis simply ends that debate.

All of this reminds me of a scene from the classic book by Douglas Adams, The Hitchhiker’s Guide to the Galaxy. A computer is about to give the answer to “Life, The Universe, and Everything.” Right before the computer is turned on, philosophers break into the room, and argue that the computer is about to put them out of a job. As the philosophers put it,”what’s the use of our sitting up half the night arguing that there may or may not be a God, if this machine only goes and gives you his bleeding phone number the next morning?”

Sports fans are just like these philosophers. And statistical analysis is just like the giant computer in the Adams classic. Sports fans want to debate the relative merits of players. And stat analysis comes along and puts the sports fans out of work.

To illustrate the speed at which statistical analysis can work—and the amount of fun it can kill–imagine you and your friends wanted to debate the merits of the following players:

If you went to boxscoregeeks.com (where one can find Wins Produced numbers back to 1977), or just clicked on the above links, you would see the following answers:

  • MJ produced more than Kobe
  • Magic produced more than Bird
  • Stockton produced more than Isiah
  • Sir Charles produced more than the Mailman
  • The Worm produced more than the Human Highlight Reel

None of the comparisons are really the close. In each case, the player who is listed as more productive is much more productive.

None of this really helps a sports fan. Comparing players on subjective terms can lead to hours of lively back-and-forth. But once you turn on the computer, an objective answer is provided, and the fun ends. Talking sports is what fans love to do, and numbers kill the conversation.

So, if you have ever wished us stats people would go away…. that’s probably not going to happen. In my defense, I really didn’t set out to answer the sort of questions sports fans most frequently talk about. The Wins Produced model was designed to answer research questions in economics, but it also helps us answer the relatively trivial ones listed above. And when I see people struggle with these questions… well, trivial questions or not, researchers love to find answers. Unfortunately, I’ve come to think that answering these questions isn’t really helping. Sports fans want to enjoy the dialogue, but the answers from statistics end the debate, and kill the fun.

***

David Berri is a professor of economics at Southern Utah University. He was the lead author of The Wages of Wins and Stumbling on Wins, and has numerous academic publications on sports and economics. In addition, he is a past president of the North American Association of Sports Economists, and continues to serve on the editorial board of both the Journal of Sports Economics and the International Journal of Sport Finance. He has written for a number of popular media outlets, most recently Time.com. You may follow him on Twitter, even if you still think Carmelo is better than LeBron.

7 Comments

  1. Post By James Perkins

    Here’s Calderon’s WP48 for 3 seasons: 2011-12 (.221); 2012-13 (.260); 2013-14 (.146). Yet according to unadjusted net plus/minus the 3 teams he played for over the those 3 seasons were all better when he was on the bench. Raw net plus/minus is not a model – it’s just a fact. Hard to believe a guy who’s elite according to WP48 would make a team worse when he’s on the court. Methinks something is terribly wrong with the WP model.

    1. Post By Joamiq

      This is a pretty easy critique to dispense with – raw net plus/minus can’t actually tell you if a team is better or worse without a player. Their scoring margin might have been better with him on the bench, but it’s easy to imagine other reasons why that might be other than that he’s not that good. E.g., if a player usually plays with the starting unit (and against other starting units), and his team’s starting lineups are not good compared to other team’s starting lineups, then his team’s scoring margins with him on the floor won’t be very good, and if his team’s benches are strong compared to other teams’ benches, then their scoring margin might be better without him on the floor. But this would have little to do with how good the player is.

      That might actually explain it for Calderon – he was the best player on a really bad Toronto starting lineup in 2011-12, and that team’s bench surely did better against other backups than its starters did against opposition starters (the Barbosa-led bench had Toronto’s 3rd and 4th leading scorers – 2nd and 3rd in pts/36). And in 2013-14, he was part of a solid Dallas starting lineup, but the bench was very good, with one of the league’s best sixth men in Vince Carter and a center (Brandan Wright) who probably outplayed the guy ahead of him (Samuel Dalembert) (and also he was playing for a coach whose strengths are going to do more to enhance a bench’s effectiveness than a starting lineup’s). (not looking into 2012-13 since he got traded in the middle of it)

      So, I can’t say for sure that WP is good, but I can say your argument against it is not.

      1. Post By James Perkins

        Calderon’s adjusted plus/minus is negative two of those three years according to 82games.com. Calderon’s WP48 over his entire career is .200 (twice the average) versus a net minus 1.3 over his career according to BRef. Wall, Curry, Lawson, Westbrook, Teague, Lowry, Conley, Williams are all plus over their careers. Last season, real plus/minus (adjusted) ranked him 46 out of 72 among PGs while WP48 had him at .148. A week ago WP48 had Calderon this season at .100 while real plus/minus ranked him 71 out of 80.

        The adjusted and unadjusted +/- stats differ sharply from the WP48 metrics. I put more faith in +/- having watched Calderon quite a bit over his career. He is an abysmal defender and he only creates on offense thru PnR, being one of the worst at breaking down defenses and initiating sequences which eventually lead to open shots (often on the second pass). Additionally, his usage tends to be much lower than top 10 PGs (his WP48 typically places him as 6th or 7th in rank in any given season). Both defense and limited shot creation capability are not really captured in box score at individual level in contrast to shooting efficiency.

        The bottom line is plus/minus stats almost certainly give a more realistic appraisal of Calderon’s effectiveness.

    2. Post By Xavier Q

      You have to understand the inherent limitations of plus/minus, which even the creator of +/- points out but people who quote +/- seem to ignore. +/- cannot substantially separate a player from his teammates without a decade of data. The case study was Derek Fisher. By +/- he was a fantastic player. But essentially it was because he was a starter with Kobe Bryant on contending teams. Throughout his career with the Lakers, +/- cannot tell the two of them apart. But strictly by that measure, Derek Fisher was a star player even in the twilight of his career BECAUSE HE STARTED ON A CONTENDER WITH KOBE BRYANT.

      The only way for plus/minus to show you anything in 3 seasons is if when Calderon goes to the bench, NO OTHER PLAYER ON EITHER TEAM EVER CHANGES. That’s the problem with it, for any instance of play there are 9 uncontrolled variables on the floor at the same time (4 players on your team, 5 on the other team).

      Looking at unadjusted plus/minus is like just looking at points scored and saying, “Oh player X had 25 points while player Y on the other team only had 20. Player X had a better game.” It’s extremely simplistic.

      1. Post By James Perkins

        But I gave you exactly what you requested: a DECADE of data. Calderon is a net minus 1.3 over a decade, almost 20,000 minutes! Then why should you be surprised that he’s a net minus over the last 7,500 minutes (4 seasons including this one or 6,500 minutes excluding the current season) playing for 3 or 4 different teams. That’s a large enough sample size on its own. And he fares just as bad on adjusted plus/minus. I even explained why he’s a net minus player but people tend to focus on what is most visible – shooting in his case at which he is elite. WP48 overrates Calderon because it effectively overrates his defense, overstates his efficiency on relatively low usage, and is unable to factor in his limited shot creation capability since it is not reflected in the box score.

  2. “… However, inviting me to the conversation creates a very big problem. …”

    Of course it will: based on this writing, Dr.Berri will boorishly insist that his notion of player value is the only valid one.

  3. Post By James Perkins

    Dennis Rodman or Michael Jordan?
    John Stockton or Michael Jordan?
    Charles Barkley or Michael Jordan?

    Hmm.. Michael Jordan was worse (produced less according to WP48 than Rodman, Stockton, and Barkley).

    Now read this Dave Berri quote in that light: “Unfortunately, I’ve come to think that answering these questions isn’t really helping. Sports fans want to enjoy the dialogue, but the answers from statistics end the debate, and kill the fun.”

Comments are closed.