wutang112878 said:
You probably know better than I do, but do you have any idea how it handles small sample size? So say you have a stable starting lineup, well those 5 guys should be playing something like 80% of their minutes together so how much data do you really have to run regression to really isolate the players true defense when not playing with those guys. And to my knowledge it doesnt address the value of the opponent, so say having a +5 differential against the 76ers is probably the same as having a 0 differential against the Spurs. I'm not just trying to poke holes in the thing, but sometimes it seems to be touted as 'look this really, really shows the effects of a player' and the data for some guys just doesnt pass the smell test. I mean Garnett was allegedly one of the best defenders in the league last year by this metric and when you watch him play that just doesnt seem to match what you see.
1. This "starting 5" issue just isn't that prevalent. Between injuries, fatigue, new guys being inserted in the lineup, it's rare to have a starting 5 for very long in the first place. The Celtics' top lineup played ~10% of all minutes together. The Spurs' played ~7.5%. The top number I can find is the Trailblazers, who played about a third of minutes together, but even then the other lineups mostly consisted of similar lineups with one or two of the starters swapped out. It's not like they swapped in 5 backups all at once. It is rare to find a player with a decent sample of minutes overall who was super isolated like this. Every lineup used in the top 20 included one of those five guys, and in 19 of those, you were getting to see the impact of playing without one or more of the top five.
However, yes, if you have 5 guys who always play together, and basically never see the court without each other, you're not going to get much teased out with this stat. You'll run into sample size issues here the same as anywhere else.
2. The value of the opponent issue is addressed. It uses a method of least squares approach across the entire league. Having a +5 as the 76ers is valued as being much worse than a +0 vs. the Spurs (since the Spurs are about 19 points/game better than the 76ers), although it's not done vs. the team, but vs. the combination of players out on the court. Being +5 vs. the Spurs starting 5 is valued differently than being +5 vs. Bonner, Joseph, Mills, etc...
3. I watch a ton of basketball, for both personal and professional reasons. I watched maybe 20-25 Nets games? The difference for someone like me, who doesn't really know what he's seeing, between someone being good and bad defensively without really watching a breakdown on them is really hard to see. You're talking an few shots a game. That adds up, but given most of us aren't watching every game for every player, it's hard for me to take the "smell test" all that seriously. I had the same sense as you with respect to the Garnett, but part of that was that he was so awful offensively (which is easier to see), which made him seem old and broken down.
We know Harden is awful offensively cause we get those fun highlight packages of him chilling out on defense rather than playing. We know Fab Melo is awful at everything cause he's such a spazz that even a few minutes on the court reveal he's basically clueless. For most guys however, the difference between good and bad is a fine line, and unless you're really watching a lot of games, specifically looking for that, I think it's hard to see. Maybe you're doing that, or maybe Garnett is so bad that it's a Fab Melo thing. I dunno. I didn't see it at least. He "looked old" cause his shot wasn't falling (reflected in RPM), and cause the Nets lost a ton of games early in the season when the spotlight was on them. A meme got started of Garnett being finished. I dunno. I'm not a scout, and wasn't much of a basketball fan until a few years ago. I can't tell what I'm looking at mostly.
The biggest "unsolved" issue with RPM to my knowledge is the game situation one. Basketball is unique among American sports + soccer in that a team with a lead is likely to see that lead shrink, while in other sports, leads tend to expand (not always obviously). This is the result of some combination of starting lineups (which wouldn't affect RPM), but also strategy/effort (which would).
It's not a perfect stat, it's far from it. As PKB notes, nothing is, we should always be skeptical of "adjustments", and results that don't look right, etc... My defense of RPM is similar to my defense of Pro Football Focus - the alternatives are mostly worse. Our other defensive metrics aren't super good. Most of us don't have synergy data, and don't have detailed breakdowns resulting from it on every player. It's not like the alternative is a detailed scouting report - if you have that, definitely go with it over RPM. But when the alternative is Sportscenter top 10 plays, and the sense you get from watching games as a fan, I think RPM adds a lot of helpful context.
Your mileage may vary.
