Math in Basketball: Garbage or Not Garbage?

[slamminsammya]

Jayson Tatum's worship thread is being tainted with bickering that historians initially dated to cerca 2003 but has been found to actually be from 2021 about whether the attempt to apply statistical models to evaluate NBA players is garbage or not.

Possible topics include: Single number impact metrics - useful or not? How to measure defense using teh maths? Pythagorean theorem - does it explain Shaq?

"Our offense is like the Pythagorean theorem - there is no answer. There is no answer to the Pythagorean theorem. Well, there is an answer, but by the time you figure it out, I got 40 points, 10 rebounds and then we’re planning for the parade."​

 

[Cesar Crespo]

Math is needed otherwise how do you have a score?


Advanced metrics that measure everything all in 1 number are hot garbage at this point though.

 

[slamminsammya]

Math is needed otherwise how do you have a score?


Advanced metrics that measure everything all in 1 number are hot garbage at this point though.

Care to elaborate on why they are garbage? Is your take that all current attempts at this endeavor suck or that any such attempt will suck because of philosophical reasons prior to any implementation?

 

[wade boggs chicken dinner]

Stats suck and honestly no one can evaluate talent like me - if DA were listening to me, the Cs would have 10 more championships and 'Arsen Edwards would be scoring 20 ppg.

Seriously, stats are at their infancy so as mentioned in the other thread, so given that so many people are spending their time looking at this, it's not going away.

I think the next generation of stats will be using tracking information not box scores; hopefully they will be something I can understand and result in more consistent evaluation of players.

 

[Cesar Crespo]

Care to elaborate on why they are garbage? Is your take that all current attempts at this endeavor suck or that any such attempt will suck because of philosophical reasons prior to any implementation?

I can't predict the future but I'd guess they will all be shoddy at best for the next few years. They still haven't figured out defensive metrics in baseball. I'm also philosophically against the idea and would rather have more numbers to look at than 1. If you need someone who can shoot the ball, PER and LEBRON are going to tell you way less than a box score. A lot of the advanced stats are also black boxes where you don't know the formula and the formula even changes with time.

I despise OPS too. Is it really that hard to post avg/obp/slug? That's especially terrible because slugging isn't even a real %.

More info is better.

 

[slamminsammya]

I can't predict the future but I'd guess they will all be shoddy at best. I'm also philosophically against the idea and would rather have more numbers to look at than 1. If you need someone who can shoot the ball, PER and LEBRON are going to tell you way less than a box score. A lot of the advanced stats are also black boxes where you don't know the formula and the formula even changes with time.

I despise OPS too. Is it really that hard to post avg/obp/slug? That's especially terrible because slugging isn't even a real %.

More info is better.

This post makes me take your opinion much less seriously on these matters. Slugging isn't a %... So your issue there is terminology? Have you attempted to look at how the advanced stats are calculated or is "black box" a euphemism for "I don't care and I don't want to learn roughly what `ridge regression` means?"

 

[RorschachsMask]

I post LEBRON quite a bit, so I take some of the blame for this. My two favorite basketball advanced stats have been RAPM, and PIPM, which LEBRON uses both of for the formula, I believe. There are going to be issues with any of them, but I personally think it’s the best of the free ones.

No advanced stat is an end all, but if you use them as tool in evaluating players, I think they are useful. Raw stats in the nba can lie, you can’t take them completely at face value. I just feel like impact stats kind of bridge the gap between raw stats, and the eye test.

 

[Cesar Crespo]

This post makes me take your opinion much less seriously on these matters. Slugging isn't a %... So your issue there is terminology? Have you attempted to look at how the advanced stats are calculated or is "black box" a euphemism for "I don't care and I don't want to learn roughly what `ridge regression` means?"

It's not terminology. You are adding a real % to a stat and calling it a %. A .940 OPS could be a .320 OBP, .620 slugging or .440 OBP, .500 slugging. One of those players is more valuable than the other. They have OPS+ which tries to balance out the difference between OBP and slugging... but why not just list the OBP and slugging numbers? Is it that much harder than listing an OPS+?

And yes, I've tried to calculate stats. Sometimes you can't because there is literally no formula, or rather they aren't giving you the info you need to actually do so.

 

[Devizier]

Defensive metrics are always going to be hurt by the non random nature of team defense*. A starter is typically going to guard opposing starters. A wing is going to guard opposing wings. Obviously there are exceptions, but it’s hard to normalize (e.g. for opponent quality) with these kinds of constraints. That’s without even weighing considerations like zones, switches, team defense schemes, etc. There are some pretty clever folks working on these problems so maybe they come up with some innovative solutions.

*Offensive stats are affected too, but offensive players have a lot more control over the outcome of a possession than defenders.

 

[Jimbodandy]

I can't predict the future but I'd guess they will all be shoddy at best for the next few years. They still haven't figured out defensive metrics in baseball. I'm also philosophically against the idea and would rather have more numbers to look at than 1. If you need someone who can shoot the ball, PER and LEBRON are going to tell you way less than a box score. A lot of the advanced stats are also black boxes where you don't know the formula and the formula even changes with time.

I despise OPS too. Is it really that hard to post avg/obp/slug? That's especially terrible because slugging isn't even a real %.

More info is better.

I agree with almost all of this.

One caveat is that while the baseball metrics for defense and baserunning are certainly imperfect, they're close enough for government work. Vorp/war isn't perfect but are damn good tools. It also took like 40 years and countless contributors to get there.

Hoop def metrics and all-in-one metrics, both for rate and counting are currently complete and utter garbage. Reasons are many, but it's largely because of a lack of ability to properly regress for teammates. The O numbers have this problem too, but not as badly.

Hamfisted attempts to normalize the numbers through various blending techniques don't help whatsoever. More harm than good imo, and even legitimate statisticians use these techniques out of desperation.

Black boxes are a significant obstacle to evolving the science.

The effort, however, is not only worthwhile but also seemingly inevitable. We should support it. We're just really like 20 years behind baseball.

 

[DeJesus Built My Hotrod]

Regarding a single number metric, a price is a stand alone number that is meant to convey some general information about a good or service available for sale. The amount will not tell the consumer every attribute or feature but it makes it easy to compare things. The work being done in this space is geared in the same direction.

Prices don't tell you if this $500 phone takes better pictures than the other $500 phone and the single metrics being discussed around the NBA don't attempt to do that either - at least not that I've seen.

 

[Cellar-Door]

I think within reason single metrics can be good, though defense in particular is nowhere close to being well quantified (and it's worse in some like LEBRON where it seems almost useless) I think it's good to discuss how multiple metrics, and underlying stats see players.
There is sometimes a tendency though to go with:

 

[slamminsammya]

It's not terminology. You are adding a real % to a stat and calling it a %. A .940 OPS could be a .320 OBP, .620 slugging or .440 OBP, .500 slugging. One of those players is more valuable than the other. They have OPS+ which tries to balance out the difference between OBP and slugging... but why not just list the OBP and slugging numbers? Is it that much harder than listing an OPS+?

And yes, I've tried to calculate stats. Sometimes you can't because there is literally no formula, or rather they aren't giving you the info you need to actually do so.

You think the available statistics are bad but also think that good statistics should be expressible in a closed form expression that you can pop into your calculator?

You have a beef with OPS because people call it a %? I have never heard of anyone referring to OPS as a %, its literally called "on base plus slugging" there is no implication of a rate or anything like that. What an odd issue to take with it.

 

[Cesar Crespo]

You think the available statistics are bad but also think that good statistics should be expressible in a closed form expression that you can pop into your calculator?

You have a beef with OPS because people call it a %? I have never heard of anyone referring to OPS as a %, its literally called "on base plus slugging" there is no implication of a rate or anything like that. What an odd issue to take with it.

I do not have a beef because people call it a %. OMG. I have a beef because they are adding a percentage to a stat that is not a percentage and giving them both equal weight.

And yes. If someone is giving me an answer, I want to see the equation. Call me crazy.

 

[luckiestman]

The critique I have is the same one I have always had. If you were going to invent a sport and the goal was to make the information useful for statistical analysis you could do a lot worse than baseball. How well statistical analysis worked to analyze individual player value in baseball gave people the idea that they could do similar stuff elsewhere because “sports”.

For basketball and football there have been good ideas on what teams should do and then finding players that can do those things. But I have yet to see many individual level metrics that beat the eye test or standard box score stuff. The only thing I kind of like is large sample on/off stuff.

 

[Jimbodandy]

I think within reason single metrics can be good, though defense in particular is nowhere close to being well quantified (and it's worse in some like LEBRON where it seems almost useless) I think it's good to discuss how multiple metrics, and underlying stats see players.
There is sometimes a tendency though to go with:

We have no choice but to use multiple metrics at this time, and we get something of a picture that way. More is always better. We look at a lot of hitter advanced numbers, even though the baseball O numbers are rock solid.

 

[nighthob]

You think the available statistics are bad but also think that good statistics should be expressible in a closed form expression that you can pop into your calculator?

A few things, the term black box number refers to something derived from a formula that isn’t public. In other words, something I can’t calculate myself. All in ones tend to be garbage because half of basketball is defensive and we have no way of accurately quantifying it.

Basketball breaks down very cleanly at the micro level, especially these days with the Synergy data. With the synergy play by play data we can even see how players perform defensively by types of plays. But no one’s figured out how to turn that into a single number reflecting all defense (or even justify why it’s necessary when the play by play analysis provides more useful data).

 

[HowBoutDemSox]

I guess my question would be how well any of these metrics predict future performance (or, when applied retroactively, accurately describe past results). Does comparing a team’s aggregate LEBRON (or, pick your advance stat) better predict who will win a game than their point differentials? Does comparing the LEBRON (or, pick your advance stat) of traded players or free agent signings accurately tell you how much better or worse the teams will perform? I honestly don’t know the answer to those questions and wouldn’t be qualified enough to analyze any proposed responses, but that’s where my immediate thinking goes.

 

[DeJesus Built My Hotrod]

A few things, the term black box number refers to something derived from a formula that isn’t public. In other words, something I can’t calculate myself. All in ones tend to be garbage because half of basketball is defensive and we have no way of accurately quantifying it.

Basketball breaks down very cleanly at the micro level, especially these days with the Synergy data. With the synergy play by play data we can even see how players perform defensively by types of plays. But no one’s figured out how to turn that into a single number reflecting all defense (or even justify why it’s necessary when the play by play analysis provides more useful data).

Single metrics aren't designed to convey attributes. They are used as a basis for comparisons. We compare contracts all the time and its the same thing. Big X got 2yrs and $20mm and Big Y got the same but we like Big Y better because he is a better perimeter defender or has a better post game or can get the team into great clubs on the road.

Contrary to the opinions of some here, people aren't citing these numbers as definitive proof of anything at all. They are just incorporating them into their decision making processes.

 

[slamminsammya]

A few things, the term black box number refers to something derived from a formula that isn’t public. In other words, something I can’t calculate myself. All in ones tend to be garbage because half of basketball is defensive and we have no way of accurately quantifying it.

Basketball breaks down very cleanly at the micro level, especially these days with the Synergy data. With the synergy play by play data we can even see how players perform defensively by types of plays. But no one’s figured out how to turn that into a single number reflecting all defense (or even justify why it’s necessary when the play by play analysis provides more useful data).

There is a difference between non-public methodology and something you can't calculate yourself (at least to my understanding of the term "black box"). If I posted to github the code I used to calculate slamminsammya's favorite metric, I wouldn't consider that a black box - maybe you'd disagree if you can't read code or aren't familiar with the statistical techniques?

For example, I cannot personally perform ridge regression since I don't know which python library to use and I can't recall off the top of my head how ridge regression is performed (its a regularization with L1 or L2, one is ridge one is lasso and I can't remember which is which). But ridge regression is not a black box. Its a known statistical technique.

I think this matters for this conversation because "its a black box" meaning "no one knows how its computed besides its creator" is a legitimate argument against a metric's validity. On the other hand "I don't know what ridge regression is and I don't care to learn, and there is no formula I can look at" holds no weight to me as an argument.

 

[Cesar Crespo]

There is a difference between non-public methodology and something you can't calculate yourself (at least to my understanding of the term "black box"). If I posted to github the code I used to calculate slamminsammya's favorite metric, I wouldn't consider that a black box - maybe you'd disagree if you can't read code or aren't familiar with the statistical techniques?

For example, I cannot personally perform ridge regression since I don't know which python library to use and I can't recall off the top of my head how ridge regression is performed (its a regularization with L1 or L0, one is ridge one is lasso and I can't remember which is which). But ridge regression is not a black box. Its a known statistical technique.

I think this matters for this conversation because "its a black box" meaning "no one knows how its computed besides its creator" is a legitimate argument against a metric's validity. On the other hand "I don't know what ridge regression is and I don't care to learn, and there is no formula I can look at" holds no weight to me as an argument.

I meant the former. I said they aren't giving you the info you need to actually do the math.

 

[lexrageorge]

I don't understand the beef w/ OPS. It is a sum of two of the most influential statistics related to offense in baseball. Is it flawless? o. Does it tell the whole story? No, not even on offense. I don't think anyone claims either. But it does provide a useful metric when comparing relative offensive impact of different players. Compact, easily understand stats do offer value. Glad that the Red Sox forum doesn't have any restrictions about citing OPS when making arguments.

With regards to basketball, my view is the questions we should ask our ourselves is (1) what makes a good statistic and (2) when is it appropriate to use a specific stat when making a post on this board?

On the first point, my own personal view on a good statistic:

1.) Meaningful. Seems obvious, but a good stat should provide more info than just looking at a box score or calculating PPG and EFG. Part of this is that outliers need justification. I personally don't consider a stat that puts Jaylen Brown and Tacko Fall in the same category as providing much meaningful information until and unless a solid explanation can be provided as to why that happens and why there is still some meaning in the comparison.

2.) Objective. A few years back, one of the football stats graded Tom Brady as being outside the top 10 in the league, and subsequently gave Brady a bad grade in the Super Bowl against Seattle. They did this because their methodology was to watch the games and make arbitrary decisions on whether a receiver should have caught a ball thrown 5 yards outside his grasp, or whether the defensive lineman should have been able to shed his pair of blockers and get to the QB. Such ratings are obviously not repeatable; other people watching the same game could easily come up with very different grades.

3.) Not a black box. I use @slamminsammya's definition above. While I can see why people may want to keep their regression formulas private, the resulting calculations become instantly suspect, as there is no way to know if methodology changes beyond the good graces of the inventor.

4.) Predictive. This means that the stat truly stabilizes over a reasonable sample size to the point where one has an idea of what type of player they are dealing with. Pitcher BABIP in baseball never stabilizes to any value, so its predictive utility is very low.

 

[slamminsammya]

I meant the former. I said they aren't giving you the info you need to actually do the math.

Here ya go! They even give you the value of lambda used for the regularization in one version of RAPM. Which other ones would you like? PIPM - https://fansided.com/2018/01/11/nylon-calculus-introducing-player-impact-plus-minus/ . Now you have almost all you need to compute LEBRON, modulo some parameter choices for padding the priors and doing luck adjustment.

https://basketballstat.home.blog/2019/08/14/regularized-adjusted-plus-minus-rapm/ View: https://www.youtube.com/watch?v=OuC0YZTADcE&ab_channel=MarkGlickman

 

[Cellar-Door]

Single metrics aren't designed to convey attributes. They are designed as a basis to begin comparisons. We compare contracts all the time and its the same thing. Big X got 2yrs and $20mm and Big Y got the same but we like Big Y better because he is a better perimeter defender or has a better post game or can get the team into great clubs on the road.

Contrary to the opinions of some here, people aren't citing these numbers as definitive proof of anything at all. They are just incorporating them into their decision making processes.

But the thing is... if the metric is garbage then using it for comparison is useless.
Let me move to a different area and give an example...
There are a bunch of "rankings" of graduate schools, like say law schools.
The most prominent of them uses a bunch of metrics with the heaviest weight being... "what other law schools think of them" and "what lawyers think of them" with the next highest being "how hard is it to get admitted"
Another focuses heavily on average cost of attendance versus average employment outcome
A third focuses on a bunch of esoterics, like how much library space there is, how new the facilities are, etc.

So the question is... if you pull 1 or 2, you get somewhat similar results, as reputation helps get you a job, but 2 tells you more about value since it takes into account how that job market relates to cost.
The third is a hack job put together by a for-profit law school to make themselves score highly....

That's the thing with all in ones... they aren't inherently useful if they are broken. And they have no value unless we know what they are measuring. LEBRON is moderately useful, but we know it has significant flaws with it's defensive formula which makes it the equivalent of the law school library square footage criteria (one reason I think overall LEBRON has little use, but Offensive LEBRON has some value).

But looking at the thread this debate came out of, one of the things that started it was a garbage poorly labeled chart with no explanation of what it was measuring or how, which lead to discussions about why players weren't on it (because it was allegedly measuring an all in one of only certain play types on defense buy all offense, which is dumb) and that I think more than anything is what annoys people in the forum.

 

[slamminsammya]

4.) Predictive. This means that the stat truly stabilizes over a reasonable sample size to the point where one has an idea of what type of player they are dealing with. Pitcher BABIP in baseball never stabilizes to any value, so its predictive utility is very low.

This one is a quite subtle point I am not sure I agree with. The question "how good was this player in the games we watched" is different from "how good do we expect this player to be in the future" to the extent that player skill and performance is variable, and in basketball I think this varies quite a bit through time due to injuries, mental engagement, and many other factors. In particular, the margins for athleticism in basketball are so great that true performance can really vary - just look at what adding 5 pounds to Grant Williams has done this year

That being the case, asking for a metric that is descriptive and asking for a metric that is predictive are different problems and even more so in basketball. People do evaluate their models according to predictive power which is fine since there isn't much other choice, but not every model is meant to be used for both cases.

 

[Cesar Crespo]

I don't understand the beef w/ OPS. It is a sum of two of the most influential statistics related to offense in baseball. Is it flawless? o. Does it tell the whole story? No, not even on offense. I don't think anyone claims either. But it does provide a useful metric when comparing relative offensive impact of different players. Compact, easily understand stats do offer value. Glad that the Red Sox forum doesn't have any restrictions about citing OPS when making arguments.

With regards to basketball, my view is the questions we should ask our ourselves is (1) what makes a good statistic and (2) when is it appropriate to use a specific stat when making a post on this board?

On the first point, my own personal view on a good statistic:

1.) Meaningful. Seems obvious, but a good stat should provide more info than just looking at a box score or calculating PPG and EFG. Part of this is that outliers need justification. I personally don't consider a stat that puts Jaylen Brown and Tacko Fall in the same category as providing much meaningful information until and unless a solid explanation can be provided as to why that happens and why there is still some meaning in the comparison.

2.) Objective. A few years back, one of the football stats graded Tom Brady as being outside the top 10 in the league, and subsequently gave Brady a bad grade in the Super Bowl against Seattle. They did this because their methodology was to watch the games and make arbitrary decisions on whether a receiver should have caught a ball thrown 5 yards outside his grasp, or whether the defensive lineman should have been able to shed his pair of blockers and get to the QB. Such ratings are obviously not repeatable; other people watching the same game could easily come up with very different grades.

3.) Not a black box. I use @slamminsammya's definition above. While I can see why people may want to keep their regression formulas private, the resulting calculations become instantly suspect, as there is no way to know if methodology changes beyond the good graces of the inventor.

4.) Predictive. This means that the stat truly stabilizes over a reasonable sample size to the point where one has an idea of what type of player they are dealing with. Pitcher BABIP in baseball never stabilizes to any value, so its predictive utility is very low.

The OPS beef is 1 poster from what I can tell. People don't even really use it anymore anyway.

1. I think PPG, EFG%, etc are meaningful, most people just arrive at the wrong meaning. PPG tells you exactly how many points a person scored in a game (duh) and nothing more. A box score can somewhat tell you if a player is good on offense but it can't really tell you anything about defense. Steals and blocks don't measure defensive value, though terrible block and steal rates in college are usually predictive of busts and what not. Jaylen Brown is a pretty big exception. A player who scores 20 a game and is 4/9 or 4/10 from 3 every night is a player you can safely assume is a pretty good offensive player. A player who gets 2 blocks and 2 steals every night tells you nothing. He could be good, he could be gambling and getting burned most of the time.

 

[nighthob]

Single metrics aren't designed to convey attributes. They are used as a basis for comparisons. We compare contracts all the time and its the same thing.

We all understand that they're used for rough and dirty comparisons. Some of us just understand that the numbers aren't particularly useful for determining how good a player is, especially given the limitations of quantifying defense at the macro level.

There is a difference between non-public methodology and something you can't calculate yourself (at least to my understanding of the term "black box"). If I posted to github the code I used to calculate slamminsammya's favorite metric, I wouldn't consider that a black box - maybe you'd disagree if you can't read code or aren't familiar with the statistical techniques?

A black box number is literally derived from a formula that others aren't allowed to see. That's why we call it a black box number, we can't see how it's produced.

 

[chilidawg]

[QUOTE="DeJesus Built My Hotrod, post: 4382998, member: ]

Contrary to the opinions of some here, people aren't citing these numbers as definitive proof of anything at all. They are just incorporating them into their decision making processes.
[/QUOTE]

This, exactly.

To me the best use of these metrics is to open your eyes to things you might be missing. Wood was a good example of this last year, a guy starting to put up modest counting stats but with excellent advanced metrics. Made me think there might be something worth checking out.

I also like On-Off +/- numbers as they are a non black box measure of how a team does when a player is on the court relative to off. The non bbref O/D ratings are similarly useful.

 

[RorschachsMask]

But looking at the thread this debate came out of, one of the things that started it was a garbage poorly labeled chart with no explanation of what it was measuring or how, which lead to discussions about why players weren't on it (because it was allegedly measuring an all in one of only certain play types on defense buy all offense, which is dumb) and that I think more than anything is what annoys people in the forum.

I don’t think the chart is poorly labeled, it very clearly defines what it’s showing in the first sentence of the tweet. They also had other tweets showing point of attack defenders, and their offensive value. I get not liking that they’re just measuring one part of defense as opposed to the whole deal offensively, but it’s just one of many charts/breakdowns. I just posted that one, because it shows where the Jays stand. I don’t like defensive impact stats, but b-ball index is pretty open about their formula.

View: https://twitter.com/The_BBall_Index/status/1383865413742723076?s=20

 

[DeJesus Built My Hotrod]

But the thing is... if the metric is garbage then using it for comparison is useless.
Let me move to a different area and give an example...
There are a bunch of "rankings" of graduate schools, like say law schools.
The most prominent of them uses a bunch of metrics with the heaviest weight being... "what other law schools think of them" and "what lawyers think of them" with the next highest being "how hard is it to get admitted"
Another focuses heavily on average cost of attendance versus average employment outcome
A third focuses on a bunch of esoterics, like how much library space there is, how new the facilities are, etc.

So the question is... if you pull 1 or 2, you get somewhat similar results, as reputation helps get you a job, but 2 tells you more about value since it takes into account how that job market relates to cost.
The third is a hack job put together by a for-profit law school to make themselves score highly....

That's the thing with all in ones... they aren't inherently useful if they are broken. And they have no value unless we know what they are measuring. LEBRON is moderately useful, but we know it has significant flaws with it's defensive formula which makes it the equivalent of the law school library square footage criteria (one reason I think overall LEBRON has little use, but Offensive LEBRON has some value).

But looking at the thread this debate came out of, one of the things that started it was a garbage poorly labeled chart with no explanation of what it was measuring or how, which lead to discussions about why players weren't on it (because it was allegedly measuring an all in one of only certain play types on defense buy all offense, which is dumb) and that I think more than anything is what annoys people in the forum.

I cannot speak to law school rankings.

However in my line of work and for analysis in general, I see the value of a system which uses common denominators as its foundation. IMO, even if they perfectly measure what you are attempting to evaluate, single number metrics should not be used in isolation - no advanced metrics should really override how you optimize what is at its core a very human endeavor (at the highest level, the NBA is just another sport that allows humans to "fight" without the gruesome consequences).

But again, people should be thinking about this in terms of their day to day consumption. Chicken sandwiches seem to be all the rage right now and I haven't checked but I suspect you aren't going to find a meaningful price delta from, say Wendy's to McDonalds. Since I no longer have to worry about the cost, I can now select based on whatever else I value (bun, sauce, proximity etc).

We have a long way to go before we get to that point with sports metrics but that is the general idea.

 

[Jimbodandy]

I meant the former. I said they aren't giving you the info you need to actually do the math.

Exactly.

And fwiw, I don't use that is an argument that the metric is flawed. I hold it against the creators for not adding to the scholarship. If they published, we could build on it.

 

[Tony C]

... They have OPS+ which tries to balance out the difference between OBP and slugging... but why not just list the OBP and slugging numbers? Is it that much harder than listing an OPS+?

You mean, list OBP and slugging as is done on virtually every site that gives stats? I mean, here's a sample from an obscure site called ESPN:
https://www.espn.com/mlb/player/splits/_/id/6389/ryan-zimmerman

Well, there you go (and, sure, it adds them up to an OPS, too, which is kind of a convenient albeit imperfect shorthand, tho agree OPS+ would be better). This is another example of "the internet goes in search of an argument."

 

[RorschachsMask]

This is funny timing with the tweet.

View: https://twitter.com/The_BBall_Index/status/1384906050936115206?s=20

 

[slamminsammya]

A black box number is literally derived from a formula that others aren't allowed to see. That's why we call it a black box number, we can't see how it's produced.

We agree. You equated this with something you can't compute yourself. Those two things are not identical, although a black box statistic is also something you can't compute yourself.

 

[Cesar Crespo]

This, exactly.

To me the best use of these metrics is to open your eyes to things you might be missing. Wood was a good example of this last year, a guy starting to put up modest counting stats but with excellent advanced metrics. Made me think there might be something worth checking out.

I also like On-Off +/- numbers as they are a non black box measure of how a team does when a player is on the court relative to off. The non bbref O/D ratings are similarly useful.

Traditional box scores would have told you the same thing about Christian Wood in 19/20. Weird thing about Christian Wood is his advanced rates haven't really changed much since entering the pros. HIs assist rate % has improved a little.

When Harden was on OKC, he was an advanced metric darling but even using traditional stats, one could tell Harden was primed for a break out.

Young player improves year over year and takes a biggish leap in age <25 season. Player comes back even better the following season.

I think anyone who follows the NBA an unhealthy amount knew all about Christian Wood. A few posters here were calling it the best signing of the offseason right when he signed.

I'm trying to think of a player who went from trash to great in one off season and I can't. Trash to good, yes. And then from good to great. Not trash to great though. I'm sure there are some and I'm just blanking. It would be interesting if the Metrics picked up on those players.

 

[Cesar Crespo]

You mean, list OBP and slugging as is done on virtually every site that gives stats? I mean, here's a sample from an obscure site called ESPN:
https://www.espn.com/mlb/player/splits/_/id/6389/ryan-zimmerman

Well, there you go (and, sure, it adds them up to an OPS, too, which is kind of a convenient albeit imperfect shorthand, tho agree OPS+ would be better). This is another example of "the internet goes in search of an argument."

I already said it was a me thing and I wasn't referring to websites. I was referring to posters who just use OPS and nothing more. I also noted OPS is not really used anymore.

This whole argument is pointless if you are going to go by what every website has up for stats. I'm not against OPS being listed as a stat on baseball reference. I'm against using it to talk about how good a player is offensively (unless listed with lots of other stats) when there are far better stats to do so.

Long story short, I'm not against any stat. I'm against people using them incorrectly. Let's use 25 games of data in CF to see how good Jackie Bradley is even though the author of whatever formula says it takes 3 years to stabilize. The more numbers, the more info, the better. I'd rather have the O and D Ratings on basketball reference than not, even though they are totally useless and make TL, Capela, Mitchell Robinson types the best players in the NBA.

 

[Cesar Crespo]

Exactly.

And fwiw, I don't use that is an argument that the metric is flawed. I hold it against the creators for not adding to the scholarship. If they published, we could build on it.

Not a perfect analogy, but most video games are better with user mods.

 

[nighthob]

We agree. You equated this with something you can't compute yourself. Those two things are not identical, although a black box statistic is also something you can't compute yourself.

I mean my math is good enough to understand formulae. Show me one and I can do the calculations myself. But black box numbers are still garbage.

 

[Cesar Crespo]

If the formula isn't public knowledge, how can anyone compute anything? I'm confused. Is this a semantics argument?

Like, if someone had the formula, it doesn't mean they could compute it? Ok.

 

[slamminsammya]

If the formula isn't public knowledge, how can anyone compute anything? I'm confused. Is this a semantics argument?

Like, if someone had the formula, it doesn't mean they could compute it? Ok.

I am not sure if its a semantics argument? I guess my point is you don't need the actual formula if you know what the methodology is at a high level, and I think the fixation on a formula is indicative of a bit of a lack of experience with these sorts of things, but I could be wrong here. Like, there is technically a formula for computing the linear regression solution for a given set of data - but giving you the formula for m and b for the best fit line is less helpful than just saying "go into excel and take a linear regression".

Similarly, for something like RAPM, they aren't giving you the formula because it is actually less transparent / helpful / illuminating than just saying "Look at their on/off data and do ridge regression with respect to their teammates" which is actually telling you exactly what is happening at a level humans can understand, instead of giving you the fOrMulA with like a million sigmas, various indices, various statistical notations etc.

If I read "take a players on/off data, blend it with a prior that is computed in this way and blended according to these weights, and do ridge regression" that is, to me, totally transparent even if it is going to take some work on my part to actually recreate the math, more so than an fOrMulA.

So I dont know, is this a semantics thing or do we actually disagree here?

EDIT: I should also mention I posted a series of articles that tell you exactly how to compute the various ingredients for LEBRON in the Tatum thread, do you want a walkthrough?

 

[Cesar Crespo]

"take a players on/off data, blend it with a prior that is computed in this way and blended according to these weights, and do ridge regression"

Is that not the formula? The computed in this way is the whole point.

 

[slamminsammya]

Is that not the formula? The computed in this way is the whole point.

OK, here we go then:

• RAPM - do standard linear regression on a matrix where each row represents a chunk of 10 players on the floor together and the target column is the point differential over that time. Regularize your regression using ridge method and a lambda of something like 2000 or 3000. Add a prior if you'd like to using previous seasons.
• PIPM - same as above except instead of target column the raw +/-, adjust for shot luck and also add some columns for box score statistics.
• LEBRON - take the box score component of the above, smooth it out by adding dummy data for cases where players have few minutes where the dummy data is computed according to offensive role, where offensive role in turn is computed using K-means clustering on synergy play by play categorization of offensive scoring actions for each player. This part requires a little thinking - the value of K he uses is going to be equal to the number of named roles he has here which isn't given explicitly, and his tableau dash shows all the play types that go into vectorizing each player. This is your prior you use for RAPM - just do RAPM after.

 

[bowiac]

I just posted in the Tatum thread about some of this, so maybe it's worth moving over here. Mods feel free to do that if appropriate.

FWIW, PIPM did not use a linear regression. It was not like RAPM in that way. Rather, it could be computed from simple box-score and on/off data for a single game, but didn't require 'play-by-play' level granularity to create 10-man stints, etc... It performed impressively well despite this limitation, largely due to the luck adjustments Jacob was doing.

I like LEBRON, but I do think it's fair to call it a black box. For a user, it's hard to understand what's driving a given player's LEBRON. It could be the box prior, Or it could be the on/off data, or it could be the offensive role? Etc... That's fine - black boxes are useful, but that's a legitimate limitation of LEBRON (and all these stats to some degree other than raw RAPM).

 

[DeJesus Built My Hotrod]

I don't get the debate here. Data is being collected and mined literally every second. People are using the output to guide the actions of other humans. Its the foundation of our global economy.

This dynamic is happening in the NBA as it is in every walk of life. As bowiac notes Jacob Goldstein, who created PIPM, now works for the Wizards. My understanding is that the majority of clubs use some form of advanced metrics for things like play calling as well as roster construction/player evaluation.

Teams measure and analyze every inch of an NBA playing surface as well as the humans who occupy it. In short, the math nerds have already won.

 

[slamminsammya]

I just posted in the Tatum thread about some of this, so maybe it's worth moving over here. Mods feel free to do that if appropriate.

FWIW, PIPM did not use a linear regression. It was not like RAPM in that way. Rather, it could be computed from simple box-score and on/off data for a single game, but didn't require 'play-by-play' level granularity to create 10-man stints, etc... It performed impressively well despite this limitation, largely due to the luck adjustments Jacob was doing.

I like LEBRON, but I do think it's fair to call it a black box. For a user, it's hard to understand what's driving a given player's LEBRON. It could be the box prior, Or it could be the on/off data, or it could be the offensive role? Etc... That's fine - black boxes are useful, but that's a legitimate limitation of LEBRON (and all these stats to some degree other than raw RAPM).

We get another distinction in the notion of black box here. There is the reproducibility question which is quite distinct from the interpretability question. If you really wanted you could add interpretability for this kind of model, I don't think its inherently black boxy in the interpretability sense. Actually it would be really interesting to run SHAP on any of these models to see what exactly they are learning about basketball on an aggregate scale. I realize that isn't the type of interpretability you were pointing to but I would find that actually more interesting than what an individual player's value comes from.

 

[HomeRunBaker]

I don't get the debate here. Data is being collected and mined literally every second. People are using the output to guide the actions of other humans. Its the foundation of our global economy.

This dynamic is happening in the NBA as it is in every walk of life. As bowiac notes Jacob Goldstein, who created PIPM, now works for the Wizards. My understanding is that the majority of clubs use some form of advanced metrics for things like play calling as well as roster construction/player evaluation.

Teams measure and analyze every inch of an NBA playing surface as well as the humans who occupy it. In short, the math nerds have already won.

It is definitely a piece but I wonder how much value they place on them and/or how much they are utilized.

The Wizards, for example, hire the creator of PIPM then trade for Westbrook so he can take 10 mid-range jumpers each night and turn the ball over 6 times. Many teams employ sleep coaches or performance health directors yet players are out at bars, strip clubs or casinos until 4am when they are on the road. Like all performance tools, they are only valuable if they are implemented.

 

[DeJesus Built My Hotrod]

There is no argument that some organizations are better than others at getting the mix of stats and individual creativity right. However the question is whether math, and advanced metrics are garbage or not garbage? Let's find out.

Most teams use some form of it so it holds some value. It is therefore not garbage.

 

[HomeRunBaker]

There is no argument that some organizations are better than others at getting the mix of stats and individual creativity right. However the question is whether math, and advanced metrics are garbage or not garbage? Let's find out.

Most teams use some form of it so it holds some value. It is therefore not garbage.

Def not garbage.

I think teams utilize numbers a lot differently than we speak about them or how we see them listed in a chart. An example would be using data such as a players shooting numbers from different spots on the floor and actively implementing this data into designed sets. Another in training/recuperation using cryotheraphy, scheduling of their road trips to best accommodate sleep, etc etc.

IIRC, there was a site about a decade ago that tracked professional sports teams flights. I recall speaking to a friend of mine who used this information as a tool for gambling purposes. These are things we don’t talk about here but is calculated in a precise manner by these organizations. It’s all a part of one big picture.

 

[Auger34]

There is no argument that some organizations are better than others at getting the mix of stats and individual creativity right. However the question is whether math, and advanced metrics are garbage or not garbage? Let's find out.

Most teams use some form of it so it holds some value. It is therefore not garbage.

I thought the argument was more “are the advanced stats that are available to the general public garbage?” There’s actually arguments to be made for each side of that.

As you point out above, literally every team uses advanced stats in evaluations. I think a lot of them are proprietary and way better than the stats we have access too but they’re clearly being used and definitely not garbage

 

[wade boggs chicken dinner]

I'll move Bowiac's post over:

LEBRON is also using BBall Index's "player role stuff" to influence the prior, not just boxPIPM (I helped Krishna put some of LEBRON together, and did most of the backtesting).

With respect to their luck adjustment as flagged by @slamminsammya - I agree this is DIPS-like, and drops meaningful data because we don't yet know how to quantify it. On the other hand, DIPS remains hugely influential (probably the most influential sports stat finding of my lifetime), and is like ~85% true in the purest Voros form. We're probably in a similar place on luck adjustments right now. We're losing meaningful signal by adjusting opposing shooting percentages, but mostly we're getting rid of noise, such that it's worth it on net.

In my testing, it's a good stat, in the top ~3 most predictive public metrics at predicting future game outcomes (along with my own DARKO, and Taylor Snarr's EPM). I do agree there's a somewhat unfortunate proliferation of these stats right now, which leads to some cherry picking to find the stat which best fits the narrative someone wants to tell. I don't really know the answer to that issue, and I'm not obviously not helping myself (although DARKO is a bit different, in that it's fundamentally predictive).

Some quick specific thoughts:

- I don't think RPM should be cited anymore. There have been some significant changes to the metric recently since the creator was hired by the Mavericks. Nobody has a real understanding of what the stat is doing anymore, and it doesn't perform well in testing.
- The same goes for RAPTOR. The stat has not held up well in out-of-sample testing, and is prone to wild swings in the playoffs in particular. I respect 538 a lot, but they're not an NBA shop, and don't seem to put in consistent effort to keeping RAPTOR reliable. They have also made some strange decisions in not running a true RAPM model here.
- BPM remains useful in my opinion for being a box-score-only metric. If you see a player doing well in BPM, but poorly in LEBRON or EPM, that's a sign that their on-off data is not very good for instance. It's helpful in adding interpretability to understand what's going on with some players. Note VORP is the same thing as BPM. It's just the counting stat version. Your VORP is a linear function of your BPM and minutes played.
- RAPM is likewise useful as a pure on-off stat. It's the complement to BPM, ignoring box stats entirely and just surmising how good a player is from on/off data.
- Nobody takes PIE seriously.