How Likely Are You to Draft a Starter?

BaseballJones

ivanvamp
SoSH Member
Oct 1, 2015
24,369
I've spent lots of time evaluating NFL drafts from 2000-present. For this particular exercise, I'm asking the question of: How likely are you to draft a starter? I'm using data from pro-football-reference.com. And honestly, I have it all in an excel spreadsheet but the data in the spreadsheet is a year old, because I started this project a year ago. So here's my methodology:

(1) I am only looking at drafts from 2000-2013. This is because drafts after 2013 are much more impacted by the fact that I don't have the 2022 season data in it, which does change some things, but also by the fact that the longer the career, the better chance to hit the two benchmarks I'm about to look for.

(2) Benchmark #1 is a 5-year starter. I don't want to go fewer than 5 years, because being a 5-year starter indicates to me that you aren't flukey, and that you did this for a decent length of time, with some consistency. But I didn't want to make the threshold any higher because 5 years as a starter in the NFL is really good.

(3) Benchmark #2 is a 2x Pro Bowler. I didn't want to go with All Pro because that's a super high bar to hit. And we all know that the Pro Bowl is weird, and that sometimes undeserving players make a Pro Bowl. But it does at least give us SOME indication of who is good. Generally, with enough data, being a Pro Bowler is going to give you a reasonable indication of whether a guy is good. Making it once is potentially a fluke. Making it twice shows you're pretty good. So that's my second benchmark.

You can see why I wanted to look at drafts that are 10 or more years old. Because evaluating the 2020 draft with these benchmarks is obviously stupid, as players haven't even been in the league long enough to achieve 5 years as a starter. And it's also unreasonable to look at a guy who has been in the league 7 years and hold them to a 5-year starter benchmark. So that's at least the method behind my madness here.

So with that being said, from the 2000-2013 drafts, here's the data:

64332

So the "best" draft of the bunch, from these two benchmarks, was the 2006 draft, where 65 out of 255 players drafted (25.5%) were at least 5-year starters, and 29 players (11.4%) made at least 2 Pro Bowls. Depending on how you look at it, the "worst" draft years here were probably 2008 (17.9% 5-year starters, 6.7% 2x PB), 2009 (19.5% 5-year starters, 4.3% 2x PB), or 2013 (18.9% 5-year starters, 7.1% 2x PB).

So out of these 14 drafts, there were 3,567 players taken. 761 of them (21.3%) started at least 5 seasons, and 280 (7.8%) made at least 2 Pro Bowls.

The average team made 8 picks per year, and had 1.7 5-year starters come from that, and fewer than one player per draft (about one every two drafts, as it were) make at least 2 Pro Bowls.

So a "good" draft means that you draft two guys who end up starting for 5 years in the NFL, and a really good draft means you draft two guys who end up starting 5 years in the NFL and one of them makes 2 Pro Bowls.
 

BaseballJones

ivanvamp
SoSH Member
Oct 1, 2015
24,369
The Patriots, from 2000-2013, drafted 124 guys.

- 5-year starters: 29 (23.4%)
- 2x Pro Bowl: 14 (11.3%)

Both those numbers are above the NFL averages of 21.3% and 7.8%. And keep in mind that the Patriots' average draft position over this time is among the worst in the NFL, because the Patriots, more than any other team, consistently had the best seasons, thus putting them at the bottom of each round as a starting draft position.
 

luckiestman

Son of the Harpy
SoSH Member
Jul 15, 2005
32,617
Cool data. Makes me wonder how much you could narrow it down without much loss. Like how many guys started 48 games but didn’t get to 80 (16 game seasons your set ).

you could also profile positions and questions like didn’t start year one but then was an X year starter.

or you could know when to cut bait. If didnt start X games within first X years the likelihood of ever starting is ?.

it’s a good data set
 

BaseballJones

ivanvamp
SoSH Member
Oct 1, 2015
24,369
Cool data. Makes me wonder how much you could narrow it down without much loss. Like how many guys started 48 games but didn’t get to 80 (16 game seasons your set ).

you could also profile positions and questions like didn’t start year one but then was an X year starter.

or you could know when to cut bait. If didnt start X games within first X years the likelihood of ever starting is ?.

it’s a good data set
Yeah, it's a TON of data. The file takes up a lot of space in my hard drive, and there's SO many ways to parse it out. I'll do this round by round at some point as well. It may feel weird to cut it off after 2013 but in order to do analyses like this, you have to allow players to get through a good length of a career. It really just doesn't do much for this kind of analysis to examine the 2021 draft, you know? So I give it ten years, which makes the cutoff 2013. Obviously, though, that data is "old" and the game has changed some since then, so it's going to be incomplete. I guess I could do this same thing with more recent data but just use lower thresholds.
 

joe dokes

Member
SoSH Member
Jul 18, 2005
30,233
That's great stuff.
I wonder if its possible to get more granular by some sort of points system assigned to fewer-than-5-year starters. My sense is that 5-year starters are stars or near-stars. Or at a minimum, there just aren't that many of them. And while teams should get credit in this context for drafting them, I wonder if only counting *them* is the best way to assess drafting. Believe me, this is not a criticism of your heavy lifting here. (lifting I am not capable of).
 

BaseballJones

ivanvamp
SoSH Member
Oct 1, 2015
24,369
That's great stuff.
I wonder if its possible to get more granular by some sort of points system assigned to fewer-than-5-year starters. My sense is that 5-year starters are stars or near-stars. Or at a minimum, there just aren't that many of them. And while teams should get credit in this context for drafting them, I wonder if only counting *them* is the best way to assess drafting. Believe me, this is not a criticism of your heavy lifting here. (lifting I am not capable of).
Yeah later I can run the numbers for, like, 3-year (or more) starters. I admit that my 5-year starter threshold is arbitrary, but then, so is any other, right? Nothing "magical" about 5 years. I just thought it signified a good player who starts for a good length of time. But I can run the numbers for lower thresholds too.
 

Morgan's Magic Snowplow

Member
SoSH Member
Jul 2, 2006
22,345
Philadelphia
Cool data. How are you defining a starter? For example, if I guy starts 9 games out of 16 in a given year, is he classified as a starter? What if a guy plays only 8 games due to a big injury but starts them all?
 

pappymojo

Member
SoSH Member
Jul 28, 2010
6,667
This is very interesting. I don't want to ask you to do work, but I have always wondered about the ability to draft a good player based on draft slot. Basically, for so many years we have made excuses for the Patriots because they have been picking so late in the first round, is there any truth to that, and if so would it make more sense to consider picks 1-20 as the true first round, and to consider 21-40 as the true second round, etc.?
 

SMU_Sox

queer eye for the next pats guy
SoSH Member
Jul 20, 2009
8,878
Dallas
I would check out this study - do you use Python at all?

You can build and refine tiers and definitions too.
 

Ferm Sheller

Member
SoSH Member
Mar 5, 2007
20,404
This is very interesting. I don't want to ask you to do work, but I have always wondered about the ability to draft a good player based on draft slot. Basically, for so many years we have made excuses for the Patriots because they have been picking so late in the first round, is there any truth to that, and if so would it make more sense to consider picks 1-20 as the true first round, and to consider 21-40 as the true second round, etc.?
I think that this data gives some indication. If you look at just round one, it appears that HoFers skew heavily toward being top 10-12 draft picks (but you also have to factor in every picked in round two and later -- I haven't looked at it closely yet).
 

Dotrat

Well-Known Member
Lifetime Member
SoSH Member
Jul 11, 2002
2,135
Morris County NJ
This is great stuff. I wonder if it's also possible to weight teams' records. I may be wrong, but it seems to me that more players are more likely to start for losing teams, and especially for teams that consistently finish at or below .500.
 

InstaFace

The Ultimate One
SoSH Member
Sep 27, 2016
21,753
Pittsburgh, PA
The high variance on the back end of that decay is as good evidence for the "lottery tickets" argument as any.
yeah, it took me a second, but those diamonds on the back end of the decay chart are not individual player outcomes plotted as a scatter, they're average-AV (so average-of-an-approximation :) ) for all players drafted at that pick #, meaning it gives no real sense of the probability of drafting someone useful, because 10 guys who are zeroes are averaged with one guy who's a 30 AV and one guy who's a 12 AV and so the average is a 4. Or rather, it hides that information behind the averaging.

BJ, this is cool info, but I would probably do this from a # of starts perspective (to capture useful players whose career was cut short by injury, or other quirks of a career that often has ups and downs). Some threshold like 30 career starts (or 20, or 50, I dunno). But I think "5-year starter" has a lot of noise associated with the definition that obscures what we're really trying to get at, which is a draft success that is materially useful to the drafting team.

For that matter, the 5-year threshold is odd too, since players who aren't stars (but might be useful) tend to change teams after 3 or 4 years. Only first-rounders come with a 5th year team option, right? So I would at least dial that back to 4-year starters.

Then for me, the next step I'd take with your data would be calculating some value-above-replacement-draft kind of metric, where you have some sort of set expectation based on the draft slots that a team is drafting with, and they have some outcomes (in AV, or just the binary "was he a starter long enough to clear this bar?" flag), and we can calculate the extent to which every team's drafts each year were successes or failures. And add up which team has done better or worse or by how much. And assess whether success or failure is something that can be consistently replicated by an organization or is just kinda random. All of which would be very cool insights.
 

BaseballJones

ivanvamp
SoSH Member
Oct 1, 2015
24,369
Thanks for all the replies and suggestions. I’m going to do much more with this. One of the requirements for me though is that the data needs to be easily accessible. So like I’m not sure how pro-football-reference defines what player constitutes a “starter” but they apply whatever reasoning they have across the board. And because they give that metric (“starter”) in easy to access form, that’s what I’m going to use. Hope that makes sense why.
 

BaseballJones

ivanvamp
SoSH Member
Oct 1, 2015
24,369
I don’t want to go too low on the number of years being a starter. Take a crap team that’s always drafting in the single digits. They’d BETTER draft a guy who is more likely to be a starter than the team that picks 32nd, for two reasons: first, the current players on the team drafting 4th aren’t very good (or they wouldn’t be drafting 4th), and second, the guy they get SHOULD be significantly better than the guy they’d get at #32. So it’s much more likely that the guy drafted 4th is going to replace the schlep currently playing for that lousy team. And the converse is true for the guy drafted at 32…. Most like he’s a worse player (than the 4th pick) trying to replace a current player who’s better than the guy on the crappy team picking 4th.

So you need to see if the guy can at least be a starter for a few years before you determine if the pick was actually any good.
 

InstaFace

The Ultimate One
SoSH Member
Sep 27, 2016
21,753
Pittsburgh, PA
From the explanation that PFR's founder, Doug Drinen, gave about the Approximate Value metric:

"Essentially, AV is a substitute for --- and a significant improvement upon, in my opinion --- metrics like 'number of seasons as a starter' or 'number of times making the pro bowl' or the like. You should think of it as being essentially like those two metrics, but with interpolation in between. That is, 'number of seasons as a starter' is a reasonable starting point if you're trying to measure, say, how good a particular draft class is, or what kind of player you can expect to get with the #13 pick in the draft. But obviously some starters are better than others. Starters on good teams are, as a group, better than starters on bad teams. Starting WRs who had lots of receiving yards are, as a group, better than starting WRs who did not have many receiving yards. Starters who made the pro bowl are, as a group, better than starters who didn't, and so on. And non-starters aren't worthless, so they get some points too."

I submit that their "Drafting-team AV" number, which should be in the same dataset you're using, is probably a better metric than either of the ones you're looking at. It better-captures the full range of player quality, makes allowances for players who are just-under or massively-above your thresholds, and allows us to more precisely answer the questions I posed in my last reply. It's also usable only 4 seasons of play removed from the draft year, so although the story isn't fully closed on the careers of players in the 2018 draft, it pretty much is for the purpose of assessing what the drafting team got from them, for all the players who weren't so good that the drafting team decided to sign them to an extension or keep them in free agency (which is a small minority). You definitely wouldn't have to stop at the 2013 Draft.

More significantly, it gets at the problem you're talking about in post #16, which is that the draftees of crap teams don't have as high a bar to clear to be a starter as the draftees on good teams (in general), and answers it through the AV metric. Teams and positional units that are collectively crap will generate less AV to be distributed among their various players, even if any team's stars will still come off well by the metric.
 

BaseballJones

ivanvamp
SoSH Member
Oct 1, 2015
24,369
Yep and I’ve been working with AV as well. One problem with it is you could have one or two massive AV seasons and that’s it and the overall AV number turns out ok.

But more than that…. I just don’t know how they calculate AV. Especially when it comes to, say, offensive linemen. It’s a black box for me. Unless they’ve shared their formula somewhere and I’ve just missed it.

Edit: and mainly, the OP was just meant to answer the question that actually was debated in another thread about how likely it is to draft a starter. I added in Pro Bowls as a bonus. It’s definitely more comprehensive to look at it from a host of angles, including AV.

I’ll get there.
 

BaseballJones

ivanvamp
SoSH Member
Oct 1, 2015
24,369
Ok after some more work on this, taking some suggestions into account, here's some more insight...

64373

First, if we're looking at drafting a 3-year starter, we can see that on average, teams draft 2.5 players per draft that end up starting 3 or more seasons for them. So in two drafts, an average team will draft five "starters".

Second, if we're talking about players who make it to just *one* Pro Bowl, the average team drafts one player per year that makes it to one Pro Bowl over the course of their career.

Third, if you're talking about AV....the average team drafts 138.3 career AV each draft. That comes to about 17.3 career AV per pick per year.

Again, you can see what years have been better and worse. The 2006 draft really stands out as an excellent draft. 2007 was a bad draft across the NFL.

Now how about the Pats over this stretch of time?

- 3-year starters: 33 out of 124 picks (26.6%), which is below the league average.

- 1x Pro Bowl: 17 out of 124 picks (13.7%), which is above the league average.

- AV: 18.8 career AV per pick, which is above the league average. 167.3 career AV per draft, again above average.

NOW...on that last point... obviously Tom Brady skews this number in a tremendous way. Not counting 2022 (as you know already, this data was gathered before the 2022 season), he accounted for 179 career AV. So not counting him, the Pats averaged 17.6 career AV per pick, which is still above average - just a tick above average, but still, above average.

But of course, we shouldn't discount the Brady pick. It was, quite literally, in virtually every way possible, the single best draft pick in the history of the NFL. Brady has won more than anyone in NFL history, and has the highest career AV in NFL history. And not only did the Patriots draft him, they drafted him in a spot where you'd expect very little career AV. So he's not only the most productive draft pick in history, he's also by FAR the greatest VALUE ever in an NFL draft pick.

But even if you remove him from the equation (while leaving Drew Brees, Aaron Rodgers, Calvin Johnson, etc., for other teams), the Pats have drafted generally above average, and again, that doesn't even take into account the Pats' draft slot. Which is the next phase of my project.
 
Last edited:

BaseballJones

ivanvamp
SoSH Member
Oct 1, 2015
24,369
They have, in fact, shared how they arrive at the AV numbers.

https://www.pro-football-reference.com/about/approximate_value.htm

Doug Drinen developed it in a series of forum threads which are also linked at the bottom of that page, if you want to read the back-and-forth between him and other nutcases.
Thanks. That seems pretty complicated, and reliant on TEAM success. Which I guess makes sense in some way. As long as it's a reasonable metric and applied across the board, it's fine. Just remember that he himself says he's not using it as any sort of definitive statement on a player's value, but rather just a quick estimate that should line up roughly with what we think. As he puts it, "Again remember that the goal here is not to forever put an end to the debate about whether Daniel Graham or Antonio Bryant has had the better career. That’s too ambitious a goal. We simply want to classify them both as being a bit better than Michael Bishop or Travis Dorsch, but not as good as Terry Glenn or Carson Palmer."

But it works for our discussion here.
 

InstaFace

The Ultimate One
SoSH Member
Sep 27, 2016
21,753
Pittsburgh, PA
So I didn't want to be left out of the fun, so I went through the minor bother of downloading all the draft histories and evaluations from PFR, from 1995 (when they went from 28 to 30 teams) to 2018 (the most recent year we could reasonably start to evaluate the quality of a pick retrospectively). And I've put my analysis up at this google sheet which is publicly accessible. This yielded a little over 6000 picks worth of data.

Here are some probabilities of a pick at a given pick # resulting in a thing, plus some regression curves which I am using to give a fairer estimate against which to evaluate a team's pick quality.

value-by-pick-regressions.png

Those measures are, respectively, "Starter 3+ years", "Starter 5+ years", "Pro Bowl 1+ time", "All-Pro 1st Team 1+ time", "Played in at least 30 games", and "Accumulated at least 10 AV". I have a separate one for the average expected EV by draft pick, for the drafting team (DrAV).

The regressions are either Exponential or Logarithmic, whatever resulted in higher R^2. Interestingly, the regressions for the DrAV one were about the same for Exponential and Logarithmic and so I got to decide whether I'd prefer slightly greater accuracy at the top end (early 1st round) of the draft, or better fit for the vast majority of the back end of the drafts. I chose the latter.

drav-by-pick-regression.png

Next up I'm going to:
- Assess variance of pick #s, using these value measures
- use those regression values as a baseline upon which to evaluate each team's picks over this window
- Calculate a team's Value Over Replacement Pick that they've created, positive or negative, using some weighted average of some of these measures. I'll probably give people a settings panel where they can adjust weights to their liking and decide how teams have done.

Notes:
- I'm going to focus on picks 1-256, even though a few drafts had more (and frankly I might reduce to 1-240 except that it doesn't really change anything).
- My input to the regressions started with a nearest-neighbor smoothing using the adjacent two values, even though I probably shouldn't have done that before the regression.
- AP1 is super random, Pro Bowl not much less so. AV, Games Played and Years Started are definitely better measures.
- There was such a big jump in average AV by pick right at Pick #199, even averaged over 24 years, that it was actually the impetus for me to do regression smoothing rather than use the real-world (noisier) values.

If you want to play around with this data or take advantage of some of the data-prep steps I've done, feel free to Make A Copy on google sheets and go to town on it.
 

BaseballJones

ivanvamp
SoSH Member
Oct 1, 2015
24,369
Great stuff, @InstaFace and I definitely want more people in on this kind of analysis.

I'm working through the 2000-2018 drafts (the caveat being, again, that this was data culled and sorted before the 2022 season, so if you look it all up now it'll take some adjustments, but it's still plenty big enough of a sample size for the purposes of this study). Here's some more thoughts.

I looked at every single pick and slotted them over that 19-year period. In other words, looking at all 19 #1 picks, all 19 #117 picks, all 19 #232 picks, etc. From 1 through 261 (there weren't many pick number 261 but still). Over this time period, here's all the draft slots the Pats have had from 2000-2018:

Round 1 - 6, 10, 13, 17, 21, 21, 21, 21, 23, 24, 25, 27, 29, 31, 32, 32, 32
Round 2 - 33, 34, 36, 36, 40, 41, 42, 45, 46, 48, 48, 52, 53, 56, 56, 58, 59, 60, 62, 62, 63, 64, 65
Round 3 - 73, 74, 76, 78, 78, 83, 83, 83, 84, 85, 86, 86, 90, 90, 91, 91, 94, 95, 96, 97, 97
Round 4 - 96, 101, 102, 105, 106, 111, 112, 113, 113, 117, 117, 118, 119, 120, 123, 126, 127, 127, 128, 129, 130, 131, 131, 133, 140
Round 5 - 136, 138, 141, 143, 150, 153, 159, 161, 163, 164, 164, 166, 170, 170, 171
Round 6 - 178, 178, 179, 180, 180, 187, 191, 194, 197, 197, 198, 198, 199, 200, 201, 201, 202, 202, 205, 205, 206, 206, 207, 208, 208, 209, 210, 211, 214, 221
Round 7 - 208, 211, 216, 219, 219, 224, 225, 226, 226, 229, 230, 232, 233, 234, 234, 235, 235, 237, 239, 239, 239, 243, 243, 244, 247, 247, 247, 248, 250, 250, 253, 253, 255

So here's their average draft slot for these rounds:

Round 1 (17 picks): 23
Round 2 (23 picks): 50
Round 3 (21 picks): 82
Round 4 (25 picks): 119
Round 5 (15 picks): 157
Round 6 (30 picks): 199
Round 7 (33 picks): 235

Here's the average career AV for those draft slots from 2000-2018:

#23: 31.0
#50: 27.8
#82: 12.4
#119: 11.0
#157: 5.4
#199: 11.8 (this is the Tom Brady slot, which warps EVERYTHING)
#235: 5.1

So if you take the Pats' picks from 2000-2018, here's the expected career AV from all those picks:

Round 1 (17 picks at average pick #23): 527
Round 2 (23 picks at average pick #50): 639
Round 3 (21 picks at average pick #82): 260
Round 4 (25 picks at average pick #119): 275
Round 5 (15 picks at average pick #157): 81
Round 6 (30 picks at average pick #199): 354
Round 7 (33 picks at average pick #235): 168

Here's what the Pats, over that time, have actually gotten in career AV.

Round 1 (17 picks): 827 (+300)
Round 2 (23 picks): 584 (-55)
Round 3 (21 picks): 260 (-1)
Round 4 (25 picks): 409 (+134)
Round 5 (15 picks): 156 (+75)
Round 6 (30 picks): 360 (+6)
Round 7 (33 picks): 206 (+38)

But #199 is Brady, which throws EVERYTHING off. So let's get rid of pick #199 and pretend it's pick #198, and remove the Brady pick entirely. The average career AV for pick #198 is 4.9, so if the Pats had taken 29 picks at an average pick of #198 their expected career AV from round 6 would be 142.0. And we'll eliminate Brady from the Pats' calculation. Their round 6 then looks like this:

Round 6 (29 picks): 59 (-83)


So if we include the Brady pick (and we SHOULD, but there will be detractors from that argument), the Pats have:

- Gotten a career AV of 2,801
- Had an expected career AV of 2,305
- Gotten a net career AV of +496 over expected (which comes to +3.0 per pick)

If you remove Brady then the Pats have:

- Gotten a career AV of 2,500
- Had an expected career AV of 2,092
- Gotten a net career AV of +408 over expected (which comes to +2.5 per pick)


LONG story short....... The Patriots have drafted very very well over the years, especially when you factor in WHERE in the draft they have been. They kill it in the first round, they struggle in the second round (and a tiny bit in the third), and they're really solid everywhere else.
 

InstaFace

The Ultimate One
SoSH Member
Sep 27, 2016
21,753
Pittsburgh, PA
I'm working through the 2000-2018 drafts (the caveat being, again, that this was data culled and sorted before the 2022 season, so if you look it all up now it'll take some adjustments, but it's still plenty big enough of a sample size for the purposes of this study). Here's some more thoughts.
So firstly, yeah, my data was pulled fresh last night, and it's just a straight copy/paste from the PFR Draft pages for each year. So it should be the same underlying format as whatever you're using (I added a Year column and a sequence number in the front for easier sorting, but that's it), if you want to update yours, or just start from mine.

I looked at every single pick and slotted them over that 19-year period. In other words, looking at all 19 #1 picks, all 19 #117 picks, all 19 #232 picks, etc. From 1 through 261 (there weren't many pick number 261 but still). Over this time period, here's all the draft slots the Pats have had from 2000-2018:

...

So here's their average draft slot for these rounds:

....

Here's the average career AV for those draft slots from 2000-2018:

...

So if you take the Pats' picks from 2000-2018, here's the expected career AV from all those picks:

...

Here's what the Pats, over that time, have actually gotten in career AV.
So I think this sort of pick-position-averaging actually tends to remove a lot of information. For one thing, the decay in pick value as pick number increases is exponential, not linear, so an arithmetic averaging probably under-sells the expected value of picking towards the end of rounds as the Pats usually did. For just about any measure other than "Odds of playing 30 Games", which is, somewhat oddly, almost linear.

What I'm going to do instead is calculate the expected value of each specific pick the teams each made, whatever number it was in that year, across a range of value measures. And then look at what they got with that pick, above/below expectation. I think that'll reduce the noise and also permit easier subdividing by era, by position, by pick #, by team, and so on.

Where I need advice (or clearer thinking) is on two things:

(1) Binary or continuous value measures? Simple flags of "did they get a starter who started for X years?" ignore the value difference between getting, say, a starter for 2 years vs 0, or a starter for 5 years vs 10. It's fine to use that in some contexts, but measures that have a fuller value range are probably conveying more information. But it's using those simple thresholds, of did the guy clear this bar or not, do end up somewhat mitigating the concerns about "a bad team will play more draft picks because their existing players suck more" (because a team's context usually changes every few years, and because a simple Yes/No threshold test reduces the need to even think about the team context factors that might turn a 30 AV player into a 50 AV player, or a would-be 3-year starter into a 7-year starter (or the reverse)). It focuses the information on less noisy facts. It's been a while since information theory class, but I think that helps if and only you have enough data points to get a good estimate of central tendency even with the information loss of reducing it to a boolean Yes/No. Do we? Each pick position only has N=24 (or fewer at the tail end). I just don't know whether binary measures will end up having more or less information loss than the continuous ones.

(2) So those continuous value measures - things like AV, Games Played, or # of Years Started - might be preferable. On the third hand, per the PFR commentary above, half the reason AV was created was to be an improvement over the cruder measures like Games or Years Started. So maybe AV is our do-it-all stat for value capture? Or should we take several of them and weight them together to assess a pick's value more holistically? A high-AV player who doesn't end up starting that many years because he gets hurt (say, star RBs) is still very valuable for the years he does play. A lower-end defensive lineman who might not make the pro bowl but is very good at avoiding injury and staying on the field and accumulates lots of games played, even if he's not always the primary starter, has value too, just in a different way. AV does attempt to interpolate between these different approaches to value, but maybe we should be using all of them anyway, just so we don't over- or under-state a player's value to a team. Which ones would you focus on?

So if we include the Brady pick (and we SHOULD, but there will be detractors from that argument), the Pats have:

- Gotten a career AV of 2,801
- Had an expected career AV of 2,305
- Gotten a net career AV of +496 over expected (which comes to +3.0 per pick)

If you remove Brady then the Pats have:

- Gotten a career AV of 2,500
- Had an expected career AV of 2,092
- Gotten a net career AV of +408 over expected (which comes to +2.5 per pick)
I'm having a bit of trouble here, because Brady's AV was 184 for his career, 168 for the Pats, and yet the difference between your with-Brady and without-Brady expected career AVs is 213, and actual-AV numbers is 301. If we replace Brady with an average pick at that position (with AV around 4.9), the difference should be less, if all else was held the same. What happened?


As for your musings about adjusting for Brady: There's no reason to ignore the Brady pick, imo. It's the most consequential outcome-altering team decision in the history of the NFL. It, like, happened, and then he went and did all that shit. The flags will fly forever. But also, at this point, it's far from the entire story of the Patriots' drafting over the last few decades. As you point out, his 168 AV for the Patriots is amazing but it's also like ~5% of the total AV you'd have expected for the Patriots over the course of time. The data is big enough at this point that conclusions won't be significantly altered even by an outlier as big as that one. So I think we keep him in, and give the Pats full credit for that miracle (which was of course the result of careful scouting and Belichick listening to different perspectives).
 

BaseballJones

ivanvamp
SoSH Member
Oct 1, 2015
24,369
So firstly, yeah, my data was pulled fresh last night, and it's just a straight copy/paste from the PFR Draft pages for each year. So it should be the same underlying format as whatever you're using (I added a Year column and a sequence number in the front for easier sorting, but that's it), if you want to update yours, or just start from mine.
Thanks for that!

So I think this sort of pick-position-averaging actually tends to remove a lot of information. For one thing, the decay in pick value as pick number increases is exponential, not linear, so an arithmetic averaging probably under-sells the expected value of picking towards the end of rounds as the Pats usually did. For just about any measure other than "Odds of playing 30 Games", which is, somewhat oddly, almost linear.

What I'm going to do instead is calculate the expected value of each specific pick the teams each made, whatever number it was in that year, across a range of value measures. And then look at what they got with that pick, above/below expectation. I think that'll reduce the noise and also permit easier subdividing by era, by position, by pick #, by team, and so on.
Here's the reality: you are smarter than me about how to really analyze this data. I'm a regular guy that barely got through calculus, so when I do this kind of work, I really have to keep things simple. I wish I knew this stuff as well as you do, but alas..... :)

Where I need advice (or clearer thinking) is on two things:

(1) Binary or continuous value measures? Simple flags of "did they get a starter who started for X years?" ignore the value difference between getting, say, a starter for 2 years vs 0, or a starter for 5 years vs 10. It's fine to use that in some contexts, but measures that have a fuller value range are probably conveying more information. But it's using those simple thresholds, of did the guy clear this bar or not, do end up somewhat mitigating the concerns about "a bad team will play more draft picks because their existing players suck more" (because a team's context usually changes every few years, and because a simple Yes/No threshold test reduces the need to even think about the team context factors that might turn a 30 AV player into a 50 AV player, or a would-be 3-year starter into a 7-year starter (or the reverse)). It focuses the information on less noisy facts. It's been a while since information theory class, but I think that helps if and only you have enough data points to get a good estimate of central tendency even with the information loss of reducing it to a boolean Yes/No. Do we? Each pick position only has N=24 (or fewer at the tail end). I just don't know whether binary measures will end up having more or less information loss than the continuous ones.

(2) So those continuous value measures - things like AV, Games Played, or # of Years Started - might be preferable. On the third hand, per the PFR commentary above, half the reason AV was created was to be an improvement over the cruder measures like Games or Years Started. So maybe AV is our do-it-all stat for value capture? Or should we take several of them and weight them together to assess a pick's value more holistically? A high-AV player who doesn't end up starting that many years because he gets hurt (say, star RBs) is still very valuable for the years he does play. A lower-end defensive lineman who might not make the pro bowl but is very good at avoiding injury and staying on the field and accumulates lots of games played, even if he's not always the primary starter, has value too, just in a different way. AV does attempt to interpolate between these different approaches to value, but maybe we should be using all of them anyway, just so we don't over- or under-state a player's value to a team. Which ones would you focus on?
I think looking at it in a variety of ways gives you the best overall picture. Just as a different example, imagine a team making 70 draft picks over a ten year period. It tells one story to look at how many starters they drafted. It tells another story to look at how many superstars they drafted. AV kind of gives you both - a big AV number is only possible if a guy is (a) really, really good, and (b) plays a long time. But it doesn't tell you everything and in many ways is team dependent, so it's not necessarily a fair reflection on any individual player taken in isolation. I don't think we need to boil it down to one metric.

I know that means more work, but it appears that you and I are enjoying it so..... :)

I'm having a bit of trouble here, because Brady's AV was 184 for his career, 168 for the Pats, and yet the difference between your with-Brady and without-Brady expected career AVs is 213, and actual-AV numbers is 301. If we replace Brady with an average pick at that position (with AV around 4.9), the difference should be less, if all else was held the same. What happened?
I just removed the pick entirely and didn't replace Brady with an average pick. I just took the entirety of Brady's career (2000-2021, because, again, I am using the data from before the 2022 season) of 301 AV. I just took all 301 AV off the board from the Pats' total.

As for your musings about adjusting for Brady: There's no reason to ignore the Brady pick, imo. It's the most consequential outcome-altering team decision in the history of the NFL. It, like, happened, and then he went and did all that shit. The flags will fly forever. But also, at this point, it's far from the entire story of the Patriots' drafting over the last few decades. As you point out, his 168 AV for the Patriots is amazing but it's also like ~5% of the total AV you'd have expected for the Patriots over the course of time. The data is big enough at this point that conclusions won't be significantly altered even by an outlier as big as that one. So I think we keep him in, and give the Pats full credit for that miracle (which was of course the result of careful scouting and Belichick listening to different perspectives).
I agree that we should keep Brady in the discussion, because he was by far the greatest draft pick in NFL history (and there is NOBODY that's even close), and it DID happen. It's just that they made so many other picks that Brady's monumental amount of AV impacts the entire data set and I don't want people to respond with, "Well, it's because they got lucky with Brady" as if BB didn't also make a ton of other great picks over that stretch of time. It's like...Wilt's 100 point game DID happen and it absolutely DOES count, but it also skews his season scoring average and doesn't really give you the picture of what his whole body of work for that season was, so if you take it out, it's a little more reflective of his every day work.
 

BaseballJones

ivanvamp
SoSH Member
Oct 1, 2015
24,369
Regarding Brady...

He produced 326 career AV. The average #199 pick (not counting Brady) produces an expected career AV of about 4.9. So Brady produced an excess of 321.1 career AV.

Peyton Manning, in comparison, produced 277 career AV (44 fewer than Brady), but he was the #1 pick in the NFL draft - a spot where the expected career AV is 63.3. Thus, Peyton produced an excess of 213.7 career AV.

So the difference in those two excess AV numbers is 107.4. To give you a sense of how big that gap of excess is, consider that Hall of Famer Andre Tippett had a career AV of 109.

Brady produced basically as much *excess* AV (relative to their respective draft slots) over Peyton Manning as Andre Tippett accumulated in his entire career.
 

InstaFace

The Ultimate One
SoSH Member
Sep 27, 2016
21,753
Pittsburgh, PA
In fairness, Wilt Chamberlain averaged over 50 points a game that season, so while the 100 point game was an outlier and a target they decided to hit, it doesn't give you a misleading impression of his season, either :)
 

MainerInExile

Well-Known Member
Lifetime Member
SoSH Member
Nov 21, 2003
4,825
Bay Area
The round-by-round breakdown seems to correspond well to the conventional wisdom around here. They hit in the first round more often than not, and they do well to find contributors in later rounds. Round two is maddening.
 

ZMart100

Member
SoSH Member
Aug 15, 2008
3,188
I did something similar to what I think Instaface is suggesting a few years ago looking at starts. Position matters. Examples: WR
64434
64435
The parameterization I used did lead to some funky things happening late in the draft though.
64436
 

ZMart100

Member
SoSH Member
Aug 15, 2008
3,188
Seems like you wanna enforce some monotone condition - how did you parametrize these?
It was awhile ago and I don't want to dig up the code. I have a vague memory that it was actually non-parametric loess, but I'm sure I was playing with other things too.
 

InstaFace

The Ultimate One
SoSH Member
Sep 27, 2016
21,753
Pittsburgh, PA
Here's the reality: you are smarter than me about how to really analyze this data. I'm a regular guy that barely got through calculus, so when I do this kind of work, I really have to keep things simple. I wish I knew this stuff as well as you do, but alas..... :)
Thanks for your kind words. I think I'm about at the same level as you and the other data-literate people around here when it comes to crunching sports data, not least because I have a lot less experience in that data realm, and also have less subtle context about the game itself or how to control for different variables. And there's certainly nothing at the level of calculus that we're doing here regardless. Anyway, we're on the same page and same team here, let's see what we can learn together.

I think looking at it in a variety of ways gives you the best overall picture. Just as a different example, imagine a team making 70 draft picks over a ten year period. It tells one story to look at how many starters they drafted. It tells another story to look at how many superstars they drafted. AV kind of gives you both - a big AV number is only possible if a guy is (a) really, really good, and (b) plays a long time. But it doesn't tell you everything and in many ways is team dependent, so it's not necessarily a fair reflection on any individual player taken in isolation. I don't think we need to boil it down to one metric.

I know that means more work, but it appears that you and I are enjoying it so..... :)
This is fair. I think I will:

- Add a few other continuous metrics, such as # Years Starter, # Games Played, to go with our binary / threshold flags (and the DrAV stat which is already continuous). More recent years will have more noisy data for those stats, because ideally you'd want them to accumulate over a few more years, but we can probably make that adjustable.
- Concoct some sort of weighted average for them, but put out multiple sets of conclusions shaped by different sets of metrics or combinations thereof
- Try to get some sort of team-performance context into there, because like the Browns have been terrible forever so a metric that heavily weights Games Played will make them look better, while a metric that weights AV will make them look worse

I just removed the pick entirely and didn't replace Brady with an average pick. I just took the entirety of Brady's career (2000-2021, because, again, I am using the data from before the 2022 season) of 301 AV. I just took all 301 AV off the board from the Pats' total.
I must be seeing some different data here because on Tom Brady's PFR page it's got him at 326 career AV (316 excluding 2022, given your data), and when you look at... oh wait, OK, the column in the draft page is actually "weighted career AV", downweighting his lesser years, and for the handful of players who played over 20 years excluding his worst years entirely. So that's an odd stat, I wish they'd just put in career AV as a column. But then you've got the DrAV column, AV accumulated for the team that Drafted this player - it's got Brady at 168. You go and look at his page, though, and his career AV for the Patriots was 285, a number you should have the same as I see it there. That's a huge difference! What on earth is this site doing here? I'm going to file a support ticket. DrAV is broken or poorly documented, and they need to just have Career AV in there as a no-shenanigans option.


Here's the latest progress.

- Switched to Logarithmic formula for (Drafting-team) AV by pick #, as that's a lot more accurate if you don't do moving-average smoothing first
- Added continuous-value metrics for # Games Played and # Years Started, per the above
- Couldn't decide from among two different methods for scoring a pick, so the spreadsheet's control panel now offers both. For continuous (counting-stat) metrics, the choices are a 100-based index or a z-score (recentered to 100). For threshold (binary yes/no) metrics, there are both linear and exponential scoring choices.
- Unsurprisingly, the standard deviations for AV, GP and YS (Years Started) all have a huge spike at pick #199. :) So I went and fit StDev to a trendline and calculated the (regression-expected) StDevs for those, as well.
- While AV had a steady slope to the standard deviation (R^2 = 0.568 on a logarithmic fit), as did YS (R^2 = 0.589 on an exponential fit), GP meanwhile had basically no pattern at all, with R^2s all right around 0.1 for every regression type. So I'm plugging in a baseline StDev of 46.9 (the average StDev for all pick #s) for Games Played, and calling it a day. Any pick is equally high-variance when it comes to games played, it seems.
- For the very tail end of the regression-driven formulas for predicting odds for Pro Bowl and All-Pro selections, the predicted odds went negative, which thwarts the scoring. I have switched the regression for Pro Bowl from Logarithmic to Exponential starting after the 4th round, which doesn't fit as well overall but does not decay to negatives. For All-Pro, I switched to a Power Series fit line partway through the 2nd round (picks 39+), which fits better overall and also does not decay to negatives.
- Added scoring evaluations (based on these metrics) for each pick in our range
- Created average scores for each team

Results are visible at the same sheet.

View: https://docs.google.com/spreadsheets/d/1C9vUQZtSFjQ-3REqvLzvNwGmyQLN3fUKrszx2TPzXu8/edit#gid=949691895


And the latest dilemma:

The "Drafting-team AV" metric is wonky because of draft-day trades. For example, Eli Manning is a DrAV of 0 for the team that drafted him (the San Diego Chargers), but it's not fair to say they "missed" on him (nor that the Giants "missed" on Rivers): the Chargers got Philip Rivers and 1st- and 3rd-rounders, pretty close to full value; the Chargers are also getting credit for the quality of those picks received in trade. My sense is, the number of Eli Manning situations (draft-and-trade; shouldn't penalize drafting team) greatly outnumber the number of Lawrence Guy situations (underutilized, released / waived from a couple of teams before later finding success; drafting team should get no credit, was a 0 for the drafting team). So for the ~10% of picks where DrAV = 0 but Career AV > 0, which includes both Eli and Guy, I'm giving the team full credit for the Career AV, because between the two imperfect options that seems like the less-bad choice. A third option would be "ignore", i.e. act as if the team never made the pick, they don't get a zero and they don't get the player's result. But I'm very unsure about this. I'm happy to provide a list of the 31 players who have > 20 AV in this situation (i.e. the ones that matter) and let people look through and decide if, in general and as a group, the drafting team should get full credit, or none, or ignore. It's a reversible decision, but not one I can make into a flexible user option all that easily.

2017 7 233 CAR Harrison Butker K
2016 2 41 BUF Reggie Ragland ILB
2016 6 208 NWE Kamu Grugier-Hill OLB
2015 7 222 WAS Austin Reiter C
2011 7 233 GNB Lawrence Guy DT
2010 4 123 NOR Al Woods DT
2010 6 205 NWE Ted Larsen C
2009 3 79 PIT Kraig Urbik T
2009 7 226 PIT A.Q. Shipley C
2008 6 167 DAL Erik Walden DE
2006 6 191 NWE Jeremy Mincey DE
2005 6 207 CAR Joe Berger T
2005 6 213 BAL Derek Anderson QB
2005 7 216 MIA Kevin Vickerson DT
2004 1 1 SDG Eli Manning QB
2004 1 4 NYG Philip Rivers QB
2003 5 166 GNB Hunter Hillenmeyer LB
2003 6 185 PHI Jeremy Bridges G
2003 7 255 NYG Kevin Walter WR
2002 7 238 PHI Raheem Brock DE
2000 6 168 NOR Marc Bulger QB
2000 7 253 DET Alfonso Boone DT
1999 4 131 GNB Aaron Brooks QB
1999 6 173 CIN Kelly Gregg NT
1998 6 169 SEA Bobby Shaw WR
1998 7 207 DET Chris Liwienski G
1997 4 119 PHI Damien Robinson DB
1997 6 169 TAM Al Harris DB
1995 4 122 MIA Pete Mitchell TE
1995 7 241 ARI Chad Eaton DT
1995 7 242 NOR Travis Davis DB
 

Eddie Jurak

canderson-lite
Lifetime Member
SoSH Member
Dec 12, 2002
44,475
Melrose, MA
Has anyone looked at player age when drafted vs. overrall value and 4-year (ie, first contract) value?

It has sort of struck me in the past draft or two that the Pats seem to be drafting some older players... but at least some of that seemed possibly related to Covid disruptions and it isn't clear to me whether they are doing anything different than the rest of the league in this regard.

I would also assume that the pool of players who choose to enter the draft early (and thus younger) tend to be more talented, and this would have to be filtered out to really assess whether older is better.
 

BaseballJones

ivanvamp
SoSH Member
Oct 1, 2015
24,369
Has anyone looked at player age when drafted vs. overrall value and 4-year (ie, first contract) value?

It has sort of struck me in the past draft or two that the Pats seem to be drafting some older players... but at least some of that seemed possibly related to Covid disruptions and it isn't clear to me whether they are doing anything different than the rest of the league in this regard.

I would also assume that the pool of players who choose to enter the draft early (and thus younger) tend to be more talented, and this would have to be filtered out to really assess whether older is better.
This is a question I want to look at soon. I have argued that, given the relatively short careers most NFL players have, it may be preferable to draft an older player who by and large (there are obviously many exceptions) is more ready to help you now, without needing the kind of seasoning that a younger player might need. And though his ceiling may not be quite as high, the overall value he brings you is better, especially on that first contract.

But I don't know if my hypothesis is true, and I'll be looking at that soon.

It's a great question.
 

SMU_Sox

queer eye for the next pats guy
SoSH Member
Jul 20, 2009
8,878
Dallas
I can’t find it but the Patriots in the last 5 years or so and I believe overall tend to draft older players. In the last 5-6 years or whatever they have been the oldest in the NFL. I’ll try and find it. They love senior bowl guys.
 

SMU_Sox

queer eye for the next pats guy
SoSH Member
Jul 20, 2009
8,878
Dallas
Almost everyone they drafted this year who I watched was a senior bowl or shrine bowl guy including 3/3 of their day 1 and 2 picks being senior bowl guys.
 

luckiestman

Son of the Harpy
SoSH Member
Jul 15, 2005
32,617
Almost everyone they drafted this year who I watched was a senior bowl or shrine bowl guy including 3/3 of their day 1 and 2 picks being senior bowl guys.

Do you listen to Bucky and Jeremiah? If so, what did you think about their discussion of the Colts' trait-based approach?
 

SMU_Sox

queer eye for the next pats guy
SoSH Member
Jul 20, 2009
8,878
Dallas
Do you listen to Bucky and Jeremiah? If so, what did you think about their discussion of the Colts' trait-based approach?
I think there isn’t anything new about having traits based scouting and drafting - BB has tended towards high athleticism in early picks even more than average. Think WRs round 2 for them for example or their OL picks. Bill also likes to take athletes late. Keion Crossen comes to mind. A lot of other teams and GMs take similar approaches. Coaches have been putting athletes through development for a long time. But here Bucky believes the Colts are going “all in” on this approach. I’m skeptical that going all-in on this theory is going to work out. For one even with position coaching depth most of these guys are refining core skills in the off-season with private coaches. You also have lower success rates with guys who don’t have production in college. Taking both great athletes with production is ideal but Bucky is saying the Colts are trading the former for the latter. You also get into complications with how much guys can develop on the fly. You aren’t going to start a developmental OT because his ass is going to give up way too many sacks and pressures.
Committing to developing a high traits QB isn’t new either and teams are doing s better job integrating QBs now than they were in the mid 2000s for sure. The NFL game is closer to the college game offensively than it has been in a long time. So I do think you can develop AR in that way by letting him develop with reps vs being able to do that vs the developmental OT. Some positions you can’t or shouldn’t develop in live action.
I’m curious to see how it works out as well. I believe it will work out for AR and SS did wonders for JH in Philly. But the other positions? I don’t know. Also I think AR wasn’t necessarily raw but just inexperienced. Guys who still look raw but have experience and are good athletes imo are much less likely to succeed.
 

BaseballJones

ivanvamp
SoSH Member
Oct 1, 2015
24,369
This is a question I want to look at soon. I have argued that, given the relatively short careers most NFL players have, it may be preferable to draft an older player who by and large (there are obviously many exceptions) is more ready to help you now, without needing the kind of seasoning that a younger player might need. And though his ceiling may not be quite as high, the overall value he brings you is better, especially on that first contract.

But I don't know if my hypothesis is true, and I'll be looking at that soon.

It's a great question.
I'll just say that as I get started on this, it's difficult to parse out people's contributions on a first contract basis. I will not go through literally thousands of players and look at each of their year-to-year AV scores. That would take me weeks and months and maybe someone else has the time to do that, but that info is not available in an easy to read and sort form like the main draft data is (which takes into account their career AV numbers, not year-to-year numbers).

I'll still do my best with this but it won't be quite what some are looking for. Apologies.
 

BaseballJones

ivanvamp
SoSH Member
Oct 1, 2015
24,369
A bunch of guys in the PFR draft data don't have ages attached to them (255 of them to be exact). But of the rest drafted from 2000-2018, I grouped them by age and then by career wAV. Here's how it broke down by draft age and their average career AV for that age group:

Age 29 (1 player): 9.0 AV
Age 28 (4 players): 6.5 AV
Age 27 (2 players): 1.5 AV
Age 26 (24 players): 10.0 AV
Age 25 (70 players): 8.6 AV
Age 24 (619 players): 11.1 AV
Age 23 (2,625 players): 14.7 AV
Age 22 (2,470 players): 19.2 AV
Age 21 (773 players): 26.2 AV
Age 20 (47 players): 28.0 AV

So yeah, the younger you draft them, the more likely it is that they'll give you more career AV than if you draft them older. Which makes sense because they can play in the league longer and accumulate more AV.

But what I want to figure out is what @Eddie Jurak asked, which is, what is their AV for their first contract? I am not sure I can do THAT without literally going through every player's page and add up their AV their first four years (and for some it would be five years, and no way can I parse THAT). But what I can do is look at their DrAV, which is their value to their DRAFTED team. This is not perfect, because it says nothing about their first contracts. A player might get moved in year 3 of his 4 year rookie deal, and this doesn't help us with that. Or a player could play 8 years for their team and this reflects those 8 years but doesn't tell us about his first contract.

In any case, here's the DrAV data for those age groups:

Age 29 - 9.0 DrAV
Age 28 - 5.0 DrAV
Age 27 - 1.5 DrAV
Age 26 - 7.8 DrAV
Age 25 - 6.7 DrAV
Age 24 - 8.0 DrAV
Age 23 - 10.3 DrAV
Age 22 - 13.4 DrAV
Age 21 - 19.9 DrAV
Age 20 - 21.5 DrAV

Same pattern. Draft younger players and you get more out of them, both for the team drafting them and the NFL as a whole.

But that still doesn't tell me what I really want to know. I'll keep working on this.
 

Eddie Jurak

canderson-lite
Lifetime Member
SoSH Member
Dec 12, 2002
44,475
Melrose, MA
A bunch of guys in the PFR draft data don't have ages attached to them (255 of them to be exact). But of the rest drafted from 2000-2018, I grouped them by age and then by career wAV. Here's how it broke down by draft age and their average career AV for that age group:

Age 29 (1 player): 9.0 AV
Age 28 (4 players): 6.5 AV
Age 27 (2 players): 1.5 AV
Age 26 (24 players): 10.0 AV
Age 25 (70 players): 8.6 AV
Age 24 (619 players): 11.1 AV
Age 23 (2,625 players): 14.7 AV
Age 22 (2,470 players): 19.2 AV
Age 21 (773 players): 26.2 AV
Age 20 (47 players): 28.0 AV

So yeah, the younger you draft them, the more likely it is that they'll give you more career AV than if you draft them older. Which makes sense because they can play in the league longer and accumulate more AV.

But what I want to figure out is what @Eddie Jurak asked, which is, what is their AV for their first contract? I am not sure I can do THAT without literally going through every player's page and add up their AV their first four years (and for some it would be five years, and no way can I parse THAT). But what I can do is look at their DrAV, which is their value to their DRAFTED team. This is not perfect, because it says nothing about their first contracts. A player might get moved in year 3 of his 4 year rookie deal, and this doesn't help us with that. Or a player could play 8 years for their team and this reflects those 8 years but doesn't tell us about his first contract.

In any case, here's the DrAV data for those age groups:

Age 29 - 9.0 DrAV
Age 28 - 5.0 DrAV
Age 27 - 1.5 DrAV
Age 26 - 7.8 DrAV
Age 25 - 6.7 DrAV
Age 24 - 8.0 DrAV
Age 23 - 10.3 DrAV
Age 22 - 13.4 DrAV
Age 21 - 19.9 DrAV
Age 20 - 21.5 DrAV

Same pattern. Draft younger players and you get more out of them, both for the team drafting them and the NFL as a whole.

But that still doesn't tell me what I really want to know. I'll keep working on this.
Nice work, though the inability to easily do certain data cuts makes it harder. I think using total value is probably an adequate proxy for 4-year value, at least in the early going when that is what you have. :)

I think there might be some confounding issues. To enter the draft at age 20 means the player declared early, and early-declarers are likely to be better players than those who do their full time in college. The other potential issue that comes to mind is draft position.
 

BaseballJones

ivanvamp
SoSH Member
Oct 1, 2015
24,369
Nice work, though the inability to easily do certain data cuts makes it harder. I think using total value is probably an adequate proxy for 4-year value, at least in the early going when that is what you have. :)

I think there might be some confounding issues. To enter the draft at age 20 means the player declared early, and early-declarers are likely to be better players than those who do their full time in college. The other potential issue that comes to mind is draft position.
Yep, agreed. It's really hard to parse through all this to answer certain specific questions. I mean, if I had someone willing to go through every single player's page and track their draft position, their age, and their first four years' worth of AV, that would be awesome. But that would take weeks of effort. And I just don't have time for that. I don't know who does, to be honest.
 

InstaFace

The Ultimate One
SoSH Member
Sep 27, 2016
21,753
Pittsburgh, PA
Bumping this thread, because it needed an update in light of the debate over Bill Belichick. I've refreshed the data on the spreadsheet, and its calculations. My goal was to assess the question "is Belichick a bad drafter now / recently?", trying to put some quantification around it.

Recall that we have two categories of success measure:

1) "Threshold" measures: do picks hit a threshold of 'useful starter' as often as you'd expect based on draft slot? (e.g. 3+ years as starter, Pro Bowl selections, 10+ AV)
2) "Counting" measures: how many counting stats do picks rack up relative to expectations based on draft slot? (e.g. AV for the drafting team, Games Played)

Teams' draft histories that tend to be measured as good by one category tend to be rated good by the other, but it is not a super strong correlation (or else we'd have no need for both).

When considering seasons 2000 through 2018, New England ranks 4th and 2nd respectively, with index scores of 126.4 (26.4% above average expectation) and 120.9. Even if you adjust it to 2001 through 2018 (deliberately excluding the outlier Brady pick), New England would rank 7th (117.1) and 8th (109.6). Meaning, even leaving aside the greatest draft pick in NFL history (which Belichick did in fact make), Belichick was a well-above-average drafter.

When considering only draft years 2019 through 2021*, this now includes a lot of picks who haven't had the years in the league to hit the "useful starter" thresholds as much as their predecessors did. We're measuring over an average of ~2.5 years of performance, vs a model that is benchmarking against players with 4+ years to have hit milestones and racked up stats. As such, the baseline index values are not 100 (by definition), instead here they are 42.9 and 52.5 respectively. And it's a LOT noisier, as we're measuring over an average of about 24 draft picks per team, instead of 200+. But those caveats aside, by these two groups of measures, in this time period New England is 2nd in the league (!) by threshold measures, with a 72.4 score... but dead last, 32nd in the league, by counting measures, with a 39.1 score.

Diving into those a bit more, the threshold numbers are buoyed substantially by punter Jake Bailey making AP1 and Pro Bowl out of the 5th round, and CB Marcus Jones making AP1 last year. Only 7 of 28 picks have hit 10 AV thus far, and 15 of 28 have hit 30 games played. But by counting measures, we have only 2 players who have exceeded their expected counting stats by this point: Bailey, and OG Michael Onwenu. And by round, New England has been better than expected in the last 3 rounds of the draft (rounds 5-7), and worse in the first 2 rounds.

If we change our scoring approach for these success measures, going from "power-law" to "linear" for threshold, and going from "index" to "z-score" for counting measures, then the teams' average scores become very bunched up together. Nevertheless, New England ranks 22nd and 29th in the league under these scoring methods. Same story.

Lastly, if we switch to a middle-of-the-road year range of 2015 through 2021, on the assumption that Bill falling off the table wouldn't have happened all at once but rather gradually, it gives us more stability but still keeping a focus on recent picks. Under these ranges, and using the original scoring methods, New England ranks 6th by threshold measures (98.8 score vs league average of 72.2), and 12th by counting measures (78.3 score vs league average of 72.3). They are, again, rated strongly vs expectations in rounds 5 and 6, while well below expectations in their picks in rounds 1, 2 and 7 (especially #2).

Propping up Bill's ratings in that time window are Bailey, Elandon Roberts, Shaq Mason, Ted Karras... and ironically, WR Braxton Berrios, who of course was drafted here in 2018, spent the season on IR, was cut during the 2019 preseason, and then became an all-pro on the NYJ (as a kick returner). By my rating system, New England gets credit for drafting him, particularly in the 6th round; that's just a data limitation here. We thus need to rely on larger sample sizes in order for such anomalies to even out in both directions over time. I could also downweight noisier success measures like All-Pro and Pro Bowl, and upweight things like "threshold of 10 AV".

Conclusion: There is some weak evidence that in the most recent years we can measure, Bill Belichick has fallen off the table in terms of being able to generate value from draft picks, beyond that which you'd expect from merely what pick # you've got. However, over any longer of a time window (e.g. 7 years from 2015-2021), his team is still right up there near the top of the rankings. Reasonable people can disagree about the sample sizes involved, and the nuances of what sort of objective achievements are good measures of draft value.


* I had originally done it through 2022, but the data on the 2022 draftees is SO thin at this point that basically nobody has hit thresholds or gotten to meaningful differences in counting stats, so it's just adding noise. Revised the above to exclude 2022 and finish with 2021, which is still very aggressively early, but not stupidly so.
 

FL4WL3SS

Member
SoSH Member
Jul 31, 2006
14,913
Andy Brickley's potty mouth
I don't think it's Bill in as much as there has been a ton of FO turnover in recent years. I've been beating this drum, but they need to replace some folks advising BB in the FO. I don't think this is necessarily a BB problem directly.
 

JimD

Member
SoSH Member
Nov 29, 2001
8,681
I don't think it's Bill in as much as there has been a ton of FO turnover in recent years. I've been beating this drum, but they need to replace some folks advising BB in the FO. I don't think this is necessarily a BB problem directly.
Given that Belichick has full control of all aspects of football operations, it absolutely is his problem. Time will tell if Matt Groh and company are the answer to producing more impactful draft results.
 

BaseballJones

ivanvamp
SoSH Member
Oct 1, 2015
24,369
Given that Belichick has full control of all aspects of football operations, it absolutely is his problem. Time will tell if Matt Groh and company are the answer to producing more impactful draft results.
I agree. The buck stops with Bill, plain and simple. He gets credit when they draft well, and he gets blame when they don't.

One interesting note to that is that there were stories (that resurfaced this week) that Mac Jones was really Robert Kraft's draft pick, not Belichick's. That is, Belichick didn't mind the player but wasn't enamored with taking him at 15, but Kraft LOVED Mac and insisted on the Pats taking him.

I have no idea if that's true. But if it is, and if BB is giving Mac all these chances because Kraft is insisting on it, and BB is basically in "I'm showing Kraft how wrong he was about this" mode, then that's a totally different picture when it comes to looking at BB, in my opinion. And it might explain why a guy like Kraft would still be willing to stay with Belichick, because he would know that Mac was HIS pick, not BB's, and that it hasn't worked out obviously isn't therefore something to blame BB for.

Again, no idea if it's true, but it's not totally implausible.

Whatever the real deal is...the Pats are about to face maybe their most important draft in franchise history. They had better not screw it up.
 

tims4wins

PN23's replacement
SoSH Member
Jul 15, 2005
37,054
Hingham, MA
I agree. The buck stops with Bill, plain and simple. He gets credit when they draft well, and he gets blame when they don't.

One interesting note to that is that there were stories (that resurfaced this week) that Mac Jones was really Robert Kraft's draft pick, not Belichick's. That is, Belichick didn't mind the player but wasn't enamored with taking him at 15, but Kraft LOVED Mac and insisted on the Pats taking him.

I have no idea if that's true. But if it is, and if BB is giving Mac all these chances because Kraft is insisting on it, and BB is basically in "I'm showing Kraft how wrong he was about this" mode, then that's a totally different picture when it comes to looking at BB, in my opinion. And it might explain why a guy like Kraft would still be willing to stay with Belichick, because he would know that Mac was HIS pick, not BB's, and that it hasn't worked out obviously isn't therefore something to blame BB for.

Again, no idea if it's true, but it's not totally implausible.

Whatever the real deal is...the Pats are about to face maybe their most important draft in franchise history. They had better not screw it up.
Can you link any of those stories? This is the first I've read of this.