Let's talk (again) about PFF - BSMW article

Smiling Joe Hesketh

All Hail King Boron
Dope
May 20, 2003
31,600
Deep inside Muppet Labs
Bruce Allen over at BSMW ran a long piece today about PFF and their grading system. It's an interesting read and touches on quite a few points we've brought up here. It's well worth checking out the entire thing, then coming back here.
 
Some quotes:
 
 
It is important to note what Pro Football Focus is. Actually, first we’ll define what they are NOT.  They are not taking raw numbers and data and crunching them into new and exotic formulas to provide a different sort of insight into player performance. This is not sabermetrics for football.
 
No, their methods are different. They are a UK-based company, who obtain games through NFL Rewind and sit and watch and grade each player on each play. Their dedication to this is admirable, as I can’t imagine sitting down and doing this kind of deep grading for every play, every game week after week.
 
 
Allen then touches on his key question: how can they grade each player on the field if they don't know what his assignement was or where coverage/protection/route running broke down?
 
 
So sometimes even the team itself doesn’t know exactly where things broke down and who did what wrong. Belichick then went on to talk about watching opposing team’s game films and the impossibilities of knowing what happened:

But believe me, I’ve watched plenty of preseason games this time of year and you’re looking at all the other teams in the league and you try to evaluate players and you’re watching the teams that we’re going to play early in the season and there are plenty of plays where I have no idea what went wrong. Something’s wrong but I don’t…these two guys made a mistake but I don’t know which guy it was or if it was both of them. You just don’t know that. I don’t know how you can know that unless you’re really part of the team and know exactly what was supposed to happen on that play. I know there are a lot of experts out there that have it all figured out but I definitely don’t. This time of year, sometimes it’s hard to figure that out, exactly what they’re trying to do. When somebody makes a mistake, whose mistake is it?
Bill Belichick doesn’t have it figured out. But Pro Football Focus does? They can provide a grade on every play?
 
Another problem is that the NFL just recently added the coaches film to Game Rewind, so before that, the PFF graders could not even see the entire field. I don’t know if they currently even utilize the overhead game film, or just rely on the standard HD game telecasts.
 
There HAS to be a subjective element in the grading process. They have to be making conclusions based on conjecture and assumption or what they “think” the player was attempting to do or was assigned to do on any given play.

 
 
 
Again, this is not taking actual numbers and using them to come up with new stats to use in analytics. This is not taking passes complete and passes attempted and breaking it down into the various lengths of throws and spots on the field. This is sitting down in front of the monitor, forming an opinion and making up their own stats based on what they think is happening on each play.
 
 
It's long so I'm not going to quote the whole thing. Allen raises some very worthwhile questions (qualifications of the reviewers, surety of their conclusions, etc). Go check it out, I'd like to hear people's thoughts on this, because it's come up here before.
 

cromulence

Member
SoSH Member
Aug 25, 2009
5,192
These are the same questions/problems I've always had about them and it's why my eyes glaze over when people immediately point to PFF rankings as gospel. I don't know how it came to be that everyone blindly accepted their stuff but over the last couple years their influence has grown more than it should have. There's definitely some value in breaking down video of every play, but distilling it down to a grade is truly impossible in my opinion.
 

Vinho Tinto

Member
SoSH Member
Dec 9, 2003
6,216
Auburn, MA
Smiling Joe Hesketh said:
It's long so I'm not going to quote the whole thing. Allen raises some very worthwhile questions (qualifications of the reviewers, surety of their conclusions, etc). Go check it out, I'd like to hear people's thoughts on this, because it's come up here before.
 
Couldn't the same critiques be placed on Football Outsiders? IIRC, they review old VHS tapes to calculate team DVOA.
 

cromulence

Member
SoSH Member
Aug 25, 2009
5,192
Vinho Tinto said:
 
Couldn't the same critiques be placed on Football Outsiders? IIRC, they review old VHS tapes to calculate team DVOA.
 
My biggest gripe with DVOA is that it's a proprietary black box. I just don't know how we're supposed to trust a stat someone created when they won't even tell us exactly how it's calculated. Maybe this has changed of late but it's always bugged me about FO.
 

bowiac

I've been living a lie.
Lifetime Member
SoSH Member
Dec 18, 2003
12,722
New York, NY
This article is an example of Betteridge's law of headlines: "Any headline which ends in a question mark can be answered by the word no."
 
Can Pro Football Focus Stats Be Blindly Trusted?
 
Well, no. Of course not. That could apply to literally any topic however. Nothing can or should be blindly trusted. The entire article follows that same strawman style:
 
"Bill Belichick doesn’t have it figured out. But Pro Football Focus does?"
"From all of this, the national media are using PFF stats as gospel?"
 
Who is saying they should be blindly trusted? Who is saying they have it all figured out? Who is saying it's gospel?
 
Like anything else, it's a tool. It can be misleading, it has biases, flaws and it can be intentionally mis-used. Is anyone saying otherwise? I'm not sure what the point of this article is in the other words.  Saying it's not perfect? Okay... Nobody thought it was, did they?
 
What I will say with some confidence is that Pro Football Focus grades, in the aggregate, have a lot of predictive value for how a team is going to perform on the field in the future. This predictive value is separate from a team's box score stats.
 

mpx42

lurker
Apr 23, 2010
2,684
Seattle, WA
I'm not positive, but AFAIK Football Outsiders has a lengthy, complicated formula they use that calculates things like DVOA, and all they do is just input the game data from week to week.
 

Morgan's Magic Snowplow

Member
SoSH Member
Jul 2, 2006
18,626
Philadelphia
PFF's other stats - coverage stats, drops, slot performance, etc - are a million times more useful than their player scores.
 
The other big issue with their ratings is the notion of using only 1s and 0s (and -1s or whatever) assigned to each play and then creating an additive scale of performance.
 
So a DE beats his man, makes a strip sack, recovers a fumble, and scores a TD.  He gets a positive grade on the play.  Next play that DE loses contain slightly and allows a 5-yard running gain.  He gets a negative grade on the play and grades out evenly for the sequence.  And that's indistinguishable from another player who was just basically nondescript for two plays running and has a similar 0 score.
 
Big plays are so important in football - TDs, first downs, turnovers, other negative plays on defense that are effective drive killers - that grading everything the same and then adding up scores really doesn't make a lot of sense.
 
Edit: Remembering more now, I think they might use a 2,1,0,-1,-2 scale but the point still remains.
 

SMU_Sox

loves his fluffykins
SoSH Member
Jul 20, 2009
6,768
Dallas
Vinho Tinto said:
 
Couldn't the same critiques be placed on Football Outsiders? IIRC, they review old VHS tapes to calculate team DVOA.
 
Huh? No - they are all data driven.
 

mpx42

lurker
Apr 23, 2010
2,684
Seattle, WA
cromulence said:
 
My biggest gripe with DVOA is that it's a proprietary black box. I just don't know how we're supposed to trust a stat someone created when they won't even tell us exactly how it's calculated. Maybe this has changed of late but it's always bugged me about FO.
I would agree with this completely - that being said, the play-by-play grading by PFF is sort of the same way, there's no way to dispute those numbers when you can't tell what goes into each individual graded component.
 

cromulence

Member
SoSH Member
Aug 25, 2009
5,192
bowiac said:
Who is saying they should be blindly trusted? Who is saying they have it all figured out? Who is saying it's gospel?
 
Obviously no one is declaring "PFF IS GOSPEL!!" but come on - look around. It's become an extremely popular tool/crutch for writers to use when discussing a player. Rotoworld started more or less parroting their rankings last season and treating them as unquestionable fact. The rankings are definitely being given too much importance in the media.
 

SMU_Sox

loves his fluffykins
SoSH Member
Jul 20, 2009
6,768
Dallas
Morgan's Magic Snowplow said:
The other big issue is the notion of using only 1s and 0s (and -1s or whatever) assigned to each play and then creating an additive scale of performance.
 
So a DE beats his man, makes a strip sack, recovers a fumble, and scores a TD.  He gets a positive grade on the play.  Next play that DE loses contain slightly and allows a 5-yard running gain.  He gets a negative grade on the play and grades out evenly for the sequence.  And that's indistinguishable from another player who was just basically nondescript for two plays running and has a similar 0 score.
 
Big plays are so important in football - TDs, first downs, turnovers, other negative plays on defense that are effective drive killers - that grading everything the same and then adding up scores really doesn't make a lot of sense.
 
Edit: Remembering more now, I think they might use a 2,1,0,-1,-2 scale but the point still remains.
 
PFF needs scope or magnitude - which, in order to not be subjective, they could easily make mathematical. Did you prevent a success? Did you achieve a success? OK - you get 3 points instead. Basically I think the process would be better if FO and PFF blended their methodologies AND stopped the whole black box thing ;P. Neither will happen though so wishful thinking.
 

bowiac

I've been living a lie.
Lifetime Member
SoSH Member
Dec 18, 2003
12,722
New York, NY
cromulence said:
Obviously no one is declaring "PFF IS GOSPEL!!" but come on - look around. It's become an extremely popular tool/crutch for writers to use when discussing a player. Rotoworld started more or less parroting their rankings last season and treating them as unquestionable fact. The rankings are definitely being given too much importance in the media.
I guess. I don't do much fantasy football, so maybe I'm just not seeing that aspect of it. Most what I've seen is people saying "he seems really good/bad, but his PFF number is otherwise. I wonder what that means?"
 
I think it's a fine tool to use when discussing a player however.
 

Kenny F'ing Powers

posts 18% useful shit
SoSH Member
Nov 17, 2010
11,340
cromulence said:
 
Obviously no one is declaring "PFF IS GOSPEL!!" but come on - look around. It's become an extremely popular tool/crutch for writers to use when discussing a player. Rotoworld started more or less parroting their rankings last season and treating them as unquestionable fact. The rankings are definitely being given too much importance in the media.
 
That's not PFF's, fault though. 
 
I've said it before, and I'll say it again. PFF is just another data point to grade players. It's easy to use it as the end-all-be-all because it's a clean and quick. "Player A is a +4, Player B is a -2. Player A is better then player B."
 
PFF and FO are the first iteration of this process, and it has a very long way to go. There is also so much ambiguity in every play that it's difficult to ever be 100% correct. Still, there's no reason to assume that the number of missed plays varies wildly from game to game so, even though the numbers may not be completely accurate, everyone across the board is affected almost equally by them and that should largely negate the issue.
 
The other thought here is that a large portion of plays run by teams (both offense and defense) are shit that have been run out onto the field before. Being able to establish that "The offense is running a weak side dive" helps predict what each player should be doing in the play. It also should help decide, based on the defensive formation, how each player should respond. It obviously gets more convoluted as the plays, blitzes, schemes become more exotic, but a large majority of schemes/plays run on a football field are ones that the PFF/FO folks should have seen hundreds of times by now.
 

Smiling Joe Hesketh

All Hail King Boron
Dope
May 20, 2003
31,600
Deep inside Muppet Labs
bowiac said:
I guess. I don't do much fantasy football, so maybe I'm just not seeing that aspect of it. Most what I've seen is people saying "he seems really good/bad, but his PFF number is otherwise. I wonder what that means?"
 
I think it's a fine tool to use when discussing a player however.
 
But the question raised by the article is: how trustworthy are PFF's player evaluations? This is who they are, here's the way they do it, and here are their guidelines. Are they trustworthy, given these facts, to product an accurate number for a player?
 
And my thought based on reading this is: I'm not really sure they are.
 
I mean, this, from their own website, is just so gunshy:
 
 
It is, however, crucial that this is not seen as an excuse to shy away from making a judgement. What we definitely do not do is raise or lower the grading because we’re not sure. Giving a grade of -0.5 rather than -1.5 for a player on an individual play because you’re unsure is the wrong grade to give. If the grader is 95 percent sure of the severe fault on the play, the grade is -1.5. If, however, the grader is unsure of his judgment, the correct grade is 0.
 
A grader, whose qualifications are totally unknown to us, judging himself to be 95% sure on a play, when he has no idea about the required assignments on that play, just leads to all sorts of biases just begging to creep in.
 

bowiac

I've been living a lie.
Lifetime Member
SoSH Member
Dec 18, 2003
12,722
New York, NY
Smiling Joe Hesketh said:
But the question raised by the article is: how trustworthy are PFF's player evaluations? This is who they are, here's the way they do it, and here are their guidelines. Are they trustworthy, given these facts, to product an accurate number for a player?
That's what I was getting at earlier - in the aggregate, the PFF player evaluations have real predictive value. That value is separate from wins & losses, box score stats, and more advanced DVOA-type stats. If you are building a model to predict future team performance, you will get better results including PFF data in your model than if you don't.
 
More generally however, I think an "accurate number" is too high a standard. No, it's not accurate in almost any sense of the word. With football player evaluations however, unless you're a professional, the bar is exceptionally low. PFF numbers don't need to be especially "accurate" in order to contribute a lot of value to the conversation.
 

SMU_Sox

loves his fluffykins
SoSH Member
Jul 20, 2009
6,768
Dallas
Precisely. If you built a model that is say 70% predictive do you just throw it out because it isn't 100% there? Now before you get your pitchforks, I'm not saying off is 70% accurate that's just a hypothetical number. Progress comes slowly. There's a ton of white noisehere and cutting through it you'll probably never get to 100%.
 

Smiling Joe Hesketh

All Hail King Boron
Dope
May 20, 2003
31,600
Deep inside Muppet Labs
bowiac said:
That's what I was getting at earlier - in the aggregate, the PFF player evaluations have real predictive value. That value is separate from wins & losses, box score stats, and more advanced DVOA-type stats. If you are building a model to predict future team performance, you will get better results including PFF data in your model than if you don't.
 
Have you done this? Has anyone done this?
 
That's not meant to be a dickish question, it's an honest one. How do we know the bolded is true?
 

SMU_Sox

loves his fluffykins
SoSH Member
Jul 20, 2009
6,768
Dallas
Smiling Joe Hesketh said:
 
Have you done this? Has anyone done this?
 
That's not meant to be a dickish question, it's an honest one. How do we know the bolded is true?
look at their team rankings last year. Their stats actually had predictive value from week to week. I used them a bit to gamble. I used FO a bit more but I still used PFF to great success. What I'd do with PFF is see strength on weakness values. Like Bufs pass rush vs. Miami' s oline. Predictive and successful.
 

Super Nomario

Member
SoSH Member
Nov 5, 2000
13,096
Mansfield MA
I subscribe to PFF, and find it useful and interesting. It does need to be taken with a grain of salt. Allen's article smacks of a lot of the "get off my lawn" quality that FJM made a living criticizing.
 
Morgan's Magic Snowplow said:
So a DE beats his man, makes a strip sack, recovers a fumble, and scores a TD.  He gets a positive grade on the play.  Next play that DE loses contain slightly and allows a 5-yard running gain.  He gets a negative grade on the play and grades out evenly for the sequence.  And that's indistinguishable from another player who was just basically nondescript for two plays running and has a similar 0 score.
 
Big plays are so important in football - TDs, first downs, turnovers, other negative plays on defense that are effective drive killers - that grading everything the same and then adding up scores really doesn't make a lot of sense.
There's a valid question as to whether the big plays are repeatable. But PFF's ratings suffer from the same incoherence that DVOA does where it's unclear whether it's predictive or descriptive (as opposed to the work Brian Burke does at advancednflstats.com, love him or hate him).
 
My biggest criticism is that while PFF purports to separate individual player contributions, it does not do this effectively. Brady had much better PFF grades in the games where he had weapons available last year. All Denver's OL had a massive jump when they went from Tebow at QB to Manning. PFF doesn't factor in "degree of difficulty" at all - beating Joe Thomas for a sack doesn't get you a better grade than beating Lamar Holmes. That means at some positions the grading is almost backwards - such as TE blocking, where the poorer blockers get better grades than the better blockers because they aren't asked to execute the same challenging assignments. Aqib Talib graded negatively last year - some of that was a few ineffective weeks when he returned from injury, but a lot of it was that PFF doesn't factor in that he was covering opposing #1s. According to PFF, Talib was the worst coverage player on the Patriots' defense last year at -6.0 - at the same time, Bill Belichick was asking him to defend the offense's best receiver!
 
bowiac said:
That's what I was getting at earlier - in the aggregate, the PFF player evaluations have real predictive value. That value is separate from wins & losses, box score stats, and more advanced DVOA-type stats. If you are building a model to predict future team performance, you will get better results including PFF data in your model than if you don't.
I believe you, but I'm curious as to your evidence for this.
 
bowiac said:
More generally however, I think an "accurate number" is too high a standard. No, it's not accurate in almost any sense of the word. With football player evaluations however, unless you're a professional, the bar is exceptionally low. PFF numbers don't need to be especially "accurate" in order to contribute a lot of value to the conversation.
I agree with this, but I think more humility needs to be applied in the use of such stats. I think there also needs to be more realism about what sample sizes look like in football statistics and how certain we can be about one-year results. Last week a PFF writer penned an article saying Brady is no longer a top-five QB on the basis of one down year and a SSS negative "trend" in "under pressure" results (and putting Rivers, who was worse vs pressure in '12 than Brady was in '13, ahead of him). That kind of argument puts way too much stock in PFF's methods, in one year results, and in the predictability of fairly small samples.
 

Stitch01

Member
SoSH Member
Jul 15, 2005
18,155
Boston
Can you share your study on how these were successful and predictive?  That is really interesting to me
 

Kenny F'ing Powers

posts 18% useful shit
SoSH Member
Nov 17, 2010
11,340
Super Nomario said:
Aqib Talib graded negatively last year - some of that was a few ineffective weeks when he returned from injury, but a lot of it was that PFF doesn't factor in that he was covering opposing #1s. According to PFF, Talib was the worst coverage player on the Patriots' defense last year at -6.0 - at the same time, Bill Belichick was asking him to defend the offense's best receiver!
 
 
 
But if you dig deeper and look at his "signature stats", they have him 18th in QB rating against, 11th in cover snaps/target, 12th in cover snaps/receptions, and 41st in yards in cover/snaps.
 
There were 81 CB's that qualified with 50% of defensive snaps. I would think if you assigned a ranking in each category (81 points for highest ranking player, 1 point for lowest ranking player), those numbers would bear out that Talib was a top 10 CB in the NFL last year. That sounds about right.
 
Moreover, those numbers certainly pass the smell test to me. He wasn't targeted very often before he was hurt, gave up some big YAC in a few games ('sup Josh Gordon). 
 
Again, this isn't to say that PFF is perfect, but if you dig deep enough with these guys, I think there's some real value to the site.
 

SMU_Sox

loves his fluffykins
SoSH Member
Jul 20, 2009
6,768
Dallas
SN, I get what you're saying but PFF is a business - of course they rely too much on their own product. They're going to make some, what we might consider, off the wall claims. They get attention for their product. It sells. Profit. That's the missing step the underwear gnomes never quite grasped. 
 
If you want to see something interesting look at last year week 16 MIA @ BUF. The spread ended (on average) with BUF -1. I believe I got the line at BUF +1 or +2 or +3.
 
I put a lot of weight on blocking stats and offensive and defensive lines. BUF had a better pass rush and run defense then MIA had pass blocking and run blocking. Miami also, on defense, matched up better against BUF on offense but not to the degree that BUF outmatched MIA's offense with their defense. So you have two somewhat similar teams in that their strength is their defense. One team has a rather large advantage on defense going against an overmatched offense. Now dig into the key players - you're going to see that BUF's front 4-5 all have individual stats that are much higher than their MIA offensive linemen opponents - wonder what's going to happen? Basically we anticipated a low scoring matchup and one which BUF should win by around 3.
 
The O/U was 40. 
 
We ended up taking the following bets:
 
BUF moneyline - not a huge play.
BUF +X (whatever I got it at) just the spread.
Teaser: BUF +X and under 46.
Under 40.
Parlay: Buf +X and under 40
 
Final score? Buf 19 - Mia 0.
 
It isn't always sunshine and roses for me but combining week to week earnings plus survivor pool payouts I made 10x what I invested last year. Right now I got an engineer/programmer at LM to start working with me on how to use FO, PFF, and ingame data to make a much more predictive model. I know Phragle is interested in joining up with us... anyway... look I want to give you more info but right now we're in our infancy. Stay tuned.
 

bowiac

I've been living a lie.
Lifetime Member
SoSH Member
Dec 18, 2003
12,722
New York, NY
Smiling Joe Hesketh said:
Have you done this? Has anyone done this?
 
That's not meant to be a dickish question, it's an honest one. How do we know the bolded is true?
I have run the study myself, backtested with out of sample data. (i.e. I tested how 2010 data predicted 2011 data relative to Vegas, how 2011 data predicted 2012 data relative to Vegas, etc...). I don't particularly want to share the spreadsheet (yes, I still use excel for my crap), but I'll present top line data if people are interested.
 
Importantly (for this discussion), this is aggregate data only. I don't split out by player or anything.
 

soxfan121

JAG
Lifetime Member
SoSH Member
Dec 22, 2002
23,043
I rely on Mel Kiper, Todd McShay and Pro Bowl rosters to tell me who's good. 
 
bowiac and SN covered everything serious I would want to say, although I have had a great time listening to Scott Zolak come to grips with PFF scores the last few days...."they don't even factor in wins for the QB position!! How can you evaluate a QB without wins and losses!?!!?"
 

Super Nomario

Member
SoSH Member
Nov 5, 2000
13,096
Mansfield MA
Kenny F'ing Powers said:
 
 
But if you dig deeper and look at his "signature stats", they have him 18th in QB rating against, 11th in cover snaps/target, 12th in cover snaps/receptions, and 41st in yards in cover/snaps.
 
There were 81 CB's that qualified with 50% of defensive snaps. I would think if you assigned a ranking in each category (81 points for highest ranking player, 1 point for lowest ranking player), those numbers would bear out that Talib was a top 10 CB in the NFL last year. That sounds about right.
 
Moreover, those numbers certainly pass the smell test to me. He wasn't targeted very often before he was hurt, gave up some big YAC in a few games ('sup Josh Gordon). 
 
Again, this isn't to say that PFF is perfect, but if you dig deep enough with these guys, I think there's some real value to the site.
I agree with all this. But PFF will tell you that the grades are a better, more accurate source of what the player did than these stats. "We collect lots of extra statistics such as yards after catch, yards after contact, missed tackles, dropped passes etc., but our real focus is on grading individual performance on each play."
 

SMU_Sox

loves his fluffykins
SoSH Member
Jul 20, 2009
6,768
Dallas
Super Nomario said:
I agree with all this. But PFF will tell you that the grades are a better, more accurate source of what the player did than these stats. "We collect lots of extra statistics such as yards after catch, yards after contact, missed tackles, dropped passes etc., but our real focus is on grading individual performance on each play."
 
Boiling it down to a one number answer. Isn't that the American way? This is where they can say whatever they want to. That one number answer fits in with their overall approach to their brand. I get that. But don't listen as much to that, imo. Look at the other stats. Some of the gems KFP and I found in RFP last year had great sig stats that didn't match their grades. Look at Chris D. Clemons. Good signature pass coverage stats in 2012 but only a +2.9 grade. He improved that this year to a +7.2 - good for 9th best overall for safeties. 
 

Super Nomario

Member
SoSH Member
Nov 5, 2000
13,096
Mansfield MA
SMU_Sox said:
 
Boiling it down to a one number answer. Isn't that the American way? This is where they can say whatever they want to. That one number answer fits in with their overall approach to their brand. I get that. But don't listen as much to that, imo. Look at the other stats. Some of the gems KFP and I found in RFP last year had great sig stats that didn't match their grades. Look at Chris D. Clemons. Good signature pass coverage stats in 2012 but only a +2.9 grade. He improved that this year to a +7.2 - good for 9th best overall for safeties. 
I don't think much of coverage stats for free safeties, to be honest. Suffer from small sample size, plus in many schemes a FS is only "in coverage" if it's a double-team.
 
(what are you guys going to do about S anyway, with Clemons gone? Yikes)
 

Kenny F'ing Powers

posts 18% useful shit
SoSH Member
Nov 17, 2010
11,340
Kenny F'ing Powers said:
There were 81 CB's that qualified with 50% of defensive snaps. I would think if you assigned a ranking in each category (81 points for highest ranking player, 1 point for lowest ranking player), those numbers would bear out that Talib was a top 10 CB in the NFL last year. That sounds about right.
 
 
Because I'm at work and I don't want to work, I just put together a spreadsheet and did this.
 
Here is a table of the top 30 cumulative Signature stat scores. 
 
Players were given a score based on their overall ranking in each category - QB Rating against, Cover snaps per target, yards per cover snap, and cover snap per reception. As there were 81 CB's who played in at least 50% of the snaps, players were scored from 1 - 81. The top player in each category (almost always Richard Sherman, fwiw) was given a score of 81. The second highest player in the category was given an 80, all the way down to the worst player in a category (who received 1 point). 
 
Here is a list of the top 30 players based on cumulative signature stats.
 
If you want the entire list, PM me. Our table's can only fit so much information in here, so I cut the list to the top 30 players.
 
[tablegrid= Top 30 Cumulative PFF "signature stat" scores (50% snaps) ]Name Score NFL Rating Score Cover Snaps / Target Score Yards / Cover Snap Score Cover Snaps / Rec Final Score  Richard Sherman 81 47.3 81 9.5 80 0.77 81 18.3 323  Keenan Lewis 72 67.4 79 7.7 77 0.89 76 13.8 304  Alterraun Verner 79 55.8 76 7.4 68 0.99 78 15 301  Dominique Rodgers-Cromartie 71 67.8 71 6.9 73 0.96 79 15.7 294  Logan Ryan 80 53.3 73 7 66 1 75 13.8 294  Darrelle Revis 51 81.4 80 8.8 81 0.72 80 16.4 292  Drayton Florence 69 68.8 77 7.5 72 0.97 74 13.7 292  Alan Ball 57 75.3 75 7.1 65 1.01 72 13 269  Chris Harris Jr. 76 64.9 58 6.5 70 0.97 62 11.6 266  Nickell Robey 68 71.7 48 6.3 78 0.88 65 12 259  Patrick Peterson 27 91.3 78 7.7 67 1 77 14.1 249  Trumaine Johnson 60 74.5 67 6.8 64 1.02 57 11.2 248  Aqib Talib 64 72.3 70 6.9 41 1.22 70 12.8 245  Melvin White 63 72.4 68 6.8 71 0.97 41 10.3 243  Joe Haden 59 74.9 47 6.3 76 0.89 58 11.3 240  Tyrann Mathieu 41 85 72 7 75 0.9 51 10.7 239  Tim Jennings 70 68.2 49 6.3 55 1.09 63 11.7 237  Jimmy Wilson 66 72 57 6.5 63 1.02 50 10.7 236  Trumaine McBride 78 57.4 13 5.2 69 0.97 64 11.9 224  Desmond Trufant 58 75.2 45 6.2 59 1.03 61 11.6 223  Sean Smith 42 84.7 60 6.6 46 1.17 73 13.1 221  William Gay 61 72.9 41 6.1 74 0.91 45 10.4 221  Jimmy Smith 56 76.3 46 6.3 57 1.04 60 11.6 219  Carlos Rogers 48 82 63 6.7 53 1.1 52 10.9 216  Alfonzo Dennard 46 82.7 56 6.5 45 1.17 66 12.1 213  Brent Grimes 74 66.3 50 6.3 44 1.19 42 10.4 210  Sam Shields 62 72.7 42 6.1 36 1.29 67 12.3 207  Adam Jones 50 81.4 52 6.4 43 1.2 59 11.4 204  Prince Amukamara 34 87.7 62 6.7 60 1.03 44 10.4 200  [/tablegrid] 
 

bowiac

I've been living a lie.
Lifetime Member
SoSH Member
Dec 18, 2003
12,722
New York, NY
Stitch01 said:
Can you share your study on how these were successful and predictive?  That is really interesting to me
In a top line sense: 
 
I set up a non-linear regression, using data such as wins, losses, points scored/allowed, yards gained/allowed, strength of schedule, whether a team made the playoffs, whether they changed coaches, as well as various football outsider stats. I used these stats to predict NFL wins for the next season. I specifically focused on the places where these results deviated from Vegas O/Us. Using the coefficients from 2008 to predict the 2009 data, then the combined coefficients from 2008-2009 to predict 2010 data, then the combined 2008-2010 coefficients to predict 2011, etc... I got a root mean square error of 2.48 wins. This model was relative to Vegas in 56.3% of cases overall, and correct in 73% of cases where the prediction differed from the Vegas O/U by at least 1.3 wins. (There was a slight bias towards the vigged side, for a 34% ROI overall in these cases).
 
When I added PFF data into the model (I picked those years because those are the only years for which PFF data is available), and redid the coefficients, the RMSE dropped to 2.3, improved its accuracy to 60%, and was correct in 87% of cases where it differed from Vegas by at least 1.3 wins (62% ROI in those cases).
 
It's not a massive improvement, except in an ROI sense. Part of that is that the base model was already pretty good, and there's only so much extra you can do beyond there. But at the margins, it's a pretty strong improvement, and represents either a fluke (possible), or actual value added from PFF data.
 
For what it's worth, both models performed identically (lost money) in 2013.
 
I can share more details upon request.
 

Eck'sSneakyCheese

Member
SoSH Member
May 11, 2011
9,296
NH
I am not a big fan of PFF and their methodology but their signature stats seem to carry a little weight and I like the table that KFP put together ( of course because on that list I have two of the top 5 CBs in the RFP.) Their overall scores don't make sense because you can't base a play by play or game by game performance by adding or subtracting the numbers from each instance. It leaves the final number with way too much noise.

Morgan's Magic Snowplow said:
PFF's other stats - coverage stats, drops, slot performance, etc - are a million times more useful than their player scores.
 
The other big issue with their ratings is the notion of using only 1s and 0s (and -1s or whatever) assigned to each play and then creating an additive scale of performance.
 
So a DE beats his man, makes a strip sack, recovers a fumble, and scores a TD.  He gets a positive grade on the play.  Next play that DE loses contain slightly and allows a 5-yard running gain.  He gets a negative grade on the play and grades out evenly for the sequence.  And that's indistinguishable from another player who was just basically nondescript for two plays running and has a similar 0 score.
 
Big plays are so important in football - TDs, first downs, turnovers, other negative plays on defense that are effective drive killers - that grading everything the same and then adding up scores really doesn't make a lot of sense.
 
Edit: Remembering more now, I think they might use a 2,1,0,-1,-2 scale but the point still remains.
Or basically what MMS said here.

I think PFF is headed in the right direction but they're still pretty far away from a finished product and their overall individual player grades shouldn't be taken seriously.
 

bowiac

I've been living a lie.
Lifetime Member
SoSH Member
Dec 18, 2003
12,722
New York, NY
One more data nugget: a model based purely on FO data performs almost identically to a model based purely on PFF data. The average error increases to 2.63 wins in both cases when I strip out the other sources. Both models underperform basic Pro-football reference data however, which got to an average error of 2.51 all by itself.
 

Phragle

wild card bitches
SoSH Member
Jan 1, 2009
13,018
Carmine's closet
Super Nomario said:
I agree with all this. But PFF will tell you that the grades are a better, more accurate source of what the player did than these stats. "We collect lots of extra statistics such as yards after catch, yards after contact, missed tackles, dropped passes etc., but our real focus is on grading individual performance on each play."
 
I would say PFF's opinion is fairly biased.
 
Kenny F'ing Powers said:
 
Because I'm at work and I don't want to work, I just put together a spreadsheet and did this.
 
Here is a table of the top 30 cumulative Signature stat scores. 
 
Players were given a score based on their overall ranking in each category - QB Rating against, Cover snaps per target, yards per cover snap, and cover snap per reception. As there were 81 CB's who played in at least 50% of the snaps, players were scored from 1 - 81. The top player in each category (almost always Richard Sherman, fwiw) was given a score of 81. The second highest player in the category was given an 80, all the way down to the worst player in a category (who received 1 point). 
 
Here is a list of the top 30 players based on cumulative signature stats.
 
If you want the entire list, PM me. Our table's can only fit so much information in here, so I cut the list to the top 30 players.
 
[tablegrid= Top 30 Cumulative PFF "signature stat" scores (50% snaps) ]Name Score NFL Rating Score Cover Snaps / Target Score Yards / Cover Snap Score Cover Snaps / Rec Final Score  Richard Sherman 81 47.3 81 9.5 80 0.77 81 18.3 323  Keenan Lewis 72 67.4 79 7.7 77 0.89 76 13.8 304  Alterraun Verner 79 55.8 76 7.4 68 0.99 78 15 301  Dominique Rodgers-Cromartie 71 67.8 71 6.9 73 0.96 79 15.7 294  Logan Ryan 80 53.3 73 7 66 1 75 13.8 294  Darrelle Revis 51 81.4 80 8.8 81 0.72 80 16.4 292  Drayton Florence 69 68.8 77 7.5 72 0.97 74 13.7 292  Alan Ball 57 75.3 75 7.1 65 1.01 72 13 269  Chris Harris Jr. 76 64.9 58 6.5 70 0.97 62 11.6 266  Nickell Robey 68 71.7 48 6.3 78 0.88 65 12 259  Patrick Peterson 27 91.3 78 7.7 67 1 77 14.1 249  Trumaine Johnson 60 74.5 67 6.8 64 1.02 57 11.2 248  Aqib Talib 64 72.3 70 6.9 41 1.22 70 12.8 245  Melvin White 63 72.4 68 6.8 71 0.97 41 10.3 243  Joe Haden 59 74.9 47 6.3 76 0.89 58 11.3 240  Tyrann Mathieu 41 85 72 7 75 0.9 51 10.7 239  Tim Jennings 70 68.2 49 6.3 55 1.09 63 11.7 237  Jimmy Wilson 66 72 57 6.5 63 1.02 50 10.7 236  Trumaine McBride 78 57.4 13 5.2 69 0.97 64 11.9 224  Desmond Trufant 58 75.2 45 6.2 59 1.03 61 11.6 223  Sean Smith 42 84.7 60 6.6 46 1.17 73 13.1 221  William Gay 61 72.9 41 6.1 74 0.91 45 10.4 221  Jimmy Smith 56 76.3 46 6.3 57 1.04 60 11.6 219  Carlos Rogers 48 82 63 6.7 53 1.1 52 10.9 216  Alfonzo Dennard 46 82.7 56 6.5 45 1.17 66 12.1 213  Brent Grimes 74 66.3 50 6.3 44 1.19 42 10.4 210  Sam Shields 62 72.7 42 6.1 36 1.29 67 12.3 207  Adam Jones 50 81.4 52 6.4 43 1.2 59 11.4 204  Prince Amukamara 34 87.7 62 6.7 60 1.03 44 10.4 200  [/tablegrid] 
 
Good stuff, but even this has to be viewed in context. It's not too surprising Sherman looks good. He doesn't follow the best WR - he always plays on the left side, and he benefits from great pass rush and safety play in Seattle. Then go down to Verner. He doesn't get the same pass rush or safety play benefits, but he's always taking the second best WR. McCourty takes the best guy in Tennessee. Here Peterson and Talib are better than they look because they both follow the number one WR on every snap. That's hard to do and their stats still look good.
 
If we're starting a team from scratch I'll take Peterson over Sherman.
 
PFF is a very valuable tool when used correctly, but it's still just a tool.
 

SMU_Sox

loves his fluffykins
SoSH Member
Jul 20, 2009
6,768
Dallas
Context and white noise. Have fun getting that perfect with stats in football. Here's a hint: that's really hard to do. Phragle has the right attitude here. People love pff because it's easy. One number. I think if you blended pffs individual approach with fos adjustments you'd get some dawn good stuff. That's, right now, the direction I'm heading.
 

Stitch01

Member
SoSH Member
Jul 15, 2005
18,155
Boston
bowiac said:
In a top line sense: 
 
I set up a non-linear regression, using data such as wins, losses, points scored/allowed, yards gained/allowed, strength of schedule, whether a team made the playoffs, whether they changed coaches, as well as various football outsider stats. I used these stats to predict NFL wins for the next season. I specifically focused on the places where these results deviated from Vegas O/Us. Using the coefficients from 2008 to predict the 2009 data, then the combined coefficients from 2008-2009 to predict 2010 data, then the combined 2008-2010 coefficients to predict 2011, etc... I got a root mean square error of 2.48 wins. This model was relative to Vegas in 56.3% of cases overall, and correct in 73% of cases where the prediction differed from the Vegas O/U by at least 1.3 wins. (There was a slight bias towards the vigged side, for a 34% ROI overall in these cases).
 
When I added PFF data into the model (I picked those years because those are the only years for which PFF data is available), and redid the coefficients, the RMSE dropped to 2.3, improved its accuracy to 60%, and was correct in 87% of cases where it differed from Vegas by at least 1.3 wins (62% ROI in those cases).
 
It's not a massive improvement, except in an ROI sense. Part of that is that the base model was already pretty good, and there's only so much extra you can do beyond there. But at the margins, it's a pretty strong improvement, and represents either a fluke (possible), or actual value added from PFF data.
 
For what it's worth, both models performed identically (lost money) in 2013.
 
I can share more details upon request.
Interesting stuff, thanks.  Wonder if data becoming more widely available has eroded the ROI. Feels to me like O/U lines aren't nearly as soft as five years ago, but don't have numbers to back that intuition.  I was in a group in 2010 that did a basic study using DVOA data for betting individual games and it looked like it could beat the vig but it looked like the edge was actively eroding (or, more accurately, there were fewer games where the line differed materially from where DVOA) so I wondered if it was self defeating.
 
I view PFF as an interesting, but super imprecise, tool and think the drilled down stats are more interesting analytically then the raw +/- number. 
 

bowiac

I've been living a lie.
Lifetime Member
SoSH Member
Dec 18, 2003
12,722
New York, NY
Stitch01 said:
Interesting stuff, thanks.  Wonder if data becoming more widely available has eroded the ROI. Feels to me like O/U lines aren't nearly as soft as five years ago, but don't have numbers to back that intuition.  I was in a group in 2010 that did a basic study using DVOA data for betting individual games and it looked like it could beat the vig but it looked like the edge was actively eroding (or, more accurately, there were fewer games where the line differed materially from where DVOA) so I wondered if it was self defeating.
I don't know honestly. The top end picks last year, Atlanta (U), Houston (U), Jacksonville (O), New England (O), St. Louis (U), and the Steelers (U), still went 5 of 6, with a 62% ROI after the vig.
 

SMU_Sox

loves his fluffykins
SoSH Member
Jul 20, 2009
6,768
Dallas
Context and white noise. Have fun getting that perfect with stats in football. Here's a hint: that's really hard to do. Phragle has the right attitude here. People love pff because it's easy. One number. I think if you blended pffs individual approach with fos adjustments you'd get some dawn good stuff. That's, right now, the direction I'm heading.
 

SMU_Sox

loves his fluffykins
SoSH Member
Jul 20, 2009
6,768
Dallas
Guys, excel is a dumpster fire for stats. Use r. It's free and extremely dynamic. Steep but worthwhile learning curve.
 

BSF34

lurker
Oct 4, 2008
2
Super Nomario said:
I subscribe to PFF, and find it useful and interesting. It does need to be taken with a grain of salt. Allen's article smacks of a lot of the "get off my lawn" quality that FJM made a living criticizing.
 
 
I don't see it that way at all. IMO, Bruce is saying some in the the media pick-and-choose which kind of data that it uses to support their positions. I mean, Shaughnessey,Ryan, etc.. ridicule advanced stats in baseball all the time, yet PFF is widely accepted as advanced stats by guys like these?
 

SMU_Sox

loves his fluffykins
SoSH Member
Jul 20, 2009
6,768
Dallas
Because it's straightforward man. That's what makes pff easy to accept. It's simple and one number. Everyone, not just numbers people, can understand their methodology. It's a brilliant idea. Execution needs work.
 

bowiac

I've been living a lie.
Lifetime Member
SoSH Member
Dec 18, 2003
12,722
New York, NY
SMU_Sox said:
Guys, excel is a dumpster fire for stats. Use r. It's free and extremely dynamic. Steep but worthwhile learning curve.
While I agree with this in general, and I kick around with R somewhat, I have a finite amount of time, and have invested a lot of time getting pretty good with excel & visual basic. If I were doing this professionally, sure. But as someone who plays with this stuff a few hours a week, learning R to get up where I am with excel just doesn't make sense. Excel is surprisingly versatile, although admittedly a lot things require awkward workarounds.
 

SMU_Sox

loves his fluffykins
SoSH Member
Jul 20, 2009
6,768
Dallas
So you're captain macro, huh? ;)

I didn't mean to come off in a bad way. I got introduced to r in February and have had an unhealthy obsession with it since. We're talking like wake up at 4am and play around with it instead of sleeping.
 

Devizier

Member
SoSH Member
Jul 3, 2000
13,690
Somewhere
What strikes me about PFF's explosion in popularity is that ESPN writers use their numbers pretty extensively. Why is that significant? Because a massive media company like Disney/ESPN should be able to employ a scouting department on par with those of individual teams in the NFL. Maybe they do, but their columnists are essentially relying on amateur scouting. I guess they don't have to pay (much) for it.
 

Super Nomario

Member
SoSH Member
Nov 5, 2000
13,096
Mansfield MA
BSF34 said:
 
I don't see it that way at all. IMO, Bruce is saying some in the the media pick-and-choose which kind of data that it uses to support their positions. I mean, Shaughnessey,Ryan, etc.. ridicule advanced stats in baseball all the time, yet PFF is widely accepted as advanced stats by guys like these?
Are Shaughnessey and Ryan really quoting PFF? I haven't read CHB in years; I can't say I've read much Ryan, either, of late (didn't he retire?). I agree with Allen's conclusion; I just thought some of the specific points (they aren't even American! Why don't they have NFL scouts [as if scouts are just out there helping develop stats]!) were unfair.
 

Vinho Tinto

Member
SoSH Member
Dec 9, 2003
6,216
Auburn, MA
SMU_Sox said:
 
Huh? No - they are all data driven.
I'm going to blame my confusion on posting in between meetings at work. I was thinking of their project for game charting prior seasons where they do not have play by play data for all the games.  
 

kolbitr

lurker
Jul 20, 2005
682
Providence, RI
I'm not a statistician, nor a gambler, so forgive me if this is a dumb question...but is it possible to deduce that at the present PFF stats are more useful on the aggregate than individually? I mean, it seems that both bowiac and smu_sox have found their info significantly useful for their gambling on team win-loss bets...but we are also seeing a lot of noise regarding individual statistics, no?
 
I guess I'm moving toward a question about the significance of their stats for players versus teams...is it probable that (at the present) the individual noise that strikes some posters as inexplicable or wrong is incorporated usefully in the overall team outlooks?
 
I don't think that either Shaughnessy or Ryan have ever referenced PFF (I can't think of a Globe writer who has, actually), but it is certainly what ESPN writers use with some frequency when they wish to "kick-start" a discussion...
 
(NB: yes, PFF and ESPN writers have some significant overlap these days...)
 

bowiac

I've been living a lie.
Lifetime Member
SoSH Member
Dec 18, 2003
12,722
New York, NY
kolbitr said:
I'm not a statistician, nor a gambler, so forgive me if this is a dumb question...but is it possible to deduce that at the present PFF stats are more useful on the aggregate than individually? I mean, it seems that both bowiac and smu_sox have found their info significantly useful for their gambling on team win-loss bets...but we are also seeing a lot of noise regarding individual statistics, no?
This is almost certainly true, and almost certainly true of most stats, basically regardless of sport. To take the most obvious example, team points per game in basketball is way more useful than individual points per game. But the principle should apply through almost any stat I'd imagine.
 

mascho

Kane is Able
SoSH Member
Nov 30, 2007
14,952
Silver Spring, Maryland
Bumping this.  
 
It was mentioned in MMQB that Collinsworth has bought in on PFF.  Not in terms of using it, but he is now a part owner.
 


In this morning’s Monday Morning Quarterback, Peter King forwards a very interesting story involving PFF and one of the NFL’s most well-known analysts.  According to King, Cris Collinsworth was so impressed by Pro Football Focus that he didn’t just become a subscriber – he became a part-owner.
 
Article from Awful Announcing.