5 replies to this topic

[bowiac]

Football isn't baseball, so not every thread needs a lengthy consideration of various statistical results, but at the same time, they can be pretty informative. I thought I'd start a thread to talk about the more interesting ideas that people come across.

To kick it off, I wanted to link to this post by Brett Thiessen about success rates, and why they might be pretty useless. This is a personal bone I've had to pick with Football Outsiders for some time, and Thiessen explains in more depth why we should doubt the value of the success rate statistic in football. He misses the mark I think with the "Football is not baseball" moniker, but that's neither here nor there. I think he raises a very good point when he writes:

A couple of examples of how success rate can be misleading (first down gain, second down gain, third down gain):
4,3,2: This is a 67% success rate but is a three and out.

3,3,4: This is a 33% success rate but a first down, plus the first two plays are nearly identical but the first two downs of the first group are both successes and the second group are both failures. Over a large group of data some of these will iron themselves out, but why put such a black and white metric over something that is not. 2nd and 7 is almost the same as 2nd and 6, but 2nd and 1 is very different from 2nd and 6. Success rate completely misses the magnitude of plays.

This has come up extensively in the Tebow thread, but in many other debates as well, like about the relative value of BJGE vs. other running backs. That said, while I'm a big fan of the point Thiessen is making here, it's a little light on analytics, and is more of a deductive explanation of why success rates shouldn't be used. So I'd be interested in hearing what other people think - is there a response to this? Football Outsiders has consistently refused to explain why success rates should be used, but maybe there's a good reason. If so, what is it?

[Dehere]

I think success rates are a little like batting average and what you're looking for is something more like slugging percentage or total average where successful plays can be weighted according to their relative success.

Let's take Thiessen's example to a greater extreme. What if you start a possession with three plays that go for 1, 0, and a 79 yard touchdown? That's a success rate of 33%. Surely some of that does come out in the wash over an entire season but couldn't we get to a better place by weighting the 79 yard play differently? What if every ten yards gained beyond the first-down marker put another 1 in the numerator of the success rate, so that in the extreme example the success rate of that series would be 8/3 instead of 1/3?

Another question - a trickier one, I think - is does a successful third down conversion mitigate the failure of previous plays? Broadly speaking I think it does. Let's say you pick up 4 on 2nd and 10. By current definitions that play is a failure, but it puts you in 3rd and manageable and if you then pick up the 3rd and 6 I think the efficiency of the 2nd down play has to be reevaluated. Another example: you make an aggressive play call on 2nd and 1 because you're confident you can pick it up on third down. Result: incompletion and then a third down conversion. Was the second down play call truly a total failure? I'd like to see success rate account for success on both a by-play and by-series basis, so that any series resulting in a first down is weighted more favorable. For example:

- 15 yard completion on first down. 1/1 success rate by plays. 1/1 success rate by series. 2/2 overall.
- Two incompletions followed by a 15 yard completion. 1/3 success rate by plays. 1/1 success rate by series. 2/4 overall.
- Three incompletions. 0/3 success rate by plays. 0/1 success rate by series. 0/4 overall.

I like the concept of success rate and I wouldn't discard it entirely. Football metrics are rarely going to be as clean and elegant as baseball metrics. You have to start somewhere and take progress incrementally. Weighted success rates would be a good step.

[Shelterdog]

Football Outsiders has consistently refused to explain why success rates should be used, but maybe there's a good reason. If so, what is it?

This isn't remotely fair.

Their claim is and has always been that DVOA is more predictive of winning than other measures-and success rate is the core of DVOA.

I'd grant you that it's a conclusion based on a foundation of sand and the support is flimsy (the only support they ever show is some 2000-2005 numbers that show DVOA correlates less to winning than point differential although it does better (not well but better) than other stats in predicting wins the next year).

[Super Nomario]

There are two factors here. 1) does success rate effectively capture what happened? 2) is success rate predictive of future events?

Clearly it doesn't tell a complete picture for #1. A five-yard run on 1st-and-10 is the same as a conversion on 4th-and-20, or a fifty-yard TD bomb. It is a good measure of a team's ability to consistently make plays that get it closer to winning, but it doesn't capture the magnitude of success. That doesn't invalidate it as a statistic; OBP counts a walk the same as a HR but it's not useless for doing so.

As far as #2, success rate has some predictive value that isn't captured in conventional stats. One example is the correlation Brian Burke found between run success rate and passing efficiency - a correlation that isn't found looking at YPC and passing efficiency. This makes sense. In terms of one series, getting sacked twice and then throwing a 30-yard completion is no different than throwing a 20-yard completion on first down, but when we want to think about what might happen next week, it's worth knowing that the QB was sacked 2/3 of those plays. The baseball analogy I'd make is strikeout rate for pitchers; two guys can throw six scoreless, but the guy who struck out 10 is more likely to reproduce it than the guy who fanned 4.

As for DVOA specifically, success rate is one factor, but the ultimate value is more complex than that. Look at the first possession for the Jets in the first Pats / Jets game this year: http://www.footballo...d-jets-patriots

QTR	DRV	SITUATION	1/TD	PLAY	PLAYER	YD	RECEPT	INTENDED	SUC	PLAY VALUE	BASE LINE	VALUE OVER BASELINE
1	1	1-10-NYJ20		pass	6-M.Sanchez	5	complete	23-S.Greene	1	1.17	0.81	0.35
1	1	2-5-NYJ25		rushed	23-S.Greene	3			1	1	1.05	-0.05
1	1	3-2-NYJ28		pass	6-M.Sanchez	0	incomplete	81-D.Keller	0	0	1.09	-1.09

The first two plays are "successful" (suc = 1), but the value over baseline for the second play is negative, because the success of each plays is evaluated relative to an average performance. Gaining three yards on 2nd-and-5 is a "success," but it's also not very impressive, so it's a (slightly) negatively play by DVOA.

[bowiac]

As far as #2, success rate has some predictive value that isn't captured in conventional stats. One example is the correlation Brian Burke found between run success rate and passing efficiency - a correlation that isn't found looking at YPC and passing efficiency. This makes sense. In terms of one series, getting sacked twice and then throwing a 30-yard completion is no different than throwing a 20-yard completion on first down, but when we want to think about what might happen next week, it's worth knowing that the QB was sacked 2/3 of those plays. The baseball analogy I'd make is strikeout rate for pitchers; two guys can throw six scoreless, but the guy who struck out 10 is more likely to reproduce it than the guy who fanned 4.

I don't really agree with the analogy, but your point is well taken. If indeed SR has significant predictive value, then it obviously has a lot of value. I'm going to e-mail Thiessen and ask him to comment on that post by Burke. Or maybe he'll address it himself in his followup.

Their claim is and has always been that DVOA is more predictive of winning than other measures-and success rate is the core of DVOA.

My issue with this is that I strongly suspect that 90% of the value of DVOA are the opponent adjustments. Something like Jeff Sagarin's data also beats out point differential, because it adds in those opponent adjustments. But if Sagarin is beating out DVOA (I've seen some indications of this, albeit indirect), then FO is doing it wrong. Sagarin's method is extremely simple, doesn't use play by play data or SR, and yet works pretty well.

[Alternate34]

I don't really agree with the analogy, but your point is well taken. If indeed SR has significant predictive value, then it obviously has a lot of value. I'm going to e-mail Thiessen and ask him to comment on that post by Burke. Or maybe he'll address it himself in his followup.

My issue with this is that I strongly suspect that 90% of the value of DVOA are the opponent adjustments. Something like Jeff Sagarin's data also beats out point differential, because it adds in those opponent adjustments. But if Sagarin is beating out DVOA (I've seen some indications of this, albeit indirect), then FO is doing it wrong. Sagarin's method is extremely simple, doesn't use play by play data or SR, and yet works pretty well.

Sometimes I disagree with you on DVOA. For example, I think the 4, 3, 2 v. 3, 3, 4 argument against Success Rate unless one can demonstrate that teams that go 3, 3 are more likely to get 4 yards on 3rd down than a team who goes 4, 3 to get those two yards, or more simply that teams have a repeatable ability to repeat success on third down beyond their ability to do well on 1st and 2nd down. Additionally, as mentioned above, DVOA is not based solely on success rate as a binary variable. There are gradations of success. A -5 yard loss and a 0 yard gain on first down are both failures, but the first gets more negative DVOA than the second. The better criticism which I think Football Outsiders does address is that per drive statistics generally do a better job than per play statistics. I say they address this because they have excellent drive statistics on their website available.

However, you are absolutely right about the predictability argument. They haphazardly mix their recipe with outcomes that are repeatable and outcomes that are not repeatable. The reason they do this is to produce a metric that can both accurately reflect who won a game and also predict who would win a game. Aaron addressed this in the chat here, but it is a problem. Turnover luck is something they don't deal with consistently. They have data that indicates that forced fumbles are a skill, fumble recoveries are luck. They do adjust DVOA for that. However, they also have data that the difference between a dropped INT and an INT is luck for a QB. However, I am pretty sure their metric does not reflect that, as a dropped INT has the same value as an incomplete pass rather than a value in between an incomplete pass and an INT, similar to fumble forced v. fumbles recovered.

Turnovers in particular are a salient example because those are plays that are highly variable from game to game, depending in large degree on luck, that significantly alter the course of a game. It is hard to know what else gets weighted as luck and not luck because your other criticism, that DVOA is a black box to a degree, is also true.