To kick it off, I wanted to link to this post by Brett Thiessen about success rates, and why they might be pretty useless. This is a personal bone I've had to pick with Football Outsiders for some time, and Thiessen explains in more depth why we should doubt the value of the success rate statistic in football. He misses the mark I think with the "Football is not baseball" moniker, but that's neither here nor there. I think he raises a very good point when he writes:
A couple of examples of how success rate can be misleading (first down gain, second down gain, third down gain):
4,3,2: This is a 67% success rate but is a three and out.
3,3,4: This is a 33% success rate but a first down, plus the first two plays are nearly identical but the first two downs of the first group are both successes and the second group are both failures. Over a large group of data some of these will iron themselves out, but why put such a black and white metric over something that is not. 2nd and 7 is almost the same as 2nd and 6, but 2nd and 1 is very different from 2nd and 6. Success rate completely misses the magnitude of plays.
This has come up extensively in the Tebow thread, but in many other debates as well, like about the relative value of BJGE vs. other running backs. That said, while I'm a big fan of the point Thiessen is making here, it's a little light on analytics, and is more of a deductive explanation of why success rates shouldn't be used. So I'd be interested in hearing what other people think - is there a response to this? Football Outsiders has consistently refused to explain why success rates should be used, but maybe there's a good reason. If so, what is it?