coremiller said:
DotB, I don't actually disagree with much here, except that I think a lot of the statistics while imperfect, add value to the discussion, so long as you recognize their limitations. [SIZE=14.4444446563721px]I learn things from them. [/SIZE]Even misleading statistics can be educational, because figuring out how they are misleading often teaches you more than the statistics themselves do. Mostly what bothers me are your claims that "there is only one right answer when you look at it like that, and it's Brady" and "I'm not so blind as to dismiss other's arguments, but at this point (and for me, this was the case long before yesterday), I can't see anyone rationally being able to leave Brady out over someone else." I just don't think it's that clear cut, and I definitely don't think your observational evidence proves those conclusions. I guess what I don't understand is how you can be so thoroughly aware of all the methodological problems with statistical analysis but not similarly discount observational analysis, which has its own similar problems.
I don't have an issue discounting observational analysis, but I find that when it comes to football, it's much harder to do. For example, let's look briefly at the studies you talked about above that conclude that Montana is world's better than everyone else. To accept the conclusions from those studies, you have to accept as gospel that Adjusted yards per pass attempt is basically the greatest statistic when judging a quarterback's postseason abilities. If I remember correctly, this is a statistic that provides for an arbitrary 45 yard penalty for interceptions and 20 yard bonus for touchdowns. Don't really remember why, don't really care, just know that it's because someone chose that. Why is it the one they used in those studies? Because it's his "favorite" and apparently, is the stat that has the highest correlation to wins. Not sure why we don't just go back to wins and losses if that's the goal, but I'll play along a bit more. But the bottom line is that the entire statistical analysis is based on yards per pass attempt or adjusted net yards per pass attempt. The problem is those stats both have the exact same problems that almost every other statistic does that we talk about, weather, opponents, teammates, etc. The entire study starts with a flawed statistic as it's basis.
Then from there, he "adjusts" for opponent and era in one swoop. How does he do that? By looking at the opponents net adjusted yards per pass attempt. That's it. Nothing else. Then they factor in "leverage" which is nothing more than assigning a value to each game based on the number of teams left at that moment and thus, their "odds" of winning the Super Bowl. For example, at the super bowl level, there are only two teams left, so the expected delta is 50%. At the Championship game level, it's a 25% because there is 4 teams left. I don't think I have to spend too much time explaining why there are inherent flaws in assuming just because a team is in the conference championship, the odds of them winning the Super Bowl is not exactly 25% for every team, every year.
I'm not trying to get into a battle of semantics over the methodology of this study, and if I made some minor mistake in my paraphrasing it from memory, I apologize, but that's not the point. The point is that these are fun little exercises that in the end spit out a list, and we all know everyone loves lists, but these lists don't really tell us anything except that if you accept what the author is choosing to use as his formula, that's the answer you get. It's not conclusive as to anything else, and when I see folks cite something like that, it gets the hair on the back of my neck to stand up because there are a whole lot of folks who see that list and just assume, "Hey, this guy did a ton of work and spent a lot of time on this, so I guess Montana really is the best," when a study like that couldn't possibly prove that. I mean shit, any equation/study or whatever you want to call it that concludes that Jim Plunkett's 1980 playoff season was the 2nd best playoff season EVER for a QB in the NFL just doesn't even pass the smell test, never mind when you dig deeper into the methodology. For those that don't know, Plunkett was 49-92 (53% completion percentage) for 839 yards, 7td's and 3int's and a 96.1 QB rating, through 4 games. He was absolutely dismal in the first two games of that playoffs, played ok in the conference championship, and then went crazy in the Super Bowl, and because of his high pass per attempt and great Super Bowl (which is weighted so much heavier than every other game, even though the playoffs are one and done anyway), he ends up at #2, higher than guys like Aaron Rodgers in 2010, Aikman in 92, and even Montana in 89.
Anyway, I was trying to keep that brief, but obviously failed a bit. For me, observational data is so much more important and easier to confirm or deny than these kinds of studies, which is why I've come to the point where I prefer them. When I write that Tom Brady doesn't get sacked and fumble with the game on the line, or throw a pick that results in his team losing, there may not be a stat for it, but it's not hard to counter. The guy only has 48 losses in his career, including the post-season. How many times did he blow the game down the stretch in those losses? I don't need a statistic to tell me it's not that many by comparison to a bunch of guys with twice as many losses. I can go by memory (for example, the Pats lost 4 games this year, one of which he didn't play, 2 of which they were blown out, which leaves Green Bay, and he did neither in that game), go read a play by play or go watch the film. I think folks think that because observational data is not easy to locate in a numerical form, that makes it subjective or unreliable or impossible to prove or deny, but that's not really true. It's just not as easy to disprove, which bothers a lot of math folks. Sure, it's easy to say, but I don't write things that I don't know to be true. I've done the research and know that when I write "you don't see many broken plays in New England like you do around the league on any given Sunday", it's the truth. Just because the "research" in that case is watching thousands of hours of football or reading hundreds of play-by-play charts and box scores as opposed to taking some stats, smashing them together and calling it a study, doesn't make it any less true. It just makes it harder for someone to disprove.
There is place for observational data, and there is a place for statistical analysis. And IMO, there is room for both, and both should be used, but with football statistics, I don't believe you can necessarily form many conclusions without also observational data, which is not the case in a sport like baseball. I also believe that football is so much harder for people to understand on a micro level that they want to be able to use statistics to tell them what they can't figure out on their own. I just don't think it's possible.