52 replies to this topic

[philly sox fan]

We're happy to have Nate Silver of Baseball Prospectus (and more specifically the creator of the PECOTA projections) stop by to do a chat.

People who are subscribers know that the PECOTA cards have been a drawn out delivery this year but the hitter cards were posted this weekend with the pitcher cards expected soon-ish. In part due to that we haven't yet settled on an exact time, but we'll be handling this like previous "chats" where we simply ask for questions and let Nate answer at his convenience. As a result we'll open this thread up to questions.

I assume there will be a lot of specific interest in the PECOTA process, but any baseball related questions are welcome.

Just a few things I've noticed perusing the hitter cards and some things that Nate has mentioned previously:

- the new cards feature platoon splits
- because predicting 5 years of performance was apparently too easy the new cards feature 7(!) year forcasts
- draft status and signing bonus totals are now included which has led to projections for players without professional experience like David Price
- as a result of a change in the underlying Davenport Translations very young minor leaguers project much worse than they used too

And a whole lot more, I'm sure.

Edit: And thanks to Hairps for facilitating the chat.

[philly sox fan]

I'll start off by touching on that last issue. Assuming the UPSIDE calculation is the same I was immediately struck by the decreased UPSIDE figures for most prospects in the Projection Spreadsheet released a couple of weeks ago. And that difference has carried through onto the player cards with much lower MORP totals and "star" chances in the "Stars and Scrub" chart.

Here's a table showing the difference at each age level below 25.

Table

			2008 PECOTA					2007 PECOTA
Age	Players	Upside	UPSIDE/player		Age	Players	Upside	UPSIDE/player	%of2007
17	1	23.9	23.9		17
18	8	184.5	23.1		18	6	516.4	86.1	27
19	21	385.8	18.4		19	14	649.9	46.4	40
20	25	823.7	32.9		20	24	951.5	39.6	83
21	38	846.9	22.3		21	44	1751	39.8	56
22	66	2192.6	33.2		22	51	2989.1	58.6	57
23	72	3030.4	42.1		23	93	4849.8	52.1	81
24	105	4724.2	45.0		24	97	4870.9	50.2	90
Total	336	12,212	36.3			329	16,579	50.4	72

It looks like a nearly 30% across the board decrease in UPSIDE for young players. At a time when MLB teams seem to hoarding their prospects more than ever, are you concerned that PECOTA has swung the pendulum too far the other way?

[ScubaSteveAvery]

I have a couple questions:

1. Regarding the new cards for minor leaguers, is there a timetable to know when PECOTA expects them to break into the majors? Also, is the seven year forecast for those players the stats of what we would expect from those players if they were in the majors at that point in time? Take for example Lars Anderson's 2008 line in his seven year forecast - Is that what we would expect if he were in MLB this upcoming year? Or what we should expect him to do in the minors in 2008? My apologies if that was really vague or confusing.

2. Why is David Ortiz' similarity index so low (15), yet Manny's kinda high (42)? Why is Ortiz' career much more unique than somebody who has raked since he has been called up? I don't object to the scores, but am just curious as what PECOTA sees in Ortiz.

Thank you much for your time.

[Rough Carrigan]

One question right now.

Does PECOTA take into account a change in playing surface when a player goes to a new team, Ortiz going form Minnesota to Boston or now Glaus going from Toronto to St. Louis and Rolen going from St. Louis to Toronto?

[DSG]

What is an average breakout score? Collapse? Improve? Attrition? How confident are you in the accuracy of these metrics?

Does draft position get factored in for every player, or only players of a certain age? Do you discount it as sample size goes up?

Does it make sense for Manny to have a higher projected EqA and PA in 2011 than 2010? And why is his projected VORP lower?

[philly sox fan]

In looking at some of these very long term projections I was struck by how little the projected EQAs change for many players deep into their 30s. A few examples - Mike Lowell 276 at 34 and 271 at 37, JD Drew 283 at 32 and 281 at 37, Julio Lugo 263 at 32 and 261 at 36. The overall value projections decrease substantially in those intervals because the defense worsens and the projected playing time plummets, but has your PECOTA research suggested that offensive production on a per PA basis changes very little deep into a player's 30s? Or are the consistent EQAs a testament to how hard it is to do long term projections that deviate from established projection levels?

[Rough Carrigan]

Josh Beckett walked 40 batters last year. Yet, even his 90th percentile stat line has him with 50 something walks. Why does even the most optimistic projection not have him with the same control as last year?

(And the same is true of the PECOTA projection for Johan Santana. Even his 90th percentile stat line has him walking more men than last year.)

Edited by Rough Carrigan, 25 February 2008 - 12:59 PM.

[Rough Carrigan]

Have you considered making more specific adjustments to the comparable players based on such league wide conditions like the use of a livelier ball in 1911 and 1930, the national league's use of a less lively ball for most of the 30's while the AL was using one that jumped more, the dead ball of the WWII era? What about the high mound era from 1962-68? Is the effect too difficult to quantify?

[Worst Trade Evah]

My question is about the quality of age progression studies. Age has always been identified as a key component of ranking prospect quality -- a good 20 year old is generally regarded as a much better prospect than a good 22 or 23 year old. But have the studies that show this accounted for 1. position changes (high school outfielders that start pitching late) and 2. college players vs. similar age players who didn't go to college? If the pool of 23 year old pitchers compared to 20 year old pitchers includes 23 year olds who didn't go to college, isn't that a problem? If you compare the future careers of 20 year olds who pitch in MLB with 22 or 23 year olds who pitch in MLB -- but you only count college grads -- would you get a different result? How refined or granular are age progression studies?

Edited by Worst Trade Evah, 25 February 2008 - 03:08 PM.

[Rough Carrigan]

Could you please explain how a player's fielding Runs Above Average is calculated and is there another system of calculating defensive value that's similar to the one BP uses?

[philly sox fan]

Note that there is now a 3 pm Wed time for a "live" portion, but pre-submitted questions are easier all around so don't be shy.

[philly sox fan]

Do you plan to another series of PECOTA prospect posts and look back at how last year's PECOTA Top 100 compared to other lists? If I recall, wasn't there a bet between you and Kevin Goldstein about whose list would be better?

For subscribers (I think) here's the link to the 2007 PECOTA Top 100:

2007 PECOTA Top 100

Just doing a quick scan I'll note that PECOTA received a lot of credit for having Pedroia at #6, but not much criticism for having Alexi Casilla right behind him at #7.

Actually you did address those guys a bit"

In summary, I endorse most of these ratings, with the potential exceptions being the quartet of middle infield prospects (Pedroia, Casilla, and Patterson, and Lillibridge to a lesser extent) that don’t hit for power.

With the happy exception of Pedroia that quartet disappointed overall. Has their been an adjustment to better model middle of the diamond players with marginal power?

And one more not so happy excerpt:

Troy Tulowitzki (PECOTA #78, BA #15, KG #24). Tulo creates arguments not just between scouts and PECOTA, but also within the scouting community. Some dissidents in the later group concur with PECOTA’s conclusion that he just isn’t that special. The sample size isn’t large enough to make much out of Tulowitzki’s .210 career EqA in the big leagues, but this is a player who was supposed to have a smooth transition to the majors.

In this case though it's not just the relatively low placement it's that even amongst SS prospects he was rated below Marcus Lemon (63), Chris Valakia (66), Joel Guzman (68) and Paul Kelly (75). Did/does PECOTA have trouble with up the middle position prospects?

And that's the end of that run on question...

[gammoseditor]

Coco Crisp's FRAA went from -1 in his first year as a full time center fielder for the Red Sox in 2006 to 27 last year. While its obvious to look at these numbers and think that he got better from more experience in CF, do you think there should be similar skepticism in Coco maintaining the same level of defense as there would be if a player's batting average or home runs jumped significantly in a single season?

[Worst Trade Evah]

In Bill James's recent essay on Craig Biggio he writes that

He [Biggio]couldn't hit a good pitcher—never could, really. His career batting average in post-season play was .234, OPS somewhere around .600. His clutch hitting record is miserable.

We have this profile in the online … Batting Performance by Quality of Opposing Pitcher. Of course, over time, almost everybody is going to hit better against weaker pitchers. I doubt that anybody was as consistent or extreme about it as Biggio was. In 2003 he hit .354 against pitchers with ERAs over 5.25 (64 for 181), but .143 against pitchers with ERAs under 3.50 (19 for 133). In 2004 he hit .382 with 10 homers in 110 at bats against pitchers with ERAs over 5.25. Every year he has had huge good pitcher/bad pitcher splits...but the reason that Biggio struggled in clutch situations and against good pitchers couldn't be more obvious. He was an overachiever, and he knew what he was doing. Against a weak pitcher, a pitcher not really in command of his material, Biggio could take control of the at bat and drive it toward a good conclusion. When the pitcher was not really focused, Biggio was. But when the pressure was on and there was somebody on the mound who knew what he was doing, Biggio had limited ability to step up.

I've seen arguments like this around, but I don't think I've seen it so prominently placed in a close evaluation of a player by a major analyst. First, do you agree with this assessment of Biggio -- that he had limited ability to control at bats when a quality pitcher was actually focussed on him?

More generally, how significant are the differences in player ability to hit "quality" or step up, and how much should they inform roster construction? Here's a first ballot Hall of Famer with limited ability to step up, apparently. Is this kind of a sideways clutch discussion? Is this a reason why players like Manny may in fact be worth a lot more than their WARP/$ numbers suggest? Could a team, like say the Red Sox, deliberately collect a mix of grinding over-achievers (to get through the regular season) and coasting super-talents who might have some ability to turn it on in the post-season?

Should this factor into player valuation?

Edited by Worst Trade Evah, 26 February 2008 - 01:15 PM.

[Rough Carrigan]

Do the PECOTA comparables recognize groundball vs. flyball tendencies in pitchers?

[Hairps]

- How far away are we from being able to integrate, with confidence, injury history information into player projections?

- How far away are we from being able to integrate, with confidence, "scouting" observations (for example, Pitch F/X data on a pitcher's repertoire or the "type" of pitcher they are) into player projections?

- Which is likely to deteriorate more quickly, a "Groundball pitcher"'s GB% or a "power pitcher"'s K rate?

Edited by Hairps, 26 February 2008 - 08:22 PM.

[PedroKsBambino]

To what degree do you find that pitchers performance follows an age curve similar or different than hitters? IOW, for hitters it is generally agreed I think that the typical peak is a period around ages 26-30 and most hitters younger than this will have projected improvement, especially in the area of power, and then a decline of some trajectory after that range. For pitchers, is a similar curve accurate and if so, what is the peak of it? Does the progression for pitchers occur in particular ways (e.g. more strikeouts, better control, etc) or is it a general growth?

Also, shifting gears a bit, how would you rate your confidence level in the offensive and defensive components of the projections? I see many people using the PECTOA MORP data to evalute trades and I wonder if perhaps there should be a different regression applied to offensive and defensive numbers comprising it.

[Kevin Youkulele]

For hitters, "old player skills" have been characterized such as increased plate discipline. Are there also "old pitcher skills" that tend to grow even as velocity, etc., decline?

[Worst Trade Evah]

What is the marginal value of a sabermetrician to a team? Is 99% of the value in modern analysis available to anyone with access to a few web sites? Why wouldn't a team offer $1,000,000 to someone like you or Tom Tango? If they did, would you take it? How much do you think you could impact a team's chances to win if you focussed exclusively on that team's issues?

What would be your priorities if you were given control of the analysis department for a major league team? What team would you really like to have control over -- aside (or including) from being a fan, what teams are interesting from your perspective?

Edited by Worst Trade Evah, 26 February 2008 - 04:01 PM.

[Worst Trade Evah]

I guess I'm on a roll, so...

You feel value is non-linear, as you represent it in MORP, which results in significant bending at the margins (ARod's "value" at $38 million or whatever). Tango argues that value is actually pretty linear. It seems a fundamental issue. Why are you right and he's wrong? (Clash of the analysts!)

Edited by Worst Trade Evah, 26 February 2008 - 04:12 PM.

[Rough Carrigan]

In the 2008 Baseball Prospectus writeup on Vernon Wells, it says, explicitly, that his defense is "good". Yet, looking at the last column of his stats we're told that he's been 7, 10 and 3 runs below average defensively for a centerfielder the last 3 seasons. Does the author of the paragraph about Vernon Wells disagree with your system of calculating defensive performance?

[Tony C]

My question is how much is Pecota a quantitative exercise, and how much qualitative? By analogy, political polling, for example, is seen as a purely quantitative exercise, but in fact the most important element is a qualitative estimate about turnout among various demographics and good pollers have good qualitative political instincts to make the right calls on that. Along the lines of that analogy, I understand that Pecota this year is giving a lot less value to minor league prospects. It was, indeed, notable last year that kids scheduled for AA seemed to have some inflated projections, so clearly alterations were needed. So, similar to adjusting polling demographics, what were the adjustments that were made to reduce Pecota numbers for kids and, with that in mind, how much of a qualitative rather than quantitative exercise is putting together Pecota?

[TomRicardo]

When you evaluate a pitcher who was injured last year what methods do you use to predict his effectiveness going forward?

[Kevin Youkulele]

How much does the formula for WARP get recalibrated from year to year? Do the weights of individual components change, as opposed to the replacement level baseline? Is WARP3 a static formula while WARP1 is recalculated based on each year's environment? Presumably the accuracy of WARP, in terms of representing actual wins, could be checked by adding the WARPs of each player on a team to replacement level wins (30 or so) and seeing how that compared to real records, or third order wins to reduce the effect of luck; do you know how close it gets?

[Frisbetarian]

Nate should be here shortly to answer your questions. Please do not post any new questions until he is finished with the existing ones. Hopefully, Nate will be able to stick around for a while and answer some follow-ups.

I would like to thank Nate for taking the time to come here and educate our unwashed masses. Nate - if you have any problems with posting, please contact me at Frisbetarian@gmail.com and we can take care of them. I will be gone (driving the kid around) until about 3:10.

[NateSilver]

Testing 1, 2, 3. Testing.

[NateSilver]

Testing 1, 2, 3. Testing.

Cool, looks like I'm fully functional. Guys, I'm just going to start picking off some of your questions -- feel free to add others, naturally, and thanks to Philly and everyone else for the opportunity to be here today.

[NateSilver]

I'll start off by touching on that last issue. Assuming the UPSIDE calculation is the same I was immediately struck by the decreased UPSIDE figures for most prospects in the Projection Spreadsheet released a couple of weeks ago. And that difference has carried through onto the player cards with much lower MORP totals and "star" chances in the "Stars and Scrub" chart.

Here's a table showing the difference at each age level below 25.

Table

			2008 PECOTA					2007 PECOTA
Age	Players	Upside	UPSIDE/player		Age	Players	Upside	UPSIDE/player	%of2007
17	1	23.9	23.9		17
18	8	184.5	23.1		18	6	516.4	86.1	27
19	21	385.8	18.4		19	14	649.9	46.4	40
20	25	823.7	32.9		20	24	951.5	39.6	83
21	38	846.9	22.3		21	44	1751	39.8	56
22	66	2192.6	33.2		22	51	2989.1	58.6	57
23	72	3030.4	42.1		23	93	4849.8	52.1	81
24	105	4724.2	45.0		24	97	4870.9	50.2	90
Total	336	12,212	36.3			329	16,579	50.4	72

It looks like a nearly 30% across the board decrease in UPSIDE for young players. At a time when MLB teams seem to hoarding their prospects more than ever, are you concerned that PECOTA has swung the pendulum too far the other way?

Philly,

The principle reason that the Upside scores are lower is because Clay has revised his minor league translations to make them more sensitive to changes in league difficulty. This is because, quite frankly, the minor league projections were coming in too high overall before (by roughly 25 points of OPS per hitter, by my calculations). But what this seems to be doing is giving PECOTA a greater opportunity to do its job, which is to differentiate between the wheat and the chaff. If you look at someone like Homer Bailey, for instance -- Bailey gets a very harsh projection for 2008, with an EqERA in the low 5's. But PECOTA recognizes that this is a skill set that can grow prodigiously, and so it has him shaving nearly a whole point off his ERA by the time he reaches age 25 or 26. PECOTA was having difficulty doing this before, particularly with pitchers, when the translations were too flat'.

[NateSilver]

I have a couple questions:

1. Regarding the new cards for minor leaguers, is there a timetable to know when PECOTA expects them to break into the majors? Also, is the seven year forecast for those players the stats of what we would expect from those players if they were in the majors at that point in time? Take for example Lars Anderson's 2008 line in his seven year forecast - Is that what we would expect if he were in MLB this upcoming year? Or what we should expect him to do in the minors in 2008? My apologies if that was really vague or confusing.

2. Why is David Ortiz' similarity index so low (15), yet Manny's kinda high (42)? Why is Ortiz' career much more unique than somebody who has raked since he has been called up? I don't object to the scores, but am just curious as what PECOTA sees in Ortiz.

Thank you much for your time.

Scuba,

Y'know, it would be fascinating to add a variable that indicated when a guy was expected to break into the major leagues, and perhaps that's something we can incorporate in future versions. But as a broad gauge, when a guy's weighted-median VORP gets substantially above zero is probably a pretty good gauge. The market for baseball talent *is* relatively efficient -- or at least it has become that way -- and players rarely languish in the minor leagues too long when they're capable of a performance above MLB replacement level, nor do they tend to stick in the major leagues if they can't meet that description.

With respect to Ortiz and Manny -- David Ortiz has a couple of skills (isolated power and drawing walks) that essentially are off the charts -- not without precedent by any means, but a couple of good standard deviations away from the mean. The PECOTA comparable scored are based, essentially, are based on how many standard deviations a guy is away from the mean across different categories, so these sort of performances behave less linearly than you might think -- there is a big difference, in PECOTA terms, between a guy who performs at the 90th-95th percentile (like Manny) and one who performs at the 99th percentile (like Ortiz). Also, Ortiz is a DH, and there aren't as many DHes throughout history as there are players at other positions, so PECOTA dings his comp scores as a result.

[NateSilver]

One question right now.

Does PECOTA take into account a change in playing surface when a player goes to a new team, Ortiz going form Minnesota to Boston or now Glaus going from Toronto to St. Louis and Rolen going from St. Louis to Toronto?

RC,

The playing surface is one of the factors that PECOTA uses in developing its park factors (which are essentially estimates of 'true' park factors rather than anything purely empirical). But apart from that, there's no direct attempt to account for playing surface. Also, we do not park factor stolen bases (which maybe we should), where the playing surface presumably might have the most effect.

[NateSilver]

What is an average breakout score? Collapse? Improve? Attrition? How confident are you in the accuracy of these metrics?

Does draft position get factored in for every player, or only players of a certain age? Do you discount it as sample size goes up?

Does it make sense for Manny to have a higher projected EqA and PA in 2011 than 2010? And why is his projected VORP lower?

The average breakout and collapse scores are so dependent on a guy's age and profile, that I'm not sure it helps to point to some sort of average number. From a 30,000-foot level, I suppose I consider anything above 20-25% in these departments to be noteworthy for an established player, but many players who are not established can have scores quite a bit higher than that.

The draft position variable gets factored in so long as PECOTA treats a player as a 'minor leaguer', which is a somewhat arbitrary distinction depending on his age and the amount of playing time he got between the majors and the minors in the previous season. So, in terms of that variable being present, it's an on-off thing rather than some sort of gradient.

With respect to Manny's numbers -- one of the major issues you run into when you're projecting a guy's performance 5-6 years out is that there are substantial selection effects. Basically, a guy will either continue playing *and still be pretty good*, or he'll retire (or get benched, etc.). So when you're looking at a rate variable like EqA, you're seeing only the performance of the guys who stayed in the sampe, and there may *not* be all that big of a drop-off because of these selection effects. On the other hand, if we're looking at something like WARP, we're able to handle the players who dropped from the dataset by assigning them a score of zero, which is factored into the denominator. This is really the more desirable way to do things, so I would tend to look at the progress of a guy's VORP and WARP rather than his rate numbers when evaluating his long-term performance.

[NateSilver]

Josh Beckett walked 40 batters last year. Yet, even his 90th percentile stat line has him with 50 something walks. Why does even the most optimistic projection not have him with the same control as last year?

(And the same is true of the PECOTA projection for Johan Santana. Even his 90th percentile stat line has him walking more men than last year.)

Wow, these are some pretty technical questions -- you folks have really done your homework. The thing to be aware of when looking at the percentile forecasts is that they are calibrated around one specific variable and one specific variable only -- that variable is EqA for hitters and EqERA for pitchers. The percentiles are *not* designed to work for the component statistics. If Beckett hit his 90th percentile in walk rate *and* strikeout rate *and* home run rate *and* BABIP and everything else, the cumulative effect would be much greater than his hitting his 90th percentile in terms of EqERA alone. So what PECOTA does is to calibrate things based on EqERA, and then it develops a best-fit set of peripheral statistics to match that EqERA estimate.

[NateSilver]

Have you considered making more specific adjustments to the comparable players based on such league wide conditions like the use of a livelier ball in 1911 and 1930, the national league's use of a less lively ball for most of the 30's while the AL was using one that jumped more, the dead ball of the WWII era? What about the high mound era from 1962-68? Is the effect too difficult to quantify?

We don't use any comparables from before World War II, precisely because IMO there were points in time before that in the game's history (e.g. the deadball era) where the nature of the game was just too different from the way it is played today. Something like the high mound era, on the other hand, is still recognizable as modern baseball, and I think can be handled adequately through standard normalization techniques.

[NateSilver]

My question is about the quality of age progression studies. Age has always been identified as a key component of ranking prospect quality -- a good 20 year old is generally regarded as a much better prospect than a good 22 or 23 year old. But have the studies that show this accounted for 1. position changes (high school outfielders that start pitching late) and 2. college players vs. similar age players who didn't go to college? If the pool of 23 year old pitchers compared to 20 year old pitchers includes 23 year olds who didn't go to college, isn't that a problem? If you compare the future careers of 20 year olds who pitch in MLB with 22 or 23 year olds who pitch in MLB -- but you only count college grads -- would you get a different result? How refined or granular are age progression studies?

WTE,

Well, fundamentally this is exactly the reason that I started working on PECOTA -- to avoid one-sized-fits-all type of aging curves.

Right now, however, we do not make a distinction between high school and college players, principally because I do not have a complete database of who went to college and who did not.

[NateSilver]

Could you please explain how a player's fielding Runs Above Average is calculated and is there another system of calculating defensive value that's similar to the one BP uses?

That's more a question for Clay Davenport; FRAA is not my baby. There are lots of other systems out there to calculate defensive value -- and frankly, the ones that take advantage of detailed play-by-play data (which FRAA does not) are liable to be more accurate. We are working on some PBP-based defensive metrics to compliment FRAA, however.

[NateSilver]

Do you plan to another series of PECOTA prospect posts and look back at how last year's PECOTA Top 100 compared to other lists? If I recall, wasn't there a bet between you and Kevin Goldstein about whose list would be better?

For subscribers (I think) here's the link to the 2007 PECOTA Top 100:

2007 PECOTA Top 100

Just doing a quick scan I'll note that PECOTA received a lot of credit for having Pedroia at #6, but not much criticism for having Alexi Casilla right behind him at #7.

Actually you did address those guys a bit"

With the happy exception of Pedroia that quartet disappointed overall. Has their been an adjustment to better model middle of the diamond players with marginal power?

And one more not so happy excerpt:
In this case though it's not just the relatively low placement it's that even amongst SS prospects he was rated below Marcus Lemon (63), Chris Valakia (66), Joel Guzman (68) and Paul Kelly (75). Did/does PECOTA have trouble with up the middle position prospects?

And that's the end of that run on question...

Of that quartet of players you mention -- Pedroia, Casilla, Lillibridge, Patterson -- I think it's a little early to write off Lillibridge and Patterson, both of whom had pretty good seasons, and to a lesser extent Casilla.

But, it's worth asking whether PECOTA had overrated this type of player in general, and it may be a consequence of our translations having tended to come in slightly too high overall for minor leaguers. The whole argument for a guy like Patterson was that while he might not have as much upside, he essentially guaranteed you an MLB average level of performance. But if you knock 10-15 points off his EqA as the result of a harsher translation schedule, then all of the sudden you're locked into an MLB fringe performance, rather than an MLB average performance, which is not such a good thing.

[NateSilver]

Coco Crisp's FRAA went from -1 in his first year as a full time center fielder for the Red Sox in 2006 to 27 last year. While its obvious to look at these numbers and think that he got better from more experience in CF, do you think there should be similar skepticism in Coco maintaining the same level of defense as there would be if a player's batting average or home runs jumped significantly in a single season?

Indeed, I think some skepticism is warranted, at least from a statistical standpoint. Precisely because defensive statistics are so crude, people tend to overlook that players can have fluke years in the field just like they have at the plate. But I have no doubt that Coco's D has substantially improved.

[NateSilver]

In Bill James's recent essay on Craig Biggio he writes that
I've seen arguments like this around, but I don't think I've seen it so prominently placed in a close evaluation of a player by a major analyst. First, do you agree with this assessment of Biggio -- that he had limited ability to control at bats when a quality pitcher was actually focussed on him?

More generally, how significant are the differences in player ability to hit "quality" or step up, and how much should they inform roster construction? Here's a first ballot Hall of Famer with limited ability to step up, apparently. Is this kind of a sideways clutch discussion? Is this a reason why players like Manny may in fact be worth a lot more than their WARP/$ numbers suggest? Could a team, like say the Red Sox, deliberately collect a mix of grinding over-achievers (to get through the regular season) and coasting super-talents who might have some ability to turn it on in the post-season?

Should this factor into player valuation?

Let me be blunt: I think this kind of debate about a player's clutch abilities is pretty close to worthless unless you're willing to evaluate a guy's entire career, and do so in a systematic, coherent way. James is cherry-picking a couple of statistics from a couple of seasons in a 20-year career, and gives us no framework to evaluate these statistics (what is a standard "Good pitcher / Bad pitcher" split, for instance?).

If there is such a thing as clutch ability, then certainly it is worth something. And I'm on record as saying that clutch ability exists -- it's just very small compared to what most people believe. And I don't think cherry-picking statistics like the way James did for Biggio is the right way to find it, whether or not it's there.

[NateSilver]

Do the PECOTA comparables recognize groundball vs. flyball tendencies in pitchers?

Yes. This might be the third-most important way to differentiate pitchers, after stuff/strikeout rate and control/walk rate.

[NateSilver]

- How far away are we from being able to integrate, with confidence, injury history information into player projections?

- How far away are we from being able to integrate, with confidence, "scouting" observations (for example, Pitch F/X data on a pitcher's repertoire or the "type" of pitcher they are) into player projections?

- Which is likely to deteriorate more quickly, a "Groundball pitcher"'s GB% or a "power pitcher"'s K rate?

The problem with injury data is simply that very few people have compiled any sort of comprehensive injury database. I'm sure there are all sorts of fun things we could do with that data if we had it. Maybe one year, we'll hire a particularly masochistic intern and have him sort through 40 years of the Sporting News archive on microfiche, but we haven't done so yet.

The Pitch F/x type of data ... I suspect it's only a matter of time before forecasting systems are starting to incorporate that kind of stuff. Maybe not next year, but in 3-5 year's sure.

...Groundball rates are just about the most stable statistic known to man, so they tend to stay intact longer than strikeout rates.

[NateSilver]

To what degree do you find that pitchers performance follows an age curve similar or different than hitters? IOW, for hitters it is generally agreed I think that the typical peak is a period around ages 26-30 and most hitters younger than this will have projected improvement, especially in the area of power, and then a decline of some trajectory after that range. For pitchers, is a similar curve accurate and if so, what is the peak of it? Does the progression for pitchers occur in particular ways (e.g. more strikeouts, better control, etc) or is it a general growth?

Also, shifting gears a bit, how would you rate your confidence level in the offensive and defensive components of the projections? I see many people using the PECTOA MORP data to evalute trades and I wonder if perhaps there should be a different regression applied to offensive and defensive numbers comprising it.

The pitcher aging curve is far more irregular than the hitter aging curve, and basically tends to resemble a very long, very slow, and relatively linear decline when taken in the aggregate, at least from the age of 24/25 or so onward. At certain points in a player's career, you might actually prefer the slow and steady decline to the steeper decline that hitters have. For example, I've never quite looked at it like this, but I *suspect* a 37-year old All-Star pitcher is a better bet to still be an All-Star pitcher at age 40 than a 37 year-old All-Star hitter is.

On the other question, while I recognize the disagreements that people have with FRAA and therefore WARP, the thing to recognize is that the PECOTA defensive forecasts are already regressed quite heavily to the mean, so I'm not sure if there's an inherent need to do that above and beyond what the system does. Of course, if you know that FRAA is "wrong" about a player's defense, you should always want to take that into account.

[NateSilver]

For hitters, "old player skills" have been characterized such as increased plate discipline. Are there also "old pitcher skills" that tend to grow even as velocity, etc., decline?

A pitcher's "old player skill" is basically his walk rate. Also, pitchers tend to get smarter about situational pitching as they age, which means that older pitchers tend to have ERAs that are better than their PERAs (component ERAs). This effect is very minor, however.

[NateSilver]

What is the marginal value of a sabermetrician to a team? Is 99% of the value in modern analysis available to anyone with access to a few web sites? Why wouldn't a team offer $1,000,000 to someone like you or Tom Tango? If they did, would you take it? How much do you think you could impact a team's chances to win if you focussed exclusively on that team's issues?

What would be your priorities if you were given control of the analysis department for a major league team? What team would you really like to have control over -- aside (or including) from being a fan, what teams are interesting from your perspective?

Baseball teams tend to have a sort of Chinese Wall between their on-field and off-field P&Ls, so the same teams that can spend millions of dollars on the latest overrated toolsy outfielder may at the same time be very cheap with its front office expenditures. But the real problem is simply that baseball teams don't have to pay market rates because there are lots of smart young people who are willing to work for substantially less than that, with baseball being a high-status job. We sabermetricians should unionize, damnit!

[NateSilver]

I guess I'm on a roll, so...

You feel value is non-linear, as you represent it in MORP, which results in significant bending at the margins (ARod's "value" at $38 million or whatever). Tango argues that value is actually pretty linear. It seems a fundamental issue. Why are you right and he's wrong? (Clash of the analysts!)

We've actually re-done the WARP formula this year so that it's based on a combination of WARP and VORP. And there *is* still some non-linearity in there, even when we have the luxury of accounting for both variables. In fact, the amount of nonlinearity seems to be increasing, as baseball teams are increasingly adopting a 'stars and scrubs' approach when it comes to paying talent -- there aren't that many contracts being signed in the 3-year, $17 million range anymore.

[NateSilver]

In the 2008 Baseball Prospectus writeup on Vernon Wells, it says, explicitly, that his defense is "good". Yet, looking at the last column of his stats we're told that he's been 7, 10 and 3 runs below average defensively for a centerfielder the last 3 seasons. Does the author of the paragraph about Vernon Wells disagree with your system of calculating defensive performance?

Yes.

[NateSilver]

My question is how much is Pecota a quantitative exercise, and how much qualitative? By analogy, political polling, for example, is seen as a purely quantitative exercise, but in fact the most important element is a qualitative estimate about turnout among various demographics and good pollers have good qualitative political instincts to make the right calls on that. Along the lines of that analogy, I understand that Pecota this year is giving a lot less value to minor league prospects. It was, indeed, notable last year that kids scheduled for AA seemed to have some inflated projections, so clearly alterations were needed. So, similar to adjusting polling demographics, what were the adjustments that were made to reduce Pecota numbers for kids and, with that in mind, how much of a qualitative rather than quantitative exercise is putting together Pecota?

Tony,

That's a fairly interesting analogy with political polling. Ostensibly, both are entirely quantitative exercises. You don't take a poll that has Obama 2 points ahead of McCain and say that "you know what, I think that Obama should be 5 points ahead of McCain, so that's the number I'm going to publish". But you *might* be able to fiddle with your turnout model instead to sort of reverse-engineer that result.

I tend to think that this is a bad thing when you're doing political polling. But for a baseball forecasting system, this feedback process can be highly useful. A lot of the improvements I've made to PECOTA are essentially the result of my saying: "You know what -- this forecast just doesn't look right to me. What sort of assumptions might have produced it? Are there reasonable arguments for changing those assumptions?". So there's a certain amount of trial-and-error there. But the thing with PECOTA is that if you make a change that affects one forecast, it's going to affect all 1,000 forecasts, so that provides a reality check against trying to engineer a particular result for a particular player.

[NateSilver]

When you evaluate a pitcher who was injured last year what methods do you use to predict his effectiveness going forward?

We hold our nose and take our best guess. But, I'm also of the belief that there's a sort of continuum between health and performance, so sometimes the statistical record can tell you more than you might think.

[NateSilver]

How much does the formula for WARP get recalibrated from year to year? Do the weights of individual components change, as opposed to the replacement level baseline? Is WARP3 a static formula while WARP1 is recalculated based on each year's environment? Presumably the accuracy of WARP, in terms of representing actual wins, could be checked by adding the WARPs of each player on a team to replacement level wins (30 or so) and seeing how that compared to real records, or third order wins to reduce the effect of luck; do you know how close it gets?

'Lele:

That's another question for Clay Davenport, but I know generally speaking he's very aggressive about making tweaks of all kinds to the WARP formula. Both Clay and I tend to be perfectionists about our algorithms and programs, which leads quite literally to a lot of sleepless nights.

...

Looks like that's the last question. Thanks for joining me, everyone, and I hope I'll see you over at the BP side soon!

[Rough Carrigan]

Ron Shandler had or perhaps still has a practice of trying to predict hitter performance using, as part of the data, hitter groundball and flyball rates. What do you think of this? Have you considered using that data to better match comparables and better project how a hitter's underlying skill set will come out in future performance?

[Worst Trade Evah]

Aside from the "clutch" problem, do some hitters have an ability to hit "quality" that isn't just a function of their normal day-to-day performance level?