[BosoxBob]

I am planning on tracking the performance of all Red Sox players this year using Win Probability Added or WPA (see this article at The Hardball Times for a good overview). I first got interested in WPA in light of the 2005 AL MVP award when I determined that David Ortiz's clutch hits made him more valuable than ARod and his overall performance, even taking into account ARod's defense and the relative replacement levels for DH and 3B. As part of that study, I developed win probability tables using data from the 1993-2005 seasons - the period of time covering today's higher scoring environment.

In any case, what I'll be doing is tracking the game situation and outcome for each Red Sox players' at bats and computing the WPA. I'll also do the same for pitching (avoiding the pitching vs. fielding issue for the time being by awarding all credit/blame to the pitchers). Then for each game, I'll normalize the sum of all WPA scores to +1 (for wins) or -1 (for losses). I intend to post the WPA values for each game, as well as accumulated totals for the year.

Game 1: April 3, 2006
Red Sox 7 Rangers 3

Hitting                Pitching

Jason Varitek   .425   Curt Schilling    .476
David Ortiz     .332   Jonathan Papelbon .007
Mark Loretta    .036   Keith Foulke      .000
Trot Nixon      .033
Alex Gonzalez   .000
Coco Crisp     -.070
Kevin Youlikis -.075
Manny Ramirez  -.081
Mike Lowell    -.084

Game 2: April 4, 2006
Rangers 10 Red Sox 4

Hitting                Pitching

Coco Crisp      .143   Rudy Seanez       .000
David Ortiz     .115   David Riske      -.003
Mike Lowell     .016   Lenny DiNardo    -.038
Wily Mo Pena    .000   Tim Wakefield    -.648
J.T. Snow       .000
Kevin Youlikis -.003
Josh Bard      -.034
Alex Cora      -.105
Mark Loretta   -.119
Trot Nixon     -.142
Manny Ramirez  -.183

This post has been edited by BosoxBob: 06 April 2006 - 11:35 AM

[templeUsox]

I don't have much to add yet. But I think this is fascinating, and I look forward to following it all year.

[WSD2]

Very Interesting, I'll be watching this.....why is Foulke @ zero? He gave up a run and even though we had a large lead, I would think that should negatively impact the team since the probability of loss, though remote for the Sox, should have increased slightly as a result of the additional run.

This post has been edited by WSD2: 05 April 2006 - 04:41 PM

[BosoxBob]

WSD2, on Apr 5 2006, 01:40 PM, said:

Very Interesting, I'll be watching this.....why is Foulke @ zero?  He gave up a run and even though we had a large lead, I would think that should negatively impact the team since the probability of loss, though remote for the Sox, should have increased slightly as a result of the additional run.
<{POST_SNAPBACK}>

The thing to remember is that I'm computing the difference in win probability between when Foulke entered the game (win probability virtually equally to 1) and when he finished the game (win probability exactly 1). I'm not looking at intermediate points during the inning, although if I did, then Foulke would have been at -.013 after giving up the double to get to the 2nd and 3rd/1 out situation.

[PedroSpecialK]

Great article, I too will be very interested to see where this goes. Those numbers seem pretty accurate to me, though I do think Loretta should be slightly higher with Timlin a good bit lower.

Nice find

[Bucknahs Bum Ankle]

I too find this very interesting. Is it possible you could enter the data for each game in a new post and keep the old ones up so that we could see past results/how things change.

[Worst Trade Evah]

Bucknahs Bum Ankle, on Apr 6 2006, 09:57 AM, said:

I too find this very interesting.  Is it possible you could enter the data for each game in a new post and keep the old ones up so that we could see past results/how things change.
<{POST_SNAPBACK}>

Yes, please don't over-write the old games! You have the log and the data, let's keep it. Who knows? Might be useful for some other purpose.

Great job.

[BosoxBob]

PedroSpecialK, on Apr 6 2006, 05:58 AM, said:

Great article, I too will be very interested to see where this goes. Those numbers seem pretty accurate to me, though I do think Loretta should be slightly higher with Timlin a good bit lower.<{POST_SNAPBACK}>

Loretta's case illustrates the value provided by this sort of analysis. Mark came to bat three times with the bases empty and reached base (WPAs of .064, .074, .077). Obviously all good contributions. But then there's his at-bat in the 6th inning, which came with 1 out, a runner at third, and his team down by a run. In looking at last year's Expected Runs Matrix, Loretta's making an out while not generating a run cost about 0.6 runs (.9795-.3718). The context of the situation (trailing by one in the later half of the game) just magnified the failure, and resulted in a WPA of -.209. If the same thing happened in the 9th inning of Monday's blowout, the WPA would have been zero, since it was a virtual certainty that the Sox would win holding a 5 run lead.

In Timlin's case, the bottom line is whether any runs were allowed. So no matter how harrowing the experience, Timlin gets full credit for pitching a scoreless eighth. Now if he continues to pitch that way, eventually it will catch up with him (and be reflected by runs allowed and negative WPA).

Bucknahs Bum Ankle, on Apr 6 2006, 06:57 AM, said:

I too find this very interesting.  Is it possible you could enter the data for each game in a new post and keep the old ones up so that we could see past results/how things change.<{POST_SNAPBACK}>

No problem. I was debating which way to go, and also waiting to see what level of interest there would be in this. Here's last night's entry (which I'll now replace up top with the entries from the first two games:

Game 3: April 5, 2006
Red Sox 2 Rangers 1

Hitting                Pitching

Trot Nixon      .520   Josh Beckett      .596
Manny Ramirez   .079   Jonathan Papelbon .386
Coco Crisp      .025   Mike Timlin       .322
Mark Loretta    .007
Jason Varitek  -.060
Mike Lowell    -.088
Alex Gonzalez  -.139
J.T. Snow      -.160
David Ortiz    -.487

The story tonight - pitching, pitching and more pitching. Our new (co-)ace mowed down the opposition until Trot's shot could give the Sox the lead. Timlin picks up some undeserved credit for his rocky eighth inning, and Papelbon picks up some deserved credit for nailing down the win. On offense, not much to speak of beyond the homer. Loretta reaches base three times, but gets hurt badly by failing to bring home Crisp from third to tie in the sixth. And Big Papi was uncharacteristically little tonight, with two GIDPs, a stranded runner at third in the sixth, and a failure to even move Loretta (who represented an insurance run) up a base in the eighth.

Here are the season totals:

Hitting                Pitching

Trot Nixon      .411   Josh Beckett      .596
Jason Varitek   .365   Curt Schilling    .476
Coco Crisp      .098   Jonathan Papelbon .393
Wily Mo Pena    .000   Mike Timlin       .322
Josh Bard      -.034   Keith Foulke      .000
David Ortiz    -.040   Rudy Seanez       .000
Mark Loretta   -.076   David Riske      -.003
Kevin Youkilis -.077   Lenny DiNardo    -.038
Alex Cora      -.105   Tim Wakefield    -.648
Alex Gonzalez  -.139
Mike Lowell    -.155
J.T. Snow      -.160
Manny Ramirez  -.185

[Monster Dick Radatz]

BosoxBob, on Apr 6 2006, 11:20 AM, said:

No problem. I was debating which way to go, and also waiting to see what level of interest there would be in this. Here's last night's entry (which I'll now replace up top with the entries from the first two games:
<{POST_SNAPBACK}>

Kudos for putting this out for us, Bob. Fascinating stuff with a definite audience. Thanks!

My only comment is that 162 games will turn this into a very long post very quick. We'll need a good way to archive the individual games, but still keep it manageable. How about pinning two posts at the top? The first can track the last three games with a current total and the second could keep a game-by-game log. Maybe they both should have current totals... What do you guys think? Any interest in weekly or monthly totals?

[BosoxBob]

Game 4: April 7, 2006
Red Sox 14 Orioles 8

Hitting                Pitching

Manny Ramirez   .224   Matt Clement      .132
Jason Varitek   .173   Keith Foulke      .000
Mike Lowell     .149   Rudy Seanez       .000
David Ortiz     .126
Mark Loretta    .110
Coco Crisp      .084
Kevin Youkilis  .055
Alex Cora       .003
J.T. Snow       .000
Adam Stern      .000
Alex Gonzalez  -.010
Trot Nixon     -.046

Pretty much everybody gets credit for this victory, although O's starter Daniel Cabrera's name should somehow appear on this list (it's amazing what Leo Mazzone has done with this pitching staff). About the only things worth mentioning are Rudy Seanez, who escaped blame for coughing up four runs (due to the 10 run lead), and Trot Nixon, who finished with the lowest score despite hitting a two-run homer (that'll teach you to strike out with the bases loaded in a one-run game).

Here are the season totals:

Hitting                Pitching

Jason Varitek   .538   Josh Beckett      .596
Trot Nixon      .365   Curt Schilling    .476
Coco Crisp      .182   Jonathan Papelbon .393
David Ortiz     .087   Mike Timlin       .322
Manny Ramirez   .040   Matt Clement      .132
Mark Loretta    .034   Keith Foulke      .000
Wily Mo Pena    .000   Rudy Seanez       .000
Adam Stern      .000   David Riske      -.003
Mike Lowell    -.007   Lenny DiNardo    -.038
Kevin Youkilis -.022   Tim Wakefield    -.648
Josh Bard      -.034
Alex Cora      -.102
Alex Gonzalez  -.149
J.T. Snow      -.160

[finnVT]

BosoxBob, on Apr 8 2006, 11:43 AM, said:

Game 4: April 7, 2006
Red Sox 14 Orioles 8

Hitting                Pitching

Manny Ramirez   .224   Matt Clement      .132
Jason Varitek   .173   Keith Foulke      .000
Mike Lowell     .149   Rudy Seanez       .000
David Ortiz     .126
Mark Loretta    .110
Coco Crisp      .084
Kevin Youkilis  .055
Alex Cora       .003
J.T. Snow       .000
Adam Stern      .000
Alex Gonzalez  -.010
Trot Nixon     -.046

<{POST_SNAPBACK}>

Interesting, but one question. At the start of the game, win probability should be 0.5, right? And at the end, it should be either 0 or 1, depending on whether they lost or won. So, shouldn't the sum of all events be -0.5 for a loss or +0.5 for a win? That the numbers above don't add up to 0.5 makes me a little concerned, but maybe I'm just misunderstanding something.

[OttoC]

finnVT, on Apr 8 2006, 12:08 PM, said:

Interesting, but one question.  At the start of the game, win probability should be 0.5, right?  And at the end, it should be either 0 or 1, depending on whether they lost or won.  So, shouldn't the sum of all events be -0.5 for a loss or +0.5 for a win?  That the numbers above don't add up to 0.5 makes me a little concerned, but maybe I'm just misunderstanding something.
<{POST_SNAPBACK}>

Is the win probability 0.5 at the g=beginning of the game? Has there not been historically a home-field advantage of 53 to 54 percent?

[Timmeh49]

finnVT, on Apr 8 2006, 01:08 PM, said:

Interesting, but one question.  At the start of the game, win probability should be 0.5, right?  And at the end, it should be either 0 or 1, depending on whether they lost or won.  So, shouldn't the sum of all events be -0.5 for a loss or +0.5 for a win?  That the numbers above don't add up to 0.5 makes me a little concerned, but maybe I'm just misunderstanding something.<{POST_SNAPBACK}>

I think the answer is in BB's initial post:

BosoxBob, on Apr 4 2006, 05:14 PM, said:

In any case, what I'll be doing is tracking the game situation and outcome for each Red Sox players' at bats and computing the WPA. I'll also do the same for pitching (avoiding the pitching vs. fielding issue for the time being by awarding all credit/blame to the pitchers). Then for each game, I'll normalize the sum of all WPA scores to +1 (for wins) or -1 (for losses).<{POST_SNAPBACK}>

So my guess is that you are right... the overall change should be +0.5 for a win and -0.5 for a loss (notwithstanding OttoC's point), but BB multiplies the number by two to get a scale of -1 to +1.

[Timmeh49]

BB -- Great work and interesting metric.

A couple of questions...

Have you or anyone else used this metric to assess "clutch pitching ability" (a.k.a. the ability to "pitch to the score", and a.k.a. the Jack Morris Power)?

How about using it to measure a team's ability to fail to take advantage of its opportunities? For example, if the Sox load the bases with no one out in the bottom of the ninth in a game they trail by one run, but end up not scoring, it seems like WPA could be used to measure events like this over the course of the season.

[Eric Van]

Fabulous, but why should we wimp out? Let's include fielding.

The simplest way would be to only look at errors (by calculating WP with and without, charging the fielder and crediting the pitcher). But I think it's time for SoSH to implement a crude subjective version of UZR methodology in the game threads. I'd bet a bunch of eyeballs would be a vast improvement over ignoring fielding, probably more than halfway to being as accurate as a statistically based system.

Let's start with good plays. You award 1, 2, 3, or 4 stars, which approximately mean:

1: that play's only made 75% of the time
2: that play's only made half the time
3: that play's only made 25% of the time
4: nobody makes that play

Obviously, the actual brackets are:

0: plays made >87.5% of the time
1: plays made 62.5 - 87.5% of the time
2. plays made 37.5 - 62.5%
3. plays made 12.5 - 37.5%
4. plays made < 12.5%

For plays not made, we would have 1 to 4 daggers:

1: that play's made 25% of the time
2: that play's made half the time
3: that play's made 75% of the time
4: everybody makes that play

which is to say:

0. plays made < 12.5%
1. plays made 12.5 - 37.5%
2. plays made 37.5 - 62.5%
3: plays made 62.5 - 87.5% of the time
4: plays made >87.5% of the time

However, it would probably be easier to use the same scale for plays made and unmade rather than reversing them. Every play that isn't routine is graded on a Degree of Diffiiculty Scale, which is the 4 star scale above.

If this is feasible, we could do the first 4 games from scoresheets (even if I'm the only one who notes good and bad plays).

BTW, this scoring system is so straightforward that I'm ashamed to admit I just made it up. I really wish I'd thought of it years ago (although my "good play" scoring system is essentially a little ! for plays made 50-87% of the time, a big ! for plays made 16-50% of the time, a big and little ! for plays made <16% of the time and two big !! for historically great plays).

And if this is rejected as unfeasible -- and I think it may well be too much of a challenge -- then let it serve as the argument for using errors as the crudest, binary implementation of this methodology. IOW, the usual saber argument that we should ignore errors because we're not also looking at good plays, or at fielding range, is plain wrong. Throwing out a small bit of good data because you don't have more complete data is just perverse.

Finally, if we do decide to simply use errors, then we should at least do what MLB should have maybe be doing all along -- also score defensive gems. Just as an error is a routine play not made, a gem is a routine hit turned into an out, a ball that in the opinion of the scorer is a hit 75% (80%? 90%? What do people think the standard for errors is, in practice?) or more of the time.

I believe we haven't had an obvious defensive gem yet this year. The one candidate is Crisp's catch in deep CF off of Nix, but that may be as as low as a 50-50 ball.

[mr guido]

Timmeh49, on Apr 8 2006, 01:42 PM, said:

Have you or anyone else used this metric to assess "clutch pitching ability" (a.k.a. the ability to "pitch to the score", and a.k.a. the Jack Morris Power)? 
<{POST_SNAPBACK}>

BP does something along this line, tracking the win probability added for relievers, while also accounting for quality of opponent. They also track the leverage of the situation each reliever is used in.

Quote

WX

Expected wins added over an average pitcher. WX uses win expectancy calculations to assess how relievers have changed the outcome of games. Win expectancy looks at the inning, score, and runners on base when the reliever entered the game, and determines the probability of the team winning the game from that point with an average pitcher. Then it looks at how the reliever actually did, and how that changes the probability of winning. The difference between how the reliever improved the chances of winning and how an average pitcher would is his WX.

http://www.baseballprospectus.com/statisti..._winexp2005.php

[Pumpsie]

Mike Timlin .322
Matt Clement .132

Well, I have a problem with this metric already. If Mike Timlin can get a mark three times what Matt Clement gets for pitching one quite lousy and lucky inning while Clement shut down the O's for six innings and then got left in by his manager too long, and was unlucky to boot, in the 7th, also in a win, then I'm not sure we're getting it all in here.

Still doesn't factor out "luck" does it? In fact, this metric may exacerbate the luck element. And, btw, luck won't "balance out" over just one season. It takes many seasons to do that.

[Cumberland Blues]

You're completely ignoring the context though, Pumpsie. When a team scores 14 runs (and especially when most of 'em come early) - a bunch of offensive players get a share of the 100% win probability. In a 2-1 game like Timlin pitched in, the pitchers, particularly the pitchers working the crucial late innings, are going to get more of the credit because the offense didn't help much. Clement's share likely would've been pretty similar with or without the 4 runs in the 7th, cuz the offense did the heavy lifting opening up the big lead early.

[Timmeh49]

mr guido, on Apr 8 2006, 02:22 PM, said:

Timmeh49, on Apr 8 2006, 01:42 PM, said:

Have you or anyone else used this metric to assess "clutch pitching ability" (a.k.a. the ability to "pitch to the score", and a.k.a. the Jack Morris Power)?<{POST_SNAPBACK}>

BP does something along this line, tracking the win probability added for relievers, while also accounting for quality of opponent. They also track the leverage of the situation each reliever is used in.
http://www.baseballprospectus.com/statisti..._winexp2005.php
<{POST_SNAPBACK}>

Yeah... Forgot about this. But I was wondering about using such a measure on starting pitchers in particular. For a reliever, the leverage of the situation is dictated by when they are brought into the game (i.e. by the manager). But for a starter, the leverage unfolds as the game progresses. Actually, Pumpsie's question and Cumby's response illustrates my point. Because the Sox scored so many runs, Clement was pitching much of the game in a low leverage situation. And so any offensive success that was had against him would not have decreased the win probability very much at all. Clement could have given up more runs, which would hurt his ERA, but wouldn't have decreased the Sox chance of winning very much. This is the very definition of "pitching to the score". Maybe some simple ratio like WPA to runs allowed would be a start?

[Eric Van]

Pumpsie, on Apr 8 2006, 03:21 PM, said:

Still doesn't factor out "luck" does it?  In fact, this metric may exacerbate the luck element.  And, btw, luck won't "balance out" over just one season.  It takes many seasons to do that.

Absolutely. It's not a measure of projectable value. It's a measure of actual achieved value in the actual context.

To get the full picture, you want 3 numbers:

1) A theoretical value (in runs) calculated by a formula like Contextual Runs, BaseRuns, or Runs Created.

2) A value in runs calculated by changes in Run Expectancies (like I calculate and posted here in 2003 and 2004). Factors in base/out situational performance, but not game-score / inning situational performance. Since much of situational hitting difference is random but some is real, differences between this value and #1 value shouldn't be ignored but still need to be taken with a grain of salt.

3) The WPA values (converted back to runs for the sake of comparisons) which are the last word in actual (achieved) value, but are so subject to the influence of luck that they don't have much predictive value.

An MVP award should be based on WPA. For something like an All-Star team, you'd start with 1 and look at 2.

Now, BoSox Bob -- what's the exact run-per win vaalue in the data set the WPA values are drawn from?