Robo strikezone: Not as simple as you think -- Baseball Prospectus

charlieoscar

Member
Sep 28, 2014
1,339
This is a free article from Baseball Prospectus of a paper submitted to SABR's Analytics Research Conference. It discusses some of the problems that could be encountered in trying to develop an automated ball/strike calling system.

https://www.baseballprospectus.com/news/article/37347/robo-strike-zone-not-simple-think/

One thing that is not spelled out is the definition of a non-swinging strike, which is any portion of the ball passing through any portion of the strike zone. A baseball must have a circumference measuring from 9.0 inches to 9.25 inches. If you work out the diameter of a ball from those numbers, you find it to be from 2.864789... to 2.944366...", which is a latitude of almost 1/12 of an inch. A question I have never seen answered is whether allowance is made in the circumference for the seams.

A question or two comes to mind but I'm not a subscriber to BP, so I cannot ask there. However, if the plate was slightly skewed towards left field (a few hundredths of an inch), and there was a right-handed pitcher facinf a right-handed batter. If a pitch was thrown to the outside of the plate, just missing the front left corner and breaking away from the batter, would/could the robo-ump pick that up?

Are the plate and batter's boxes perfectly flat in both the x- and y-axes? What if the back of the plate were slightly higher than front of the plate? A higher pitch would not have to drop quite as far to catch the top of the zone than if the back of the plate was slightly lower than the front. The rear left corner of the plate could be lower than the front right corner, which also could cause an error.

These may be nit-picking questions but if you want perfection....
 
Last edited:

InstaFace

MDLzera
SoSH Member
Sep 27, 2016
8,626
yeah, if it's 98% accurate to within 1/8", the umps' union might as well fold up shop. These worries are angels-dancing-on-the-head-of-a-pin in terms of impact on the overall experience.
 

edoug

Member
SoSH Member
Jul 15, 2005
2,170
What about foul tips? Wild pitches or passed balls with a man on 3rd? Wild pitches or passed balls with men on 2nd and 3rd? HBP that graze the batters uniform?
 

jercra

Member
SoSH Member
Jul 31, 2006
1,906
Arvada, Co
What about foul tips? Wild pitches or passed balls with a man on 3rd? Wild pitches or passed balls with men on 2nd and 3rd? HBP that graze the batters uniform?
I don't think the proposal needs to be that the home plate umpire is eliminated, just that they don't call balls and strikes anymore. The first base ump still exists and he doesn't call balls and strikes.
 

Beomoose

Member
SoSH Member
May 28, 2006
17,058
Wherabouts Unknown
How about a compromise: the Ump wears an AR headset that projects the zone into the world so he can clearly see the ball in relation to it, but he's still the one that makes the call.
 

charlieoscar

Member
Sep 28, 2014
1,339
Overall, batters hit .248 with an on-base percentage of .318 during the 2018 regular season. Who made more mistakes...the batters of the home plate umpires? I've been watching baseball for a long, long time and missed calls by umpires (and mistakes by everyone from the owners on down) have been part of the game. What I want to see is getting the game time back down to about two-and-a-half hours. I posted a link to that article to bring it to the attention of those people who think that robo-calling balls and strikes is a piece of cake to implement.
 

Joe Sixpack

Well-Known Member
Lifetime Member
SoSH Member
Sep 9, 2002
4,049
Mansfield, MA
Pace of play is an entirely separate issue being discussed elsewhere on this site, and I think an important one, but not to be conflated with the missed calls issue.

This article is highly flawed from my perspective. For example:

"It is widely known that both Doppler-based and camera-based machines are not accurately reporting 100 percent of their measured data 100 percent of the time. Even if they were accurate 99.9 percent of the time, what if that 0.1 percent is a strike but is called a ball on a 3-2 count in the bottom of the ninth inning with bases loaded and two outs in a tie game?"

This argument makes no sense, because I think we all know that no human umpire is 99.9% accurate, so why would a robo ump be held to that standard? Not to mention what the example used brings to mind since we saw a perfect game ruined by a human umpire in real life. (I know, not a ball/strike call)

Worrying about whether the seams of the baseball will be picked up is silly because right now the entire baseball can be over the plate and the human ump may call it a ball.
 

Lose Remerswaal

Leaves after the 8th inning
Lifetime Member
SoSH Member
Overall, batters hit .248 with an on-base percentage of .318 during the 2018 regular season. Who made more mistakes...the batters of the home plate umpires? I've been watching baseball for a long, long time and missed calls by umpires (and mistakes by everyone from the owners on down) have been part of the game. What I want to see is getting the game time back down to about two-and-a-half hours. I posted a link to that article to bring it to the attention of those people who think that robo-calling balls and strikes is a piece of cake to implement.
Welcome to SoSH Mr. Cafardo!

Doing it quickly is better than doing it right? I hope I never meet your surgeon!

Of course if batters hit 1.000 or had an OBP of 1.000 you could turn your argument around to say the pitchers are making too many mistakes . . . .
 

Van Everyman

Member
SoSH Member
Apr 30, 2009
17,589
Newton
My assumption all along has been that umpires will wear a Google Glass-type of eyewear that allows them to see the ball go over the plate Gameday-style.

Really, it’s not that hard.
 

Max Power

thai good. you like shirt?
SoSH Member
Jul 20, 2005
3,579
Boston, MA
Welcome to SoSH Mr. Cafardo!

Doing it quickly is better than doing it right? I hope I never meet your surgeon!

Of course if batters hit 1.000 or had an OBP of 1.000 you could turn your argument around to say the pitchers are making too many mistakes . . . .
Imagine how long the games would be if that were the case.
 

DrewDawg

Dorito Dink
SoSH Member
Dec 16, 2010
36,116
People thought robot umps would be a piece of cake to implement?
 
Last edited:

TomTerrific

Member
SoSH Member
Jul 15, 2005
1,384
Wayland, MA
The article isn't bad for the most part (see below), but it kind of misses the forest for the trees.

(BTW, one of the bad parts of the article is the machine calibration part--calibrating to the tolerances required is not challenging at all. And the discussion of RF issues is uninformed, to put it mildly.)

What a baseball umpire calling balls and strikes needs to have, more than anything else, is consistency within a game, and consistency from game to game, and park to park. This is very much easier to achieve with automated equipment than humans. Even if the strike zone is different than the average one called by umpires, given the tolerances already achievable by machines, I'll bet the in-game variance for them is way smaller than the variance from ump to ump. And as I said above, the game-to-game and park-to-park variance just isn't going to be anything nearly as big as the in-game variance if any reasonable effort is put into it.

The thing that drives people nuts aren't any of the things focused on in the article. What bugs everyone are lack of consistency and glaring errors--and humans just can't do nearly as good a job at avoiding those two things. As for any residual errors--if batters can adjust to the vagaries (i.e., biases) of human umpires, they can adjust to the ones caused by machines. Which will be far more constant and predictable than the human ones. And BTW, if the K-zone shown on the screen is drawn from the same sensing that calls the balls and strikes, it will look right to everyone and stop pissing people off. Which people, BTW, who have a really, really poor idea based on just their eyeballs as to whether the pitch really was a ball or strike.

TECHNICAL RANT I CAN'T RESIST MAKING:

From a physics point of view, humans are kind of poorly designed relative to machines. We depend almost exclusively on two angles that our eyes measure, and a not-very-well-stablized platform (our heads) to track an object that is decreasing in range by almost 98%. Anyone who's ever worked this kind of problem knows you're going to be heavily dependent on estimating the entire trajectory in order to try and get that last bit (where the ball crosses the strike zone) right, and humans definitely do that. Given the forces at play (e.g., baseball's response to the spin vector, air motion-temp-pressure-humidity) this is a really variable process, too.

In contrast, these other systems are range-Doppler multilateration systems (meaning they construct a trajectory based on a series of range-Doppler measurements from a set of fixed locations). There are very few situations where we cannot reconstruct trajectories far, far better using range-Doppler multilateration than we can using angle information along, even multi-angular measurements (aka triangulation). The basic reason is that, for most situations, and relative to the required accuracy, we can measure range far, far better than we can angles.
 

Marbleheader

Dope
Dope
Sep 27, 2004
10,231
How about a compromise: the Ump wears an AR headset that projects the zone into the world so he can clearly see the ball in relation to it, but he's still the one that makes the call.
All he's really need is an earpiece. The ump gets a ding for a strike and a buzzer for a ball but still relays the call on the field.
 

charlieoscar

Member
Sep 28, 2014
1,339
Welcome to SoSH Mr. Cafardo!

Doing it quickly is better than doing it right? I hope I never meet your surgeon!

Of course if batters hit 1.000 or had an OBP of 1.000 you could turn your argument around to say the pitchers are making too many mistakes . . . .
Why does a batter take a pitch? Sometimes it's because it wasn't the pitch he was looking for, sometimes because he was fooled, sometimes because he thought it would be called a ball. In this latter case, he is umpiring. So, who makes more mistakes umpiring? The batter or the umpire? I'd say the batter and if you had a perfect ball/strike calling robo-umpire you would still see batters misjudge close pitches. For one reason, batters are watching the ball and especially with pitches that are up a bit, they won't be seeing the edge of the strike zone.

Oh, the umpire missed a call. Boo! But does anyone boo the batter? You have these fans watching a depiction of the strike zone and path of the ball in a little display on the corner of their television screen, a display that is not always accurate, calling for the head of the umpire. Or better, yet, someone down past the foul pole in right field screaming at the umpire for missing a ball or strike call.

I think baseball has bigger things to worry about, and doing things quicker is one of them.
 

geoduck no quahog

not particularly consistent
Lifetime Member
SoSH Member
Nov 8, 2002
11,381
Seattle, WA
What on earth is relevant about "batters making more mistakes than umpires"? Seriously...what's your point?

That aside, the killer paragraph for me is:

With human umpires the zone expands and contracts depending on the count and whether the batter or pitcher has the advantage. Knowing that, pitchers who fall behind know that an expanding zone can help them battle back. When the zone doesn’t change size based on count, pitchers will be incentivized to try to get ahead early so that hitters have to cover areas they haven’t traditionally had to defend when behind in the count (e.g., the top corners of the zone).
There is nothing fair or sporting about that sentence. Is that the way players should be trained? The zone changes with the count? Robo-Umps aside, it's ridiculous that an umpire should change the zone within a single plate appearance.

I'll repeat what an NCAA umpire once told me (take it for what it's worth - I'd be interested in a more direct interpretation): Umpires have a hard time judging pitch height, particularly in the 2-dimensions that include plate depth. He says he was taught to call a pitch caught above the catcher's knees and below the catcher's eyes a strike. He said that college umps give catchers a lot of shit if they try to repeatedly game that system. Anyone have any insight?

Everything about that article led me to believe automatic strike zone calls are more fair than human perception.
 

Bergs

Member
SoSH Member
Jul 22, 2005
13,086
Why does a batter take a pitch? Sometimes it's because it wasn't the pitch he was looking for, sometimes because he was fooled, sometimes because he thought it would be called a ball. In this latter case, he is umpiring. So, who makes more mistakes umpiring? The batter or the umpire? I'd say the batter and if you had a perfect ball/strike calling robo-umpire you would still see batters misjudge close pitches. For one reason, batters are watching the ball and especially with pitches that are up a bit, they won't be seeing the edge of the strike zone.

Oh, the umpire missed a call. Boo! But does anyone boo the batter? You have these fans watching a depiction of the strike zone and path of the ball in a little display on the corner of their television screen, a display that is not always accurate, calling for the head of the umpire. Or better, yet, someone down past the foul pole in right field screaming at the umpire for missing a ball or strike call.

I think baseball has bigger things to worry about, and doing things quicker is one of them.
This is an argument against the very concept of logic.
 

Plympton91

bubble burster
SoSH Member
Oct 19, 2008
11,864
Oh, the umpire missed a call. Boo! But does anyone boo the batter? You have these fans watching a depiction of the strike zone and path of the ball in a little display on the corner of their television screen, a display that is not always accurate, calling for the head of the umpire. Or better, yet, someone down past the foul pole in right field screaming at the umpire for missing a ball or strike call.

.
It’s been about 20 years now that we’ve had games with that display screen for balls and strikes at least intermittently. In all the games I’ve watched, there was 1 time - one - where I said, “boy, the computer clearly missed that call, I’m glad the home plate umpire was there.” There is at least 1 time per game where the umpire calls a pitch completely and obviously wrong.

The argument against computerized strike zones is, “Old man yells at cloud.” That’s it.
 

joe dokes

Member
SoSH Member
Jul 18, 2005
13,009
Why does a batter take a pitch? Sometimes it's because it wasn't the pitch he was looking for, sometimes because he was fooled, sometimes because he thought it would be called a ball. In this latter case, he is umpiring. So, who makes more mistakes umpiring? The batter or the umpire? I'd say the batter and if you had a perfect ball/strike calling robo-umpire you would still see batters misjudge close pitches. For one reason, batters are watching the ball and especially with pitches that are up a bit, they won't be seeing the edge of the strike zone.

Oh, the umpire missed a call. Boo! But does anyone boo the batter? You have these fans watching a depiction of the strike zone and path of the ball in a little display on the corner of their television screen, a display that is not always accurate, calling for the head of the umpire. Or better, yet, someone down past the foul pole in right field screaming at the umpire for missing a ball or strike call.

I think baseball has bigger things to worry about, and doing things quicker is one of them.
I dont know how hard it would be to implement, but I'm not sure your points go to that issue.

Who makes more mistakes delivering babies? Obstetricians or orthodontists?

Does anyone boo the batter? In my experience, that happens occasionally.

How does this do things not quicker?

This is an argument against the very concept of logic.
#17 Bergs, 27 minutes ago
If true, it is even an argument?
(see "The Argument Sketch," Monty Python.)
 

Dewey'sCannon

lurker
Jul 18, 2005
598
Maryland
Look at these shoes, I've only worn them a week and already the soles are worn out. Why bother to complain, no one even listens. . . .

FWIW, I always though the first step should be rob-assisted umpiring - rigging their ball/strike counters so that they would vibrate if the ball is within the automated strike zone. Ump can rely on this, over override if they really think the robot-call is wrong. And that data would be available for review, at least by MLB.
 

Plympton91

bubble burster
SoSH Member
Oct 19, 2008
11,864
It just seems like a vibration signal, or a beep, or a heads up display would be so distracting that it would make the umpires moot anyway. I can’t imagine that something that breaks your concentration as the pitch crosses home plate is conducive to making a better call. And, if they had the buzzer or the vibration on a one-second delay, it’s just going to train umps to not make a borderline call and rely on the computer anyway.

There is no reason to not make this change other than nastalgia for a less sophisticated time.
 

geoduck no quahog

not particularly consistent
Lifetime Member
SoSH Member
Nov 8, 2002
11,381
Seattle, WA
...and they'd still be responsible for check swings (along with another umpire), plays at the plate, HBP, 3rd strikes in the dirt, foul balls up to the bag, calling time and generally running the game...a little more than they do the other 3 games they officiate - and they could now do better.
 

edoug

Member
SoSH Member
Jul 15, 2005
2,170
...and they'd still be responsible for check swings (along with another umpire), plays at the plate, HBP, 3rd strikes in the dirt, foul balls up to the bag, calling time and generally running the game...a little more than they do the other 3 games they officiate - and they could now do better.
Yeah, there absolutely has to be someone behind home plate.
 

InstaFace

MDLzera
SoSH Member
Sep 27, 2016
8,626
It’s been about 20 years now that we’ve had games with that display screen for balls and strikes at least intermittently. In all the games I’ve watched, there was 1 time - one - where I said, “boy, the computer clearly missed that call, I’m glad the home plate umpire was there.” There is at least 1 time per game where the umpire calls a pitch completely and obviously wrong.

The argument against computerized strike zones is, “Old man yells at cloud.” That’s it.
Since it was clearly so memorable, though, I'm now kinda curious what that one instance was.
 

uncannymanny

Well-Known Member
Gold Supporter
SoSH Member
Jan 12, 2007
6,193
Boston --> NYC --> LA --> NYC
Why does a batter take a pitch? Sometimes it's because it wasn't the pitch he was looking for, sometimes because he was fooled, sometimes because he thought it would be called a ball. In this latter case, he is umpiring. So, who makes more mistakes umpiring? The batter or the umpire? I'd say the batter and if you had a perfect ball/strike calling robo-umpire you would still see batters misjudge close pitches. For one reason, batters are watching the ball and especially with pitches that are up a bit, they won't be seeing the edge of the strike zone.

Oh, the umpire missed a call. Boo! But does anyone boo the batter?
The batter is also trying to decide whether or not to swing and to get his body parts into position to do so in a split second. The umpire has the luxury of having one job as the pitch is delivered. This is a really odd argument.
 

Plympton91

bubble burster
SoSH Member
Oct 19, 2008
11,864
Since it was clearly so memorable, though, I'm now kinda curious what that one instance was.
It was the Ortiz at bat against Miller early in 2016 season. Of course, the umpire missed that call too, so it didn’t really matter.
 
Last edited:

shaggydog2000

Member
SoSH Member
Apr 5, 2007
5,958
Trackman/Statcast, which I assume would be the robo-ump of choice given that it is in current use at ball-parks, is off vertically and horizontally by up to 3 inches:

https://tht.fangraphs.com/the-lurking-error-in-statcast-pitch-data/

Now, those three inch errors are pretty rare. There are most likely errors that are even worse, but happen even more rarely. But we've all seen Umpires make even worse mistakes. I don't know of data on umpire accuracy, but I am sure the MLB has it somewhere. I'd guess it's somewhat equal, but not fairly distributed error. Combining an Ump with an ability to over-rule the system with the trackman buzzer would be fine by me. The Ump could catch the obvious errors, and the robo-ump could be used to train the Ump to be more accurate and consistent. I know they do post-game training involving video and the Umpire calls, but a buzzer in your ear is a good reminder of how you're supposed to be calling things. It's going to come eventually, let's figure out how to do it in the most fair and accurate manner.
 

charlieoscar

Member
Sep 28, 2014
1,339
The argument against computerized strike zones is, “Old man yells at cloud.” That’s it.
Captain Benjamin Sisko said "absolutely not" when Odo suggested using a holographic umpire in the baseball match between the Vulcans and and the team from Deep Space Nine.
 

santadevil

Member
SoSH Member
Aug 1, 2006
3,890
Saskatchestan
Good umpire breakdown of numbers article from ESPN

http://www.espn.com/mlb/story/_/id/26467659/study-young-umps-top-older-peers-calls

A new study reports that younger umpires are better at calling balls and strikes than older umpires and that umpires selected for recent World Series games have not necessarily been the best performers.

Umpires showed a persistent two-strike bias. With two strikes, umpires were twice as likely to call a true ball a strike (29 percent of the time) than with a lower count (15 percent). The error rates have declined since 2008, when it was 35.2 percent, but over the 11 seasons, nearly one-third of the batters who struck out looking were called out on miscalls.
 

InstaFace

MDLzera
SoSH Member
Sep 27, 2016
8,626
Robot Umpire debuts in Atlantic League ASG.

Umpires can overrule the computer.

"Until we can trust this system 100 percent, I still have to go back there with the intention of getting a pitch correct because if the system fails, it doesn't pick a pitch up, or if it registers a pitch that's a foot-and-a-half off the plate as a strike, I have to be prepared to correct that," deBrauwere said before the game.

The umpires have the ability to override the computer, which considers a pitch a strike when the ball bounces and then crosses the zone. TrackMan also does not evaluate check swings.
Some interesting quotes from Rob Manfred at the end there.
 

mauf

Anderson Cooper x Mr. Rogers
Staff member
Dope
Gold Supporter
As with automobiles, the talk is all about automating the job and eliminating the human element entirely, but in the near term some combination of human and machine is likely to yield the best results. To some degree, this is already happening with balls and strikes — no one who follows baseball closely would dispute that ball-strike calls have improved greatly in the past 20 years, and technology has been a big part of that. Giving umpires the benefit of real-time technological input is the logical next step.
 

soxhop411

Member
SoSH Member
Dec 4, 2009
34,172
The first pitch from the All-Star Game in the Atlantic League was a called strike. https://mobile.twitter.com/RobRoseSports/status/1149099581667446784
But did they get it right?
Close call sports has a bunch of highlights from the debut of the robo ump in the Atlantic league.


Cc @budcrew08
 
Last edited:

Mueller's Twin Grannies

Member
SoSH Member
Dec 19, 2009
4,849
This is basically going to eliminate the need for pitch framing as a skill for catchers, which is kind of sad. Stealing a strike is a great skill that has made the difference for a lot of pitchers over the years and catchers too.

I know some that can throw strikes...

I don't think that's legal since it never touches the lane. It's also terrifying while simultaneously reminding me of Walter throw the ball into the Nihilist (Flea)'s stomach in TBL.
 

InstaFace

MDLzera
SoSH Member
Sep 27, 2016
8,626
Ok, mark it zero. As you were.

Anyone have opinions (yours or someone else's) as to how Trackman did against their own assessment of pitch calling? The video up there had some called strikes that would surely have been low balls to an MLB ump. That's surprising, I always figured the most squeezing happened on high pitches, where umps have collectively decided to ignore the rulebook. Left/right looked good to me.

Wonder where else they'll use it next.
 

CFB_Rules

Member
SoSH Member
Nov 29, 2016
181
I think having robots call balls and strikes is certainly doable and desirable, but there are a few things that would have to be demonstrated for it to work well.

1) I'm less concerned with the error bars of the machine, because humans have those too. However, it needs to be shown that machine errors are essentially randomly distributed. If the machine is more likely to make mistakes with one pitcher because it doesn't like the spin on his pitches or something, that's not acceptable.

2) The machine needs to be able to make decisions quickly. Less than a second after the ball is caught the umpire should know the result from the computer.

The umpire also needs to have the final say to potentially overrule the machine. Overrules should be rare and documented (such as a ball bouncing in the dirt and then catching part of the zone).

One thing that I haven't heard a lot of people talk about...as machines become more proficient at calling strikes correctly according to rule, pitching and hitting will change as both become accustomed to the "new" zone. Pitches up and away have historically almost always been called balls since they are practically unhittable, and breaking pitches just off the plate have been called strikes. Will players adjust the way they've played to the newly called strike zone, or will they adjust the machine parameters to match what players expect?
 

InstaFace

MDLzera
SoSH Member
Sep 27, 2016
8,626
how would you conclusively determine whether a given machine pitch call is an error? frame-by-frame review of the video?

I believe your #2 is already the case, it just took the Atlantic League ump more time to communicate than players were used to. And your #3 is the case in the Atlantic League's implementation - see my quote in post #34.
 

CFB_Rules

Member
SoSH Member
Nov 29, 2016
181
how would you conclusively determine whether a given machine pitch call is an error? frame-by-frame review of the video?
The article cites this as their source for PitchFX accuracy: https://www.baseballprospectus.com/news/article/13109/spinning-yarn-how-accurate-is-pitchtrax/

Some of the data in the article would cause concern. For example, it is obviously unacceptable that Milwaukee's PitchFX system was shifted 2 inches to the 3rd base side for 4 straight years. Hopefully that is something that routine calibrations, done by MLB and not the home club, would fix.
 

geoduck no quahog

not particularly consistent
Lifetime Member
SoSH Member
Nov 8, 2002
11,381
Seattle, WA
Don't know if they're accurate or not, but the 3D strike zone graphics used by ESPN are spectacular. I wonder if teams use that technology to help train/tune-up their pitchers instead of coaches or catchers calling out location and, if they don't - why not?

It seems incredibly useful for a pitcher to see that he missed his target, say, by a half a baseball width inside, but still high enough to be called a strike...or if he caught the front end of the strike zone on a low curve.
 

InstaFace

MDLzera
SoSH Member
Sep 27, 2016
8,626
Great article.

Also love that tidbit: the worst park for calibration and systematic error? Florida, of course.
 

HowBoutDemSox

Member
SoSH Member
Aug 12, 2009
4,001
Interesting piece of umpire accuracy and the elusive “perfect game” of umpiring:
Since the start of 2008, the first full season when pitch-tracking systems were installed in every stadium, MLB players have recorded 51 cycles, 48 triple plays, and 38 no-hitters (not counting combined no-hitters). All those infrequent events have occurred more often than the umpire perfect game. Over the same span, home-plate umpires have made every pitch call correctly on one team only 24 times in games that lasted at least nine innings, with West’s the most recent. That’s roughly once per 1,200 games, or a little more than twice per season, on average. It’s not quite as uncommon as the pitcher perfect game—we’ve seen only six of those since ’08—but it’s rarer than most of the events considered deserving of push notifications.
No umpire on record has achieved perfection when forced to make more than 71 calls on a team. Naturally, no umpire has been perfect for both teams in a game; Scott Barry came closest on July 22, 2017, when he made 138 of 142 calls correctly (97.2 percent). When West was perfect on Cubs calls, he nailed only 87.5 percent of his calls on Pirates pitches.