There is very little disagreement, among sabermetricians, as to how to figure out who the best hitter is. (“Very little” does not translate to none, however.) There is somewhat more disagreement over how to figure out who the best pitcher is, but one can broadly say there is more agreement than disagreement.
There is, however, considerably more controversy over how to measure who the best fielder is. Understandably, this controversy has made the general public more reticent to trust sabermetricians on the subject of fielding than they are on hitting and pitching. (And they are right to be skeptical, of course – the lack of consensus is indicative of a lack of convincing evidence that any approach is more correct than the others.)
It’s nearly enough to make you throw up your hands and give up on the subject. But defense is no less important to understand for being hard to measure. So let’s start by asking what are some common areas on how to measure defense that we can get everyone to agree on.
The fundamental insight into measuring defense was (as with so many things) conceived of by Bill James. James reasoned that if you wanted to tell how good a team was defensively, you’d want to find a measure that has a strong relationship with how many runs a team allows. The traditional measures of player fielding – things like errors and fielding percentage – do an absolutely awful job of relating with runs allowed. So James surmised that team errors are a bad measure of team fielding (and by extension, individual player errors are a bad measure of individual player fielding).
So what’s a good predictor of team runs allowed, that also measures things largely under the control of the defense? James came up with Defensive Efficiency Rating, which is the number of outs recorded per ball in play (in other words, plate appearances minus strikeouts, walks and home runs). James found that DER does a much better job than fielding percentage of predicting how many runs a team allows. One could argue that it’s not entirely the defense that’s responsible for DER – the pitcher is involved in every one of those plays. But for a team in a whole season, there are at least five starters (and a team that uses only five starters is about as rare as a triple crown winner) and a whole host of relievers, so taken in the aggregate, we can say that DER is likely to be primarily a measure of what the defense, rather than the pitching, has done. (There is also the question of home park, which is why Baseball Prospectus offers Park Adjusted Defensive Efficiency on its sortable reports.)
So that gives us a measure of team defense that is relatively uncontroversial – if nothing else, most sabermetricians will agree that it’s a reasonable measure. So what can DER tell us about player fielding metrics? There’s two key elements of DER – outs, which can also be called plays made, and balls in play. If you consider the various defensive measures out there, you’ll find that these are what those systems all share in common as well – a record of the player’s plays made, and how many balls in play he was on the field for. Where they differ is in their estimate of how many plays the average player at that position would have made, given those same balls in play.
And the difficulty is, you have to estimate. With every plate appearance, you know exactly which batter was at the plate. Baseball’s records are not and can not be as clear about which fielder was responsible for the ball. (And in fact, unlike hitters, more than one fielder can have a chance to field each ball in play.) So you have to try and figure out how to split credit for all those balls in play among the fielders. That’s the tricky part, and the part that leads to the majority of disagreements between metrics.
So if you want somewhere to start, start with the parts that are both uncontroversial and in common between all measures. For plays made (in other words, outs) you want to look almost entirely at putouts for outfielders, and primarily assists for infielders. Balls in play is a bit trickier to come by, but innings played will do in a pinch (just remember to account for a team’s strikeout rate). Will that tell you everything you want to know about defense?
Certainly not. But if a defensive measure disagrees significantly from this, it’s going to be because of how it comes to its estimate of expected plays made. And you may not be able to find out why a measure comes to that. But at least know you have a place to start asking questions.