When I finished the 2015 season preview prediction analysis in April, I promised to come back at the end of the season and find out just how wrong everyone was. For those of you who were really, really wrong, time to look away. Everyone else can laugh at how bad we still are at trying to predict baseball.

In order to evaluate how successful our previewers and projection systems were, each team’s final win total was subtracted from the projected total from each set. The average difference of all 30 predictions were then calculated, as well as the correlation of each set of predictions to the actual win totals. The results can be seen below.

TeamActualBttPEssayPECOTAFangraphsAverage Diff
Cardinals100-9-4-11-129
Pirates98-11-3-17-1311
Cubs97-15-13-12-1313.25
Royals95-17-11-22-1716.75
Blue Jays93-7-5-13-119
Dodgers92546-14
Mets90-6-6-7-97
Rangers88-8-6-9-149.25
Yankees87-11-4-8-57
Astros86-6-9-9-77.75
Angels8577625.5
Giants84200-31.25
Nationals8316991011
Twins83-15-5-12-910.25
Orioles8158-2-24.25
Indians8179-145.25
Rays80-31713
Diamondbacks79-1-7-5-44.25
Red Sox781171099.25
White Sox7637223.5
Mariners76154111210.5
Tigers74131291011
Padres74111310910.75
Marlins71157101010.5
Brewers681315131113
Athletics681517161515.75
Rockies6884375.5
Braves6765565.5
Reds641914121214.25
Phillies6313453.25
Average Diff9.3677.38.78.167
Correlation0.3470.6290.3770.441

In a stunning turn of events, the BP Annual essay writers/Effectively Wild podcast preview guests crushed not only the other human predictions (we’ll gloss over where those came from), but both the PECOTA and Fangraphs predictions, finishing with the lowest average difference of 7.3 wins. The EW set was also the only set to finish with a correlation stronger than 0.5. PECOTA wasn’t all that far ahead of the less successful human predictors, while Fangraphs came in a comfortable second. The composite projections I calculated in the original post would have been the second-best system by this methodology, and successfully predicted the Rays’ total after some significant disagreement between the four sets.

Also included are the averages per team, to give an indication of just how predictable each team was. Although this author was quite pleased with only missing the Giants’ total of 84 by two wins, the truth is that the Giants were clearly the most predictable team this season. Both PECOTA and EW guest Grant Brisbee nailed the win total precisely, and Fangraphs’ 3-win miss looked positively huge by comparison. No other prediction from any of the sets was spot on, but several only missed by one, including BttP alum and current BP star Matt Trueblood’s Phillies total of 64.

At the other end of the spectrum was, of course, the Kansas City Royals, and no system hates the Royals more than PECOTA, which whiffed on the AL Central champs’ total by an incredible 22 wins. If PECOTA hadn’t disliked KC and Pittsburgh so much, it certainly would have made a contest of this. Let that be a lesson to you, PECOTA. All four sets were also much too optimistic about the Athletics and Reds, with double-digit misses all around.

Readers of the original prediction post or EW Facebook group members may also remember that listeners were pitted against PECOTA in an over/under poll for each of the original 30 projected win totals from early February. Initially I just calculated whether the highest percentage of voters was correct on the over/under in each case, but that didn’t strike me as an entirely satisfactory way to determine a winner, especially considering the score came out at 15-15. The over/unders were therefore also evaluated using the following entirely arbitrary scoring system:

-PECOTA was automatically awarded a win for correctly predicting a win total.

-If PECOTA was within six wins (just under one standard deviation) of a team’s actual total and the majority of listeners failed to correctly pick the over/under, PECOTA took the win.

-If PECOTA was within six wins of a team’s actual total and the majority of listeners did correctly pick the over/under, it was scored as a tie

-If PECOTA missed by more than six wins and the majority of listeners correctly picked the over/under, the listeners won.

-If PECOTA missed by more than six wins and the majority of listeners was also wrong with the over/under, neither side received a point.

The results were as follows:

TeamPECOTA Feb 6thListeners OverListeners UnderListeners AgreeActualOver/Under Correct?Winner
Tigers8275.16%16.99%7.84%74NoNeither
Indians8146.41%37.91%15.69%81NoPECOTA
White Sox7856.86%29.41%13.73%76NoPECOTA
Royals7274.51%9.15%16.34%95YesListeners
Twins7025.49%54.25%20.26%83NoNeither
Red Sox8746.41%38.56%15.03%78NoNeither
Rays867.84%88.24%3.92%80YesListeners
Blue Jays8272.55%18.30%9.15%93YesListeners
Yankees8041.18%49.02%9.80%87YesNeither
Orioles7872.55%17.65%9.80%81YesTie
Angels9026.80%58.17%15.03%85YesTie
Mariners8736.60%43.14%20.26%76YesNeither
Athletics8441.83%35.29%22.88%68NoNeither
Rangers8228.10%51.63%20.26%88NoPECOTA
Astros7723.53%58.17%18.30%86NoNeither
Cardinals8932.68%43.14%24.18%100NoNeither
Cubs8252.94%35.95%11.11%97YesListeners
Pirates8079.08%12.42%8.50%98YesListeners
Brewers8027.45%50.98%21.57%68YesListeners
Reds7922.88%62.75%14.38%64YesListeners
Nationals9254.25%24.84%20.92%83NoNeither
Mets8229.41%51.63%18.95%90NoNeither
Marlins8160.13%30.07%9.80%71NoNeither
Braves7333.99%44.44%21.57%67YesPECOTA
Phillies7015.03%64.71%20.26%63YesListeners
Dodgers977.19%83.66%9.15%92YesPECOTA
Giants8448.37%32.03%19.61%84NoPECOTA
Padres8345.10%32.68%22.22%74NoNeither
Diamondbacks7423.53%56.86%19.61%79NoPECOTA
Rockies7228.76%48.37%22.88%68YesPECOTA

Of course, after my efforts to decide a winner, the final score works out at 8-8. with two ties and ten totals both sides were pretty wrong about. Frankly, PECOTA got away with a couple, because had just a few more listeners come to their senses and realised that the Rockies would be terrible, it would have lost a win. It was also six wins away from both the Rangers and Braves totals, which I generously awarded as wins to make PECOTA feel slightly less bad after receiving such a drubbing from the essay writers. I’m therefore calling this for the listeners; make that 2-0 to the humans.

Predicting win totals isn’t the easiest job in the world. Were our sets any better at predicting the overall standings? Below is the same table as our win total comparison, but this time comparing the predicted rank with the final rank of all 30 teams.

TeamActual RankBttPEssayPECOTAFangraphsAverage Diff
Cardinals130322
Pirates2611256
Cubs31510569
Royals4199221817
Blue Jays5531287
Dodgers6-4-5-5-44.5
Mets786586.75
Rangers81212111913.5
Yankees91781049.75
Astros10101613912
Angels11-8-7-8-67.25
Giants12-21-332.25
Twins131610151413.75
Nationals13-12-9-11-1211
Cleveland15-8-92-86.75
Orioles15-5-8445.25
Rays1784-11-26.25
Diamondbacks1859777
Red Sox19-13-8-14-1412.25
Mariners20-162-14-1712.25
White Sox202-3222.25
Padres22-8-13-13-1111.25
Tigers22-14-12-10-1312.25
Marlins24-14-1-10-98.5
Rockies2512301.5
Athletics25-9-14-16-1413.25
Brewers25-6-8-11-67.75
Braves280-1-110.75
Reds29-13-6-5-57.25
Phillies3000000
Average Diff8.6336.5678.6007.700
Correlation0.3060.3130.5670.403

The overall ranking of the four sets doesn’t change, but BttP’s predictions nearly overtake PECOTA. Baseball Prospectus Annual essay writers can congratulate themselves on comfortably defeating two projection systems, and now must carry the incredible weight of the baseball community’s expectation going into 2016.

This second table perhaps makes it even more obvious how impossible it was to predict the AL this year. The Nationals and Padres aside, the NL predictions had an average difference well into the single digits; our projections collectively missed by double-digits for more than half of the AL teams. There was not a single division winner correctly predicted by any set in the AL, and all four also missed on the Nationals. At least both humans and projection systems alike got the Cardinals and Dodgers right.

Finally, congratulations must go to the Phillies, who successfully proved all four projections right by finishing dead last. Sometimes, you can predict baseball.

 

Next post:
Previous post:

Trackbacks/Pingbacks

  1.  Analysing the 2016 Season Preview Predictions | Banished to the Pen
  2.  Evaluating the 2016 Season Preview Predictions | Banished to the Pen

Leave a Reply

Your email address will not be published. Required fields are marked *