With the regular season over, it’s the most exciting time of the year: we get to find out which predictions or projections from our preseason contestants were the most accurate. Apparently there are some playoffs going on as well, if you like that kind of thing. This is the fourth year running we’ve carried out this exercise, with PECOTA and FanGraphs performing most impressively from a win total perspective last season, while the Composite set most accurately predicted the rankings.

A quick recap of how this works for the new reader. For each of the sets of predictions and projections that featured in our preseason analysis, the mean absolute error (MAE) and root mean squared error (RMSE) has been calculated. MAE is the average difference between the predicted total and the actual, while RMSE is the square root of the average of the squares of all the differences. RMSE gives greater weight to large errors because they are squared, so if you think bigger misses should be punished more heavily, this is the more relevant number.

Before we get to the results, here’s a reminder of the predictions and projections featured and their abbreviations here:

PECOTA (PEC): Baseball Prospectus‘ projected win totals based on their in-house projection system.

FanGraphs (FG): The FanGraphs Depth Charts projected totals, which are a combination of the Steamer and ZiPS projection systems, with an additional playing time adjustment applied by FanGraphs staff.

Davenport (Dav): Totals based on Clay Davenport’s projection system, with Clay’s own playing time estimates.

Banished to the Pen writers (BttP): Predictions from each of our writers from our season preview series.

Effectively Wild guests (EW): Predictions from each of Effectively Wild‘s team preview podcast guests.

Composite (Comp): The average of the five projection/prediction sets above.

Adjusted PECOTA (APEC): A modified PECOTA prediction set, based on EW listener responses to a preseason over/under poll using the first PECOTA release, as discussed in the preseason edition of this series. Each PECOTA total is modified by up to ten wins, based on the net total percentage of listeners taking the over/under; for example, if 70% of listeners took the over, 20% under, and 10% said push, the PECOTA total would be moved up by 5 wins.

Let’s find out how the contenders got on. First, performance against the final team win totals:

SetMAEMAE RankRMSERMSE Rank
PEC7.5319.311
Comp7.8429.772
EW8.0039.863
APEC8.40610.264
FG8.13410.375
Dav8.30510.676
BttP8.73710.697
DivTeamActualPEC DiffFG DiffDav DiffBttP DiffEW DiffComp DiffAPEC Diff
ALCCLE9153117-241
ALCMIN783386958
ALCDET644635030
ALCCHW62944813710
ALCKCR588131210131113
ALEBOS108-20-15-13-14-3-14-14
ALENYY100-3-5-8-2-2-5-11
ALETBR90-7-12-17-16-11-13-14
ALETOR7371391211105
ALEBAL4724313231282927
ALWHOU103-4-3-9-5-5-6-6
ALWOAK97-21-17-23-22-16-20-23
ALWSEA89-7-9-8-8-10-9-9
ALWLAA80-1424429
ALWTEX67812151212117
DivTeamActualPEC DiffFG DiffDav DiffBttP DiffEW DiffComp DiffAPEC Diff
NLCMIL96-12-17-9-12-10-13-8
NLCCHC95-4-1114-14
NLCSTL88-3-3-100-2-3
NLCPIT82-5-6-6-4-6-6-11
NLCCIN677449764
NLEATL90-9-14-10-14-13-13-11
NLEWAS8277101011814
NLEPHI80-4-8-8-5-5-7-7
NLENYM7735171153
NLEMIA6321-1-5-4-2-3
NLWLAD925146333
NLWCOL91-13-11-5-80-8-7
NLWARI825-126734
NLWSFG7398681189
NLWSDP6677-75944

After narrowly winning by RMSE last year and just falling behind FanGraphs in MAE, PECOTA romped to¬† victory this season. It beat both of the other projection systems in RMSE by over a full win, and only the Composite projections, naturally influenced by PECOTA itself, got to within even half a win. With PECOTA’s success, the Adjusted PECOTA set improved on just 5 PECOTA projections by three or more wins. Of the most vehement overs and unders, only Boston’s prediction was significantly improved, with most moving in the wrong direction.

Some classic PECOTA trademarks from recent years really helped it to outperform the other efforts, as that AL table shows: being down on the Royals and Orioles, and relatively high on the Rays (the most popular APEC under). Although the system wasn’t especially close on any of those teams, and everyone missed Baltimore by record-breaking amounts, it was comfortably the closest in all three cases. Davenport once again finished last of the three systems, and this marked the first time that FanGraphs’ Depth Charts were not a top-two finisher.

The Effectively Wild guests bounced back from last season’s basement finish with an impressive third, and also took the most coveted and nebulous prize: most correct win totals. Jesse Spector of Rockies Magazine picked Colorado to win 91, MLB.com’s Jason Beck was spot on about the terrible 64-win Tigers (as was the Adjusted PECOTA prediction), and Craig Edwards of FanGraphs joined BttP’s Alex Crisafulli in predicting the correct 88 wins for the Cardinals. Oddly enough, not a single projection system total was correct in 2018.

As that might suggest, St Louis were the most predictable team this year, with an average miss of just 1.7 wins. The Cubs were right behind at 2.2 wins, with the Marlins the only other team under three wins at 2.6. Baltimore’s average miss was an unprecedented 28.8 wins, blowing the Giants’ 24.3 from last year out of the water. The A’s also confounded everyone, at 20.4. 2018 as a whole was slightly harder to predict than 2017, which in turn was significantly more thorny than 2016, so from a win total perspective, baseball has been more unpredictable recently, not less.

Special mention must once again go to Alex Speier of the Boston Globe. Although Speier did not predict the Red Sox total for the second year running, he did boldly predict 105 wins, missing their actual total by just 3 and beating all other predictions and projections by at least 10 wins.

What about the final ranks? While surprises like the Orioles and the A’s made this a really tough season for win totals, using final finishing position strips out the extremity of the totals and allows us to see how the sets did from a pure rankings perspective.

SetMAEMAE RankRMSERMSE Rank
PEC4.0015.421
EW4.9726.002
Comp5.0036.043
APEC5.3756.424
Dav5.3346.675
FG5.8066.916
BttP5.977.277
DivTeamRankPEC DiffFG DiffDav DiffBttP DiffEW DiffComp DiffAPEC Diff
ALCCLE84577052
ALCMIN195798789
ALCDET26-2-2-1-2-3-3-3
ALCCHW282-101514
ALCKCR290330114
ALEBOS1-6-4-3-50-5-4
ALENYY311-3201-4
ALETBR10-1-10-13-15-8-10-9
ALETOR21513811783
ALEBAL3041012117710
ALWHOU211-31-110
ALWOAK4-17-11-18-19-13-17-16
ALWSEA120-3-4-4-6-5-3
ALWLAA17-17463310
ALWTEX2305105553
DivTeamRankPEC DiffFG DiffDav DiffBttP DiffEW DiffComp DiffAPEC Diff
NLCMIL5-5-13-3-6-8-5-4
NLCCHC61341415
NLCSTL134455351
NLCPIT14-6-8-7-5-8-8-11
NLCCIN23-1-3-22-4-2-2
NLEATL10-4-12-7-11-11-9-7
NLEWAS1487878711
NLEPHI17-4-8-7-6-6-7-6
NLENYM2049091055
NLEMIA27-3-3-2-3-3-3-3
NLWLAD75256233
NLWCOL8-11-7-2-71-4-5
NLWARI146226654
NLWSFG219935757
NLWSDP2501-5-12-1-3

It’s another comfortable win for PECOTA, giving it the clean sweep. The ranking method aided the EW guests and Davenport, both of which tend more towards the extremes with regards to the win totals.

If the win total results suggested a league that was rather surprising, these results line up far more with the common perception of the current environment as a predictable league led by several super teams, and propped up by tanking rosters. The A’s were the only team that every set missed on by double digits.

This does raise some questions about what we even mean by predictable and the perceived problems that predictability represents. The Astros still won the AL West with one of the league’s best win totals, but because one team pushed them all the way, it wasn’t a predictable race. The Cubs almost finished exactly where everyone predicted in both wins and rankings, yet that cost them a division title. Only four of the six divisions (and it was almost just the three AL divisions) finished with winners that were widely-expected, but these rankings results indicate the most predictable set of standings in any of the last four years.

The AL Central does represent the one truly predictable division in many ways. At the same time, it was unique, in that it was arguably the worst division of all time. Even when it is predictable, baseball still isn’t predictable.

Next post:
Previous post:

Trackbacks/Pingbacks

  1.  Bat Flips and Nerds Versus the Projections: The Results – Bat Flips and Nerds

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.