Accuracy of Vegas' Win Projections
In the post about the accuracy of PFP's 2006 win projections, I just got a comment linking to another interesting analysis of their accuracy as well as Vegas' win projections for 2005 and 2006. There's an Excel file with those numbers at the link.
The projections are within the 4.5-11.5 range, so they cover a pretty good range. It's about as wide as you'd want it. What surprised me is that in both years, exactly half the teams performed over and half the teams performed under their projection. Perhaps it is just a mathematical certainty that it will happen, but I would have expected random perturbations either way as in coin flips. But 2007 might buck that trend, so we'll see.
2005:
Mean abs. err.: 3.0469 games
Corr. coef. with actual wins: 0.22331
Largest error: 5.5 games (Jets, Texans, Eagles)
Smallest error: 0.5 games (Patriots, Chiefs, Vikings, 49ers)
No. of predictions within:
0-1 games: 5
1-2 games: 4
2-3 games: 5
3-4 games: 8
4-5 games: 7
2006:
Mean abs. err.: 2.3438 games
Corr. coef. with actual wins: 0.36088
Largest error: 5.5 games (Ravens)
Smallest error: 0.5 games (Bills, Texans, Colts, Chiefs)
No. of predictions within:
0-1 games: 8
1-2 games: 9
2-3 games: 6
3-4 games: 5
4-5 games: 3
In 2006, both PFP's projections and Vegas' projections were within 2 games of being exactly right for 17 out of 32 teams. But the one thing that strikes me about PFP's numbers is that the number of predictions curve is monotonically (i.e. always) decreasing as the error becomes larger (10,7,5,4,3). In the two years shown here, the curve for Vegas is more irregular, though 2006 is better than 2005. In 2005, more teams' win projection errors were within 3-4 games than any other margin (4-5 games followed closely). The PFP projections correctly predicted the Over 8 times and the Under 8 times, so using PFP to make all your over/under season win total bets, you would have been exactly 50% correct. Overall, I'd say that PFP's projections are better, and I'm really surprised to see that Vegas had a mean absolute error of 3 games in 2005. But the gap between the two is not that wide.
Since I was curious, here's how the Vegas regular season win total lines project the final standings by division:
AFC EAST
New England
New York Jets
Miami
Buffalo
AFC NORTH
Baltimore/Cincinnati (tie)
Pittsburgh
Cleveland
AFC SOUTH
Indianapolis
Jacksonville
Tennessee
Houston
AFC WEST
San Diego
Denver
Kansas City
Oakland
NFC EAST
Philidelphia
Dallas
New York Giants
Washington
NFC NORTH
Chicago
Green Bay
Minnesota
Detroit
NFC SOUTH
Carolina/New Orleans (tie)
Atlanta
Tampa Bay
NFC WEST
Seattle
Arizona/St. Louis/San Francisco (tie)
2 comments:
I don't think either Vegas or the FO predictions (or any others) are any better than simply using last year's wins to predict next year's wins.
Notice that in the division rankings you posted, they are exactly the same as last year's standings, except for the couple of ties.
If I do a similar regression exercise using just last year's wins from the 01-05 seasons, I get:
correlation = 0.26
r-squared = 0.13
mean absolute error = 2.51
That's no better or worse than either of the expert predictions.
Here's what I would do to make pre-season predicitons:
1. Take last year's wins.
2. Regress to the mean by 10% or so.
3. Identify which teams had a sure-thing upgrade/downgrade at QB (e.g. McNair/Brees in '06). Add 2 wins for the upgrade, subtract 2 for the downgrade.
Voila. Maybe I'll actually do this and post it. I'll go back see how it compares to Vegas/FO for 2007.
If you knew Drew Brees's full recovery from that shoulder injury was a "sure thing", then you are a wiser man than most of us. Or that an aging McNair actually still had something in the tank. Or, conversely, that Culpepper had not recovered from his injuries at all.
I'm not really arguing your core point - that most predictions are lousy to the point of being useless. Rather, I'm arguing that you lack an ability to make a clear-eyed subjective projection on one aspect of football (upgrade/downgrade at QB) in a way that's superior to everyone else. These things are, after all, the same things that the Vegas oddsmakers and FO are considering.
If you selectively pick the teams that actually did experience an improvement at QB after acquiring someone, then you're obviously going to get better results than Vegas. But you won't convince me of anything in the process.
What would impress me is if you could find a measure of past team performance that actually does improve predictions in a statisticaly significant fashion.
Post a Comment