Tuesday, August 7, 2007

Do Stats Need to be Adjusted for Conference?

In this post and this post, I've found that the proportion of games won by the home team and those won by the "better" team (according to DVOA) is smaller for interconference games with a larger variance in the former proportion than for intraconference games. Games between divisional opponents have the highest of both proportions and the smallest variance in the former proportion. This led me to hypothesize that coaches are able to better adapt to teams they see more frequently, which makes inuitive sense, but I was also curious whether or not stats such as DVOA (or my VOLA stats) needed to be adjusted based on conference averages rather than league averages. Are NFC teams rewarded unfairly by DVOA for playing three quarters of its season in a conference full of weak teams? Ideally, opponent adjustments will filter out those effects, even if it's by league average. But perhaps 4 games against a stronger conference is not enough to filter out the effect. NFC Team X has played Y% better than league average when adjusted for opponent, but they were more likely to because of an easy schedule. Against AFC opponents, they'd be less likely to play at that level, even when adjusted for opponent strength. This effect might pop up more in games between average to below average opponents rather than when the NFC team is legitimately good. If the effect is real, then we'd see a greater proportion of interconference games being lost by the team with the higher DVOA when that team is in the NFC rather than the AFC.


I went back and looked at games from the entire history of DVOA (1996-2006). The DVOA totals used reflect the entire season's performance, including postseason. We should see the balance of power shift in favor of the AFC in 2000-6, given that only 1 NFC team ('02 Bucs) won the Super Bowl in that time span, and that team had a large and rare tactical advantage.

















Year% Was Better Team% Was Better Team
and At Home
Better Team Win %
Overall
AFCNFCAFCNFCAFCNFC
19960.566670.433330.283330.216670.676470.65385
19970.566670.433330.250.183330.558820.53846
19980.550.450.283330.233330.696970.7037
19990.650.350.316670.166670.743590.57143
20000.650.350.316670.166670.641030.7619
20010.50.50.250.250.80.8
20020.593750.406250.296880.203130.631580.57692
20030.609380.3750.281250.156250.666670.66667
20040.750.250.406250.156250.770830.5625
20050.656250.343750.328130.171880.619050.63636
20060.593750.406250.296880.203130.710530.5



The first table shows that the AFC might have been the better conference overall since at least 1996. For both conferences, the number of games in which their team was the better team is split evenly between home and away in almost every year, so neither conference's numbers should be unduly affected by home field advantage. For 6 out of 11 years, the AFC had a better win percentage when they were the better team than the NFC did when they were. In 2 of those seasons, the difference amounted to less than one game ('96 and '97). Looking at 2002-6, after the realignment, the AFC has had the better win percentage in 3 out of the 5 seasons. In 2003 and 2005, they were essentially even. In 2004, the NFC would have had to have won 3.33 more games in which they had the better team to match the AFC's win percentage. In 2006, they would have needed 5.4737 games. The predictive power of DVOA (where prediction = team with better total DVOA wins) was only 55.56% in 2006, compared to 66.67% in 2005 and 65% in 2004. Based on the table above, I'm guessing a large amount of the dropoff was because of interconference games, valuing NFC teams too highly given the gap between the conferences. That the year-to-year differences in better team winning percentage favor the AFC more strongly on average also indicates NFC teams being rewarded by a weak conference.

















YearBetter Team Win %
Home Better
Better Team Win %
Away Better
AFCNFCAFCNFC
19960.764710.769230.588240.53846
19970.60.818180.526320.33333
19980.823530.785710.56250.61538
19990.789470.70.70.45455
20000.842110.80.450.72727
20010.80.866670.80.73333
20020.789470.769230.473680.38462
20030.833330.90.523810.5
20040.807690.60.727270.5
20050.76190.818180.476190.45455
20060.736840.461540.684210.53846



The NFC teams see a bigger gap in better team win % for home and away games. The average difference is 18.517% for the AFC and 22.807% for the NFC. Small sample size has some effect for the games in which the NFC team is better, however. On average, the difference seems to be one or two games that could go either way.

So DVOA does seem to unfairly reward some NFC teams for playing in a weaker conference. Which teams are throwing off the system, though: the good teams being bumped up to very good or the mediocre teams being bumped up to good? In most years, the average gap in DVOA between teams in games in which the AFC was the better team and lost was larger than that for the NFC. And in most years, the average DVOA of the better teams that lost was higher for the AFC than the NFC. I would say, then, that the NFC teams that are mediocre to slightly above average (Rams, Falcons, Panthers) are unfairly rewarded more than good NFC teams (Eagles, Cowboys) for playing in a weaker conference.

Though DVOA is far more sophisticated than my methods, I believe I can apply the lessons learned here to my prediction model, specifically my VOLA stats. For intraconference games, I might start using Value Over Conference Average rather than VOLA. For interconference games, I could use Value Over Other Conference Average. So if Team A average 6.5 yards per play, it could be average in the AFC, but it might above average in the weaker NFC. Perhaps you could do a similar thing for DVOA, but the math of it is rather hazy to me right now.

4 comments:

Brian Burke said...

I guess you read my comment from yesterday. We pretty much said the same thing.

I don't think adjusting for conference is the key, but it would help. It would affect teams differently. Some teams start off the year with 2 or 3 out of 6 inter-conf games. Some teams don't have their first inter-conf game until week 10. Were the first couple inter-conf games against doormats in an otherwise strong conference, or against the 2 strongest teams in the league?

I think the best way would be to have a very good opponent adjustment. Most strength of schedule adjustments are based on opponent records, but W-L records are full of randomness. I'd recommend game-by-game adjustments based on opponents' stats (not including the game against the team in question). But that's a lot of work. I don't know much about DVOA, but I suspect FO doesn't rely on just w-l records for their DVOA.

Now I'm doubting myself. Maybe a sound adjustment for conference is in order, at least toward the end of the season when we know which teams are any good.

Brian Burke said...

By the way, there seems to be something wrong with your main page.

Anonymous said...

wow, you need to click that link. the models they have are killing me.

Anonymous said...

I think your thought of creating value over conference is a good one. I'm not sure that it is a matter of DVOA systemically favoring a weaker conference, though. I think there are just more unusual results in AFC-NFC matchups that could skew the results of a rating system, sometimes in favor of overvaluing the better conference overall, and sometimes in favor of the weaker conference overall.

Here are the home records in intraconference matchups, sorted by DVOA difference, the first is AFC, followed by NFC, since 2002:

+40.1 or better, AFC 20-0, NFC 3-0
+30.1 to +40.0, AFC 17-2, NFC 5-1
+20.1 to +30.0, AFC 16-2, NFC 7-5
+10.1 to +20.0, AFC 11-7, NFC 11-4
within +/-10.0, AFC 26-20-1, NFC 33-21
-10.1 to -20.0, AFC 10-4, NFC 8-6
-20.1 to -30.0, AFC 8-5, NFC 8-13
-30.1 to -40.0, AFC 1-5, NFC 8-9
-40.1 or worse, AFC 1-4, NFC 1-17

In some groups, the AFC was better at home, in some, the NFC was better. Look at the NFC group at -30.1 to -40.0, these are teams that are significantly worse than their opponent, and are 8-9 at home. Aaron Schatz of FO has said home field is around +15.0% DVOA on average. However, it appears stronger in intraconference games where the home team is the relatively worse team. Home Teams in intraconference games with a DVOA difference of -10.1 to -40.0 have a winning record 43-42. I guarantee you would not see that kind of winning percentage for worse teams in divisional matchups.

I think uncertainty and unfamiliarity is playing a role here. Thus, I would put less stock in an unusual result in intraconference play than in divisional or conference play.

Also, because there is far less cross over, one result can skew things. Take Oakland's 45-0 win over Tampa in 1999. I dare say that win was not representative of those teams strengths that year, it was a great game for Oakland and a poor one for Tampa. That happens in a conference matchup, and things even themselves out more, because of the number of common opponents. But it happens in an AFC-NFC matchup, and it might cause overvaluing of the AFC West in that case, and undervaluing of the NFC Central.