My NFL model forecasts the line, not the game

I built an NFL power rating. Over 3,331 out-of-sample games across 19 seasons, it produced 63.16% closing line value — meaning the market moved toward my number, after I’d written it down, about 63% of the time. The 95% confidence interval is [0.615, 0.648], comfortably clear of coin-flip.

It gets better. The effect is monotone in the size of the disagreement: 65.9% at 1+ points, 68.0% at 2+, 71.0% at 3+, 79.55% at 4+, where the average game moved 1.61 points in my direction.

It survives the obvious confound. The first thing anyone says is “you’re just picking up injury news.” So I stripped the quarterback block out entirely — CLV barely moved (0.6211 vs 0.6356). Then I split by whether the announced starter actually changed:

CLV is higher when nothing changed at quarterback. Whatever the model is seeing, it isn’t the injury report.

This is, by every conventional standard, a real and well-validated signal.

It is also worth nothing, and understanding why took me longer than building the model.

The number that ended it

I regressed what the model wanted against two different things: what the market subsequently did, and what actually happened on the field.

It tracks the market’s own weekly repricing about seven times better than it tracks the game.

That single comparison reframes everything above it. The 63% CLV isn’t evidence that I know something about football. It’s evidence that my rating and the market’s repricing are computing a similar function of the same recent-form inputs — and that I happen to compute it a few days earlier in the week. With a 6-game half-life, my rating is recent-form-heavy. So is the money that moves lines Tuesday through Sunday. We’re correlated because we’re reading the same box scores, not because I’m right.

The magnitude tells the same story. Per one point the model wants to move the line:

• the market delivers +0.217 points
• reality delivers +0.248 points

A stated 4-point edge is really worth about 1. Every number this model produces needs shrinking by roughly three quarters before it means anything. That’s the one durable finding in the whole project, and it’s a correction factor, not an edge.

Why “the market agrees with me later” isn’t the same as “I’m right”

Closing line value is the metric everyone reaches for when raw accuracy is unimpressive, and mine was unimpressive: RMSE 12.753 against the actual margin, versus 12.590 for the closing line and 12.721 for the opener. The model beats neither.

CLV is genuinely predictive of outcomes — bets that ended with positive CLV covered 56.79% (n=2,062) against 45.36% (n=1,197) for negative CLV. That’s a big spread, and it’s real.

But look at the tense. You know which bucket you’re in after the line has moved. At the moment of the forecast, you hold a mixture of the two, and the mixture is approximately break-even. CLV is a post-hoc partition of your own predictions into the good ones and the bad ones. It cannot be acted on prospectively, which is the only way it would matter.

I now think of CLV as a diagnostic for whether your model is plugged into the same information stream as the market. It is not a measure of whether you have an insight the market lacks. Those are different questions, and the metric that answers the first one is very good at impersonating an answer to the second.

The bucket that looked like the whole thing

Before I got to any of that, I ran a threshold search over edge sizes and found something that made me sit up. Bets where the model disagreed with the line by 4 or more points covered at a rate that implied double-digit returns.

Then I stopped using cumulative buckets — which smear everything above a threshold together — and looked at disjoint bins:

Four bins of dead flat, two spectacular bins, then partial regression. Written down from my own notes at the time:

There is no mechanism that makes a 3.9-point disagreement worthless and a 4.1-point disagreement worth 13% ROI. Two bins out of seven, found by a search over thresholds, is what noise looks like.

The cumulative view hid the shape. Because every “4+” bucket contains the 4–5 and 5–6 bins, the anomaly propagated upward into every threshold I tested and made the effect look monotone and robust. It wasn’t monotone; it was two adjacent cells doing all the work in a 4,560-game sample.

I’ve since made disjoint binning the default. Cumulative buckets are how a local anomaly disguises itself as a trend.

What I do with a model that doesn’t work

The honest scoreboard, which is generated automatically and published with the project:

Margin RMSE 12.890 vs closing line 12.423 (+0.467) — the closing line is more accurate. Beat the closing line on margin RMSE (2025 holdout): FAIL — 12.896 vs 12.245. Disagree-bucket ATS clears the vig (pooled): FAIL — 0.4691, 95% CI [0.400, 0.539]. No layer beats the closing line.

Every ablation layer fails. The table has a “Clears vig?” column and it reads no in all eight rows.

So the interesting question stopped being “can this win” and became “how do I keep myself from fooling myself next season.” Which produced the part of this project I’d actually defend.

Pre-registering the next test

The 2025 season had been scored three times — once each for three model layers. Every look erodes a holdout. A fourth number from 2025 would have meant nothing, and I’d have had no way to prove to anyone, including myself, that I hadn’t tuned until it looked good.

So on 5 August 2026, 35 days before the first kickoff, I locked predictions for all 272 regular-season games. Home margin, win probability, and pick for every game, written to a CSV, SHA-256 hashed, with the hash recorded in a separate lock file. The scoring script re-hashes the file on every run and refuses to score if a single byte changed. I verified this by appending one newline and watching it fail.

Two gates, defined before the season:

• Gate 1 — beat the market: margin RMSE lower than the closing line’s on the same games.
• Gate 2 — bettable: on games where the model disagrees with the close, the Wilson 95% lower bound on the cover rate exceeds the −110 break-even of 0.5238.

And explicitly not a gate: performance when agreeing with the closing line. Agreeing with the close means picking the favourite, which lands near the Vegas baseline no matter how good or bad your model is. It’s the metric that flatters everybody.

Then the line I’m proudest of in the whole repository, written into the lock file in advance:

I expect GATE 1 and GATE 2 both to fail. A pre-season Elo that matched the market would be a genuinely surprising result. The purpose of this lock is to make that prediction falsifiable rather than to hope for an upset.

A pre-registration where you expect to succeed is a press release. One where you expect to fail is an experiment.

The postscript that makes the point better than the lock does

After locking, I built a better model — an efficiency-and-QB stack that beats the locked one on every out-of-sample split.

Which means the lock now tests a model I no longer consider my best. The obvious move is to re-lock with the better stack before kickoff. I haven’t, and the reason is written into the docs:

It should either be re-locked with this model before kickoff, or explicitly treated as a test of L1 alone. It has not been re-locked, because doing so silently would defeat the purpose of locking it.

Quietly swapping the contents of a sealed envelope is not a stronger test than the one you sealed. If I re-lock, it gets a new date, a new hash, and a note saying what it replaced and why. Otherwise the season scores the model I actually committed to in August — a worse model, honestly evaluated, which is worth more than a better model evaluated on a technicality.

What I take from it

The model is a competent power rating that does not beat the market, closing or opening. That’s the finding. I spent a season getting to it, and I’d spend it again, because the three things it taught me transfer to work that isn’t football:

A metric can be real, robust, monotone, confound-resistant — and still be measuring your correlation with the incumbent rather than your accuracy. Ask what your headline metric would look like if you were merely similar to the thing you’re trying to beat, and check whether that’s distinguishable from success.

Cumulative buckets launder noise into trends. Bin disjointly, always.

Pre-register before you have a result, and state the outcome you expect. It costs nothing when you’re right and it’s the only thing that makes a null result credible when you’re wrong.