How I proved my model couldn't get better, then spent four attempts proving it again

I built a model to predict UFC fight outcomes. It’s prediction-only — no betting logic, no staking, no market modeling, and that’s a scope decision written into the spec rather than something I drifted into.

Over 1,532 out-of-sample fights, refit every six months from January 2023 to July 2025, it lands at:

That accuracy number is unremarkable. Roughly 62% on fights where the naive base rate is 50%. I spent a long time trying to move it, and the most useful thing I did was stop and ask a different question: is this number low because my model is bad, or because the sport is hard?

There’s a decomposition that answers that, and I wish I’d run it a year earlier.

Splitting the error into three pieces

Murphy’s decomposition breaks a Brier score into three terms:

Brier  =  uncertainty  −  resolution  +  reliability
0.2259 =    0.2500     −    0.0244    +    0.0003

Each term means something specific:

Uncertainty (0.2500) is the variance of the outcome itself. A 50/50 binary event has uncertainty 0.25, and that is the maximum possible. This is the difficulty of the sport, and no model of any kind can reduce it. It’s a property of MMA, not of me.

Reliability (0.0003) is the calibration penalty — how far my stated probabilities drift from observed frequencies. Lower is better, zero is perfect. Mine is three ten-thousandths. When this model says 65%, the fighter wins about 65% of the time. There is essentially nothing left to fix here.

Resolution (0.0244) is the payload: how far my predictions move away from the base rate in the correct direction. This is the only term that represents knowing something. Mine is 0.0244 out of a theoretical maximum of 0.25.

That’s 9–10%. The model explains roughly a tenth of the available variance. The other ~90% is information I don’t have.

Why this changed what I did next

Before the decomposition, “62% accuracy” was ambiguous. It could have meant my priors were miscalibrated, or my likelihood was misspecified, or my features were weak, or the sport is close to a coin flip. Those diagnoses point at completely different work.

After the decomposition, only one of those is live. Reliability at 0.0003 rules out calibration problems. Uncertainty at the theoretical maximum rules out “the outcome is more predictable than I’m treating it as.” The entire deficit is resolution, and resolution is bought with information, not with better machinery.

So the question became concrete and falsifiable: is there information about UFC fights that I’m not using?

I made four attempts to find some. All four failed, and the way they failed is the interesting part.

Attempt 1 — a Bayesian latent-skill model

The obvious move. Replace the feature-based model with a proper hierarchical latent-skill model — every fighter gets a latent ability that evolves over time, Bradley-Terry style, fit with NUTS.

Badly worse. Test accuracy of 53.7% is barely above guessing, against a baseline at 67%.

The failure mode is textbook — train log loss of 0.492 against test 0.720 is a model memorising individual fighters. But I wanted to be sure it wasn’t a sampler artifact, so I swept the drift parameter down to 1e-6 (effectively freezing skill over time) with no change, and then re-implemented the whole thing as a one-hot Bradley-Terry logistic regression in scikit-learn as an independent check. Same answer.

Fighters don’t have enough bouts. A latent-skill model needs repeated matchups to separate ability from noise, and the UFC schedule doesn’t supply them.

Attempt 2 — a round-level simulator

If I can’t predict the fight directly, maybe I can simulate it — model round-level striking and grappling exchanges, then aggregate up to a winner.

It tied. Never beat. The reason, once I saw it, was obvious enough that I should have predicted it:

We’re conditioning on the same information twice.

The round-level statistics are computed from the same fighter histories that the winner model already uses. Simulating forward from them doesn’t add information; it just re-expresses the existing features through a more elaborate generative story. More parameters, same evidence. I canceled the phase.

Attempt 3 — external event metadata

Scraped Wikipedia for event context — venue, card position, attendance, event significance.

Change in log loss: 0.0000386.

I include this one because the null was so complete it’s almost funny, and because it took a day to establish. That’s the correct price for ruling something out.

Attempt 4 — pre-UFC fight history

The best remaining idea. Fighters arrive in the UFC with records from other promotions. That’s real information the model ignores, and it should be most valuable exactly where the model is weakest: debutants and low-experience fighters.

I built it. It made things worse. And the coverage-stratified breakdown is the most instructive table in the project:

Read that ordering carefully. The features hurt most in the rows where they actually exist, and did nothing where they were absent. If this had been a coverage problem — good signal diluted by missing data — the ordering would be reversed. It isn’t. The pre-UFC records are, for prediction purposes, noise: different competition quality, different rules, wildly inconsistent record-keeping.

That was the fourth failure, and it was predicted by the same decomposition that predicted the first three.

The debutant problem, and shipping a suppression instead of a fix

One thing I did fix, though not in the way I wanted.

The winner model extrapolates catastrophically when a fighter has zero prior UFC bouts — every difference feature saturates and the logit runs past anything in the training range, producing confident nonsense at 0.9999+.

The empirical check is brutal. On fights where both competitors are debuting (37 fights), accuracy is 0.270. Not 0.5. The model is systematically picking the wrong fighter, with high confidence, because the feature differences it’s reading are artifacts of missing history rather than signal.

I didn’t find a way to make it correct. So I suppressed it: fighters with fewer than three prior bouts are flagged sparse, confidence bands are capped at “low” whenever either fighter is sparse, and both-debutant predictions are withheld entirely.

Shipping a suppression instead of a correction feels like a failure, and in a sense it is. But a model that abstains on the cases it demonstrably can’t handle is more useful than one that answers everything at uniform confidence. The tiers are visible in the product, which means users can see where the model claims to know something and where it doesn’t.

The finding I didn’t expect: retraining is pointless here

One more measurement worth recording. I ran the walk-forward protocol — refit every six months — against a single static model trained once in 2023 and never updated.

The static model wins, slightly. Which means the environment isn’t drifting in any way the model can detect. It’s information-limited, not concept-drift-limited — and all the retraining infrastructure I’d built was solving a problem I didn’t have.

That distinction is worth carrying into any production ML context. “Retrain more often” is the reflexive answer to a model that underperforms, and it’s the right answer only when performance is decaying over time. If your gap is flat across the whole history, retraining is machinery you’ll maintain forever for nothing.

The part I’d defend hardest: stopping

The spec set a target of log loss below 0.62. I’m at 0.643. I missed, and the docs say so.

I could keep going. There’s always another feature set, another architecture, another ensemble. But the decomposition told me what I’d be buying with that time, and four independent attempts confirmed it empirically. Reliability is already ~0. The uncertainty term is at its theoretical maximum and is a fact about cage fighting. The only lever is resolution, and I’ve now failed to move it four different ways, including the two most promising ones.

At some point the honest read is that the model is close to the information ceiling of the data available to me, and further work is not modeling work — it’s data-acquisition work I’m not in a position to do. Fight-camp information, injury status, weight-cut severity, training-partner quality: that’s what would move resolution, and none of it is public.

Knowing when you’ve hit a ceiling is worth more than one more percentage point of accuracy, and the decomposition is what let me say it with a number instead of a feeling. I’d run it first next time — before building anything, as soon as there’s a baseline to decompose. It costs a few lines and it tells you which of the three possible problems you actually have.