Prediction Model Metrics: Brier Score, Log Loss, Calibration

Prediction model metrics tell you whether a forecaster’s probabilities deserve trust, and the three that matter most for football are the Brier score, log loss and the reliability curve. Each one scores the stated probabilities against what actually happened, so a model cannot look good by picking easy matches or hiding its misses. This guide explains what each metric rewards, what it punishes and where you can see them applied to a real, public record.

What Does the Brier Score Measure?

The Brier score is the average squared gap between the probability a forecaster gave and the outcome, counted as one if it happened and zero if it did not. Lower is better. A forecaster who says a home win is very likely and sees it happen barely loses anything; one who says the same and sees an away win loses a lot.

Worked example: a 0.7 chance for an outcome that happens scores (1 − 0.7)² = 0.09; the same 0.7 for an outcome that does not happen scores 0.7² = 0.49. Always answering 50/50 on a two-way question scores 0.25 every time.

Because the penalty is squared, the Brier score is forgiving of small errors and harsh on large ones, but it never punishes a forecast infinitely. That makes it a steady, readable measure over thousands of matches.

How Is Log Loss Different From the Brier Score?

Log loss scores each forecast by the logarithm of the probability it gave to what actually happened. It rewards the same thing as the Brier score, honest and sharp probabilities, but it punishes confident mistakes far more savagely. A forecaster that gives an outcome almost no chance and then sees it happen takes a very large penalty, and giving an outcome zero chance scores infinitely badly.

That is why log loss is the favourite metric for training models: it teaches them never to rule an outcome out entirely in a football match. It is harder to read at a glance than the Brier score, so the two are often reported side by side.

What Is a Reliability Curve?

A reliability curve, or calibration curve, groups predictions by the probability they were given and plots each group’s stated probability against how often those outcomes actually happened. A perfectly calibrated forecaster sits on the diagonal: its sixty-per-cent calls come true about sixty per cent of the time.

The calibration chart on the accuracy page is a reliability curve drawn from every settled TACTIX prediction, with the number of predictions in each bucket shown beside it. Points above the diagonal mean the model was too cautious in that band; points below mean it was too confident.

Why Do Some Models Look Brilliant and Then Fail?

Overfitting is when a model learns the quirks of the matches it was trained on instead of the patterns that carry forward. It looks superb when tested on the same data and falls apart on new fixtures. The defence is to judge a model only on matches it never saw during training, in the order they were played, which is called out-of-time validation.

The TACTIX methodology describes how every model change is validated this way before it ships. A back-test that is not out of time is the first thing to question in any prediction model metrics a site quotes.

What Is an Ensemble Model?

An ensemble combines several different models into one forecast: averaging them, weighting them by recent skill, or letting one model correct another. Different models make different mistakes, so a sensible blend is usually steadier than its best single member. TACTIX does not average several models. It serves one base model, a Poisson goal matrix with a Dixon–Coles correction, and adds two learned layers: a stacking layer for both teams to score and over/under 2.5 goals, and a three-way match-result model that uses Elo ratings. Each layer is kept only where it beats the base model on matches it had not seen, and the output is calibrated market by market.

Where Can You Check These Prediction Model Metrics?

Look for a record that includes the losing predictions, states its sample size and shows its calibration by probability band. The scoreboard grades every TACTIX prediction after full time, and the checklist for judging a prediction model turns these metrics into five questions to ask any site.

18+. Information and analysis only, not betting advice. If gambling stops being fun, free and confidential help is at BeGambleAware.org.