Brier score and calibration: why 84% should mean 84%

When a model says "84%", what should that mean? It should mean that, across many answers given with 84% confidence, about 84% turn out right. That property is called calibration. Two numbers measure it: the Brier score and the expected calibration error (ECE).

This post explains both in plain English. Every number in it comes from training Coeus, our open Video Jev model. Coeus is fine-tuned from Qwen3-Omni, with Clef's answer head. It answers typed questions about a video with a probability for every answer, so these two numbers are on every row of our results. The engagement numbers come from the same recipe trained on one task, predicting which short videos people keep watching.

The code and the model: the metrics code, the experiment log and the paper are on GitHub. The weights and the results table are on Hugging Face.

The numbers in this post

What we measured	Brier	ECE
Uniform guess over 10 engagement levels	0.900	-
Our engagement model (Coeus recipe) on its test, 1,736 clips, no scaling	0.818	0.028
CLIP + MLP, after temperature scaling, same test	0.843	0.034
CLIP + MLP before scaling (500 val clips)	0.876	0.144
Untrained Qwen3-Omni, AI-video quality scores (test)	-	0.29 to 0.56
Coeus, AI-video quality scores (test)	-	0.031 to 0.045

Lower is better for both. One training seed per result, so small gaps are inside the noise.

What calibration means

Take every answer a model gave with about 84% confidence. Count how many were right. If about 84% were, the model is calibrated at that level. If only 60% were, it is overconfident. If 95% were, it is underconfident.

Calibration is not the same as being accurate. A weather forecaster who says "30% chance of rain" every day in a city where it rains on 30% of days is perfectly calibrated, and useless. A model can also be very accurate and still overconfident: right most of the time, but sure of itself when it is wrong.

You want both: answers that are often right, and confidence you can take at face value. That is why a calibrated number can drive code. You can write "approve above 0.9, send 0.4 to 0.6 to a person" and know roughly how many mistakes each rule lets through. More on why this matters for typed answers is in What is a Jev model?.

The Brier score

The Brier score grades each probability against what actually happened. For each answer option, take the probability the model gave, subtract 1 if that option was the right one (0 if not), and square it. Add those up for the question, then average over all questions.

A worked yes-or-no example:

The model said	The answer was	Brier for this question
90% yes	yes	(0.9 - 1)² + (0.1 - 0)² = 0.02
60% yes	yes	(0.6 - 1)² + (0.4 - 0)² = 0.32
90% yes	no	(0.9 - 0)² + (0.1 - 1)² = 1.62

Confident and right costs almost nothing. Unsure costs a little. Confident and wrong costs a lot. So the Brier score rewards being right and punishes false confidence in the same number. On this scale it runs from 0 (perfect) to 2 (fully sure of the wrong answer).

The same rule works for more than two answers. Our engagement question has 10 levels, from least to most engaging. A model that gives every level 10% scores 0.9² + 9 × 0.1² = 0.900, whatever the answer. That is the bar to beat: below 0.900 means the probabilities carry real information.

OursBaselines

Ours (Coeus recipe), no scaling

0.818

CLIP + MLP + temperature

0.843

CLIP + text + MLP + temperature

0.859

Text + XGBoost + temperature

0.884

Uniform guess

0.900

Clef-flash, untrained (video + text)

0.912

SnapUGC engagement test split, 1,736 clips, 10 levels. A uniform guess scores 0.900. One seed per model.

The untrained model scores worse than a uniform guess. Its probabilities are worse than saying nothing.

Expected calibration error (ECE)

ECE asks only one question: does the model's confidence match how often it is right?

1. For each answer, take the model's top choice and its confidence in it.
2. Sort the answers into bins by confidence: 0 to 10%, 10 to 20%, and so on.
3. In each bin, compare the average confidence with the share that were right.
4. Average those gaps, weighting each bin by how many answers are in it.

If the model says 80% on a batch of answers and 80% are right, that bin adds nothing. If it says 80% and only 50% are right, that bin adds a 0.30 gap. An ECE of 0.03 means confidence and accuracy are about 3 points apart on average.

Untrained, Qwen3-Omni's confidence on our quality questions was off by 29 to 56 points. After training, Coeus is off by 3 to 4.5.

Why you need both numbers

ECE can be gamed by hedging. A model that always says "about 10% for each of the 10 levels" has a top-choice confidence near 10%, and it is right about 10% of the time. Its ECE is low. It has told you nothing.

We saw exactly this. Asked our engagement question without training, the untrained Clef-flash model, given only the video, scored ECE 0.051. That looks fine. Its Brier score was 0.910, worse than the 0.900 of a uniform guess. The low ECE meant hedging, not skill.

The Brier score catches that, because it also rewards being right. ECE catches what Brier hides: two models can have the same Brier score while one of them is badly overconfident. So we report both on every result, and we read them together:

• Low Brier and low ECE: useful and honest.
• Low ECE, Brier near the uniform guess: hedging.
• Good ranking, high ECE: the order is right, but the numbers cannot be used as thresholds.

Temperature scaling: what it fixes and what it does not

Temperature scaling is the standard fix for an overconfident model. Before turning scores into probabilities, divide them by one number, the temperature. A temperature above 1 flattens the probabilities; below 1 sharpens them. You fit that one number on held-out data, so the model's confidence matches its accuracy there.

Our cheap baselines needed it. They were trained on the same engagement training split and were overconfident:

Baseline (500 val clips)	Temperature	ECE before	ECE after	Brier before	Brier after
CLIP + MLP	2.04	0.144	0.046	0.876	0.845
CLIP + text + MLP	5.88	0.465	0.037	1.136	0.852
Text + XGBoost	1.32	0.055	0.044	0.899	0.891

Adding text made the CLIP model far more overconfident (ECE 0.465), and a temperature of 5.88 was needed to undo it.

What temperature scaling does not do is make a model better at the task. It never changes which answer the model rates highest. It moved the CLIP + MLP ranking score only a little, from 0.662 to 0.673 on val. So it fixes how sure the model sounds, not how well it ranks videos. On the untouched engagement test, our engagement model still ranked better than the scaled CLIP + MLP: a final score of 0.709 against 0.643 (gap +0.065, 95% interval 0.041 to 0.088). More on that comparison is in Can one model predict which short videos people keep watching?.

Calibration from the loss

Coeus, and the engagement model built on its recipe, have no temperature step. Their calibration comes from how they were trained. The training loss adds three parts:

• Cross-entropy with light label smoothing (0.05): be right.
• Brier score, weighted 1.0: do not be falsely sure.
• Ordinal distance, weighted 1.0: on ordered answers, being one level off is better than being five off.

Because the Brier score is in the loss, the model is pushed toward honest probabilities while it learns. On engagement, ECE stayed near 0.03 through training, and it was 0.028 on the untouched test. How we set up that training, and the failures on the way, are in Fine-tuning Qwen3-Omni on video and audio.

Where it is not calibrated yet

Calibration is not solved everywhere. From the working paper's test results:

Question (test)	Untrained ECE	Coeus ECE
AI-video quality, 5 scores	0.29 to 0.56	0.031 to 0.045
Real or AI	0.034	0.023
Which sound (VGGSound)	-	0.157
Laughing (never trained)	0.112	0.101

The sound question is the weak one. VGGSound's official test is harder than its validation set for every model: even the untrained backbone drops from 0.66 to 0.50 accuracy. Calibration suffers with it. A question the model was never trained on (laughing) is also poorly calibrated. So check calibration on your own question before you set a threshold on it.

Two more caveats. Every result here is one training seed, with no run-to-run spread yet. And calibration is measured on the same kind of data the model was tested on; on very different videos it can drift.

A short checklist

1. Report Brier and ECE next to accuracy, on held-out data.
2. Compare the Brier score with a uniform guess. If it is close, the model is hedging.
3. If ECE is high and you cannot retrain, fit a temperature on held-out data. Expect better confidence, not better ranking.
4. If you can retrain, put the Brier score in the loss.
5. Check calibration per question before using a threshold on it.

Run it yourself: the metrics code (Brier and ECE), the training code and the paper are on GitHub. The weights and the quick start are on Hugging Face, for non-commercial use.

More on Coeus:

• Coeus, our open Video Jev model: what it does and every benchmark.
• What is a Jev model?
• Can an open model spot AI-generated video?
• Can one model predict which short videos people keep watching?
• Qwen3-Omni fine-tuning on video and audio: our LoRA recipe
• Video moment retrieval: finding the funniest second
• AI video ad analysis: 777 Meta Ad Library ads, scored