Can one model predict which short videos people keep watching?

Every short video lives or dies in its first few seconds. Theia makes short films, so the question we care about most is simple: before a video is posted, can a model tell whether people will keep watching it?

We tested that on a public benchmark, using the same method as Coeus, our Video Jev model. One model, trained once, scored 0.709. The winner of the 2025 challenge on this task scored 0.710 with an ensemble of four models.

The released model: Coeus and its full results table are on Hugging Face. The code and the SnapUGC experiments are on GitHub. This SnapUGC run uses the same recipe, trained on this one task.

The benchmark

SnapUGC is the dataset behind the VQualA 2025 engagement challenge. It holds short user-made Snapchat videos, each labelled with its engagement continuation rate (ECR): the share of viewers who keep watching past the first 5 seconds. That is close to what TikTok calls retention.

The score is the challenge's own: the mean of SROCC and PLCC between predicted and real ECR. Both measure how well the model's ranking and values match reality, from 0 (no link) to 1 (perfect).

What we did

• Model: Qwen3-Omni-30B-A3B, an open model that reads video and audio together, with a LoRA adapter and a typed answer head. This is the Coeus recipe, trained on this one task.
• Input: the first 16 seconds at 2 frames per second, the soundtrack, and the title and description.
• Question: "What is this video's ECR?" as a 10-level score question. The levels are ECR deciles, so the model returns a probability for each level, not a single guess.
• Training: 30,631 clips, one epoch, 7.5 hours on one H200 GPU, about $43.
• Test: 1,736 held-out clips, scored once, after every choice was made on a separate validation set.

Results

Model	Input	Score	Calibration error
Coeus recipe: Qwen3-Omni + typed head (ours)	video, audio, text	0.709	0.028
CLIP + MLP	8 frames	0.643	0.034
CLIP + text + MLP	8 frames, text	0.639	0.029
Text + XGBoost	title and description	0.347	0.029
Clef-flash, no training	video, text	0.127	0.064

Our model beats the strongest baseline we trained, CLIP + MLP on the same clips, by 0.065. A paired bootstrap puts that gap between 0.041 and 0.088, so it is not noise.

For context, the official VQualA 2025 leaderboard:

Rank	Team	Score
1	ECNU-SJTU VQA (ensemble of 3 VideoLLaMA2 + Qwen2.5-VL, with audio)	0.710
2	IMCL-DAMO	0.698
3	HKUST-Cardiff-MI-BAAI	0.680
4	MCCE / EasyVQA	0.667

What we are careful not to claim

We say "level with" the winner, not "beats". The official test labels are withheld, so we made our own test split from the labelled data. Our 0.709 and their 0.710 are on different clips.

The data has a survivorship bias. Only 32% of the SnapUGC videos could still be downloaded, and low-engagement videos were deleted more often. The average ECR is 0.47 on our clips against 0.50 over all labels.

It is one training run. We report the confidence interval for the gap to the baseline, but not yet for the score itself.

What we learned

1. Training is the whole game. The untrained model scores 0.13. Trained on 10,000 clips it reaches 0.65 on validation, and on all 30,631 it reaches 0.72 on validation and 0.709 on test.
2. The video carries the signal. Title and description alone reach only 0.35.
3. Calibration comes from the loss. The model's calibration error stays near 0.03 without any extra step. The baselines needed temperature scaling to get there.
4. Ensembles stop helping as the model gets stronger. Adding CLIP helped the small pilot by 0.04 and the full model by 0.001.
5. Audio helped a little at most. Removing it cost 0.026 in one early run, which is within the noise.

Why it matters to us

A calibrated engagement score is a decision tool. "This cut has a 70% chance of landing in the top half" is something code can act on: keep it, re-cut it, or show it to a person.

We are continuing to train on more data, and we will publish the next numbers here, including the ones that do not improve.

The method, every experiment and the raw results are in the Coeus repository.

See every number: the results table, the weak spots and the quick start are on Hugging Face. The training code and the full paper are on GitHub.

More on Coeus:

• Coeus, our open Video Jev model
• What is a Jev model?
• Can an open model spot AI-generated video?
• Fine-tuning Qwen3-Omni on video and audio: our recipe