Coeus, our open Video Jev model that watches and listens

Jev is a format for decisions. A Jev record holds some context and a set of typed questions, and each question lists its allowed answers. A Jev model returns a probability for every answer. Code then acts on those numbers: a threshold, a ranking, or "send this one to a person".

That only works if the probabilities are honest. When a Jev model says 84%, it should be right about 84% of the time.

The existing Jev model, Cloudflare's Clef, reads text and images. Theia makes short films, so we wanted one that could watch a video and hear it at the same time. So we trained one, and we have released it.

We call it Coeus. In Greek myth, Coeus is the Titan of the inquiring mind, the one who asks questions, and he is Theia's brother. Theia makes the films. Coeus watches them, listens to them, and answers questions about them.

What it is, in two words: a Video Jev. You can get it on Hugging Face as video-jev.

Full breakdown of results: every benchmark, split and baseline is on the Coeus model card on Hugging Face, with the weights. The code, the paper and every experiment are on GitHub.

What Coeus does

Coeus reads raw video frames and the soundtrack on one timeline. You write your questions as a Jev record, and one forward pass answers all of them.

Question type	You get	Example
noul	P(yes) and P(no)	"Can you hear someone laughing?"
choice	P for each named option	"Which sound is this: laughing, cheering, speech?"
score	P for each ordered level	"Visual quality: bad, average, good or perfect?"
moment	P for each second of video	"When is the funniest moment?"

The moment type is new. Each second of the video is an answer option, scored on that second's own video and audio tokens. There is no separate detector and no timestamp written out as text, so "when" questions get probabilities like every other question.

A record looks like this:

{"state": {"title": "my dog vs the hoover"},
 "questions": {
  "funny_moment": {"type": "moment", "instructions": "When is the funniest moment in this video?"},
  "laugh": {"type": "noul", "instructions": "Can you hear someone laughing in this clip?",
            "criteria": {"true": "yes, someone laughs", "false": "no laughter"}},
  "quality": {"type": "score", "instructions": "Rate how clean and sharp this video looks.",
              "criteria": ["bad", "average", "good", "perfect"]}}}

How we built it

Coeus is a fine-tune, and we want to be exact about what that means.

• Backbone: Qwen3-Omni-30B-A3B from the Qwen team, frozen. It is an open model that already reads video and audio together.
• What we trained: a LoRA adapter (rank 64) on the language model's attention, and Clef's joint answer head, trained from scratch.
• Loss: cross-entropy plus Brier score plus an ordinal distance for score and moment questions. The last two are what keep the probabilities calibrated.
• Data: 26,564 clips from five public tasks, one epoch, about 10 hours on one H200 GPU. Every question was asked in one of three wordings at random, so the model learns the question and not the phrasing.

The full recipe, and the two failures we hit on the way, are in Fine-tuning Qwen3-Omni on video and audio.

Results

Official test splits, each scored once. "Untrained" is the same Qwen3-Omni model asked the same question as text, reading off the probability of each answer.

Task	Metric	Untrained	Coeus	Reference
VideoFeedback, 5 quality questions	mean Spearman	0.340	0.743	VideoScore 0.771
VideoFeedback, real or AI	AUC	0.905	0.996
GenVideo, real or AI, generators never seen in training	AUC		0.942
QVHighlights, find the moment for a text query	R1@0.5	0.528	0.679	Moment-DETR 0.539
FunQA, funny and magic moments	HIT@1	0.504	0.776
VGGSound, 8 people sounds	accuracy	0.503	0.590	chance 0.125

Three results matter most to us:

1. It is calibrated. Expected calibration error is 0.02 to 0.05 on the quality and real-or-AI questions. The untrained model sits at 0.29 to 0.56. Calibrated numbers can be used as thresholds without a person checking each one.
2. It answers questions it was never trained on. Rewording every question in ways it never saw costs nothing (0.728 against 0.730). Asked "Can you hear someone laughing?", a question it was never trained on, it reaches AUC 0.970 against 0.873 untrained.
3. "When" questions work. On FunQA it picks the right funny moment 78% of the time, up from 50%.

A model trained the same way on one task goes further: 0.775 on VideoFeedback, level with VideoScore. On short-video engagement it matches the winner of the 2025 challenge. That result has its own post.

What it does not do well

We would rather you read this here than find it out the hard way.

• Text-query moments: it finds the right moment often (R1), but its ranking of many candidate spans (mAP) is weak.
• Real-or-AI was trained on 2023 to 2024 video generators. Do not rely on it for current ones.
• Accuracy falls as videos get longer. In a 10-minute video it finds the right second 13% of the time (chance is 0.9%).
• Every number comes from one training run, with no confidence intervals yet.

Try it

You need one GPU with 80 GB or more, and ffmpeg.

git clone https://huggingface.co/sp1d5r/video-jev && cd video-jev
python run.py --video clip.mp4 --record questions.json

It answers a clip of up to 60 seconds in 130 to 290 ms on one H200, after the first call.

• Model and model card: huggingface.co/sp1d5r/video-jev
• Code and the full paper: github.com/sp1d5r/vjev

The weights are licensed CC BY-NC-SA 4.0, for non-commercial use, because two of the training sets carry those terms.

Credit

Coeus stands on other people's work. The backbone is Qwen3-Omni by the Qwen team (Apache 2.0). The answer head comes from Cloudflare's Clef (Apache 2.0). The training data comes from VideoFeedback, QVHighlights, FunQA and VGGSound. Please cite them if you use this model.

See every number: the results table, the weak spots and the quick start are on Hugging Face. The training code and the full paper are on GitHub.

More on Coeus:

• What is a Jev model?
• Can an open model spot AI-generated video?
• Can one model predict which short videos people keep watching?
• Fine-tuning Qwen3-Omni on video and audio: our recipe