Video moment retrieval: finding the funniest second

Most video models answer "what" questions. "Is this clip funny?" "Is it AI-generated?" For editing, the more useful question is "when". When is the funny moment? When does the man open the fridge?

This is called moment retrieval. Coeus, our open Video Jev model, answers it with a probability for each second of the video. Coeus is a fine-tune: a LoRA adapter and Cloudflare's Clef answer head, trained on top of Qwen3-Omni-30B-A3B. We did not build a foundation model.

This post covers how the moment question works, and what it scored on two public benchmarks, FunQA and QVHighlights.

The code and the model: the moment code, the experiment log and the paper are on GitHub. The weights and every result, with its split and baseline, are on Hugging Face.

Two kinds of "when"

There are two kinds of moment question in this post.

1. A moment you describe. "When does a man open the fridge?" The answer is in the words. QVHighlights tests this.
2. A moment you judge. "When is the funniest moment?" Nobody tells the model what happens. It has to decide what is funny. FunQA tests this, with funny, creative and magic clips.

How a moment question works

A Jev model answers typed questions: yes/no, choice, ordered score. The Jev post explains the idea. We added a fourth type, moment.

For a moment question, each second of video is an answer option. Qwen3-Omni already places a clip's video tokens in short blocks, with each block's audio tokens next to it. So each option's evidence is that second's own video and audio tokens. The head scores every second against the question, and the answer is a probability for each second.

There is no separate detector and no timestamp written as text. The head itself did not change.

From one distribution you can take:

• the top second,
• a span around the peak (every second at 50% or more of the peak's probability),
• or the top few candidates, with how sure the model is about each.

A model that writes "00:05 to 00:07" as text gives one answer and no confidence. This one gives the whole curve.

In training, the target for each second is weighted by how much it overlaps the true answer window, or by human highlight scores where the dataset has them.

How we scored it

• HIT@1: the top second falls inside the true moment.
• R1@0.5: the span around the peak overlaps the true moment by at least 50% (IoU 0.5). R1@0.7 is the same at 70%.
• mIoU: the average overlap of that span with the true moment.
• mAP: how well the model ranks many candidate spans. Dedicated moment models are built for this.

We always compare against content-free guesses on the same clips. On short clips a guess can score well, and that matters below.

FunQA: the funny moment

FunQA has short clips of pranks, tricks and magic, each with a marked answer window. One question per clip, worded as in FunQA ("Identify the video's funny moment."), at 2 frames per second, up to 60 s.

First, a single-task run on the funny, creative and magic clips: 3,072 train videos, scored on 400 val videos.

Method (val, 400 videos)	HIT@1	R1@0.5	mIoU
Uniform guess	0.435	0.320	0.380
Position prior (the average answer position)	0.533	0.135	0.190
Untrained Qwen3-Omni, asked for start and end times	0.545	0.295	0.350
Coeus, video only	0.792	0.580	0.537
Coeus, video and audio	0.807	0.580	0.538

That is 27 points of HIT@1 and 26 points of R1@0.5 above the best content-free guess.

Audio added almost nothing here: every gap was within 0.015, which is inside the noise at 400 clips. FunQA moments are visual. A prank is something you see. On VGGSound, where the answer is in the sound, audio added 6.4 points.

Then the official test split, scored once, on the funny and magic clips (339):

Model (test, 339 videos)	HIT@1	R1@0.5
Untrained Qwen3-Omni	0.504	0.398
Single-task model	0.767	0.507
Coeus (one joint model, five tasks)	0.776	0.504

The joint model answers four other tasks with the same weights, and matched the single-task model on FunQA. We have not compared against published FunQA timestamp numbers yet, because their metric definitions differ.

The subset that measured nothing

We held out FunQA's creative clips, to test a question family the joint model never trained on. It scored 0.000 HIT@1. The untrained model scored 0.635.

Then we looked at the answers. Every creative answer window in FunQA starts within the first 3 seconds. A model that always says "the start" wins, and the single-task model, trained on them, scored 1.000. So that test measures position, not content, and we call it void.

The lesson: before you trust a moment benchmark, score a position prior on it.

QVHighlights: the moment you describe

QVHighlights pairs videos of about 150 s with text queries that describe what happens in one moment. Its test labels are withheld, so its official val split (1,550 queries) is our test.

The first run failed. With 2,000 training queries, the model was far below the untrained backbone on spans: R1@0.5 0.264 against 0.539 on val. Its curve was too flat, so the span around the peak was too wide. The untrained Qwen3-Omni was already good at this. It was likely trained on grounding data.

With all 6,862 training queries, inside the joint model, it was well above:

Model (val split, 1,550 queries)	R1@0.5	R1@0.7	mIoU	HIT@1	mAP
Untrained Qwen3-Omni	0.528	0.354	0.515	0.181	0.043
Coeus	0.679	0.463	0.601	0.358	0.055
Moment-DETR (published, same split)	0.539	0.348		0.556	0.322

R1 is above Moment-DETR's published result on the same split. Newer dedicated models report more than Moment-DETR, and on HIT@1 and mAP Coeus is well below even Moment-DETR. In plain words: it often finds the right span, but it ranks the other candidate spans and single seconds poorly. Text-query moments need data, and ranking needs more work.

What the output looks like

This is real output from the released run.py, on a 7 s FunQA test clip whose funny moment runs from 0 to 6.7 s:

{"funny_moment": [{"from_s": 6.0, "to_s": 7.0, "p": 0.181},
                  {"from_s": 5.0, "to_s": 6.0, "p": 0.18},
                  {"from_s": 4.0, "to_s": 5.0, "p": 0.171}, ...]}

Almost the whole clip is the answer here, so the curve is nearly flat, and that is the right shape. On a longer clip with a short moment, the mass gathers around the peak.

You ask for it with one line in the question record:

{"questions": {
  "funny_moment": {"type": "moment", "instructions": "When is the funniest moment in this video?"}}}

What it does not do well

• Ranking. mAP on QVHighlights is 0.055, against 0.322 for Moment-DETR.
• Long video. It reads 60 s windows. Hidden in a 10-minute video, one moment is found 13% of the time (chance is 0.9%).
• Data. Text-query moments only worked with the full training set.
• One seed. No confidence intervals on these numbers yet.
• Licence. The weights are for non-commercial use (CC BY-NC-SA), like the FunQA and QVHighlights data.

See every number: the moment results, the baselines and the quick start are on Hugging Face. The moment code, the experiment log and the paper are on GitHub.

More on Coeus:

• Coeus, our open Video Jev model: what it does and every benchmark.
• What is a Jev model?
• Fine-tuning Qwen3-Omni on video and audio: our recipe
• Can an open model spot AI-generated video?
• Can one model predict which short videos people keep watching?