Fine-tuning Qwen3-Omni on video and audio: our recipe

Coeus, our open Video Jev model, is a fine-tune of Qwen3-Omni. We did not build a foundation model. We trained a LoRA adapter and Cloudflare's Clef answer head on top of Qwen3-Omni-30B-A3B, and left the rest frozen.

This post is the practical recipe. It covers the settings we ended on, and the two mistakes that cost us about $46 of GPU time before we found them.

The code and the model: every training script and the full experiment log are on GitHub. The adapter, the head and the results table are on Hugging Face.

Why Qwen3-Omni

Qwen3-Omni reads video frames and the clip's sound on one timeline. For questions like "can you hear someone laughing?" or "when is the funny moment?", the sound matters.

It is also open (Apache 2.0) and a mixture of experts: 30B parameters in total, about 3B active per token. Running it in bf16 needs one 80 GB GPU. We trained on one H200.

The recipe that worked

Setting	Value
Backbone	Qwen3-Omni-30B-A3B-Instruct thinker, bf16, talker off, frozen
Adapter	LoRA rank 64, alpha 128, dropout 0.05, on q/k/v/o of the language model
Head	Clef joint schema head, trained from scratch, in fp32
Loss	Cross-entropy (label smoothing 0.05) + Brier score + ordinal earth mover's distance
Optimiser	AdamW, weight decay 0.01
Learning rate	LoRA 3e-5, head 1e-4, warmup 50 steps, cosine to 0
Batch	1 clip, 8-step gradient accumulation, gradient clip 1.0
Data	26,564 clips, 1 epoch, 3,348 steps
Hardware	One H200, about 10 hours

We chose these settings on one task (SnapUGC engagement) and reused them unchanged on every other task.

Step 1: load only what you train

Qwen3-Omni ships with a "talker" that speaks answers out loud. We only need the "thinker", the part that reads. We also do not need its 152k-way vocabulary output, because the head reads hidden states.

full = Qwen3OmniMoeForConditionalGeneration.from_pretrained(
    path, dtype=torch.bfloat16, device_map={"": "cuda"}, attn_implementation="sdpa")
full.disable_talker()
thinker = full.thinker
thinker.requires_grad_(False)
thinker.gradient_checkpointing_enable(gradient_checkpointing_kwargs={"use_reentrant": False})
thinker.lm_head = torch.nn.Identity()  # the head reads hidden states, not logits

# Text attention only. The regex is a full match, so the audio tower is left alone.
targets = r"model\.layers\.\d+\.self_attn\.(q|k|v|o)_proj"
thinker = get_peft_model(thinker, LoraConfig(r=64, lora_alpha=128, lora_dropout=0.05,
                                              target_modules=targets))

Pass use_audio_in_video=True to the processor and the forward call, so frames and sound are interleaved on one timeline.

Step 2: decode every clip once

In our first baseline, decoding the video took about 3.5 times as long as the forward pass. A GPU that waits on ffmpeg is an expensive GPU.

So we decode each clip once, before training, into JPEG frames and 16 kHz mono audio, and save them as small files. For engagement we kept the first 16 seconds at 2 frames per second, with the longest side at 448 pixels. Training then reads those files and never touches an mp4.

What went wrong 1: two runs went NaN

Our first two long runs both hit a NaN loss near step 720, about 2.5 hours in. A NaN step corrupts the weights for good, so both runs were lost. That cost about $23.

The likely cause was the head running in bf16, like the backbone. The fix had two parts:

• Run the head in fp32. The 30B thinker stays in bf16.
• Skip any step whose loss or gradient norm is not finite, and count the skips in your logs.

hidden = thinker(..., use_audio_in_video=True).logits   # bf16, under autocast
probs = head(hidden.float(), ...)                       # fp32

if torch.isfinite(loss):
    (loss / accum).backward()
...
grad_norm = torch.nn.utils.clip_grad_norm_(params, 1.0)
if torch.isfinite(grad_norm):
    opt.step()

None of our later engagement runs skipped a step.

What went wrong 2: the learning rate collapsed a long run

Our pilot used a LoRA rate of 1e-4 (head 3e-4) on 10,000 clips and worked well. So we ran the same settings on all 30,631 clips. It collapsed.

Step	LoRA 1e-4 (stopped)	LoRA 3e-5 (kept)
400	0.437	0.557
800	0.185	0.641
1,200	0.238	0.682
1,600	0.259	0.672
2,000	0.213	0.700
2,400	0.261	0.704
3,200	-	0.718

Val final score (mean of Spearman and Pearson correlation), SnapUGC engagement, the same 500 val clips, one seed. Higher is better.

We stopped the first run at step 2,440 under a rule we set before it started. That was about $23 lost.

The surprising part was the sweep we ran next. Over 250 steps, the highest rate learned fastest. Lower rates looked slower, not safer.

The failure only shows up after 400 to 800 steps while the LoRA rate stays above about 7e-5. Short runs survive because their cosine schedule decays quickly. A full epoch keeps the rate high for about 2,000 steps, and that is where it breaks.

The fix was a lower peak: LoRA 3e-5, head 1e-4. An 800-step test passed, then the full run trained cleanly for the whole epoch. It scored 0.709 on the untouched test split.

The low-rate end of the schedule also mattered. Steps 2,400 to 3,200 added 0.014 to the score and lowered the Brier score from 0.821 to 0.808.

What did not help

• LoRA on the vision tower. At 500 val clips it scored 0.663 against 0.677 without it, and trained 20% slower. It did give the lowest calibration error (0.023), so it may be worth another look.
• Audio, for some tasks. Audio added 6.4 points on VGGSound, where the answer is in the sound. On FunQA it added little, and on engagement the gain was inside the noise. Turn it off if your question is purely visual: video-only training was about 1.8 times faster.

What it got us

One joint model, trained with this recipe on five public tasks, beats the untrained Qwen3-Omni on every task, by 0.09 to 0.40. Its calibration error on quality questions fell from 0.29 to 0.56 untrained to 0.031 to 0.045. The full results, with splits and baselines, are in the Coeus post.

The checks we now run first

1. Run the head in fp32 and skip non-finite steps from day one.
2. Decode every clip once, before training.
3. Test new settings on 2,000 clips (about $4) before any run over about $20.
4. Then run a medium test long enough to pass step 800 at the peak rate. A 250-step sweep cannot show a late collapse.
5. Set the stop rule before the run starts, and write it down.

Run it yourself: the training code, the experiment log and the paper are on GitHub. The weights and the quick start are on Hugging Face, for non-commercial use.

More on Coeus:

• Coeus, our open Video Jev model: what it does and every benchmark.
• What is a Jev model?
• Can an open model spot AI-generated video?
• Can one model predict which short videos people keep watching?