File size: 5,180 Bytes
a0325d0 6fa2496 a0325d0 6fa2496 a0325d0 6fa2496 a0325d0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 | ---
license: apache-2.0
base_model: Qwen/Qwen3.5-4B
library_name: transformers
pipeline_tag: video-text-to-text
tags:
- streaming-video
- video-understanding
- on-policy-distillation
- knowledge-distillation
- qwen3_5
- arxiv:2608.16320
language:
- en
---
# StreamOPD-4B-ST-CueGate
A 4B streaming video-understanding model, post-trained from **Qwen3.5-4B** with on-policy
distillation from a **Qwen3.5-9B** teacher under **ST-CueGate**, the spatio-temporal cue
gating method from *StreamOPD: A Post-Training Recipe with Spatio-Temporal Cue Gating for
Streaming Video Understanding*.
- **Paper:** https://arxiv.org/abs/2608.16320
- **Code:** https://github.com/UniX-AI-Lab/StreamOPD
- **Project page:** https://unix-ai-lab.github.io/StreamOPD/
## What is different about this model
The model is trained for a deliberately austere streaming setting: at answer time it sees
only the **4 most recent frames at 1 fps** — no memory bank, no retrieval, no KV-cache
compression, and no generated reasoning trace. All of the capability lives in the weights
rather than in test-time machinery.
Training uses on-policy distillation in *thinking* mode, while deployment is in *instruct*
mode. During training the frozen teacher scores the student's own response twice — once
conditioned on a grounded spatio-temporal cue and once without it — and the resulting
per-token likelihood ratio is aggregated into a response-level weight that gates the
distillation advantage. The cue is a training-time signal only: it is never present at
inference, so the deployment path is identical to a plain Qwen3.5-4B.
## Results
Evaluated in instruct mode with greedy decoding under the recent-4-frame protocol.
StreamingBench and OVO-Bench use recent-4 frames at 1 fps; Video-MME and LongVideoBench use
their standard protocols with at most 32 frames.
| Model | StreamingBench | OVO-Bench (excl. HLD) | Video-MME | LongVideoBench |
|-------|:---:|:---:|:---:|:---:|
| Qwen3.5-4B (untrained) | 77.87 | 59.94 | 64.22 | 57.74 |
| Qwen3.5-9B (teacher) | 84.15 | — | — | — |
| **This checkpoint** | **84.19** | **70.48** | **64.85** | **60.36** |
The full comparison, including per-subtask OVO-Bench breakdowns, the teacher-conditioning
ablations, and the self-distillation variant, is in the paper.
## Usage
```python
from transformers import AutoProcessor, Qwen3_5ForConditionalGeneration
model_id = "UniX-Lab/StreamOPD-4B-ST-CueGate"
model = Qwen3_5ForConditionalGeneration.from_pretrained(
model_id, torch_dtype="bfloat16", device_map="auto"
).eval()
processor = AutoProcessor.from_pretrained(model_id)
messages = [{
"role": "user",
"content": [
{"type": "video", "video": "clip.mp4"},
{"type": "text", "text": "What did the person pick up?\nA. book B. cup C. apple\n"
"Only give the best option's letter directly."},
],
}]
inputs = processor.apply_chat_template(
messages, add_generation_prompt=True, tokenize=True,
return_dict=True, return_tensors="pt",
).to(model.device)
out = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```
Run with **thinking disabled** — the model is trained to answer directly, and the reported
scores use instruct-mode greedy decoding. Set `FORCE_QWENVL_VIDEO_READER=decord` for video
decoding.
To reproduce the streaming evaluation exactly, use the evaluators in the code repository,
which implement the recent-window protocol and the OVO-Bench B+R macro:
```bash
bash scripts/eval/run_all.sh <path-to-this-model> my_run 0,1,2,3
bash scripts/eval/score_all.sh my_run
```
## Training
| | |
|---|---|
| Student | Qwen3.5-4B |
| Teacher | Qwen3.5-9B (frozen) |
| Objective | on-policy distillation, sampled-token k1 reverse KL via policy gradient |
| Gating | ST-CueGate, α=0.5, gate range [0, 2], UID sibling normalization, response-level |
| Rollouts | n=4 |
| Data | 25,118 verifiable video QA items (multiple choice / binary / counting) |
| Mode | trained in thinking mode, deployed in instruct mode |
Training data is derived from public sources (LLaVA-Video-178K and Kinetics-700 clips); the
data pipeline, the parquets, and the cue-generation tooling are in the code repository.
## Limitations
- Answers come from a 4-frame window, so questions needing evidence outside that window are
out of scope by construction.
- Distillation from a larger teacher reduces the willingness to abstain on unanswerable
queries (OVO-Bench's HLD subtask) relative to the untrained base model. The paper reports
a self-distillation variant that recovers this.
- Trained and evaluated on English data.
## Citation
```bibtex
@article{wu2026streamopd,
title = {StreamOPD: A Post-Training Recipe with Spatio-Temporal Cue Gating
for Streaming Video Understanding},
author = {Wu, Keming and Wang, Baoyi and Zhang, Kaichen and An, Xiang and
Yang, Zuhao and Wang, Sudong and Zhu, Haowei and Huang, Tingxuan and
Gao, Hongcheng and Wang, Bin},
journal = {arXiv preprint arXiv:2608.16320},
year = {2026}
}
```
|