| --- |
| base_model: Qwen/Qwen3-Omni-30B-A3B-Instruct |
| library_name: transformers |
| tags: |
| - qwen3-omni |
| - audio |
| - video |
| - multimodal |
| - temporal-grounding |
| - dense-video-captioning |
| - reinforcement-learning |
| --- |
| |
| # AVTime-RL |
|
|
| `AVTime-RL-BiTSC-GRPO-v1` is a merged, bf16 milestone for time-aware |
| audio-video understanding. It starts from `Qwen/Qwen3-Omni-30B-A3B-Instruct`, |
| uses one-round AVTime/LongVALE supervision, and is then optimized with |
| multi-task GRPO for timestamped dense video captioning, temporal video |
| grounding, and segment captioning. |
|
|
| ## Milestone identity |
|
|
| | Field | Value | |
| | --- | --- | |
| | Release | AVTime-RL-BiTSC-GRPO-v1 | |
| | Checkpoint | merged checkpoint 60 | |
| | Precision | bfloat16 | |
| | Serialization | 15 safetensors shards, about 66 GB total | |
| | Architecture | `Qwen3OmniMoeForConditionalGeneration` | |
| | Release date | 2026-08-11 | |
|
|
| This is a fully merged checkpoint; no LoRA adapter needs to be applied at |
| inference time. Use a recent `transformers` or vLLM build that supports |
| Qwen3-Omni MoE models and provide both video and audio when available. |
|
|
| ## Training |
|
|
| The GRPO mixture contains 12,484 examples: 70.0% dense video captioning (DVC), |
| 15.0% temporal video grounding (TVG), and 15.0% segment captioning (SC). |
| It combines 7,239 LongVALE DVC examples, 1,500 AVTime DVC examples, 1,872 |
| LongVALE TVG examples, and 1,873 LongVALE SC examples. |
|
|
| The main DVC reward is BiTSC (bidirectional time-semantic consistency): a |
| local RLAIF judge matches captions to events, while deterministic code measures |
| the resulting temporal overlap in both Time-to-Event and Event-to-Time |
| directions. TVG uses official-format parsing plus temporal IoU and boundary |
| proximity; SC uses deterministic lexical coverage and length calibration. |
|
|
| ## LongVALE evaluation |
|
|
| The following results use the LongVALE test split (1,171 videos and 13,867 |
| events) and the official metric implementation. |
|
|
| | Task | Metric | Score | |
| | --- | ---: | ---: | |
| | Omni-TVG | R@0.3 | 26.13 | |
| | Omni-TVG | R@0.5 | 16.85 | |
| | Omni-TVG | R@0.7 | 8.71 | |
| | Omni-TVG | mIoU | 17.76 | |
| | Omni-SC | BLEU-4 | 7.79 | |
| | Omni-SC | ROUGE-L | 24.84 | |
| | Omni-SC | CIDEr | 31.28 | |
| | Omni-SC | METEOR | 13.20 | |
|
|
| On the clean 512-video DVC evaluation using the historical one-round |
| generation protocol, the model achieves SODA-C 5.057, METEOR 7.033, and |
| CIDEr 12.156. |
|
|
| For TVG, evaluation used constrained decoding to enforce LongVALE's official |
| `From XX to YY` response syntax. Fair comparisons should use the same output |
| constraint for every compared model; this checkpoint should not be interpreted |
| as a claim about unconstrained free-form timestamp formatting. |
|
|
| ## Limitations and use |
|
|
| This is a research checkpoint, not a safety-validated production system. Its |
| timestamps and captions can be inaccurate, especially for subtle audio events, |
| very long videos, or videos outside the training distribution. The checkpoint |
| does not include any LongVALE or AVTime media or annotations. Users must |
| comply with the terms of the upstream base model and any input datasets. |
|
|
|
|
