Refresh RAVEN model card

README.md

@@ -1,174 +1,92 @@
 ---
 license: cc-by-nc-4.0
-library_name: pytorch
-tags:
-  - text-to-video
-  - video-generation
-  - diffusion
-  - autoregressive
-  - consistency-model
-  - grpo
-  - wan2.1
-  - raven
+pipeline_tag: text-to-video
 base_model:
   - Wan-AI/Wan2.1-T2V-1.3B
-pipeline_tag: text-to-video
+tags:
+  - autoregressive
+  - acceleration
+  - streaming
+  - realtime
+  - diffusion
 ---
 
 # RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPO
 
 [Yanzuo Lu](https://yanzuo.lu/) · [Ronglai Zuo](https://2000zrl.github.io/) · [Jiankang Deng](https://jiankangdeng.github.io/) — Imperial College London
 
-Project page: https://yanzuo.lu/raven  
+Project page: <https://yanzuo.lu/raven>
+
+[![arXiv](https://img.shields.io/badge/arXiv-2605.15190-b31b1b.svg)](https://arxiv.org/abs/2605.15190) [![GitHub](https://img.shields.io/badge/GitHub-mvp--ai--lab%2FRAVEN-181717)](https://github.com/mvp-ai-lab/RAVEN)
 
 ## Overview
 
-RAVEN is a causal autoregressive text-to-video generation model built on Wan2.1-T2V-1.3B. It is designed for real-time streaming video generation by extrapolating future video chunks from previously generated content.
+**Wan2.1-T2V-1.3B release.** RAVEN is a causal autoregressive text-to-video model for real-time streaming generation. It extrapolates future chunks from already-generated content, and it is trained with a training-time test framework that aligns training attention with inference-time extrapolation.
 
-The release contains the RAVEN checkpoint plus three interchangeable CM-GRPO variants:
+CM-GRPO is the reinforcement-learning stage on top of RAVEN. It treats a consistency-model sampling step as a conditional Gaussian transition and applies online Group Relative Policy Optimization directly to that kernel.
 
-| File | Description |
-| --- | --- |
-| `raven_model.pt` | Full RAVEN backbone for causal autoregressive text-to-video generation. |
-| `cmgrpo_raven_lora.safetensors` | CM-GRPO LoRA adapter only. Load `raven_model.pt` as the base weight and this file through the LoRA path. |
-| `cmgrpo_raven_full.pt` | RAVEN base and CM-GRPO LoRA adapter packed into one PEFT-wrapped state dict. Load this file through the LoRA path without a separate base weight. |
-| `cmgrpo_raven_merge.pt` | Full CM-GRPO backbone with the adapter already merged into RAVEN. Load this file as the base weight, with no LoRA block. |
+This repository hosts weights only. Training, inference, and evaluation code live in the [code repository](https://github.com/mvp-ai-lab/RAVEN). Other artifacts are listed in the [RAVEN collection](https://huggingface.co/collections/mvp-lab/raven).
 
-RAVEN trains a causal video generator using a training-time test framework that repacks each self rollout into an interleaved sequence of clean historical endpoints and noisy denoising states. This aligns the model's training attention pattern with inference-time autoregressive extrapolation and allows downstream chunk losses to supervise the historical representations used for future predictions.
+## Files
 
-We also release CM-GRPO weights. CM-GRPO formulates a consistency-model sampling step as a conditional Gaussian transition and applies online Group Relative Policy Optimization directly to this kernel.
+| File | Description |
+| --- | --- |
+| `raven_model.pt` | RAVEN backbone (post-distillation, pre-RL). Full `CausalWanModel` state dict. |
+| `cmgrpo_raven_merge.pt` | CM-GRPO backbone with the LoRA already folded into RAVEN. Same key set as `raven_model.pt`. |
+| `cmgrpo_raven_lora.safetensors` | CM-GRPO adapter only. Use together with `raven_model.pt` as the base. |
+| `cmgrpo_raven_full.pt` | PEFT-wrapped bundle of base + adapter, kept as an archival interchange format. The current code cannot load it directly. Convert or merge it offline first. |
 
 ## Model details
 
-- Base architecture: Wan2.1-T2V-1.3B DiT
-- Task: text-to-video generation
-- Generation mode: causal autoregressive video extrapolation
-- Resolution used in released configs: 480 x 832
-- Frames: 81
-- FPS: 16
-- Sampling steps: 4
-- Sampler: consistency sampler
-- Schedule: linear interpolation schedule, `v_lerp` prediction type
-- Classifier-free guidance: not used; the `guidance_scale=3.0` value in the configs is a placeholder for interface compatibility
-- Causal chunking: `chunk_size=3`, `independent_first_chunk=3`, `sink=0`, `window_size=null`
-- VAE stride: `[4, 8, 8]`
-- Latent channels: 16
-- DiT config: dim 1536, 30 layers, 12 heads, FFN dim 8960, text length 512
+| Item | Value |
+| --- | --- |
+| Base model | `Wan-AI/Wan2.1-T2V-1.3B` DiT, causal autoregressive variant |
+| Task / mode | Text-to-video, causal streaming extrapolation |
+| Resolution | 480 x 832 |
+| Frames | 81 |
+| Frame rate | 16 fps |
+| Sampling | 4 NFE with a consistency sampler |
 
 ## Usage
 
-This repository only hosts the released model weights. Please use the RAVEN codebase for inference and evaluation:
+### Setup
 
-```bash
-git clone https://github.com/YanzuoLu/RAVEN.git
+```sh
+git clone https://github.com/mvp-ai-lab/RAVEN.git
 cd RAVEN
-```
-
-Set up the environment:
-
-```bash
 conda env create -f tools/environment.yaml
-conda activate raven
 bash tools/prepare_venv.sh
 source venv/bin/activate
 ```
 
-Download this model repository:
+### Download
 
-```bash
+```sh
 hf download mvp-lab/RAVEN --local-dir /path/to/RAVEN-weights
 ```
 
-Then point the relevant config files to the downloaded checkpoints. RAVEN itself (`raven_model.pt`) is a single full backbone:
-
-```jsonc
-"backbone": {
-    "weight": "/path/to/RAVEN-weights/raven_model.pt"
-}
-```
-
-CM-GRPO can be loaded in any of three equivalent forms:
-
-Adapter only (`cmgrpo_raven_lora.safetensors`):
-
-```jsonc
-"backbone": {
-    "weight": "/path/to/RAVEN-weights/raven_model.pt",
-    "lora": {
-        "enabled": true,
-        "weight": "/path/to/RAVEN-weights/cmgrpo_raven_lora.safetensors"
-    }
-}
-```
-
-Base + LoRA bundle (`cmgrpo_raven_full.pt`):
-
-```jsonc
-"backbone": {
-    "lora": {
-        "enabled": true,
-        "weight": "/path/to/RAVEN-weights/cmgrpo_raven_full.pt"
-    }
-}
-```
-
-Merged backbone (`cmgrpo_raven_merge.pt`):
-
-```jsonc
-"backbone": {
-    "weight": "/path/to/RAVEN-weights/cmgrpo_raven_merge.pt"
-}
-```
-
-The released CM-GRPO configs use the base + LoRA bundle form by default.
-
-Reference configs:
-
-```bash
-configs/trials/generate_t2v/causal_wan2.1_1.3B_t2v/raven_baseline_prompts.jsonc
-configs/trials/generate_t2v/causal_wan2.1_1.3B_t2v/cmgrpo_baseline_prompts.jsonc
-configs/trials/vbench_t2v/causal_wan2.1_1.3B_t2v/raven.jsonc
-configs/trials/vbench_t2v/causal_wan2.1_1.3B_t2v/cmgrpo.jsonc
-```
+### Configure and run
 
-Run qualitative generation:
+Trial YAML files keep site-specific absolute paths (backbone, VAE, text encoder, tokenizer, prompt lists), so edit them or override the keys on the command line before running. The loading schema depends on which checkpoint you use.
 
-```bash
-bash tools/multi_run.sh configs/trials/generate_t2v/causal_wan2.1_1.3B_t2v/raven_baseline_prompts.jsonc
-bash tools/multi_run.sh configs/trials/generate_t2v/causal_wan2.1_1.3B_t2v/cmgrpo_baseline_prompts.jsonc
-```
+- Base or merged checkpoint → `models.backbone.weight.path`.
+- Adapter-only → `raven_model.pt` as the base plus `models.backbone.adapter.weight`, with `r`, `lora_alpha`, and `target_modules` matching the CM-GRPO training trial (`r = lora_alpha = 256`).
+- Legacy `lora: {enabled, weight}` blocks are not part of the current schema and are ignored.
 
-Run VBench prompt-suite sampling:
+```sh
+bash tools/multi_run.sh \
+  projects/wan_t2v/trials/dmd/wan2_1_1_3B/causal_wan_t2v_dmd/val_only/raven_sample100.yaml
 
-```bash
-bash tools/multi_run.sh configs/trials/vbench_t2v/causal_wan2.1_1.3B_t2v/raven.jsonc
-bash tools/multi_run.sh configs/trials/vbench_t2v/causal_wan2.1_1.3B_t2v/cmgrpo.jsonc
+bash tools/multi_run.sh \
+  projects/wan_t2v/trials/dmd/wan2_1_1_3B/causal_wan_t2v_dmd/val_only/cmgrpo_raven_sample100.yaml
 ```
 
-## Requirements
-
-The released configs depend on the RAVEN codebase and the upstream Wan2.1-T2V-1.3B components, including:
-
-- Wan2.1-T2V-1.3B diffusion backbone / DiT config
-- Wan2.1 VAE
-- UMT5-XXL tokenizer and text encoder
-- Python 3.10
-- CUDA 12.8
-- PyTorch 2.11 + cu128
-- flash-attention 2/3 and magi-attention as built by `tools/prepare_venv.sh`
-
-See the code repository README for full setup and evaluation instructions.
-
 ## License
 
-This model is released under the Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0). See the `LICENSE` file in the code repository for details.
-
-The upstream Wan2.1 components are subject to their own licenses and terms. Users are responsible for complying with all applicable licenses for the base model, code, data, and dependencies.
+These weights are released under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/). The upstream Wan2.1 components (DiT initialization, VAE, UMT5-XXL text encoder) remain subject to their own licenses and terms. You are responsible for complying with all applicable licenses for the base model, code, data, and dependencies.
 
 ## Citation
 
-If you find this work useful, please cite RAVEN. A BibTeX entry will be added when available.
-
 ```bibtex
 @article{lu2026raven,
   title = {RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPO},