Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,68 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
library_name: peft
|
| 3 |
+
base_model: Wan-AI/Wan2.1-I2V-14B-480P
|
| 4 |
+
tags:
|
| 5 |
+
- video-generation
|
| 6 |
+
- robot-manipulation
|
| 7 |
+
- world-model
|
| 8 |
+
- lora
|
| 9 |
+
- videox-fun
|
| 10 |
+
license: apache-2.0
|
| 11 |
+
---
|
| 12 |
+
|
| 13 |
+
# Wan2.1 LoRA SFT for Robot Video Generation
|
| 14 |
+
|
| 15 |
+
Fine-tuned from Wan2.1-I2V-14B-480P using VideoX-Fun framework on RoboTwin dataset.
|
| 16 |
+
|
| 17 |
+
Part of the [WorldArena World Model Challenge](https://github.com/cjgogo27/worldarena-wm).
|
| 18 |
+
|
| 19 |
+
## Training Details
|
| 20 |
+
|
| 21 |
+
| Parameter | Value |
|
| 22 |
+
|-----------|-------|
|
| 23 |
+
| Framework | VideoX-Fun |
|
| 24 |
+
| Base Model | Wan2.1-I2V-14B-480P |
|
| 25 |
+
| Training Data | RoboTwin aloha-agilex_clean_50 (2,500 videos, 50 tasks) |
|
| 26 |
+
| LoRA Rank | 32 |
|
| 27 |
+
| LoRA Alpha | 16 |
|
| 28 |
+
| LoRA Targets | q, k, v, ffn.0, ffn.2 |
|
| 29 |
+
| Learning Rate | 1e-4 (constant with warmup) |
|
| 30 |
+
| Warmup Steps | 100 |
|
| 31 |
+
| Precision | bf16 |
|
| 32 |
+
| Resolution | 640 × 640 |
|
| 33 |
+
| Frames | 81 per video |
|
| 34 |
+
| Batch Size | 1 per GPU (2 GPUs) |
|
| 35 |
+
| Total Steps | 12,500 (planned), 2,200 (completed) |
|
| 36 |
+
|
| 37 |
+
## Checkpoints Available
|
| 38 |
+
|
| 39 |
+
| File | Training Steps | Description |
|
| 40 |
+
|------|---------------|-------------|
|
| 41 |
+
| checkpoint-1200.safetensors | 1,200 | ~1 epoch of training |
|
| 42 |
+
| checkpoint-2200.safetensors | 2,200 | Latest checkpoint, used for 1000-video inference |
|
| 43 |
+
|
| 44 |
+
Each checkpoint has a ComfyUI-compatible version.
|
| 45 |
+
|
| 46 |
+
## Usage
|
| 47 |
+
|
| 48 |
+
Inference with VideoX-Fun:
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
## Evaluation Results (WorldArena Track 1, 10 samples)
|
| 53 |
+
|
| 54 |
+
| Metric | Baseline Wan2.1 | SFT-Wan2.1 (ckpt300) |
|
| 55 |
+
|--------|----------------|---------------------|
|
| 56 |
+
| Image Quality | 70.44 | 48.80 |
|
| 57 |
+
| Background Consistency | 69.58 | **90.40** |
|
| 58 |
+
| Subject Consistency | 59.90 | **82.33** |
|
| 59 |
+
| Flow Score | 16.18 | **21.29** |
|
| 60 |
+
|
| 61 |
+
SFT-Wan2.1 shows significant improvements in background consistency (+20.8), subject consistency (+22.4), and flow score (+5.1).
|
| 62 |
+
|
| 63 |
+
## Limitations
|
| 64 |
+
|
| 65 |
+
- Intended for research on robot video world models
|
| 66 |
+
- May fail on long-horizon manipulation and complex physical contact
|
| 67 |
+
- Instruction following accuracy still limited (~54%)
|
| 68 |
+
- Full 1,000-video WorldArena evaluation in progress
|