robbyant/lingbot-video-rewriter-lora

LingBot-Video

🌐 Project Page | 🤗 Hugging Face | 🤖 ModelScope | 📄 Paper | ⚖️ License

We are excited to introduce LingBot-Video, the first open-source large-scale MoE (Mixture-of-Experts) video generation model dedicated to embodied intelligence. As a top-tier video model, LingBot-Video is designed to bridge the gap between video synthesis and physical world understanding.

🔥 Key Highlights

• 🚀 Efficient MoE Architecture: Scaled from scratch; balanced between capacity and cost with ~3x faster inference.
• 📦 Data Engine: Trained on massive web videos integrated with 70,000+ hours of embodied data.
• ⚖️ Multi Reward System: Rewarded for high aesthetics, physical rationality, and task completion.

🔥 Latest News

• July 9, 2026: 🎉 We release the technical report, code, models, rewriters for LingBot-Video.

📦 Model Download

Model Name	Components	Tasks	Download
⚡ LingBot-Video-Dense	Dense (1.3B)	T2I, T2V, TI2V	🤗 Huggingface   🤖 ModelScope
💪 LingBot-Video-MoE	MoE (30B-A3B) + Refiner	T2I, T2V, TI2V, Refinement	🤗 Huggingface   🤖 ModelScope
📝 LingBot-Video-Rewriter-Base	Qwen3.6-27B official	Prompt rewriter (Expand)	🤗 Huggingface   🤖 ModelScope
📝 LingBot-Video-Rewriter-Adapter	Qwen3.6-27B LoRA	Prompt rewriter (Json)	🤗 Huggingface   🤖 ModelScope

🚀 Quick Start

🛠️ Installation

The root requirements.txt includes the recommended PyTorch build for LingBot-Video inference.

git clone https://github.com/Robbyant/lingbot-video
cd lingbot-video

python -m venv .venv
source .venv/bin/activate
python -m pip install -U pip

# Base requirements cover direct DiT inference and rewriter --backend transformers.
pip install -r requirements.txt
pip install -e .

💡 Rewriter deployment: the bundled rewriter uses the single-process transformers backend. For higher throughput, deploy the VLM yourself and call it through an OpenAI-compatible API. Preserve the two-stage semantics: step 1 must use the base VLM without the rewriter LoRA, while step 2 must use the same base VLM with the rewriter LoRA enabled. This can be implemented with two endpoints, or with one server that can select the adapter per request. See vLLM / SGLang official docs.

Install the optional SGLang dependencies only when using SGLang Diffusion or the fused / FP8 MoE runtime:

python -m pip install --no-deps -r requirements-sglang.txt

Recommended runtime versions:

Package	Version
Python	>=3.10
torch	2.12.0.dev20260220+cu130 (recommended)
torchvision	0.26.0.dev20260220+cu130 (recommended)
transformers	5.8.1
diffusers	0.39.0
peft	0.19.1
json_repair	>=0.30
decord	>=0.6.0
safetensors	>=0.4.5

🎬 Inference

🧭 Recommended Inference Workflow

LingBot-Video DiT inference is designed to consume structured JSON captions, not casual natural-language prompts. The recommended public workflow is:

1. Rewrite the user's plain prompt with Prompt Rewriter. For TI2V, pass the same first frame to the rewriter.
2. Run Auto Negative by default to prune the negative prompt for this specific caption.
3. Run the unified inference runner with --prompt_json and select direct diffusers or SGLang Diffusion through --backend.

Backend choices:

• diffusers: direct diffusers reference path.
• sglang: SGLang Diffusion path. If the optional SGLang package is not installed, it automatically falls back to direct diffusers and prints a warning. Install requirements-sglang.txt to enable the SGLang runtime.

For multi-GPU inference, add --enable_fsdp_inference to shard the base DiT and refiner DiT on GPU. This reduces GPU memory pressure after loading, but each rank still constructs the transformer on host memory before FSDP sharding; make sure the machine has enough system RAM for large MoE checkpoints.

# Model root (released Dense or MoE package) and rewriter weights.
export MODEL_DIR="<path_to_lingbot-video-model>"
export REWRITER_BASE_MODEL="<path_to_rewriter_base_vlm>"
export REWRITER_ADAPTER="<path_to_rewriter_lora>"

python rewriter/inference.py --backend transformers --mode t2v \
  --prompt "<plain_user_prompt>" --duration 5 --output prompt.json

# Recommended Auto Negative block. If skipped, remove --negative_prompt_json from
# the DiT inference command.
python rewriter/auto_negative.py --backend transformers --mode t2v \
  --caption prompt.json --output negative.json

export BACKEND=diffusers  # or: sglang

python scripts/inference.py \
  --backend "$BACKEND" \
  --model_dir "$MODEL_DIR" \
  --run_refiner \
  --mode t2v \
  --prompt_json prompt.json \
  --negative_prompt_json negative.json \
  --output "<output_dir>/base.mp4" \
  --refiner_output "<output_dir>/refined.mp4" \
  --height 480 \
  --width 832 \
  --fps 24 \
  --steps 40 \
  --refiner_steps 8 \
  --guidance_scale 3 \
  --refiner_guidance_scale 3 \
  --shift 3 \
  --refiner_shift 3 \
  --transformer_dtype bf16 \
  --text_encoder_dtype bf16 \
  --vae_dtype fp32 \
  --refiner_vae_dtype fp32 \
  --reuse_condition_features

Ready-to-run scripts are provided for single-GPU and multi-GPU inference. Set your environment and model path first:

source .venv/bin/activate
export PYTHON_BIN=python
export DENSE_MODEL_DIR="<path_to_lingbot-video-dense>"
export MOE_MODEL_DIR="<path_to_lingbot-video-moe>"

Single-GPU scripts use direct diffusers and batched CFG by default. They run base generation only.

MODEL_DIR="$DENSE_MODEL_DIR" ./scripts/single-gpu/run_dense_t2i.sh
MODEL_DIR="$DENSE_MODEL_DIR" ./scripts/single-gpu/run_dense_t2v.sh
MODEL_DIR="$DENSE_MODEL_DIR" ./scripts/single-gpu/run_dense_ti2v.sh

MODEL_DIR="$MOE_MODEL_DIR" ./scripts/single-gpu/run_moe_t2i.sh
MODEL_DIR="$MOE_MODEL_DIR" ./scripts/single-gpu/run_moe_t2v.sh
MODEL_DIR="$MOE_MODEL_DIR" ./scripts/single-gpu/run_moe_ti2v.sh

Multi-GPU no-refiner scripts use the same inference arguments as the single-GPU scripts, plus CP8 and FSDP:

MODEL_DIR="$DENSE_MODEL_DIR" ./scripts/multi-gpus-no-refiner/run_dense_t2i_fsdp_cp8.sh
MODEL_DIR="$DENSE_MODEL_DIR" ./scripts/multi-gpus-no-refiner/run_dense_t2v_fsdp_cp8.sh
MODEL_DIR="$DENSE_MODEL_DIR" ./scripts/multi-gpus-no-refiner/run_dense_ti2v_fsdp_cp8.sh

MODEL_DIR="$MOE_MODEL_DIR" ./scripts/multi-gpus-no-refiner/run_moe_t2i_fsdp_cp8.sh
MODEL_DIR="$MOE_MODEL_DIR" ./scripts/multi-gpus-no-refiner/run_moe_t2v_fsdp_cp8.sh
MODEL_DIR="$MOE_MODEL_DIR" ./scripts/multi-gpus-no-refiner/run_moe_ti2v_fsdp_cp8.sh

Multi-GPU refiner scripts use CP8 + FSDP + batched CFG by default. They also default to direct diffusers; set BACKEND=sglang externally when you want to exercise SGLang Diffusion. MoE multi-GPU T2V/TI2V scripts additionally run the refiner.

MODEL_DIR="$DENSE_MODEL_DIR" ./scripts/multi-gpus/run_dense_t2i_fsdp_cp8.sh
MODEL_DIR="$DENSE_MODEL_DIR" ./scripts/multi-gpus/run_dense_t2v_fsdp_cp8.sh
MODEL_DIR="$DENSE_MODEL_DIR" ./scripts/multi-gpus/run_dense_ti2v_fsdp_cp8.sh

MODEL_DIR="$MOE_MODEL_DIR" ./scripts/multi-gpus/run_moe_t2i_fsdp_cp8.sh
MODEL_DIR="$MOE_MODEL_DIR" ./scripts/multi-gpus/run_moe_t2v_refiner_fsdp_cp8.sh
MODEL_DIR="$MOE_MODEL_DIR" ./scripts/multi-gpus/run_moe_ti2v_refiner_fsdp_cp8.sh

All scripts accept the same environment overrides, such as PROMPT_JSON, IMAGE, OUT_DIR, HEIGHT, WIDTH, STEPS, GUIDANCE_SCALE, SHIFT, SEED, FPS, BACKEND, and PYTHON_BIN. Refiner scripts also accept REFINER_HEIGHT, REFINER_WIDTH, REFINER_STEPS, REFINER_GUIDANCE_SCALE, REFINER_SHIFT, REFINER_T_THRESH, and REFINER_SIGMA_TAIL_STEPS. MoE scripts default to grouped expert execution (LINGBOT_MOE_EXPERT_BACKEND=grouped_mm).

See English Docs or 中文文档 for the detailed prompt rewrite, auto-negative, TI2V, base-only/refiner, distributed SGLang, and speed-first FP8 workflows.

📊 Benchmarks

🏛️ Public Benchmark

As of July 9th, 2026, LingBot-Video ranks top in RBench Leaderboard.

Models	Open-source	Avg.	Manip.	Spatial	Multi-entity	Long-hor.	Reasoning	Single arm	Dual arm	Quadruped	Humanoid
LingBot-Video (Ours)	✅	0.620	0.578	0.643	0.444	0.634	0.505	0.636	0.639	0.758	0.689
Cosmos3 Super	✅	0.581	0.487	0.642	0.444	0.591	0.395	0.615	0.623	0.739	0.691
LongCat-Video	✅	0.437	0.372	0.310	0.220	0.384	0.186	0.586	0.576	0.681	0.621
Wan 2.2 A14B	✅	0.507	0.381	0.454	0.373	0.501	0.330	0.608	0.582	0.690	0.648
HunyuanVideo 1.5	✅	0.460	0.442	0.316	0.312	0.438	0.364	0.513	0.526	0.634	0.595
Wan 2.6	❌	0.607	0.546	0.656	0.479	0.514	0.531	0.666	0.681	0.723	0.667
Seedance 1.5 pro	❌	0.584	0.577	0.495	0.484	0.570	0.470	0.648	0.641	0.680	0.692
Veo 3	❌	0.563	0.521	0.508	0.430	0.530	0.504	0.634	0.610	0.689	0.637

Note: Bold indicates the best performance, and underline indicates the second best.

⚖️ License

This project is licensed under the Apache 2.0 License. Please refer to the LICENSE file for the full text, including details on rights and restrictions.

📚 Citation

If you find this work useful for your research, please cite our paper:

@article{lingbot-video,
  title = {Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence},
  author = {Shuailei Ma and Jiaqi Liao and Xinyang Wang and Jingjing Wang and Chaoran Feng and Zijing Hu and Chong Bao and Zichen Xi and Yuqi Gan and Weisen Wang and Yanhong Zeng and Qin Zhao and Zifan Shi and Wei Wu and Hao Ouyang and Qiuyu Wang and Shangzhan Zhang and Jiahao Shao and Yipengjing Sun and Liangxiao Hu and Lunke Pan and Nan Xue and Kecheng Zheng and Yinghao Xu and Xing Zhu and Yujun Shen and Ka Leong Cheng},
  journal={arXiv preprint arXiv:2607.xxxxx},
  year = {2026}
}