Instructions to use gangweix/next-forcing-base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use gangweix/next-forcing-base with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("gangweix/next-forcing-base", dtype=torch.bfloat16, device_map="cuda") prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k" image = pipe(prompt).images[0] - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -6,19 +6,20 @@ tags:
|
|
| 6 |
- robotics
|
| 7 |
- world-model
|
| 8 |
- video-generation
|
| 9 |
-
- vision-
|
| 10 |
- next-forcing
|
| 11 |
-
- robotwin
|
| 12 |
---
|
| 13 |
|
| 14 |
-
# Next Forcing β
|
| 15 |
|
| 16 |
-
|
| 17 |
-
|
|
|
|
| 18 |
|
| 19 |
- π Paper: https://arxiv.org/pdf/2606.11187
|
| 20 |
- π Project page: https://gangweix.github.io/next-forcing/
|
| 21 |
- π» Code: https://github.com/gangweix/next-forcing
|
|
|
|
| 22 |
|
| 23 |
## Model Description
|
| 24 |
|
|
@@ -29,31 +30,23 @@ instead of long-range dynamics, especially at high frame rates. Lightweight
|
|
| 29 |
a causal chain during training, providing dense temporal supervision to the
|
| 30 |
backbone.
|
| 31 |
|
| 32 |
-
This checkpoint
|
| 33 |
-
|
| 34 |
-
|
|
|
|
| 35 |
|
| 36 |
| | |
|
| 37 |
| --- | --- |
|
| 38 |
-
| Parameters |
|
| 39 |
| Backbone layers | 30 |
|
| 40 |
-
| MCP
|
| 41 |
-
| Benchmark | RoboTwin 2.0, 50 bimanual manipulation tasks |
|
| 42 |
-
| Initialized from | [`next-forcing-base`](https://huggingface.co/gangweix/next-forcing-base) (5.1B) |
|
| 43 |
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
Average success rate on RoboTwin 2.0:
|
| 47 |
-
|
| 48 |
-
| Setting | LingBot-VA | **Next Forcing** |
|
| 49 |
-
| --- | ---: | ---: |
|
| 50 |
-
| Clean | 92.9 | **94.1** |
|
| 51 |
-
| Random | 91.5 | **93.5** |
|
| 52 |
|
| 53 |
## Repository Layout
|
| 54 |
|
| 55 |
```text
|
| 56 |
-
transformer/
|
| 57 |
vae/
|
| 58 |
text_encoder/
|
| 59 |
tokenizer/
|
|
@@ -66,27 +59,26 @@ Clone the code and install the dependencies as described in the
|
|
| 66 |
|
| 67 |
```bash
|
| 68 |
python -m pip install "huggingface_hub[cli]"
|
| 69 |
-
hf download gangweix/next-forcing-
|
| 70 |
-
--local-dir ./checkpoints/next-forcing-
|
| 71 |
```
|
| 72 |
|
| 73 |
-
The
|
| 74 |
-
`
|
| 75 |
-
id:
|
| 76 |
|
| 77 |
```bash
|
| 78 |
-
export
|
| 79 |
-
export
|
| 80 |
-
|
| 81 |
-
# Start the inference server on one GPU
|
| 82 |
-
CUDA_VISIBLE_DEVICES=0 bash evaluation/robotwin/launch_server.sh
|
| 83 |
|
| 84 |
-
|
| 85 |
-
bash
|
| 86 |
```
|
| 87 |
|
| 88 |
-
|
| 89 |
-
[
|
|
|
|
| 90 |
|
| 91 |
## License
|
| 92 |
|
|
|
|
| 6 |
- robotics
|
| 7 |
- world-model
|
| 8 |
- video-generation
|
| 9 |
+
- vision-action
|
| 10 |
- next-forcing
|
|
|
|
| 11 |
---
|
| 12 |
|
| 13 |
+
# Next Forcing β Base Checkpoint
|
| 14 |
|
| 15 |
+
Base checkpoint for **Next Forcing: Causal World Modeling with Multi-Chunk
|
| 16 |
+
Prediction**. This is the causal video-action backbone used as the
|
| 17 |
+
initialization for post-training.
|
| 18 |
|
| 19 |
- π Paper: https://arxiv.org/pdf/2606.11187
|
| 20 |
- π Project page: https://gangweix.github.io/next-forcing/
|
| 21 |
- π» Code: https://github.com/gangweix/next-forcing
|
| 22 |
+
- π€ RoboTwin post-trained model: [`gangweix/next-forcing-posttrain-robotwin`](https://huggingface.co/gangweix/next-forcing-posttrain-robotwin)
|
| 23 |
|
| 24 |
## Model Description
|
| 25 |
|
|
|
|
| 30 |
a causal chain during training, providing dense temporal supervision to the
|
| 31 |
backbone.
|
| 32 |
|
| 33 |
+
This base checkpoint contains the backbone **without** MCP modules. The MCP
|
| 34 |
+
modules are created at the start of post-training and initialized from the last
|
| 35 |
+
`mcp_blocks_per_depth` backbone blocks (`mcp_init_from_backbone = True`), so the
|
| 36 |
+
post-trained checkpoint is correspondingly larger (6.7B vs 5.1B).
|
| 37 |
|
| 38 |
| | |
|
| 39 |
| --- | --- |
|
| 40 |
+
| Parameters | 5.1B (BF16) |
|
| 41 |
| Backbone layers | 30 |
|
| 42 |
+
| MCP modules | Not included (added during post-training) |
|
|
|
|
|
|
|
| 43 |
|
| 44 |
+
Built on top of the [LingBot-VA](https://github.com/Robbyant/lingbot-va) codebase.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 45 |
|
| 46 |
## Repository Layout
|
| 47 |
|
| 48 |
```text
|
| 49 |
+
transformer/ Causal video-action backbone
|
| 50 |
vae/
|
| 51 |
text_encoder/
|
| 52 |
tokenizer/
|
|
|
|
| 59 |
|
| 60 |
```bash
|
| 61 |
python -m pip install "huggingface_hub[cli]"
|
| 62 |
+
hf download gangweix/next-forcing-base \
|
| 63 |
+
--local-dir ./checkpoints/next-forcing-base
|
| 64 |
```
|
| 65 |
|
| 66 |
+
The training code resolves model subfolders by path, so point
|
| 67 |
+
`NEXT_FORCING_PRETRAINED_MODEL_PATH` at the **local directory**, not at the Hub
|
| 68 |
+
repository id:
|
| 69 |
|
| 70 |
```bash
|
| 71 |
+
export NEXT_FORCING_PRETRAINED_MODEL_PATH=$PWD/checkpoints/next-forcing-base
|
| 72 |
+
export NEXT_FORCING_DATASET_PATH=/path/to/your/dataset
|
| 73 |
+
export NEXT_FORCING_SAVE_ROOT=/path/to/your/output
|
|
|
|
|
|
|
| 74 |
|
| 75 |
+
NGPU=8 CONFIG_NAME=robotwin_train \
|
| 76 |
+
bash script/run_va_posttrain.sh --init-worker 1
|
| 77 |
```
|
| 78 |
|
| 79 |
+
MCP defaults (`wan_va/configs/mcp_train_config.py`): `num_mcp_depths = 3`,
|
| 80 |
+
`mcp_blocks_per_depth = 3`, `mcp_hidden_collect_layers = [3, 11, 19, 29]`,
|
| 81 |
+
`mcp_loss_weights = [0.5, 0.2, 0.1]`.
|
| 82 |
|
| 83 |
## License
|
| 84 |
|