gangweix commited on
Commit
305a3fe
Β·
verified Β·
1 Parent(s): b22eec3

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +27 -35
README.md CHANGED
@@ -6,19 +6,20 @@ tags:
6
  - robotics
7
  - world-model
8
  - video-generation
9
- - vision-language-action
10
  - next-forcing
11
- - robotwin
12
  ---
13
 
14
- # Next Forcing β€” RoboTwin Post-trained Checkpoint
15
 
16
- Post-trained checkpoint for **Next Forcing: Causal World Modeling with
17
- Multi-Chunk Prediction**, evaluated on the RoboTwin 2.0 benchmark.
 
18
 
19
  - πŸ“„ Paper: https://arxiv.org/pdf/2606.11187
20
  - 🌐 Project page: https://gangweix.github.io/next-forcing/
21
  - πŸ’» Code: https://github.com/gangweix/next-forcing
 
22
 
23
  ## Model Description
24
 
@@ -29,31 +30,23 @@ instead of long-range dynamics, especially at high frame rates. Lightweight
29
  a causal chain during training, providing dense temporal supervision to the
30
  backbone.
31
 
32
- This checkpoint is the RoboTwin post-trained model, built on top of the
33
- [LingBot-VA](https://github.com/Robbyant/lingbot-va) codebase. It was post-trained
34
- from [`gangweix/next-forcing-base`](https://huggingface.co/gangweix/next-forcing-base).
 
35
 
36
  | | |
37
  | --- | --- |
38
- | Parameters | 6.7B (BF16) |
39
  | Backbone layers | 30 |
40
- | MCP depths | 3 (`mcp_blocks_per_depth=3`, collect layers `[3, 11, 19, 29]`) |
41
- | Benchmark | RoboTwin 2.0, 50 bimanual manipulation tasks |
42
- | Initialized from | [`next-forcing-base`](https://huggingface.co/gangweix/next-forcing-base) (5.1B) |
43
 
44
- ## Results
45
-
46
- Average success rate on RoboTwin 2.0:
47
-
48
- | Setting | LingBot-VA | **Next Forcing** |
49
- | --- | ---: | ---: |
50
- | Clean | 92.9 | **94.1** |
51
- | Random | 91.5 | **93.5** |
52
 
53
  ## Repository Layout
54
 
55
  ```text
56
- transformer/ Next Forcing backbone with MCP modules (enable_mcp=true)
57
  vae/
58
  text_encoder/
59
  tokenizer/
@@ -66,27 +59,26 @@ Clone the code and install the dependencies as described in the
66
 
67
  ```bash
68
  python -m pip install "huggingface_hub[cli]"
69
- hf download gangweix/next-forcing-posttrain-robotwin \
70
- --local-dir ./checkpoints/next-forcing-posttrain-robotwin
71
  ```
72
 
73
- The evaluation code resolves model subfolders by path, so point
74
- `NEXT_FORCING_MODEL_PATH` at the **local directory**, not at the Hub repository
75
- id:
76
 
77
  ```bash
78
- export NEXT_FORCING_MODEL_PATH=$PWD/checkpoints/next-forcing-posttrain-robotwin
79
- export ROBOTWIN_ROOT=/path/to/your/RoboTwin
80
-
81
- # Start the inference server on one GPU
82
- CUDA_VISIBLE_DEVICES=0 bash evaluation/robotwin/launch_server.sh
83
 
84
- # In another terminal, evaluate one task for 100 trials
85
- bash evaluation/robotwin/launch_client.sh /path/to/eval_results adjust_bottle
86
  ```
87
 
88
- RoboTwin evaluation requires a working RoboTwin 2.0 installation; see the
89
- [official guide](https://robotwin-platform.github.io/doc/usage/robotwin-install.html).
 
90
 
91
  ## License
92
 
 
6
  - robotics
7
  - world-model
8
  - video-generation
9
+ - vision-action
10
  - next-forcing
 
11
  ---
12
 
13
+ # Next Forcing β€” Base Checkpoint
14
 
15
+ Base checkpoint for **Next Forcing: Causal World Modeling with Multi-Chunk
16
+ Prediction**. This is the causal video-action backbone used as the
17
+ initialization for post-training.
18
 
19
  - πŸ“„ Paper: https://arxiv.org/pdf/2606.11187
20
  - 🌐 Project page: https://gangweix.github.io/next-forcing/
21
  - πŸ’» Code: https://github.com/gangweix/next-forcing
22
+ - πŸ€– RoboTwin post-trained model: [`gangweix/next-forcing-posttrain-robotwin`](https://huggingface.co/gangweix/next-forcing-posttrain-robotwin)
23
 
24
  ## Model Description
25
 
 
30
  a causal chain during training, providing dense temporal supervision to the
31
  backbone.
32
 
33
+ This base checkpoint contains the backbone **without** MCP modules. The MCP
34
+ modules are created at the start of post-training and initialized from the last
35
+ `mcp_blocks_per_depth` backbone blocks (`mcp_init_from_backbone = True`), so the
36
+ post-trained checkpoint is correspondingly larger (6.7B vs 5.1B).
37
 
38
  | | |
39
  | --- | --- |
40
+ | Parameters | 5.1B (BF16) |
41
  | Backbone layers | 30 |
42
+ | MCP modules | Not included (added during post-training) |
 
 
43
 
44
+ Built on top of the [LingBot-VA](https://github.com/Robbyant/lingbot-va) codebase.
 
 
 
 
 
 
 
45
 
46
  ## Repository Layout
47
 
48
  ```text
49
+ transformer/ Causal video-action backbone
50
  vae/
51
  text_encoder/
52
  tokenizer/
 
59
 
60
  ```bash
61
  python -m pip install "huggingface_hub[cli]"
62
+ hf download gangweix/next-forcing-base \
63
+ --local-dir ./checkpoints/next-forcing-base
64
  ```
65
 
66
+ The training code resolves model subfolders by path, so point
67
+ `NEXT_FORCING_PRETRAINED_MODEL_PATH` at the **local directory**, not at the Hub
68
+ repository id:
69
 
70
  ```bash
71
+ export NEXT_FORCING_PRETRAINED_MODEL_PATH=$PWD/checkpoints/next-forcing-base
72
+ export NEXT_FORCING_DATASET_PATH=/path/to/your/dataset
73
+ export NEXT_FORCING_SAVE_ROOT=/path/to/your/output
 
 
74
 
75
+ NGPU=8 CONFIG_NAME=robotwin_train \
76
+ bash script/run_va_posttrain.sh --init-worker 1
77
  ```
78
 
79
+ MCP defaults (`wan_va/configs/mcp_train_config.py`): `num_mcp_depths = 3`,
80
+ `mcp_blocks_per_depth = 3`, `mcp_hidden_collect_layers = [3, 11, 19, 29]`,
81
+ `mcp_loss_weights = [0.5, 0.2, 0.1]`.
82
 
83
  ## License
84