Update checkpoint documentation
Browse files
README.md
CHANGED
|
@@ -18,17 +18,21 @@ The model uses a dual-stream MoT to jointly model video latents and action token
|
|
| 18 |
|
| 19 |
- Code: [flyingbitac/FlowPilot](https://github.com/flyingbitac/FlowPilot)
|
| 20 |
|
| 21 |
-
##
|
| 22 |
|
| 23 |
-
|
|
|
|
|
|
|
| 24 |
| --- | --- | --- |
|
| 25 |
-
| `ckpt/phase2/ckpt_final.pt` | 651 MB |
|
| 26 |
-
| `ckpt/
|
| 27 |
-
| `ckpt/
|
|
|
|
|
|
|
| 28 |
|
| 29 |
-
|
| 30 |
|
| 31 |
-
##
|
| 32 |
|
| 33 |
Clone the code repository first, then download the checkpoints into its root:
|
| 34 |
|
|
@@ -44,7 +48,7 @@ If `hf` is not available, install it with `pip install -U huggingface_hub`.
|
|
| 44 |
|
| 45 |
The `--include "ckpt/*"` filter matters: without it, `hf download` would also write this model card as `README.md` and overwrite the README of the code repository.
|
| 46 |
|
| 47 |
-
The files land under `ckpt/`, the default location expected by `configs/runtime/rviz.py`:
|
| 48 |
|
| 49 |
```
|
| 50 |
ckpt/
|
|
@@ -54,17 +58,40 @@ ckpt/
|
|
| 54 |
└── coeff_stats.pt
|
| 55 |
```
|
| 56 |
|
| 57 |
-
`
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 58 |
|
|
|
|
| 59 |
|
|
|
|
| 60 |
|
| 61 |
-
|
| 62 |
|
| 63 |
```bash
|
| 64 |
python docker/docker.py build
|
|
|
|
| 65 |
python docker/docker.py start
|
| 66 |
```
|
| 67 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 68 |
## Intended Use
|
| 69 |
|
| 70 |
These weights are released for research on learning-based UAV navigation, mainly for simulation and onboard inference experiments. They are not certified for safety-critical flight.
|
|
|
|
| 18 |
|
| 19 |
- Code: [flyingbitac/FlowPilot](https://github.com/flyingbitac/FlowPilot)
|
| 20 |
|
| 21 |
+
## Checkpoint roles
|
| 22 |
|
| 23 |
+
**Phase 1 is a training intermediate and is not used for evaluation.** Evaluation uses Phase 2 for both Flightmare trajectory collection and future-video prediction. The Phase-1 checkpoint is optional and is not loaded by any evaluation script.
|
| 24 |
+
|
| 25 |
+
| File | Size | Role |
|
| 26 |
| --- | --- | --- |
|
| 27 |
+
| `ckpt/phase2/ckpt_final.pt` | 651 MB | **Evaluation checkpoint.** Joint video/action model used for control and future-video prediction. The inference state dict is stored under `model`; `step`, `optimizer`, and `lr_scheduler` are also included. |
|
| 28 |
+
| `ckpt/phase1.pt` | 539 MB | Optional video-pretraining intermediate. Not required or used for evaluation. |
|
| 29 |
+
| `ckpt/action_stats.pt` | 3.1 KB | State/goal and action normalization statistics used by Phase 2. |
|
| 30 |
+
| `ckpt/coeff_stats.pt` | 2.7 KB | Bernstein coefficient normalization statistics used by Phase 2. |
|
| 31 |
+
| `ckpt/Wan2.2_VAE.pth` | about 2.8 GB | Depth encoder/decoder used by both phases. Required for evaluation. |
|
| 32 |
|
| 33 |
+
Only the Phase-2 checkpoint, Wan2.2 VAE, and normalization statistics are needed for evaluation.
|
| 34 |
|
| 35 |
+
## Download
|
| 36 |
|
| 37 |
Clone the code repository first, then download the checkpoints into its root:
|
| 38 |
|
|
|
|
| 48 |
|
| 49 |
The `--include "ckpt/*"` filter matters: without it, `hf download` would also write this model card as `README.md` and overwrite the README of the code repository.
|
| 50 |
|
| 51 |
+
The evaluation files land under `ckpt/`, the default location expected by `configs/runtime/rviz.py`:
|
| 52 |
|
| 53 |
```
|
| 54 |
ckpt/
|
|
|
|
| 58 |
└── coeff_stats.pt
|
| 59 |
```
|
| 60 |
|
| 61 |
+
`ckpt/phase1.pt` may also be present for training analysis, but can be omitted for evaluation.
|
| 62 |
+
|
| 63 |
+
`hf download` also creates a `.cache/huggingface/` directory in the current directory. It only holds download metadata and can be removed afterwards. Weights can instead be kept outside the repository and symlinked at the paths above. For example:
|
| 64 |
+
|
| 65 |
+
```bash
|
| 66 |
+
ln -s /path/to/Wan2.2_VAE.pth ckpt/Wan2.2_VAE.pth
|
| 67 |
+
```
|
| 68 |
|
| 69 |
+
`python docker/docker.py start` resolves model symlinks and mounts external targets read-only at the same absolute path inside the container.
|
| 70 |
|
| 71 |
+
## Phase-2 evaluation
|
| 72 |
|
| 73 |
+
Build and start the simulation container:
|
| 74 |
|
| 75 |
```bash
|
| 76 |
python docker/docker.py build
|
| 77 |
+
export DISPLAY=:1
|
| 78 |
python docker/docker.py start
|
| 79 |
```
|
| 80 |
|
| 81 |
+
The standard container exposes all GPUs, allowing Flightmare to use NVIDIA rendering. Inside the container, run the offline collection and future-video comparison without ROS or RViz:
|
| 82 |
+
|
| 83 |
+
```bash
|
| 84 |
+
python scripts/collect_phase2_trajectory.py
|
| 85 |
+
python scripts/replay_phase2_comparison.py --start-time 0.0
|
| 86 |
+
```
|
| 87 |
+
|
| 88 |
+
Outputs are written to `runs/phase2_replay/`:
|
| 89 |
+
|
| 90 |
+
- `trajectory.npz`: 30 Hz states, goals, waypoints, and grayscale depth frames; no RGB data.
|
| 91 |
+
- `comparison.png`: two rows and nine columns, with ground-truth depth above Phase-2 predictions.
|
| 92 |
+
|
| 93 |
+
The comparison covers 0 to 1.6 seconds at 0.2-second intervals. The first frame is the observed condition, followed by eight generated future frames. Phase 2 uses the recorded state/goal at the condition frame. Select another condition time with `--start-time`.
|
| 94 |
+
|
| 95 |
## Intended Use
|
| 96 |
|
| 97 |
These weights are released for research on learning-based UAV navigation, mainly for simulation and onboard inference experiments. They are not certified for safety-critical flight.
|