File size: 4,450 Bytes
644f343 0be59bf 644f343 0be59bf c16b901 0be59bf c16b901 0be59bf c16b901 0be59bf c16b901 0be59bf c16b901 0be59bf c16b901 0be59bf c16b901 0be59bf c16b901 0be59bf c16b901 0be59bf c16b901 0be59bf c16b901 0be59bf c16b901 0be59bf | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 | ---
license: apache-2.0
library_name: pytorch
tags:
- robotics
- uav
- drone
- flow-matching
- autonomous-navigation
- flightmare
---
# FlowPilot Checkpoints
Model checkpoints for **FlowPilot**, a Flow Matching based world-action model for real-time onboard UAV navigation.
The model uses a dual-stream MoT to jointly model video latents and action tokens. Actions are parameterized by a 7th-order Bernstein polynomial with 8 control points (3 fixed by the initial state and 5 predicted by the network), producing 48 waypoints at 30 Hz.
- Code: [flyingbitac/FlowPilot](https://github.com/flyingbitac/FlowPilot)
## Checkpoint roles
**Phase 1 is a training intermediate and is not used for evaluation.** Evaluation uses Phase 2 for both Flightmare trajectory collection and future-video prediction. The Phase-1 checkpoint is optional and is not loaded by any evaluation script.
| File | Size | Role |
| --- | --- | --- |
| `ckpt/phase2/ckpt_final.pt` | 651 MB | **Evaluation checkpoint.** Joint video/action model used for control and future-video prediction. The inference state dict is stored under `model`; `step`, `optimizer`, and `lr_scheduler` are also included. |
| `ckpt/phase1.pt` | 539 MB | Optional video-pretraining intermediate. Not required or used for evaluation. |
| `ckpt/action_stats.pt` | 3.1 KB | State/goal and action normalization statistics used by Phase 2. |
| `ckpt/coeff_stats.pt` | 2.7 KB | Bernstein coefficient normalization statistics used by Phase 2. |
| `ckpt/Wan2.2_VAE.pth` | about 2.8 GB | Depth encoder/decoder used by both phases. Required for evaluation. |
Only the Phase-2 checkpoint, Wan2.2 VAE, and normalization statistics are needed for evaluation.
## Download
Clone the code repository first, then download the checkpoints into its root:
```bash
git clone https://github.com/flyingbitac/FlowPilot.git
cd FlowPilot
hf download YuDinging/FlowPilot --include "ckpt/*" --local-dir .
hf download Wan-AI/Wan2.2-TI2V-5B Wan2.2_VAE.pth --local-dir ckpt
```
If `hf` is not available, install it with `pip install -U huggingface_hub`.
The `--include "ckpt/*"` filter matters: without it, `hf download` would also write this model card as `README.md` and overwrite the README of the code repository.
The evaluation files land under `ckpt/`, the default location expected by `configs/runtime/rviz.py`:
```
ckpt/
βββ phase2/ckpt_final.pt
βββ Wan2.2_VAE.pth
βββ action_stats.pt
βββ coeff_stats.pt
```
`ckpt/phase1.pt` may also be present for training analysis, but can be omitted for evaluation.
`hf download` also creates a `.cache/huggingface/` directory in the current directory. It only holds download metadata and can be removed afterwards. Weights can instead be kept outside the repository and symlinked at the paths above. For example:
```bash
ln -s /path/to/Wan2.2_VAE.pth ckpt/Wan2.2_VAE.pth
```
`python docker/docker.py start` resolves model symlinks and mounts external targets read-only at the same absolute path inside the container.
## Phase-2 evaluation
Build and start the simulation container:
```bash
python docker/docker.py build
export DISPLAY=:1
python docker/docker.py start
```
The standard container exposes all GPUs, allowing Flightmare to use NVIDIA rendering. Inside the container, run the offline collection and future-video comparison without ROS or RViz:
```bash
python scripts/collect_phase2_trajectory.py
python scripts/replay_phase2_comparison.py --start-time 0.0
```
Outputs are written to `runs/phase2_replay/`:
- `trajectory.npz`: 30 Hz states, goals, waypoints, and grayscale depth frames; no RGB data.
- `comparison.png`: two rows and nine columns, with ground-truth depth above Phase-2 predictions.
The comparison covers 0 to 1.6 seconds at 0.2-second intervals. The first frame is the observed condition, followed by eight generated future frames. Phase 2 uses the recorded state/goal at the condition frame. Select another condition time with `--start-time`.
## Intended Use
These weights are released for research on learning-based UAV navigation, mainly for simulation and onboard inference experiments. They are not certified for safety-critical flight.
## License
The checkpoints are released under the Apache License 2.0.
`ckpt/Wan2.2_VAE.pth` is downloaded from the official [Wan2.2 model repository](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B) (Alibaba Wan Team) and is also licensed under Apache-2.0.
|