| --- |
| license: cc-by-nc-sa-4.0 |
| library_name: pytorch |
| tags: |
| - lidar |
| - autonomous-driving |
| - trajectory-refinement |
| - auto-labeling |
| - argoverse-2 |
| - mojo |
| - smoke-checkpoint |
| --- |
| |
| # LabelFormer — AV2 smoke checkpoint |
|
|
| > ⚠️ **This is a small-scale smoke checkpoint, not a paper reproduction.** It is a 1.16M-parameter model trained locally on a laptop (Apple Silicon, MPS) on **12 of the 700** ArgoVerse 2 sensor train logs for 30 epochs (~11 minutes). It exists so that the reference implementations below can be run end-to-end without training. Do not use it to benchmark against the paper. |
|
|
| Reference checkpoint for two open-source implementations of **LabelFormer** (*Object Trajectory Refinement for Offboard Perception from LiDAR Point Clouds*, Yang et al., CoRL 2023, [arXiv:2311.01444](https://arxiv.org/abs/2311.01444)): |
|
|
| - **[labelrefinery/LabelFormer.py](https://github.com/labelrefinery/LabelFormer.py)** — PyTorch implementation + ArgoVerse 2 training pipeline (this checkpoint was trained with it). |
| - **[labelrefinery/LabelFormer.mojo](https://github.com/labelrefinery/LabelFormer.mojo)** — pure-Mojo inferencer, verified to match PyTorch within 1.5e-6 on the refined boxes using the `mojo/` files here. |
|
|
| ## What the model does |
|
|
| LabelFormer refines noisy BEV object trajectories (auto-labels) from LiDAR point clouds: per-frame boxes and object points are encoded independently (box MLP + PointPillars-style CNN), a transformer with ALiBi relative position biases reasons over the whole trajectory, and the model decodes per-frame pose residuals plus one trajectory-level object size. |
|
|
| ## Training setup (differs from the paper) |
|
|
| | | this checkpoint | paper | |
| |---|---|---| |
| | params | 1.16M (d=128, 3 layers) | ~6M (d=256, 6 layers) | |
| | data | 12 AV2 train logs (739 vehicle tracks) | 700 logs | |
| | initial noisy tracks | perturbed ground truth (±0.25 m, ±10°, size jitter) | detector + tracker outputs | |
| | training | 30 epochs, ~11 min, Apple M-series (MPS) | 40 epochs, GPU | |
| | pillar grid | 0.2 m, 19.2 m × 6.4 m | 0.1 m, 24 m × 8 m | |
|
|
| ## Results (4 AV2 val logs, 261 tracks, refined vs. perturbed input) |
|
|
| | metric | initial | refined | |
| |---|---|---| |
| | mean IoU | 0.794 | 0.939 | |
| | recall@0.7 | 0.966 | 0.999 | |
| | recall@0.8 | 0.450 | 0.992 | |
|
|
| Full numbers in `eval_val.json`; per-epoch curves in `history.json`. |
|
|
| ## Files |
|
|
| - `best.pt` — PyTorch checkpoint (`model` state dict + `config`), load with LabelFormer.py's `evaluate.py` or `torch.load`. |
| - `config.yaml` — the exact training config (`configs/smoke.yaml`). |
| - `mojo/weights.lft` — weights with BatchNorms folded into convs, in the LFT1 container consumed by LabelFormer.mojo. |
| - `mojo/sample_*.lft` — three real AV2 val trajectories with expected per-stage outputs, for parity testing the Mojo inferencer. |
|
|
| ## License and data provenance |
|
|
| Trained on the [ArgoVerse 2 sensor dataset](https://www.argoverse.org/av2.html), which is released under **CC BY-NC-SA 4.0**; these weights are published under the same license and are intended for **non-commercial research use**. |
|
|
