File size: 3,620 Bytes
9720148 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 | ---
license: apache-2.0
tags:
- robotics
- traversability
- sam2
- frodobots
- earth-rover
library_name: sam-tp
---
# SAM-TP Mini+ Traversability (checkpoint_finetuned_v2)
Image-space traversability segmentation for the **FrodoBots Earth Rover Mini+**
front camera. One RGB frame in, per-pixel drivability out.
## Model details
- **Architecture**: SAM 2.1 image branch, Hiera-**tiny** backbone (embed_dim 96),
with the prompt encoder replaced by a learned "traversability prompt"
(GeNIE SAM-TP, `CustomPromptEncoderLarger`, `want_custom_prompt_encoder: 2`).
Prompt-free: point/box inputs are ignored; output is deterministic per image.
- **Init**: `facebook/sam2.1-hiera-tiny`
- **Fine-tuning data**: ~50k front-camera frames from Earth Rover Mini+ footage
(Mini-4K derived) with binary drivable-ground masks. (v1 used ~5k frames;
this v2 checkpoint used a larger set with cleaner labels.)
- **File**: `checkpoint_finetuned_v2.pt` — torch save with a single top-level
`model` key holding the state dict. 136,622,641 bytes.
- **sha256**: `44e508da3d36a63431f8197f16784c980abf43ea94fc4e524bcd19d0646692bd`
## Required inference config
This checkpoint ONLY loads against the tiny SAM-TP inference config
(`sam2/configs/sam2.1_inference_tiny/sam2.1_custom2.yaml` in the GeNIE sam2
fork). Loading it with a base+/small/large config — or loading the public GeNIE
`checkpoint_2.pt` (base+) with the tiny config — fails with a state-dict
mismatch.
## Usage
Via the `rover-traversability` package (in the team repo under `traversability/`):
```bash
pip install 'rover-traversability[hf]' # from the repo: pip install -e ./traversability[hf]
python -c "
from rover_traversability import TraversabilityPredictor
p = TraversabilityPredictor() # auto-downloads this checkpoint
result = p.predict('frame.jpg')
print(result.mask.shape, result.mask.mean())
"
```
Or manually: download `checkpoint_finetuned_v2.pt` and set
`SAMTP_CHECKPOINT=/path/to/checkpoint_finetuned_v2.pt`.
Output contract: `mask` is HxW float32 in [0, 1], 1 = drivable (sigmoid of the
raw logits, resized to the input frame size).
## Fine-tuning on top
This is a full model state dict — use it directly as the init checkpoint in
Meta's SAM2 training harness (`training.*` from facebookresearch/sam2) with the
`sam2.1_training_tiny` configs from the GeNIE fork (`ckpt_state_dict_keys:
['model']`). Dataset format: image folder + binary PNG masks (MOSE/PNG-VOS
layout). Reference hyperparameters from this checkpoint's training: 1024 res,
batch 8, AdamW, base_lr 5e-6 / vision_lr 3e-6, 5 epochs.
## Performance (Apple M-series, 1024x576 input)
| Device | Latency/frame |
| ------ | ------------- |
| MPS | ~0.16–0.23 s (4–6 Hz) |
| CPU | ~0.44 s (~2.3 Hz) |
## Known limitations
- Trained as "ground vs. above-ground": dark objects sitting on light ground
(other rovers, low obstacles) can be labeled drivable. The
`rover-traversability` wrapper applies a per-frame luminance-contrast
refinement to mitigate this — keep it enabled.
- Monocular, image-space only: no metric depth. Pair with camera calibration
for BEV projection.
## Licensing & provenance
- SAM 2 / SAM 2.1 base weights and code: Apache-2.0 (Meta Platforms).
- SAM-TP architecture: GeNIE (Wang, Liu, Chen, et al.).
- Fine-tuning data: FrodoBots Earth Rover Mini footage. The related public
dataset is [`BitRobot/FrodoBots-Mini-4K`](https://huggingface.co/datasets/BitRobot/FrodoBots-Mini-4K)
(**CC-BY-SA**) — if you redistribute or build on these weights, carry this
provenance note and attribution with them.
|