README.md

---
license: apache-2.0
tags:
  - robotics
  - traversability
  - sam2
  - frodobots
  - earth-rover
library_name: sam-tp
---

# SAM-TP Mini+ Traversability (checkpoint_finetuned_v2)

Image-space traversability segmentation for the **FrodoBots Earth Rover Mini+**
front camera. One RGB frame in, per-pixel drivability out.

## Model details

- **Architecture**: SAM 2.1 image branch, Hiera-**tiny** backbone (embed_dim 96),
  with the prompt encoder replaced by a learned "traversability prompt"
  (GeNIE SAM-TP, `CustomPromptEncoderLarger`, `want_custom_prompt_encoder: 2`).
  Prompt-free: point/box inputs are ignored; output is deterministic per image.
- **Init**: `facebook/sam2.1-hiera-tiny`
- **Fine-tuning data**: ~50k front-camera frames from Earth Rover Mini+ footage
  (Mini-4K derived) with binary drivable-ground masks. (v1 used ~5k frames;
  this v2 checkpoint used a larger set with cleaner labels.)
- **File**: `checkpoint_finetuned_v2.pt` — torch save with a single top-level
  `model` key holding the state dict. 136,622,641 bytes.
- **sha256**: `44e508da3d36a63431f8197f16784c980abf43ea94fc4e524bcd19d0646692bd`

## Required inference config

This checkpoint ONLY loads against the tiny SAM-TP inference config
(`sam2/configs/sam2.1_inference_tiny/sam2.1_custom2.yaml` in the GeNIE sam2
fork). Loading it with a base+/small/large config — or loading the public GeNIE
`checkpoint_2.pt` (base+) with the tiny config — fails with a state-dict
mismatch.

## Usage

Via the `rover-traversability` package (in the team repo under `traversability/`):

```bash
pip install 'rover-traversability[hf]'   # from the repo: pip install -e ./traversability[hf]
python -c "
from rover_traversability import TraversabilityPredictor
p = TraversabilityPredictor()            # auto-downloads this checkpoint
result = p.predict('frame.jpg')
print(result.mask.shape, result.mask.mean())
"
```

Or manually: download `checkpoint_finetuned_v2.pt` and set
`SAMTP_CHECKPOINT=/path/to/checkpoint_finetuned_v2.pt`.

Output contract: `mask` is HxW float32 in [0, 1], 1 = drivable (sigmoid of the
raw logits, resized to the input frame size).

## Fine-tuning on top

This is a full model state dict — use it directly as the init checkpoint in
Meta's SAM2 training harness (`training.*` from facebookresearch/sam2) with the
`sam2.1_training_tiny` configs from the GeNIE fork (`ckpt_state_dict_keys:
['model']`). Dataset format: image folder + binary PNG masks (MOSE/PNG-VOS
layout). Reference hyperparameters from this checkpoint's training: 1024 res,
batch 8, AdamW, base_lr 5e-6 / vision_lr 3e-6, 5 epochs.

## Performance (Apple M-series, 1024x576 input)

| Device | Latency/frame |
| ------ | ------------- |
| MPS    | ~0.16–0.23 s (4–6 Hz) |
| CPU    | ~0.44 s (~2.3 Hz) |

## Known limitations

- Trained as "ground vs. above-ground": dark objects sitting on light ground
  (other rovers, low obstacles) can be labeled drivable. The
  `rover-traversability` wrapper applies a per-frame luminance-contrast
  refinement to mitigate this — keep it enabled.
- Monocular, image-space only: no metric depth. Pair with camera calibration
  for BEV projection.

## Licensing & provenance

- SAM 2 / SAM 2.1 base weights and code: Apache-2.0 (Meta Platforms).
- SAM-TP architecture: GeNIE (Wang, Liu, Chen, et al.).
- Fine-tuning data: FrodoBots Earth Rover Mini footage. The related public
  dataset is [`BitRobot/FrodoBots-Mini-4K`](https://huggingface.co/datasets/BitRobot/FrodoBots-Mini-4K)
  (**CC-BY-SA**) — if you redistribute or build on these weights, carry this
  provenance note and attribution with them.