# Use SAM2 with videos
import torch
from sam2.sam2_video_predictor import SAM2VideoPredictor
predictor = SAM2VideoPredictor.from_pretrained(sanatem/samtp-mini-traversability)
with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
state = predictor.init_state(<your_video>)
# add new prompts and instantly get the output on the same frame
frame_idx, object_ids, masks = predictor.add_new_points(state, <your_prompts>):
# propagate the prompts to get masklets throughout the video
for frame_idx, object_ids, masks in predictor.propagate_in_video(state):
...SAM-TP Mini+ Traversability (checkpoint_finetuned_v2)
Image-space traversability segmentation for the FrodoBots Earth Rover Mini+ front camera. One RGB frame in, per-pixel drivability out.
Model details
- Architecture: SAM 2.1 image branch, Hiera-tiny backbone (embed_dim 96),
with the prompt encoder replaced by a learned "traversability prompt"
(GeNIE SAM-TP,
CustomPromptEncoderLarger,want_custom_prompt_encoder: 2). Prompt-free: point/box inputs are ignored; output is deterministic per image. - Init:
facebook/sam2.1-hiera-tiny - Fine-tuning data: ~50k front-camera frames from Earth Rover Mini+ footage (Mini-4K derived) with binary drivable-ground masks. (v1 used ~5k frames; this v2 checkpoint used a larger set with cleaner labels.)
- File:
checkpoint_finetuned_v2.pt— torch save with a single top-levelmodelkey holding the state dict. 136,622,641 bytes. - sha256:
44e508da3d36a63431f8197f16784c980abf43ea94fc4e524bcd19d0646692bd
Required inference config
This checkpoint ONLY loads against the tiny SAM-TP inference config
(sam2/configs/sam2.1_inference_tiny/sam2.1_custom2.yaml in the GeNIE sam2
fork). Loading it with a base+/small/large config — or loading the public GeNIE
checkpoint_2.pt (base+) with the tiny config — fails with a state-dict
mismatch.
Usage
Via the rover-traversability package (in the team repo under traversability/):
pip install 'rover-traversability[hf]' # from the repo: pip install -e ./traversability[hf]
python -c "
from rover_traversability import TraversabilityPredictor
p = TraversabilityPredictor() # auto-downloads this checkpoint
result = p.predict('frame.jpg')
print(result.mask.shape, result.mask.mean())
"
Or manually: download checkpoint_finetuned_v2.pt and set
SAMTP_CHECKPOINT=/path/to/checkpoint_finetuned_v2.pt.
Output contract: mask is HxW float32 in [0, 1], 1 = drivable (sigmoid of the
raw logits, resized to the input frame size).
Fine-tuning on top
This is a full model state dict — use it directly as the init checkpoint in
Meta's SAM2 training harness (training.* from facebookresearch/sam2) with the
sam2.1_training_tiny configs from the GeNIE fork (ckpt_state_dict_keys: ['model']). Dataset format: image folder + binary PNG masks (MOSE/PNG-VOS
layout). Reference hyperparameters from this checkpoint's training: 1024 res,
batch 8, AdamW, base_lr 5e-6 / vision_lr 3e-6, 5 epochs.
Performance (Apple M-series, 1024x576 input)
| Device | Latency/frame |
|---|---|
| MPS | ~0.16–0.23 s (4–6 Hz) |
| CPU |
Known limitations
- Trained as "ground vs. above-ground": dark objects sitting on light ground
(other rovers, low obstacles) can be labeled drivable. The
rover-traversabilitywrapper applies a per-frame luminance-contrast refinement to mitigate this — keep it enabled. - Monocular, image-space only: no metric depth. Pair with camera calibration for BEV projection.
Licensing & provenance
- SAM 2 / SAM 2.1 base weights and code: Apache-2.0 (Meta Platforms).
- SAM-TP architecture: GeNIE (Wang, Liu, Chen, et al.).
- Fine-tuning data: FrodoBots Earth Rover Mini footage. The related public
dataset is
BitRobot/FrodoBots-Mini-4K(CC-BY-SA) — if you redistribute or build on these weights, carry this provenance note and attribution with them.
# Use SAM2 with images import torch from sam2.sam2_image_predictor import SAM2ImagePredictor predictor = SAM2ImagePredictor.from_pretrained(sanatem/samtp-mini-traversability) with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16): predictor.set_image(<your_image>) masks, _, _ = predictor.predict(<input_prompts>)