README.md

---
title: Molmo2Fish Tracking
emoji: 🐟
colorFrom: purple
colorTo: blue
sdk: gradio
sdk_version: 6.15.1
app_file: app.py
short_description: Track fish in sonar video, fix it with plain English
python_version: "3.12"
startup_duration_timeout: 1h
---

# 🐟 Molmo2Fish — interactive fish tracking with natural language guidance

Demo of [**tidalove/Molmo2Fish**](https://huggingface.co/tidalove/Molmo2Fish), the model from
*"Teach a Molmo2Fish: Towards interactive fish tracking with natural language guidance"*
([paper](https://huggingface.co/papers/2608.18602) ·
[code](https://github.com/tidalove/molmo2fish)).

Molmo2Fish is a LoRA-finetuned [Molmo2](https://huggingface.co/allenai) VLM that tracks
salmon in ARIS **sonar** video and — crucially — accepts a plain-English critique of its
own output and re-emits corrected tracks. The paper reports tracking accuracy going from
~5% to ~79% over a handful of conversational correction turns.

## How the demo works

1. **① Track all fish** sends the clip with the prompt `track all fish`. The model replies
   with the html-v2 pointing format used in training:
   `<tracks coords="0.0 1 409 852;0.5 1 436 890 2 300 120;…">fish</tracks>`
   (timestamps in seconds at 2 FPS, ids, and x/y normalised to 0–1000).
2. **② Apply correction** rebuilds the conversation — video on the first user turn, the
   model's previous `<tracks …>` answer as the assistant turn, your critique as the new
   user turn — exactly as in `olmo/eval/vllm_runner.py::build_multi_turn_chat`, and the
   model regenerates the track set.

Points are parsed with the same regexes as
`olmo/preprocessing/point_formatter.py` and drawn back onto the 6 FPS source video.

## Example clips

The three sonar clips shipped with this Space are re-encoded (G channel, 6 FPS, libx264,
matching the repo's `encode_frames_to_video` recipe) from the Caltech Fish Counting
release [**perona-lab/cfc26**](https://huggingface.co/datasets/perona-lab/cfc26), which is
distributed under **CC-BY-4.0** — credit to the Caltech Fish Counting / CFC26 authors.
The correction prompts pre-filled with each example are verbatim from the validation split
of [tidalove/cfc-track-instruction](https://huggingface.co/datasets/tidalove/cfc-track-instruction).

## Notes

- Runs on ZeroGPU; the model is loaded in bfloat16 (~16 GB VRAM).
- Video is sampled at 2 FPS, max 128 frames, per the model's
  `video_preprocessor_config.json`.
- The released checkpoint ships a config mismatch: `processor_config.json` sets
  `use_frame_special_tokens: true` (so each frame is wrapped in
  `<frame_start>`/`<frame_end>`, matching training) while `config.json` sets it
  `false`, which makes the model count `<im_end>` instead and abort with
  `AssertionError: Expected 0 videos, but got 1`. `app.py` aligns the two at
  load time.
- A truncated answer (no closing `</tracks>`) means you hit the *Max new tokens* cap —
  raise it in **Advanced**.