README.md

---
license: apache-2.0
base_model: allenai/Molmo2-8B
pipeline_tag: video-text-to-text
library_name: transformers
tags:
- molmo2
- video
- object-tracking
- ecology
datasets:
- tidalove/cfc-track-instruction
- perona-lab/cfc26
---

# Molmo2Fish

Molmo2Fish is [Molmo2-8B](https://github.com/allenai/molmo2) fine-tuned to **track fish in
sonar video, and to edit those tracks in response to natural-language feedback**.

Track correction is treated as a conversation: the model is shown a set of existing tracks —
its own earlier predictions, another tracker's output, or corrupted ground truth — is told in
words what is wrong with them, and returns a repaired set of tracks.

| | |
|---|---|
| Code | [github.com/tidalove/molmo2fish](https://github.com/tidalove/molmo2fish) |
| Data | [tidalove/cfc-track-instruction](https://huggingface.co/datasets/tidalove/cfc-track-instruction) |
| Source video | [Caltech Fish Counting](https://huggingface.co/datasets/perona-lab/cfc26) |

## Files

| file | what it is |
|---|---|
| `*.safetensors`, `config.json`, … | HuggingFace-format weights at the repo root — what vLLM and `launch_scripts/hf_eval.py` consume |
| `Molmo2Fish-step420-raw.tar` | the raw training checkpoint (sharded model + optimizer state), for resuming fine-tuning |

## Training

Rank 64 LoRA fine-tuning of Molmo2-8B on the full CFC mixture, step 420. Adapters on all
three components — LLM, ViT, and connector — with the base weights frozen:

```bash
torchrun --nproc-per-node=8 launch_scripts/sft.py /path/to/Molmo2-8B cfc_correction \
  --lora_llm --lora_vit --lora_connector --lora_rank 64 \
  --save_folder=/path/to/save/folder
```

The `cfc_correction` mixture combines pure tracking, targeted correction, synthetically
corrupted correction, correction of real model predictions at two quality levels
(`molmo_high` / `molmo_low`), and text-only correction. See the
[dataset card](https://huggingface.co/datasets/tidalove/cfc-track-instruction) for what each
config contains.

## Citation

```bibtex
@article{molmo2fish,
    title={Teach a Molmo2Fish: Towards interactive fish tracking with natural language guidance},
    author={Kai van Brunt and Justin Kay and Sara Beery},
    year={2026},
    url={https://arxiv.org/abs/2608.18602}
}
```