File size: 2,275 Bytes
742d21f 2f60eae 742d21f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 | ---
license: apache-2.0
base_model: allenai/Molmo2-8B
pipeline_tag: video-text-to-text
library_name: transformers
tags:
- molmo2
- video
- object-tracking
- ecology
datasets:
- tidalove/cfc-track-instruction
- perona-lab/cfc26
---
# Molmo2Fish
Molmo2Fish is [Molmo2-8B](https://github.com/allenai/molmo2) fine-tuned to **track fish in
sonar video, and to edit those tracks in response to natural-language feedback**.
Track correction is treated as a conversation: the model is shown a set of existing tracks —
its own earlier predictions, another tracker's output, or corrupted ground truth — is told in
words what is wrong with them, and returns a repaired set of tracks.
| | |
|---|---|
| Code | [github.com/tidalove/molmo2fish](https://github.com/tidalove/molmo2fish) |
| Data | [tidalove/cfc-track-instruction](https://huggingface.co/datasets/tidalove/cfc-track-instruction) |
| Source video | [Caltech Fish Counting](https://huggingface.co/datasets/perona-lab/cfc26) |
## Files
| file | what it is |
|---|---|
| `*.safetensors`, `config.json`, … | HuggingFace-format weights at the repo root — what vLLM and `launch_scripts/hf_eval.py` consume |
| `Molmo2Fish-step420-raw.tar` | the raw training checkpoint (sharded model + optimizer state), for resuming fine-tuning |
## Training
Rank 64 LoRA fine-tuning of Molmo2-8B on the full CFC mixture, step 420. Adapters on all
three components — LLM, ViT, and connector — with the base weights frozen:
```bash
torchrun --nproc-per-node=8 launch_scripts/sft.py /path/to/Molmo2-8B cfc_correction \
--lora_llm --lora_vit --lora_connector --lora_rank 64 \
--save_folder=/path/to/save/folder
```
The `cfc_correction` mixture combines pure tracking, targeted correction, synthetically
corrupted correction, correction of real model predictions at two quality levels
(`molmo_high` / `molmo_low`), and text-only correction. See the
[dataset card](https://huggingface.co/datasets/tidalove/cfc-track-instruction) for what each
config contains.
## Citation
```bibtex
@article{molmo2fish,
title={Teach a Molmo2Fish: Towards interactive fish tracking with natural language guidance},
author={Kai van Brunt and Justin Kay and Sara Beery},
year={2026},
url={https://arxiv.org/abs/2608.18602}
}
```
|