Model card

README.md

@@ -0,0 +1,84 @@
+---
+license: apache-2.0
+base_model: allenai/Molmo2-8B
+pipeline_tag: video-text-to-text
+library_name: transformers
+tags:
+- molmo2
+- video
+- object-tracking
+- ecology
+datasets:
+- tidalove/cfc-track-instruction
+- perona-lab/cfc26
+---
+
+# Molmo2Fish
+
+Molmo2Fish is [Molmo2-8B](https://github.com/allenai/molmo2) fine-tuned to **track fish in
+sonar video, and to edit those tracks in response to natural-language feedback**.
+
+Track correction is treated as a conversation: the model is shown a set of existing tracks —
+its own earlier predictions, another tracker's output, or corrupted ground truth — is told in
+words what is wrong with them, and returns a repaired set of tracks.
+
+| | |
+|---|---|
+| Code | [github.com/tidalove/molmo2fish](https://github.com/tidalove/molmo2fish) |
+| Data | [tidalove/cfc-track-instruction](https://huggingface.co/datasets/tidalove/cfc-track-instruction) |
+| Source video | [Caltech Fish Counting](https://huggingface.co/datasets/perona-lab/cfc26) |
+
+## Files
+
+| file | what it is |
+|---|---|
+| `*.safetensors`, `config.json`, … | HuggingFace-format weights at the repo root — what vLLM and `launch_scripts/hf_eval.py` consume |
+| `Molmo2Fish-step420-raw.tar` | the raw training checkpoint (sharded model + optimizer state), for resuming fine-tuning |
+
+## Training
+
+Rank 64 LoRA fine-tuning of Molmo2-8B on the full CFC mixture, step 420. Adapters on all
+three components — LLM, ViT, and connector — with the base weights frozen:
+
+```bash
+torchrun --nproc-per-node=8 launch_scripts/sft.py /path/to/Molmo2-8B cfc_correction \
+  --lora_llm --lora_vit --lora_connector --lora_rank 64 \
+  --save_folder=/path/to/save/folder
+```
+
+The `cfc_correction` mixture combines pure tracking, targeted correction, synthetically
+corrupted correction, correction of real model predictions at two quality levels
+(`molmo_high` / `molmo_low`), and text-only correction. See the
+[dataset card](https://huggingface.co/datasets/tidalove/cfc-track-instruction) for what each
+config contains.
+
+## Usage
+
+```bash
+git clone https://github.com/tidalove/molmo2fish.git && cd molmo2fish
+pip install torchcodec && pip install -e .[all]
+
+export MOLMO_DATA_DIR=./data
+python -m scripts.download_datasets cfc --n-procs 8
+
+hf download tidalove/Molmo2Fish --local-dir Molmo2Fish-HF/step420-hf
+python launch_scripts/hf_eval.py Molmo2Fish-HF/step420-hf \
+  cfc_hf_correction_molmo_low_full_eval_2fps
+```
+
+Correction tasks report `HOTA_before` (the tracks the model was handed), `HOTA_after` (what
+it returned), and `norm_delta_HOTA` (the fraction of available headroom it closed), alongside
+a per-river breakdown and the directional net-count error `nMAE`.
+
+Evaluation runs on 6 fps clips with tracks annotated at 2 fps, sliced from the original CFC
+videos — metrics are not comparable to those computed on the original CFC release.
+
+## Citation
+
+```bibtex
+@article{molmo2fish,
+    title={Teach a Molmo2Fish: Towards interactive fish tracking with natural language guidance},
+    author={Kai van Brunt and Justin Kay and Sara Beery},
+    year={2026}
+}
+```