| --- |
| library_name: peft |
| license: llama2 |
| base_model: |
| - Qwen/Qwen3-VL-8B-Instruct |
| - openbmb/MiniCPM-V-4_5 |
| - OpenGVLab/InternVL3_5-8B-HF |
| - llava-hf/llava-v1.6-vicuna-13b-hf |
| - llava-hf/llava-onevision-qwen2-7b-ov-hf |
| datasets: |
| - ReefNet/ReefNet-1.0 |
| pipeline_tag: image-text-to-text |
| tags: |
| - lora |
| - llama-factory |
| - coral-reef |
| - biodiversity |
| - fine-grained-classification |
| - reefnet |
| --- |
| |
| # ReefNet LoRA adapters: few-shot, variance and rebuttal runs |
|
|
| Supplementary LoRA adapters from the ReefNet paper (NeurIPS 2026, Evaluations & Datasets track) and its rebuttal. |
| The main fine-tuned models have their own repos: |
| [Qwen3-VL](https://huggingface.co/ReefNet/reefnet-qwen3-vl-8b-lora), |
| [MiniCPM-V 4.5](https://huggingface.co/ReefNet/reefnet-minicpm-v-4.5-lora), |
| [InternVL3.5](https://huggingface.co/ReefNet/reefnet-internvl3.5-8b-lora). |
|
|
| Each subfolder is a self-contained PEFT adapter with its `train_config.yaml`, trainer logs and `predictions/`. |
| Few-shot support sets are nested: k=1 is a subset of k=5, which is a subset of k=10 and of k=20. They are sampled from the ReefNet |
| train split, and the sample lists are in the reefnet-mllm repository. The LLaVA-NeXT adapter inherits the Llama 2 |
| licence of its base model. The other adapters are Apache-2.0. |
|
|
| | subfolder | base model | what | ReefNet | Al-Wajh | |
| |---|---|---|---|---| |
| | `few_shot/k1/qwen3_vl` | Qwen3-VL-8B-Instruct | Table 4, 1-shot | 5.56 | 10.93 | |
| | `few_shot/k1/minicpm_v` | MiniCPM-V 4.5 | Table 4, 1-shot | 2.85 | 7.68 | |
| | `few_shot/k1/internvl3_5` | InternVL3.5-8B | Table 4, 1-shot | 2.00 | 10.31 | |
| | `few_shot/k5/qwen3_vl` | Qwen3-VL-8B-Instruct | Table 4, 5-shot | 9.15 | 13.77 | |
| | `few_shot/k5/minicpm_v` | MiniCPM-V 4.5 | Table 4, 5-shot | 7.32 | 8.52 | |
| | `few_shot/k5/internvl3_5` | InternVL3.5-8B | Table 4, 5-shot | 2.87 | 9.01 | |
| | `few_shot/k10/qwen3_vl` | Qwen3-VL-8B-Instruct | Table 4, 10-shot | 17.24 | 25.87 | |
| | `few_shot/k10/minicpm_v` | MiniCPM-V 4.5 | Table 4, 10-shot | 14.95 | 12.77 | |
| | `few_shot/k10/internvl3_5` | InternVL3.5-8B | Table 4, 10-shot | 10.44 | 12.80 | |
| | `few_shot/k20/qwen3_vl` | Qwen3-VL-8B-Instruct | Table 4, 20-shot | 27.22 | 31.27 | |
| | `few_shot/k20/minicpm_v` | MiniCPM-V 4.5 | Table 4, 20-shot | 21.95 | 20.88 | |
| | `few_shot/k20/internvl3_5` | InternVL3.5-8B | Table 4, 20-shot | 18.34 | 12.87 | |
| | `seed_variance/qwen3_vl_cap300_seed1` | Qwen3-VL-8B-Instruct | rebuttal: run-to-run variance, class-capped subsample (<=300/genus) | 50.96 | 34.72 | |
| | `seed_variance/qwen3_vl_cap300_seed2` | Qwen3-VL-8B-Instruct | rebuttal: run-to-run variance, class-capped subsample (<=300/genus) | 50.45 | 34.68 | |
| | `seed_variance/qwen3_vl_cap300_seed3` | Qwen3-VL-8B-Instruct | rebuttal: run-to-run variance, class-capped subsample (<=300/genus) | 50.16 | 32.66 | |
| | `convergence/qwen3_vl_cont_2p5ep` | Qwen3-VL-8B-Instruct | rebuttal: +0.5 epoch continuation | 72.75 | 56.94 | |
| | `llava/llava_next_13b_ckpt14000` | LLaVA-NeXT (Vicuna-13B) | rebuttal: LLaVA-NeXT, step 14000 | 61.70 | 38.62 | |
| | `llava/llava_onevision_7b_ckpt26000` | LLaVA-OneVision-7B | rebuttal: LLaVA-OneVision, step 26000, 3 GPUs (effective batch 24) | 60.50 | 35.39 | |
|
|
| Base model revisions: `Qwen/Qwen3-VL-8B-Instruct@0c351dd0`, `openbmb/MiniCPM-V-4_5@c1a6986b`, `OpenGVLab/InternVL3_5-8B-HF@741a7d03`, `llava-hf/llava-v1.6-vicuna-13b-hf@745cfbdb`, `llava-hf/llava-onevision-qwen2-7b-ov-hf@0d506805` |
|
|
| ## Usage (example: 20-shot Qwen3-VL) |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForImageTextToText, AutoProcessor |
| from peft import PeftModel |
| |
| base, rev = "Qwen/Qwen3-VL-8B-Instruct", "0c351dd01ed87e9c1b53cbc748cba10e6187ff3b" |
| model = AutoModelForImageTextToText.from_pretrained(base, revision=rev, torch_dtype=torch.bfloat16, device_map="auto") |
| model = PeftModel.from_pretrained(model, "ReefNet/reefnet-mllm-lora-extras", subfolder="few_shot/k20/qwen3_vl").eval() |
| processor = AutoProcessor.from_pretrained(base, revision=rev) |
| |
| # input: a 1024x1024 crop centred on the annotated point |
| messages = [{"role": "user", "content": [ |
| {"type": "image", "image": "crop.png"}, |
| {"type": "text", "text": "What is the genus in the center of the image? Answer with one word"}]}] |
| inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, |
| return_dict=True, return_tensors="pt").to(model.device) |
| out = model.generate(**inputs, max_new_tokens=32, do_sample=False) |
| print(processor.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)) |
| ``` |
| During training the image token sat inside the prompt text (see Prompts below). This snippet puts the image first, which |
| is how the chat template inserts it. |
|
|
| ## Training |
|
|
| LoRA fine-tuning with [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) @ `589da21d`. |
| The exact config is in `train_config.yaml`: rank 8 on all linear layers, AdamW, lr 5e-5, cosine schedule, |
| warmup ratio 0.03, bf16, per-device batch 8, no gradient accumulation, `image_max_pixels` 1024x1024, no augmentation. |
| Software: transformers 4.57.1, peft 0.18.1, torch 2.9.1. |
| Model selection used the ReefNet validation split. Al-Wajh is never used for training, tuning or selection. |
|
|
| ## Prompts (exact strings, `<image>` marks the image position) |
|
|
| | split | user prompt | target | |
| |---|---|---| |
| | train / val / ReefNet test | `What is the genus in the center of the image <image>? Answer with one word` | genus name | |
| | Al-Wajh test | `<image>Identify the hard coral genus in this image. Respond with only the genus name.` | genus name | |
|
|
| Decoding: greedy (`top_k=1`), `max_new_tokens=32`. The output is normalised (strip, capitalise) and must match a genus exactly. |
|
|
| ## Metric |
|
|
| Macro recall (balanced accuracy). ReefNet: 39 labels, 23,043 test crops. Al-Wajh: averaged over the 13 genera present |
| (4,624 crops), while the model keeps the full 39-way output. A prediction outside the 13 counts as an error. |
| `predictions/` holds the per-sample outputs behind the reported numbers. Recompute them with `tools/recompute_macro_recall.py` |
| from the reefnet-mllm repository. |
|
|
| ## Data |
|
|
| [ReefNet-1.0](https://huggingface.co/datasets/ReefNet/ReefNet-1.0): 1024x1024 crops centred on each point annotation. |
| Training uses the high-confidence benchmark split (445,985 train / 9,999 val). |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{felemban2026reefnet, |
| title = {ReefNet: A Large-Scale Dataset and Benchmark for Fine-Grained Coral Reef Recognition}, |
| author = {Felemban, Abdulwahab and Battach, Yahia and Khan, Faizan Farooq and Fu, Yuqian and Liu, Xuhui and |
| Khattab, Yesmeen and Radwan, Yousef A. and Li, Xiang and Dunn, Natalie and B{\"a}hr, Susanne and |
| Terraneo, Tullia Isotta and Marchese, Fabio and Beery, Sara and Jones, Burton H. and |
| Benzoni, Francesca and Elhoseiny, Mohamed}, |
| booktitle = {Advances in Neural Information Processing Systems (NeurIPS) Evaluations and Datasets Track}, |
| year = {2026} |
| } |
| ``` |
|
|
