Add endoedge_qwen3vl8b_qlora_r512: Qwen3-VL-8B GI adapter, SAGE-benchmarked

endoedge_qwen3vl8b_qlora_r512/README.md

@@ -0,0 +1,106 @@
+---
+library_name: peft
+base_model: Qwen/Qwen3-VL-8B-Instruct
+tags: [medical, endoscopy, gastroenterology, multimodal, lora, qlora, json-extraction]
+license: apache-2.0
+---
+
+# endoedge_qwen3vl8b_qlora_r512
+
+LoRA adapter for **Qwen3-VL-8B-Instruct** fine-tuned to report gastrointestinal
+endoscopy findings as a single canonical JSON object.
+
+| | |
+|---|---|
+| method | QLORA, rank 512, alpha 256 |
+| trainable parameters | 1,396,703,232 (21.81% of total) |
+| adapter size | 2.60 GiB |
+| resident weights (base + adapter) | 10.87 GiB |
+| base model | `Qwen/Qwen3-VL-8B-Instruct`, revision `0c351dd01ed87e9c1b53cbc748cba10e6187ff3b` |
+| checkpoint published | `step_001000` (optimizer step 1000) |
+| training | 1,200 optimizer updates, effective global batch 64, 2xH100 |
+
+## Output format
+
+The model emits exactly eight keys, in this fixed order, `null` permitted:
+
+```json
+{"tract": ..., "landmark": ..., "abnormality_present": ..., "abnormality_type": ...,
+ "grade": ..., "prep_quality": ..., "procedure": ..., "instrument_present": ...}
+```
+
+Use the prompt it was trained with, or output quality will not match the numbers below:
+
+```
+system:  You are an expert gastrointestinal endoscopist. You read a single endoscopy
+         frame and report your findings as one JSON object.
+user:    <image>
+         Report the findings in this GI endoscopy frame as a single JSON object with
+         exactly these keys, in this order: tract, landmark, abnormality_present,
+         abnormality_type, grade, prep_quality, procedure, instrument_present.
+         Use null for any key this frame does not establish.
+```
+
+## Results
+
+**Held-out population shift — SAGE test split (420 images, South Asian GI endoscopy,
+never seen in training), projected onto SAGE's 18 multi-label classes:**
+
+| metric | value |
+|---|---|
+| macro-F1 | **0.2509** |
+| macro-F1, reachable labels only | 0.2657 |
+| micro-F1 | 0.3713 |
+| mAP weighted / macro / micro | 0.389 / 0.262 / 0.317 |
+| schema validity | 1.000 |
+
+Published baselines on the same split, trained on HyperKvasir or GastroVision
+(Oli et al. 2026): macro-F1 **0.1164 – 0.2431**.
+
+**In-domain** (stratified validation subset, mean per-field macro-F1): 0.6366
+
+**Inference** (NVIDIA H100, batch 16, greedy): 662 ms/image, 1.49 img/s,
+peak VRAM 23.17 GiB.
+
+## Known limitation
+
+On out-of-distribution images the model fills **all eight keys**, including fields the
+frame does not establish — measured unsupported-field fill rate is 1.000 on SAGE.
+Training masked unsupervised fields out of the loss, so the model received no gradient
+teaching it to abstain. Treat every non-null value as an assertion to verify, and gate
+on token confidence if precision matters.
+
+## Training data and attribution
+
+Trained on LIMUC, GastroVision and HyperKvasir (29,938 images), all **CC BY 4.0**.
+The attribution obligation travels with these weights — if you use them, cite:
+
+- Borgli H. et al. *HyperKvasir.* Scientific Data 7, 283 (2020).
+- Jha D. et al. *GastroVision.* ICML ML4MHD Workshop, 2023.
+- Polat G. et al. *LIMUC.* Inflammatory Bowel Diseases, izac196, 2022.
+
+Evaluated on SAGE (CC BY-SA 4.0; Oli et al. 2026, arXiv:2606.22144). SAGE was used
+for evaluation only and never for training or model selection, so no ShareAlike
+obligation attaches to these weights.
+
+## Intended use
+
+Research only. Not a medical device, not validated for clinical use, and not a
+substitute for endoscopist judgement.
+
+## Usage
+
+```python
+from peft import PeftModel
+from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
+
+base = Qwen3VLForConditionalGeneration.from_pretrained(
+    "Qwen/Qwen3-VL-8B-Instruct", dtype="bfloat16", device_map="auto")
+model = PeftModel.from_pretrained(base, "GutVLMmodels/experiments_checkpoints", subfolder="endoedge_qwen3vl8b_qlora_r512")
+processor = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-8B-Instruct")
+```
+
+## A note on which checkpoint this is
+
+The published adapter is `step_001000`, **not** the last one. Scored on a held-out in-domain validation subset, this run peaked before the end of training and the final step was worse. The step-1200 adapter is in `final_step_001200/` for completeness.
+

endoedge_qwen3vl8b_qlora_r512/adapter_config.json

@@ -0,0 +1,50 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen3-VL-8B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 256,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 512,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "up_proj",
+    "v_proj",
+    "k_proj",
+    "down_proj",
+    "o_proj",
+    "q_proj",
+    "gate_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

endoedge_qwen3vl8b_qlora_r512/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c95b1ecb135639c73be25f7d6c6b7b952d1a741d3f05018639494640f87fd64
+size 2793483904

endoedge_qwen3vl8b_qlora_r512/final_step_001200/adapter_config.json

@@ -0,0 +1,50 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen3-VL-8B-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 256,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.20.0",
+  "qalora_group_size": 16,
+  "r": 512,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "up_proj",
+    "v_proj",
+    "k_proj",
+    "down_proj",
+    "o_proj",
+    "q_proj",
+    "gate_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

endoedge_qwen3vl8b_qlora_r512/final_step_001200/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2ced45fb011e24a9e73f828c42ee34e42157d4767f73d5d9628d58d9ede7d6c7
+size 2793483904