| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen3-VL-8B-Instruct |
| tags: |
| - visual-anomaly-detection |
| - mvtec |
| - opsd |
| - irpo |
| --- |
| |
| # IRPO MVTec checkpoints (full-FT, 4-shot, `combined` excluded) |
|
|
| Full fine-tuned **Qwen3-VL-8B-Instruct** checkpoints from the IRPO deductive-stage work, trained on the MVTec |
| **single-defect** split: 15 categories, 84 (category, class) pairs, **`combined` excluded**, **4-shot** (4 support |
| images/class = 336 training images). Held-out test = 1334 single-defect images. |
|
|
| > **Not comparable to the older `combined`-included checkpoints.** The earlier NCHC `sft-mvtec` / `rft-mvtec` |
| > were trained *with* `combined` as a defect class (88 pairs, ~1373 test images) under DeepSpeed ZeRO (fp32 |
| > master). These are a different label space / test set — do not subtract their numbers against these. |
|
|
| ## Checkpoints |
|
|
| | dir | what it is | key config | eval | |
| |---|---|---|---| |
| | `sft/` | supervised fine-tune (direct `<think>/<answer>`) | full-FT, 4 ep, lr 1e-5, bf16 | 0.6304 macro / 0.6184 micro (direct family) | |
| | `rft/` | GRPO answer-reward (`direct_reward`) | full-FT, 4 ep (1344 st), lr 1e-6, β0, temp0.9, num_gen 8, vLLM colocate | **≈ base — see warning** | |
| | `opsd-golden-rule-gt/` | OPSD self-distillation, teacher = **golden rule + GT** | full-FT, 4 ep, lr 1e-6, lmbda 1.0, β0.1, kl 1.0, frozen teacher | **0.7340** (names-only, best) | |
| | `opsd-base-rule-gt/` | OPSD, teacher = base-model rule + GT | same as above | 0.6667 | |
| | `opsd-gt-only/` | OPSD, teacher = GT label only (no rule) | same as above (step 252) | 0.6870 | |
| |
| OPSD eval = `opsd_train.evaluate` names-only, mean-over-categories on the 1334-image test set (base = 0.4939). |
| SFT/RFT eval = direct-family `direct_eval.py` (base-direct = 0.4568 macro). **The two families are not |
| subtractable** (different system prompt + answer contract). |
|
|
| ### ⚠️ `rft/` is effectively UNTRAINED |
| It was full-FT at lr 1e-6 in **raw bf16 with no fp32 master copy** (DeepSpeed was dropped for the single-GPU |
| vLLM-colocate setup), so ~all of each optimizer step rounds away below bf16 resolution. Its eval (0.4697 macro |
| / 0.5045 micro) is within noise of the untrained base (0.4568 / 0.4880). Treat it as a **base-model copy**, not |
| a trained RFT. (SFT escaped this because its lr is 1e-5.) A genuinely-trained RFT needs a fp32 master (DeepSpeed) |
| or lr ≥ ~1e-4. |
|
|
| ## The golden-rule finding |
| With the **base model's own** induced rules, OPSD `rule+GT` (0.6667) *lost* to `GT-only` (0.6870). With |
| **golden** rules (Opus-4.8, generated by looking at the support images; same all-pairs induction), everything |
| else identical, `rule+GT` becomes the **best** condition (**0.7340**, +0.067 from rule quality alone). So rule |
| value is bounded by rule quality, not by the "written-rule channel." Rule banks are in the IRPO repo under |
| `deductive_stage/data/` (`mvtec_rulebank_golden_opus.json`, `_codex`, `_base`). |
|
|
| ## ⚠️ Precision — OPSD checkpoints are fp32 on purpose |
| The three `opsd-*` checkpoints are **fp32** (33 GB). This is required to reproduce the reported macro numbers. |
| The OPSD students were full-fine-tuned in fp32 at lr 1e-6, so the learned delta is **tiny**: ‖dW‖/‖W‖ ≈ **4e-4**, |
| an order of magnitude *below* bf16's ~3.9e-3 relative resolution. Exporting them to bf16 **erases ~36% of that |
| delta and reverts ~87% of the changed weights toward base**, which measurably lowers eval. So keep them fp32. |
| `sft/` and `rft/` are **bf16** — SFT's delta is large enough to survive (RFT is an untrained no-op anyway). |
|
|
| ## Usage |
| ```python |
| import torch |
| from transformers import AutoModelForImageTextToText, AutoProcessor |
| # OPSD -> load fp32 to reproduce the numbers |
| m = AutoModelForImageTextToText.from_pretrained("andyqmongo/IRPO-mvtec-checkpoints", |
| subfolder="opsd-golden-rule-gt", dtype=torch.float32) |
| p = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-8B-Instruct") |
| ``` |
| Base model + eval protocol: see the IRPO repo (`deductive_stage/opsd_train.py` / `opsd_axis_eval.py`, |
| `inductive_stage/eval_variants/direct_eval.py`). OPSD eval is **names-only** (no rule at inference). |
|
|
