README.md

---
license: apache-2.0
base_model: Qwen/Qwen3-VL-8B-Instruct
tags:
  - visual-anomaly-detection
  - mvtec
  - opsd
  - irpo
---

# IRPO MVTec checkpoints (full-FT, 4-shot, `combined` excluded)

Full fine-tuned **Qwen3-VL-8B-Instruct** checkpoints from the IRPO deductive-stage work, trained on the MVTec
**single-defect** split: 15 categories, 84 (category, class) pairs, **`combined` excluded**, **4-shot** (4 support
images/class = 336 training images). Held-out test = 1334 single-defect images.

> **Not comparable to the older `combined`-included checkpoints.** The earlier NCHC `sft-mvtec` / `rft-mvtec`
> were trained *with* `combined` as a defect class (88 pairs, ~1373 test images) under DeepSpeed ZeRO (fp32
> master). These are a different label space / test set — do not subtract their numbers against these.

## Checkpoints

| dir | what it is | key config | eval |
|---|---|---|---|
| `sft/` | supervised fine-tune (direct `<think>/<answer>`) | full-FT, 4 ep, lr 1e-5, bf16 | 0.6304 macro / 0.6184 micro (direct family) |
| `rft/` | GRPO answer-reward (`direct_reward`) | full-FT, 4 ep (1344 st), lr 1e-6, β0, temp0.9, num_gen 8, vLLM colocate | **≈ base — see warning** |
| `opsd-golden-rule-gt/` | OPSD self-distillation, teacher = **golden rule + GT** | full-FT, 4 ep, lr 1e-6, lmbda 1.0, β0.1, kl 1.0, frozen teacher | **0.7340** (names-only, best) |
| `opsd-base-rule-gt/` | OPSD, teacher = base-model rule + GT | same as above | 0.6667 |
| `opsd-gt-only/` | OPSD, teacher = GT label only (no rule) | same as above (step 252) | 0.6870 |

OPSD eval = `opsd_train.evaluate` names-only, mean-over-categories on the 1334-image test set (base = 0.4939).
SFT/RFT eval = direct-family `direct_eval.py` (base-direct = 0.4568 macro). **The two families are not
subtractable** (different system prompt + answer contract).

### ⚠️ `rft/` is effectively UNTRAINED
It was full-FT at lr 1e-6 in **raw bf16 with no fp32 master copy** (DeepSpeed was dropped for the single-GPU
vLLM-colocate setup), so ~all of each optimizer step rounds away below bf16 resolution. Its eval (0.4697 macro
/ 0.5045 micro) is within noise of the untrained base (0.4568 / 0.4880). Treat it as a **base-model copy**, not
a trained RFT. (SFT escaped this because its lr is 1e-5.) A genuinely-trained RFT needs a fp32 master (DeepSpeed)
or lr ≥ ~1e-4.

## The golden-rule finding
With the **base model's own** induced rules, OPSD `rule+GT` (0.6667) *lost* to `GT-only` (0.6870). With
**golden** rules (Opus-4.8, generated by looking at the support images; same all-pairs induction), everything
else identical, `rule+GT` becomes the **best** condition (**0.7340**, +0.067 from rule quality alone). So rule
value is bounded by rule quality, not by the "written-rule channel." Rule banks are in the IRPO repo under
`deductive_stage/data/` (`mvtec_rulebank_golden_opus.json`, `_codex`, `_base`).

## ⚠️ Precision — OPSD checkpoints are fp32 on purpose
The three `opsd-*` checkpoints are **fp32** (33 GB). This is required to reproduce the reported macro numbers.
The OPSD students were full-fine-tuned in fp32 at lr 1e-6, so the learned delta is **tiny**: ‖dW‖/‖W‖ ≈ **4e-4**,
an order of magnitude *below* bf16's ~3.9e-3 relative resolution. Exporting them to bf16 **erases ~36% of that
delta and reverts ~87% of the changed weights toward base**, which measurably lowers eval. So keep them fp32.
`sft/` and `rft/` are **bf16** — SFT's delta is large enough to survive (RFT is an untrained no-op anyway).

## Usage
```python
import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
# OPSD -> load fp32 to reproduce the numbers
m = AutoModelForImageTextToText.from_pretrained("andyqmongo/IRPO-mvtec-checkpoints",
        subfolder="opsd-golden-rule-gt", dtype=torch.float32)
p = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-8B-Instruct")
```
Base model + eval protocol: see the IRPO repo (`deductive_stage/opsd_train.py` / `opsd_axis_eval.py`,
`inductive_stage/eval_variants/direct_eval.py`). OPSD eval is **names-only** (no rule at inference).