andyqmongo/IRPO-mvtec-checkpoints

IRPO MVTec checkpoints (full-FT, 4-shot, combined excluded)

Full fine-tuned Qwen3-VL-8B-Instruct checkpoints from the IRPO deductive-stage work, trained on the MVTec single-defect split: 15 categories, 84 (category, class) pairs, combined excluded, 4-shot (4 support images/class = 336 training images). Held-out test = 1334 single-defect images.

Not comparable to the older combined-included checkpoints. The earlier NCHC sft-mvtec / rft-mvtec were trained with combined as a defect class (88 pairs, ~1373 test images) under DeepSpeed ZeRO (fp32 master). These are a different label space / test set — do not subtract their numbers against these.

Checkpoints

dir	what it is	key config	eval
sft/	supervised fine-tune (direct <think>/<answer>)	full-FT, 4 ep, lr 1e-5, bf16	0.6304 macro / 0.6184 micro (direct family)
rft/	GRPO answer-reward (direct_reward)	full-FT, 4 ep (1344 st), lr 1e-6, β0, temp0.9, num_gen 8, vLLM colocate	≈ base — see warning
opsd-golden-rule-gt/	OPSD self-distillation, teacher = golden rule + GT	full-FT, 4 ep, lr 1e-6, lmbda 1.0, β0.1, kl 1.0, frozen teacher	0.7340 (names-only, best)
opsd-base-rule-gt/	OPSD, teacher = base-model rule + GT	same as above	0.6667
opsd-gt-only/	OPSD, teacher = GT label only (no rule)	same as above (step 252)	0.6870

OPSD eval = opsd_train.evaluate names-only, mean-over-categories on the 1334-image test set (base = 0.4939). SFT/RFT eval = direct-family direct_eval.py (base-direct = 0.4568 macro). The two families are not subtractable (different system prompt + answer contract).

⚠️ rft/ is effectively UNTRAINED

It was full-FT at lr 1e-6 in raw bf16 with no fp32 master copy (DeepSpeed was dropped for the single-GPU vLLM-colocate setup), so ~all of each optimizer step rounds away below bf16 resolution. Its eval (0.4697 macro / 0.5045 micro) is within noise of the untrained base (0.4568 / 0.4880). Treat it as a base-model copy, not a trained RFT. (SFT escaped this because its lr is 1e-5.) A genuinely-trained RFT needs a fp32 master (DeepSpeed) or lr ≥ ~1e-4.

The golden-rule finding

With the base model's own induced rules, OPSD rule+GT (0.6667) lost to GT-only (0.6870). With golden rules (Opus-4.8, generated by looking at the support images; same all-pairs induction), everything else identical, rule+GT becomes the best condition (0.7340, +0.067 from rule quality alone). So rule value is bounded by rule quality, not by the "written-rule channel." Rule banks are in the IRPO repo under deductive_stage/data/ (mvtec_rulebank_golden_opus.json, _codex, _base).

⚠️ Precision — OPSD checkpoints are fp32 on purpose

The three opsd-* checkpoints are fp32 (33 GB). This is required to reproduce the reported macro numbers. The OPSD students were full-fine-tuned in fp32 at lr 1e-6, so the learned delta is tiny: ‖dW‖/‖W‖ ≈ 4e-4, an order of magnitude below bf16's ~3.9e-3 relative resolution. Exporting them to bf16 erases ~36% of that delta and reverts ~87% of the changed weights toward base, which measurably lowers eval. So keep them fp32. sft/ and rft/ are bf16 — SFT's delta is large enough to survive (RFT is an untrained no-op anyway).

Usage

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
# OPSD -> load fp32 to reproduce the numbers
m = AutoModelForImageTextToText.from_pretrained("andyqmongo/IRPO-mvtec-checkpoints",
        subfolder="opsd-golden-rule-gt", dtype=torch.float32)
p = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-8B-Instruct")

Base model + eval protocol: see the IRPO repo (deductive_stage/opsd_train.py / opsd_axis_eval.py, inductive_stage/eval_variants/direct_eval.py). OPSD eval is names-only (no rule at inference).