Add evidence caveat, calibration/abstention (91% precision at 50% coverage) and improvement roadmap

IMPROVEMENT-PLAN.md

@@ -0,0 +1,126 @@
+# MasterFormat classifier — improvement plan
+
+**Date:** 2026-10-05 · **Owner:** Constructelligence
+**Published:** [`constructelligence/masterformat-classifier`](https://huggingface.co/constructelligence/masterformat-classifier)
+**Current:** `mf-0.2` transformer + `ensemble/` (word+char TF-IDF blend).
+
+| metric | published value | 95 % CI | n |
+|---|---:|---|---:|
+| line-item top-1 (hand-labelled estimates) | 0.702 | **[0.633, 0.762]** | 194 (191 in-taxonomy) |
+| whole-section top-1 | 0.620 | **[0.540, 0.694]** | 153 |
+| whole-section division | 0.804 | ±~6 pts | 153 |
+| UFGS val top-1 | 0.588 | [0.579, 0.596] | 11,598 |
+| manual-chunk top-1 | 0.330 | [0.319, 0.341] | 6,890 |
+
+---
+
+## 1. The binding constraint is evidence, not modelling
+
+The two metrics that matter for the product — **hand-labelled line items** and **whole real spec sections** —
+come from tiny samples. Their confidence intervals are **±6–8 points**, which means:
+
+- the recent `0.686 → 0.702` ensemble change is **statistically indistinguishable from noise**;
+- any model change smaller than ~10 points on line items cannot be validated on this eval;
+- the tight, large-sample metric (`val`, ±0.9 pts) is **in-domain UFGS text**, not the target distribution —
+  a model can improve `val` a lot (see `mf-0.3`: 0.466 → 0.556) while getting *worse* on real manuals.
+
+**Consequence:** the next gains are gated on **a bigger, real-world evaluation set**, not on architecture.
+Chasing sub-CI improvements (more TF-IDF features, more seeds) is a trap we have already fallen into once.
+
+## 2. Where the headroom actually is
+
+1. **Line items (estimating).** TF-IDF dominates; the transformer adds little. True ceiling unknown because
+   the eval is underpowered. Needs *real* labelled items, not synthetic augmentation (`mf-0.3` proved
+   synthetic terse augmentation does not transfer: line items stayed 0.597 and manuals regressed).
+2. **Section-level *top-1*** (0.62) is the weak spot, while *division* (0.81) is strong → most residual error
+   is **confusion within a division** (e.g. `03 30 00` vs `03 50 00`). A coarse-to-fine or contrastive
+   approach targets exactly this.
+3. **Out-of-taxonomy** sections only score at division level; there is no `other`/hierarchical fallback.
+4. **No level-3 / full 8-digit section codes** — real spec books use them.
+
+## 3. Definition of done (gates for any new model)
+
+| Gate | Target | How to verify |
+|---|---|---|
+| Powered eval | line items ≥ 1,000; sections ≥ 400 | Wilson CI ≤ ±3 pts |
+| Measurable lift | ≥ +3 pts line-item top-1 vs published, 95 % CI non-overlapping | paired bootstrap / McNemar |
+| No regression | manuals + section division within 1 pt | fixed eval sets |
+| Calibrated | ECE ≤ 0.05 after temperature scaling | reliability curve |
+| Useful abstention | ≥ 90 % precision at ≥ 50 % coverage on line items | coverage–precision curve |
+| Reproducible | one `cloud/run_kaggle.py --run <name>` + seed | identical metrics ± noise |
+
+## 4. Plan
+
+### P0 — Make the measurement trustworthy, then feed it real data
+
+**P0.1 — Enlarge the held-out eval (do first).**
+*Why:* everything downstream is unverifiable until this exists. *How:* assemble ≥1,000 real estimate line
+items and ≥400 real spec sections with true MasterFormat labels from public sources (§5); store as
+`eval/line_items_v2.tsv`, `eval/manuals_v2.jsonl`; report Wilson CIs + paired bootstrap for every comparison.
+*Done when:* CIs ≤ ±3 pts and the published vs candidate comparison reports a p-value. *Effort:* M.
+
+**P0.2 — Acquire real line-item training labels (not just eval).**
+*Why:* the transformer's line-item ceiling is a data problem, not a capacity problem. *How:* mine public bid
+tabulations / agency item catalogs; map agency item codes to MasterFormat where a crosswalk exists; treat the
+rest as weak/self-supervised signal. Keep a strict train/eval split by project. *Done when:* ≥10k labelled
+items, none from the eval projects. *Effort:* L.
+
+**P0.3 — Domain-adaptive pretraining corpus.**
+*Why:* the encoder only ever saw UFGS. *How:* collect public construction prose (specs, bid tabs, RFIs, submittal
+logs) beyond UFGS — VA design manuals, state DOT standard specs, Corps/UFC, NASA. Continued-MLM the encoder,
+then fine-tune. *Done when:* mf-0.5 beats the current model on the *enlarged* eval with non-overlapping CI.
+*Effort:* L (compute: 1–2 GPU-days).
+
+**P0.4 — Label space: level-3 sections + explicit out-of-taxonomy handling.**
+*Why:* real specs use 8-digit codes and many sections outside the 171 groups. *How:* extend the taxonomy to
+full sections; add an `other/<division>` fallback and score it honestly; revisit `group_of()`. *Effort:* M.
+
+### P1 — Modelling (only once P0.1 is in place)
+
+- **P1.1 Distillation** from the TF-IDF/ensemble teacher using **out-of-fold** soft targets (never on-data
+  teacher labels) into the transformer. Replaces the current brittle log-prob blend with one model.
+- **P1.2 Coarse-to-fine / hierarchical head** (division → group), directly attacking within-division confusion
+  and giving graceful out-of-taxonomy fallback.
+- **P1.3 Retrieval + rerank:** kNN over labelled embeddings for the top-k, plus a cross-encoder to reorder
+  them. Often the cheapest way to absorb new labelled data as it arrives.
+- **P1.4 Calibration + abstention:** temperature scaling on `val`; per-class thresholds; a `REVIEW` queue.
+  Ship this regardless of accuracy — it converts a probabilistic classifier into a usable one.
+- **P1.5 Multi-task:** shared encoder with group + division + section heads; consistency regularisation between
+  them.
+
+### P2 — Serving / ops
+
+- **P2.1 Routing policy:** transformer for whole sections / long text, ensemble for terse items (already the
+  evidence-backed split).
+- **P2.2 Quantization-aware / int8-calibrated ONNX** (the int8 export currently drifts; add QAT or
+  per-channel calibration and a browser benchmark).
+- **P2.3 Golden-file CI** on the enlarged eval; nightly regression that fails on a gate breach.
+- **P2.4 Metrics dashboard** with coverage/precision so abstention is tuned in production.
+
+## 5. Data sourcing shortlist (public unless noted)
+
+| Source | Use | Note |
+|---|---|---|
+| UFGS + UFC | train (in use) | US federal, public domain |
+| State DOT standard specs (Caltrans, TxDOT, WSDOT, MnDOT…) | pretrain / train | mostly public |
+| VA design manuals & specifications | pretrain / train | US federal, public |
+| NASA / DoD / Corps specs | pretrain | public |
+| Public **bid tabulations** (agency open-data portals; city/state capital projects) | line-item text + weak labels | item number + description; codes vary by agency |
+| SAM.gov / FPDS contract line items | weak labels | huge, noisy |
+| Public BOQ / cost datasets (e.g. Kaggle construction cost) | line items | check licence |
+| RSMeans / Gordian cost codes | labels | **proprietary** — licence required |
+
+## 6. Sequenced milestones
+
+1. **M1 (now):** publish the evaluation caveat + `cloud/stats.py` (CIs, abstention). Freeze `mf-0.2`+ensemble
+   as the reference. ✅ started in this change.
+2. **M2:** enlarged eval (P0.1) from public bid tabs + manuals; re-score the reference.
+3. **M3:** domain-adaptive pretrain + retrain (`mf-0.5`) on P0.2/P0.3 data; validate with CIs (P0.3).
+4. **M4:** distillation + hierarchical head (`mf-0.6`); pick the best single model (P1.1/P1.2).
+5. **M5:** calibration/abstention + routing in serving; publish with a coverage–precision card (P1.4, P2.1).
+
+## 7. Explicitly not doing
+
+- More synthetic line-item augmentation (tested: `mf-0.3` — no line-item transfer, manual regression).
+- More TF-IDF feature engineering (already saturated; gains below the CI).
+- Reporting sub-CI improvements as wins on the model card.

README.md

@@ -155,6 +155,11 @@ sets are **fixed**, so those columns are directly comparable for every scorer. `
 commercial project manuals; out-of-taxonomy gold labels (e.g. `22 40 00`) count only toward the division
 score, so top-1 is over in-taxonomy items only.
 
+**Evidence caveat.** The line-item and section sets are small. At n = 191 the 95 % Wilson interval on the
+line-item top-1 is **[0.633, 0.762] (±6.4 pts)**, and whole-section is ±~8 pts — so differences of a few
+points between models here are **noise**. Treat the `val` and `manual-chunk` rows (±1 pt) as the measurable
+ones, and the line-item/section rows as indicative. Enlarging these eval sets is the top item on the roadmap.
+
 **Reading it:** `mf-0.2` is the best scorer overall at section-level *division* accuracy (0.810). On short
 line items a lexical TF-IDF model is stronger (0.696 alone), so the shipped **transformer + TF-IDF ensemble**
 reaches **0.702** and is the production candidate; see *Ensemble*.
@@ -194,6 +199,34 @@ the weights on the 194-item test itself and does not generalise. The transformer
 section *division* accuracy (0.810). A reasonable production split is the transformer for whole sections, the
 ensemble for terse line items.
 
+## Calibration and abstention
+
+The ensemble is already close to calibrated; a single **temperature** `T = 0.90` (fit on `val`) lowers the
+expected calibration error on `val` from **0.060 to 0.022**. More useful in production: send the
+lowest-confidence predictions to a human instead of filing them.
+
+Coverage → precision on the 194 line items (predictions sorted by confidence):
+
+| auto-filed (coverage) | 100 % | 90 % | 80 % | 70 % | 50 % | 30 % |
+|---|---:|---:|---:|---:|---:|---:|
+| precision | 0.69 | 0.75 | 0.81 | **0.85** | **0.91** | 0.97 |
+
+Routing the least-confident **30 %** to review lifts precision to **91 %**; routing 50 % gives **97 %**. So
+the model is usable today as an **assist that flags its own uncertainty**, even though it is not accurate
+enough to file unattended. Reproduce with `python cloud/stats.py`.
+
+## Roadmap
+
+Further gains are **evidence-bound**, not architecture-bound (see the repo's `IMPROVEMENT-PLAN.md`):
+
+1. **Enlarge the eval** (≥1,000 line items, ≥400 sections) so changes are measurable at ±3 pts — currently
+   line-item changes below ~10 pts cannot be validated.
+2. **Real line-item training data** (public bid tabulations, agency item catalogs) — the transformer's
+   line-item ceiling is a data problem; synthetic augmentation was tested and did not transfer (`mf-0.3`).
+3. **Domain-adaptive pretraining** on construction prose beyond UFGS, then retrain.
+4. **Distillation + hierarchical (division → group) head** to beat the TF-IDF blend with a single model.
+5. **Level-3 / full-section codes** and an explicit out-of-taxonomy fallback.
+
 ## Intended use
 
 - **Use it for:** tagging construction text with a candidate MasterFormat group (top-3 shown), an
@@ -268,6 +301,7 @@ manual-chunk and whole-section accuracy. Constructelligence's proprietary models
 - `onnx/` — ONNX fp32 and int8 exports.
 - `ensemble/` — `tfidf.joblib` (word + char n-grams, ~112 MB), `blend.json`, `predict_ensemble.py`: the
   transformer + TF-IDF line-item ensemble.
+- `IMPROVEMENT-PLAN.md` — the prioritised roadmap (enlarged eval, data, domain-adaptive pretraining, distillation).
 - `CITATION.cff`, `requirements.txt`.
 
 ## Citation