IMPROVEMENT-PLAN.md

# MasterFormat classifier — improvement plan

**Date:** 2026-10-05 · **Owner:** Constructelligence
**Published:** [`constructelligence/masterformat-classifier`](https://huggingface.co/constructelligence/masterformat-classifier)
**Current:** `mf-0.2` transformer + `ensemble/` (word+char TF-IDF blend).

| metric | published value | 95 % CI | n |
|---|---:|---|---:|
| line-item top-1 (hand-labelled estimates) | 0.702 | **[0.633, 0.762]** | 194 (191 in-taxonomy) |
| whole-section top-1 | 0.620 | **[0.540, 0.694]** | 153 |
| whole-section division | 0.804 | ±~6 pts | 153 |
| UFGS val top-1 | 0.588 | [0.579, 0.596] | 11,598 |
| manual-chunk top-1 | 0.330 | [0.319, 0.341] | 6,890 |

---

## 1. The binding constraint is evidence, not modelling

The two metrics that matter for the product — **hand-labelled line items** and **whole real spec sections** —
come from tiny samples. Their confidence intervals are **±6–8 points**, which means:

- the recent `0.686 → 0.702` ensemble change is **statistically indistinguishable from noise**;
- any model change smaller than ~10 points on line items cannot be validated on this eval;
- the tight, large-sample metric (`val`, ±0.9 pts) is **in-domain UFGS text**, not the target distribution —
  a model can improve `val` a lot (see `mf-0.3`: 0.466 → 0.556) while getting *worse* on real manuals.

**Consequence:** the next gains are gated on **a bigger, real-world evaluation set**, not on architecture.
Chasing sub-CI improvements (more TF-IDF features, more seeds) is a trap we have already fallen into once.

## 2. Where the headroom actually is

1. **Line items (estimating).** TF-IDF dominates; the transformer adds little. True ceiling unknown because
   the eval is underpowered. Needs *real* labelled items, not synthetic augmentation (`mf-0.3` proved
   synthetic terse augmentation does not transfer: line items stayed 0.597 and manuals regressed).
2. **Section-level *top-1*** (0.62) is the weak spot, while *division* (0.81) is strong → most residual error
   is **confusion within a division** (e.g. `03 30 00` vs `03 50 00`). A coarse-to-fine or contrastive
   approach targets exactly this.
3. **Out-of-taxonomy** sections only score at division level; there is no `other`/hierarchical fallback.
4. **No level-3 / full 8-digit section codes** — real spec books use them.

## 3. Definition of done (gates for any new model)

| Gate | Target | How to verify |
|---|---|---|
| Powered eval | line items ≥ 1,000; sections ≥ 400 | Wilson CI ≤ ±3 pts |
| Measurable lift | ≥ +3 pts line-item top-1 vs published, 95 % CI non-overlapping | paired bootstrap / McNemar |
| No regression | manuals + section division within 1 pt | fixed eval sets |
| Calibrated | ECE ≤ 0.05 after temperature scaling | reliability curve |
| Useful abstention | ≥ 90 % precision at ≥ 50 % coverage on line items | coverage–precision curve |
| Reproducible | one `cloud/run_kaggle.py --run <name>` + seed | identical metrics ± noise |

## 4. Plan

### P0 — Make the measurement trustworthy, then feed it real data

**P0.1 — Enlarge the held-out eval (do first).**
*Why:* everything downstream is unverifiable until this exists. *How:* assemble ≥1,000 real estimate line
items and ≥400 real spec sections with true MasterFormat labels from public sources (§5); store as
`eval/line_items_v2.tsv`, `eval/manuals_v2.jsonl`; report Wilson CIs + paired bootstrap for every comparison.
*Done when:* CIs ≤ ±3 pts and the published vs candidate comparison reports a p-value. *Effort:* M.

**P0.2 — Acquire real line-item training labels (not just eval).**
*Why:* the transformer's line-item ceiling is a data problem, not a capacity problem. *How:* mine public bid
tabulations / agency item catalogs; map agency item codes to MasterFormat where a crosswalk exists; treat the
rest as weak/self-supervised signal. Keep a strict train/eval split by project. *Done when:* ≥10k labelled
items, none from the eval projects. *Effort:* L.

**P0.3 — Domain-adaptive pretraining corpus.**
*Why:* the encoder only ever saw UFGS. *How:* collect public construction prose (specs, bid tabs, RFIs, submittal
logs) beyond UFGS — VA design manuals, state DOT standard specs, Corps/UFC, NASA. Continued-MLM the encoder,
then fine-tune. *Done when:* mf-0.5 beats the current model on the *enlarged* eval with non-overlapping CI.
*Effort:* L (compute: 1–2 GPU-days).

**P0.4 — Label space: level-3 sections + explicit out-of-taxonomy handling.**
*Why:* real specs use 8-digit codes and many sections outside the 171 groups. *How:* extend the taxonomy to
full sections; add an `other/<division>` fallback and score it honestly; revisit `group_of()`. *Effort:* M.

### P1 — Modelling (only once P0.1 is in place)

- **P1.1 Distillation** from the TF-IDF/ensemble teacher using **out-of-fold** soft targets (never on-data
  teacher labels) into the transformer. Replaces the current brittle log-prob blend with one model.
- **P1.2 Coarse-to-fine / hierarchical head** (division → group), directly attacking within-division confusion
  and giving graceful out-of-taxonomy fallback.
- **P1.3 Retrieval + rerank:** kNN over labelled embeddings for the top-k, plus a cross-encoder to reorder
  them. Often the cheapest way to absorb new labelled data as it arrives.
- **P1.4 Calibration + abstention:** temperature scaling on `val`; per-class thresholds; a `REVIEW` queue.
  Ship this regardless of accuracy — it converts a probabilistic classifier into a usable one.
- **P1.5 Multi-task:** shared encoder with group + division + section heads; consistency regularisation between
  them.

### P2 — Serving / ops

- **P2.1 Routing policy:** transformer for whole sections / long text, ensemble for terse items (already the
  evidence-backed split).
- **P2.2 Quantization-aware / int8-calibrated ONNX** (the int8 export currently drifts; add QAT or
  per-channel calibration and a browser benchmark).
- **P2.3 Golden-file CI** on the enlarged eval; nightly regression that fails on a gate breach.
- **P2.4 Metrics dashboard** with coverage/precision so abstention is tuned in production.

## 5. Data sourcing shortlist (public unless noted)

| Source | Use | Note |
|---|---|---|
| UFGS + UFC | train (in use) | US federal, public domain |
| State DOT standard specs (Caltrans, TxDOT, WSDOT, MnDOT…) | pretrain / train | mostly public |
| VA design manuals & specifications | pretrain / train | US federal, public |
| NASA / DoD / Corps specs | pretrain | public |
| Public **bid tabulations** (agency open-data portals; city/state capital projects) | line-item text + weak labels | item number + description; codes vary by agency |
| SAM.gov / FPDS contract line items | weak labels | huge, noisy |
| Public BOQ / cost datasets (e.g. Kaggle construction cost) | line items | check licence |
| RSMeans / Gordian cost codes | labels | **proprietary** — licence required |

## 6. Sequenced milestones

1. **M1 (now):** publish the evaluation caveat + `cloud/stats.py` (CIs, abstention). Freeze `mf-0.2`+ensemble
   as the reference. ✅ started in this change.
2. **M2:** enlarged eval (P0.1) from public bid tabs + manuals; re-score the reference.
3. **M3:** domain-adaptive pretrain + retrain (`mf-0.5`) on P0.2/P0.3 data; validate with CIs (P0.3).
4. **M4:** distillation + hierarchical head (`mf-0.6`); pick the best single model (P1.1/P1.2).
5. **M5:** calibration/abstention + routing in serving; publish with a coverage–precision card (P1.4, P2.1).

## 7. Explicitly not doing

- More synthetic line-item augmentation (tested: `mf-0.3` — no line-item transfer, manual regression).
- More TF-IDF feature engineering (already saturated; gains below the CI).
- Reporting sub-CI improvements as wins on the model card.