Publish mf-0.2: completed GPU fine-tune (val 46.6%, 2.3x mf-0.1) + new model card with metrics/ONNX

CITATION.cff

@@ -1,5 +1,5 @@
 cff-version: 1.2.0
-title: "MasterFormat Classifier (mf-0.1)"
+title: "MasterFormat Classifier (mf-0.2)"
 message: "If you use this model, please cite it as below."
 type: software
 authors:
@@ -12,7 +12,7 @@ abstract: >-
   A BERT-based text classifier that maps construction line items, specification
   paragraphs and section titles to one of 171 MasterFormat level-2 groups across
   32 divisions. Fine-tuned from BAAI/bge-small-en-v1.5 on public-domain UFGS
-  specification text. Published as an early (step-700) checkpoint.
+  specification text. mf-0.2 is the completed 12-epoch GPU fine-tune.
 keywords:
   - MasterFormat
   - construction

README.md

@@ -33,41 +33,42 @@ model-index:
       type: text-classification
       name: MasterFormat level-2 group classification
     dataset:
-      type: ufgs-val
-      name: UFGS held-out text units (13,518)
+      type: ufgs-val-v2
+      name: UFGS held-out text units (11,598)
       split: validation
     metrics:
     - type: accuracy
-      value: 0.184
-      name: Top-1 accuracy (step-700 checkpoint)
+      value: 0.466
+      name: Top-1 accuracy (mf-0.2)
 widget:
-- text: 4000 psi concrete slab on grade
-  example_title: Cast-in-place concrete
-- text: TPO roofing, 60 mil, fully adhered
+- text: EPDM membrane roofing
   example_title: Membrane roofing
-- text: Cat 6 data cabling and jacks
-  example_title: Structured cabling
 - text: Wet pipe sprinkler system, light hazard
   example_title: Fire suppression
+- text: 8" CMU wall, grout filled at 32" o.c.
+  example_title: Unit masonry
+- text: Addressable fire alarm system, devices and panel
+  example_title: Fire alarm
 ---
 
-# MasterFormat classifier (mf-0.1) — construction spec & line-item classification
+# MasterFormat classifier (mf-0.2) — construction spec & line-item classification
 
 A BERT text classifier that maps **construction line items, specification paragraphs, submittals and
 section titles** to one of **171 MasterFormat level-2 groups** (e.g. `03 30 00 Cast-in-Place Concrete`,
 `23 30 00 HVAC Air Distribution`) across **32 divisions**. It is the machine-learning counterpart to a
-cost-code lookup table: hand it "TPO roofing, 60 mil, fully adhered" and it returns the MasterFormat group,
-ranked. English text, one label per input.
+cost-code lookup table: hand it "EPDM membrane roofing" and it returns the MasterFormat group, ranked.
+English text, one label per input.
 
 Built for estimating, takeoff, spec-writing and RAG pipelines that need to tag free text with CSI
 MasterFormat codes without a human choosing from a 171-row list.
 
-> **Early checkpoint, published as-is.** This is **step 700 of a planned 4,150-step fine-tune**
-> (`mf-0.1`); training was paused on an 8 GB CPU-only Mac. Top-1 accuracy on the UFGS validation split is
-> **18.4 %** (v1 split; **20.0 %** on the rebuilt v2 split) — well below this project's TF-IDF baseline
-> (**53.1 %** / **57.5 %**) on the same splits. Treat this release
-> as a reproducible starting point and a working deployment format, **not as a finished classifier**. A v2
-> dataset and trainer are in the repository; see *Model* and *Limitations*.
+> **Completed GPU fine-tune.** `mf-0.2` is the full run that `mf-0.1` (step 700) was an early checkpoint of:
+> **6,132 steps / 12 epochs** on a Kaggle T4 over the rebuilt v2 dataset. Top-1 accuracy on the v2 UFGS
+> validation split is **46.6 %** — **2.3×** `mf-0.1` (20.0 %). Line-item accuracy on 194 hand-labelled
+> estimate items is **59.7 %** (was 28.3 %), and whole-section accuracy is **60.7 %** with **81.0 %** at
+> division level. It is still **below this project's TF-IDF + embedding ensemble** on line items (70.2 %) —
+> see *Results* — but it now matches TF-IDF on section-level top-1 and beats every baseline on section-level
+> division accuracy.
 
 MasterFormat is a registered trademark of CSI / CSC. This project is independent and not affiliated with or
 endorsed by them.
@@ -78,19 +79,19 @@ endorsed by them.
 from transformers import pipeline
 
 clf = pipeline("text-classification", model="constructelligence/masterformat-classifier", top_k=3)
-for p in clf("4000 psi concrete slab on grade"):
+for p in clf("EPDM membrane roofing"):
     print(p["label"], round(p["score"], 3))
-# 04 20 00 Unit Masonry 0.082
-# 03 60 00 Grouting 0.076
-# 09 30 00 Tiling 0.063   <- not competitive yet; this is the step-700 checkpoint
+# 07 50 00 Membrane Roofing 0.863
+# 07 10 00 Dampproofing and Waterproofing 0.068
+# 07 30 00 Steep Slope Roofing 0.047
 ```
 
 Batched, with the level-1 division roll-up and JSON output:
 
 ```bash
 pip install -r requirements.txt        # transformers + torch
-python predict.py --top-k 5 --divisions "8\" CMU wall, grout filled at 32\" o.c."
-echo "Cat 6 data cabling and jacks" | python predict.py -
+python predict.py --top-k 5 --divisions "Addressable fire alarm system, devices and panel"
+echo "12\" RCP storm drain pipe" | python predict.py -
 python predict.py --file items.txt --json > out.json
 ```
 
@@ -98,7 +99,7 @@ ONNX (no torch, ~34 MB int8):
 
 ```bash
 pip install onnxruntime transformers
-python predict.py --onnx onnx/model_quantized.onnx "Panelboards 208/120V 42 circuit"
+python predict.py --onnx onnx/model_quantized.onnx "Wet pipe sprinkler system, light hazard"
 ```
 
 The encoder is English-only; inputs should be English.
@@ -107,71 +108,78 @@ The encoder is English-only; inputs should be English.
 
 `onnx/model.onnx` (fp32, 134 MB) and `onnx/model_quantized.onnx` (dynamic int8, 34 MB) are exported with
 dynamic batch and sequence axes, in the layout `transformers.js` / Optimum expect — inputs `input_ids`,
-`attention_mask`, `token_type_ids`, output `logits`. The fp32 export matches PyTorch on a 5-text smoke set
-(max |Δ logit| 5e-6, argmax agreement 5/5). Dynamic int8 quantization shifts low-confidence logits
-(max |Δ| ≈ 1.5 on the same smoke set, argmax agreement 3/5), so verify the quantized model on your own
-inputs before relying on it. Reproduce with `python scripts/export_onnx.py runs/mf-0.1`.
+`attention_mask`, `token_type_ids`, output `logits`. The fp32 export matches PyTorch on a 4-text smoke set
+(max |Δ logit| 1e-5, argmax agreement 4/4). Dynamic int8 quantization shifts the logits more than it did for
+`mf-0.1` — the better-trained head is more confident — so verify the quantized model on your own inputs;
+the fp32 export is the safer default. Reproduce with `python scripts/export_onnx.py runs/mf-0.2`.
 
 ## Model
 
 - **Base:** [`BAAI/bge-small-en-v1.5`](https://huggingface.co/BAAI/bge-small-en-v1.5) (BERT, 33M parameters, 384-d, MIT).
 - **Head:** 171-way sequence-classification layer; `id2label` / `label2id` are in [`config.json`](config.json).
-- **Recipe:** 2 planned epochs over the training set (per-class balance capped at 450 / floored at 300,
-  estimate-style augmentation), batch 32, max length 96, AdamW, 6 % linear warmup; body LR 6e-5, head LR 2e-3;
-  embeddings and the first 4 encoder layers frozen.
-- **Checkpoint:** stopped at step 700; `train_state.json` records `{"step": 700, "val_acc": 0.18725}`.
-- **A stronger v2 recipe already exists** in the source project (`label smoothing 0.05`, sentence-window
-  dataset, max length 64, lower learning rates) but no `mf-0.2` run has been completed yet.
+- **Recipe:** `label smoothing 0.05`, max length 64, batch 128, 12 epochs (6,132 steps), fp16 autocast,
+  AdamW with 6 % linear warmup; body LR 5e-5, head LR 1e-3; embeddings and the first 4 encoder layers frozen.
+- **Checkpoint:** `train_state.json` records `{"step": 6132, "val_acc": 0.4661}`.
+- **Trained on:** Kaggle T4, ~16.5 min wall-clock, from `BAAI/bge-small-en-v1.5` (not resumed from `mf-0.1`).
 
 ## Results
 
-All numbers are measured in this project on held-out data. `val` = UFGS text units, held out by a hash of
-the normalised source text; `line_items` = 194 hand-labelled estimate line items; `manuals` = 6,890 chunks
-from 153 sections of two real commercial project manuals. Out-of-taxonomy gold labels (e.g. `22 40 00`)
-count only toward the division score, so a row's top-1 is computed over in-taxonomy items only.
+All numbers are measured in this project on held-out data. The `line_items`, `manuals` and `manual_sections`
+sets are **fixed**, so those columns are directly comparable for every scorer. `val` depends on the dataset
+version: `mf-0.2` and `mf-0.1` below are on the **v2** split (11,598 units); the embeddings/ensemble rows
+were tuned on the v1 split (13,518 units) and are marked *v1*.
 
-| Scorer | val top-1 | val top-3 | val division | line-item top-1 | manual-chunk top-1 | manual-section top-1 |
-|---|---:|---:|---:|---:|---:|---:|
-| **mf-0.1 (step 700)** | **0.184** | 0.345 | 0.321 | 0.283 | 0.175 | 0.287 |
-| TF-IDF + linear SGD (full run) | 0.531 | 0.690 | 0.634 | 0.665 | 0.290 | 0.607 |
-| bge-small embeddings + logistic regression | 0.349 | 0.517 | 0.458 | 0.576 | 0.256 | 0.593 |
-| Ensemble (0.25 embedding + 0.75 TF-IDF) | **0.532** | **0.700** | **0.637** | **0.702** | **0.318** | **0.640** |
+**v2 validation split**
 
-The published checkpoint is the weakest scorer in this table. The best result on the real-world target
-(hand-labelled estimate line items) is the **ensemble at 70.2 % top-1**, which is why the project's next
-milestone is to finish the transformer run rather than ship `mf-0.1` as the answer.
+| Scorer | val top-1 | val top-3 | val division |
+|---|---:|---:|---:|
+| **mf-0.2 (step 6132)** | **0.466** | **0.641** | **0.593** |
+| mf-0.1 (step 700) | 0.200 | 0.372 | 0.356 |
+| TF-IDF + linear SGD | 0.575 | 0.732 | 0.675 |
 
-`val` is the only column that depends on the dataset version: `line_items`, `manuals` and `manual_sections`
-are fixed held-out sets, so those columns are directly comparable for every scorer. On the rebuilt **v2**
-validation split (11,598 units) the same `mf-0.1` checkpoint scores **0.200 top-1 / 0.372 top-3 / 0.356
-division**, against **0.575 / 0.732 / 0.675** for TF-IDF on that split. The `val` columns above are on the
-original v1 split (13,518 units).
+**Fixed held-out sets (line items · manual chunks · whole sections)**
+
+| Scorer | line-item top-1 | manual-chunk top-1 | manual-section top-1 | manual-section division |
+|---|---:|---:|---:|---:|
+| **mf-0.2 (step 6132)** | **0.597** | 0.280 | **0.607** | **0.810** |
+| mf-0.1 (step 700) | 0.283 | 0.176 | 0.287 | 0.588 |
+| TF-IDF + linear SGD | 0.665 | **0.290** | **0.607** | 0.732 |
+| bge-small embeddings + logistic regression *(v1)* | 0.576 | 0.256 | 0.593 | 0.725 |
+| Ensemble 0.25 embedding + 0.75 TF-IDF *(v1)* | **0.702** | **0.318** | 0.640 | 0.771 |
+
+`line_items` = 194 hand-labelled estimate line items; `manuals` = 6,890 chunks from 153 sections of two real
+commercial project manuals; out-of-taxonomy gold labels (e.g. `22 40 00`) count only toward the division
+score, so top-1 is over in-taxonomy items only.
+
+**Reading it:** `mf-0.2` is the best *single transformer* here and the best scorer overall at section-level
+division accuracy (0.810). The TF-IDF + embedding **ensemble still leads on short line items** (0.702 vs
+0.597), which is the real-world target — so the ensemble remains the production candidate, with `mf-0.2` a
+much stronger transformer baseline than `mf-0.1`.
 
 ## Intended use
 
-- **Use it for:** tagging construction text with a candidate MasterFormat group (top-3 shown), prototyping
-  an auto-classification step for estimating or spec workflows, and as a reproducible baseline to fine-tune
-  further.
+- **Use it for:** tagging construction text with a candidate MasterFormat group (top-3 shown), an
+  auto-classification step for estimating or spec workflows, and as a strong base to fine-tune or distil.
 - **Do not use it for:** unattended production takeoff, bid pricing, code compliance, or anything where a
-  wrong cost code has financial or contractual consequences. At 18.4 % top-1 it is not accurate enough.
-- **Not a substitute for review.** Always keep a human in the loop for classification decisions.
+  wrong cost code has financial or contractual consequences. Keep a human in the loop.
+- **Not a substitute for review.** Classifies text content only — if the input already contains a
+  MasterFormat number, read the number instead.
 
 ## Training data and taxonomy
 
 - **Source:** UFGS (Unified Facilities Guide Specifications) `.SEC` files — US federal works in the
   **public domain**. Paragraphs and titles are parsed into labelled text units; boilerplate shared by
   multiple sections is dropped, cross-references are stripped so section numbers cannot leak labels, and
-  long paragraphs are windowed into 20–60-word units.
+  long paragraphs are cut on sentence boundaries into 8–60-word windows.
 - **Split:** held out by a hash of the normalised source text (10 %), so a paragraph and its augmentations
-  stay on the same side. The `mf-0.1` run used 66,396 train / 13,518 val rows; the rebuilt v2 dataset has
-  65,496 / 11,598.
+  stay on the same side. **65,496 train / 11,598 val** rows (the v2 set).
 - **Taxonomy:** 171 level-2 groups over 32 MasterFormat divisions; group numbers follow the MasterFormat
   numbering convention and the short names are this project's own (see `config.json`).
 
 ## Limitations
 
-- **Early checkpoint.** The 18.4 % top-1 is a snapshot of an interrupted run; it is not competitive with
-  the TF-IDF baseline yet. Do not use it for production takeoff without further training and re-evaluation.
+- **Below the ensemble on line items.** 59.7 % vs 70.2 % on 194 hand-labelled estimate items; do not deploy
+  as the sole classifier.
 - **Domain skew.** UFGS over-represents heavy-civil, water/wastewater and process work relative to commercial
   building estimates; the training set is class-balanced, so raw predictions do not reflect building-project
   priors.
@@ -179,8 +187,7 @@ original v1 split (13,518 units).
   division level.
 - **Short, terse line items are the hardest inputs**; division (2-digit) accuracy is consistently higher than
   group (6-digit) accuracy.
-- **No section-number leakage, but no section-number help either.** The model classifies text content only;
-  if your input already contains a MasterFormat number, don't ask the model — read the number.
+- **Quantized ONNX drifts.** The int8 export is smaller but less faithful than `mf-0.1`'s; prefer fp32.
 
 ## Bias, risks and safety
 
@@ -196,9 +203,11 @@ original v1 split (13,518 units).
 specifications and cost data into numbered divisions and sections. This model predicts the **level-2 group**
 (a 6-digit code such as `03 30 00 Cast-in-Place Concrete`), not the full section number.
 
-**Can it classify a whole specification section?** Partly. Averaging the model's log-probabilities over a
-section's chunks is the intended way to score a section; at step 700 that gets 28.7 % top-1 on two real
-project manuals (53 % at division level). The TF-IDF baseline reaches 60.7 %.
+**How is this different from `mf-0.1`?** `mf-0.1` was step 700 of an interrupted CPU run; `mf-0.2` is the
+completed 12-epoch GPU fine-tune on the rebuilt dataset. Same architecture, ~2.3× the validation accuracy.
+
+**Can it classify a whole specification section?** Yes — average the model's log-probabilities over a
+section's chunks. On two real project manuals that gives 60.7 % top-1 and 81.0 % at division level.
 
 **Can it read a MasterFormat number out of the text?** No. It classifies the description. If the number is
 already present, parse it directly.
@@ -206,17 +215,16 @@ already present, parse it directly.
 **Does it work offline / in the browser?** Yes — the ONNX exports are intended for `onnxruntime` and
 `transformers.js`.
 
-**Is a better model available?** Not yet as a published checkpoint. The project's TF-IDF + embedding
-**ensemble scores 70.2 % top-1 on hand-labelled line items** and is the current production-candidate; the
-next transformer run (`mf-0.2`) is expected to close most of the gap. Constructelligence's proprietary models
-are at [constructelligence.co](https://constructelligence.co).
+**Is a better model available?** The project's TF-IDF + embedding **ensemble scores 70.2 % top-1 on
+hand-labelled line items** and remains the production candidate. Constructelligence's proprietary models are
+at [constructelligence.co](https://constructelligence.co).
 
 ## Files
 
 - `model.safetensors`, `config.json`, `tokenizer.json`, `tokenizer_config.json`, `vocab.txt`,
   `special_tokens_map.json` — standard `transformers` checkpoint.
-- `train_state.json` — step and validation accuracy of the saved checkpoint (text logs and optimizer state
-  are not published).
+- `train_state.json` — step and validation accuracy of the saved checkpoint.
+- `metrics.json` — full held-out evaluation (`val`, `line_items`, `manuals`, `manual_sections`).
 - `predict.py` — CLI example: batching, `--top-k`, `--divisions`, `--json`, stdin/file input, `--onnx`.
 - `onnx/` — ONNX fp32 and int8 exports.
 - `CITATION.cff`, `requirements.txt`.
@@ -225,7 +233,7 @@ are at [constructelligence.co](https://constructelligence.co).
 
 ```bibtex
 @misc{constructelligence_masterformat_classifier,
-  title        = {MasterFormat Classifier (mf-0.1): construction spec and line-item classification},
+  title        = {MasterFormat Classifier (mf-0.2): construction spec and line-item classification},
   author       = {Constructelligence},
   year         = {2026},
   howpublished = {\url{https://huggingface.co/constructelligence/masterformat-classifier}},

config.json

@@ -1,10 +1,13 @@
 {
+  "add_cross_attention": false,
   "architectures": [
     "BertForSequenceClassification"
   ],
   "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": null,
   "classifier_dropout": null,
   "dtype": "float32",
+  "eos_token_id": null,
   "hidden_act": "gelu",
   "hidden_dropout_prob": 0.1,
   "hidden_size": 384,
@@ -183,6 +186,7 @@
   },
   "initializer_range": 0.02,
   "intermediate_size": 1536,
+  "is_decoder": false,
   "label2id": {
     "01 10 00 Summary of Work": 0,
     "01 20 00 Price and Payment Procedures": 1,
@@ -363,8 +367,8 @@
   "num_hidden_layers": 12,
   "pad_token_id": 0,
   "position_embedding_type": "absolute",
-  "problem_type": "single_label_classification",
-  "transformers_version": "4.57.6",
+  "tie_word_embeddings": true,
+  "transformers_version": "5.16.1",
   "type_vocab_size": 2,
   "use_cache": true,
   "vocab_size": 30522

metrics.json

@@ -0,0 +1,25 @@
+{
+ "val": {
+  "n": 11598,
+  "top1": 0.4661148473874806,
+  "top3": 0.6408001379548198,
+  "division": 0.5929470598379031
+ },
+ "line_items": {
+  "n": 194,
+  "top1": 0.5968586387434555,
+  "top3": 0.7591623036649214,
+  "division": 0.7371134020618557
+ },
+ "manuals": {
+  "n": 6890,
+  "top1": 0.2797942689199118,
+  "top3": 0.44584864070536373,
+  "division": 0.46357039187227866
+ },
+ "manual_sections": {
+  "sections": 153,
+  "top1": 0.6066666666666667,
+  "division": 0.8104575163398693
+ }
+}

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0770e36bd9b8e8f054ac5750ab0da3078c1e9b4d00ce907a90e9b9f59fa048ca
+oid sha256:47d0e681abcfa7f5c588035bd451cc963e335d46bc1de91ef505cccc025a0af9
 size 133726636

onnx/model.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c359f7e6387049e671c1c2039c9e65003f80205183a5702134ebf0b4a39e3fe8
+oid sha256:10628d545f4b23125421a30ebd11678d7f20911d52c0a7d2e4c893f52c916827
 size 133960852

onnx/model_quantized.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3096cf891670cf74be258a21c860825dd3e4bd3aa3f4b48bb5e6d716ff4fca5a
+oid sha256:18a2d9bc236170e7ee6a06f27f7e1a73073d651d6e85dc2f0a8eb998dcd30055
 size 34067631

predict.py

@@ -8,7 +8,7 @@ Examples
     echo "Cat 6 data cabling and jacks" | python predict.py -
     python predict.py --file items.txt --json > out.json
 
-ONNX (no torch, ~4 MB quantized weights):
+ONNX (no torch, ~34 MB int8 weights):
     pip install onnxruntime transformers
     python predict.py --onnx onnx/model_quantized.onnx "Panelboards 208/120V 42 circuit"
 
@@ -116,7 +116,7 @@ def predict(texts, score, top_k, want_div):
 
 
 def main(argv=None):
-    ap = argparse.ArgumentParser(description="MasterFormat level-2 classifier (mf-0.1).")
+    ap = argparse.ArgumentParser(description="MasterFormat level-2 classifier (171 groups, 32 divisions).")
     ap.add_argument("text", nargs="*", help="text to classify; use '-' to read lines from stdin")
     ap.add_argument("--model", default=MODEL, help="HF repo id or local checkpoint dir")
     ap.add_argument("--onnx", metavar="PATH", help="classify with an ONNX model instead of PyTorch")

tokenizer.json

@@ -2,7 +2,7 @@
   "version": "1.0",
   "truncation": {
     "direction": "Right",
-    "max_length": 96,
+    "max_length": 128,
     "strategy": "LongestFirst",
     "stride": 0
   },

tokenizer_config.json

@@ -1,51 +1,11 @@
 {
-  "added_tokens_decoder": {
-    "0": {
-      "content": "[PAD]",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "100": {
-      "content": "[UNK]",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "101": {
-      "content": "[CLS]",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "102": {
-      "content": "[SEP]",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "103": {
-      "content": "[MASK]",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    }
-  },
+  "backend": "tokenizers",
   "clean_up_tokenization_spaces": true,
   "cls_token": "[CLS]",
   "do_basic_tokenize": true,
   "do_lower_case": true,
-  "extra_special_tokens": {},
+  "is_local": false,
+  "local_files_only": false,
   "mask_token": "[MASK]",
   "model_max_length": 512,
   "never_split": null,

train_state.json

@@ -1 +1 @@
-{"step": 700, "val_acc": 0.18725}
+{"step": 6132, "val_acc": 0.4661148473874806}