Add transformer+TF-IDF ensemble (line-item top-1 0.597 -> 0.686) and document it in the card

README.md

@@ -16,6 +16,8 @@ tags:
 - bert
 - onnx
 - safetensors
+- tfidf
+- ensemble
 - specs
 - spec-writing
 - specifications
@@ -68,7 +70,9 @@ MasterFormat codes without a human choosing from a 171-row list.
 > estimate items is **59.7 %** (was 28.3 %), and whole-section accuracy is **60.7 %** with **81.0 %** at
 > division level. It is still **below this project's TF-IDF + embedding ensemble** on line items (70.2 %) —
 > see *Results* — but it now matches TF-IDF on section-level top-1 and beats every baseline on section-level
-> division accuracy.
+> division accuracy. To close the remaining line-item gap the repo also ships a **transformer + TF-IDF
+> ensemble** (`ensemble/`) that lifts line-item top-1 to **68.6 %** and gives the best manual-chunk and
+> section-accuracy results; see *Ensemble*.
 
 MasterFormat is a registered trademark of CSI / CSC. This project is independent and not affiliated with or
 endorsed by them.
@@ -156,6 +160,38 @@ division accuracy (0.810). The TF-IDF + embedding **ensemble still leads on shor
 0.597), which is the real-world target — so the ensemble remains the production candidate, with `mf-0.2` a
 much stronger transformer baseline than `mf-0.1`.
 
+## Ensemble (closing the line-item gap)
+
+The transformer is trained on specification prose, so on **short, terse estimate line items** a lexical
+TF-IDF model is still stronger (0.675 vs 0.597 top-1). To close that gap without giving up the transformer's
+long-text accuracy, this repo ships a **log-probability ensemble** of `mf-0.2` and a TF-IDF + SGD scorer:
+
+```
+p = softmax( 0.25 · log_softmax(mf-0.2)  +  0.75 · log_softmax(tfidf) )
+```
+
+The 0.25 weight is chosen on the UFGS **validation** split — not on the line-item test set. `ensemble/tfidf.joblib`
+holds the fitted vectorizer (word 1–2 grams, 100k features, `min_df=2`, sublinear tf) and SGD logistic
+classifier; `ensemble/blend.json` records the weight and all metrics; `ensemble/predict_ensemble.py` runs the blend.
+
+| Scorer | val top-1 | val division | line-item top-1 | manual-chunk top-1 | manual-section top-1 | manual-section division |
+|---|---:|---:|---:|---:|---:|---:|
+| mf-0.2 (transformer) | 0.466 | 0.593 | 0.597 | 0.280 | 0.607 | **0.810** |
+| TF-IDF (100k features) | 0.571 | 0.663 | 0.675 | 0.288 | 0.573 | 0.726 |
+| **Ensemble (w = 0.25)** | **0.586** | **0.688** | **0.686** | **0.334** | **0.613** | 0.784 |
+
+```bash
+pip install transformers torch scikit-learn joblib
+python ensemble/predict_ensemble.py --model constructelligence/masterformat-classifier \
+    --tfidf ensemble/tfidf.joblib --top-k 3 --divisions "4000 psi concrete slab on grade"
+```
+
+**Honest caveat.** The ensemble's line-item strength comes from TF-IDF: in a three-way blend with bge-small
+embeddings the transformer receives **zero** weight on line items (the best line-item result is **0.712** at
+0.70 TF-IDF + 0.30 embeddings). The transformer's own contribution is on **longer text** — it is the best
+scorer at manual-section *division* accuracy (0.810 vs 0.784 for the ensemble). A reasonable production split
+is the transformer for whole sections and short-answer text, the ensemble for terse line items.
+
 ## Intended use
 
 - **Use it for:** tagging construction text with a candidate MasterFormat group (top-3 shown), an
@@ -215,9 +251,11 @@ already present, parse it directly.
 **Does it work offline / in the browser?** Yes — the ONNX exports are intended for `onnxruntime` and
 `transformers.js`.
 
-**Is a better model available?** The project's TF-IDF + embedding **ensemble scores 70.2 % top-1 on
-hand-labelled line items** and remains the production candidate. Constructelligence's proprietary models are
-at [constructelligence.co](https://constructelligence.co).
+**Is a better model available?** Yes — this repo ships a **transformer + TF-IDF ensemble** (`ensemble/`) that
+scores **68.6 % top-1 on hand-labelled line items** (vs 59.7 % for the transformer alone) and is the best
+scorer on manual chunks. A TF-IDF + bge-small embedding ensemble reaches **71.2 %** but the transformer takes
+no weight there. Constructelligence's proprietary models are at
+[constructelligence.co](https://constructelligence.co).
 
 ## Files
 
@@ -227,6 +265,7 @@ at [constructelligence.co](https://constructelligence.co).
 - `metrics.json` — full held-out evaluation (`val`, `line_items`, `manuals`, `manual_sections`).
 - `predict.py` — CLI example: batching, `--top-k`, `--divisions`, `--json`, stdin/file input, `--onnx`.
 - `onnx/` — ONNX fp32 and int8 exports.
+- `ensemble/` — `tfidf.joblib`, `blend.json`, `predict_ensemble.py`: the transformer + TF-IDF line-item ensemble.
 - `CITATION.cff`, `requirements.txt`.
 
 ## Citation

ensemble/blend.json

@@ -0,0 +1,27 @@
+{
+ "weight": 0.25,
+ "max_features": 100000,
+ "val": {
+  "n": 11598,
+  "top1": 0.5863942058975685,
+  "top3": 0.7549577513364373,
+  "division": 0.6883083290222453
+ },
+ "line_items": {
+  "n": 194,
+  "top1": 0.6858638743455497,
+  "top3": 0.8272251308900523,
+  "division": 0.8195876288659794
+ },
+ "manuals": {
+  "n": 6890,
+  "top1": 0.33357825128581925,
+  "top3": 0.5049228508449669,
+  "division": 0.5136429608127722
+ },
+ "manual_sections": {
+  "sections": 153,
+  "top1": 0.6133333333333333,
+  "division": 0.7843137254901961
+ }
+}

ensemble/predict_ensemble.py

@@ -0,0 +1,143 @@
+"""Ensemble predictor: the mf-0.2 transformer + a TF-IDF/SGD scorer, blended in log-prob space.
+
+The transformer alone reaches 0.597 top-1 on the 194 hand-labelled estimate line items; the fitted TF-IDF
+scorer reaches 0.675; the blend reaches 0.686 (and improves whole-section and manual-chunk accuracy). This
+script reproduces the blend. The weight is chosen on the UFGS validation split (see `blend.json`), not on the
+line-item test set.
+
+    pip install transformers torch scikit-learn joblib
+
+    python predict_ensemble.py "EPDM membrane roofing" "4000 psi concrete slab on grade"
+    python predict_ensemble.py --top-k 5 --divisions "12\" RCP storm drain pipe"
+    python predict_ensemble.py --json --file items.txt > out.json
+
+    # default TF-IDF/results live beside this script; point --model at the Hub or a local run
+    python predict_ensemble.py --model constructelligence/masterformat-classifier \
+        --tfidf tfidf.joblib --weight 0.2 "Wet pipe sprinkler system, light hazard"
+"""
+import argparse
+import json
+import sys
+from pathlib import Path
+
+import numpy as np
+
+HERE = Path(__file__).resolve().parent
+MODEL = "constructelligence/masterformat-classifier"
+
+DIVISIONS = {
+    "01": "General Requirements", "02": "Existing Conditions", "03": "Concrete",
+    "04": "Masonry", "05": "Metals", "06": "Wood, Plastics, and Composites",
+    "07": "Thermal and Moisture Protection", "08": "Openings", "09": "Finishes",
+    "10": "Specialties", "11": "Equipment", "12": "Furnishings",
+    "13": "Special Construction", "14": "Conveying Equipment", "21": "Fire Suppression",
+    "22": "Plumbing", "23": "HVAC", "25": "Integrated Automation", "26": "Electrical",
+    "27": "Communications", "28": "Electronic Safety and Security", "31": "Earthwork",
+    "32": "Exterior Improvements", "33": "Utilities", "34": "Transportation",
+    "35": "Waterway and Marine Construction", "40": "Process Interconnections",
+    "41": "Material Processing and Handling Equipment",
+    "43": "Process Gas and Liquid Handling, Purification, and Storage Equipment",
+    "44": "Pollution and Waste Control Equipment", "46": "Water and Wastewater Equipment",
+    "48": "Electrical Power Generation",
+}
+
+
+def norm(lp):
+    return lp - np.logaddexp.reduce(lp, axis=1, keepdims=True)
+
+
+class Transformer:
+    def __init__(self, model, batch=64):
+        import torch
+        from transformers import AutoModelForSequenceClassification, AutoTokenizer
+        self.torch = torch
+        self.tok = AutoTokenizer.from_pretrained(model)
+        self.m = AutoModelForSequenceClassification.from_pretrained(model).eval()
+        self.labels = {int(k): v for k, v in self.m.config.id2label.items()}
+        self.batch = batch
+
+    def logprobs(self, texts):
+        out = []
+        with self.torch.no_grad():
+            for i in range(0, len(texts), self.batch):
+                enc = self.tok(texts[i:i + self.batch], truncation=True, max_length=128,
+                               padding=True, return_tensors="pt")
+                logits = self.m(**enc).logits.float()
+                out.append(self.torch.log_softmax(logits, -1).numpy())
+        return norm(np.concatenate(out))
+
+
+class Tfidf:
+    def __init__(self, path):
+        import joblib
+        d = joblib.load(path)
+        self.vec = d["vectorizer"]
+        self.clf = d["classifier"]
+        # classifier classes_ are indices into the sorted label list; verify against the transformer later
+        self.classes = list(self.clf.classes_)
+
+    def logprobs(self, texts):
+        return norm(self.clf.predict_log_proba(self.vec.transform(texts)))
+
+
+def main():
+    ap = argparse.ArgumentParser(description="MasterFormat ensemble (mf-0.2 + TF-IDF).")
+    ap.add_argument("text", nargs="*", help="text to classify; '-' reads stdin")
+    ap.add_argument("--model", default=MODEL, help="HF repo id or local checkpoint dir")
+    ap.add_argument("--tfidf", default=str(HERE / "tfidf.joblib"))
+    ap.add_argument("--weight", type=float, default=None,
+                    help="transformer weight in the blend (default: blend.json beside --tfidf)")
+    ap.add_argument("--top-k", type=int, default=3)
+    ap.add_argument("--divisions", action="store_true")
+    ap.add_argument("--file")
+    ap.add_argument("--json", action="store_true")
+    a = ap.parse_args()
+
+    if a.weight is None:
+        bf = Path(a.tfidf).with_name("blend.json")
+        a.weight = json.loads(bf.read_text())["weight"] if bf.exists() else 0.2
+    w = a.weight
+
+    texts = list(a.text)
+    if a.file:
+        texts += [l.rstrip("\n") for l in open(a.file, encoding="utf-8") if l.strip()]
+    if "-" in texts:
+        texts = [t for t in texts if t != "-"] + [l.rstrip("\n") for l in sys.stdin if l.strip()]
+    texts = [t for t in texts if t.strip()]
+    if not texts:
+        texts = ["EPDM membrane roofing"]
+
+    tr, tf = Transformer(a.model), Tfidf(a.tfidf)
+    assert tf.classes == list(range(len(tr.labels))), "TF-IDF classes do not align with the transformer labels"
+    lp = norm(w * tr.logprobs(texts) + (1 - w) * tf.logprobs(texts))   # blend of log-probs, renormalised
+
+    results = []
+    for text, row in zip(texts, lp):
+        order = np.argsort(-row)[:a.top_k]
+        preds = [{"code": tr.labels[int(i)][:8].strip(), "label": tr.labels[int(i)],
+                  "name": tr.labels[int(i)][8:].strip(), "score": round(float(np.exp(row[i])), 4)} for i in order]
+        item = {"text": text, "weight_transformer": w, "predictions": preds}
+        if a.divisions:
+            tot = {}
+            for p in preds:
+                tot[p["code"][:2]] = tot.get(p["code"][:2], 0.0) + p["score"]
+            item["divisions"] = [{"code": d, "name": DIVISIONS.get(d, d), "score": round(s, 4)}
+                                 for d, s in sorted(tot.items(), key=lambda kv: -kv[1])]
+        results.append(item)
+
+    if a.json:
+        json.dump(results, sys.stdout, indent=2, ensure_ascii=False)
+        print()
+        return
+    for r in results:
+        print(f"\n{r['text']}  (blend, transformer weight {w:g})")
+        for p in r["predictions"]:
+            print(f"  {p['code']} {p['name']:<48.48} {p['score']:.3f}")
+        if a.divisions:
+            print("  -- divisions --")
+            for d in r.get("divisions", []):
+                print(f"  {d['code']}      {d['name']:<48.48} {d['score']:.3f}")
+
+
+if __name__ == "__main__":
+    main()

ensemble/tfidf.joblib

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:957b80ab7b61f679513daa017e7ac3732d992bccb5d98b38affbb6808cdd6f2b
+size 62395513

requirements.txt

@@ -2,6 +2,10 @@
 transformers>=4.40
 torch>=2.0
 
-# ONNX path: pip install -r requirements.txt onnxruntime
+# ONNX path: pip install onnxruntime
 # onnxruntime>=1.17
 # numpy>=1.24
+
+# Ensemble (ensemble/predict_ensemble.py)
+# scikit-learn>=1.3
+# joblib>=1.3