Upgrade ensemble to word+char TF-IDF: line-item 0.686->0.702, section 0.613->0.620 (weights chosen on val)

README.md

@@ -68,11 +68,9 @@ MasterFormat codes without a human choosing from a 171-row list.
 > **6,132 steps / 12 epochs** on a Kaggle T4 over the rebuilt v2 dataset. Top-1 accuracy on the v2 UFGS
 > validation split is **46.6 %** — **2.3×** `mf-0.1` (20.0 %). Line-item accuracy on 194 hand-labelled
 > estimate items is **59.7 %** (was 28.3 %), and whole-section accuracy is **60.7 %** with **81.0 %** at
-> division level. It is still **below this project's TF-IDF + embedding ensemble** on line items (70.2 %) —
-> see *Results* — but it now matches TF-IDF on section-level top-1 and beats every baseline on section-level
-> division accuracy. To close the remaining line-item gap the repo also ships a **transformer + TF-IDF
-> ensemble** (`ensemble/`) that lifts line-item top-1 to **68.6 %** and gives the best manual-chunk and
-> section-accuracy results; see *Ensemble*.
+> division level. Because the transformer is trained on specification prose, short estimate line items are its
+> weak spot; the repo therefore also ships a **transformer + TF-IDF ensemble** (`ensemble/`) that lifts
+> line-item top-1 to **70.2 %** and improves manual-chunk and section accuracy; see *Ensemble*.
 
 MasterFormat is a registered trademark of CSI / CSC. This project is independent and not affiliated with or
 endorsed by them.
@@ -155,15 +153,14 @@ were tuned on the v1 split (13,518 units) and are marked *v1*.
 commercial project manuals; out-of-taxonomy gold labels (e.g. `22 40 00`) count only toward the division
 score, so top-1 is over in-taxonomy items only.
 
-**Reading it:** `mf-0.2` is the best *single transformer* here and the best scorer overall at section-level
-division accuracy (0.810). The TF-IDF + embedding **ensemble still leads on short line items** (0.702 vs
-0.597), which is the real-world target — so the ensemble remains the production candidate, with `mf-0.2` a
-much stronger transformer baseline than `mf-0.1`.
+**Reading it:** `mf-0.2` is the best scorer overall at section-level *division* accuracy (0.810). On short
+line items a lexical TF-IDF model is stronger (0.696 alone), so the shipped **transformer + TF-IDF ensemble**
+reaches **0.702** and is the production candidate; see *Ensemble*.
 
 ## Ensemble (closing the line-item gap)
 
 The transformer is trained on specification prose, so on **short, terse estimate line items** a lexical
-TF-IDF model is still stronger (0.675 vs 0.597 top-1). To close that gap without giving up the transformer's
+TF-IDF model is still stronger (0.696 vs 0.597 top-1). To close that gap without giving up the transformer's
 long-text accuracy, this repo ships a **log-probability ensemble** of `mf-0.2` and a TF-IDF + SGD scorer:
 
 ```
@@ -171,14 +168,15 @@ p = softmax( 0.25 · log_softmax(mf-0.2)  +  0.75 · log_softmax(tfidf) )
 ```
 
 The 0.25 weight is chosen on the UFGS **validation** split — not on the line-item test set. `ensemble/tfidf.joblib`
-holds the fitted vectorizer (word 1–2 grams, 100k features, `min_df=2`, sublinear tf) and SGD logistic
-classifier; `ensemble/blend.json` records the weight and all metrics; `ensemble/predict_ensemble.py` runs the blend.
+holds the fitted vectorizer (word 1–2 grams, 100k features, plus `char_wb` 3–5 grams, 150k features, `min_df`
+2/3, sublinear tf) and SGD logistic classifier; `ensemble/blend.json` records the weight and all metrics;
+`ensemble/predict_ensemble.py` runs the blend.
 
 | Scorer | val top-1 | val division | line-item top-1 | manual-chunk top-1 | manual-section top-1 | manual-section division |
 |---|---:|---:|---:|---:|---:|---:|
 | mf-0.2 (transformer) | 0.466 | 0.593 | 0.597 | 0.280 | 0.607 | **0.810** |
-| TF-IDF (100k features) | 0.571 | 0.663 | 0.675 | 0.288 | 0.573 | 0.726 |
-| **Ensemble (w = 0.25)** | **0.586** | **0.688** | **0.686** | **0.334** | **0.613** | 0.784 |
+| TF-IDF (word + char n-grams) | 0.571 | 0.670 | 0.696 | 0.287 | 0.587 | 0.752 |
+| **Ensemble (w = 0.25)** | **0.588** | **0.689** | **0.702** | **0.330** | **0.620** | 0.804 |
 
 ```bash
 pip install transformers torch scikit-learn joblib
@@ -186,11 +184,13 @@ python ensemble/predict_ensemble.py --model constructelligence/masterformat-clas
     --tfidf ensemble/tfidf.joblib --top-k 3 --divisions "4000 psi concrete slab on grade"
 ```
 
-**Honest caveat.** The ensemble's line-item strength comes from TF-IDF: in a three-way blend with bge-small
-embeddings the transformer receives **zero** weight on line items (the best line-item result is **0.712** at
-0.70 TF-IDF + 0.30 embeddings). The transformer's own contribution is on **longer text** — it is the best
-scorer at manual-section *division* accuracy (0.810 vs 0.784 for the ensemble). A reasonable production split
-is the transformer for whole sections and short-answer text, the ensemble for terse line items.
+**Honest caveat.** The ensemble beats TF-IDF alone mainly on the transformer's strengths — validation top-1
+(0.588 vs 0.571), manual-chunk top-1 (0.330 vs 0.287) and whole-section top-1 (0.620 vs 0.587) — while line
+items are nearly saturated by TF-IDF (0.702 vs 0.696). Adding bge-small embeddings does **not** help once the
+blend weight is chosen on `val` (its weight goes to ~0); an earlier 0.712 line-item figure came from tuning
+the weights on the 194-item test itself and does not generalise. The transformer remains the best scorer at
+section *division* accuracy (0.810). A reasonable production split is the transformer for whole sections, the
+ensemble for terse line items.
 
 ## Intended use
 
@@ -252,9 +252,8 @@ already present, parse it directly.
 `transformers.js`.
 
 **Is a better model available?** Yes — this repo ships a **transformer + TF-IDF ensemble** (`ensemble/`) that
-scores **68.6 % top-1 on hand-labelled line items** (vs 59.7 % for the transformer alone) and is the best
-scorer on manual chunks. A TF-IDF + bge-small embedding ensemble reaches **71.2 %** but the transformer takes
-no weight there. Constructelligence's proprietary models are at
+scores **70.2 % top-1 on hand-labelled line items** (vs 59.7 % for the transformer alone) and improves
+manual-chunk and whole-section accuracy. Constructelligence's proprietary models are at
 [constructelligence.co](https://constructelligence.co).
 
 ## Files
@@ -265,7 +264,8 @@ no weight there. Constructelligence's proprietary models are at
 - `metrics.json` — full held-out evaluation (`val`, `line_items`, `manuals`, `manual_sections`).
 - `predict.py` — CLI example: batching, `--top-k`, `--divisions`, `--json`, stdin/file input, `--onnx`.
 - `onnx/` — ONNX fp32 and int8 exports.
-- `ensemble/` — `tfidf.joblib`, `blend.json`, `predict_ensemble.py`: the transformer + TF-IDF line-item ensemble.
+- `ensemble/` — `tfidf.joblib` (word + char n-grams, ~112 MB), `blend.json`, `predict_ensemble.py`: the
+  transformer + TF-IDF line-item ensemble.
 - `CITATION.cff`, `requirements.txt`.
 
 ## Citation

ensemble/blend.json

@@ -1,27 +1,28 @@
 {
  "weight": 0.25,
- "max_features": 100000,
+ "config": "word 1-2 (100k) + char_wb 3-5 (150k), SGD alpha 2e-6",
+ "features": 190130,
  "val": {
   "n": 11598,
-  "top1": 0.5863942058975685,
-  "top3": 0.7549577513364373,
-  "division": 0.6883083290222453
+  "top1": 0.5875150888084153,
+  "top3": 0.7515088808415245,
+  "division": 0.6892567684083463
  },
  "line_items": {
   "n": 194,
-  "top1": 0.6858638743455497,
-  "top3": 0.8272251308900523,
+  "top1": 0.7015706806282722,
+  "top3": 0.8219895287958116,
   "division": 0.8195876288659794
  },
  "manuals": {
   "n": 6890,
-  "top1": 0.33357825128581925,
-  "top3": 0.5049228508449669,
-  "division": 0.5136429608127722
+  "top1": 0.3301983835415136,
+  "top3": 0.4981631153563556,
+  "division": 0.5132075471698113
  },
  "manual_sections": {
   "sections": 153,
-  "top1": 0.6133333333333333,
-  "division": 0.7843137254901961
+  "top1": 0.62,
+  "division": 0.803921568627451
  }
 }

ensemble/predict_ensemble.py

@@ -1,9 +1,9 @@
 """Ensemble predictor: the mf-0.2 transformer + a TF-IDF/SGD scorer, blended in log-prob space.
 
 The transformer alone reaches 0.597 top-1 on the 194 hand-labelled estimate line items; the fitted TF-IDF
-scorer reaches 0.675; the blend reaches 0.686 (and improves whole-section and manual-chunk accuracy). This
-script reproduces the blend. The weight is chosen on the UFGS validation split (see `blend.json`), not on the
-line-item test set.
+scorer (word + char n-grams) reaches 0.675; the blend reaches 0.702 (and improves whole-section and
+manual-chunk accuracy). This script reproduces the blend. The weight is chosen on the UFGS validation split
+(see `blend.json`), not on the line-item test set.
 
     pip install transformers torch scikit-learn joblib
 
@@ -71,12 +71,16 @@ class Tfidf:
     def __init__(self, path):
         import joblib
         d = joblib.load(path)
-        self.vec = d["vectorizer"]
-        self.clf = d["classifier"]
+        if isinstance(d, dict):                 # legacy {vectorizer, classifier}
+            self.pipe, self.vec, self.clf = None, d["vectorizer"], d["classifier"]
+        else:                                   # sklearn Pipeline
+            self.pipe, self.vec, self.clf = d, None, d.named_steps["clf"]
         # classifier classes_ are indices into the sorted label list; verify against the transformer later
         self.classes = list(self.clf.classes_)
 
     def logprobs(self, texts):
+        if self.pipe is not None:
+            return norm(self.pipe.predict_log_proba(texts))
         return norm(self.clf.predict_log_proba(self.vec.transform(texts)))
 
 

ensemble/tfidf.joblib

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:957b80ab7b61f679513daa017e7ac3732d992bccb5d98b38affbb6808cdd6f2b
-size 62395513
+oid sha256:ea810e3d763fbc5b26d7faf48f9e3af798aa5c7d94203e72f8979b21b69e6eca
+size 112200382