Upload folder using huggingface_hub

COLAB_SETUP.md

@@ -30,6 +30,16 @@ print(torch.cuda.is_available(), torch.cuda.get_device_name(0) if torch.cuda.is_
 %pip install -q -r requirements.txt
 ```
 
+If you see Torch version conflicts like:
+
+- `torchvision ... requires torch==2.10.0, but you have torch 2.11.0`
+
+Pin matching versions (then restart the runtime):
+
+```python
+%pip install -q -U torch==2.10.0 torchvision==0.25.0 torchaudio==2.10.0
+```
+
 If `requirements.txt` is missing, install manually:
 
 ```python

HF_MODEL_CARD.md

@@ -43,7 +43,47 @@ Combines multitask intent modeling, supervised IAB content classification, and p
 
 ## Deployment Options
 
-### 1. `transformers.pipeline()` — one line anywhere
+### 0. Colab / Kaggle Quickstart (copy/paste)
+
+```python
+!pip -q install -U pip
+!pip -q install -U "torch==2.10.0" "torchvision==0.25.0" "torchaudio==2.10.0"
+!pip -q install -U "transformers>=4.36.0" "huggingface_hub>=0.20.0" "safetensors>=0.4.0"
+```
+
+Restart the runtime after installs (**Runtime → Restart runtime**) so the new Torch version is actually used.
+
+```python
+from transformers import pipeline
+
+clf = pipeline(
+    "admesh-intent",
+    model="admesh/agentic-intent-classifier",
+    trust_remote_code=True,  # required (custom pipeline + multi-model bundle)
+)
+
+out = clf("Which laptop should I buy for college?")
+print(out["meta"])
+print(out["model_output"]["classification"]["intent"])
+```
+
+---
+
+## Latency / inference timing (quick check)
+
+The first call includes model/code loading. Warm up once, then measure:
+
+```python
+import time
+q = "Which laptop should I buy for college?"
+
+_ = clf("warm up")
+t0 = time.perf_counter()
+out = clf(q)
+print(f"latency_ms={(time.perf_counter() - t0) * 1000:.1f}")
+```
+
+### 1. `transformers.pipeline()` — anywhere (Python)
 
 ```python
 from transformers import pipeline
@@ -142,6 +182,28 @@ clf = AdmeshIntentPipeline.from_pretrained("admesh/agentic-intent-classifier")
 
 ---
 
+## Troubleshooting (avoid environment errors)
+
+### `No module named 'combined_inference'` (or similar)
+
+This means the Hub repo root is missing required Python files. Ensure these exist at the **root of the model repo** (same level as `pipeline.py`):
+
+- `pipeline.py`, `config.json`, `config.py`
+- `combined_inference.py`, `schemas.py`
+- `model_runtime.py`, `multitask_runtime.py`, `multitask_model.py`
+- `inference_intent_type.py`, `inference_subtype.py`, `inference_decision_phase.py`, `inference_iab_classifier.py`
+- `iab_classifier.py`, `iab_taxonomy.py`
+
+### `does not appear to have a file named model.safetensors`
+
+Transformers requires a standard checkpoint at the repo root for `pipeline()` to initialize. This repo includes a **small dummy** `model.safetensors` + tokenizer files at the root for compatibility; the *real* production weights live in:
+
+- `multitask_intent_model_output/`
+- `iab_classifier_model_output/`
+- `artifacts/calibration/`
+
+---
+
 ## Example Output
 
 ```json

README.md

@@ -1,57 +1,10 @@
----
-language:
-- en
-library_name: transformers
-pipeline_tag: text-classification
-base_model: distilbert-base-uncased
-metrics:
-- accuracy
-- f1
-tags:
-- intent-classification
-- multitask
-- iab
-- conversational-ai
-- adtech
-- calibrated-confidence
-license: apache-2.0
----
-
-# admesh/agentic-intent-classifier
-
-Production-ready intent + IAB classifier bundle for conversational traffic.
-
-Combines multitask intent modeling, supervised IAB content classification, and per-head confidence calibration to support safe monetization decisions in real time.
-
-## Links
-
-- Hugging Face: https://huggingface.co/admesh/agentic-intent-classifier
-- GitHub: https://github.com/GouniManikumar12/agentic-intent-classifier
+# Agentic Intent Classifier
 
-## What It Predicts
+`agentic-intent-classifier` is a multi-head query classification stack for conversational traffic.
 
-| Field | Description |
-|---|---|
-| `intent.type` | `commercial`, `informational`, `navigational`, `transactional`, … |
-| `intent.subtype` | `product_discovery`, `comparison`, `how_to`, … |
-| `intent.decision_phase` | `awareness`, `consideration`, `decision`, … |
-| `iab_content` | IAB Content Taxonomy 3.0 tier1 / tier2 / tier3 labels |
-| `component_confidence` | Per-head calibrated confidence with threshold flags |
-| `system_decision` | Monetization eligibility, opportunity type, policy |
-
----
-
-## Deployment Options
-
-### 0. Colab / Kaggle Quickstart (copy/paste)
-
-```python
-!pip -q install -U pip
-!pip -q install -U "torch==2.10.0" "torchvision==0.25.0" "torchaudio==2.10.0"
-!pip -q install -U "transformers>=4.36.0" "huggingface_hub>=0.20.0" "safetensors>=0.4.0"
-```
+## Quickstart (recommended): run from Hugging Face Hub
 
-Restart the runtime after installs (**Runtime → Restart runtime**) so the new Torch version is actually used.
+This is the easiest way for developers to test the full production stack (multitask intent + IAB + calibration) without training locally.
 
 ```python
 from transformers import pipeline
@@ -59,212 +12,539 @@ from transformers import pipeline
 clf = pipeline(
     "admesh-intent",
     model="admesh/agentic-intent-classifier",
-    trust_remote_code=True,  # required (custom pipeline + multi-model bundle)
+    trust_remote_code=True,
 )
 
 out = clf("Which laptop should I buy for college?")
-print(out["meta"])
 print(out["model_output"]["classification"]["intent"])
+print(out["model_output"]["classification"]["iab_content"])
+print(out["meta"])
 ```
 
----
+If you’re running in Colab/Kaggle and see Torch version conflicts, follow `COLAB_SETUP.md`.
+
+## Latency / inference timing (developer quick check)
 
-## Latency / inference timing (quick check)
+The first call includes model/code loading; measure latency after a warm-up call.
 
-The first call includes model/code loading. Warm up once, then measure:
+Single query:
 
 ```python
 import time
+from transformers import pipeline
+
+clf = pipeline("admesh-intent", model="admesh/agentic-intent-classifier", trust_remote_code=True)
 q = "Which laptop should I buy for college?"
 
 _ = clf("warm up")
 t0 = time.perf_counter()
 out = clf(q)
-print(f"latency_ms={(time.perf_counter() - t0) * 1000:.1f}")
+dt_ms = (time.perf_counter() - t0) * 1000
+
+print(f"latency_ms={dt_ms:.1f}")
+print(out["model_output"]["classification"]["intent"])
 ```
 
-### 1. `transformers.pipeline()` — anywhere (Python)
+Warm p50 / p95 over 20 runs:
 
 ```python
-from transformers import pipeline
+import time, statistics
 
-clf = pipeline(
-    "admesh-intent",
-    model="admesh/agentic-intent-classifier",
-    trust_remote_code=True,
+times = []
+for _ in range(20):
+    t0 = time.perf_counter()
+    _ = clf(q)
+    times.append((time.perf_counter() - t0) * 1000)
+
+times_sorted = sorted(times)
+print(f"p50={statistics.median(times):.1f}ms p95={times_sorted[int(0.95*len(times))-1]:.1f}ms mean={statistics.mean(times):.1f}ms")
+```
+
+It currently produces:
+
+- `intent.type`
+- `intent.subtype`
+- `intent.decision_phase`
+- `iab_content`
+- calibrated confidence per head
+- combined fallback / policy / opportunity decisions
+
+The repo is beyond the original v0.1 baseline. It now includes:
+
+- shared config and label ownership
+- reusable model runtime
+- calibrated confidence and threshold gating
+- combined inference with fallback/policy logic
+- request/response validation in the demo API
+- repeatable evaluation and regression suites
+- full-TSV IAB taxonomy retrieval support through tier4
+- a local embedding index for taxonomy-node retrieval over IAB content paths
+- a separate synthetic full-intent-taxonomy augmentation dataset for non-IAB heads
+- a dedicated intent-type difficulty dataset and held-out benchmark with `easy`, `medium`, and `hard` cases
+- a dedicated decision-phase difficulty dataset and held-out benchmark with `easy`, `medium`, and `hard` cases
+
+Generated model weights are intentionally not committed.
+
+## Current Taxonomy
+
+### `intent.type`
+
+- `informational`
+- `exploratory`
+- `commercial`
+- `transactional`
+- `support`
+- `personal_reflection`
+- `creative_generation`
+- `chit_chat`
+- `ambiguous`
+- `prohibited`
+
+### `intent.decision_phase`
+
+- `awareness`
+- `research`
+- `consideration`
+- `decision`
+- `action`
+- `post_purchase`
+- `support`
+
+### `intent.subtype`
+
+- `education`
+- `product_discovery`
+- `comparison`
+- `evaluation`
+- `deal_seeking`
+- `provider_selection`
+- `signup`
+- `purchase`
+- `booking`
+- `download`
+- `contact_sales`
+- `task_execution`
+- `onboarding_setup`
+- `troubleshooting`
+- `account_help`
+- `billing_help`
+- `follow_up`
+- `emotional_reflection`
+
+### `iab_content`
+
+- candidates are derived from every row in [data/iab-content/Content Taxonomy 3.0.tsv](data/iab-content/Content%20Taxonomy%203.0.tsv)
+- retrieval output supports `tier1`, `tier2`, `tier3`, and optional `tier4`
+
+## What The System Does
+
+- runs three classifier heads:
+  - `intent_type`
+  - `intent_subtype`
+  - `decision_phase`
+- resolves `iab_content` through a local embedding index over taxonomy nodes plus generic label/path reranking
+- applies calibration artifacts when present
+- computes `commercial_score`
+- applies fallback when confidence is too weak or policy-safe blocking is required
+- emits a schema-validated combined envelope
+
+## What The System Does Not Do
+
+- it is not a multi-turn memory system
+- it is not a production-optimized low-latency serving path
+- it is not yet trained on large real-traffic human-labeled intent data
+- combined decision logic is still heuristic, even though it is materially stronger than the original baseline
+
+## Project Layout
+
+- [config.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/config.py): labels, thresholds, artifact paths, model paths
+- [model_runtime.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/model_runtime.py): shared calibrated inference runtime
+- [combined_inference.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/combined_inference.py): composed system response
+- [inference_intent_type.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/inference_intent_type.py): direct `intent_type` inference entrypoint
+- [inference_iab_classifier.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/inference_iab_classifier.py): direct supervised `iab_content` inference entrypoint
+- [schemas.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/schemas.py): request/response validation
+- [demo_api.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/demo_api.py): local validated API
+- [iab_taxonomy.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/iab_taxonomy.py): full taxonomy parser/index
+- [iab_classifier.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/iab_classifier.py): supervised IAB runtime with taxonomy-aware parent fallback
+- [iab_retrieval.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/iab_retrieval.py): optional shadow retrieval baseline
+- [training/build_full_intent_taxonomy_dataset.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/build_full_intent_taxonomy_dataset.py): separate synthetic intent augmentation dataset
+- [training/build_intent_type_difficulty_dataset.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/build_intent_type_difficulty_dataset.py): extra `intent_type` augmentation plus held-out difficulty benchmark
+- [training/build_decision_phase_difficulty_dataset.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/build_decision_phase_difficulty_dataset.py): extra `decision_phase` augmentation plus held-out difficulty benchmark
+- [training/build_subtype_difficulty_dataset.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/build_subtype_difficulty_dataset.py): extra `intent_subtype` augmentation plus held-out difficulty benchmark
+- [training/build_subtype_dataset.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/build_subtype_dataset.py): subtype dataset generation from existing corpora
+- [training/train_iab.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/train_iab.py): train the supervised IAB classifier head
+- [training/build_iab_taxonomy_embeddings.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/build_iab_taxonomy_embeddings.py): build local IAB node embedding artifacts
+- [training/run_full_training_pipeline.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/run_full_training_pipeline.py): full multi-head training/calibration/eval pipeline
+- [evaluation/run_evaluation.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/evaluation/run_evaluation.py): repeatable benchmark runner
+- [evaluation/run_regression_suite.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/evaluation/run_regression_suite.py): known-failure regression runner
+- [evaluation/run_iab_mapping_suite.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/evaluation/run_iab_mapping_suite.py): IAB behavior-lock regression runner
+- [evaluation/run_iab_quality_suite.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/evaluation/run_iab_quality_suite.py): curated IAB quality-target runner
+- [known_limitations.md](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/known_limitations.md): current gaps and caveats
+
+## Quickstart: Run From Hugging Face
+
+Download the trained bundle and run inference in three lines — no local training required.
+
+```python
+import sys
+from huggingface_hub import snapshot_download
+
+# Download the full bundle (models + calibration + code)
+local_dir = snapshot_download(
+    repo_id="admesh/agentic-intent-classifier",
+    repo_type="model",
 )
+sys.path.insert(0, local_dir)
+
+# Import and instantiate
+from pipeline import AdmeshIntentPipeline
+clf = AdmeshIntentPipeline()
 
+# Classify
+import json
 result = clf("Which laptop should I buy for college?")
+print(json.dumps(result, indent=2))
+```
+
+Or use the one-liner factory method:
+
+```python
+from pipeline import AdmeshIntentPipeline   # after sys.path.insert above
+
+clf = AdmeshIntentPipeline.from_pretrained("admesh/agentic-intent-classifier")
+result = clf("I need a CRM for a 5-person startup")
 ```
 
-Batch and custom thresholds:
+Batch mode and custom thresholds are also supported:
 
 ```python
-# batch
+# Batch
 results = clf([
     "Best running shoes under $100",
-    "How does TCP work?",
+    "How does gradient descent work?",
     "Buy noise-cancelling headphones",
 ])
 
-# custom confidence thresholds
+# Custom confidence thresholds
 result = clf(
-    "Buy headphones",
+    "Buy noise-cancelling headphones",
     threshold_overrides={"intent_type": 0.6, "intent_subtype": 0.35},
 )
 ```
 
+Verify artifacts and run a smoke test from the CLI:
+
+```bash
+cd "<local_dir>"
+python3 training/pipeline_verify.py
+python3 combined_inference.py "Which CRM should I buy for a 3-person startup?"
+```
+
+Pin a specific revision for reproducibility:
+
+```python
+local_dir = snapshot_download(
+    repo_id="admesh/agentic-intent-classifier",
+    repo_type="model",
+    revision="0584798f8efee6beccd778b0afa06782ab5add60",
+)
+```
+
 ---
 
-### 2. HF Inference Endpoints (managed, deploy to AWS / Azure / GCP)
+## Setup (for local training)
 
-1. Go to https://ui.endpoints.huggingface.co
-2. **New Endpoint** → select `admesh/agentic-intent-classifier`
-3. Framework: **PyTorch** — Task: **Text Classification**
-4. Enable **"Load with trust_remote_code"**
-5. Deploy
+```bash
+python3 -m venv .venv
+source .venv/bin/activate
+pip install -r agentic-intent-classifier/requirements.txt
+```
+
+## Inference (local training path)
 
-The endpoint serves the same `pipeline()` interface above via REST:
+Run one query locally:
 
 ```bash
-curl https://<your-endpoint>.endpoints.huggingface.cloud \
-  -H "Authorization: Bearer $HF_TOKEN" \
-  -H "Content-Type: application/json" \
-  -d '{"inputs": "Which laptop should I buy for college?"}'
+cd agentic-intent-classifier
+python3 training/train_iab.py
+python3 training/calibrate_confidence.py --head iab_content
+python3 combined_inference.py "Which CRM should I buy for a 3-person startup?"
 ```
 
----
+Run only the `intent_type` head:
 
-### 3. HF Spaces (Gradio / Streamlit demo)
+```bash
+cd agentic-intent-classifier
+python3 inference_intent_type.py "best shoes under 100"
+```
 
-```python
-# app.py for a Gradio Space
-import gradio as gr
-from transformers import pipeline
+Run the demo API:
 
-clf = pipeline(
-    "admesh-intent",
-    model="admesh/agentic-intent-classifier",
-    trust_remote_code=True,
-)
+```bash
+cd agentic-intent-classifier
+python3 demo_api.py
+```
 
-def classify(text):
-    return clf(text)
+Example request:
 
-gr.Interface(fn=classify, inputs="text", outputs="json").launch()
+```bash
+curl -sS -X POST http://127.0.0.1:8008/classify \
+  -H 'Content-Type: application/json' \
+  -d '{"text":"I cannot log into my account"}'
 ```
 
----
+Infra endpoints:
 
-### 4. Local / notebook via `snapshot_download`
+```bash
+curl -sS http://127.0.0.1:8008/health
+curl -sS http://127.0.0.1:8008/version
+```
 
-```python
-import sys
-from huggingface_hub import snapshot_download
+Train only the IAB classifier head:
 
-local_dir = snapshot_download(
-    repo_id="admesh/agentic-intent-classifier",
-    repo_type="model",
-)
-sys.path.insert(0, local_dir)
+```bash
+cd agentic-intent-classifier
+python3 training/train_iab.py
+python3 training/calibrate_confidence.py --head iab_content
+```
 
-from pipeline import AdmeshIntentPipeline
-clf = AdmeshIntentPipeline()
-result = clf("I need a CRM for a 5-person startup")
+The online `iab_content` path now uses the compact supervised classifier. Retrieval is still available as an optional shadow baseline.
+
+Build the optional retrieval shadow index:
+
+```bash
+cd agentic-intent-classifier
+python3 training/build_iab_taxonomy_embeddings.py
 ```
 
-Or the one-liner factory:
+By default the shadow retrieval path uses `Alibaba-NLP/gte-Qwen2-1.5B-instruct`. The retrieval runtime applies the model's query-side instruction format and last-token pooling, matching the Hugging Face usage guidance. If you want to point retrieval at a different embedding model, set `IAB_RETRIEVAL_MODEL_NAME_OVERRIDE` before building the index.
 
-```python
-from pipeline import AdmeshIntentPipeline
-clf = AdmeshIntentPipeline.from_pretrained("admesh/agentic-intent-classifier")
+Open-source users can swap in their own embedding model, but the contract is:
+
+- query embeddings and taxonomy-node embeddings must be produced by the same model and model revision
+- after changing models, you must rebuild `artifacts/iab/taxonomy_embeddings.pt`
+- the repository only tests and supports the default model path out of the box
+- not every Hugging Face embedding model is drop-in compatible with this runtime; some require custom pooling, query instructions, or `trust_remote_code`
+
+Example override:
+
+```bash
+cd agentic-intent-classifier
+export IAB_RETRIEVAL_MODEL_NAME_OVERRIDE=mixedbread-ai/mxbai-embed-large-v1
+python3 training/build_iab_taxonomy_embeddings.py
 ```
 
----
+This writes:
+
+- `artifacts/iab/taxonomy_nodes.json`
+- `artifacts/iab/taxonomy_embeddings.pt`
+
+## Training
+
+### Full local pipeline
+
+```bash
+cd agentic-intent-classifier
+python3 training/run_full_training_pipeline.py
+```
+
+This pipeline now does:
+
+1. build separate full-intent-taxonomy augmentation data
+2. build separate `intent_type` difficulty augmentation + benchmark
+3. train `intent_type`
+4. build subtype corpus
+5. build separate `intent_subtype` difficulty augmentation + benchmark
+6. train `intent_subtype`
+7. build separate `decision_phase` difficulty augmentation + benchmark
+8. train `decision_phase`
+9. train `iab_content`
+10. calibrate all classifier heads, including `iab_content`
+11. run regression/evaluation unless `--skip-full-eval` is used
+
+### Build datasets individually
+
+Separate full-intent augmentation:
+
+```bash
+cd agentic-intent-classifier
+python3 training/build_full_intent_taxonomy_dataset.py
+```
+
+Intent-type difficulty augmentation and benchmark:
+
+```bash
+cd agentic-intent-classifier
+python3 training/build_intent_type_difficulty_dataset.py
+```
+
+Decision-phase difficulty augmentation and benchmark:
+
+```bash
+cd agentic-intent-classifier
+python3 training/build_decision_phase_difficulty_dataset.py
+```
+
+Subtype difficulty augmentation and benchmark:
+
+```bash
+cd agentic-intent-classifier
+python3 training/build_subtype_difficulty_dataset.py
+```
+
+Subtype dataset:
+
+```bash
+cd agentic-intent-classifier
+python3 training/build_subtype_dataset.py
+```
+
+IAB embedding index:
+
+```bash
+cd agentic-intent-classifier
+python3 training/build_iab_taxonomy_embeddings.py
+```
+
+### Train heads individually
+
+```bash
+cd agentic-intent-classifier
+python3 training/train.py
+python3 training/train_subtype.py
+python3 training/train_decision_phase.py
+```
+
+### Calibration
+
+```bash
+cd agentic-intent-classifier
+python3 training/calibrate_confidence.py --head intent_type
+python3 training/calibrate_confidence.py --head intent_subtype
+python3 training/calibrate_confidence.py --head decision_phase
+```
+
+## Evaluation
+
+Full evaluation:
+
+```bash
+cd agentic-intent-classifier
+python3 evaluation/run_evaluation.py
+```
 
-## Troubleshooting (avoid environment errors)
+Known-failure regression:
 
-### `No module named 'combined_inference'` (or similar)
+```bash
+cd agentic-intent-classifier
+python3 evaluation/run_regression_suite.py
+```
 
-This means the Hub repo root is missing required Python files. Ensure these exist at the **root of the model repo** (same level as `pipeline.py`):
+IAB behavior-lock regression:
+
+```bash
+cd agentic-intent-classifier
+python3 evaluation/run_iab_mapping_suite.py
+```
 
-- `pipeline.py`, `config.json`, `config.py`
-- `combined_inference.py`, `schemas.py`
-- `model_runtime.py`, `multitask_runtime.py`, `multitask_model.py`
-- `inference_intent_type.py`, `inference_subtype.py`, `inference_decision_phase.py`, `inference_iab_classifier.py`
-- `iab_classifier.py`, `iab_taxonomy.py`
+IAB quality-target evaluation:
 
-### `does not appear to have a file named model.safetensors`
+```bash
+cd agentic-intent-classifier
+python3 evaluation/run_iab_quality_suite.py
+```
+
+Threshold sweeps:
+
+```bash
+cd agentic-intent-classifier
+python3 evaluation/sweep_intent_threshold.py
+```
 
-Transformers requires a standard checkpoint at the repo root for `pipeline()` to initialize. This repo includes a **small dummy** `model.safetensors` + tokenizer files at the root for compatibility; the *real* production weights live in:
+Artifacts are written to:
 
-- `multitask_intent_model_output/`
-- `iab_classifier_model_output/`
 - `artifacts/calibration/`
+- `artifacts/evaluation/latest/`
 
----
+## Google Colab
 
-## Example Output
-
-```json
-{
-  "model_output": {
-    "classification": {
-      "iab_content": {
-        "taxonomy": "IAB Content Taxonomy",
-        "taxonomy_version": "3.0",
-        "tier1": {"id": "552", "label": "Style & Fashion"},
-        "tier2": {"id": "579", "label": "Men's Fashion"},
-        "mapping_mode": "exact",
-        "mapping_confidence": 0.73
-      },
-      "intent": {
-        "type": "commercial",
-        "subtype": "product_discovery",
-        "decision_phase": "consideration",
-        "confidence": 0.9549,
-        "commercial_score": 0.656
-      }
-    }
-  },
-  "system_decision": {
-    "policy": {
-      "monetization_eligibility": "allowed_with_caution",
-      "eligibility_reason": "commercial_discovery_signal_present"
-    },
-    "opportunity": {"type": "soft_recommendation", "strength": "medium"}
-  },
-  "meta": {
-    "system_version": "0.6.0-phase4",
-    "calibration_enabled": true,
-    "iab_mapping_is_placeholder": false
-  }
-}
-```
-
-## Reproducible Revision
+Use Colab for the full retraining pass if local memory is limited.
 
-```python
-from huggingface_hub import snapshot_download
-local_dir = snapshot_download(
-    repo_id="admesh/agentic-intent-classifier",
-    repo_type="model",
-    revision="0584798f8efee6beccd778b0afa06782ab5add60",
-)
+Clone once:
+
+```bash
+%cd /content
+!git clone https://github.com/GouniManikumar12/agentic-intent-classifier.git
+%cd /content/agentic-intent-classifier
+```
+
+If the repo is already cloned and you want the latest code, pull manually:
+
+```bash
+!git pull origin main
 ```
 
-## Included Artifacts
+Full pipeline:
+
+```bash
+!python training/run_full_training_pipeline.py
+```
+
+If full evaluation is too heavy for the current Colab runtime:
+
+```bash
+!python training/run_full_training_pipeline.py \
+  --iab-embedding-batch-size 32 \
+  --skip-full-eval
+```
+
+Then run eval separately after training:
+
+```bash
+!python evaluation/run_regression_suite.py
+!python evaluation/run_iab_mapping_suite.py
+!python evaluation/run_iab_quality_suite.py
+!python evaluation/run_evaluation.py
+```
+
+## Current Saved Metrics
+
+Generate fresh metrics with:
+
+```bash
+cd agentic-intent-classifier
+python3 evaluation/run_evaluation.py
+```
+
+Do not treat any checked-in summary as canonical unless it was regenerated after the current code and artifacts were built. The IAB path is now retrieval-based, so older saved reports from the deleted hierarchy stack are not meaningful.
+
+## Latency Note
+
+`combined_inference.py` is a debugging/offline path, not a production latency path.
+
+Current production truth:
+
+- per-request CLI execution is not a sub-50ms architecture
+- production serving should use a long-lived API process with preloaded models
+- if sub-50ms becomes a hard requirement, the serving path will need:
+  - persistent loaded models
+  - runtime optimization
+  - likely fewer model passes or a shared multi-head model
+
+## Current Status
+
+Current repo status:
 
-| Path | Contents |
-|---|---|
-| `multitask_intent_model_output/` | DistilBERT multitask weights + tokenizer |
-| `iab_classifier_model_output/` | IAB content classifier weights + tokenizer |
-| `artifacts/calibration/` | Per-head temperature + threshold JSONs |
-| `pipeline.py` | `AdmeshIntentPipeline` (transformers.Pipeline subclass) |
-| `combined_inference.py` | Core inference logic |
+- full 10-class `intent.type` taxonomy is wired
+- subtype and phase heads are present
+- difficulty benchmarks are wired for `intent_type`, `intent_subtype`, and `decision_phase`
+- full-TSV IAB taxonomy retrieval is wired through tier4
+- separate full-intent augmentation dataset is in place
+- evaluation/runtime memory handling is improved for large IAB splits
 
-## Notes
+The main remaining gap is not basic infrastructure anymore. It is improving real-world robustness, especially for:
 
-- `trust_remote_code=True` is required because this model uses a custom multi-head architecture that does not map to a single standard `AutoModel` checkpoint.
-- `meta.iab_mapping_is_placeholder: true` means IAB artifacts were missing or skipped; train and calibrate IAB for full production accuracy.
-- For long-running servers, instantiate once and reuse — models are cached in memory after the first call.
+- `decision_phase`
+- `intent_subtype`
+- confidence quality on borderline commercial queries
+- real-traffic supervision beyond synthetic data

artifacts/calibration/decision_phase.json

@@ -1,20 +1,20 @@
 {
   "calibrated": true,
   "confidence_threshold": 0.22,
-  "generated_at": "2026-03-25T16:15:10.949430+00:00",
+  "generated_at": "2026-03-25T19:06:15.261600+00:00",
   "head": "decision_phase",
   "metrics": {
     "calibrated_accuracy": 0.8621,
-    "calibrated_expected_calibration_error": 0.0672,
-    "calibrated_negative_log_likelihood": 0.4798,
-    "mean_calibrated_confidence": 0.868,
-    "mean_raw_confidence": 0.868,
+    "calibrated_expected_calibration_error": 0.047,
+    "calibrated_negative_log_likelihood": 0.5014,
+    "mean_calibrated_confidence": 0.8653,
+    "mean_raw_confidence": 0.8672,
     "raw_accuracy": 0.8621,
-    "raw_expected_calibration_error": 0.0672,
-    "raw_negative_log_likelihood": 0.4798
+    "raw_expected_calibration_error": 0.0325,
+    "raw_negative_log_likelihood": 0.5015
   },
   "minimum_threshold_floor": 0.22,
-  "optimized_temperature_candidate": 1.000144,
+  "optimized_temperature_candidate": 1.007321,
   "selected_threshold_before_floor": {
     "accepted_accuracy": 0.8621,
     "coverage": 1.0,
@@ -22,7 +22,7 @@
   },
   "selection_split": "val",
   "selection_target_precision": 0.75,
-  "temperature": 1.000144,
+  "temperature": 1.007321,
   "temperature_scaling_applied": true,
   "threshold_summary": {
     "accepted_accuracy": 0.8621,

artifacts/calibration/iab_content.json

@@ -1,32 +1,32 @@
 {
   "calibrated": true,
   "confidence_threshold": 0.12,
-  "generated_at": "2026-03-25T16:17:01.813766+00:00",
+  "generated_at": "2026-03-25T19:08:04.708996+00:00",
   "head": "iab_content",
   "metrics": {
-    "calibrated_accuracy": 0.9308,
-    "calibrated_expected_calibration_error": 0.2692,
-    "calibrated_negative_log_likelihood": 0.5893,
-    "mean_calibrated_confidence": 0.6617,
-    "mean_raw_confidence": 0.1809,
-    "raw_accuracy": 0.9308,
-    "raw_expected_calibration_error": 0.75,
-    "raw_negative_log_likelihood": 1.9134
+    "calibrated_accuracy": 0.9442,
+    "calibrated_expected_calibration_error": 0.0283,
+    "calibrated_negative_log_likelihood": 0.2113,
+    "mean_calibrated_confidence": 0.9159,
+    "mean_raw_confidence": 0.1856,
+    "raw_accuracy": 0.9442,
+    "raw_expected_calibration_error": 0.7587,
+    "raw_negative_log_likelihood": 1.8642
   },
   "minimum_threshold_floor": 0.12,
-  "optimized_temperature_candidate": 0.55869,
+  "optimized_temperature_candidate": 0.26014,
   "selected_threshold_before_floor": {
-    "accepted_accuracy": 0.9308,
+    "accepted_accuracy": 0.9442,
     "coverage": 1.0,
     "threshold": 0.0
   },
   "selection_split": "val",
   "selection_target_precision": 0.7,
-  "temperature": 0.55869,
+  "temperature": 0.26014,
   "temperature_scaling_applied": true,
   "threshold_summary": {
-    "accepted_accuracy": 0.9436,
-    "coverage": 0.9775,
+    "accepted_accuracy": 0.9442,
+    "coverage": 1.0,
     "threshold": 0.12
   }
 }

artifacts/calibration/intent_subtype.json

@@ -1,31 +1,31 @@
 {
   "calibrated": true,
   "confidence_threshold": 0.25,
-  "generated_at": "2026-03-25T16:15:00.986765+00:00",
+  "generated_at": "2026-03-25T19:06:04.625853+00:00",
   "head": "intent_subtype",
   "metrics": {
-    "calibrated_accuracy": 0.875,
-    "calibrated_expected_calibration_error": 0.0667,
-    "calibrated_negative_log_likelihood": 0.3811,
-    "mean_calibrated_confidence": 0.8307,
-    "mean_raw_confidence": 0.7584,
-    "raw_accuracy": 0.875,
-    "raw_expected_calibration_error": 0.1314,
-    "raw_negative_log_likelihood": 0.4541
+    "calibrated_accuracy": 0.85,
+    "calibrated_expected_calibration_error": 0.0782,
+    "calibrated_negative_log_likelihood": 0.451,
+    "mean_calibrated_confidence": 0.826,
+    "mean_raw_confidence": 0.775,
+    "raw_accuracy": 0.85,
+    "raw_expected_calibration_error": 0.124,
+    "raw_negative_log_likelihood": 0.4945
   },
   "minimum_threshold_floor": 0.25,
-  "optimized_temperature_candidate": 0.824082,
+  "optimized_temperature_candidate": 0.868223,
   "selected_threshold_before_floor": {
-    "accepted_accuracy": 0.875,
+    "accepted_accuracy": 0.85,
     "coverage": 1.0,
     "threshold": 0.0
   },
   "selection_split": "val",
   "selection_target_precision": 0.75,
-  "temperature": 0.824082,
+  "temperature": 0.868223,
   "temperature_scaling_applied": true,
   "threshold_summary": {
-    "accepted_accuracy": 0.8734,
+    "accepted_accuracy": 0.8608,
     "coverage": 0.9875,
     "threshold": 0.25
   }

artifacts/calibration/intent_type.json

@@ -1,20 +1,20 @@
 {
   "calibrated": true,
   "confidence_threshold": 0.4,
-  "generated_at": "2026-03-25T16:14:49.053223+00:00",
+  "generated_at": "2026-03-25T19:05:52.623864+00:00",
   "head": "intent_type",
   "metrics": {
     "calibrated_accuracy": 0.9362,
-    "calibrated_expected_calibration_error": 0.0715,
-    "calibrated_negative_log_likelihood": 0.2384,
-    "mean_calibrated_confidence": 0.917,
-    "mean_raw_confidence": 0.8891,
+    "calibrated_expected_calibration_error": 0.0845,
+    "calibrated_negative_log_likelihood": 0.2091,
+    "mean_calibrated_confidence": 0.9221,
+    "mean_raw_confidence": 0.8936,
     "raw_accuracy": 0.9362,
-    "raw_expected_calibration_error": 0.0807,
-    "raw_negative_log_likelihood": 0.257
+    "raw_expected_calibration_error": 0.0771,
+    "raw_negative_log_likelihood": 0.2295
   },
   "minimum_threshold_floor": 0.4,
-  "optimized_temperature_candidate": 0.901567,
+  "optimized_temperature_candidate": 0.895563,
   "selected_threshold_before_floor": {
     "accepted_accuracy": 0.9362,
     "coverage": 1.0,
@@ -22,7 +22,7 @@
   },
   "selection_split": "val",
   "selection_target_precision": 0.8,
-  "temperature": 0.901567,
+  "temperature": 0.895563,
   "temperature_scaling_applied": true,
   "threshold_summary": {
     "accepted_accuracy": 0.9362,

artifacts/evaluation/latest/combined_demo_benchmark.json

@@ -11,8 +11,8 @@
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.676,
-            "mapping_mode": "nearest_equivalent",
+            "mapping_confidence": 0.5429,
+            "mapping_mode": "exact",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
             "tier1": {
@@ -22,6 +22,10 @@
             "tier2": {
               "id": "599",
               "label": "Computing"
+            },
+            "tier3": {
+              "id": "602",
+              "label": "Software and Applications"
             }
           },
           "intent": {
@@ -29,31 +33,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.961,
+                "confidence": 0.962,
                 "confidence_threshold": 0.22,
                 "label": "awareness",
                 "meets_threshold": true,
-                "raw_confidence": 0.9611
+                "raw_confidence": 0.9633
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.9853,
+                "confidence": 0.9805,
                 "confidence_threshold": 0.25,
                 "label": "education",
                 "meets_threshold": true,
-                "raw_confidence": 0.9516
+                "raw_confidence": 0.9549
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.9807,
+                "confidence": 0.9817,
                 "confidence_threshold": 0.4,
                 "label": "informational",
                 "meets_threshold": true,
-                "raw_confidence": 0.9655
+                "raw_confidence": 0.9658
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.961,
+            "confidence": 0.962,
             "decision_phase": "awareness",
             "subtype": "education",
             "summary": "Classified as informational intent with subtype education in the awareness phase.",
@@ -98,8 +102,8 @@
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.4448,
-            "mapping_mode": "nearest_equivalent",
+            "mapping_confidence": 0.4784,
+            "mapping_mode": "exact",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
             "tier1": {
@@ -112,31 +116,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.9381,
+                "confidence": 0.9277,
                 "confidence_threshold": 0.22,
                 "label": "awareness",
                 "meets_threshold": true,
-                "raw_confidence": 0.9381
+                "raw_confidence": 0.9297
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.9753,
+                "confidence": 0.9749,
                 "confidence_threshold": 0.25,
                 "label": "education",
                 "meets_threshold": true,
-                "raw_confidence": 0.9275
+                "raw_confidence": 0.9445
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.9768,
+                "confidence": 0.9797,
                 "confidence_threshold": 0.4,
                 "label": "informational",
                 "meets_threshold": true,
-                "raw_confidence": 0.9597
+                "raw_confidence": 0.9626
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.9381,
+            "confidence": 0.9277,
             "decision_phase": "awareness",
             "subtype": "education",
             "summary": "Classified as informational intent with subtype education in the awareness phase.",
@@ -181,13 +185,17 @@
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.7819,
-            "mapping_mode": "nearest_equivalent",
+            "mapping_confidence": 0.2179,
+            "mapping_mode": "exact",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
             "tier1": {
               "id": "483",
               "label": "Sports"
+            },
+            "tier2": {
+              "id": "496",
+              "label": "Equine Sports"
             }
           },
           "intent": {
@@ -195,31 +203,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.9691,
+                "confidence": 0.9444,
                 "confidence_threshold": 0.22,
                 "label": "consideration",
                 "meets_threshold": true,
-                "raw_confidence": 0.9691
+                "raw_confidence": 0.9461
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.563,
+                "confidence": 0.4804,
                 "confidence_threshold": 0.25,
                 "label": "comparison",
                 "meets_threshold": true,
-                "raw_confidence": 0.4806
+                "raw_confidence": 0.4327
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.9869,
+                "confidence": 0.981,
                 "confidence_threshold": 0.4,
                 "label": "commercial",
                 "meets_threshold": true,
-                "raw_confidence": 0.9756
+                "raw_confidence": 0.9653
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.563,
+            "confidence": 0.4804,
             "decision_phase": "consideration",
             "subtype": "comparison",
             "summary": "Classified as commercial intent with subtype comparison in the consideration phase.",
@@ -264,13 +272,17 @@
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.3576,
-            "mapping_mode": "nearest_equivalent",
+            "mapping_confidence": 0.3122,
+            "mapping_mode": "exact",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
             "tier1": {
               "id": "596",
               "label": "Technology & Computing"
+            },
+            "tier2": {
+              "id": "638",
+              "label": "Robotics"
             }
           },
           "intent": {
@@ -278,31 +290,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.9283,
+                "confidence": 0.8858,
                 "confidence_threshold": 0.22,
                 "label": "consideration",
                 "meets_threshold": true,
-                "raw_confidence": 0.9284
+                "raw_confidence": 0.8885
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.9727,
+                "confidence": 0.9538,
                 "confidence_threshold": 0.25,
                 "label": "comparison",
                 "meets_threshold": true,
-                "raw_confidence": 0.9236
+                "raw_confidence": 0.9083
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.9734,
+                "confidence": 0.9676,
                 "confidence_threshold": 0.4,
                 "label": "commercial",
                 "meets_threshold": true,
-                "raw_confidence": 0.954
+                "raw_confidence": 0.9435
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.9283,
+            "confidence": 0.8858,
             "decision_phase": "consideration",
             "subtype": "comparison",
             "summary": "Classified as commercial intent with subtype comparison in the consideration phase.",
@@ -347,17 +359,13 @@
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.3542,
-            "mapping_mode": "nearest_equivalent",
+            "mapping_confidence": 0.5309,
+            "mapping_mode": "exact",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
             "tier1": {
-              "id": "52",
-              "label": "Business and Finance"
-            },
-            "tier2": {
-              "id": "53",
-              "label": "Business"
+              "id": "596",
+              "label": "Technology & Computing"
             }
           },
           "intent": {
@@ -365,31 +373,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.8012,
+                "confidence": 0.6077,
                 "confidence_threshold": 0.22,
                 "label": "decision",
                 "meets_threshold": true,
-                "raw_confidence": 0.8012
+                "raw_confidence": 0.6097
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.9028,
+                "confidence": 0.7801,
                 "confidence_threshold": 0.25,
                 "label": "provider_selection",
                 "meets_threshold": true,
-                "raw_confidence": 0.8041
+                "raw_confidence": 0.6968
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.9759,
+                "confidence": 0.9843,
                 "confidence_threshold": 0.4,
                 "label": "commercial",
                 "meets_threshold": true,
-                "raw_confidence": 0.9582
+                "raw_confidence": 0.9703
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.8012,
+            "confidence": 0.6077,
             "decision_phase": "decision",
             "subtype": "provider_selection",
             "summary": "Classified as commercial intent with subtype provider_selection in the decision phase.",
@@ -434,13 +442,17 @@
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.2259,
-            "mapping_mode": "nearest_equivalent",
+            "mapping_confidence": 0.2299,
+            "mapping_mode": "exact",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
             "tier1": {
-              "id": "483",
-              "label": "Sports"
+              "id": "v9i3On",
+              "label": "Sensitive Topics"
+            },
+            "tier2": {
+              "id": "XtODT3",
+              "label": "Crime & Harmful Acts to Individuals, Society & Human Right Violations"
             }
           },
           "intent": {
@@ -448,31 +460,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.9176,
+                "confidence": 0.9662,
                 "confidence_threshold": 0.22,
                 "label": "action",
                 "meets_threshold": true,
-                "raw_confidence": 0.9176
+                "raw_confidence": 0.9674
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.9675,
+                "confidence": 0.9473,
                 "confidence_threshold": 0.25,
                 "label": "signup",
                 "meets_threshold": true,
-                "raw_confidence": 0.9135
+                "raw_confidence": 0.8993
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.9416,
+                "confidence": 0.9788,
                 "confidence_threshold": 0.4,
                 "label": "transactional",
                 "meets_threshold": true,
-                "raw_confidence": 0.909
+                "raw_confidence": 0.9614
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.9176,
+            "confidence": 0.9473,
             "decision_phase": "action",
             "subtype": "signup",
             "summary": "Classified as transactional intent with subtype signup in the action phase.",
@@ -517,7 +529,7 @@
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.5455,
+            "mapping_confidence": 0.8304,
             "mapping_mode": "exact",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
@@ -535,31 +547,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.9628,
+                "confidence": 0.9595,
                 "confidence_threshold": 0.22,
                 "label": "action",
                 "meets_threshold": true,
-                "raw_confidence": 0.9628
+                "raw_confidence": 0.9608
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.7841,
+                "confidence": 0.8434,
                 "confidence_threshold": 0.25,
                 "label": "booking",
                 "meets_threshold": true,
-                "raw_confidence": 0.6676
+                "raw_confidence": 0.7616
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.9761,
+                "confidence": 0.9805,
                 "confidence_threshold": 0.4,
                 "label": "transactional",
                 "meets_threshold": true,
-                "raw_confidence": 0.9583
+                "raw_confidence": 0.9649
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.7841,
+            "confidence": 0.8434,
             "decision_phase": "action",
             "subtype": "booking",
             "summary": "Classified as transactional intent with subtype booking in the action phase.",
@@ -604,8 +616,8 @@
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.3927,
-            "mapping_mode": "nearest_equivalent",
+            "mapping_confidence": 0.5261,
+            "mapping_mode": "exact",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
             "tier1": {
@@ -618,31 +630,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.9539,
+                "confidence": 0.9573,
                 "confidence_threshold": 0.22,
                 "label": "post_purchase",
                 "meets_threshold": true,
-                "raw_confidence": 0.9539
+                "raw_confidence": 0.9587
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.9652,
+                "confidence": 0.967,
                 "confidence_threshold": 0.25,
                 "label": "onboarding_setup",
                 "meets_threshold": true,
-                "raw_confidence": 0.9053
+                "raw_confidence": 0.9306
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.7786,
+                "confidence": 0.5834,
                 "confidence_threshold": 0.4,
                 "label": "transactional",
                 "meets_threshold": true,
-                "raw_confidence": 0.7173
+                "raw_confidence": 0.5253
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.7786,
+            "confidence": 0.5834,
             "decision_phase": "post_purchase",
             "subtype": "onboarding_setup",
             "summary": "Classified as transactional intent with subtype onboarding_setup in the post_purchase phase.",
@@ -687,13 +699,21 @@
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.2935,
-            "mapping_mode": "nearest_equivalent",
+            "mapping_confidence": 0.272,
+            "mapping_mode": "exact",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
             "tier1": {
               "id": "52",
               "label": "Business and Finance"
+            },
+            "tier2": {
+              "id": "53",
+              "label": "Business"
+            },
+            "tier3": {
+              "id": "72",
+              "label": "Business I.T."
             }
           },
           "intent": {
@@ -701,31 +721,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.9528,
+                "confidence": 0.9589,
                 "confidence_threshold": 0.22,
                 "label": "support",
                 "meets_threshold": true,
-                "raw_confidence": 0.9528
+                "raw_confidence": 0.9603
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.894,
+                "confidence": 0.8859,
                 "confidence_threshold": 0.25,
                 "label": "account_help",
                 "meets_threshold": true,
-                "raw_confidence": 0.8027
+                "raw_confidence": 0.8147
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.9636,
+                "confidence": 0.9699,
                 "confidence_threshold": 0.4,
                 "label": "support",
                 "meets_threshold": true,
-                "raw_confidence": 0.9403
+                "raw_confidence": 0.9476
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.894,
+            "confidence": 0.8859,
             "decision_phase": "support",
             "subtype": "account_help",
             "summary": "Classified as support intent with subtype account_help in the support phase.",
@@ -776,13 +796,13 @@
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.1521,
-            "mapping_mode": "nearest_equivalent",
+            "mapping_confidence": 0.7892,
+            "mapping_mode": "exact",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
             "tier1": {
-              "id": "v9i3On",
-              "label": "Sensitive Topics"
+              "id": "186",
+              "label": "Family and Relationships"
             }
           },
           "intent": {
@@ -790,31 +810,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.843,
+                "confidence": 0.9219,
                 "confidence_threshold": 0.22,
                 "label": "awareness",
                 "meets_threshold": true,
-                "raw_confidence": 0.843
+                "raw_confidence": 0.9239
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.9678,
+                "confidence": 0.9492,
                 "confidence_threshold": 0.25,
                 "label": "emotional_reflection",
                 "meets_threshold": true,
-                "raw_confidence": 0.9123
+                "raw_confidence": 0.9021
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.929,
+                "confidence": 0.9388,
                 "confidence_threshold": 0.4,
                 "label": "personal_reflection",
                 "meets_threshold": true,
-                "raw_confidence": 0.8937
+                "raw_confidence": 0.9059
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.843,
+            "confidence": 0.9219,
             "decision_phase": "awareness",
             "subtype": "emotional_reflection",
             "summary": "Classified as personal_reflection intent with subtype emotional_reflection in the awareness phase.",
@@ -865,8 +885,8 @@
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.0846,
-            "mapping_mode": "nearest_equivalent",
+            "mapping_confidence": 0.2238,
+            "mapping_mode": "exact",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
             "tier1": {
@@ -879,31 +899,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.8336,
+                "confidence": 0.8763,
                 "confidence_threshold": 0.22,
                 "label": "research",
                 "meets_threshold": true,
-                "raw_confidence": 0.8336
+                "raw_confidence": 0.8791
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.9685,
+                "confidence": 0.9683,
                 "confidence_threshold": 0.25,
                 "label": "follow_up",
                 "meets_threshold": true,
-                "raw_confidence": 0.9121
+                "raw_confidence": 0.9314
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.9583,
+                "confidence": 0.9623,
                 "confidence_threshold": 0.4,
                 "label": "ambiguous",
                 "meets_threshold": true,
-                "raw_confidence": 0.9339
+                "raw_confidence": 0.9367
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.8336,
+            "confidence": 0.8763,
             "decision_phase": "research",
             "subtype": "follow_up",
             "summary": "Classified as ambiguous intent with subtype follow_up in the research phase.",
@@ -954,13 +974,17 @@
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.2223,
-            "mapping_mode": "nearest_equivalent",
+            "mapping_confidence": 0.2371,
+            "mapping_mode": "exact",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
             "tier1": {
               "id": "391",
               "label": "Personal Finance"
+            },
+            "tier2": {
+              "id": "396",
+              "label": "Financial Planning"
             }
           },
           "intent": {
@@ -968,31 +992,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.9337,
+                "confidence": 0.9225,
                 "confidence_threshold": 0.22,
                 "label": "research",
                 "meets_threshold": true,
-                "raw_confidence": 0.9337
+                "raw_confidence": 0.9246
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.9493,
+                "confidence": 0.9586,
                 "confidence_threshold": 0.25,
                 "label": "follow_up",
                 "meets_threshold": true,
-                "raw_confidence": 0.875
+                "raw_confidence": 0.9146
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.9454,
+                "confidence": 0.9488,
                 "confidence_threshold": 0.4,
                 "label": "ambiguous",
                 "meets_threshold": true,
-                "raw_confidence": 0.9149
+                "raw_confidence": 0.9179
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.9337,
+            "confidence": 0.9225,
             "decision_phase": "research",
             "subtype": "follow_up",
             "summary": "Classified as ambiguous intent with subtype follow_up in the research phase.",
@@ -1043,17 +1067,13 @@
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.1233,
-            "mapping_mode": "exact",
+            "mapping_confidence": 0.2131,
+            "mapping_mode": "nearest_equivalent",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
             "tier1": {
-              "id": "239",
-              "label": "Hobbies & Interests"
-            },
-            "tier2": {
-              "id": "264",
-              "label": "Content Production"
+              "id": "42",
+              "label": "Books and Literature"
             }
           },
           "intent": {
@@ -1061,31 +1081,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.9632,
+                "confidence": 0.9861,
                 "confidence_threshold": 0.22,
                 "label": "action",
                 "meets_threshold": true,
-                "raw_confidence": 0.9632
+                "raw_confidence": 0.9867
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.9349,
+                "confidence": 0.7335,
                 "confidence_threshold": 0.25,
                 "label": "signup",
                 "meets_threshold": true,
-                "raw_confidence": 0.8563
+                "raw_confidence": 0.6454
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.9378,
+                "confidence": 0.9628,
                 "confidence_threshold": 0.4,
                 "label": "transactional",
                 "meets_threshold": true,
-                "raw_confidence": 0.9047
+                "raw_confidence": 0.938
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.9349,
+            "confidence": 0.7335,
             "decision_phase": "action",
             "subtype": "signup",
             "summary": "Classified as transactional intent with subtype signup in the action phase.",
@@ -1130,13 +1150,17 @@
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.2185,
-            "mapping_mode": "nearest_equivalent",
+            "mapping_confidence": 0.3327,
+            "mapping_mode": "exact",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
             "tier1": {
-              "id": "52",
-              "label": "Business and Finance"
+              "id": "596",
+              "label": "Technology & Computing"
+            },
+            "tier2": {
+              "id": "639",
+              "label": "Virtual Reality"
             }
           },
           "intent": {
@@ -1144,31 +1168,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.9572,
+                "confidence": 0.9295,
                 "confidence_threshold": 0.22,
                 "label": "consideration",
                 "meets_threshold": true,
-                "raw_confidence": 0.9572
+                "raw_confidence": 0.9315
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.9432,
+                "confidence": 0.9374,
                 "confidence_threshold": 0.25,
                 "label": "comparison",
                 "meets_threshold": true,
-                "raw_confidence": 0.8708
+                "raw_confidence": 0.8838
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.9622,
+                "confidence": 0.9602,
                 "confidence_threshold": 0.4,
                 "label": "commercial",
                 "meets_threshold": true,
-                "raw_confidence": 0.9374
+                "raw_confidence": 0.9329
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.9432,
+            "confidence": 0.9295,
             "decision_phase": "consideration",
             "subtype": "comparison",
             "summary": "Classified as commercial intent with subtype comparison in the consideration phase.",
@@ -1213,21 +1237,13 @@
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.1513,
+            "mapping_confidence": 0.3227,
             "mapping_mode": "exact",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
             "tier1": {
               "id": "596",
               "label": "Technology & Computing"
-            },
-            "tier2": {
-              "id": "599",
-              "label": "Computing"
-            },
-            "tier3": {
-              "id": "618",
-              "label": "Information and Network Security"
             }
           },
           "intent": {
@@ -1235,31 +1251,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.9609,
+                "confidence": 0.9535,
                 "confidence_threshold": 0.22,
                 "label": "awareness",
                 "meets_threshold": true,
-                "raw_confidence": 0.961
+                "raw_confidence": 0.955
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.9867,
+                "confidence": 0.9793,
                 "confidence_threshold": 0.25,
                 "label": "education",
                 "meets_threshold": true,
-                "raw_confidence": 0.9556
+                "raw_confidence": 0.9527
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.975,
+                "confidence": 0.9769,
                 "confidence_threshold": 0.4,
                 "label": "informational",
                 "meets_threshold": true,
-                "raw_confidence": 0.9567
+                "raw_confidence": 0.9584
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.9609,
+            "confidence": 0.9535,
             "decision_phase": "awareness",
             "subtype": "education",
             "summary": "Classified as informational intent with subtype education in the awareness phase.",

artifacts/evaluation/latest/decision_phase_difficulty_benchmark_confusion_matrix.csv

@@ -1,8 +1,8 @@
 ,awareness,research,consideration,decision,action,post_purchase,support
 awareness,14,1,0,0,0,0,0
 research,0,15,0,0,0,0,0
-consideration,0,1,14,0,0,0,0
-decision,0,0,0,15,0,0,0
-action,0,0,0,0,15,0,0
+consideration,0,2,13,0,0,0,0
+decision,0,1,0,14,0,0,0
+action,0,0,0,1,14,0,0
 post_purchase,0,0,0,0,0,15,0
 support,0,0,0,0,0,0,15

artifacts/evaluation/latest/decision_phase_difficulty_benchmark_report.json

@@ -1,7 +1,7 @@
 {
-  "accepted_accuracy": 0.981,
+  "accepted_accuracy": 0.9524,
   "accepted_coverage": 1.0,
-  "accuracy": 0.981,
+  "accuracy": 0.9524,
   "confusion_matrix_path": "/content/agentic-intent-classifier/artifacts/evaluation/latest/decision_phase_difficulty_benchmark_confusion_matrix.csv",
   "count": 105,
   "dataset_path": "/content/agentic-intent-classifier/data/decision_phase_benchmark.jsonl",
@@ -15,12 +15,12 @@
       "macro_f1": 0.9711
     },
     "hard": {
-      "accepted_accuracy": 0.9714,
+      "accepted_accuracy": 0.8857,
       "accepted_coverage": 1.0,
-      "accuracy": 0.9714,
+      "accuracy": 0.8857,
       "count": 35,
       "fallback_rate": 0.0,
-      "macro_f1": 0.9711
+      "macro_f1": 0.8908
     },
     "medium": {
       "accepted_accuracy": 1.0,
@@ -33,13 +33,13 @@
   },
   "fallback_rate": 0.0,
   "head": "decision_phase",
-  "macro_f1": 0.9812,
+  "macro_f1": 0.9536,
   "per_class_metrics": {
-    "accuracy": 0.9809523809523809,
+    "accuracy": 0.9523809523809523,
     "action": {
-      "f1-score": 1.0,
+      "f1-score": 0.9655172413793104,
       "precision": 1.0,
-      "recall": 1.0,
+      "recall": 0.9333333333333333,
       "support": 15.0
     },
     "awareness": {
@@ -49,21 +49,21 @@
       "support": 15.0
     },
     "consideration": {
-      "f1-score": 0.9655172413793104,
+      "f1-score": 0.9285714285714286,
       "precision": 1.0,
-      "recall": 0.9333333333333333,
+      "recall": 0.8666666666666667,
       "support": 15.0
     },
     "decision": {
-      "f1-score": 1.0,
-      "precision": 1.0,
-      "recall": 1.0,
+      "f1-score": 0.9333333333333333,
+      "precision": 0.9333333333333333,
+      "recall": 0.9333333333333333,
       "support": 15.0
     },
     "macro avg": {
-      "f1-score": 0.9812192118226601,
-      "precision": 0.9831932773109244,
-      "recall": 0.980952380952381,
+      "f1-score": 0.9536131694056934,
+      "precision": 0.9604010025062657,
+      "recall": 0.9523809523809524,
       "support": 105.0
     },
     "post_purchase": {
@@ -73,8 +73,8 @@
       "support": 15.0
     },
     "research": {
-      "f1-score": 0.9375,
-      "precision": 0.8823529411764706,
+      "f1-score": 0.8823529411764706,
+      "precision": 0.7894736842105263,
       "recall": 1.0,
       "support": 15.0
     },
@@ -85,9 +85,9 @@
       "support": 15.0
     },
     "weighted avg": {
-      "f1-score": 0.9812192118226601,
-      "precision": 0.9831932773109243,
-      "recall": 0.9809523809523809,
+      "f1-score": 0.9536131694056934,
+      "precision": 0.9604010025062656,
+      "recall": 0.9523809523809523,
       "support": 105.0
     }
   },

artifacts/evaluation/latest/decision_phase_final_wave_cases_confusion_matrix.csv

@@ -1,7 +1,7 @@
 ,awareness,research,consideration,decision,action,post_purchase,support
 awareness,5,0,0,0,0,0,0
-research,2,2,0,0,0,0,0
-consideration,0,1,4,0,0,0,0
+research,1,3,0,0,0,0,0
+consideration,0,2,3,0,0,0,0
 decision,0,0,0,5,0,0,0
 action,0,0,0,0,0,0,0
 post_purchase,0,0,0,0,0,4,0

artifacts/evaluation/latest/decision_phase_final_wave_cases_report.json

@@ -7,7 +7,7 @@
   "dataset_path": "/content/agentic-intent-classifier/data/decision_phase/final_wave_cases.jsonl",
   "fallback_rate": 0.0,
   "head": "decision_phase",
-  "macro_f1": 0.8823,
+  "macro_f1": 0.8876,
   "per_class_metrics": {
     "accuracy": 0.8888888888888888,
     "action": {
@@ -17,15 +17,15 @@
       "support": 0.0
     },
     "awareness": {
-      "f1-score": 0.8333333333333334,
-      "precision": 0.7142857142857143,
+      "f1-score": 0.9090909090909091,
+      "precision": 0.8333333333333334,
       "recall": 1.0,
       "support": 5.0
     },
     "consideration": {
-      "f1-score": 0.8888888888888888,
+      "f1-score": 0.75,
       "precision": 1.0,
-      "recall": 0.8,
+      "recall": 0.6,
       "support": 5.0
     },
     "decision": {
@@ -35,9 +35,9 @@
       "support": 5.0
     },
     "macro avg": {
-      "f1-score": 0.7562358276643991,
-      "precision": 0.7687074829931974,
-      "recall": 0.7571428571428571,
+      "f1-score": 0.7608225108225108,
+      "precision": 0.7761904761904762,
+      "recall": 0.7642857142857142,
       "support": 27.0
     },
     "post_purchase": {
@@ -47,9 +47,9 @@
       "support": 4.0
     },
     "research": {
-      "f1-score": 0.5714285714285714,
-      "precision": 0.6666666666666666,
-      "recall": 0.5,
+      "f1-score": 0.6666666666666666,
+      "precision": 0.6,
+      "recall": 0.75,
       "support": 4.0
     },
     "support": {
@@ -59,8 +59,8 @@
       "support": 4.0
     },
     "weighted avg": {
-      "f1-score": 0.8850676072898296,
-      "precision": 0.8977072310405644,
+      "f1-score": 0.8874859708193041,
+      "precision": 0.9098765432098765,
       "recall": 0.8888888888888888,
       "support": 27.0
     }

artifacts/evaluation/latest/decision_phase_test_confusion_matrix.csv

@@ -2,7 +2,7 @@
 awareness,3,0,0,0,0,0,0
 research,3,2,0,0,0,0,0
 consideration,0,2,3,0,0,0,0
-decision,0,0,0,5,0,0,0
+decision,0,0,0,4,0,1,0
 action,0,0,0,0,3,0,0
 post_purchase,0,0,0,0,0,4,0
 support,0,0,0,0,0,1,3

artifacts/evaluation/latest/decision_phase_test_report.json

@@ -1,15 +1,15 @@
 {
-  "accepted_accuracy": 0.7931,
+  "accepted_accuracy": 0.7586,
   "accepted_coverage": 1.0,
-  "accuracy": 0.7931,
+  "accuracy": 0.7586,
   "confusion_matrix_path": "/content/agentic-intent-classifier/artifacts/evaluation/latest/decision_phase_test_confusion_matrix.csv",
   "count": 29,
   "dataset_path": "/content/agentic-intent-classifier/data/decision_phase/test.jsonl",
   "fallback_rate": 0.0,
   "head": "decision_phase",
-  "macro_f1": 0.801,
+  "macro_f1": 0.7724,
   "per_class_metrics": {
-    "accuracy": 0.7931034482758621,
+    "accuracy": 0.7586206896551724,
     "action": {
       "f1-score": 1.0,
       "precision": 1.0,
@@ -29,20 +29,20 @@
       "support": 5.0
     },
     "decision": {
-      "f1-score": 1.0,
+      "f1-score": 0.8888888888888888,
       "precision": 1.0,
-      "recall": 1.0,
+      "recall": 0.8,
       "support": 5.0
     },
     "macro avg": {
-      "f1-score": 0.8010204081632653,
-      "precision": 0.8285714285714285,
-      "recall": 0.8214285714285714,
+      "f1-score": 0.7724489795918367,
+      "precision": 0.8095238095238095,
+      "recall": 0.7928571428571428,
       "support": 29.0
     },
     "post_purchase": {
-      "f1-score": 0.8888888888888888,
-      "precision": 0.8,
+      "f1-score": 0.8,
+      "precision": 0.6666666666666666,
       "recall": 1.0,
       "support": 4.0
     },
@@ -59,9 +59,9 @@
       "support": 4.0
     },
     "weighted avg": {
-      "f1-score": 0.7915982484948002,
-      "precision": 0.8344827586206897,
-      "recall": 0.7931034482758621,
+      "f1-score": 0.7601806239737274,
+      "precision": 0.8160919540229885,
+      "recall": 0.7586206896551724,
       "support": 29.0
     }
   },

artifacts/evaluation/latest/decision_phase_train_confusion_matrix.csv

@@ -1,8 +1,8 @@
 ,awareness,research,consideration,decision,action,post_purchase,support
 awareness,16,0,0,0,0,0,0
-research,1,13,1,0,0,0,0
-consideration,0,3,14,0,0,0,0
+research,1,14,0,0,0,0,0
+consideration,0,4,13,0,0,0,0
 decision,0,0,1,15,0,0,0
 action,0,0,0,0,10,0,0
-post_purchase,0,1,0,0,0,13,0
+post_purchase,0,0,0,0,0,14,0
 support,0,0,0,0,0,0,14

artifacts/evaluation/latest/decision_phase_train_report.json

@@ -1,15 +1,15 @@
 {
-  "accepted_accuracy": 0.9314,
+  "accepted_accuracy": 0.9412,
   "accepted_coverage": 1.0,
-  "accuracy": 0.9314,
+  "accuracy": 0.9412,
   "confusion_matrix_path": "/content/agentic-intent-classifier/artifacts/evaluation/latest/decision_phase_train_confusion_matrix.csv",
   "count": 102,
   "dataset_path": "/content/agentic-intent-classifier/data/decision_phase/train.jsonl",
   "fallback_rate": 0.0,
   "head": "decision_phase",
-  "macro_f1": 0.9373,
+  "macro_f1": 0.9464,
   "per_class_metrics": {
-    "accuracy": 0.9313725490196079,
+    "accuracy": 0.9411764705882353,
     "action": {
       "f1-score": 1.0,
       "precision": 1.0,
@@ -23,9 +23,9 @@
       "support": 16.0
     },
     "consideration": {
-      "f1-score": 0.8484848484848485,
-      "precision": 0.875,
-      "recall": 0.8235294117647058,
+      "f1-score": 0.8387096774193549,
+      "precision": 0.9285714285714286,
+      "recall": 0.7647058823529411,
       "support": 17.0
     },
     "decision": {
@@ -35,21 +35,21 @@
       "support": 16.0
     },
     "macro avg": {
-      "f1-score": 0.9373409595183789,
-      "precision": 0.9401260504201681,
-      "recall": 0.9366096438575431,
+      "f1-score": 0.9463762044407206,
+      "precision": 0.9496465252767774,
+      "recall": 0.9479341736694679,
       "support": 102.0
     },
     "post_purchase": {
-      "f1-score": 0.9629629629629629,
+      "f1-score": 1.0,
       "precision": 1.0,
-      "recall": 0.9285714285714286,
+      "recall": 1.0,
       "support": 14.0
     },
     "research": {
-      "f1-score": 0.8125,
-      "precision": 0.7647058823529411,
-      "recall": 0.8666666666666667,
+      "f1-score": 0.8484848484848485,
+      "precision": 0.7777777777777778,
+      "recall": 0.9333333333333333,
       "support": 15.0
     },
     "support": {
@@ -59,9 +59,9 @@
       "support": 14.0
     },
     "weighted avg": {
-      "f1-score": 0.9322769253786015,
-      "precision": 0.9353373702422145,
-      "recall": 0.9313725490196079,
+      "f1-score": 0.9410231345715216,
+      "precision": 0.946188279233262,
+      "recall": 0.9411764705882353,
       "support": 102.0
     }
   },

artifacts/evaluation/latest/decision_phase_val_confusion_matrix.csv

@@ -1,8 +1,8 @@
 ,awareness,research,consideration,decision,action,post_purchase,support
 awareness,5,0,0,0,0,0,0
-research,2,2,0,0,0,0,0
+research,1,3,0,0,0,0,0
 consideration,0,0,5,0,0,0,0
 decision,0,0,1,3,0,0,0
 action,0,0,0,0,3,0,0
 post_purchase,0,1,0,0,0,3,0
-support,0,0,0,0,0,0,4
+support,0,0,0,0,1,0,3

artifacts/evaluation/latest/decision_phase_val_report.json

@@ -7,18 +7,18 @@
   "dataset_path": "/content/agentic-intent-classifier/data/decision_phase/val.jsonl",
   "fallback_rate": 0.0,
   "head": "decision_phase",
-  "macro_f1": 0.8612,
+  "macro_f1": 0.8567,
   "per_class_metrics": {
     "accuracy": 0.8620689655172413,
     "action": {
-      "f1-score": 1.0,
-      "precision": 1.0,
+      "f1-score": 0.8571428571428571,
+      "precision": 0.75,
       "recall": 1.0,
       "support": 3.0
     },
     "awareness": {
-      "f1-score": 0.8333333333333334,
-      "precision": 0.7142857142857143,
+      "f1-score": 0.9090909090909091,
+      "precision": 0.8333333333333334,
       "recall": 1.0,
       "support": 5.0
     },
@@ -35,8 +35,8 @@
       "support": 4.0
     },
     "macro avg": {
-      "f1-score": 0.8611626468769326,
-      "precision": 0.8877551020408163,
+      "f1-score": 0.8566790352504637,
+      "precision": 0.880952380952381,
       "recall": 0.8571428571428571,
       "support": 29.0
     },
@@ -47,20 +47,20 @@
       "support": 4.0
     },
     "research": {
-      "f1-score": 0.5714285714285714,
-      "precision": 0.6666666666666666,
-      "recall": 0.5,
+      "f1-score": 0.75,
+      "precision": 0.75,
+      "recall": 0.75,
       "support": 4.0
     },
     "support": {
-      "f1-score": 1.0,
+      "f1-score": 0.8571428571428571,
       "precision": 1.0,
-      "recall": 1.0,
+      "recall": 0.75,
       "support": 4.0
     },
     "weighted avg": {
-      "f1-score": 0.8570682191371846,
-      "precision": 0.8760262725779967,
+      "f1-score": 0.8602776533811015,
+      "precision": 0.8821839080459771,
       "recall": 0.8620689655172413,
       "support": 29.0
     }

artifacts/evaluation/latest/iab_behavior_lock_regression.json

@@ -13,7 +13,7 @@
   "results": [
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Automotive",
         "model_output.classification.iab_content.tier2.label": null
       },
@@ -24,6 +24,11 @@
       },
       "id": "car-buying-maps-to-automotive-buying",
       "mismatches": [
+        {
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
         {
           "actual": null,
           "expected": "Auto Type",
@@ -63,7 +68,7 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": null,
         "model_output.classification.iab_content.tier3.label": null
@@ -76,6 +81,11 @@
       },
       "id": "labtop-buying-maps-to-laptops",
       "mismatches": [
+        {
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
         {
           "actual": null,
           "expected": "Computing",
@@ -94,10 +104,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier3.label": "Software and Applications"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -108,9 +118,9 @@
       "id": "crm-awareness-maps-to-sales",
       "mismatches": [
         {
-          "actual": null,
-          "expected": "Software and Applications",
-          "path": "model_output.classification.iab_content.tier3.label"
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
         }
       ],
       "notes": "CRM education should resolve to the closest business/sales path, not generic software.",
@@ -120,9 +130,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
-        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier2.label": "Robotics",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -134,7 +144,12 @@
       "id": "crm-comparison-maps-to-sales",
       "mismatches": [
         {
-          "actual": null,
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "Robotics",
           "expected": "Computing",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -151,9 +166,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Careers",
-        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier2.label": "Job Search",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -170,7 +185,12 @@
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": null,
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "Job Search",
           "expected": "Computing",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -188,7 +208,7 @@
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Sports"
+        "model_output.classification.iab_content.tier1.label": "Science"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -197,7 +217,7 @@
       "id": "ml-explanation-maps-to-ai",
       "mismatches": [
         {
-          "actual": "Sports",
+          "actual": "Science",
           "expected": "Technology & Computing",
           "path": "model_output.classification.iab_content.tier1.label"
         }
@@ -209,10 +229,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier3.label": "Information and Network Security"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -223,7 +243,12 @@
       "id": "support-credential-help-maps-to-business-it",
       "mismatches": [
         {
-          "actual": null,
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "Information and Network Security",
           "expected": "Internet",
           "path": "model_output.classification.iab_content.tier3.label"
         }
@@ -259,9 +284,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Sports",
-        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Sensitive Topics",
+        "model_output.classification.iab_content.tier2.label": "Crime & Harmful Acts to Individuals, Society & Human Right Violations",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -273,12 +298,17 @@
       "id": "trial-signup-maps-to-software",
       "mismatches": [
         {
-          "actual": "Sports",
+          "actual": "Sensitive Topics",
           "expected": "Hobbies & Interests",
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": null,
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "Crime & Harmful Acts to Individuals, Society & Human Right Violations",
           "expected": "Content Production",
           "path": "model_output.classification.iab_content.tier2.label"
         },

artifacts/evaluation/latest/iab_content_cross_vertical_benchmark_report.json

@@ -1,90 +1,90 @@
 {
-  "accepted_accuracy": 0.3,
-  "accepted_coverage": 0.8889,
-  "accuracy": 0.2667,
+  "accepted_accuracy": 0.427,
+  "accepted_coverage": 0.9889,
+  "accuracy": 0.4222,
   "count": 90,
   "dataset_path": "/content/agentic-intent-classifier/data/iab_cross_vertical_benchmark.jsonl",
   "difficulty_breakdown": {
     "easy": {
-      "accepted_accuracy": 0.28,
-      "accepted_coverage": 0.8333,
-      "accuracy": 0.2333,
+      "accepted_accuracy": 0.4138,
+      "accepted_coverage": 0.9667,
+      "accuracy": 0.4,
       "count": 30,
-      "fallback_rate": 0.1667,
-      "macro_f1": 0.1556
+      "fallback_rate": 0.0333,
+      "macro_f1": 0.2727
     },
     "hard": {
-      "accepted_accuracy": 0.3846,
-      "accepted_coverage": 0.8667,
-      "accuracy": 0.3333,
+      "accepted_accuracy": 0.4667,
+      "accepted_coverage": 1.0,
+      "accuracy": 0.4667,
       "count": 30,
-      "fallback_rate": 0.1333,
-      "macro_f1": 0.2083
+      "fallback_rate": 0.0,
+      "macro_f1": 0.3106
     },
     "medium": {
-      "accepted_accuracy": 0.2414,
-      "accepted_coverage": 0.9667,
-      "accuracy": 0.2333,
+      "accepted_accuracy": 0.4,
+      "accepted_coverage": 1.0,
+      "accuracy": 0.4,
       "count": 30,
-      "fallback_rate": 0.0333,
-      "macro_f1": 0.1458
+      "fallback_rate": 0.0,
+      "macro_f1": 0.2667
     }
   },
-  "fallback_rate": 0.1111,
+  "fallback_rate": 0.0111,
   "head": "iab_content",
-  "macro_f1": 0.1418,
+  "macro_f1": 0.227,
   "primary_source": "supervised_classifier",
   "suite": "cross_vertical_benchmark",
   "tier_metrics": {
-    "average_prediction_depth": 1.9556,
+    "average_prediction_depth": 2.4,
     "error_buckets": {
-      "exact_match": 24,
-      "parent_safe_stop": 10,
-      "right_tier1_wrong_tier2": 15,
-      "wrong_deep_leaf": 12,
+      "exact_match": 38,
+      "parent_safe_stop": 1,
+      "right_tier1_wrong_tier2": 14,
+      "wrong_deep_leaf": 8,
       "wrong_tier1": 29
     },
-    "exact_path_accuracy": 0.2667,
-    "parent_safe_accuracy": 0.4556,
+    "exact_path_accuracy": 0.4222,
+    "parent_safe_accuracy": 0.4444,
     "tier1_accuracy": 0.6778,
-    "tier2_accuracy": 0.4762,
-    "tier3_accuracy": 0.2143,
-    "tier4_accuracy": 0.0
+    "tier2_accuracy": 0.4881,
+    "tier3_accuracy": 0.5238,
+    "tier4_accuracy": 0.5
   },
   "view_metrics": {
     "classifier": {
-      "average_prediction_depth": 1.9556,
+      "average_prediction_depth": 2.4,
       "error_buckets": {
-        "exact_match": 24,
-        "parent_safe_stop": 10,
-        "right_tier1_wrong_tier2": 15,
-        "wrong_deep_leaf": 12,
+        "exact_match": 37,
+        "parent_safe_stop": 1,
+        "right_tier1_wrong_tier2": 14,
+        "wrong_deep_leaf": 9,
         "wrong_tier1": 29
       },
-      "exact_path_accuracy": 0.2667,
-      "parent_safe_accuracy": 0.4556,
+      "exact_path_accuracy": 0.4111,
+      "parent_safe_accuracy": 0.4333,
       "tier1_accuracy": 0.6778,
-      "tier2_accuracy": 0.4762,
-      "tier3_accuracy": 0.2143,
-      "tier4_accuracy": 0.0
+      "tier2_accuracy": 0.4881,
+      "tier3_accuracy": 0.4762,
+      "tier4_accuracy": 0.5
     },
     "combined_path": {
-      "average_prediction_depth": 1.9556,
+      "average_prediction_depth": 2.4,
       "error_buckets": {
-        "exact_match": 24,
-        "parent_safe_stop": 10,
-        "right_tier1_wrong_tier2": 15,
-        "wrong_deep_leaf": 12,
+        "exact_match": 37,
+        "parent_safe_stop": 1,
+        "right_tier1_wrong_tier2": 14,
+        "wrong_deep_leaf": 9,
         "wrong_tier1": 29
       },
-      "exact_path_accuracy": 0.2667,
-      "fallback_overuse_count": 17,
-      "fallback_rate": 0.1889,
-      "parent_safe_accuracy": 0.4556,
+      "exact_path_accuracy": 0.4111,
+      "fallback_overuse_count": 25,
+      "fallback_rate": 0.2778,
+      "parent_safe_accuracy": 0.4333,
       "tier1_accuracy": 0.6778,
-      "tier2_accuracy": 0.4762,
-      "tier3_accuracy": 0.2143,
-      "tier4_accuracy": 0.0
+      "tier2_accuracy": 0.4881,
+      "tier3_accuracy": 0.4762,
+      "tier4_accuracy": 0.5
     },
     "disagreements": {
       "classifier_vs_combined": 0

artifacts/evaluation/latest/iab_content_difficulty_benchmark_report.json

@@ -1,90 +1,90 @@
 {
-  "accepted_accuracy": 0.4219,
-  "accepted_coverage": 0.8205,
-  "accuracy": 0.3462,
+  "accepted_accuracy": 0.4231,
+  "accepted_coverage": 1.0,
+  "accuracy": 0.4231,
   "count": 156,
   "dataset_path": "/content/agentic-intent-classifier/data/iab_benchmark.jsonl",
   "difficulty_breakdown": {
     "easy": {
-      "accepted_accuracy": 0.4889,
-      "accepted_coverage": 0.8654,
-      "accuracy": 0.4231,
+      "accepted_accuracy": 0.4615,
+      "accepted_coverage": 1.0,
+      "accuracy": 0.4615,
       "count": 52,
-      "fallback_rate": 0.1346,
-      "macro_f1": 0.2305
+      "fallback_rate": 0.0,
+      "macro_f1": 0.2359
     },
     "hard": {
-      "accepted_accuracy": 0.3846,
-      "accepted_coverage": 0.75,
-      "accuracy": 0.2885,
+      "accepted_accuracy": 0.3654,
+      "accepted_coverage": 1.0,
+      "accuracy": 0.3654,
       "count": 52,
-      "fallback_rate": 0.25,
-      "macro_f1": 0.1638
+      "fallback_rate": 0.0,
+      "macro_f1": 0.1892
     },
     "medium": {
-      "accepted_accuracy": 0.3864,
-      "accepted_coverage": 0.8462,
-      "accuracy": 0.3269,
+      "accepted_accuracy": 0.4423,
+      "accepted_coverage": 1.0,
+      "accuracy": 0.4423,
       "count": 52,
-      "fallback_rate": 0.1538,
-      "macro_f1": 0.1819
+      "fallback_rate": 0.0,
+      "macro_f1": 0.2338
     }
   },
-  "fallback_rate": 0.1795,
+  "fallback_rate": 0.0,
   "head": "iab_content",
-  "macro_f1": 0.1478,
+  "macro_f1": 0.1524,
   "primary_source": "supervised_classifier",
   "suite": "difficulty_benchmark",
   "tier_metrics": {
-    "average_prediction_depth": 2.0256,
+    "average_prediction_depth": 2.4103,
     "error_buckets": {
-      "exact_match": 54,
-      "parent_safe_stop": 21,
-      "right_tier1_wrong_tier2": 37,
-      "wrong_deep_leaf": 3,
-      "wrong_tier1": 41
+      "exact_match": 66,
+      "parent_safe_stop": 1,
+      "right_tier1_wrong_tier2": 42,
+      "wrong_deep_leaf": 8,
+      "wrong_tier1": 39
     },
-    "exact_path_accuracy": 0.3462,
-    "parent_safe_accuracy": 0.6603,
-    "tier1_accuracy": 0.7372,
-    "tier2_accuracy": 0.5,
-    "tier3_accuracy": 0.3519,
-    "tier4_accuracy": 0.2917
+    "exact_path_accuracy": 0.4231,
+    "parent_safe_accuracy": 0.5385,
+    "tier1_accuracy": 0.75,
+    "tier2_accuracy": 0.4808,
+    "tier3_accuracy": 0.5093,
+    "tier4_accuracy": 0.4583
   },
   "view_metrics": {
     "classifier": {
-      "average_prediction_depth": 2.0256,
+      "average_prediction_depth": 2.4103,
       "error_buckets": {
-        "exact_match": 49,
-        "parent_safe_stop": 21,
-        "right_tier1_wrong_tier2": 37,
-        "wrong_deep_leaf": 8,
-        "wrong_tier1": 41
+        "exact_match": 59,
+        "parent_safe_stop": 1,
+        "right_tier1_wrong_tier2": 42,
+        "wrong_deep_leaf": 15,
+        "wrong_tier1": 39
       },
-      "exact_path_accuracy": 0.3141,
-      "parent_safe_accuracy": 0.6282,
-      "tier1_accuracy": 0.7372,
-      "tier2_accuracy": 0.5,
-      "tier3_accuracy": 0.3056,
-      "tier4_accuracy": 0.2083
+      "exact_path_accuracy": 0.3782,
+      "parent_safe_accuracy": 0.4936,
+      "tier1_accuracy": 0.75,
+      "tier2_accuracy": 0.4808,
+      "tier3_accuracy": 0.4259,
+      "tier4_accuracy": 0.1667
     },
     "combined_path": {
-      "average_prediction_depth": 2.0256,
+      "average_prediction_depth": 2.4103,
       "error_buckets": {
-        "exact_match": 49,
-        "parent_safe_stop": 21,
-        "right_tier1_wrong_tier2": 37,
-        "wrong_deep_leaf": 8,
-        "wrong_tier1": 41
+        "exact_match": 59,
+        "parent_safe_stop": 1,
+        "right_tier1_wrong_tier2": 42,
+        "wrong_deep_leaf": 15,
+        "wrong_tier1": 39
       },
-      "exact_path_accuracy": 0.3141,
-      "fallback_overuse_count": 14,
-      "fallback_rate": 0.0897,
-      "parent_safe_accuracy": 0.6282,
-      "tier1_accuracy": 0.7372,
-      "tier2_accuracy": 0.5,
-      "tier3_accuracy": 0.3056,
-      "tier4_accuracy": 0.2083
+      "exact_path_accuracy": 0.3782,
+      "fallback_overuse_count": 15,
+      "fallback_rate": 0.0962,
+      "parent_safe_accuracy": 0.4936,
+      "tier1_accuracy": 0.75,
+      "tier2_accuracy": 0.4808,
+      "tier3_accuracy": 0.4259,
+      "tier4_accuracy": 0.1667
     },
     "disagreements": {
       "classifier_vs_combined": 0

artifacts/evaluation/latest/iab_content_extended_cases_report.json

@@ -1,24 +1,24 @@
 {
-  "accepted_accuracy": 0.4286,
-  "accepted_coverage": 0.875,
-  "accuracy": 0.375,
+  "accepted_accuracy": 0.5,
+  "accepted_coverage": 1.0,
+  "accuracy": 0.5,
   "count": 8,
   "dataset_path": "/content/agentic-intent-classifier/data/iab/extended_cases.jsonl",
-  "fallback_rate": 0.125,
+  "fallback_rate": 0.0,
   "head": "iab_content",
-  "macro_f1": 0.2308,
+  "macro_f1": 0.3333,
   "primary_source": "supervised_classifier",
   "suite": "extended_cases",
   "tier_metrics": {
-    "average_prediction_depth": 2.0,
+    "average_prediction_depth": 2.125,
     "error_buckets": {
-      "exact_match": 3,
+      "exact_match": 4,
       "right_tier1_wrong_tier2": 2,
-      "wrong_deep_leaf": 2,
+      "wrong_deep_leaf": 1,
       "wrong_tier1": 1
     },
-    "exact_path_accuracy": 0.375,
-    "parent_safe_accuracy": 0.375,
+    "exact_path_accuracy": 0.5,
+    "parent_safe_accuracy": 0.5,
     "tier1_accuracy": 0.875,
     "tier2_accuracy": 0.5714,
     "tier3_accuracy": 0.0,
@@ -26,32 +26,32 @@
   },
   "view_metrics": {
     "classifier": {
-      "average_prediction_depth": 2.0,
+      "average_prediction_depth": 2.125,
       "error_buckets": {
-        "exact_match": 3,
+        "exact_match": 4,
         "right_tier1_wrong_tier2": 2,
-        "wrong_deep_leaf": 2,
+        "wrong_deep_leaf": 1,
         "wrong_tier1": 1
       },
-      "exact_path_accuracy": 0.375,
-      "parent_safe_accuracy": 0.375,
+      "exact_path_accuracy": 0.5,
+      "parent_safe_accuracy": 0.5,
       "tier1_accuracy": 0.875,
       "tier2_accuracy": 0.5714,
       "tier3_accuracy": 0.0,
       "tier4_accuracy": 0.0
     },
     "combined_path": {
-      "average_prediction_depth": 2.0,
+      "average_prediction_depth": 2.125,
       "error_buckets": {
-        "exact_match": 3,
+        "exact_match": 4,
         "right_tier1_wrong_tier2": 2,
-        "wrong_deep_leaf": 2,
+        "wrong_deep_leaf": 1,
         "wrong_tier1": 1
       },
-      "exact_path_accuracy": 0.375,
+      "exact_path_accuracy": 0.5,
       "fallback_overuse_count": 2,
       "fallback_rate": 0.25,
-      "parent_safe_accuracy": 0.375,
+      "parent_safe_accuracy": 0.5,
       "tier1_accuracy": 0.875,
       "tier2_accuracy": 0.5714,
       "tier3_accuracy": 0.0,

artifacts/evaluation/latest/iab_content_hard_cases_report.json

@@ -1,16 +1,16 @@
 {
-  "accepted_accuracy": 0.5,
-  "accepted_coverage": 0.75,
+  "accepted_accuracy": 0.4286,
+  "accepted_coverage": 0.875,
   "accuracy": 0.375,
   "count": 8,
   "dataset_path": "/content/agentic-intent-classifier/data/iab/hard_cases.jsonl",
-  "fallback_rate": 0.25,
+  "fallback_rate": 0.125,
   "head": "iab_content",
-  "macro_f1": 0.25,
+  "macro_f1": 0.2308,
   "primary_source": "supervised_classifier",
   "suite": "hard_cases",
   "tier_metrics": {
-    "average_prediction_depth": 2.0,
+    "average_prediction_depth": 2.25,
     "error_buckets": {
       "exact_match": 3,
       "right_tier1_wrong_tier2": 1,
@@ -25,7 +25,7 @@
   },
   "view_metrics": {
     "classifier": {
-      "average_prediction_depth": 2.0,
+      "average_prediction_depth": 2.25,
       "error_buckets": {
         "exact_match": 3,
         "right_tier1_wrong_tier2": 1,
@@ -39,7 +39,7 @@
       "tier4_accuracy": 0.0
     },
     "combined_path": {
-      "average_prediction_depth": 2.0,
+      "average_prediction_depth": 2.25,
       "error_buckets": {
         "exact_match": 3,
         "right_tier1_wrong_tier2": 1,

artifacts/evaluation/latest/iab_content_test_report.json

@@ -1,46 +1,46 @@
 {
-  "accepted_accuracy": 0.9242,
-  "accepted_coverage": 0.9973,
-  "accuracy": 0.922,
+  "accepted_accuracy": 0.943,
+  "accepted_coverage": 1.0,
+  "accuracy": 0.943,
   "count": 3282,
   "dataset_path": "/content/agentic-intent-classifier/data/iab/test.jsonl",
-  "fallback_rate": 0.0027,
+  "fallback_rate": 0.0,
   "head": "iab_content",
-  "macro_f1": 0.8741,
+  "macro_f1": 0.911,
   "primary_source": "supervised_classifier",
   "suite": "test",
   "tier_metrics": {
-    "average_prediction_depth": 2.1789,
+    "average_prediction_depth": 2.213,
     "error_buckets": {
-      "exact_match": 3026,
-      "parent_safe_stop": 68,
-      "right_tier1_wrong_tier2": 59,
-      "wrong_deep_leaf": 96,
-      "wrong_tier1": 33
+      "exact_match": 3095,
+      "parent_safe_stop": 45,
+      "right_tier1_wrong_tier2": 41,
+      "wrong_deep_leaf": 72,
+      "wrong_tier1": 29
     },
-    "exact_path_accuracy": 0.922,
-    "parent_safe_accuracy": 0.9509,
-    "tier1_accuracy": 0.9899,
-    "tier2_accuracy": 0.9693,
-    "tier3_accuracy": 0.8477,
-    "tier4_accuracy": 0.5143
+    "exact_path_accuracy": 0.943,
+    "parent_safe_accuracy": 0.958,
+    "tier1_accuracy": 0.9912,
+    "tier2_accuracy": 0.9776,
+    "tier3_accuracy": 0.9078,
+    "tier4_accuracy": 0.7
   },
   "view_metrics": {
     "classifier": {
-      "average_prediction_depth": 2.1789,
+      "average_prediction_depth": 2.213,
       "error_buckets": {
-        "exact_match": 2995,
-        "parent_safe_stop": 62,
-        "right_tier1_wrong_tier2": 71,
-        "wrong_deep_leaf": 121,
-        "wrong_tier1": 33
+        "exact_match": 3052,
+        "parent_safe_stop": 44,
+        "right_tier1_wrong_tier2": 53,
+        "wrong_deep_leaf": 104,
+        "wrong_tier1": 29
       },
-      "exact_path_accuracy": 0.9126,
-      "parent_safe_accuracy": 0.9397,
-      "tier1_accuracy": 0.9899,
-      "tier2_accuracy": 0.9651,
-      "tier3_accuracy": 0.8218,
-      "tier4_accuracy": 0.3929
+      "exact_path_accuracy": 0.9299,
+      "parent_safe_accuracy": 0.9445,
+      "tier1_accuracy": 0.9912,
+      "tier2_accuracy": 0.9734,
+      "tier3_accuracy": 0.8725,
+      "tier4_accuracy": 0.5
     },
     "combined_path": {
       "count": 3282,

artifacts/evaluation/latest/iab_content_train_report.json

@@ -1,46 +1,46 @@
 {
-  "accepted_accuracy": 0.93,
-  "accepted_coverage": 0.9978,
-  "accuracy": 0.9282,
+  "accepted_accuracy": 0.9459,
+  "accepted_coverage": 1.0,
+  "accuracy": 0.9459,
   "count": 13211,
   "dataset_path": "/content/agentic-intent-classifier/data/iab/train.jsonl",
-  "fallback_rate": 0.0022,
+  "fallback_rate": 0.0,
   "head": "iab_content",
-  "macro_f1": 0.8851,
+  "macro_f1": 0.9194,
   "primary_source": "supervised_classifier",
   "suite": "train",
   "tier_metrics": {
-    "average_prediction_depth": 2.172,
+    "average_prediction_depth": 2.2105,
     "error_buckets": {
-      "exact_match": 12263,
-      "parent_safe_stop": 259,
-      "right_tier1_wrong_tier2": 229,
-      "wrong_deep_leaf": 356,
-      "wrong_tier1": 104
+      "exact_match": 12496,
+      "parent_safe_stop": 162,
+      "right_tier1_wrong_tier2": 144,
+      "wrong_deep_leaf": 284,
+      "wrong_tier1": 125
     },
-    "exact_path_accuracy": 0.9282,
-    "parent_safe_accuracy": 0.9572,
-    "tier1_accuracy": 0.9921,
-    "tier2_accuracy": 0.9726,
-    "tier3_accuracy": 0.8565,
-    "tier4_accuracy": 0.5518
+    "exact_path_accuracy": 0.9459,
+    "parent_safe_accuracy": 0.9585,
+    "tier1_accuracy": 0.9905,
+    "tier2_accuracy": 0.9805,
+    "tier3_accuracy": 0.9135,
+    "tier4_accuracy": 0.7268
   },
   "view_metrics": {
     "classifier": {
-      "average_prediction_depth": 2.172,
+      "average_prediction_depth": 2.2105,
       "error_buckets": {
-        "exact_match": 12130,
-        "parent_safe_stop": 238,
-        "right_tier1_wrong_tier2": 277,
-        "wrong_deep_leaf": 462,
-        "wrong_tier1": 104
+        "exact_match": 12323,
+        "parent_safe_stop": 157,
+        "right_tier1_wrong_tier2": 192,
+        "wrong_deep_leaf": 414,
+        "wrong_tier1": 125
       },
-      "exact_path_accuracy": 0.9182,
-      "parent_safe_accuracy": 0.9456,
-      "tier1_accuracy": 0.9921,
-      "tier2_accuracy": 0.9685,
-      "tier3_accuracy": 0.829,
-      "tier4_accuracy": 0.4214
+      "exact_path_accuracy": 0.9328,
+      "parent_safe_accuracy": 0.945,
+      "tier1_accuracy": 0.9905,
+      "tier2_accuracy": 0.9764,
+      "tier3_accuracy": 0.8777,
+      "tier4_accuracy": 0.525
     },
     "combined_path": {
       "count": 13211,

artifacts/evaluation/latest/iab_content_val_report.json

@@ -1,46 +1,46 @@
 {
-  "accepted_accuracy": 0.9246,
-  "accepted_coverage": 0.9979,
-  "accuracy": 0.9229,
+  "accepted_accuracy": 0.9442,
+  "accepted_coverage": 1.0,
+  "accuracy": 0.9442,
   "count": 3282,
   "dataset_path": "/content/agentic-intent-classifier/data/iab/val.jsonl",
-  "fallback_rate": 0.0021,
+  "fallback_rate": 0.0,
   "head": "iab_content",
-  "macro_f1": 0.8789,
+  "macro_f1": 0.9166,
   "primary_source": "supervised_classifier",
   "suite": "val",
   "tier_metrics": {
-    "average_prediction_depth": 2.1789,
+    "average_prediction_depth": 2.2151,
     "error_buckets": {
-      "exact_match": 3029,
-      "parent_safe_stop": 69,
-      "right_tier1_wrong_tier2": 67,
-      "wrong_deep_leaf": 91,
-      "wrong_tier1": 26
+      "exact_match": 3099,
+      "parent_safe_stop": 35,
+      "right_tier1_wrong_tier2": 45,
+      "wrong_deep_leaf": 72,
+      "wrong_tier1": 31
     },
-    "exact_path_accuracy": 0.9229,
-    "parent_safe_accuracy": 0.9549,
-    "tier1_accuracy": 0.9921,
-    "tier2_accuracy": 0.9686,
-    "tier3_accuracy": 0.8549,
-    "tier4_accuracy": 0.5286
+    "exact_path_accuracy": 0.9442,
+    "parent_safe_accuracy": 0.9576,
+    "tier1_accuracy": 0.9906,
+    "tier2_accuracy": 0.9769,
+    "tier3_accuracy": 0.9088,
+    "tier4_accuracy": 0.7286
   },
   "view_metrics": {
     "classifier": {
-      "average_prediction_depth": 2.1789,
+      "average_prediction_depth": 2.2151,
       "error_buckets": {
-        "exact_match": 2997,
-        "parent_safe_stop": 64,
-        "right_tier1_wrong_tier2": 79,
-        "wrong_deep_leaf": 116,
-        "wrong_tier1": 26
+        "exact_match": 3056,
+        "parent_safe_stop": 34,
+        "right_tier1_wrong_tier2": 57,
+        "wrong_deep_leaf": 104,
+        "wrong_tier1": 31
       },
-      "exact_path_accuracy": 0.9132,
-      "parent_safe_accuracy": 0.9436,
-      "tier1_accuracy": 0.9921,
-      "tier2_accuracy": 0.9644,
-      "tier3_accuracy": 0.829,
-      "tier4_accuracy": 0.4071
+      "exact_path_accuracy": 0.9311,
+      "parent_safe_accuracy": 0.9442,
+      "tier1_accuracy": 0.9906,
+      "tier2_accuracy": 0.9727,
+      "tier3_accuracy": 0.8736,
+      "tier4_accuracy": 0.5286
     },
     "combined_path": {
       "count": 3282,

artifacts/evaluation/latest/iab_cross_vertical_behavior_lock_regression.json

@@ -1,21 +1,21 @@
 {
   "by_status": {
     "must_fix": {
-      "failed": 88,
-      "passed": 2,
+      "failed": 90,
+      "passed": 0,
       "total": 90
     }
   },
   "cases_path": "/content/agentic-intent-classifier/examples/iab_cross_vertical_behavior_lock_cases.json",
   "count": 90,
-  "failed": 88,
-  "passed": 2,
+  "failed": 90,
+  "passed": 0,
   "results": [
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "exact",
-        "model_output.classification.iab_content.tier1.label": "Automotive",
-        "model_output.classification.iab_content.tier2.label": "Auto Rentals"
+        "model_output.classification.iab_content.tier1.label": "Personal Finance",
+        "model_output.classification.iab_content.tier2.label": "Insurance"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -25,7 +25,7 @@
       "id": "auto-buying-easy",
       "mismatches": [
         {
-          "actual": "Automotive",
+          "actual": "Personal Finance",
           "expected": "Travel",
           "path": "model_output.classification.iab_content.tier1.label"
         },
@@ -35,7 +35,7 @@
           "path": "model_output.classification.iab_content.mapping_mode"
         },
         {
-          "actual": "Auto Rentals",
+          "actual": "Insurance",
           "expected": "Travel Type",
           "path": "model_output.classification.iab_content.tier2.label"
         }
@@ -47,7 +47,7 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Automotive",
         "model_output.classification.iab_content.tier2.label": "Auto Body Styles"
       },
@@ -57,17 +57,23 @@
         "model_output.classification.iab_content.tier2.label": "Auto Body Styles"
       },
       "id": "auto-buying-medium",
-      "mismatches": [],
+      "mismatches": [
+        {
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        }
+      ],
       "notes": "Cross-vertical medium IAB mapping case for Automotive > Auto Buying and Selling.",
-      "pass": true,
+      "pass": false,
       "status": "must_fix",
       "text": "Best used SUV for a family of four"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Automotive",
-        "model_output.classification.iab_content.tier2.label": "Auto Type"
+        "model_output.classification.iab_content.tier2.label": "Car Culture"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -75,18 +81,29 @@
         "model_output.classification.iab_content.tier2.label": "Auto Type"
       },
       "id": "auto-buying-hard",
-      "mismatches": [],
+      "mismatches": [
+        {
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "Car Culture",
+          "expected": "Auto Type",
+          "path": "model_output.classification.iab_content.tier2.label"
+        }
+      ],
       "notes": "Cross-vertical hard IAB mapping case for Automotive > Auto Buying and Selling.",
-      "pass": true,
+      "pass": false,
       "status": "must_fix",
       "text": "I need a shortlist of practical cars before making a purchase this month"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier3.label": "Software and Applications"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -97,9 +114,9 @@
       "id": "sales-crm-easy",
       "mismatches": [
         {
-          "actual": null,
-          "expected": "Software and Applications",
-          "path": "model_output.classification.iab_content.tier3.label"
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
         }
       ],
       "notes": "Cross-vertical easy IAB mapping case for Business and Finance > Business > Sales.",
@@ -109,9 +126,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
-        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier2.label": "Robotics",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -123,7 +140,12 @@
       "id": "sales-crm-medium",
       "mismatches": [
         {
-          "actual": null,
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "Robotics",
           "expected": "Computing",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -140,10 +162,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Business and Finance",
         "model_output.classification.iab_content.tier2.label": "Business",
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier3.label": "Sales"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -154,9 +176,9 @@
       "id": "sales-crm-hard",
       "mismatches": [
         {
-          "actual": null,
-          "expected": "Sales",
-          "path": "model_output.classification.iab_content.tier3.label"
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
         }
       ],
       "notes": "Cross-vertical hard IAB mapping case for Business and Finance > Business > Sales.",
@@ -166,9 +188,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Hobbies & Interests",
-        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Careers",
+        "model_output.classification.iab_content.tier2.label": "Job Search",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -180,12 +202,17 @@
       "id": "marketing-tools-easy",
       "mismatches": [
         {
-          "actual": "Hobbies & Interests",
+          "actual": "Careers",
           "expected": "Technology & Computing",
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": null,
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "Job Search",
           "expected": "Computing",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -202,9 +229,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Careers",
-        "model_output.classification.iab_content.tier2.label": null
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Sensitive Topics",
+        "model_output.classification.iab_content.tier2.label": "Terrorism"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -214,12 +241,17 @@
       "id": "marketing-tools-medium",
       "mismatches": [
         {
-          "actual": "Careers",
+          "actual": "Sensitive Topics",
           "expected": "Business and Finance",
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": null,
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "Terrorism",
           "expected": "Business",
           "path": "model_output.classification.iab_content.tier2.label"
         }
@@ -231,9 +263,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Personal Finance",
-        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier2.label": "Home Utilities",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -250,7 +282,12 @@
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": null,
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "Home Utilities",
           "expected": "Computing",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -267,10 +304,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier3.label": "Information and Network Security"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -281,7 +318,12 @@
       "id": "business-it-easy",
       "mismatches": [
         {
-          "actual": null,
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "Information and Network Security",
           "expected": "Internet",
           "path": "model_output.classification.iab_content.tier3.label"
         }
@@ -293,9 +335,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Personal Finance",
-        "model_output.classification.iab_content.tier2.label": null
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Careers",
+        "model_output.classification.iab_content.tier2.label": "Job Search"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -305,14 +347,9 @@
       "id": "business-it-medium",
       "mismatches": [
         {
-          "actual": "Personal Finance",
-          "expected": "Careers",
-          "path": "model_output.classification.iab_content.tier1.label"
-        },
-        {
-          "actual": null,
-          "expected": "Job Search",
-          "path": "model_output.classification.iab_content.tier2.label"
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
         }
       ],
       "notes": "Cross-vertical medium IAB mapping case for Business and Finance > Business > Business I.T..",
@@ -412,8 +449,8 @@
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "exact",
-        "model_output.classification.iab_content.tier1.label": "Food & Drink",
-        "model_output.classification.iab_content.tier2.label": "Dining Out"
+        "model_output.classification.iab_content.tier1.label": "Attractions",
+        "model_output.classification.iab_content.tier2.label": "Bars & Restaurants"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -422,10 +459,20 @@
       },
       "id": "dining-out-hard",
       "mismatches": [
+        {
+          "actual": "Attractions",
+          "expected": "Food & Drink",
+          "path": "model_output.classification.iab_content.tier1.label"
+        },
         {
           "actual": "exact",
           "expected": "nearest_equivalent",
           "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "Bars & Restaurants",
+          "expected": "Dining Out",
+          "path": "model_output.classification.iab_content.tier2.label"
         }
       ],
       "notes": "Cross-vertical hard IAB mapping case for Food & Drink > Dining Out.",
@@ -502,7 +549,7 @@
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Sports"
+        "model_output.classification.iab_content.tier1.label": "Science"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -511,7 +558,7 @@
       "id": "artificial-intelligence-easy",
       "mismatches": [
         {
-          "actual": "Sports",
+          "actual": "Science",
           "expected": "Technology & Computing",
           "path": "model_output.classification.iab_content.tier1.label"
         }
@@ -571,7 +618,7 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing"
       },
@@ -587,6 +634,11 @@
           "expected": "Business and Finance",
           "path": "model_output.classification.iab_content.tier1.label"
         },
+        {
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
         {
           "actual": "Computing",
           "expected": "Business",
@@ -600,10 +652,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": null,
+        "model_output.classification.iab_content.tier3.label": "Software and Applications",
         "model_output.classification.iab_content.tier4.label": null
       },
       "expected": {
@@ -616,7 +668,12 @@
       "id": "software-apps-medium",
       "mismatches": [
         {
-          "actual": null,
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "Software and Applications",
           "expected": "Internet",
           "path": "model_output.classification.iab_content.tier3.label"
         },
@@ -633,9 +690,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Business and Finance",
-        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Virtual Reality",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -647,12 +704,12 @@
       "id": "software-apps-hard",
       "mismatches": [
         {
-          "actual": "Business and Finance",
-          "expected": "Technology & Computing",
-          "path": "model_output.classification.iab_content.tier1.label"
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
         },
         {
-          "actual": null,
+          "actual": "Virtual Reality",
           "expected": "Computing",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -717,10 +774,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": null,
+        "model_output.classification.iab_content.tier3.label": "Information and Network Security",
         "model_output.classification.iab_content.tier4.label": null
       },
       "expected": {
@@ -733,7 +790,12 @@
       "id": "communication-software-medium",
       "mismatches": [
         {
-          "actual": null,
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "Information and Network Security",
           "expected": "Software and Applications",
           "path": "model_output.classification.iab_content.tier3.label"
         },
@@ -750,9 +812,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Careers",
-        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Virtual Reality",
         "model_output.classification.iab_content.tier3.label": null,
         "model_output.classification.iab_content.tier4.label": null
       },
@@ -766,12 +828,12 @@
       "id": "communication-software-hard",
       "mismatches": [
         {
-          "actual": "Careers",
-          "expected": "Technology & Computing",
-          "path": "model_output.classification.iab_content.tier1.label"
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
         },
         {
-          "actual": null,
+          "actual": "Virtual Reality",
           "expected": "Computing",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -793,11 +855,11 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": null,
-        "model_output.classification.iab_content.tier4.label": null
+        "model_output.classification.iab_content.tier3.label": "Internet",
+        "model_output.classification.iab_content.tier4.label": "Web Hosting"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -809,14 +871,9 @@
       "id": "web-hosting-easy",
       "mismatches": [
         {
-          "actual": null,
-          "expected": "Internet",
-          "path": "model_output.classification.iab_content.tier3.label"
-        },
-        {
-          "actual": null,
-          "expected": "Web Hosting",
-          "path": "model_output.classification.iab_content.tier4.label"
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
         }
       ],
       "notes": "Cross-vertical easy IAB mapping case for Technology & Computing > Computing > Internet > Web Hosting.",
@@ -826,11 +883,11 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": null,
-        "model_output.classification.iab_content.tier4.label": null
+        "model_output.classification.iab_content.tier3.label": "Internet",
+        "model_output.classification.iab_content.tier4.label": "Web Hosting"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -842,14 +899,9 @@
       "id": "web-hosting-medium",
       "mismatches": [
         {
-          "actual": null,
-          "expected": "Internet",
-          "path": "model_output.classification.iab_content.tier3.label"
-        },
-        {
-          "actual": null,
-          "expected": "Web Hosting",
-          "path": "model_output.classification.iab_content.tier4.label"
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
         }
       ],
       "notes": "Cross-vertical medium IAB mapping case for Technology & Computing > Computing > Internet > Web Hosting.",
@@ -859,11 +911,11 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": null,
-        "model_output.classification.iab_content.tier4.label": null
+        "model_output.classification.iab_content.tier3.label": "Internet",
+        "model_output.classification.iab_content.tier4.label": "Web Hosting"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -875,14 +927,9 @@
       "id": "web-hosting-hard",
       "mismatches": [
         {
-          "actual": null,
-          "expected": "Internet",
-          "path": "model_output.classification.iab_content.tier3.label"
-        },
-        {
-          "actual": null,
-          "expected": "Web Hosting",
-          "path": "model_output.classification.iab_content.tier4.label"
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
         }
       ],
       "notes": "Cross-vertical hard IAB mapping case for Technology & Computing > Computing > Internet > Web Hosting.",
@@ -892,10 +939,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
-        "model_output.classification.iab_content.tier2.label": null,
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Laptops"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -906,14 +953,9 @@
       "id": "laptops-easy",
       "mismatches": [
         {
-          "actual": null,
-          "expected": "Computing",
-          "path": "model_output.classification.iab_content.tier2.label"
-        },
-        {
-          "actual": null,
-          "expected": "Laptops",
-          "path": "model_output.classification.iab_content.tier3.label"
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
         }
       ],
       "notes": "Cross-vertical easy IAB mapping case for Technology & Computing > Computing > Laptops.",
@@ -949,10 +991,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier3.label": "Laptops"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -962,13 +1004,18 @@
       },
       "id": "laptops-hard",
       "mismatches": [
+        {
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
         {
           "actual": "Computing",
           "expected": "Consumer Electronics",
           "path": "model_output.classification.iab_content.tier2.label"
         },
         {
-          "actual": null,
+          "actual": "Laptops",
           "expected": "Smartphones",
           "path": "model_output.classification.iab_content.tier3.label"
         }
@@ -980,11 +1027,11 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
         "model_output.classification.iab_content.tier3.label": "Software and Applications",
-        "model_output.classification.iab_content.tier4.label": null
+        "model_output.classification.iab_content.tier4.label": "Computer Animation"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -996,7 +1043,12 @@
       "id": "desktops-easy",
       "mismatches": [
         {
-          "actual": null,
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "Computer Animation",
           "expected": "Photo Editing Software",
           "path": "model_output.classification.iab_content.tier4.label"
         }
@@ -1008,10 +1060,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier3.label": "Desktops"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -1022,9 +1074,9 @@
       "id": "desktops-medium",
       "mismatches": [
         {
-          "actual": null,
-          "expected": "Desktops",
-          "path": "model_output.classification.iab_content.tier3.label"
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
         }
       ],
       "notes": "Cross-vertical medium IAB mapping case for Technology & Computing > Computing > Desktops.",
@@ -1087,7 +1139,7 @@
         "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Consumer Electronics",
-        "model_output.classification.iab_content.tier3.label": "Wearable Technology"
+        "model_output.classification.iab_content.tier3.label": "Smartphones"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -1101,11 +1153,6 @@
           "actual": "exact",
           "expected": "nearest_equivalent",
           "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": "Wearable Technology",
-          "expected": "Smartphones",
-          "path": "model_output.classification.iab_content.tier3.label"
         }
       ],
       "notes": "Cross-vertical medium IAB mapping case for Technology & Computing > Consumer Electronics > Smartphones.",
@@ -1118,7 +1165,7 @@
         "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Consumer Electronics",
-        "model_output.classification.iab_content.tier3.label": "Wearable Technology"
+        "model_output.classification.iab_content.tier3.label": "Smartphones"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -1132,11 +1179,6 @@
           "actual": "exact",
           "expected": "nearest_equivalent",
           "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": "Wearable Technology",
-          "expected": "Smartphones",
-          "path": "model_output.classification.iab_content.tier3.label"
         }
       ],
       "notes": "Cross-vertical hard IAB mapping case for Technology & Computing > Consumer Electronics > Smartphones.",
@@ -1148,8 +1190,8 @@
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Style & Fashion",
-        "model_output.classification.iab_content.tier2.label": "Designer Clothing",
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier2.label": "Men's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Men's Shoes and Footwear"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -1165,12 +1207,12 @@
           "path": "model_output.classification.iab_content.mapping_mode"
         },
         {
-          "actual": "Designer Clothing",
+          "actual": "Men's Fashion",
           "expected": "Women's Fashion",
           "path": "model_output.classification.iab_content.tier2.label"
         },
         {
-          "actual": null,
+          "actual": "Men's Shoes and Footwear",
           "expected": "Women's Shoes and Footwear",
           "path": "model_output.classification.iab_content.tier3.label"
         }
@@ -1220,8 +1262,8 @@
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Style & Fashion",
-        "model_output.classification.iab_content.tier2.label": "Designer Clothing",
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier2.label": "Women's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Women's Shoes and Footwear"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -1235,16 +1277,6 @@
           "actual": "exact",
           "expected": "nearest_equivalent",
           "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": "Designer Clothing",
-          "expected": "Women's Fashion",
-          "path": "model_output.classification.iab_content.tier2.label"
-        },
-        {
-          "actual": null,
-          "expected": "Women's Shoes and Footwear",
-          "path": "model_output.classification.iab_content.tier3.label"
         }
       ],
       "notes": "Cross-vertical hard IAB mapping case for Style & Fashion.",
@@ -1254,9 +1286,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Sports",
-        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier2.label": "Bodybuilding",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -1273,7 +1305,12 @@
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": null,
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "Bodybuilding",
           "expected": "Women's Fashion",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -1357,9 +1394,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Style & Fashion",
-        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier2.label": "Children's Clothing",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -1371,7 +1408,12 @@
       "id": "mens-shoes-easy",
       "mismatches": [
         {
-          "actual": null,
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "Children's Clothing",
           "expected": "Men's Fashion",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -1419,9 +1461,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Style & Fashion",
-        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier2.label": "Children's Clothing",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -1433,7 +1475,12 @@
       "id": "mens-shoes-hard",
       "mismatches": [
         {
-          "actual": null,
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "Children's Clothing",
           "expected": "Men's Fashion",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -1512,7 +1559,7 @@
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Travel",
-        "model_output.classification.iab_content.tier2.label": null
+        "model_output.classification.iab_content.tier2.label": "Travel Type"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -1525,11 +1572,6 @@
           "actual": "exact",
           "expected": "nearest_equivalent",
           "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
-          "expected": "Travel Type",
-          "path": "model_output.classification.iab_content.tier2.label"
         }
       ],
       "notes": "Cross-vertical hard IAB mapping case for Travel > Travel Type > Hotels and Motels.",
@@ -1628,10 +1670,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Healthy Living",
-        "model_output.classification.iab_content.tier2.label": null,
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier2.label": "Fitness and Exercise",
+        "model_output.classification.iab_content.tier3.label": "Running and Jogging"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -1647,12 +1689,17 @@
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": null,
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "Fitness and Exercise",
           "expected": "Business",
           "path": "model_output.classification.iab_content.tier2.label"
         },
         {
-          "actual": null,
+          "actual": "Running and Jogging",
           "expected": "Green Solutions",
           "path": "model_output.classification.iab_content.tier3.label"
         }
@@ -1706,8 +1753,8 @@
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "exact",
-        "model_output.classification.iab_content.tier1.label": "Healthy Living",
-        "model_output.classification.iab_content.tier2.label": "Fitness and Exercise",
+        "model_output.classification.iab_content.tier1.label": "Sports",
+        "model_output.classification.iab_content.tier2.label": "Walking",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -1718,11 +1765,21 @@
       },
       "id": "running-and-jogging-hard",
       "mismatches": [
+        {
+          "actual": "Sports",
+          "expected": "Healthy Living",
+          "path": "model_output.classification.iab_content.tier1.label"
+        },
         {
           "actual": "exact",
           "expected": "nearest_equivalent",
           "path": "model_output.classification.iab_content.mapping_mode"
         },
+        {
+          "actual": "Walking",
+          "expected": "Fitness and Exercise",
+          "path": "model_output.classification.iab_content.tier2.label"
+        },
         {
           "actual": null,
           "expected": "Running and Jogging",
@@ -1841,8 +1898,8 @@
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "exact",
-        "model_output.classification.iab_content.tier1.label": "Books and Literature",
-        "model_output.classification.iab_content.tier2.label": "Fiction"
+        "model_output.classification.iab_content.tier1.label": "Travel",
+        "model_output.classification.iab_content.tier2.label": "Travel Type"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -1851,10 +1908,20 @@
       },
       "id": "fiction-medium",
       "mismatches": [
+        {
+          "actual": "Travel",
+          "expected": "Books and Literature",
+          "path": "model_output.classification.iab_content.tier1.label"
+        },
         {
           "actual": "exact",
           "expected": "nearest_equivalent",
           "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "Travel Type",
+          "expected": "Fiction",
+          "path": "model_output.classification.iab_content.tier2.label"
         }
       ],
       "notes": "Cross-vertical medium IAB mapping case for Books and Literature > Fiction.",
@@ -1888,7 +1955,7 @@
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Home & Garden",
-        "model_output.classification.iab_content.tier2.label": "Remodeling & Construction"
+        "model_output.classification.iab_content.tier2.label": "Interior Decorating"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -1901,6 +1968,11 @@
           "actual": "exact",
           "expected": "nearest_equivalent",
           "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "Interior Decorating",
+          "expected": "Remodeling & Construction",
+          "path": "model_output.classification.iab_content.tier2.label"
         }
       ],
       "notes": "Cross-vertical easy IAB mapping case for Home & Garden > Home Improvement.",
@@ -1910,9 +1982,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Home & Garden",
-        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier2.label": "Interior Decorating",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -1929,7 +2001,12 @@
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": null,
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "Interior Decorating",
           "expected": "Personal Care",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -1980,9 +2057,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
-        "model_output.classification.iab_content.tier2.label": null
+        "model_output.classification.iab_content.tier2.label": "Augmented Reality"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -1997,7 +2074,12 @@
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": null,
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "Augmented Reality",
           "expected": "Language Learning",
           "path": "model_output.classification.iab_content.tier2.label"
         }
@@ -2033,8 +2115,8 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Healthy Living"
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Careers"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -2043,9 +2125,9 @@
       "id": "online-education-hard",
       "mismatches": [
         {
-          "actual": "Healthy Living",
-          "expected": "Careers",
-          "path": "model_output.classification.iab_content.tier1.label"
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
         }
       ],
       "notes": "Cross-vertical hard IAB mapping case for Education > Online Education.",
@@ -2136,10 +2218,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Medical Health",
         "model_output.classification.iab_content.tier2.label": "Diseases and Conditions",
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier3.label": "Allergies"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -2150,9 +2232,9 @@
       "id": "medical-health-easy",
       "mismatches": [
         {
-          "actual": null,
-          "expected": "Allergies",
-          "path": "model_output.classification.iab_content.tier3.label"
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
         }
       ],
       "notes": "Cross-vertical easy IAB mapping case for Medical Health.",
@@ -2165,7 +2247,7 @@
         "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Medical Health",
         "model_output.classification.iab_content.tier2.label": "Diseases and Conditions",
-        "model_output.classification.iab_content.tier3.label": "Injuries",
+        "model_output.classification.iab_content.tier3.label": "Bone and Joint Conditions",
         "model_output.classification.iab_content.tier4.label": null
       },
       "expected": {
@@ -2182,6 +2264,11 @@
           "expected": "nearest_equivalent",
           "path": "model_output.classification.iab_content.mapping_mode"
         },
+        {
+          "actual": "Bone and Joint Conditions",
+          "expected": "Injuries",
+          "path": "model_output.classification.iab_content.tier3.label"
+        },
         {
           "actual": null,
           "expected": "First Aid",
@@ -2197,7 +2284,7 @@
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Medical Health",
-        "model_output.classification.iab_content.tier2.label": "Surgery",
+        "model_output.classification.iab_content.tier2.label": null,
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -2219,7 +2306,7 @@
           "path": "model_output.classification.iab_content.mapping_mode"
         },
         {
-          "actual": "Surgery",
+          "actual": null,
           "expected": "Wellness",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -2260,10 +2347,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Business and Finance",
         "model_output.classification.iab_content.tier2.label": "Business",
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier3.label": "Sales"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -2274,9 +2361,9 @@
       "id": "careers-job-search-medium",
       "mismatches": [
         {
-          "actual": null,
-          "expected": "Sales",
-          "path": "model_output.classification.iab_content.tier3.label"
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
         }
       ],
       "notes": "Cross-vertical medium IAB mapping case for Careers > Job Search.",
@@ -2286,9 +2373,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Genres",
-        "model_output.classification.iab_content.tier2.label": null
+        "model_output.classification.iab_content.tier2.label": "Talk Show"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -2303,7 +2390,12 @@
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": null,
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "Talk Show",
           "expected": "Job Search",
           "path": "model_output.classification.iab_content.tier2.label"
         }
@@ -2315,9 +2407,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Personal Celebrations & Life Events",
-        "model_output.classification.iab_content.tier2.label": null
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Holidays",
+        "model_output.classification.iab_content.tier2.label": "National & Civic Holidays"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -2327,12 +2419,17 @@
       "id": "personal-finance-easy",
       "mismatches": [
         {
-          "actual": "Personal Celebrations & Life Events",
+          "actual": "Holidays",
           "expected": "Food & Drink",
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": null,
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "National & Civic Holidays",
           "expected": "Food Movements",
           "path": "model_output.classification.iab_content.tier2.label"
         }
@@ -2434,8 +2531,8 @@
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "exact",
-        "model_output.classification.iab_content.tier1.label": "Hobbies & Interests",
-        "model_output.classification.iab_content.tier2.label": "Content Production"
+        "model_output.classification.iab_content.tier1.label": "Genres",
+        "model_output.classification.iab_content.tier2.label": "Family/Children"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -2445,7 +2542,7 @@
       "id": "parenting-medium",
       "mismatches": [
         {
-          "actual": "Hobbies & Interests",
+          "actual": "Genres",
           "expected": "Family and Relationships",
           "path": "model_output.classification.iab_content.tier1.label"
         },
@@ -2455,7 +2552,7 @@
           "path": "model_output.classification.iab_content.mapping_mode"
         },
         {
-          "actual": "Content Production",
+          "actual": "Family/Children",
           "expected": "Parenting",
           "path": "model_output.classification.iab_content.tier2.label"
         }
@@ -2470,7 +2567,7 @@
         "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Family and Relationships",
         "model_output.classification.iab_content.tier2.label": "Parenting",
-        "model_output.classification.iab_content.tier3.label": "Special Needs Kids"
+        "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -2484,6 +2581,11 @@
           "actual": "exact",
           "expected": "nearest_equivalent",
           "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": null,
+          "expected": "Special Needs Kids",
+          "path": "model_output.classification.iab_content.tier3.label"
         }
       ],
       "notes": "Cross-vertical hard IAB mapping case for Family and Relationships > Parenting.",
@@ -2515,9 +2617,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Food & Drink",
-        "model_output.classification.iab_content.tier2.label": null
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Home & Garden",
+        "model_output.classification.iab_content.tier2.label": "Gardening"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -2527,14 +2629,9 @@
       "id": "gardening-medium",
       "mismatches": [
         {
-          "actual": "Food & Drink",
-          "expected": "Home & Garden",
-          "path": "model_output.classification.iab_content.tier1.label"
-        },
-        {
-          "actual": null,
-          "expected": "Gardening",
-          "path": "model_output.classification.iab_content.tier2.label"
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
         }
       ],
       "notes": "Cross-vertical medium IAB mapping case for Home & Garden > Gardening.",
@@ -2591,8 +2688,8 @@
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "exact",
-        "model_output.classification.iab_content.tier1.label": "Genres",
-        "model_output.classification.iab_content.tier2.label": "Horror",
+        "model_output.classification.iab_content.tier1.label": "Entertainment",
+        "model_output.classification.iab_content.tier2.label": "Movies",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -2604,7 +2701,7 @@
       "id": "movies-medium",
       "mismatches": [
         {
-          "actual": "Genres",
+          "actual": "Entertainment",
           "expected": "Video Gaming",
           "path": "model_output.classification.iab_content.tier1.label"
         },
@@ -2614,7 +2711,7 @@
           "path": "model_output.classification.iab_content.mapping_mode"
         },
         {
-          "actual": "Horror",
+          "actual": "Movies",
           "expected": "Video Game Genres",
           "path": "model_output.classification.iab_content.tier2.label"
         },

artifacts/evaluation/latest/iab_cross_vertical_quality_target_eval.json

@@ -1,21 +1,21 @@
 {
   "by_status": {
     "must_fix": {
-      "failed": 63,
-      "passed": 27,
+      "failed": 49,
+      "passed": 41,
       "total": 90
     }
   },
   "cases_path": "/content/agentic-intent-classifier/examples/iab_cross_vertical_mapping_cases.json",
   "count": 90,
-  "failed": 63,
-  "passed": 27,
+  "failed": 49,
+  "passed": 41,
   "results": [
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "exact",
-        "model_output.classification.iab_content.tier1.label": "Automotive",
-        "model_output.classification.iab_content.tier2.label": "Auto Rentals"
+        "model_output.classification.iab_content.tier1.label": "Personal Finance",
+        "model_output.classification.iab_content.tier2.label": "Insurance"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -24,13 +24,18 @@
       },
       "id": "auto-buying-easy",
       "mismatches": [
+        {
+          "actual": "Personal Finance",
+          "expected": "Automotive",
+          "path": "model_output.classification.iab_content.tier1.label"
+        },
         {
           "actual": "exact",
           "expected": "nearest_equivalent",
           "path": "model_output.classification.iab_content.mapping_mode"
         },
         {
-          "actual": "Auto Rentals",
+          "actual": "Insurance",
           "expected": "Auto Buying and Selling",
           "path": "model_output.classification.iab_content.tier2.label"
         }
@@ -42,7 +47,7 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Automotive",
         "model_output.classification.iab_content.tier2.label": "Auto Body Styles"
       },
@@ -53,6 +58,11 @@
       },
       "id": "auto-buying-medium",
       "mismatches": [
+        {
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
         {
           "actual": "Auto Body Styles",
           "expected": "Auto Buying and Selling",
@@ -66,9 +76,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Automotive",
-        "model_output.classification.iab_content.tier2.label": "Auto Type"
+        "model_output.classification.iab_content.tier2.label": "Car Culture"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -78,7 +88,12 @@
       "id": "auto-buying-hard",
       "mismatches": [
         {
-          "actual": "Auto Type",
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "Car Culture",
           "expected": "Auto Buying and Selling",
           "path": "model_output.classification.iab_content.tier2.label"
         }
@@ -90,10 +105,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier3.label": "Software and Applications"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -108,18 +123,13 @@
           "expected": "Business and Finance",
           "path": "model_output.classification.iab_content.tier1.label"
         },
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
         {
           "actual": "Computing",
           "expected": "Business",
           "path": "model_output.classification.iab_content.tier2.label"
         },
         {
-          "actual": null,
+          "actual": "Software and Applications",
           "expected": "Sales",
           "path": "model_output.classification.iab_content.tier3.label"
         }
@@ -131,9 +141,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
-        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier2.label": "Robotics",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -150,12 +160,7 @@
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
+          "actual": "Robotics",
           "expected": "Business",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -172,10 +177,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Business and Finance",
         "model_output.classification.iab_content.tier2.label": "Business",
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier3.label": "Sales"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -184,28 +189,17 @@
         "model_output.classification.iab_content.tier3.label": "Sales"
       },
       "id": "sales-crm-hard",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
-          "expected": "Sales",
-          "path": "model_output.classification.iab_content.tier3.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical hard IAB mapping case for Business and Finance > Business > Sales.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Need software to manage leads and pipeline for a startup sales team"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Hobbies & Interests",
-        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Careers",
+        "model_output.classification.iab_content.tier2.label": "Job Search",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -217,17 +211,12 @@
       "id": "marketing-tools-easy",
       "mismatches": [
         {
-          "actual": "Hobbies & Interests",
+          "actual": "Careers",
           "expected": "Business and Finance",
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
+          "actual": "Job Search",
           "expected": "Business",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -244,9 +233,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Careers",
-        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Sensitive Topics",
+        "model_output.classification.iab_content.tier2.label": "Terrorism",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -258,17 +247,12 @@
       "id": "marketing-tools-medium",
       "mismatches": [
         {
-          "actual": "Careers",
+          "actual": "Sensitive Topics",
           "expected": "Business and Finance",
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
+          "actual": "Terrorism",
           "expected": "Business",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -285,9 +269,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Personal Finance",
-        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier2.label": "Home Utilities",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -304,12 +288,7 @@
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
+          "actual": "Home Utilities",
           "expected": "Business",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -326,10 +305,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier3.label": "Information and Network Security"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -344,18 +323,13 @@
           "expected": "Business and Finance",
           "path": "model_output.classification.iab_content.tier1.label"
         },
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
         {
           "actual": "Computing",
           "expected": "Business",
           "path": "model_output.classification.iab_content.tier2.label"
         },
         {
-          "actual": null,
+          "actual": "Information and Network Security",
           "expected": "Business I.T.",
           "path": "model_output.classification.iab_content.tier3.label"
         }
@@ -367,9 +341,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Personal Finance",
-        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Careers",
+        "model_output.classification.iab_content.tier2.label": "Job Search",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -381,17 +355,12 @@
       "id": "business-it-medium",
       "mismatches": [
         {
-          "actual": "Personal Finance",
+          "actual": "Careers",
           "expected": "Business and Finance",
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
+          "actual": "Job Search",
           "expected": "Business",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -492,8 +461,8 @@
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "exact",
-        "model_output.classification.iab_content.tier1.label": "Food & Drink",
-        "model_output.classification.iab_content.tier2.label": "Dining Out"
+        "model_output.classification.iab_content.tier1.label": "Attractions",
+        "model_output.classification.iab_content.tier2.label": "Bars & Restaurants"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -501,9 +470,20 @@
         "model_output.classification.iab_content.tier2.label": "Dining Out"
       },
       "id": "dining-out-hard",
-      "mismatches": [],
+      "mismatches": [
+        {
+          "actual": "Attractions",
+          "expected": "Food & Drink",
+          "path": "model_output.classification.iab_content.tier1.label"
+        },
+        {
+          "actual": "Bars & Restaurants",
+          "expected": "Dining Out",
+          "path": "model_output.classification.iab_content.tier2.label"
+        }
+      ],
       "notes": "Cross-vertical hard IAB mapping case for Food & Drink > Dining Out.",
-      "pass": true,
+      "pass": false,
       "status": "must_fix",
       "text": "Need a place to eat tonight where I can make a reservation online"
     },
@@ -547,7 +527,7 @@
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Food & Drink",
-        "model_output.classification.iab_content.tier2.label": "Alcoholic Beverages"
+        "model_output.classification.iab_content.tier2.label": "Non-Alcoholic Beverages"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -555,16 +535,22 @@
         "model_output.classification.iab_content.tier2.label": "Alcoholic Beverages"
       },
       "id": "alcoholic-beverages-hard",
-      "mismatches": [],
+      "mismatches": [
+        {
+          "actual": "Non-Alcoholic Beverages",
+          "expected": "Alcoholic Beverages",
+          "path": "model_output.classification.iab_content.tier2.label"
+        }
+      ],
       "notes": "Cross-vertical hard IAB mapping case for Food & Drink > Alcoholic Beverages.",
-      "pass": true,
+      "pass": false,
       "status": "must_fix",
       "text": "Want a spirit-forward drink recommendation, not a restaurant suggestion"
     },
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Sports",
+        "model_output.classification.iab_content.tier1.label": "Science",
         "model_output.classification.iab_content.tier2.label": null
       },
       "expected": {
@@ -575,7 +561,7 @@
       "id": "artificial-intelligence-easy",
       "mismatches": [
         {
-          "actual": "Sports",
+          "actual": "Science",
           "expected": "Technology & Computing",
           "path": "model_output.classification.iab_content.tier1.label"
         },
@@ -655,10 +641,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier3.label": "Software and Applications"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -667,29 +653,18 @@
         "model_output.classification.iab_content.tier3.label": "Software and Applications"
       },
       "id": "software-apps-easy",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
-          "expected": "Software and Applications",
-          "path": "model_output.classification.iab_content.tier3.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical easy IAB mapping case for Technology & Computing > Computing > Computer Software and Applications.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Best workflow software for a small operations team"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier3.label": "Software and Applications"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -698,28 +673,17 @@
         "model_output.classification.iab_content.tier3.label": "Software and Applications"
       },
       "id": "software-apps-medium",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
-          "expected": "Software and Applications",
-          "path": "model_output.classification.iab_content.tier3.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical medium IAB mapping case for Technology & Computing > Computing > Computer Software and Applications.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Need project management software for a distributed team"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Business and Finance",
-        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Virtual Reality",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -731,17 +695,7 @@
       "id": "software-apps-hard",
       "mismatches": [
         {
-          "actual": "Business and Finance",
-          "expected": "Technology & Computing",
-          "path": "model_output.classification.iab_content.tier1.label"
-        },
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
+          "actual": "Virtual Reality",
           "expected": "Computing",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -801,10 +755,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": null,
+        "model_output.classification.iab_content.tier3.label": "Information and Network Security",
         "model_output.classification.iab_content.tier4.label": null
       },
       "expected": {
@@ -817,12 +771,7 @@
       "id": "communication-software-medium",
       "mismatches": [
         {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
+          "actual": "Information and Network Security",
           "expected": "Software and Applications",
           "path": "model_output.classification.iab_content.tier3.label"
         },
@@ -839,9 +788,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Careers",
-        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Virtual Reality",
         "model_output.classification.iab_content.tier3.label": null,
         "model_output.classification.iab_content.tier4.label": null
       },
@@ -855,17 +804,7 @@
       "id": "communication-software-hard",
       "mismatches": [
         {
-          "actual": "Careers",
-          "expected": "Technology & Computing",
-          "path": "model_output.classification.iab_content.tier1.label"
-        },
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
+          "actual": "Virtual Reality",
           "expected": "Computing",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -887,11 +826,11 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": null,
-        "model_output.classification.iab_content.tier4.label": null
+        "model_output.classification.iab_content.tier3.label": "Internet",
+        "model_output.classification.iab_content.tier4.label": "Web Hosting"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -901,35 +840,19 @@
         "model_output.classification.iab_content.tier4.label": "Web Hosting"
       },
       "id": "web-hosting-easy",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
-          "expected": "Internet",
-          "path": "model_output.classification.iab_content.tier3.label"
-        },
-        {
-          "actual": null,
-          "expected": "Web Hosting",
-          "path": "model_output.classification.iab_content.tier4.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical easy IAB mapping case for Technology & Computing > Computing > Internet > Web Hosting.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Vercel vs Netlify for website hosting"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": null,
-        "model_output.classification.iab_content.tier4.label": null
+        "model_output.classification.iab_content.tier3.label": "Internet",
+        "model_output.classification.iab_content.tier4.label": "Web Hosting"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -939,35 +862,19 @@
         "model_output.classification.iab_content.tier4.label": "Web Hosting"
       },
       "id": "web-hosting-medium",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
-          "expected": "Internet",
-          "path": "model_output.classification.iab_content.tier3.label"
-        },
-        {
-          "actual": null,
-          "expected": "Web Hosting",
-          "path": "model_output.classification.iab_content.tier4.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical medium IAB mapping case for Technology & Computing > Computing > Internet > Web Hosting.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Best hosting platform for a startup website"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": null,
-        "model_output.classification.iab_content.tier4.label": null
+        "model_output.classification.iab_content.tier3.label": "Internet",
+        "model_output.classification.iab_content.tier4.label": "Web Hosting"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -977,34 +884,18 @@
         "model_output.classification.iab_content.tier4.label": "Web Hosting"
       },
       "id": "web-hosting-hard",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
-          "expected": "Internet",
-          "path": "model_output.classification.iab_content.tier3.label"
-        },
-        {
-          "actual": null,
-          "expected": "Web Hosting",
-          "path": "model_output.classification.iab_content.tier4.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical hard IAB mapping case for Technology & Computing > Computing > Internet > Web Hosting.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Need a managed hosting provider to deploy and run our marketing site"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
-        "model_output.classification.iab_content.tier2.label": null,
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Laptops"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1013,25 +904,9 @@
         "model_output.classification.iab_content.tier3.label": "Laptops"
       },
       "id": "laptops-easy",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
-          "expected": "Computing",
-          "path": "model_output.classification.iab_content.tier2.label"
-        },
-        {
-          "actual": null,
-          "expected": "Laptops",
-          "path": "model_output.classification.iab_content.tier3.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical easy IAB mapping case for Technology & Computing > Computing > Laptops.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Which laptop should I buy for college?"
     },
@@ -1057,10 +932,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier3.label": "Laptops"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1069,26 +944,15 @@
         "model_output.classification.iab_content.tier3.label": "Laptops"
       },
       "id": "laptops-hard",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
-          "expected": "Laptops",
-          "path": "model_output.classification.iab_content.tier3.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical hard IAB mapping case for Technology & Computing > Computing > Laptops.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Need a portable computer with good battery life for everyday work"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
         "model_output.classification.iab_content.tier3.label": "Software and Applications"
@@ -1101,11 +965,6 @@
       },
       "id": "desktops-easy",
       "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
         {
           "actual": "Software and Applications",
           "expected": "Desktops",
@@ -1119,10 +978,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier3.label": "Desktops"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1131,20 +990,9 @@
         "model_output.classification.iab_content.tier3.label": "Desktops"
       },
       "id": "desktops-medium",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
-          "expected": "Desktops",
-          "path": "model_output.classification.iab_content.tier3.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical medium IAB mapping case for Technology & Computing > Computing > Desktops.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Which desktop computer should I buy for a home office?"
     },
@@ -1173,7 +1021,7 @@
         "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Consumer Electronics",
-        "model_output.classification.iab_content.tier3.label": "Wearable Technology"
+        "model_output.classification.iab_content.tier3.label": "Smartphones"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1182,15 +1030,9 @@
         "model_output.classification.iab_content.tier3.label": "Smartphones"
       },
       "id": "smartphones-easy",
-      "mismatches": [
-        {
-          "actual": "Wearable Technology",
-          "expected": "Smartphones",
-          "path": "model_output.classification.iab_content.tier3.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical easy IAB mapping case for Technology & Computing > Consumer Electronics > Smartphones.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Best phone with a good camera under 700"
     },
@@ -1199,7 +1041,7 @@
         "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Consumer Electronics",
-        "model_output.classification.iab_content.tier3.label": "Wearable Technology"
+        "model_output.classification.iab_content.tier3.label": "Smartphones"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1208,15 +1050,9 @@
         "model_output.classification.iab_content.tier3.label": "Smartphones"
       },
       "id": "smartphones-medium",
-      "mismatches": [
-        {
-          "actual": "Wearable Technology",
-          "expected": "Smartphones",
-          "path": "model_output.classification.iab_content.tier3.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical medium IAB mapping case for Technology & Computing > Consumer Electronics > Smartphones.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Should I buy an iPhone or Pixel this year?"
     },
@@ -1225,7 +1061,7 @@
         "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Consumer Electronics",
-        "model_output.classification.iab_content.tier3.label": "Wearable Technology"
+        "model_output.classification.iab_content.tier3.label": "Smartphones"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1234,15 +1070,9 @@
         "model_output.classification.iab_content.tier3.label": "Smartphones"
       },
       "id": "smartphones-hard",
-      "mismatches": [
-        {
-          "actual": "Wearable Technology",
-          "expected": "Smartphones",
-          "path": "model_output.classification.iab_content.tier3.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical hard IAB mapping case for Technology & Computing > Consumer Electronics > Smartphones.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Need a new smartphone with strong battery life and a clean software experience"
     },
@@ -1314,9 +1144,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Sports",
-        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier2.label": "Bodybuilding",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -1333,12 +1163,7 @@
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
+          "actual": "Bodybuilding",
           "expected": "Women's Fashion",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -1411,9 +1236,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Style & Fashion",
-        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier2.label": "Children's Clothing",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -1425,12 +1250,7 @@
       "id": "mens-shoes-easy",
       "mismatches": [
         {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
+          "actual": "Children's Clothing",
           "expected": "Men's Fashion",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -1467,9 +1287,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Style & Fashion",
-        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier2.label": "Children's Clothing",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -1481,12 +1301,7 @@
       "id": "mens-shoes-hard",
       "mismatches": [
         {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
+          "actual": "Children's Clothing",
           "expected": "Men's Fashion",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -1545,7 +1360,7 @@
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Travel",
-        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier2.label": "Travel Type",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -1556,11 +1371,6 @@
       },
       "id": "hotels-hard",
       "mismatches": [
-        {
-          "actual": null,
-          "expected": "Travel Type",
-          "path": "model_output.classification.iab_content.tier2.label"
-        },
         {
           "actual": null,
           "expected": "Hotels and Motels",
@@ -1645,10 +1455,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Healthy Living",
-        "model_output.classification.iab_content.tier2.label": null,
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier2.label": "Fitness and Exercise",
+        "model_output.classification.iab_content.tier3.label": "Running and Jogging"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1657,25 +1467,9 @@
         "model_output.classification.iab_content.tier3.label": "Running and Jogging"
       },
       "id": "running-and-jogging-easy",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
-          "expected": "Fitness and Exercise",
-          "path": "model_output.classification.iab_content.tier2.label"
-        },
-        {
-          "actual": null,
-          "expected": "Running and Jogging",
-          "path": "model_output.classification.iab_content.tier3.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical easy IAB mapping case for Healthy Living > Fitness and Exercise > Running and Jogging.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Best running plan for a first 10k"
     },
@@ -1718,8 +1512,8 @@
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "exact",
-        "model_output.classification.iab_content.tier1.label": "Healthy Living",
-        "model_output.classification.iab_content.tier2.label": "Fitness and Exercise",
+        "model_output.classification.iab_content.tier1.label": "Sports",
+        "model_output.classification.iab_content.tier2.label": "Walking",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -1730,6 +1524,16 @@
       },
       "id": "running-and-jogging-hard",
       "mismatches": [
+        {
+          "actual": "Sports",
+          "expected": "Healthy Living",
+          "path": "model_output.classification.iab_content.tier1.label"
+        },
+        {
+          "actual": "Walking",
+          "expected": "Fitness and Exercise",
+          "path": "model_output.classification.iab_content.tier2.label"
+        },
         {
           "actual": null,
           "expected": "Running and Jogging",
@@ -1827,8 +1631,8 @@
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "exact",
-        "model_output.classification.iab_content.tier1.label": "Books and Literature",
-        "model_output.classification.iab_content.tier2.label": "Fiction"
+        "model_output.classification.iab_content.tier1.label": "Travel",
+        "model_output.classification.iab_content.tier2.label": "Travel Type"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -1837,10 +1641,20 @@
       },
       "id": "fiction-medium",
       "mismatches": [
+        {
+          "actual": "Travel",
+          "expected": "Books and Literature",
+          "path": "model_output.classification.iab_content.tier1.label"
+        },
         {
           "actual": "exact",
           "expected": "nearest_equivalent",
           "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "Travel Type",
+          "expected": "Fiction",
+          "path": "model_output.classification.iab_content.tier2.label"
         }
       ],
       "notes": "Cross-vertical medium IAB mapping case for Books and Literature > Fiction.",
@@ -1870,7 +1684,7 @@
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Home & Garden",
-        "model_output.classification.iab_content.tier2.label": "Remodeling & Construction"
+        "model_output.classification.iab_content.tier2.label": "Interior Decorating"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1880,7 +1694,7 @@
       "id": "home-improvement-easy",
       "mismatches": [
         {
-          "actual": "Remodeling & Construction",
+          "actual": "Interior Decorating",
           "expected": "Home Improvement",
           "path": "model_output.classification.iab_content.tier2.label"
         }
@@ -1892,9 +1706,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Home & Garden",
-        "model_output.classification.iab_content.tier2.label": null
+        "model_output.classification.iab_content.tier2.label": "Interior Decorating"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1904,12 +1718,7 @@
       "id": "home-improvement-medium",
       "mismatches": [
         {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
+          "actual": "Interior Decorating",
           "expected": "Home Improvement",
           "path": "model_output.classification.iab_content.tier2.label"
         }
@@ -1950,9 +1759,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
-        "model_output.classification.iab_content.tier2.label": null
+        "model_output.classification.iab_content.tier2.label": "Augmented Reality"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1967,12 +1776,7 @@
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
+          "actual": "Augmented Reality",
           "expected": "Online Education",
           "path": "model_output.classification.iab_content.tier2.label"
         }
@@ -2013,9 +1817,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Healthy Living",
-        "model_output.classification.iab_content.tier2.label": null
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Careers",
+        "model_output.classification.iab_content.tier2.label": "Vocational Training"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -2025,17 +1829,12 @@
       "id": "online-education-hard",
       "mismatches": [
         {
-          "actual": "Healthy Living",
+          "actual": "Careers",
           "expected": "Education",
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
+          "actual": "Vocational Training",
           "expected": "Online Education",
           "path": "model_output.classification.iab_content.tier2.label"
         }
@@ -2107,7 +1906,7 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Medical Health"
       },
       "expected": {
@@ -2115,15 +1914,9 @@
         "model_output.classification.iab_content.tier1.label": "Medical Health"
       },
       "id": "medical-health-easy",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical easy IAB mapping case for Medical Health.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "what do these allergy symptoms mean"
     },
@@ -2185,7 +1978,7 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Business and Finance",
         "model_output.classification.iab_content.tier2.label": "Business"
       },
@@ -2201,11 +1994,6 @@
           "expected": "Careers",
           "path": "model_output.classification.iab_content.tier1.label"
         },
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
         {
           "actual": "Business",
           "expected": "Job Search",
@@ -2219,9 +2007,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Genres",
-        "model_output.classification.iab_content.tier2.label": null
+        "model_output.classification.iab_content.tier2.label": "Talk Show"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -2236,12 +2024,7 @@
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
+          "actual": "Talk Show",
           "expected": "Job Search",
           "path": "model_output.classification.iab_content.tier2.label"
         }
@@ -2253,9 +2036,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Personal Celebrations & Life Events",
-        "model_output.classification.iab_content.tier2.label": null
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Holidays",
+        "model_output.classification.iab_content.tier2.label": "National & Civic Holidays"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -2265,17 +2048,12 @@
       "id": "personal-finance-easy",
       "mismatches": [
         {
-          "actual": "Personal Celebrations & Life Events",
+          "actual": "Holidays",
           "expected": "Personal Finance",
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
+          "actual": "National & Civic Holidays",
           "expected": "Financial Planning",
           "path": "model_output.classification.iab_content.tier2.label"
         }
@@ -2354,8 +2132,8 @@
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "exact",
-        "model_output.classification.iab_content.tier1.label": "Hobbies & Interests",
-        "model_output.classification.iab_content.tier2.label": "Content Production"
+        "model_output.classification.iab_content.tier1.label": "Genres",
+        "model_output.classification.iab_content.tier2.label": "Family/Children"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -2365,12 +2143,12 @@
       "id": "parenting-medium",
       "mismatches": [
         {
-          "actual": "Hobbies & Interests",
+          "actual": "Genres",
           "expected": "Family and Relationships",
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": "Content Production",
+          "actual": "Family/Children",
           "expected": "Parenting",
           "path": "model_output.classification.iab_content.tier2.label"
         }
@@ -2418,9 +2196,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Food & Drink",
-        "model_output.classification.iab_content.tier2.label": null
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Home & Garden",
+        "model_output.classification.iab_content.tier2.label": "Gardening"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -2428,25 +2206,9 @@
         "model_output.classification.iab_content.tier2.label": "Gardening"
       },
       "id": "gardening-medium",
-      "mismatches": [
-        {
-          "actual": "Food & Drink",
-          "expected": "Home & Garden",
-          "path": "model_output.classification.iab_content.tier1.label"
-        },
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
-          "expected": "Gardening",
-          "path": "model_output.classification.iab_content.tier2.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical medium IAB mapping case for Home & Garden > Gardening.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "how often should i water tomato plants"
     },
@@ -2489,8 +2251,8 @@
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "exact",
-        "model_output.classification.iab_content.tier1.label": "Genres",
-        "model_output.classification.iab_content.tier2.label": "Horror"
+        "model_output.classification.iab_content.tier1.label": "Entertainment",
+        "model_output.classification.iab_content.tier2.label": "Movies"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -2498,20 +2260,9 @@
         "model_output.classification.iab_content.tier2.label": "Movies"
       },
       "id": "movies-medium",
-      "mismatches": [
-        {
-          "actual": "Genres",
-          "expected": "Entertainment",
-          "path": "model_output.classification.iab_content.tier1.label"
-        },
-        {
-          "actual": "Horror",
-          "expected": "Movies",
-          "path": "model_output.classification.iab_content.tier2.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical medium IAB mapping case for Entertainment > Movies.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Best thriller movies from the last few years"
     },

artifacts/evaluation/latest/iab_quality_target_eval.json

@@ -13,7 +13,7 @@
   "results": [
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Automotive",
         "model_output.classification.iab_content.tier2.label": null
       },
@@ -28,11 +28,6 @@
           "actual": null,
           "expected": "Auto Buying and Selling",
           "path": "model_output.classification.iab_content.tier2.label"
-        },
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
         }
       ],
       "notes": "Vehicle shopping queries should map into the automotive buying branch, not business sales.",
@@ -62,7 +57,7 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": null,
         "model_output.classification.iab_content.tier3.label": null
@@ -84,11 +79,6 @@
           "actual": null,
           "expected": "Laptops",
           "path": "model_output.classification.iab_content.tier3.label"
-        },
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
         }
       ],
       "notes": "Common typo handling should still land in the laptops branch.",
@@ -98,10 +88,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier3.label": "Software and Applications"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -122,9 +112,14 @@
           "path": "model_output.classification.iab_content.tier2.label"
         },
         {
-          "actual": null,
+          "actual": "Software and Applications",
           "expected": "Sales",
           "path": "model_output.classification.iab_content.tier3.label"
+        },
+        {
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
         }
       ],
       "notes": "CRM education should resolve to the closest business/sales path, not generic software.",
@@ -134,9 +129,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
-        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier2.label": "Robotics",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -153,7 +148,7 @@
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": null,
+          "actual": "Robotics",
           "expected": "Business",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -161,11 +156,6 @@
           "actual": null,
           "expected": "Sales",
           "path": "model_output.classification.iab_content.tier3.label"
-        },
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
         }
       ],
       "notes": "Direct CRM vendor comparison should map cleanly into the sales domain.",
@@ -175,9 +165,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Careers",
-        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier2.label": "Job Search",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -194,7 +184,7 @@
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": null,
+          "actual": "Job Search",
           "expected": "Business",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -202,11 +192,6 @@
           "actual": null,
           "expected": "Marketing and Advertising",
           "path": "model_output.classification.iab_content.tier3.label"
-        },
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
         }
       ],
       "notes": "Marketing tool discovery should map to the marketing and advertising branch.",
@@ -217,7 +202,7 @@
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Sports",
+        "model_output.classification.iab_content.tier1.label": "Science",
         "model_output.classification.iab_content.tier2.label": null
       },
       "expected": {
@@ -228,7 +213,7 @@
       "id": "ml-explanation-maps-to-ai",
       "mismatches": [
         {
-          "actual": "Sports",
+          "actual": "Science",
           "expected": "Technology & Computing",
           "path": "model_output.classification.iab_content.tier1.label"
         },
@@ -250,10 +235,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier3.label": "Information and Network Security"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -274,9 +259,14 @@
           "path": "model_output.classification.iab_content.tier2.label"
         },
         {
-          "actual": null,
+          "actual": "Information and Network Security",
           "expected": "Business I.T.",
           "path": "model_output.classification.iab_content.tier3.label"
+        },
+        {
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
         }
       ],
       "notes": "Credential and account help should map to business IT rather than generic business.",
@@ -304,9 +294,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Sports",
-        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Sensitive Topics",
+        "model_output.classification.iab_content.tier2.label": "Crime & Harmful Acts to Individuals, Society & Human Right Violations",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -318,12 +308,12 @@
       "id": "trial-signup-maps-to-software",
       "mismatches": [
         {
-          "actual": "Sports",
+          "actual": "Sensitive Topics",
           "expected": "Technology & Computing",
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": null,
+          "actual": "Crime & Harmful Acts to Individuals, Society & Human Right Violations",
           "expected": "Computing",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -331,6 +321,11 @@
           "actual": null,
           "expected": "Software and Applications",
           "path": "model_output.classification.iab_content.tier3.label"
+        },
+        {
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
         }
       ],
       "notes": "Software action queries should map to the software/application branch.",

artifacts/evaluation/latest/intent_subtype_difficulty_benchmark_confusion_matrix.csv

@@ -1,19 +1,19 @@
 ,education,product_discovery,comparison,evaluation,deal_seeking,provider_selection,signup,purchase,booking,download,contact_sales,task_execution,onboarding_setup,troubleshooting,account_help,billing_help,follow_up,emotional_reflection
 education,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
-product_discovery,0,13,0,1,0,0,0,0,0,0,0,1,0,0,0,0,0,0
-comparison,1,0,11,2,0,0,0,0,0,0,0,0,0,0,0,1,0,0
-evaluation,1,0,0,14,0,0,0,0,0,0,0,0,0,0,0,0,0,0
-deal_seeking,0,1,1,0,13,0,0,0,0,0,0,0,0,0,0,0,0,0
+product_discovery,0,14,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+comparison,2,1,11,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0
+evaluation,1,0,0,13,0,0,0,0,0,0,0,0,1,0,0,0,0,0
+deal_seeking,0,1,0,0,13,1,0,0,0,0,0,0,0,0,0,0,0,0
 provider_selection,0,0,0,0,0,16,0,0,0,0,0,0,0,0,0,0,0,0
-signup,0,0,0,0,0,0,14,0,0,1,0,0,0,0,1,0,0,0
-purchase,0,0,0,0,0,0,1,13,0,0,0,1,0,0,0,0,0,0
-booking,0,0,0,0,0,0,5,0,8,0,1,1,0,0,0,0,0,0
-download,0,0,0,0,0,0,0,0,0,13,0,1,0,0,1,0,0,0
-contact_sales,0,0,0,1,0,0,0,0,0,0,14,0,0,0,0,0,0,0
-task_execution,0,0,0,0,0,0,0,0,0,0,0,17,1,0,0,0,0,0
-onboarding_setup,0,0,0,0,0,0,0,0,0,0,0,1,15,1,0,0,0,0
-troubleshooting,0,0,0,0,0,0,0,0,0,0,0,0,0,15,0,0,0,0
-account_help,0,0,0,0,0,0,0,0,0,0,0,0,0,1,12,2,0,0
-billing_help,0,0,0,0,0,0,0,0,0,0,0,0,0,2,1,12,0,0
-follow_up,0,0,0,0,0,0,0,0,0,0,0,1,0,0,1,0,13,0
+signup,0,0,0,0,0,0,15,1,0,0,0,0,0,0,0,0,0,0
+purchase,0,0,0,0,0,0,0,13,0,0,0,0,2,0,0,0,0,0
+booking,0,0,0,0,0,0,1,0,13,0,1,0,0,0,0,0,0,0
+download,0,0,0,0,0,0,0,0,0,13,1,1,0,0,0,0,0,0
+contact_sales,0,0,0,0,0,0,0,0,0,0,15,0,0,0,0,0,0,0
+task_execution,0,0,0,0,0,0,0,0,0,0,0,18,0,0,0,0,0,0
+onboarding_setup,0,0,0,0,0,0,0,0,0,0,0,1,16,0,0,0,0,0
+troubleshooting,0,0,0,0,0,0,0,0,0,0,0,0,0,12,1,2,0,0
+account_help,0,0,0,0,0,0,2,0,0,0,0,1,0,3,8,1,0,0
+billing_help,0,0,0,0,0,0,0,0,0,0,0,0,1,0,1,13,0,0
+follow_up,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,14,0
 emotional_reflection,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,15

artifacts/evaluation/latest/intent_subtype_difficulty_benchmark_report.json

@@ -1,7 +1,7 @@
 {
-  "accepted_accuracy": 0.8967,
-  "accepted_coverage": 0.9783,
-  "accuracy": 0.8773,
+  "accepted_accuracy": 0.9104,
+  "accepted_coverage": 0.9675,
+  "accuracy": 0.8917,
   "confusion_matrix_path": "/content/agentic-intent-classifier/artifacts/evaluation/latest/intent_subtype_difficulty_benchmark_confusion_matrix.csv",
   "count": 277,
   "dataset_path": "/content/agentic-intent-classifier/data/subtype_benchmark.jsonl",
@@ -12,15 +12,15 @@
       "accuracy": 0.913,
       "count": 92,
       "fallback_rate": 0.0,
-      "macro_f1": 0.9111
+      "macro_f1": 0.9109
     },
     "hard": {
-      "accepted_accuracy": 0.814,
-      "accepted_coverage": 0.9451,
-      "accuracy": 0.7692,
+      "accepted_accuracy": 0.8554,
+      "accepted_coverage": 0.9121,
+      "accuracy": 0.8132,
       "count": 91,
-      "fallback_rate": 0.0549,
-      "macro_f1": 0.7704
+      "fallback_rate": 0.0879,
+      "macro_f1": 0.8025
     },
     "medium": {
       "accepted_accuracy": 0.957,
@@ -28,30 +28,30 @@
       "accuracy": 0.9468,
       "count": 94,
       "fallback_rate": 0.0106,
-      "macro_f1": 0.9453
+      "macro_f1": 0.9469
     }
   },
-  "fallback_rate": 0.0217,
+  "fallback_rate": 0.0325,
   "head": "intent_subtype",
-  "macro_f1": 0.8767,
+  "macro_f1": 0.8886,
   "per_class_metrics": {
     "account_help": {
-      "f1-score": 0.7741935483870968,
-      "precision": 0.75,
-      "recall": 0.8,
+      "f1-score": 0.64,
+      "precision": 0.8,
+      "recall": 0.5333333333333333,
       "support": 15.0
     },
-    "accuracy": 0.8772563176895307,
+    "accuracy": 0.8916967509025271,
     "billing_help": {
-      "f1-score": 0.8,
-      "precision": 0.8,
-      "recall": 0.8,
+      "f1-score": 0.8387096774193549,
+      "precision": 0.8125,
+      "recall": 0.8666666666666667,
       "support": 15.0
     },
     "booking": {
-      "f1-score": 0.6956521739130435,
+      "f1-score": 0.9285714285714286,
       "precision": 1.0,
-      "recall": 0.5333333333333333,
+      "recall": 0.8666666666666667,
       "support": 15.0
     },
     "comparison": {
@@ -61,26 +61,26 @@
       "support": 15.0
     },
     "contact_sales": {
-      "f1-score": 0.9333333333333333,
-      "precision": 0.9333333333333333,
-      "recall": 0.9333333333333333,
+      "f1-score": 0.9375,
+      "precision": 0.8823529411764706,
+      "recall": 1.0,
       "support": 15.0
     },
     "deal_seeking": {
-      "f1-score": 0.9285714285714286,
-      "precision": 1.0,
+      "f1-score": 0.896551724137931,
+      "precision": 0.9285714285714286,
       "recall": 0.8666666666666667,
       "support": 15.0
     },
     "download": {
-      "f1-score": 0.896551724137931,
-      "precision": 0.9285714285714286,
+      "f1-score": 0.9285714285714286,
+      "precision": 1.0,
       "recall": 0.8666666666666667,
       "support": 15.0
     },
     "education": {
-      "f1-score": 0.9375,
-      "precision": 0.8823529411764706,
+      "f1-score": 0.9090909090909091,
+      "precision": 0.8333333333333334,
       "recall": 1.0,
       "support": 15.0
     },
@@ -91,69 +91,69 @@
       "support": 15.0
     },
     "evaluation": {
-      "f1-score": 0.8484848484848485,
-      "precision": 0.7777777777777778,
-      "recall": 0.9333333333333333,
+      "f1-score": 0.896551724137931,
+      "precision": 0.9285714285714286,
+      "recall": 0.8666666666666667,
       "support": 15.0
     },
     "follow_up": {
-      "f1-score": 0.9285714285714286,
+      "f1-score": 0.9655172413793104,
       "precision": 1.0,
-      "recall": 0.8666666666666667,
+      "recall": 0.9333333333333333,
       "support": 15.0
     },
     "macro avg": {
-      "f1-score": 0.8767381318695205,
-      "precision": 0.893520983060569,
-      "recall": 0.8760257806826435,
+      "f1-score": 0.8886471209737711,
+      "precision": 0.8965122159975102,
+      "recall": 0.8895561002178651,
       "support": 277.0
     },
     "onboarding_setup": {
-      "f1-score": 0.9090909090909091,
-      "precision": 0.9375,
-      "recall": 0.8823529411764706,
+      "f1-score": 0.8648648648648649,
+      "precision": 0.8,
+      "recall": 0.9411764705882353,
       "support": 17.0
     },
     "product_discovery": {
-      "f1-score": 0.896551724137931,
-      "precision": 0.9285714285714286,
-      "recall": 0.8666666666666667,
+      "f1-score": 0.9032258064516129,
+      "precision": 0.875,
+      "recall": 0.9333333333333333,
       "support": 15.0
     },
     "provider_selection": {
-      "f1-score": 1.0,
-      "precision": 1.0,
+      "f1-score": 0.9696969696969697,
+      "precision": 0.9411764705882353,
       "recall": 1.0,
       "support": 16.0
     },
     "purchase": {
-      "f1-score": 0.9285714285714286,
-      "precision": 1.0,
+      "f1-score": 0.896551724137931,
+      "precision": 0.9285714285714286,
       "recall": 0.8666666666666667,
       "support": 15.0
     },
     "signup": {
-      "f1-score": 0.7777777777777778,
-      "precision": 0.7,
-      "recall": 0.875,
+      "f1-score": 0.8823529411764706,
+      "precision": 0.8333333333333334,
+      "recall": 0.9375,
       "support": 16.0
     },
     "task_execution": {
-      "f1-score": 0.8292682926829268,
-      "precision": 0.7391304347826086,
-      "recall": 0.9444444444444444,
+      "f1-score": 0.9230769230769231,
+      "precision": 0.8571428571428571,
+      "recall": 1.0,
       "support": 18.0
     },
     "troubleshooting": {
-      "f1-score": 0.8823529411764706,
-      "precision": 0.7894736842105263,
-      "recall": 1.0,
+      "f1-score": 0.8,
+      "precision": 0.8,
+      "recall": 0.8,
       "support": 15.0
     },
     "weighted avg": {
-      "f1-score": 0.8765453432446891,
-      "precision": 0.8918521903635431,
-      "recall": 0.8772563176895307,
+      "f1-score": 0.8891181699377334,
+      "precision": 0.8953221541324111,
+      "recall": 0.8916967509025271,
       "support": 277.0
     }
   },

artifacts/evaluation/latest/intent_subtype_extended_cases_confusion_matrix.csv

@@ -15,5 +15,5 @@ onboarding_setup,0,0,0,0,0,0,0,0,0,0,0,0,4,0,0,0,0,0
 troubleshooting,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0
 account_help,0,0,0,0,0,0,0,0,0,0,0,0,0,1,2,0,0,0
 billing_help,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
-follow_up,1,0,0,0,2,0,0,0,0,0,0,1,0,0,0,0,8,0
+follow_up,1,0,0,0,3,0,0,0,0,0,0,1,0,0,0,0,7,0
 emotional_reflection,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0

artifacts/evaluation/latest/intent_subtype_extended_cases_report.json

@@ -1,13 +1,13 @@
 {
-  "accepted_accuracy": 0.8491,
+  "accepted_accuracy": 0.8302,
   "accepted_coverage": 1.0,
-  "accuracy": 0.8491,
+  "accuracy": 0.8302,
   "confusion_matrix_path": "/content/agentic-intent-classifier/artifacts/evaluation/latest/intent_subtype_extended_cases_confusion_matrix.csv",
   "count": 53,
   "dataset_path": "/content/agentic-intent-classifier/data/subtype/extended_cases.jsonl",
   "fallback_rate": 0.0,
   "head": "intent_subtype",
-  "macro_f1": 0.7764,
+  "macro_f1": 0.7668,
   "per_class_metrics": {
     "account_help": {
       "f1-score": 0.8,
@@ -15,7 +15,7 @@
       "recall": 0.6666666666666666,
       "support": 3.0
     },
-    "accuracy": 0.8490566037735849,
+    "accuracy": 0.8301886792452831,
     "billing_help": {
       "f1-score": 0.0,
       "precision": 0.0,
@@ -41,8 +41,8 @@
       "support": 0.0
     },
     "deal_seeking": {
-      "f1-score": 0.9,
-      "precision": 0.8181818181818182,
+      "f1-score": 0.8571428571428571,
+      "precision": 0.75,
       "recall": 1.0,
       "support": 9.0
     },
@@ -71,15 +71,15 @@
       "support": 3.0
     },
     "follow_up": {
-      "f1-score": 0.8,
+      "f1-score": 0.7368421052631579,
       "precision": 1.0,
-      "recall": 0.6666666666666666,
+      "recall": 0.5833333333333334,
       "support": 12.0
     },
     "macro avg": {
-      "f1-score": 0.474472286972287,
-      "precision": 0.46035754369087706,
-      "recall": 0.5185185185185186,
+      "f1-score": 0.46858256266151,
+      "precision": 0.4565696649029982,
+      "recall": 0.513888888888889,
       "support": 53.0
     },
     "onboarding_setup": {
@@ -125,9 +125,9 @@
       "support": 1.0
     },
     "weighted avg": {
-      "f1-score": 0.823438668249989,
-      "precision": 0.8324076342944268,
-      "recall": 0.8490566037735849,
+      "f1-score": 0.8018611395225099,
+      "precision": 0.8208295896975142,
+      "recall": 0.8301886792452831,
       "support": 53.0
     }
   },

artifacts/evaluation/latest/intent_subtype_hard_cases_confusion_matrix.csv

@@ -2,11 +2,11 @@
 education,29,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
 product_discovery,0,8,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
 comparison,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
-evaluation,2,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+evaluation,3,2,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0
 deal_seeking,0,0,0,0,2,1,0,0,0,0,0,0,0,0,0,0,0,0
 provider_selection,0,0,0,1,0,9,0,0,0,0,0,0,0,0,0,0,0,0
 signup,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0,0
-purchase,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0
+purchase,0,0,0,0,0,0,1,2,0,0,0,0,0,0,0,0,0,0
 booking,0,0,0,0,0,0,0,0,4,0,0,0,0,0,0,0,0,0
 download,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
 contact_sales,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0

artifacts/evaluation/latest/intent_subtype_hard_cases_report.json

@@ -7,7 +7,7 @@
   "dataset_path": "/content/agentic-intent-classifier/data/subtype/hard_cases.jsonl",
   "fallback_rate": 0.0,
   "head": "intent_subtype",
-  "macro_f1": 0.846,
+  "macro_f1": 0.8447,
   "per_class_metrics": {
     "account_help": {
       "f1-score": 0.8,
@@ -53,8 +53,8 @@
       "support": 0.0
     },
     "education": {
-      "f1-score": 0.9666666666666667,
-      "precision": 0.9354838709677419,
+      "f1-score": 0.9508196721311475,
+      "precision": 0.90625,
       "recall": 1.0,
       "support": 29.0
     },
@@ -65,9 +65,9 @@
       "support": 0.0
     },
     "evaluation": {
-      "f1-score": 0.0,
-      "precision": 0.0,
-      "recall": 0.0,
+      "f1-score": 0.25,
+      "precision": 0.5,
+      "recall": 0.16666666666666666,
       "support": 6.0
     },
     "follow_up": {
@@ -77,9 +77,9 @@
       "support": 12.0
     },
     "macro avg": {
-      "f1-score": 0.7049919484702094,
-      "precision": 0.7038231780167263,
-      "recall": 0.7305555555555556,
+      "f1-score": 0.7038911013311109,
+      "precision": 0.7234953703703704,
+      "recall": 0.7212962962962962,
       "support": 94.0
     },
     "onboarding_setup": {
@@ -89,8 +89,8 @@
       "support": 6.0
     },
     "product_discovery": {
-      "f1-score": 0.8,
-      "precision": 0.6666666666666666,
+      "f1-score": 0.8888888888888888,
+      "precision": 0.8,
       "recall": 1.0,
       "support": 8.0
     },
@@ -101,14 +101,14 @@
       "support": 10.0
     },
     "purchase": {
-      "f1-score": 1.0,
+      "f1-score": 0.8,
       "precision": 1.0,
-      "recall": 1.0,
+      "recall": 0.6666666666666666,
       "support": 3.0
     },
     "signup": {
-      "f1-score": 1.0,
-      "precision": 1.0,
+      "f1-score": 0.8571428571428571,
+      "precision": 0.75,
       "recall": 1.0,
       "support": 3.0
     },
@@ -125,8 +125,8 @@
       "support": 3.0
     },
     "weighted avg": {
-      "f1-score": 0.8721091581868641,
-      "precision": 0.8648478609013955,
+      "f1-score": 0.8798004011763282,
+      "precision": 0.8911125886524823,
       "recall": 0.8936170212765957,
       "support": 94.0
     }

artifacts/evaluation/latest/intent_subtype_test_confusion_matrix.csv

@@ -2,9 +2,9 @@
 education,14,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
 product_discovery,0,8,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
 comparison,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
-evaluation,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+evaluation,0,0,1,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0
 deal_seeking,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0
-provider_selection,0,0,0,1,0,5,0,0,0,0,0,0,0,0,0,0,0,0
+provider_selection,0,0,0,1,0,4,0,0,0,0,0,0,1,0,0,0,0,0
 signup,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0
 purchase,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
 booking,0,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0
@@ -12,7 +12,7 @@ download,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
 contact_sales,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
 task_execution,0,0,0,0,0,0,0,0,0,0,0,6,0,0,0,0,0,0
 onboarding_setup,0,0,0,0,0,0,0,0,0,0,0,0,4,0,0,0,0,0
-troubleshooting,0,0,0,0,0,0,0,0,0,0,0,1,0,1,0,0,0,0
+troubleshooting,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0,0
 account_help,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0
 billing_help,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
 follow_up,1,0,0,0,1,0,0,0,0,0,0,1,0,0,0,0,8,0

artifacts/evaluation/latest/intent_subtype_test_report.json

@@ -1,13 +1,13 @@
 {
-  "accepted_accuracy": 0.9143,
+  "accepted_accuracy": 0.9,
   "accepted_coverage": 1.0,
-  "accuracy": 0.9143,
+  "accuracy": 0.9,
   "confusion_matrix_path": "/content/agentic-intent-classifier/artifacts/evaluation/latest/intent_subtype_test_confusion_matrix.csv",
   "count": 70,
   "dataset_path": "/content/agentic-intent-classifier/data/subtype/test.jsonl",
   "fallback_rate": 0.0,
   "head": "intent_subtype",
-  "macro_f1": 0.8855,
+  "macro_f1": 0.8531,
   "per_class_metrics": {
     "account_help": {
       "f1-score": 1.0,
@@ -15,7 +15,7 @@
       "recall": 1.0,
       "support": 2.0
     },
-    "accuracy": 0.9142857142857143,
+    "accuracy": 0.9,
     "billing_help": {
       "f1-score": 0.0,
       "precision": 0.0,
@@ -41,8 +41,8 @@
       "support": 0.0
     },
     "deal_seeking": {
-      "f1-score": 0.8,
-      "precision": 0.6666666666666666,
+      "f1-score": 0.6666666666666666,
+      "precision": 0.5,
       "recall": 1.0,
       "support": 2.0
     },
@@ -65,9 +65,9 @@
       "support": 5.0
     },
     "evaluation": {
-      "f1-score": 0.5,
-      "precision": 0.5,
-      "recall": 0.5,
+      "f1-score": 0.0,
+      "precision": 0.0,
+      "recall": 0.0,
       "support": 2.0
     },
     "follow_up": {
@@ -77,14 +77,14 @@
       "support": 11.0
     },
     "macro avg": {
-      "f1-score": 0.6887592108100274,
-      "precision": 0.7,
-      "recall": 0.6978114478114478,
+      "f1-score": 0.6635221022395855,
+      "precision": 0.6578042328042328,
+      "recall": 0.6885521885521885,
       "support": 70.0
     },
     "onboarding_setup": {
-      "f1-score": 1.0,
-      "precision": 1.0,
+      "f1-score": 0.8888888888888888,
+      "precision": 0.8,
       "recall": 1.0,
       "support": 4.0
     },
@@ -95,9 +95,9 @@
       "support": 8.0
     },
     "provider_selection": {
-      "f1-score": 0.9090909090909091,
+      "f1-score": 0.8,
       "precision": 1.0,
-      "recall": 0.8333333333333334,
+      "recall": 0.6666666666666666,
       "support": 6.0
     },
     "purchase": {
@@ -113,21 +113,21 @@
       "support": 2.0
     },
     "task_execution": {
-      "f1-score": 0.8571428571428571,
-      "precision": 0.75,
+      "f1-score": 0.9230769230769231,
+      "precision": 0.8571428571428571,
       "recall": 1.0,
       "support": 6.0
     },
     "troubleshooting": {
-      "f1-score": 0.6666666666666666,
+      "f1-score": 1.0,
       "precision": 1.0,
-      "recall": 0.5,
+      "recall": 1.0,
       "support": 2.0
     },
     "weighted avg": {
-      "f1-score": 0.9126080539458813,
-      "precision": 0.9307142857142858,
-      "recall": 0.9142857142857143,
+      "f1-score": 0.8939882610403741,
+      "precision": 0.9094217687074829,
+      "recall": 0.9,
       "support": 70.0
     }
   },

artifacts/evaluation/latest/intent_subtype_train_confusion_matrix.csv

@@ -1,19 +1,19 @@
 ,education,product_discovery,comparison,evaluation,deal_seeking,provider_selection,signup,purchase,booking,download,contact_sales,task_execution,onboarding_setup,troubleshooting,account_help,billing_help,follow_up,emotional_reflection
 education,52,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
-product_discovery,0,30,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0
+product_discovery,0,29,1,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0
 comparison,0,0,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
-evaluation,3,6,0,8,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+evaluation,4,4,0,9,0,0,0,0,0,0,0,0,0,0,0,0,0,0
 deal_seeking,0,0,0,0,10,1,0,0,0,0,0,0,0,0,0,0,0,0
 provider_selection,0,0,0,0,0,23,0,0,0,0,0,0,0,0,0,0,2,0
 signup,0,0,0,0,0,0,16,0,0,0,0,0,0,0,0,0,0,0
-purchase,0,0,0,0,0,0,0,6,0,0,0,0,0,0,0,0,0,0
-booking,0,0,0,0,0,0,2,0,3,0,0,0,0,0,0,0,0,0
+purchase,0,0,0,0,0,0,2,4,0,0,0,0,0,0,0,0,0,0
+booking,0,0,0,0,0,0,0,0,5,0,0,0,0,0,0,0,0,0
 download,0,0,0,0,0,0,0,0,0,8,0,0,0,0,0,0,0,0
 contact_sales,0,0,0,0,0,0,2,0,1,0,6,0,0,0,0,0,0,0
-task_execution,0,0,0,0,0,0,0,0,0,0,0,19,0,0,0,0,0,0
-onboarding_setup,0,0,0,0,0,0,0,0,1,1,0,0,15,0,0,0,0,0
-troubleshooting,0,0,0,0,0,0,0,0,0,0,0,0,0,10,1,0,2,0
-account_help,0,0,0,0,0,0,0,0,0,0,0,0,0,0,7,0,0,0
-billing_help,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,4,0,0
-follow_up,0,0,0,0,0,0,0,0,0,0,0,5,0,0,0,0,31,0
+task_execution,0,0,0,0,0,0,1,0,0,1,0,17,0,0,0,0,0,0
+onboarding_setup,0,0,0,0,0,0,0,0,1,0,0,0,16,0,0,0,0,0
+troubleshooting,0,0,0,0,0,0,0,0,0,0,0,0,0,10,2,0,1,0
+account_help,0,0,0,0,0,0,0,0,0,0,0,0,0,2,5,0,0,0
+billing_help,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,6,0,0
+follow_up,0,0,0,0,1,0,0,0,0,0,0,5,0,0,0,0,30,0
 emotional_reflection,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,20

artifacts/evaluation/latest/intent_subtype_train_report.json

@@ -1,36 +1,36 @@
 {
-  "accepted_accuracy": 0.9042,
+  "accepted_accuracy": 0.8978,
   "accepted_coverage": 1.0,
-  "accuracy": 0.9042,
+  "accuracy": 0.8978,
   "confusion_matrix_path": "/content/agentic-intent-classifier/artifacts/evaluation/latest/intent_subtype_train_confusion_matrix.csv",
   "count": 313,
   "dataset_path": "/content/agentic-intent-classifier/data/subtype/train.jsonl",
   "fallback_rate": 0.0,
   "head": "intent_subtype",
-  "macro_f1": 0.8789,
+  "macro_f1": 0.877,
   "per_class_metrics": {
     "account_help": {
-      "f1-score": 0.875,
-      "precision": 0.7777777777777778,
-      "recall": 1.0,
+      "f1-score": 0.7142857142857143,
+      "precision": 0.7142857142857143,
+      "recall": 0.7142857142857143,
       "support": 7.0
     },
-    "accuracy": 0.9041533546325878,
+    "accuracy": 0.8977635782747604,
     "billing_help": {
-      "f1-score": 0.8,
+      "f1-score": 1.0,
       "precision": 1.0,
-      "recall": 0.6666666666666666,
+      "recall": 1.0,
       "support": 6.0
     },
     "booking": {
-      "f1-score": 0.6,
-      "precision": 0.6,
-      "recall": 0.6,
+      "f1-score": 0.8333333333333334,
+      "precision": 0.7142857142857143,
+      "recall": 1.0,
       "support": 5.0
     },
     "comparison": {
-      "f1-score": 1.0,
-      "precision": 1.0,
+      "f1-score": 0.967741935483871,
+      "precision": 0.9375,
       "recall": 1.0,
       "support": 15.0
     },
@@ -41,8 +41,8 @@
       "support": 9.0
     },
     "deal_seeking": {
-      "f1-score": 0.9523809523809523,
-      "precision": 1.0,
+      "f1-score": 0.9090909090909091,
+      "precision": 0.9090909090909091,
       "recall": 0.9090909090909091,
       "support": 11.0
     },
@@ -53,8 +53,8 @@
       "support": 8.0
     },
     "education": {
-      "f1-score": 0.9719626168224299,
-      "precision": 0.9454545454545454,
+      "f1-score": 0.9629629629629629,
+      "precision": 0.9285714285714286,
       "recall": 1.0,
       "support": 52.0
     },
@@ -65,33 +65,33 @@
       "support": 20.0
     },
     "evaluation": {
-      "f1-score": 0.64,
+      "f1-score": 0.6923076923076923,
       "precision": 1.0,
-      "recall": 0.47058823529411764,
+      "recall": 0.5294117647058824,
       "support": 17.0
     },
     "follow_up": {
-      "f1-score": 0.8611111111111112,
-      "precision": 0.8611111111111112,
-      "recall": 0.8611111111111112,
+      "f1-score": 0.8571428571428571,
+      "precision": 0.8823529411764706,
+      "recall": 0.8333333333333334,
       "support": 36.0
     },
     "macro avg": {
-      "f1-score": 0.8788540112011829,
-      "precision": 0.9092031425364758,
-      "recall": 0.8729694019289211,
+      "f1-score": 0.8770498618135788,
+      "precision": 0.8988923431325393,
+      "recall": 0.876671278202288,
       "support": 313.0
     },
     "onboarding_setup": {
-      "f1-score": 0.9375,
+      "f1-score": 0.9696969696969697,
       "precision": 1.0,
-      "recall": 0.8823529411764706,
+      "recall": 0.9411764705882353,
       "support": 17.0
     },
     "product_discovery": {
-      "f1-score": 0.8955223880597015,
-      "precision": 0.8333333333333334,
-      "recall": 0.967741935483871,
+      "f1-score": 0.90625,
+      "precision": 0.8787878787878788,
+      "recall": 0.9354838709677419,
       "support": 31.0
     },
     "provider_selection": {
@@ -101,33 +101,33 @@
       "support": 25.0
     },
     "purchase": {
-      "f1-score": 1.0,
+      "f1-score": 0.8,
       "precision": 1.0,
-      "recall": 1.0,
+      "recall": 0.6666666666666666,
       "support": 6.0
     },
     "signup": {
-      "f1-score": 0.8888888888888888,
-      "precision": 0.8,
+      "f1-score": 0.8648648648648649,
+      "precision": 0.7619047619047619,
       "recall": 1.0,
       "support": 16.0
     },
     "task_execution": {
-      "f1-score": 0.8837209302325582,
-      "precision": 0.7916666666666666,
-      "recall": 1.0,
+      "f1-score": 0.8292682926829268,
+      "precision": 0.7727272727272727,
+      "recall": 0.8947368421052632,
       "support": 19.0
     },
     "troubleshooting": {
-      "f1-score": 0.8333333333333334,
-      "precision": 0.9090909090909091,
+      "f1-score": 0.8,
+      "precision": 0.8333333333333334,
       "recall": 0.7692307692307693,
       "support": 13.0
     },
     "weighted avg": {
-      "f1-score": 0.8995750585641842,
-      "precision": 0.9142834091715881,
-      "recall": 0.9041533546325878,
+      "f1-score": 0.894423568060199,
+      "precision": 0.9063956713482179,
+      "recall": 0.8977635782747604,
       "support": 313.0
     }
   },

artifacts/evaluation/latest/intent_subtype_val_confusion_matrix.csv

@@ -1,6 +1,6 @@
 ,education,product_discovery,comparison,evaluation,deal_seeking,provider_selection,signup,purchase,booking,download,contact_sales,task_execution,onboarding_setup,troubleshooting,account_help,billing_help,follow_up,emotional_reflection
 education,14,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
-product_discovery,0,10,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0
+product_discovery,0,9,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
 comparison,0,0,2,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0
 evaluation,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0
 deal_seeking,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0
@@ -14,6 +14,6 @@ task_execution,0,0,0,0,0,0,0,0,0,0,0,8,0,0,0,0,0,0
 onboarding_setup,0,1,0,0,0,0,0,0,0,0,0,0,4,0,0,0,0,0
 troubleshooting,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0
 account_help,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0
-billing_help,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0
+billing_help,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0
 follow_up,0,0,0,0,1,0,0,0,0,0,0,1,0,0,0,0,9,0
 emotional_reflection,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,5

artifacts/evaluation/latest/intent_subtype_val_report.json

@@ -1,13 +1,13 @@
 {
-  "accepted_accuracy": 0.8734,
+  "accepted_accuracy": 0.8608,
   "accepted_coverage": 0.9875,
-  "accuracy": 0.875,
+  "accuracy": 0.85,
   "confusion_matrix_path": "/content/agentic-intent-classifier/artifacts/evaluation/latest/intent_subtype_val_confusion_matrix.csv",
   "count": 80,
   "dataset_path": "/content/agentic-intent-classifier/data/subtype/val.jsonl",
   "fallback_rate": 0.0125,
   "head": "intent_subtype",
-  "macro_f1": 0.7429,
+  "macro_f1": 0.6722,
   "per_class_metrics": {
     "account_help": {
       "f1-score": 0.5,
@@ -15,11 +15,11 @@
       "recall": 0.5,
       "support": 2.0
     },
-    "accuracy": 0.875,
+    "accuracy": 0.85,
     "billing_help": {
-      "f1-score": 1.0,
-      "precision": 1.0,
-      "recall": 1.0,
+      "f1-score": 0.0,
+      "precision": 0.0,
+      "recall": 0.0,
       "support": 1.0
     },
     "booking": {
@@ -29,8 +29,8 @@
       "support": 3.0
     },
     "comparison": {
-      "f1-score": 0.6666666666666666,
-      "precision": 1.0,
+      "f1-score": 0.5,
+      "precision": 0.5,
       "recall": 0.5,
       "support": 4.0
     },
@@ -41,8 +41,8 @@
       "support": 0.0
     },
     "deal_seeking": {
-      "f1-score": 0.6666666666666666,
-      "precision": 0.5,
+      "f1-score": 0.8,
+      "precision": 0.6666666666666666,
       "recall": 1.0,
       "support": 2.0
     },
@@ -77,9 +77,9 @@
       "support": 11.0
     },
     "macro avg": {
-      "f1-score": 0.660381593714927,
-      "precision": 0.6597643097643098,
-      "recall": 0.695959595959596,
+      "f1-score": 0.5974890931031281,
+      "precision": 0.5811447811447812,
+      "recall": 0.6353535353535353,
       "support": 80.0
     },
     "onboarding_setup": {
@@ -89,9 +89,9 @@
       "support": 5.0
     },
     "product_discovery": {
-      "f1-score": 0.9090909090909091,
-      "precision": 0.9090909090909091,
-      "recall": 0.9090909090909091,
+      "f1-score": 0.8571428571428571,
+      "precision": 0.9,
+      "recall": 0.8181818181818182,
       "support": 11.0
     },
     "provider_selection": {
@@ -113,8 +113,8 @@
       "support": 2.0
     },
     "task_execution": {
-      "f1-score": 0.8888888888888888,
-      "precision": 0.8,
+      "f1-score": 0.8421052631578947,
+      "precision": 0.7272727272727273,
       "recall": 1.0,
       "support": 8.0
     },
@@ -125,9 +125,9 @@
       "support": 1.0
     },
     "weighted avg": {
-      "f1-score": 0.8673611111111112,
-      "precision": 0.8841666666666667,
-      "recall": 0.875,
+      "f1-score": 0.8380398913951546,
+      "precision": 0.8423106060606059,
+      "recall": 0.85,
       "support": 80.0
     }
   },

artifacts/evaluation/latest/intent_type_hard_cases_report.json

@@ -1,11 +1,11 @@
 {
   "accepted_accuracy": 1.0,
-  "accepted_coverage": 1.0,
+  "accepted_coverage": 0.9836,
   "accuracy": 1.0,
   "confusion_matrix_path": "/content/agentic-intent-classifier/artifacts/evaluation/latest/intent_type_hard_cases_confusion_matrix.csv",
   "count": 61,
   "dataset_path": "/content/agentic-intent-classifier/data/hard_cases.jsonl",
-  "fallback_rate": 0.0,
+  "fallback_rate": 0.0164,
   "head": "intent_type",
   "macro_f1": 1.0,
   "per_class_metrics": {

training/run_full_training_pipeline.py

@@ -3,17 +3,27 @@ from __future__ import annotations
 import argparse
 import subprocess
 import sys
+import time
+from datetime import datetime, timezone
 from pathlib import Path
 
 BASE_DIR = Path(__file__).resolve().parent.parent
 
 
 def run_step(args: list[str]) -> None:
-    print(f"\n==> Running: {' '.join(args)}")
+    cmd = " ".join(args)
+    started_at = time.perf_counter()
+    started_wall = datetime.now(timezone.utc).isoformat()
+    print(f"\n==> Running: {cmd}\n    start: {started_wall}")
     subprocess.run(args, cwd=BASE_DIR, check=True)
+    elapsed_s = time.perf_counter() - started_at
+    ended_wall = datetime.now(timezone.utc).isoformat()
+    print(f"    end:   {ended_wall}\n    took:  {elapsed_s:.2f}s")
 
 
 def main() -> None:
+    pipeline_start = time.perf_counter()
+    pipeline_start_wall = datetime.now(timezone.utc).isoformat()
     parser = argparse.ArgumentParser(
         description=(
             "Run the full multi-head training pipeline: multitask intent, IAB classifier, calibration for all heads, "
@@ -106,6 +116,15 @@ def main() -> None:
     if args.smoke_test:
         run_step([python, "combined_inference.py", args.smoke_test_query])
 
+    pipeline_elapsed_s = time.perf_counter() - pipeline_start
+    pipeline_end_wall = datetime.now(timezone.utc).isoformat()
+    print(
+        f"\n==> Pipeline complete\n"
+        f"    start: {pipeline_start_wall}\n"
+        f"    end:   {pipeline_end_wall}\n"
+        f"    total: {pipeline_elapsed_s:.2f}s"
+    )
+
 
 if __name__ == "__main__":
     main()

training/upload_to_hf.py

@@ -11,6 +11,8 @@ from __future__ import annotations
 import argparse
 import os
 import sys
+import time
+from datetime import datetime, timezone
 from pathlib import Path
 
 
@@ -46,6 +48,16 @@ def _parse_args() -> argparse.Namespace:
         action="store_true",
         help="Upload artifacts/calibration directory.",
     )
+    parser.add_argument(
+        "--include-hf-readme",
+        action="store_true",
+        help="Upload a Hugging Face model card file as README.md in the Hub repo root.",
+    )
+    parser.add_argument(
+        "--hf-readme-path",
+        default="HF_MODEL_CARD.md",
+        help="Local path to the HF model card markdown to upload as README.md (relative to repo root).",
+    )
     parser.add_argument(
         "--multitask-dir",
         default="multitask_intent_model_output",
@@ -70,6 +82,8 @@ def _parse_args() -> argparse.Namespace:
 
 
 def main() -> int:
+    started_at = time.perf_counter()
+    started_wall = datetime.now(timezone.utc).isoformat()
     args = _parse_args()
     if not args.token:
         print("Missing HF token. Provide --token or set env HF_TOKEN.", file=sys.stderr)
@@ -80,6 +94,7 @@ def main() -> int:
     multitask_dir = (repo_root / args.multitask_dir).resolve()
     iab_dir = (repo_root / args.iab_dir).resolve()
     calibration_dir = (repo_root / args.calibration_dir).resolve()
+    hf_readme_path = (repo_root / args.hf_readme_path).resolve()
 
     to_upload: list[tuple[str, Path]] = []
     if args.include_multitask:
@@ -88,6 +103,8 @@ def main() -> int:
         to_upload.append(("iab_classifier_model_output", iab_dir))
     if args.include_calibration:
         to_upload.append(("artifacts/calibration", calibration_dir))
+    if args.include_hf_readme:
+        to_upload.append(("README.md", hf_readme_path))
 
     if not to_upload:
         print("Nothing to upload. Pass --include-multitask, --include-iab, and/or --include-calibration.", file=sys.stderr)
@@ -110,6 +127,20 @@ def main() -> int:
         if args.dry_run:
             print(f"[DRY] Would upload {local_dir} -> {args.repo_id}:{repo_path}")
             continue
+        # Upload single README.md file (Hub model card) vs directories
+        if repo_path == "README.md":
+            step_start = time.perf_counter()
+            print(f"[UPLOAD] {local_dir} -> {args.repo_id}:README.md")
+            api.upload_file(
+                repo_id=args.repo_id,
+                repo_type="model",
+                path_or_fileobj=str(local_dir),
+                path_in_repo="README.md",
+            )
+            print(f"[DONE ] README.md took {(time.perf_counter() - step_start):.2f}s")
+            continue
+
+        step_start = time.perf_counter()
         print(f"[UPLOAD] {local_dir} -> {args.repo_id}:{repo_path}")
         api.upload_folder(
             repo_id=args.repo_id,
@@ -117,8 +148,11 @@ def main() -> int:
             folder_path=str(local_dir),
             path_in_repo=repo_path,
         )
+        print(f"[DONE ] {repo_path} took {(time.perf_counter() - step_start):.2f}s")
 
-    print("Upload complete.")
+    ended_wall = datetime.now(timezone.utc).isoformat()
+    elapsed_s = time.perf_counter() - started_at
+    print(f"Upload complete.\nstart: {started_wall}\nend:   {ended_wall}\ntotal: {elapsed_s:.2f}s")
     return 0