Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+multitask_intent_model_output/multitask_intent.onnx.data filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,9 @@
+__pycache__/
+*.pyc
+.DS_Store
+model_output/
+decision_phase_model_output/
+subtype_model_output/
+iab_model_output/
+*/model.safetensors
+iab_hierarchy_model_output/

COLAB_SETUP.md

@@ -0,0 +1,105 @@
+# Google Colab setup — agentic-intent-classifier
+
+## 1. Runtime
+
+**Runtime → Change runtime type → GPU** (T4/L4/A100). Then verify:
+
+```python
+import torch
+print(torch.cuda.is_available(), torch.cuda.get_device_name(0) if torch.cuda.is_available() else "CPU")
+```
+
+## 2. Get the code
+
+**Option A — clone (if the repo is public or you use a token):**
+
+```python
+!git clone <YOUR_REPO_URL> protocol
+%cd protocol/agentic-intent-classifier
+```
+
+**Option B — upload:** Zip `agentic-intent-classifier/` (including `data/`, `examples/`, taxonomy TSV under `data/iab-content/` if you use IAB), unzip in Colab, then:
+
+```python
+%cd /content/agentic-intent-classifier
+```
+
+## 3. Install dependencies
+
+```python
+%pip install -q -r requirements.txt
+```
+
+If `requirements.txt` is missing, install manually:
+
+```python
+%pip install -q torch transformers datasets accelerate scikit-learn numpy pandas safetensors
+```
+
+## 4. Optional: quieter TensorFlow / XLA logs
+
+Run **before** importing `combined_inference` or anything that pulls TensorFlow:
+
+```python
+import os
+os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3"
+os.environ["ABSL_MIN_LOG_LEVEL"] = "3"
+```
+
+Harmless CUDA “already registered” lines may still appear; they do not mean training failed.
+
+## 5. Optional: persist artifacts on Google Drive
+
+```python
+from google.colab import drive
+drive.mount("/content/drive")
+```
+
+Copy outputs to Drive after training, or symlink `multitask_intent_model_output` / `artifacts` / `iab_classifier_model_output` to a Drive folder.
+
+## 6. Full pipeline (train + IAB + calibrate + verify + ONNX + smoke test)
+
+From `agentic-intent-classifier/`:
+
+```python
+!python training/run_full_training_pipeline.py --skip-full-eval --complete
+```
+
+- `--skip-full-eval` avoids the heaviest eval pass (OOM on small RAM); remove when you have headroom.
+- `--complete` = export multitask ONNX + `pipeline_verify.py` + one `combined_inference` query.
+
+**Artifacts-only check (after copying weights in):**
+
+```python
+!python training/pipeline_verify.py
+```
+
+**Single query:**
+
+```python
+!python combined_inference.py "Which laptop should I buy for college?"
+```
+
+Check `meta.iab_mapping_is_placeholder`: `false` only if IAB was trained and calibration exists.
+
+## 7. Minimal path (intent multitask + calibrate only)
+
+If you only run multitask training and calibration in Colab (no full orchestrator):
+
+```text
+python training/train_multitask_intent.py
+python training/calibrate_confidence.py --head intent_type
+python training/calibrate_confidence.py --head intent_subtype
+python training/calibrate_confidence.py --head decision_phase
+```
+
+Production “complete” stack still needs **IAB train + IAB calibrate** (see `run_full_training_pipeline.py`).
+
+## 8. Working directory
+
+Always `cd` to the folder that contains `config.py`, `training/`, and `data/`:
+
+```python
+import os
+assert os.path.isfile("config.py"), "Wrong directory — cd into agentic-intent-classifier"
+```

README.md

@@ -1,3 +1,421 @@
----
-license: apache-2.0
----
+# Agentic Intent Classifier
+
+`agentic-intent-classifier` is a multi-head query classification stack for conversational traffic.
+
+It currently produces:
+
+- `intent.type`
+- `intent.subtype`
+- `intent.decision_phase`
+- `iab_content`
+- calibrated confidence per head
+- combined fallback / policy / opportunity decisions
+
+The repo is beyond the original v0.1 baseline. It now includes:
+
+- shared config and label ownership
+- reusable model runtime
+- calibrated confidence and threshold gating
+- combined inference with fallback/policy logic
+- request/response validation in the demo API
+- repeatable evaluation and regression suites
+- full-TSV IAB taxonomy retrieval support through tier4
+- a local embedding index for taxonomy-node retrieval over IAB content paths
+- a separate synthetic full-intent-taxonomy augmentation dataset for non-IAB heads
+- a dedicated intent-type difficulty dataset and held-out benchmark with `easy`, `medium`, and `hard` cases
+- a dedicated decision-phase difficulty dataset and held-out benchmark with `easy`, `medium`, and `hard` cases
+
+Generated model weights are intentionally not committed.
+
+## Current Taxonomy
+
+### `intent.type`
+
+- `informational`
+- `exploratory`
+- `commercial`
+- `transactional`
+- `support`
+- `personal_reflection`
+- `creative_generation`
+- `chit_chat`
+- `ambiguous`
+- `prohibited`
+
+### `intent.decision_phase`
+
+- `awareness`
+- `research`
+- `consideration`
+- `decision`
+- `action`
+- `post_purchase`
+- `support`
+
+### `intent.subtype`
+
+- `education`
+- `product_discovery`
+- `comparison`
+- `evaluation`
+- `deal_seeking`
+- `provider_selection`
+- `signup`
+- `purchase`
+- `booking`
+- `download`
+- `contact_sales`
+- `task_execution`
+- `onboarding_setup`
+- `troubleshooting`
+- `account_help`
+- `billing_help`
+- `follow_up`
+- `emotional_reflection`
+
+### `iab_content`
+
+- candidates are derived from every row in [data/iab-content/Content Taxonomy 3.0.tsv](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/iab-content/Content%20Taxonomy%203.0.tsv)
+- retrieval output supports `tier1`, `tier2`, `tier3`, and optional `tier4`
+
+## What The System Does
+
+- runs three classifier heads:
+  - `intent_type`
+  - `intent_subtype`
+  - `decision_phase`
+- resolves `iab_content` through a local embedding index over taxonomy nodes plus generic label/path reranking
+- applies calibration artifacts when present
+- computes `commercial_score`
+- applies fallback when confidence is too weak or policy-safe blocking is required
+- emits a schema-validated combined envelope
+
+## What The System Does Not Do
+
+- it is not a multi-turn memory system
+- it is not a production-optimized low-latency serving path
+- it is not yet trained on large real-traffic human-labeled intent data
+- combined decision logic is still heuristic, even though it is materially stronger than the original baseline
+
+## Project Layout
+
+- [config.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/config.py): labels, thresholds, artifact paths, model paths
+- [model_runtime.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/model_runtime.py): shared calibrated inference runtime
+- [combined_inference.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/combined_inference.py): composed system response
+- [inference_intent_type.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/inference_intent_type.py): direct `intent_type` inference entrypoint
+- [inference_iab_classifier.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/inference_iab_classifier.py): direct supervised `iab_content` inference entrypoint
+- [schemas.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/schemas.py): request/response validation
+- [demo_api.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/demo_api.py): local validated API
+- [iab_taxonomy.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/iab_taxonomy.py): full taxonomy parser/index
+- [iab_classifier.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/iab_classifier.py): supervised IAB runtime with taxonomy-aware parent fallback
+- [iab_retrieval.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/iab_retrieval.py): optional shadow retrieval baseline
+- [training/build_full_intent_taxonomy_dataset.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/build_full_intent_taxonomy_dataset.py): separate synthetic intent augmentation dataset
+- [training/build_intent_type_difficulty_dataset.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/build_intent_type_difficulty_dataset.py): extra `intent_type` augmentation plus held-out difficulty benchmark
+- [training/build_decision_phase_difficulty_dataset.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/build_decision_phase_difficulty_dataset.py): extra `decision_phase` augmentation plus held-out difficulty benchmark
+- [training/build_subtype_difficulty_dataset.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/build_subtype_difficulty_dataset.py): extra `intent_subtype` augmentation plus held-out difficulty benchmark
+- [training/build_subtype_dataset.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/build_subtype_dataset.py): subtype dataset generation from existing corpora
+- [training/train_iab.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/train_iab.py): train the supervised IAB classifier head
+- [training/build_iab_taxonomy_embeddings.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/build_iab_taxonomy_embeddings.py): build local IAB node embedding artifacts
+- [training/run_full_training_pipeline.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/run_full_training_pipeline.py): full multi-head training/calibration/eval pipeline
+- [evaluation/run_evaluation.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/evaluation/run_evaluation.py): repeatable benchmark runner
+- [evaluation/run_regression_suite.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/evaluation/run_regression_suite.py): known-failure regression runner
+- [evaluation/run_iab_mapping_suite.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/evaluation/run_iab_mapping_suite.py): IAB behavior-lock regression runner
+- [evaluation/run_iab_quality_suite.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/evaluation/run_iab_quality_suite.py): curated IAB quality-target runner
+- [known_limitations.md](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/known_limitations.md): current gaps and caveats
+
+## Setup
+
+```bash
+python3 -m venv .venv
+source .venv/bin/activate
+pip install -r agentic-intent-classifier/requirements.txt
+```
+
+## Inference
+
+Run one query locally:
+
+```bash
+cd agentic-intent-classifier
+python3 training/train_iab.py
+python3 training/calibrate_confidence.py --head iab_content
+python3 combined_inference.py "Which CRM should I buy for a 3-person startup?"
+```
+
+Run only the `intent_type` head:
+
+```bash
+cd agentic-intent-classifier
+python3 inference_intent_type.py "best shoes under 100"
+```
+
+Run the demo API:
+
+```bash
+cd agentic-intent-classifier
+python3 demo_api.py
+```
+
+Example request:
+
+```bash
+curl -sS -X POST http://127.0.0.1:8008/classify \
+  -H 'Content-Type: application/json' \
+  -d '{"text":"I cannot log into my account"}'
+```
+
+Infra endpoints:
+
+```bash
+curl -sS http://127.0.0.1:8008/health
+curl -sS http://127.0.0.1:8008/version
+```
+
+Train only the IAB classifier head:
+
+```bash
+cd agentic-intent-classifier
+python3 training/train_iab.py
+python3 training/calibrate_confidence.py --head iab_content
+```
+
+The online `iab_content` path now uses the compact supervised classifier. Retrieval is still available as an optional shadow baseline.
+
+Build the optional retrieval shadow index:
+
+```bash
+cd agentic-intent-classifier
+python3 training/build_iab_taxonomy_embeddings.py
+```
+
+By default the shadow retrieval path uses `Alibaba-NLP/gte-Qwen2-1.5B-instruct`. The retrieval runtime applies the model's query-side instruction format and last-token pooling, matching the Hugging Face usage guidance. If you want to point retrieval at a different embedding model, set `IAB_RETRIEVAL_MODEL_NAME_OVERRIDE` before building the index.
+
+Open-source users can swap in their own embedding model, but the contract is:
+
+- query embeddings and taxonomy-node embeddings must be produced by the same model and model revision
+- after changing models, you must rebuild `artifacts/iab/taxonomy_embeddings.pt`
+- the repository only tests and supports the default model path out of the box
+- not every Hugging Face embedding model is drop-in compatible with this runtime; some require custom pooling, query instructions, or `trust_remote_code`
+
+Example override:
+
+```bash
+cd agentic-intent-classifier
+export IAB_RETRIEVAL_MODEL_NAME_OVERRIDE=mixedbread-ai/mxbai-embed-large-v1
+python3 training/build_iab_taxonomy_embeddings.py
+```
+
+This writes:
+
+- `artifacts/iab/taxonomy_nodes.json`
+- `artifacts/iab/taxonomy_embeddings.pt`
+
+## Training
+
+### Full local pipeline
+
+```bash
+cd agentic-intent-classifier
+python3 training/run_full_training_pipeline.py
+```
+
+This pipeline now does:
+
+1. build separate full-intent-taxonomy augmentation data
+2. build separate `intent_type` difficulty augmentation + benchmark
+3. train `intent_type`
+4. build subtype corpus
+5. build separate `intent_subtype` difficulty augmentation + benchmark
+6. train `intent_subtype`
+7. build separate `decision_phase` difficulty augmentation + benchmark
+8. train `decision_phase`
+9. train `iab_content`
+10. calibrate all classifier heads, including `iab_content`
+11. run regression/evaluation unless `--skip-full-eval` is used
+
+### Build datasets individually
+
+Separate full-intent augmentation:
+
+```bash
+cd agentic-intent-classifier
+python3 training/build_full_intent_taxonomy_dataset.py
+```
+
+Intent-type difficulty augmentation and benchmark:
+
+```bash
+cd agentic-intent-classifier
+python3 training/build_intent_type_difficulty_dataset.py
+```
+
+Decision-phase difficulty augmentation and benchmark:
+
+```bash
+cd agentic-intent-classifier
+python3 training/build_decision_phase_difficulty_dataset.py
+```
+
+Subtype difficulty augmentation and benchmark:
+
+```bash
+cd agentic-intent-classifier
+python3 training/build_subtype_difficulty_dataset.py
+```
+
+Subtype dataset:
+
+```bash
+cd agentic-intent-classifier
+python3 training/build_subtype_dataset.py
+```
+
+IAB embedding index:
+
+```bash
+cd agentic-intent-classifier
+python3 training/build_iab_taxonomy_embeddings.py
+```
+
+### Train heads individually
+
+```bash
+cd agentic-intent-classifier
+python3 training/train.py
+python3 training/train_subtype.py
+python3 training/train_decision_phase.py
+```
+
+### Calibration
+
+```bash
+cd agentic-intent-classifier
+python3 training/calibrate_confidence.py --head intent_type
+python3 training/calibrate_confidence.py --head intent_subtype
+python3 training/calibrate_confidence.py --head decision_phase
+```
+
+## Evaluation
+
+Full evaluation:
+
+```bash
+cd agentic-intent-classifier
+python3 evaluation/run_evaluation.py
+```
+
+Known-failure regression:
+
+```bash
+cd agentic-intent-classifier
+python3 evaluation/run_regression_suite.py
+```
+
+IAB behavior-lock regression:
+
+```bash
+cd agentic-intent-classifier
+python3 evaluation/run_iab_mapping_suite.py
+```
+
+IAB quality-target evaluation:
+
+```bash
+cd agentic-intent-classifier
+python3 evaluation/run_iab_quality_suite.py
+```
+
+Threshold sweeps:
+
+```bash
+cd agentic-intent-classifier
+python3 evaluation/sweep_intent_threshold.py
+```
+
+Artifacts are written to:
+
+- `artifacts/calibration/`
+- `artifacts/evaluation/latest/`
+
+## Google Colab
+
+Use Colab for the full retraining pass if local memory is limited.
+
+Clone once:
+
+```bash
+%cd /content
+!git clone https://github.com/GouniManikumar12/agentic-intent-classifier.git
+%cd /content/agentic-intent-classifier
+```
+
+If the repo is already cloned and you want the latest code, pull manually:
+
+```bash
+!git pull origin main
+```
+
+Full pipeline:
+
+```bash
+!python training/run_full_training_pipeline.py
+```
+
+If full evaluation is too heavy for the current Colab runtime:
+
+```bash
+!python training/run_full_training_pipeline.py \
+  --iab-embedding-batch-size 32 \
+  --skip-full-eval
+```
+
+Then run eval separately after training:
+
+```bash
+!python evaluation/run_regression_suite.py
+!python evaluation/run_iab_mapping_suite.py
+!python evaluation/run_iab_quality_suite.py
+!python evaluation/run_evaluation.py
+```
+
+## Current Saved Metrics
+
+Generate fresh metrics with:
+
+```bash
+cd agentic-intent-classifier
+python3 evaluation/run_evaluation.py
+```
+
+Do not treat any checked-in summary as canonical unless it was regenerated after the current code and artifacts were built. The IAB path is now retrieval-based, so older saved reports from the deleted hierarchy stack are not meaningful.
+
+## Latency Note
+
+`combined_inference.py` is a debugging/offline path, not a production latency path.
+
+Current production truth:
+
+- per-request CLI execution is not a sub-50ms architecture
+- production serving should use a long-lived API process with preloaded models
+- if sub-50ms becomes a hard requirement, the serving path will need:
+  - persistent loaded models
+  - runtime optimization
+  - likely fewer model passes or a shared multi-head model
+
+## Current Status
+
+Current repo status:
+
+- full 10-class `intent.type` taxonomy is wired
+- subtype and phase heads are present
+- difficulty benchmarks are wired for `intent_type`, `intent_subtype`, and `decision_phase`
+- full-TSV IAB taxonomy retrieval is wired through tier4
+- separate full-intent augmentation dataset is in place
+- evaluation/runtime memory handling is improved for large IAB splits
+
+The main remaining gap is not basic infrastructure anymore. It is improving real-world robustness, especially for:
+
+- `decision_phase`
+- `intent_subtype`
+- confidence quality on borderline commercial queries
+- real-traffic supervision beyond synthetic data

artifacts/calibration/decision_phase.json

@@ -0,0 +1,32 @@
+{
+  "calibrated": true,
+  "confidence_threshold": 0.22,
+  "generated_at": "2026-03-25T05:10:14.092098+00:00",
+  "head": "decision_phase",
+  "metrics": {
+    "calibrated_accuracy": 0.8621,
+    "calibrated_expected_calibration_error": 0.0877,
+    "calibrated_negative_log_likelihood": 0.5315,
+    "mean_calibrated_confidence": 0.866,
+    "mean_raw_confidence": 0.8684,
+    "raw_accuracy": 0.8621,
+    "raw_expected_calibration_error": 0.087,
+    "raw_negative_log_likelihood": 0.5317
+  },
+  "minimum_threshold_floor": 0.22,
+  "optimized_temperature_candidate": 1.008347,
+  "selected_threshold_before_floor": {
+    "accepted_accuracy": 0.8621,
+    "coverage": 1.0,
+    "threshold": 0.0
+  },
+  "selection_split": "val",
+  "selection_target_precision": 0.75,
+  "temperature": 1.008347,
+  "temperature_scaling_applied": true,
+  "threshold_summary": {
+    "accepted_accuracy": 0.8621,
+    "coverage": 1.0,
+    "threshold": 0.22
+  }
+}

artifacts/calibration/iab_content.json

@@ -0,0 +1,32 @@
+{
+  "calibrated": true,
+  "confidence_threshold": 0.12,
+  "generated_at": "2026-03-25T05:12:02.550364+00:00",
+  "head": "iab_content",
+  "metrics": {
+    "calibrated_accuracy": 0.9442,
+    "calibrated_expected_calibration_error": 0.2773,
+    "calibrated_negative_log_likelihood": 0.5519,
+    "mean_calibrated_confidence": 0.6669,
+    "mean_raw_confidence": 0.2286,
+    "raw_accuracy": 0.9442,
+    "raw_expected_calibration_error": 0.7157,
+    "raw_negative_log_likelihood": 1.6567
+  },
+  "minimum_threshold_floor": 0.12,
+  "optimized_temperature_candidate": 0.607335,
+  "selected_threshold_before_floor": {
+    "accepted_accuracy": 0.9442,
+    "coverage": 1.0,
+    "threshold": 0.0
+  },
+  "selection_split": "val",
+  "selection_target_precision": 0.7,
+  "temperature": 0.607335,
+  "temperature_scaling_applied": true,
+  "threshold_summary": {
+    "accepted_accuracy": 0.9478,
+    "coverage": 0.9915,
+    "threshold": 0.12
+  }
+}

artifacts/calibration/intent_subtype.json

@@ -0,0 +1,32 @@
+{
+  "calibrated": true,
+  "confidence_threshold": 0.25,
+  "generated_at": "2026-03-25T05:09:58.809351+00:00",
+  "head": "intent_subtype",
+  "metrics": {
+    "calibrated_accuracy": 0.875,
+    "calibrated_expected_calibration_error": 0.0778,
+    "calibrated_negative_log_likelihood": 0.4165,
+    "mean_calibrated_confidence": 0.8225,
+    "mean_raw_confidence": 0.7521,
+    "raw_accuracy": 0.875,
+    "raw_expected_calibration_error": 0.1475,
+    "raw_negative_log_likelihood": 0.4843
+  },
+  "minimum_threshold_floor": 0.25,
+  "optimized_temperature_candidate": 0.834211,
+  "selected_threshold_before_floor": {
+    "accepted_accuracy": 0.875,
+    "coverage": 1.0,
+    "threshold": 0.0
+  },
+  "selection_split": "val",
+  "selection_target_precision": 0.75,
+  "temperature": 0.834211,
+  "temperature_scaling_applied": true,
+  "threshold_summary": {
+    "accepted_accuracy": 0.875,
+    "coverage": 1.0,
+    "threshold": 0.25
+  }
+}

artifacts/calibration/intent_type.json

@@ -0,0 +1,32 @@
+{
+  "calibrated": true,
+  "confidence_threshold": 0.4,
+  "generated_at": "2026-03-25T05:09:42.900721+00:00",
+  "head": "intent_type",
+  "metrics": {
+    "calibrated_accuracy": 0.9362,
+    "calibrated_expected_calibration_error": 0.0424,
+    "calibrated_negative_log_likelihood": 0.3117,
+    "mean_calibrated_confidence": 0.8993,
+    "mean_raw_confidence": 0.8741,
+    "raw_accuracy": 0.9362,
+    "raw_expected_calibration_error": 0.0788,
+    "raw_negative_log_likelihood": 0.3262
+  },
+  "minimum_threshold_floor": 0.4,
+  "optimized_temperature_candidate": 0.916196,
+  "selected_threshold_before_floor": {
+    "accepted_accuracy": 0.9362,
+    "coverage": 1.0,
+    "threshold": 0.0
+  },
+  "selection_split": "val",
+  "selection_target_precision": 0.8,
+  "temperature": 0.916196,
+  "temperature_scaling_applied": true,
+  "threshold_summary": {
+    "accepted_accuracy": 0.9362,
+    "coverage": 1.0,
+    "threshold": 0.4
+  }
+}

artifacts/evaluation/intent_threshold_sweep.json

@@ -0,0 +1,1294 @@
+{
+  "applied_threshold": 0.4,
+  "recommended_threshold": 0.4,
+  "results": [
+    {
+      "combined": {
+        "bad_allow_rate_on_safe_prompts": 0.0,
+        "benchmark_fallback_rate": 0.6,
+        "benchmark_intent_only_fallback_rate": 0.0,
+        "benchmark_phase_only_fallback_rate": 0.5333,
+        "fallback_responsibility": {
+          "both": 0,
+          "fallback_rate": 0.6,
+          "intent_only": 0,
+          "intent_share_of_threshold_fallbacks": 0.0,
+          "phase_only": 5,
+          "phase_share_of_threshold_fallbacks": 1.0,
+          "policy_safe": 1
+        },
+        "false_fallback_rate_on_obvious_prompts": 0.3333,
+        "obvious_prompt_count": 6,
+        "safe_prompt_count": 4,
+        "suite_outputs": [
+          {
+            "decision_phase": "awareness",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "What is CRM software?",
+            "intent_confidence": 0.8488,
+            "intent_type": "informational",
+            "phase_confidence": 0.4134
+          },
+          {
+            "decision_phase": "research",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "Help me understand CRM basics",
+            "intent_confidence": 0.4226,
+            "intent_type": "informational",
+            "phase_confidence": 0.2602
+          },
+          {
+            "decision_phase": "consideration",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "HubSpot vs Zoho for a small team",
+            "intent_confidence": 0.6819,
+            "intent_type": "commercial",
+            "phase_confidence": 0.3331
+          },
+          {
+            "decision_phase": "decision",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "Which CRM should I buy for a 3-person startup?",
+            "intent_confidence": 0.7691,
+            "intent_type": "informational",
+            "phase_confidence": 0.2879
+          },
+          {
+            "decision_phase": "action",
+            "expected_outcome": "pass",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "Start my free trial",
+            "intent_confidence": 0.5429,
+            "intent_type": "transactional",
+            "phase_confidence": 0.1875
+          },
+          {
+            "decision_phase": "action",
+            "expected_outcome": "pass",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "Book a table for 2 tonight",
+            "intent_confidence": 0.5284,
+            "intent_type": "transactional",
+            "phase_confidence": 0.1716
+          },
+          {
+            "decision_phase": "support",
+            "expected_outcome": "fallback",
+            "failed_components": [],
+            "fallback_applied": true,
+            "input": "I cannot log into my account",
+            "intent_confidence": 0.3314,
+            "intent_type": "personal_reflection",
+            "phase_confidence": 0.275
+          },
+          {
+            "decision_phase": "action",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "go deeper",
+            "intent_confidence": 0.6527,
+            "intent_type": "ambiguous",
+            "phase_confidence": 0.1638
+          },
+          {
+            "decision_phase": "research",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "say more about that",
+            "intent_confidence": 0.6538,
+            "intent_type": "ambiguous",
+            "phase_confidence": 0.1674
+          },
+          {
+            "decision_phase": "awareness",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "what do you mean by that",
+            "intent_confidence": 0.4595,
+            "intent_type": "ambiguous",
+            "phase_confidence": 0.1718
+          }
+        ],
+        "suite_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/examples/intent_threshold_sweep_suite.json"
+      },
+      "head": {
+        "ambiguous_bad_allow_rate": 0.1842,
+        "ambiguous_prompt_count": 38,
+        "obvious_false_fallback_rate": 0.0104,
+        "obvious_prompt_count": 96,
+        "safe_predicate_rate": 0.2917
+      },
+      "threshold": 0.0
+    },
+    {
+      "combined": {
+        "bad_allow_rate_on_safe_prompts": 0.0,
+        "benchmark_fallback_rate": 0.6,
+        "benchmark_intent_only_fallback_rate": 0.0,
+        "benchmark_phase_only_fallback_rate": 0.5333,
+        "fallback_responsibility": {
+          "both": 0,
+          "fallback_rate": 0.6,
+          "intent_only": 0,
+          "intent_share_of_threshold_fallbacks": 0.0,
+          "phase_only": 5,
+          "phase_share_of_threshold_fallbacks": 1.0,
+          "policy_safe": 1
+        },
+        "false_fallback_rate_on_obvious_prompts": 0.3333,
+        "obvious_prompt_count": 6,
+        "safe_prompt_count": 4,
+        "suite_outputs": [
+          {
+            "decision_phase": "awareness",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "What is CRM software?",
+            "intent_confidence": 0.8488,
+            "intent_type": "informational",
+            "phase_confidence": 0.4134
+          },
+          {
+            "decision_phase": "research",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "Help me understand CRM basics",
+            "intent_confidence": 0.4226,
+            "intent_type": "informational",
+            "phase_confidence": 0.2602
+          },
+          {
+            "decision_phase": "consideration",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "HubSpot vs Zoho for a small team",
+            "intent_confidence": 0.6819,
+            "intent_type": "commercial",
+            "phase_confidence": 0.3331
+          },
+          {
+            "decision_phase": "decision",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "Which CRM should I buy for a 3-person startup?",
+            "intent_confidence": 0.7691,
+            "intent_type": "informational",
+            "phase_confidence": 0.2879
+          },
+          {
+            "decision_phase": "action",
+            "expected_outcome": "pass",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "Start my free trial",
+            "intent_confidence": 0.5429,
+            "intent_type": "transactional",
+            "phase_confidence": 0.1875
+          },
+          {
+            "decision_phase": "action",
+            "expected_outcome": "pass",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "Book a table for 2 tonight",
+            "intent_confidence": 0.5284,
+            "intent_type": "transactional",
+            "phase_confidence": 0.1716
+          },
+          {
+            "decision_phase": "support",
+            "expected_outcome": "fallback",
+            "failed_components": [],
+            "fallback_applied": true,
+            "input": "I cannot log into my account",
+            "intent_confidence": 0.3314,
+            "intent_type": "personal_reflection",
+            "phase_confidence": 0.275
+          },
+          {
+            "decision_phase": "action",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "go deeper",
+            "intent_confidence": 0.6527,
+            "intent_type": "ambiguous",
+            "phase_confidence": 0.1638
+          },
+          {
+            "decision_phase": "research",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "say more about that",
+            "intent_confidence": 0.6538,
+            "intent_type": "ambiguous",
+            "phase_confidence": 0.1674
+          },
+          {
+            "decision_phase": "awareness",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "what do you mean by that",
+            "intent_confidence": 0.4595,
+            "intent_type": "ambiguous",
+            "phase_confidence": 0.1718
+          }
+        ],
+        "suite_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/examples/intent_threshold_sweep_suite.json"
+      },
+      "head": {
+        "ambiguous_bad_allow_rate": 0.1842,
+        "ambiguous_prompt_count": 38,
+        "obvious_false_fallback_rate": 0.0104,
+        "obvious_prompt_count": 96,
+        "safe_predicate_rate": 0.2917
+      },
+      "threshold": 0.1
+    },
+    {
+      "combined": {
+        "bad_allow_rate_on_safe_prompts": 0.0,
+        "benchmark_fallback_rate": 0.6,
+        "benchmark_intent_only_fallback_rate": 0.0,
+        "benchmark_phase_only_fallback_rate": 0.5333,
+        "fallback_responsibility": {
+          "both": 0,
+          "fallback_rate": 0.6,
+          "intent_only": 0,
+          "intent_share_of_threshold_fallbacks": 0.0,
+          "phase_only": 5,
+          "phase_share_of_threshold_fallbacks": 1.0,
+          "policy_safe": 1
+        },
+        "false_fallback_rate_on_obvious_prompts": 0.3333,
+        "obvious_prompt_count": 6,
+        "safe_prompt_count": 4,
+        "suite_outputs": [
+          {
+            "decision_phase": "awareness",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "What is CRM software?",
+            "intent_confidence": 0.8488,
+            "intent_type": "informational",
+            "phase_confidence": 0.4134
+          },
+          {
+            "decision_phase": "research",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "Help me understand CRM basics",
+            "intent_confidence": 0.4226,
+            "intent_type": "informational",
+            "phase_confidence": 0.2602
+          },
+          {
+            "decision_phase": "consideration",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "HubSpot vs Zoho for a small team",
+            "intent_confidence": 0.6819,
+            "intent_type": "commercial",
+            "phase_confidence": 0.3331
+          },
+          {
+            "decision_phase": "decision",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "Which CRM should I buy for a 3-person startup?",
+            "intent_confidence": 0.7691,
+            "intent_type": "informational",
+            "phase_confidence": 0.2879
+          },
+          {
+            "decision_phase": "action",
+            "expected_outcome": "pass",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "Start my free trial",
+            "intent_confidence": 0.5429,
+            "intent_type": "transactional",
+            "phase_confidence": 0.1875
+          },
+          {
+            "decision_phase": "action",
+            "expected_outcome": "pass",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "Book a table for 2 tonight",
+            "intent_confidence": 0.5284,
+            "intent_type": "transactional",
+            "phase_confidence": 0.1716
+          },
+          {
+            "decision_phase": "support",
+            "expected_outcome": "fallback",
+            "failed_components": [],
+            "fallback_applied": true,
+            "input": "I cannot log into my account",
+            "intent_confidence": 0.3314,
+            "intent_type": "personal_reflection",
+            "phase_confidence": 0.275
+          },
+          {
+            "decision_phase": "action",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "go deeper",
+            "intent_confidence": 0.6527,
+            "intent_type": "ambiguous",
+            "phase_confidence": 0.1638
+          },
+          {
+            "decision_phase": "research",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "say more about that",
+            "intent_confidence": 0.6538,
+            "intent_type": "ambiguous",
+            "phase_confidence": 0.1674
+          },
+          {
+            "decision_phase": "awareness",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "what do you mean by that",
+            "intent_confidence": 0.4595,
+            "intent_type": "ambiguous",
+            "phase_confidence": 0.1718
+          }
+        ],
+        "suite_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/examples/intent_threshold_sweep_suite.json"
+      },
+      "head": {
+        "ambiguous_bad_allow_rate": 0.1842,
+        "ambiguous_prompt_count": 38,
+        "obvious_false_fallback_rate": 0.0104,
+        "obvious_prompt_count": 96,
+        "safe_predicate_rate": 0.2917
+      },
+      "threshold": 0.15
+    },
+    {
+      "combined": {
+        "bad_allow_rate_on_safe_prompts": 0.0,
+        "benchmark_fallback_rate": 0.6,
+        "benchmark_intent_only_fallback_rate": 0.0,
+        "benchmark_phase_only_fallback_rate": 0.5333,
+        "fallback_responsibility": {
+          "both": 0,
+          "fallback_rate": 0.6,
+          "intent_only": 0,
+          "intent_share_of_threshold_fallbacks": 0.0,
+          "phase_only": 5,
+          "phase_share_of_threshold_fallbacks": 1.0,
+          "policy_safe": 1
+        },
+        "false_fallback_rate_on_obvious_prompts": 0.3333,
+        "obvious_prompt_count": 6,
+        "safe_prompt_count": 4,
+        "suite_outputs": [
+          {
+            "decision_phase": "awareness",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "What is CRM software?",
+            "intent_confidence": 0.8488,
+            "intent_type": "informational",
+            "phase_confidence": 0.4134
+          },
+          {
+            "decision_phase": "research",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "Help me understand CRM basics",
+            "intent_confidence": 0.4226,
+            "intent_type": "informational",
+            "phase_confidence": 0.2602
+          },
+          {
+            "decision_phase": "consideration",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "HubSpot vs Zoho for a small team",
+            "intent_confidence": 0.6819,
+            "intent_type": "commercial",
+            "phase_confidence": 0.3331
+          },
+          {
+            "decision_phase": "decision",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "Which CRM should I buy for a 3-person startup?",
+            "intent_confidence": 0.7691,
+            "intent_type": "informational",
+            "phase_confidence": 0.2879
+          },
+          {
+            "decision_phase": "action",
+            "expected_outcome": "pass",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "Start my free trial",
+            "intent_confidence": 0.5429,
+            "intent_type": "transactional",
+            "phase_confidence": 0.1875
+          },
+          {
+            "decision_phase": "action",
+            "expected_outcome": "pass",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "Book a table for 2 tonight",
+            "intent_confidence": 0.5284,
+            "intent_type": "transactional",
+            "phase_confidence": 0.1716
+          },
+          {
+            "decision_phase": "support",
+            "expected_outcome": "fallback",
+            "failed_components": [],
+            "fallback_applied": true,
+            "input": "I cannot log into my account",
+            "intent_confidence": 0.3314,
+            "intent_type": "personal_reflection",
+            "phase_confidence": 0.275
+          },
+          {
+            "decision_phase": "action",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "go deeper",
+            "intent_confidence": 0.6527,
+            "intent_type": "ambiguous",
+            "phase_confidence": 0.1638
+          },
+          {
+            "decision_phase": "research",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "say more about that",
+            "intent_confidence": 0.6538,
+            "intent_type": "ambiguous",
+            "phase_confidence": 0.1674
+          },
+          {
+            "decision_phase": "awareness",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "what do you mean by that",
+            "intent_confidence": 0.4595,
+            "intent_type": "ambiguous",
+            "phase_confidence": 0.1718
+          }
+        ],
+        "suite_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/examples/intent_threshold_sweep_suite.json"
+      },
+      "head": {
+        "ambiguous_bad_allow_rate": 0.1842,
+        "ambiguous_prompt_count": 38,
+        "obvious_false_fallback_rate": 0.0104,
+        "obvious_prompt_count": 96,
+        "safe_predicate_rate": 0.2917
+      },
+      "threshold": 0.2
+    },
+    {
+      "combined": {
+        "bad_allow_rate_on_safe_prompts": 0.0,
+        "benchmark_fallback_rate": 0.6,
+        "benchmark_intent_only_fallback_rate": 0.0,
+        "benchmark_phase_only_fallback_rate": 0.5333,
+        "fallback_responsibility": {
+          "both": 0,
+          "fallback_rate": 0.6,
+          "intent_only": 0,
+          "intent_share_of_threshold_fallbacks": 0.0,
+          "phase_only": 5,
+          "phase_share_of_threshold_fallbacks": 1.0,
+          "policy_safe": 1
+        },
+        "false_fallback_rate_on_obvious_prompts": 0.3333,
+        "obvious_prompt_count": 6,
+        "safe_prompt_count": 4,
+        "suite_outputs": [
+          {
+            "decision_phase": "awareness",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "What is CRM software?",
+            "intent_confidence": 0.8488,
+            "intent_type": "informational",
+            "phase_confidence": 0.4134
+          },
+          {
+            "decision_phase": "research",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "Help me understand CRM basics",
+            "intent_confidence": 0.4226,
+            "intent_type": "informational",
+            "phase_confidence": 0.2602
+          },
+          {
+            "decision_phase": "consideration",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "HubSpot vs Zoho for a small team",
+            "intent_confidence": 0.6819,
+            "intent_type": "commercial",
+            "phase_confidence": 0.3331
+          },
+          {
+            "decision_phase": "decision",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "Which CRM should I buy for a 3-person startup?",
+            "intent_confidence": 0.7691,
+            "intent_type": "informational",
+            "phase_confidence": 0.2879
+          },
+          {
+            "decision_phase": "action",
+            "expected_outcome": "pass",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "Start my free trial",
+            "intent_confidence": 0.5429,
+            "intent_type": "transactional",
+            "phase_confidence": 0.1875
+          },
+          {
+            "decision_phase": "action",
+            "expected_outcome": "pass",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "Book a table for 2 tonight",
+            "intent_confidence": 0.5284,
+            "intent_type": "transactional",
+            "phase_confidence": 0.1716
+          },
+          {
+            "decision_phase": "support",
+            "expected_outcome": "fallback",
+            "failed_components": [],
+            "fallback_applied": true,
+            "input": "I cannot log into my account",
+            "intent_confidence": 0.3314,
+            "intent_type": "personal_reflection",
+            "phase_confidence": 0.275
+          },
+          {
+            "decision_phase": "action",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "go deeper",
+            "intent_confidence": 0.6527,
+            "intent_type": "ambiguous",
+            "phase_confidence": 0.1638
+          },
+          {
+            "decision_phase": "research",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "say more about that",
+            "intent_confidence": 0.6538,
+            "intent_type": "ambiguous",
+            "phase_confidence": 0.1674
+          },
+          {
+            "decision_phase": "awareness",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "what do you mean by that",
+            "intent_confidence": 0.4595,
+            "intent_type": "ambiguous",
+            "phase_confidence": 0.1718
+          }
+        ],
+        "suite_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/examples/intent_threshold_sweep_suite.json"
+      },
+      "head": {
+        "ambiguous_bad_allow_rate": 0.1842,
+        "ambiguous_prompt_count": 38,
+        "obvious_false_fallback_rate": 0.0104,
+        "obvious_prompt_count": 96,
+        "safe_predicate_rate": 0.2917
+      },
+      "threshold": 0.25
+    },
+    {
+      "combined": {
+        "bad_allow_rate_on_safe_prompts": 0.0,
+        "benchmark_fallback_rate": 0.6,
+        "benchmark_intent_only_fallback_rate": 0.0,
+        "benchmark_phase_only_fallback_rate": 0.5333,
+        "fallback_responsibility": {
+          "both": 0,
+          "fallback_rate": 0.6,
+          "intent_only": 0,
+          "intent_share_of_threshold_fallbacks": 0.0,
+          "phase_only": 5,
+          "phase_share_of_threshold_fallbacks": 1.0,
+          "policy_safe": 1
+        },
+        "false_fallback_rate_on_obvious_prompts": 0.3333,
+        "obvious_prompt_count": 6,
+        "safe_prompt_count": 4,
+        "suite_outputs": [
+          {
+            "decision_phase": "awareness",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "What is CRM software?",
+            "intent_confidence": 0.8488,
+            "intent_type": "informational",
+            "phase_confidence": 0.4134
+          },
+          {
+            "decision_phase": "research",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "Help me understand CRM basics",
+            "intent_confidence": 0.4226,
+            "intent_type": "informational",
+            "phase_confidence": 0.2602
+          },
+          {
+            "decision_phase": "consideration",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "HubSpot vs Zoho for a small team",
+            "intent_confidence": 0.6819,
+            "intent_type": "commercial",
+            "phase_confidence": 0.3331
+          },
+          {
+            "decision_phase": "decision",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "Which CRM should I buy for a 3-person startup?",
+            "intent_confidence": 0.7691,
+            "intent_type": "informational",
+            "phase_confidence": 0.2879
+          },
+          {
+            "decision_phase": "action",
+            "expected_outcome": "pass",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "Start my free trial",
+            "intent_confidence": 0.5429,
+            "intent_type": "transactional",
+            "phase_confidence": 0.1875
+          },
+          {
+            "decision_phase": "action",
+            "expected_outcome": "pass",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "Book a table for 2 tonight",
+            "intent_confidence": 0.5284,
+            "intent_type": "transactional",
+            "phase_confidence": 0.1716
+          },
+          {
+            "decision_phase": "support",
+            "expected_outcome": "fallback",
+            "failed_components": [],
+            "fallback_applied": true,
+            "input": "I cannot log into my account",
+            "intent_confidence": 0.3314,
+            "intent_type": "personal_reflection",
+            "phase_confidence": 0.275
+          },
+          {
+            "decision_phase": "action",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "go deeper",
+            "intent_confidence": 0.6527,
+            "intent_type": "ambiguous",
+            "phase_confidence": 0.1638
+          },
+          {
+            "decision_phase": "research",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "say more about that",
+            "intent_confidence": 0.6538,
+            "intent_type": "ambiguous",
+            "phase_confidence": 0.1674
+          },
+          {
+            "decision_phase": "awareness",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "what do you mean by that",
+            "intent_confidence": 0.4595,
+            "intent_type": "ambiguous",
+            "phase_confidence": 0.1718
+          }
+        ],
+        "suite_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/examples/intent_threshold_sweep_suite.json"
+      },
+      "head": {
+        "ambiguous_bad_allow_rate": 0.1842,
+        "ambiguous_prompt_count": 38,
+        "obvious_false_fallback_rate": 0.0104,
+        "obvious_prompt_count": 96,
+        "safe_predicate_rate": 0.2917
+      },
+      "threshold": 0.3
+    },
+    {
+      "combined": {
+        "bad_allow_rate_on_safe_prompts": 0.0,
+        "benchmark_fallback_rate": 0.6,
+        "benchmark_intent_only_fallback_rate": 0.0667,
+        "benchmark_phase_only_fallback_rate": 0.5333,
+        "fallback_responsibility": {
+          "both": 0,
+          "fallback_rate": 0.6,
+          "intent_only": 1,
+          "intent_share_of_threshold_fallbacks": 0.1667,
+          "phase_only": 5,
+          "phase_share_of_threshold_fallbacks": 0.8333,
+          "policy_safe": 0
+        },
+        "false_fallback_rate_on_obvious_prompts": 0.3333,
+        "obvious_prompt_count": 6,
+        "safe_prompt_count": 4,
+        "suite_outputs": [
+          {
+            "decision_phase": "awareness",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "What is CRM software?",
+            "intent_confidence": 0.8488,
+            "intent_type": "informational",
+            "phase_confidence": 0.4134
+          },
+          {
+            "decision_phase": "research",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "Help me understand CRM basics",
+            "intent_confidence": 0.4226,
+            "intent_type": "informational",
+            "phase_confidence": 0.2602
+          },
+          {
+            "decision_phase": "consideration",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "HubSpot vs Zoho for a small team",
+            "intent_confidence": 0.6819,
+            "intent_type": "commercial",
+            "phase_confidence": 0.3331
+          },
+          {
+            "decision_phase": "decision",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "Which CRM should I buy for a 3-person startup?",
+            "intent_confidence": 0.7691,
+            "intent_type": "informational",
+            "phase_confidence": 0.2879
+          },
+          {
+            "decision_phase": "action",
+            "expected_outcome": "pass",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "Start my free trial",
+            "intent_confidence": 0.5429,
+            "intent_type": "transactional",
+            "phase_confidence": 0.1875
+          },
+          {
+            "decision_phase": "action",
+            "expected_outcome": "pass",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "Book a table for 2 tonight",
+            "intent_confidence": 0.5284,
+            "intent_type": "transactional",
+            "phase_confidence": 0.1716
+          },
+          {
+            "decision_phase": "support",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "intent_type"
+            ],
+            "fallback_applied": true,
+            "input": "I cannot log into my account",
+            "intent_confidence": 0.3314,
+            "intent_type": "personal_reflection",
+            "phase_confidence": 0.275
+          },
+          {
+            "decision_phase": "action",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "go deeper",
+            "intent_confidence": 0.6527,
+            "intent_type": "ambiguous",
+            "phase_confidence": 0.1638
+          },
+          {
+            "decision_phase": "research",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "say more about that",
+            "intent_confidence": 0.6538,
+            "intent_type": "ambiguous",
+            "phase_confidence": 0.1674
+          },
+          {
+            "decision_phase": "awareness",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "what do you mean by that",
+            "intent_confidence": 0.4595,
+            "intent_type": "ambiguous",
+            "phase_confidence": 0.1718
+          }
+        ],
+        "suite_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/examples/intent_threshold_sweep_suite.json"
+      },
+      "head": {
+        "ambiguous_bad_allow_rate": 0.1842,
+        "ambiguous_prompt_count": 38,
+        "obvious_false_fallback_rate": 0.0417,
+        "obvious_prompt_count": 96,
+        "safe_predicate_rate": 0.2917
+      },
+      "threshold": 0.35
+    },
+    {
+      "combined": {
+        "bad_allow_rate_on_safe_prompts": 0.0,
+        "benchmark_fallback_rate": 0.6,
+        "benchmark_intent_only_fallback_rate": 0.0667,
+        "benchmark_phase_only_fallback_rate": 0.4667,
+        "fallback_responsibility": {
+          "both": 0,
+          "fallback_rate": 0.6,
+          "intent_only": 1,
+          "intent_share_of_threshold_fallbacks": 0.1667,
+          "phase_only": 5,
+          "phase_share_of_threshold_fallbacks": 0.8333,
+          "policy_safe": 0
+        },
+        "false_fallback_rate_on_obvious_prompts": 0.3333,
+        "obvious_prompt_count": 6,
+        "safe_prompt_count": 4,
+        "suite_outputs": [
+          {
+            "decision_phase": "awareness",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "What is CRM software?",
+            "intent_confidence": 0.8488,
+            "intent_type": "informational",
+            "phase_confidence": 0.4134
+          },
+          {
+            "decision_phase": "research",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "Help me understand CRM basics",
+            "intent_confidence": 0.4226,
+            "intent_type": "informational",
+            "phase_confidence": 0.2602
+          },
+          {
+            "decision_phase": "consideration",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "HubSpot vs Zoho for a small team",
+            "intent_confidence": 0.6819,
+            "intent_type": "commercial",
+            "phase_confidence": 0.3331
+          },
+          {
+            "decision_phase": "decision",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "Which CRM should I buy for a 3-person startup?",
+            "intent_confidence": 0.7691,
+            "intent_type": "informational",
+            "phase_confidence": 0.2879
+          },
+          {
+            "decision_phase": "action",
+            "expected_outcome": "pass",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "Start my free trial",
+            "intent_confidence": 0.5429,
+            "intent_type": "transactional",
+            "phase_confidence": 0.1875
+          },
+          {
+            "decision_phase": "action",
+            "expected_outcome": "pass",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "Book a table for 2 tonight",
+            "intent_confidence": 0.5284,
+            "intent_type": "transactional",
+            "phase_confidence": 0.1716
+          },
+          {
+            "decision_phase": "support",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "intent_type"
+            ],
+            "fallback_applied": true,
+            "input": "I cannot log into my account",
+            "intent_confidence": 0.3314,
+            "intent_type": "personal_reflection",
+            "phase_confidence": 0.275
+          },
+          {
+            "decision_phase": "action",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "go deeper",
+            "intent_confidence": 0.6527,
+            "intent_type": "ambiguous",
+            "phase_confidence": 0.1638
+          },
+          {
+            "decision_phase": "research",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "say more about that",
+            "intent_confidence": 0.6538,
+            "intent_type": "ambiguous",
+            "phase_confidence": 0.1674
+          },
+          {
+            "decision_phase": "awareness",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "what do you mean by that",
+            "intent_confidence": 0.4595,
+            "intent_type": "ambiguous",
+            "phase_confidence": 0.1718
+          }
+        ],
+        "suite_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/examples/intent_threshold_sweep_suite.json"
+      },
+      "head": {
+        "ambiguous_bad_allow_rate": 0.1053,
+        "ambiguous_prompt_count": 38,
+        "obvious_false_fallback_rate": 0.0625,
+        "obvious_prompt_count": 96,
+        "safe_predicate_rate": 0.2917
+      },
+      "threshold": 0.4
+    },
+    {
+      "combined": {
+        "bad_allow_rate_on_safe_prompts": 0.0,
+        "benchmark_fallback_rate": 0.6667,
+        "benchmark_intent_only_fallback_rate": 0.1333,
+        "benchmark_phase_only_fallback_rate": 0.4667,
+        "fallback_responsibility": {
+          "both": 0,
+          "fallback_rate": 0.7,
+          "intent_only": 2,
+          "intent_share_of_threshold_fallbacks": 0.2857,
+          "phase_only": 5,
+          "phase_share_of_threshold_fallbacks": 0.7143,
+          "policy_safe": 0
+        },
+        "false_fallback_rate_on_obvious_prompts": 0.5,
+        "obvious_prompt_count": 6,
+        "safe_prompt_count": 4,
+        "suite_outputs": [
+          {
+            "decision_phase": "awareness",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "What is CRM software?",
+            "intent_confidence": 0.8488,
+            "intent_type": "informational",
+            "phase_confidence": 0.4134
+          },
+          {
+            "decision_phase": "research",
+            "expected_outcome": "pass",
+            "failed_components": [
+              "intent_type"
+            ],
+            "fallback_applied": true,
+            "input": "Help me understand CRM basics",
+            "intent_confidence": 0.4226,
+            "intent_type": "informational",
+            "phase_confidence": 0.2602
+          },
+          {
+            "decision_phase": "consideration",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "HubSpot vs Zoho for a small team",
+            "intent_confidence": 0.6819,
+            "intent_type": "commercial",
+            "phase_confidence": 0.3331
+          },
+          {
+            "decision_phase": "decision",
+            "expected_outcome": "pass",
+            "failed_components": [],
+            "fallback_applied": false,
+            "input": "Which CRM should I buy for a 3-person startup?",
+            "intent_confidence": 0.7691,
+            "intent_type": "informational",
+            "phase_confidence": 0.2879
+          },
+          {
+            "decision_phase": "action",
+            "expected_outcome": "pass",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "Start my free trial",
+            "intent_confidence": 0.5429,
+            "intent_type": "transactional",
+            "phase_confidence": 0.1875
+          },
+          {
+            "decision_phase": "action",
+            "expected_outcome": "pass",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "Book a table for 2 tonight",
+            "intent_confidence": 0.5284,
+            "intent_type": "transactional",
+            "phase_confidence": 0.1716
+          },
+          {
+            "decision_phase": "support",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "intent_type"
+            ],
+            "fallback_applied": true,
+            "input": "I cannot log into my account",
+            "intent_confidence": 0.3314,
+            "intent_type": "personal_reflection",
+            "phase_confidence": 0.275
+          },
+          {
+            "decision_phase": "action",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "go deeper",
+            "intent_confidence": 0.6527,
+            "intent_type": "ambiguous",
+            "phase_confidence": 0.1638
+          },
+          {
+            "decision_phase": "research",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "say more about that",
+            "intent_confidence": 0.6538,
+            "intent_type": "ambiguous",
+            "phase_confidence": 0.1674
+          },
+          {
+            "decision_phase": "awareness",
+            "expected_outcome": "fallback",
+            "failed_components": [
+              "decision_phase"
+            ],
+            "fallback_applied": true,
+            "input": "what do you mean by that",
+            "intent_confidence": 0.4595,
+            "intent_type": "ambiguous",
+            "phase_confidence": 0.1718
+          }
+        ],
+        "suite_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/examples/intent_threshold_sweep_suite.json"
+      },
+      "head": {
+        "ambiguous_bad_allow_rate": 0.1053,
+        "ambiguous_prompt_count": 38,
+        "obvious_false_fallback_rate": 0.1042,
+        "obvious_prompt_count": 96,
+        "safe_predicate_rate": 0.2917
+      },
+      "threshold": 0.45
+    }
+  ],
+  "thresholds": [
+    0.0,
+    0.1,
+    0.15,
+    0.2,
+    0.25,
+    0.3,
+    0.35,
+    0.4,
+    0.45
+  ]
+}

artifacts/evaluation/latest/combined_demo_benchmark.json

@@ -0,0 +1,1352 @@
+[
+  {
+    "expected_behavior": "Should read as informational awareness and fail safe via fallback because confidence is still low.",
+    "input": "What is CRM software?",
+    "response": {
+      "meta": {
+        "calibration_enabled": true,
+        "system_version": "0.6.0-phase4"
+      },
+      "model_output": {
+        "classification": {
+          "iab_content": {
+            "mapping_confidence": 0.9035,
+            "mapping_mode": "nearest_equivalent",
+            "taxonomy": "IAB Content Taxonomy",
+            "taxonomy_version": "3.0",
+            "tier1": {
+              "id": "596",
+              "label": "Technology & Computing"
+            },
+            "tier2": {
+              "id": "599",
+              "label": "Computing"
+            },
+            "tier3": {
+              "id": "602",
+              "label": "Software and Applications"
+            }
+          },
+          "intent": {
+            "commercial_score": 0.0875,
+            "component_confidence": {
+              "decision_phase": {
+                "calibrated": true,
+                "confidence": 0.9947,
+                "confidence_threshold": 0.22,
+                "label": "awareness",
+                "meets_threshold": true,
+                "raw_confidence": 0.9788
+              },
+              "intent_subtype": {
+                "calibrated": true,
+                "confidence": 0.9547,
+                "confidence_threshold": 0.25,
+                "label": "education",
+                "meets_threshold": true,
+                "raw_confidence": 0.9547
+              },
+              "intent_type": {
+                "calibrated": true,
+                "confidence": 0.9972,
+                "confidence_threshold": 0.4,
+                "label": "informational",
+                "meets_threshold": true,
+                "raw_confidence": 0.9662
+              },
+              "overall_strategy": "min_required_component_confidence"
+            },
+            "confidence": 0.9947,
+            "decision_phase": "awareness",
+            "subtype": "education",
+            "summary": "Classified as informational intent with subtype education in the awareness phase.",
+            "type": "informational"
+          }
+        },
+        "fallback": null
+      },
+      "system_decision": {
+        "intent_trajectory": [
+          "awareness"
+        ],
+        "opportunity": {
+          "strength": "low",
+          "type": "none"
+        },
+        "policy": {
+          "applied_thresholds": {
+            "commercial_score_min": 0.6,
+            "decision_phase_confidence_min": 0.22,
+            "intent_subtype_confidence_min": 0.25,
+            "intent_type_confidence_min": 0.4
+          },
+          "decision_basis": "score_threshold",
+          "eligibility_reason": "commercial_signal_below_threshold",
+          "monetization_eligibility": "restricted",
+          "regulated_vertical": false,
+          "sensitivity": "low"
+        }
+      }
+    }
+  },
+  {
+    "expected_behavior": "Should lean informational research and still fall back conservatively.",
+    "input": "Help me understand CRM basics",
+    "response": {
+      "meta": {
+        "calibration_enabled": true,
+        "system_version": "0.6.0-phase4"
+      },
+      "model_output": {
+        "classification": {
+          "iab_content": {
+            "mapping_confidence": 0.8427,
+            "mapping_mode": "nearest_equivalent",
+            "taxonomy": "IAB Content Taxonomy",
+            "taxonomy_version": "3.0",
+            "tier1": {
+              "id": "1",
+              "label": "Automotive"
+            }
+          },
+          "intent": {
+            "commercial_score": 0.0875,
+            "component_confidence": {
+              "decision_phase": {
+                "calibrated": true,
+                "confidence": 0.9944,
+                "confidence_threshold": 0.22,
+                "label": "awareness",
+                "meets_threshold": true,
+                "raw_confidence": 0.9779
+              },
+              "intent_subtype": {
+                "calibrated": true,
+                "confidence": 0.955,
+                "confidence_threshold": 0.25,
+                "label": "education",
+                "meets_threshold": true,
+                "raw_confidence": 0.955
+              },
+              "intent_type": {
+                "calibrated": true,
+                "confidence": 0.9969,
+                "confidence_threshold": 0.4,
+                "label": "informational",
+                "meets_threshold": true,
+                "raw_confidence": 0.9637
+              },
+              "overall_strategy": "min_required_component_confidence"
+            },
+            "confidence": 0.9944,
+            "decision_phase": "awareness",
+            "subtype": "education",
+            "summary": "Classified as informational intent with subtype education in the awareness phase.",
+            "type": "informational"
+          }
+        },
+        "fallback": null
+      },
+      "system_decision": {
+        "intent_trajectory": [
+          "awareness"
+        ],
+        "opportunity": {
+          "strength": "low",
+          "type": "none"
+        },
+        "policy": {
+          "applied_thresholds": {
+            "commercial_score_min": 0.6,
+            "decision_phase_confidence_min": 0.22,
+            "intent_subtype_confidence_min": 0.25,
+            "intent_type_confidence_min": 0.4
+          },
+          "decision_basis": "score_threshold",
+          "eligibility_reason": "commercial_signal_below_threshold",
+          "monetization_eligibility": "restricted",
+          "regulated_vertical": false,
+          "sensitivity": "low"
+        }
+      }
+    }
+  },
+  {
+    "expected_behavior": "Should behave like a commercial comparison or selection query, but v0.1 still falls back because confidence is low.",
+    "input": "Best CRM for small teams",
+    "response": {
+      "meta": {
+        "calibration_enabled": true,
+        "system_version": "0.6.0-phase4"
+      },
+      "model_output": {
+        "classification": {
+          "iab_content": {
+            "mapping_confidence": 0.7798,
+            "mapping_mode": "nearest_equivalent",
+            "taxonomy": "IAB Content Taxonomy",
+            "taxonomy_version": "3.0",
+            "tier1": {
+              "id": "483",
+              "label": "Sports"
+            }
+          },
+          "intent": {
+            "commercial_score": 0.656,
+            "component_confidence": {
+              "decision_phase": {
+                "calibrated": true,
+                "confidence": 0.9965,
+                "confidence_threshold": 0.22,
+                "label": "consideration",
+                "meets_threshold": true,
+                "raw_confidence": 0.9846
+              },
+              "intent_subtype": {
+                "calibrated": true,
+                "confidence": 0.4682,
+                "confidence_threshold": 0.25,
+                "label": "product_discovery",
+                "meets_threshold": true,
+                "raw_confidence": 0.4682
+              },
+              "intent_type": {
+                "calibrated": true,
+                "confidence": 0.9995,
+                "confidence_threshold": 0.4,
+                "label": "commercial",
+                "meets_threshold": true,
+                "raw_confidence": 0.9895
+              },
+              "overall_strategy": "min_required_component_confidence"
+            },
+            "confidence": 0.4682,
+            "decision_phase": "consideration",
+            "subtype": "product_discovery",
+            "summary": "Classified as commercial intent with subtype product_discovery in the consideration phase.",
+            "type": "commercial"
+          }
+        },
+        "fallback": null
+      },
+      "system_decision": {
+        "intent_trajectory": [
+          "consideration"
+        ],
+        "opportunity": {
+          "strength": "medium",
+          "type": "soft_recommendation"
+        },
+        "policy": {
+          "applied_thresholds": {
+            "commercial_score_min": 0.6,
+            "decision_phase_confidence_min": 0.22,
+            "intent_subtype_confidence_min": 0.25,
+            "intent_type_confidence_min": 0.4
+          },
+          "decision_basis": "score_threshold",
+          "eligibility_reason": "commercial_discovery_signal_present",
+          "monetization_eligibility": "allowed_with_caution",
+          "regulated_vertical": false,
+          "sensitivity": "low"
+        }
+      }
+    }
+  },
+  {
+    "expected_behavior": "Should read as a commercial comparison in consideration and remain conservative if confidence stays low.",
+    "input": "HubSpot vs Zoho for a small team",
+    "response": {
+      "meta": {
+        "calibration_enabled": true,
+        "system_version": "0.6.0-phase4"
+      },
+      "model_output": {
+        "classification": {
+          "iab_content": {
+            "mapping_confidence": 0.8606,
+            "mapping_mode": "nearest_equivalent",
+            "taxonomy": "IAB Content Taxonomy",
+            "taxonomy_version": "3.0",
+            "tier1": {
+              "id": "596",
+              "label": "Technology & Computing"
+            },
+            "tier2": {
+              "id": "599",
+              "label": "Computing"
+            },
+            "tier3": {
+              "id": "619",
+              "label": "Internet"
+            }
+          },
+          "intent": {
+            "commercial_score": 0.728,
+            "component_confidence": {
+              "decision_phase": {
+                "calibrated": true,
+                "confidence": 0.9964,
+                "confidence_threshold": 0.22,
+                "label": "consideration",
+                "meets_threshold": true,
+                "raw_confidence": 0.9842
+              },
+              "intent_subtype": {
+                "calibrated": true,
+                "confidence": 0.9449,
+                "confidence_threshold": 0.25,
+                "label": "comparison",
+                "meets_threshold": true,
+                "raw_confidence": 0.9449
+              },
+              "intent_type": {
+                "calibrated": true,
+                "confidence": 0.9995,
+                "confidence_threshold": 0.4,
+                "label": "commercial",
+                "meets_threshold": true,
+                "raw_confidence": 0.9892
+              },
+              "overall_strategy": "min_required_component_confidence"
+            },
+            "confidence": 0.9449,
+            "decision_phase": "consideration",
+            "subtype": "comparison",
+            "summary": "Classified as commercial intent with subtype comparison in the consideration phase.",
+            "type": "commercial"
+          }
+        },
+        "fallback": null
+      },
+      "system_decision": {
+        "intent_trajectory": [
+          "consideration"
+        ],
+        "opportunity": {
+          "strength": "high",
+          "type": "comparison_slot"
+        },
+        "policy": {
+          "applied_thresholds": {
+            "commercial_score_min": 0.6,
+            "decision_phase_confidence_min": 0.22,
+            "intent_subtype_confidence_min": 0.25,
+            "intent_type_confidence_min": 0.4
+          },
+          "decision_basis": "score_threshold",
+          "eligibility_reason": "commercial_comparison_signal_present",
+          "monetization_eligibility": "allowed_with_caution",
+          "regulated_vertical": false,
+          "sensitivity": "low"
+        }
+      }
+    }
+  },
+  {
+    "expected_behavior": "Should preserve a decision-stage buying signal even if the intent-type head still undercalls it.",
+    "input": "Which CRM should I buy for a 3-person startup?",
+    "response": {
+      "meta": {
+        "calibration_enabled": true,
+        "system_version": "0.6.0-phase4"
+      },
+      "model_output": {
+        "classification": {
+          "iab_content": {
+            "mapping_confidence": 0.8737,
+            "mapping_mode": "nearest_equivalent",
+            "taxonomy": "IAB Content Taxonomy",
+            "taxonomy_version": "3.0",
+            "tier1": {
+              "id": "52",
+              "label": "Business and Finance"
+            },
+            "tier2": {
+              "id": "53",
+              "label": "Business"
+            }
+          },
+          "intent": {
+            "commercial_score": 0.8525,
+            "component_confidence": {
+              "decision_phase": {
+                "calibrated": true,
+                "confidence": 0.963,
+                "confidence_threshold": 0.22,
+                "label": "decision",
+                "meets_threshold": true,
+                "raw_confidence": 0.9122
+              },
+              "intent_subtype": {
+                "calibrated": true,
+                "confidence": 0.9119,
+                "confidence_threshold": 0.25,
+                "label": "provider_selection",
+                "meets_threshold": true,
+                "raw_confidence": 0.9119
+              },
+              "intent_type": {
+                "calibrated": true,
+                "confidence": 0.9994,
+                "confidence_threshold": 0.4,
+                "label": "commercial",
+                "meets_threshold": true,
+                "raw_confidence": 0.9874
+              },
+              "overall_strategy": "min_required_component_confidence"
+            },
+            "confidence": 0.9119,
+            "decision_phase": "decision",
+            "subtype": "provider_selection",
+            "summary": "Classified as commercial intent with subtype provider_selection in the decision phase.",
+            "type": "commercial"
+          }
+        },
+        "fallback": null
+      },
+      "system_decision": {
+        "intent_trajectory": [
+          "decision"
+        ],
+        "opportunity": {
+          "strength": "high",
+          "type": "decision_moment"
+        },
+        "policy": {
+          "applied_thresholds": {
+            "commercial_score_min": 0.6,
+            "decision_phase_confidence_min": 0.22,
+            "intent_subtype_confidence_min": 0.25,
+            "intent_type_confidence_min": 0.4
+          },
+          "decision_basis": "score_threshold",
+          "eligibility_reason": "high_intent_subtype_signal",
+          "monetization_eligibility": "allowed",
+          "regulated_vertical": false,
+          "sensitivity": "low"
+        }
+      }
+    }
+  },
+  {
+    "expected_behavior": "Should read as an action-stage transactional query and fail safe because confidence is low.",
+    "input": "Start my free trial",
+    "response": {
+      "meta": {
+        "calibration_enabled": true,
+        "system_version": "0.6.0-phase4"
+      },
+      "model_output": {
+        "classification": {
+          "iab_content": {
+            "mapping_confidence": 0.7133,
+            "mapping_mode": "nearest_equivalent",
+            "taxonomy": "IAB Content Taxonomy",
+            "taxonomy_version": "3.0",
+            "tier1": {
+              "id": "239",
+              "label": "Hobbies & Interests"
+            },
+            "tier2": {
+              "id": "264",
+              "label": "Content Production"
+            },
+            "tier3": {
+              "id": "266",
+              "label": "Freelance Writing"
+            }
+          },
+          "intent": {
+            "commercial_score": 0.954,
+            "component_confidence": {
+              "decision_phase": {
+                "calibrated": true,
+                "confidence": 0.9991,
+                "confidence_threshold": 0.22,
+                "label": "action",
+                "meets_threshold": true,
+                "raw_confidence": 0.9947
+              },
+              "intent_subtype": {
+                "calibrated": true,
+                "confidence": 0.9382,
+                "confidence_threshold": 0.25,
+                "label": "signup",
+                "meets_threshold": true,
+                "raw_confidence": 0.9382
+              },
+              "intent_type": {
+                "calibrated": true,
+                "confidence": 0.9996,
+                "confidence_threshold": 0.4,
+                "label": "transactional",
+                "meets_threshold": true,
+                "raw_confidence": 0.9902
+              },
+              "overall_strategy": "min_required_component_confidence"
+            },
+            "confidence": 0.9382,
+            "decision_phase": "action",
+            "subtype": "signup",
+            "summary": "Classified as transactional intent with subtype signup in the action phase.",
+            "type": "transactional"
+          }
+        },
+        "fallback": null
+      },
+      "system_decision": {
+        "intent_trajectory": [
+          "action"
+        ],
+        "opportunity": {
+          "strength": "high",
+          "type": "transaction_trigger"
+        },
+        "policy": {
+          "applied_thresholds": {
+            "commercial_score_min": 0.6,
+            "decision_phase_confidence_min": 0.22,
+            "intent_subtype_confidence_min": 0.25,
+            "intent_type_confidence_min": 0.4
+          },
+          "decision_basis": "score_threshold",
+          "eligibility_reason": "high_intent_subtype_signal",
+          "monetization_eligibility": "allowed",
+          "regulated_vertical": false,
+          "sensitivity": "low"
+        }
+      }
+    }
+  },
+  {
+    "expected_behavior": "Should preserve generic action-stage transactional behavior outside SaaS language.",
+    "input": "Book a table for 2 tonight",
+    "response": {
+      "meta": {
+        "calibration_enabled": true,
+        "system_version": "0.6.0-phase4"
+      },
+      "model_output": {
+        "classification": {
+          "iab_content": {
+            "mapping_confidence": 0.7997,
+            "mapping_mode": "nearest_equivalent",
+            "taxonomy": "IAB Content Taxonomy",
+            "taxonomy_version": "3.0",
+            "tier1": {
+              "id": "210",
+              "label": "Food & Drink"
+            },
+            "tier2": {
+              "id": "218",
+              "label": "Dining Out"
+            }
+          },
+          "intent": {
+            "commercial_score": 0.963,
+            "component_confidence": {
+              "decision_phase": {
+                "calibrated": true,
+                "confidence": 0.999,
+                "confidence_threshold": 0.22,
+                "label": "action",
+                "meets_threshold": true,
+                "raw_confidence": 0.9945
+              },
+              "intent_subtype": {
+                "calibrated": true,
+                "confidence": 0.8724,
+                "confidence_threshold": 0.25,
+                "label": "booking",
+                "meets_threshold": true,
+                "raw_confidence": 0.8724
+              },
+              "intent_type": {
+                "calibrated": true,
+                "confidence": 0.9996,
+                "confidence_threshold": 0.4,
+                "label": "transactional",
+                "meets_threshold": true,
+                "raw_confidence": 0.9901
+              },
+              "overall_strategy": "min_required_component_confidence"
+            },
+            "confidence": 0.8724,
+            "decision_phase": "action",
+            "subtype": "booking",
+            "summary": "Classified as transactional intent with subtype booking in the action phase.",
+            "type": "transactional"
+          }
+        },
+        "fallback": null
+      },
+      "system_decision": {
+        "intent_trajectory": [
+          "action"
+        ],
+        "opportunity": {
+          "strength": "high",
+          "type": "transaction_trigger"
+        },
+        "policy": {
+          "applied_thresholds": {
+            "commercial_score_min": 0.6,
+            "decision_phase_confidence_min": 0.22,
+            "intent_subtype_confidence_min": 0.25,
+            "intent_type_confidence_min": 0.4
+          },
+          "decision_basis": "score_threshold",
+          "eligibility_reason": "high_intent_subtype_signal",
+          "monetization_eligibility": "allowed",
+          "regulated_vertical": false,
+          "sensitivity": "low"
+        }
+      }
+    }
+  },
+  {
+    "expected_behavior": "Should preserve a post-purchase lifecycle signal and remain restricted under low confidence.",
+    "input": "How do I set up my new CRM?",
+    "response": {
+      "meta": {
+        "calibration_enabled": true,
+        "system_version": "0.6.0-phase4"
+      },
+      "model_output": {
+        "classification": {
+          "iab_content": {
+            "mapping_confidence": 0.8423,
+            "mapping_mode": "nearest_equivalent",
+            "taxonomy": "IAB Content Taxonomy",
+            "taxonomy_version": "3.0",
+            "tier1": {
+              "id": "596",
+              "label": "Technology & Computing"
+            },
+            "tier2": {
+              "id": "599",
+              "label": "Computing"
+            },
+            "tier3": {
+              "id": "619",
+              "label": "Internet"
+            },
+            "tier4": {
+              "id": "620",
+              "label": "Cloud Computing"
+            }
+          },
+          "intent": {
+            "commercial_score": 0.3235,
+            "component_confidence": {
+              "decision_phase": {
+                "calibrated": true,
+                "confidence": 0.9736,
+                "confidence_threshold": 0.22,
+                "label": "post_purchase",
+                "meets_threshold": true,
+                "raw_confidence": 0.9264
+              },
+              "intent_subtype": {
+                "calibrated": true,
+                "confidence": 0.921,
+                "confidence_threshold": 0.25,
+                "label": "onboarding_setup",
+                "meets_threshold": true,
+                "raw_confidence": 0.921
+              },
+              "intent_type": {
+                "calibrated": true,
+                "confidence": 0.9935,
+                "confidence_threshold": 0.4,
+                "label": "transactional",
+                "meets_threshold": true,
+                "raw_confidence": 0.9448
+              },
+              "overall_strategy": "min_required_component_confidence"
+            },
+            "confidence": 0.9736,
+            "decision_phase": "post_purchase",
+            "subtype": "onboarding_setup",
+            "summary": "Classified as transactional intent with subtype onboarding_setup in the post_purchase phase.",
+            "type": "transactional"
+          }
+        },
+        "fallback": null
+      },
+      "system_decision": {
+        "intent_trajectory": [
+          "post_purchase"
+        ],
+        "opportunity": {
+          "strength": "low",
+          "type": "soft_recommendation"
+        },
+        "policy": {
+          "applied_thresholds": {
+            "commercial_score_min": 0.6,
+            "decision_phase_confidence_min": 0.22,
+            "intent_subtype_confidence_min": 0.25,
+            "intent_type_confidence_min": 0.4
+          },
+          "decision_basis": "score_threshold",
+          "eligibility_reason": "post_purchase_setup_query",
+          "monetization_eligibility": "restricted",
+          "regulated_vertical": false,
+          "sensitivity": "low"
+        }
+      }
+    }
+  },
+  {
+    "expected_behavior": "Should fail safe on a support-like query and keep monetization blocked.",
+    "input": "I cannot log into my account",
+    "response": {
+      "meta": {
+        "calibration_enabled": true,
+        "system_version": "0.6.0-phase4"
+      },
+      "model_output": {
+        "classification": {
+          "iab_content": {
+            "mapping_confidence": 0.8039,
+            "mapping_mode": "nearest_equivalent",
+            "taxonomy": "IAB Content Taxonomy",
+            "taxonomy_version": "3.0",
+            "tier1": {
+              "id": "596",
+              "label": "Technology & Computing"
+            },
+            "tier2": {
+              "id": "599",
+              "label": "Computing"
+            },
+            "tier3": {
+              "id": "619",
+              "label": "Internet"
+            }
+          },
+          "intent": {
+            "commercial_score": 0.0,
+            "component_confidence": {
+              "decision_phase": {
+                "calibrated": true,
+                "confidence": 0.9969,
+                "confidence_threshold": 0.22,
+                "label": "support",
+                "meets_threshold": true,
+                "raw_confidence": 0.9863
+              },
+              "intent_subtype": {
+                "calibrated": true,
+                "confidence": 0.923,
+                "confidence_threshold": 0.25,
+                "label": "account_help",
+                "meets_threshold": true,
+                "raw_confidence": 0.923
+              },
+              "intent_type": {
+                "calibrated": true,
+                "confidence": 0.9988,
+                "confidence_threshold": 0.4,
+                "label": "support",
+                "meets_threshold": true,
+                "raw_confidence": 0.9811
+              },
+              "overall_strategy": "min_required_component_confidence"
+            },
+            "confidence": 0.923,
+            "decision_phase": "support",
+            "subtype": "account_help",
+            "summary": "Classified as support intent with subtype account_help in the support phase.",
+            "type": "support"
+          }
+        },
+        "fallback": {
+          "applied": true,
+          "failed_components": [],
+          "fallback_intent_type": "support",
+          "fallback_monetization_eligibility": "not_allowed",
+          "reason": "policy_default"
+        }
+      },
+      "system_decision": {
+        "intent_trajectory": [
+          "support"
+        ],
+        "opportunity": {
+          "strength": "low",
+          "type": "none"
+        },
+        "policy": {
+          "applied_thresholds": {
+            "commercial_score_min": 0.6,
+            "decision_phase_confidence_min": 0.22,
+            "intent_subtype_confidence_min": 0.25,
+            "intent_type_confidence_min": 0.4
+          },
+          "decision_basis": "fallback_policy_default",
+          "eligibility_reason": "policy_default",
+          "monetization_eligibility": "not_allowed",
+          "regulated_vertical": false,
+          "sensitivity": "high"
+        }
+      }
+    }
+  },
+  {
+    "expected_behavior": "Should trigger a sensitive fallback path and block monetization.",
+    "input": "I feel depressed and need help",
+    "response": {
+      "meta": {
+        "calibration_enabled": true,
+        "system_version": "0.6.0-phase4"
+      },
+      "model_output": {
+        "classification": {
+          "iab_content": {
+            "mapping_confidence": 0.7854,
+            "mapping_mode": "nearest_equivalent",
+            "taxonomy": "IAB Content Taxonomy",
+            "taxonomy_version": "3.0",
+            "tier1": {
+              "id": "286",
+              "label": "Medical Health"
+            },
+            "tier2": {
+              "id": "287",
+              "label": "Diseases and Conditions"
+            }
+          },
+          "intent": {
+            "commercial_score": 0.035,
+            "component_confidence": {
+              "decision_phase": {
+                "calibrated": true,
+                "confidence": 0.9699,
+                "confidence_threshold": 0.22,
+                "label": "awareness",
+                "meets_threshold": true,
+                "raw_confidence": 0.9258
+              },
+              "intent_subtype": {
+                "calibrated": true,
+                "confidence": 0.9435,
+                "confidence_threshold": 0.25,
+                "label": "emotional_reflection",
+                "meets_threshold": true,
+                "raw_confidence": 0.9435
+              },
+              "intent_type": {
+                "calibrated": true,
+                "confidence": 0.9916,
+                "confidence_threshold": 0.4,
+                "label": "personal_reflection",
+                "meets_threshold": true,
+                "raw_confidence": 0.9406
+              },
+              "overall_strategy": "min_required_component_confidence"
+            },
+            "confidence": 0.9435,
+            "decision_phase": "awareness",
+            "subtype": "emotional_reflection",
+            "summary": "Classified as personal_reflection intent with subtype emotional_reflection in the awareness phase.",
+            "type": "personal_reflection"
+          }
+        },
+        "fallback": {
+          "applied": true,
+          "failed_components": [],
+          "fallback_intent_type": "personal_reflection",
+          "fallback_monetization_eligibility": "not_allowed",
+          "reason": "policy_default"
+        }
+      },
+      "system_decision": {
+        "intent_trajectory": [
+          "awareness"
+        ],
+        "opportunity": {
+          "strength": "low",
+          "type": "none"
+        },
+        "policy": {
+          "applied_thresholds": {
+            "commercial_score_min": 0.6,
+            "decision_phase_confidence_min": 0.22,
+            "intent_subtype_confidence_min": 0.25,
+            "intent_type_confidence_min": 0.4
+          },
+          "decision_basis": "fallback_policy_default",
+          "eligibility_reason": "policy_default",
+          "monetization_eligibility": "not_allowed",
+          "regulated_vertical": false,
+          "sensitivity": "high"
+        }
+      }
+    }
+  },
+  {
+    "expected_behavior": "Should be treated as ambiguous and fall back safely.",
+    "input": "Tell me more",
+    "response": {
+      "meta": {
+        "calibration_enabled": true,
+        "system_version": "0.6.0-phase4"
+      },
+      "model_output": {
+        "classification": {
+          "iab_content": {
+            "mapping_confidence": 0.7304,
+            "mapping_mode": "nearest_equivalent",
+            "taxonomy": "IAB Content Taxonomy",
+            "taxonomy_version": "3.0",
+            "tier1": {
+              "id": "SPSHQ5",
+              "label": "Genres"
+            }
+          },
+          "intent": {
+            "commercial_score": 0.165,
+            "component_confidence": {
+              "decision_phase": {
+                "calibrated": true,
+                "confidence": 0.9934,
+                "confidence_threshold": 0.22,
+                "label": "research",
+                "meets_threshold": true,
+                "raw_confidence": 0.9746
+              },
+              "intent_subtype": {
+                "calibrated": true,
+                "confidence": 0.9631,
+                "confidence_threshold": 0.25,
+                "label": "follow_up",
+                "meets_threshold": true,
+                "raw_confidence": 0.9631
+              },
+              "intent_type": {
+                "calibrated": true,
+                "confidence": 0.9934,
+                "confidence_threshold": 0.4,
+                "label": "ambiguous",
+                "meets_threshold": true,
+                "raw_confidence": 0.9405
+              },
+              "overall_strategy": "min_required_component_confidence"
+            },
+            "confidence": 0.9934,
+            "decision_phase": "research",
+            "subtype": "follow_up",
+            "summary": "Classified as ambiguous intent with subtype follow_up in the research phase.",
+            "type": "ambiguous"
+          }
+        },
+        "fallback": {
+          "applied": true,
+          "failed_components": [],
+          "fallback_intent_type": "ambiguous",
+          "fallback_monetization_eligibility": "not_allowed",
+          "reason": "ambiguous_query"
+        }
+      },
+      "system_decision": {
+        "intent_trajectory": [
+          "research"
+        ],
+        "opportunity": {
+          "strength": "low",
+          "type": "none"
+        },
+        "policy": {
+          "applied_thresholds": {
+            "commercial_score_min": 0.6,
+            "decision_phase_confidence_min": 0.22,
+            "intent_subtype_confidence_min": 0.25,
+            "intent_type_confidence_min": 0.4
+          },
+          "decision_basis": "fallback_ambiguous_intent",
+          "eligibility_reason": "ambiguous_query",
+          "monetization_eligibility": "not_allowed",
+          "regulated_vertical": false,
+          "sensitivity": "medium"
+        }
+      }
+    }
+  },
+  {
+    "expected_behavior": "Should currently fall back because the prompt is short and context-dependent.",
+    "input": "What about pricing",
+    "response": {
+      "meta": {
+        "calibration_enabled": true,
+        "system_version": "0.6.0-phase4"
+      },
+      "model_output": {
+        "classification": {
+          "iab_content": {
+            "mapping_confidence": 0.7779,
+            "mapping_mode": "nearest_equivalent",
+            "taxonomy": "IAB Content Taxonomy",
+            "taxonomy_version": "3.0",
+            "tier1": {
+              "id": "52",
+              "label": "Business and Finance"
+            },
+            "tier2": {
+              "id": "53",
+              "label": "Business"
+            }
+          },
+          "intent": {
+            "commercial_score": 0.165,
+            "component_confidence": {
+              "decision_phase": {
+                "calibrated": true,
+                "confidence": 0.9888,
+                "confidence_threshold": 0.22,
+                "label": "research",
+                "meets_threshold": true,
+                "raw_confidence": 0.9639
+              },
+              "intent_subtype": {
+                "calibrated": true,
+                "confidence": 0.9487,
+                "confidence_threshold": 0.25,
+                "label": "follow_up",
+                "meets_threshold": true,
+                "raw_confidence": 0.9487
+              },
+              "intent_type": {
+                "calibrated": true,
+                "confidence": 0.9916,
+                "confidence_threshold": 0.4,
+                "label": "ambiguous",
+                "meets_threshold": true,
+                "raw_confidence": 0.9321
+              },
+              "overall_strategy": "min_required_component_confidence"
+            },
+            "confidence": 0.9888,
+            "decision_phase": "research",
+            "subtype": "follow_up",
+            "summary": "Classified as ambiguous intent with subtype follow_up in the research phase.",
+            "type": "ambiguous"
+          }
+        },
+        "fallback": {
+          "applied": true,
+          "failed_components": [],
+          "fallback_intent_type": "ambiguous",
+          "fallback_monetization_eligibility": "not_allowed",
+          "reason": "ambiguous_query"
+        }
+      },
+      "system_decision": {
+        "intent_trajectory": [
+          "research"
+        ],
+        "opportunity": {
+          "strength": "low",
+          "type": "none"
+        },
+        "policy": {
+          "applied_thresholds": {
+            "commercial_score_min": 0.6,
+            "decision_phase_confidence_min": 0.22,
+            "intent_subtype_confidence_min": 0.25,
+            "intent_type_confidence_min": 0.4
+          },
+          "decision_basis": "fallback_ambiguous_intent",
+          "eligibility_reason": "ambiguous_query",
+          "monetization_eligibility": "not_allowed",
+          "regulated_vertical": false,
+          "sensitivity": "medium"
+        }
+      }
+    }
+  },
+  {
+    "expected_behavior": "Should preserve transactional intent even though the phase is still noisy and fallback remains active.",
+    "input": "Sign me up for the newsletter",
+    "response": {
+      "meta": {
+        "calibration_enabled": true,
+        "system_version": "0.6.0-phase4"
+      },
+      "model_output": {
+        "classification": {
+          "iab_content": {
+            "mapping_confidence": 0.7753,
+            "mapping_mode": "nearest_equivalent",
+            "taxonomy": "IAB Content Taxonomy",
+            "taxonomy_version": "3.0",
+            "tier1": {
+              "id": "596",
+              "label": "Technology & Computing"
+            },
+            "tier2": {
+              "id": "599",
+              "label": "Computing"
+            },
+            "tier3": {
+              "id": "619",
+              "label": "Internet"
+            },
+            "tier4": {
+              "id": "623",
+              "label": "Email"
+            }
+          },
+          "intent": {
+            "commercial_score": 0.954,
+            "component_confidence": {
+              "decision_phase": {
+                "calibrated": true,
+                "confidence": 0.9991,
+                "confidence_threshold": 0.22,
+                "label": "action",
+                "meets_threshold": true,
+                "raw_confidence": 0.9948
+              },
+              "intent_subtype": {
+                "calibrated": true,
+                "confidence": 0.8874,
+                "confidence_threshold": 0.25,
+                "label": "signup",
+                "meets_threshold": true,
+                "raw_confidence": 0.8874
+              },
+              "intent_type": {
+                "calibrated": true,
+                "confidence": 0.9996,
+                "confidence_threshold": 0.4,
+                "label": "transactional",
+                "meets_threshold": true,
+                "raw_confidence": 0.9908
+              },
+              "overall_strategy": "min_required_component_confidence"
+            },
+            "confidence": 0.8874,
+            "decision_phase": "action",
+            "subtype": "signup",
+            "summary": "Classified as transactional intent with subtype signup in the action phase.",
+            "type": "transactional"
+          }
+        },
+        "fallback": null
+      },
+      "system_decision": {
+        "intent_trajectory": [
+          "action"
+        ],
+        "opportunity": {
+          "strength": "high",
+          "type": "transaction_trigger"
+        },
+        "policy": {
+          "applied_thresholds": {
+            "commercial_score_min": 0.6,
+            "decision_phase_confidence_min": 0.22,
+            "intent_subtype_confidence_min": 0.25,
+            "intent_type_confidence_min": 0.4
+          },
+          "decision_basis": "score_threshold",
+          "eligibility_reason": "high_intent_subtype_signal",
+          "monetization_eligibility": "allowed",
+          "regulated_vertical": false,
+          "sensitivity": "low"
+        }
+      }
+    }
+  },
+  {
+    "expected_behavior": "Should behave like a commercial comparison in the middle of the buying journey and still fail safe if confidence is low.",
+    "input": "Compare AI search monetization platforms for publishers",
+    "response": {
+      "meta": {
+        "calibration_enabled": true,
+        "system_version": "0.6.0-phase4"
+      },
+      "model_output": {
+        "classification": {
+          "iab_content": {
+            "mapping_confidence": 0.8626,
+            "mapping_mode": "nearest_equivalent",
+            "taxonomy": "IAB Content Taxonomy",
+            "taxonomy_version": "3.0",
+            "tier1": {
+              "id": "596",
+              "label": "Technology & Computing"
+            },
+            "tier2": {
+              "id": "599",
+              "label": "Computing"
+            },
+            "tier3": {
+              "id": "619",
+              "label": "Internet"
+            },
+            "tier4": {
+              "id": "627",
+              "label": "Search"
+            }
+          },
+          "intent": {
+            "commercial_score": 0.728,
+            "component_confidence": {
+              "decision_phase": {
+                "calibrated": true,
+                "confidence": 0.9966,
+                "confidence_threshold": 0.22,
+                "label": "consideration",
+                "meets_threshold": true,
+                "raw_confidence": 0.9852
+              },
+              "intent_subtype": {
+                "calibrated": true,
+                "confidence": 0.9415,
+                "confidence_threshold": 0.25,
+                "label": "comparison",
+                "meets_threshold": true,
+                "raw_confidence": 0.9415
+              },
+              "intent_type": {
+                "calibrated": true,
+                "confidence": 0.9994,
+                "confidence_threshold": 0.4,
+                "label": "commercial",
+                "meets_threshold": true,
+                "raw_confidence": 0.9884
+              },
+              "overall_strategy": "min_required_component_confidence"
+            },
+            "confidence": 0.9415,
+            "decision_phase": "consideration",
+            "subtype": "comparison",
+            "summary": "Classified as commercial intent with subtype comparison in the consideration phase.",
+            "type": "commercial"
+          }
+        },
+        "fallback": null
+      },
+      "system_decision": {
+        "intent_trajectory": [
+          "consideration"
+        ],
+        "opportunity": {
+          "strength": "high",
+          "type": "comparison_slot"
+        },
+        "policy": {
+          "applied_thresholds": {
+            "commercial_score_min": 0.6,
+            "decision_phase_confidence_min": 0.22,
+            "intent_subtype_confidence_min": 0.25,
+            "intent_type_confidence_min": 0.4
+          },
+          "decision_basis": "score_threshold",
+          "eligibility_reason": "commercial_comparison_signal_present",
+          "monetization_eligibility": "allowed_with_caution",
+          "regulated_vertical": false,
+          "sensitivity": "low"
+        }
+      }
+    }
+  },
+  {
+    "expected_behavior": "Should stay in informational awareness and remain non-monetized under low confidence.",
+    "input": "Why do businesses use CRM systems?",
+    "response": {
+      "meta": {
+        "calibration_enabled": true,
+        "system_version": "0.6.0-phase4"
+      },
+      "model_output": {
+        "classification": {
+          "iab_content": {
+            "mapping_confidence": 0.8741,
+            "mapping_mode": "nearest_equivalent",
+            "taxonomy": "IAB Content Taxonomy",
+            "taxonomy_version": "3.0",
+            "tier1": {
+              "id": "596",
+              "label": "Technology & Computing"
+            },
+            "tier2": {
+              "id": "599",
+              "label": "Computing"
+            },
+            "tier3": {
+              "id": "619",
+              "label": "Internet"
+            },
+            "tier4": {
+              "id": "620",
+              "label": "Cloud Computing"
+            }
+          },
+          "intent": {
+            "commercial_score": 0.0875,
+            "component_confidence": {
+              "decision_phase": {
+                "calibrated": true,
+                "confidence": 0.9939,
+                "confidence_threshold": 0.22,
+                "label": "awareness",
+                "meets_threshold": true,
+                "raw_confidence": 0.9764
+              },
+              "intent_subtype": {
+                "calibrated": true,
+                "confidence": 0.9545,
+                "confidence_threshold": 0.25,
+                "label": "education",
+                "meets_threshold": true,
+                "raw_confidence": 0.9545
+              },
+              "intent_type": {
+                "calibrated": true,
+                "confidence": 0.9964,
+                "confidence_threshold": 0.4,
+                "label": "informational",
+                "meets_threshold": true,
+                "raw_confidence": 0.961
+              },
+              "overall_strategy": "min_required_component_confidence"
+            },
+            "confidence": 0.9939,
+            "decision_phase": "awareness",
+            "subtype": "education",
+            "summary": "Classified as informational intent with subtype education in the awareness phase.",
+            "type": "informational"
+          }
+        },
+        "fallback": null
+      },
+      "system_decision": {
+        "intent_trajectory": [
+          "awareness"
+        ],
+        "opportunity": {
+          "strength": "low",
+          "type": "none"
+        },
+        "policy": {
+          "applied_thresholds": {
+            "commercial_score_min": 0.6,
+            "decision_phase_confidence_min": 0.22,
+            "intent_subtype_confidence_min": 0.25,
+            "intent_type_confidence_min": 0.4
+          },
+          "decision_basis": "score_threshold",
+          "eligibility_reason": "commercial_signal_below_threshold",
+          "monetization_eligibility": "restricted",
+          "regulated_vertical": false,
+          "sensitivity": "low"
+        }
+      }
+    }
+  }
+]

artifacts/evaluation/latest/decision_phase_difficulty_benchmark_confusion_matrix.csv

@@ -0,0 +1,8 @@
+,awareness,research,consideration,decision,action,post_purchase,support
+awareness,14,1,0,0,0,0,0
+research,0,14,0,0,0,1,0
+consideration,0,1,14,0,0,0,0
+decision,0,0,0,15,0,0,0
+action,0,0,0,1,13,1,0
+post_purchase,0,0,0,0,0,15,0
+support,0,0,0,0,0,0,15

artifacts/evaluation/latest/decision_phase_difficulty_benchmark_report.json

@@ -0,0 +1,95 @@
+{
+  "accepted_accuracy": 0.9524,
+  "accepted_coverage": 1.0,
+  "accuracy": 0.9524,
+  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/decision_phase_difficulty_benchmark_confusion_matrix.csv",
+  "count": 105,
+  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/decision_phase_benchmark.jsonl",
+  "difficulty_breakdown": {
+    "easy": {
+      "accepted_accuracy": 0.9714,
+      "accepted_coverage": 1.0,
+      "accuracy": 0.9714,
+      "count": 35,
+      "fallback_rate": 0.0,
+      "macro_f1": 0.9711
+    },
+    "hard": {
+      "accepted_accuracy": 0.8857,
+      "accepted_coverage": 1.0,
+      "accuracy": 0.8857,
+      "count": 35,
+      "fallback_rate": 0.0,
+      "macro_f1": 0.883
+    },
+    "medium": {
+      "accepted_accuracy": 1.0,
+      "accepted_coverage": 1.0,
+      "accuracy": 1.0,
+      "count": 35,
+      "fallback_rate": 0.0,
+      "macro_f1": 1.0
+    }
+  },
+  "fallback_rate": 0.0,
+  "head": "decision_phase",
+  "macro_f1": 0.9526,
+  "per_class_metrics": {
+    "accuracy": 0.9523809523809523,
+    "action": {
+      "f1-score": 0.9285714285714286,
+      "precision": 1.0,
+      "recall": 0.8666666666666667,
+      "support": 15.0
+    },
+    "awareness": {
+      "f1-score": 0.9655172413793104,
+      "precision": 1.0,
+      "recall": 0.9333333333333333,
+      "support": 15.0
+    },
+    "consideration": {
+      "f1-score": 0.9655172413793104,
+      "precision": 1.0,
+      "recall": 0.9333333333333333,
+      "support": 15.0
+    },
+    "decision": {
+      "f1-score": 0.967741935483871,
+      "precision": 0.9375,
+      "recall": 1.0,
+      "support": 15.0
+    },
+    "macro avg": {
+      "f1-score": 0.9525819504665047,
+      "precision": 0.9564075630252101,
+      "recall": 0.9523809523809523,
+      "support": 105.0
+    },
+    "post_purchase": {
+      "f1-score": 0.9375,
+      "precision": 0.8823529411764706,
+      "recall": 1.0,
+      "support": 15.0
+    },
+    "research": {
+      "f1-score": 0.9032258064516129,
+      "precision": 0.875,
+      "recall": 0.9333333333333333,
+      "support": 15.0
+    },
+    "support": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 15.0
+    },
+    "weighted avg": {
+      "f1-score": 0.9525819504665048,
+      "precision": 0.9564075630252101,
+      "recall": 0.9523809523809523,
+      "support": 105.0
+    }
+  },
+  "suite": "difficulty_benchmark"
+}

artifacts/evaluation/latest/decision_phase_final_wave_cases_confusion_matrix.csv

@@ -0,0 +1,8 @@
+,awareness,research,consideration,decision,action,post_purchase,support
+awareness,5,0,0,0,0,0,0
+research,1,3,0,0,0,0,0
+consideration,0,0,5,0,0,0,0
+decision,0,0,0,5,0,0,0
+action,0,0,0,0,0,0,0
+post_purchase,0,0,0,0,0,4,0
+support,0,0,0,0,0,0,4

artifacts/evaluation/latest/decision_phase_final_wave_cases_report.json

@@ -0,0 +1,69 @@
+{
+  "accepted_accuracy": 0.963,
+  "accepted_coverage": 1.0,
+  "accuracy": 0.963,
+  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/decision_phase_final_wave_cases_confusion_matrix.csv",
+  "count": 27,
+  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/decision_phase/final_wave_cases.jsonl",
+  "fallback_rate": 0.0,
+  "head": "decision_phase",
+  "macro_f1": 0.961,
+  "per_class_metrics": {
+    "accuracy": 0.9629629629629629,
+    "action": {
+      "f1-score": 0.0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "support": 0.0
+    },
+    "awareness": {
+      "f1-score": 0.9090909090909091,
+      "precision": 0.8333333333333334,
+      "recall": 1.0,
+      "support": 5.0
+    },
+    "consideration": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 5.0
+    },
+    "decision": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 5.0
+    },
+    "macro avg": {
+      "f1-score": 0.823747680890538,
+      "precision": 0.8333333333333334,
+      "recall": 0.8214285714285714,
+      "support": 27.0
+    },
+    "post_purchase": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 4.0
+    },
+    "research": {
+      "f1-score": 0.8571428571428571,
+      "precision": 1.0,
+      "recall": 0.75,
+      "support": 4.0
+    },
+    "support": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 4.0
+    },
+    "weighted avg": {
+      "f1-score": 0.962000962000962,
+      "precision": 0.9691358024691359,
+      "recall": 0.9629629629629629,
+      "support": 27.0
+    }
+  },
+  "suite": "final_wave_cases"
+}

artifacts/evaluation/latest/decision_phase_hard_cases_confusion_matrix.csv

@@ -0,0 +1,8 @@
+,awareness,research,consideration,decision,action,post_purchase,support
+awareness,6,0,0,0,0,0,0
+research,2,5,0,0,0,0,0
+consideration,0,1,6,0,0,0,0
+decision,0,0,0,7,0,0,0
+action,0,0,0,0,0,0,0
+post_purchase,0,0,0,0,0,6,0
+support,0,0,0,0,0,0,6

artifacts/evaluation/latest/decision_phase_hard_cases_report.json

@@ -0,0 +1,69 @@
+{
+  "accepted_accuracy": 0.9231,
+  "accepted_coverage": 1.0,
+  "accuracy": 0.9231,
+  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/decision_phase_hard_cases_confusion_matrix.csv",
+  "count": 39,
+  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/decision_phase/hard_cases.jsonl",
+  "fallback_rate": 0.0,
+  "head": "decision_phase",
+  "macro_f1": 0.9249,
+  "per_class_metrics": {
+    "accuracy": 0.9230769230769231,
+    "action": {
+      "f1-score": 0.0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "support": 0.0
+    },
+    "awareness": {
+      "f1-score": 0.8571428571428571,
+      "precision": 0.75,
+      "recall": 1.0,
+      "support": 6.0
+    },
+    "consideration": {
+      "f1-score": 0.9230769230769231,
+      "precision": 1.0,
+      "recall": 0.8571428571428571,
+      "support": 7.0
+    },
+    "decision": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 7.0
+    },
+    "macro avg": {
+      "f1-score": 0.792778649921507,
+      "precision": 0.7976190476190477,
+      "recall": 0.7959183673469388,
+      "support": 39.0
+    },
+    "post_purchase": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 6.0
+    },
+    "research": {
+      "f1-score": 0.7692307692307693,
+      "precision": 0.8333333333333334,
+      "recall": 0.7142857142857143,
+      "support": 7.0
+    },
+    "support": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 6.0
+    },
+    "weighted avg": {
+      "f1-score": 0.9227951535643845,
+      "precision": 0.9316239316239316,
+      "recall": 0.9230769230769231,
+      "support": 39.0
+    }
+  },
+  "suite": "hard_cases"
+}

artifacts/evaluation/latest/decision_phase_test_confusion_matrix.csv

@@ -0,0 +1,8 @@
+,awareness,research,consideration,decision,action,post_purchase,support
+awareness,3,0,0,0,0,0,0
+research,3,2,0,0,0,0,0
+consideration,0,1,4,0,0,0,0
+decision,0,0,0,5,0,0,0
+action,0,0,0,0,3,0,0
+post_purchase,0,0,0,0,0,4,0
+support,0,0,0,0,0,0,4

artifacts/evaluation/latest/decision_phase_test_report.json

@@ -0,0 +1,69 @@
+{
+  "accepted_accuracy": 0.8621,
+  "accepted_coverage": 1.0,
+  "accuracy": 0.8621,
+  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/decision_phase_test_confusion_matrix.csv",
+  "count": 29,
+  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/decision_phase/test.jsonl",
+  "fallback_rate": 0.0,
+  "head": "decision_phase",
+  "macro_f1": 0.8651,
+  "per_class_metrics": {
+    "accuracy": 0.8620689655172413,
+    "action": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 3.0
+    },
+    "awareness": {
+      "f1-score": 0.6666666666666666,
+      "precision": 0.5,
+      "recall": 1.0,
+      "support": 3.0
+    },
+    "consideration": {
+      "f1-score": 0.8888888888888888,
+      "precision": 1.0,
+      "recall": 0.8,
+      "support": 5.0
+    },
+    "decision": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 5.0
+    },
+    "macro avg": {
+      "f1-score": 0.865079365079365,
+      "precision": 0.8809523809523808,
+      "recall": 0.8857142857142858,
+      "support": 29.0
+    },
+    "post_purchase": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 4.0
+    },
+    "research": {
+      "f1-score": 0.5,
+      "precision": 0.6666666666666666,
+      "recall": 0.4,
+      "support": 5.0
+    },
+    "support": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 4.0
+    },
+    "weighted avg": {
+      "f1-score": 0.8601532567049808,
+      "precision": 0.8908045977011494,
+      "recall": 0.8620689655172413,
+      "support": 29.0
+    }
+  },
+  "suite": "test"
+}

artifacts/evaluation/latest/decision_phase_train_confusion_matrix.csv

@@ -0,0 +1,8 @@
+,awareness,research,consideration,decision,action,post_purchase,support
+awareness,16,0,0,0,0,0,0
+research,1,14,0,0,0,0,0
+consideration,0,0,17,0,0,0,0
+decision,0,0,0,16,0,0,0
+action,0,0,0,0,10,0,0
+post_purchase,0,0,0,0,0,14,0
+support,0,0,0,0,0,0,14

artifacts/evaluation/latest/decision_phase_train_report.json

@@ -0,0 +1,69 @@
+{
+  "accepted_accuracy": 0.9902,
+  "accepted_coverage": 1.0,
+  "accuracy": 0.9902,
+  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/decision_phase_train_confusion_matrix.csv",
+  "count": 102,
+  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/decision_phase/train.jsonl",
+  "fallback_rate": 0.0,
+  "head": "decision_phase",
+  "macro_f1": 0.9907,
+  "per_class_metrics": {
+    "accuracy": 0.9901960784313726,
+    "action": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 10.0
+    },
+    "awareness": {
+      "f1-score": 0.9696969696969697,
+      "precision": 0.9411764705882353,
+      "recall": 1.0,
+      "support": 16.0
+    },
+    "consideration": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 17.0
+    },
+    "decision": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 16.0
+    },
+    "macro avg": {
+      "f1-score": 0.9907448872966115,
+      "precision": 0.9915966386554622,
+      "recall": 0.9904761904761905,
+      "support": 102.0
+    },
+    "post_purchase": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 14.0
+    },
+    "research": {
+      "f1-score": 0.9655172413793104,
+      "precision": 1.0,
+      "recall": 0.9333333333333333,
+      "support": 15.0
+    },
+    "support": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 14.0
+    },
+    "weighted avg": {
+      "f1-score": 0.9901755895670704,
+      "precision": 0.9907727797001153,
+      "recall": 0.9901960784313726,
+      "support": 102.0
+    }
+  },
+  "suite": "train"
+}

artifacts/evaluation/latest/decision_phase_val_confusion_matrix.csv

@@ -0,0 +1,8 @@
+,awareness,research,consideration,decision,action,post_purchase,support
+awareness,5,0,0,0,0,0,0
+research,1,3,0,0,0,0,0
+consideration,0,0,5,0,0,0,0
+decision,0,0,1,3,0,0,0
+action,0,0,0,0,3,0,0
+post_purchase,0,1,0,0,0,3,0
+support,0,0,0,0,0,0,4

artifacts/evaluation/latest/decision_phase_val_report.json

@@ -0,0 +1,69 @@
+{
+  "accepted_accuracy": 0.8966,
+  "accepted_coverage": 1.0,
+  "accuracy": 0.8966,
+  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/decision_phase_val_confusion_matrix.csv",
+  "count": 29,
+  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/decision_phase/val.jsonl",
+  "fallback_rate": 0.0,
+  "head": "decision_phase",
+  "macro_f1": 0.8975,
+  "per_class_metrics": {
+    "accuracy": 0.896551724137931,
+    "action": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 3.0
+    },
+    "awareness": {
+      "f1-score": 0.9090909090909091,
+      "precision": 0.8333333333333334,
+      "recall": 1.0,
+      "support": 5.0
+    },
+    "consideration": {
+      "f1-score": 0.9090909090909091,
+      "precision": 0.8333333333333334,
+      "recall": 1.0,
+      "support": 5.0
+    },
+    "decision": {
+      "f1-score": 0.8571428571428571,
+      "precision": 1.0,
+      "recall": 0.75,
+      "support": 4.0
+    },
+    "macro avg": {
+      "f1-score": 0.8974953617810761,
+      "precision": 0.9166666666666667,
+      "recall": 0.8928571428571429,
+      "support": 29.0
+    },
+    "post_purchase": {
+      "f1-score": 0.8571428571428571,
+      "precision": 1.0,
+      "recall": 0.75,
+      "support": 4.0
+    },
+    "research": {
+      "f1-score": 0.75,
+      "precision": 0.75,
+      "recall": 0.75,
+      "support": 4.0
+    },
+    "support": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 4.0
+    },
+    "weighted avg": {
+      "f1-score": 0.8947604120017911,
+      "precision": 0.9080459770114944,
+      "recall": 0.896551724137931,
+      "support": 29.0
+    }
+  },
+  "suite": "val"
+}

artifacts/evaluation/latest/iab_behavior_lock_regression.json

@@ -0,0 +1,245 @@
+{
+  "by_status": {
+    "must_fix": {
+      "failed": 0,
+      "passed": 12,
+      "total": 12
+    }
+  },
+  "cases_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/examples/iab_behavior_lock_cases.json",
+  "count": 12,
+  "failed": 0,
+  "passed": 12,
+  "results": [
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Automotive",
+        "model_output.classification.iab_content.tier2.label": "Auto Type"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Automotive",
+        "model_output.classification.iab_content.tier2.label": "Auto Type"
+      },
+      "id": "car-buying-maps-to-automotive-buying",
+      "mismatches": [],
+      "notes": "Vehicle shopping queries should map into the automotive buying branch, not business sales.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Which car to buy in 2026"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Laptops"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Laptops"
+      },
+      "id": "laptop-buying-maps-to-laptops",
+      "mismatches": [],
+      "notes": "Laptop shopping should resolve into the laptops branch, not business sales.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Which laptop to buy in 2026"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Laptops"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Laptops"
+      },
+      "id": "labtop-buying-maps-to-laptops",
+      "mismatches": [],
+      "notes": "Common typo handling should still land in the laptops branch.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Which labtop to buy in 2026"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+      },
+      "id": "crm-awareness-maps-to-sales",
+      "mismatches": [],
+      "notes": "CRM education should resolve to the closest business/sales path, not generic software.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "What is CRM software?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "id": "crm-comparison-maps-to-sales",
+      "mismatches": [],
+      "notes": "Direct CRM vendor comparison should map cleanly into the sales domain.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "HubSpot vs Zoho for a small team"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "id": "marketing-tools-map-to-marketing",
+      "mismatches": [],
+      "notes": "Marketing tool discovery should map to the marketing and advertising branch.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best AI SEO tools for content teams"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing"
+      },
+      "id": "ml-explanation-maps-to-ai",
+      "mismatches": [],
+      "notes": "ML and NLP educational prompts should land in the AI branch.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "What is intent classification in NLP?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "id": "support-credential-help-maps-to-business-it",
+      "mismatches": [],
+      "notes": "Credential and account help should map to business IT rather than generic business.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "How do I reset my password?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink",
+        "model_output.classification.iab_content.tier2.label": "Dining Out"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink",
+        "model_output.classification.iab_content.tier2.label": "Dining Out"
+      },
+      "id": "restaurant-booking-maps-to-dining-out",
+      "mismatches": [],
+      "notes": "Generic dining requests should not inherit the repo's business default.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Book a table for 2 tonight"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Hobbies & Interests",
+        "model_output.classification.iab_content.tier2.label": "Content Production",
+        "model_output.classification.iab_content.tier3.label": "Freelance Writing"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Hobbies & Interests",
+        "model_output.classification.iab_content.tier2.label": "Content Production",
+        "model_output.classification.iab_content.tier3.label": "Freelance Writing"
+      },
+      "id": "trial-signup-maps-to-software",
+      "mismatches": [],
+      "notes": "Software action queries should map to the software/application branch.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Start my free trial"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications",
+        "model_output.classification.iab_content.tier4.label": "Communication"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications",
+        "model_output.classification.iab_content.tier4.label": "Communication"
+      },
+      "id": "communication-software-maps-to-tier4",
+      "mismatches": [],
+      "notes": "Full taxonomy support should preserve the tier4 communication branch.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "best communication software for remote teams"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink"
+      },
+      "id": "vodka-query-maps-to-alcoholic-beverages",
+      "mismatches": [],
+      "notes": "Food and beverage prompts should not fall through to the business default.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "what is best vodka drink should i try"
+    }
+  ]
+}

artifacts/evaluation/latest/iab_content_cross_vertical_benchmark_report.json

@@ -0,0 +1,93 @@
+{
+  "accepted_accuracy": 0.3444,
+  "accepted_coverage": 1.0,
+  "accuracy": 0.3444,
+  "count": 90,
+  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/iab_cross_vertical_benchmark.jsonl",
+  "difficulty_breakdown": {
+    "easy": {
+      "accepted_accuracy": 0.2667,
+      "accepted_coverage": 1.0,
+      "accuracy": 0.2667,
+      "count": 30,
+      "fallback_rate": 0.0,
+      "macro_f1": 0.1633
+    },
+    "hard": {
+      "accepted_accuracy": 0.3667,
+      "accepted_coverage": 1.0,
+      "accuracy": 0.3667,
+      "count": 30,
+      "fallback_rate": 0.0,
+      "macro_f1": 0.2174
+    },
+    "medium": {
+      "accepted_accuracy": 0.4,
+      "accepted_coverage": 1.0,
+      "accuracy": 0.4,
+      "count": 30,
+      "fallback_rate": 0.0,
+      "macro_f1": 0.2667
+    }
+  },
+  "fallback_rate": 0.0,
+  "head": "iab_content",
+  "macro_f1": 0.1808,
+  "primary_source": "embedding_retrieval",
+  "suite": "cross_vertical_benchmark",
+  "tier_metrics": {
+    "average_prediction_depth": 2.5333,
+    "error_buckets": {
+      "exact_match": 31,
+      "parent_safe_stop": 5,
+      "right_tier1_wrong_tier2": 19,
+      "wrong_deep_leaf": 13,
+      "wrong_tier1": 22
+    },
+    "exact_path_accuracy": 0.3444,
+    "parent_safe_accuracy": 0.4889,
+    "tier1_accuracy": 0.7556,
+    "tier2_accuracy": 0.5238,
+    "tier3_accuracy": 0.4762,
+    "tier4_accuracy": 1.0
+  },
+  "view_metrics": {
+    "combined_path": {
+      "average_prediction_depth": 2.5333,
+      "error_buckets": {
+        "exact_match": 27,
+        "parent_safe_stop": 5,
+        "right_tier1_wrong_tier2": 19,
+        "wrong_deep_leaf": 17,
+        "wrong_tier1": 22
+      },
+      "exact_path_accuracy": 0.3,
+      "fallback_overuse_count": 12,
+      "fallback_rate": 0.1333,
+      "parent_safe_accuracy": 0.4444,
+      "tier1_accuracy": 0.7556,
+      "tier2_accuracy": 0.5238,
+      "tier3_accuracy": 0.381,
+      "tier4_accuracy": 0.5
+    },
+    "disagreements": {
+      "retrieval_vs_combined": 0
+    },
+    "embedding_retrieval": {
+      "average_prediction_depth": 2.5333,
+      "error_buckets": {
+        "exact_match": 27,
+        "parent_safe_stop": 5,
+        "right_tier1_wrong_tier2": 19,
+        "wrong_deep_leaf": 17,
+        "wrong_tier1": 22
+      },
+      "exact_path_accuracy": 0.3,
+      "parent_safe_accuracy": 0.4444,
+      "tier1_accuracy": 0.7556,
+      "tier2_accuracy": 0.5238,
+      "tier3_accuracy": 0.381,
+      "tier4_accuracy": 0.5
+    }
+  }
+}

artifacts/evaluation/latest/iab_content_difficulty_benchmark_report.json

@@ -0,0 +1,93 @@
+{
+  "accepted_accuracy": 0.3782,
+  "accepted_coverage": 1.0,
+  "accuracy": 0.3782,
+  "count": 156,
+  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/iab_benchmark.jsonl",
+  "difficulty_breakdown": {
+    "easy": {
+      "accepted_accuracy": 0.4038,
+      "accepted_coverage": 1.0,
+      "accuracy": 0.4038,
+      "count": 52,
+      "fallback_rate": 0.0,
+      "macro_f1": 0.2171
+    },
+    "hard": {
+      "accepted_accuracy": 0.3077,
+      "accepted_coverage": 1.0,
+      "accuracy": 0.3077,
+      "count": 52,
+      "fallback_rate": 0.0,
+      "macro_f1": 0.1626
+    },
+    "medium": {
+      "accepted_accuracy": 0.4231,
+      "accepted_coverage": 1.0,
+      "accuracy": 0.4231,
+      "count": 52,
+      "fallback_rate": 0.0,
+      "macro_f1": 0.2265
+    }
+  },
+  "fallback_rate": 0.0,
+  "head": "iab_content",
+  "macro_f1": 0.1593,
+  "primary_source": "embedding_retrieval",
+  "suite": "difficulty_benchmark",
+  "tier_metrics": {
+    "average_prediction_depth": 2.5833,
+    "error_buckets": {
+      "exact_match": 59,
+      "parent_safe_stop": 17,
+      "right_tier1_wrong_tier2": 42,
+      "wrong_deep_leaf": 13,
+      "wrong_tier1": 25
+    },
+    "exact_path_accuracy": 0.3782,
+    "parent_safe_accuracy": 0.6154,
+    "tier1_accuracy": 0.8397,
+    "tier2_accuracy": 0.5705,
+    "tier3_accuracy": 0.5648,
+    "tier4_accuracy": 0.5833
+  },
+  "view_metrics": {
+    "combined_path": {
+      "average_prediction_depth": 2.5833,
+      "error_buckets": {
+        "exact_match": 48,
+        "parent_safe_stop": 17,
+        "right_tier1_wrong_tier2": 42,
+        "wrong_deep_leaf": 24,
+        "wrong_tier1": 25
+      },
+      "exact_path_accuracy": 0.3077,
+      "fallback_overuse_count": 11,
+      "fallback_rate": 0.0705,
+      "parent_safe_accuracy": 0.5449,
+      "tier1_accuracy": 0.8397,
+      "tier2_accuracy": 0.5705,
+      "tier3_accuracy": 0.4352,
+      "tier4_accuracy": 0.25
+    },
+    "disagreements": {
+      "retrieval_vs_combined": 0
+    },
+    "embedding_retrieval": {
+      "average_prediction_depth": 2.5833,
+      "error_buckets": {
+        "exact_match": 48,
+        "parent_safe_stop": 17,
+        "right_tier1_wrong_tier2": 42,
+        "wrong_deep_leaf": 24,
+        "wrong_tier1": 25
+      },
+      "exact_path_accuracy": 0.3077,
+      "parent_safe_accuracy": 0.5449,
+      "tier1_accuracy": 0.8397,
+      "tier2_accuracy": 0.5705,
+      "tier3_accuracy": 0.4352,
+      "tier4_accuracy": 0.25
+    }
+  }
+}

artifacts/evaluation/latest/iab_content_extended_cases_report.json

@@ -0,0 +1,64 @@
+{
+  "accepted_accuracy": 0.25,
+  "accepted_coverage": 1.0,
+  "accuracy": 0.25,
+  "count": 8,
+  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/iab/extended_cases.jsonl",
+  "fallback_rate": 0.0,
+  "head": "iab_content",
+  "macro_f1": 0.1429,
+  "primary_source": "embedding_retrieval",
+  "suite": "extended_cases",
+  "tier_metrics": {
+    "average_prediction_depth": 2.375,
+    "error_buckets": {
+      "exact_match": 2,
+      "right_tier1_wrong_tier2": 3,
+      "wrong_deep_leaf": 2,
+      "wrong_tier1": 1
+    },
+    "exact_path_accuracy": 0.25,
+    "parent_safe_accuracy": 0.375,
+    "tier1_accuracy": 0.875,
+    "tier2_accuracy": 0.4286,
+    "tier3_accuracy": 1.0,
+    "tier4_accuracy": 0.0
+  },
+  "view_metrics": {
+    "combined_path": {
+      "average_prediction_depth": 2.375,
+      "error_buckets": {
+        "exact_match": 2,
+        "right_tier1_wrong_tier2": 3,
+        "wrong_deep_leaf": 2,
+        "wrong_tier1": 1
+      },
+      "exact_path_accuracy": 0.25,
+      "fallback_overuse_count": 1,
+      "fallback_rate": 0.125,
+      "parent_safe_accuracy": 0.375,
+      "tier1_accuracy": 0.875,
+      "tier2_accuracy": 0.4286,
+      "tier3_accuracy": 0.0,
+      "tier4_accuracy": 0.0
+    },
+    "disagreements": {
+      "retrieval_vs_combined": 0
+    },
+    "embedding_retrieval": {
+      "average_prediction_depth": 2.375,
+      "error_buckets": {
+        "exact_match": 2,
+        "right_tier1_wrong_tier2": 3,
+        "wrong_deep_leaf": 2,
+        "wrong_tier1": 1
+      },
+      "exact_path_accuracy": 0.25,
+      "parent_safe_accuracy": 0.375,
+      "tier1_accuracy": 0.875,
+      "tier2_accuracy": 0.4286,
+      "tier3_accuracy": 0.0,
+      "tier4_accuracy": 0.0
+    }
+  }
+}

artifacts/evaluation/latest/iab_content_hard_cases_report.json

@@ -0,0 +1,66 @@
+{
+  "accepted_accuracy": 0.25,
+  "accepted_coverage": 1.0,
+  "accuracy": 0.25,
+  "count": 8,
+  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/iab/hard_cases.jsonl",
+  "fallback_rate": 0.0,
+  "head": "iab_content",
+  "macro_f1": 0.1429,
+  "primary_source": "embedding_retrieval",
+  "suite": "hard_cases",
+  "tier_metrics": {
+    "average_prediction_depth": 2.375,
+    "error_buckets": {
+      "exact_match": 2,
+      "parent_safe_stop": 1,
+      "right_tier1_wrong_tier2": 2,
+      "wrong_tier1": 3
+    },
+    "exact_path_accuracy": 0.25,
+    "parent_safe_accuracy": 0.5,
+    "tier1_accuracy": 0.625,
+    "tier2_accuracy": 0.375,
+    "tier3_accuracy": 0.2,
+    "tier4_accuracy": 1.0
+  },
+  "view_metrics": {
+    "combined_path": {
+      "average_prediction_depth": 2.375,
+      "error_buckets": {
+        "exact_match": 1,
+        "parent_safe_stop": 1,
+        "right_tier1_wrong_tier2": 2,
+        "wrong_deep_leaf": 1,
+        "wrong_tier1": 3
+      },
+      "exact_path_accuracy": 0.125,
+      "fallback_overuse_count": 1,
+      "fallback_rate": 0.125,
+      "parent_safe_accuracy": 0.375,
+      "tier1_accuracy": 0.625,
+      "tier2_accuracy": 0.375,
+      "tier3_accuracy": 0.0,
+      "tier4_accuracy": 0.0
+    },
+    "disagreements": {
+      "retrieval_vs_combined": 0
+    },
+    "embedding_retrieval": {
+      "average_prediction_depth": 2.375,
+      "error_buckets": {
+        "exact_match": 1,
+        "parent_safe_stop": 1,
+        "right_tier1_wrong_tier2": 2,
+        "wrong_deep_leaf": 1,
+        "wrong_tier1": 3
+      },
+      "exact_path_accuracy": 0.125,
+      "parent_safe_accuracy": 0.375,
+      "tier1_accuracy": 0.625,
+      "tier2_accuracy": 0.375,
+      "tier3_accuracy": 0.0,
+      "tier4_accuracy": 0.0
+    }
+  }
+}

artifacts/evaluation/latest/iab_content_test_report.json

@@ -0,0 +1,58 @@
+{
+  "accepted_accuracy": 0.6527,
+  "accepted_coverage": 1.0,
+  "accuracy": 0.6527,
+  "count": 3282,
+  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/iab/test.jsonl",
+  "fallback_rate": 0.0,
+  "head": "iab_content",
+  "macro_f1": 0.6922,
+  "primary_source": "embedding_retrieval",
+  "suite": "test",
+  "tier_metrics": {
+    "average_prediction_depth": 2.1889,
+    "error_buckets": {
+      "exact_match": 2142,
+      "parent_safe_stop": 115,
+      "right_tier1_wrong_tier2": 674,
+      "wrong_deep_leaf": 236,
+      "wrong_tier1": 115
+    },
+    "exact_path_accuracy": 0.6527,
+    "parent_safe_accuracy": 0.7721,
+    "tier1_accuracy": 0.965,
+    "tier2_accuracy": 0.7587,
+    "tier3_accuracy": 0.8041,
+    "tier4_accuracy": 0.7929
+  },
+  "view_metrics": {
+    "combined_path": {
+      "count": 3282,
+      "max_combined_rows": 500,
+      "reason": "dataset_too_large_for_combined_view",
+      "skipped": true
+    },
+    "disagreements": {
+      "count": 3282,
+      "max_combined_rows": 500,
+      "reason": "dataset_too_large_for_combined_view",
+      "skipped": true
+    },
+    "embedding_retrieval": {
+      "average_prediction_depth": 2.1889,
+      "error_buckets": {
+        "exact_match": 2107,
+        "parent_safe_stop": 109,
+        "right_tier1_wrong_tier2": 680,
+        "wrong_deep_leaf": 271,
+        "wrong_tier1": 115
+      },
+      "exact_path_accuracy": 0.642,
+      "parent_safe_accuracy": 0.7596,
+      "tier1_accuracy": 0.965,
+      "tier2_accuracy": 0.7566,
+      "tier3_accuracy": 0.7679,
+      "tier4_accuracy": 0.6071
+    }
+  }
+}

artifacts/evaluation/latest/iab_content_train_report.json

@@ -0,0 +1,67 @@
+{
+  "accepted_accuracy": 0.8115,
+  "accepted_coverage": 1.0,
+  "accuracy": 0.8115,
+  "count": 13211,
+  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/iab/train.jsonl",
+  "fallback_rate": 0.0,
+  "head": "iab_content",
+  "macro_f1": 0.8293,
+  "primary_source": "embedding_retrieval",
+  "suite": "train",
+  "tier_metrics": {
+    "average_prediction_depth": 2.2368,
+    "error_buckets": {
+      "exact_match": 10721,
+      "parent_safe_stop": 346,
+      "right_tier1_wrong_tier2": 812,
+      "wrong_deep_leaf": 809,
+      "wrong_tier1": 523
+    },
+    "exact_path_accuracy": 0.8115,
+    "parent_safe_accuracy": 0.8753,
+    "tier1_accuracy": 0.9604,
+    "tier2_accuracy": 0.9208,
+    "tier3_accuracy": 0.8788,
+    "tier4_accuracy": 0.8732
+  },
+  "view_metrics": {
+    "combined_path": {
+      "average_prediction_depth": 2.2368,
+      "error_buckets": {
+        "exact_match": 10569,
+        "parent_safe_stop": 338,
+        "right_tier1_wrong_tier2": 834,
+        "wrong_deep_leaf": 947,
+        "wrong_tier1": 523
+      },
+      "exact_path_accuracy": 0.8,
+      "fallback_overuse_count": 1123,
+      "fallback_rate": 0.085,
+      "parent_safe_accuracy": 0.8631,
+      "tier1_accuracy": 0.9604,
+      "tier2_accuracy": 0.9189,
+      "tier3_accuracy": 0.843,
+      "tier4_accuracy": 0.6589
+    },
+    "disagreements": {
+      "retrieval_vs_combined": 0
+    },
+    "embedding_retrieval": {
+      "average_prediction_depth": 2.2368,
+      "error_buckets": {
+        "exact_match": 10569,
+        "parent_safe_stop": 338,
+        "right_tier1_wrong_tier2": 834,
+        "wrong_deep_leaf": 947,
+        "wrong_tier1": 523
+      },
+      "exact_path_accuracy": 0.8,
+      "parent_safe_accuracy": 0.8631,
+      "tier1_accuracy": 0.9604,
+      "tier2_accuracy": 0.9189,
+      "tier3_accuracy": 0.843,
+      "tier4_accuracy": 0.6589
+    }
+  }
+}

artifacts/evaluation/latest/iab_content_val_report.json

@@ -0,0 +1,67 @@
+{
+  "accepted_accuracy": 0.6545,
+  "accepted_coverage": 1.0,
+  "accuracy": 0.6545,
+  "count": 3282,
+  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/iab/val.jsonl",
+  "fallback_rate": 0.0,
+  "head": "iab_content",
+  "macro_f1": 0.6957,
+  "primary_source": "embedding_retrieval",
+  "suite": "val",
+  "tier_metrics": {
+    "average_prediction_depth": 2.1813,
+    "error_buckets": {
+      "exact_match": 2148,
+      "parent_safe_stop": 105,
+      "right_tier1_wrong_tier2": 684,
+      "wrong_deep_leaf": 234,
+      "wrong_tier1": 111
+    },
+    "exact_path_accuracy": 0.6545,
+    "parent_safe_accuracy": 0.7821,
+    "tier1_accuracy": 0.9662,
+    "tier2_accuracy": 0.7577,
+    "tier3_accuracy": 0.8352,
+    "tier4_accuracy": 0.7214
+  },
+  "view_metrics": {
+    "combined_path": {
+      "average_prediction_depth": 2.1813,
+      "error_buckets": {
+        "exact_match": 2116,
+        "parent_safe_stop": 100,
+        "right_tier1_wrong_tier2": 689,
+        "wrong_deep_leaf": 266,
+        "wrong_tier1": 111
+      },
+      "exact_path_accuracy": 0.6447,
+      "fallback_overuse_count": 413,
+      "fallback_rate": 0.1258,
+      "parent_safe_accuracy": 0.7709,
+      "tier1_accuracy": 0.9662,
+      "tier2_accuracy": 0.756,
+      "tier3_accuracy": 0.799,
+      "tier4_accuracy": 0.55
+    },
+    "disagreements": {
+      "retrieval_vs_combined": 0
+    },
+    "embedding_retrieval": {
+      "average_prediction_depth": 2.1813,
+      "error_buckets": {
+        "exact_match": 2116,
+        "parent_safe_stop": 100,
+        "right_tier1_wrong_tier2": 689,
+        "wrong_deep_leaf": 266,
+        "wrong_tier1": 111
+      },
+      "exact_path_accuracy": 0.6447,
+      "parent_safe_accuracy": 0.7709,
+      "tier1_accuracy": 0.9662,
+      "tier2_accuracy": 0.756,
+      "tier3_accuracy": 0.799,
+      "tier4_accuracy": 0.55
+    }
+  }
+}

artifacts/evaluation/latest/iab_cross_vertical_behavior_lock_regression.json

@@ -0,0 +1,1731 @@
+{
+  "by_status": {
+    "must_fix": {
+      "failed": 0,
+      "passed": 90,
+      "total": 90
+    }
+  },
+  "cases_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/examples/iab_cross_vertical_behavior_lock_cases.json",
+  "count": 90,
+  "failed": 0,
+  "passed": 90,
+  "results": [
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Travel",
+        "model_output.classification.iab_content.tier2.label": "Travel Type"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Travel",
+        "model_output.classification.iab_content.tier2.label": "Travel Type"
+      },
+      "id": "auto-buying-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Automotive > Auto Buying and Selling.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Which car should I buy for commuting?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Automotive",
+        "model_output.classification.iab_content.tier2.label": "Auto Body Styles"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Automotive",
+        "model_output.classification.iab_content.tier2.label": "Auto Body Styles"
+      },
+      "id": "auto-buying-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Automotive > Auto Buying and Selling.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best used SUV for a family of four"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Automotive",
+        "model_output.classification.iab_content.tier2.label": "Auto Type"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Automotive",
+        "model_output.classification.iab_content.tier2.label": "Auto Type"
+      },
+      "id": "auto-buying-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Automotive > Auto Buying and Selling.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "I need a shortlist of practical cars before making a purchase this month"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+      },
+      "id": "sales-crm-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Business and Finance > Business > Sales.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "What is CRM software?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "id": "sales-crm-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Business and Finance > Business > Sales.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "HubSpot vs Zoho for a small team"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Business",
+        "model_output.classification.iab_content.tier3.label": "Sales"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Business",
+        "model_output.classification.iab_content.tier3.label": "Sales"
+      },
+      "id": "sales-crm-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Business and Finance > Business > Sales.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need software to manage leads and pipeline for a startup sales team"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "id": "marketing-tools-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Business and Finance > Business > Marketing and Advertising.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best SEO tools for content teams"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Business"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Business"
+      },
+      "id": "marketing-tools-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Business and Finance > Business > Marketing and Advertising.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "How should I compare ad attribution platforms?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+      },
+      "id": "marketing-tools-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Business and Finance > Business > Marketing and Advertising.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need software to measure channel performance across paid and organic campaigns"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "id": "business-it-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Business and Finance > Business > Business I.T..",
+      "pass": true,
+      "status": "must_fix",
+      "text": "How do I reset my work password?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Careers",
+        "model_output.classification.iab_content.tier2.label": "Job Search"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Careers",
+        "model_output.classification.iab_content.tier2.label": "Job Search"
+      },
+      "id": "business-it-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Business and Finance > Business > Business I.T..",
+      "pass": true,
+      "status": "must_fix",
+      "text": "My employees keep getting locked out of their accounts"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "id": "business-it-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Business and Finance > Business > Business I.T..",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need identity and access software for login, permissions, and account security"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink",
+        "model_output.classification.iab_content.tier2.label": "Dining Out"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink",
+        "model_output.classification.iab_content.tier2.label": "Dining Out"
+      },
+      "id": "dining-out-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Food & Drink > Dining Out.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Book a table for six tonight"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink",
+        "model_output.classification.iab_content.tier2.label": "Dining Out"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink",
+        "model_output.classification.iab_content.tier2.label": "Dining Out"
+      },
+      "id": "dining-out-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Food & Drink > Dining Out.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Good restaurants for a client dinner downtown"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink",
+        "model_output.classification.iab_content.tier2.label": "Dining Out"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink",
+        "model_output.classification.iab_content.tier2.label": "Dining Out"
+      },
+      "id": "dining-out-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Food & Drink > Dining Out.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need a place to eat tonight where I can make a reservation online"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink"
+      },
+      "id": "alcoholic-beverages-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Food & Drink > Alcoholic Beverages.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Which whiskey cocktail should I order?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink"
+      },
+      "id": "alcoholic-beverages-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Food & Drink > Alcoholic Beverages.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best vodka drinks for beginners"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink"
+      },
+      "id": "alcoholic-beverages-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Food & Drink > Alcoholic Beverages.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Want a spirit-forward drink recommendation, not a restaurant suggestion"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing"
+      },
+      "id": "artificial-intelligence-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Technology & Computing > Artificial Intelligence.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "What is intent classification in NLP?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Education",
+        "model_output.classification.iab_content.tier2.label": "Language Learning"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Education",
+        "model_output.classification.iab_content.tier2.label": "Language Learning"
+      },
+      "id": "artificial-intelligence-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Technology & Computing > Artificial Intelligence.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "How do large language models handle text classification?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Education",
+        "model_output.classification.iab_content.tier2.label": "Language Learning"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Education",
+        "model_output.classification.iab_content.tier2.label": "Language Learning"
+      },
+      "id": "artificial-intelligence-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Technology & Computing > Artificial Intelligence.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need the machine learning concept behind language understanding, not software to buy"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Business"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Business"
+      },
+      "id": "software-apps-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Technology & Computing > Computing > Computer Software and Applications.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best workflow software for a small operations team"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet",
+        "model_output.classification.iab_content.tier4.label": "Cloud Computing"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet",
+        "model_output.classification.iab_content.tier4.label": "Cloud Computing"
+      },
+      "id": "software-apps-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Technology & Computing > Computing > Computer Software and Applications.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need project management software for a distributed team"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+      },
+      "id": "software-apps-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Technology & Computing > Computing > Computer Software and Applications.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Looking for a business software platform to organize internal workflows"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications",
+        "model_output.classification.iab_content.tier4.label": "Communication"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications",
+        "model_output.classification.iab_content.tier4.label": "Communication"
+      },
+      "id": "communication-software-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Technology & Computing > Computing > Computer Software and Applications > Communication.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best communication software for remote teams"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications",
+        "model_output.classification.iab_content.tier4.label": "Communication"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications",
+        "model_output.classification.iab_content.tier4.label": "Communication"
+      },
+      "id": "communication-software-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Technology & Computing > Computing > Computer Software and Applications > Communication.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Slack vs Teams for internal messaging"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications",
+        "model_output.classification.iab_content.tier4.label": "Communication"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications",
+        "model_output.classification.iab_content.tier4.label": "Communication"
+      },
+      "id": "communication-software-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Technology & Computing > Computing > Computer Software and Applications > Communication.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need a workplace chat tool for cross-functional collaboration"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet",
+        "model_output.classification.iab_content.tier4.label": "Web Hosting"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet",
+        "model_output.classification.iab_content.tier4.label": "Web Hosting"
+      },
+      "id": "web-hosting-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Technology & Computing > Computing > Internet > Web Hosting.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Vercel vs Netlify for website hosting"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet",
+        "model_output.classification.iab_content.tier4.label": "Web Hosting"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet",
+        "model_output.classification.iab_content.tier4.label": "Web Hosting"
+      },
+      "id": "web-hosting-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Technology & Computing > Computing > Internet > Web Hosting.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best hosting platform for a startup website"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet",
+        "model_output.classification.iab_content.tier4.label": "Web Hosting"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet",
+        "model_output.classification.iab_content.tier4.label": "Web Hosting"
+      },
+      "id": "web-hosting-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Technology & Computing > Computing > Internet > Web Hosting.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need a managed hosting provider to deploy and run our marketing site"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Laptops"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Laptops"
+      },
+      "id": "laptops-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Technology & Computing > Computing > Laptops.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Which laptop should I buy for college?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Laptops"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Laptops"
+      },
+      "id": "laptops-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Technology & Computing > Computing > Laptops.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best laptop for work and study under 1200"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Consumer Electronics",
+        "model_output.classification.iab_content.tier3.label": "Smartphones"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Consumer Electronics",
+        "model_output.classification.iab_content.tier3.label": "Smartphones"
+      },
+      "id": "laptops-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Technology & Computing > Computing > Laptops.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need a portable computer with good battery life for everyday work"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications",
+        "model_output.classification.iab_content.tier4.label": "Photo Editing Software"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications",
+        "model_output.classification.iab_content.tier4.label": "Photo Editing Software"
+      },
+      "id": "desktops-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Technology & Computing > Computing > Desktops.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best desktop for video editing"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Desktops"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Desktops"
+      },
+      "id": "desktops-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Technology & Computing > Computing > Desktops.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Which desktop computer should I buy for a home office?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Desktops"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Desktops"
+      },
+      "id": "desktops-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Technology & Computing > Computing > Desktops.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need a desktop PC with strong performance for creative work"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Consumer Electronics"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Consumer Electronics"
+      },
+      "id": "smartphones-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Technology & Computing > Consumer Electronics > Smartphones.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best phone with a good camera under 700"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Consumer Electronics",
+        "model_output.classification.iab_content.tier3.label": "Smartphones"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Consumer Electronics",
+        "model_output.classification.iab_content.tier3.label": "Smartphones"
+      },
+      "id": "smartphones-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Technology & Computing > Consumer Electronics > Smartphones.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Should I buy an iPhone or Pixel this year?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Consumer Electronics",
+        "model_output.classification.iab_content.tier3.label": "Smartphones"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Consumer Electronics",
+        "model_output.classification.iab_content.tier3.label": "Smartphones"
+      },
+      "id": "smartphones-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Technology & Computing > Consumer Electronics > Smartphones.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need a new smartphone with strong battery life and a clean software experience"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Women's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Women's Shoes and Footwear"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Women's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Women's Shoes and Footwear"
+      },
+      "id": "style-fashion-parent-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Style & Fashion.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best shoes under 100 dollars"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Women's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Women's Clothing"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Women's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Women's Clothing"
+      },
+      "id": "style-fashion-parent-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Style & Fashion.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Affordable fashion accessories for everyday wear"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Women's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Women's Shoes and Footwear"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Women's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Women's Shoes and Footwear"
+      },
+      "id": "style-fashion-parent-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Style & Fashion.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need style recommendations for clothing and footwear without a specific brand in mind"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Women's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Women's Clothing"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Women's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Women's Clothing"
+      },
+      "id": "womens-shoes-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Style & Fashion > Women's Fashion > Women's Shoes and Footwear.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best women's running shoes under 100 dollars"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Women's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Women's Clothing"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Women's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Women's Clothing"
+      },
+      "id": "womens-shoes-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Style & Fashion > Women's Fashion > Women's Shoes and Footwear.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Comfortable women's sneakers for walking all day"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Women's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Women's Shoes and Footwear"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Women's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Women's Shoes and Footwear"
+      },
+      "id": "womens-shoes-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Style & Fashion > Women's Fashion > Women's Shoes and Footwear.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need women's footwear for commuting that looks polished but feels comfortable"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Men's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Men's Clothing"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Men's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Men's Clothing"
+      },
+      "id": "mens-shoes-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Style & Fashion > Men's Fashion > Men's Shoes and Footwear.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best men's sneakers for daily wear"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Men's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Men's Clothing"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Men's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Men's Clothing"
+      },
+      "id": "mens-shoes-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Style & Fashion > Men's Fashion > Men's Shoes and Footwear.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Good men's dress shoes for office use"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Men's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Men's Shoes and Footwear"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Men's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Men's Shoes and Footwear"
+      },
+      "id": "mens-shoes-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Style & Fashion > Men's Fashion > Men's Shoes and Footwear.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need men's footwear that works for workdays and weekend walking"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Real Estate",
+        "model_output.classification.iab_content.tier2.label": "Hotel Properties"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Real Estate",
+        "model_output.classification.iab_content.tier2.label": "Hotel Properties"
+      },
+      "id": "hotels-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Travel > Travel Type > Hotels and Motels.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need a hotel in Chicago for two nights"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Travel",
+        "model_output.classification.iab_content.tier2.label": "Travel Type",
+        "model_output.classification.iab_content.tier3.label": "Hotels and Motels"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Travel",
+        "model_output.classification.iab_content.tier2.label": "Travel Type",
+        "model_output.classification.iab_content.tier3.label": "Hotels and Motels"
+      },
+      "id": "hotels-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Travel > Travel Type > Hotels and Motels.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best hotels near Times Square for a weekend trip"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Travel",
+        "model_output.classification.iab_content.tier2.label": "Travel Type"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Travel",
+        "model_output.classification.iab_content.tier2.label": "Travel Type"
+      },
+      "id": "hotels-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Travel > Travel Type > Hotels and Motels.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Looking for a place to stay during a work trip, not general travel advice"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Real Estate",
+        "model_output.classification.iab_content.tier2.label": "Apartments"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Real Estate",
+        "model_output.classification.iab_content.tier2.label": "Apartments"
+      },
+      "id": "real-estate-rentals-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Real Estate > Real Estate Renting and Leasing.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Apartments for rent near downtown Austin"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Real Estate",
+        "model_output.classification.iab_content.tier2.label": "Apartments"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Real Estate",
+        "model_output.classification.iab_content.tier2.label": "Apartments"
+      },
+      "id": "real-estate-rentals-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Real Estate > Real Estate Renting and Leasing.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best neighborhoods to lease a two-bedroom apartment in Seattle"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Personal Finance",
+        "model_output.classification.iab_content.tier2.label": "Personal Debt",
+        "model_output.classification.iab_content.tier3.label": "Home Financing"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Personal Finance",
+        "model_output.classification.iab_content.tier2.label": "Personal Debt",
+        "model_output.classification.iab_content.tier3.label": "Home Financing"
+      },
+      "id": "real-estate-rentals-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Real Estate > Real Estate Renting and Leasing.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need rental listings for a short move, not home-buying advice"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Business",
+        "model_output.classification.iab_content.tier3.label": "Green Solutions"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Business",
+        "model_output.classification.iab_content.tier3.label": "Green Solutions"
+      },
+      "id": "running-and-jogging-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Healthy Living > Fitness and Exercise > Running and Jogging.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best running plan for a first 10k"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Healthy Living",
+        "model_output.classification.iab_content.tier2.label": "Fitness and Exercise",
+        "model_output.classification.iab_content.tier3.label": "Running and Jogging"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Healthy Living",
+        "model_output.classification.iab_content.tier2.label": "Fitness and Exercise",
+        "model_output.classification.iab_content.tier3.label": "Running and Jogging"
+      },
+      "id": "running-and-jogging-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Healthy Living > Fitness and Exercise > Running and Jogging.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "How should I train for a half marathon as a beginner?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Healthy Living",
+        "model_output.classification.iab_content.tier2.label": "Fitness and Exercise",
+        "model_output.classification.iab_content.tier3.label": "Running and Jogging"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Healthy Living",
+        "model_output.classification.iab_content.tier2.label": "Fitness and Exercise",
+        "model_output.classification.iab_content.tier3.label": "Running and Jogging"
+      },
+      "id": "running-and-jogging-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Healthy Living > Fitness and Exercise > Running and Jogging.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need guidance on building a weekly jogging routine without getting injured"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Sports",
+        "model_output.classification.iab_content.tier2.label": "Soccer"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Sports",
+        "model_output.classification.iab_content.tier2.label": "Soccer"
+      },
+      "id": "soccer-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Sports > Soccer.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "How do offside rules work in soccer?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Sports",
+        "model_output.classification.iab_content.tier2.label": "Soccer"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Sports",
+        "model_output.classification.iab_content.tier2.label": "Soccer"
+      },
+      "id": "soccer-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Sports > Soccer.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best soccer drills for beginner players"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Sports"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Sports"
+      },
+      "id": "soccer-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Sports > Soccer.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need help understanding football tactics for the Premier League, not fantasy sports"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Books and Literature",
+        "model_output.classification.iab_content.tier2.label": "Fiction"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Books and Literature",
+        "model_output.classification.iab_content.tier2.label": "Fiction"
+      },
+      "id": "fiction-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Books and Literature > Fiction.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Recommend a good fantasy novel to read"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Books and Literature",
+        "model_output.classification.iab_content.tier2.label": "Fiction"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Books and Literature",
+        "model_output.classification.iab_content.tier2.label": "Fiction"
+      },
+      "id": "fiction-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Books and Literature > Fiction.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best fiction books for a long flight"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Books and Literature"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Books and Literature"
+      },
+      "id": "fiction-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Books and Literature > Fiction.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Looking for a character-driven novel, not comics or poetry"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Home & Garden",
+        "model_output.classification.iab_content.tier2.label": "Remodeling & Construction"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Home & Garden",
+        "model_output.classification.iab_content.tier2.label": "Remodeling & Construction"
+      },
+      "id": "home-improvement-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Home & Garden > Home Improvement.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "How much does a kitchen remodel usually cost?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Personal Care",
+        "model_output.classification.iab_content.tier3.label": "Bath and Shower"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Personal Care",
+        "model_output.classification.iab_content.tier3.label": "Bath and Shower"
+      },
+      "id": "home-improvement-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Home & Garden > Home Improvement.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best tools for a DIY bathroom renovation"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Home & Garden",
+        "model_output.classification.iab_content.tier2.label": "Interior Decorating"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Home & Garden",
+        "model_output.classification.iab_content.tier2.label": "Interior Decorating"
+      },
+      "id": "home-improvement-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Home & Garden > Home Improvement.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need practical advice for upgrading an older house, not interior decor inspiration"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Education",
+        "model_output.classification.iab_content.tier2.label": "Language Learning"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Education",
+        "model_output.classification.iab_content.tier2.label": "Language Learning"
+      },
+      "id": "online-education-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Education > Online Education.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best online courses for learning Python"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Careers",
+        "model_output.classification.iab_content.tier2.label": "Remote Working"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Careers",
+        "model_output.classification.iab_content.tier2.label": "Remote Working"
+      },
+      "id": "online-education-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Education > Online Education.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "What are good platforms for remote professional classes?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Careers"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Careers"
+      },
+      "id": "online-education-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Education > Online Education.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need internet-based training options I can finish after work hours"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Education",
+        "model_output.classification.iab_content.tier2.label": "College Education"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Education",
+        "model_output.classification.iab_content.tier2.label": "College Education"
+      },
+      "id": "postgraduate-education-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Education > College Education > Postgraduate Education.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "best universities to study masters"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Education",
+        "model_output.classification.iab_content.tier2.label": "College Education",
+        "model_output.classification.iab_content.tier3.label": "Postgraduate Education",
+        "model_output.classification.iab_content.tier4.label": "Professional School"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Education",
+        "model_output.classification.iab_content.tier2.label": "College Education",
+        "model_output.classification.iab_content.tier3.label": "Postgraduate Education",
+        "model_output.classification.iab_content.tier4.label": "Professional School"
+      },
+      "id": "postgraduate-education-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Education > College Education > Postgraduate Education.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "which graduate schools have strong data science programs"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Education",
+        "model_output.classification.iab_content.tier2.label": "College Education"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Education",
+        "model_output.classification.iab_content.tier2.label": "College Education"
+      },
+      "id": "postgraduate-education-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Education > College Education > Postgraduate Education.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "need postgraduate options for a master's degree, not short online courses"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Medical Health",
+        "model_output.classification.iab_content.tier2.label": "Diseases and Conditions",
+        "model_output.classification.iab_content.tier3.label": "Allergies"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Medical Health",
+        "model_output.classification.iab_content.tier2.label": "Diseases and Conditions",
+        "model_output.classification.iab_content.tier3.label": "Allergies"
+      },
+      "id": "medical-health-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Medical Health.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "what do these allergy symptoms mean"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Medical Health",
+        "model_output.classification.iab_content.tier2.label": "Diseases and Conditions",
+        "model_output.classification.iab_content.tier3.label": "Injuries",
+        "model_output.classification.iab_content.tier4.label": "First Aid"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Medical Health",
+        "model_output.classification.iab_content.tier2.label": "Diseases and Conditions",
+        "model_output.classification.iab_content.tier3.label": "Injuries",
+        "model_output.classification.iab_content.tier4.label": "First Aid"
+      },
+      "id": "medical-health-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Medical Health.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "when should i see a doctor for persistent knee pain"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Healthy Living",
+        "model_output.classification.iab_content.tier2.label": "Wellness",
+        "model_output.classification.iab_content.tier3.label": "Physical Therapy"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Healthy Living",
+        "model_output.classification.iab_content.tier2.label": "Wellness",
+        "model_output.classification.iab_content.tier3.label": "Physical Therapy"
+      },
+      "id": "medical-health-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Medical Health.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "need medical advice about symptoms, not wellness or fitness tips"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Careers",
+        "model_output.classification.iab_content.tier2.label": "Remote Working"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Careers",
+        "model_output.classification.iab_content.tier2.label": "Remote Working"
+      },
+      "id": "careers-job-search-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Careers > Job Search.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "best remote jobs for data analysts"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Business",
+        "model_output.classification.iab_content.tier3.label": "Sales"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Business",
+        "model_output.classification.iab_content.tier3.label": "Sales"
+      },
+      "id": "careers-job-search-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Careers > Job Search.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "where should i look for product manager openings"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Careers",
+        "model_output.classification.iab_content.tier2.label": "Job Search"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Careers",
+        "model_output.classification.iab_content.tier2.label": "Job Search"
+      },
+      "id": "careers-job-search-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Careers > Job Search.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "need help finding a new role and preparing for interviews"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink",
+        "model_output.classification.iab_content.tier2.label": "Food Movements"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink",
+        "model_output.classification.iab_content.tier2.label": "Food Movements"
+      },
+      "id": "personal-finance-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Personal Finance > Financial Planning.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "how much should i save each month"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Economy",
+        "model_output.classification.iab_content.tier3.label": "Financial Reform"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Economy",
+        "model_output.classification.iab_content.tier3.label": "Financial Reform"
+      },
+      "id": "personal-finance-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Personal Finance > Financial Planning.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "best budgeting approach for a growing family"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Personal Finance",
+        "model_output.classification.iab_content.tier2.label": "Retirement Planning"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Personal Finance",
+        "model_output.classification.iab_content.tier2.label": "Retirement Planning"
+      },
+      "id": "personal-finance-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Personal Finance > Financial Planning.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "need help planning savings and retirement, not business finance advice"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Family and Relationships",
+        "model_output.classification.iab_content.tier2.label": "Parenting"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Family and Relationships",
+        "model_output.classification.iab_content.tier2.label": "Parenting"
+      },
+      "id": "parenting-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Family and Relationships > Parenting.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "tips for parenting a toddler"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Family and Relationships",
+        "model_output.classification.iab_content.tier2.label": "Parenting"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Family and Relationships",
+        "model_output.classification.iab_content.tier2.label": "Parenting"
+      },
+      "id": "parenting-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Family and Relationships > Parenting.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "how do i help my teenager spend less time online"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Family and Relationships",
+        "model_output.classification.iab_content.tier2.label": "Parenting",
+        "model_output.classification.iab_content.tier3.label": "Special Needs Kids"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Family and Relationships",
+        "model_output.classification.iab_content.tier2.label": "Parenting",
+        "model_output.classification.iab_content.tier3.label": "Special Needs Kids"
+      },
+      "id": "parenting-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Family and Relationships > Parenting.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "need parenting advice for a child starting preschool"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Home & Garden"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Home & Garden"
+      },
+      "id": "gardening-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Home & Garden > Gardening.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "best plants for a small balcony garden"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Home & Garden",
+        "model_output.classification.iab_content.tier2.label": "Gardening"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Home & Garden",
+        "model_output.classification.iab_content.tier2.label": "Gardening"
+      },
+      "id": "gardening-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Home & Garden > Gardening.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "how often should i water tomato plants"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Home & Garden"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Home & Garden"
+      },
+      "id": "gardening-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Home & Garden > Gardening.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "need gardening advice for a shady backyard, not interior decor ideas"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Entertainment",
+        "model_output.classification.iab_content.tier2.label": "Movies"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Entertainment",
+        "model_output.classification.iab_content.tier2.label": "Movies"
+      },
+      "id": "movies-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Entertainment > Movies.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "What movie should we watch tonight?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Video Gaming",
+        "model_output.classification.iab_content.tier2.label": "Video Game Genres",
+        "model_output.classification.iab_content.tier3.label": "Horror Video Games"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Video Gaming",
+        "model_output.classification.iab_content.tier2.label": "Video Game Genres",
+        "model_output.classification.iab_content.tier3.label": "Horror Video Games"
+      },
+      "id": "movies-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Entertainment > Movies.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best thriller movies from the last few years"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Entertainment",
+        "model_output.classification.iab_content.tier2.label": "Music",
+        "model_output.classification.iab_content.tier3.label": "Soundtracks, TV and Showtunes"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Entertainment",
+        "model_output.classification.iab_content.tier2.label": "Music",
+        "model_output.classification.iab_content.tier3.label": "Soundtracks, TV and Showtunes"
+      },
+      "id": "movies-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Entertainment > Movies.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Looking for film recommendations, not TV shows or music"
+    }
+  ]
+}

artifacts/evaluation/latest/iab_cross_vertical_regression.json

@@ -0,0 +1,1731 @@
+{
+  "by_status": {
+    "must_fix": {
+      "failed": 0,
+      "passed": 90,
+      "total": 90
+    }
+  },
+  "cases_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/examples/iab_cross_vertical_mapping_cases.json",
+  "count": 90,
+  "failed": 0,
+  "passed": 90,
+  "results": [
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Travel",
+        "model_output.classification.iab_content.tier2.label": "Travel Type"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Travel",
+        "model_output.classification.iab_content.tier2.label": "Travel Type"
+      },
+      "id": "auto-buying-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Automotive > Auto Buying and Selling.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Which car should I buy for commuting?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Automotive",
+        "model_output.classification.iab_content.tier2.label": "Auto Body Styles"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Automotive",
+        "model_output.classification.iab_content.tier2.label": "Auto Body Styles"
+      },
+      "id": "auto-buying-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Automotive > Auto Buying and Selling.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best used SUV for a family of four"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Automotive",
+        "model_output.classification.iab_content.tier2.label": "Auto Type"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Automotive",
+        "model_output.classification.iab_content.tier2.label": "Auto Type"
+      },
+      "id": "auto-buying-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Automotive > Auto Buying and Selling.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "I need a shortlist of practical cars before making a purchase this month"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+      },
+      "id": "sales-crm-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Business and Finance > Business > Sales.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "What is CRM software?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "id": "sales-crm-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Business and Finance > Business > Sales.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "HubSpot vs Zoho for a small team"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Business",
+        "model_output.classification.iab_content.tier3.label": "Sales"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Business",
+        "model_output.classification.iab_content.tier3.label": "Sales"
+      },
+      "id": "sales-crm-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Business and Finance > Business > Sales.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need software to manage leads and pipeline for a startup sales team"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "id": "marketing-tools-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Business and Finance > Business > Marketing and Advertising.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best SEO tools for content teams"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Business"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Business"
+      },
+      "id": "marketing-tools-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Business and Finance > Business > Marketing and Advertising.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "How should I compare ad attribution platforms?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+      },
+      "id": "marketing-tools-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Business and Finance > Business > Marketing and Advertising.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need software to measure channel performance across paid and organic campaigns"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "id": "business-it-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Business and Finance > Business > Business I.T..",
+      "pass": true,
+      "status": "must_fix",
+      "text": "How do I reset my work password?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Careers",
+        "model_output.classification.iab_content.tier2.label": "Job Search"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Careers",
+        "model_output.classification.iab_content.tier2.label": "Job Search"
+      },
+      "id": "business-it-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Business and Finance > Business > Business I.T..",
+      "pass": true,
+      "status": "must_fix",
+      "text": "My employees keep getting locked out of their accounts"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "id": "business-it-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Business and Finance > Business > Business I.T..",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need identity and access software for login, permissions, and account security"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink",
+        "model_output.classification.iab_content.tier2.label": "Dining Out"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink",
+        "model_output.classification.iab_content.tier2.label": "Dining Out"
+      },
+      "id": "dining-out-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Food & Drink > Dining Out.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Book a table for six tonight"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink",
+        "model_output.classification.iab_content.tier2.label": "Dining Out"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink",
+        "model_output.classification.iab_content.tier2.label": "Dining Out"
+      },
+      "id": "dining-out-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Food & Drink > Dining Out.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Good restaurants for a client dinner downtown"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink",
+        "model_output.classification.iab_content.tier2.label": "Dining Out"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink",
+        "model_output.classification.iab_content.tier2.label": "Dining Out"
+      },
+      "id": "dining-out-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Food & Drink > Dining Out.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need a place to eat tonight where I can make a reservation online"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink"
+      },
+      "id": "alcoholic-beverages-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Food & Drink > Alcoholic Beverages.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Which whiskey cocktail should I order?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink"
+      },
+      "id": "alcoholic-beverages-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Food & Drink > Alcoholic Beverages.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best vodka drinks for beginners"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink"
+      },
+      "id": "alcoholic-beverages-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Food & Drink > Alcoholic Beverages.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Want a spirit-forward drink recommendation, not a restaurant suggestion"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing"
+      },
+      "id": "artificial-intelligence-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Technology & Computing > Artificial Intelligence.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "What is intent classification in NLP?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Education",
+        "model_output.classification.iab_content.tier2.label": "Language Learning"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Education",
+        "model_output.classification.iab_content.tier2.label": "Language Learning"
+      },
+      "id": "artificial-intelligence-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Technology & Computing > Artificial Intelligence.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "How do large language models handle text classification?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Education",
+        "model_output.classification.iab_content.tier2.label": "Language Learning"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Education",
+        "model_output.classification.iab_content.tier2.label": "Language Learning"
+      },
+      "id": "artificial-intelligence-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Technology & Computing > Artificial Intelligence.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need the machine learning concept behind language understanding, not software to buy"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Business"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Business"
+      },
+      "id": "software-apps-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Technology & Computing > Computing > Computer Software and Applications.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best workflow software for a small operations team"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet",
+        "model_output.classification.iab_content.tier4.label": "Cloud Computing"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet",
+        "model_output.classification.iab_content.tier4.label": "Cloud Computing"
+      },
+      "id": "software-apps-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Technology & Computing > Computing > Computer Software and Applications.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need project management software for a distributed team"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+      },
+      "id": "software-apps-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Technology & Computing > Computing > Computer Software and Applications.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Looking for a business software platform to organize internal workflows"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications",
+        "model_output.classification.iab_content.tier4.label": "Communication"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications",
+        "model_output.classification.iab_content.tier4.label": "Communication"
+      },
+      "id": "communication-software-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Technology & Computing > Computing > Computer Software and Applications > Communication.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best communication software for remote teams"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications",
+        "model_output.classification.iab_content.tier4.label": "Communication"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications",
+        "model_output.classification.iab_content.tier4.label": "Communication"
+      },
+      "id": "communication-software-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Technology & Computing > Computing > Computer Software and Applications > Communication.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Slack vs Teams for internal messaging"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications",
+        "model_output.classification.iab_content.tier4.label": "Communication"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications",
+        "model_output.classification.iab_content.tier4.label": "Communication"
+      },
+      "id": "communication-software-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Technology & Computing > Computing > Computer Software and Applications > Communication.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need a workplace chat tool for cross-functional collaboration"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet",
+        "model_output.classification.iab_content.tier4.label": "Web Hosting"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet",
+        "model_output.classification.iab_content.tier4.label": "Web Hosting"
+      },
+      "id": "web-hosting-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Technology & Computing > Computing > Internet > Web Hosting.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Vercel vs Netlify for website hosting"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet",
+        "model_output.classification.iab_content.tier4.label": "Web Hosting"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet",
+        "model_output.classification.iab_content.tier4.label": "Web Hosting"
+      },
+      "id": "web-hosting-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Technology & Computing > Computing > Internet > Web Hosting.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best hosting platform for a startup website"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet",
+        "model_output.classification.iab_content.tier4.label": "Web Hosting"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet",
+        "model_output.classification.iab_content.tier4.label": "Web Hosting"
+      },
+      "id": "web-hosting-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Technology & Computing > Computing > Internet > Web Hosting.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need a managed hosting provider to deploy and run our marketing site"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Laptops"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Laptops"
+      },
+      "id": "laptops-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Technology & Computing > Computing > Laptops.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Which laptop should I buy for college?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Laptops"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Laptops"
+      },
+      "id": "laptops-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Technology & Computing > Computing > Laptops.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best laptop for work and study under 1200"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Consumer Electronics",
+        "model_output.classification.iab_content.tier3.label": "Smartphones"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Consumer Electronics",
+        "model_output.classification.iab_content.tier3.label": "Smartphones"
+      },
+      "id": "laptops-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Technology & Computing > Computing > Laptops.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need a portable computer with good battery life for everyday work"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications",
+        "model_output.classification.iab_content.tier4.label": "Photo Editing Software"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications",
+        "model_output.classification.iab_content.tier4.label": "Photo Editing Software"
+      },
+      "id": "desktops-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Technology & Computing > Computing > Desktops.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best desktop for video editing"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Desktops"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Desktops"
+      },
+      "id": "desktops-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Technology & Computing > Computing > Desktops.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Which desktop computer should I buy for a home office?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Desktops"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Desktops"
+      },
+      "id": "desktops-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Technology & Computing > Computing > Desktops.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need a desktop PC with strong performance for creative work"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Consumer Electronics"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Consumer Electronics"
+      },
+      "id": "smartphones-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Technology & Computing > Consumer Electronics > Smartphones.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best phone with a good camera under 700"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Consumer Electronics",
+        "model_output.classification.iab_content.tier3.label": "Smartphones"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Consumer Electronics",
+        "model_output.classification.iab_content.tier3.label": "Smartphones"
+      },
+      "id": "smartphones-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Technology & Computing > Consumer Electronics > Smartphones.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Should I buy an iPhone or Pixel this year?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Consumer Electronics",
+        "model_output.classification.iab_content.tier3.label": "Smartphones"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Consumer Electronics",
+        "model_output.classification.iab_content.tier3.label": "Smartphones"
+      },
+      "id": "smartphones-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Technology & Computing > Consumer Electronics > Smartphones.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need a new smartphone with strong battery life and a clean software experience"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Women's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Women's Shoes and Footwear"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Women's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Women's Shoes and Footwear"
+      },
+      "id": "style-fashion-parent-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Style & Fashion.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best shoes under 100 dollars"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Women's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Women's Clothing"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Women's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Women's Clothing"
+      },
+      "id": "style-fashion-parent-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Style & Fashion.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Affordable fashion accessories for everyday wear"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Women's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Women's Shoes and Footwear"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Women's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Women's Shoes and Footwear"
+      },
+      "id": "style-fashion-parent-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Style & Fashion.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need style recommendations for clothing and footwear without a specific brand in mind"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Women's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Women's Clothing"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Women's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Women's Clothing"
+      },
+      "id": "womens-shoes-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Style & Fashion > Women's Fashion > Women's Shoes and Footwear.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best women's running shoes under 100 dollars"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Women's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Women's Clothing"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Women's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Women's Clothing"
+      },
+      "id": "womens-shoes-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Style & Fashion > Women's Fashion > Women's Shoes and Footwear.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Comfortable women's sneakers for walking all day"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Women's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Women's Shoes and Footwear"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Women's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Women's Shoes and Footwear"
+      },
+      "id": "womens-shoes-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Style & Fashion > Women's Fashion > Women's Shoes and Footwear.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need women's footwear for commuting that looks polished but feels comfortable"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Men's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Men's Clothing"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Men's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Men's Clothing"
+      },
+      "id": "mens-shoes-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Style & Fashion > Men's Fashion > Men's Shoes and Footwear.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best men's sneakers for daily wear"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Men's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Men's Clothing"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Men's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Men's Clothing"
+      },
+      "id": "mens-shoes-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Style & Fashion > Men's Fashion > Men's Shoes and Footwear.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Good men's dress shoes for office use"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Men's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Men's Shoes and Footwear"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Men's Fashion",
+        "model_output.classification.iab_content.tier3.label": "Men's Shoes and Footwear"
+      },
+      "id": "mens-shoes-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Style & Fashion > Men's Fashion > Men's Shoes and Footwear.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need men's footwear that works for workdays and weekend walking"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Real Estate",
+        "model_output.classification.iab_content.tier2.label": "Hotel Properties"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Real Estate",
+        "model_output.classification.iab_content.tier2.label": "Hotel Properties"
+      },
+      "id": "hotels-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Travel > Travel Type > Hotels and Motels.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need a hotel in Chicago for two nights"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Travel",
+        "model_output.classification.iab_content.tier2.label": "Travel Type",
+        "model_output.classification.iab_content.tier3.label": "Hotels and Motels"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Travel",
+        "model_output.classification.iab_content.tier2.label": "Travel Type",
+        "model_output.classification.iab_content.tier3.label": "Hotels and Motels"
+      },
+      "id": "hotels-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Travel > Travel Type > Hotels and Motels.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best hotels near Times Square for a weekend trip"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Travel",
+        "model_output.classification.iab_content.tier2.label": "Travel Type"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Travel",
+        "model_output.classification.iab_content.tier2.label": "Travel Type"
+      },
+      "id": "hotels-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Travel > Travel Type > Hotels and Motels.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Looking for a place to stay during a work trip, not general travel advice"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Real Estate",
+        "model_output.classification.iab_content.tier2.label": "Apartments"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Real Estate",
+        "model_output.classification.iab_content.tier2.label": "Apartments"
+      },
+      "id": "real-estate-rentals-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Real Estate > Real Estate Renting and Leasing.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Apartments for rent near downtown Austin"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Real Estate",
+        "model_output.classification.iab_content.tier2.label": "Apartments"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Real Estate",
+        "model_output.classification.iab_content.tier2.label": "Apartments"
+      },
+      "id": "real-estate-rentals-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Real Estate > Real Estate Renting and Leasing.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best neighborhoods to lease a two-bedroom apartment in Seattle"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Personal Finance",
+        "model_output.classification.iab_content.tier2.label": "Personal Debt",
+        "model_output.classification.iab_content.tier3.label": "Home Financing"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Personal Finance",
+        "model_output.classification.iab_content.tier2.label": "Personal Debt",
+        "model_output.classification.iab_content.tier3.label": "Home Financing"
+      },
+      "id": "real-estate-rentals-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Real Estate > Real Estate Renting and Leasing.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need rental listings for a short move, not home-buying advice"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Business",
+        "model_output.classification.iab_content.tier3.label": "Green Solutions"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Business",
+        "model_output.classification.iab_content.tier3.label": "Green Solutions"
+      },
+      "id": "running-and-jogging-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Healthy Living > Fitness and Exercise > Running and Jogging.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best running plan for a first 10k"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Healthy Living",
+        "model_output.classification.iab_content.tier2.label": "Fitness and Exercise",
+        "model_output.classification.iab_content.tier3.label": "Running and Jogging"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Healthy Living",
+        "model_output.classification.iab_content.tier2.label": "Fitness and Exercise",
+        "model_output.classification.iab_content.tier3.label": "Running and Jogging"
+      },
+      "id": "running-and-jogging-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Healthy Living > Fitness and Exercise > Running and Jogging.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "How should I train for a half marathon as a beginner?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Healthy Living",
+        "model_output.classification.iab_content.tier2.label": "Fitness and Exercise",
+        "model_output.classification.iab_content.tier3.label": "Running and Jogging"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Healthy Living",
+        "model_output.classification.iab_content.tier2.label": "Fitness and Exercise",
+        "model_output.classification.iab_content.tier3.label": "Running and Jogging"
+      },
+      "id": "running-and-jogging-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Healthy Living > Fitness and Exercise > Running and Jogging.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need guidance on building a weekly jogging routine without getting injured"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Sports",
+        "model_output.classification.iab_content.tier2.label": "Soccer"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Sports",
+        "model_output.classification.iab_content.tier2.label": "Soccer"
+      },
+      "id": "soccer-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Sports > Soccer.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "How do offside rules work in soccer?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Sports",
+        "model_output.classification.iab_content.tier2.label": "Soccer"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Sports",
+        "model_output.classification.iab_content.tier2.label": "Soccer"
+      },
+      "id": "soccer-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Sports > Soccer.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best soccer drills for beginner players"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Sports"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Sports"
+      },
+      "id": "soccer-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Sports > Soccer.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need help understanding football tactics for the Premier League, not fantasy sports"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Books and Literature",
+        "model_output.classification.iab_content.tier2.label": "Fiction"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Books and Literature",
+        "model_output.classification.iab_content.tier2.label": "Fiction"
+      },
+      "id": "fiction-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Books and Literature > Fiction.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Recommend a good fantasy novel to read"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Books and Literature",
+        "model_output.classification.iab_content.tier2.label": "Fiction"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Books and Literature",
+        "model_output.classification.iab_content.tier2.label": "Fiction"
+      },
+      "id": "fiction-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Books and Literature > Fiction.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best fiction books for a long flight"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Books and Literature"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Books and Literature"
+      },
+      "id": "fiction-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Books and Literature > Fiction.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Looking for a character-driven novel, not comics or poetry"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Home & Garden",
+        "model_output.classification.iab_content.tier2.label": "Remodeling & Construction"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Home & Garden",
+        "model_output.classification.iab_content.tier2.label": "Remodeling & Construction"
+      },
+      "id": "home-improvement-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Home & Garden > Home Improvement.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "How much does a kitchen remodel usually cost?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Personal Care",
+        "model_output.classification.iab_content.tier3.label": "Bath and Shower"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
+        "model_output.classification.iab_content.tier2.label": "Personal Care",
+        "model_output.classification.iab_content.tier3.label": "Bath and Shower"
+      },
+      "id": "home-improvement-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Home & Garden > Home Improvement.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best tools for a DIY bathroom renovation"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Home & Garden",
+        "model_output.classification.iab_content.tier2.label": "Interior Decorating"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Home & Garden",
+        "model_output.classification.iab_content.tier2.label": "Interior Decorating"
+      },
+      "id": "home-improvement-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Home & Garden > Home Improvement.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need practical advice for upgrading an older house, not interior decor inspiration"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Education",
+        "model_output.classification.iab_content.tier2.label": "Language Learning"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Education",
+        "model_output.classification.iab_content.tier2.label": "Language Learning"
+      },
+      "id": "online-education-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Education > Online Education.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best online courses for learning Python"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Careers",
+        "model_output.classification.iab_content.tier2.label": "Remote Working"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Careers",
+        "model_output.classification.iab_content.tier2.label": "Remote Working"
+      },
+      "id": "online-education-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Education > Online Education.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "What are good platforms for remote professional classes?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Careers"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Careers"
+      },
+      "id": "online-education-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Education > Online Education.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Need internet-based training options I can finish after work hours"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Education",
+        "model_output.classification.iab_content.tier2.label": "College Education"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Education",
+        "model_output.classification.iab_content.tier2.label": "College Education"
+      },
+      "id": "postgraduate-education-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Education > College Education > Postgraduate Education.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "best universities to study masters"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Education",
+        "model_output.classification.iab_content.tier2.label": "College Education",
+        "model_output.classification.iab_content.tier3.label": "Postgraduate Education",
+        "model_output.classification.iab_content.tier4.label": "Professional School"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Education",
+        "model_output.classification.iab_content.tier2.label": "College Education",
+        "model_output.classification.iab_content.tier3.label": "Postgraduate Education",
+        "model_output.classification.iab_content.tier4.label": "Professional School"
+      },
+      "id": "postgraduate-education-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Education > College Education > Postgraduate Education.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "which graduate schools have strong data science programs"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Education",
+        "model_output.classification.iab_content.tier2.label": "College Education"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Education",
+        "model_output.classification.iab_content.tier2.label": "College Education"
+      },
+      "id": "postgraduate-education-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Education > College Education > Postgraduate Education.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "need postgraduate options for a master's degree, not short online courses"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Medical Health",
+        "model_output.classification.iab_content.tier2.label": "Diseases and Conditions",
+        "model_output.classification.iab_content.tier3.label": "Allergies"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Medical Health",
+        "model_output.classification.iab_content.tier2.label": "Diseases and Conditions",
+        "model_output.classification.iab_content.tier3.label": "Allergies"
+      },
+      "id": "medical-health-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Medical Health.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "what do these allergy symptoms mean"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Medical Health",
+        "model_output.classification.iab_content.tier2.label": "Diseases and Conditions",
+        "model_output.classification.iab_content.tier3.label": "Injuries",
+        "model_output.classification.iab_content.tier4.label": "First Aid"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Medical Health",
+        "model_output.classification.iab_content.tier2.label": "Diseases and Conditions",
+        "model_output.classification.iab_content.tier3.label": "Injuries",
+        "model_output.classification.iab_content.tier4.label": "First Aid"
+      },
+      "id": "medical-health-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Medical Health.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "when should i see a doctor for persistent knee pain"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Healthy Living",
+        "model_output.classification.iab_content.tier2.label": "Wellness",
+        "model_output.classification.iab_content.tier3.label": "Physical Therapy"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Healthy Living",
+        "model_output.classification.iab_content.tier2.label": "Wellness",
+        "model_output.classification.iab_content.tier3.label": "Physical Therapy"
+      },
+      "id": "medical-health-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Medical Health.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "need medical advice about symptoms, not wellness or fitness tips"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Careers",
+        "model_output.classification.iab_content.tier2.label": "Remote Working"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Careers",
+        "model_output.classification.iab_content.tier2.label": "Remote Working"
+      },
+      "id": "careers-job-search-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Careers > Job Search.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "best remote jobs for data analysts"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Business",
+        "model_output.classification.iab_content.tier3.label": "Sales"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Business",
+        "model_output.classification.iab_content.tier3.label": "Sales"
+      },
+      "id": "careers-job-search-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Careers > Job Search.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "where should i look for product manager openings"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Careers",
+        "model_output.classification.iab_content.tier2.label": "Job Search"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Careers",
+        "model_output.classification.iab_content.tier2.label": "Job Search"
+      },
+      "id": "careers-job-search-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Careers > Job Search.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "need help finding a new role and preparing for interviews"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink",
+        "model_output.classification.iab_content.tier2.label": "Food Movements"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink",
+        "model_output.classification.iab_content.tier2.label": "Food Movements"
+      },
+      "id": "personal-finance-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Personal Finance > Financial Planning.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "how much should i save each month"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Economy",
+        "model_output.classification.iab_content.tier3.label": "Financial Reform"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Economy",
+        "model_output.classification.iab_content.tier3.label": "Financial Reform"
+      },
+      "id": "personal-finance-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Personal Finance > Financial Planning.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "best budgeting approach for a growing family"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Personal Finance",
+        "model_output.classification.iab_content.tier2.label": "Retirement Planning"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Personal Finance",
+        "model_output.classification.iab_content.tier2.label": "Retirement Planning"
+      },
+      "id": "personal-finance-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Personal Finance > Financial Planning.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "need help planning savings and retirement, not business finance advice"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Family and Relationships",
+        "model_output.classification.iab_content.tier2.label": "Parenting"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Family and Relationships",
+        "model_output.classification.iab_content.tier2.label": "Parenting"
+      },
+      "id": "parenting-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Family and Relationships > Parenting.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "tips for parenting a toddler"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Family and Relationships",
+        "model_output.classification.iab_content.tier2.label": "Parenting"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Family and Relationships",
+        "model_output.classification.iab_content.tier2.label": "Parenting"
+      },
+      "id": "parenting-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Family and Relationships > Parenting.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "how do i help my teenager spend less time online"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Family and Relationships",
+        "model_output.classification.iab_content.tier2.label": "Parenting",
+        "model_output.classification.iab_content.tier3.label": "Special Needs Kids"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Family and Relationships",
+        "model_output.classification.iab_content.tier2.label": "Parenting",
+        "model_output.classification.iab_content.tier3.label": "Special Needs Kids"
+      },
+      "id": "parenting-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Family and Relationships > Parenting.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "need parenting advice for a child starting preschool"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Home & Garden"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Home & Garden"
+      },
+      "id": "gardening-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Home & Garden > Gardening.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "best plants for a small balcony garden"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Home & Garden",
+        "model_output.classification.iab_content.tier2.label": "Gardening"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Home & Garden",
+        "model_output.classification.iab_content.tier2.label": "Gardening"
+      },
+      "id": "gardening-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Home & Garden > Gardening.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "how often should i water tomato plants"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Home & Garden"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Home & Garden"
+      },
+      "id": "gardening-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Home & Garden > Gardening.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "need gardening advice for a shady backyard, not interior decor ideas"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Entertainment",
+        "model_output.classification.iab_content.tier2.label": "Movies"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Entertainment",
+        "model_output.classification.iab_content.tier2.label": "Movies"
+      },
+      "id": "movies-easy",
+      "mismatches": [],
+      "notes": "Cross-vertical easy IAB mapping case for Entertainment > Movies.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "What movie should we watch tonight?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Video Gaming",
+        "model_output.classification.iab_content.tier2.label": "Video Game Genres",
+        "model_output.classification.iab_content.tier3.label": "Horror Video Games"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Video Gaming",
+        "model_output.classification.iab_content.tier2.label": "Video Game Genres",
+        "model_output.classification.iab_content.tier3.label": "Horror Video Games"
+      },
+      "id": "movies-medium",
+      "mismatches": [],
+      "notes": "Cross-vertical medium IAB mapping case for Entertainment > Movies.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best thriller movies from the last few years"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Entertainment",
+        "model_output.classification.iab_content.tier2.label": "Music",
+        "model_output.classification.iab_content.tier3.label": "Soundtracks, TV and Showtunes"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Entertainment",
+        "model_output.classification.iab_content.tier2.label": "Music",
+        "model_output.classification.iab_content.tier3.label": "Soundtracks, TV and Showtunes"
+      },
+      "id": "movies-hard",
+      "mismatches": [],
+      "notes": "Cross-vertical hard IAB mapping case for Entertainment > Movies.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Looking for film recommendations, not TV shows or music"
+    }
+  ]
+}

artifacts/evaluation/latest/iab_mapping_regression.json

@@ -0,0 +1,245 @@
+{
+  "by_status": {
+    "must_fix": {
+      "failed": 0,
+      "passed": 12,
+      "total": 12
+    }
+  },
+  "cases_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/examples/iab_mapping_cases.json",
+  "count": 12,
+  "failed": 0,
+  "passed": 12,
+  "results": [
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Automotive",
+        "model_output.classification.iab_content.tier2.label": "Auto Type"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Automotive",
+        "model_output.classification.iab_content.tier2.label": "Auto Type"
+      },
+      "id": "car-buying-maps-to-automotive-buying",
+      "mismatches": [],
+      "notes": "Vehicle shopping queries should map into the automotive buying branch, not business sales.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Which car to buy in 2026"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Laptops"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Laptops"
+      },
+      "id": "laptop-buying-maps-to-laptops",
+      "mismatches": [],
+      "notes": "Laptop shopping should resolve into the laptops branch, not business sales.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Which laptop to buy in 2026"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Laptops"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Laptops"
+      },
+      "id": "labtop-buying-maps-to-laptops",
+      "mismatches": [],
+      "notes": "Common typo handling should still land in the laptops branch.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Which labtop to buy in 2026"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+      },
+      "id": "crm-awareness-maps-to-sales",
+      "mismatches": [],
+      "notes": "CRM education should resolve to the closest business/sales path, not generic software.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "What is CRM software?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "id": "crm-comparison-maps-to-sales",
+      "mismatches": [],
+      "notes": "Direct CRM vendor comparison should map cleanly into the sales domain.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "HubSpot vs Zoho for a small team"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "id": "marketing-tools-map-to-marketing",
+      "mismatches": [],
+      "notes": "Marketing tool discovery should map to the marketing and advertising branch.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Best AI SEO tools for content teams"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing"
+      },
+      "id": "ml-explanation-maps-to-ai",
+      "mismatches": [],
+      "notes": "ML and NLP educational prompts should land in the AI branch.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "What is intent classification in NLP?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "id": "support-credential-help-maps-to-business-it",
+      "mismatches": [],
+      "notes": "Credential and account help should map to business IT rather than generic business.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "How do I reset my password?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink",
+        "model_output.classification.iab_content.tier2.label": "Dining Out"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink",
+        "model_output.classification.iab_content.tier2.label": "Dining Out"
+      },
+      "id": "restaurant-booking-maps-to-dining-out",
+      "mismatches": [],
+      "notes": "Generic dining requests should not inherit the repo's business default.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Book a table for 2 tonight"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Hobbies & Interests",
+        "model_output.classification.iab_content.tier2.label": "Content Production",
+        "model_output.classification.iab_content.tier3.label": "Freelance Writing"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Hobbies & Interests",
+        "model_output.classification.iab_content.tier2.label": "Content Production",
+        "model_output.classification.iab_content.tier3.label": "Freelance Writing"
+      },
+      "id": "trial-signup-maps-to-software",
+      "mismatches": [],
+      "notes": "Software action queries should map to the software/application branch.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "Start my free trial"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications",
+        "model_output.classification.iab_content.tier4.label": "Communication"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications",
+        "model_output.classification.iab_content.tier4.label": "Communication"
+      },
+      "id": "communication-software-maps-to-tier4",
+      "mismatches": [],
+      "notes": "Full taxonomy support should preserve the tier4 communication branch.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "best communication software for remote teams"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink"
+      },
+      "id": "vodka-query-maps-to-alcoholic-beverages",
+      "mismatches": [],
+      "notes": "Food and beverage prompts should not fall through to the business default.",
+      "pass": true,
+      "status": "must_fix",
+      "text": "what is best vodka drink should i try"
+    }
+  ]
+}

artifacts/evaluation/latest/iab_quality_target_eval.json

@@ -0,0 +1,411 @@
+{
+  "by_status": {
+    "must_fix": {
+      "failed": 12,
+      "passed": 0,
+      "total": 12
+    }
+  },
+  "cases_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/examples/iab_mapping_cases.json",
+  "count": 12,
+  "failed": 12,
+  "passed": 0,
+  "results": [
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Automotive",
+        "model_output.classification.iab_content.tier2.label": null
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Automotive",
+        "model_output.classification.iab_content.tier2.label": "Auto Buying and Selling"
+      },
+      "id": "car-buying-maps-to-automotive-buying",
+      "mismatches": [
+        {
+          "actual": null,
+          "expected": "Auto Buying and Selling",
+          "path": "model_output.classification.iab_content.tier2.label"
+        },
+        {
+          "actual": "nearest_equivalent",
+          "expected": "exact",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        }
+      ],
+      "notes": "Vehicle shopping queries should map into the automotive buying branch, not business sales.",
+      "pass": false,
+      "status": "must_fix",
+      "text": "Which car to buy in 2026"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Laptops"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Laptops"
+      },
+      "id": "laptop-buying-maps-to-laptops",
+      "mismatches": [
+        {
+          "actual": "nearest_equivalent",
+          "expected": "exact",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        }
+      ],
+      "notes": "Laptop shopping should resolve into the laptops branch, not business sales.",
+      "pass": false,
+      "status": "must_fix",
+      "text": "Which laptop to buy in 2026"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": null
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Laptops"
+      },
+      "id": "labtop-buying-maps-to-laptops",
+      "mismatches": [
+        {
+          "actual": null,
+          "expected": "Laptops",
+          "path": "model_output.classification.iab_content.tier3.label"
+        },
+        {
+          "actual": "nearest_equivalent",
+          "expected": "exact",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        }
+      ],
+      "notes": "Common typo handling should still land in the laptops branch.",
+      "pass": false,
+      "status": "must_fix",
+      "text": "Which labtop to buy in 2026"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Business",
+        "model_output.classification.iab_content.tier3.label": "Sales"
+      },
+      "id": "crm-awareness-maps-to-sales",
+      "mismatches": [
+        {
+          "actual": "Technology & Computing",
+          "expected": "Business and Finance",
+          "path": "model_output.classification.iab_content.tier1.label"
+        },
+        {
+          "actual": "Computing",
+          "expected": "Business",
+          "path": "model_output.classification.iab_content.tier2.label"
+        },
+        {
+          "actual": "Software and Applications",
+          "expected": "Sales",
+          "path": "model_output.classification.iab_content.tier3.label"
+        }
+      ],
+      "notes": "CRM education should resolve to the closest business/sales path, not generic software.",
+      "pass": false,
+      "status": "must_fix",
+      "text": "What is CRM software?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Business",
+        "model_output.classification.iab_content.tier3.label": "Sales"
+      },
+      "id": "crm-comparison-maps-to-sales",
+      "mismatches": [
+        {
+          "actual": "Technology & Computing",
+          "expected": "Business and Finance",
+          "path": "model_output.classification.iab_content.tier1.label"
+        },
+        {
+          "actual": "Computing",
+          "expected": "Business",
+          "path": "model_output.classification.iab_content.tier2.label"
+        },
+        {
+          "actual": "Software and Applications",
+          "expected": "Sales",
+          "path": "model_output.classification.iab_content.tier3.label"
+        },
+        {
+          "actual": "nearest_equivalent",
+          "expected": "exact",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        }
+      ],
+      "notes": "Direct CRM vendor comparison should map cleanly into the sales domain.",
+      "pass": false,
+      "status": "must_fix",
+      "text": "HubSpot vs Zoho for a small team"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Hobbies & Interests",
+        "model_output.classification.iab_content.tier2.label": "Content Production",
+        "model_output.classification.iab_content.tier3.label": null
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Business",
+        "model_output.classification.iab_content.tier3.label": "Marketing and Advertising"
+      },
+      "id": "marketing-tools-map-to-marketing",
+      "mismatches": [
+        {
+          "actual": "Hobbies & Interests",
+          "expected": "Business and Finance",
+          "path": "model_output.classification.iab_content.tier1.label"
+        },
+        {
+          "actual": "Content Production",
+          "expected": "Business",
+          "path": "model_output.classification.iab_content.tier2.label"
+        },
+        {
+          "actual": null,
+          "expected": "Marketing and Advertising",
+          "path": "model_output.classification.iab_content.tier3.label"
+        },
+        {
+          "actual": "nearest_equivalent",
+          "expected": "exact",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        }
+      ],
+      "notes": "Marketing tool discovery should map to the marketing and advertising branch.",
+      "pass": false,
+      "status": "must_fix",
+      "text": "Best AI SEO tools for content teams"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Artificial Intelligence"
+      },
+      "id": "ml-explanation-maps-to-ai",
+      "mismatches": [
+        {
+          "actual": "Computing",
+          "expected": "Artificial Intelligence",
+          "path": "model_output.classification.iab_content.tier2.label"
+        },
+        {
+          "actual": "nearest_equivalent",
+          "expected": "exact",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        }
+      ],
+      "notes": "ML and NLP educational prompts should land in the AI branch.",
+      "pass": false,
+      "status": "must_fix",
+      "text": "What is intent classification in NLP?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Internet"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Business",
+        "model_output.classification.iab_content.tier3.label": "Business I.T."
+      },
+      "id": "support-credential-help-maps-to-business-it",
+      "mismatches": [
+        {
+          "actual": "Technology & Computing",
+          "expected": "Business and Finance",
+          "path": "model_output.classification.iab_content.tier1.label"
+        },
+        {
+          "actual": "Computing",
+          "expected": "Business",
+          "path": "model_output.classification.iab_content.tier2.label"
+        },
+        {
+          "actual": "Internet",
+          "expected": "Business I.T.",
+          "path": "model_output.classification.iab_content.tier3.label"
+        }
+      ],
+      "notes": "Credential and account help should map to business IT rather than generic business.",
+      "pass": false,
+      "status": "must_fix",
+      "text": "How do I reset my password?"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink",
+        "model_output.classification.iab_content.tier2.label": "Dining Out"
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink",
+        "model_output.classification.iab_content.tier2.label": "Dining Out"
+      },
+      "id": "restaurant-booking-maps-to-dining-out",
+      "mismatches": [
+        {
+          "actual": "nearest_equivalent",
+          "expected": "exact",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        }
+      ],
+      "notes": "Generic dining requests should not inherit the repo's business default.",
+      "pass": false,
+      "status": "must_fix",
+      "text": "Book a table for 2 tonight"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": "Business",
+        "model_output.classification.iab_content.tier3.label": null
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+      },
+      "id": "trial-signup-maps-to-software",
+      "mismatches": [
+        {
+          "actual": "Business and Finance",
+          "expected": "Technology & Computing",
+          "path": "model_output.classification.iab_content.tier1.label"
+        },
+        {
+          "actual": "Business",
+          "expected": "Computing",
+          "path": "model_output.classification.iab_content.tier2.label"
+        },
+        {
+          "actual": null,
+          "expected": "Software and Applications",
+          "path": "model_output.classification.iab_content.tier3.label"
+        }
+      ],
+      "notes": "Software action queries should map to the software/application branch.",
+      "pass": false,
+      "status": "must_fix",
+      "text": "Start my free trial"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Software and Applications",
+        "model_output.classification.iab_content.tier4.label": null
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier3.label": "Computer Software and Applications",
+        "model_output.classification.iab_content.tier4.label": "Communication"
+      },
+      "id": "communication-software-maps-to-tier4",
+      "mismatches": [
+        {
+          "actual": "Software and Applications",
+          "expected": "Computer Software and Applications",
+          "path": "model_output.classification.iab_content.tier3.label"
+        },
+        {
+          "actual": null,
+          "expected": "Communication",
+          "path": "model_output.classification.iab_content.tier4.label"
+        },
+        {
+          "actual": "nearest_equivalent",
+          "expected": "exact",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        }
+      ],
+      "notes": "Full taxonomy support should preserve the tier4 communication branch.",
+      "pass": false,
+      "status": "must_fix",
+      "text": "best communication software for remote teams"
+    },
+    {
+      "actual": {
+        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink",
+        "model_output.classification.iab_content.tier2.label": null
+      },
+      "expected": {
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink",
+        "model_output.classification.iab_content.tier2.label": "Alcoholic Beverages"
+      },
+      "id": "vodka-query-maps-to-alcoholic-beverages",
+      "mismatches": [
+        {
+          "actual": null,
+          "expected": "Alcoholic Beverages",
+          "path": "model_output.classification.iab_content.tier2.label"
+        },
+        {
+          "actual": "nearest_equivalent",
+          "expected": "exact",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        }
+      ],
+      "notes": "Food and beverage prompts should not fall through to the business default.",
+      "pass": false,
+      "status": "must_fix",
+      "text": "what is best vodka drink should i try"
+    }
+  ]
+}

artifacts/evaluation/latest/intent_subtype_difficulty_benchmark_confusion_matrix.csv

@@ -0,0 +1,19 @@
+,education,product_discovery,comparison,evaluation,deal_seeking,provider_selection,signup,purchase,booking,download,contact_sales,task_execution,onboarding_setup,troubleshooting,account_help,billing_help,follow_up,emotional_reflection
+education,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+product_discovery,0,13,0,0,1,0,0,0,0,0,0,1,0,0,0,0,0,0
+comparison,2,0,13,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+evaluation,1,0,0,14,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+deal_seeking,0,0,0,0,14,1,0,0,0,0,0,0,0,0,0,0,0,0
+provider_selection,0,0,1,0,0,15,0,0,0,0,0,0,0,0,0,0,0,0
+signup,0,0,0,0,0,0,16,0,0,0,0,0,0,0,0,0,0,0
+purchase,0,0,0,0,0,0,0,14,0,0,0,0,1,0,0,0,0,0
+booking,0,0,0,0,0,0,1,0,13,0,0,1,0,0,0,0,0,0
+download,0,0,0,0,0,0,0,0,0,14,0,1,0,0,0,0,0,0
+contact_sales,0,0,0,0,0,0,0,0,0,0,15,0,0,0,0,0,0,0
+task_execution,0,0,0,0,0,0,1,0,0,0,0,17,0,0,0,0,0,0
+onboarding_setup,0,0,0,0,0,0,0,0,0,0,0,1,16,0,0,0,0,0
+troubleshooting,0,0,0,0,0,0,0,0,0,0,0,0,0,14,0,0,0,1
+account_help,0,0,0,0,0,0,2,0,0,0,0,0,0,0,12,1,0,0
+billing_help,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,15,0,0
+follow_up,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,15,0
+emotional_reflection,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,15

artifacts/evaluation/latest/intent_subtype_difficulty_benchmark_report.json

@@ -0,0 +1,161 @@
+{
+  "accepted_accuracy": 0.9386,
+  "accepted_coverage": 1.0,
+  "accuracy": 0.9386,
+  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/intent_subtype_difficulty_benchmark_confusion_matrix.csv",
+  "count": 277,
+  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/subtype_benchmark.jsonl",
+  "difficulty_breakdown": {
+    "easy": {
+      "accepted_accuracy": 0.9565,
+      "accepted_coverage": 1.0,
+      "accuracy": 0.9565,
+      "count": 92,
+      "fallback_rate": 0.0,
+      "macro_f1": 0.9579
+    },
+    "hard": {
+      "accepted_accuracy": 0.8901,
+      "accepted_coverage": 1.0,
+      "accuracy": 0.8901,
+      "count": 91,
+      "fallback_rate": 0.0,
+      "macro_f1": 0.8913
+    },
+    "medium": {
+      "accepted_accuracy": 0.9681,
+      "accepted_coverage": 1.0,
+      "accuracy": 0.9681,
+      "count": 94,
+      "fallback_rate": 0.0,
+      "macro_f1": 0.9671
+    }
+  },
+  "fallback_rate": 0.0,
+  "head": "intent_subtype",
+  "macro_f1": 0.9401,
+  "per_class_metrics": {
+    "account_help": {
+      "f1-score": 0.8888888888888888,
+      "precision": 1.0,
+      "recall": 0.8,
+      "support": 15.0
+    },
+    "accuracy": 0.9386281588447654,
+    "billing_help": {
+      "f1-score": 0.967741935483871,
+      "precision": 0.9375,
+      "recall": 1.0,
+      "support": 15.0
+    },
+    "booking": {
+      "f1-score": 0.9285714285714286,
+      "precision": 1.0,
+      "recall": 0.8666666666666667,
+      "support": 15.0
+    },
+    "comparison": {
+      "f1-score": 0.896551724137931,
+      "precision": 0.9285714285714286,
+      "recall": 0.8666666666666667,
+      "support": 15.0
+    },
+    "contact_sales": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 15.0
+    },
+    "deal_seeking": {
+      "f1-score": 0.9333333333333333,
+      "precision": 0.9333333333333333,
+      "recall": 0.9333333333333333,
+      "support": 15.0
+    },
+    "download": {
+      "f1-score": 0.9655172413793104,
+      "precision": 1.0,
+      "recall": 0.9333333333333333,
+      "support": 15.0
+    },
+    "education": {
+      "f1-score": 0.9090909090909091,
+      "precision": 0.8333333333333334,
+      "recall": 1.0,
+      "support": 15.0
+    },
+    "emotional_reflection": {
+      "f1-score": 0.967741935483871,
+      "precision": 0.9375,
+      "recall": 1.0,
+      "support": 15.0
+    },
+    "evaluation": {
+      "f1-score": 0.9655172413793104,
+      "precision": 1.0,
+      "recall": 0.9333333333333333,
+      "support": 15.0
+    },
+    "follow_up": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 15.0
+    },
+    "macro avg": {
+      "f1-score": 0.9401067100194944,
+      "precision": 0.9476910208527856,
+      "recall": 0.9383215323166303,
+      "support": 277.0
+    },
+    "onboarding_setup": {
+      "f1-score": 0.9411764705882353,
+      "precision": 0.9411764705882353,
+      "recall": 0.9411764705882353,
+      "support": 17.0
+    },
+    "product_discovery": {
+      "f1-score": 0.9285714285714286,
+      "precision": 1.0,
+      "recall": 0.8666666666666667,
+      "support": 15.0
+    },
+    "provider_selection": {
+      "f1-score": 0.9375,
+      "precision": 0.9375,
+      "recall": 0.9375,
+      "support": 16.0
+    },
+    "purchase": {
+      "f1-score": 0.9655172413793104,
+      "precision": 1.0,
+      "recall": 0.9333333333333333,
+      "support": 15.0
+    },
+    "signup": {
+      "f1-score": 0.8888888888888888,
+      "precision": 0.8,
+      "recall": 1.0,
+      "support": 16.0
+    },
+    "task_execution": {
+      "f1-score": 0.8717948717948718,
+      "precision": 0.8095238095238095,
+      "recall": 0.9444444444444444,
+      "support": 18.0
+    },
+    "troubleshooting": {
+      "f1-score": 0.9655172413793104,
+      "precision": 1.0,
+      "recall": 0.9333333333333333,
+      "support": 15.0
+    },
+    "weighted avg": {
+      "f1-score": 0.9391802821325396,
+      "precision": 0.9455776173285199,
+      "recall": 0.9386281588447654,
+      "support": 277.0
+    }
+  },
+  "suite": "difficulty_benchmark"
+}

artifacts/evaluation/latest/intent_subtype_extended_cases_confusion_matrix.csv

@@ -0,0 +1,19 @@
+,education,product_discovery,comparison,evaluation,deal_seeking,provider_selection,signup,purchase,booking,download,contact_sales,task_execution,onboarding_setup,troubleshooting,account_help,billing_help,follow_up,emotional_reflection
+education,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+product_discovery,0,6,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+comparison,0,0,1,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0
+evaluation,1,0,0,1,0,1,0,0,0,0,0,0,0,0,0,0,0,0
+deal_seeking,0,0,0,0,9,0,0,0,0,0,0,0,0,0,0,0,0,0
+provider_selection,0,0,0,0,0,5,0,0,0,0,0,0,0,0,0,0,0,0
+signup,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+purchase,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+booking,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+download,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+contact_sales,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+task_execution,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0
+onboarding_setup,0,0,0,0,0,0,0,0,0,0,0,0,4,0,0,0,0,0
+troubleshooting,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0
+account_help,0,0,0,0,0,0,0,0,0,0,0,0,0,1,2,0,0,0
+billing_help,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+follow_up,0,0,0,0,3,0,0,0,0,0,0,0,0,0,1,0,8,0
+emotional_reflection,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0

artifacts/evaluation/latest/intent_subtype_extended_cases_report.json

@@ -0,0 +1,135 @@
+{
+  "accepted_accuracy": 0.8491,
+  "accepted_coverage": 1.0,
+  "accuracy": 0.8491,
+  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/intent_subtype_extended_cases_confusion_matrix.csv",
+  "count": 53,
+  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/subtype/extended_cases.jsonl",
+  "fallback_rate": 0.0,
+  "head": "intent_subtype",
+  "macro_f1": 0.8146,
+  "per_class_metrics": {
+    "account_help": {
+      "f1-score": 0.6666666666666666,
+      "precision": 0.6666666666666666,
+      "recall": 0.6666666666666666,
+      "support": 3.0
+    },
+    "accuracy": 0.8490566037735849,
+    "billing_help": {
+      "f1-score": 0.0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "support": 0.0
+    },
+    "booking": {
+      "f1-score": 0.0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "support": 0.0
+    },
+    "comparison": {
+      "f1-score": 0.6666666666666666,
+      "precision": 1.0,
+      "recall": 0.5,
+      "support": 2.0
+    },
+    "contact_sales": {
+      "f1-score": 0.0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "support": 0.0
+    },
+    "deal_seeking": {
+      "f1-score": 0.8181818181818182,
+      "precision": 0.6923076923076923,
+      "recall": 1.0,
+      "support": 9.0
+    },
+    "download": {
+      "f1-score": 0.0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "support": 0.0
+    },
+    "education": {
+      "f1-score": 0.9333333333333333,
+      "precision": 0.875,
+      "recall": 1.0,
+      "support": 7.0
+    },
+    "emotional_reflection": {
+      "f1-score": 0.0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "support": 0.0
+    },
+    "evaluation": {
+      "f1-score": 0.5,
+      "precision": 1.0,
+      "recall": 0.3333333333333333,
+      "support": 3.0
+    },
+    "follow_up": {
+      "f1-score": 0.8,
+      "precision": 1.0,
+      "recall": 0.6666666666666666,
+      "support": 12.0
+    },
+    "macro avg": {
+      "f1-score": 0.4978114478114478,
+      "precision": 0.531517094017094,
+      "recall": 0.5092592592592592,
+      "support": 53.0
+    },
+    "onboarding_setup": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 4.0
+    },
+    "product_discovery": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 6.0
+    },
+    "provider_selection": {
+      "f1-score": 0.9090909090909091,
+      "precision": 0.8333333333333334,
+      "recall": 1.0,
+      "support": 5.0
+    },
+    "purchase": {
+      "f1-score": 0.0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "support": 0.0
+    },
+    "signup": {
+      "f1-score": 0.0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "support": 0.0
+    },
+    "task_execution": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 1.0
+    },
+    "troubleshooting": {
+      "f1-score": 0.6666666666666666,
+      "precision": 0.5,
+      "recall": 1.0,
+      "support": 1.0
+    },
+    "weighted avg": {
+      "f1-score": 0.8404230989136648,
+      "precision": 0.887215771649734,
+      "recall": 0.8490566037735849,
+      "support": 53.0
+    }
+  },
+  "suite": "extended_cases"
+}

artifacts/evaluation/latest/intent_subtype_hard_cases_confusion_matrix.csv

@@ -0,0 +1,19 @@
+,education,product_discovery,comparison,evaluation,deal_seeking,provider_selection,signup,purchase,booking,download,contact_sales,task_execution,onboarding_setup,troubleshooting,account_help,billing_help,follow_up,emotional_reflection
+education,29,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+product_discovery,0,8,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+comparison,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+evaluation,0,2,0,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+deal_seeking,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0
+provider_selection,0,0,0,1,0,9,0,0,0,0,0,0,0,0,0,0,0,0
+signup,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0,0
+purchase,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0
+booking,0,0,0,0,0,0,0,0,4,0,0,0,0,0,0,0,0,0
+download,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+contact_sales,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+task_execution,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0
+onboarding_setup,0,0,0,0,0,0,0,0,0,0,0,0,6,0,0,0,0,0
+troubleshooting,0,0,0,0,0,0,0,0,0,0,0,0,0,2,1,0,0,0
+account_help,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0
+billing_help,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0
+follow_up,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,11,0
+emotional_reflection,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0

artifacts/evaluation/latest/intent_subtype_hard_cases_report.json

@@ -0,0 +1,135 @@
+{
+  "accepted_accuracy": 0.9468,
+  "accepted_coverage": 1.0,
+  "accuracy": 0.9468,
+  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/intent_subtype_hard_cases_confusion_matrix.csv",
+  "count": 94,
+  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/subtype/hard_cases.jsonl",
+  "fallback_rate": 0.0,
+  "head": "intent_subtype",
+  "macro_f1": 0.9191,
+  "per_class_metrics": {
+    "account_help": {
+      "f1-score": 0.8,
+      "precision": 0.6666666666666666,
+      "recall": 1.0,
+      "support": 2.0
+    },
+    "accuracy": 0.9468085106382979,
+    "billing_help": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 1.0
+    },
+    "booking": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 4.0
+    },
+    "comparison": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 3.0
+    },
+    "contact_sales": {
+      "f1-score": 0.0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "support": 0.0
+    },
+    "deal_seeking": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 3.0
+    },
+    "download": {
+      "f1-score": 0.0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "support": 0.0
+    },
+    "education": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 29.0
+    },
+    "emotional_reflection": {
+      "f1-score": 0.0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "support": 0.0
+    },
+    "evaluation": {
+      "f1-score": 0.7272727272727273,
+      "precision": 0.8,
+      "recall": 0.6666666666666666,
+      "support": 6.0
+    },
+    "follow_up": {
+      "f1-score": 0.9565217391304348,
+      "precision": 1.0,
+      "recall": 0.9166666666666666,
+      "support": 12.0
+    },
+    "macro avg": {
+      "f1-score": 0.7659288023895194,
+      "precision": 0.7648148148148147,
+      "recall": 0.786111111111111,
+      "support": 94.0
+    },
+    "onboarding_setup": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 6.0
+    },
+    "product_discovery": {
+      "f1-score": 0.8888888888888888,
+      "precision": 0.8,
+      "recall": 1.0,
+      "support": 8.0
+    },
+    "provider_selection": {
+      "f1-score": 0.9473684210526315,
+      "precision": 1.0,
+      "recall": 0.9,
+      "support": 10.0
+    },
+    "purchase": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 3.0
+    },
+    "signup": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 3.0
+    },
+    "task_execution": {
+      "f1-score": 0.6666666666666666,
+      "precision": 0.5,
+      "recall": 1.0,
+      "support": 1.0
+    },
+    "troubleshooting": {
+      "f1-score": 0.8,
+      "precision": 1.0,
+      "recall": 0.6666666666666666,
+      "support": 3.0
+    },
+    "weighted avg": {
+      "f1-score": 0.9478016938458051,
+      "precision": 0.9578014184397163,
+      "recall": 0.9468085106382979,
+      "support": 94.0
+    }
+  },
+  "suite": "hard_cases"
+}

artifacts/evaluation/latest/intent_subtype_test_confusion_matrix.csv

@@ -0,0 +1,19 @@
+,education,product_discovery,comparison,evaluation,deal_seeking,provider_selection,signup,purchase,booking,download,contact_sales,task_execution,onboarding_setup,troubleshooting,account_help,billing_help,follow_up,emotional_reflection
+education,14,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+product_discovery,0,7,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+comparison,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+evaluation,0,1,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+deal_seeking,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0
+provider_selection,0,0,0,1,0,5,0,0,0,0,0,0,0,0,0,0,0,0
+signup,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0
+purchase,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+booking,0,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0
+download,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+contact_sales,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+task_execution,0,0,0,0,0,0,0,0,0,0,0,6,0,0,0,0,0,0
+onboarding_setup,0,0,0,0,0,0,0,0,0,0,0,0,4,0,0,0,0,0
+troubleshooting,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0
+account_help,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0
+billing_help,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+follow_up,0,0,0,0,2,0,0,0,0,0,0,0,0,1,0,0,8,0
+emotional_reflection,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,5

artifacts/evaluation/latest/intent_subtype_test_report.json

@@ -0,0 +1,135 @@
+{
+  "accepted_accuracy": 0.9,
+  "accepted_coverage": 1.0,
+  "accuracy": 0.9,
+  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/intent_subtype_test_confusion_matrix.csv",
+  "count": 70,
+  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/subtype/test.jsonl",
+  "fallback_rate": 0.0,
+  "head": "intent_subtype",
+  "macro_f1": 0.863,
+  "per_class_metrics": {
+    "account_help": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 2.0
+    },
+    "accuracy": 0.9,
+    "billing_help": {
+      "f1-score": 0.0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "support": 0.0
+    },
+    "booking": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 3.0
+    },
+    "comparison": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 3.0
+    },
+    "contact_sales": {
+      "f1-score": 0.0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "support": 0.0
+    },
+    "deal_seeking": {
+      "f1-score": 0.6666666666666666,
+      "precision": 0.5,
+      "recall": 1.0,
+      "support": 2.0
+    },
+    "download": {
+      "f1-score": 0.0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "support": 0.0
+    },
+    "education": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 14.0
+    },
+    "emotional_reflection": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 5.0
+    },
+    "evaluation": {
+      "f1-score": 0.4,
+      "precision": 0.3333333333333333,
+      "recall": 0.5,
+      "support": 2.0
+    },
+    "follow_up": {
+      "f1-score": 0.8421052631578947,
+      "precision": 1.0,
+      "recall": 0.7272727272727273,
+      "support": 11.0
+    },
+    "macro avg": {
+      "f1-score": 0.6712084293224644,
+      "precision": 0.6671296296296296,
+      "recall": 0.6908670033670034,
+      "support": 70.0
+    },
+    "onboarding_setup": {
+      "f1-score": 0.8888888888888888,
+      "precision": 0.8,
+      "recall": 1.0,
+      "support": 4.0
+    },
+    "product_discovery": {
+      "f1-score": 0.875,
+      "precision": 0.875,
+      "recall": 0.875,
+      "support": 8.0
+    },
+    "provider_selection": {
+      "f1-score": 0.9090909090909091,
+      "precision": 1.0,
+      "recall": 0.8333333333333334,
+      "support": 6.0
+    },
+    "purchase": {
+      "f1-score": 0.0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "support": 0.0
+    },
+    "signup": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 2.0
+    },
+    "task_execution": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 6.0
+    },
+    "troubleshooting": {
+      "f1-score": 0.5,
+      "precision": 0.5,
+      "recall": 0.5,
+      "support": 2.0
+    },
+    "weighted avg": {
+      "f1-score": 0.9058084605453025,
+      "precision": 0.9266666666666667,
+      "recall": 0.9,
+      "support": 70.0
+    }
+  },
+  "suite": "test"
+}

artifacts/evaluation/latest/intent_subtype_train_confusion_matrix.csv

@@ -0,0 +1,19 @@
+,education,product_discovery,comparison,evaluation,deal_seeking,provider_selection,signup,purchase,booking,download,contact_sales,task_execution,onboarding_setup,troubleshooting,account_help,billing_help,follow_up,emotional_reflection
+education,52,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+product_discovery,0,31,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+comparison,0,0,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+evaluation,1,2,0,14,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+deal_seeking,0,0,0,0,11,0,0,0,0,0,0,0,0,0,0,0,0,0
+provider_selection,0,0,0,0,0,24,0,0,0,0,0,0,0,0,0,0,1,0
+signup,0,0,0,0,0,0,16,0,0,0,0,0,0,0,0,0,0,0
+purchase,0,0,0,0,0,0,0,6,0,0,0,0,0,0,0,0,0,0
+booking,0,0,0,0,0,0,0,0,5,0,0,0,0,0,0,0,0,0
+download,0,0,0,0,0,0,0,0,0,8,0,0,0,0,0,0,0,0
+contact_sales,0,0,0,0,0,0,2,0,0,0,7,0,0,0,0,0,0,0
+task_execution,0,0,0,0,0,0,0,0,0,0,0,19,0,0,0,0,0,0
+onboarding_setup,0,0,0,0,0,0,0,0,0,0,0,0,17,0,0,0,0,0
+troubleshooting,0,0,0,0,0,0,0,0,0,0,0,0,0,12,1,0,0,0
+account_help,0,0,0,0,0,0,0,0,0,0,0,0,0,0,7,0,0,0
+billing_help,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,6,0,0
+follow_up,0,0,0,0,0,0,0,0,0,0,0,3,0,1,0,0,32,0
+emotional_reflection,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,20

artifacts/evaluation/latest/intent_subtype_train_report.json

@@ -0,0 +1,135 @@
+{
+  "accepted_accuracy": 0.9649,
+  "accepted_coverage": 1.0,
+  "accuracy": 0.9649,
+  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/intent_subtype_train_confusion_matrix.csv",
+  "count": 313,
+  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/subtype/train.jsonl",
+  "fallback_rate": 0.0,
+  "head": "intent_subtype",
+  "macro_f1": 0.9649,
+  "per_class_metrics": {
+    "account_help": {
+      "f1-score": 0.9333333333333333,
+      "precision": 0.875,
+      "recall": 1.0,
+      "support": 7.0
+    },
+    "accuracy": 0.9648562300319489,
+    "billing_help": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 6.0
+    },
+    "booking": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 5.0
+    },
+    "comparison": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 15.0
+    },
+    "contact_sales": {
+      "f1-score": 0.875,
+      "precision": 1.0,
+      "recall": 0.7777777777777778,
+      "support": 9.0
+    },
+    "deal_seeking": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 11.0
+    },
+    "download": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 8.0
+    },
+    "education": {
+      "f1-score": 0.9904761904761905,
+      "precision": 0.9811320754716981,
+      "recall": 1.0,
+      "support": 52.0
+    },
+    "emotional_reflection": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 20.0
+    },
+    "evaluation": {
+      "f1-score": 0.9032258064516129,
+      "precision": 1.0,
+      "recall": 0.8235294117647058,
+      "support": 17.0
+    },
+    "follow_up": {
+      "f1-score": 0.927536231884058,
+      "precision": 0.9696969696969697,
+      "recall": 0.8888888888888888,
+      "support": 36.0
+    },
+    "macro avg": {
+      "f1-score": 0.9649442256020961,
+      "precision": 0.9689347311202658,
+      "recall": 0.9651818334171275,
+      "support": 313.0
+    },
+    "onboarding_setup": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 17.0
+    },
+    "product_discovery": {
+      "f1-score": 0.96875,
+      "precision": 0.9393939393939394,
+      "recall": 1.0,
+      "support": 31.0
+    },
+    "provider_selection": {
+      "f1-score": 0.9795918367346939,
+      "precision": 1.0,
+      "recall": 0.96,
+      "support": 25.0
+    },
+    "purchase": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 6.0
+    },
+    "signup": {
+      "f1-score": 0.9411764705882353,
+      "precision": 0.8888888888888888,
+      "recall": 1.0,
+      "support": 16.0
+    },
+    "task_execution": {
+      "f1-score": 0.926829268292683,
+      "precision": 0.8636363636363636,
+      "recall": 1.0,
+      "support": 19.0
+    },
+    "troubleshooting": {
+      "f1-score": 0.9230769230769231,
+      "precision": 0.9230769230769231,
+      "recall": 0.9230769230769231,
+      "support": 13.0
+    },
+    "weighted avg": {
+      "f1-score": 0.9643733669039578,
+      "precision": 0.967429661617075,
+      "recall": 0.9648562300319489,
+      "support": 313.0
+    }
+  },
+  "suite": "train"
+}

artifacts/evaluation/latest/intent_subtype_val_confusion_matrix.csv

@@ -0,0 +1,19 @@
+,education,product_discovery,comparison,evaluation,deal_seeking,provider_selection,signup,purchase,booking,download,contact_sales,task_execution,onboarding_setup,troubleshooting,account_help,billing_help,follow_up,emotional_reflection
+education,14,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+product_discovery,0,10,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0
+comparison,0,0,1,2,1,0,0,0,0,0,0,0,0,0,0,0,0,0
+evaluation,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+deal_seeking,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0
+provider_selection,0,0,0,0,0,7,0,0,0,0,0,0,0,0,0,0,0,0
+signup,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0
+purchase,0,0,0,0,0,0,1,0,0,0,0,0,1,0,0,0,0,0
+booking,0,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0
+download,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+contact_sales,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+task_execution,0,0,0,0,0,0,0,0,0,0,0,8,0,0,0,0,0,0
+onboarding_setup,0,1,0,0,0,0,0,0,0,0,0,0,4,0,0,0,0,0
+troubleshooting,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0
+account_help,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0
+billing_help,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0
+follow_up,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,10,0
+emotional_reflection,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,5

artifacts/evaluation/latest/intent_subtype_val_report.json

@@ -0,0 +1,135 @@
+{
+  "accepted_accuracy": 0.875,
+  "accepted_coverage": 1.0,
+  "accuracy": 0.875,
+  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/intent_subtype_val_confusion_matrix.csv",
+  "count": 80,
+  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/subtype/val.jsonl",
+  "fallback_rate": 0.0,
+  "head": "intent_subtype",
+  "macro_f1": 0.725,
+  "per_class_metrics": {
+    "account_help": {
+      "f1-score": 0.5,
+      "precision": 0.5,
+      "recall": 0.5,
+      "support": 2.0
+    },
+    "accuracy": 0.875,
+    "billing_help": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 1.0
+    },
+    "booking": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 3.0
+    },
+    "comparison": {
+      "f1-score": 0.4,
+      "precision": 1.0,
+      "recall": 0.25,
+      "support": 4.0
+    },
+    "contact_sales": {
+      "f1-score": 0.0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "support": 0.0
+    },
+    "deal_seeking": {
+      "f1-score": 0.5714285714285714,
+      "precision": 0.4,
+      "recall": 1.0,
+      "support": 2.0
+    },
+    "download": {
+      "f1-score": 0.0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "support": 0.0
+    },
+    "education": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 14.0
+    },
+    "emotional_reflection": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 5.0
+    },
+    "evaluation": {
+      "f1-score": 0.6666666666666666,
+      "precision": 0.5,
+      "recall": 1.0,
+      "support": 2.0
+    },
+    "follow_up": {
+      "f1-score": 0.9523809523809523,
+      "precision": 1.0,
+      "recall": 0.9090909090909091,
+      "support": 11.0
+    },
+    "macro avg": {
+      "f1-score": 0.6444203944203944,
+      "precision": 0.6542087542087542,
+      "recall": 0.687121212121212,
+      "support": 80.0
+    },
+    "onboarding_setup": {
+      "f1-score": 0.8,
+      "precision": 0.8,
+      "recall": 0.8,
+      "support": 5.0
+    },
+    "product_discovery": {
+      "f1-score": 0.9090909090909091,
+      "precision": 0.9090909090909091,
+      "recall": 0.9090909090909091,
+      "support": 11.0
+    },
+    "provider_selection": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 7.0
+    },
+    "purchase": {
+      "f1-score": 0.0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "support": 2.0
+    },
+    "signup": {
+      "f1-score": 0.8,
+      "precision": 0.6666666666666666,
+      "recall": 1.0,
+      "support": 2.0
+    },
+    "task_execution": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 8.0
+    },
+    "troubleshooting": {
+      "f1-score": 0.0,
+      "precision": 0.0,
+      "recall": 0.0,
+      "support": 1.0
+    },
+    "weighted avg": {
+      "f1-score": 0.8644047619047619,
+      "precision": 0.8891666666666665,
+      "recall": 0.875,
+      "support": 80.0
+    }
+  },
+  "suite": "val"
+}

artifacts/evaluation/latest/intent_type_difficulty_benchmark_confusion_matrix.csv

@@ -0,0 +1,11 @@
+,informational,exploratory,commercial,transactional,support,personal_reflection,creative_generation,chit_chat,ambiguous,prohibited
+informational,15,0,0,0,0,0,0,0,0,0
+exploratory,0,15,0,0,0,0,0,0,0,0
+commercial,0,0,15,0,0,0,0,0,0,0
+transactional,0,0,0,15,0,0,0,0,0,0
+support,0,0,0,0,15,0,0,0,0,0
+personal_reflection,0,0,0,0,0,15,0,0,0,0
+creative_generation,0,0,0,0,0,0,15,0,0,0
+chit_chat,0,0,0,0,0,1,0,14,0,0
+ambiguous,0,0,0,0,0,0,0,0,15,0
+prohibited,0,0,0,0,1,0,0,0,0,14

artifacts/evaluation/latest/intent_type_difficulty_benchmark_report.json

@@ -0,0 +1,113 @@
+{
+  "accepted_accuracy": 0.9867,
+  "accepted_coverage": 1.0,
+  "accuracy": 0.9867,
+  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/intent_type_difficulty_benchmark_confusion_matrix.csv",
+  "count": 150,
+  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/intent_type_benchmark.jsonl",
+  "difficulty_breakdown": {
+    "easy": {
+      "accepted_accuracy": 1.0,
+      "accepted_coverage": 1.0,
+      "accuracy": 1.0,
+      "count": 50,
+      "fallback_rate": 0.0,
+      "macro_f1": 1.0
+    },
+    "hard": {
+      "accepted_accuracy": 0.96,
+      "accepted_coverage": 1.0,
+      "accuracy": 0.96,
+      "count": 50,
+      "fallback_rate": 0.0,
+      "macro_f1": 0.9596
+    },
+    "medium": {
+      "accepted_accuracy": 1.0,
+      "accepted_coverage": 1.0,
+      "accuracy": 1.0,
+      "count": 50,
+      "fallback_rate": 0.0,
+      "macro_f1": 1.0
+    }
+  },
+  "fallback_rate": 0.0,
+  "head": "intent_type",
+  "macro_f1": 0.9867,
+  "per_class_metrics": {
+    "accuracy": 0.9866666666666667,
+    "ambiguous": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 15.0
+    },
+    "chit_chat": {
+      "f1-score": 0.9655172413793104,
+      "precision": 1.0,
+      "recall": 0.9333333333333333,
+      "support": 15.0
+    },
+    "commercial": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 15.0
+    },
+    "creative_generation": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 15.0
+    },
+    "exploratory": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 15.0
+    },
+    "informational": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 15.0
+    },
+    "macro avg": {
+      "f1-score": 0.9866518353726363,
+      "precision": 0.9875,
+      "recall": 0.9866666666666667,
+      "support": 150.0
+    },
+    "personal_reflection": {
+      "f1-score": 0.967741935483871,
+      "precision": 0.9375,
+      "recall": 1.0,
+      "support": 15.0
+    },
+    "prohibited": {
+      "f1-score": 0.9655172413793104,
+      "precision": 1.0,
+      "recall": 0.9333333333333333,
+      "support": 15.0
+    },
+    "support": {
+      "f1-score": 0.967741935483871,
+      "precision": 0.9375,
+      "recall": 1.0,
+      "support": 15.0
+    },
+    "transactional": {
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
+      "support": 15.0
+    },
+    "weighted avg": {
+      "f1-score": 0.9866518353726362,
+      "precision": 0.9875,
+      "recall": 0.9866666666666667,
+      "support": 150.0
+    }
+  },
+  "suite": "difficulty_benchmark"
+}

artifacts/evaluation/latest/intent_type_hard_cases_confusion_matrix.csv

@@ -0,0 +1,11 @@
+,informational,exploratory,commercial,transactional,support,personal_reflection,creative_generation,chit_chat,ambiguous,prohibited
+informational,20,0,0,0,0,0,0,0,0,0
+exploratory,0,1,0,0,0,0,0,0,0,0
+commercial,0,0,13,0,0,0,0,0,0,0
+transactional,0,0,0,10,0,0,0,0,0,0
+support,0,0,0,0,4,0,0,0,0,0
+personal_reflection,0,0,0,0,0,0,0,0,0,0
+creative_generation,0,0,0,0,0,0,1,0,0,0
+chit_chat,0,0,0,0,0,0,0,1,0,0
+ambiguous,0,0,0,0,0,0,0,0,10,0
+prohibited,0,0,0,0,0,0,0,0,0,1