Upload folder using huggingface_hub

.gitignore

@@ -7,3 +7,4 @@ subtype_model_output/
 iab_model_output/
 */model.safetensors
 iab_hierarchy_model_output/
+model.safetensors

README.md

@@ -1,208 +1,492 @@
----
-language:
-- en
-library_name: transformers
-pipeline_tag: text-classification
-base_model: distilbert-base-uncased
-metrics:
-- accuracy
-- f1
-tags:
-- intent-classification
-- multitask
-- iab
-- conversational-ai
-- adtech
-- calibrated-confidence
-license: apache-2.0
----
-
-# admesh/agentic-intent-classifier
-
-Production-ready intent + IAB classifier bundle for conversational traffic.
-
-Combines multitask intent modeling, supervised IAB content classification, and per-head confidence calibration to support safe monetization decisions in real time.
-
-## Links
+# Agentic Intent Classifier
+
+`agentic-intent-classifier` is a multi-head query classification stack for conversational traffic.
+
+It currently produces:
+
+- `intent.type`
+- `intent.subtype`
+- `intent.decision_phase`
+- `iab_content`
+- calibrated confidence per head
+- combined fallback / policy / opportunity decisions
+
+The repo is beyond the original v0.1 baseline. It now includes:
+
+- shared config and label ownership
+- reusable model runtime
+- calibrated confidence and threshold gating
+- combined inference with fallback/policy logic
+- request/response validation in the demo API
+- repeatable evaluation and regression suites
+- full-TSV IAB taxonomy retrieval support through tier4
+- a local embedding index for taxonomy-node retrieval over IAB content paths
+- a separate synthetic full-intent-taxonomy augmentation dataset for non-IAB heads
+- a dedicated intent-type difficulty dataset and held-out benchmark with `easy`, `medium`, and `hard` cases
+- a dedicated decision-phase difficulty dataset and held-out benchmark with `easy`, `medium`, and `hard` cases
+
+Generated model weights are intentionally not committed.
+
+## Current Taxonomy
+
+### `intent.type`
+
+- `informational`
+- `exploratory`
+- `commercial`
+- `transactional`
+- `support`
+- `personal_reflection`
+- `creative_generation`
+- `chit_chat`
+- `ambiguous`
+- `prohibited`
+
+### `intent.decision_phase`
+
+- `awareness`
+- `research`
+- `consideration`
+- `decision`
+- `action`
+- `post_purchase`
+- `support`
+
+### `intent.subtype`
+
+- `education`
+- `product_discovery`
+- `comparison`
+- `evaluation`
+- `deal_seeking`
+- `provider_selection`
+- `signup`
+- `purchase`
+- `booking`
+- `download`
+- `contact_sales`
+- `task_execution`
+- `onboarding_setup`
+- `troubleshooting`
+- `account_help`
+- `billing_help`
+- `follow_up`
+- `emotional_reflection`
+
+### `iab_content`
+
+- candidates are derived from every row in [data/iab-content/Content Taxonomy 3.0.tsv](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/iab-content/Content%20Taxonomy%203.0.tsv)
+- retrieval output supports `tier1`, `tier2`, `tier3`, and optional `tier4`
+
+## What The System Does
+
+- runs three classifier heads:
+  - `intent_type`
+  - `intent_subtype`
+  - `decision_phase`
+- resolves `iab_content` through a local embedding index over taxonomy nodes plus generic label/path reranking
+- applies calibration artifacts when present
+- computes `commercial_score`
+- applies fallback when confidence is too weak or policy-safe blocking is required
+- emits a schema-validated combined envelope
+
+## What The System Does Not Do
+
+- it is not a multi-turn memory system
+- it is not a production-optimized low-latency serving path
+- it is not yet trained on large real-traffic human-labeled intent data
+- combined decision logic is still heuristic, even though it is materially stronger than the original baseline
+
+## Project Layout
+
+- [config.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/config.py): labels, thresholds, artifact paths, model paths
+- [model_runtime.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/model_runtime.py): shared calibrated inference runtime
+- [combined_inference.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/combined_inference.py): composed system response
+- [inference_intent_type.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/inference_intent_type.py): direct `intent_type` inference entrypoint
+- [inference_iab_classifier.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/inference_iab_classifier.py): direct supervised `iab_content` inference entrypoint
+- [schemas.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/schemas.py): request/response validation
+- [demo_api.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/demo_api.py): local validated API
+- [iab_taxonomy.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/iab_taxonomy.py): full taxonomy parser/index
+- [iab_classifier.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/iab_classifier.py): supervised IAB runtime with taxonomy-aware parent fallback
+- [iab_retrieval.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/iab_retrieval.py): optional shadow retrieval baseline
+- [training/build_full_intent_taxonomy_dataset.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/build_full_intent_taxonomy_dataset.py): separate synthetic intent augmentation dataset
+- [training/build_intent_type_difficulty_dataset.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/build_intent_type_difficulty_dataset.py): extra `intent_type` augmentation plus held-out difficulty benchmark
+- [training/build_decision_phase_difficulty_dataset.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/build_decision_phase_difficulty_dataset.py): extra `decision_phase` augmentation plus held-out difficulty benchmark
+- [training/build_subtype_difficulty_dataset.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/build_subtype_difficulty_dataset.py): extra `intent_subtype` augmentation plus held-out difficulty benchmark
+- [training/build_subtype_dataset.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/build_subtype_dataset.py): subtype dataset generation from existing corpora
+- [training/train_iab.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/train_iab.py): train the supervised IAB classifier head
+- [training/build_iab_taxonomy_embeddings.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/build_iab_taxonomy_embeddings.py): build local IAB node embedding artifacts
+- [training/run_full_training_pipeline.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/run_full_training_pipeline.py): full multi-head training/calibration/eval pipeline
+- [evaluation/run_evaluation.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/evaluation/run_evaluation.py): repeatable benchmark runner
+- [evaluation/run_regression_suite.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/evaluation/run_regression_suite.py): known-failure regression runner
+- [evaluation/run_iab_mapping_suite.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/evaluation/run_iab_mapping_suite.py): IAB behavior-lock regression runner
+- [evaluation/run_iab_quality_suite.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/evaluation/run_iab_quality_suite.py): curated IAB quality-target runner
+- [known_limitations.md](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/known_limitations.md): current gaps and caveats
+
+## Quickstart: Run From Hugging Face
+
+Download the trained bundle and run inference in three lines — no local training required.
 
-- Hugging Face: https://huggingface.co/admesh/agentic-intent-classifier
-- GitHub: https://github.com/GouniManikumar12/agentic-intent-classifier
-
-## What It Predicts
+```python
+import sys
+from huggingface_hub import snapshot_download
 
-| Field | Description |
-|---|---|
-| `intent.type` | `commercial`, `informational`, `navigational`, `transactional`, … |
-| `intent.subtype` | `product_discovery`, `comparison`, `how_to`, … |
-| `intent.decision_phase` | `awareness`, `consideration`, `decision`, … |
-| `iab_content` | IAB Content Taxonomy 3.0 tier1 / tier2 / tier3 labels |
-| `component_confidence` | Per-head calibrated confidence with threshold flags |
-| `system_decision` | Monetization eligibility, opportunity type, policy |
+# Download the full bundle (models + calibration + code)
+local_dir = snapshot_download(
+    repo_id="admesh/agentic-intent-classifier",
+    repo_type="model",
+)
+sys.path.insert(0, local_dir)
 
----
+# Import and instantiate
+from pipeline import AdmeshIntentPipeline
+clf = AdmeshIntentPipeline()
 
-## Deployment Options
+# Classify
+import json
+result = clf("Which laptop should I buy for college?")
+print(json.dumps(result, indent=2))
+```
 
-### 1. `transformers.pipeline()` — one line anywhere
+Or use the one-liner factory method:
 
 ```python
-from transformers import pipeline
+from pipeline import AdmeshIntentPipeline   # after sys.path.insert above
 
-clf = pipeline(
-    "admesh-intent",
-    model="admesh/agentic-intent-classifier",
-    trust_remote_code=True,
-)
-
-result = clf("Which laptop should I buy for college?")
+clf = AdmeshIntentPipeline.from_pretrained("admesh/agentic-intent-classifier")
+result = clf("I need a CRM for a 5-person startup")
 ```
 
-Batch and custom thresholds:
+Batch mode and custom thresholds are also supported:
 
 ```python
-# batch
+# Batch
 results = clf([
     "Best running shoes under $100",
-    "How does TCP work?",
+    "How does gradient descent work?",
     "Buy noise-cancelling headphones",
 ])
 
-# custom confidence thresholds
+# Custom confidence thresholds
 result = clf(
-    "Buy headphones",
+    "Buy noise-cancelling headphones",
     threshold_overrides={"intent_type": 0.6, "intent_subtype": 0.35},
 )
 ```
 
+Verify artifacts and run a smoke test from the CLI:
+
+```bash
+cd "<local_dir>"
+python3 training/pipeline_verify.py
+python3 combined_inference.py "Which CRM should I buy for a 3-person startup?"
+```
+
+Pin a specific revision for reproducibility:
+
+```python
+local_dir = snapshot_download(
+    repo_id="admesh/agentic-intent-classifier",
+    repo_type="model",
+    revision="0584798f8efee6beccd778b0afa06782ab5add60",
+)
+```
+
 ---
 
-### 2. HF Inference Endpoints (managed, deploy to AWS / Azure / GCP)
+## Setup (for local training)
 
-1. Go to https://ui.endpoints.huggingface.co
-2. **New Endpoint** → select `admesh/agentic-intent-classifier`
-3. Framework: **PyTorch** — Task: **Text Classification**
-4. Enable **"Load with trust_remote_code"**
-5. Deploy
+```bash
+python3 -m venv .venv
+source .venv/bin/activate
+pip install -r agentic-intent-classifier/requirements.txt
+```
+
+## Inference (local training path)
 
-The endpoint serves the same `pipeline()` interface above via REST:
+Run one query locally:
 
 ```bash
-curl https://<your-endpoint>.endpoints.huggingface.cloud \
-  -H "Authorization: Bearer $HF_TOKEN" \
-  -H "Content-Type: application/json" \
-  -d '{"inputs": "Which laptop should I buy for college?"}'
+cd agentic-intent-classifier
+python3 training/train_iab.py
+python3 training/calibrate_confidence.py --head iab_content
+python3 combined_inference.py "Which CRM should I buy for a 3-person startup?"
 ```
 
----
+Run only the `intent_type` head:
+
+```bash
+cd agentic-intent-classifier
+python3 inference_intent_type.py "best shoes under 100"
+```
 
-### 3. HF Spaces (Gradio / Streamlit demo)
+Run the demo API:
 
-```python
-# app.py for a Gradio Space
-import gradio as gr
-from transformers import pipeline
-
-clf = pipeline(
-    "admesh-intent",
-    model="admesh/agentic-intent-classifier",
-    trust_remote_code=True,
-)
+```bash
+cd agentic-intent-classifier
+python3 demo_api.py
+```
 
-def classify(text):
-    return clf(text)
+Example request:
 
-gr.Interface(fn=classify, inputs="text", outputs="json").launch()
+```bash
+curl -sS -X POST http://127.0.0.1:8008/classify \
+  -H 'Content-Type: application/json' \
+  -d '{"text":"I cannot log into my account"}'
 ```
 
----
+Infra endpoints:
 
-### 4. Local / notebook via `snapshot_download`
+```bash
+curl -sS http://127.0.0.1:8008/health
+curl -sS http://127.0.0.1:8008/version
+```
 
-```python
-import sys
-from huggingface_hub import snapshot_download
+Train only the IAB classifier head:
 
-local_dir = snapshot_download(
-    repo_id="admesh/agentic-intent-classifier",
-    repo_type="model",
-)
-sys.path.insert(0, local_dir)
+```bash
+cd agentic-intent-classifier
+python3 training/train_iab.py
+python3 training/calibrate_confidence.py --head iab_content
+```
 
-from pipeline import AdmeshIntentPipeline
-clf = AdmeshIntentPipeline()
-result = clf("I need a CRM for a 5-person startup")
+The online `iab_content` path now uses the compact supervised classifier. Retrieval is still available as an optional shadow baseline.
+
+Build the optional retrieval shadow index:
+
+```bash
+cd agentic-intent-classifier
+python3 training/build_iab_taxonomy_embeddings.py
 ```
 
-Or the one-liner factory:
+By default the shadow retrieval path uses `Alibaba-NLP/gte-Qwen2-1.5B-instruct`. The retrieval runtime applies the model's query-side instruction format and last-token pooling, matching the Hugging Face usage guidance. If you want to point retrieval at a different embedding model, set `IAB_RETRIEVAL_MODEL_NAME_OVERRIDE` before building the index.
 
-```python
-from pipeline import AdmeshIntentPipeline
-clf = AdmeshIntentPipeline.from_pretrained("admesh/agentic-intent-classifier")
+Open-source users can swap in their own embedding model, but the contract is:
+
+- query embeddings and taxonomy-node embeddings must be produced by the same model and model revision
+- after changing models, you must rebuild `artifacts/iab/taxonomy_embeddings.pt`
+- the repository only tests and supports the default model path out of the box
+- not every Hugging Face embedding model is drop-in compatible with this runtime; some require custom pooling, query instructions, or `trust_remote_code`
+
+Example override:
+
+```bash
+cd agentic-intent-classifier
+export IAB_RETRIEVAL_MODEL_NAME_OVERRIDE=mixedbread-ai/mxbai-embed-large-v1
+python3 training/build_iab_taxonomy_embeddings.py
 ```
 
----
+This writes:
 
-## Example Output
-
-```json
-{
-  "model_output": {
-    "classification": {
-      "iab_content": {
-        "taxonomy": "IAB Content Taxonomy",
-        "taxonomy_version": "3.0",
-        "tier1": {"id": "552", "label": "Style & Fashion"},
-        "tier2": {"id": "579", "label": "Men's Fashion"},
-        "mapping_mode": "exact",
-        "mapping_confidence": 0.73
-      },
-      "intent": {
-        "type": "commercial",
-        "subtype": "product_discovery",
-        "decision_phase": "consideration",
-        "confidence": 0.9549,
-        "commercial_score": 0.656
-      }
-    }
-  },
-  "system_decision": {
-    "policy": {
-      "monetization_eligibility": "allowed_with_caution",
-      "eligibility_reason": "commercial_discovery_signal_present"
-    },
-    "opportunity": {"type": "soft_recommendation", "strength": "medium"}
-  },
-  "meta": {
-    "system_version": "0.6.0-phase4",
-    "calibration_enabled": true,
-    "iab_mapping_is_placeholder": false
-  }
-}
-```
-
-## Reproducible Revision
+- `artifacts/iab/taxonomy_nodes.json`
+- `artifacts/iab/taxonomy_embeddings.pt`
 
-```python
-from huggingface_hub import snapshot_download
-local_dir = snapshot_download(
-    repo_id="admesh/agentic-intent-classifier",
-    repo_type="model",
-    revision="0584798f8efee6beccd778b0afa06782ab5add60",
-)
+## Training
+
+### Full local pipeline
+
+```bash
+cd agentic-intent-classifier
+python3 training/run_full_training_pipeline.py
 ```
 
-## Included Artifacts
+This pipeline now does:
+
+1. build separate full-intent-taxonomy augmentation data
+2. build separate `intent_type` difficulty augmentation + benchmark
+3. train `intent_type`
+4. build subtype corpus
+5. build separate `intent_subtype` difficulty augmentation + benchmark
+6. train `intent_subtype`
+7. build separate `decision_phase` difficulty augmentation + benchmark
+8. train `decision_phase`
+9. train `iab_content`
+10. calibrate all classifier heads, including `iab_content`
+11. run regression/evaluation unless `--skip-full-eval` is used
+
+### Build datasets individually
+
+Separate full-intent augmentation:
+
+```bash
+cd agentic-intent-classifier
+python3 training/build_full_intent_taxonomy_dataset.py
+```
+
+Intent-type difficulty augmentation and benchmark:
+
+```bash
+cd agentic-intent-classifier
+python3 training/build_intent_type_difficulty_dataset.py
+```
+
+Decision-phase difficulty augmentation and benchmark:
+
+```bash
+cd agentic-intent-classifier
+python3 training/build_decision_phase_difficulty_dataset.py
+```
+
+Subtype difficulty augmentation and benchmark:
+
+```bash
+cd agentic-intent-classifier
+python3 training/build_subtype_difficulty_dataset.py
+```
+
+Subtype dataset:
+
+```bash
+cd agentic-intent-classifier
+python3 training/build_subtype_dataset.py
+```
+
+IAB embedding index:
+
+```bash
+cd agentic-intent-classifier
+python3 training/build_iab_taxonomy_embeddings.py
+```
+
+### Train heads individually
+
+```bash
+cd agentic-intent-classifier
+python3 training/train.py
+python3 training/train_subtype.py
+python3 training/train_decision_phase.py
+```
+
+### Calibration
+
+```bash
+cd agentic-intent-classifier
+python3 training/calibrate_confidence.py --head intent_type
+python3 training/calibrate_confidence.py --head intent_subtype
+python3 training/calibrate_confidence.py --head decision_phase
+```
+
+## Evaluation
+
+Full evaluation:
+
+```bash
+cd agentic-intent-classifier
+python3 evaluation/run_evaluation.py
+```
+
+Known-failure regression:
+
+```bash
+cd agentic-intent-classifier
+python3 evaluation/run_regression_suite.py
+```
+
+IAB behavior-lock regression:
+
+```bash
+cd agentic-intent-classifier
+python3 evaluation/run_iab_mapping_suite.py
+```
+
+IAB quality-target evaluation:
+
+```bash
+cd agentic-intent-classifier
+python3 evaluation/run_iab_quality_suite.py
+```
+
+Threshold sweeps:
+
+```bash
+cd agentic-intent-classifier
+python3 evaluation/sweep_intent_threshold.py
+```
+
+Artifacts are written to:
+
+- `artifacts/calibration/`
+- `artifacts/evaluation/latest/`
+
+## Google Colab
+
+Use Colab for the full retraining pass if local memory is limited.
+
+Clone once:
+
+```bash
+%cd /content
+!git clone https://github.com/GouniManikumar12/agentic-intent-classifier.git
+%cd /content/agentic-intent-classifier
+```
+
+If the repo is already cloned and you want the latest code, pull manually:
+
+```bash
+!git pull origin main
+```
+
+Full pipeline:
+
+```bash
+!python training/run_full_training_pipeline.py
+```
+
+If full evaluation is too heavy for the current Colab runtime:
+
+```bash
+!python training/run_full_training_pipeline.py \
+  --iab-embedding-batch-size 32 \
+  --skip-full-eval
+```
+
+Then run eval separately after training:
+
+```bash
+!python evaluation/run_regression_suite.py
+!python evaluation/run_iab_mapping_suite.py
+!python evaluation/run_iab_quality_suite.py
+!python evaluation/run_evaluation.py
+```
+
+## Current Saved Metrics
+
+Generate fresh metrics with:
+
+```bash
+cd agentic-intent-classifier
+python3 evaluation/run_evaluation.py
+```
+
+Do not treat any checked-in summary as canonical unless it was regenerated after the current code and artifacts were built. The IAB path is now retrieval-based, so older saved reports from the deleted hierarchy stack are not meaningful.
+
+## Latency Note
+
+`combined_inference.py` is a debugging/offline path, not a production latency path.
+
+Current production truth:
+
+- per-request CLI execution is not a sub-50ms architecture
+- production serving should use a long-lived API process with preloaded models
+- if sub-50ms becomes a hard requirement, the serving path will need:
+  - persistent loaded models
+  - runtime optimization
+  - likely fewer model passes or a shared multi-head model
+
+## Current Status
+
+Current repo status:
 
-| Path | Contents |
-|---|---|
-| `multitask_intent_model_output/` | DistilBERT multitask weights + tokenizer |
-| `iab_classifier_model_output/` | IAB content classifier weights + tokenizer |
-| `artifacts/calibration/` | Per-head temperature + threshold JSONs |
-| `pipeline.py` | `AdmeshIntentPipeline` (transformers.Pipeline subclass) |
-| `combined_inference.py` | Core inference logic |
+- full 10-class `intent.type` taxonomy is wired
+- subtype and phase heads are present
+- difficulty benchmarks are wired for `intent_type`, `intent_subtype`, and `decision_phase`
+- full-TSV IAB taxonomy retrieval is wired through tier4
+- separate full-intent augmentation dataset is in place
+- evaluation/runtime memory handling is improved for large IAB splits
 
-## Notes
+The main remaining gap is not basic infrastructure anymore. It is improving real-world robustness, especially for:
 
-- `trust_remote_code=True` is required because this model uses a custom multi-head architecture that does not map to a single standard `AutoModel` checkpoint.
-- `meta.iab_mapping_is_placeholder: true` means IAB artifacts were missing or skipped; train and calibrate IAB for full production accuracy.
-- For long-running servers, instantiate once and reuse — models are cached in memory after the first call.
+- `decision_phase`
+- `intent_subtype`
+- confidence quality on borderline commercial queries
+- real-traffic supervision beyond synthetic data

artifacts/calibration/decision_phase.json

@@ -1,32 +1,32 @@
 {
   "calibrated": true,
   "confidence_threshold": 0.22,
-  "generated_at": "2026-03-22T17:27:04.263877+00:00",
+  "generated_at": "2026-03-25T16:15:10.949430+00:00",
   "head": "decision_phase",
   "metrics": {
-    "calibrated_accuracy": 0.6552,
-    "calibrated_expected_calibration_error": 0.3883,
-    "calibrated_negative_log_likelihood": 1.4596,
-    "mean_calibrated_confidence": 0.2668,
-    "mean_raw_confidence": 0.2383,
-    "raw_accuracy": 0.6552,
-    "raw_expected_calibration_error": 0.4169,
-    "raw_negative_log_likelihood": 1.5421
+    "calibrated_accuracy": 0.8621,
+    "calibrated_expected_calibration_error": 0.0672,
+    "calibrated_negative_log_likelihood": 0.4798,
+    "mean_calibrated_confidence": 0.868,
+    "mean_raw_confidence": 0.868,
+    "raw_accuracy": 0.8621,
+    "raw_expected_calibration_error": 0.0672,
+    "raw_negative_log_likelihood": 0.4798
   },
   "minimum_threshold_floor": 0.22,
-  "optimized_temperature_candidate": 0.798354,
+  "optimized_temperature_candidate": 1.000144,
   "selected_threshold_before_floor": {
-    "accepted_accuracy": 0.75,
-    "coverage": 0.6897,
-    "threshold": 0.22
+    "accepted_accuracy": 0.8621,
+    "coverage": 1.0,
+    "threshold": 0.0
   },
   "selection_split": "val",
   "selection_target_precision": 0.75,
-  "temperature": 0.798354,
+  "temperature": 1.000144,
   "temperature_scaling_applied": true,
   "threshold_summary": {
-    "accepted_accuracy": 0.75,
-    "coverage": 0.6897,
+    "accepted_accuracy": 0.8621,
+    "coverage": 1.0,
     "threshold": 0.22
   }
 }

artifacts/calibration/iab_content.json

@@ -1,32 +1,32 @@
 {
   "calibrated": true,
   "confidence_threshold": 0.12,
-  "generated_at": "2026-03-25T07:04:35.676097+00:00",
+  "generated_at": "2026-03-25T16:17:01.813766+00:00",
   "head": "iab_content",
   "metrics": {
-    "calibrated_accuracy": 0.9159,
-    "calibrated_expected_calibration_error": 0.2475,
-    "calibrated_negative_log_likelihood": 0.5736,
-    "mean_calibrated_confidence": 0.6684,
-    "mean_raw_confidence": 0.1932,
-    "raw_accuracy": 0.9159,
-    "raw_expected_calibration_error": 0.7227,
-    "raw_negative_log_likelihood": 1.8448
+    "calibrated_accuracy": 0.9308,
+    "calibrated_expected_calibration_error": 0.2692,
+    "calibrated_negative_log_likelihood": 0.5893,
+    "mean_calibrated_confidence": 0.6617,
+    "mean_raw_confidence": 0.1809,
+    "raw_accuracy": 0.9308,
+    "raw_expected_calibration_error": 0.75,
+    "raw_negative_log_likelihood": 1.9134
   },
   "minimum_threshold_floor": 0.12,
-  "optimized_temperature_candidate": 0.562804,
+  "optimized_temperature_candidate": 0.55869,
   "selected_threshold_before_floor": {
-    "accepted_accuracy": 0.9159,
+    "accepted_accuracy": 0.9308,
     "coverage": 1.0,
     "threshold": 0.0
   },
   "selection_split": "val",
   "selection_target_precision": 0.7,
-  "temperature": 0.562804,
+  "temperature": 0.55869,
   "temperature_scaling_applied": true,
   "threshold_summary": {
-    "accepted_accuracy": 0.921,
-    "coverage": 0.9878,
+    "accepted_accuracy": 0.9436,
+    "coverage": 0.9775,
     "threshold": 0.12
   }
 }

artifacts/calibration/intent_subtype.json

@@ -1,32 +1,32 @@
 {
   "calibrated": true,
   "confidence_threshold": 0.25,
-  "generated_at": "2026-03-22T17:27:03.399767+00:00",
+  "generated_at": "2026-03-25T16:15:00.986765+00:00",
   "head": "intent_subtype",
   "metrics": {
-    "calibrated_accuracy": 0.6667,
-    "calibrated_expected_calibration_error": 0.2996,
-    "calibrated_negative_log_likelihood": 1.5163,
-    "mean_calibrated_confidence": 0.3894,
-    "mean_raw_confidence": 0.3894,
-    "raw_accuracy": 0.6667,
-    "raw_expected_calibration_error": 0.2996,
-    "raw_negative_log_likelihood": 1.5163
+    "calibrated_accuracy": 0.875,
+    "calibrated_expected_calibration_error": 0.0667,
+    "calibrated_negative_log_likelihood": 0.3811,
+    "mean_calibrated_confidence": 0.8307,
+    "mean_raw_confidence": 0.7584,
+    "raw_accuracy": 0.875,
+    "raw_expected_calibration_error": 0.1314,
+    "raw_negative_log_likelihood": 0.4541
   },
   "minimum_threshold_floor": 0.25,
-  "optimized_temperature_candidate": 0.001,
+  "optimized_temperature_candidate": 0.824082,
   "selected_threshold_before_floor": {
-    "accepted_accuracy": 0.75,
-    "coverage": 0.8485,
-    "threshold": 0.13
+    "accepted_accuracy": 0.875,
+    "coverage": 1.0,
+    "threshold": 0.0
   },
   "selection_split": "val",
   "selection_target_precision": 0.75,
-  "temperature": 1.0,
-  "temperature_scaling_applied": false,
+  "temperature": 0.824082,
+  "temperature_scaling_applied": true,
   "threshold_summary": {
-    "accepted_accuracy": 0.878,
-    "coverage": 0.6212,
+    "accepted_accuracy": 0.8734,
+    "coverage": 0.9875,
     "threshold": 0.25
   }
 }

artifacts/calibration/intent_type.json

@@ -1,32 +1,32 @@
 {
   "calibrated": true,
   "confidence_threshold": 0.4,
-  "generated_at": "2026-03-22T17:27:02.517983+00:00",
+  "generated_at": "2026-03-25T16:14:49.053223+00:00",
   "head": "intent_type",
   "metrics": {
-    "calibrated_accuracy": 0.975,
-    "calibrated_expected_calibration_error": 0.3585,
-    "calibrated_negative_log_likelihood": 0.5728,
-    "mean_calibrated_confidence": 0.6165,
-    "mean_raw_confidence": 0.4936,
-    "raw_accuracy": 0.975,
-    "raw_expected_calibration_error": 0.4814,
-    "raw_negative_log_likelihood": 0.7776
+    "calibrated_accuracy": 0.9362,
+    "calibrated_expected_calibration_error": 0.0715,
+    "calibrated_negative_log_likelihood": 0.2384,
+    "mean_calibrated_confidence": 0.917,
+    "mean_raw_confidence": 0.8891,
+    "raw_accuracy": 0.9362,
+    "raw_expected_calibration_error": 0.0807,
+    "raw_negative_log_likelihood": 0.257
   },
   "minimum_threshold_floor": 0.4,
-  "optimized_temperature_candidate": 0.68171,
+  "optimized_temperature_candidate": 0.901567,
   "selected_threshold_before_floor": {
-    "accepted_accuracy": 0.975,
+    "accepted_accuracy": 0.9362,
     "coverage": 1.0,
     "threshold": 0.0
   },
   "selection_split": "val",
   "selection_target_precision": 0.8,
-  "temperature": 0.68171,
+  "temperature": 0.901567,
   "temperature_scaling_applied": true,
   "threshold_summary": {
-    "accepted_accuracy": 1.0,
-    "coverage": 0.775,
+    "accepted_accuracy": 0.9362,
+    "coverage": 1.0,
     "threshold": 0.4
   }
 }

artifacts/evaluation/latest/combined_demo_benchmark.json

@@ -5,12 +5,13 @@
     "response": {
       "meta": {
         "calibration_enabled": true,
+        "iab_mapping_is_placeholder": false,
         "system_version": "0.6.0-phase4"
       },
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.9035,
+            "mapping_confidence": 0.676,
             "mapping_mode": "nearest_equivalent",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
@@ -21,10 +22,6 @@
             "tier2": {
               "id": "599",
               "label": "Computing"
-            },
-            "tier3": {
-              "id": "602",
-              "label": "Software and Applications"
             }
           },
           "intent": {
@@ -32,31 +29,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.9947,
+                "confidence": 0.961,
                 "confidence_threshold": 0.22,
                 "label": "awareness",
                 "meets_threshold": true,
-                "raw_confidence": 0.9788
+                "raw_confidence": 0.9611
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.9547,
+                "confidence": 0.9853,
                 "confidence_threshold": 0.25,
                 "label": "education",
                 "meets_threshold": true,
-                "raw_confidence": 0.9547
+                "raw_confidence": 0.9516
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.9972,
+                "confidence": 0.9807,
                 "confidence_threshold": 0.4,
                 "label": "informational",
                 "meets_threshold": true,
-                "raw_confidence": 0.9662
+                "raw_confidence": 0.9655
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.9947,
+            "confidence": 0.961,
             "decision_phase": "awareness",
             "subtype": "education",
             "summary": "Classified as informational intent with subtype education in the awareness phase.",
@@ -95,18 +92,19 @@
     "response": {
       "meta": {
         "calibration_enabled": true,
+        "iab_mapping_is_placeholder": false,
         "system_version": "0.6.0-phase4"
       },
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.8427,
+            "mapping_confidence": 0.4448,
             "mapping_mode": "nearest_equivalent",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
             "tier1": {
-              "id": "1",
-              "label": "Automotive"
+              "id": "596",
+              "label": "Technology & Computing"
             }
           },
           "intent": {
@@ -114,31 +112,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.9944,
+                "confidence": 0.9381,
                 "confidence_threshold": 0.22,
                 "label": "awareness",
                 "meets_threshold": true,
-                "raw_confidence": 0.9779
+                "raw_confidence": 0.9381
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.955,
+                "confidence": 0.9753,
                 "confidence_threshold": 0.25,
                 "label": "education",
                 "meets_threshold": true,
-                "raw_confidence": 0.955
+                "raw_confidence": 0.9275
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.9969,
+                "confidence": 0.9768,
                 "confidence_threshold": 0.4,
                 "label": "informational",
                 "meets_threshold": true,
-                "raw_confidence": 0.9637
+                "raw_confidence": 0.9597
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.9944,
+            "confidence": 0.9381,
             "decision_phase": "awareness",
             "subtype": "education",
             "summary": "Classified as informational intent with subtype education in the awareness phase.",
@@ -177,12 +175,13 @@
     "response": {
       "meta": {
         "calibration_enabled": true,
+        "iab_mapping_is_placeholder": false,
         "system_version": "0.6.0-phase4"
       },
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.7798,
+            "mapping_confidence": 0.7819,
             "mapping_mode": "nearest_equivalent",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
@@ -192,38 +191,38 @@
             }
           },
           "intent": {
-            "commercial_score": 0.656,
+            "commercial_score": 0.728,
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.9965,
+                "confidence": 0.9691,
                 "confidence_threshold": 0.22,
                 "label": "consideration",
                 "meets_threshold": true,
-                "raw_confidence": 0.9846
+                "raw_confidence": 0.9691
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.4682,
+                "confidence": 0.563,
                 "confidence_threshold": 0.25,
-                "label": "product_discovery",
+                "label": "comparison",
                 "meets_threshold": true,
-                "raw_confidence": 0.4682
+                "raw_confidence": 0.4806
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.9995,
+                "confidence": 0.9869,
                 "confidence_threshold": 0.4,
                 "label": "commercial",
                 "meets_threshold": true,
-                "raw_confidence": 0.9895
+                "raw_confidence": 0.9756
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.4682,
+            "confidence": 0.563,
             "decision_phase": "consideration",
-            "subtype": "product_discovery",
-            "summary": "Classified as commercial intent with subtype product_discovery in the consideration phase.",
+            "subtype": "comparison",
+            "summary": "Classified as commercial intent with subtype comparison in the consideration phase.",
             "type": "commercial"
           }
         },
@@ -234,8 +233,8 @@
           "consideration"
         ],
         "opportunity": {
-          "strength": "medium",
-          "type": "soft_recommendation"
+          "strength": "high",
+          "type": "comparison_slot"
         },
         "policy": {
           "applied_thresholds": {
@@ -245,7 +244,7 @@
             "intent_type_confidence_min": 0.4
           },
           "decision_basis": "score_threshold",
-          "eligibility_reason": "commercial_discovery_signal_present",
+          "eligibility_reason": "commercial_comparison_signal_present",
           "monetization_eligibility": "allowed_with_caution",
           "regulated_vertical": false,
           "sensitivity": "low"
@@ -259,26 +258,19 @@
     "response": {
       "meta": {
         "calibration_enabled": true,
+        "iab_mapping_is_placeholder": false,
         "system_version": "0.6.0-phase4"
       },
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.8606,
+            "mapping_confidence": 0.3576,
             "mapping_mode": "nearest_equivalent",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
             "tier1": {
               "id": "596",
               "label": "Technology & Computing"
-            },
-            "tier2": {
-              "id": "599",
-              "label": "Computing"
-            },
-            "tier3": {
-              "id": "619",
-              "label": "Internet"
             }
           },
           "intent": {
@@ -286,31 +278,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.9964,
+                "confidence": 0.9283,
                 "confidence_threshold": 0.22,
                 "label": "consideration",
                 "meets_threshold": true,
-                "raw_confidence": 0.9842
+                "raw_confidence": 0.9284
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.9449,
+                "confidence": 0.9727,
                 "confidence_threshold": 0.25,
                 "label": "comparison",
                 "meets_threshold": true,
-                "raw_confidence": 0.9449
+                "raw_confidence": 0.9236
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.9995,
+                "confidence": 0.9734,
                 "confidence_threshold": 0.4,
                 "label": "commercial",
                 "meets_threshold": true,
-                "raw_confidence": 0.9892
+                "raw_confidence": 0.954
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.9449,
+            "confidence": 0.9283,
             "decision_phase": "consideration",
             "subtype": "comparison",
             "summary": "Classified as commercial intent with subtype comparison in the consideration phase.",
@@ -349,12 +341,13 @@
     "response": {
       "meta": {
         "calibration_enabled": true,
+        "iab_mapping_is_placeholder": false,
         "system_version": "0.6.0-phase4"
       },
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.8737,
+            "mapping_confidence": 0.3542,
             "mapping_mode": "nearest_equivalent",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
@@ -372,31 +365,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.963,
+                "confidence": 0.8012,
                 "confidence_threshold": 0.22,
                 "label": "decision",
                 "meets_threshold": true,
-                "raw_confidence": 0.9122
+                "raw_confidence": 0.8012
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.9119,
+                "confidence": 0.9028,
                 "confidence_threshold": 0.25,
                 "label": "provider_selection",
                 "meets_threshold": true,
-                "raw_confidence": 0.9119
+                "raw_confidence": 0.8041
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.9994,
+                "confidence": 0.9759,
                 "confidence_threshold": 0.4,
                 "label": "commercial",
                 "meets_threshold": true,
-                "raw_confidence": 0.9874
+                "raw_confidence": 0.9582
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.9119,
+            "confidence": 0.8012,
             "decision_phase": "decision",
             "subtype": "provider_selection",
             "summary": "Classified as commercial intent with subtype provider_selection in the decision phase.",
@@ -435,26 +428,19 @@
     "response": {
       "meta": {
         "calibration_enabled": true,
+        "iab_mapping_is_placeholder": false,
         "system_version": "0.6.0-phase4"
       },
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.7133,
+            "mapping_confidence": 0.2259,
             "mapping_mode": "nearest_equivalent",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
             "tier1": {
-              "id": "239",
-              "label": "Hobbies & Interests"
-            },
-            "tier2": {
-              "id": "264",
-              "label": "Content Production"
-            },
-            "tier3": {
-              "id": "266",
-              "label": "Freelance Writing"
+              "id": "483",
+              "label": "Sports"
             }
           },
           "intent": {
@@ -462,31 +448,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.9991,
+                "confidence": 0.9176,
                 "confidence_threshold": 0.22,
                 "label": "action",
                 "meets_threshold": true,
-                "raw_confidence": 0.9947
+                "raw_confidence": 0.9176
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.9382,
+                "confidence": 0.9675,
                 "confidence_threshold": 0.25,
                 "label": "signup",
                 "meets_threshold": true,
-                "raw_confidence": 0.9382
+                "raw_confidence": 0.9135
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.9996,
+                "confidence": 0.9416,
                 "confidence_threshold": 0.4,
                 "label": "transactional",
                 "meets_threshold": true,
-                "raw_confidence": 0.9902
+                "raw_confidence": 0.909
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.9382,
+            "confidence": 0.9176,
             "decision_phase": "action",
             "subtype": "signup",
             "summary": "Classified as transactional intent with subtype signup in the action phase.",
@@ -525,13 +511,14 @@
     "response": {
       "meta": {
         "calibration_enabled": true,
+        "iab_mapping_is_placeholder": false,
         "system_version": "0.6.0-phase4"
       },
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.7997,
-            "mapping_mode": "nearest_equivalent",
+            "mapping_confidence": 0.5455,
+            "mapping_mode": "exact",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
             "tier1": {
@@ -548,31 +535,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.999,
+                "confidence": 0.9628,
                 "confidence_threshold": 0.22,
                 "label": "action",
                 "meets_threshold": true,
-                "raw_confidence": 0.9945
+                "raw_confidence": 0.9628
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.8724,
+                "confidence": 0.7841,
                 "confidence_threshold": 0.25,
                 "label": "booking",
                 "meets_threshold": true,
-                "raw_confidence": 0.8724
+                "raw_confidence": 0.6676
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.9996,
+                "confidence": 0.9761,
                 "confidence_threshold": 0.4,
                 "label": "transactional",
                 "meets_threshold": true,
-                "raw_confidence": 0.9901
+                "raw_confidence": 0.9583
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.8724,
+            "confidence": 0.7841,
             "decision_phase": "action",
             "subtype": "booking",
             "summary": "Classified as transactional intent with subtype booking in the action phase.",
@@ -611,30 +598,19 @@
     "response": {
       "meta": {
         "calibration_enabled": true,
+        "iab_mapping_is_placeholder": false,
         "system_version": "0.6.0-phase4"
       },
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.8423,
+            "mapping_confidence": 0.3927,
             "mapping_mode": "nearest_equivalent",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
             "tier1": {
               "id": "596",
               "label": "Technology & Computing"
-            },
-            "tier2": {
-              "id": "599",
-              "label": "Computing"
-            },
-            "tier3": {
-              "id": "619",
-              "label": "Internet"
-            },
-            "tier4": {
-              "id": "620",
-              "label": "Cloud Computing"
             }
           },
           "intent": {
@@ -642,31 +618,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.9736,
+                "confidence": 0.9539,
                 "confidence_threshold": 0.22,
                 "label": "post_purchase",
                 "meets_threshold": true,
-                "raw_confidence": 0.9264
+                "raw_confidence": 0.9539
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.921,
+                "confidence": 0.9652,
                 "confidence_threshold": 0.25,
                 "label": "onboarding_setup",
                 "meets_threshold": true,
-                "raw_confidence": 0.921
+                "raw_confidence": 0.9053
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.9935,
+                "confidence": 0.7786,
                 "confidence_threshold": 0.4,
                 "label": "transactional",
                 "meets_threshold": true,
-                "raw_confidence": 0.9448
+                "raw_confidence": 0.7173
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.9736,
+            "confidence": 0.7786,
             "decision_phase": "post_purchase",
             "subtype": "onboarding_setup",
             "summary": "Classified as transactional intent with subtype onboarding_setup in the post_purchase phase.",
@@ -705,26 +681,19 @@
     "response": {
       "meta": {
         "calibration_enabled": true,
+        "iab_mapping_is_placeholder": false,
         "system_version": "0.6.0-phase4"
       },
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.8039,
+            "mapping_confidence": 0.2935,
             "mapping_mode": "nearest_equivalent",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
             "tier1": {
-              "id": "596",
-              "label": "Technology & Computing"
-            },
-            "tier2": {
-              "id": "599",
-              "label": "Computing"
-            },
-            "tier3": {
-              "id": "619",
-              "label": "Internet"
+              "id": "52",
+              "label": "Business and Finance"
             }
           },
           "intent": {
@@ -732,31 +701,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.9969,
+                "confidence": 0.9528,
                 "confidence_threshold": 0.22,
                 "label": "support",
                 "meets_threshold": true,
-                "raw_confidence": 0.9863
+                "raw_confidence": 0.9528
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.923,
+                "confidence": 0.894,
                 "confidence_threshold": 0.25,
                 "label": "account_help",
                 "meets_threshold": true,
-                "raw_confidence": 0.923
+                "raw_confidence": 0.8027
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.9988,
+                "confidence": 0.9636,
                 "confidence_threshold": 0.4,
                 "label": "support",
                 "meets_threshold": true,
-                "raw_confidence": 0.9811
+                "raw_confidence": 0.9403
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.923,
+            "confidence": 0.894,
             "decision_phase": "support",
             "subtype": "account_help",
             "summary": "Classified as support intent with subtype account_help in the support phase.",
@@ -801,22 +770,19 @@
     "response": {
       "meta": {
         "calibration_enabled": true,
+        "iab_mapping_is_placeholder": false,
         "system_version": "0.6.0-phase4"
       },
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.7854,
+            "mapping_confidence": 0.1521,
             "mapping_mode": "nearest_equivalent",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
             "tier1": {
-              "id": "286",
-              "label": "Medical Health"
-            },
-            "tier2": {
-              "id": "287",
-              "label": "Diseases and Conditions"
+              "id": "v9i3On",
+              "label": "Sensitive Topics"
             }
           },
           "intent": {
@@ -824,31 +790,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.9699,
+                "confidence": 0.843,
                 "confidence_threshold": 0.22,
                 "label": "awareness",
                 "meets_threshold": true,
-                "raw_confidence": 0.9258
+                "raw_confidence": 0.843
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.9435,
+                "confidence": 0.9678,
                 "confidence_threshold": 0.25,
                 "label": "emotional_reflection",
                 "meets_threshold": true,
-                "raw_confidence": 0.9435
+                "raw_confidence": 0.9123
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.9916,
+                "confidence": 0.929,
                 "confidence_threshold": 0.4,
                 "label": "personal_reflection",
                 "meets_threshold": true,
-                "raw_confidence": 0.9406
+                "raw_confidence": 0.8937
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.9435,
+            "confidence": 0.843,
             "decision_phase": "awareness",
             "subtype": "emotional_reflection",
             "summary": "Classified as personal_reflection intent with subtype emotional_reflection in the awareness phase.",
@@ -893,18 +859,19 @@
     "response": {
       "meta": {
         "calibration_enabled": true,
+        "iab_mapping_is_placeholder": false,
         "system_version": "0.6.0-phase4"
       },
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.7304,
+            "mapping_confidence": 0.0846,
             "mapping_mode": "nearest_equivalent",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
             "tier1": {
-              "id": "SPSHQ5",
-              "label": "Genres"
+              "id": "42",
+              "label": "Books and Literature"
             }
           },
           "intent": {
@@ -912,31 +879,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.9934,
+                "confidence": 0.8336,
                 "confidence_threshold": 0.22,
                 "label": "research",
                 "meets_threshold": true,
-                "raw_confidence": 0.9746
+                "raw_confidence": 0.8336
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.9631,
+                "confidence": 0.9685,
                 "confidence_threshold": 0.25,
                 "label": "follow_up",
                 "meets_threshold": true,
-                "raw_confidence": 0.9631
+                "raw_confidence": 0.9121
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.9934,
+                "confidence": 0.9583,
                 "confidence_threshold": 0.4,
                 "label": "ambiguous",
                 "meets_threshold": true,
-                "raw_confidence": 0.9405
+                "raw_confidence": 0.9339
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.9934,
+            "confidence": 0.8336,
             "decision_phase": "research",
             "subtype": "follow_up",
             "summary": "Classified as ambiguous intent with subtype follow_up in the research phase.",
@@ -981,22 +948,19 @@
     "response": {
       "meta": {
         "calibration_enabled": true,
+        "iab_mapping_is_placeholder": false,
         "system_version": "0.6.0-phase4"
       },
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.7779,
+            "mapping_confidence": 0.2223,
             "mapping_mode": "nearest_equivalent",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
             "tier1": {
-              "id": "52",
-              "label": "Business and Finance"
-            },
-            "tier2": {
-              "id": "53",
-              "label": "Business"
+              "id": "391",
+              "label": "Personal Finance"
             }
           },
           "intent": {
@@ -1004,31 +968,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.9888,
+                "confidence": 0.9337,
                 "confidence_threshold": 0.22,
                 "label": "research",
                 "meets_threshold": true,
-                "raw_confidence": 0.9639
+                "raw_confidence": 0.9337
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.9487,
+                "confidence": 0.9493,
                 "confidence_threshold": 0.25,
                 "label": "follow_up",
                 "meets_threshold": true,
-                "raw_confidence": 0.9487
+                "raw_confidence": 0.875
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.9916,
+                "confidence": 0.9454,
                 "confidence_threshold": 0.4,
                 "label": "ambiguous",
                 "meets_threshold": true,
-                "raw_confidence": 0.9321
+                "raw_confidence": 0.9149
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.9888,
+            "confidence": 0.9337,
             "decision_phase": "research",
             "subtype": "follow_up",
             "summary": "Classified as ambiguous intent with subtype follow_up in the research phase.",
@@ -1073,30 +1037,23 @@
     "response": {
       "meta": {
         "calibration_enabled": true,
+        "iab_mapping_is_placeholder": false,
         "system_version": "0.6.0-phase4"
       },
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.7753,
-            "mapping_mode": "nearest_equivalent",
+            "mapping_confidence": 0.1233,
+            "mapping_mode": "exact",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
             "tier1": {
-              "id": "596",
-              "label": "Technology & Computing"
+              "id": "239",
+              "label": "Hobbies & Interests"
             },
             "tier2": {
-              "id": "599",
-              "label": "Computing"
-            },
-            "tier3": {
-              "id": "619",
-              "label": "Internet"
-            },
-            "tier4": {
-              "id": "623",
-              "label": "Email"
+              "id": "264",
+              "label": "Content Production"
             }
           },
           "intent": {
@@ -1104,31 +1061,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.9991,
+                "confidence": 0.9632,
                 "confidence_threshold": 0.22,
                 "label": "action",
                 "meets_threshold": true,
-                "raw_confidence": 0.9948
+                "raw_confidence": 0.9632
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.8874,
+                "confidence": 0.9349,
                 "confidence_threshold": 0.25,
                 "label": "signup",
                 "meets_threshold": true,
-                "raw_confidence": 0.8874
+                "raw_confidence": 0.8563
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.9996,
+                "confidence": 0.9378,
                 "confidence_threshold": 0.4,
                 "label": "transactional",
                 "meets_threshold": true,
-                "raw_confidence": 0.9908
+                "raw_confidence": 0.9047
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.8874,
+            "confidence": 0.9349,
             "decision_phase": "action",
             "subtype": "signup",
             "summary": "Classified as transactional intent with subtype signup in the action phase.",
@@ -1167,30 +1124,19 @@
     "response": {
       "meta": {
         "calibration_enabled": true,
+        "iab_mapping_is_placeholder": false,
         "system_version": "0.6.0-phase4"
       },
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.8626,
+            "mapping_confidence": 0.2185,
             "mapping_mode": "nearest_equivalent",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
             "tier1": {
-              "id": "596",
-              "label": "Technology & Computing"
-            },
-            "tier2": {
-              "id": "599",
-              "label": "Computing"
-            },
-            "tier3": {
-              "id": "619",
-              "label": "Internet"
-            },
-            "tier4": {
-              "id": "627",
-              "label": "Search"
+              "id": "52",
+              "label": "Business and Finance"
             }
           },
           "intent": {
@@ -1198,31 +1144,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.9966,
+                "confidence": 0.9572,
                 "confidence_threshold": 0.22,
                 "label": "consideration",
                 "meets_threshold": true,
-                "raw_confidence": 0.9852
+                "raw_confidence": 0.9572
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.9415,
+                "confidence": 0.9432,
                 "confidence_threshold": 0.25,
                 "label": "comparison",
                 "meets_threshold": true,
-                "raw_confidence": 0.9415
+                "raw_confidence": 0.8708
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.9994,
+                "confidence": 0.9622,
                 "confidence_threshold": 0.4,
                 "label": "commercial",
                 "meets_threshold": true,
-                "raw_confidence": 0.9884
+                "raw_confidence": 0.9374
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.9415,
+            "confidence": 0.9432,
             "decision_phase": "consideration",
             "subtype": "comparison",
             "summary": "Classified as commercial intent with subtype comparison in the consideration phase.",
@@ -1261,13 +1207,14 @@
     "response": {
       "meta": {
         "calibration_enabled": true,
+        "iab_mapping_is_placeholder": false,
         "system_version": "0.6.0-phase4"
       },
       "model_output": {
         "classification": {
           "iab_content": {
-            "mapping_confidence": 0.8741,
-            "mapping_mode": "nearest_equivalent",
+            "mapping_confidence": 0.1513,
+            "mapping_mode": "exact",
             "taxonomy": "IAB Content Taxonomy",
             "taxonomy_version": "3.0",
             "tier1": {
@@ -1279,12 +1226,8 @@
               "label": "Computing"
             },
             "tier3": {
-              "id": "619",
-              "label": "Internet"
-            },
-            "tier4": {
-              "id": "620",
-              "label": "Cloud Computing"
+              "id": "618",
+              "label": "Information and Network Security"
             }
           },
           "intent": {
@@ -1292,31 +1235,31 @@
             "component_confidence": {
               "decision_phase": {
                 "calibrated": true,
-                "confidence": 0.9939,
+                "confidence": 0.9609,
                 "confidence_threshold": 0.22,
                 "label": "awareness",
                 "meets_threshold": true,
-                "raw_confidence": 0.9764
+                "raw_confidence": 0.961
               },
               "intent_subtype": {
                 "calibrated": true,
-                "confidence": 0.9545,
+                "confidence": 0.9867,
                 "confidence_threshold": 0.25,
                 "label": "education",
                 "meets_threshold": true,
-                "raw_confidence": 0.9545
+                "raw_confidence": 0.9556
               },
               "intent_type": {
                 "calibrated": true,
-                "confidence": 0.9964,
+                "confidence": 0.975,
                 "confidence_threshold": 0.4,
                 "label": "informational",
                 "meets_threshold": true,
-                "raw_confidence": 0.961
+                "raw_confidence": 0.9567
               },
               "overall_strategy": "min_required_component_confidence"
             },
-            "confidence": 0.9939,
+            "confidence": 0.9609,
             "decision_phase": "awareness",
             "subtype": "education",
             "summary": "Classified as informational intent with subtype education in the awareness phase.",

artifacts/evaluation/latest/decision_phase_difficulty_benchmark_confusion_matrix.csv

@@ -1,8 +1,8 @@
 ,awareness,research,consideration,decision,action,post_purchase,support
 awareness,14,1,0,0,0,0,0
-research,0,14,0,0,0,1,0
+research,0,15,0,0,0,0,0
 consideration,0,1,14,0,0,0,0
 decision,0,0,0,15,0,0,0
-action,0,0,0,1,13,1,0
+action,0,0,0,0,15,0,0
 post_purchase,0,0,0,0,0,15,0
 support,0,0,0,0,0,0,15

artifacts/evaluation/latest/decision_phase_difficulty_benchmark_report.json

@@ -1,10 +1,10 @@
 {
-  "accepted_accuracy": 0.9524,
+  "accepted_accuracy": 0.981,
   "accepted_coverage": 1.0,
-  "accuracy": 0.9524,
-  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/decision_phase_difficulty_benchmark_confusion_matrix.csv",
+  "accuracy": 0.981,
+  "confusion_matrix_path": "/content/agentic-intent-classifier/artifacts/evaluation/latest/decision_phase_difficulty_benchmark_confusion_matrix.csv",
   "count": 105,
-  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/decision_phase_benchmark.jsonl",
+  "dataset_path": "/content/agentic-intent-classifier/data/decision_phase_benchmark.jsonl",
   "difficulty_breakdown": {
     "easy": {
       "accepted_accuracy": 0.9714,
@@ -15,12 +15,12 @@
       "macro_f1": 0.9711
     },
     "hard": {
-      "accepted_accuracy": 0.8857,
+      "accepted_accuracy": 0.9714,
       "accepted_coverage": 1.0,
-      "accuracy": 0.8857,
+      "accuracy": 0.9714,
       "count": 35,
       "fallback_rate": 0.0,
-      "macro_f1": 0.883
+      "macro_f1": 0.9711
     },
     "medium": {
       "accepted_accuracy": 1.0,
@@ -33,13 +33,13 @@
   },
   "fallback_rate": 0.0,
   "head": "decision_phase",
-  "macro_f1": 0.9526,
+  "macro_f1": 0.9812,
   "per_class_metrics": {
-    "accuracy": 0.9523809523809523,
+    "accuracy": 0.9809523809523809,
     "action": {
-      "f1-score": 0.9285714285714286,
+      "f1-score": 1.0,
       "precision": 1.0,
-      "recall": 0.8666666666666667,
+      "recall": 1.0,
       "support": 15.0
     },
     "awareness": {
@@ -55,27 +55,27 @@
       "support": 15.0
     },
     "decision": {
-      "f1-score": 0.967741935483871,
-      "precision": 0.9375,
+      "f1-score": 1.0,
+      "precision": 1.0,
       "recall": 1.0,
       "support": 15.0
     },
     "macro avg": {
-      "f1-score": 0.9525819504665047,
-      "precision": 0.9564075630252101,
-      "recall": 0.9523809523809523,
+      "f1-score": 0.9812192118226601,
+      "precision": 0.9831932773109244,
+      "recall": 0.980952380952381,
       "support": 105.0
     },
     "post_purchase": {
-      "f1-score": 0.9375,
-      "precision": 0.8823529411764706,
+      "f1-score": 1.0,
+      "precision": 1.0,
       "recall": 1.0,
       "support": 15.0
     },
     "research": {
-      "f1-score": 0.9032258064516129,
-      "precision": 0.875,
-      "recall": 0.9333333333333333,
+      "f1-score": 0.9375,
+      "precision": 0.8823529411764706,
+      "recall": 1.0,
       "support": 15.0
     },
     "support": {
@@ -85,9 +85,9 @@
       "support": 15.0
     },
     "weighted avg": {
-      "f1-score": 0.9525819504665048,
-      "precision": 0.9564075630252101,
-      "recall": 0.9523809523809523,
+      "f1-score": 0.9812192118226601,
+      "precision": 0.9831932773109243,
+      "recall": 0.9809523809523809,
       "support": 105.0
     }
   },

artifacts/evaluation/latest/decision_phase_final_wave_cases_confusion_matrix.csv

@@ -1,7 +1,7 @@
 ,awareness,research,consideration,decision,action,post_purchase,support
 awareness,5,0,0,0,0,0,0
-research,1,3,0,0,0,0,0
-consideration,0,0,5,0,0,0,0
+research,2,2,0,0,0,0,0
+consideration,0,1,4,0,0,0,0
 decision,0,0,0,5,0,0,0
 action,0,0,0,0,0,0,0
 post_purchase,0,0,0,0,0,4,0

artifacts/evaluation/latest/decision_phase_final_wave_cases_report.json

@@ -1,15 +1,15 @@
 {
-  "accepted_accuracy": 0.963,
+  "accepted_accuracy": 0.8889,
   "accepted_coverage": 1.0,
-  "accuracy": 0.963,
-  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/decision_phase_final_wave_cases_confusion_matrix.csv",
+  "accuracy": 0.8889,
+  "confusion_matrix_path": "/content/agentic-intent-classifier/artifacts/evaluation/latest/decision_phase_final_wave_cases_confusion_matrix.csv",
   "count": 27,
-  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/decision_phase/final_wave_cases.jsonl",
+  "dataset_path": "/content/agentic-intent-classifier/data/decision_phase/final_wave_cases.jsonl",
   "fallback_rate": 0.0,
   "head": "decision_phase",
-  "macro_f1": 0.961,
+  "macro_f1": 0.8823,
   "per_class_metrics": {
-    "accuracy": 0.9629629629629629,
+    "accuracy": 0.8888888888888888,
     "action": {
       "f1-score": 0.0,
       "precision": 0.0,
@@ -17,15 +17,15 @@
       "support": 0.0
     },
     "awareness": {
-      "f1-score": 0.9090909090909091,
-      "precision": 0.8333333333333334,
+      "f1-score": 0.8333333333333334,
+      "precision": 0.7142857142857143,
       "recall": 1.0,
       "support": 5.0
     },
     "consideration": {
-      "f1-score": 1.0,
+      "f1-score": 0.8888888888888888,
       "precision": 1.0,
-      "recall": 1.0,
+      "recall": 0.8,
       "support": 5.0
     },
     "decision": {
@@ -35,9 +35,9 @@
       "support": 5.0
     },
     "macro avg": {
-      "f1-score": 0.823747680890538,
-      "precision": 0.8333333333333334,
-      "recall": 0.8214285714285714,
+      "f1-score": 0.7562358276643991,
+      "precision": 0.7687074829931974,
+      "recall": 0.7571428571428571,
       "support": 27.0
     },
     "post_purchase": {
@@ -47,9 +47,9 @@
       "support": 4.0
     },
     "research": {
-      "f1-score": 0.8571428571428571,
-      "precision": 1.0,
-      "recall": 0.75,
+      "f1-score": 0.5714285714285714,
+      "precision": 0.6666666666666666,
+      "recall": 0.5,
       "support": 4.0
     },
     "support": {
@@ -59,9 +59,9 @@
       "support": 4.0
     },
     "weighted avg": {
-      "f1-score": 0.962000962000962,
-      "precision": 0.9691358024691359,
-      "recall": 0.9629629629629629,
+      "f1-score": 0.8850676072898296,
+      "precision": 0.8977072310405644,
+      "recall": 0.8888888888888888,
       "support": 27.0
     }
   },

artifacts/evaluation/latest/decision_phase_hard_cases_confusion_matrix.csv

@@ -1,7 +1,7 @@
 ,awareness,research,consideration,decision,action,post_purchase,support
 awareness,6,0,0,0,0,0,0
 research,2,5,0,0,0,0,0
-consideration,0,1,6,0,0,0,0
+consideration,0,2,5,0,0,0,0
 decision,0,0,0,7,0,0,0
 action,0,0,0,0,0,0,0
 post_purchase,0,0,0,0,0,6,0

artifacts/evaluation/latest/decision_phase_hard_cases_report.json

@@ -1,15 +1,15 @@
 {
-  "accepted_accuracy": 0.9231,
+  "accepted_accuracy": 0.8974,
   "accepted_coverage": 1.0,
-  "accuracy": 0.9231,
-  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/decision_phase_hard_cases_confusion_matrix.csv",
+  "accuracy": 0.8974,
+  "confusion_matrix_path": "/content/agentic-intent-classifier/artifacts/evaluation/latest/decision_phase_hard_cases_confusion_matrix.csv",
   "count": 39,
-  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/decision_phase/hard_cases.jsonl",
+  "dataset_path": "/content/agentic-intent-classifier/data/decision_phase/hard_cases.jsonl",
   "fallback_rate": 0.0,
   "head": "decision_phase",
-  "macro_f1": 0.9249,
+  "macro_f1": 0.9008,
   "per_class_metrics": {
-    "accuracy": 0.9230769230769231,
+    "accuracy": 0.8974358974358975,
     "action": {
       "f1-score": 0.0,
       "precision": 0.0,
@@ -23,9 +23,9 @@
       "support": 6.0
     },
     "consideration": {
-      "f1-score": 0.9230769230769231,
+      "f1-score": 0.8333333333333334,
       "precision": 1.0,
-      "recall": 0.8571428571428571,
+      "recall": 0.7142857142857143,
       "support": 7.0
     },
     "decision": {
@@ -35,9 +35,9 @@
       "support": 7.0
     },
     "macro avg": {
-      "f1-score": 0.792778649921507,
-      "precision": 0.7976190476190477,
-      "recall": 0.7959183673469388,
+      "f1-score": 0.772108843537415,
+      "precision": 0.7806122448979592,
+      "recall": 0.7755102040816327,
       "support": 39.0
     },
     "post_purchase": {
@@ -47,8 +47,8 @@
       "support": 6.0
     },
     "research": {
-      "f1-score": 0.7692307692307693,
-      "precision": 0.8333333333333334,
+      "f1-score": 0.7142857142857143,
+      "precision": 0.7142857142857143,
       "recall": 0.7142857142857143,
       "support": 7.0
     },
@@ -59,9 +59,9 @@
       "support": 6.0
     },
     "weighted avg": {
-      "f1-score": 0.9227951535643845,
-      "precision": 0.9316239316239316,
-      "recall": 0.9230769230769231,
+      "f1-score": 0.8968253968253967,
+      "precision": 0.9102564102564102,
+      "recall": 0.8974358974358975,
       "support": 39.0
     }
   },

artifacts/evaluation/latest/decision_phase_test_confusion_matrix.csv

@@ -1,8 +1,8 @@
 ,awareness,research,consideration,decision,action,post_purchase,support
 awareness,3,0,0,0,0,0,0
 research,3,2,0,0,0,0,0
-consideration,0,1,4,0,0,0,0
+consideration,0,2,3,0,0,0,0
 decision,0,0,0,5,0,0,0
 action,0,0,0,0,3,0,0
 post_purchase,0,0,0,0,0,4,0
-support,0,0,0,0,0,0,4
+support,0,0,0,0,0,1,3

artifacts/evaluation/latest/decision_phase_test_report.json

@@ -1,15 +1,15 @@
 {
-  "accepted_accuracy": 0.8621,
+  "accepted_accuracy": 0.7931,
   "accepted_coverage": 1.0,
-  "accuracy": 0.8621,
-  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/decision_phase_test_confusion_matrix.csv",
+  "accuracy": 0.7931,
+  "confusion_matrix_path": "/content/agentic-intent-classifier/artifacts/evaluation/latest/decision_phase_test_confusion_matrix.csv",
   "count": 29,
-  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/decision_phase/test.jsonl",
+  "dataset_path": "/content/agentic-intent-classifier/data/decision_phase/test.jsonl",
   "fallback_rate": 0.0,
   "head": "decision_phase",
-  "macro_f1": 0.8651,
+  "macro_f1": 0.801,
   "per_class_metrics": {
-    "accuracy": 0.8620689655172413,
+    "accuracy": 0.7931034482758621,
     "action": {
       "f1-score": 1.0,
       "precision": 1.0,
@@ -23,9 +23,9 @@
       "support": 3.0
     },
     "consideration": {
-      "f1-score": 0.8888888888888888,
+      "f1-score": 0.75,
       "precision": 1.0,
-      "recall": 0.8,
+      "recall": 0.6,
       "support": 5.0
     },
     "decision": {
@@ -35,33 +35,33 @@
       "support": 5.0
     },
     "macro avg": {
-      "f1-score": 0.865079365079365,
-      "precision": 0.8809523809523808,
-      "recall": 0.8857142857142858,
+      "f1-score": 0.8010204081632653,
+      "precision": 0.8285714285714285,
+      "recall": 0.8214285714285714,
       "support": 29.0
     },
     "post_purchase": {
-      "f1-score": 1.0,
-      "precision": 1.0,
+      "f1-score": 0.8888888888888888,
+      "precision": 0.8,
       "recall": 1.0,
       "support": 4.0
     },
     "research": {
-      "f1-score": 0.5,
-      "precision": 0.6666666666666666,
+      "f1-score": 0.4444444444444444,
+      "precision": 0.5,
       "recall": 0.4,
       "support": 5.0
     },
     "support": {
-      "f1-score": 1.0,
+      "f1-score": 0.8571428571428571,
       "precision": 1.0,
-      "recall": 1.0,
+      "recall": 0.75,
       "support": 4.0
     },
     "weighted avg": {
-      "f1-score": 0.8601532567049808,
-      "precision": 0.8908045977011494,
-      "recall": 0.8620689655172413,
+      "f1-score": 0.7915982484948002,
+      "precision": 0.8344827586206897,
+      "recall": 0.7931034482758621,
       "support": 29.0
     }
   },

artifacts/evaluation/latest/decision_phase_train_confusion_matrix.csv

@@ -1,8 +1,8 @@
 ,awareness,research,consideration,decision,action,post_purchase,support
 awareness,16,0,0,0,0,0,0
-research,1,14,0,0,0,0,0
-consideration,0,0,17,0,0,0,0
-decision,0,0,0,16,0,0,0
+research,1,13,1,0,0,0,0
+consideration,0,3,14,0,0,0,0
+decision,0,0,1,15,0,0,0
 action,0,0,0,0,10,0,0
-post_purchase,0,0,0,0,0,14,0
+post_purchase,0,1,0,0,0,13,0
 support,0,0,0,0,0,0,14

artifacts/evaluation/latest/decision_phase_train_report.json

@@ -1,15 +1,15 @@
 {
-  "accepted_accuracy": 0.9902,
+  "accepted_accuracy": 0.9314,
   "accepted_coverage": 1.0,
-  "accuracy": 0.9902,
-  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/decision_phase_train_confusion_matrix.csv",
+  "accuracy": 0.9314,
+  "confusion_matrix_path": "/content/agentic-intent-classifier/artifacts/evaluation/latest/decision_phase_train_confusion_matrix.csv",
   "count": 102,
-  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/decision_phase/train.jsonl",
+  "dataset_path": "/content/agentic-intent-classifier/data/decision_phase/train.jsonl",
   "fallback_rate": 0.0,
   "head": "decision_phase",
-  "macro_f1": 0.9907,
+  "macro_f1": 0.9373,
   "per_class_metrics": {
-    "accuracy": 0.9901960784313726,
+    "accuracy": 0.9313725490196079,
     "action": {
       "f1-score": 1.0,
       "precision": 1.0,
@@ -23,33 +23,33 @@
       "support": 16.0
     },
     "consideration": {
-      "f1-score": 1.0,
-      "precision": 1.0,
-      "recall": 1.0,
+      "f1-score": 0.8484848484848485,
+      "precision": 0.875,
+      "recall": 0.8235294117647058,
       "support": 17.0
     },
     "decision": {
-      "f1-score": 1.0,
+      "f1-score": 0.967741935483871,
       "precision": 1.0,
-      "recall": 1.0,
+      "recall": 0.9375,
       "support": 16.0
     },
     "macro avg": {
-      "f1-score": 0.9907448872966115,
-      "precision": 0.9915966386554622,
-      "recall": 0.9904761904761905,
+      "f1-score": 0.9373409595183789,
+      "precision": 0.9401260504201681,
+      "recall": 0.9366096438575431,
       "support": 102.0
     },
     "post_purchase": {
-      "f1-score": 1.0,
+      "f1-score": 0.9629629629629629,
       "precision": 1.0,
-      "recall": 1.0,
+      "recall": 0.9285714285714286,
       "support": 14.0
     },
     "research": {
-      "f1-score": 0.9655172413793104,
-      "precision": 1.0,
-      "recall": 0.9333333333333333,
+      "f1-score": 0.8125,
+      "precision": 0.7647058823529411,
+      "recall": 0.8666666666666667,
       "support": 15.0
     },
     "support": {
@@ -59,9 +59,9 @@
       "support": 14.0
     },
     "weighted avg": {
-      "f1-score": 0.9901755895670704,
-      "precision": 0.9907727797001153,
-      "recall": 0.9901960784313726,
+      "f1-score": 0.9322769253786015,
+      "precision": 0.9353373702422145,
+      "recall": 0.9313725490196079,
       "support": 102.0
     }
   },

artifacts/evaluation/latest/decision_phase_val_confusion_matrix.csv

@@ -1,6 +1,6 @@
 ,awareness,research,consideration,decision,action,post_purchase,support
 awareness,5,0,0,0,0,0,0
-research,1,3,0,0,0,0,0
+research,2,2,0,0,0,0,0
 consideration,0,0,5,0,0,0,0
 decision,0,0,1,3,0,0,0
 action,0,0,0,0,3,0,0

artifacts/evaluation/latest/decision_phase_val_report.json

@@ -1,15 +1,15 @@
 {
-  "accepted_accuracy": 0.8966,
+  "accepted_accuracy": 0.8621,
   "accepted_coverage": 1.0,
-  "accuracy": 0.8966,
-  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/decision_phase_val_confusion_matrix.csv",
+  "accuracy": 0.8621,
+  "confusion_matrix_path": "/content/agentic-intent-classifier/artifacts/evaluation/latest/decision_phase_val_confusion_matrix.csv",
   "count": 29,
-  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/decision_phase/val.jsonl",
+  "dataset_path": "/content/agentic-intent-classifier/data/decision_phase/val.jsonl",
   "fallback_rate": 0.0,
   "head": "decision_phase",
-  "macro_f1": 0.8975,
+  "macro_f1": 0.8612,
   "per_class_metrics": {
-    "accuracy": 0.896551724137931,
+    "accuracy": 0.8620689655172413,
     "action": {
       "f1-score": 1.0,
       "precision": 1.0,
@@ -17,8 +17,8 @@
       "support": 3.0
     },
     "awareness": {
-      "f1-score": 0.9090909090909091,
-      "precision": 0.8333333333333334,
+      "f1-score": 0.8333333333333334,
+      "precision": 0.7142857142857143,
       "recall": 1.0,
       "support": 5.0
     },
@@ -35,9 +35,9 @@
       "support": 4.0
     },
     "macro avg": {
-      "f1-score": 0.8974953617810761,
-      "precision": 0.9166666666666667,
-      "recall": 0.8928571428571429,
+      "f1-score": 0.8611626468769326,
+      "precision": 0.8877551020408163,
+      "recall": 0.8571428571428571,
       "support": 29.0
     },
     "post_purchase": {
@@ -47,9 +47,9 @@
       "support": 4.0
     },
     "research": {
-      "f1-score": 0.75,
-      "precision": 0.75,
-      "recall": 0.75,
+      "f1-score": 0.5714285714285714,
+      "precision": 0.6666666666666666,
+      "recall": 0.5,
       "support": 4.0
     },
     "support": {
@@ -59,9 +59,9 @@
       "support": 4.0
     },
     "weighted avg": {
-      "f1-score": 0.8947604120017911,
-      "precision": 0.9080459770114944,
-      "recall": 0.896551724137931,
+      "f1-score": 0.8570682191371846,
+      "precision": 0.8760262725779967,
+      "recall": 0.8620689655172413,
       "support": 29.0
     }
   },

artifacts/evaluation/latest/iab_behavior_lock_regression.json

@@ -1,21 +1,21 @@
 {
   "by_status": {
     "must_fix": {
-      "failed": 0,
-      "passed": 12,
+      "failed": 12,
+      "passed": 0,
       "total": 12
     }
   },
-  "cases_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/examples/iab_behavior_lock_cases.json",
+  "cases_path": "/content/agentic-intent-classifier/examples/iab_behavior_lock_cases.json",
   "count": 12,
-  "failed": 0,
-  "passed": 12,
+  "failed": 12,
+  "passed": 0,
   "results": [
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
         "model_output.classification.iab_content.tier1.label": "Automotive",
-        "model_output.classification.iab_content.tier2.label": "Auto Type"
+        "model_output.classification.iab_content.tier2.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -23,15 +23,21 @@
         "model_output.classification.iab_content.tier2.label": "Auto Type"
       },
       "id": "car-buying-maps-to-automotive-buying",
-      "mismatches": [],
+      "mismatches": [
+        {
+          "actual": null,
+          "expected": "Auto Type",
+          "path": "model_output.classification.iab_content.tier2.label"
+        }
+      ],
       "notes": "Vehicle shopping queries should map into the automotive buying branch, not business sales.",
-      "pass": true,
+      "pass": false,
       "status": "must_fix",
       "text": "Which car to buy in 2026"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
         "model_output.classification.iab_content.tier3.label": "Laptops"
@@ -43,9 +49,15 @@
         "model_output.classification.iab_content.tier3.label": "Laptops"
       },
       "id": "laptop-buying-maps-to-laptops",
-      "mismatches": [],
+      "mismatches": [
+        {
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        }
+      ],
       "notes": "Laptop shopping should resolve into the laptops branch, not business sales.",
-      "pass": true,
+      "pass": false,
       "status": "must_fix",
       "text": "Which laptop to buy in 2026"
     },
@@ -53,8 +65,8 @@
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
-        "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": "Laptops"
+        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -63,9 +75,20 @@
         "model_output.classification.iab_content.tier3.label": "Laptops"
       },
       "id": "labtop-buying-maps-to-laptops",
-      "mismatches": [],
+      "mismatches": [
+        {
+          "actual": null,
+          "expected": "Computing",
+          "path": "model_output.classification.iab_content.tier2.label"
+        },
+        {
+          "actual": null,
+          "expected": "Laptops",
+          "path": "model_output.classification.iab_content.tier3.label"
+        }
+      ],
       "notes": "Common typo handling should still land in the laptops branch.",
-      "pass": true,
+      "pass": false,
       "status": "must_fix",
       "text": "Which labtop to buy in 2026"
     },
@@ -74,7 +97,7 @@
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+        "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -83,9 +106,15 @@
         "model_output.classification.iab_content.tier3.label": "Software and Applications"
       },
       "id": "crm-awareness-maps-to-sales",
-      "mismatches": [],
+      "mismatches": [
+        {
+          "actual": null,
+          "expected": "Software and Applications",
+          "path": "model_output.classification.iab_content.tier3.label"
+        }
+      ],
       "notes": "CRM education should resolve to the closest business/sales path, not generic software.",
-      "pass": true,
+      "pass": false,
       "status": "must_fix",
       "text": "What is CRM software?"
     },
@@ -93,8 +122,8 @@
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
-        "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": "Internet"
+        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -103,18 +132,29 @@
         "model_output.classification.iab_content.tier3.label": "Internet"
       },
       "id": "crm-comparison-maps-to-sales",
-      "mismatches": [],
+      "mismatches": [
+        {
+          "actual": null,
+          "expected": "Computing",
+          "path": "model_output.classification.iab_content.tier2.label"
+        },
+        {
+          "actual": null,
+          "expected": "Internet",
+          "path": "model_output.classification.iab_content.tier3.label"
+        }
+      ],
       "notes": "Direct CRM vendor comparison should map cleanly into the sales domain.",
-      "pass": true,
+      "pass": false,
       "status": "must_fix",
       "text": "HubSpot vs Zoho for a small team"
     },
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
-        "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": "Internet"
+        "model_output.classification.iab_content.tier1.label": "Careers",
+        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -123,25 +163,47 @@
         "model_output.classification.iab_content.tier3.label": "Internet"
       },
       "id": "marketing-tools-map-to-marketing",
-      "mismatches": [],
+      "mismatches": [
+        {
+          "actual": "Careers",
+          "expected": "Technology & Computing",
+          "path": "model_output.classification.iab_content.tier1.label"
+        },
+        {
+          "actual": null,
+          "expected": "Computing",
+          "path": "model_output.classification.iab_content.tier2.label"
+        },
+        {
+          "actual": null,
+          "expected": "Internet",
+          "path": "model_output.classification.iab_content.tier3.label"
+        }
+      ],
       "notes": "Marketing tool discovery should map to the marketing and advertising branch.",
-      "pass": true,
+      "pass": false,
       "status": "must_fix",
       "text": "Best AI SEO tools for content teams"
     },
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Technology & Computing"
+        "model_output.classification.iab_content.tier1.label": "Sports"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing"
       },
       "id": "ml-explanation-maps-to-ai",
-      "mismatches": [],
+      "mismatches": [
+        {
+          "actual": "Sports",
+          "expected": "Technology & Computing",
+          "path": "model_output.classification.iab_content.tier1.label"
+        }
+      ],
       "notes": "ML and NLP educational prompts should land in the AI branch.",
-      "pass": true,
+      "pass": false,
       "status": "must_fix",
       "text": "What is intent classification in NLP?"
     },
@@ -150,7 +212,7 @@
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": "Internet"
+        "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -159,15 +221,21 @@
         "model_output.classification.iab_content.tier3.label": "Internet"
       },
       "id": "support-credential-help-maps-to-business-it",
-      "mismatches": [],
+      "mismatches": [
+        {
+          "actual": null,
+          "expected": "Internet",
+          "path": "model_output.classification.iab_content.tier3.label"
+        }
+      ],
       "notes": "Credential and account help should map to business IT rather than generic business.",
-      "pass": true,
+      "pass": false,
       "status": "must_fix",
       "text": "How do I reset my password?"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Food & Drink",
         "model_output.classification.iab_content.tier2.label": "Dining Out"
       },
@@ -177,18 +245,24 @@
         "model_output.classification.iab_content.tier2.label": "Dining Out"
       },
       "id": "restaurant-booking-maps-to-dining-out",
-      "mismatches": [],
+      "mismatches": [
+        {
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        }
+      ],
       "notes": "Generic dining requests should not inherit the repo's business default.",
-      "pass": true,
+      "pass": false,
       "status": "must_fix",
       "text": "Book a table for 2 tonight"
     },
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Hobbies & Interests",
-        "model_output.classification.iab_content.tier2.label": "Content Production",
-        "model_output.classification.iab_content.tier3.label": "Freelance Writing"
+        "model_output.classification.iab_content.tier1.label": "Sports",
+        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -197,19 +271,35 @@
         "model_output.classification.iab_content.tier3.label": "Freelance Writing"
       },
       "id": "trial-signup-maps-to-software",
-      "mismatches": [],
+      "mismatches": [
+        {
+          "actual": "Sports",
+          "expected": "Hobbies & Interests",
+          "path": "model_output.classification.iab_content.tier1.label"
+        },
+        {
+          "actual": null,
+          "expected": "Content Production",
+          "path": "model_output.classification.iab_content.tier2.label"
+        },
+        {
+          "actual": null,
+          "expected": "Freelance Writing",
+          "path": "model_output.classification.iab_content.tier3.label"
+        }
+      ],
       "notes": "Software action queries should map to the software/application branch.",
-      "pass": true,
+      "pass": false,
       "status": "must_fix",
       "text": "Start my free trial"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
-        "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": "Software and Applications",
-        "model_output.classification.iab_content.tier4.label": "Communication"
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Careers",
+        "model_output.classification.iab_content.tier2.label": "Remote Working",
+        "model_output.classification.iab_content.tier3.label": null,
+        "model_output.classification.iab_content.tier4.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -219,15 +309,41 @@
         "model_output.classification.iab_content.tier4.label": "Communication"
       },
       "id": "communication-software-maps-to-tier4",
-      "mismatches": [],
+      "mismatches": [
+        {
+          "actual": "Careers",
+          "expected": "Technology & Computing",
+          "path": "model_output.classification.iab_content.tier1.label"
+        },
+        {
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "Remote Working",
+          "expected": "Computing",
+          "path": "model_output.classification.iab_content.tier2.label"
+        },
+        {
+          "actual": null,
+          "expected": "Software and Applications",
+          "path": "model_output.classification.iab_content.tier3.label"
+        },
+        {
+          "actual": null,
+          "expected": "Communication",
+          "path": "model_output.classification.iab_content.tier4.label"
+        }
+      ],
       "notes": "Full taxonomy support should preserve the tier4 communication branch.",
-      "pass": true,
+      "pass": false,
       "status": "must_fix",
       "text": "best communication software for remote teams"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Food & Drink"
       },
       "expected": {
@@ -235,9 +351,15 @@
         "model_output.classification.iab_content.tier1.label": "Food & Drink"
       },
       "id": "vodka-query-maps-to-alcoholic-beverages",
-      "mismatches": [],
+      "mismatches": [
+        {
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        }
+      ],
       "notes": "Food and beverage prompts should not fall through to the business default.",
-      "pass": true,
+      "pass": false,
       "status": "must_fix",
       "text": "what is best vodka drink should i try"
     }

artifacts/evaluation/latest/iab_content_cross_vertical_benchmark_report.json

@@ -1,93 +1,98 @@
 {
-  "accepted_accuracy": 0.3444,
-  "accepted_coverage": 1.0,
-  "accuracy": 0.3444,
+  "accepted_accuracy": 0.3,
+  "accepted_coverage": 0.8889,
+  "accuracy": 0.2667,
   "count": 90,
-  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/iab_cross_vertical_benchmark.jsonl",
+  "dataset_path": "/content/agentic-intent-classifier/data/iab_cross_vertical_benchmark.jsonl",
   "difficulty_breakdown": {
     "easy": {
-      "accepted_accuracy": 0.2667,
-      "accepted_coverage": 1.0,
-      "accuracy": 0.2667,
+      "accepted_accuracy": 0.28,
+      "accepted_coverage": 0.8333,
+      "accuracy": 0.2333,
       "count": 30,
-      "fallback_rate": 0.0,
-      "macro_f1": 0.1633
+      "fallback_rate": 0.1667,
+      "macro_f1": 0.1556
     },
     "hard": {
-      "accepted_accuracy": 0.3667,
-      "accepted_coverage": 1.0,
-      "accuracy": 0.3667,
+      "accepted_accuracy": 0.3846,
+      "accepted_coverage": 0.8667,
+      "accuracy": 0.3333,
       "count": 30,
-      "fallback_rate": 0.0,
-      "macro_f1": 0.2174
+      "fallback_rate": 0.1333,
+      "macro_f1": 0.2083
     },
     "medium": {
-      "accepted_accuracy": 0.4,
-      "accepted_coverage": 1.0,
-      "accuracy": 0.4,
+      "accepted_accuracy": 0.2414,
+      "accepted_coverage": 0.9667,
+      "accuracy": 0.2333,
       "count": 30,
-      "fallback_rate": 0.0,
-      "macro_f1": 0.2667
+      "fallback_rate": 0.0333,
+      "macro_f1": 0.1458
     }
   },
-  "fallback_rate": 0.0,
+  "fallback_rate": 0.1111,
   "head": "iab_content",
-  "macro_f1": 0.1808,
-  "primary_source": "embedding_retrieval",
+  "macro_f1": 0.1418,
+  "primary_source": "supervised_classifier",
   "suite": "cross_vertical_benchmark",
   "tier_metrics": {
-    "average_prediction_depth": 2.5333,
+    "average_prediction_depth": 1.9556,
     "error_buckets": {
-      "exact_match": 31,
-      "parent_safe_stop": 5,
-      "right_tier1_wrong_tier2": 19,
-      "wrong_deep_leaf": 13,
-      "wrong_tier1": 22
+      "exact_match": 24,
+      "parent_safe_stop": 10,
+      "right_tier1_wrong_tier2": 15,
+      "wrong_deep_leaf": 12,
+      "wrong_tier1": 29
     },
-    "exact_path_accuracy": 0.3444,
-    "parent_safe_accuracy": 0.4889,
-    "tier1_accuracy": 0.7556,
-    "tier2_accuracy": 0.5238,
-    "tier3_accuracy": 0.4762,
-    "tier4_accuracy": 1.0
+    "exact_path_accuracy": 0.2667,
+    "parent_safe_accuracy": 0.4556,
+    "tier1_accuracy": 0.6778,
+    "tier2_accuracy": 0.4762,
+    "tier3_accuracy": 0.2143,
+    "tier4_accuracy": 0.0
   },
   "view_metrics": {
+    "classifier": {
+      "average_prediction_depth": 1.9556,
+      "error_buckets": {
+        "exact_match": 24,
+        "parent_safe_stop": 10,
+        "right_tier1_wrong_tier2": 15,
+        "wrong_deep_leaf": 12,
+        "wrong_tier1": 29
+      },
+      "exact_path_accuracy": 0.2667,
+      "parent_safe_accuracy": 0.4556,
+      "tier1_accuracy": 0.6778,
+      "tier2_accuracy": 0.4762,
+      "tier3_accuracy": 0.2143,
+      "tier4_accuracy": 0.0
+    },
     "combined_path": {
-      "average_prediction_depth": 2.5333,
+      "average_prediction_depth": 1.9556,
       "error_buckets": {
-        "exact_match": 27,
-        "parent_safe_stop": 5,
-        "right_tier1_wrong_tier2": 19,
-        "wrong_deep_leaf": 17,
-        "wrong_tier1": 22
+        "exact_match": 24,
+        "parent_safe_stop": 10,
+        "right_tier1_wrong_tier2": 15,
+        "wrong_deep_leaf": 12,
+        "wrong_tier1": 29
       },
-      "exact_path_accuracy": 0.3,
-      "fallback_overuse_count": 12,
-      "fallback_rate": 0.1333,
-      "parent_safe_accuracy": 0.4444,
-      "tier1_accuracy": 0.7556,
-      "tier2_accuracy": 0.5238,
-      "tier3_accuracy": 0.381,
-      "tier4_accuracy": 0.5
+      "exact_path_accuracy": 0.2667,
+      "fallback_overuse_count": 17,
+      "fallback_rate": 0.1889,
+      "parent_safe_accuracy": 0.4556,
+      "tier1_accuracy": 0.6778,
+      "tier2_accuracy": 0.4762,
+      "tier3_accuracy": 0.2143,
+      "tier4_accuracy": 0.0
     },
     "disagreements": {
-      "retrieval_vs_combined": 0
+      "classifier_vs_combined": 0
     },
-    "embedding_retrieval": {
-      "average_prediction_depth": 2.5333,
-      "error_buckets": {
-        "exact_match": 27,
-        "parent_safe_stop": 5,
-        "right_tier1_wrong_tier2": 19,
-        "wrong_deep_leaf": 17,
-        "wrong_tier1": 22
-      },
-      "exact_path_accuracy": 0.3,
-      "parent_safe_accuracy": 0.4444,
-      "tier1_accuracy": 0.7556,
-      "tier2_accuracy": 0.5238,
-      "tier3_accuracy": 0.381,
-      "tier4_accuracy": 0.5
+    "shadow_embedding_retrieval": {
+      "hint": "Set IAB_EVAL_INCLUDE_SHADOW_RETRIEVAL=1 to run shadow embedding retrieval (downloads/loads gte-Qwen2 when index is present).",
+      "reason": "disabled_by_default",
+      "skipped": true
     }
   }
 }

artifacts/evaluation/latest/iab_content_difficulty_benchmark_report.json

@@ -1,93 +1,98 @@
 {
-  "accepted_accuracy": 0.3782,
-  "accepted_coverage": 1.0,
-  "accuracy": 0.3782,
+  "accepted_accuracy": 0.4219,
+  "accepted_coverage": 0.8205,
+  "accuracy": 0.3462,
   "count": 156,
-  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/iab_benchmark.jsonl",
+  "dataset_path": "/content/agentic-intent-classifier/data/iab_benchmark.jsonl",
   "difficulty_breakdown": {
     "easy": {
-      "accepted_accuracy": 0.4038,
-      "accepted_coverage": 1.0,
-      "accuracy": 0.4038,
+      "accepted_accuracy": 0.4889,
+      "accepted_coverage": 0.8654,
+      "accuracy": 0.4231,
       "count": 52,
-      "fallback_rate": 0.0,
-      "macro_f1": 0.2171
+      "fallback_rate": 0.1346,
+      "macro_f1": 0.2305
     },
     "hard": {
-      "accepted_accuracy": 0.3077,
-      "accepted_coverage": 1.0,
-      "accuracy": 0.3077,
+      "accepted_accuracy": 0.3846,
+      "accepted_coverage": 0.75,
+      "accuracy": 0.2885,
       "count": 52,
-      "fallback_rate": 0.0,
-      "macro_f1": 0.1626
+      "fallback_rate": 0.25,
+      "macro_f1": 0.1638
     },
     "medium": {
-      "accepted_accuracy": 0.4231,
-      "accepted_coverage": 1.0,
-      "accuracy": 0.4231,
+      "accepted_accuracy": 0.3864,
+      "accepted_coverage": 0.8462,
+      "accuracy": 0.3269,
       "count": 52,
-      "fallback_rate": 0.0,
-      "macro_f1": 0.2265
+      "fallback_rate": 0.1538,
+      "macro_f1": 0.1819
     }
   },
-  "fallback_rate": 0.0,
+  "fallback_rate": 0.1795,
   "head": "iab_content",
-  "macro_f1": 0.1593,
-  "primary_source": "embedding_retrieval",
+  "macro_f1": 0.1478,
+  "primary_source": "supervised_classifier",
   "suite": "difficulty_benchmark",
   "tier_metrics": {
-    "average_prediction_depth": 2.5833,
+    "average_prediction_depth": 2.0256,
     "error_buckets": {
-      "exact_match": 59,
-      "parent_safe_stop": 17,
-      "right_tier1_wrong_tier2": 42,
-      "wrong_deep_leaf": 13,
-      "wrong_tier1": 25
+      "exact_match": 54,
+      "parent_safe_stop": 21,
+      "right_tier1_wrong_tier2": 37,
+      "wrong_deep_leaf": 3,
+      "wrong_tier1": 41
     },
-    "exact_path_accuracy": 0.3782,
-    "parent_safe_accuracy": 0.6154,
-    "tier1_accuracy": 0.8397,
-    "tier2_accuracy": 0.5705,
-    "tier3_accuracy": 0.5648,
-    "tier4_accuracy": 0.5833
+    "exact_path_accuracy": 0.3462,
+    "parent_safe_accuracy": 0.6603,
+    "tier1_accuracy": 0.7372,
+    "tier2_accuracy": 0.5,
+    "tier3_accuracy": 0.3519,
+    "tier4_accuracy": 0.2917
   },
   "view_metrics": {
+    "classifier": {
+      "average_prediction_depth": 2.0256,
+      "error_buckets": {
+        "exact_match": 49,
+        "parent_safe_stop": 21,
+        "right_tier1_wrong_tier2": 37,
+        "wrong_deep_leaf": 8,
+        "wrong_tier1": 41
+      },
+      "exact_path_accuracy": 0.3141,
+      "parent_safe_accuracy": 0.6282,
+      "tier1_accuracy": 0.7372,
+      "tier2_accuracy": 0.5,
+      "tier3_accuracy": 0.3056,
+      "tier4_accuracy": 0.2083
+    },
     "combined_path": {
-      "average_prediction_depth": 2.5833,
+      "average_prediction_depth": 2.0256,
       "error_buckets": {
-        "exact_match": 48,
-        "parent_safe_stop": 17,
-        "right_tier1_wrong_tier2": 42,
-        "wrong_deep_leaf": 24,
-        "wrong_tier1": 25
+        "exact_match": 49,
+        "parent_safe_stop": 21,
+        "right_tier1_wrong_tier2": 37,
+        "wrong_deep_leaf": 8,
+        "wrong_tier1": 41
       },
-      "exact_path_accuracy": 0.3077,
-      "fallback_overuse_count": 11,
-      "fallback_rate": 0.0705,
-      "parent_safe_accuracy": 0.5449,
-      "tier1_accuracy": 0.8397,
-      "tier2_accuracy": 0.5705,
-      "tier3_accuracy": 0.4352,
-      "tier4_accuracy": 0.25
+      "exact_path_accuracy": 0.3141,
+      "fallback_overuse_count": 14,
+      "fallback_rate": 0.0897,
+      "parent_safe_accuracy": 0.6282,
+      "tier1_accuracy": 0.7372,
+      "tier2_accuracy": 0.5,
+      "tier3_accuracy": 0.3056,
+      "tier4_accuracy": 0.2083
     },
     "disagreements": {
-      "retrieval_vs_combined": 0
+      "classifier_vs_combined": 0
     },
-    "embedding_retrieval": {
-      "average_prediction_depth": 2.5833,
-      "error_buckets": {
-        "exact_match": 48,
-        "parent_safe_stop": 17,
-        "right_tier1_wrong_tier2": 42,
-        "wrong_deep_leaf": 24,
-        "wrong_tier1": 25
-      },
-      "exact_path_accuracy": 0.3077,
-      "parent_safe_accuracy": 0.5449,
-      "tier1_accuracy": 0.8397,
-      "tier2_accuracy": 0.5705,
-      "tier3_accuracy": 0.4352,
-      "tier4_accuracy": 0.25
+    "shadow_embedding_retrieval": {
+      "hint": "Set IAB_EVAL_INCLUDE_SHADOW_RETRIEVAL=1 to run shadow embedding retrieval (downloads/loads gte-Qwen2 when index is present).",
+      "reason": "disabled_by_default",
+      "skipped": true
     }
   }
 }

artifacts/evaluation/latest/iab_content_extended_cases_report.json

@@ -1,64 +1,69 @@
 {
-  "accepted_accuracy": 0.25,
-  "accepted_coverage": 1.0,
-  "accuracy": 0.25,
+  "accepted_accuracy": 0.4286,
+  "accepted_coverage": 0.875,
+  "accuracy": 0.375,
   "count": 8,
-  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/iab/extended_cases.jsonl",
-  "fallback_rate": 0.0,
+  "dataset_path": "/content/agentic-intent-classifier/data/iab/extended_cases.jsonl",
+  "fallback_rate": 0.125,
   "head": "iab_content",
-  "macro_f1": 0.1429,
-  "primary_source": "embedding_retrieval",
+  "macro_f1": 0.2308,
+  "primary_source": "supervised_classifier",
   "suite": "extended_cases",
   "tier_metrics": {
-    "average_prediction_depth": 2.375,
+    "average_prediction_depth": 2.0,
     "error_buckets": {
-      "exact_match": 2,
-      "right_tier1_wrong_tier2": 3,
+      "exact_match": 3,
+      "right_tier1_wrong_tier2": 2,
       "wrong_deep_leaf": 2,
       "wrong_tier1": 1
     },
-    "exact_path_accuracy": 0.25,
+    "exact_path_accuracy": 0.375,
     "parent_safe_accuracy": 0.375,
     "tier1_accuracy": 0.875,
-    "tier2_accuracy": 0.4286,
-    "tier3_accuracy": 1.0,
+    "tier2_accuracy": 0.5714,
+    "tier3_accuracy": 0.0,
     "tier4_accuracy": 0.0
   },
   "view_metrics": {
-    "combined_path": {
-      "average_prediction_depth": 2.375,
+    "classifier": {
+      "average_prediction_depth": 2.0,
       "error_buckets": {
-        "exact_match": 2,
-        "right_tier1_wrong_tier2": 3,
+        "exact_match": 3,
+        "right_tier1_wrong_tier2": 2,
         "wrong_deep_leaf": 2,
         "wrong_tier1": 1
       },
-      "exact_path_accuracy": 0.25,
-      "fallback_overuse_count": 1,
-      "fallback_rate": 0.125,
+      "exact_path_accuracy": 0.375,
       "parent_safe_accuracy": 0.375,
       "tier1_accuracy": 0.875,
-      "tier2_accuracy": 0.4286,
+      "tier2_accuracy": 0.5714,
       "tier3_accuracy": 0.0,
       "tier4_accuracy": 0.0
     },
-    "disagreements": {
-      "retrieval_vs_combined": 0
-    },
-    "embedding_retrieval": {
-      "average_prediction_depth": 2.375,
+    "combined_path": {
+      "average_prediction_depth": 2.0,
       "error_buckets": {
-        "exact_match": 2,
-        "right_tier1_wrong_tier2": 3,
+        "exact_match": 3,
+        "right_tier1_wrong_tier2": 2,
         "wrong_deep_leaf": 2,
         "wrong_tier1": 1
       },
-      "exact_path_accuracy": 0.25,
+      "exact_path_accuracy": 0.375,
+      "fallback_overuse_count": 2,
+      "fallback_rate": 0.25,
       "parent_safe_accuracy": 0.375,
       "tier1_accuracy": 0.875,
-      "tier2_accuracy": 0.4286,
+      "tier2_accuracy": 0.5714,
       "tier3_accuracy": 0.0,
       "tier4_accuracy": 0.0
+    },
+    "disagreements": {
+      "classifier_vs_combined": 0
+    },
+    "shadow_embedding_retrieval": {
+      "hint": "Set IAB_EVAL_INCLUDE_SHADOW_RETRIEVAL=1 to run shadow embedding retrieval (downloads/loads gte-Qwen2 when index is present).",
+      "reason": "disabled_by_default",
+      "skipped": true
     }
   }
 }

artifacts/evaluation/latest/iab_content_hard_cases_report.json

@@ -1,66 +1,66 @@
 {
-  "accepted_accuracy": 0.25,
-  "accepted_coverage": 1.0,
-  "accuracy": 0.25,
+  "accepted_accuracy": 0.5,
+  "accepted_coverage": 0.75,
+  "accuracy": 0.375,
   "count": 8,
-  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/iab/hard_cases.jsonl",
-  "fallback_rate": 0.0,
+  "dataset_path": "/content/agentic-intent-classifier/data/iab/hard_cases.jsonl",
+  "fallback_rate": 0.25,
   "head": "iab_content",
-  "macro_f1": 0.1429,
-  "primary_source": "embedding_retrieval",
+  "macro_f1": 0.25,
+  "primary_source": "supervised_classifier",
   "suite": "hard_cases",
   "tier_metrics": {
-    "average_prediction_depth": 2.375,
+    "average_prediction_depth": 2.0,
     "error_buckets": {
-      "exact_match": 2,
-      "parent_safe_stop": 1,
-      "right_tier1_wrong_tier2": 2,
-      "wrong_tier1": 3
+      "exact_match": 3,
+      "right_tier1_wrong_tier2": 1,
+      "wrong_tier1": 4
     },
-    "exact_path_accuracy": 0.25,
+    "exact_path_accuracy": 0.375,
     "parent_safe_accuracy": 0.5,
-    "tier1_accuracy": 0.625,
+    "tier1_accuracy": 0.5,
     "tier2_accuracy": 0.375,
-    "tier3_accuracy": 0.2,
-    "tier4_accuracy": 1.0
+    "tier3_accuracy": 0.4,
+    "tier4_accuracy": 0.0
   },
   "view_metrics": {
+    "classifier": {
+      "average_prediction_depth": 2.0,
+      "error_buckets": {
+        "exact_match": 3,
+        "right_tier1_wrong_tier2": 1,
+        "wrong_tier1": 4
+      },
+      "exact_path_accuracy": 0.375,
+      "parent_safe_accuracy": 0.5,
+      "tier1_accuracy": 0.5,
+      "tier2_accuracy": 0.375,
+      "tier3_accuracy": 0.4,
+      "tier4_accuracy": 0.0
+    },
     "combined_path": {
-      "average_prediction_depth": 2.375,
+      "average_prediction_depth": 2.0,
       "error_buckets": {
-        "exact_match": 1,
-        "parent_safe_stop": 1,
-        "right_tier1_wrong_tier2": 2,
-        "wrong_deep_leaf": 1,
-        "wrong_tier1": 3
+        "exact_match": 3,
+        "right_tier1_wrong_tier2": 1,
+        "wrong_tier1": 4
       },
-      "exact_path_accuracy": 0.125,
+      "exact_path_accuracy": 0.375,
       "fallback_overuse_count": 1,
       "fallback_rate": 0.125,
-      "parent_safe_accuracy": 0.375,
-      "tier1_accuracy": 0.625,
+      "parent_safe_accuracy": 0.5,
+      "tier1_accuracy": 0.5,
       "tier2_accuracy": 0.375,
-      "tier3_accuracy": 0.0,
+      "tier3_accuracy": 0.4,
       "tier4_accuracy": 0.0
     },
     "disagreements": {
-      "retrieval_vs_combined": 0
+      "classifier_vs_combined": 0
     },
-    "embedding_retrieval": {
-      "average_prediction_depth": 2.375,
-      "error_buckets": {
-        "exact_match": 1,
-        "parent_safe_stop": 1,
-        "right_tier1_wrong_tier2": 2,
-        "wrong_deep_leaf": 1,
-        "wrong_tier1": 3
-      },
-      "exact_path_accuracy": 0.125,
-      "parent_safe_accuracy": 0.375,
-      "tier1_accuracy": 0.625,
-      "tier2_accuracy": 0.375,
-      "tier3_accuracy": 0.0,
-      "tier4_accuracy": 0.0
+    "shadow_embedding_retrieval": {
+      "hint": "Set IAB_EVAL_INCLUDE_SHADOW_RETRIEVAL=1 to run shadow embedding retrieval (downloads/loads gte-Qwen2 when index is present).",
+      "reason": "disabled_by_default",
+      "skipped": true
     }
   }
 }

artifacts/evaluation/latest/iab_content_test_report.json

@@ -1,31 +1,47 @@
 {
-  "accepted_accuracy": 0.6527,
-  "accepted_coverage": 1.0,
-  "accuracy": 0.6527,
+  "accepted_accuracy": 0.9242,
+  "accepted_coverage": 0.9973,
+  "accuracy": 0.922,
   "count": 3282,
-  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/iab/test.jsonl",
-  "fallback_rate": 0.0,
+  "dataset_path": "/content/agentic-intent-classifier/data/iab/test.jsonl",
+  "fallback_rate": 0.0027,
   "head": "iab_content",
-  "macro_f1": 0.6922,
-  "primary_source": "embedding_retrieval",
+  "macro_f1": 0.8741,
+  "primary_source": "supervised_classifier",
   "suite": "test",
   "tier_metrics": {
-    "average_prediction_depth": 2.1889,
+    "average_prediction_depth": 2.1789,
     "error_buckets": {
-      "exact_match": 2142,
-      "parent_safe_stop": 115,
-      "right_tier1_wrong_tier2": 674,
-      "wrong_deep_leaf": 236,
-      "wrong_tier1": 115
+      "exact_match": 3026,
+      "parent_safe_stop": 68,
+      "right_tier1_wrong_tier2": 59,
+      "wrong_deep_leaf": 96,
+      "wrong_tier1": 33
     },
-    "exact_path_accuracy": 0.6527,
-    "parent_safe_accuracy": 0.7721,
-    "tier1_accuracy": 0.965,
-    "tier2_accuracy": 0.7587,
-    "tier3_accuracy": 0.8041,
-    "tier4_accuracy": 0.7929
+    "exact_path_accuracy": 0.922,
+    "parent_safe_accuracy": 0.9509,
+    "tier1_accuracy": 0.9899,
+    "tier2_accuracy": 0.9693,
+    "tier3_accuracy": 0.8477,
+    "tier4_accuracy": 0.5143
   },
   "view_metrics": {
+    "classifier": {
+      "average_prediction_depth": 2.1789,
+      "error_buckets": {
+        "exact_match": 2995,
+        "parent_safe_stop": 62,
+        "right_tier1_wrong_tier2": 71,
+        "wrong_deep_leaf": 121,
+        "wrong_tier1": 33
+      },
+      "exact_path_accuracy": 0.9126,
+      "parent_safe_accuracy": 0.9397,
+      "tier1_accuracy": 0.9899,
+      "tier2_accuracy": 0.9651,
+      "tier3_accuracy": 0.8218,
+      "tier4_accuracy": 0.3929
+    },
     "combined_path": {
       "count": 3282,
       "max_combined_rows": 500,
@@ -38,21 +54,10 @@
       "reason": "dataset_too_large_for_combined_view",
       "skipped": true
     },
-    "embedding_retrieval": {
-      "average_prediction_depth": 2.1889,
-      "error_buckets": {
-        "exact_match": 2107,
-        "parent_safe_stop": 109,
-        "right_tier1_wrong_tier2": 680,
-        "wrong_deep_leaf": 271,
-        "wrong_tier1": 115
-      },
-      "exact_path_accuracy": 0.642,
-      "parent_safe_accuracy": 0.7596,
-      "tier1_accuracy": 0.965,
-      "tier2_accuracy": 0.7566,
-      "tier3_accuracy": 0.7679,
-      "tier4_accuracy": 0.6071
+    "shadow_embedding_retrieval": {
+      "hint": "Set IAB_EVAL_INCLUDE_SHADOW_RETRIEVAL=1 to run shadow embedding retrieval (downloads/loads gte-Qwen2 when index is present).",
+      "reason": "disabled_by_default",
+      "skipped": true
     }
   }
 }

artifacts/evaluation/latest/iab_content_train_report.json

@@ -1,67 +1,63 @@
 {
-  "accepted_accuracy": 0.8115,
-  "accepted_coverage": 1.0,
-  "accuracy": 0.8115,
+  "accepted_accuracy": 0.93,
+  "accepted_coverage": 0.9978,
+  "accuracy": 0.9282,
   "count": 13211,
-  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/iab/train.jsonl",
-  "fallback_rate": 0.0,
+  "dataset_path": "/content/agentic-intent-classifier/data/iab/train.jsonl",
+  "fallback_rate": 0.0022,
   "head": "iab_content",
-  "macro_f1": 0.8293,
-  "primary_source": "embedding_retrieval",
+  "macro_f1": 0.8851,
+  "primary_source": "supervised_classifier",
   "suite": "train",
   "tier_metrics": {
-    "average_prediction_depth": 2.2368,
+    "average_prediction_depth": 2.172,
     "error_buckets": {
-      "exact_match": 10721,
-      "parent_safe_stop": 346,
-      "right_tier1_wrong_tier2": 812,
-      "wrong_deep_leaf": 809,
-      "wrong_tier1": 523
+      "exact_match": 12263,
+      "parent_safe_stop": 259,
+      "right_tier1_wrong_tier2": 229,
+      "wrong_deep_leaf": 356,
+      "wrong_tier1": 104
     },
-    "exact_path_accuracy": 0.8115,
-    "parent_safe_accuracy": 0.8753,
-    "tier1_accuracy": 0.9604,
-    "tier2_accuracy": 0.9208,
-    "tier3_accuracy": 0.8788,
-    "tier4_accuracy": 0.8732
+    "exact_path_accuracy": 0.9282,
+    "parent_safe_accuracy": 0.9572,
+    "tier1_accuracy": 0.9921,
+    "tier2_accuracy": 0.9726,
+    "tier3_accuracy": 0.8565,
+    "tier4_accuracy": 0.5518
   },
   "view_metrics": {
-    "combined_path": {
-      "average_prediction_depth": 2.2368,
+    "classifier": {
+      "average_prediction_depth": 2.172,
       "error_buckets": {
-        "exact_match": 10569,
-        "parent_safe_stop": 338,
-        "right_tier1_wrong_tier2": 834,
-        "wrong_deep_leaf": 947,
-        "wrong_tier1": 523
+        "exact_match": 12130,
+        "parent_safe_stop": 238,
+        "right_tier1_wrong_tier2": 277,
+        "wrong_deep_leaf": 462,
+        "wrong_tier1": 104
       },
-      "exact_path_accuracy": 0.8,
-      "fallback_overuse_count": 1123,
-      "fallback_rate": 0.085,
-      "parent_safe_accuracy": 0.8631,
-      "tier1_accuracy": 0.9604,
-      "tier2_accuracy": 0.9189,
-      "tier3_accuracy": 0.843,
-      "tier4_accuracy": 0.6589
+      "exact_path_accuracy": 0.9182,
+      "parent_safe_accuracy": 0.9456,
+      "tier1_accuracy": 0.9921,
+      "tier2_accuracy": 0.9685,
+      "tier3_accuracy": 0.829,
+      "tier4_accuracy": 0.4214
+    },
+    "combined_path": {
+      "count": 13211,
+      "max_combined_rows": 500,
+      "reason": "dataset_too_large_for_combined_view",
+      "skipped": true
     },
     "disagreements": {
-      "retrieval_vs_combined": 0
+      "count": 13211,
+      "max_combined_rows": 500,
+      "reason": "dataset_too_large_for_combined_view",
+      "skipped": true
     },
-    "embedding_retrieval": {
-      "average_prediction_depth": 2.2368,
-      "error_buckets": {
-        "exact_match": 10569,
-        "parent_safe_stop": 338,
-        "right_tier1_wrong_tier2": 834,
-        "wrong_deep_leaf": 947,
-        "wrong_tier1": 523
-      },
-      "exact_path_accuracy": 0.8,
-      "parent_safe_accuracy": 0.8631,
-      "tier1_accuracy": 0.9604,
-      "tier2_accuracy": 0.9189,
-      "tier3_accuracy": 0.843,
-      "tier4_accuracy": 0.6589
+    "shadow_embedding_retrieval": {
+      "hint": "Set IAB_EVAL_INCLUDE_SHADOW_RETRIEVAL=1 to run shadow embedding retrieval (downloads/loads gte-Qwen2 when index is present).",
+      "reason": "disabled_by_default",
+      "skipped": true
     }
   }
 }

artifacts/evaluation/latest/iab_content_val_report.json

@@ -1,67 +1,63 @@
 {
-  "accepted_accuracy": 0.6545,
-  "accepted_coverage": 1.0,
-  "accuracy": 0.6545,
+  "accepted_accuracy": 0.9246,
+  "accepted_coverage": 0.9979,
+  "accuracy": 0.9229,
   "count": 3282,
-  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/iab/val.jsonl",
-  "fallback_rate": 0.0,
+  "dataset_path": "/content/agentic-intent-classifier/data/iab/val.jsonl",
+  "fallback_rate": 0.0021,
   "head": "iab_content",
-  "macro_f1": 0.6957,
-  "primary_source": "embedding_retrieval",
+  "macro_f1": 0.8789,
+  "primary_source": "supervised_classifier",
   "suite": "val",
   "tier_metrics": {
-    "average_prediction_depth": 2.1813,
+    "average_prediction_depth": 2.1789,
     "error_buckets": {
-      "exact_match": 2148,
-      "parent_safe_stop": 105,
-      "right_tier1_wrong_tier2": 684,
-      "wrong_deep_leaf": 234,
-      "wrong_tier1": 111
+      "exact_match": 3029,
+      "parent_safe_stop": 69,
+      "right_tier1_wrong_tier2": 67,
+      "wrong_deep_leaf": 91,
+      "wrong_tier1": 26
     },
-    "exact_path_accuracy": 0.6545,
-    "parent_safe_accuracy": 0.7821,
-    "tier1_accuracy": 0.9662,
-    "tier2_accuracy": 0.7577,
-    "tier3_accuracy": 0.8352,
-    "tier4_accuracy": 0.7214
+    "exact_path_accuracy": 0.9229,
+    "parent_safe_accuracy": 0.9549,
+    "tier1_accuracy": 0.9921,
+    "tier2_accuracy": 0.9686,
+    "tier3_accuracy": 0.8549,
+    "tier4_accuracy": 0.5286
   },
   "view_metrics": {
-    "combined_path": {
-      "average_prediction_depth": 2.1813,
+    "classifier": {
+      "average_prediction_depth": 2.1789,
       "error_buckets": {
-        "exact_match": 2116,
-        "parent_safe_stop": 100,
-        "right_tier1_wrong_tier2": 689,
-        "wrong_deep_leaf": 266,
-        "wrong_tier1": 111
+        "exact_match": 2997,
+        "parent_safe_stop": 64,
+        "right_tier1_wrong_tier2": 79,
+        "wrong_deep_leaf": 116,
+        "wrong_tier1": 26
       },
-      "exact_path_accuracy": 0.6447,
-      "fallback_overuse_count": 413,
-      "fallback_rate": 0.1258,
-      "parent_safe_accuracy": 0.7709,
-      "tier1_accuracy": 0.9662,
-      "tier2_accuracy": 0.756,
-      "tier3_accuracy": 0.799,
-      "tier4_accuracy": 0.55
+      "exact_path_accuracy": 0.9132,
+      "parent_safe_accuracy": 0.9436,
+      "tier1_accuracy": 0.9921,
+      "tier2_accuracy": 0.9644,
+      "tier3_accuracy": 0.829,
+      "tier4_accuracy": 0.4071
+    },
+    "combined_path": {
+      "count": 3282,
+      "max_combined_rows": 500,
+      "reason": "dataset_too_large_for_combined_view",
+      "skipped": true
     },
     "disagreements": {
-      "retrieval_vs_combined": 0
+      "count": 3282,
+      "max_combined_rows": 500,
+      "reason": "dataset_too_large_for_combined_view",
+      "skipped": true
     },
-    "embedding_retrieval": {
-      "average_prediction_depth": 2.1813,
-      "error_buckets": {
-        "exact_match": 2116,
-        "parent_safe_stop": 100,
-        "right_tier1_wrong_tier2": 689,
-        "wrong_deep_leaf": 266,
-        "wrong_tier1": 111
-      },
-      "exact_path_accuracy": 0.6447,
-      "parent_safe_accuracy": 0.7709,
-      "tier1_accuracy": 0.9662,
-      "tier2_accuracy": 0.756,
-      "tier3_accuracy": 0.799,
-      "tier4_accuracy": 0.55
+    "shadow_embedding_retrieval": {
+      "hint": "Set IAB_EVAL_INCLUDE_SHADOW_RETRIEVAL=1 to run shadow embedding retrieval (downloads/loads gte-Qwen2 when index is present).",
+      "reason": "disabled_by_default",
+      "skipped": true
     }
   }
 }

artifacts/evaluation/latest/iab_cross_vertical_quality_target_eval.json

@@ -1,21 +1,21 @@
 {
   "by_status": {
     "must_fix": {
-      "failed": 86,
-      "passed": 4,
+      "failed": 63,
+      "passed": 27,
       "total": 90
     }
   },
-  "cases_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/examples/iab_cross_vertical_mapping_cases.json",
+  "cases_path": "/content/agentic-intent-classifier/examples/iab_cross_vertical_mapping_cases.json",
   "count": 90,
-  "failed": 86,
-  "passed": 4,
+  "failed": 63,
+  "passed": 27,
   "results": [
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Automotive",
-        "model_output.classification.iab_content.tier2.label": "Auto Type"
+        "model_output.classification.iab_content.tier2.label": "Auto Rentals"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -25,7 +25,12 @@
       "id": "auto-buying-easy",
       "mismatches": [
         {
-          "actual": "Auto Type",
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": "Auto Rentals",
           "expected": "Auto Buying and Selling",
           "path": "model_output.classification.iab_content.tier2.label"
         }
@@ -88,7 +93,7 @@
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+        "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -114,7 +119,7 @@
           "path": "model_output.classification.iab_content.tier2.label"
         },
         {
-          "actual": "Software and Applications",
+          "actual": null,
           "expected": "Sales",
           "path": "model_output.classification.iab_content.tier3.label"
         }
@@ -128,8 +133,8 @@
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
-        "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": "Internet"
+        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -150,12 +155,12 @@
           "path": "model_output.classification.iab_content.mapping_mode"
         },
         {
-          "actual": "Computing",
+          "actual": null,
           "expected": "Business",
           "path": "model_output.classification.iab_content.tier2.label"
         },
         {
-          "actual": "Internet",
+          "actual": null,
           "expected": "Sales",
           "path": "model_output.classification.iab_content.tier3.label"
         }
@@ -200,7 +205,7 @@
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
         "model_output.classification.iab_content.tier1.label": "Hobbies & Interests",
-        "model_output.classification.iab_content.tier2.label": "Content Production",
+        "model_output.classification.iab_content.tier2.label": null,
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -222,7 +227,7 @@
           "path": "model_output.classification.iab_content.mapping_mode"
         },
         {
-          "actual": "Content Production",
+          "actual": null,
           "expected": "Business",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -240,8 +245,8 @@
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Sensitive Topics",
-        "model_output.classification.iab_content.tier2.label": "Online Piracy",
+        "model_output.classification.iab_content.tier1.label": "Careers",
+        "model_output.classification.iab_content.tier2.label": null,
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -253,7 +258,7 @@
       "id": "marketing-tools-medium",
       "mismatches": [
         {
-          "actual": "Sensitive Topics",
+          "actual": "Careers",
           "expected": "Business and Finance",
           "path": "model_output.classification.iab_content.tier1.label"
         },
@@ -263,7 +268,7 @@
           "path": "model_output.classification.iab_content.mapping_mode"
         },
         {
-          "actual": "Online Piracy",
+          "actual": null,
           "expected": "Business",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -281,9 +286,9 @@
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Genres",
-        "model_output.classification.iab_content.tier2.label": "Talk Radio",
-        "model_output.classification.iab_content.tier3.label": "Public Radio"
+        "model_output.classification.iab_content.tier1.label": "Personal Finance",
+        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -294,7 +299,7 @@
       "id": "marketing-tools-hard",
       "mismatches": [
         {
-          "actual": "Genres",
+          "actual": "Personal Finance",
           "expected": "Business and Finance",
           "path": "model_output.classification.iab_content.tier1.label"
         },
@@ -304,12 +309,12 @@
           "path": "model_output.classification.iab_content.mapping_mode"
         },
         {
-          "actual": "Talk Radio",
+          "actual": null,
           "expected": "Business",
           "path": "model_output.classification.iab_content.tier2.label"
         },
         {
-          "actual": "Public Radio",
+          "actual": null,
           "expected": "Marketing and Advertising",
           "path": "model_output.classification.iab_content.tier3.label"
         }
@@ -322,8 +327,8 @@
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Careers",
-        "model_output.classification.iab_content.tier2.label": "Job Search",
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": "Computing",
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -335,7 +340,7 @@
       "id": "business-it-easy",
       "mismatches": [
         {
-          "actual": "Careers",
+          "actual": "Technology & Computing",
           "expected": "Business and Finance",
           "path": "model_output.classification.iab_content.tier1.label"
         },
@@ -345,7 +350,7 @@
           "path": "model_output.classification.iab_content.mapping_mode"
         },
         {
-          "actual": "Job Search",
+          "actual": "Computing",
           "expected": "Business",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -363,8 +368,8 @@
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Business and Finance",
-        "model_output.classification.iab_content.tier2.label": "Business",
+        "model_output.classification.iab_content.tier1.label": "Personal Finance",
+        "model_output.classification.iab_content.tier2.label": null,
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -375,11 +380,21 @@
       },
       "id": "business-it-medium",
       "mismatches": [
+        {
+          "actual": "Personal Finance",
+          "expected": "Business and Finance",
+          "path": "model_output.classification.iab_content.tier1.label"
+        },
         {
           "actual": "nearest_equivalent",
           "expected": "exact",
           "path": "model_output.classification.iab_content.mapping_mode"
         },
+        {
+          "actual": null,
+          "expected": "Business",
+          "path": "model_output.classification.iab_content.tier2.label"
+        },
         {
           "actual": null,
           "expected": "Business I.T.",
@@ -393,10 +408,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+        "model_output.classification.iab_content.tier3.label": "Information and Network Security"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -411,18 +426,13 @@
           "expected": "Business and Finance",
           "path": "model_output.classification.iab_content.tier1.label"
         },
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
         {
           "actual": "Computing",
           "expected": "Business",
           "path": "model_output.classification.iab_content.tier2.label"
         },
         {
-          "actual": "Software and Applications",
+          "actual": "Information and Network Security",
           "expected": "Business I.T.",
           "path": "model_output.classification.iab_content.tier3.label"
         }
@@ -434,9 +444,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Sports",
-        "model_output.classification.iab_content.tier2.label": "Table Tennis"
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink",
+        "model_output.classification.iab_content.tier2.label": "Dining Out"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -444,33 +454,17 @@
         "model_output.classification.iab_content.tier2.label": "Dining Out"
       },
       "id": "dining-out-easy",
-      "mismatches": [
-        {
-          "actual": "Sports",
-          "expected": "Food & Drink",
-          "path": "model_output.classification.iab_content.tier1.label"
-        },
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": "Table Tennis",
-          "expected": "Dining Out",
-          "path": "model_output.classification.iab_content.tier2.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical easy IAB mapping case for Food & Drink > Dining Out.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Book a table for six tonight"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Attractions",
-        "model_output.classification.iab_content.tier2.label": null
+        "model_output.classification.iab_content.tier2.label": "Bars & Restaurants"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -485,12 +479,7 @@
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
+          "actual": "Bars & Restaurants",
           "expected": "Dining Out",
           "path": "model_output.classification.iab_content.tier2.label"
         }
@@ -502,9 +491,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Food & Drink",
-        "model_output.classification.iab_content.tier2.label": null
+        "model_output.classification.iab_content.tier2.label": "Dining Out"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -512,28 +501,17 @@
         "model_output.classification.iab_content.tier2.label": "Dining Out"
       },
       "id": "dining-out-hard",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
-          "expected": "Dining Out",
-          "path": "model_output.classification.iab_content.tier2.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical hard IAB mapping case for Food & Drink > Dining Out.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Need a place to eat tonight where I can make a reservation online"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
-        "model_output.classification.iab_content.tier2.label": null
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink",
+        "model_output.classification.iab_content.tier2.label": "Alcoholic Beverages"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -541,33 +519,17 @@
         "model_output.classification.iab_content.tier2.label": "Alcoholic Beverages"
       },
       "id": "alcoholic-beverages-easy",
-      "mismatches": [
-        {
-          "actual": "Style & Fashion",
-          "expected": "Food & Drink",
-          "path": "model_output.classification.iab_content.tier1.label"
-        },
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
-          "expected": "Alcoholic Beverages",
-          "path": "model_output.classification.iab_content.tier2.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical easy IAB mapping case for Food & Drink > Alcoholic Beverages.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Which whiskey cocktail should I order?"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Food & Drink",
-        "model_output.classification.iab_content.tier2.label": null
+        "model_output.classification.iab_content.tier2.label": "Alcoholic Beverages"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -575,28 +537,17 @@
         "model_output.classification.iab_content.tier2.label": "Alcoholic Beverages"
       },
       "id": "alcoholic-beverages-medium",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
-          "expected": "Alcoholic Beverages",
-          "path": "model_output.classification.iab_content.tier2.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical medium IAB mapping case for Food & Drink > Alcoholic Beverages.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Best vodka drinks for beginners"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Food & Drink",
-        "model_output.classification.iab_content.tier2.label": null
+        "model_output.classification.iab_content.tier2.label": "Alcoholic Beverages"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -604,27 +555,16 @@
         "model_output.classification.iab_content.tier2.label": "Alcoholic Beverages"
       },
       "id": "alcoholic-beverages-hard",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
-          "expected": "Alcoholic Beverages",
-          "path": "model_output.classification.iab_content.tier2.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical hard IAB mapping case for Food & Drink > Alcoholic Beverages.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Want a spirit-forward drink recommendation, not a restaurant suggestion"
     },
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Sensitive Topics",
+        "model_output.classification.iab_content.tier1.label": "Sports",
         "model_output.classification.iab_content.tier2.label": null
       },
       "expected": {
@@ -635,7 +575,7 @@
       "id": "artificial-intelligence-easy",
       "mismatches": [
         {
-          "actual": "Sensitive Topics",
+          "actual": "Sports",
           "expected": "Technology & Computing",
           "path": "model_output.classification.iab_content.tier1.label"
         },
@@ -657,9 +597,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Sensitive Topics",
-        "model_output.classification.iab_content.tier2.label": null
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Education",
+        "model_output.classification.iab_content.tier2.label": "Language Learning"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -669,17 +609,12 @@
       "id": "artificial-intelligence-medium",
       "mismatches": [
         {
-          "actual": "Sensitive Topics",
+          "actual": "Education",
           "expected": "Technology & Computing",
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
+          "actual": "Language Learning",
           "expected": "Artificial Intelligence",
           "path": "model_output.classification.iab_content.tier2.label"
         }
@@ -691,7 +626,7 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Education",
         "model_output.classification.iab_content.tier2.label": "Language Learning"
       },
@@ -707,11 +642,6 @@
           "expected": "Technology & Computing",
           "path": "model_output.classification.iab_content.tier1.label"
         },
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
         {
           "actual": "Language Learning",
           "expected": "Artificial Intelligence",
@@ -728,7 +658,7 @@
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+        "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -742,6 +672,11 @@
           "actual": "nearest_equivalent",
           "expected": "exact",
           "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": null,
+          "expected": "Software and Applications",
+          "path": "model_output.classification.iab_content.tier3.label"
         }
       ],
       "notes": "Cross-vertical easy IAB mapping case for Technology & Computing > Computing > Computer Software and Applications.",
@@ -754,7 +689,7 @@
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+        "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -768,6 +703,11 @@
           "actual": "nearest_equivalent",
           "expected": "exact",
           "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": null,
+          "expected": "Software and Applications",
+          "path": "model_output.classification.iab_content.tier3.label"
         }
       ],
       "notes": "Cross-vertical medium IAB mapping case for Technology & Computing > Computing > Computer Software and Applications.",
@@ -778,9 +718,9 @@
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
-        "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+        "model_output.classification.iab_content.tier1.label": "Business and Finance",
+        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -790,10 +730,25 @@
       },
       "id": "software-apps-hard",
       "mismatches": [
+        {
+          "actual": "Business and Finance",
+          "expected": "Technology & Computing",
+          "path": "model_output.classification.iab_content.tier1.label"
+        },
         {
           "actual": "nearest_equivalent",
           "expected": "exact",
           "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": null,
+          "expected": "Computing",
+          "path": "model_output.classification.iab_content.tier2.label"
+        },
+        {
+          "actual": null,
+          "expected": "Software and Applications",
+          "path": "model_output.classification.iab_content.tier3.label"
         }
       ],
       "notes": "Cross-vertical hard IAB mapping case for Technology & Computing > Computing > Computer Software and Applications.",
@@ -803,7 +758,7 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Careers",
         "model_output.classification.iab_content.tier2.label": "Remote Working",
         "model_output.classification.iab_content.tier3.label": null,
@@ -823,11 +778,6 @@
           "expected": "Technology & Computing",
           "path": "model_output.classification.iab_content.tier1.label"
         },
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
         {
           "actual": "Remote Working",
           "expected": "Computing",
@@ -854,7 +804,7 @@
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": "Internet",
+        "model_output.classification.iab_content.tier3.label": null,
         "model_output.classification.iab_content.tier4.label": null
       },
       "expected": {
@@ -872,7 +822,7 @@
           "path": "model_output.classification.iab_content.mapping_mode"
         },
         {
-          "actual": "Internet",
+          "actual": null,
           "expected": "Software and Applications",
           "path": "model_output.classification.iab_content.tier3.label"
         },
@@ -891,7 +841,7 @@
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
         "model_output.classification.iab_content.tier1.label": "Careers",
-        "model_output.classification.iab_content.tier2.label": "Remote Working",
+        "model_output.classification.iab_content.tier2.label": null,
         "model_output.classification.iab_content.tier3.label": null,
         "model_output.classification.iab_content.tier4.label": null
       },
@@ -915,7 +865,7 @@
           "path": "model_output.classification.iab_content.mapping_mode"
         },
         {
-          "actual": "Remote Working",
+          "actual": null,
           "expected": "Computing",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -978,7 +928,7 @@
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": "Internet",
+        "model_output.classification.iab_content.tier3.label": null,
         "model_output.classification.iab_content.tier4.label": null
       },
       "expected": {
@@ -995,6 +945,11 @@
           "expected": "exact",
           "path": "model_output.classification.iab_content.mapping_mode"
         },
+        {
+          "actual": null,
+          "expected": "Internet",
+          "path": "model_output.classification.iab_content.tier3.label"
+        },
         {
           "actual": null,
           "expected": "Web Hosting",
@@ -1011,7 +966,7 @@
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": "Internet",
+        "model_output.classification.iab_content.tier3.label": null,
         "model_output.classification.iab_content.tier4.label": null
       },
       "expected": {
@@ -1030,9 +985,14 @@
         },
         {
           "actual": null,
-          "expected": "Web Hosting",
-          "path": "model_output.classification.iab_content.tier4.label"
-        }
+          "expected": "Internet",
+          "path": "model_output.classification.iab_content.tier3.label"
+        },
+        {
+          "actual": null,
+          "expected": "Web Hosting",
+          "path": "model_output.classification.iab_content.tier4.label"
+        }
       ],
       "notes": "Cross-vertical hard IAB mapping case for Technology & Computing > Computing > Internet > Web Hosting.",
       "pass": false,
@@ -1043,8 +1003,8 @@
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
-        "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": "Laptops"
+        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1058,6 +1018,16 @@
           "actual": "nearest_equivalent",
           "expected": "exact",
           "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": null,
+          "expected": "Computing",
+          "path": "model_output.classification.iab_content.tier2.label"
+        },
+        {
+          "actual": null,
+          "expected": "Laptops",
+          "path": "model_output.classification.iab_content.tier3.label"
         }
       ],
       "notes": "Cross-vertical easy IAB mapping case for Technology & Computing > Computing > Laptops.",
@@ -1067,7 +1037,7 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
         "model_output.classification.iab_content.tier3.label": "Laptops"
@@ -1079,15 +1049,9 @@
         "model_output.classification.iab_content.tier3.label": "Laptops"
       },
       "id": "laptops-medium",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical medium IAB mapping case for Technology & Computing > Computing > Laptops.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Best laptop for work and study under 1200"
     },
@@ -1096,7 +1060,7 @@
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": "Laptops"
+        "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1110,6 +1074,11 @@
           "actual": "nearest_equivalent",
           "expected": "exact",
           "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": null,
+          "expected": "Laptops",
+          "path": "model_output.classification.iab_content.tier3.label"
         }
       ],
       "notes": "Cross-vertical hard IAB mapping case for Technology & Computing > Computing > Laptops.",
@@ -1153,7 +1122,7 @@
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+        "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1169,7 +1138,7 @@
           "path": "model_output.classification.iab_content.mapping_mode"
         },
         {
-          "actual": "Software and Applications",
+          "actual": null,
           "expected": "Desktops",
           "path": "model_output.classification.iab_content.tier3.label"
         }
@@ -1181,10 +1150,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier3.label": "Desktops"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1193,29 +1162,18 @@
         "model_output.classification.iab_content.tier3.label": "Desktops"
       },
       "id": "desktops-hard",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
-          "expected": "Desktops",
-          "path": "model_output.classification.iab_content.tier3.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical hard IAB mapping case for Technology & Computing > Computing > Desktops.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Need a desktop PC with strong performance for creative work"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Consumer Electronics",
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier3.label": "Wearable Technology"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1226,12 +1184,7 @@
       "id": "smartphones-easy",
       "mismatches": [
         {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
+          "actual": "Wearable Technology",
           "expected": "Smartphones",
           "path": "model_output.classification.iab_content.tier3.label"
         }
@@ -1243,10 +1196,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Consumer Electronics",
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier3.label": "Wearable Technology"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1257,12 +1210,7 @@
       "id": "smartphones-medium",
       "mismatches": [
         {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
+          "actual": "Wearable Technology",
           "expected": "Smartphones",
           "path": "model_output.classification.iab_content.tier3.label"
         }
@@ -1274,10 +1222,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Consumer Electronics",
-        "model_output.classification.iab_content.tier3.label": "Smartphones"
+        "model_output.classification.iab_content.tier3.label": "Wearable Technology"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1288,9 +1236,9 @@
       "id": "smartphones-hard",
       "mismatches": [
         {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
+          "actual": "Wearable Technology",
+          "expected": "Smartphones",
+          "path": "model_output.classification.iab_content.tier3.label"
         }
       ],
       "notes": "Cross-vertical hard IAB mapping case for Technology & Computing > Consumer Electronics > Smartphones.",
@@ -1300,7 +1248,7 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Style & Fashion"
       },
       "expected": {
@@ -1308,15 +1256,21 @@
         "model_output.classification.iab_content.tier1.label": "Style & Fashion"
       },
       "id": "style-fashion-parent-easy",
-      "mismatches": [],
+      "mismatches": [
+        {
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        }
+      ],
       "notes": "Cross-vertical easy IAB mapping case for Style & Fashion.",
-      "pass": true,
+      "pass": false,
       "status": "must_fix",
       "text": "Best shoes under 100 dollars"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Style & Fashion"
       },
       "expected": {
@@ -1324,15 +1278,21 @@
         "model_output.classification.iab_content.tier1.label": "Style & Fashion"
       },
       "id": "style-fashion-parent-medium",
-      "mismatches": [],
+      "mismatches": [
+        {
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        }
+      ],
       "notes": "Cross-vertical medium IAB mapping case for Style & Fashion.",
-      "pass": true,
+      "pass": false,
       "status": "must_fix",
       "text": "Affordable fashion accessories for everyday wear"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Style & Fashion"
       },
       "expected": {
@@ -1340,18 +1300,24 @@
         "model_output.classification.iab_content.tier1.label": "Style & Fashion"
       },
       "id": "style-fashion-parent-hard",
-      "mismatches": [],
+      "mismatches": [
+        {
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        }
+      ],
       "notes": "Cross-vertical hard IAB mapping case for Style & Fashion.",
-      "pass": true,
+      "pass": false,
       "status": "must_fix",
       "text": "Need style recommendations for clothing and footwear without a specific brand in mind"
     },
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
-        "model_output.classification.iab_content.tier2.label": "Women's Fashion",
-        "model_output.classification.iab_content.tier3.label": "Women's Shoes and Footwear"
+        "model_output.classification.iab_content.tier1.label": "Sports",
+        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1361,10 +1327,25 @@
       },
       "id": "womens-shoes-easy",
       "mismatches": [
+        {
+          "actual": "Sports",
+          "expected": "Style & Fashion",
+          "path": "model_output.classification.iab_content.tier1.label"
+        },
         {
           "actual": "nearest_equivalent",
           "expected": "exact",
           "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": null,
+          "expected": "Women's Fashion",
+          "path": "model_output.classification.iab_content.tier2.label"
+        },
+        {
+          "actual": null,
+          "expected": "Women's Shoes and Footwear",
+          "path": "model_output.classification.iab_content.tier3.label"
         }
       ],
       "notes": "Cross-vertical easy IAB mapping case for Style & Fashion > Women's Fashion > Women's Shoes and Footwear.",
@@ -1374,10 +1355,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
-        "model_output.classification.iab_content.tier2.label": "Women's Fashion",
-        "model_output.classification.iab_content.tier3.label": "Women's Shoes and Footwear"
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Sports",
+        "model_output.classification.iab_content.tier2.label": "Walking",
+        "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1388,9 +1369,19 @@
       "id": "womens-shoes-medium",
       "mismatches": [
         {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
+          "actual": "Sports",
+          "expected": "Style & Fashion",
+          "path": "model_output.classification.iab_content.tier1.label"
+        },
+        {
+          "actual": "Walking",
+          "expected": "Women's Fashion",
+          "path": "model_output.classification.iab_content.tier2.label"
+        },
+        {
+          "actual": null,
+          "expected": "Women's Shoes and Footwear",
+          "path": "model_output.classification.iab_content.tier3.label"
         }
       ],
       "notes": "Cross-vertical medium IAB mapping case for Style & Fashion > Women's Fashion > Women's Shoes and Footwear.",
@@ -1400,7 +1391,7 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Style & Fashion",
         "model_output.classification.iab_content.tier2.label": "Women's Fashion",
         "model_output.classification.iab_content.tier3.label": "Women's Shoes and Footwear"
@@ -1412,15 +1403,9 @@
         "model_output.classification.iab_content.tier3.label": "Women's Shoes and Footwear"
       },
       "id": "womens-shoes-hard",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical hard IAB mapping case for Style & Fashion > Women's Fashion > Women's Shoes and Footwear.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Need women's footwear for commuting that looks polished but feels comfortable"
     },
@@ -1428,8 +1413,8 @@
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
         "model_output.classification.iab_content.tier1.label": "Style & Fashion",
-        "model_output.classification.iab_content.tier2.label": "Men's Fashion",
-        "model_output.classification.iab_content.tier3.label": "Men's Shoes and Footwear"
+        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1443,6 +1428,16 @@
           "actual": "nearest_equivalent",
           "expected": "exact",
           "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": null,
+          "expected": "Men's Fashion",
+          "path": "model_output.classification.iab_content.tier2.label"
+        },
+        {
+          "actual": null,
+          "expected": "Men's Shoes and Footwear",
+          "path": "model_output.classification.iab_content.tier3.label"
         }
       ],
       "notes": "Cross-vertical easy IAB mapping case for Style & Fashion > Men's Fashion > Men's Shoes and Footwear.",
@@ -1452,7 +1447,7 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Style & Fashion",
         "model_output.classification.iab_content.tier2.label": "Men's Fashion",
         "model_output.classification.iab_content.tier3.label": "Men's Shoes and Footwear"
@@ -1464,15 +1459,9 @@
         "model_output.classification.iab_content.tier3.label": "Men's Shoes and Footwear"
       },
       "id": "mens-shoes-medium",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical medium IAB mapping case for Style & Fashion > Men's Fashion > Men's Shoes and Footwear.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Good men's dress shoes for office use"
     },
@@ -1480,8 +1469,8 @@
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
         "model_output.classification.iab_content.tier1.label": "Style & Fashion",
-        "model_output.classification.iab_content.tier2.label": "Men's Fashion",
-        "model_output.classification.iab_content.tier3.label": "Men's Shoes and Footwear"
+        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1495,6 +1484,16 @@
           "actual": "nearest_equivalent",
           "expected": "exact",
           "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": null,
+          "expected": "Men's Fashion",
+          "path": "model_output.classification.iab_content.tier2.label"
+        },
+        {
+          "actual": null,
+          "expected": "Men's Shoes and Footwear",
+          "path": "model_output.classification.iab_content.tier3.label"
         }
       ],
       "notes": "Cross-vertical hard IAB mapping case for Style & Fashion > Men's Fashion > Men's Shoes and Footwear.",
@@ -1504,10 +1503,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Attractions",
-        "model_output.classification.iab_content.tier2.label": "Nightclubs",
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Travel",
+        "model_output.classification.iab_content.tier2.label": "Travel Type",
+        "model_output.classification.iab_content.tier3.label": "Hotels and Motels"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1516,39 +1515,18 @@
         "model_output.classification.iab_content.tier3.label": "Hotels and Motels"
       },
       "id": "hotels-easy",
-      "mismatches": [
-        {
-          "actual": "Attractions",
-          "expected": "Travel",
-          "path": "model_output.classification.iab_content.tier1.label"
-        },
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": "Nightclubs",
-          "expected": "Travel Type",
-          "path": "model_output.classification.iab_content.tier2.label"
-        },
-        {
-          "actual": null,
-          "expected": "Hotels and Motels",
-          "path": "model_output.classification.iab_content.tier3.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical easy IAB mapping case for Travel > Travel Type > Hotels and Motels.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Need a hotel in Chicago for two nights"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Travel",
         "model_output.classification.iab_content.tier2.label": "Travel Type",
-        "model_output.classification.iab_content.tier3.label": null
+        "model_output.classification.iab_content.tier3.label": "Hotels and Motels"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1557,26 +1535,15 @@
         "model_output.classification.iab_content.tier3.label": "Hotels and Motels"
       },
       "id": "hotels-medium",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
-          "expected": "Hotels and Motels",
-          "path": "model_output.classification.iab_content.tier3.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical medium IAB mapping case for Travel > Travel Type > Hotels and Motels.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Best hotels near Times Square for a weekend trip"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Travel",
         "model_output.classification.iab_content.tier2.label": null,
         "model_output.classification.iab_content.tier3.label": null
@@ -1589,11 +1556,6 @@
       },
       "id": "hotels-hard",
       "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
         {
           "actual": null,
           "expected": "Travel Type",
@@ -1612,7 +1574,7 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Real Estate",
         "model_output.classification.iab_content.tier2.label": "Apartments"
       },
@@ -1623,11 +1585,6 @@
       },
       "id": "real-estate-rentals-easy",
       "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
         {
           "actual": "Apartments",
           "expected": "Real Estate Renting and Leasing",
@@ -1641,7 +1598,7 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Real Estate",
         "model_output.classification.iab_content.tier2.label": "Apartments"
       },
@@ -1652,6 +1609,11 @@
       },
       "id": "real-estate-rentals-medium",
       "mismatches": [
+        {
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        },
         {
           "actual": "Apartments",
           "expected": "Real Estate Renting and Leasing",
@@ -1665,9 +1627,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Real Estate",
-        "model_output.classification.iab_content.tier2.label": null
+        "model_output.classification.iab_content.tier2.label": "Real Estate Renting and Leasing"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1675,20 +1637,9 @@
         "model_output.classification.iab_content.tier2.label": "Real Estate Renting and Leasing"
       },
       "id": "real-estate-rentals-hard",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
-          "expected": "Real Estate Renting and Leasing",
-          "path": "model_output.classification.iab_content.tier2.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical hard IAB mapping case for Real Estate > Real Estate Renting and Leasing.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Need rental listings for a short move, not home-buying advice"
     },
@@ -1696,8 +1647,8 @@
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
         "model_output.classification.iab_content.tier1.label": "Healthy Living",
-        "model_output.classification.iab_content.tier2.label": "Fitness and Exercise",
-        "model_output.classification.iab_content.tier3.label": "Running and Jogging"
+        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1711,6 +1662,16 @@
           "actual": "nearest_equivalent",
           "expected": "exact",
           "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": null,
+          "expected": "Fitness and Exercise",
+          "path": "model_output.classification.iab_content.tier2.label"
+        },
+        {
+          "actual": null,
+          "expected": "Running and Jogging",
+          "path": "model_output.classification.iab_content.tier3.label"
         }
       ],
       "notes": "Cross-vertical easy IAB mapping case for Healthy Living > Fitness and Exercise > Running and Jogging.",
@@ -1720,10 +1681,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Healthy Living",
-        "model_output.classification.iab_content.tier2.label": "Fitness and Exercise",
-        "model_output.classification.iab_content.tier3.label": "Running and Jogging"
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Sports",
+        "model_output.classification.iab_content.tier2.label": "Walking",
+        "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1734,9 +1695,19 @@
       "id": "running-and-jogging-medium",
       "mismatches": [
         {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
+          "actual": "Sports",
+          "expected": "Healthy Living",
+          "path": "model_output.classification.iab_content.tier1.label"
+        },
+        {
+          "actual": "Walking",
+          "expected": "Fitness and Exercise",
+          "path": "model_output.classification.iab_content.tier2.label"
+        },
+        {
+          "actual": null,
+          "expected": "Running and Jogging",
+          "path": "model_output.classification.iab_content.tier3.label"
         }
       ],
       "notes": "Cross-vertical medium IAB mapping case for Healthy Living > Fitness and Exercise > Running and Jogging.",
@@ -1746,10 +1717,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Healthy Living",
         "model_output.classification.iab_content.tier2.label": "Fitness and Exercise",
-        "model_output.classification.iab_content.tier3.label": "Running and Jogging"
+        "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1760,9 +1731,9 @@
       "id": "running-and-jogging-hard",
       "mismatches": [
         {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
+          "actual": null,
+          "expected": "Running and Jogging",
+          "path": "model_output.classification.iab_content.tier3.label"
         }
       ],
       "notes": "Cross-vertical hard IAB mapping case for Healthy Living > Fitness and Exercise > Running and Jogging.",
@@ -1772,9 +1743,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Sports",
-        "model_output.classification.iab_content.tier2.label": "Australian Rules Football"
+        "model_output.classification.iab_content.tier2.label": "Soccer"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1782,28 +1753,17 @@
         "model_output.classification.iab_content.tier2.label": "Soccer"
       },
       "id": "soccer-easy",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": "Australian Rules Football",
-          "expected": "Soccer",
-          "path": "model_output.classification.iab_content.tier2.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical easy IAB mapping case for Sports > Soccer.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "How do offside rules work in soccer?"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Sports",
-        "model_output.classification.iab_content.tier2.label": null
+        "model_output.classification.iab_content.tier2.label": "Soccer"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1811,28 +1771,17 @@
         "model_output.classification.iab_content.tier2.label": "Soccer"
       },
       "id": "soccer-medium",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
-          "expected": "Soccer",
-          "path": "model_output.classification.iab_content.tier2.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical medium IAB mapping case for Sports > Soccer.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Best soccer drills for beginner players"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Sports",
-        "model_output.classification.iab_content.tier2.label": "Fantasy Sports"
+        "model_output.classification.iab_content.tier2.label": "Soccer"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1840,26 +1789,15 @@
         "model_output.classification.iab_content.tier2.label": "Soccer"
       },
       "id": "soccer-hard",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": "Fantasy Sports",
-          "expected": "Soccer",
-          "path": "model_output.classification.iab_content.tier2.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical hard IAB mapping case for Sports > Soccer.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Need help understanding football tactics for the Premier League, not fantasy sports"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Genres",
         "model_output.classification.iab_content.tier2.label": "Fantasy"
       },
@@ -1875,11 +1813,6 @@
           "expected": "Books and Literature",
           "path": "model_output.classification.iab_content.tier1.label"
         },
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
         {
           "actual": "Fantasy",
           "expected": "Fiction",
@@ -1893,7 +1826,7 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Books and Literature",
         "model_output.classification.iab_content.tier2.label": "Fiction"
       },
@@ -1903,17 +1836,23 @@
         "model_output.classification.iab_content.tier2.label": "Fiction"
       },
       "id": "fiction-medium",
-      "mismatches": [],
+      "mismatches": [
+        {
+          "actual": "exact",
+          "expected": "nearest_equivalent",
+          "path": "model_output.classification.iab_content.mapping_mode"
+        }
+      ],
       "notes": "Cross-vertical medium IAB mapping case for Books and Literature > Fiction.",
-      "pass": true,
+      "pass": false,
       "status": "must_fix",
       "text": "Best fiction books for a long flight"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Books and Literature",
-        "model_output.classification.iab_content.tier2.label": "Comics and Graphic Novels"
+        "model_output.classification.iab_content.tier2.label": "Fiction"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1921,26 +1860,15 @@
         "model_output.classification.iab_content.tier2.label": "Fiction"
       },
       "id": "fiction-hard",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": "Comics and Graphic Novels",
-          "expected": "Fiction",
-          "path": "model_output.classification.iab_content.tier2.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical hard IAB mapping case for Books and Literature > Fiction.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Looking for a character-driven novel, not comics or poetry"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Home & Garden",
         "model_output.classification.iab_content.tier2.label": "Remodeling & Construction"
       },
@@ -1951,11 +1879,6 @@
       },
       "id": "home-improvement-easy",
       "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
         {
           "actual": "Remodeling & Construction",
           "expected": "Home Improvement",
@@ -1970,8 +1893,8 @@
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Style & Fashion",
-        "model_output.classification.iab_content.tier2.label": "Personal Care"
+        "model_output.classification.iab_content.tier1.label": "Home & Garden",
+        "model_output.classification.iab_content.tier2.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -1980,18 +1903,13 @@
       },
       "id": "home-improvement-medium",
       "mismatches": [
-        {
-          "actual": "Style & Fashion",
-          "expected": "Home & Garden",
-          "path": "model_output.classification.iab_content.tier1.label"
-        },
         {
           "actual": "nearest_equivalent",
           "expected": "exact",
           "path": "model_output.classification.iab_content.mapping_mode"
         },
         {
-          "actual": "Personal Care",
+          "actual": null,
           "expected": "Home Improvement",
           "path": "model_output.classification.iab_content.tier2.label"
         }
@@ -2003,9 +1921,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Home & Garden",
-        "model_output.classification.iab_content.tier2.label": "Interior Decorating"
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Real Estate",
+        "model_output.classification.iab_content.tier2.label": "Houses"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -2015,12 +1933,12 @@
       "id": "home-improvement-hard",
       "mismatches": [
         {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
+          "actual": "Real Estate",
+          "expected": "Home & Garden",
+          "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": "Interior Decorating",
+          "actual": "Houses",
           "expected": "Home Improvement",
           "path": "model_output.classification.iab_content.tier2.label"
         }
@@ -2033,8 +1951,8 @@
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Education",
-        "model_output.classification.iab_content.tier2.label": "Online Education"
+        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
+        "model_output.classification.iab_content.tier2.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -2043,10 +1961,20 @@
       },
       "id": "online-education-easy",
       "mismatches": [
+        {
+          "actual": "Technology & Computing",
+          "expected": "Education",
+          "path": "model_output.classification.iab_content.tier1.label"
+        },
         {
           "actual": "nearest_equivalent",
           "expected": "exact",
           "path": "model_output.classification.iab_content.mapping_mode"
+        },
+        {
+          "actual": null,
+          "expected": "Online Education",
+          "path": "model_output.classification.iab_content.tier2.label"
         }
       ],
       "notes": "Cross-vertical easy IAB mapping case for Education > Online Education.",
@@ -2056,7 +1984,7 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Careers",
         "model_output.classification.iab_content.tier2.label": "Remote Working"
       },
@@ -2072,11 +2000,6 @@
           "expected": "Education",
           "path": "model_output.classification.iab_content.tier1.label"
         },
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
         {
           "actual": "Remote Working",
           "expected": "Online Education",
@@ -2091,8 +2014,8 @@
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
-        "model_output.classification.iab_content.tier2.label": "Computing"
+        "model_output.classification.iab_content.tier1.label": "Healthy Living",
+        "model_output.classification.iab_content.tier2.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -2102,7 +2025,7 @@
       "id": "online-education-hard",
       "mismatches": [
         {
-          "actual": "Technology & Computing",
+          "actual": "Healthy Living",
           "expected": "Education",
           "path": "model_output.classification.iab_content.tier1.label"
         },
@@ -2112,7 +2035,7 @@
           "path": "model_output.classification.iab_content.mapping_mode"
         },
         {
-          "actual": "Computing",
+          "actual": null,
           "expected": "Online Education",
           "path": "model_output.classification.iab_content.tier2.label"
         }
@@ -2124,7 +2047,7 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Education",
         "model_output.classification.iab_content.tier2.label": "College Education",
         "model_output.classification.iab_content.tier3.label": "Postgraduate Education"
@@ -2136,21 +2059,15 @@
         "model_output.classification.iab_content.tier3.label": "Postgraduate Education"
       },
       "id": "postgraduate-education-easy",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical easy IAB mapping case for Education > College Education > Postgraduate Education.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "best universities to study masters"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Education",
         "model_output.classification.iab_content.tier2.label": "College Education",
         "model_output.classification.iab_content.tier3.label": "Postgraduate Education"
@@ -2162,21 +2079,15 @@
         "model_output.classification.iab_content.tier3.label": "Postgraduate Education"
       },
       "id": "postgraduate-education-medium",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical medium IAB mapping case for Education > College Education > Postgraduate Education.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "which graduate schools have strong data science programs"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Education",
         "model_output.classification.iab_content.tier2.label": "College Education",
         "model_output.classification.iab_content.tier3.label": "Postgraduate Education"
@@ -2188,15 +2099,9 @@
         "model_output.classification.iab_content.tier3.label": "Postgraduate Education"
       },
       "id": "postgraduate-education-hard",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical hard IAB mapping case for Education > College Education > Postgraduate Education.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "need postgraduate options for a master's degree, not short online courses"
     },
@@ -2224,7 +2129,7 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Medical Health"
       },
       "expected": {
@@ -2232,21 +2137,15 @@
         "model_output.classification.iab_content.tier1.label": "Medical Health"
       },
       "id": "medical-health-medium",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical medium IAB mapping case for Medical Health.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "when should i see a doctor for persistent knee pain"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Medical Health"
       },
       "expected": {
@@ -2254,21 +2153,15 @@
         "model_output.classification.iab_content.tier1.label": "Medical Health"
       },
       "id": "medical-health-hard",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical hard IAB mapping case for Medical Health.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "need medical advice about symptoms, not wellness or fitness tips"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Careers",
         "model_output.classification.iab_content.tier2.label": "Remote Working"
       },
@@ -2279,11 +2172,6 @@
       },
       "id": "careers-job-search-easy",
       "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
         {
           "actual": "Remote Working",
           "expected": "Job Search",
@@ -2299,7 +2187,7 @@
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
         "model_output.classification.iab_content.tier1.label": "Business and Finance",
-        "model_output.classification.iab_content.tier2.label": "Industries"
+        "model_output.classification.iab_content.tier2.label": "Business"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -2319,7 +2207,7 @@
           "path": "model_output.classification.iab_content.mapping_mode"
         },
         {
-          "actual": "Industries",
+          "actual": "Business",
           "expected": "Job Search",
           "path": "model_output.classification.iab_content.tier2.label"
         }
@@ -2332,8 +2220,8 @@
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Business and Finance",
-        "model_output.classification.iab_content.tier2.label": "Industries"
+        "model_output.classification.iab_content.tier1.label": "Genres",
+        "model_output.classification.iab_content.tier2.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -2343,7 +2231,7 @@
       "id": "careers-job-search-hard",
       "mismatches": [
         {
-          "actual": "Business and Finance",
+          "actual": "Genres",
           "expected": "Careers",
           "path": "model_output.classification.iab_content.tier1.label"
         },
@@ -2353,7 +2241,7 @@
           "path": "model_output.classification.iab_content.mapping_mode"
         },
         {
-          "actual": "Industries",
+          "actual": null,
           "expected": "Job Search",
           "path": "model_output.classification.iab_content.tier2.label"
         }
@@ -2366,7 +2254,7 @@
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Personal Finance",
+        "model_output.classification.iab_content.tier1.label": "Personal Celebrations & Life Events",
         "model_output.classification.iab_content.tier2.label": null
       },
       "expected": {
@@ -2376,6 +2264,11 @@
       },
       "id": "personal-finance-easy",
       "mismatches": [
+        {
+          "actual": "Personal Celebrations & Life Events",
+          "expected": "Personal Finance",
+          "path": "model_output.classification.iab_content.tier1.label"
+        },
         {
           "actual": "nearest_equivalent",
           "expected": "exact",
@@ -2394,7 +2287,7 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Personal Finance",
         "model_output.classification.iab_content.tier2.label": null
       },
@@ -2405,11 +2298,6 @@
       },
       "id": "personal-finance-medium",
       "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
         {
           "actual": null,
           "expected": "Financial Planning",
@@ -2423,7 +2311,7 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Personal Finance",
         "model_output.classification.iab_content.tier2.label": "Retirement Planning"
       },
@@ -2434,11 +2322,6 @@
       },
       "id": "personal-finance-hard",
       "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
         {
           "actual": "Retirement Planning",
           "expected": "Financial Planning",
@@ -2452,7 +2335,7 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Family and Relationships",
         "model_output.classification.iab_content.tier2.label": "Parenting"
       },
@@ -2462,23 +2345,17 @@
         "model_output.classification.iab_content.tier2.label": "Parenting"
       },
       "id": "parenting-easy",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical easy IAB mapping case for Family and Relationships > Parenting.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "tips for parenting a toddler"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Education",
-        "model_output.classification.iab_content.tier2.label": "Online Education"
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Hobbies & Interests",
+        "model_output.classification.iab_content.tier2.label": "Content Production"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -2488,17 +2365,12 @@
       "id": "parenting-medium",
       "mismatches": [
         {
-          "actual": "Education",
+          "actual": "Hobbies & Interests",
           "expected": "Family and Relationships",
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": "Online Education",
+          "actual": "Content Production",
           "expected": "Parenting",
           "path": "model_output.classification.iab_content.tier2.label"
         }
@@ -2510,7 +2382,7 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Family and Relationships",
         "model_output.classification.iab_content.tier2.label": "Parenting"
       },
@@ -2520,21 +2392,15 @@
         "model_output.classification.iab_content.tier2.label": "Parenting"
       },
       "id": "parenting-hard",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical hard IAB mapping case for Family and Relationships > Parenting.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "need parenting advice for a child starting preschool"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Home & Garden",
         "model_output.classification.iab_content.tier2.label": "Gardening"
       },
@@ -2544,22 +2410,16 @@
         "model_output.classification.iab_content.tier2.label": "Gardening"
       },
       "id": "gardening-easy",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical easy IAB mapping case for Home & Garden > Gardening.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "best plants for a small balcony garden"
     },
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Personal Finance",
+        "model_output.classification.iab_content.tier1.label": "Food & Drink",
         "model_output.classification.iab_content.tier2.label": null
       },
       "expected": {
@@ -2570,7 +2430,7 @@
       "id": "gardening-medium",
       "mismatches": [
         {
-          "actual": "Personal Finance",
+          "actual": "Food & Drink",
           "expected": "Home & Garden",
           "path": "model_output.classification.iab_content.tier1.label"
         },
@@ -2592,9 +2452,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Home & Garden",
-        "model_output.classification.iab_content.tier2.label": null
+        "model_output.classification.iab_content.tier2.label": "Gardening"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -2602,28 +2462,17 @@
         "model_output.classification.iab_content.tier2.label": "Gardening"
       },
       "id": "gardening-hard",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
-          "expected": "Gardening",
-          "path": "model_output.classification.iab_content.tier2.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical hard IAB mapping case for Home & Garden > Gardening.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "need gardening advice for a shady backyard, not interior decor ideas"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Genres",
-        "model_output.classification.iab_content.tier2.label": null
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Entertainment",
+        "model_output.classification.iab_content.tier2.label": "Movies"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -2631,31 +2480,15 @@
         "model_output.classification.iab_content.tier2.label": "Movies"
       },
       "id": "movies-easy",
-      "mismatches": [
-        {
-          "actual": "Genres",
-          "expected": "Entertainment",
-          "path": "model_output.classification.iab_content.tier1.label"
-        },
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": null,
-          "expected": "Movies",
-          "path": "model_output.classification.iab_content.tier2.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical easy IAB mapping case for Entertainment > Movies.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "What movie should we watch tonight?"
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Genres",
         "model_output.classification.iab_content.tier2.label": "Horror"
       },
@@ -2671,11 +2504,6 @@
           "expected": "Entertainment",
           "path": "model_output.classification.iab_content.tier1.label"
         },
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
         {
           "actual": "Horror",
           "expected": "Movies",
@@ -2689,9 +2517,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Entertainment",
-        "model_output.classification.iab_content.tier2.label": "Music"
+        "model_output.classification.iab_content.tier2.label": "Movies"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -2699,20 +2527,9 @@
         "model_output.classification.iab_content.tier2.label": "Movies"
       },
       "id": "movies-hard",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        },
-        {
-          "actual": "Music",
-          "expected": "Movies",
-          "path": "model_output.classification.iab_content.tier2.label"
-        }
-      ],
+      "mismatches": [],
       "notes": "Cross-vertical hard IAB mapping case for Entertainment > Movies.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Looking for film recommendations, not TV shows or music"
     }

artifacts/evaluation/latest/iab_quality_target_eval.json

@@ -1,15 +1,15 @@
 {
   "by_status": {
     "must_fix": {
-      "failed": 12,
-      "passed": 0,
+      "failed": 9,
+      "passed": 3,
       "total": 12
     }
   },
-  "cases_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/examples/iab_mapping_cases.json",
+  "cases_path": "/content/agentic-intent-classifier/examples/iab_mapping_cases.json",
   "count": 12,
-  "failed": 12,
-  "passed": 0,
+  "failed": 9,
+  "passed": 3,
   "results": [
     {
       "actual": {
@@ -42,7 +42,7 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
         "model_output.classification.iab_content.tier3.label": "Laptops"
@@ -54,15 +54,9 @@
         "model_output.classification.iab_content.tier3.label": "Laptops"
       },
       "id": "laptop-buying-maps-to-laptops",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        }
-      ],
+      "mismatches": [],
       "notes": "Laptop shopping should resolve into the laptops branch, not business sales.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Which laptop to buy in 2026"
     },
@@ -70,7 +64,7 @@
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
-        "model_output.classification.iab_content.tier2.label": "Computing",
+        "model_output.classification.iab_content.tier2.label": null,
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -81,6 +75,11 @@
       },
       "id": "labtop-buying-maps-to-laptops",
       "mismatches": [
+        {
+          "actual": null,
+          "expected": "Computing",
+          "path": "model_output.classification.iab_content.tier2.label"
+        },
         {
           "actual": null,
           "expected": "Laptops",
@@ -102,7 +101,7 @@
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+        "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -123,7 +122,7 @@
           "path": "model_output.classification.iab_content.tier2.label"
         },
         {
-          "actual": "Software and Applications",
+          "actual": null,
           "expected": "Sales",
           "path": "model_output.classification.iab_content.tier3.label"
         }
@@ -137,8 +136,8 @@
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
-        "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": "Software and Applications"
+        "model_output.classification.iab_content.tier2.label": null,
+        "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -154,12 +153,12 @@
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": "Computing",
+          "actual": null,
           "expected": "Business",
           "path": "model_output.classification.iab_content.tier2.label"
         },
         {
-          "actual": "Software and Applications",
+          "actual": null,
           "expected": "Sales",
           "path": "model_output.classification.iab_content.tier3.label"
         },
@@ -177,8 +176,8 @@
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Hobbies & Interests",
-        "model_output.classification.iab_content.tier2.label": "Content Production",
+        "model_output.classification.iab_content.tier1.label": "Careers",
+        "model_output.classification.iab_content.tier2.label": null,
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -190,12 +189,12 @@
       "id": "marketing-tools-map-to-marketing",
       "mismatches": [
         {
-          "actual": "Hobbies & Interests",
+          "actual": "Careers",
           "expected": "Business and Finance",
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": "Content Production",
+          "actual": null,
           "expected": "Business",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -218,8 +217,8 @@
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
-        "model_output.classification.iab_content.tier2.label": "Computing"
+        "model_output.classification.iab_content.tier1.label": "Sports",
+        "model_output.classification.iab_content.tier2.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -229,7 +228,12 @@
       "id": "ml-explanation-maps-to-ai",
       "mismatches": [
         {
-          "actual": "Computing",
+          "actual": "Sports",
+          "expected": "Technology & Computing",
+          "path": "model_output.classification.iab_content.tier1.label"
+        },
+        {
+          "actual": null,
           "expected": "Artificial Intelligence",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -249,7 +253,7 @@
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
         "model_output.classification.iab_content.tier1.label": "Technology & Computing",
         "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": "Internet"
+        "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
@@ -270,7 +274,7 @@
           "path": "model_output.classification.iab_content.tier2.label"
         },
         {
-          "actual": "Internet",
+          "actual": null,
           "expected": "Business I.T.",
           "path": "model_output.classification.iab_content.tier3.label"
         }
@@ -282,7 +286,7 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Food & Drink",
         "model_output.classification.iab_content.tier2.label": "Dining Out"
       },
@@ -292,23 +296,17 @@
         "model_output.classification.iab_content.tier2.label": "Dining Out"
       },
       "id": "restaurant-booking-maps-to-dining-out",
-      "mismatches": [
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        }
-      ],
+      "mismatches": [],
       "notes": "Generic dining requests should not inherit the repo's business default.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "Book a table for 2 tonight"
     },
     {
       "actual": {
         "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Business and Finance",
-        "model_output.classification.iab_content.tier2.label": "Business",
+        "model_output.classification.iab_content.tier1.label": "Sports",
+        "model_output.classification.iab_content.tier2.label": null,
         "model_output.classification.iab_content.tier3.label": null
       },
       "expected": {
@@ -320,12 +318,12 @@
       "id": "trial-signup-maps-to-software",
       "mismatches": [
         {
-          "actual": "Business and Finance",
+          "actual": "Sports",
           "expected": "Technology & Computing",
           "path": "model_output.classification.iab_content.tier1.label"
         },
         {
-          "actual": "Business",
+          "actual": null,
           "expected": "Computing",
           "path": "model_output.classification.iab_content.tier2.label"
         },
@@ -342,10 +340,10 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
-        "model_output.classification.iab_content.tier1.label": "Technology & Computing",
-        "model_output.classification.iab_content.tier2.label": "Computing",
-        "model_output.classification.iab_content.tier3.label": "Software and Applications",
+        "model_output.classification.iab_content.mapping_mode": "exact",
+        "model_output.classification.iab_content.tier1.label": "Careers",
+        "model_output.classification.iab_content.tier2.label": "Remote Working",
+        "model_output.classification.iab_content.tier3.label": null,
         "model_output.classification.iab_content.tier4.label": null
       },
       "expected": {
@@ -358,7 +356,17 @@
       "id": "communication-software-maps-to-tier4",
       "mismatches": [
         {
-          "actual": "Software and Applications",
+          "actual": "Careers",
+          "expected": "Technology & Computing",
+          "path": "model_output.classification.iab_content.tier1.label"
+        },
+        {
+          "actual": "Remote Working",
+          "expected": "Computing",
+          "path": "model_output.classification.iab_content.tier2.label"
+        },
+        {
+          "actual": null,
           "expected": "Computer Software and Applications",
           "path": "model_output.classification.iab_content.tier3.label"
         },
@@ -366,11 +374,6 @@
           "actual": null,
           "expected": "Communication",
           "path": "model_output.classification.iab_content.tier4.label"
-        },
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
         }
       ],
       "notes": "Full taxonomy support should preserve the tier4 communication branch.",
@@ -380,9 +383,9 @@
     },
     {
       "actual": {
-        "model_output.classification.iab_content.mapping_mode": "nearest_equivalent",
+        "model_output.classification.iab_content.mapping_mode": "exact",
         "model_output.classification.iab_content.tier1.label": "Food & Drink",
-        "model_output.classification.iab_content.tier2.label": null
+        "model_output.classification.iab_content.tier2.label": "Alcoholic Beverages"
       },
       "expected": {
         "model_output.classification.iab_content.mapping_mode": "exact",
@@ -390,20 +393,9 @@
         "model_output.classification.iab_content.tier2.label": "Alcoholic Beverages"
       },
       "id": "vodka-query-maps-to-alcoholic-beverages",
-      "mismatches": [
-        {
-          "actual": null,
-          "expected": "Alcoholic Beverages",
-          "path": "model_output.classification.iab_content.tier2.label"
-        },
-        {
-          "actual": "nearest_equivalent",
-          "expected": "exact",
-          "path": "model_output.classification.iab_content.mapping_mode"
-        }
-      ],
+      "mismatches": [],
       "notes": "Food and beverage prompts should not fall through to the business default.",
-      "pass": false,
+      "pass": true,
       "status": "must_fix",
       "text": "what is best vodka drink should i try"
     }

artifacts/evaluation/latest/intent_subtype_difficulty_benchmark_confusion_matrix.csv

@@ -1,19 +1,19 @@
 ,education,product_discovery,comparison,evaluation,deal_seeking,provider_selection,signup,purchase,booking,download,contact_sales,task_execution,onboarding_setup,troubleshooting,account_help,billing_help,follow_up,emotional_reflection
 education,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
-product_discovery,0,13,0,0,1,0,0,0,0,0,0,1,0,0,0,0,0,0
-comparison,2,0,13,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+product_discovery,0,13,0,1,0,0,0,0,0,0,0,1,0,0,0,0,0,0
+comparison,1,0,11,2,0,0,0,0,0,0,0,0,0,0,0,1,0,0
 evaluation,1,0,0,14,0,0,0,0,0,0,0,0,0,0,0,0,0,0
-deal_seeking,0,0,0,0,14,1,0,0,0,0,0,0,0,0,0,0,0,0
-provider_selection,0,0,1,0,0,15,0,0,0,0,0,0,0,0,0,0,0,0
-signup,0,0,0,0,0,0,16,0,0,0,0,0,0,0,0,0,0,0
-purchase,0,0,0,0,0,0,0,14,0,0,0,0,1,0,0,0,0,0
-booking,0,0,0,0,0,0,1,0,13,0,0,1,0,0,0,0,0,0
-download,0,0,0,0,0,0,0,0,0,14,0,1,0,0,0,0,0,0
-contact_sales,0,0,0,0,0,0,0,0,0,0,15,0,0,0,0,0,0,0
-task_execution,0,0,0,0,0,0,1,0,0,0,0,17,0,0,0,0,0,0
-onboarding_setup,0,0,0,0,0,0,0,0,0,0,0,1,16,0,0,0,0,0
-troubleshooting,0,0,0,0,0,0,0,0,0,0,0,0,0,14,0,0,0,1
-account_help,0,0,0,0,0,0,2,0,0,0,0,0,0,0,12,1,0,0
-billing_help,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,15,0,0
-follow_up,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,15,0
+deal_seeking,0,1,1,0,13,0,0,0,0,0,0,0,0,0,0,0,0,0
+provider_selection,0,0,0,0,0,16,0,0,0,0,0,0,0,0,0,0,0,0
+signup,0,0,0,0,0,0,14,0,0,1,0,0,0,0,1,0,0,0
+purchase,0,0,0,0,0,0,1,13,0,0,0,1,0,0,0,0,0,0
+booking,0,0,0,0,0,0,5,0,8,0,1,1,0,0,0,0,0,0
+download,0,0,0,0,0,0,0,0,0,13,0,1,0,0,1,0,0,0
+contact_sales,0,0,0,1,0,0,0,0,0,0,14,0,0,0,0,0,0,0
+task_execution,0,0,0,0,0,0,0,0,0,0,0,17,1,0,0,0,0,0
+onboarding_setup,0,0,0,0,0,0,0,0,0,0,0,1,15,1,0,0,0,0
+troubleshooting,0,0,0,0,0,0,0,0,0,0,0,0,0,15,0,0,0,0
+account_help,0,0,0,0,0,0,0,0,0,0,0,0,0,1,12,2,0,0
+billing_help,0,0,0,0,0,0,0,0,0,0,0,0,0,2,1,12,0,0
+follow_up,0,0,0,0,0,0,0,0,0,0,0,1,0,0,1,0,13,0
 emotional_reflection,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,15

artifacts/evaluation/latest/intent_subtype_difficulty_benchmark_report.json

@@ -1,159 +1,159 @@
 {
-  "accepted_accuracy": 0.9386,
-  "accepted_coverage": 1.0,
-  "accuracy": 0.9386,
-  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/intent_subtype_difficulty_benchmark_confusion_matrix.csv",
+  "accepted_accuracy": 0.8967,
+  "accepted_coverage": 0.9783,
+  "accuracy": 0.8773,
+  "confusion_matrix_path": "/content/agentic-intent-classifier/artifacts/evaluation/latest/intent_subtype_difficulty_benchmark_confusion_matrix.csv",
   "count": 277,
-  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/subtype_benchmark.jsonl",
+  "dataset_path": "/content/agentic-intent-classifier/data/subtype_benchmark.jsonl",
   "difficulty_breakdown": {
     "easy": {
-      "accepted_accuracy": 0.9565,
+      "accepted_accuracy": 0.913,
       "accepted_coverage": 1.0,
-      "accuracy": 0.9565,
+      "accuracy": 0.913,
       "count": 92,
       "fallback_rate": 0.0,
-      "macro_f1": 0.9579
+      "macro_f1": 0.9111
     },
     "hard": {
-      "accepted_accuracy": 0.8901,
-      "accepted_coverage": 1.0,
-      "accuracy": 0.8901,
+      "accepted_accuracy": 0.814,
+      "accepted_coverage": 0.9451,
+      "accuracy": 0.7692,
       "count": 91,
-      "fallback_rate": 0.0,
-      "macro_f1": 0.8913
+      "fallback_rate": 0.0549,
+      "macro_f1": 0.7704
     },
     "medium": {
-      "accepted_accuracy": 0.9681,
-      "accepted_coverage": 1.0,
-      "accuracy": 0.9681,
+      "accepted_accuracy": 0.957,
+      "accepted_coverage": 0.9894,
+      "accuracy": 0.9468,
       "count": 94,
-      "fallback_rate": 0.0,
-      "macro_f1": 0.9671
+      "fallback_rate": 0.0106,
+      "macro_f1": 0.9453
     }
   },
-  "fallback_rate": 0.0,
+  "fallback_rate": 0.0217,
   "head": "intent_subtype",
-  "macro_f1": 0.9401,
+  "macro_f1": 0.8767,
   "per_class_metrics": {
     "account_help": {
-      "f1-score": 0.8888888888888888,
-      "precision": 1.0,
+      "f1-score": 0.7741935483870968,
+      "precision": 0.75,
       "recall": 0.8,
       "support": 15.0
     },
-    "accuracy": 0.9386281588447654,
+    "accuracy": 0.8772563176895307,
     "billing_help": {
-      "f1-score": 0.967741935483871,
-      "precision": 0.9375,
-      "recall": 1.0,
+      "f1-score": 0.8,
+      "precision": 0.8,
+      "recall": 0.8,
       "support": 15.0
     },
     "booking": {
-      "f1-score": 0.9285714285714286,
+      "f1-score": 0.6956521739130435,
       "precision": 1.0,
-      "recall": 0.8666666666666667,
+      "recall": 0.5333333333333333,
       "support": 15.0
     },
     "comparison": {
-      "f1-score": 0.896551724137931,
-      "precision": 0.9285714285714286,
-      "recall": 0.8666666666666667,
+      "f1-score": 0.8148148148148148,
+      "precision": 0.9166666666666666,
+      "recall": 0.7333333333333333,
       "support": 15.0
     },
     "contact_sales": {
-      "f1-score": 1.0,
-      "precision": 1.0,
-      "recall": 1.0,
-      "support": 15.0
-    },
-    "deal_seeking": {
       "f1-score": 0.9333333333333333,
       "precision": 0.9333333333333333,
       "recall": 0.9333333333333333,
       "support": 15.0
     },
-    "download": {
-      "f1-score": 0.9655172413793104,
+    "deal_seeking": {
+      "f1-score": 0.9285714285714286,
       "precision": 1.0,
-      "recall": 0.9333333333333333,
+      "recall": 0.8666666666666667,
+      "support": 15.0
+    },
+    "download": {
+      "f1-score": 0.896551724137931,
+      "precision": 0.9285714285714286,
+      "recall": 0.8666666666666667,
       "support": 15.0
     },
     "education": {
-      "f1-score": 0.9090909090909091,
-      "precision": 0.8333333333333334,
+      "f1-score": 0.9375,
+      "precision": 0.8823529411764706,
       "recall": 1.0,
       "support": 15.0
     },
     "emotional_reflection": {
-      "f1-score": 0.967741935483871,
-      "precision": 0.9375,
+      "f1-score": 1.0,
+      "precision": 1.0,
       "recall": 1.0,
       "support": 15.0
     },
     "evaluation": {
-      "f1-score": 0.9655172413793104,
-      "precision": 1.0,
+      "f1-score": 0.8484848484848485,
+      "precision": 0.7777777777777778,
       "recall": 0.9333333333333333,
       "support": 15.0
     },
     "follow_up": {
-      "f1-score": 1.0,
+      "f1-score": 0.9285714285714286,
       "precision": 1.0,
-      "recall": 1.0,
+      "recall": 0.8666666666666667,
       "support": 15.0
     },
     "macro avg": {
-      "f1-score": 0.9401067100194944,
-      "precision": 0.9476910208527856,
-      "recall": 0.9383215323166303,
+      "f1-score": 0.8767381318695205,
+      "precision": 0.893520983060569,
+      "recall": 0.8760257806826435,
       "support": 277.0
     },
     "onboarding_setup": {
-      "f1-score": 0.9411764705882353,
-      "precision": 0.9411764705882353,
-      "recall": 0.9411764705882353,
+      "f1-score": 0.9090909090909091,
+      "precision": 0.9375,
+      "recall": 0.8823529411764706,
       "support": 17.0
     },
     "product_discovery": {
-      "f1-score": 0.9285714285714286,
-      "precision": 1.0,
+      "f1-score": 0.896551724137931,
+      "precision": 0.9285714285714286,
       "recall": 0.8666666666666667,
       "support": 15.0
     },
     "provider_selection": {
-      "f1-score": 0.9375,
-      "precision": 0.9375,
-      "recall": 0.9375,
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
       "support": 16.0
     },
     "purchase": {
-      "f1-score": 0.9655172413793104,
+      "f1-score": 0.9285714285714286,
       "precision": 1.0,
-      "recall": 0.9333333333333333,
+      "recall": 0.8666666666666667,
       "support": 15.0
     },
     "signup": {
-      "f1-score": 0.8888888888888888,
-      "precision": 0.8,
-      "recall": 1.0,
+      "f1-score": 0.7777777777777778,
+      "precision": 0.7,
+      "recall": 0.875,
       "support": 16.0
     },
     "task_execution": {
-      "f1-score": 0.8717948717948718,
-      "precision": 0.8095238095238095,
+      "f1-score": 0.8292682926829268,
+      "precision": 0.7391304347826086,
       "recall": 0.9444444444444444,
       "support": 18.0
     },
     "troubleshooting": {
-      "f1-score": 0.9655172413793104,
-      "precision": 1.0,
-      "recall": 0.9333333333333333,
+      "f1-score": 0.8823529411764706,
+      "precision": 0.7894736842105263,
+      "recall": 1.0,
       "support": 15.0
     },
     "weighted avg": {
-      "f1-score": 0.9391802821325396,
-      "precision": 0.9455776173285199,
-      "recall": 0.9386281588447654,
+      "f1-score": 0.8765453432446891,
+      "precision": 0.8918521903635431,
+      "recall": 0.8772563176895307,
       "support": 277.0
     }
   },

artifacts/evaluation/latest/intent_subtype_extended_cases_confusion_matrix.csv

@@ -1,8 +1,8 @@
 ,education,product_discovery,comparison,evaluation,deal_seeking,provider_selection,signup,purchase,booking,download,contact_sales,task_execution,onboarding_setup,troubleshooting,account_help,billing_help,follow_up,emotional_reflection
 education,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
 product_discovery,0,6,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
-comparison,0,0,1,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0
-evaluation,1,0,0,1,0,1,0,0,0,0,0,0,0,0,0,0,0,0
+comparison,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+evaluation,1,1,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0
 deal_seeking,0,0,0,0,9,0,0,0,0,0,0,0,0,0,0,0,0,0
 provider_selection,0,0,0,0,0,5,0,0,0,0,0,0,0,0,0,0,0,0
 signup,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
@@ -15,5 +15,5 @@ onboarding_setup,0,0,0,0,0,0,0,0,0,0,0,0,4,0,0,0,0,0
 troubleshooting,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0
 account_help,0,0,0,0,0,0,0,0,0,0,0,0,0,1,2,0,0,0
 billing_help,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
-follow_up,0,0,0,0,3,0,0,0,0,0,0,0,0,0,1,0,8,0
+follow_up,1,0,0,0,2,0,0,0,0,0,0,1,0,0,0,0,8,0
 emotional_reflection,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0

artifacts/evaluation/latest/intent_subtype_extended_cases_report.json

@@ -2,16 +2,16 @@
   "accepted_accuracy": 0.8491,
   "accepted_coverage": 1.0,
   "accuracy": 0.8491,
-  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/intent_subtype_extended_cases_confusion_matrix.csv",
+  "confusion_matrix_path": "/content/agentic-intent-classifier/artifacts/evaluation/latest/intent_subtype_extended_cases_confusion_matrix.csv",
   "count": 53,
-  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/subtype/extended_cases.jsonl",
+  "dataset_path": "/content/agentic-intent-classifier/data/subtype/extended_cases.jsonl",
   "fallback_rate": 0.0,
   "head": "intent_subtype",
-  "macro_f1": 0.8146,
+  "macro_f1": 0.7764,
   "per_class_metrics": {
     "account_help": {
-      "f1-score": 0.6666666666666666,
-      "precision": 0.6666666666666666,
+      "f1-score": 0.8,
+      "precision": 1.0,
       "recall": 0.6666666666666666,
       "support": 3.0
     },
@@ -29,9 +29,9 @@
       "support": 0.0
     },
     "comparison": {
-      "f1-score": 0.6666666666666666,
+      "f1-score": 1.0,
       "precision": 1.0,
-      "recall": 0.5,
+      "recall": 1.0,
       "support": 2.0
     },
     "contact_sales": {
@@ -41,8 +41,8 @@
       "support": 0.0
     },
     "deal_seeking": {
-      "f1-score": 0.8181818181818182,
-      "precision": 0.6923076923076923,
+      "f1-score": 0.9,
+      "precision": 0.8181818181818182,
       "recall": 1.0,
       "support": 9.0
     },
@@ -53,8 +53,8 @@
       "support": 0.0
     },
     "education": {
-      "f1-score": 0.9333333333333333,
-      "precision": 0.875,
+      "f1-score": 0.875,
+      "precision": 0.7777777777777778,
       "recall": 1.0,
       "support": 7.0
     },
@@ -65,9 +65,9 @@
       "support": 0.0
     },
     "evaluation": {
-      "f1-score": 0.5,
-      "precision": 1.0,
-      "recall": 0.3333333333333333,
+      "f1-score": 0.0,
+      "precision": 0.0,
+      "recall": 0.0,
       "support": 3.0
     },
     "follow_up": {
@@ -77,9 +77,9 @@
       "support": 12.0
     },
     "macro avg": {
-      "f1-score": 0.4978114478114478,
-      "precision": 0.531517094017094,
-      "recall": 0.5092592592592592,
+      "f1-score": 0.474472286972287,
+      "precision": 0.46035754369087706,
+      "recall": 0.5185185185185186,
       "support": 53.0
     },
     "onboarding_setup": {
@@ -89,8 +89,8 @@
       "support": 4.0
     },
     "product_discovery": {
-      "f1-score": 1.0,
-      "precision": 1.0,
+      "f1-score": 0.9230769230769231,
+      "precision": 0.8571428571428571,
       "recall": 1.0,
       "support": 6.0
     },
@@ -113,8 +113,8 @@
       "support": 0.0
     },
     "task_execution": {
-      "f1-score": 1.0,
-      "precision": 1.0,
+      "f1-score": 0.6666666666666666,
+      "precision": 0.5,
       "recall": 1.0,
       "support": 1.0
     },
@@ -125,8 +125,8 @@
       "support": 1.0
     },
     "weighted avg": {
-      "f1-score": 0.8404230989136648,
-      "precision": 0.887215771649734,
+      "f1-score": 0.823438668249989,
+      "precision": 0.8324076342944268,
       "recall": 0.8490566037735849,
       "support": 53.0
     }

artifacts/evaluation/latest/intent_subtype_hard_cases_confusion_matrix.csv

@@ -2,8 +2,8 @@
 education,29,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
 product_discovery,0,8,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
 comparison,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
-evaluation,0,2,0,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0
-deal_seeking,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0
+evaluation,2,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+deal_seeking,0,0,0,0,2,1,0,0,0,0,0,0,0,0,0,0,0,0
 provider_selection,0,0,0,1,0,9,0,0,0,0,0,0,0,0,0,0,0,0
 signup,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0,0
 purchase,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0

artifacts/evaluation/latest/intent_subtype_hard_cases_report.json

@@ -1,13 +1,13 @@
 {
-  "accepted_accuracy": 0.9468,
+  "accepted_accuracy": 0.8936,
   "accepted_coverage": 1.0,
-  "accuracy": 0.9468,
-  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/intent_subtype_hard_cases_confusion_matrix.csv",
+  "accuracy": 0.8936,
+  "confusion_matrix_path": "/content/agentic-intent-classifier/artifacts/evaluation/latest/intent_subtype_hard_cases_confusion_matrix.csv",
   "count": 94,
-  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/subtype/hard_cases.jsonl",
+  "dataset_path": "/content/agentic-intent-classifier/data/subtype/hard_cases.jsonl",
   "fallback_rate": 0.0,
   "head": "intent_subtype",
-  "macro_f1": 0.9191,
+  "macro_f1": 0.846,
   "per_class_metrics": {
     "account_help": {
       "f1-score": 0.8,
@@ -15,7 +15,7 @@
       "recall": 1.0,
       "support": 2.0
     },
-    "accuracy": 0.9468085106382979,
+    "accuracy": 0.8936170212765957,
     "billing_help": {
       "f1-score": 1.0,
       "precision": 1.0,
@@ -41,9 +41,9 @@
       "support": 0.0
     },
     "deal_seeking": {
-      "f1-score": 1.0,
+      "f1-score": 0.8,
       "precision": 1.0,
-      "recall": 1.0,
+      "recall": 0.6666666666666666,
       "support": 3.0
     },
     "download": {
@@ -53,8 +53,8 @@
       "support": 0.0
     },
     "education": {
-      "f1-score": 1.0,
-      "precision": 1.0,
+      "f1-score": 0.9666666666666667,
+      "precision": 0.9354838709677419,
       "recall": 1.0,
       "support": 29.0
     },
@@ -65,9 +65,9 @@
       "support": 0.0
     },
     "evaluation": {
-      "f1-score": 0.7272727272727273,
-      "precision": 0.8,
-      "recall": 0.6666666666666666,
+      "f1-score": 0.0,
+      "precision": 0.0,
+      "recall": 0.0,
       "support": 6.0
     },
     "follow_up": {
@@ -77,9 +77,9 @@
       "support": 12.0
     },
     "macro avg": {
-      "f1-score": 0.7659288023895194,
-      "precision": 0.7648148148148147,
-      "recall": 0.786111111111111,
+      "f1-score": 0.7049919484702094,
+      "precision": 0.7038231780167263,
+      "recall": 0.7305555555555556,
       "support": 94.0
     },
     "onboarding_setup": {
@@ -89,14 +89,14 @@
       "support": 6.0
     },
     "product_discovery": {
-      "f1-score": 0.8888888888888888,
-      "precision": 0.8,
+      "f1-score": 0.8,
+      "precision": 0.6666666666666666,
       "recall": 1.0,
       "support": 8.0
     },
     "provider_selection": {
-      "f1-score": 0.9473684210526315,
-      "precision": 1.0,
+      "f1-score": 0.9,
+      "precision": 0.9,
       "recall": 0.9,
       "support": 10.0
     },
@@ -125,9 +125,9 @@
       "support": 3.0
     },
     "weighted avg": {
-      "f1-score": 0.9478016938458051,
-      "precision": 0.9578014184397163,
-      "recall": 0.9468085106382979,
+      "f1-score": 0.8721091581868641,
+      "precision": 0.8648478609013955,
+      "recall": 0.8936170212765957,
       "support": 94.0
     }
   },

artifacts/evaluation/latest/intent_subtype_test_confusion_matrix.csv

@@ -1,8 +1,8 @@
 ,education,product_discovery,comparison,evaluation,deal_seeking,provider_selection,signup,purchase,booking,download,contact_sales,task_execution,onboarding_setup,troubleshooting,account_help,billing_help,follow_up,emotional_reflection
 education,14,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
-product_discovery,0,7,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+product_discovery,0,8,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
 comparison,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
-evaluation,0,1,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+evaluation,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0
 deal_seeking,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0
 provider_selection,0,0,0,1,0,5,0,0,0,0,0,0,0,0,0,0,0,0
 signup,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0
@@ -12,8 +12,8 @@ download,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
 contact_sales,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
 task_execution,0,0,0,0,0,0,0,0,0,0,0,6,0,0,0,0,0,0
 onboarding_setup,0,0,0,0,0,0,0,0,0,0,0,0,4,0,0,0,0,0
-troubleshooting,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0
+troubleshooting,0,0,0,0,0,0,0,0,0,0,0,1,0,1,0,0,0,0
 account_help,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,0,0,0
 billing_help,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
-follow_up,0,0,0,0,2,0,0,0,0,0,0,0,0,1,0,0,8,0
+follow_up,1,0,0,0,1,0,0,0,0,0,0,1,0,0,0,0,8,0
 emotional_reflection,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,5

artifacts/evaluation/latest/intent_subtype_test_report.json

@@ -1,13 +1,13 @@
 {
-  "accepted_accuracy": 0.9,
+  "accepted_accuracy": 0.9143,
   "accepted_coverage": 1.0,
-  "accuracy": 0.9,
-  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/intent_subtype_test_confusion_matrix.csv",
+  "accuracy": 0.9143,
+  "confusion_matrix_path": "/content/agentic-intent-classifier/artifacts/evaluation/latest/intent_subtype_test_confusion_matrix.csv",
   "count": 70,
-  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/subtype/test.jsonl",
+  "dataset_path": "/content/agentic-intent-classifier/data/subtype/test.jsonl",
   "fallback_rate": 0.0,
   "head": "intent_subtype",
-  "macro_f1": 0.863,
+  "macro_f1": 0.8855,
   "per_class_metrics": {
     "account_help": {
       "f1-score": 1.0,
@@ -15,7 +15,7 @@
       "recall": 1.0,
       "support": 2.0
     },
-    "accuracy": 0.9,
+    "accuracy": 0.9142857142857143,
     "billing_help": {
       "f1-score": 0.0,
       "precision": 0.0,
@@ -29,8 +29,8 @@
       "support": 3.0
     },
     "comparison": {
-      "f1-score": 1.0,
-      "precision": 1.0,
+      "f1-score": 0.8571428571428571,
+      "precision": 0.75,
       "recall": 1.0,
       "support": 3.0
     },
@@ -41,8 +41,8 @@
       "support": 0.0
     },
     "deal_seeking": {
-      "f1-score": 0.6666666666666666,
-      "precision": 0.5,
+      "f1-score": 0.8,
+      "precision": 0.6666666666666666,
       "recall": 1.0,
       "support": 2.0
     },
@@ -53,8 +53,8 @@
       "support": 0.0
     },
     "education": {
-      "f1-score": 1.0,
-      "precision": 1.0,
+      "f1-score": 0.9655172413793104,
+      "precision": 0.9333333333333333,
       "recall": 1.0,
       "support": 14.0
     },
@@ -65,8 +65,8 @@
       "support": 5.0
     },
     "evaluation": {
-      "f1-score": 0.4,
-      "precision": 0.3333333333333333,
+      "f1-score": 0.5,
+      "precision": 0.5,
       "recall": 0.5,
       "support": 2.0
     },
@@ -77,21 +77,21 @@
       "support": 11.0
     },
     "macro avg": {
-      "f1-score": 0.6712084293224644,
-      "precision": 0.6671296296296296,
-      "recall": 0.6908670033670034,
+      "f1-score": 0.6887592108100274,
+      "precision": 0.7,
+      "recall": 0.6978114478114478,
       "support": 70.0
     },
     "onboarding_setup": {
-      "f1-score": 0.8888888888888888,
-      "precision": 0.8,
+      "f1-score": 1.0,
+      "precision": 1.0,
       "recall": 1.0,
       "support": 4.0
     },
     "product_discovery": {
-      "f1-score": 0.875,
-      "precision": 0.875,
-      "recall": 0.875,
+      "f1-score": 1.0,
+      "precision": 1.0,
+      "recall": 1.0,
       "support": 8.0
     },
     "provider_selection": {
@@ -113,21 +113,21 @@
       "support": 2.0
     },
     "task_execution": {
-      "f1-score": 1.0,
-      "precision": 1.0,
+      "f1-score": 0.8571428571428571,
+      "precision": 0.75,
       "recall": 1.0,
       "support": 6.0
     },
     "troubleshooting": {
-      "f1-score": 0.5,
-      "precision": 0.5,
+      "f1-score": 0.6666666666666666,
+      "precision": 1.0,
       "recall": 0.5,
       "support": 2.0
     },
     "weighted avg": {
-      "f1-score": 0.9058084605453025,
-      "precision": 0.9266666666666667,
-      "recall": 0.9,
+      "f1-score": 0.9126080539458813,
+      "precision": 0.9307142857142858,
+      "recall": 0.9142857142857143,
       "support": 70.0
     }
   },

artifacts/evaluation/latest/intent_subtype_train_confusion_matrix.csv

@@ -1,19 +1,19 @@
 ,education,product_discovery,comparison,evaluation,deal_seeking,provider_selection,signup,purchase,booking,download,contact_sales,task_execution,onboarding_setup,troubleshooting,account_help,billing_help,follow_up,emotional_reflection
 education,52,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
-product_discovery,0,31,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+product_discovery,0,30,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0
 comparison,0,0,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
-evaluation,1,2,0,14,0,0,0,0,0,0,0,0,0,0,0,0,0,0
-deal_seeking,0,0,0,0,11,0,0,0,0,0,0,0,0,0,0,0,0,0
-provider_selection,0,0,0,0,0,24,0,0,0,0,0,0,0,0,0,0,1,0
+evaluation,3,6,0,8,0,0,0,0,0,0,0,0,0,0,0,0,0,0
+deal_seeking,0,0,0,0,10,1,0,0,0,0,0,0,0,0,0,0,0,0
+provider_selection,0,0,0,0,0,23,0,0,0,0,0,0,0,0,0,0,2,0
 signup,0,0,0,0,0,0,16,0,0,0,0,0,0,0,0,0,0,0
 purchase,0,0,0,0,0,0,0,6,0,0,0,0,0,0,0,0,0,0
-booking,0,0,0,0,0,0,0,0,5,0,0,0,0,0,0,0,0,0
+booking,0,0,0,0,0,0,2,0,3,0,0,0,0,0,0,0,0,0
 download,0,0,0,0,0,0,0,0,0,8,0,0,0,0,0,0,0,0
-contact_sales,0,0,0,0,0,0,2,0,0,0,7,0,0,0,0,0,0,0
+contact_sales,0,0,0,0,0,0,2,0,1,0,6,0,0,0,0,0,0,0
 task_execution,0,0,0,0,0,0,0,0,0,0,0,19,0,0,0,0,0,0
-onboarding_setup,0,0,0,0,0,0,0,0,0,0,0,0,17,0,0,0,0,0
-troubleshooting,0,0,0,0,0,0,0,0,0,0,0,0,0,12,1,0,0,0
+onboarding_setup,0,0,0,0,0,0,0,0,1,1,0,0,15,0,0,0,0,0
+troubleshooting,0,0,0,0,0,0,0,0,0,0,0,0,0,10,1,0,2,0
 account_help,0,0,0,0,0,0,0,0,0,0,0,0,0,0,7,0,0,0
-billing_help,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,6,0,0
-follow_up,0,0,0,0,0,0,0,0,0,0,0,3,0,1,0,0,32,0
+billing_help,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,4,0,0
+follow_up,0,0,0,0,0,0,0,0,0,0,0,5,0,0,0,0,31,0
 emotional_reflection,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,20

artifacts/evaluation/latest/intent_subtype_train_report.json

@@ -1,31 +1,31 @@
 {
-  "accepted_accuracy": 0.9649,
+  "accepted_accuracy": 0.9042,
   "accepted_coverage": 1.0,
-  "accuracy": 0.9649,
-  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/intent_subtype_train_confusion_matrix.csv",
+  "accuracy": 0.9042,
+  "confusion_matrix_path": "/content/agentic-intent-classifier/artifacts/evaluation/latest/intent_subtype_train_confusion_matrix.csv",
   "count": 313,
-  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/subtype/train.jsonl",
+  "dataset_path": "/content/agentic-intent-classifier/data/subtype/train.jsonl",
   "fallback_rate": 0.0,
   "head": "intent_subtype",
-  "macro_f1": 0.9649,
+  "macro_f1": 0.8789,
   "per_class_metrics": {
     "account_help": {
-      "f1-score": 0.9333333333333333,
-      "precision": 0.875,
+      "f1-score": 0.875,
+      "precision": 0.7777777777777778,
       "recall": 1.0,
       "support": 7.0
     },
-    "accuracy": 0.9648562300319489,
+    "accuracy": 0.9041533546325878,
     "billing_help": {
-      "f1-score": 1.0,
+      "f1-score": 0.8,
       "precision": 1.0,
-      "recall": 1.0,
+      "recall": 0.6666666666666666,
       "support": 6.0
     },
     "booking": {
-      "f1-score": 1.0,
-      "precision": 1.0,
-      "recall": 1.0,
+      "f1-score": 0.6,
+      "precision": 0.6,
+      "recall": 0.6,
       "support": 5.0
     },
     "comparison": {
@@ -35,26 +35,26 @@
       "support": 15.0
     },
     "contact_sales": {
-      "f1-score": 0.875,
+      "f1-score": 0.8,
       "precision": 1.0,
-      "recall": 0.7777777777777778,
+      "recall": 0.6666666666666666,
       "support": 9.0
     },
     "deal_seeking": {
-      "f1-score": 1.0,
+      "f1-score": 0.9523809523809523,
       "precision": 1.0,
-      "recall": 1.0,
+      "recall": 0.9090909090909091,
       "support": 11.0
     },
     "download": {
-      "f1-score": 1.0,
-      "precision": 1.0,
+      "f1-score": 0.9411764705882353,
+      "precision": 0.8888888888888888,
       "recall": 1.0,
       "support": 8.0
     },
     "education": {
-      "f1-score": 0.9904761904761905,
-      "precision": 0.9811320754716981,
+      "f1-score": 0.9719626168224299,
+      "precision": 0.9454545454545454,
       "recall": 1.0,
       "support": 52.0
     },
@@ -65,39 +65,39 @@
       "support": 20.0
     },
     "evaluation": {
-      "f1-score": 0.9032258064516129,
+      "f1-score": 0.64,
       "precision": 1.0,
-      "recall": 0.8235294117647058,
+      "recall": 0.47058823529411764,
       "support": 17.0
     },
     "follow_up": {
-      "f1-score": 0.927536231884058,
-      "precision": 0.9696969696969697,
-      "recall": 0.8888888888888888,
+      "f1-score": 0.8611111111111112,
+      "precision": 0.8611111111111112,
+      "recall": 0.8611111111111112,
       "support": 36.0
     },
     "macro avg": {
-      "f1-score": 0.9649442256020961,
-      "precision": 0.9689347311202658,
-      "recall": 0.9651818334171275,
+      "f1-score": 0.8788540112011829,
+      "precision": 0.9092031425364758,
+      "recall": 0.8729694019289211,
       "support": 313.0
     },
     "onboarding_setup": {
-      "f1-score": 1.0,
+      "f1-score": 0.9375,
       "precision": 1.0,
-      "recall": 1.0,
+      "recall": 0.8823529411764706,
       "support": 17.0
     },
     "product_discovery": {
-      "f1-score": 0.96875,
-      "precision": 0.9393939393939394,
-      "recall": 1.0,
+      "f1-score": 0.8955223880597015,
+      "precision": 0.8333333333333334,
+      "recall": 0.967741935483871,
       "support": 31.0
     },
     "provider_selection": {
-      "f1-score": 0.9795918367346939,
-      "precision": 1.0,
-      "recall": 0.96,
+      "f1-score": 0.9387755102040817,
+      "precision": 0.9583333333333334,
+      "recall": 0.92,
       "support": 25.0
     },
     "purchase": {
@@ -107,27 +107,27 @@
       "support": 6.0
     },
     "signup": {
-      "f1-score": 0.9411764705882353,
-      "precision": 0.8888888888888888,
+      "f1-score": 0.8888888888888888,
+      "precision": 0.8,
       "recall": 1.0,
       "support": 16.0
     },
     "task_execution": {
-      "f1-score": 0.926829268292683,
-      "precision": 0.8636363636363636,
+      "f1-score": 0.8837209302325582,
+      "precision": 0.7916666666666666,
       "recall": 1.0,
       "support": 19.0
     },
     "troubleshooting": {
-      "f1-score": 0.9230769230769231,
-      "precision": 0.9230769230769231,
-      "recall": 0.9230769230769231,
+      "f1-score": 0.8333333333333334,
+      "precision": 0.9090909090909091,
+      "recall": 0.7692307692307693,
       "support": 13.0
     },
     "weighted avg": {
-      "f1-score": 0.9643733669039578,
-      "precision": 0.967429661617075,
-      "recall": 0.9648562300319489,
+      "f1-score": 0.8995750585641842,
+      "precision": 0.9142834091715881,
+      "recall": 0.9041533546325878,
       "support": 313.0
     }
   },

artifacts/evaluation/latest/intent_subtype_val_confusion_matrix.csv

@@ -1,12 +1,12 @@
 ,education,product_discovery,comparison,evaluation,deal_seeking,provider_selection,signup,purchase,booking,download,contact_sales,task_execution,onboarding_setup,troubleshooting,account_help,billing_help,follow_up,emotional_reflection
 education,14,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
 product_discovery,0,10,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0
-comparison,0,0,1,2,1,0,0,0,0,0,0,0,0,0,0,0,0,0
+comparison,0,0,2,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0
 evaluation,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0,0
 deal_seeking,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0,0,0
 provider_selection,0,0,0,0,0,7,0,0,0,0,0,0,0,0,0,0,0,0
 signup,0,0,0,0,0,0,2,0,0,0,0,0,0,0,0,0,0,0
-purchase,0,0,0,0,0,0,1,0,0,0,0,0,1,0,0,0,0,0
+purchase,0,0,0,0,0,0,1,0,0,0,0,1,0,0,0,0,0,0
 booking,0,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0
 download,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
 contact_sales,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
@@ -15,5 +15,5 @@ onboarding_setup,0,1,0,0,0,0,0,0,0,0,0,0,4,0,0,0,0,0
 troubleshooting,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0
 account_help,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0
 billing_help,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0
-follow_up,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,10,0
+follow_up,0,0,0,0,1,0,0,0,0,0,0,1,0,0,0,0,9,0
 emotional_reflection,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,5

artifacts/evaluation/latest/intent_subtype_val_report.json

@@ -1,13 +1,13 @@
 {
-  "accepted_accuracy": 0.875,
-  "accepted_coverage": 1.0,
+  "accepted_accuracy": 0.8734,
+  "accepted_coverage": 0.9875,
   "accuracy": 0.875,
-  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/intent_subtype_val_confusion_matrix.csv",
+  "confusion_matrix_path": "/content/agentic-intent-classifier/artifacts/evaluation/latest/intent_subtype_val_confusion_matrix.csv",
   "count": 80,
-  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/subtype/val.jsonl",
-  "fallback_rate": 0.0,
+  "dataset_path": "/content/agentic-intent-classifier/data/subtype/val.jsonl",
+  "fallback_rate": 0.0125,
   "head": "intent_subtype",
-  "macro_f1": 0.725,
+  "macro_f1": 0.7429,
   "per_class_metrics": {
     "account_help": {
       "f1-score": 0.5,
@@ -29,9 +29,9 @@
       "support": 3.0
     },
     "comparison": {
-      "f1-score": 0.4,
+      "f1-score": 0.6666666666666666,
       "precision": 1.0,
-      "recall": 0.25,
+      "recall": 0.5,
       "support": 4.0
     },
     "contact_sales": {
@@ -41,8 +41,8 @@
       "support": 0.0
     },
     "deal_seeking": {
-      "f1-score": 0.5714285714285714,
-      "precision": 0.4,
+      "f1-score": 0.6666666666666666,
+      "precision": 0.5,
       "recall": 1.0,
       "support": 2.0
     },
@@ -71,20 +71,20 @@
       "support": 2.0
     },
     "follow_up": {
-      "f1-score": 0.9523809523809523,
+      "f1-score": 0.9,
       "precision": 1.0,
-      "recall": 0.9090909090909091,
+      "recall": 0.8181818181818182,
       "support": 11.0
     },
     "macro avg": {
-      "f1-score": 0.6444203944203944,
-      "precision": 0.6542087542087542,
-      "recall": 0.687121212121212,
+      "f1-score": 0.660381593714927,
+      "precision": 0.6597643097643098,
+      "recall": 0.695959595959596,
       "support": 80.0
     },
     "onboarding_setup": {
-      "f1-score": 0.8,
-      "precision": 0.8,
+      "f1-score": 0.8888888888888888,
+      "precision": 1.0,
       "recall": 0.8,
       "support": 5.0
     },
@@ -113,8 +113,8 @@
       "support": 2.0
     },
     "task_execution": {
-      "f1-score": 1.0,
-      "precision": 1.0,
+      "f1-score": 0.8888888888888888,
+      "precision": 0.8,
       "recall": 1.0,
       "support": 8.0
     },
@@ -125,8 +125,8 @@
       "support": 1.0
     },
     "weighted avg": {
-      "f1-score": 0.8644047619047619,
-      "precision": 0.8891666666666665,
+      "f1-score": 0.8673611111111112,
+      "precision": 0.8841666666666667,
       "recall": 0.875,
       "support": 80.0
     }

artifacts/evaluation/latest/intent_type_difficulty_benchmark_report.json

@@ -2,9 +2,9 @@
   "accepted_accuracy": 0.9867,
   "accepted_coverage": 1.0,
   "accuracy": 0.9867,
-  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/intent_type_difficulty_benchmark_confusion_matrix.csv",
+  "confusion_matrix_path": "/content/agentic-intent-classifier/artifacts/evaluation/latest/intent_type_difficulty_benchmark_confusion_matrix.csv",
   "count": 150,
-  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/intent_type_benchmark.jsonl",
+  "dataset_path": "/content/agentic-intent-classifier/data/intent_type_benchmark.jsonl",
   "difficulty_breakdown": {
     "easy": {
       "accepted_accuracy": 1.0,

artifacts/evaluation/latest/intent_type_hard_cases_report.json

@@ -2,9 +2,9 @@
   "accepted_accuracy": 1.0,
   "accepted_coverage": 1.0,
   "accuracy": 1.0,
-  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/intent_type_hard_cases_confusion_matrix.csv",
+  "confusion_matrix_path": "/content/agentic-intent-classifier/artifacts/evaluation/latest/intent_type_hard_cases_confusion_matrix.csv",
   "count": 61,
-  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/hard_cases.jsonl",
+  "dataset_path": "/content/agentic-intent-classifier/data/hard_cases.jsonl",
   "fallback_rate": 0.0,
   "head": "intent_type",
   "macro_f1": 1.0,

artifacts/evaluation/latest/intent_type_test_confusion_matrix.csv

@@ -1,7 +1,7 @@
 ,informational,exploratory,commercial,transactional,support,personal_reflection,creative_generation,chit_chat,ambiguous,prohibited
 informational,8,0,0,0,0,0,0,0,0,0
 exploratory,0,1,0,0,0,0,0,0,0,0
-commercial,1,0,9,0,0,0,0,0,0,0
+commercial,0,0,10,0,0,0,0,0,0,0
 transactional,0,0,0,8,0,0,0,0,0,0
 support,0,0,0,0,2,0,0,0,0,1
 personal_reflection,0,0,0,0,0,5,0,0,0,0

artifacts/evaluation/latest/intent_type_test_report.json

@@ -1,15 +1,15 @@
 {
-  "accepted_accuracy": 0.9149,
+  "accepted_accuracy": 0.9362,
   "accepted_coverage": 1.0,
-  "accuracy": 0.9149,
-  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/intent_type_test_confusion_matrix.csv",
+  "accuracy": 0.9362,
+  "confusion_matrix_path": "/content/agentic-intent-classifier/artifacts/evaluation/latest/intent_type_test_confusion_matrix.csv",
   "count": 47,
-  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/test.jsonl",
+  "dataset_path": "/content/agentic-intent-classifier/data/test.jsonl",
   "fallback_rate": 0.0,
   "head": "intent_type",
-  "macro_f1": 0.9131,
+  "macro_f1": 0.9235,
   "per_class_metrics": {
-    "accuracy": 0.9148936170212766,
+    "accuracy": 0.9361702127659575,
     "ambiguous": {
       "f1-score": 0.875,
       "precision": 1.0,
@@ -23,9 +23,9 @@
       "support": 1.0
     },
     "commercial": {
-      "f1-score": 0.9,
-      "precision": 0.9,
-      "recall": 0.9,
+      "f1-score": 0.9523809523809523,
+      "precision": 0.9090909090909091,
+      "recall": 1.0,
       "support": 10.0
     },
     "creative_generation": {
@@ -41,15 +41,15 @@
       "support": 1.0
     },
     "informational": {
-      "f1-score": 0.8888888888888888,
-      "precision": 0.8,
+      "f1-score": 0.9411764705882353,
+      "precision": 0.8888888888888888,
       "recall": 1.0,
       "support": 8.0
     },
     "macro avg": {
-      "f1-score": 0.9130555555555555,
-      "precision": 0.9199999999999999,
-      "recall": 0.9344444444444445,
+      "f1-score": 0.9235224089635853,
+      "precision": 0.9297979797979797,
+      "recall": 0.9444444444444444,
       "support": 47.0
     },
     "personal_reflection": {
@@ -77,9 +77,9 @@
       "support": 8.0
     },
     "weighted avg": {
-      "f1-score": 0.916016548463357,
-      "precision": 0.9340425531914893,
-      "recall": 0.9148936170212766,
+      "f1-score": 0.9360614458549377,
+      "precision": 0.9511068128089405,
+      "recall": 0.9361702127659575,
       "support": 47.0
     }
   },

artifacts/evaluation/latest/intent_type_third_wave_cases_confusion_matrix.csv

@@ -1,11 +1,11 @@
 ,informational,exploratory,commercial,transactional,support,personal_reflection,creative_generation,chit_chat,ambiguous,prohibited
 informational,0,0,0,0,0,0,0,0,0,0
 exploratory,0,1,0,0,0,0,0,0,0,0
-commercial,1,0,11,0,0,0,0,0,0,0
+commercial,0,0,12,0,0,0,0,0,0,0
 transactional,0,0,0,0,0,0,0,0,0,0
 support,0,0,0,0,0,0,0,0,0,0
 personal_reflection,0,0,0,0,0,0,0,0,0,0
 creative_generation,0,0,0,0,0,0,1,0,0,0
 chit_chat,0,0,0,0,0,0,0,1,0,0
-ambiguous,1,0,1,0,0,0,0,0,8,0
+ambiguous,1,0,2,0,0,0,0,0,7,0
 prohibited,0,0,0,0,0,0,0,0,0,1

artifacts/evaluation/latest/intent_type_third_wave_cases_report.json

@@ -2,18 +2,18 @@
   "accepted_accuracy": 0.8846,
   "accepted_coverage": 1.0,
   "accuracy": 0.8846,
-  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/intent_type_third_wave_cases_confusion_matrix.csv",
+  "confusion_matrix_path": "/content/agentic-intent-classifier/artifacts/evaluation/latest/intent_type_third_wave_cases_confusion_matrix.csv",
   "count": 26,
-  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/third_wave_cases.jsonl",
+  "dataset_path": "/content/agentic-intent-classifier/data/third_wave_cases.jsonl",
   "fallback_rate": 0.0,
   "head": "intent_type",
-  "macro_f1": 0.8294,
+  "macro_f1": 0.8209,
   "per_class_metrics": {
     "accuracy": 0.8846153846153846,
     "ambiguous": {
-      "f1-score": 0.8888888888888888,
+      "f1-score": 0.8235294117647058,
       "precision": 1.0,
-      "recall": 0.8,
+      "recall": 0.7,
       "support": 10.0
     },
     "chit_chat": {
@@ -23,9 +23,9 @@
       "support": 1.0
     },
     "commercial": {
-      "f1-score": 0.9166666666666666,
-      "precision": 0.9166666666666666,
-      "recall": 0.9166666666666666,
+      "f1-score": 0.9230769230769231,
+      "precision": 0.8571428571428571,
+      "recall": 1.0,
       "support": 12.0
     },
     "creative_generation": {
@@ -47,9 +47,9 @@
       "support": 0.0
     },
     "macro avg": {
-      "f1-score": 0.5805555555555555,
-      "precision": 0.5916666666666666,
-      "recall": 0.5716666666666667,
+      "f1-score": 0.5746606334841629,
+      "precision": 0.5857142857142857,
+      "recall": 0.5700000000000001,
       "support": 26.0
     },
     "personal_reflection": {
@@ -77,8 +77,8 @@
       "support": 0.0
     },
     "weighted avg": {
-      "f1-score": 0.9188034188034189,
-      "precision": 0.9615384615384616,
+      "f1-score": 0.8966237382526975,
+      "precision": 0.9340659340659341,
       "recall": 0.8846153846153846,
       "support": 26.0
     }

artifacts/evaluation/latest/intent_type_train_report.json

@@ -1,11 +1,11 @@
 {
   "accepted_accuracy": 1.0,
-  "accepted_coverage": 1.0,
+  "accepted_coverage": 0.9945,
   "accuracy": 1.0,
-  "confusion_matrix_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/artifacts/evaluation/latest/intent_type_train_confusion_matrix.csv",
+  "confusion_matrix_path": "/content/agentic-intent-classifier/artifacts/evaluation/latest/intent_type_train_confusion_matrix.csv",
   "count": 183,
-  "dataset_path": "/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/train.jsonl",
-  "fallback_rate": 0.0,
+  "dataset_path": "/content/agentic-intent-classifier/data/train.jsonl",
+  "fallback_rate": 0.0055,
   "head": "intent_type",
   "macro_f1": 1.0,
   "per_class_metrics": {

artifacts/evaluation/latest/intent_type_val_confusion_matrix.csv

@@ -1,11 +1,11 @@
 ,informational,exploratory,commercial,transactional,support,personal_reflection,creative_generation,chit_chat,ambiguous,prohibited
 informational,8,0,0,0,0,0,0,0,0,0
 exploratory,0,1,0,0,0,0,0,0,0,0
-commercial,0,0,10,0,0,0,0,0,0,0
+commercial,0,1,9,0,0,0,0,0,0,0
 transactional,0,0,0,7,0,0,1,0,0,0
-support,0,0,0,0,2,0,0,0,0,1
+support,0,0,0,0,3,0,0,0,0,0
 personal_reflection,0,0,0,0,0,5,0,0,0,0
 creative_generation,0,0,0,0,0,0,1,0,0,0
 chit_chat,0,0,0,0,0,0,0,1,0,0
-ambiguous,0,0,0,0,0,0,0,0,9,0
+ambiguous,0,0,1,0,0,0,0,0,8,0
 prohibited,0,0,0,0,0,0,0,0,0,1