admesh
/

agentic-intent-classifier

@@ -1,492 +1,208 @@
-# Agentic Intent Classifier
-`agentic-intent-classifier` is a multi-head query classification stack for conversational traffic.
-It currently produces:
-- `intent.type`
-- `intent.subtype`
-- `intent.decision_phase`
-- `iab_content`
-- calibrated confidence per head
-- combined fallback / policy / opportunity decisions
-The repo is beyond the original v0.1 baseline. It now includes:
-- shared config and label ownership
-- reusable model runtime
-- calibrated confidence and threshold gating
-- combined inference with fallback/policy logic
-- request/response validation in the demo API
-- repeatable evaluation and regression suites
-- full-TSV IAB taxonomy retrieval support through tier4
-- a local embedding index for taxonomy-node retrieval over IAB content paths
-- a separate synthetic full-intent-taxonomy augmentation dataset for non-IAB heads
-- a dedicated intent-type difficulty dataset and held-out benchmark with `easy`, `medium`, and `hard` cases
-- a dedicated decision-phase difficulty dataset and held-out benchmark with `easy`, `medium`, and `hard` cases
-Generated model weights are intentionally not committed.
-## Current Taxonomy
-### `intent.type`
-- `informational`
-- `exploratory`
-- `commercial`
-- `transactional`
-- `support`
-- `personal_reflection`
-- `creative_generation`
-- `chit_chat`
-- `ambiguous`
-- `prohibited`
-### `intent.decision_phase`
-- `awareness`
-- `research`
-- `consideration`
-- `decision`
-- `action`
-- `post_purchase`
-- `support`
-### `intent.subtype`
-- `education`
-- `product_discovery`
-- `comparison`
-- `evaluation`
-- `deal_seeking`
-- `provider_selection`
-- `signup`
-- `purchase`
-- `booking`
-- `download`
-- `contact_sales`
-- `task_execution`
-- `onboarding_setup`
-- `troubleshooting`
-- `account_help`
-- `billing_help`
-- `follow_up`
-- `emotional_reflection`
-### `iab_content`
-- candidates are derived from every row in [data/iab-content/Content Taxonomy 3.0.tsv](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/data/iab-content/Content%20Taxonomy%203.0.tsv)
-- retrieval output supports `tier1`, `tier2`, `tier3`, and optional `tier4`
-## What The System Does
-- runs three classifier heads:
-  - `intent_type`
-  - `intent_subtype`
-  - `decision_phase`
-- resolves `iab_content` through a local embedding index over taxonomy nodes plus generic label/path reranking
-- applies calibration artifacts when present
-- computes `commercial_score`
-- applies fallback when confidence is too weak or policy-safe blocking is required
-- emits a schema-validated combined envelope
-## What The System Does Not Do
-- it is not a multi-turn memory system
-- it is not a production-optimized low-latency serving path
-- it is not yet trained on large real-traffic human-labeled intent data
-- combined decision logic is still heuristic, even though it is materially stronger than the original baseline
-## Project Layout
-- [config.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/config.py): labels, thresholds, artifact paths, model paths
-- [model_runtime.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/model_runtime.py): shared calibrated inference runtime
-- [combined_inference.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/combined_inference.py): composed system response
-- [inference_intent_type.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/inference_intent_type.py): direct `intent_type` inference entrypoint
-- [inference_iab_classifier.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/inference_iab_classifier.py): direct supervised `iab_content` inference entrypoint
-- [schemas.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/schemas.py): request/response validation
-- [demo_api.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/demo_api.py): local validated API
-- [iab_taxonomy.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/iab_taxonomy.py): full taxonomy parser/index
-- [iab_classifier.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/iab_classifier.py): supervised IAB runtime with taxonomy-aware parent fallback
-- [iab_retrieval.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/iab_retrieval.py): optional shadow retrieval baseline
-- [training/build_full_intent_taxonomy_dataset.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/build_full_intent_taxonomy_dataset.py): separate synthetic intent augmentation dataset
-- [training/build_intent_type_difficulty_dataset.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/build_intent_type_difficulty_dataset.py): extra `intent_type` augmentation plus held-out difficulty benchmark
-- [training/build_decision_phase_difficulty_dataset.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/build_decision_phase_difficulty_dataset.py): extra `decision_phase` augmentation plus held-out difficulty benchmark
-- [training/build_subtype_difficulty_dataset.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/build_subtype_difficulty_dataset.py): extra `intent_subtype` augmentation plus held-out difficulty benchmark
-- [training/build_subtype_dataset.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/build_subtype_dataset.py): subtype dataset generation from existing corpora
-- [training/train_iab.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/train_iab.py): train the supervised IAB classifier head
-- [training/build_iab_taxonomy_embeddings.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/build_iab_taxonomy_embeddings.py): build local IAB node embedding artifacts
-- [training/run_full_training_pipeline.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/training/run_full_training_pipeline.py): full multi-head training/calibration/eval pipeline
-- [evaluation/run_evaluation.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/evaluation/run_evaluation.py): repeatable benchmark runner
-- [evaluation/run_regression_suite.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/evaluation/run_regression_suite.py): known-failure regression runner
-- [evaluation/run_iab_mapping_suite.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/evaluation/run_iab_mapping_suite.py): IAB behavior-lock regression runner
-- [evaluation/run_iab_quality_suite.py](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/evaluation/run_iab_quality_suite.py): curated IAB quality-target runner
-- [known_limitations.md](/Users/manikumargouni/Desktop/AdMesh/protocol/agentic-intent-classifier/known_limitations.md): current gaps and caveats
-## Quickstart: Run From Hugging Face
-Download the trained bundle and run inference in three lines — no local training required.
-```python
-import sys
-from huggingface_hub import snapshot_download
-# Download the full bundle (models + calibration + code)
-local_dir = snapshot_download(
-    repo_id="admesh/agentic-intent-classifier",
-    repo_type="model",
-)
-sys.path.insert(0, local_dir)
-# Import and instantiate
-from pipeline import AdmeshIntentPipeline
-clf = AdmeshIntentPipeline()
-# Classify
-import json
-result = clf("Which laptop should I buy for college?")
-print(json.dumps(result, indent=2))
-```
-Or use the one-liner factory method:
 ```python
-from pipeline import AdmeshIntentPipeline   # after sys.path.insert above
-clf = AdmeshIntentPipeline.from_pretrained("admesh/agentic-intent-classifier")
-result = clf("I need a CRM for a 5-person startup")
 ```
-Batch mode and custom thresholds are also supported:
 ```python
-# Batch
 results = clf([
     "Best running shoes under $100",
-    "How does gradient descent work?",
     "Buy noise-cancelling headphones",
 ])
-# Custom confidence thresholds
 result = clf(
-    "Buy noise-cancelling headphones",
     threshold_overrides={"intent_type": 0.6, "intent_subtype": 0.35},
 )
 ```
-Verify artifacts and run a smoke test from the CLI:
-```bash
-cd "<local_dir>"
-python3 training/pipeline_verify.py
-python3 combined_inference.py "Which CRM should I buy for a 3-person startup?"
-```
-Pin a specific revision for reproducibility:
-```python
-local_dir = snapshot_download(
-    repo_id="admesh/agentic-intent-classifier",
-    repo_type="model",
-    revision="0584798f8efee6beccd778b0afa06782ab5add60",
-)
-```
 ---
-## Setup (for local training)
-```bash
-python3 -m venv .venv
-source .venv/bin/activate
-pip install -r agentic-intent-classifier/requirements.txt
-```
-## Inference (local training path)
-Run one query locally:
-```bash
-cd agentic-intent-classifier
-python3 training/train_iab.py
-python3 training/calibrate_confidence.py --head iab_content
-python3 combined_inference.py "Which CRM should I buy for a 3-person startup?"
-```
-Run only the `intent_type` head:
-```bash
-cd agentic-intent-classifier
-python3 inference_intent_type.py "best shoes under 100"
-```
-Run the demo API:
-```bash
-cd agentic-intent-classifier
-python3 demo_api.py
-```
-Example request:
 ```bash
-curl -sS -X POST http://127.0.0.1:8008/classify \
-  -H 'Content-Type: application/json' \
-  -d '{"text":"I cannot log into my account"}'
 ```
-Infra endpoints:
-```bash
-curl -sS http://127.0.0.1:8008/health
-curl -sS http://127.0.0.1:8008/version
-```
-Train only the IAB classifier head:
-```bash
-cd agentic-intent-classifier
-python3 training/train_iab.py
-python3 training/calibrate_confidence.py --head iab_content
-```
-The online `iab_content` path now uses the compact supervised classifier. Retrieval is still available as an optional shadow baseline.
-Build the optional retrieval shadow index:
-```bash
-cd agentic-intent-classifier
-python3 training/build_iab_taxonomy_embeddings.py
-```
-By default the shadow retrieval path uses `Alibaba-NLP/gte-Qwen2-1.5B-instruct`. The retrieval runtime applies the model's query-side instruction format and last-token pooling, matching the Hugging Face usage guidance. If you want to point retrieval at a different embedding model, set `IAB_RETRIEVAL_MODEL_NAME_OVERRIDE` before building the index.
-Open-source users can swap in their own embedding model, but the contract is:
-- query embeddings and taxonomy-node embeddings must be produced by the same model and model revision
-- after changing models, you must rebuild `artifacts/iab/taxonomy_embeddings.pt`
-- the repository only tests and supports the default model path out of the box
-- not every Hugging Face embedding model is drop-in compatible with this runtime; some require custom pooling, query instructions, or `trust_remote_code`
-Example override:
-```bash
-cd agentic-intent-classifier
-export IAB_RETRIEVAL_MODEL_NAME_OVERRIDE=mixedbread-ai/mxbai-embed-large-v1
-python3 training/build_iab_taxonomy_embeddings.py
-```
-This writes:
-- `artifacts/iab/taxonomy_nodes.json`
-- `artifacts/iab/taxonomy_embeddings.pt`
-## Training
-### Full local pipeline
-```bash
-cd agentic-intent-classifier
-python3 training/run_full_training_pipeline.py
-```
-This pipeline now does:
-1. build separate full-intent-taxonomy augmentation data
-2. build separate `intent_type` difficulty augmentation + benchmark
-3. train `intent_type`
-4. build subtype corpus
-5. build separate `intent_subtype` difficulty augmentation + benchmark
-6. train `intent_subtype`
-7. build separate `decision_phase` difficulty augmentation + benchmark
-8. train `decision_phase`
-9. train `iab_content`
-10. calibrate all classifier heads, including `iab_content`
-11. run regression/evaluation unless `--skip-full-eval` is used
-### Build datasets individually
-Separate full-intent augmentation:
-```bash
-cd agentic-intent-classifier
-python3 training/build_full_intent_taxonomy_dataset.py
-```
-Intent-type difficulty augmentation and benchmark:
-```bash
-cd agentic-intent-classifier
-python3 training/build_intent_type_difficulty_dataset.py
-```
-Decision-phase difficulty augmentation and benchmark:
-```bash
-cd agentic-intent-classifier
-python3 training/build_decision_phase_difficulty_dataset.py
-```
-Subtype difficulty augmentation and benchmark:
-```bash
-cd agentic-intent-classifier
-python3 training/build_subtype_difficulty_dataset.py
-```
-Subtype dataset:
-```bash
-cd agentic-intent-classifier
-python3 training/build_subtype_dataset.py
-```
-IAB embedding index:
-```bash
-cd agentic-intent-classifier
-python3 training/build_iab_taxonomy_embeddings.py
-```
-### Train heads individually
-```bash
-cd agentic-intent-classifier
-python3 training/train.py
-python3 training/train_subtype.py
-python3 training/train_decision_phase.py
-```
-### Calibration
-```bash
-cd agentic-intent-classifier
-python3 training/calibrate_confidence.py --head intent_type
-python3 training/calibrate_confidence.py --head intent_subtype
-python3 training/calibrate_confidence.py --head decision_phase
-```
-## Evaluation
-Full evaluation:
-```bash
-cd agentic-intent-classifier
-python3 evaluation/run_evaluation.py
-```
-Known-failure regression:
-```bash
-cd agentic-intent-classifier
-python3 evaluation/run_regression_suite.py
-```
-IAB behavior-lock regression:
-```bash
-cd agentic-intent-classifier
-python3 evaluation/run_iab_mapping_suite.py
-```
-IAB quality-target evaluation:
-```bash
-cd agentic-intent-classifier
-python3 evaluation/run_iab_quality_suite.py
-```
-Threshold sweeps:
-```bash
-cd agentic-intent-classifier
-python3 evaluation/sweep_intent_threshold.py
-```
-Artifacts are written to:
-- `artifacts/calibration/`
-- `artifacts/evaluation/latest/`
-## Google Colab
-Use Colab for the full retraining pass if local memory is limited.
-Clone once:
-```bash
-%cd /content
-!git clone https://github.com/GouniManikumar12/agentic-intent-classifier.git
-%cd /content/agentic-intent-classifier
 ```
-If the repo is already cloned and you want the latest code, pull manually:
-```bash
-!git pull origin main
-```
-Full pipeline:
-```bash
-!python training/run_full_training_pipeline.py
-```
-If full evaluation is too heavy for the current Colab runtime:
-```bash
-!python training/run_full_training_pipeline.py \
-  --iab-embedding-batch-size 32 \
-  --skip-full-eval
 ```
-Then run eval separately after training:
-```bash
-!python evaluation/run_regression_suite.py
-!python evaluation/run_iab_mapping_suite.py
-!python evaluation/run_iab_quality_suite.py
-!python evaluation/run_evaluation.py
 ```
-## Current Saved Metrics
-Generate fresh metrics with:
-```bash
-cd agentic-intent-classifier
-python3 evaluation/run_evaluation.py
 ```
-Do not treat any checked-in summary as canonical unless it was regenerated after the current code and artifacts were built. The IAB path is now retrieval-based, so older saved reports from the deleted hierarchy stack are not meaningful.
-## Latency Note
-`combined_inference.py` is a debugging/offline path, not a production latency path.
-Current production truth:
-- per-request CLI execution is not a sub-50ms architecture
-- production serving should use a long-lived API process with preloaded models
-- if sub-50ms becomes a hard requirement, the serving path will need:
-  - persistent loaded models
-  - runtime optimization
-  - likely fewer model passes or a shared multi-head model
-## Current Status
-Current repo status:
-- full 10-class `intent.type` taxonomy is wired
-- subtype and phase heads are present
-- difficulty benchmarks are wired for `intent_type`, `intent_subtype`, and `decision_phase`
-- full-TSV IAB taxonomy retrieval is wired through tier4
-- separate full-intent augmentation dataset is in place
-- evaluation/runtime memory handling is improved for large IAB splits
-The main remaining gap is not basic infrastructure anymore. It is improving real-world robustness, especially for:
-- `decision_phase`
-- `intent_subtype`
-- confidence quality on borderline commercial queries
-- real-traffic supervision beyond synthetic data

+---
+language:
+- en
+library_name: transformers
+pipeline_tag: text-classification
+base_model: distilbert-base-uncased
+metrics:
+- accuracy
+- f1
+tags:
+- intent-classification
+- multitask
+- iab
+- conversational-ai
+- adtech
+- calibrated-confidence
+license: apache-2.0
+---
+# admesh/agentic-intent-classifier
+Production-ready intent + IAB classifier bundle for conversational traffic.
+Combines multitask intent modeling, supervised IAB content classification, and per-head confidence calibration to support safe monetization decisions in real time.
+## Links
+- Hugging Face: https://huggingface.co/admesh/agentic-intent-classifier
+- GitHub: https://github.com/GouniManikumar12/agentic-intent-classifier
+## What It Predicts
+| Field | Description |
+|---|---|
+| `intent.type` | `commercial`, `informational`, `navigational`, `transactional`, … |
+| `intent.subtype` | `product_discovery`, `comparison`, `how_to`, … |
+| `intent.decision_phase` | `awareness`, `consideration`, `decision`, … |
+| `iab_content` | IAB Content Taxonomy 3.0 tier1 / tier2 / tier3 labels |
+| `component_confidence` | Per-head calibrated confidence with threshold flags |
+| `system_decision` | Monetization eligibility, opportunity type, policy |
+---
+## Deployment Options
+### 1. `transformers.pipeline()` — one line anywhere
 ```python
+from transformers import pipeline
+clf = pipeline(
+    "admesh-intent",
+    model="admesh/agentic-intent-classifier",
+    trust_remote_code=True,
+)
+result = clf("Which laptop should I buy for college?")
 ```
+Batch and custom thresholds:
 ```python
+# batch
 results = clf([
     "Best running shoes under $100",
+    "How does TCP work?",
     "Buy noise-cancelling headphones",
 ])
+# custom confidence thresholds
 result = clf(
+    "Buy headphones",
     threshold_overrides={"intent_type": 0.6, "intent_subtype": 0.35},
 )
 ```
 ---
+### 2. HF Inference Endpoints (managed, deploy to AWS / Azure / GCP)
+1. Go to https://ui.endpoints.huggingface.co
+2. **New Endpoint** → select `admesh/agentic-intent-classifier`
+3. Framework: **PyTorch** — Task: **Text Classification**
+4. Enable **"Load with trust_remote_code"**
+5. Deploy
+The endpoint serves the same `pipeline()` interface above via REST:
 ```bash
+curl https://<your-endpoint>.endpoints.huggingface.cloud \
+  -H "Authorization: Bearer $HF_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"inputs": "Which laptop should I buy for college?"}'
 ```
+---
+### 3. HF Spaces (Gradio / Streamlit demo)
+```python
+# app.py for a Gradio Space
+import gradio as gr
+from transformers import pipeline
+clf = pipeline(
+    "admesh-intent",
+    model="admesh/agentic-intent-classifier",
+    trust_remote_code=True,
+)
+def classify(text):
+    return clf(text)
+gr.Interface(fn=classify, inputs="text", outputs="json").launch()
 ```
+---
+### 4. Local / notebook via `snapshot_download`
+```python
+import sys
+from huggingface_hub import snapshot_download
+local_dir = snapshot_download(
+    repo_id="admesh/agentic-intent-classifier",
+    repo_type="model",
+)
+sys.path.insert(0, local_dir)
+from pipeline import AdmeshIntentPipeline
+clf = AdmeshIntentPipeline()
+result = clf("I need a CRM for a 5-person startup")
 ```
+Or the one-liner factory:
+```python
+from pipeline import AdmeshIntentPipeline
+clf = AdmeshIntentPipeline.from_pretrained("admesh/agentic-intent-classifier")
 ```
+---
+## Example Output
+```json
+{
+  "model_output": {
+    "classification": {
+      "iab_content": {
+        "taxonomy": "IAB Content Taxonomy",
+        "taxonomy_version": "3.0",
+        "tier1": {"id": "552", "label": "Style & Fashion"},
+        "tier2": {"id": "579", "label": "Men's Fashion"},
+        "mapping_mode": "exact",
+        "mapping_confidence": 0.73
+      },
+      "intent": {
+        "type": "commercial",
+        "subtype": "product_discovery",
+        "decision_phase": "consideration",
+        "confidence": 0.9549,
+        "commercial_score": 0.656
+      }
+    }
+  },
+  "system_decision": {
+    "policy": {
+      "monetization_eligibility": "allowed_with_caution",
+      "eligibility_reason": "commercial_discovery_signal_present"
+    },
+    "opportunity": {"type": "soft_recommendation", "strength": "medium"}
+  },
+  "meta": {
+    "system_version": "0.6.0-phase4",
+    "calibration_enabled": true,
+    "iab_mapping_is_placeholder": false
+  }
+}
+```
+## Reproducible Revision
+```python
+from huggingface_hub import snapshot_download
+local_dir = snapshot_download(
+    repo_id="admesh/agentic-intent-classifier",
+    repo_type="model",
+    revision="0584798f8efee6beccd778b0afa06782ab5add60",
+)
 ```
+## Included Artifacts
+| Path | Contents |
+|---|---|
+| `multitask_intent_model_output/` | DistilBERT multitask weights + tokenizer |
+| `iab_classifier_model_output/` | IAB content classifier weights + tokenizer |
+| `artifacts/calibration/` | Per-head temperature + threshold JSONs |
+| `pipeline.py` | `AdmeshIntentPipeline` (transformers.Pipeline subclass) |
+| `combined_inference.py` | Core inference logic |
+## Notes
+- `trust_remote_code=True` is required because this model uses a custom multi-head architecture that does not map to a single standard `AutoModel` checkpoint.
+- `meta.iab_mapping_is_placeholder: true` means IAB artifacts were missing or skipped; train and calibrate IAB for full production accuracy.
+- For long-running servers, instantiate once and reuse — models are cached in memory after the first call.