🚀 Enterprise NLP Query Router

A lightweight, high-performance query routing microservice powered by fine-tuned XLM-RoBERTa, post-hoc temperature calibration, and rule-based fast paths. It classifies enterprise incoming requests into 11 specialized service domains or flags them as out-of-scope / uncertain.

📌 Overview & Features

11 Domain Service Coverage: Intelligently classifies user queries into core business functions or flags out-of-scope requests.
Calibrated Confidence: Calibrated via validation temperature scaling ($T = 0.7468$), ensuring probability outputs reflect true likelihood with minimal calibration error ($ECE = 0.0141$).
Dual-Gated Decision Engine: Enforces strict confidence thresholds (CONF_TH >= 0.90) and margin thresholds (MARGIN_TH >= 0.10) between Top-1 and Top-2 predictions to prevent hallucinated routings on ambiguous queries.
Deterministic Rule Intercept: Immediate regex bypass for high-priority keyword triggers (e.g., regex \bsmartsheet\b routes with confidence 1.0 without model overhead).
Single-File Microservice: Self-contained api.py built on FastAPI with single-query and batch (up to 64 queries) endpoints.

📊 Benchmark & Evaluation

Evaluated on an independent test set of 1,251 samples.

Key Evaluation Metrics

Metric	Score	Note
Accuracy	94.56%	Overall correct routing rate
Macro F1	94.32%	Balanced across all 12 classes (95% CI: `[92.92%, 95.64%]`)
Weighted F1	94.51%	Volume-weighted precision & recall balance
Top-2 Accuracy	98.88%	Correct domain present in top 2 predictions
Top-3 Accuracy	99.68%	Correct domain present in top 3 predictions
Matthews Correlation (MCC)	0.9395	High quality multiclass correlation
Expected Calibration Error (ECE)	0.0141	Calibrated posterior probabilities
Log Loss	0.2070	Multi-class cross entropy loss

Visualizations

1. Confusion Matrix

Confusion matrix across all 11 enterprise service domains and the Other (Out-of-Scope) category:

2. Performance Dashboard

Comprehensive metrics dashboard highlighting per-class accuracy, precision, recall, and calibration curves:

🏷️ Service Domain Taxonomy

The router categorizes incoming requests into the following 12 classes:

Domain	Description / Scope	Example Query
Finance	Budgets, revenue, forecasting, payments, invoices, accounting	"How do I submit an invoice for vendor payment?"
Healthcare	Patients, clinical workflows, hospitals, medical records	"Update patient clinical record access policies."
HR	Recruitment, onboarding, payroll, leave, performance reviews	"How do I automate employee onboarding?"
IT Operations	Servers, infrastructure, monitoring, incidents, helpdesk	"Our production server CPU usage spiked to 100%."
Customer Experience	Customer feedback, surveys, support tickets, satisfaction	"Analyze our latest customer CSAT ratings."
Document Intelligence	OCR, extracting tables and forms from PDFs and scanned files	"Extract line-item tables from this invoice PDF."
Supply Chain	Inventory, shipments, logistics, procurement, demand planning	"Optimize warehouse re-order points for next quarter."
Data & Analytics	Dashboards, BI reporting, data analysis, KPI visualization	"Build a dashboard to monitor sales conversion trends."
GenAI Platform	LLM applications, prompts, chatbots, RAG pipelines, model serving	"How do we set up RAG with chunking for documentation?"
Smartsheet	Smartsheet tool, automations, project sheets, workspace tracking	"Create an automated Smartsheet workflow for deadlines."
Agentic AI	Autonomous AI agents, multi-agent workflows, tool execution	"Deploy an autonomous agent to triage support tickets."
Other	Out-of-scope chit-chat, greetings, or completely unrelated topics	"What's the weather like in Seattle today?"

⚙️ Routing Decision Logic

The router returns one of three statuses based on the prediction scores:

Status	Returned Label	Meaning & Action
`routed`	`<Service Name>`	Confident prediction ($p \ge 0.90$ and margin $\ge 0.10$). Route user directly to service.
`out_of_scope`	`null`	Query identified as off-topic or chit-chat ($p(\text{Other}) \ge 0.50$).
`uncertain`	`null`	Low confidence or ambiguous query. Ask the user for clarification or display `top_k` candidates.

Note: If a query mentions "Smartsheet", it is immediately routed via deterministic rule with source: "rule" and confidence: 1.0.

🚀 Setup & Installation

Directory Structure

.
├── api.py
├── requirements.txt
├── README.md
└── models/router/
    ├── benchmark/
    │   ├── 1_confusion.png
    │   ├── 2_dashboard.png
    │   ├── errors.csv
    │   └── metrics.json
    ├── calib.json
    ├── config.json
    ├── model.safetensors
    ├── tokenizer.json
    ├── tokenizer_config.json
    └── training_args.bin

Installation

Clone the repository & create environment:

git clone https://huggingface.co/Diksha-xvg/nlp_Router.git
cd nlp_Router
python3 -m venv .venv
source .venv/bin/activate

Install dependencies:

pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install -r requirements.txt

Start the API server:

uvicorn api:app --host 0.0.0.0 --port 8000

Interactive Swagger documentation is available at http://localhost:8000/docs.

📡 API Usage

1. Single Query Classification

curl -X POST http://localhost:8000/classify \
  -H "Content-Type: application/json" \
  -d '{"text": "How do I automate employee onboarding?"}'

Response:

{
  "label": "HR",
  "status": "routed",
  "source": "model",
  "confidence": 0.9731,
  "top_k": [
    {"label": "HR", "score": 0.9731},
    {"label": "Data & Analytics", "score": 0.0102},
    {"label": "Finance", "score": 0.0061}
  ]
}

2. Batch Classification (Up to 64 queries)

curl -X POST http://localhost:8000/classify/batch \
  -H "Content-Type: application/json" \
  -d '{
    "texts": [
      "Why is our Kubernetes node restarting?",
      "Can we extract structured tables from invoice PDFs?",
      "Tell me a joke"
    ]
  }'

Response:

{
  "results": [
    {
      "label": "IT Operations",
      "status": "routed",
      "source": "model",
      "confidence": 0.9854,
      "top_k": [{"label": "IT Operations", "score": 0.9854}]
    },
    {
      "label": "Document Intelligence",
      "status": "routed",
      "source": "model",
      "confidence": 0.9812,
      "top_k": [{"label": "Document Intelligence", "score": 0.9812}]
    },
    {
      "label": null,
      "status": "out_of_scope",
      "source": "model",
      "confidence": 0.9621,
      "top_k": [{"label": "Other", "score": 0.9621}]
    }
  ]
}

3. Health Check

curl http://localhost:8000/health

Response:

{
  "status": "ok",
  "labels": 12
}

🔧 Environment Configuration

You can customize runtime behavior using environment variables:

Variable	Default	Description
`MODEL_DIR`	`models/router`	Path to directory containing model weights and `calib.json`
`CONF_TH`	`0.90`	Minimum top-1 probability threshold required to route
`MARGIN_TH`	`0.10`	Minimum difference between top-1 and top-2 probability
`OOS_TH`	`0.50`	Minimum "Other" probability to classify as `out_of_scope`

Example:

CONF_TH=0.85 MARGIN_TH=0.15 uvicorn api:app --port 8000

☁️ Hugging Face Upload Command

To sync your model and benchmark assets to Hugging Face Hub:

HF_HUB_DISABLE_XET=1 hf upload Diksha-xvg/nlp_Router . \
  --include "api.py" \
  --include "requirements.txt" \
  --include "README.md" \
  --include "models/router/**"

🛡️ Production Notes

CPU Optimization: Pre-calibrated temperature scaling enables fast single-digit millisecond latency per query on standard modern CPUs without requiring GPU acceleration.
Security & Networking: Designed for internal service mesh or VPC deployments. Place behind an API gateway (e.g., Kong, Traefik, or NGINX) for rate limiting and authentication.
Calibration File: Ensure calib.json is located in the model directory alongside model.safetensors and tokenizer configs for startup verification.

Downloads last month: -; Downloads are not tracked for this model. How to track