File size: 3,923 Bytes
6784fa4 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 | ---
language:
- en
- hi
- multilingual
license: apache-2.0
library_name: transformers
pipeline_tag: text-classification
tags:
- query-classification
- intent-detection
- memory-scope
- modernbert
- onnx
- quantized
metrics:
- accuracy
- f1
model_name: Query Scope Classifier (ModernBERT-base)
---
# Multi-lingual Query Scope Classifier (`addyo07/query-scope-classifier`)
A production-grade, fast, multi-lingual single-pass sequence classifier fine-tuned from `answerdotai/ModernBERT-base` to categorize incoming user queries into 4 distinct scope categories across English, Devanagari Hindi, and Hinglish.
## 🏷️ 4-Class Taxonomy
1. **`ChitChat`** (Label `0`): Casual greetings, small talk, AI identity questions, emotional banter.
2. **`User`** (Label `1`): Personal facts, user preferences, memory updates, user profile instructions.
3. **`Domain`** (Label `2`, **Primary Default**): Code execution, math formulas, general domain task queries, technical instructions.
4. **`Temporal`** (Label `3`): Time-sensitive queries, schedules, dates, past session history, reminders.
---
## 📊 Performance & SLA Benchmarks
- **Base Architecture**: `answerdotai/ModernBERT-base` (149M parameters, RoPE, Unpadded FlashAttention-2).
- **Holdout Test Accuracy**: **96.18%** across 2,201 holdout samples.
- **Macro F1 Score**: **0.9619**
- **Calibrated Non-Default Precision**: **98.01%** at confidence threshold tau* = 0.81 (with automatic safe fallback to Domain when uncertain).
- **Quantized INT8 ONNX File Size**: **143.67 MB**
### Per-Class Recall Breakdown
| Scope Class | Recall | Precision | F1-Score |
|---|---|---|---|
| **ChitChat** | **98.00%** | **98.50%** | **0.9825** |
| **Temporal** | **97.28%** | **97.80%** | **0.9754** |
| **User** | **95.27%** | **97.73%** | **0.9648** |
| **Domain** (Default) | **94.18%** | **95.20%** | **0.9469** |
---
## 📁 Repository Structure
```
.gitattributes
README.md
model/
onnx/
config.json
model_quantized.onnx # 143.67 MB Dynamic INT8 ONNX model
pytorch/
config.json
model.safetensors # 571 MB PyTorch BFloat16 weights
tokenizer.json
tokenizer_config.json
scripts/ # Full fine-tuning, dataset audit & quantization pipeline
```
---
## 💻 Python / PyTorch Usage
```python
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
MODEL_NAME = "addyo07/query-scope-classifier"
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, subfolder="model/pytorch")
model = AutoModelForSequenceClassification.from_pretrained(MODEL_NAME, subfolder="model/pytorch")
labels = ["ChitChat", "User", "Domain", "Temporal"]
query = "aaj sham ko mera schedule kya hai?"
inputs = tokenizer(query, return_tensors="pt")
with torch.no_grad():
logits = model(**inputs).logits
probs = torch.softmax(logits, dim=-1)
pred_idx = torch.argmax(probs, dim=-1).item()
print(f"Predicted Scope: {labels[pred_idx]} (Confidence: {probs[0][pred_idx].item():.4f})")
```
---
## ⚡ ONNX Runtime Usage (Fast CPU Inference)
```python
import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("addyo07/query-scope-classifier", subfolder="model/pytorch")
session = ort.InferenceSession("model/onnx/model_quantized.onnx", providers=["CPUExecutionProvider"])
query = "Remind me to submit the quarterly tax report tomorrow at 5pm"
inputs = tokenizer(query, return_tensors="np", max_length=64, truncation=True)
onnx_inputs = {
"input_ids": inputs["input_ids"].astype(np.int64),
"attention_mask": inputs["attention_mask"].astype(np.int64)
}
outputs = session.run(None, onnx_inputs)
logits = outputs[0][0]
probs = np.exp(logits) / np.sum(np.exp(logits))
pred_id = np.argmax(probs)
labels = ["ChitChat", "User", "Domain", "Temporal"]
print(f"Scope: {labels[pred_id]}, Confidence: {probs[pred_id]:.4f}")
```
|