qwen-intent-classifier

This model is a fine-tuned version of Qwen/Qwen2.5-1.5B on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.0422
• Accuracy: 0.9910
• F1: 0.9910

Model description

This model classifies customer support queries into 5 intents before routing them to the appropriate pipeline in the SmartAssist AI agent system.

Intent	Description
product_info	Questions about products, pricing, availability
order_status	Tracking, delivery, shipping inquiries
refund_request	Returns, refunds, cancellations
complaint	Negative experiences, disputes
escalate	Requests for human agent

Intended uses

This model is part of the SmartAssistant AI customer support agent. It acts as the first routing layer — classifying each user query before deciding whether to call the RAG pipeline or escalate to a human agent.

Training and evaluation data

The model was trained on ~5,000 balanced examples merged from two datasets:

• Bitext Customer Support Dataset — 27k examples across 27 intents, mapped to our 5 categories
• Banking77 (mteb) — 10k examples from the financial/banking support domain

Both datasets were balanced to 999 examples per intent to avoid class imbalance, resulting in ~4,995 training examples split 80/20 for train/test.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	F1
0.0059	1.0	500	0.0405	0.9910	0.9910
0.0017	2.0	1000	0.0671	0.9890	0.9890
0.0006	3.0	1500	0.0422	0.9910	0.9910

Framework versions

• PEFT 0.18.1
• Transformers 5.0.0
• Pytorch 2.10.0+cu128
• Datasets 4.0.0
• Tokenizers 0.22.2

How to Use

from transformers import AutoTokenizer, AutoModelForSequenceClassification
from peft import PeftModel
import torch
import torch.nn.functional as F

LABELS = ["complaint", "escalate", "order_status", "product_info", "refund_request"]
id2label = {i: l for i, l in enumerate(LABELS)}

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-1.5B")
tokenizer.pad_token = tokenizer.eos_token

base_model = AutoModelForSequenceClassification.from_pretrained(
    "Qwen/Qwen2.5-1.5B",
    num_labels=5,
    ignore_mismatched_sizes=True,
)
model = PeftModel.from_pretrained(base_model, "manarsaber11/qwen-intent-classifier")
model.eval()

def classify(text):
    inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128, padding=True)
    with torch.no_grad():
        logits = model(**inputs).logits
    probs = F.softmax(logits, dim=-1).squeeze()
    confidence, pred_id = probs.max(dim=-1)
    return id2label[pred_id.item()], confidence.item()

print(classify("Where is my order?"))
# ('order_status', 0.9998)