intent-crossencoder-deberta-v3-base

Cross-encoder reranker for enterprise intent detection (DLP / security). Fine-tuned from cross-encoder/nli-deberta-v3-base on a synthetic intent-detection dataset.

Intended use

Binary classification: given a (user_input, intent_description) pair, predict whether the user input matches the intent. Designed as Stage-2 in a cascading firewall: Stage-1 (fast heuristic) → this model (reranker) → Stage-2 LLM (Qwen).

Input format

user_input [SEP] intent_description

Performance (held-out test set, threshold=0.3)

Metric	Value
Recall	0.9978
Precision	0.9966
F1	0.9972
AUC-ROC	1.0000
PR-AUC	1.0000
Best threshold (F1-optimal)	0.9699 → F1=0.9983

Training config

Parameter	Value
Base model	cross-encoder/nli-deberta-v3-base
Batch size	16
Grad accum steps	4
Effective batch	64
Learning rate	1e-05
Label smoothing	0.05
Warmup ratio	0.06
Max sequence length	256
Early stopping	recall@0.3 (patience=3)
Epochs trained	10
Training time	13.5 min

Inference snippet

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id  = "aryasuneesh-quilr/intent-crossencoder-deberta-v3-base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model     = AutoModelForSequenceClassification.from_pretrained(model_id)
model.eval()

def score(user_input: str, intent_description: str) -> float:
    pair = f"{user_input} [SEP] {intent_description}"
    enc  = tokenizer(pair, return_tensors="pt", truncation=True, max_length=256)
    with torch.no_grad():
        logits = model(**enc).logits
    return torch.softmax(logits, dim=1)[0, 1].item()   # P(match)

# Example
s = score(
    "Our AWS_SECRET_ACCESS_KEY was found in a public repo",
    "Identify exposure of authentication credentials or API keys"
)
print(f"Match probability: {s:.4f}")   # use threshold 0.9699 for best F1

Files in this repo

File	Description
model.safetensors	HF-native weights
best_model.pt	Raw PyTorch state_dict (for resuming training)
training_config.json	Full hyperparameter record
metrics/	Per-epoch + test-set evaluation CSVs

---

Generated 2026-02-23 08:35 UTC by ablation_reranker_training.py