intent-router-zh-setfit-v1

SetFit intent router for AI coding-agent traffic, fine-tuned from Qwen/Qwen3-Embedding-0.6B. It maps a user prompt to 21 intents across coding / ops / general_control plus out_of_scope, and ships with a v4 routing policy (temperature scaling + optional top-2 flips + precision-first keyword overrides).

基于 Qwen3-Embedding-0.6B 的中英双语意图路由器（21 类）。
主评测集 gold_test n=695；独立短指令 coding OOD n=360；action–object probe n=45。
发布数字（CPU 复算，与 metrics.json 一致）：closed-set 0.705 / routed v4 0.675 / domain 0.847。

Datasets & test sizes (release record)

Split / set	n	Role	Leakage / notes
Train (raw labeled)	3203	Contrastive body + linear head source	Pre-oversample distribution below
Body train (capped balanced)	3640	SetFit body	body_max_samples_per_label=250
Head train (oversampled)	4437	SetFitHead	min 150 samples/class
Calibration	473	Policy temperature / thresholds	metrics.json
Rule validation	416	Keyword mining validation	metrics.json
Gold test (release)	695	Primary metrics	intent-router/gold_test.jsonl
Gold test (legacy v1 report)	611	Historical baseline only	dataset_splits/gold_test.jsonl; overlap with 695 = 518 texts
Action–object probe	45	Held-out disambiguation	15 objects × 3
Independent coding OOD	360	Short-instruction stress test	9 coding intents × 40; 0 train/gold leakage

Gold test composition (n=695)

Domain (true label)	n	Share
coding (9 intents)	466	67.1%
ops (8 intents)	166	23.9%
general_control (3 intents)	56	8.1%
out_of_scope	7	1.0%

Length (characters): min 4 · median 290 · mean 1418 · max 36248.

Length bin	n
<100	240
100–800	180
800+	275

Per-class support on gold (closed-set evaluation order):

Intent	Domain	n
code_review	coding	149
code_modification	coding	84
code_explanation	coding	74
monitoring_query	ops	55
code_authoring	coding	50
code_repair	coding	38
code_search	coding	32
architecture_consultation	coding	31
context_specification	general_control	31
log_analysis	ops	26
deployment	ops	22
config_change	ops	19
security_operation	ops	15
general_inquiry	general_control	14
infrastructure_provisioning	ops	14
workflow_control	general_control	11
incident_response	ops	9
out_of_scope	fallback	7
pipeline_operation	ops	6
test_generation	coding	5
dependency_management	coding	3

Independent coding OOD (n=360)

Axis	Distribution
Labels	9 coding intents × 40 (balanced)
Language	zh 129 (35.8%) · en 158 (43.9%) · mixed 73 (20.3%)
Difficulty	easy 216 (60%) · medium 108 (30%) · hard 36 (10%)
Length	min 12 · median 204 · p90 922 · max 1195 · <100: 92 · 100–800: 160 · 800+: 108
Leakage	0 vs train (3203) and gold after text normalization

Action–object probe (n=45)

Held-out triples that keep the object fixed and vary the action intent (e.g. explain / search / monitor the same resource). Accuracy 0.978, macro-F1 0.741.

Release status

Domain	Status	Evidence on gold n=695
coding	Usable on long, agent-style inputs	Intent closed 0.725 (n=466); 800+ overall closed 0.782
ops	Conditionally usable	Intent closed 0.711 (n=166); rare classes still weak
general_control	Weak / limited	Intent closed 0.571 (n=56)
short standalone instructions	Not reliable	Coding OOD closed 0.117 / routed 0.203 (n=360)

Labels

Domain	Intents
coding (9)	code_authoring, code_modification, code_repair, code_review, code_explanation, test_generation, code_search, architecture_consultation, dependency_management
ops (8)	deployment, infrastructure_provisioning, monitoring_query, incident_response, pipeline_operation, config_change, security_operation, log_analysis
general_control (3)	context_specification, workflow_control, general_inquiry
fallback	out_of_scope

Training data (original train split, n=3203)

Label	Domain	Samples	Status
code_review	coding	741	adequate
code_modification	coding	420	adequate
code_explanation	coding	368	adequate
code_authoring	coding	249	adequate
code_repair	coding	187	adequate
code_search	coding	158	adequate
context_specification	general_control	155	adequate
architecture_consultation	coding	154	adequate
log_analysis	ops	129	marginal
out_of_scope	fallback	125	adequate
config_change	ops	94	marginal
general_inquiry	general_control	70	marginal
workflow_control	general_control	54	marginal
monitoring_query	ops	46	insufficient
pipeline_operation	ops	45	insufficient
test_generation	coding	45	insufficient
deployment	ops	36	insufficient
security_operation	ops	32	insufficient
incident_response	ops	32	insufficient
dependency_management	coding	32	insufficient
infrastructure_provisioning	ops	31	insufficient

Training recipe oversamples the head to min 150/class (n_head_train=4437) and uses a capped class-balanced contrastive body set (n_body_train=3640). Oversampling cannot invent evidence for long-tail classes.

Intended use

• Route prompts of an AI coding assistant to specialized handlers.
• Best on long, context-rich, agent-style prompts (system context, diffs, stack traces). On gold prompts ≥800 chars, closed-set accuracy is 0.782.
• Max input length for best quality: 1024 tokens (training setting).
• Not a general short-prompt router without keyword rules / OOS fallback.

Usage (SetFit / PyTorch)

import os
os.environ["SETFIT_DEVICE"] = "cpu"
import torch
from setfit import SetFitModel

model = SetFitModel.from_pretrained(
    "snival/intent-router-zh-setfit-v1",
    trust_remote_code=True,
)
model.to(torch.device("cpu"))

body = model.model_body
body.max_seq_length = 1024
body.tokenizer.model_max_length = 1024
transformer = body[0]
transformer.max_seq_length = 1024
transformer.tokenizer.model_max_length = 1024
transformer.tokenizer.padding_side = "left"

probas = model.predict_proba([
    "帮我 review 一下这个 PR，重点关注线程安全",
    "Traceback (most recent call last): ... NullPointerException at OrderService.java:148",
])

Framework-free path: encode with the body, then apply head_coef.npy / head_intercept.npy + softmax.

ONNX / ONNX Runtime (CPU)

Encoder graphs are shipped for edge/CPU deployment. The head is not fused into ONNX — apply head_coef.npy + head_intercept.npy after sentence_embedding. Details: ONNX_INFERENCE.md.

Artifact	Approx. size	Role
encoder.onnx + encoder.onnx.data	~4 MB + ~2.4 GB	FP32 reference (external data)
encoder-int8.onnx	~600 MB	Recommended CPU path (weight-only int8, single file)
encoder-woq8.onnx + encoder-woq8.onnx.data	~19 MB + ~596 MB	Alternative WOQ export
head_meta.json	small	Labels + head temperature / mode

I/O: input_ids / attention_mask (int64, [batch, sequence]) → sentence_embedding (float32, [batch, 1024]); pooling lasttoken, L2 normalize True, left padding, max_length=1024.

import json
from pathlib import Path

import numpy as np
import onnxruntime as ort
from huggingface_hub import snapshot_download
from transformers import AutoTokenizer

model_dir = Path(
    snapshot_download(
        "snival/intent-router-zh-setfit-v1",
        allow_patterns=[
            "encoder-int8.onnx",
            "head_coef.npy",
            "head_intercept.npy",
            "head_meta.json",
            "tokenizer*",
            "vocab.json",
            "merges.txt",
            "added_tokens.json",
            "special_tokens_map.json",
            "chat_template.jinja",
            "config.json",
        ],
    )
)

tok = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)
tok.padding_side = "left"
sess = ort.InferenceSession(
    str(model_dir / "encoder-int8.onnx"),
    providers=["CPUExecutionProvider"],
)
coef = np.load(model_dir / "head_coef.npy")
intercept = np.load(model_dir / "head_intercept.npy")
labels = json.loads((model_dir / "head_meta.json").read_text(encoding="utf-8"))["labels"]

enc = tok(
    ["帮我 review 一下这个 PR，重点关注线程安全"],
    padding=True,
    truncation=True,
    max_length=1024,
    return_tensors="np",
)
emb = sess.run(
    ["sentence_embedding"],
    {
        "input_ids": enc["input_ids"].astype("int64"),
        "attention_mask": enc["attention_mask"].astype("int64"),
    },
)[0]
logits = emb @ coef.T + intercept
probs = np.exp(logits - logits.max(axis=-1, keepdims=True))
probs /= probs.sum(axis=-1, keepdims=True)
print(labels[int(probs.argmax())], float(probs.max()))

End-to-end routing (recommended)

Ship files: labels.json, metrics.json, rejection_policy.json (v4).

1. Keyword hard-routing (657 rules in this release)
2. Optional top-2 flip rules for known near-neighbors
3. Temperature scaling (temperature ≈ 1.745)
4. Short-text / model-OOS → out_of_scope (short_text_max_chars=3; score-threshold rejection is off: max_probability=0.0)

Reference: routing_policy.decide_route(). On gold n=695, route reasons were:

Reason	Count
argmax (no override)	506
keyword_override	184
predicted_out_of_scope	5

Smoke examples (CPU)

Input	Top-1 (raw)	v4 route
帮我 review 一下这个 PR，重点关注线程安全	code_review ~97.6%	code_review (keyword_override)
traceback / NullPointerException style	often log_analysis	often code_repair (keyword_override)
short “写一个防抖函数…”	often out_of_scope / off-domain	policy may keep OOS or keyword-correct

Evaluation

All primary numbers below were recomputed on CPU from the shipped model + eval_proba_695_Qwen--Qwen3-Embedding-0.6B.npy and match metrics.json.

1) Gold test — n=695 (primary)

Metric	Closed-set	Routed (v4)
Accuracy	0.705	0.675
Macro-F1	0.536	0.504 (all) / 0.512 (in-scope)
Domain accuracy	0.850	0.847
OOS precision / recall / F1	—	0.400 / 0.286 / 0.333 (support 7)

By length (chars):

Length	n	Closed-set	Routed
<100	240	0.658	0.650
100–800	180	0.650	0.633
800+	275	0.782	0.724

Intent accuracy by true domain:

Domain	n	Closed-set	Routed
coding	466	0.725	0.691
ops	166	0.711	0.705
general_control	56	0.571	0.500
out_of_scope	7	0.286	0.286

Per-class closed-set (precision / recall / F1 / support):

Intent	P	R	F1	n
code_review	0.928	0.953	0.940	149
monitoring_query	0.852	0.945	0.897	55
security_operation	0.778	0.933	0.848	15
code_authoring	0.791	0.680	0.731	50
code_explanation	0.720	0.730	0.725	74
infrastructure_provisioning	0.818	0.643	0.720	14
deployment	0.750	0.682	0.714	22
code_modification	0.670	0.702	0.686	84
context_specification	0.605	0.742	0.667	31
log_analysis	0.533	0.615	0.571	26
code_search	0.652	0.469	0.545	32
incident_response	0.667	0.444	0.533	9
code_repair	0.696	0.421	0.525	38
architecture_consultation	0.486	0.548	0.515	31
workflow_control	0.429	0.545	0.480	11
out_of_scope	0.400	0.286	0.333	7
config_change	0.242	0.421	0.308	19
test_generation	0.500	0.200	0.286	5
general_inquiry	0.231	0.214	0.222	14
dependency_management	0.000	0.000	0.000	3
pipeline_operation	0.000	0.000	0.000	6

Top closed-set confusions:

Count	True → Pred
10	code_authoring → code_modification
9	code_modification → config_change
8	code_search → code_explanation
8	code_repair → log_analysis

Action–object probe (n=45): accuracy 0.978, macro-F1 0.741.

Calibration (policy metadata): after temperature scaling, ECE ≈ 0.055, NLL ≈ 1.103 (before: ECE ≈ 0.192 / NLL ≈ 1.466).

Prefer the 695-row gold set + metrics.json as the release record. The older 611-row split is not identical (overlap 518 texts).

2) Independent coding OOD — n=360 (short-instruction stress)

Zero leakage vs train/gold. Median length 204 chars (short-heavy). Source report: coding_test_report.md / coding_eval_results.json.

Metric	Closed-set	Routed
Accuracy	0.117	0.203
Macro-F1	0.134	0.246
Weighted-F1	0.134	0.246

Per-class routed F1 (best → worst): code_review 0.548 · code_repair 0.333 · code_search 0.256 · code_modification 0.254 · code_authoring 0.222 · test_generation 0.222 · architecture_consultation 0.188 · code_explanation 0.187 · dependency_management 0.000.

Slice	Routed acc
easy / medium / hard	0.181 / 0.241 / 0.222
zh / en / mixed	0.256 / 0.133 / 0.260

Routed gain is mostly keyword rules; a large share is honestly rejected or corrected rather than pure argmax. This set is the model’s weakest region.

3) CPU speed (Intel i9-14900KF, 32 threads; coding eval)

Metric	Value
Cold start (warm weights local)	~0.3 s
Single-sample latency p50 / p90 / p99	108 / 489 / 647 ms
Throughput (batch=8, n=360)	≈4.4 samples/s

Full cold from_pretrained on this machine is ~tens of seconds depending on disk/cache.

Limitations

• Short standalone prompts are OOD. Training is dominated by long agent-style prompts. Short imperatives collapse in embedding space; do not deploy as a general short-prompt router without short-instruction training data.
• Long-tail intents remain weak: dependency_management, test_generation, pipeline_operation (and other low-support classes).
• Cross-domain near-neighbors: code_repair ↔ log_analysis, code_modification ↔ config_change, code_authoring ↔ code_modification, code_search ↔ code_explanation.
• Keyword overrides help explicit patterns but can override a higher-probability neighbor; audit mined rules before production.
• Score-threshold rejection is disabled (max_probability=0.0); re-tune if you need aggressive abstention. OOS support on gold is only 7 rows.
• On gold, v4 routed accuracy is slightly below closed-set (−3.0pp): keywords fix specific phrases more than they lift overall accuracy.

Training

• Method: SetFit (contrastive body + linear SetFitHead)
• Base: Qwen/Qwen3-Embedding-0.6B, max_length=1024, pooling=lasttoken, padding_side=left
• Data: train 3203 → body 3640 (capped) / head 4437 (min 150/class); calibration 473; rule validation 416
• Head: lr=5e-3, l2=0.01, 8 epochs, head_class_weighting=none; body 2 epochs, 8 iterations; batch sizes [2, 64]; AMP + gradient checkpointing
• Device: CUDA (NVIDIA GeForce RTX 5090 D); inference reference target is CPU
• Keyword mining: 647 accepted at train time → 657 rules in shipped rejection_policy.json

Files

File	Purpose
model.safetensors + transformer/tokenizer configs	Fine-tuned embedding body
classification_head.pkl/.pt, model_head.pkl	SetFit head (native)
head_coef.npy, head_intercept.npy, head_meta.json	Head arrays + metadata
encoder.onnx + encoder.onnx.data	FP32 ONNX encoder
encoder-int8.onnx	Weight-only int8 ONNX (recommended CPU)
encoder-woq8.onnx + encoder-woq8.onnx.data	Alternative quantized ONNX
ONNX_INFERENCE.md	ONNX Runtime usage
labels.json	Labels, display names, intent→domain
rejection_policy.json	v4 routing policy
metrics.json	Training-time evaluation record (n_gold_test=695)
action_object_probe_metrics.json	Probe metrics (n=45)

Not uploaded by default (local eval assets): gold_test.jsonl (n=695), eval_proba_695_*.npy, train logs, conflict dumps.

Comparison vs first trained release (v1)

Baseline: post-training report on gold n=611 (2026-08-09).
Current: gold n=695 (metrics.json). Write-up: docs/v1_to_current_comparison.md.

Metric	v1 closed	v1 routed v3.2	Current closed	Current routed v4	Δ closed	Δ routed
Accuracy	0.558	0.584	0.705	0.675	+14.7pp	+9.0pp
Macro-F1	0.296	0.304	0.536	0.504	+0.240	+0.200
Domain Acc	—	0.818	—	0.847	—	+2.9pp
OOS F1	—	0.611	—	0.333	—	-0.278
Gold n	611	611	695	695	+84	+84

Length closed-set: <100 +22.8pp · 100–800 +25.8pp · 800+ +4.9pp.

On 518 overlapping texts present in both gold sets, current closed-set is 0.710 (≈ +15.2pp vs v1 closed 0.558) — main gain is classifier quality, not only gold composition. Independent short-instruction coding OOD (n=360) is unchanged (closed 0.117 / routed 0.203).

How this card was verified

• Recomputed closed-set + v4 routed metrics on gold n=695 from the shipped probability cache; matches metrics.json (closed 0.7050, routed 0.6748, domain 0.8475).
• Gold label / length / domain counts taken from intent-router/gold_test.jsonl (n=695).
• Coding OOD numbers from coding_eval_results.json / coding_test_report.md (n=360; prior CPU↔CUDA argmax agreement 100% on gold).
• Action–object probe from action_object_probe_metrics.json (n=45).
• ONNX artifacts documented in ONNX_INFERENCE.md (fp32 / int8 / woq8).