Decisum v10+swa — decision model for e-commerce support tickets (PT-BR)

by Syvor AI (Raphael Pizzo)

Decisum v10+swa is the production champion of the Decisum lineage: an ensemble of two lightweight classifiers (322M params each) that read a Portuguese e-commerce support conversation and answer three typed questions with calibrated probabilities:

question	type	labels
category	choice (17)	reembolso, troca_devolucao, fraude_suspeita, prazo_envio, pagamento, frete, produto_defeito, chargeback, duvida_comercial, outro, incidente, whatsapp_meta, bug_produto, integracao, conta_config, plano_creditos, duvida_produto
decision	choice (5)	aprovar, rejeitar, escalar, executar_acao, pedir_mais_info
urgency	score (0–3)	baixa · moderada · alta · crítica

The ensemble averages the temperature-scaled probability distributions of both members (checkpoints/v10 and checkpoints/swa) and is the strongest configuration we measured — it emerged from a challenge campaign of nine challengers as the new champion (see Benchmark).

Routing confidence is the minimum across the three answers: above a threshold (default 0.8) the case goes fully automatic, otherwise it is escalated to a bigger model or a human ("System 1 / System 2" architecture). This model runs in production as the System 1 stage of a real ticket-triage pipeline as the always-on first stage.

Architecture

• Ensemble members: two jhu-clsp/mmBERT-base encoders (multilingual ModernBERT, 322M, 256k vocab, 8k context) with option-markers heads: each question is answered by scoring option markers in the text — one forward pass per question, no generation. Members were trained on the same recipe (v10: hybrid corpus + KL corpus; swa: same data + Stochastic Weight Averaging over the last 25% of training), making them diverse enough to gain from averaging.
• Calibration: per-question temperature scaling per member (temperaturas.pt, calibracao.json inside each checkpoint); the ensemble averages the calibrated distributions.
• Footprint: 2 × 1.2 GB fp32 weights; ~2×16 ms per question on GPU — still System 1 territory; runs on CPU too.

Latency by hardware (ensemble, all 3 questions per case)

hardware	ensemble	single member	notes
NVIDIA RTX PRO 6000 (Blackwell)	~56 ms	~28 ms	measured; fastest
AMD RX 7900 XTX (ROCm)	~111 ms	~55 ms	measured; production serving
AMD RX 7900 XT (ROCm)	~130 ms	~65 ms	measured
Intel Arc Pro B60 (XPU)	~140 ms	~69 ms	measured; numeric parity with ROCm; serves the production mirror (v10+swa)
Apple M2 Max (MPS)	~168 ms	~84 ms	measured; fastest Apple — memory-bandwidth-bound workload
Apple M4 Pro (MPS)	~205 ms	~103 ms	measured
Apple M5 (MPS)	~339 ms	~165 ms	measured
Apple M4 Pro (CPU)	~405 ms	~202 ms	measured
Apple M2 Max (CPU)	~596 ms	~299 ms	measured
Apple M5 (CPU)	~612 ms	~305 ms	measured
AMD Threadripper 2970WX (24C, x86 CPU)	~2,900 ms	~1,450 ms	measured; stock torch CPU wheel (24 threads) — the old ROCm container's CPU path was 12× slower

All measurements are for the full ensemble (2× 322M models, 3 forward passes each, 6 forward passes total per case). Single-member = one model, 3 passes. ROCm / XPU / Apple rows were all measured with the same script (scripts/bench_latencia_ensemble.py, torch 2.14, batch 1) on a representative ~430-token PT-BR support conversation; the RTX PRO 6000 row comes from an earlier, lighter workload. On Apple Silicon this encoder is memory-bandwidth-bound, which is why the M2 Max (400 GB/s) edges out the newer M4 Pro and M5 on the GPU path.

Benchmark

Official ruler (private, human-curated — 1,499 real cases, fixed seed-42 split)

The evaluation benchmark was never used for training; temperature scaling follows the documented protocol (fit on the calibration split).

This release:

metric	v10+swa (this model)	best single member (swa)
Overall accuracy (all questions)	0.6339	0.6242
Accuracy on the human-adjudicated hard core	0.6170	0.6150
Category accuracy	0.5618	0.5551
Decision accuracy	0.6481	0.6295
Urgency accuracy	0.6919	0.6879
Category macro-F1 (17 classes)	0.4301	0.4693

Per-question accuracy of the members (test split): v10 — category 0.5471 / decision 0.6441 / urgency 0.6786; swa — 0.5551 / 0.6295 / 0.6879.

The challenge campaign — ten challengers, one champion

Every challenger was evaluated on the same ruler with the same protocol (same seed, same split, temperature scaling on the calibration split):

challenger	idea	overall acc	verdict
self-training (born-again)	retrain on the champion's own predictions	0.6087	best hard-core ever (0.6016) but tail recall collapsed
alternative label source (26B Imperium)	relabel training data with our own 26B model	0.6184	strong overall but loses category-tail recall
training stack (focal + R-Drop + FGM + SWA)	regularize/optimize the full training step	0.5644	over-regularized (−4.8 pp); but SWA inside it was gold
active-learning slice (+6k uncertain cases)	add uncertainty×disagreement-selected cases	0.5985	degraded category-tail performance
Veredicto (9B sibling)	scale up to a 9B backbone	0.3838	rejected as classifier; urgency record 0.745
Veredicto 9B v5 (clean retrain)	dedup the data, retrain from scratch	0.3316	legacy pipeline; architecture not isolated (category 0.097)
DAPT encoder (1.08M-doc MLM)	domain-adaptive pretraining before fine-tuning	0.4188	over-trained encoder → catastrophic forgetting (−20.5 pp); DAPT+stack canceled
urgency distillation into 322M (v13urg)	transfer the 9B urgency prior via KL (α=1.0, 125k cases)	0.6122	rejected — urgency −3.3 pp vs swa, macro-F1 collapsed; teacher prior contradicts 94% of inherited "critical" labels
alternative ensemble combinations	average challenger + champion probs	≤ 0.6130	none beats v10+swa
SWA-only ablation	isolate SWA from the failed stack	0.6242	best single model ever;
v10+swa ensemble	average v10 + swa probs	0.6339	CHAMPION (+1.1 pp over best single)

The consistent negative results are themselves the finding: past this point, neither scale-of-labels, nor self-training, nor training-stack optimization, nor domain-adaptive pretraining moved the ruler — the open front is category-tail data and cross-architecture knowledge transfer (see Limitations).

Local update — 20 October 2026: an audit of the observed v4/v5 pipeline copies found targets misaligned with reordered option_ids, a trained head omitted from adapter saves, and hard-label category cases omitted when probs_alvo was absent. The historical scores therefore do not establish an architectural limit; the contribution of each defect was not quantified, and the historical remote head was not recovered. A clean rebuild — the Veredicto v6 A1 complete checkpoint (step-2208, selected on dev), trained head_only with a frozen backbone — went to the same official ruler (753 seed-42 test cases, T=1, no fitting or calibration on the gold set). The full lineage picture:

system	category	decision	urgency	overall
JEV (frontier API, teacher)	0.7251	0.6892	0.7238	0.7127
Veredicto 9B (v6 A1, unpublished)	0.6826	0.6494	0.6521	0.6614
Decisum v10+swa (this model)	0.5618	0.6481	0.6919	0.6339
Veredicto 9B backbone (zero-shot)	0.5817	0.2988	0.5511	0.4772
Veredicto 9B (v4)	0.2430	0.1633	0.7450	0.3838
Veredicto 9B (v5)	0.0969	0.1700	0.7278	0.3316

That is +27.8 pp over v4 and +18.4 pp over the zero-shot base, refuting the absolute category/decision limitation claimed above (the effect of each individual fix is not isolated). On category, the 9B student now beats this champion by 12.1 pp and sits 4.3 pp from the teacher itself; overall it beats the champion by 2.8 pp. The weak spot moved to urgency: 65.2% vs 69.2% — a 4.0 pp regression, beyond the 2 pp promotion margin, which blocks promotion. The v6 checkpoint is not yet published and is distinct from the legacy v4/v5 adapter; point comparisons against historical references carry no paired confidence interval. No production promotion or switch was made.

Second update — 20 October 2026 (night): the reverse transfer also left the paper and failed the ruler. The v13urg challenger distilled the Veredicto urgency teacher (the 0.745 adapter above) into the 322M over 125k corpus cases with kl_alpha=1.0, trained with the exact champion SWA recipe. Same ruler, same paired run (probability averaging, no temperature scaling — hence the champion ensemble reads 0.6357 here vs 0.6339 in the calibrated evaluation):

system	category	decision	urgency	overall
v10+swa (champion ensemble)	0.5604	0.6534	0.6932	0.6357
v10+v13urg (ensemble)	0.5644	0.6560	0.6799	0.6335
swa (single)	0.5551	0.6295	0.6879	0.6242
v13urg (single)	0.5299	0.6521	0.6547	0.6122

Rejected on all fronts: urgency drops 3.3 pp single and 1.3 pp in ensemble, and multi-task interference drags even category down (macro-F1 0.327 vs 0.469 for swa) — the same failure mode as the active-learning slice. Root-cause hypothesis: inflated urgency labels — the teacher agrees with 95.6% of the corpus "low" labels but only 6.2% of the inherited "critical" ones, so with kl_alpha=1.0 the student trained entirely against a prior contradicting nearly every hard urgency label. A retry would need kl_alpha < 1 or distillation restricted to high teacher/label agreement. The checkpoint is archived for the record and production (v10+swa) remains untouched.

Reference-system agreement (context)

Labeling the same 30,195 public-complaint cases with both the frontier decision API (JEV) and our 26B Imperium: 79.2% agreement on category, 46.8% on decision, 45.1% on urgency — decision/urgency are partially subjective even between strong reference systems, which bounds how much any supervised model can learn from either.

Head-to-head vs. the frontier decision API (JEV)

Full official ruler (same seed-42 test split, n=753 cases, one scoring pass of the API):

system	category	decision	urgency	overall
JEV (frontier API)	0.7251	0.6892	0.7238	0.7127
Decisum v10+swa (this model)	0.5618	0.6481	0.6919	0.6339
Veredicto 9B (urgency specialist)	0.2430	0.1633	0.7450	0.3838

The API leads overall (−7.9 pp vs v10+swa) — concentrated on category tails. Notably, our 9B urgency specialist beats the API on urgency itself.

An older, deliberately tail-weighted 76-case stratified sample of the hard core showed the same ordering (JEV 0.754 / chat judge 0.711 / single member ~0.465 on its out-of-train subset) — high variance, kept for the record.

Why deploy this instead of the API, then

	JEV (decision API)	Decisum v10+swa
Latency per case	seconds	~100 ms (3 questions × 2 members, GPU)
Cost per 1k cases	~US$0.08 (measured)	0 (local)
Data leaves your infra	yes	no
Calibrated probabilities	yes	yes (temperature-scaled, averaged)
Deployable offline / on-prem	no	yes (CPU-capable)

In production, Decisum runs as the always-on System 1, outperforming JEV on cost and latency by orders of magnitude while delivering comparable quality for the confident majority of cases.

Usage

The repo ships both ensemble members and the inference scripts (the heads are custom, so inference goes through the included scripts/):

checkpoints/
  v10/        model.safetensors, cabeca_decisao.pt, temperaturas.pt, calibracao.json, tokenizer…

scripts/
  servir_ensemble.py       # HTTP API — the ensemble (GET /saude, POST /decidir)
  servir_decisor.py        # HTTP API — single member
  inferencia_decisor.py    # DecisorCPU class + batch CLI (single member)
  treinar_decisor.py       # model class + training (reference)
  chamados_para_decisor.py # vocabularies (single source of truth)
  bench_latencia_ensemble.py # latency benchmark used in the table above

Serve the ensemble (recommended)

pip install torch transformers
python scripts/servir_ensemble.py --checkpoints checkpoints/v10 checkpoints/swa --porta 8100

curl -s localhost:8100/decidir -H 'Content-Type: application/json' -d '{
  "estado": "Cliente: meu canal de mensagens parou de enviar, a loja inteira está parada, estou perdendo vendas agora",
  "limiar": 0.8
}'

Real response for the request above (device auto-detected: cuda → mps → cpu):

{
  "respostas": {
    "categoria": {"resposta": "outro", "confianca": 0.2387,
                  "distribuicao": {"outro": 0.2387, "prazo_envio": 0.1719,
                                   "duvida_comercial": 0.1529, "whatsapp_meta": 0.0266}},
    "decisao":   {"resposta": "pedir_mais_info", "confianca": 0.7493},
    "urgencia":  {"resposta": "1 - moderada", "confianca": 0.6295}
  },
  "confianca_roteamento": 0.2387,
  "limiar": 0.8,
  "destino": "system2"
}

destino is the routing decision: "automatico" (all three answers above the threshold — act without review) or "system2" (escalate). The example shows a classic pattern: a category doubt (0.24) drags the minimum confidence down, so an operationally critical ticket still gets escalated.

Single-member triage in Python

import sys; sys.path.insert(0, "scripts")   # modules live in scripts/
from inferencia_decisor import DecisorCPU

dec = DecisorCPU("checkpoints/swa")   # auto-loads temperature scaling
r = dec.prever(conversation, {
    "tipo": "choice",
    "instrucao": "Qual a decisão tomar para este caso?",
    "opcoes": ["aprovar", "rejeitar", "escalar", "executar_acao", "pedir_mais_info"]})
print(r["resposta"], r["confianca"])

Batch scoring a JSONL of tickets

python scripts/inferencia_decisor.py --checkpoint checkpoints/swa \
    --casos meus_chamados.jsonl --limiar 0.8

Training data

• Real anonymized support conversations from an e-commerce platform (PII scrubbed, credentials masked, no company/customer names).

• Public e-commerce complaint corpora (marketplace ticket classification).

• LLM-assisted labeling: decision/urgency labels enriched with calibrated probability distributions from large language models, audited and quality-controlled.

• Verified synthetic conversations targeting category tails and decision boundaries.

Raw data, benchmarks and label sets are not distributed.

Limitations

• Category confusion axis: the hardest distinctions concentrate in bug_produto × duvida_produto × conta_config × incidente.
• Data-poor tails: duvida_comercial and fraude_suspeita have few examples, which is reflected in their scores on the benchmark.
• Ensemble latency: 2× a single member; both members ship in the repo if you prefer half the latency in exchange for ~1 pp of accuracy.
• Urgency headroom: our 9B sibling (Veredicto) reaches 0.745 urgency accuracy, suggesting room to grow via per-question knowledge transfer (work in progress).
• Domain: PT-BR e-commerce support. Calibration guarantees hold only in-domain.
• Not intended for sensitive decisions (financial/legal) without human review.

Ethics & privacy

Trained on real support conversations under data minimization: PII removed, credentials masked before any internal or external use, no real company/platform names in any public artifact. The goal is faster, better routed support — not replacing human judgment on sensitive cases.

License & credits

• Apache 2.0 (compatible with mmBERT-base's license).
• Base: mmBERT (multilingual ModernBERT) — please cite the original mmBERT and ModernBERT works when deriving from this model.
• Model, data work and training: Raphael Pizzo (2026).
• The full technical report (v3→v10+swa lineage, evaluation protocol, the nine-challenger campaign, negative results on scaling) is in preparation and will be linked here.

Citation (provisional)

@software{pizzo2026decisum,
  author  = {Raphael Pizzo},
  title   = {Decisum v10+swa: ensemble decision model for PT-BR e-commerce support tickets},
  year    = {2026},
  url     = {https://huggingface.co/syvorai/decisum},
  license = {Apache-2.0}
}