language:
- pt
license: apache-2.0
base_model:
- Cloudflare/clef-flash
library_name: peft
pipeline_tag: text-classification
tags:
- text-classification
- support-automation
- e-commerce
- portuguese
- ticket-routing
- decision-model
- multi-task
- lora
- multi-specialist
metrics:
- accuracy
- f1
model-index:
- name: Veredicto
results:
- task:
type: text-classification
metrics:
- type: accuracy
name: overall accuracy (human-curated ruler, seed-42 test split)
value: 0.7118
- type: accuracy
name: accuracy on the human-adjudicated hard core
value: 0.6895
- type: accuracy
name: urgency accuracy (same split)
value: 0.745
- type: f1
name: category macro-F1 (classes present in the test split)
value: 0.6018
Veredicto — 9B multi-specialist decision model for support tickets (PT-BR)
by Syvor AI (Raphael Pizzo)
Veredicto is the 9B member of the Syvor decision family. The current release — codenamed Trium — is a multi-specialist composition on a single Cloudflare/clef-flash backbone (9.4B, bf16, hybrid linear-attention, 128k context): one shared backbone, three specialist heads and one LoRA adapter, routed per question. It reads a Portuguese support conversation and answers three typed questions with probabilities:
| question | type | labels |
|---|---|---|
| category | choice (17) | reembolso, troca_devolucao, fraude_suspeita, prazo_envio, pagamento, frete, produto_defeito, chargeback, duvida_comercial, outro, incidente, whatsapp_meta, bug_produto, integracao, conta_config, plano_creditos, duvida_produto |
| decision | choice (5) | aprovar, rejeitar, escalar, executar_acao, pedir_mais_info |
| urgency | score (0–3) | baixa · moderada · alta · crÃtica |
Routing confidence is the minimum across the three answers: above a threshold (default 0.8) the case goes fully automatic, otherwise it escalates ("System 1 / System 2" architecture — Veredicto is the System 2 intermediate between the 322M Decisum and the 26B Imperium).
Architecture (Trium composition)
A week of challenge campaigns showed that no single head wins all three questions — but every question has a champion. Trium routes each question to its champion over the shared backbone:
| question | answered by |
|---|---|
| category | specialist head B2 (frozen-base checkpoint, step-2208) |
| decision | specialist head E1 (frozen-base checkpoint, step-2576) |
| urgency | LoRA adapter v4 (r=16, α=32; ~29M params) + release head |
This is inference-time composition, not a merged model: the merge experiment showed the specialist heads collapse on the adapted backbone (category 0.68 → 0.24), so the server keeps one backbone in memory and toggles the adapter per question group — 3 forward passes per case. The joint-schema head scores all options in a single pass per question, no generation.
Latency by hardware (full composition: 3 forwards per case, bf16, warm server)
| hardware | p50 | p95 | notes |
|---|---|---|---|
| NVIDIA RTX PRO 6000 (96 GB) | ~400 ms | ~738 ms | measured (100 cases, reproduced on two separate runs); ~147 ms per forward |
| AMD RX 7900 XTX (24 GB, ROCm) | ~947 ms | ~2,178 ms | measured (100 cases after 5-case warm-up, dedicated server); ~311 ms per forward; occasional multi-second outliers remain even after warm-up (see note) |
| AMD RX 7900 XT (20 GB, ROCm) | ~1,106 ms | ~2,630 ms | measured twice (100 cases after 5-case warm-up, reproduced); ~364 ms per forward; occasional multi-second outliers remain even after warm-up (see note) |
| Apple M2 Max (32 GB, MPS) | — | — | serving correctness verified on MPS; latency measurement deliberately not pursued |
24 GB Apple Silicon (e.g. M4 Pro): not supported for this artifact. bf16 (~19.5 GB) loads and answers correctly but the host swaps heavily and throughput collapses to unusable levels — we measured it and deliberately do not publish a latency number for a machine that cannot serve this model well. Use ≥ 32 GB unified memory, or wait for the quantized build (roadmap).
Cold-start / outlier note: the first case after boot is a cold-start
outlier (up to 120 s on ROCm) — warm the server with a few cases before
measuring or serving. Even after warm-up, one isolated multi-second
outlier (60 s) appeared per 100-case run on both AMD boards, likely
kernel autotuning on first encounter of an unusual sequence length; we
report p50/p95 as the contract numbers.
Memory requirement: ~19.5 GB of VRAM/RAM in bf16 (plan for 24 GB). 16 GB machines (e.g. M5 MacBook Air) and the 14 GB RAM / 24 GB VRAM Intel Arc Pro B60 box (Battlemage G21) do not fit this artifact without quantization — a quantized build is on the roadmap. All latency numbers in this card are measured, never estimated.
Benchmark
Official ruler (private, human-curated — 1,499 real cases, fixed seed-42 split, n=753 test)
The ruler was never used for training or fitting; all numbers at T=1, same split, same protocol, paired per-case predictions.
This release:
| metric | Trium (this release) | best single model (E1) |
|---|---|---|
| Overall accuracy (all questions) | 0.7118 | 0.6835 |
| Accuracy on the human-adjudicated hard core | 0.6895 | 0.6566 |
| Category accuracy | 0.7012 | 0.6773 |
| Decision accuracy | 0.6892 | 0.6892 |
| Urgency accuracy | 0.7450 | 0.6839 |
| Category macro-F1 (classes present in the test split) | 0.6018 | 0.5714 |
The hard core is the 497 human-adjudicated hard cases of the ruler
(subset nucleo, 1,491 of the 2,259 test answers); macro-F1 follows the
lineage's standard rule (mean F1 over every category appearing in the
split's gold or predictions — 13 here). Category accuracy/macro-F1 of
Trium come from its B2 head; decision from E1; urgency from the v4
adapter.
Head-to-head on the six metrics — the frontier teacher API and the untrained base, same ruler, same split, everything from paired per-case predictions:
| metric | Veredicto Trium | frontier API (JEV, teacher) | clef-flash base (zero-shot) |
|---|---|---|---|
| Overall accuracy | 0.7118 | 0.7127 | 0.4772 |
| Hard-core accuracy | 0.6895 | 0.6922 | 0.4681 |
| Category accuracy | 0.7012 | 0.7251 | 0.5817 |
| Decision accuracy | 0.6892 | 0.6892 | 0.2988 |
| Urgency accuracy | 0.7450 | 0.7238 | 0.5511 |
| Category macro-F1 | 0.6018 | 0.6718 | 0.4760 |
Read of the dispute: on overall and hard-core accuracy Trium and the
teacher tie (Δ −0.09 pp, CI [−1.86, +1.68]; hard core Δ −0.27 pp,
CI [−2.48, +1.95]), with an exact tie on decision and a +2.1 pp win on
urgency; the teacher keeps the edge on category (accuracy and macro-F1).
Against the untrained base, Trium wins every metric: +23.5 pp overall
(CI [+21.16, +25.72]), +22.1 pp on the hard core (CI [+19.25, +24.88]),
+39.0 pp on decision. (In the shipped base predictions, rotulo/pred
index the canonical business label order — validated by reproducing
0.5817/0.2988/0.5511 exactly from the gold.)
| system | category | decision | urgency | overall |
|---|---|---|---|---|
| JEV (frontier decision API, teacher/reference) | 0.7251 | 0.6892 | 0.7238 | 0.7127 |
| Veredicto Trium (this release) | 0.7012 | 0.6892 | 0.7450 | 0.7118 |
| Composition A (v6 head cat/dec + v4 adapter urgency) | 0.6826 | 0.6494 | 0.7450 | 0.6923 |
| Veredicto E1 (best single trained model) | 0.6773 | 0.6892 | 0.6839 | 0.6835 |
| Veredicto B2 | 0.7012 | 0.6680 | 0.6707 | 0.6799 |
| Veredicto B0 | 0.6853 | 0.6680 | 0.6786 | 0.6773 |
| Veredicto B1 | 0.6919 | 0.6746 | 0.6521 | 0.6729 |
| Veredicto D (relabel) | 0.6946 | 0.6786 | 0.6401 | 0.6711 |
| Veredicto E2F (relabel from scratch) | 0.6906 | 0.6866 | 0.6547 | 0.6773 |
| Veredicto v6 A1 | 0.6826 | 0.6494 | 0.6521 | 0.6614 |
| Decisum v10+swa (production champion, 322M ×2) †| 0.5604 | 0.6534 | 0.6932 | 0.6357 |
| clef-flash base, zero-shot (no training) | 0.5817 | 0.2988 | 0.5511 | 0.4772 |
| Veredicto v4 (legacy adapter alone) | 0.2470 | 0.1633 | 0.7450 | 0.3851 |
†Decisum re-measured in the same paired run (same 753 cases, same protocol); its earlier standalone measurement was 0.6339 overall — same model, run-to-run calibration noise.
Paired bootstrap (10k resamples by case, seed 42):
vs the untrained base (clef-flash zero-shot): +23.5 pp overall (0.4772 → 0.7118), 95% CI [+21.16, +25.72]; +39.0 pp on the decision question (0.2988 → 0.6892) — this training is what turns the base into a decision model.
vs the frontier API: statistical tie — Δ −0.10 pp, 95% CI [−1.86, +1.68]. Exact tie on decision, +2.11 pp win on urgency, −2.39 pp on category.
vs Decisum v10+swa (production champion): +7.6 pp overall.
vs composition A: Δ +1.95 pp, 95% CI [+1.11, +2.79].
vs v6: Δ +5.05 pp, 95% CI [+3.54, +6.60].
Serving parity: the production server answered all 2,259/2,259 test answers through its HTTP API byte-identical to the offline evaluation.
Reproduce every number in this card from the raw per-case predictions
shipped under eval/preds/ (ruler identities only, no raw text):
python3 scripts/auditoria_card.py --dir eval/preds --ouro <ouro_gold.jsonl>
Lineage in one paragraph
The legacy v4/v5 pipeline trained Veredicto as an end-to-end classifier and produced an accidental urgency specialist (0.745) with collapsed category/decision. An audit of the pipeline copies found target/option-id misalignment, a trained head omitted from adapter saves and dropped hard-label cases — the historical scores never isolated architecture from pipeline defects. A clean head-only rebuild on a frozen backbone (v6) refuted the architectural limit; a family of distillation/relabel ablations (B0–D, E1, E2F) pushed single-model quality to 0.6835 but none recovered urgency past the promotion gate; question routing (Trium) closed the gap with zero additional training. E2 also left a methodological finding: a distilled checkpoint cannot re-memorize hard labels under pure cross-entropy — the teacher's prior stays stuck in the weights.
Usage
The repo ships the three specialist heads, the v4 adapter and the audited
serving script. The base weights are not redistributed: download
Cloudflare/clef-flash once (its license applies) and point --base-path
at the local snapshot.
pip install torch transformers peft safetensors
python3 scripts/servir_veredicto_multi.py \
--checkpoint-cat heads/veredicto_b2_step-2208 \
--checkpoint-dec heads/veredicto_e1_step-2576 \
--base-path <local snapshot of Cloudflare/clef-flash> \
--adapter adapters/veredicto-lora-v4 \
--porta 8200 --device cuda # cuda (ROCm included) or mps (Apple Silicon ≥ 32 GB)
curl -s localhost:8200/decidir -H 'Content-Type: application/json' -d '{
"estado": "Cliente: meu canal de mensagens parou de enviar, a loja inteira está parada, estou perdendo vendas agora",
"limiar": 0.8
}'
Real captured response (from the 7900 XTX server; distribution tails condensed for readability):
{
"respostas": {
"categoria": {"resposta": "whatsapp_meta", "confianca": 0.7757,
"distribuicao": {"whatsapp_meta": 0.7757, "incidente": 0.1092,
"integracao": 0.0604, "conta_config": 0.0418,
"...": "..."}},
"decisao": {"resposta": "escalar", "confianca": 0.426,
"distribuicao": {"escalar": 0.426, "executar_acao": 0.2794,
"pedir_mais_info": 0.2176, "aprovar": 0.077,
"rejeitar": 0.0}},
"urgencia": {"resposta": "3 - crÃtica", "confianca": 0.9569,
"distribuicao": {"3 - crÃtica": 0.9569, "2 - alta": 0.0302,
"1 - moderada": 0.0088, "0 - baixa": 0.0041}}
},
"confianca_roteamento": 0.426,
"limiar": 0.8,
"destino": "system2",
"latencia_ms": 616
}
Note the routing behaviour: high category and urgency confidence but only
0.43 on the decision question → confianca_roteamento takes the minimum
across questions and the case is routed to a human (system2).
On Apple Silicon set
PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.9 PYTORCH_MPS_LOW_WATERMARK_RATIO=0.8
and raise the GPU wired limit — sudo sysctl iogpu.wired_limit_mb=28672
(32 GB machines; on 24 GB machines the same setting is required,
without it loading ~19.5 GB in bf16 aborts in the MPS allocator). Always
warm the server with a few cases before measuring or serving (first case
after boot is a cold-start outlier).
Training data
- Real anonymized support conversations from an e-commerce platform (PII scrubbed, credentials masked, no company/customer names).
- Public e-commerce complaint corpora.
- LLM-assisted labeling: probability distributions from large language models, audited and quality-controlled; the urgency specialist descends from the legacy v4 adapter distilled from the frontier teacher.
- Verified synthetic conversations targeting category tails and decision boundaries.
Raw data, benchmarks and label sets are not distributed.
Limitations
- Category tails: the frontier API still leads on data-poor categories; Trium closes most of the gap but trails by ~2.4 pp on category overall.
- 3× compute: three forwards per case vs one for a single-head model — the price of per-question specialization.
- Domain: PT-BR e-commerce support; out-of-domain use is unmeasured.
- Requires the base model's custom code and its joint-schema processor.
- Not intended for sensitive decisions (financial/legal) without human review.
Ethics & privacy
Trained on real support conversations under data minimization: PII removed, credentials masked before any internal or external use, no real company/platform names in any public artifact. The goal is faster, better routed support — not replacing human judgment on sensitive cases.
License & credits
- Apache 2.0. Base: Cloudflare/clef-flash — please follow the base model's license and citation.
- Heads, adapter, decision schema, evaluation protocol and serving stack: Raphael Pizzo / Syvor AI (2026).
- The full technical report (the Veredicto lineage article) is linked from
the project page;
REPRODUCAO.mdin this repo shows how to recompute every number from the shipped predictions.
Citation
@software{pizzo2026veredicto,
author = {Raphael Pizzo},
title = {Veredicto: 9B multi-specialist decision model for PT-BR support tickets},
year = {2026},
url = {https://huggingface.co/syvorai/veredicto},
license = {Apache-2.0}
}