mmbert-small-support-triage

Routes a customer message to one of 27 request types in 11 departments of an online shop, or to out_of_scope when it is not a support request. Ten languages: en, pl, de, fr, es, it, nl, pt, cs, sv. The first 16 of mmBERT-small's 22 layers, fine-tuned; ONNX weight-only int8 for transformers.js, a 55 MB download (52 MB weights, 3 MB tokenizer). For phones and low-memory devices there is an extra-light version, mmbert-small-support-triage-extra-light. The model was trained on normalised text: lower-case, no punctuation except apostrophes, single spaces. Normalise before calling it, as below.

Code, data pipeline and benchmark: github.com/balticml/support-triage.

Use

import { pipeline } from "@huggingface/transformers";
const classify = await pipeline("text-classification", "balticml/mmbert-small-support-triage", { dtype: "q8" });
const normalise = (t) => t.toLowerCase().replace(/[^\P{P}']/gu, "").replace(/\s+/g, " ").trim();
await classify(normalise("Hi, where is my order #48213? It was supposed to arrive on Monday."), { top_k: 3 });

Results

	test split, mean of 10 locales	160 hand-written messages	not-a-request recall (false reject)	CPU latency
this model, 16 layers, 49M params, int8 52 MB	99.4	96.2	98.9 (0.10)	5.7 ms (ORT, 4 threads)
mmBERT-base, same recipe without the hand-written training messages	99.4	75.6	99.6 (0.14)	31.7 ms (PyTorch fp32)
Qwen 2.5 7B instruct (int4, Ollama), zero-shot with the label list	73.0	86.9	—	91 ms (GPU, 25 per request)

Accuracy in %. The test split is English Bitext requests (2,569) plus 1,080 machine-translated requests per other locale. The hand-written messages were written by the author of the training messages (disjoint scenarios), so 96.2 is an upper bound for a real inbox. int8 agrees with PyTorch fp32 on 400 of 400 parity messages. A hosted LLM at list price would cost about $35 per million messages of the hand-written length (estimate from measured prompt lengths).

In Chrome (WASM, one thread) a page with the model loaded holds about 240 MB, and it does not grow with use; up to v1.2.0, with the full tokenizer, it was about 310 MB.

The tokenizer (since v1.3.0)

The model only knows the 46,681 tokens seen in training; every other id becomes [UNK] inside the graph. Since v1.3.0 the tokenizer keeps just those tokens, every token BPE forms on the way to one of them, the byte-fallback and special tokens, and the merges between them: 51,032 of 256,000 tokens and 128,416 of 580,604 merges, 3 MB instead of 34 MB, ids unchanged. Any text whose full tokenization uses only known tokens comes out identical; a word the model never saw is split into known pieces instead of becoming [UNK]. Measured with the int8 model on 500 test requests and 500 not-a-request sentences per locale: test 99.26 (full tokenizer 99.28), hand-written 96.9 (96.9), not-a-request recall 98.6 (98.4). v1.2.0 keeps the full tokenizer.

Training

Bitext customer support (English, CDLA-Sharing-1.0), machine-translated with MADLAD-400 3B into nine languages; FLORES-200 and Tatoeba sentences as out_of_scope; 1,465 hand-written messages. 70% of the training rows are wrapped into a message (greeting, order number or name, sign-off), 40% of requests get an unrelated sentence attached, 30% get a typo or a swapped word. 4 epochs, lr 5e-5, batch 64, label smoothing 0.1, dropout 0.1, out-of-scope loss weight 0.3, seed 42; recipe in configs/mmbert-small.yaml.

Limits

Nine of the ten test languages are machine translations; the 160 hand-written messages are the only measurement on human text. One e-commerce taxonomy: product questions, warranty or B2B topics have no class. One label per message. Pruned vocabulary: words unseen in training are read as smaller known pieces.

Licences

Model: Apache-2.0. mmBERT: Apache-2.0. Bitext: CDLA-Sharing-1.0. MADLAD-400: Apache-2.0. FLORES-200: CC BY-SA 4.0. Tatoeba: CC BY 2.0.