ner-bert-lenerbr-onnx-int8

ONNX export of pierreguillou/ner-bert-base-cased-pt-lenerbr, graph-fused and dynamically quantized to int8, for CPU inference under ONNX Runtime. No PyTorch, no transformers — the runtime is onnxruntime + tokenizers.

No weights were retrained or fine-tuned. This is a format conversion of an existing model, plus quantization of its MatMul weights.

Licensing: deliberately no license: field — see Provenance and licensing before redistributing.

Files

file	size	sha256
model.onnx	179.3 MB	360e0669d9a75358e796178e69f1592f212883a1edc4a46de5bd03a57d1997aa
tokenizer.json	677.9 KB	45ccb3e514665f3b11b5e2a8e1a3e29765d94330bc7232aa46c4cdc485bc8881
labels.json	275 B	c0d966b01bc360473ced7847e98f203d9374a4ff08a09d2fef7ca8e43f6c7ab8

labels.json is the id2label map lifted out of the source model's config.json, so the runtime needs neither transformers nor a config parser.

Inputs: input_ids, attention_mask, token_type_ids (int64, [batch, seq], both axes dynamic). Output: logits, [batch, seq, 13]. Max sequence length 512, including [CLS]/[SEP] — BERT position embeddings, unchanged.

Labels

LeNER-Br's scheme, unchanged: O, and B-/I- pairs for PESSOA, ORGANIZACAO, TEMPO, LOCAL, LEGISLACAO, JURISPRUDENCIA.

How it was produced

export (attn_implementation="eager", opset 17)
  -> onnxruntime.transformers.optimizer.optimize_model(model_type="bert")
  -> quantize_dynamic(op_types_to_quantize=["MatMul"],
                      per_channel=True, reduce_range=False,
                      weight_type=QuantType.QUInt8)

Three choices in there are load-bearing:

attn_implementation="eager". transformers ≥5 defaults to SDPA attention, which exports as a graph shape ONNX Runtime's BERT fusion does not pattern-match. Fusion then silently no-ops — it returns an unfused model without raising. Exporting eager is what makes fusion apply at all. Verify by asserting the fused op counts (BiasGelu ×12, SkipLayerNormalization ×24), not by trusting it.

per_channel=True. One scale per output channel instead of one per tensor. Measured on the evaluation below, per-tensor scales cost roughly 3× more missed spans at the same speed. This is the single biggest accuracy lever in the quantization step.

Gather excluded from quantization. The 29,794-row embedding table stays fp32. It costs ~70 MB of artifact size and preserves precision on the rare wordpieces that proper nouns decompose into.

Deliberately not used: reduce_range and symmetric QInt8. Both are faster specifically on pre-VNNI x86, and both become accuracy loss for no gain on a VNNI-capable CPU. They tune for one generation of hardware and quietly mistune for the next.

Evaluation

Evaluated against the fp32 ONNX export of the same model on 223 chunks of Brazilian legal / transcript Portuguese, at a decision threshold of 0.5.

The fp32 export was first verified against the original transformers pipeline(..., aggregation_strategy="max"): 0 mismatches over 807 spans, exact on span boundaries and scores. So fp32 ONNX is a faithful stand-in for the source model, and the numbers below isolate the cost of quantization.

Spans found by fp32 with no overlapping same-label span from this int8 model:

entity	spans	missed	rate
PESSOA	343	0	0.0%
ORGANIZACAO	322	6	1.9%
LOCAL	154	4	2.6%
total	819	10	1.2%

Person names survive quantization intact — that held across every quantization configuration tested, and is why Gather is excluded. The loss is concentrated in organizations and places.

Speed, same corpus, single-threaded-ish on a VNNI-capable desktop CPU:

	size	relative
transformers + torch pipeline	433 MB weights	1.0×
ONNX fp32	433.6 MB	~2.0×
ONNX fp32 + fusion	433.5 MB	~2.3×
this model	179.3 MB	~2.6–3.0×

Read the speed numbers loosely. They were measured on a 12th-gen Intel CPU with avx_vnni, which has hardware int8 dot-product instructions. On pre-VNNI hardware the int8 gain comes from halved memory traffic rather than faster arithmetic, and will be smaller. Run-to-run variance on a loaded machine was ±15%. The fp32 and fusion figures do not depend on VNNI and transfer cleanly.

No absolute recall is claimed. There is no gold-annotated PII corpus here — every number is relative to fp32, which is the right frame for "what did quantization cost" and the wrong frame for "how good is this model". For the latter, see the source model's card.

Usage

The model emits raw per-token logits. Reproducing transformers' NER output takes softmax, then max word aggregation, then B-/I- grouping — plain argmax per token will not match, and any threshold you calibrated against the transformers pipeline will not transfer.

import json
import numpy as np
import onnxruntime as ort
from tokenizers import Tokenizer

labels = {int(k): v for k, v in json.load(open("labels.json")).items()}
tok = Tokenizer.from_file("tokenizer.json")
tok.enable_truncation(max_length=512)
sess = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])

enc = tok.encode("A reclamante Maria Aparecida da Silva trabalhou na Zanchetta Ltda.")
logits = sess.run(["logits"], {
    "input_ids":      np.array([enc.ids], dtype=np.int64),
    "attention_mask": np.array([enc.attention_mask], dtype=np.int64),
    "token_type_ids": np.array([enc.type_ids], dtype=np.int64),
})[0][0]
scores = np.exp(logits - logits.max(-1, keepdims=True))
scores /= scores.sum(-1, keepdims=True)

A complete implementation — word aggregation with the max strategy, B-/I- grouping, and 512-token sliding windows with offset remapping — is about 150 lines. Two details that are easy to get wrong:

• Subword detection. For WordPiece, a token continues the previous word when len(token) != len(text[start:end]) (the ## prefix makes it longer than the text it covers). [UNK] is never a continuation.
• Windowing. A window whose first token is a mid-word fragment re-tokenizes one token longer than planned, because the continuation context is gone. Plan for 510 and you can land on 513 and overflow the position embeddings. Enable truncation at 512 as a hard guard and overlap your windows so nothing is lost to the cut.

Intended use and limitations

Built for redacting personal data from Brazilian labour-law consultation transcripts (LGPD). Suitable for any Portuguese legal-domain NER on CPU where transformers + torch are too heavy to ship.

• The source model is trained on formal written legal Portuguese. Speech transcripts are out of domain, and lowercase or uncased text degrades it badly — it is a cased model.
• Not a compliance control on its own. A 1.2% relative miss rate on organizations and places means it will leave identifying data in text. Pair it with deterministic rules for structured identifiers (CPF, CNPJ, CEP, e-mail, phone) and treat the neural stage as recall assistance, not a guarantee.
• Inherits every bias and failure mode of the source model, including its documented overfitting on a small fine-tuning set.
• TEMPO, LEGISLACAO and JURISPRUDENCIA are emitted but were not evaluated here.

Provenance and licensing

neuralmind/bert-base-portuguese-cased        (BERTimbau)      — MIT
  └─ pierreguillou/bert-base-cased-pt-lenerbr (MLM finetune)  — no license declared
      └─ pierreguillou/ner-bert-base-cased-pt-lenerbr (NER)   — no license declared
          └─ this repository (ONNX export + int8 quantization)

The source model declares no license. The MIT grant at the root of the chain does not carry through it: the fine-tuned weights are the author's own work, and no permission to redistribute them was granted. This repository therefore declares no license of its own — doing so would assert a permission that is not ours to give.

If you intend to redistribute these weights, or to build on them, resolve that with the source model's author (@pierreguillou) first. The conversion recipe above is fully specified, so anyone can reproduce this artifact from the source model themselves without relying on a copy from here.

Training data is LeNER-Br; check its terms independently if they matter to your use.

Citation

Cite the source model and the dataset, not this conversion.

@inproceedings{luz_de_araujo_lener_2018,
  title     = {{LeNER-Br}: A Dataset for Named Entity Recognition in {Brazilian} Legal Text},
  author    = {Luz de Araujo, Pedro H. and de Campos, Teófilo E. and
               de Oliveira, Renato R. R. and Stauffer, Matheus and
               Couto, Samuel and Bermejo, Paulo},
  booktitle = {International Conference on the Computational Processing of Portuguese (PROPOR)},
  year      = {2018},
  publisher = {Springer}
}