File size: 13,153 Bytes
e9c4ecd 70c0253 e9c4ecd 70c0253 e9c4ecd 70c0253 e0aa0f1 70c0253 26bccb1 e9c4ecd | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 | ---
library_name: transformers
base_model:
- dmis-lab/UltraBERT-150M
license: apache-2.0
language:
- en
pipeline_tag: token-classification
datasets:
- tasksource/tasksource-jev-typed-decisions
tags:
- information-extraction
- extractive-qa
- ner
- span-extraction
- tasksource
---
# UltraBERT-150M-clairic-extraction
A 150M-parameter extractor for Clairic-style requests: give it a text and a set of fields, each described by a short instruction, and it returns the matching spans for each field (one answer, a list, or nothing when the text has no answer). Fields can be questions ("What is the total due?") or entity types ("Every company mentioned in the text"), so one model covers extractive question answering and named-entity recognition.
[UltraBERT-150M](https://huggingface.co/dmis-lab/UltraBERT-150M) trained on the `extraction` config of [tasksource-jev-typed-decisions](https://huggingface.co/datasets/tasksource/tasksource-jev-typed-decisions): extractive QA and span-tagging datasets from [tasksource](https://github.com/sileod/tasksource), rewritten as (instruction, text, spans) pairs. The model tags the answer tokens of the text (B/I/O) conditioned on the instruction. Inputs up to 2048 tokens.
Not affiliated with Cloudflare; "Clairic-style" refers to the request format.
**[Try it in your browser](https://huggingface.co/spaces/sileod/UltraBERT-clairic-extraction-demo)**: the int8 ONNX export below, run client-side with onnxruntime-web.
## Usage
```python
import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer
name = "tasksource/UltraBERT-150M-clairic-extraction"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForTokenClassification.from_pretrained(name, trust_remote_code=True).eval()
def extract(text, fields):
"""fields: {name: instruction}. Returns {name: [extracted spans]}, [] when the text has no answer."""
enc = tok(list(fields.values()), [text] * len(fields), truncation="only_second", max_length=2048,
padding=True, return_offsets_mapping=True, return_tensors="pt")
offsets = enc.pop("offset_mapping")
with torch.no_grad():
tags = model(**enc).logits.argmax(-1).tolist() # 0 = O, 1 = B, 2 = I
out = {}
for f, field in enumerate(fields):
spans, start, seq = [], None, enc.sequence_ids(f) + [None]
for i, tag in enumerate(tags[f] + [0]):
if start is not None and (seq[i] != 1 or tag != 2):
spans.append(text[offsets[f][start][0]:offsets[f][i - 1][1]].strip().rstrip(",;:"))
start = None
if seq[i] == 1 and tag in (1, 2) and start is None:
start = i
out[field] = spans
return out
text = "Invoice INV-2291 from Acme Corp to Globex, issued 3 March 2025. Total due: $1,240.50. Contact: billing@acme.com."
print(extract(text, {
"invoice_number": "The invoice number.",
"companies": "Every company mentioned in the text, as written.",
"total": "The total amount due.",
"phone": "The contact phone number.",
}))
# {'invoice_number': ['INV-2291'], 'companies': ['Acme Corp', 'Globex'], 'total': ['$1,240.50'], 'phone': []}
```
`trust_remote_code=True` is needed for the UltraBERT architecture.
### ONNX int8
`onnx/model_quantized.onnx` (204 MB, vs 604 MB in fp32) takes `input_ids` and `attention_mask` and returns the O/B/I `logits`. Linear weights are quantized to 8 bits per block of 32 (weight-only, `MatMulNBits`; activations stay in float) and the embedding table is stored in fp16. Its tags match the PyTorch model on 99.6% of tokens and on 100% of answer tokens in our checks, and it scores the same on the evaluation below (20 rows per source: QA F1 0.730 vs 0.728 for PyTorch fp16, list F1 0.701 vs 0.700); plain dynamic int8 quantization (int8 activations) breaks the model (78% agreement), so prefer this file to re-quantizing. Build the input as the tokenizer's pair template: `<|endoftext|>` instruction `<|sep|>` text.
```python
import onnxruntime
from huggingface_hub import hf_hub_download
session = onnxruntime.InferenceSession(hf_hub_download(name, "onnx/model_quantized.onnx"))
enc = tok(list(fields.values()), [text] * len(fields), truncation="only_second", max_length=2048,
padding=True, return_offsets_mapping=True, return_tensors="np")
tags = session.run(None, {"input_ids": enc["input_ids"], "attention_mask": enc["attention_mask"]})[0].argmax(-1).tolist()
# then read the spans off the text tokens as in extract() above (tags outside the text are meaningless)
```
## LocalLLaMA/typed-extraction (zero-shot)
On the `test` split of [LocalLLaMA/typed-extraction](https://huggingface.co/datasets/LocalLLaMA/typed-extraction) (1,690 documents, 17,733 typed fields), which this model never trained on. Scores come from its own `score.py`, and spans are mapped to typed values with plain code (numbers parsed, dates through `dateutil`). Submitted in [discussion #1](https://huggingface.co/datasets/LocalLLaMA/typed-extraction/discussions/1).
| Model | Headline ↑ | Fields with a value ↑ | Hallucination ↓ | Miss ↓ | Computed ↑ |
|---|---|---|---|---|---|
| ScaleDown Extract (API) | 0.782 | 0.723 | 0.117 | 0.200 | 0.570 |
| **UltraBERT-150M-clairic-extraction** | **0.616** | 0.548 | 0.141 | 0.297 | 0.014 |
| Schematron V2 Small (LLM) | 0.562 | 0.529 | 0.445 | 0.303 | 0.134 |
| Always null | 0.199 | 0.000 | 0.000 | 1.000 | 0.000 |
As an extractor, it can't compute values (totals, counts, day spans). Predictions are in [`eval/`](https://huggingface.co/tasksource/UltraBERT-150M-clairic-extraction/tree/main/eval).
## Evaluation
Test split of the extraction config (revision v2026.10.20.7), 100 rows per source. *text* fields (one answer or none) are scored by SQuAD F1, the empty answer counting as an answer. *list* fields (every span of a kind) are scored by micro F1 over normalized values. Scores are means over sources.
| Model | Params | QA (28 sources, F1) | NER / span lists (60 sources, F1) |
|---|---|---|---|
| **UltraBERT-150M-clairic-extraction** | 150M | **0.720** | **0.702** |
| [gliner-multitask-large-v0.5](https://huggingface.co/knowledgator/gliner-multitask-large-v0.5) | 440M | 0.474 | 0.408 |
| [gliner_medium-v2.1](https://huggingface.co/urchade/gliner_medium-v2.1) | 209M | 0.332 | 0.360 |
| [deberta-v3-base-squad2](https://huggingface.co/deepset/deberta-v3-base-squad2) | 184M | 0.498 | 0.060 |
The test sources are the training sources (their test splits), while the baselines see most of them zero-shot, so this measures performance on these tasks rather than on unseen ones. GLiNER reads about 384 words at a time and was given the field name as its label (the instruction for numbered questions); the QA models were given the instruction as the question and read long texts in overlapping windows. On SQuAD 2.0, deberta-v3-base-squad2, trained on it, stays ahead (0.876 vs 0.764).
<details>
<summary>Per-source F1</summary>
| Source | Field type | clairic-extraction | gliner-multitask-large | gliner_medium | deberta-squad2 |
|---|---|---|---|---|---|
| adversarial-qa | text | **0.373** | 0.251 | 0.008 | 0.320 |
| anatem | list | **0.787** | 0.305 | 0.337 | 0.000 |
| bc2gm | list | **0.776** | 0.506 | 0.575 | 0.000 |
| bc4chemd | list | **0.778** | 0.521 | 0.561 | 0.092 |
| bc5cdr | list | **0.845** | 0.641 | 0.665 | 0.083 |
| bioinfer/entities | list | **0.413** | 0.117 | 0.318 | 0.005 |
| bionlp-cg/entities | list | **0.740** | 0.546 | 0.300 | 0.003 |
| biored/entities | list | **0.775** | 0.332 | 0.307 | 0.006 |
| broad-twitter-corpus | list | 0.664 | **0.727** | 0.725 | 0.228 |
| buster | list | **0.720** | 0.540 | 0.395 | 0.047 |
| concurrentqa | list | **0.768** | 0.553 | 0.100 | 0.516 |
| conll04 | list | **0.808** | 0.702 | 0.719 | 0.120 |
| conll2000 | list | **0.950** | 0.030 | 0.010 | 0.002 |
| conll2003/ner_tags | list | **0.894** | 0.717 | 0.584 | 0.219 |
| crossner/ai | list | 0.650 | **0.675** | 0.667 | 0.075 |
| crossner/literature | list | 0.760 | **0.789** | 0.737 | 0.064 |
| crossner/music | list | 0.713 | **0.850** | 0.823 | 0.050 |
| crossner/politics | list | 0.798 | **0.853** | 0.806 | 0.036 |
| crossner/science | list | 0.735 | **0.761** | 0.755 | 0.036 |
| cuad/spans | list | **0.454** | 0.025 | 0.000 | 0.032 |
| duorc/ParaphraseRC | text | **0.517** | 0.458 | 0.290 | 0.385 |
| duorc/SelfRC | text | **0.832** | 0.708 | 0.077 | 0.519 |
| ebm-nlp/interventions | list | **0.520** | 0.155 | 0.163 | 0.000 |
| ebm-nlp/outcomes | list | **0.425** | 0.031 | 0.061 | 0.000 |
| ebm-nlp/participants | list | **0.305** | 0.006 | 0.057 | 0.006 |
| fabner | list | **0.674** | 0.268 | 0.191 | 0.013 |
| fairytaleqa/explicit | text | **0.743** | 0.537 | 0.000 | 0.627 |
| fce/errors | list | **0.377** | 0.000 | 0.000 | 0.000 |
| few-nerd/supervised | list | **0.770** | 0.725 | 0.631 | 0.043 |
| finer-139 | list | **0.906** | 0.222 | 0.164 | 0.012 |
| finer-ord | list | **0.834** | 0.672 | 0.649 | 0.141 |
| friendsqa | text | **0.660** | 0.333 | 0.005 | 0.486 |
| gum-ner | list | **0.425** | 0.287 | 0.383 | 0.032 |
| harveyner | list | **0.402** | 0.321 | 0.314 | 0.029 |
| hotpotqa | text | 0.728 | **0.753** | 0.107 | 0.586 |
| inspec/keyphrases | list | **0.431** | 0.033 | 0.000 | 0.000 |
| jnlpba/jnlpba | list | **0.723** | 0.474 | 0.581 | 0.054 |
| legallens-ner | list | **0.565** | 0.155 | 0.165 | 0.038 |
| linnaeus | list | **0.964** | 0.548 | 0.500 | 0.000 |
| medmentions/st21pv | list | **0.517** | 0.422 | 0.137 | 0.003 |
| mit-movie-trivia | list | **0.799** | 0.553 | 0.577 | 0.351 |
| mit-restaurant | list | **0.785** | 0.427 | 0.362 | 0.036 |
| molweni | text | **0.641** | 0.369 | 0.107 | 0.488 |
| multidoc2dial | text | **0.642** | 0.010 | 0.000 | 0.107 |
| natural-questions | text | **0.718** | 0.657 | 0.194 | 0.469 |
| ncbi_disease/ncbi_disease | list | **0.843** | 0.589 | 0.560 | 0.075 |
| ontonotes_english/SpeedOfMagic--ontonotes_english | list | **0.864** | 0.784 | 0.599 | 0.094 |
| opinion-target/en | list | **0.749** | 0.102 | 0.000 | 0.000 |
| policyqa | text | **0.422** | 0.052 | 0.000 | 0.197 |
| qaconv | text | 0.678 | **0.679** | 0.095 | 0.620 |
| qasper/spans | list | **0.083** | 0.034 | 0.000 | 0.005 |
| redocred/entities | list | **0.786** | 0.000 | 0.000 | 0.006 |
| scierc/entities | list | **0.464** | 0.172 | 0.150 | 0.009 |
| semeval2014/laptop-aspects | list | **0.812** | 0.132 | 0.062 | 0.032 |
| skillspan/knowledge | list | **0.737** | 0.278 | 0.056 | 0.010 |
| skillspan/skill | list | **0.469** | 0.146 | 0.048 | 0.000 |
| slurp/slots | list | **0.828** | 0.664 | 0.544 | 0.250 |
| squad-v2 | text | 0.764 | 0.585 | 0.544 | **0.876** |
| stackoverflow-ner | list | **0.715** | 0.373 | 0.384 | 0.063 |
| subjqa/books | text | **0.714** | 0.566 | 0.670 | 0.576 |
| subjqa/electronics | text | **0.559** | 0.370 | 0.480 | 0.332 |
| subjqa/grocery | text | **0.617** | 0.313 | 0.585 | 0.445 |
| subjqa/movies | text | **0.686** | 0.530 | 0.685 | 0.512 |
| subjqa/tripadvisor | text | **0.522** | 0.315 | 0.394 | 0.422 |
| swde/auto | text | **0.686** | 0.623 | 0.527 | 0.589 |
| swde/book | text | **0.991** | 0.969 | 0.875 | 0.570 |
| swde/camera | text | **0.997** | 0.736 | 0.630 | 0.682 |
| swde/job | text | **0.852** | 0.774 | 0.519 | 0.513 |
| swde/movie | text | **0.799** | 0.174 | 0.219 | 0.320 |
| swde/nbaplayer | text | **0.972** | 0.938 | 0.778 | 0.540 |
| swde/restaurant | text | **0.983** | 0.501 | 0.722 | 0.715 |
| swde/university | text | **0.999** | 0.032 | 0.053 | 0.849 |
| tab/pii | list | **0.860** | 0.330 | 0.288 | 0.000 |
| timeqa | text | **0.462** | 0.187 | 0.000 | 0.102 |
| toxic-spans | list | **0.593** | 0.000 | 0.000 | 0.000 |
| tweetner7 | list | **0.556** | 0.406 | 0.394 | 0.093 |
| ud/ewt | list | **0.896** | 0.017 | 0.037 | 0.000 |
| ud/gum | list | **0.846** | 0.039 | 0.061 | 0.000 |
| uner/en_ewt | list | **0.924** | 0.775 | 0.762 | 0.198 |
| wdc-pave | text | **0.916** | 0.699 | 0.724 | 0.621 |
| websrc/spans | text | **0.694** | 0.166 | 0.000 | 0.491 |
| wiesp2022-ner | list | **0.722** | 0.413 | 0.429 | 0.029 |
| wnut_17/wnut_17 | list | **0.594** | 0.475 | 0.525 | 0.133 |
</details>
## Training data
The extraction config gathers extractive QA (SQuAD 2.0, Natural Questions, HotpotQA, NewsQA, DuoRC, QuAC, CUAD, Qasper, TimeQA, WebSRC, SWDE, and others) and span tagging (CoNLL-2003, OntoNotes, Few-NERD, WNUT, CrossNER, biomedical, legal and financial NER, skills, aspects, keyphrases, and others), plus procedurally generated documents. A few sources (e.g. GUM, PreCo) are licensed for non-commercial use only; check the dataset card for per-source licenses.
## Citation
```bibtex
@inproceedings{sileo-2024-tasksource,
title = "tasksource: A Large Collection of {NLP} tasks with a Structured Dataset Preprocessing Framework",
author = "Sileo, Damien",
booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
year = "2024",
address = "Torino, Italia",
pages = "15655--15684",
url = "https://aclanthology.org/2024.lrec-main.1361",
}
```
|