Card: LocalLLaMA/typed-extraction zero-shot result
Browse files
README.md
CHANGED
|
@@ -82,6 +82,19 @@ tags = session.run(None, {"input_ids": enc["input_ids"], "attention_mask": enc["
|
|
| 82 |
# then read the spans off the text tokens as in extract() above (tags outside the text are meaningless)
|
| 83 |
```
|
| 84 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 85 |
## Evaluation
|
| 86 |
|
| 87 |
Test split of the extraction config (revision v2026.10.20.7), 100 rows per source. *text* fields (one answer or none) are scored by SQuAD F1, the empty answer counting as an answer. *list* fields (every span of a kind) are scored by micro F1 over normalized values. Scores are means over sources.
|
|
|
|
| 82 |
# then read the spans off the text tokens as in extract() above (tags outside the text are meaningless)
|
| 83 |
```
|
| 84 |
|
| 85 |
+
## LocalLLaMA/typed-extraction (zero-shot)
|
| 86 |
+
|
| 87 |
+
On the `test` split of [LocalLLaMA/typed-extraction](https://huggingface.co/datasets/LocalLLaMA/typed-extraction) (1,690 documents, 17,733 typed fields), which this model never trained on. Scores come from its own `score.py`, and spans are mapped to typed values with plain code (numbers parsed, dates through `dateutil`). Submitted in [discussion #1](https://huggingface.co/datasets/LocalLLaMA/typed-extraction/discussions/1).
|
| 88 |
+
|
| 89 |
+
| Model | Headline ↑ | Fields with a value ↑ | Hallucination ↓ | Miss ↓ | Computed ↑ |
|
| 90 |
+
|---|---|---|---|---|---|
|
| 91 |
+
| ScaleDown Extract (API) | 0.782 | 0.723 | 0.117 | 0.200 | 0.570 |
|
| 92 |
+
| **UltraBERT-150M-clairic-extraction** | **0.616** | 0.548 | 0.141 | 0.297 | 0.014 |
|
| 93 |
+
| Schematron V2 Small (LLM) | 0.562 | 0.529 | 0.445 | 0.303 | 0.134 |
|
| 94 |
+
| Always null | 0.199 | 0.000 | 0.000 | 1.000 | 0.000 |
|
| 95 |
+
|
| 96 |
+
As an extractor, it can't compute values (totals, counts, day spans). Predictions are in [`eval/`](https://huggingface.co/tasksource/UltraBERT-150M-clairic-extraction/tree/main/eval).
|
| 97 |
+
|
| 98 |
## Evaluation
|
| 99 |
|
| 100 |
Test split of the extraction config (revision v2026.10.20.7), 100 rows per source. *text* fields (one answer or none) are scored by SQuAD F1, the empty answer counting as an answer. *list* fields (every span of a kind) are scored by micro F1 over normalized values. Scores are means over sources.
|