File size: 4,282 Bytes
b57d5bf 0ee0441 b57d5bf 0ee0441 b57d5bf 0ee0441 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 | ---
license: apache-2.0
pipeline_tag: text-generation
tags:
- lfm2
- data-use
- data-mentions
- extraction
- sft
- lora
---
# lfm2.5-350M-datause
LoRA SFT of `LiquidAI/LFM2.5-350M` for data-mention extraction: emit the
data-bearing phrases in a text as compact JSON
(`data_mentions` with `data_mention`/`specificity_type`).
## Training
- base model: `LiquidAI/LFM2.5-350M`
- dataset: `rafmacalaba/data-use-mention-sft`
- epochs: 3
- learning rate: 0.0002
- LoRA: r=16 alpha=32 dropout=0.05
- completion-only masking (loss on assistant JSON turn)
## Evaluation (holdout, n=12531)
Jaccard entity-level matching: acronym-aware span clustering +
Hungarian optimal bipartite match (match thr=0.5), F0.5
primary. Aligned with `rafmacalaba/gliner_datause_extended`. The
holdout is `rafmacalaba/data-use-mention-sft`, so numbers are **not
directly comparable** to the GLiNER model's `data-use-mentions-extended`
holdout.
| label | tp | fp | fn | precision | recall | f0.5 | f1 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| **overall** | 13176 | 3097 | 2998 | 0.8097 | 0.8146 | 0.8107 | 0.8122 |
## Per-label
| label | tp | fp | fn | precision | recall | f0.5 | f1 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| descriptive | 5522 | 2631 | 2078 | 0.6773 | 0.7266 | 0.6866 | 0.7011 |
| named | 5344 | 1538 | 1629 | 0.7765 | 0.7664 | 0.7745 | 0.7714 |
| vague | 670 | 604 | 934 | 0.5259 | 0.4177 | 0.5000 | 0.4656 |
## Per-corpus
| label | tp | fp | fn | precision | recall | f0.5 | f1 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| fcv | 3142 | 619 | 730 | 0.8354 | 0.8115 | 0.8305 | 0.8233 |
| prwp | 10034 | 2478 | 2268 | 0.8020 | 0.8156 | 0.8047 | 0.8087 |
## Per-origin
| label | tp | fp | fn | precision | recall | f0.5 | f1 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| fcv_pads_east_asia | 728 | 96 | 110 | 0.8835 | 0.8687 | 0.8805 | 0.8761 |
| general_prwp | 10034 | 2478 | 2268 | 0.8020 | 0.8156 | 0.8047 | 0.8087 |
| jdc_operational | 149 | 25 | 24 | 0.8563 | 0.8613 | 0.8573 | 0.8588 |
| refugee_pads | 704 | 147 | 152 | 0.8273 | 0.8224 | 0.8263 | 0.8248 |
| reliefweb | 1561 | 351 | 444 | 0.8164 | 0.7786 | 0.8086 | 0.7970 |
## Sample predictions (holdout)
| gold | predicted |
| --- | --- |
| {"data_mentions":[{"data_mention":"administrative data","specificity_type":"descriptive"}]} | {"data_mentions":[{"data_mention":"administrative data","specificity_type":"descriptive"}]} |
| {"data_mentions":[{"data_mention":"World Development Indicators","specificity_type":"named"},{"data_mention":"CPHS","spe | {"data_mentions":[{"data_mention":"World Development Indicators","specificity_type":"named"},{"data_mention":"CPHS","spe |
| {"data_mentions":[{"data_mention":"ES data","specificity_type":"named"}]} | {"data_mentions":[{"data_mention":"ES data","specificity_type":"named"}]} |
| {"data_mentions":[]} | {"data_mentions":[{"data_mention":"EUROMOD I4.0+","specificity_type":"named"}]} |
| {"data_mentions":[{"data_mention":"World Development Indicators","specificity_type":"named"},{"data_mention":"IMF World | {"data_mentions":[{"data_mention":"World Development Indicators","specificity_type":"named"},{"data_mention":"IMF World |
| {"data_mentions":[]} | {"data_mentions":[]} |
| {"data_mentions":[]} | {"data_mentions":[]} |
| {"data_mentions":[{"data_mention":"UNESCO-provided data","specificity_type":"descriptive"}]} | {"data_mentions":[{"data_mention":"PIRLS","specificity_type":"named"},{"data_mention":"UNESCO-provided data","specificit |
| {"data_mentions":[{"data_mention":"IHPS dataset","specificity_type":"named"},{"data_mention":"IHS4 data","specificity_ty | {"data_mentions":[{"data_mention":"IHS4","specificity_type":"named"},{"data_mention":"IHPS dataset","specificity_type":" |
| {"data_mentions":[{"data_mention":"HFSSS","specificity_type":"named"}]} | {"data_mentions":[{"data_mention":"HFSSS","specificity_type":"named"}]} |
| {"data_mentions":[{"data_mention":"HFCS data","specificity_type":"named"}]} | {"data_mentions":[{"data_mention":"HFCS data","specificity_type":"named"}]} |
| {"data_mentions":[{"data_mention":"data on patents and publications","specificity_type":"descriptive"}]} | {"data_mentions":[{"data_mention":"data on patents and publications","specificity_type":"descriptive"}]} |
|