| --- |
| license: apache-2.0 |
| pipeline_tag: text-generation |
| tags: |
| - lfm2 |
| - data-use |
| - data-mentions |
| - extraction |
| - sft |
| - lora |
| --- |
| |
| # lfm2.5-350M-datause |
|
|
| LoRA SFT of `LiquidAI/LFM2.5-350M` for data-mention extraction: emit the |
| data-bearing phrases in a text as compact JSON |
| (`data_mentions` with `data_mention`/`specificity_type`). |
|
|
| ## Training |
| - base model: `LiquidAI/LFM2.5-350M` |
| - dataset: `rafmacalaba/data-use-mention-sft` |
| - epochs: 3 |
| - learning rate: 0.0002 |
| - LoRA: r=16 alpha=32 dropout=0.05 |
| - completion-only masking (loss on assistant JSON turn) |
|
|
| ## Evaluation (holdout, n=12531) |
|
|
| Jaccard entity-level matching: acronym-aware span clustering + |
| Hungarian optimal bipartite match (match thr=0.5), F0.5 |
| primary. Aligned with `rafmacalaba/gliner_datause_extended`. The |
| holdout is `rafmacalaba/data-use-mention-sft`, so numbers are **not |
| directly comparable** to the GLiNER model's `data-use-mentions-extended` |
| holdout. |
|
|
| | label | tp | fp | fn | precision | recall | f0.5 | f1 | |
| | --- | --- | --- | --- | --- | --- | --- | --- | |
| | **overall** | 13176 | 3097 | 2998 | 0.8097 | 0.8146 | 0.8107 | 0.8122 | |
| ## Per-label |
|
|
| | label | tp | fp | fn | precision | recall | f0.5 | f1 | |
| | --- | --- | --- | --- | --- | --- | --- | --- | |
| | descriptive | 5522 | 2631 | 2078 | 0.6773 | 0.7266 | 0.6866 | 0.7011 | |
| | named | 5344 | 1538 | 1629 | 0.7765 | 0.7664 | 0.7745 | 0.7714 | |
| | vague | 670 | 604 | 934 | 0.5259 | 0.4177 | 0.5000 | 0.4656 | |
| ## Per-corpus |
|
|
| | label | tp | fp | fn | precision | recall | f0.5 | f1 | |
| | --- | --- | --- | --- | --- | --- | --- | --- | |
| | fcv | 3142 | 619 | 730 | 0.8354 | 0.8115 | 0.8305 | 0.8233 | |
| | prwp | 10034 | 2478 | 2268 | 0.8020 | 0.8156 | 0.8047 | 0.8087 | |
| ## Per-origin |
|
|
| | label | tp | fp | fn | precision | recall | f0.5 | f1 | |
| | --- | --- | --- | --- | --- | --- | --- | --- | |
| | fcv_pads_east_asia | 728 | 96 | 110 | 0.8835 | 0.8687 | 0.8805 | 0.8761 | |
| | general_prwp | 10034 | 2478 | 2268 | 0.8020 | 0.8156 | 0.8047 | 0.8087 | |
| | jdc_operational | 149 | 25 | 24 | 0.8563 | 0.8613 | 0.8573 | 0.8588 | |
| | refugee_pads | 704 | 147 | 152 | 0.8273 | 0.8224 | 0.8263 | 0.8248 | |
| | reliefweb | 1561 | 351 | 444 | 0.8164 | 0.7786 | 0.8086 | 0.7970 | |
|
|
| ## Sample predictions (holdout) |
|
|
| | gold | predicted | |
| | --- | --- | |
| | {"data_mentions":[{"data_mention":"administrative data","specificity_type":"descriptive"}]} | {"data_mentions":[{"data_mention":"administrative data","specificity_type":"descriptive"}]} | |
| | {"data_mentions":[{"data_mention":"World Development Indicators","specificity_type":"named"},{"data_mention":"CPHS","spe | {"data_mentions":[{"data_mention":"World Development Indicators","specificity_type":"named"},{"data_mention":"CPHS","spe | |
| | {"data_mentions":[{"data_mention":"ES data","specificity_type":"named"}]} | {"data_mentions":[{"data_mention":"ES data","specificity_type":"named"}]} | |
| | {"data_mentions":[]} | {"data_mentions":[{"data_mention":"EUROMOD I4.0+","specificity_type":"named"}]} | |
| | {"data_mentions":[{"data_mention":"World Development Indicators","specificity_type":"named"},{"data_mention":"IMF World | {"data_mentions":[{"data_mention":"World Development Indicators","specificity_type":"named"},{"data_mention":"IMF World | |
| | {"data_mentions":[]} | {"data_mentions":[]} | |
| | {"data_mentions":[]} | {"data_mentions":[]} | |
| | {"data_mentions":[{"data_mention":"UNESCO-provided data","specificity_type":"descriptive"}]} | {"data_mentions":[{"data_mention":"PIRLS","specificity_type":"named"},{"data_mention":"UNESCO-provided data","specificit | |
| | {"data_mentions":[{"data_mention":"IHPS dataset","specificity_type":"named"},{"data_mention":"IHS4 data","specificity_ty | {"data_mentions":[{"data_mention":"IHS4","specificity_type":"named"},{"data_mention":"IHPS dataset","specificity_type":" | |
| | {"data_mentions":[{"data_mention":"HFSSS","specificity_type":"named"}]} | {"data_mentions":[{"data_mention":"HFSSS","specificity_type":"named"}]} | |
| | {"data_mentions":[{"data_mention":"HFCS data","specificity_type":"named"}]} | {"data_mentions":[{"data_mention":"HFCS data","specificity_type":"named"}]} | |
| | {"data_mentions":[{"data_mention":"data on patents and publications","specificity_type":"descriptive"}]} | {"data_mentions":[{"data_mention":"data on patents and publications","specificity_type":"descriptive"}]} | |
| |
