| --- |
| license: apache-2.0 |
| pipeline_tag: text-generation |
| tags: |
| - lfm2 |
| - data-use |
| - data-mentions |
| - extraction |
| - sft |
| - lora |
| --- |
| |
| # lfm2.5-350M-datause-prwp |
|
|
| LoRA SFT of `LiquidAI/LFM2.5-350M` for data-mention extraction: emit the |
| data-bearing phrases in a text as compact JSON |
| (`data_mentions` with `data_mention`/`specificity_type`). |
|
|
| ## Training |
| - base model: `LiquidAI/LFM2.5-350M` |
| - dataset: `rafmacalaba/data-use-mention-sft` |
| - epochs: 3 |
| - learning rate: 0.0002 |
| - LoRA: r=16 alpha=32 dropout=0.05 |
| - completion-only masking (loss on assistant JSON turn) |
|
|
| ## Evaluation (holdout, n=9079) |
|
|
| Jaccard entity-level matching: acronym-aware span clustering + |
| Hungarian optimal bipartite match (match thr=0.5), F0.5 |
| primary. Aligned with `rafmacalaba/gliner_datause_extended`. The |
| holdout is `rafmacalaba/data-use-mention-sft`, so numbers are **not |
| directly comparable** to the GLiNER model's `data-use-mentions-extended` |
| holdout. |
|
|
| | label | tp | fp | fn | precision | recall | f0.5 | f1 | |
| | --- | --- | --- | --- | --- | --- | --- | --- | |
| | **overall** | 9888 | 2537 | 2414 | 0.7958 | 0.8038 | 0.7974 | 0.7998 | |
| ## Per-label |
|
|
| | label | tp | fp | fn | precision | recall | f0.5 | f1 | |
| | --- | --- | --- | --- | --- | --- | --- | --- | |
| | descriptive | 4034 | 1930 | 1762 | 0.6764 | 0.6960 | 0.6802 | 0.6861 | |
| | named | 4183 | 1281 | 1175 | 0.7656 | 0.7807 | 0.7685 | 0.7731 | |
| | vague | 488 | 539 | 662 | 0.4752 | 0.4243 | 0.4641 | 0.4483 | |
| ## Per-corpus |
|
|
| | label | tp | fp | fn | precision | recall | f0.5 | f1 | |
| | --- | --- | --- | --- | --- | --- | --- | --- | |
| | prwp | 9888 | 2537 | 2414 | 0.7958 | 0.8038 | 0.7974 | 0.7998 | |
| ## Per-origin |
|
|
| | label | tp | fp | fn | precision | recall | f0.5 | f1 | |
| | --- | --- | --- | --- | --- | --- | --- | --- | |
| | general_prwp | 9888 | 2537 | 2414 | 0.7958 | 0.8038 | 0.7974 | 0.7998 | |
| |
| ## Sample predictions (holdout) |
| |
| | gold | predicted | |
| | --- | --- | |
| | {"data_mentions":[{"data_mention":"administrative data","specificity_type":"descriptive"}]} | {"data_mentions":[{"data_mention":"administrative data","specificity_type":"descriptive"}]} | |
| | {"data_mentions":[{"data_mention":"World Development Indicators","specificity_type":"named"},{"data_mention":"CPHS","spe | {"data_mentions":[{"data_mention":"World Development Indicators","specificity_type":"named"},{"data_mention":"CPHS","spe | |
| | {"data_mentions":[{"data_mention":"ES data","specificity_type":"named"}]} | {"data_mentions":[{"data_mention":"ES data","specificity_type":"named"}]} | |
| | {"data_mentions":[]} | {"data_mentions":[{"data_mention":"EUROMOD I4.0+","specificity_type":"named"}]} | |
| | {"data_mentions":[{"data_mention":"World Development Indicators","specificity_type":"named"},{"data_mention":"IMF World | {"data_mentions":[{"data_mention":"World Development Indicators","specificity_type":"named"},{"data_mention":"IMF World | |
| | {"data_mentions":[]} | {"data_mentions":[]} | |
| | {"data_mentions":[{"data_mention":"UNESCO-provided data","specificity_type":"descriptive"}]} | {"data_mentions":[{"data_mention":"PIRLS","specificity_type":"named"},{"data_mention":"UNESCO-provided data","specificit | |
| | {"data_mentions":[{"data_mention":"IHPS dataset","specificity_type":"named"},{"data_mention":"IHS4 data","specificity_ty | {"data_mentions":[{"data_mention":"IHS4 dataset","specificity_type":"named"},{"data_mention":"IHS4 data","specificity_ty | |
| | {"data_mentions":[{"data_mention":"HFSSS","specificity_type":"named"}]} | {"data_mentions":[{"data_mention":"HFSSS","specificity_type":"named"}]} | |
| | {"data_mentions":[{"data_mention":"HFCS data","specificity_type":"named"}]} | {"data_mentions":[{"data_mention":"HFCS data","specificity_type":"named"}]} | |
| | {"data_mentions":[{"data_mention":"data on patents and publications","specificity_type":"descriptive"}]} | {"data_mentions":[{"data_mention":"data on patents and publications","specificity_type":"descriptive"}]} | |
| | {"data_mentions":[{"data_mention":"UN COMTRADE Statistics","specificity_type":"named"}]} | {"data_mentions":[{"data_mention":"UN COMTRADE Statistics","specificity_type":"named"}]} | |
|
|
