| --- |
| license: apache-2.0 |
| pipeline_tag: text-generation |
| tags: |
| - lfm2 |
| - data-use |
| - provenance |
| - usage |
| - impact |
| - sft |
| - lora |
| --- |
| |
| # lfm2.5-350M-datause-multitask |
|
|
| LoRA SFT of `LiquidAI/LFM2.5-350M` for data-mention provenance attributes |
| (producer / year / geography / acronym) and usage/impact classification |
| (data_type / usage_action / impact_label / usage_summary). |
|
|
| ## Training |
| - base model: `LiquidAI/LFM2.5-350M` |
| - dataset: `rafmacalaba/data-use-sft` |
| - epochs: 3 |
| - learning rate: 0.0002 |
| - LoRA: r=16 alpha=32 dropout=0.05 |
| - completion-only masking (loss on assistant JSON turn) |
|
|
| ## real holdout |
|
|
| Holdout n=5464. Exact string match of each emitted attribute against the gold label. |
|
|
| | attribute | tp | fp | fn | precision | recall | f0.5 | f1 | |
| | --- | --- | --- | --- | --- | --- | --- | --- | |
| | producer | 1081 | 374 | 368 | 0.7430 | 0.7460 | 0.7436 | 0.7445 | |
| | year | 1002 | 230 | 348 | 0.8133 | 0.7422 | 0.7980 | 0.7761 | |
| | geography | 1620 | 446 | 407 | 0.7841 | 0.7992 | 0.7871 | 0.7916 | |
| | acronym | 1270 | 246 | 146 | 0.8377 | 0.8969 | 0.8489 | 0.8663 | |
| | **overall** | 4973 | 1296 | 1269 | 0.7933 | 0.7967 | 0.7940 | 0.7950 | |
|
|
| Usage/impact macro-F1 (per head): |
| - data_type: 0.6005 |
| - usage_action: 0.5624 |
| - impact_label: 0.4933 |
| - usage_summary: mean_sim=0.6581 grounded_rate=0.9165 |
|
|
| Verbatim rate (emitted values that are substrings of the context): 6249/6269 = 0.9968 |
|
|
| ## synthetic holdout |
|
|
| Holdout n=1564. Exact string match of each emitted attribute against the gold label. |
|
|
| | attribute | tp | fp | fn | precision | recall | f0.5 | f1 | |
| | --- | --- | --- | --- | --- | --- | --- | --- | |
| | producer | 126 | 23 | 26 | 0.8456 | 0.8289 | 0.8422 | 0.8372 | |
| | year | 123 | 24 | 21 | 0.8367 | 0.8542 | 0.8402 | 0.8454 | |
| | geography | 163 | 38 | 36 | 0.8109 | 0.8191 | 0.8126 | 0.8150 | |
| | acronym | 78 | 17 | 13 | 0.8211 | 0.8571 | 0.8280 | 0.8387 | |
| | **overall** | 490 | 102 | 96 | 0.8277 | 0.8362 | 0.8294 | 0.8319 | |
|
|
| Usage/impact macro-F1 (per head): |
| - data_type: 0.8559 |
| - usage_action: 0.7706 |
| - impact_label: 0.7985 |
| - usage_summary: mean_sim=0.5276 grounded_rate=0.5898 |
|
|
| Verbatim rate (emitted values that are substrings of the context): 592/592 = 1.0000 |
|
|
|
|
