| --- |
| license: apache-2.0 |
| pipeline_tag: text-generation |
| tags: |
| - lfm2 |
| - data-use |
| - provenance |
| - usage |
| - impact |
| - sft |
| - lora |
| --- |
| |
| # lfm2.5-350M-multitask-datause |
|
|
| LoRA SFT of `LiquidAI/LFM2.5-350M` for data-mention provenance attributes |
| (producer / year / geography / acronym) and usage/impact classification |
| (data_type / usage_action / impact_label / usage_summary). |
|
|
| ## Training |
| - base model: `LiquidAI/LFM2.5-350M` |
| - dataset: `rafmacalaba/data-use-sft` |
| - epochs: 5 |
| - learning rate: 0.0002 |
| - LoRA: r=16 alpha=32 dropout=0.05 |
| - completion-only masking (loss on assistant JSON turn) |
|
|
| ## real holdout |
|
|
| Holdout n=14776. Exact string match of each emitted attribute against the gold label. |
|
|
| | attribute | tp | fp | fn | precision | recall | f0.5 | f1 | |
| | --- | --- | --- | --- | --- | --- | --- | --- | |
| | producer | 1909 | 968 | 914 | 0.6635 | 0.6762 | 0.6660 | 0.6698 | |
| | year | 2264 | 520 | 631 | 0.8132 | 0.7820 | 0.8068 | 0.7973 | |
| | geography | 2632 | 984 | 968 | 0.7279 | 0.7311 | 0.7285 | 0.7295 | |
| | acronym | 1973 | 456 | 449 | 0.8123 | 0.8146 | 0.8127 | 0.8134 | |
| | **overall** | 8778 | 2928 | 2962 | 0.7499 | 0.7477 | 0.7494 | 0.7488 | |
|
|
| Usage/impact macro-F1 (per head): |
| - data_type: 0.7005 |
| - usage_action: 0.6554 |
| - impact_label: 0.5842 |
| - usage_summary: mean_sim=0.5522 grounded_rate=0.6513 |
|
|
| Verbatim rate (emitted values that are substrings of the context): 11659/11706 = 0.9960 |
|
|
| ## synthetic holdout |
|
|
| Holdout n=2305. Exact string match of each emitted attribute against the gold label. |
|
|
| | attribute | tp | fp | fn | precision | recall | f0.5 | f1 | |
| | --- | --- | --- | --- | --- | --- | --- | --- | |
| | producer | 231 | 39 | 56 | 0.8556 | 0.8049 | 0.8449 | 0.8294 | |
| | year | 265 | 50 | 40 | 0.8413 | 0.8689 | 0.8466 | 0.8548 | |
| | geography | 344 | 63 | 53 | 0.8452 | 0.8665 | 0.8494 | 0.8557 | |
| | acronym | 145 | 84 | 36 | 0.6332 | 0.8011 | 0.6609 | 0.7073 | |
| | **overall** | 985 | 236 | 185 | 0.8067 | 0.8419 | 0.8135 | 0.8239 | |
|
|
| Usage/impact macro-F1 (per head): |
| - data_type: 0.7818 |
| - usage_action: 0.6489 |
| - impact_label: 0.5147 |
| - usage_summary: mean_sim=0.5752 grounded_rate=0.6768 |
|
|
| Verbatim rate (emitted values that are substrings of the context): 1216/1221 = 0.9959 |
|
|
|
|
