lfm2-provenance-smoke
LoRA SFT of LiquidAI/LFM2.5-350M for data-mention provenance attribute extraction
(producer / year / geography / acronym, verbatim from context).
Training
- base model:
LiquidAI/LFM2.5-350M - dataset:
rafmacalaba/data-use-provenance-sft - epochs: 2
- learning rate: 0.0002
- LoRA: r=16 alpha=32 dropout=0.05
- completion-only masking (loss on assistant JSON turn)
Evaluation (holdout, n=1000)
Exact string match of each emitted attribute against the gold label.
| attribute | tp | fp | fn | precision | recall | f0.5 | f1 |
|---|---|---|---|---|---|---|---|
| producer | 209 | 243 | 203 | 0.4624 | 0.5073 | 0.4707 | 0.4838 |
| year | 268 | 147 | 118 | 0.6458 | 0.6943 | 0.6549 | 0.6692 |
| geography | 399 | 357 | 172 | 0.5278 | 0.6988 | 0.5549 | 0.6014 |
| acronym | 363 | 154 | 51 | 0.7021 | 0.8768 | 0.7313 | 0.7798 |
| overall | 1239 | 901 | 544 | 0.5790 | 0.6949 | 0.5990 | 0.6317 |
Verbatim rate (emitted values that are substrings of the context): 2077/2140 = 0.9706
Sample predictions (holdout)
| mention | target | predicted | correct | verbatim |
|---|---|---|---|---|
| plant-level data | {"geography": "Indonesian"} | {"geography": "Indonesian", "producer": "NTMit"} | geography | producer,geography |
| GIDD database | {"geography": "all regions in the world"} | {"acronym": "GIDD", "geography": "all regions in the world"} | geography | geography,acronym |
| NFHS-3 data | {"acronym": "NFHS-3", "geography": "India"} | {"acronym": "NFHS-3", "geography": "India"} | geography,acronym | geography,acronym |
| data from Hallegatte ( 2012 ) | {"year": "2012"} | {"geography": "LICs", "year": "2012"} | year | year,geography |
| national accounts | {"geography": "Turkey"} | {"geography": "Turkey"} | geography | geography |
| SRF registry of firms | {"acronym": "SRF", "geography": "Brazil", "producer": "Brazilian tax authority", "year": "2012"} | {"acronym": "SRF", "geography": "Brazil", "producer": "Complementary Law 128", "year": "2008"} | geography,acronym | producer,year,geography,acronym |
| Human Capital Index | {"geography": "Kenya", "producer": "World Bank", "year": "2020"} | {"geography": "Kenya", "producer": "World Bank", "year": "2020"} | producer,year,geography | producer,year,geography |
| United Nations population counts by age | {"producer": "United Nations"} | {"geography": "Afghanistan", "producer": "United Nations"} | producer | producer,geography |
| 2002 Census | {"geography": "Rwanda", "year": "2002"} | {"geography": "Rwanda", "year": "2002"} | year,geography | year,geography |
| ENCASEH 1997 | {"acronym": "ENCASEH", "year": "1997"} | {"acronym": "ENCASEH", "geography": "country", "year": "1997"} | year,acronym | year,geography,acronym |
| Agricultural Wages in India | {"acronym": "AWI", "geography": "India", "producer": "Ministry of Agriculture"} | {"acronym": "AWI", "geography": "India"} | geography,acronym | geography,acronym |
| COVID-19 Enterprise Survey | {"geography": "Chad", "year": "2020"} | {"acronym": "ES", "geography": "Chad", "year": "2020"} | year,geography | year,geography,acronym |