FinDan-lora / README.md
Algorex's picture
Card: memorisation caveat stated as a possibility, not a conclusion
70e6e3c verified
|
Raw History Blame Contribute Delete
8.42 kB
---
license: cc-by-nc-sa-3.0
base_model: ChanceFocus/finma-7b-nlp
library_name: peft
pipeline_tag: text-generation
language:
- en
datasets:
- TheFinAI/en-fpb
tags:
- lora
- finance
- sentiment-analysis
- financial-phrasebank
- finben
- negative-result
model-index:
- name: FinDan-lora
results:
- task:
type: text-classification
name: Financial sentence sentiment (negative / neutral / positive)
dataset:
type: TheFinAI/en-fpb
name: FinBen FPB, test split (970 sentences)
split: test
metrics:
- type: f1
name: weighted F1
value: 0.9326
- type: accuracy
name: accuracy
value: 0.9330
---
# FinDan: a LoRA adapter for Financial PhraseBank sentiment
FinDan is a LoRA adapter trained on the FinBen FPB train and validation splits (3,876 sentences from Financial PhraseBank).
Given a financial news sentence, the model answers with one word: `negative`, `neutral` or `positive`.
**Read this first.** FinDan is a research artifact from a pre-registered study, and the study's result was negative.
On the FinBen FPB test split the adapter scores **0.9326 weighted F1 against 0.9382 for the base model it sits on**.
The difference is within noise (95% CI of the paired difference: −0.016 to +0.005). It is published so that the study
can be checked, not as a drop-in improvement.
## Model card
| | |
|---|---|
| Base | `ChanceFocus/finma-7b-nlp`, revision `94be8e68ca0b5fd90fbfb8cb87b130ea9e423270`, loaded 4-bit NF4 with float16 compute (the exact loading used for both training and the test) |
| Adapter | LoRA rank 16, alpha 32, dropout 0.05 on the `q, k, v, o, gate, up, down` projections of all 32 layers: 39,976,960 trainable parameters (0.59% of 6.74 B), stored float32, 160 MB |
| Training data | FinBen FPB train + validation splits (3,100 + 776 = 3,876 sentences), label distribution of the train split: 60% neutral, 28% positive, 12% negative |
| Recipe | 2 epochs = 484 steps, effective batch 16 (8 × 2 accumulation), AdamW, lr 5e-5, cosine schedule with 3% warm-up, weight decay 0, fp16 autocast, seed 3407 |
| Training run | 42 min on one RTX 5060 Laptop GPU (8 GB), peak 6.9 GiB, final training loss 0.0518 |
| Software | transformers 4.40.2, peft 0.9.0, torch 2.7.1+cu128, bitsandbytes 0.46.1 (`recipe.json` lists the rest) |
| Identity of the tested weights | sha256 of `adapter_model.safetensors`: `96dc4729755dc3b5738281394055db39558546ce65059f7fbe6c54d0f3f5b9cf` |
| Licence | CC BY-NC-SA 3.0 (see "Data and licences") |
## Results
Both systems were scored by the original PIXIU/FinBen evaluation harness, pinned to commit `da45ac46` (with its
financial-evaluation fork at `f605cc8e`), with the base model's own `Human: / Assistant:` prompt wrapper, the FinBen
FPB instruction, greedy decoding and FinBen's parser and metric. The test split was scored once per system.
| System (970 test sentences) | weighted F1 | accuracy | macro F1 | MCC | missing |
|---|---|---|---|---|---|
| Base model | 0.9382 | 0.9381 | 0.9347 | 0.8876 | 0 |
| **FinDan (this adapter)** | **0.9326** | 0.9330 | 0.9224 | 0.8769 | 0 |
| Paired difference | −0.0057, 95% bootstrap CI [−0.0163, +0.0047] (10,000 resamples, seed 2501) | | | | |
Rows only the base got right: 16. Rows only FinDan got right: 11.
Before the test, the same recipe trained on the 3,100-sentence train split alone was selected on the validation split
(776 sentences): three seeds averaged 0.9639 weighted F1 against the base model's 0.9499. The published adapter is the
retrained train + validation version, so by design it has **no held-out validation score** of its own.
### What the adapter actually learned
A post-hoc diagnosis shows that almost every prediction the adapter changes is moved **to `neutral`**. On validation this
fixed 10 to 13 neutral sentences and broke 1. On the test split it fixed 10 and broke 14 positive or negative sentences,
mostly sentences in the lowest annotator-agreement band (50%), where accuracy fell from 0.822 to 0.767. The validation
gain was a label-convention bias that did not carry over, not a better reading of the sentences.
### Caveats you should know
- **Not a pristine test.** The FinBen FPB test split had been looked at in an earlier experiment by the same author
before this study was pre-registered. The study discloses this; treat the test number as a confirmatory re-use.
- **The base model has seen the test sentences.** Its instruction-tuning data lists all Financial PhraseBank sentences,
of which the FinBen test split is a subset. Both numbers above may therefore include memorisation of the test
sentences rather than pure generalisation; the size of that effect is not measured here.
- **Scope.** English financial news sentences in Financial PhraseBank's style. Anything else is a distribution shift.
## Prompt format
The adapter was trained and scored with this exact wrapper (`{query}` is FinBen's FPB query):
```
Human:
{query}
Assistant:
```
where the query is
```
Analyze the sentiment of this statement extracted from a financial news article. Provide your answer as either negative, positive, or neutral.
Text: {sentence}
Answer:
```
Note the trailing space after `Human:` and `Assistant:` and the newline after each. The model answers with one
lowercase word.
## Use it
The adapter contains no base weights. Load the base checkpoint, then the adapter (current `peft` versions give the same
logits as the 0.9.0 used for training):
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
BASE = "ChanceFocus/finma-7b-nlp"
REV = "94be8e68ca0b5fd90fbfb8cb87b130ea9e423270"
tok = AutoTokenizer.from_pretrained(BASE, revision=REV, use_fast=False)
model = AutoModelForCausalLM.from_pretrained(
BASE, revision=REV, torch_dtype=torch.float16, device_map="auto",
quantization_config=BitsAndBytesConfig(
load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.float16),
)
model = PeftModel.from_pretrained(model, "Algorex/FinDan-lora")
sentence = "Quarterly revenue rose 12% on strong demand for the new product line."
query = ("Analyze the sentiment of this statement extracted from a financial news article. "
"Provide your answer as either negative, positive, or neutral.\n"
f"Text: {sentence}\nAnswer:")
prompt = f"Human: \n{query}\n\nAssistant: \n"
inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=4, do_sample=False)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip())
```
Pin the base revision: the harness that produced the numbers above refuses to run on any other.
## Files
| File | What it is |
|---|---|
| `adapter_model.safetensors`, `adapter_config.json` | The adapter, byte-identical to the tested weights |
| `recipe.json` | The frozen training recipe and environment, written before training; local directory paths reduced to file names |
| `train_log.jsonl` | Loss, gradient norm and learning rate every 10 steps |
The registry name inside `recipe.json` (`finma-finma_wrapper-r16-a32-attn_mlp-lr5e-5-e2-cosine-s3407-trainval`) is the
study's internal identifier and was left unchanged so that it matches the study's locked records.
## Data and licences
- **Financial PhraseBank** (Malo, Sinha, Korhonen, Wallenius and Takala, 2014) is licensed CC BY-NC-SA 3.0 by its
authors. The FinBen copy used here (`TheFinAI/en-fpb`, formerly `flare-fpb`) currently carries a CC BY-SA 3.0 tag
and earlier carried MIT; a redistributor cannot loosen the original terms, so this adapter is released under
**CC BY-NC-SA 3.0**: research and academic use, attribution, share-alike. No dataset text is included in this repo.
- **Base checkpoint** `ChanceFocus/finma-7b-nlp` is tagged MIT on its model card and is a LLaMA-architecture model.
This repo contains none of its weights; obtain it from its own page under its own terms.
- The FinBen prompt and scoring code come from PIXIU / FinBen (The-FinAI), MIT.
## Citation
Malo, P., Sinha, A., Korhonen, P., Wallenius, J. and Takala, P. (2014). Good debt or bad debt: Detecting semantic
orientations in economic texts. *Journal of the Association for Information Science and Technology*, 65(4), 782–796.
Xie, Q. et al. (2024). FinBen: A holistic financial benchmark for large language models.