FinDan: a LoRA adapter for Financial PhraseBank sentiment

FinDan is a LoRA adapter trained on the FinBen FPB train and validation splits (3,876 sentences from Financial PhraseBank). Given a financial news sentence, the model answers with one word: negative, neutral or positive.

Read this first. FinDan is a research artifact from a pre-registered study, and the study's result was negative. On the FinBen FPB test split the adapter scores 0.9326 weighted F1 against 0.9382 for the base model it sits on. The difference is within noise (95% CI of the paired difference: βˆ’0.016 to +0.005). It is published so that the study can be checked, not as a drop-in improvement.

Model card

Base ChanceFocus/finma-7b-nlp, revision 94be8e68ca0b5fd90fbfb8cb87b130ea9e423270, loaded 4-bit NF4 with float16 compute (the exact loading used for both training and the test)
Adapter LoRA rank 16, alpha 32, dropout 0.05 on the q, k, v, o, gate, up, down projections of all 32 layers: 39,976,960 trainable parameters (0.59% of 6.74 B), stored float32, 160 MB
Training data FinBen FPB train + validation splits (3,100 + 776 = 3,876 sentences), label distribution of the train split: 60% neutral, 28% positive, 12% negative
Recipe 2 epochs = 484 steps, effective batch 16 (8 Γ— 2 accumulation), AdamW, lr 5e-5, cosine schedule with 3% warm-up, weight decay 0, fp16 autocast, seed 3407
Training run 42 min on one RTX 5060 Laptop GPU (8 GB), peak 6.9 GiB, final training loss 0.0518
Software transformers 4.40.2, peft 0.9.0, torch 2.7.1+cu128, bitsandbytes 0.46.1 (recipe.json lists the rest)
Identity of the tested weights sha256 of adapter_model.safetensors: 96dc4729755dc3b5738281394055db39558546ce65059f7fbe6c54d0f3f5b9cf
Licence CC BY-NC-SA 3.0 (see "Data and licences")

Results

Both systems were scored by the original PIXIU/FinBen evaluation harness, pinned to commit da45ac46 (with its financial-evaluation fork at f605cc8e), with the base model's own Human: / Assistant: prompt wrapper, the FinBen FPB instruction, greedy decoding and FinBen's parser and metric. The test split was scored once per system.

System (970 test sentences) weighted F1 accuracy macro F1 MCC missing
Base model 0.9382 0.9381 0.9347 0.8876 0
FinDan (this adapter) 0.9326 0.9330 0.9224 0.8769 0
Paired difference βˆ’0.0057, 95% bootstrap CI [βˆ’0.0163, +0.0047] (10,000 resamples, seed 2501)

Rows only the base got right: 16. Rows only FinDan got right: 11.

Before the test, the same recipe trained on the 3,100-sentence train split alone was selected on the validation split (776 sentences): three seeds averaged 0.9639 weighted F1 against the base model's 0.9499. The published adapter is the retrained train + validation version, so by design it has no held-out validation score of its own.

What the adapter actually learned

A post-hoc diagnosis shows that almost every prediction the adapter changes is moved to neutral. On validation this fixed 10 to 13 neutral sentences and broke 1. On the test split it fixed 10 and broke 14 positive or negative sentences, mostly sentences in the lowest annotator-agreement band (50%), where accuracy fell from 0.822 to 0.767. The validation gain was a label-convention bias that did not carry over, not a better reading of the sentences.

Caveats you should know

  • Not a pristine test. The FinBen FPB test split had been looked at in an earlier experiment by the same author before this study was pre-registered. The study discloses this; treat the test number as a confirmatory re-use.
  • The base model has seen the test sentences. Its instruction-tuning data lists all Financial PhraseBank sentences, of which the FinBen test split is a subset. Both numbers above may therefore include memorisation of the test sentences rather than pure generalisation; the size of that effect is not measured here.
  • Scope. English financial news sentences in Financial PhraseBank's style. Anything else is a distribution shift.

Prompt format

The adapter was trained and scored with this exact wrapper ({query} is FinBen's FPB query):

Human: 
{query}

Assistant: 

where the query is

Analyze the sentiment of this statement extracted from a financial news article. Provide your answer as either negative, positive, or neutral.
Text: {sentence}
Answer:

Note the trailing space after Human: and Assistant: and the newline after each. The model answers with one lowercase word.

Use it

The adapter contains no base weights. Load the base checkpoint, then the adapter (current peft versions give the same logits as the 0.9.0 used for training):

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

BASE = "ChanceFocus/finma-7b-nlp"
REV = "94be8e68ca0b5fd90fbfb8cb87b130ea9e423270"

tok = AutoTokenizer.from_pretrained(BASE, revision=REV, use_fast=False)
model = AutoModelForCausalLM.from_pretrained(
    BASE, revision=REV, torch_dtype=torch.float16, device_map="auto",
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.float16),
)
model = PeftModel.from_pretrained(model, "Algorex/FinDan-lora")

sentence = "Quarterly revenue rose 12% on strong demand for the new product line."
query = ("Analyze the sentiment of this statement extracted from a financial news article. "
         "Provide your answer as either negative, positive, or neutral.\n"
         f"Text: {sentence}\nAnswer:")
prompt = f"Human: \n{query}\n\nAssistant: \n"

inputs = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=4, do_sample=False)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip())

Pin the base revision: the harness that produced the numbers above refuses to run on any other.

Files

File What it is
adapter_model.safetensors, adapter_config.json The adapter, byte-identical to the tested weights
recipe.json The frozen training recipe and environment, written before training; local directory paths reduced to file names
train_log.jsonl Loss, gradient norm and learning rate every 10 steps

The registry name inside recipe.json (finma-finma_wrapper-r16-a32-attn_mlp-lr5e-5-e2-cosine-s3407-trainval) is the study's internal identifier and was left unchanged so that it matches the study's locked records.

Data and licences

  • Financial PhraseBank (Malo, Sinha, Korhonen, Wallenius and Takala, 2014) is licensed CC BY-NC-SA 3.0 by its authors. The FinBen copy used here (TheFinAI/en-fpb, formerly flare-fpb) currently carries a CC BY-SA 3.0 tag and earlier carried MIT; a redistributor cannot loosen the original terms, so this adapter is released under CC BY-NC-SA 3.0: research and academic use, attribution, share-alike. No dataset text is included in this repo.
  • Base checkpoint ChanceFocus/finma-7b-nlp is tagged MIT on its model card and is a LLaMA-architecture model. This repo contains none of its weights; obtain it from its own page under its own terms.
  • The FinBen prompt and scoring code come from PIXIU / FinBen (The-FinAI), MIT.

Citation

Malo, P., Sinha, A., Korhonen, P., Wallenius, J. and Takala, P. (2014). Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65(4), 782–796.

Xie, Q. et al. (2024). FinBen: A holistic financial benchmark for large language models.

Downloads last month
8
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for Algorex/FinDan-lora

Adapter
(1)
this model

Dataset used to train Algorex/FinDan-lora

Evaluation results