Text Generation
PEFT
Safetensors
English
lora
finance
sentiment-analysis
financial-phrasebank
finben
negative-result
Eval Results (legacy)
Instructions to use Algorex/FinDan-lora with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use Algorex/FinDan-lora with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("ChanceFocus/finma-7b-nlp") model = PeftModel.from_pretrained(base_model, "Algorex/FinDan-lora") - Notebooks
- Google Colab
- Kaggle
|
Download README.md from Algorex/FinDan-lora: direct link, hf CLI and curl.
- Browser
- Download file 8.42 kB
-
https://huggingface.co/Algorex/FinDan-lora/resolve/main/README.md
- Command line
-
hf download hf://Algorex/FinDan-lora/README.md
-
curl -L -o README.md https://huggingface.co/Algorex/FinDan-lora/resolve/main/README.md
8.42 kB
| license: cc-by-nc-sa-3.0 | |
| base_model: ChanceFocus/finma-7b-nlp | |
| library_name: peft | |
| pipeline_tag: text-generation | |
| language: | |
| - en | |
| datasets: | |
| - TheFinAI/en-fpb | |
| tags: | |
| - lora | |
| - finance | |
| - sentiment-analysis | |
| - financial-phrasebank | |
| - finben | |
| - negative-result | |
| model-index: | |
| - name: FinDan-lora | |
| results: | |
| - task: | |
| type: text-classification | |
| name: Financial sentence sentiment (negative / neutral / positive) | |
| dataset: | |
| type: TheFinAI/en-fpb | |
| name: FinBen FPB, test split (970 sentences) | |
| split: test | |
| metrics: | |
| - type: f1 | |
| name: weighted F1 | |
| value: 0.9326 | |
| - type: accuracy | |
| name: accuracy | |
| value: 0.9330 | |
| # FinDan: a LoRA adapter for Financial PhraseBank sentiment | |
| FinDan is a LoRA adapter trained on the FinBen FPB train and validation splits (3,876 sentences from Financial PhraseBank). | |
| Given a financial news sentence, the model answers with one word: `negative`, `neutral` or `positive`. | |
| **Read this first.** FinDan is a research artifact from a pre-registered study, and the study's result was negative. | |
| On the FinBen FPB test split the adapter scores **0.9326 weighted F1 against 0.9382 for the base model it sits on**. | |
| The difference is within noise (95% CI of the paired difference: −0.016 to +0.005). It is published so that the study | |
| can be checked, not as a drop-in improvement. | |
| ## Model card | |
| | | | | |
| |---|---| | |
| | Base | `ChanceFocus/finma-7b-nlp`, revision `94be8e68ca0b5fd90fbfb8cb87b130ea9e423270`, loaded 4-bit NF4 with float16 compute (the exact loading used for both training and the test) | | |
| | Adapter | LoRA rank 16, alpha 32, dropout 0.05 on the `q, k, v, o, gate, up, down` projections of all 32 layers: 39,976,960 trainable parameters (0.59% of 6.74 B), stored float32, 160 MB | | |
| | Training data | FinBen FPB train + validation splits (3,100 + 776 = 3,876 sentences), label distribution of the train split: 60% neutral, 28% positive, 12% negative | | |
| | Recipe | 2 epochs = 484 steps, effective batch 16 (8 × 2 accumulation), AdamW, lr 5e-5, cosine schedule with 3% warm-up, weight decay 0, fp16 autocast, seed 3407 | | |
| | Training run | 42 min on one RTX 5060 Laptop GPU (8 GB), peak 6.9 GiB, final training loss 0.0518 | | |
| | Software | transformers 4.40.2, peft 0.9.0, torch 2.7.1+cu128, bitsandbytes 0.46.1 (`recipe.json` lists the rest) | | |
| | Identity of the tested weights | sha256 of `adapter_model.safetensors`: `96dc4729755dc3b5738281394055db39558546ce65059f7fbe6c54d0f3f5b9cf` | | |
| | Licence | CC BY-NC-SA 3.0 (see "Data and licences") | | |
| ## Results | |
| Both systems were scored by the original PIXIU/FinBen evaluation harness, pinned to commit `da45ac46` (with its | |
| financial-evaluation fork at `f605cc8e`), with the base model's own `Human: / Assistant:` prompt wrapper, the FinBen | |
| FPB instruction, greedy decoding and FinBen's parser and metric. The test split was scored once per system. | |
| | System (970 test sentences) | weighted F1 | accuracy | macro F1 | MCC | missing | | |
| |---|---|---|---|---|---| | |
| | Base model | 0.9382 | 0.9381 | 0.9347 | 0.8876 | 0 | | |
| | **FinDan (this adapter)** | **0.9326** | 0.9330 | 0.9224 | 0.8769 | 0 | | |
| | Paired difference | −0.0057, 95% bootstrap CI [−0.0163, +0.0047] (10,000 resamples, seed 2501) | | | | | | |
| Rows only the base got right: 16. Rows only FinDan got right: 11. | |
| Before the test, the same recipe trained on the 3,100-sentence train split alone was selected on the validation split | |
| (776 sentences): three seeds averaged 0.9639 weighted F1 against the base model's 0.9499. The published adapter is the | |
| retrained train + validation version, so by design it has **no held-out validation score** of its own. | |
| ### What the adapter actually learned | |
| A post-hoc diagnosis shows that almost every prediction the adapter changes is moved **to `neutral`**. On validation this | |
| fixed 10 to 13 neutral sentences and broke 1. On the test split it fixed 10 and broke 14 positive or negative sentences, | |
| mostly sentences in the lowest annotator-agreement band (50%), where accuracy fell from 0.822 to 0.767. The validation | |
| gain was a label-convention bias that did not carry over, not a better reading of the sentences. | |
| ### Caveats you should know | |
| - **Not a pristine test.** The FinBen FPB test split had been looked at in an earlier experiment by the same author | |
| before this study was pre-registered. The study discloses this; treat the test number as a confirmatory re-use. | |
| - **The base model has seen the test sentences.** Its instruction-tuning data lists all Financial PhraseBank sentences, | |
| of which the FinBen test split is a subset. Both numbers above may therefore include memorisation of the test | |
| sentences rather than pure generalisation; the size of that effect is not measured here. | |
| - **Scope.** English financial news sentences in Financial PhraseBank's style. Anything else is a distribution shift. | |
| ## Prompt format | |
| The adapter was trained and scored with this exact wrapper (`{query}` is FinBen's FPB query): | |
| ``` | |
| Human: | |
| {query} | |
| Assistant: | |
| ``` | |
| where the query is | |
| ``` | |
| Analyze the sentiment of this statement extracted from a financial news article. Provide your answer as either negative, positive, or neutral. | |
| Text: {sentence} | |
| Answer: | |
| ``` | |
| Note the trailing space after `Human:` and `Assistant:` and the newline after each. The model answers with one | |
| lowercase word. | |
| ## Use it | |
| The adapter contains no base weights. Load the base checkpoint, then the adapter (current `peft` versions give the same | |
| logits as the 0.9.0 used for training): | |
| ```python | |
| import torch | |
| from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig | |
| from peft import PeftModel | |
| BASE = "ChanceFocus/finma-7b-nlp" | |
| REV = "94be8e68ca0b5fd90fbfb8cb87b130ea9e423270" | |
| tok = AutoTokenizer.from_pretrained(BASE, revision=REV, use_fast=False) | |
| model = AutoModelForCausalLM.from_pretrained( | |
| BASE, revision=REV, torch_dtype=torch.float16, device_map="auto", | |
| quantization_config=BitsAndBytesConfig( | |
| load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.float16), | |
| ) | |
| model = PeftModel.from_pretrained(model, "Algorex/FinDan-lora") | |
| sentence = "Quarterly revenue rose 12% on strong demand for the new product line." | |
| query = ("Analyze the sentiment of this statement extracted from a financial news article. " | |
| "Provide your answer as either negative, positive, or neutral.\n" | |
| f"Text: {sentence}\nAnswer:") | |
| prompt = f"Human: \n{query}\n\nAssistant: \n" | |
| inputs = tok(prompt, return_tensors="pt").to(model.device) | |
| out = model.generate(**inputs, max_new_tokens=4, do_sample=False) | |
| print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip()) | |
| ``` | |
| Pin the base revision: the harness that produced the numbers above refuses to run on any other. | |
| ## Files | |
| | File | What it is | | |
| |---|---| | |
| | `adapter_model.safetensors`, `adapter_config.json` | The adapter, byte-identical to the tested weights | | |
| | `recipe.json` | The frozen training recipe and environment, written before training; local directory paths reduced to file names | | |
| | `train_log.jsonl` | Loss, gradient norm and learning rate every 10 steps | | |
| The registry name inside `recipe.json` (`finma-finma_wrapper-r16-a32-attn_mlp-lr5e-5-e2-cosine-s3407-trainval`) is the | |
| study's internal identifier and was left unchanged so that it matches the study's locked records. | |
| ## Data and licences | |
| - **Financial PhraseBank** (Malo, Sinha, Korhonen, Wallenius and Takala, 2014) is licensed CC BY-NC-SA 3.0 by its | |
| authors. The FinBen copy used here (`TheFinAI/en-fpb`, formerly `flare-fpb`) currently carries a CC BY-SA 3.0 tag | |
| and earlier carried MIT; a redistributor cannot loosen the original terms, so this adapter is released under | |
| **CC BY-NC-SA 3.0**: research and academic use, attribution, share-alike. No dataset text is included in this repo. | |
| - **Base checkpoint** `ChanceFocus/finma-7b-nlp` is tagged MIT on its model card and is a LLaMA-architecture model. | |
| This repo contains none of its weights; obtain it from its own page under its own terms. | |
| - The FinBen prompt and scoring code come from PIXIU / FinBen (The-FinAI), MIT. | |
| ## Citation | |
| Malo, P., Sinha, A., Korhonen, P., Wallenius, J. and Takala, P. (2014). Good debt or bad debt: Detecting semantic | |
| orientations in economic texts. *Journal of the Association for Information Science and Technology*, 65(4), 782–796. | |
| Xie, Q. et al. (2024). FinBen: A holistic financial benchmark for large language models. | |