Upload BART LoRA finance model

README.md

@@ -1,123 +1,202 @@
 ---
-license: apache-2.0
+base_model: facebook/bart-large-cnn
+library_name: peft
 ---
-# Financial Question Answering with BART + LoRA Fine-Tuning
 
-## 1. Introduction
-This project focuses on financial question answering using recent news articles from trusted sources. Large language models (LLMs) often hallucinate or fail to produce factually grounded answers when it comes to domain-specific tasks like equity research. This is especially problematic in financial contexts where accuracy is critical. To address this, we fine-tuned the `facebook/bart-large-cnn` model using Low-Rank Adaptation (LoRA) on a custom financial QA dataset. After training, the model achieved a BLEU score of 0.218 and a ROUGE-L score of 0.431, significantly outperforming the base model and other comparison models.
+# Model Card for Model ID
 
-## 2. Training Data
-The dataset comprises eight financial news articles sourced from platforms such as Moneycontrol, Bloomberg, CNBC, and Yahoo Finance. Texts were loaded using LangChain's `UnstructuredURLLoader` and chunked using `RecursiveCharacterTextSplitter`. We then manually constructed QA pairs based on these article chunks. The dataset was split into training and validation sets using an 80/20 ratio with a fixed random seed (`random_state=42`). The data was stored in `financial_train.csv` and `financial_test.csv`.
+<!-- Provide a quick summary of what the model is/does. -->
 
-## 3. Training Method
-The chosen training method was LoRA (Low-Rank Adaptation), applied to the facebook/bart-large-cnn base model. LoRA was selected based on empirical results from prior experiments (mentioned in Step 1 rationale) where it consistently outperformed few-shot prompting and prompt tuning on metrics like BLEU, ROUGE-L, and Precision@1 for this financial QA task. It demonstrated better fluency, factual correctness, generalization to unseen questions (based on validation performance), and retention of pre-trained knowledge. LoRA's parameter efficiency was also a key advantage, allowing effective fine-tuning of a large model on limited data and computational resources. 
 
-We chose LoRA fine-tuning after comparing its performance to prompt tuning. LoRA allows efficient adaptation of large models using low-rank updates, keeping most parameters frozen. We fine-tuned `facebook/bart-large-cnn` using the following configuration:
 
-- `r=8`, `lora_alpha=16`, `lora_dropout=0.1`
-- `task_type=SEQ_2_SEQ_LM`, `target_modules=["q_proj", "v_proj"]`
-- Learning rate: 2e-4
-- Epochs: 3
-- Batch size: 2 per device
+## Model Details
 
-The model was trained using `Seq2SeqTrainer` with `load_best_model_at_end=True` and `evaluation_strategy="epoch"`.
+### Model Description
 
-## 4. Evaluation
-Evaluation focused on a custom financial question-answering task using the test/validation splits derived from the curated training data. The primary metrics used were BLEU (to assess n-gram overlap and fluency), ROUGE-L (to measure semantic similarity based on the longest common subsequence), and Precision@1 (to check if at least one relevant word from the reference answer was present in the prediction). These metrics were chosen to provide a quantitative measure of the model's ability to generate relevant, accurate, and well-formed answers based on the provided context. The base model (facebook/bart-large-cnn) was evaluated pre-training, and the LoRA-tuned model was evaluated post-training.
+<!-- Provide a longer summary of what this model is. -->
 
-We evaluated the model on three benchmark tasks:
-1. **Financial QA Test Set** (custom dataset)
-2. **GPT-2 Medium** (baseline)
-3. **FLAN-T5 Small** (comparison model)
-4. **BART-Large Base** (untuned version)
 
-### Results Table
-| Task                        | Model             | BLEU  | ROUGE-L | Precision@1 |
-|-----------------------------|-------------------|-------|----------|--------------|
-| Financial QA Test Set       | BART-Large (LoRA) | 0.218 | 0.431    | 1.000        |
-| Financial QA Test Set       | BART-Large (Base) | 0.000 | 0.056    | 0.667        |
-| Financial QA Test Set       | GPT-2 Medium      | 0.049 | 0.201    | 0.778        |
-| Financial QA Test Set       | FLAN-T5 Small     | 0.117 | 0.305    | 0.889        |
 
-**Summary:** The LoRA-tuned BART-Large model consistently outperformed all comparison models across all metrics.
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
 
-## 5. Usage and Intended Uses
-The model is intended for use in applications requiring question answering based on provided financial text, such as assisting equity research analysts, summarizing key points from news articles, or powering financial chatbots. It takes a question and a context passage as input and generates a concise answer based only on the information within that context.
-```python
+### Model Sources [optional]
 
+<!-- Provide the basic links for the model. -->
 
-import torch
-from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
-from peft import PeftModel, PeftConfig
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
 
-# Specify the path to your saved model repository on Hugging Face directory
-peft_model_path = ""deoleojr/bart-finance-lora"" 
-device = "cuda" if torch.cuda.is_available() else "cpu"
+## Uses
 
-# Load the configuration from the PEFT model path
-config = PeftConfig.from_pretrained(peft_model_path)
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
 
-# Load the base model
-base_model = AutoModelForSeq2SeqLM.from_pretrained(config.base_model_name_or_path).to(device)
+### Direct Use
 
-# Load the PEFT model (LoRA layers) on top of the base model
-model = PeftModel.from_pretrained(base_model, peft_model_path).to(device)
-model.eval() # Set model to evaluation mode
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
 
-# Load the tokenizer
-tokenizer = AutoTokenizer.from_pretrained(peft_model_path)
+[More Information Needed]
 
-# Example usage
-question = "What was the main reason for Tesla's stock rally?"
-context = "Tesla (TSLA.O) rallied 10% after Morgan Stanley upgraded the electric car maker to 'overweight' from 'equal-weight', saying its Dojo supercomputer could boost the company's market value by nearly $600 billion."
-prompt = f"Instruction: {question}\n\n[Context Information]\n{context}"
+### Downstream Use [optional]
 
-inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=512).to(device)
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
 
-with torch.no_grad():
-    outputs = model.generate(
-        **inputs,
-        max_new_tokens=100,
-        temperature=0.7,
-        top_k=50,
-        top_p=0.95,
-        do_sample=True # Set to False for deterministic output if preferred
-    )
+[More Information Needed]
 
-prediction = tokenizer.decode(outputs[0], skip_special_tokens=True).strip()
-print(f"Question: {question}")
-print(f"Generated Answer: {prediction}")
-```
-**Intended Use:** Financial analysts, researchers, or fintech developers needing factual QA over financial news articles. Not intended for real-time trading or automated financial advice.
+### Out-of-Scope Use
 
-## 6. Prompt Format
-The model expects prompts formatted with an "Instruction:" prefix for the question, followed by two newlines, the label "[Context Information]", a newline, and then the context text itself.
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
 
-```
-Instruction: {question}
+[More Information Needed]
 
-[Context Information]
-{news_article_chunk}
+## Bias, Risks, and Limitations
 
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
 
-Instruction: What is the impact of AI on Tesla’s stock movement?
+[More Information Needed]
 
-[Context Information]
-Tesla (TSLA.O) rallied 10% after Morgan Stanley upgraded the electric car maker to "overweight" from "equal-weight". Morgan Stanley said Tesla's Dojo supercomputer could boost the company's market value by nearly $600 billion.
+### Recommendations
 
-```
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
 
-## 7. Expected Output Format
-The model outputs a single block of text representing the answer generated based on the provided question and context.
-```
-A concise, fact-based answer derived from the provided context.
-Example:
-"Tesla's stock rose 10% after Morgan Stanley cited the potential of its Dojo supercomputer."
-```
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
 
-## 8. Limitations
-This model is trained on a limited set of eight news articles, which may restrict generalization to unseen financial domains. It is also susceptible to hallucination if the input lacks sufficient context. Furthermore, open-ended evaluation metrics like BLEU and ROUGE don't fully capture factual accuracy, so human evaluation or external fact-checking (e.g., via GPT-4) is still recommended for high-stakes applications.
+## How to Get Started with the Model
 
----
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
 
+[More Information Needed]
+### Framework versions
 
+- PEFT 0.14.0

adapter_config.json

@@ -0,0 +1,32 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "facebook/bart-large-cnn",
+  "bias": "none",
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "q_proj",
+    "v_proj"
+  ],
+  "task_type": "SEQ_2_SEQ_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:955937c095f977679f96b8effaf925e0595af29bbea963d4a13ccfcc405dae2c
+size 4738744

special_tokens_map.json

@@ -0,0 +1,15 @@
+{
+  "bos_token": "<s>",
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "</s>",
+  "sep_token": "</s>",
+  "unk_token": "<unk>"
+}

tokenizer_config.json

@@ -0,0 +1,58 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50264": {
+      "content": "<mask>",
+      "lstrip": true,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "mask_token": "<mask>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "</s>",
+  "sep_token": "</s>",
+  "tokenizer_class": "BartTokenizer",
+  "trim_offsets": true,
+  "unk_token": "<unk>"
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:effe253ae3437ff7bacb3654d592ac9f0855b840855638fa3748183202727d95
+size 5432