README.md

---
license: mit
tags:
- finance
- llm
- lora
- sentiment-analysis
- named-entity-recognition
- xbrl
- apollo
- rag
pipeline_tag: text-generation
---

# FinLoRA: Financial Large Language Models with LoRA Adaptation

[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
[![PyTorch](https://img.shields.io/badge/PyTorch-2.0+-red.svg)](https://pytorch.org/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

## Overview

FinLoRA is a comprehensive framework for fine-tuning large language models on financial tasks using Low-Rank Adaptation (LoRA). This repository contains trained LoRA adapters for various financial NLP tasks including sentiment analysis, named entity recognition, headline classification, XBRL processing, **RAG-enhanced models** for CFA knowledge and FinTagging tasks, and **APOLLO reasoning layers** for advanced numerical calculations.

## Model Architecture

- **Base Model**: Meta-Llama-3.1-8B-Instruct (downloaded locally)
- **Adaptation Method**: LoRA (Low-Rank Adaptation)
- **Quantization**: 8-bit and 4-bit quantization support
- **Multi-Layer Support**: RAG + APOLLO layered architecture
- **Local Usage**: All models run locally without requiring Hugging Face online access
- **Tasks**: Financial sentiment analysis, NER, classification, XBRL processing, CFA knowledge, FinTagging, numerical reasoning

## Available Models

### 8-bit Quantized Models (Recommended)
- `sentiment_llama_3_1_8b_8bits_r8` - Financial sentiment analysis
- `ner_llama_3_1_8b_8bits_r8` - Named entity recognition
- `headline_llama_3_1_8b_8bits_r8` - Financial headline classification
- `xbrl_extract_llama_3_1_8b_8bits_r8` - XBRL tag extraction
- `xbrl_term_llama_3_1_8b_8bits_r8` - XBRL terminology processing
- `financebench_llama_3_1_8b_8bits_r8` - Comprehensive financial benchmark
- `finer_llama_3_1_8b_8bits_r8` - Financial NER
- `formula_llama_3_1_8b_8bits_r8` - Financial formula processing

### RAG-Enhanced Models (Knowledge-Augmented)
- `cfa_rag_llama_3_1_8b_8bits_r8` - CFA knowledge-enhanced model with RAG
- `fintagging_combined_rag_llama_3_1_8b_8bits_r8` - Combined FinTagging RAG model
- `fintagging_fincl_rag_llama_3_1_8b_8bits_r8` - FinCL RAG-enhanced model
- `fintagging_finni_rag_llama_3_1_8b_8bits_r8` - FinNI RAG-enhanced model

### APOLLO Models (Advanced Reasoning Layer)
- `apollo_cfa_rag_llama_3_1_8b_8bits_r8` - APOLLO reasoning layer for CFA tasks
- `apollo_fintagging_combined_llama_3_1_8b_8bits_r8` - APOLLO reasoning layer for FinTagging tasks

**Note**: APOLLO models are designed to be loaded on top of RAG models for enhanced numerical reasoning and calculation capabilities.

### Bloomberg-Enhanced Models (Specialized Financial Tasks)
- `finlora_lora_ckpt_llama_8bit_r8` - Bloomberg FPB and FIQA specialized model
- `finlora_heads_llama_8bit_r8.pt` - Bloomberg model weights (71MB)

**Note**: Bloomberg models are specialized for Financial Phrasebank (FPB) and Financial Question Answering (FIQA) tasks.

### 4-bit Quantized Models (Memory Efficient)
- `sentiment_llama_3_1_8b_4bits_r4` - Financial sentiment analysis
- `ner_llama_3_1_8b_4bits_r4` - Named entity recognition
- `headline_llama_3_1_8b_4bits_r4` - Financial headline classification
- `xbrl_extract_llama_3_1_8b_4bits_r4` - XBRL tag extraction
- `xbrl_term_llama_3_1_8b_4bits_r4` - XBRL terminology processing
- `financebench_llama_3_1_8b_4bits_r4` - Comprehensive financial benchmark
- `finer_llama_3_1_8b_4bits_r4` - Financial NER
- `formula_llama_3_1_8b_4bits_r4` - Financial formula processing

## Quick Start

### 1. Installation

```bash
# Install dependencies
pip install -r requirements.txt
```

### 2. Local Model Setup

**Important**: This project uses locally downloaded models, not online Hugging Face models.

```bash
# The base Llama-3.1-8B-Instruct model will be automatically downloaded to local cache
# No internet connection required after initial setup
# All LoRA adapters are included in this repository
```

### 3. Basic Usage

```python
from inference import FinLoRAPredictor

# Initialize predictor with 8-bit model (recommended)
predictor = FinLoRAPredictor(
    model_name="sentiment_llama_3_1_8b_8bits_r8",
    use_4bit=False
)

# Financial sentiment analysis
sentiment = predictor.classify_sentiment(
    "The company's quarterly earnings exceeded expectations by 20%."
)
print(f"Sentiment: {sentiment}")

# Entity extraction
entities = predictor.extract_entities(
    "Apple Inc. reported revenue of $394.3 billion in 2022."
)
print(f"Entities: {entities}")
```

### 4. Run Complete Test

```bash
# Test all models (this will download the base Llama model if not present)
python inference.py

# Test specific model
python -c "
from inference import FinLoRAPredictor
predictor = FinLoRAPredictor('sentiment_llama_3_1_8b_8bits_r8')
print('Model loaded successfully!')
"
```

## Usage Examples

### Financial Sentiment Analysis

```python
predictor = FinLoRAPredictor("sentiment_llama_3_1_8b_8bits_r8")

# Test cases
test_texts = [
    "Stock prices are soaring to new heights.",
    "Revenue declined by 15% this quarter.",
    "The company maintained stable performance."
]

for text in test_texts:
    sentiment = predictor.classify_sentiment(text)
    print(f"Text: {text}")
    print(f"Sentiment: {sentiment}\n")
```

### Named Entity Recognition

```python
predictor = FinLoRAPredictor("ner_llama_3_1_8b_8bits_r8")

text = "Apple Inc. reported revenue of $394.3 billion in 2022."
entities = predictor.extract_entities(text)
print(f"Entities: {entities}")
```

### XBRL Processing

```python
predictor = FinLoRAPredictor("xbrl_extract_llama_3_1_8b_8bits_r8")

text = "Total assets: $1,234,567,890. Current assets: $456,789,123."
xbrl_tags = predictor.extract_xbrl_tags(text)
print(f"XBRL Tags: {xbrl_tags}")
```

### RAG-Enhanced Models

```python
# CFA RAG-enhanced model for financial knowledge
predictor = FinLoRAPredictor("cfa_rag_llama_3_1_8b_8bits_r8")

# Enhanced financial analysis with CFA knowledge
response = predictor.generate_response(
    "Explain the concept of discounted cash flow valuation"
)
print(f"CFA Response: {response}")

# FinTagging RAG models for financial information extraction
fintagging_predictor = FinLoRAPredictor("fintagging_combined_rag_llama_3_1_8b_8bits_r8")

# Extract financial information with enhanced context
entities = fintagging_predictor.extract_entities(
    "Apple Inc. reported revenue of $394.3 billion in 2022."
)
print(f"Enhanced Entities: {entities}")
```

### APOLLO Models (Advanced Reasoning)

**Important**: APOLLO models are designed for advanced numerical reasoning and should be used for complex financial calculations.

```python
# Load APOLLO model for advanced reasoning
apollo_predictor = FinLoRAPredictor("apollo_cfa_rag_llama_3_1_8b_8bits_r8")

# Financial calculations and reasoning
calculation = apollo_predictor.generate_response(
    "Calculate the present value of $10,000 received in 3 years with 5% annual discount rate"
)
print(f"APOLLO Calculation: {calculation}")

# Complex financial analysis
analysis = apollo_predictor.generate_response(
    "Analyze the impact of a 2% interest rate increase on a 10-year bond with 3% coupon rate"
)
print(f"APOLLO Analysis: {analysis}")

# Formula processing
formula_result = apollo_predictor.generate_response(
    "Solve: If a company has $1M revenue, 20% profit margin, and 10% growth rate, what's next year's profit?"
)
print(f"APOLLO Formula Result: {formula_result}")
```

### Multi-Layer LoRA Architecture (RAG + APOLLO)

For maximum performance, you can combine RAG and APOLLO models:

```python
# Step 1: Load RAG model for knowledge retrieval
rag_predictor = FinLoRAPredictor("cfa_rag_llama_3_1_8b_8bits_r8")

# Step 2: Load APOLLO model for reasoning (this will be layered on top)
apollo_predictor = FinLoRAPredictor("apollo_cfa_rag_llama_3_1_8b_8bits_r8")

# Use for complex financial reasoning tasks
complex_query = """
Given the following financial data:
- Revenue: $50M
- Cost of Goods Sold: $30M
- Operating Expenses: $15M
- Tax Rate: 25%

Calculate the net income and explain the calculation steps.
"""

response = apollo_predictor.generate_response(complex_query)
print(f"Multi-Layer Response: {response}")
```

### Bloomberg-Enhanced Models (FPB & FIQA Specialized Tasks)

**Important**: Bloomberg models require special environment setup and are optimized for Financial Phrasebank (FPB) and Financial Question Answering (FIQA) tasks.

#### Environment Setup for Bloomberg Models

```bash
# 1. Create conda environment using the provided configuration
conda env create -f finlora_hf_submission/Bloomberg_fpb_and_fiqa/environment_contrasim.yml

# 2. Activate the environment
conda activate finenv

# 3. Navigate to the Bloomberg evaluation directory
cd finlora_hf_submission/Bloomberg_fpb_and_fiqa/
```

#### Testing Bloomberg Models on FPB and FIQA Datasets

```bash
# Run Bloomberg model evaluation
python trytry1.py
```

**Configuration Notes for Testing:**

1. **Dataset Configuration**: In `trytry1.py`, modify the `EVAL_FILES` line:
   ```python
   # Replace with your test datasets
   EVAL_FILES = ["fiqa_test.jsonl", "fpb_test.jsonl"]
   ```

2. **Model Path Configuration**: For local testing, update the `BASE_DIR` in `trytry1.py`:
   ```python
   # For local Llama model deployment
   BASE_DIR = "path/to/your/local/llama/model"
   
   # For Hugging Face online model (original setting)
   BASE_DIR = "d04e592bb4f6aa9cfee91e2e20afa771667e1d4b"
   ```

3. **Model Components**:
   - `ADAPTER_DIR`: Points to the LoRA adapter (`finlora_lora_ckpt_llama_8bit_r8`)
   - `HEADS_PATH`: Points to the model weights (`finlora_heads_llama_8bit_r8.pt`)

#### Bloomberg Model Usage Example

```python
# Bloomberg models are specialized for FPB and FIQA tasks
# They provide enhanced performance on financial sentiment analysis
# and financial question answering compared to standard models

# The evaluation script automatically handles:
# - Model loading and configuration
# - Dataset processing
# - Performance metrics calculation
# - Memory management for large models
```


## Local Model Management

### Model Storage
- **Base Model**: Downloaded to `~/.cache/huggingface/transformers/`
- **LoRA Adapters**: Stored in `models/` directory
- **No Online Dependency**: All models run locally after initial download

### Model Loading Process
1. **Base Model**: Automatically downloaded on first use (~15GB)
2. **LoRA Adapters**: Loaded from local `models/` directory
3. **Quantization**: Applied during loading (8-bit or 4-bit)
4. **Device Detection**: Automatically uses GPU if available, falls back to CPU

### Performance Optimization
```python
# For better performance on GPU
predictor = FinLoRAPredictor(
    model_name="sentiment_llama_3_1_8b_8bits_r8",
    use_4bit=False  # Use 8-bit for better performance
)

# For memory-constrained environments
predictor = FinLoRAPredictor(
    model_name="sentiment_llama_3_1_8b_4bits_r4",
    use_4bit=True   # Use 4-bit for memory efficiency
)
```

## Evaluation

### For Competition Organizers

This section provides guidance for evaluating the submitted models:

#### 1. Quick Model Test
```bash
# Test if all models can be loaded successfully
python test_submission.py
```

#### 2. Comprehensive Evaluation
```bash
# Run full evaluation on all models and datasets
python comprehensive_evaluation.py

# Check results
cat comprehensive_evaluation_results.json
```

#### 3. Incremental Evaluation
```bash
# Run evaluation on missing tasks
python incremental_evaluation.py

# Check results
cat incremental_evaluation_results.json
```

#### 4. APOLLO Model Testing
```bash
# Test APOLLO reasoning capabilities
python -c "
from inference import FinLoRAPredictor
apollo = FinLoRAPredictor('apollo_cfa_rag_llama_3_1_8b_8bits_r8')
result = apollo.generate_response('Calculate 15% of $1000')
print(f'APOLLO Test: {result}')
"
```

#### 5. Bloomberg Model Testing (FPB & FIQA)
```bash
# Setup Bloomberg environment
conda env create -f finlora_hf_submission/Bloomberg_fpb_and_fiqa/environment_contrasim.yml
conda activate finenv

# Navigate to Bloomberg evaluation directory
cd finlora_hf_submission/Bloomberg_fpb_and_fiqa/

# Configure test datasets in trytry1.py:
# 1. Update EVAL_FILES = ["your_fiqa_test.jsonl", "your_fpb_test.jsonl"]
# 2. Update BASE_DIR for local model path or keep original for Hugging Face

# Run Bloomberg model evaluation
python trytry1.py
```


## Project Structure

```
finlora_hf_submission/
├── models/                          # 8-bit LoRA model adapters (15 models)
│   ├── sentiment_llama_3_1_8b_8bits_r8/
│   ├── ner_llama_3_1_8b_8bits_r8/
│   ├── headline_llama_3_1_8b_8bits_r8/
│   ├── xbrl_extract_llama_3_1_8b_8bits_r8/
│   ├── xbrl_term_llama_3_1_8b_8bits_r8/
│   ├── financebench_llama_3_1_8b_8bits_r8/
│   ├── finer_llama_3_1_8b_8bits_r8/
│   ├── formula_llama_3_1_8b_8bits_r8/
│   ├── cfa_rag_llama_3_1_8b_8bits_r8/          # RAG-enhanced CFA model
│   ├── fintagging_combined_rag_llama_3_1_8b_8bits_r8/  # Combined RAG
│   ├── fintagging_fincl_rag_llama_3_1_8b_8bits_r8/     # FinCL RAG
│   ├── fintagging_finni_rag_llama_3_1_8b_8bits_r8/     # FinNI RAG
│   ├── apollo_cfa_rag_llama_3_1_8b_8bits_r8/           # APOLLO reasoning layer
│   ├── apollo_fintagging_combined_llama_3_1_8b_8bits_r8/ # APOLLO reasoning layer
│   └── xbrl_train.jsonl-meta-llama-Llama-3.1-8B-Instruct-8bits_r8/
├── Bloomberg_fpb_and_fiqa/          # Bloomberg specialized models for FPB & FIQA
│   ├── finlora_heads_llama_8bit_r8.pt
│   ├── finlora_lora_ckpt_llama_8bit_r8/
│   ├── environment_contrasim.yml    # Conda environment configuration
│   └── trytry1.py                   # Bloomberg model evaluation script
├── models_4bit/                     # 4-bit LoRA model adapters (8 models)
│   ├── sentiment_llama_3_1_8b_4bits_r4/
│   ├── ner_llama_3_1_8b_4bits_r4/
│   ├── headline_llama_3_1_8b_4bits_r4/
│   ├── xbrl_extract_llama_3_1_8b_4bits_r4/
│   ├── xbrl_term_llama_3_1_8b_4bits_r4/
│   ├── financebench_llama_3_1_8b_4bits_r4/
│   ├── finer_llama_3_1_8b_4bits_r4/
│   └── formula_llama_3_1_8b_4bits_r4/
├── testdata/                        # Evaluation datasets
│   ├── FinCL-eval-subset.csv
│   └── FinNI-eval-subset.csv
├── rag_system/                      # RAG system components
├── inference.py                     # Main inference script
├── comprehensive_evaluation.py      # Full evaluation script
├── incremental_evaluation.py        # Incremental evaluation
├── robust_incremental.py           # Robust evaluation
├── missing_tests.py                # Missing test detection
├── requirements.txt                 # Python dependencies
└── README.md                       # This file
```

## Environment Requirements

### Minimum Requirements (CPU Mode)
- Python 3.8+
- PyTorch 2.0+
- 8GB RAM
- No GPU required

### Recommended Requirements (GPU Mode)
- Python 3.9+
- PyTorch 2.1+
- CUDA 11.8+ (for NVIDIA GPUs)
- 16GB+ GPU memory
- 32GB+ RAM

### Installation Instructions

```bash
# 1. Clone or download this repository
# 2. Install dependencies
pip install -r requirements.txt

# 3. For GPU support (optional but recommended)
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

# 4. Verify installation
python -c "import torch; print(f'PyTorch version: {torch.__version__}'); print(f'CUDA available: {torch.cuda.is_available()}')"
```

### Troubleshooting

**If you encounter memory issues:**
- Use 4-bit models instead of 8-bit models
- Reduce batch size in inference
- Use CPU mode if GPU memory is insufficient

**If models fail to load:**
- Ensure all model files are present in the correct directories
- Check that the base model (Llama-3.1-8B-Instruct) can be downloaded from HuggingFace
- Verify internet connection for initial model download

**Important Notes for Competition Organizers:**
- The base model (Llama-3.1-8B-Instruct) will be automatically downloaded from HuggingFace on first use (~15GB)
- All LoRA adapters are included in this submission and do not require additional downloads
- Models work in both CPU and GPU modes, with automatic device detection
- APOLLO models provide enhanced reasoning capabilities for complex financial tasks
- All models run locally without requiring ongoing internet connection

## Model Details

### Training Configuration
- **LoRA Rank**: 8
- **LoRA Alpha**: 16
- **Learning Rate**: 1e-4
- **Batch Size**: 4
- **Epochs**: 3-5
- **Quantization**: 8-bit (BitsAndBytes) / 4-bit (NF4)

### Training Data
- Financial Phrasebank
- FinGPT datasets (NER, Headline, XBRL)
- BloombergGPT financial datasets
- Custom financial text datasets
- APOLLO reasoning datasets for numerical calculations



## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

## Contact

For questions and support, please open an issue in the repository.

## Submission Summary

### What's Included
- **17 Total Models**: 15 8-bit models (9 original + 4 RAG-enhanced + 2 APOLLO) + 8 4-bit models
- **Complete Evaluation Results**: Comprehensive and incremental evaluation results
- **RAG-Enhanced Models**: CFA and FinTagging models with enhanced knowledge
- **APOLLO Reasoning**: Advanced numerical reasoning and calculation capabilities
- **Cross-Platform Support**: Works on CPU, GPU, and various memory configurations
- **Local Execution**: All models run locally without online dependencies
- **Ready-to-Use**: All dependencies specified, automatic device detection

### Quick Start for Competition Organizers
1. Install dependencies: `pip install -r requirements.txt`
2. Test submission: `python test_submission.py`
3. Run evaluation: `python comprehensive_evaluation.py`
4. Test APOLLO reasoning: `python -c "from inference import FinLoRAPredictor; apollo = FinLoRAPredictor('apollo_cfa_rag_llama_3_1_8b_8bits_r8'); print(apollo.generate_response('Calculate 10% of 500'))"`
5. Test Bloomberg models (FPB & FIQA):
   ```bash
   conda env create -f finlora_hf_submission/Bloomberg_fpb_and_fiqa/environment_contrasim.yml
   conda activate finenv
   cd finlora_hf_submission/Bloomberg_fpb_and_fiqa/
   # Configure EVAL_FILES and BASE_DIR in trytry1.py
   python trytry1.py
   ```
6. Check results: `cat comprehensive_evaluation_results.json`

### Model Categories
- **Financial NLP**: Sentiment, NER, Classification, XBRL processing
- **RAG-Enhanced**: CFA knowledge and FinTagging with retrieval augmentation
- **APOLLO Reasoning**: Advanced numerical calculations and financial reasoning
- **Memory Options**: Both 8-bit and 4-bit quantized versions available

## Acknowledgments

- Meta for the Llama-3.1-8B-Instruct base model
- Hugging Face for the transformers and PEFT libraries
- The financial NLP community for datasets and benchmarks
- APOLLO reasoning framework for enhanced numerical capabilities