README.md

metadata

license: mit
tags:
  - finance
  - llm
  - lora
  - sentiment-analysis
  - named-entity-recognition
  - xbrl
pipeline_tag: text-generation

FinLoRA: Financial Large Language Models with LoRA Adaptation

Overview

FinLoRA is a comprehensive framework for fine-tuning large language models on financial tasks using Low-Rank Adaptation (LoRA). This repository contains trained LoRA adapters for various financial NLP tasks including sentiment analysis, named entity recognition, headline classification, XBRL processing, and RAG-enhanced models for CFA knowledge and FinTagging tasks.

Model Architecture

• Base Model: Meta-Llama-3.1-8B-Instruct
• Adaptation Method: LoRA (Low-Rank Adaptation)
• Quantization: 8-bit and 4-bit quantization support
• Tasks: Financial sentiment analysis, NER, classification, XBRL processing, CFA knowledge, FinTagging

Available Models

8-bit Quantized Models (Recommended)

• sentiment_llama_3_1_8b_8bits_r8 - Financial sentiment analysis
• ner_llama_3_1_8b_8bits_r8 - Named entity recognition
• headline_llama_3_1_8b_8bits_r8 - Financial headline classification
• xbrl_extract_llama_3_1_8b_8bits_r8 - XBRL tag extraction
• xbrl_term_llama_3_1_8b_8bits_r8 - XBRL terminology processing
• financebench_llama_3_1_8b_8bits_r8 - Comprehensive financial benchmark
• finer_llama_3_1_8b_8bits_r8 - Financial NER
• formula_llama_3_1_8b_8bits_r8 - Financial formula processing

RAG-Enhanced Models (New!)

• cfa_rag_llama_3_1_8b_8bits_r8 - CFA knowledge-enhanced model with RAG
• fintagging_combined_rag_llama_3_1_8b_8bits_r8 - Combined FinTagging RAG model
• fintagging_fincl_rag_llama_3_1_8b_8bits_r8 - FinCL RAG-enhanced model
• fintagging_finni_rag_llama_3_1_8b_8bits_r8 - FinNI RAG-enhanced model

4-bit Quantized Models (Memory Efficient)

• sentiment_llama_3_1_8b_4bits_r4 - Financial sentiment analysis
• ner_llama_3_1_8b_4bits_r4 - Named entity recognition
• headline_llama_3_1_8b_4bits_r4 - Financial headline classification
• xbrl_extract_llama_3_1_8b_4bits_r4 - XBRL tag extraction
• xbrl_term_llama_3_1_8b_4bits_r4 - XBRL terminology processing
• financebench_llama_3_1_8b_4bits_r4 - Comprehensive financial benchmark
• finer_llama_3_1_8b_4bits_r4 - Financial NER
• formula_llama_3_1_8b_4bits_r4 - Financial formula processing

Quick Start

1. Installation

# Install dependencies
pip install -r requirements.txt

2. Basic Usage

from inference import FinLoRAPredictor

# Initialize predictor with 8-bit model (recommended)
predictor = FinLoRAPredictor(
    model_name="sentiment_llama_3_1_8b_8bits_r8",
    use_4bit=False
)

# Financial sentiment analysis
sentiment = predictor.classify_sentiment(
    "The company's quarterly earnings exceeded expectations by 20%."
)
print(f"Sentiment: {sentiment}")

# Entity extraction
entities = predictor.extract_entities(
    "Apple Inc. reported revenue of $394.3 billion in 2022."
)
print(f"Entities: {entities}")

# Use 4-bit model for memory efficiency (if you have limited GPU memory)
predictor_4bit = FinLoRAPredictor(
    model_name="sentiment_llama_3_1_8b_4bits_r4",
    use_4bit=True
)

# CPU-only mode (if no GPU available)
predictor_cpu = FinLoRAPredictor(
    model_name="sentiment_llama_3_1_8b_8bits_r8",
    use_4bit=False
)
# The script will automatically detect CPU and adjust accordingly

3. Run Complete Test

# Test all models (this will download the base Llama model if not present)
python inference.py

# Test specific model
python -c "
from inference import FinLoRAPredictor
predictor = FinLoRAPredictor('sentiment_llama_3_1_8b_8bits_r8')
print('Model loaded successfully!')
"

Usage Examples

Financial Sentiment Analysis

predictor = FinLoRAPredictor("sentiment_llama_3_1_8b_8bits_r8")

# Test cases
test_texts = [
    "Stock prices are soaring to new heights.",
    "Revenue declined by 15% this quarter.",
    "The company maintained stable performance."
]

for text in test_texts:
    sentiment = predictor.classify_sentiment(text)
    print(f"Text: {text}")
    print(f"Sentiment: {sentiment}\n")

Named Entity Recognition

predictor = FinLoRAPredictor("ner_llama_3_1_8b_8bits_r8")

text = "Apple Inc. reported revenue of $394.3 billion in 2022."
entities = predictor.extract_entities(text)
print(f"Entities: {entities}")

XBRL Processing

predictor = FinLoRAPredictor("xbrl_extract_llama_3_1_8b_8bits_r8")

text = "Total assets: $1,234,567,890. Current assets: $456,789,123."
xbrl_tags = predictor.extract_xbrl_tags(text)
print(f"XBRL Tags: {xbrl_tags}")

RAG-Enhanced Models

# CFA RAG-enhanced model for financial knowledge
predictor = FinLoRAPredictor("cfa_rag_llama_3_1_8b_8bits_r8")

# Enhanced financial analysis with CFA knowledge
response = predictor.generate_response(
    "Explain the concept of discounted cash flow valuation"
)
print(f"CFA Response: {response}")

# FinTagging RAG models for financial information extraction
fintagging_predictor = FinLoRAPredictor("fintagging_combined_rag_llama_3_1_8b_8bits_r8")

# Extract financial information with enhanced context
entities = fintagging_predictor.extract_entities(
    "Apple Inc. reported revenue of $394.3 billion in 2022."
)
print(f"Enhanced Entities: {entities}")

Memory-Efficient 4-bit Models

# For users with limited GPU memory
predictor = FinLoRAPredictor(
    model_name="sentiment_llama_3_1_8b_4bits_r4",
    use_4bit=True
)

# Same API as 8-bit models
sentiment = predictor.classify_sentiment("The market is performing well.")

Evaluation

For Competition Organizers

This section provides guidance for evaluating the submitted models:

1. Quick Model Test

# Test if all models can be loaded successfully
python test_submission.py

2. Comprehensive Evaluation

# Run full evaluation on all models and datasets
python comprehensive_evaluation.py

# Check results
cat comprehensive_evaluation_results.json

3. Incremental Evaluation

# Run evaluation on missing tasks
python incremental_evaluation.py

# Check results
cat incremental_evaluation_results.json

4. Evaluation Results

The evaluation results are provided in:

• comprehensive_evaluation_results.json - Complete evaluation results
• incremental_evaluation_results.json - Missing task evaluation results

5. Model Performance Summary

All models have been evaluated on multiple financial datasets. See the Performance Results section below for detailed metrics.

For Researchers

Run comprehensive evaluation on financial datasets:

# Run full evaluation
python comprehensive_evaluation.py

# Run incremental evaluation
python incremental_evaluation.py

# Run robust evaluation
python robust_incremental.py

Performance Results

The models have been evaluated on multiple financial datasets:

Task	Dataset	F1 Score	Accuracy
Sentiment Analysis	Financial Phrasebank	0.333	0.500
NER	Financial NER	0.889	0.800
Classification	Headline Classification	0.697	0.700
XBRL Processing	XBRL Tag Extraction	-	0.200
Sentiment Analysis	FIQA SA	0.727	0.700

Project Structure

finlora_hf_submission/
├── models/                          # 8-bit LoRA model adapters (13 models)
│   ├── sentiment_llama_3_1_8b_8bits_r8/
│   ├── ner_llama_3_1_8b_8bits_r8/
│   ├── headline_llama_3_1_8b_8bits_r8/
│   ├── xbrl_extract_llama_3_1_8b_8bits_r8/
│   ├── xbrl_term_llama_3_1_8b_8bits_r8/
│   ├── financebench_llama_3_1_8b_8bits_r8/
│   ├── finer_llama_3_1_8b_8bits_r8/
│   ├── formula_llama_3_1_8b_8bits_r8/
│   ├── cfa_rag_llama_3_1_8b_8bits_r8/          # NEW: CFA RAG model
│   ├── fintagging_combined_rag_llama_3_1_8b_8bits_r8/  # NEW: Combined RAG
│   ├── fintagging_fincl_rag_llama_3_1_8b_8bits_r8/     # NEW: FinCL RAG
│   ├── fintagging_finni_rag_llama_3_1_8b_8bits_r8/     # NEW: FinNI RAG
│   └── xbrl_train.jsonl-meta-llama-Llama-3.1-8B-Instruct-8bits_r8/
├── models_4bit/                     # 4-bit LoRA model adapters (8 models)
│   ├── sentiment_llama_3_1_8b_4bits_r4/
│   ├── ner_llama_3_1_8b_4bits_r4/
│   ├── headline_llama_3_1_8b_4bits_r4/
│   ├── xbrl_extract_llama_3_1_8b_4bits_r4/
│   ├── xbrl_term_llama_3_1_8b_4bits_r4/
│   ├── financebench_llama_3_1_8b_4bits_r4/
│   ├── finer_llama_3_1_8b_4bits_r4/
│   └── formula_llama_3_1_8b_4bits_r4/
├── testdata/                        # Evaluation datasets
│   ├── FinCL-eval-subset.csv
│   └── FinNI-eval-subset.csv
├── rag_system/                      # RAG system components
├── inference.py                     # Main inference script
├── comprehensive_evaluation.py      # Full evaluation script
├── incremental_evaluation.py        # Incremental evaluation
├── robust_incremental.py           # Robust evaluation
├── missing_tests.py                # Missing test detection
├── requirements.txt                 # Python dependencies
└── README.md                       # This file

Environment Requirements

Minimum Requirements (CPU Mode)

• Python 3.8+
• PyTorch 2.0+
• 8GB RAM
• No GPU required

Recommended Requirements (GPU Mode)

• Python 3.9+
• PyTorch 2.1+
• CUDA 11.8+ (for NVIDIA GPUs)
• 16GB+ GPU memory
• 32GB+ RAM

Installation Instructions

# 1. Clone or download this repository
# 2. Install dependencies
pip install -r requirements.txt

# 3. For GPU support (optional but recommended)
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

# 4. Verify installation
python -c "import torch; print(f'PyTorch version: {torch.__version__}'); print(f'CUDA available: {torch.cuda.is_available()}')"

Troubleshooting

If you encounter memory issues:

• Use 4-bit models instead of 8-bit models
• Reduce batch size in inference
• Use CPU mode if GPU memory is insufficient

If models fail to load:

• Ensure all model files are present in the correct directories
• Check that the base model (Llama-3.1-8B-Instruct) can be downloaded from HuggingFace
• Verify internet connection for model downloads

Important Notes for Competition Organizers:

• The base model (Llama-3.1-8B-Instruct) will be automatically downloaded from HuggingFace on first use (~15GB)
• All LoRA adapters are included in this submission and do not require additional downloads
• Models work in both CPU and GPU modes, with automatic device detection
• RAG-enhanced models require the same base model as regular models

Model Details

Training Configuration

• LoRA Rank: 8
• LoRA Alpha: 16
• Learning Rate: 1e-4
• Batch Size: 4
• Epochs: 3-5
• Quantization: 8-bit (BitsAndBytes) / 4-bit (NF4)

Training Data

• Financial Phrasebank
• FinGPT datasets (NER, Headline, XBRL)
• BloombergGPT financial datasets
• Custom financial text datasets

Citation

If you use this work in your research, please cite:

@article{finlora2024,
  title={FinLoRA: Financial Large Language Models with LoRA Adaptation},
  author={Your Name},
  journal={Financial AI Conference},
  year={2024}
}

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Contact

For questions and support, please open an issue or contact your-email@example.com.

Submission Summary

What's Included

• 21 Total Models: 13 8-bit models (9 original + 4 RAG-enhanced) + 8 4-bit models
• Complete Evaluation Results: Comprehensive and incremental evaluation results
• RAG-Enhanced Models: CFA and FinTagging models with enhanced knowledge
• Cross-Platform Support: Works on CPU, GPU, and various memory configurations
• Ready-to-Use: All dependencies specified, automatic device detection

Quick Start for Competition Organizers

1. Install dependencies: pip install -r requirements.txt
2. Test submission: python test_submission.py
3. Run evaluation: python comprehensive_evaluation.py
4. Check results: cat comprehensive_evaluation_results.json

Model Categories

• Financial NLP: Sentiment, NER, Classification, XBRL processing
• RAG-Enhanced: CFA knowledge and FinTagging with retrieval augmentation
• Memory Options: Both 8-bit and 4-bit quantized versions available

Acknowledgments

• Meta for the Llama-3.1-8B-Instruct base model
• Hugging Face for the transformers and PEFT libraries
• The financial NLP community for datasets and benchmarks