--- license: mit tags: - finance - llm - lora - sentiment-analysis - named-entity-recognition - xbrl - apollo - rag pipeline_tag: text-generation --- # FinLoRA: Financial Large Language Models with LoRA Adaptation [![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/) [![PyTorch](https://img.shields.io/badge/PyTorch-2.0+-red.svg)](https://pytorch.org/) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) ## Overview FinLoRA is a comprehensive framework for fine-tuning large language models on financial tasks using Low-Rank Adaptation (LoRA). This repository contains trained LoRA adapters for various financial NLP tasks including sentiment analysis, named entity recognition, headline classification, XBRL processing, **RAG-enhanced models** for CFA knowledge and FinTagging tasks, and **APOLLO reasoning layers** for advanced numerical calculations. ## Model Architecture - **Base Model**: Meta-Llama-3.1-8B-Instruct (downloaded locally) - **Adaptation Method**: LoRA (Low-Rank Adaptation) - **Quantization**: 8-bit and 4-bit quantization support - **Multi-Layer Support**: RAG + APOLLO layered architecture - **Local Usage**: All models run locally without requiring Hugging Face online access - **Tasks**: Financial sentiment analysis, NER, classification, XBRL processing, CFA knowledge, FinTagging, numerical reasoning ## Available Models ### 8-bit Quantized Models (Recommended) - `sentiment_llama_3_1_8b_8bits_r8` - Financial sentiment analysis - `ner_llama_3_1_8b_8bits_r8` - Named entity recognition - `headline_llama_3_1_8b_8bits_r8` - Financial headline classification - `xbrl_extract_llama_3_1_8b_8bits_r8` - XBRL tag extraction - `xbrl_term_llama_3_1_8b_8bits_r8` - XBRL terminology processing - `financebench_llama_3_1_8b_8bits_r8` - Comprehensive financial benchmark - `finer_llama_3_1_8b_8bits_r8` - Financial NER - `formula_llama_3_1_8b_8bits_r8` - Financial formula processing ### RAG-Enhanced Models (Knowledge-Augmented) - `cfa_rag_llama_3_1_8b_8bits_r8` - CFA knowledge-enhanced model with RAG - `fintagging_combined_rag_llama_3_1_8b_8bits_r8` - Combined FinTagging RAG model - `fintagging_fincl_rag_llama_3_1_8b_8bits_r8` - FinCL RAG-enhanced model - `fintagging_finni_rag_llama_3_1_8b_8bits_r8` - FinNI RAG-enhanced model ### APOLLO Models (Advanced Reasoning Layer) - `apollo_cfa_rag_llama_3_1_8b_8bits_r8` - APOLLO reasoning layer for CFA tasks - `apollo_fintagging_combined_llama_3_1_8b_8bits_r8` - APOLLO reasoning layer for FinTagging tasks **Note**: APOLLO models are designed to be loaded on top of RAG models for enhanced numerical reasoning and calculation capabilities. ### Bloomberg-Enhanced Models (Specialized Financial Tasks) - `finlora_lora_ckpt_llama_8bit_r8` - Bloomberg FPB and FIQA specialized model - `finlora_heads_llama_8bit_r8.pt` - Bloomberg model weights (71MB) **Note**: Bloomberg models are specialized for Financial Phrasebank (FPB) and Financial Question Answering (FIQA) tasks. ### 4-bit Quantized Models (Memory Efficient) - `sentiment_llama_3_1_8b_4bits_r4` - Financial sentiment analysis - `ner_llama_3_1_8b_4bits_r4` - Named entity recognition - `headline_llama_3_1_8b_4bits_r4` - Financial headline classification - `xbrl_extract_llama_3_1_8b_4bits_r4` - XBRL tag extraction - `xbrl_term_llama_3_1_8b_4bits_r4` - XBRL terminology processing - `financebench_llama_3_1_8b_4bits_r4` - Comprehensive financial benchmark - `finer_llama_3_1_8b_4bits_r4` - Financial NER - `formula_llama_3_1_8b_4bits_r4` - Financial formula processing ## Quick Start ### 1. Installation ```bash # Install dependencies pip install -r requirements.txt ``` ### 2. Local Model Setup **Important**: This project uses locally downloaded models, not online Hugging Face models. ```bash # The base Llama-3.1-8B-Instruct model will be automatically downloaded to local cache # No internet connection required after initial setup # All LoRA adapters are included in this repository ``` ### 3. Basic Usage ```python from inference import FinLoRAPredictor # Initialize predictor with 8-bit model (recommended) predictor = FinLoRAPredictor( model_name="sentiment_llama_3_1_8b_8bits_r8", use_4bit=False ) # Financial sentiment analysis sentiment = predictor.classify_sentiment( "The company's quarterly earnings exceeded expectations by 20%." ) print(f"Sentiment: {sentiment}") # Entity extraction entities = predictor.extract_entities( "Apple Inc. reported revenue of $394.3 billion in 2022." ) print(f"Entities: {entities}") ``` ### 4. Run Complete Test ```bash # Test all models (this will download the base Llama model if not present) python inference.py # Test specific model python -c " from inference import FinLoRAPredictor predictor = FinLoRAPredictor('sentiment_llama_3_1_8b_8bits_r8') print('Model loaded successfully!') " ``` ## Usage Examples ### Financial Sentiment Analysis ```python predictor = FinLoRAPredictor("sentiment_llama_3_1_8b_8bits_r8") # Test cases test_texts = [ "Stock prices are soaring to new heights.", "Revenue declined by 15% this quarter.", "The company maintained stable performance." ] for text in test_texts: sentiment = predictor.classify_sentiment(text) print(f"Text: {text}") print(f"Sentiment: {sentiment}\n") ``` ### Named Entity Recognition ```python predictor = FinLoRAPredictor("ner_llama_3_1_8b_8bits_r8") text = "Apple Inc. reported revenue of $394.3 billion in 2022." entities = predictor.extract_entities(text) print(f"Entities: {entities}") ``` ### XBRL Processing ```python predictor = FinLoRAPredictor("xbrl_extract_llama_3_1_8b_8bits_r8") text = "Total assets: $1,234,567,890. Current assets: $456,789,123." xbrl_tags = predictor.extract_xbrl_tags(text) print(f"XBRL Tags: {xbrl_tags}") ``` ### RAG-Enhanced Models ```python # CFA RAG-enhanced model for financial knowledge predictor = FinLoRAPredictor("cfa_rag_llama_3_1_8b_8bits_r8") # Enhanced financial analysis with CFA knowledge response = predictor.generate_response( "Explain the concept of discounted cash flow valuation" ) print(f"CFA Response: {response}") # FinTagging RAG models for financial information extraction fintagging_predictor = FinLoRAPredictor("fintagging_combined_rag_llama_3_1_8b_8bits_r8") # Extract financial information with enhanced context entities = fintagging_predictor.extract_entities( "Apple Inc. reported revenue of $394.3 billion in 2022." ) print(f"Enhanced Entities: {entities}") ``` ### APOLLO Models (Advanced Reasoning) **Important**: APOLLO models are designed for advanced numerical reasoning and should be used for complex financial calculations. ```python # Load APOLLO model for advanced reasoning apollo_predictor = FinLoRAPredictor("apollo_cfa_rag_llama_3_1_8b_8bits_r8") # Financial calculations and reasoning calculation = apollo_predictor.generate_response( "Calculate the present value of $10,000 received in 3 years with 5% annual discount rate" ) print(f"APOLLO Calculation: {calculation}") # Complex financial analysis analysis = apollo_predictor.generate_response( "Analyze the impact of a 2% interest rate increase on a 10-year bond with 3% coupon rate" ) print(f"APOLLO Analysis: {analysis}") # Formula processing formula_result = apollo_predictor.generate_response( "Solve: If a company has $1M revenue, 20% profit margin, and 10% growth rate, what's next year's profit?" ) print(f"APOLLO Formula Result: {formula_result}") ``` ### Multi-Layer LoRA Architecture (RAG + APOLLO) For maximum performance, you can combine RAG and APOLLO models: ```python # Step 1: Load RAG model for knowledge retrieval rag_predictor = FinLoRAPredictor("cfa_rag_llama_3_1_8b_8bits_r8") # Step 2: Load APOLLO model for reasoning (this will be layered on top) apollo_predictor = FinLoRAPredictor("apollo_cfa_rag_llama_3_1_8b_8bits_r8") # Use for complex financial reasoning tasks complex_query = """ Given the following financial data: - Revenue: $50M - Cost of Goods Sold: $30M - Operating Expenses: $15M - Tax Rate: 25% Calculate the net income and explain the calculation steps. """ response = apollo_predictor.generate_response(complex_query) print(f"Multi-Layer Response: {response}") ``` ### Bloomberg-Enhanced Models (FPB & FIQA Specialized Tasks) **Important**: Bloomberg models require special environment setup and are optimized for Financial Phrasebank (FPB) and Financial Question Answering (FIQA) tasks. #### Environment Setup for Bloomberg Models ```bash # 1. Create conda environment using the provided configuration conda env create -f finlora_hf_submission/Bloomberg_fpb_and_fiqa/environment_contrasim.yml # 2. Activate the environment conda activate finenv # 3. Navigate to the Bloomberg evaluation directory cd finlora_hf_submission/Bloomberg_fpb_and_fiqa/ ``` #### Testing Bloomberg Models on FPB and FIQA Datasets ```bash # Run Bloomberg model evaluation python trytry1.py ``` **Configuration Notes for Testing:** 1. **Dataset Configuration**: In `trytry1.py`, modify the `EVAL_FILES` line: ```python # Replace with your test datasets EVAL_FILES = ["fiqa_test.jsonl", "fpb_test.jsonl"] ``` 2. **Model Path Configuration**: For local testing, update the `BASE_DIR` in `trytry1.py`: ```python # For local Llama model deployment BASE_DIR = "path/to/your/local/llama/model" # For Hugging Face online model (original setting) BASE_DIR = "d04e592bb4f6aa9cfee91e2e20afa771667e1d4b" ``` 3. **Model Components**: - `ADAPTER_DIR`: Points to the LoRA adapter (`finlora_lora_ckpt_llama_8bit_r8`) - `HEADS_PATH`: Points to the model weights (`finlora_heads_llama_8bit_r8.pt`) #### Bloomberg Model Usage Example ```python # Bloomberg models are specialized for FPB and FIQA tasks # They provide enhanced performance on financial sentiment analysis # and financial question answering compared to standard models # The evaluation script automatically handles: # - Model loading and configuration # - Dataset processing # - Performance metrics calculation # - Memory management for large models ``` ## Local Model Management ### Model Storage - **Base Model**: Downloaded to `~/.cache/huggingface/transformers/` - **LoRA Adapters**: Stored in `models/` directory - **No Online Dependency**: All models run locally after initial download ### Model Loading Process 1. **Base Model**: Automatically downloaded on first use (~15GB) 2. **LoRA Adapters**: Loaded from local `models/` directory 3. **Quantization**: Applied during loading (8-bit or 4-bit) 4. **Device Detection**: Automatically uses GPU if available, falls back to CPU ### Performance Optimization ```python # For better performance on GPU predictor = FinLoRAPredictor( model_name="sentiment_llama_3_1_8b_8bits_r8", use_4bit=False # Use 8-bit for better performance ) # For memory-constrained environments predictor = FinLoRAPredictor( model_name="sentiment_llama_3_1_8b_4bits_r4", use_4bit=True # Use 4-bit for memory efficiency ) ``` ## Evaluation ### For Competition Organizers This section provides guidance for evaluating the submitted models: #### 1. Quick Model Test ```bash # Test if all models can be loaded successfully python test_submission.py ``` #### 2. Comprehensive Evaluation ```bash # Run full evaluation on all models and datasets python comprehensive_evaluation.py # Check results cat comprehensive_evaluation_results.json ``` #### 3. Incremental Evaluation ```bash # Run evaluation on missing tasks python incremental_evaluation.py # Check results cat incremental_evaluation_results.json ``` #### 4. APOLLO Model Testing ```bash # Test APOLLO reasoning capabilities python -c " from inference import FinLoRAPredictor apollo = FinLoRAPredictor('apollo_cfa_rag_llama_3_1_8b_8bits_r8') result = apollo.generate_response('Calculate 15% of $1000') print(f'APOLLO Test: {result}') " ``` #### 5. Bloomberg Model Testing (FPB & FIQA) ```bash # Setup Bloomberg environment conda env create -f finlora_hf_submission/Bloomberg_fpb_and_fiqa/environment_contrasim.yml conda activate finenv # Navigate to Bloomberg evaluation directory cd finlora_hf_submission/Bloomberg_fpb_and_fiqa/ # Configure test datasets in trytry1.py: # 1. Update EVAL_FILES = ["your_fiqa_test.jsonl", "your_fpb_test.jsonl"] # 2. Update BASE_DIR for local model path or keep original for Hugging Face # Run Bloomberg model evaluation python trytry1.py ``` ## Project Structure ``` finlora_hf_submission/ ├── models/ # 8-bit LoRA model adapters (15 models) │ ├── sentiment_llama_3_1_8b_8bits_r8/ │ ├── ner_llama_3_1_8b_8bits_r8/ │ ├── headline_llama_3_1_8b_8bits_r8/ │ ├── xbrl_extract_llama_3_1_8b_8bits_r8/ │ ├── xbrl_term_llama_3_1_8b_8bits_r8/ │ ├── financebench_llama_3_1_8b_8bits_r8/ │ ├── finer_llama_3_1_8b_8bits_r8/ │ ├── formula_llama_3_1_8b_8bits_r8/ │ ├── cfa_rag_llama_3_1_8b_8bits_r8/ # RAG-enhanced CFA model │ ├── fintagging_combined_rag_llama_3_1_8b_8bits_r8/ # Combined RAG │ ├── fintagging_fincl_rag_llama_3_1_8b_8bits_r8/ # FinCL RAG │ ├── fintagging_finni_rag_llama_3_1_8b_8bits_r8/ # FinNI RAG │ ├── apollo_cfa_rag_llama_3_1_8b_8bits_r8/ # APOLLO reasoning layer │ ├── apollo_fintagging_combined_llama_3_1_8b_8bits_r8/ # APOLLO reasoning layer │ └── xbrl_train.jsonl-meta-llama-Llama-3.1-8B-Instruct-8bits_r8/ ├── Bloomberg_fpb_and_fiqa/ # Bloomberg specialized models for FPB & FIQA │ ├── finlora_heads_llama_8bit_r8.pt │ ├── finlora_lora_ckpt_llama_8bit_r8/ │ ├── environment_contrasim.yml # Conda environment configuration │ └── trytry1.py # Bloomberg model evaluation script ├── models_4bit/ # 4-bit LoRA model adapters (8 models) │ ├── sentiment_llama_3_1_8b_4bits_r4/ │ ├── ner_llama_3_1_8b_4bits_r4/ │ ├── headline_llama_3_1_8b_4bits_r4/ │ ├── xbrl_extract_llama_3_1_8b_4bits_r4/ │ ├── xbrl_term_llama_3_1_8b_4bits_r4/ │ ├── financebench_llama_3_1_8b_4bits_r4/ │ ├── finer_llama_3_1_8b_4bits_r4/ │ └── formula_llama_3_1_8b_4bits_r4/ ├── testdata/ # Evaluation datasets │ ├── FinCL-eval-subset.csv │ └── FinNI-eval-subset.csv ├── rag_system/ # RAG system components ├── inference.py # Main inference script ├── comprehensive_evaluation.py # Full evaluation script ├── incremental_evaluation.py # Incremental evaluation ├── robust_incremental.py # Robust evaluation ├── missing_tests.py # Missing test detection ├── requirements.txt # Python dependencies └── README.md # This file ``` ## Environment Requirements ### Minimum Requirements (CPU Mode) - Python 3.8+ - PyTorch 2.0+ - 8GB RAM - No GPU required ### Recommended Requirements (GPU Mode) - Python 3.9+ - PyTorch 2.1+ - CUDA 11.8+ (for NVIDIA GPUs) - 16GB+ GPU memory - 32GB+ RAM ### Installation Instructions ```bash # 1. Clone or download this repository # 2. Install dependencies pip install -r requirements.txt # 3. For GPU support (optional but recommended) pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 # 4. Verify installation python -c "import torch; print(f'PyTorch version: {torch.__version__}'); print(f'CUDA available: {torch.cuda.is_available()}')" ``` ### Troubleshooting **If you encounter memory issues:** - Use 4-bit models instead of 8-bit models - Reduce batch size in inference - Use CPU mode if GPU memory is insufficient **If models fail to load:** - Ensure all model files are present in the correct directories - Check that the base model (Llama-3.1-8B-Instruct) can be downloaded from HuggingFace - Verify internet connection for initial model download **Important Notes for Competition Organizers:** - The base model (Llama-3.1-8B-Instruct) will be automatically downloaded from HuggingFace on first use (~15GB) - All LoRA adapters are included in this submission and do not require additional downloads - Models work in both CPU and GPU modes, with automatic device detection - APOLLO models provide enhanced reasoning capabilities for complex financial tasks - All models run locally without requiring ongoing internet connection ## Model Details ### Training Configuration - **LoRA Rank**: 8 - **LoRA Alpha**: 16 - **Learning Rate**: 1e-4 - **Batch Size**: 4 - **Epochs**: 3-5 - **Quantization**: 8-bit (BitsAndBytes) / 4-bit (NF4) ### Training Data - Financial Phrasebank - FinGPT datasets (NER, Headline, XBRL) - BloombergGPT financial datasets - Custom financial text datasets - APOLLO reasoning datasets for numerical calculations ## License This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. ## Contributing Contributions are welcome! Please feel free to submit a Pull Request. ## Contact For questions and support, please open an issue in the repository. ## Submission Summary ### What's Included - **17 Total Models**: 15 8-bit models (9 original + 4 RAG-enhanced + 2 APOLLO) + 8 4-bit models - **Complete Evaluation Results**: Comprehensive and incremental evaluation results - **RAG-Enhanced Models**: CFA and FinTagging models with enhanced knowledge - **APOLLO Reasoning**: Advanced numerical reasoning and calculation capabilities - **Cross-Platform Support**: Works on CPU, GPU, and various memory configurations - **Local Execution**: All models run locally without online dependencies - **Ready-to-Use**: All dependencies specified, automatic device detection ### Quick Start for Competition Organizers 1. Install dependencies: `pip install -r requirements.txt` 2. Test submission: `python test_submission.py` 3. Run evaluation: `python comprehensive_evaluation.py` 4. Test APOLLO reasoning: `python -c "from inference import FinLoRAPredictor; apollo = FinLoRAPredictor('apollo_cfa_rag_llama_3_1_8b_8bits_r8'); print(apollo.generate_response('Calculate 10% of 500'))"` 5. Test Bloomberg models (FPB & FIQA): ```bash conda env create -f finlora_hf_submission/Bloomberg_fpb_and_fiqa/environment_contrasim.yml conda activate finenv cd finlora_hf_submission/Bloomberg_fpb_and_fiqa/ # Configure EVAL_FILES and BASE_DIR in trytry1.py python trytry1.py ``` 6. Check results: `cat comprehensive_evaluation_results.json` ### Model Categories - **Financial NLP**: Sentiment, NER, Classification, XBRL processing - **RAG-Enhanced**: CFA knowledge and FinTagging with retrieval augmentation - **APOLLO Reasoning**: Advanced numerical calculations and financial reasoning - **Memory Options**: Both 8-bit and 4-bit quantized versions available ## Acknowledgments - Meta for the Llama-3.1-8B-Instruct base model - Hugging Face for the transformers and PEFT libraries - The financial NLP community for datasets and benchmarks - APOLLO reasoning framework for enhanced numerical capabilities