FINGPT_MIGRATION.md

FinGPT Migration Guide

Overview

This document describes the migration from Google Gemini 2.5 Pro to FinGPT model for the FinRobot Forecaster application.

Changes Made

1. Dependencies Updated

• Removed: google-generativeai>=0.3.0
• Added:
  • transformers>=4.30.0
  • torch>=2.0.0
  • accelerate>=0.20.0
  • peft>=0.4.0
  • bitsandbytes>=0.39.0

2. Model Configuration

• Old: GEMINI_MODEL = "gemini-2.5-pro"
• New: FINGPT_MODEL_NAME = "Starfish55/fingpt-complete"

3. API Key Changes

• Old: GOOGLE_API_KEYS environment variable
• New: HF_TOKEN environment variable for Hugging Face access

4. Model Loading

• Old: Google Generative AI client initialization
• New: Hugging Face Transformers with BitsAndBytesConfig for memory efficiency

5. Inference Function

• Old: genai.GenerativeModel.generate_content()
• New: model.generate() with custom tokenization and decoding

Environment Variables Required

For Hugging Face Spaces:

HF_TOKEN=your_hugging_face_token_here
FINNHUB_KEYS=your_finnhub_api_keys_here
RAPIDAPI_KEYS=your_rapidapi_keys_here

Model Features

FinGPT Advantages:

1. Specialized for Finance: Trained specifically on financial data
2. Open Source: No API rate limits or costs
3. Memory Efficient: Uses 4-bit quantization
4. Real-time Updates: Can be fine-tuned with latest financial data

Performance Considerations:

• Memory Usage: ~4-8GB GPU memory (with quantization)
• Inference Speed: Slower than API calls but more reliable
• Model Size: ~7B parameters (much smaller than Gemini)

Testing

Run the test script to verify the integration:

python test_fingpt_integration.py

Deployment Notes

1. Hugging Face Spaces: Ensure HF_TOKEN is set in secrets
2. GPU Requirements: Requires GPU with at least 8GB VRAM
3. Model Loading: First run may take longer due to model download
4. Fallback: App will use mock responses if model fails to load

Troubleshooting

Common Issues:

1. Out of Memory: Reduce MAX_LENGTH or use CPU inference
2. Model Loading Fails: Check HF_TOKEN and internet connection
3. Slow Inference: Consider using smaller model or CPU inference

Debug Mode:

Set debug=True in the app launch to see detailed error messages.

Migration Benefits

1. Cost Effective: No API costs for inference
2. Privacy: Data stays local, no external API calls
3. Reliability: No rate limits or API downtime
4. Customization: Can fine-tune for specific financial tasks
5. Transparency: Full control over model behavior

Next Steps

1. Test the application with real financial data
2. Fine-tune the model if needed for specific use cases
3. Monitor performance and adjust parameters
4. Consider implementing model caching for faster startup