| | --- |
| | language: en |
| | license: apache-2.0 |
| | tags: |
| | - finance |
| | - sentiment-analysis |
| | - text-classification |
| | - llama |
| | - qlora |
| | pipeline_tag: text-classification |
| | library_name: transformers |
| | --- |
| | |
| | # Llama Sentiment Classifier (QLoRA Fine-Tuned) |
| |
|
| | This repository provides a **LLaMA-based financial sentiment classifier** fine-tuned using **QLoRA** for **3-class sentiment classification** on finance-domain text. The model is designed for downstream applications including **sentiment-driven alpha signal generation** and market-neutral trading strategies. |
| |
|
| | --- |
| |
|
| | ## Model Summary |
| |
|
| | - **Backbone**: Meta-LLaMA3-8B |
| | - **Task**: Financial sentiment classification (3 classes) |
| | - **Fine-tuning method**: **QLoRA** (4-bit quantization + LoRA adapters) |
| | - **Training framework**: **SWIFT** (Scalable lightWeight Infrastructure for Fine-Tuning) |
| | - **Architecture**: LlamaForSequenceClassification |
| |
|
| | --- |
| |
|
| | ## Labels |
| |
|
| | This model performs **single-label classification** into 3 discrete classes. |
| |
|
| | In the paper, the unified sentiment label space is: |
| |
|
| | - **-1** = Negative |
| | - **0** = Neutral |
| | - **1** = Positive |
| |
|
| | > Note: your Hugging Face config may show `LABEL_0`, `LABEL_1`, `LABEL_2`. |
| | |
| | --- |
| | |
| | ## Datasets |
| | |
| | ### Part 1 — Fine-tuning datasets (4 sources), can be accessed through Romeo777777/FinLlaMa_Training_Dataset |
| | |
| | |
| | Training uses 4 finance-domain sentiment datasets: |
| | |
| | 1. **Financial PhraseBank v1.0** |
| | - 4,840 manually annotated sentences |
| | - 3-class sentiment (positive/neutral/negative) |
| | |
| | 2. **NASDAQ News Sentiment** |
| | - synthetic sentiment dataset generated using GPT-4o |
| | - labeled into positive/neutral/negative |
| | |
| | 3. **Twitter Financial News Sentiment** |
| | - 11,932 finance-related tweets |
| | - original label encoding: 0 bearish, 1 bullish, 2 neutral |
| | |
| | 4. **FIQA2018 (FiQA)** |
| | - financial opinion mining dataset |
| | - sentiment score in [-1, 1] (continuous), then discretized |
| | |
| | --- |
| | |
| | ### Part 2 — Real-world news evaluation dataset |
| | |
| | For downstream evaluation and trading strategy experiments, the paper uses a news + price dataset covering: |
| | |
| | - **502 S&P 500 companies** |
| | - **~77,000 news headlines** |
| | - time range: **2024-01-01 to 2025-05-30** |
| | |
| | --- |
| | |
| | ## Preprocessing |
| | |
| | ### Label normalization (unified mapping) |
| | |
| | All datasets are standardized into the unified label space: {-1, 0, 1}. |
| | |
| | Mapping rules: |
| | |
| | - **Financial PhraseBank**: |
| | `"negative" → -1`, `"neutral" → 0`, `"positive" → 1` |
| | |
| | - **FIQA2018** (score in [-1, 1]): |
| | `< -0.2 → -1`, `> 0.2 → 1`, otherwise `0` |
| | |
| | - **Twitter Financial Sentiment**: |
| | `0 → -1`, `2 → 0`, `1 → 1` |
| | |
| | - **NASDAQ News** (0~5 score): |
| | `<= 1 → -1`, `>= 4 → 1`, otherwise `0` |
| | |
| | ### Tokenization |
| | |
| | Uses the official **LLaMA 3 tokenizer**. |
| | |
| | --- |
| | |
| | ## Training Method |
| | |
| | ### QLoRA fine-tuning |
| | |
| | The model is fine-tuned with **QLoRA**, which keeps the backbone weights in 4-bit quantized form and trains LoRA adapters in higher precision. |
| | |
| | The paper’s QLoRA setup includes NF4 quantization + double quantization for memory efficiency. |
| | |
| | ### SWIFT training framework |
| | |
| | Fine-tuning is orchestrated using **SWIFT**, a modular training framework that simplifies adapter integration and efficient training for quantized models. |
| | |
| | --- |
| | |
| | ## Hyperparameters (paper) |
| | |
| | - LoRA rank **r = 16** |
| | - LoRA alpha **= 32** |
| | - LoRA dropout **= 0.1** |
| | - learning rate **1e-4**, optimizer **AdamW** |
| | - batch size **8**, epochs **1** |
| | - gradient accumulation **4** |
| | - gradient checkpointing enabled |
| | - cosine learning rate schedule |
| | |
| | --- |
| | |
| | ## Evaluation Results (paper) |
| | |
| | On a test set of **9,064** finance news samples, the fine-tuned model reports: |
| | |
| | - **Accuracy**: **92.18%** |
| | - **Micro-F1**: **0.9218** |
| | - **Macro-F1**: **0.5787** |
| | |
| | Class-wise performance highlights strong **Positive** and **Neutral** performance, but weaker **Negative** performance due to class imbalance. |
| | |
| | --- |
| | |
| | ## Usage |
| | |
| | ```python |
| | from transformers import pipeline |
| | |
| | clf = pipeline( |
| | "text-classification", |
| | model="Romeo777777/Llama_Sentiment_Classifier", |
| | tokenizer="Romeo777777/Llama_Sentiment_Classifier", |
| | return_all_scores=True, |
| | ) |
| | |
| | print(clf("Bitcoin is pumping hard today!")) |
