README.md

---
base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
library_name: peft
pipeline_tag: text-generation
license: apache-2.0
tags:
- lora
- finance
- instruction-tuning
- english
- transformers
- adapter
---

# Llama for Finance (LoRA)

A financial-domain instruction-tuned LoRA adapter for `meta-llama/Meta-Llama-3.1-8B-Instruct`, trained with length-aware batching and an English-only heuristic.

## Model Details
- **Base model:** meta-llama/Meta-Llama-3.1-8B-Instruct  
- **Adapter type:** LoRA (PEFT)  
- **Target modules:** q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj  
- **LoRA hyperparams:** r=64, alpha=128, dropout=0.1, bias=none  
- **Precision:** fp16 (bf16 when available); gradient checkpointing on  
- **Length bucketing:** enabled (`group_by_length=True`, boundaries 512/1024/1536/2048)  
- **Context length:** up to 2048 tokens  
- **Language:** English (non-English filtered via ASCII-ratio heuristic)

## Training Data & Filtering
- **Source dataset:** `Josephgflowers/Finance-Instruct-500k`
- **Sampling caps:** max_train_samples=25k, max_val_samples=2.5k after filtering
- **Chat formatting:** preformatted `text` field with system/user/assistant turns
- **Filters:**  
  - drop rows without text  
  - English-only heuristic (`min_english_ratio`≈0.85, `min_chars_for_lang_check`=40)  
  - EOS enforced at end of samples

## Training Setup
- **Epochs:** 2  
- **Batching:** per-device train 16, grad accumulation 4 (effective 64); eval batch 8  
- **Optimizer:** paged_adamw_8bit  
- **LR / schedule:** 1e-4, cosine, warmup_ratio 0.05  
- **Regularization:** weight_decay 0.01, max_grad_norm 1.0  
- **Eval/save:** eval_steps=50, save_steps=100 (load_best_model_at_end=True)  
- **Length-aware sampler:** custom bucket sampler reduces padding waste

## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base = "meta-llama/Meta-Llama-3.1-8B-Instruct"
adapter = "TimberGu/Llama_for_Finance"

tokenizer = AutoTokenizer.from_pretrained(adapter)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"  # matches training setup

dtype = torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16
base_model = AutoModelForCausalLM.from_pretrained(base, dtype=dtype, device_map="auto")
model = PeftModel.from_pretrained(base_model, adapter)
model.eval()

prompt = "Explain what a yield curve inversion implies for equities."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=256, temperature=0.8, top_p=0.9)
print(tokenizer.decode(out[0], skip_special_tokens=True))
```

## Evaluation
- Held-out validation (`eval_50_gpt_judged_raw.jsonl`): eval_loss ≈1.05 over 2 epochs. No public benchmark beyond the filtered split.

## Limitations & Risks
- Domain-focused on finance/economics; may underperform on general tasks.
- English-centric; non-English input was filtered during training.
- Hallucinations remain possible—do not use for financial advice without human review.

## Files
- `adapter_model.safetensors`, `adapter_config.json`: LoRA weights/config
- `tokenizer.json`, `tokenizer_config.json`, `special_tokens_map.json`, `chat_template.jinja`
- `training_config.json`, `training_args.bin`, `test_results.json`