Venkat9990/finance-specialist-v7

Finance Specialist v7

A finance-domain fine-tuned version of Llama 3.2 1B Instruct that adds financial knowledge while preserving the base model's general capabilities.

Model Details

• Base Model: unsloth/Llama-3.2-1B-Instruct
• Method: LoRA (r=8, alpha=16) on attention layers (q/k/v/o_proj)
• Training Data: 5,391 samples from Finance-Instruct-500K (permissive quality filtering)
• Training: 1 epoch, lr=1e-5, cosine schedule, effective batch size 16, bf16
• Hardware: NVIDIA A100 80GB (GMU Hopper HPC)
• Training Time: 3.5 minutes
• Loss: 1.562 avg training loss, 96.2% token masking (assistant-only)

Benchmark Results

Benchmark	Base Model	V7	Change
MMLU (57 tasks, 5-shot)	46.05%	45.75%	-0.30%
GSM8K (math, 5-shot)	33.59%	33.13%	-0.46%
IFEval (instruction following)	43.07%	41.22%	-1.85%
ARC-Challenge	37.88%	37.97%	+0.09%
ARC-Easy	68.81%	68.27%	-0.54%
HellaSwag	61.59%	61.11%	-0.48%
WinoGrande	61.80%	61.64%	-0.16%
TruthfulQA MC2	43.37%	42.57%	-0.80%
CFA (finance)	40.99%	42.54%	+1.55%
FOMC Sentiment (finance)	42.74%	49.60%	+6.86%

Key: Near-zero catastrophic forgetting across all general benchmarks, with meaningful finance domain improvements (CFA +1.55%, FOMC +6.86%).

Usage

With Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Venkat9990/finance-specialist-v7")
tokenizer = AutoTokenizer.from_pretrained("Venkat9990/finance-specialist-v7")

messages = [
    {"role": "system", "content": "You are a finance specialist AI assistant."},
    {"role": "user", "content": "What factors affect a company's credit rating?"}
]

input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt")
output = model.generate(input_ids, max_new_tokens=256, temperature=0.1)
print(tokenizer.decode(output[0], skip_special_tokens=True))

With Ollama

Download the GGUF version and create a Modelfile, or use the SafeTensors weights directly with llama.cpp.

Training Details

Built with llm-forge, a config-driven LLM fine-tuning platform.

• 12 root causes identified and fixed across 7 versions to achieve knowledge-preserving fine-tuning
• Conservative LoRA (r=8) prevents overwriting base model knowledge
• Assistant-only loss masking (96.2% of tokens masked) ensures the model only learns to generate responses
• Permissive data cleaning preserves training data diversity

Author

Naga Venkata Sai Chennu — George Mason University