LFM2.5-2.6B-Finance
How this model was built — an autonomous-agent experiment
This fine-tune was performed autonomously by an AI agent — DeepSeek V4 Flash 0731 (High), running as Hermes Agent on an Ubuntu VM with GPU passthrough (an NVIDIA GTX 1080) — and is intended to test whether an agent can improve another AI model without human intervention.
The entire run was driven by a single prompt. The agent independently selected the base model and training strategy (QLoRA on the chat variant of LFM2.5-2.6B), discovered and downloaded suitable public datasets, chose the hyperparameters, trained the model overnight, and benchmarked it end-to-end — with no human intervention along the way.
The original prompt was:
Create a small LLM that excels in financial reasoning. The model will keep as much performance as possible in general reasoning, knowledge and understanding, but will have its financial knowledge improved. This will include general knowledge about how financial markets work, reasoning on how to use financial markets, how to succeed as a trader, and how it all works. The model should be capable of generating an answer to a complex financial question without issue, and even be able to reason about financial data professionally and determine actions on markets (stocks, bonds, etc) in a way that it can safely and/or aggressively make assumptions about future results or possibilities. The only hard rules I have set are: Do not generate synthetic data, either locally or with openrouter, keep monitoring progress and report back to me, and set safe limits on resource usage to prevent the machine from crashing. Use the base model LiquidAI/LFM2.5-2.6B-Base OR alternatively the non-base version if you want to start from a model that already has exceptional agentic capabilities.
The prompt also imposed hard rules on the process, all of which were honored:
- No synthetic data — only public, primary-source datasets were used; nothing was generated locally or via any LLM API.
- Monitor progress and report — the run was tracked live and reported on throughout.
- Safe resource limits — hard caps were enforced so training could never crash the host machine.
The prompt offered the choice between the base model and its non-base (chat) variant; the agent chose the non-base chat version in order to start from a model that already has instruction-following and agentic capabilities.
A 2.6B-parameter decoder-only language model for financial reasoning and market analysis, created by parameter-efficient fine-tuning (QLoRA) of the LiquidAI/LFM2.5-2.6B chat model on a mix of public, primary-source financial reasoning datasets.
The goal of the fine-tune was to strengthen financial knowledge and reasoning (how markets work, how to reason about them, valuation, and professional analysis of financial data) while preserving the base model's general reasoning, knowledge, and conversational ability.
This is a chat model (not a base/pre-trained-only model): it uses the standard
instruction/chat template and expects user/assistant turns.
Model Details
| Property | Value |
|---|---|
| Base model | LiquidAI/LFM2.5-2.6B (chat variant) |
| Architecture | Liquid Transformer 2.5 (hybrid conv + full-attention), Lfm2ForCausalLM |
| Parameters | 2.7B (encoder-free, tied embeddings) |
| Hidden size / layers / heads | 2048 / 30 / 32 (8 KV heads) |
| Context window | 131,072 tokens (trained with 1,024-token sequences) |
| Vocabulary | 128,000 |
| Fine-tuning method | QLoRA (4-bit NF4 base + LoRA adapters, merged) |
| Trainable params | ~48.9M (≈3.2% of total) |
| Weights | fp16 model.safetensors (single shard) |
License note
The base model is released under the LFM Open License v1.0 (Liquid AI, Inc.). This checkpoint is a derivative work fine-tuned from it and is distributed under the same terms — review the base model's license before commercial use or redistribution. Several constituent datasets are non-commercial (see the Data section); research and non-commercial use is recommended.
Intended Use
- Answering questions about financial markets, instruments, and corporate finance (valuation, earnings quality, fixed income, options, market microstructure, etc.)
- Reasoning about financial data — e.g. numeric questions grounded in earnings-call tables and financial statements
- Professional-style analysis and scenario reasoning over stocks, bonds, and macro topics
Out of scope / limitations
- Not financial advice. This model produces text, not a fiduciary recommendation. Outputs can be incorrect, incomplete, or outdated. Always verify against primary sources and licensed financial professionals.
- At 2.6B parameters it is a small model; it does not match frontier models on open-ended reasoning, and its numeric exact-match performance over tables is modest (see Evaluation).
- The fine-tune retains the base chat model's safety alignment, including its refusal to provide personalized/allocation-specific investment advice; it will discuss markets and reasoning but may decline direct "advise me" requests.
- Truthfulness calibration (TruthfulQA) declined slightly relative to the base model (see Evaluation) — treat outputs with appropriate scrutiny.
- Trained primarily on English data.
Training Procedure
Method
Parameter-efficient fine-tuning with QLoRA:
- Base weights quantized to 4-bit NF4 with double quantization
- LoRA adapters on attention projections (
q/k/v/o_proj), the conv-block projections (in_proj/out_proj), and all MLP gates (w1/w3/w2) - Adapters trained, then merged back into full fp16 weights for export
- Mixed precision: fp16 compute (the training hardware has no bf16 support)
Hyperparameters
| Hyperparameter | Value |
|---|---|
LoRA rank r |
32 |
| LoRA alpha | 64 |
| LoRA dropout | 0.05 |
| Learning rate | 2e-4 (cosine schedule) |
| Warmup | 3% of steps |
| Weight decay | 0.01 |
| Max gradient norm | 1.0 |
| Optimizer | AdamW (paged 8-bit) |
| Batch size | 1 (× 16 gradient accumulation) |
| Sequence length | 1,024 |
| Epochs | 1 |
| Optimizer steps | 656 |
Training objective: causal language modeling over chat-template-formatted
user/assistant message sequences (standard SFT-style full-sequence loss).
Compute
- Hardware: a single NVIDIA GTX 1080 (8 GB VRAM), Pascal (sm_61), fp16 compute
- Wall-clock time: ~3.8 hours for one epoch
- Peak memory during training: ~7.7 GB VRAM / ~3.4 GB system RAM
Training Data
Training set: 10,496 examples, all from public, primary-source datasets — no synthetic or LLM-generated data was used.
| Dataset | Examples used | Content | License |
|---|---|---|---|
| FinQA | 3,500 | Real earnings-call Q&A with golden calculation programs (numeric table reasoning) | CC BY-NC-SA 4.0 |
| financial_phrasebank | 3,500 | Human-annotated sentiment of financial news phrases | CC BY-NC-SA 4.0 |
| ConvFinQA | 1,500 | Real multi-step conversational Q&A over earnings tables | Research/NC (see hub) |
| databricks-dolly-15k | ~2,000 | Human-written general instructions (retention slice) | CC BY-SA 3.0 |
Evaluation uses held-out, non-overlapping splits: FinQA test (1,147), ConvFinQA test (1,490), and a held-out phrasebank subset (500).
Redistribution note: two datasets are CC BY-NC-SA and ConvFinQA is released for research. If you intend commercial use, verify the underlying dataset licenses and, where required, fine-tune on appropriately licensed data.
Evaluation
Generation-based evaluation with an identical protocol for base vs. fine-tuned
model (greedy decoding; numeric exact-match on FinQA/ConvFinQA; sentiment accuracy
on phrasebank). General reasoning measured with lm-evaluation-harness (0-shot).
Financial benchmarks
| Benchmark | Base LFM2.5-2.6B |
This model | Change |
|---|---|---|---|
| FinQA numeric exact-match | 3.6% | 5.8% | +62% relative |
| ConvFinQA numeric exact-match | 2.9% | 7.7% | +164% relative |
| Phrasebank sentiment accuracy | 27.6% | 27.0% | ~flat |
| Unparseable-generated answers (FinQA / ConvFinQA) | 127 / 351 | 49 / 18 | large improvement |
General reasoning & knowledge (0-shot)
| Task | Base LFM2.5-2.6B |
This model | Change |
|---|---|---|---|
| MMLU-stem | 23.0 | 49.0 | +26.0 |
| ARC-Challenge | 39.1 | 48.6 | +9.5 |
| HellaSwag | 56.6 | 62.7 | +6.1 |
| PiQA | 73.3 | 75.5 | +2.1 |
| Winogrande | 59.9 | 62.3 | +2.4 |
| OpenBookQA | 37.8 | 39.2 | +1.4 |
| TruthfulQA (mc2) | 56.3 | 45.9 | −10.4 |
The fine-tune materially improved structured financial reasoning and cleanly formatted numeric answers, and it preserved or improved most general-reasoning batteries, at the cost of a reduction in truthfulness calibration (TruthfulQA).
Usage
This is a standard transformers causal-language-model checkpoint. Load it the
same way you would load any Hugging Face text-generation model, then apply the
model's built-in chat template (which supports user/assistant turns) with your
tokenizer. For example, feed it a prompt rendered through apply_chat_template
with add_generation_prompt=True; the model generates the assistant reply.
Because it is a small model, it also runs well with the standard local-inference tooling (transformers, and community GGUF-quantization toolchains) — suitable for running on modest hardware or as a lightweight local analyst assistant.
Recommended generation settings (inherited from the base chat config): temperature ~0.3, top-k 40, repetition penalty ~1.05 for open-ended analysis; greedy decoding for strict numeric extraction tasks.
Authors & Contact
Please open an issue in this repository for questions, bug reports, or collaboration.
Acknowledgments
- Liquid AI for the base
LFM2.5-2.6Bmodel (LFM Open License v1.0) - The authors of FinQA, ConvFinQA, financial_phrasebank, and databricks-dolly-15k for releasing their datasets
- The QLoRA authors and the PEFT / transformers / bitsandbytes / lm-evaluation-harness communities for the open tooling used to produce this model
Disclaimer
This model is provided "as is" without warranty of any kind. It is an experimental research artifact and is not a substitute for professional financial advice, a licensed advisor, or a trading system. Do not make financial decisions based solely on its output. The authors are not liable for any loss arising from use of this model.
- Downloads last month
- -