slm-125m-finetuned

SFT fine-tune of Probabilistr/slm-125m-base (a 125.8M-parameter from-scratch Llama-architecture model) on a synthetic instruction/response dataset for legal and financial documents.

Fine-tuning data

4,500 examples (grounded QA/RAFT, summarization, extraction-to-JSON, and plain-English rewrite tasks) generated with gpt-4o-mini from case-law and SEC-filing passages, then filtered for format, eval-set decontamination, near-duplicate removal, and LLM-judged grounding. See the project's Instructions_for_finetuning.txt and the dataset guidelines it references for the full pipeline.

Training

• 1x A10G, 3 epochs, 837 steps, ~8 minutes
• Loss masked to only the assistant-response tokens
• See metrics.jsonl / summary.json in this repo for the training curve

Known limitations

Grounding discipline is inconsistent on held-out prompts -- in particular the model doesn't reliably say "Not stated in the context" when the context doesn't contain the answer, and can conflate similar figures within a passage. This is a small model fine-tuned on a small dataset; treat outputs accordingly.

Prompt format

<|bos|><|system|>
You are a legal/financial assistant. Answer strictly using the information in the provided context. If the answer is not stated in the context, say "Not stated in the context."<|user|>
Context:
{context}

Instruction: {instruction}<|assistant|>