slm125m-live-sft · cautious

Refuses readily. Hallucinates on only 19% of unanswerable questions — by refusing 55% of the answerable ones.

What these are

Two instruction-tuned variants of seetha0712/slm125m-live-base, a 125M-parameter Llama pretrained from scratch on US case law, SEC filings and fineweb-edu.

Grounded QA only: the passage goes in the prompt. At 125M parameters neither model can answer closed-book. It reads the passage you supply and answers from it, or says it cannot.

Benchmarks

Model	SQuAD v2 EM	F1	Refuses unanswerable	Hallucinates on unanswerable	Falsely refuses answerable
Base (no SFT)	0.0	1.6	0%	100%	0%
eager (15k synthetic)	7.4	14.3	15.1%	85.0%	17.2%
cautious (+40k human)	49.1	51.7	80.7%	19.3%	55.1%
Humans	86.8	89.5	—	—	—

Measured on 1,500 SQuAD v2 dev questions (49% deliberately unanswerable), human-annotated.

Read this before quoting the 49.1. cautious was trained on SQuAD v2's train split, so its numbers here are supervised; the others are zero-shot. That is how SQuAD is normally used, but it is not an out-of-distribution comparison.

The out-of-distribution test

The honest one. 864 of each model's own answers on held-out legal/SEC passages, each required to be backed by a verbatim span a judge could actually find in the source:

	hallucination rate
eager	62.8%
cautious	62.5%

Unchanged. Human data did not teach it to read; it taught it to decline. Roughly 26% of the answers either model gives on its own domain are actually correct.

Training data

15,000 Q&A pairs synthesized from the pretraining corpus with Gemini 2.5 Flash, then filtered hard: 924 dropped by programmatic validators, 2,969 by an LLM judge, 969 of those because the judge cited a supporting quote that did not actually exist in the passage (verified by string match), 158 near-duplicates, 0 eval-contaminated.

cautious adds 40,000 human-written SQuAD v2 examples, 13,351 of them genuinely unanswerable.

Chat format

<|bos|><|system|>
You are a helpful assistant. Answer using only the passage provided. If the passage does not contain the answer, say so.
<|user|>
Passage:
{passage}

Question: {question}
<|assistant|>
{answer}<|eos|>

Use tokenizer.apply_chat_template(...) — it reproduces this exactly.

Note on the base model's config

The base repo ships eos_token_id: 2, but token 2 is <|pad|>; the real <|eos|> is 1 (HF LlamaConfig defaults leaked in at publish time). Harmless for pretraining, fatal for chat — generation would never terminate. Both repos here correct it to bos=0, eos=1, pad=2.

<|system|>, <|user|> and <|assistant|> were in the base vocabulary but never emitted during pretraining, so their embeddings were still at random init. They are mean-initialized before SFT.

Honest limitations

• It hallucinates. ~62% of the answers it gives on legal/SEC text are not supported by the passage.
• eager invents an answer for 85% of questions the passage cannot answer.
• cautious refuses 55% of questions it could have answered.
• 125M parameters is the binding constraint. Adding 45% more synthetic data made it worse; adding 40k human examples made it more cautious but no more comprehending.

A demonstration of a pipeline, not a production QA system. Not legal or financial advice.