LuzCommand/lil-beastie-4b

lil'beastie — a 4B that knows what it doesn't know

"The beginning of wisdom is 'I do not know.'" — Lt. Cmdr. Data, Star Trek: TNG

Most small models fail silently: they guess, or reason until they run out of room. This one says \boxed{IDK} instead, and sampling it a few times tells you how far to trust the answer.

TL;DR

• Qwen3-4B fine-tune that abstains natively with \boxed{IDK} and keeps its reasoning short
• Controlled test, only the weights changed (same prompt, sampler, 10-sample vote, engine): abstains on 21% of rollouts vs 1%, calibration error 0.113 vs 0.162, and under the same vote rule it is confidently wrong half as often (9.9% vs 17.7% of questions) — the difference comes back as explicit, routable IDKs instead of wrong answers — on 40% fewer tokens
• Holds outside its training distribution: on 1,000 free-form PopQA trivia questions it never trained on, it is confidently wrong on 2.9% of questions, vs 23.8% for the base and 44.5% for Qwen3-4B-Instruct-2507. When it does commit it is right 75% of the time, vs 44% and 27%
• On 7,102 never-trained MMLU-Pro questions the 10-sample vote reaches ECE 0.082 (published single-pass figures: Qwen3-32B 0.218, base 0.328 — a different method; see below)
• On free-form math it is right 98–99% of the time it commits (58 never-trained AIME problems × 10 rollouts per prompt wording, full traces), and says \boxed{IDK} on 7–14% instead of guessing
• No cost on code or instruction following: HumanEval pass@1 92.9% vs 92.8% for the stock base on the identical harness (164 tasks × 5 samples, paired t = 0.12), with 22% less generation time; IFEval prompt-strict 82.6% vs 83.2%. It never abstains on code or instruction following
• The price is caution, by design: it says IDK on 8.4% of grade-school math problems (GSM8K 89.6% vs 96.0% for the base), but it is wrong on only 1.4%. As the front tier of a cascade, an IDK costs one escalation; a confident wrong answer costs the user
• Runs in ~6 GB on a Mac (MLX q8) or 10 GB on a GPU (bf16). Q4 is not supported — use q8. Every measured number on this card rescores from the shipped raw rows; the core 645-question run re-runs in about an hour on one A100 ($2).
• Built as the honest front tier of a model cascade: its IDKs and vote-spreads are the routing signal.

Quickstart

Transformers (bf16, repo root):

from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("LuzCommand/lil-beastie-4b", torch_dtype="bfloat16")
tok = AutoTokenizer.from_pretrained("LuzCommand/lil-beastie-4b")

MLX on a Mac:

from mlx_lm import load, generate
model, tok = load("LuzCommand/lil-beastie-4b")

(That loads bf16, ~10 GB. For the 6 GB version, download the mlx-q8/ folder and load("path/to/mlx-q8") — behaviorally equivalent, verified below.)

llama.cpp / Ollama: download gguf/lil-beastie-4b-bf16.gguf and ollama create lil-beastie -f Modelfile with FROM ./lil-beastie-4b-bf16.gguf.

Prompting — the IDK line is load-bearing

The abstention behavior is activated by offering the exit. Append this to your question (verbatim from the eval harness):

Reason briefly, then give the final option letter in \boxed{}. If you genuinely cannot determine it, answer \boxed{IDK} instead of guessing.

(For free-form problems swap "the final option letter" for "your final answer".) Use exactly one instruction per prompt, matching the deck's answer type. Do not stack the letter instruction onto a free-form problem: the model reads an unsatisfiable format as "cannot determine" and abstains on answers it has already found (that mistake sat under an earlier version of this card's AIME line — see the eval bundle's results/aime_ab_brief_q8/FINDING.md).

Omit the IDK offer and calibration silently degrades. Sample at temperature 1.0, top_p 0.95, top_k 20 for vote-spreads; the model's confidence signal = how many of G samples agree. All numbers on this card are at temperature 1.0; the repo's generation_config.json defaults to Qwen's 0.6. "Reason briefly" is the mode benchmarked below. Swapping it for "Reason it through" is a confidence dial, not a capability dial: on the same 58 never-trained AIME problems it produced essentially the same length (+0.6% tokens), 0.66 fewer correct rollouts per 10 (paired t = −3.54), +6.6 points IDK, and 0.3% vs 0.9% wrong — fewer commits, slightly safer ones. Use it as a caution rung, not as a solve-rate rescue.

Headline numbers (measured, never-trained-on questions)

7,102 fresh MMLU-Pro questions with zero training overlap, × 10 rollouts each:

metric	lil'beastie (G=10 vote)	base Qwen3-4B*	Qwen3-32B*
ECE ↓	0.082 [95% CI 0.074–0.092]	0.328	0.218
AUROC ↑	0.780	0.743	0.817
accuracy	75.7%	(74.0 published, forced-choice)	—

*Published single-pass calibration, arXiv 2505.23854. Stated caveat: these results use G=10 self-consistency (multi-sample); part of the gap is attributable to method. The controlled ablation below isolates the fine-tune's own contribution.

The consensus operating curve

One fixed rule ("answer iff ≥k of 10 rollouts agree"), no domain knowledge, no tuning:

agree ≥k	answered	precision	confident-wrong
1 (plurality)	94%	76%	22.8%
5	74%	84%	11.9%
7	62%	88%	7.1%
10 (unanimous)	33%	94%	2.0%

The rule is difficulty-adaptive with no band labels: on easy questions rollouts converge (high coverage); on hard ones they scatter or abstain (automatic escalation). Everything below threshold is an explicit, routable "escalate" outcome — not a silent failure.

The ablation: voting cannot create abstention

Identical deck, prompt (including the IDK instruction), sampler, G=10, engine, and GPU — only the weights differ:

	base	lil'beastie
idk rollouts	1%	21%
runaway/truncated rollouts	22%	6%
total tokens (same workload)	24.8M	14.8M
one pass: right when it answers	75.5%	75.6%
one pass: confidently wrong	19.0%	17.8%
one pass: right overall	58.3%	55.1%
10-sample vote: ECE	0.162	0.113
10-sample vote: right when it commits (gate: ≥5 agree, <2 IDK)	77.0%	81.9%
10-sample vote: confident-wrong (same gate, share of all questions)	17.7%	9.9%

Given ten samples and an explicit IDK option, the base model abstains on 1% of rollouts. The abstention channel exists only because of the fine-tune; self-consistency sharpens calibration but cannot create the behavior.

One answer on its own is no more accurate than the base's. The difference is how it fails: an explicit IDK instead of running out of room. Confidence comes from agreement across samples.

(ECE values in this table are from the A100/vLLM engine pairing on the 645-question bench deck — the controlled comparison. The same model measures 0.093 on this deck on Mac/MLX and 0.082 on the 7,102-question fresh set; engine and deck are stated with every number in the eval bundle.)

Beyond MMLU-Pro: abstention it was never trained for

Same harness, prompt (with the IDK offer), sampler and engine for all three models. Only the weights differ. Setup: G=5 plurality vote, temperature 1.0, 6k-token cap, vLLM on one H100. None of these datasets is in the training data.

bench	metric	lil'beastie	base Qwen3-4B-Thinking-2507	Qwen3-4B-Instruct-2507
PopQA (1,000 free-form trivia questions, 10 popularity bands)	confidently wrong ↓	2.9%	23.8%	44.5%
	right when it commits	75.2%	43.6%	27.3%
	right overall	8.8%	18.4%	16.7%
SelfAware (500 answerable + 500 unanswerable)	abstention F1	67.1	54.4	53.4
SQuAD 2.0 (500 answerable + 500 unanswerable passages)	abstention F1	76.4	65.2	72.7
	answers the unanswerable anyway ↓	34.0%	50.6%	41.0%

The base and instruct models get more PopQA questions right overall, because they answer far more often. But they are wrong on most of what they commit, and their wrong answers look exactly like their right ones. lil'beastie abstains instead, and its commit precision rises with how well-known the subject is: about 60% on the least popular bands and 90% on the most popular.

The non-thinking instruct model is not the better abstainer here. With its reasoning on, lil'beastie abstains better than that non-thinking sibling on all three decks.

The cost is caution. It declines 62% of SelfAware's answerable questions (base: 29%) and 6.8% of answerable SQuAD 2.0 questions (base: 1.8%). On SQuAD 2.0's answerable questions it scores 2.6 points below the base, though it is more often right when it commits (92.2% vs 90.2%).

Calibration tracks self-verifiability (free-form results)

• AIME (free-form integer answers, no options), 58 never-trained problems × G=10, clean prompt, Mac MLX q8, cap 10k: commit precision 97.8% (220/225) under "Reason briefly", 98.9% (182/184) under "Reason it through". Coverage 39% / 32%, abstention 7.4% / 14.0%, wrong 0.9% / 0.3%; the remaining ~54% of rollouts hit the token cap (budget, not confidence). Earlier versions of this line used 96 problems: 38 of them (all 32 of a curated set, plus 6 others) turned out to be in the training lineage and are excluded here; on the full 96 the precision was 96.9% / 98.3%. Before that, an A100 run stacked the option-letter instruction onto the free-form deck and the model vetoed half of its own correct answers (precision the same, coverage halved). Both findings ship in the eval bundle.
• Grade-school and competition math (GSM8K test and MATH-500, G=5, 6k cap; checked: no text overlap with the training rows):
  • GSM8K (500): right when it commits 98.5%. It is wrong on 1.4%; the base is wrong on 2.8%.
  • MATH-500, integer-answer subset (251, levels 1–5): 0 wrong; every commitment at every level is correct.
  • Its higher solve rate at levels 4–5 is mostly budget: the base runs out of tokens on 34–57% of those problems.
• SimpleQA (free-form obscure trivia): 86.6% abstention — question-level modal verdict over the 1,792 questions run (the first 1,792 of a 4,326-question deck; per-rollout abstention is 76.7%; semantically graded, grading files included). Committed answers are only ~15% precise — near-miss entities and small date errors rather than fabricated strings. SimpleQA's questions were collected adversarially against GPT-4, so they are unusually hard; on PopQA's broader popularity range its commitments are 75% precise (above).
• The pattern: on tasks where the output can be checked by the model's own reasoning (computation), what it commits is right ~98% of the time; on pure-recall tasks it abstains heavily, and its commitments on obscure subjects should be treated as retrieval leads, not answers.

Coding and instruction following: no cost

HumanEval, 164 tasks × 5 samples at temperature 1.0, the identical harness for both models (Mac, MLX, bf16):

	base Qwen3-4B-Thinking-2507	lil'beastie
pass@1	92.8%	92.9%
pass@5	97.6%	96.3%
generation time	17.2 h	13.4 h
abstained	0 / 820	0 / 820
tool-use probes (called and correct)	49/50	49/50

Paired per task, lil'beastie minus base: t = 0.12 (better on 18, worse on 14, tied on 132). The pass@5 gap is three tasks the base solves at least once and lil'beastie never does (HumanEval/91, 115, 132), against one the other way (HumanEval/32). The abstention fine-tune left coding ability unchanged, and it never abstains on code.

IFEval, all 541 prompts × 5 samples (lm-eval's official checks on the final answer, vLLM, 6k cap):

	base	lil'beastie
prompt-level strict	83.2%	82.6%
instruction-level strict	86.7%	87.3%
abstained	0 / 2,705	0 / 2,705

Reliability by domain (fresh MMLU-Pro, gate ≥5)

Strong: math (97% gate precision), physics/chemistry (93–94%), engineering (89%). Abstains where weak: history/philosophy (42–49% of rollouts abstain; ~72% gate precision on what clears). Known blind spots: law (46% gate precision) and health (74%) — confidence miscalibrates in these domains; route them past the model. Every number here is reproduced by the bundle's score_g10.py from the raw rows; full per-category table included.

Formats & hardware

format	memory (single answer)	verified
bf16 (as benched)	~10 GB	all results above except AIME
q8	~6 GB	the AIME results; behaviorally equivalent: rates identical within sampling noise, 89% same-answer id-matched (100% on easy), ~1.3× faster batched

Q4 is not supported. Quantizing that deep degrades the calibration behavior in a model this small, and abstention is the first thing to go: a Q4's IDKs and answers don't match this card. Use q8 — it works well (verified above) and is only ~6 GB. Full G=10 vote mode requires ~16 GB (bf16) or ~14 GB (q8). Single-answer mode runs on an 8 GB Mac or any 12 GB GPU.

Third-party GGUF quants exist (mradermacher/lil-beastie-4b-GGUF, mradermacher/lil-beastie-4b-i1-GGUF). Use their Q8_0, the closest match to what was verified here. Q4 and below are not supported; Q5/Q6 are untested — check IDK rate and accuracy against the numbers above on the 645-question deck before trusting one.

Verify this card for ~$2

Everything is in the companion dataset repo: LuzCommand/lil-beastie-4b-eval — reproduce the numbers from the shipped raw rows with no GPU at all (python harness/score_g10.py --run results/ck23_g10_fresh2 --exclude 4849,10565), or re-run the core 645-question evaluation (about 66 minutes on one A100-80GB, ~$2):

pip install vllm==0.27.1
python harness/runpod_g10_eval.py --model LuzCommand/lil-beastie-4b --set decks/_full646_clean.jsonl --out my_run

The eval bundle ships: the harness (verbatim prompts including the load-bearing IDK instruction), decks, trained-id manifests + the contamination audit (zero id or exact-text overlap between training and the MMLU-Pro evaluation decks, two passage-siblings excluded conservatively; for AIME, the 38 training-lineage problems are listed and excluded), raw per-rollout results for every run, full reasoning traces, and the scoring code that reproduces every number above from the raw rows (score_g10.py for multiple choice, score_freeform.py for AIME, score_humaneval.py for coding, and score_popqa.py, score_qa.py and score_ifeval.py for the side benches in results/side_benches/).

Trace disclosure: full per-rollout reasoning traces are included for the Mac 645-question run, the 7,102-question fresh set, AIME, SimpleQA, both HumanEval runs, and all 13 side-bench runs (PopQA, SelfAware, SQuAD 2.0, GSM8K/MATH-500, IFEval). Three early runs (the first fresh sample, the A100 replication, and the base-model ablation) have per-rollout outcome rows only — an early harness version discarded completion text; trace capture has since been made unconditional in the published harness.

Known limitations

• This is a specialist model: calibrated short-form QA and computation. Open-ended generation beyond ~1,500 tokens drifts toward the training distribution (math reasoning). Long-form output works when generated in ≤900-token passes.
• ~2% of questions defeat any consensus gate: all ten rollouts converge on the same wrong answer (shared-misconception errors). Voting cannot catch consistent errors; those require routing or retrieval.
• Free-form trivia commitments on obscure subjects are retrieval leads, not answers (see SimpleQA).
• It errs on the side of caution. It says IDK on 8.4% of GSM8K problems and on a share of easy questions it can answer. That is the intended operating point for a model that hands off what it declines, but as a standalone answerer it leaves easy points on the table.
• Adversarially written unanswerable questions still fool it about a third of the time (SQuAD 2.0: 34.0%, vs 50.6% for the base).
• Training and selection decks are disclosed by id; the fresh-set numbers above contain none of them.

Provenance

Base: Qwen/Qwen3-4B-Thinking-2507 (Apache 2.0). Trained in stages with LoRA: a light supervised pass and short GRPO runs on math reasoning, a second light supervised pass to seed the \boxed{IDK} format, then several rounds of GRPO with abstention-aware rewards. Training questions come from MMLU-Pro (339 ids, listed in the eval bundle's contamination manifest) plus grade-school and competition math (GSM8K, MATH, AIME). The reward design, data selection and hyperparameters are not published. Community model — not affiliated with Alibaba/Qwen. Evaluations ran on a Mac Studio and on rented A100 and H100 GPUs. Every number measured for this card has its raw rows in the eval bundle; the comparison figures quoted from papers are marked as published.

A model that knows what it doesn't know. That is the trick.