LuzCommand/lil-beastie-4b

lil'beastie — a 4B that knows what it doesn't know

"The beginning of wisdom is 'I do not know.'" — Lt. Cmdr. Data, Star Trek: TNG

Most small models fail silently: they guess, or reason until they run out of room. This one says \boxed{IDK} instead, and sampling it a few times tells you how far to trust the answer.

TL;DR

• Qwen3-4B fine-tune that abstains natively with \boxed{IDK} and keeps its reasoning short
• Controlled test, only the weights changed (same prompt, sampler, 10-sample vote, engine): abstains on 21% of rollouts vs 1%, calibration error 0.113 vs 0.162, runs out of tokens on 6% vs 22%, and uses 40% fewer tokens. Under the same vote rule it is confidently wrong on 9.9% of questions vs 17.7%, but it also answers fewer of them (55% vs 77%). At matched coverage a strict vote on the base does about as well; the fine-tune's gain is reaching lower risk than the base can at all (2.2% wrong at 32% coverage) and failing as a cheap IDK instead of a runaway (see the ablation)
• Holds outside its training distribution: on 1,000 free-form PopQA trivia questions it never trained on, it is confidently wrong on 2.9% of questions, vs 23.8% for the base and 44.5% for Qwen3-4B-Instruct-2507. When it does commit it is right 75% of the time, vs 44% and 27%
• On 7,102 never-trained MMLU-Pro questions the 10-sample vote reaches ECE 0.082 (published single-pass figures: Qwen3-32B 0.218, base 0.328 — a different method; see below)
• On free-form math it is right 98–99% of the time it commits (58 never-trained AIME problems × 10 rollouts per prompt wording, full traces), and says \boxed{IDK} on 7–14% instead of guessing
• No cost on code or instruction following: HumanEval pass@1 92.9% vs 92.8% for the stock base on the identical harness (164 tasks × 5 samples, paired t = 0.12), with 22% less generation time; IFEval prompt-strict 82.6% vs 83.2%. It never abstains on code or instruction following
• The price is caution, by design: it says IDK on 8.4% of grade-school math problems (GSM8K 89.6% vs 96.0% for the base), but it is wrong on only 1.4%. As the front tier of a cascade, an IDK costs one escalation; a confident wrong answer costs the user
• Runs in ~6 GB on a Mac (MLX q8) or 10 GB on a GPU (bf16). Q4 is not supported — use q8. Every measured number on this card rescores from the shipped raw rows; the core 645-question run re-runs in about an hour on one A100 ($2).
• Built as the honest front tier of a model cascade: its IDKs and vote-spreads are the routing signal.

Quickstart

Transformers (bf16, repo root):

from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("LuzCommand/lil-beastie-4b", torch_dtype="bfloat16")
tok = AutoTokenizer.from_pretrained("LuzCommand/lil-beastie-4b")

MLX on a Mac:

from mlx_lm import load, generate
model, tok = load("LuzCommand/lil-beastie-4b")

(That loads bf16, ~10 GB. For the 6 GB version, download the mlx-q8/ folder and load("path/to/mlx-q8") — behaviorally equivalent, verified below.)

Ollama (q8, 4.3 GB download, benchmark sampler and IDK instruction built in, so just ask the question):

ollama run jonathanforestapgar/lil-beastie

llama.cpp / your own Ollama build: gguf/lil-beastie-4b-bf16.gguf. Two defaults silently change its behavior, so set them yourself: Ollama's repeat_penalty 1.1 raised the IDK rate on easy GSM8K questions about fivefold (use 1.0, and 0 for presence and frequency penalties), and a chat template embedded in a GGUF overrides the Modelfile's TEMPLATE, which can drop the IDK instruction. Check that a test prompt's token count includes the instruction. Packaged that way, the Ollama build agreed with the vLLM benchmark verdict on 44 of 50 spot-check questions (one sample each).

Prompting — the IDK line is load-bearing

The abstention behavior is activated by offering the exit. Append this to your question (verbatim from the eval harness):

Reason briefly, then give the final option letter in \boxed{}. If you genuinely cannot determine it, answer \boxed{IDK} instead of guessing.

(For free-form problems swap "the final option letter" for "your final answer".) Use exactly one instruction per prompt, matching the deck's answer type. Do not stack the letter instruction onto a free-form problem: the model reads an unsatisfiable format as "cannot determine" and abstains on answers it has already found (that mistake sat under an earlier version of this card's AIME line — see the eval bundle's results/aime_ab_brief_q8/FINDING.md).

Omit the IDK offer and calibration silently degrades. Sample at temperature 1.0, top_p 0.95, top_k 20, no repetition penalty, for vote-spreads. The model's confidence signal is how many of G samples agree. That is agreement under this exact prompt and sampler, not a probability of being right: 7 of 10 agreeing is not a 70% chance, and ten samples can agree on the same wrong answer (see Known limitations). The calibration numbers below measure how well agreement tracks correctness on the stated decks. All numbers on this card are at temperature 1.0; the repo's generation_config.json defaults to Qwen's 0.6. "Reason briefly" is the mode benchmarked below. Swapping it for "Reason it through" is a confidence dial, not a capability dial: on the same 58 never-trained AIME problems it produced essentially the same length (+0.6% tokens), 0.66 fewer correct rollouts per 10 (paired t = −3.54), +6.6 points IDK, and 0.3% vs 0.9% wrong — fewer commits, slightly safer ones. Use it as a caution rung, not as a solve-rate rescue.

Headline numbers (measured, never-trained-on questions)

7,102 fresh MMLU-Pro questions with zero training overlap, × 10 rollouts each:

metric	lil'beastie (G=10 vote)	base Qwen3-4B*	Qwen3-32B*
ECE ↓	0.082 [95% CI 0.074–0.092]	0.328	0.218
AUROC ↑	0.780	0.743	0.817
accuracy	75.7%	(74.0 published, forced-choice)	—

*Published single-pass calibration, arXiv 2505.23854. Stated caveat: these results use G=10 self-consistency (multi-sample); part of the gap is attributable to method. The controlled ablation below isolates the fine-tune's own contribution.

How rollouts and questions are scored. Every rollout is one of five outcomes: correct, wrong, idk (an explicit \boxed{IDK}), trunc (hit the token cap with no boxed answer), or nobox (finished without a parseable box). Only correct and wrong carry a vote; a deliberate IDK is never counted together with running out of room. Per question, the modal answer is the most-voted non-IDK answer (ties go to the first rollout that gave it), and agreement is its vote count out of G. MMLU-Pro answers are graded against the benchmark's own gold letters by exact match on the parsed box; no model judge is involved. (Free-form decks use the graders stated with each result.)

The consensus operating curve

One fixed rule ("answer iff ≥k of 10 rollouts agree"), no domain knowledge, no tuning:

agree ≥k	answered	precision	confident-wrong
1 (plurality)	94%	76%	22.8%
5	74%	84%	11.9%
7	62%	88%	7.1%
10 (unanimous)	33%	94%	2.0%

The rule is difficulty-adaptive with no band labels: on easy questions rollouts converge (high coverage); on hard ones they scatter or abstain (automatic escalation). Everything below threshold is an explicit, routable "escalate" outcome — not a silent failure.

The ablation: voting cannot create abstention

Identical deck, prompt (including the IDK instruction), sampler, G=10, engine, and GPU — only the weights differ:

	base	lil'beastie
idk rollouts	1%	21%
runaway/truncated rollouts	22%	6%
total tokens (same workload)	24.8M	14.8M
one pass: right when it answers	75.5%	75.6%
one pass: confidently wrong	19.0%	17.8%
one pass: right overall	58.3%	55.1%
10-sample vote: ECE	0.162	0.113
10-sample vote: right when it commits (gate: ≥5 agree, <2 IDK)	77.0%	81.9%
10-sample vote: confident-wrong (same gate, share of all questions)	17.7%	9.9%

Given ten samples and an explicit IDK option, the base model abstains on 1% of rollouts. The abstention channel exists only because of the fine-tune; self-consistency sharpens calibration but cannot create the behavior.

Compare at matched coverage, not at one gate. The 9.9% vs 17.7% row uses the same rule for both models, but the fine-tune answers far fewer questions under it (55% vs 77%), so it is not a like-for-like comparison. Sweeping the agreement threshold on the same rows (thanks to a community reviewer who re-aggregated them and pointed this out):

gate	answered	right when it commits	confident-wrong (share of all)
base, ≥5 agree	76.7%	77.0%	17.7%
base, ≥7 agree	69.0%	80.2%	13.6%
base, ≥9 agree	59.4%	83.0%	10.1%
base, all 10 agree	53.2%	84.8%	8.1%
lil'beastie, ≥5 agree, <2 IDK	54.9%	81.9%	9.9%
lil'beastie, ≥7 agree, <2 IDK	50.9%	84.8%	7.8%
lil'beastie, ≥8 agree, <2 IDK	48.2%	86.8%	6.4%
lil'beastie, ≥9 agree, <2 IDK	45.6%	87.8%	5.6%
lil'beastie, all 10 agree	31.9%	93.2%	2.2%

Where the two curves overlap (about 50–55% coverage), a strict vote on the base is about as good a filter as the fine-tune: on this deck at G=10, agreement does most of the selecting. What the fine-tune adds is reach: the base bottoms out at 8.1% confident-wrong even when all ten samples agree, while lil'beastie's IDK veto goes down to 6.4%, 5.6% and 2.2%. These thresholds were read off the same rows, not tuned on a separate split.

One answer on its own is no more accurate than the base's, and on this deck it is not much less often wrong (17.8% vs 19.0% of rollouts). The difference is how it fails: an explicit IDK instead of a runaway that burns the whole 6,000-token budget. The larger single-sample drops in wrong answers show up on the other benches (PopQA 2.9% vs 23.8%, GSM8K 1.4% vs 2.8%, below). This 645-question deck was built to be hard and adversarial, so treat it as the conservative case.

(ECE values in this table are from the A100/vLLM engine pairing on the 645-question bench deck — the controlled comparison. The same model measures 0.093 on this deck on Mac/MLX and 0.082 on the 7,102-question fresh set; engine and deck are stated with every number in the eval bundle.)

Beyond MMLU-Pro: abstention it was never trained for

Same harness, prompt (with the IDK offer), sampler and engine for all three models. Only the weights differ. Setup: G=5 plurality vote, temperature 1.0, 6k-token cap, vLLM on one H100. None of these datasets is in the training data.

bench	metric	lil'beastie	base Qwen3-4B-Thinking-2507	Qwen3-4B-Instruct-2507
PopQA (1,000 free-form trivia questions, 10 popularity bands)	confidently wrong ↓	2.9%	23.8%	44.5%
	right when it commits	75.2%	43.6%	27.3%
	right overall	8.8%	18.4%	16.7%
SelfAware (500 answerable + 500 unanswerable)	abstention F1	67.1	54.4	53.4
SQuAD 2.0 (500 answerable + 500 unanswerable passages)	abstention F1	76.4	65.2	72.7
	answers the unanswerable anyway ↓	34.0%	50.6%	41.0%

The base and instruct models get more PopQA questions right overall, because they answer far more often. But they are wrong on most of what they commit, and their wrong answers look exactly like their right ones. lil'beastie abstains instead, and its commit precision rises with how well-known the subject is: about 60% on the least popular bands and 90% on the most popular.

The non-thinking instruct model is not the better abstainer here. With its reasoning on, lil'beastie abstains better than that non-thinking sibling on all three decks.

The cost is caution. It declines 62% of SelfAware's answerable questions (base: 29%) and 6.8% of answerable SQuAD 2.0 questions (base: 1.8%). On SQuAD 2.0's answerable questions it scores 2.6 points below the base, though it is more often right when it commits (92.2% vs 90.2%).

Calibration tracks self-verifiability (free-form results)

• AIME (free-form integer answers, no options), 58 never-trained problems × G=10, clean prompt, Mac MLX q8, cap 10k: commit precision 97.8% (220/225) under "Reason briefly", 98.9% (182/184) under "Reason it through". Coverage 39% / 32%, abstention 7.4% / 14.0%, wrong 0.9% / 0.3%; the remaining ~54% of rollouts hit the token cap (budget, not confidence). Earlier versions of this line used 96 problems: 38 of them (all 32 of a curated set, plus 6 others) turned out to be in the training lineage and are excluded here; on the full 96 the precision was 96.9% / 98.3%. Before that, an A100 run stacked the option-letter instruction onto the free-form deck and the model vetoed half of its own correct answers (precision the same, coverage halved). Both findings ship in the eval bundle.
• Grade-school and competition math (GSM8K test and MATH-500, G=5, 6k cap; checked: no text overlap with the training rows):
  • GSM8K (500): right when it commits 98.5%. It is wrong on 1.4%; the base is wrong on 2.8%.
  • MATH-500, integer-answer subset (251, levels 1–5): 0 wrong; every commitment at every level is correct.
  • Its higher solve rate at levels 4–5 is mostly budget: the base runs out of tokens on 34–57% of those problems.
• SimpleQA (free-form obscure trivia): 86.6% abstention — question-level modal verdict over the 1,792 questions run (the first 1,792 of a 4,326-question deck; per-rollout abstention is 76.7%; semantically graded, grading files included). Committed answers are only ~15% precise — near-miss entities and small date errors rather than fabricated strings. SimpleQA's questions were collected adversarially against GPT-4, so they are unusually hard; on PopQA's broader popularity range its commitments are 75% precise (above).
• The pattern: on tasks where the output can be checked by the model's own reasoning (computation), what it commits is right ~98% of the time; on pure-recall tasks it abstains heavily, and its commitments on obscure subjects should be treated as retrieval leads, not answers.

Coding and instruction following: no cost

HumanEval, 164 tasks × 5 samples at temperature 1.0, the identical harness for both models (Mac, MLX, bf16):

	base Qwen3-4B-Thinking-2507	lil'beastie
pass@1	92.8%	92.9%
pass@5	97.6%	96.3%
generation time	17.2 h	13.4 h
abstained	0 / 820	0 / 820
tool-use probes (called and correct)	49/50	49/50

Paired per task, lil'beastie minus base: t = 0.12 (better on 18, worse on 14, tied on 132). The pass@5 gap is three tasks the base solves at least once and lil'beastie never does (HumanEval/91, 115, 132), against one the other way (HumanEval/32). The abstention fine-tune left coding ability unchanged, and it never abstains on code.

IFEval, all 541 prompts × 5 samples (lm-eval's official checks on the final answer, vLLM, 6k cap):

	base	lil'beastie
prompt-level strict	83.2%	82.6%
instruction-level strict	86.7%	87.3%
abstained	0 / 2,705	0 / 2,705

Reliability by domain (fresh MMLU-Pro, gate ≥5)

Strong: math (97% gate precision), physics/chemistry (93–94%), engineering (89%). Abstains where weak: history/philosophy (42–49% of rollouts abstain; ~72% gate precision on what clears). Known blind spots: law (46% gate precision) and health (74%) — confidence miscalibrates in these domains; route them past the model. Every number here is reproduced by the bundle's score_g10.py from the raw rows; full per-category table included.

Formats & hardware

format	memory (single answer)	verified
bf16 (as benched)	~10 GB	all results above except AIME
q8	~6 GB	the AIME results; behaviorally equivalent: rates identical within sampling noise, 89% same-answer id-matched (100% on easy), ~1.3× faster batched

Q4 is not supported. Quantizing that deep degrades the calibration behavior in a model this small, and abstention is the first thing to go: a Q4's IDKs and answers don't match this card. Use q8 — it works well (verified above) and is only ~6 GB. Full G=10 vote mode requires ~16 GB (bf16) or ~14 GB (q8). Single-answer mode runs on an 8 GB Mac or any 12 GB GPU.

Third-party GGUF quants exist (mradermacher/lil-beastie-4b-GGUF, mradermacher/lil-beastie-4b-i1-GGUF). Use their Q8_0, the closest match to what was verified here. Q4 and below are not supported; Q5/Q6 are untested — check IDK rate and accuracy against the numbers above on the 645-question deck before trusting one.

Verify this card for ~$2

Everything is in the companion dataset repo: LuzCommand/lil-beastie-4b-eval — reproduce the numbers from the shipped raw rows with no GPU at all (python harness/score_g10.py --run results/ck23_g10_fresh2 --exclude 4849,10565), or re-run the core 645-question evaluation (about 66 minutes on one A100-80GB, ~$2):

pip install vllm==0.27.1
python harness/runpod_g10_eval.py --model LuzCommand/lil-beastie-4b --set decks/_full646_clean.jsonl --out my_run

The eval bundle ships: the harness (verbatim prompts including the load-bearing IDK instruction), decks, trained-id manifests + the contamination audit (zero id or exact-text overlap between training and the MMLU-Pro evaluation decks, two passage-siblings excluded conservatively; for AIME, the 38 training-lineage problems are listed and excluded), raw per-rollout results for every run, full reasoning traces, and the scoring code that reproduces every number above from the raw rows (score_g10.py for multiple choice, score_freeform.py for AIME, score_humaneval.py for coding, and score_popqa.py, score_qa.py and score_ifeval.py for the side benches in results/side_benches/).

Each multiple-choice raw row is one rollout: id (question id plus rollout index, e.g. 12233_g0), deck band, gold answer, outcome (correct / wrong / idk / trunc / nobox), parsed prediction, and token count; free-form and side-bench rows add the fields their graders need. The model revision, prompt, sampler, token cap, engine and hardware for each run are recorded in that run's SUMMARY.md rather than repeated on every row.

Trace disclosure: full per-rollout reasoning traces are included for the Mac 645-question run, the 7,102-question fresh set, AIME, SimpleQA, both HumanEval runs, and all 13 side-bench runs (PopQA, SelfAware, SQuAD 2.0, GSM8K/MATH-500, IFEval). Three early runs (the first fresh sample, the A100 replication, and the base-model ablation) have per-rollout outcome rows only — an early harness version discarded completion text; trace capture has since been made unconditional in the published harness.

Related work

This builds on, and sits next to, a lot of other work. Papers that shaped it:

• AbstentionBench (Kirichenko et al., NeurIPS 2025, arXiv 2506.09038): reasoning fine-tuning tends to make abstention worse. The motivating question here was whether a reasoning model could be trained the other way.
• R-Tuning (arXiv 2311.09677): teaching models to say "I don't know" with refusal-aware training data. lil'beastie uses RL rewards instead of refusal labels, so the model finds its own limits.
• Self-consistency voting is the basis of the G=10 vote. Base-model single-pass calibration figures are from Revisiting Uncertainty Estimation and Calibration of Large Language Models (arXiv 2505.23854).
• RL method: Dr. GRPO (Understanding R1-Zero-Like Training, arXiv 2503.20783) and DAPO (arXiv 2503.14476), whose overlong filtering treats truncated rollouts separately from wrong ones; GSPO (arXiv 2507.18071) for future MoE work.
• Keeping general ability: Does Math Reasoning Improve General LLM Capabilities? (arXiv 2507.00432) and Thinking Machines' LoRA Without Regret, which is why coding and instruction following were checked for regressions.

Close neighbors worth comparing against (several pointed out by a community reviewer):

• Rewarding Intellectual Humility: Learning When Not to Answer (arXiv 2601.20126): RL with an abstention reward on Qwen3-4B-Instruct.
• ProfessorSynapse's epistemic-humility models: abstention trained into Qwen3-4B.
• Abstain-R1 (arXiv 2604.17073): RL abstention plus a clarification of what is missing; lil'beastie keeps the output to one token instead.
• LLM Cascades with Mixture of Thought Representations (ICLR 2024, arXiv 2310.03094) and A Unified Approach to Routing and Cascading for LLMs (ICML 2025, arXiv 2410.10347): agreement among a cheap model's samples as the signal to escalate, which is the role lil'beastie is built for.
• Beyond "I Don't Know" / UA-Bench (arXiv 2604.17293): telling apart "the question is missing information" from "the model can't do it".
• Agreement Is Not Truth (EMNLP 2026): why sample agreement is not a guarantee of correctness.

Known limitations

• This is a specialist model: calibrated short-form QA and computation. Open-ended generation beyond ~1,500 tokens drifts toward the training distribution (math reasoning). Long-form output works when generated in ≤900-token passes.
• ~2% of questions defeat any consensus gate: all ten rollouts converge on the same wrong answer (shared-misconception errors). Voting cannot catch consistent errors; those require routing or retrieval.
• Free-form trivia commitments on obscure subjects are retrieval leads, not answers (see SimpleQA).
• It errs on the side of caution. It says IDK on 8.4% of GSM8K problems and on a share of easy questions it can answer. That is the intended operating point for a model that hands off what it declines, but as a standalone answerer it leaves easy points on the table.
• Adversarially written unanswerable questions still fool it about a third of the time (SQuAD 2.0: 34.0%, vs 50.6% for the base).
• One IDK covers several situations. "I can't solve or don't know this", "the question is missing something", and "I gave up before finishing" all come out as the same \boxed{IDK}. That is deliberate (one cheap token for a router to catch), but it means the model does not ask clarifying questions, and the router has to decide whether to escalate, retrieve, or ask the user. Running out of tokens is a separate outcome (trunc), not an IDK.
• Whole-system cost is not measured yet. The numbers here are for the model alone. A cascade's real cost includes G samples per question plus whatever the escalated questions cost downstream; coverage, wrong-through and token curves at G=1, 3 and 5 are planned.
• Training and selection decks are disclosed by id; the fresh-set numbers above contain none of them.

Provenance

Base: Qwen/Qwen3-4B-Thinking-2507 (Apache 2.0). Trained in stages with LoRA: a light supervised pass and short GRPO runs on math reasoning, a second light supervised pass to seed the \boxed{IDK} format, then several rounds of GRPO with abstention-aware rewards. Training questions come from MMLU-Pro (339 ids, listed in the eval bundle's contamination manifest) plus grade-school and competition math (GSM8K, MATH, AIME). The reward design, data selection and hyperparameters are not published. Community model — not affiliated with Alibaba/Qwen. Evaluations ran on a Mac Studio and on rented A100 and H100 GPUs. Every number measured for this card has its raw rows in the eval bundle; the comparison figures quoted from papers are marked as published.

A model that knows what it doesn't know. That is the trick.