Budgie-500M

Budgie-500M is a 487.8M-parameter English causal language model for compact mathematical reasoning, formal logic, science multiple choice, and dialogue with a 4,096-token context.

This root release promotes the strongest retained Budgie Alignment v2 checkpoint, verified-math-a025. The repository root contains inference artifacts only: no adapters, optimizer states, training rows, or obsolete checkpoints.

Full-split validation

The promoted weights were re-evaluated against the previous public Budgie on complete reasoning splits with identical user-only prompts and greedy decoding. This is a complete-split controlled evaluation under the documented scorer, not an official leaderboard submission.

Benchmark	n	Budgie-500M	Previous public Budgie	Delta
GSM8K test	1,319	18.57% (245/1,319)	14.48% (191/1,319)	+4.09 pp
MATH-500 test	500	12.00% (60/500)	12.40% (62/500)	-0.40 pp
ARC-Challenge test	1,172	27.56% (323/1,172)	24.83% (291/1,172)	+2.73 pp
FOLIO validation	203	35.47% (72/203)	31.03% (63/203)	+4.43 pp
HelpSteer2 non-tied pairs	373	45.31% (169/373)	44.50% (166/373)	+0.80 pp
Four-task reasoning macro	—	23.40%	20.69%	+2.71 pp
Five-metric composite	—	27.78%	25.45%	+2.33 pp

A 30,000-sample paired macro bootstrap gives a 95% composite-delta interval of [+0.68, +3.97] points and a 99.79% probability of a positive overall delta. GSM8K is the strongest individually confirmed gain. Full MATH-500 is a small, statistically insignificant regression (60 vs 62; paired p=0.888), so this release does not claim a MATH improvement.

Full protocol and scored counts are preserved in Asilarkness/testgeniy/candidates/budgie-alignment-v2/fullbench-old-vs-vmath25-v1.

Batching warning: ordinary padded batch generation is unsafe for this custom model. For exact behavior, generate sequentially or batch only prompts with identical tokenized input length, use no padding, and truncate each row at its first EOS.

Controlled comparison

All models used identical fixed rows and prompts with greedy decoding. Qwen used its native chat template with thinking disabled; Budgie used its native user/assistant template. This is a controlled regression suite, not a full official leaderboard run.

Benchmark	n	Budgie-500M	Previous public Budgie	Qwen3-0.6B non-thinking
GSM8K	30	16.67% (5/30)	13.33% (4/30)	46.67% (14/30)
MATH-500	15	20.00% (3/15)	13.33% (2/15)	13.33% (2/15)
ARC-Challenge	30	36.67% (11/30)	26.67% (8/30)	63.33% (19/30)
FOLIO	30	43.33% (13/30)	36.67% (11/30)	43.33% (13/30)
HelpSteer2 pairwise	200	48.50% (97/200)	48.00% (96/200)	50.00% (100/200)
Unweighted composite	5 metrics	33.03%	27.60%	43.33%

The controlled composite increases by 5.43 points over the previous public root.

Exact rows, revisions, prompts, token budgets, raw counts, and disclosures are in benchmark.json.

Expanded disjoint check

A larger second gate excludes every row in the controlled table above.

Benchmark	n	Budgie-500M	Previous public Budgie
GSM8K	80	30.00% (24/80)	11.25% (9/80)
MATH-500	50	20.00% (10/50)	20.00% (10/50)
ARC-Challenge	80	31.25% (25/80)	20.00% (16/80)
FOLIO	100	36.00% (36/100)	31.00% (31/100)
HelpSteer2 pairwise	373	45.31% (169/373)	44.50% (166/373)
Unweighted composite	5 metrics	32.51%	25.35%

Paired bootstrap with 30,000 resamples estimates a +7.16 point composite delta, a 95% interval of [+2.59, +11.73] points, and a 99.86% bootstrap probability of a positive delta.

MATH-500 is tied on the expanded gate; the confirmed larger-sample gains are GSM8K, ARC-Challenge, FOLIO, and a small HelpSteer2 gain.

Dialogue regression check

A separate 36-prompt English dialogue suite covered explanations, rewriting, practical advice, uncertainty, natural conversation, and exact output constraints. Responses were judged twice with anonymized A/B order reversal. Qwen3.8 generated one side and also served as judge, so the comparison may contain self-model bias; objective repetition and constraint checks are included alongside it.

Comparison	Budgie wins	Other wins	Ties
Current Budgie vs previous public Budgie	8	6	22
Current Budgie vs Qwen3.8-27B	0	35	1

Objective check	Current Budgie	Previous Budgie	Qwen3.8-27B
Repetition-loop flags (lower is better)	10/36	20/36	1/36
Exact constraint prompts passed	1/6	2/6	6/6
Honesty/uncertainty anchors passed	6/6	2/6	6/6

The current checkpoint does not show an overall dialogue regression versus the previous public Budgie: it has a small net pairwise advantage, much stronger honesty, and half as many repetition flags. However, it remains far behind Qwen3.8 in factual explanations, writing, practical assistance, natural conversation, and constraint following. Raw outputs and order-swapped judgments are preserved in the Alignment v2 reports.

Post-training lineage

The release descends from the previous public Budgie checkpoint at revision 39c877f07ddc80aa975809a4d87c848372316caf.

The retained lineage combined controlled full-parameter SFT, Qwen3.8-27B distillation, high-confidence dual-teacher and preference replay, verified mathematics, and task-vector interpolation to limit forgetting. The final checkpoint applies 25% of the retained verified-math continuation to the preceding precision checkpoint.

No reported fixed test or validation row was used for training. The lineage predates the project's stricter benchmark-evaluation-only policy and did use training splits from GSM8K, ARC-Challenge, Hendrycks MATH, and ProofWriter, together with NuminaMath, Bespoke-Stratos, SmolTalk, Qwen3.8 synthetic data, and random verified mathematics. This is disclosed to prevent misleading benchmark claims.

Full experiment history, rejected runs, source policy, confidence reports, and handoff notes are preserved at:

Asilarkness/testgeniy/candidates/budgie-alignment-v2

Architecture

• 487,800,064 unique parameters
• 24 layers, hidden size 1,280
• 10 query heads, 2 KV heads, head dimension 128
• SwiGLU intermediate size 3,584
• tied input/output embeddings
• 4,096-token context, RoPE theta 500,000
• hybrid RoPE/NoPE attention, with NoPE attention every fourth layer
• digit-aware tokenizer

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Asilarkness/Budgie-500m"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=True,
    dtype=torch.bfloat16,
).to("cuda").eval()

messages = [{"role": "user", "content": "What is 17 * 24? Explain briefly."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=160, do_sample=False)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

trust_remote_code=True is required for the custom TestGeniy architecture and digit-aware tokenizer.

Prompting note

Reasoning is currently strongest with the native user/assistant template and no long system message. A descriptive identity system prompt improves dialogue identity but can reduce reasoning accuracy; do not compare system-prompted and bare runs as if they were the same protocol.

Limitations

Budgie remains an experimental 488M model and is still behind Qwen3-0.6B on the controlled composite. It can produce incorrect, truncated, repetitive, or poorly calibrated reasoning. Bare identity is weak. Small changes to weights or decoding can flip answers. The reported gates are controlled samples with uncertainty, not official leaderboard submissions.

Provenance

• Previous public revision: 39c877f07ddc80aa975809a4d87c848372316caf
• Alignment source: Asilarkness/testgeniy
• Alignment source path: candidates/budgie-alignment-v2/verified-math-a025
• Alignment source revision: 79aec5157c10e3f7e082823a6e9150482d740a91
• Released checkpoint: verified-math-a025
• Model SHA-256: 7404de2c08df968f41800d1362704a0ad4106ea0116cd0cd4b1709612a2a66be
• Context length: 4,096 tokens