1bitLabs/ultron-0.3b-base-experimental

1bitLabs Ultron 0.3B — Base (Experimental Research Artifact)

Experimental native-ternary dense base model, ~305M parameters. Every weight is -1, 0, or +1 from step one — not post-hoc quantized. Code-heavy pretraining corpus, 30B tokens, 2048 context.

Not an instruct or chat model. The base continues text; it does not follow requests. A Python-coding instruct variant is included and follows short specifications, but produces frequently-incorrect algorithms.

Not compatible with stock bitnet.cpp or llama.cpp. The architecture uses a two-tensor squared-ReLU FFN with no sub-norms, which does not match any BITNET graph in llama.cpp. Reference runtime is PyTorch, included.

Intended for fine-tuning, ternary scaling study, and comparison. Not for out-of-box assistant use.

Trained by one person on rented compute for about $235. Full recipe, costs, and failure log below.

---

Files

File	What it is
ultron_r008_305m.pt	Base model, step 228882
ultron_r008_305m_instruct.pt	Python instruct variant
ultron.py	Self-contained inference: model, KV cache, CLI

Tokenizer, config, and the special-token table are bundled inside each checkpoint. There are no other files to download.

pip install torch sentencepiece
python ultron.py -p "def fibonacci(n):"

python ultron.py --ckpt ultron_r008_305m_instruct.pt \
  -p "Implement a function \`reverse_string(s: str) -> str\` that returns the reverse of the input string. Raise TypeError if the input is not a string."

---

Architecture

Parameters	305.45M
Layers	26
Hidden	1024
Heads	16 query / 4 KV (GQA)
FFN	squared-ReLU, d_ff 3840 — two tensors, no gate
Norm	RMSNorm, pre-norm — no sub-norms
RoPE base	10000.0
Context	2048
Vocab	32000 (SentencePiece)
Embeddings	tied
Quantization	absmean ternary, per-output-channel scale, 8-bit activations

Why this is not a drop-in BitNet GGUF

Ternary weights are the quantization scheme. The architecture is a separate question, and this one differs from Microsoft's BitNet b1.58.

All three BITNET architectures in llama.cpp — BITNET, BITNET_25, BITNET_B158 — require FFN_GATE, ATTN_SUB_NORM, and FFN_SUB_NORM. Ultron has a two-tensor squared-ReLU FFN and no sub-norms, so it has none of them. Conversion would require a patched llama.cpp.

Native ternary != drop-in b1.58 GGUF. Use the PyTorch path.

---

Training

30B tokens on a single A100-SXM4-40GB, 14.3 days, **$235** rented.

Domain	Share	Source
Code	65%	The Stack (10 languages)
General	20%	FineWeb-Edu
Math	10%	open-web-math, finemath
Science	5%	peS2o

228,882 steps at 131,072 tokens/step. MFU 14.3%. Two OOM restarts, both recovered from checkpoints.

Final val_loss 2.5064, inside a pre-registered band of 2.35-2.55 with a primary prediction of 2.42, registered before any training data existed.

What val_loss 2.5064 does and does not mean

The validation set is 100% peS2o science while training was 65% code. This is an out-of-domain number and there is no in-domain generalization measurement. That is a design mistake — a stratified holdout should have been cut at build time and cannot be recovered retroactively.

More importantly: on peS2o-style prompts the model reproduces academic register with real fidelity — section numbering, statistical boilerplate, citation conventions — and says nothing of substance. Cross-entropy rewards exactly that. 2.5064 is a real, honestly obtained, pre-registered number that is compatible with output that is fluent, well-formatted, and empty.

---

Behaviour

Measured on 30 frozen probes, greedy and sampled, 256 tokens each.

Form >> content

The base has learned the shape of Python — imports, class structure, dunder conventions, docstring formats, the __main__ guard, where a file ends — much more strongly than what code computes.

Greedy, def fibonacci(n): after a partial loop body:

    a, b = b, a + b
    return a

Correct. Then it writes fibonacci_2, fibonacci_3, fibonacci_4 with identical bodies until the token cap.

Sampled (temp 0.8, top-k 40), same prompt:

    if a < b:
        a = b
    else:
        b = a
    return a

def main():
    fibonacci(10)

if __name__ == "__main__":
    main()

Wrong algorithm. Well-formed file. Terminates.

Neither decoding mode is "the truth." Greedy is deterministic and sometimes exactly right on short completions, but locks into verbatim repetition. Sampling produces better document structure and terminates more often, at some cost to correctness. Both are shown deliberately.

Termination

The model emits <|endoftext|> (token 4) at document boundaries. Teacher-forced at 20 real corpus boundaries: median rank 2, top-1 in 6/20, max p 0.88, from a 32,000-token vocabulary.

In practice: 3/30 sampled probes terminated, all three code completions. 0/24 on prose, math, or science. Greedy: 0/30 — under argmax a rank-2 token never wins.

Always set a length cap.

Where the base fails

• Synthesis from specification. All 7 docstring-to-function probes wrong: s.lower() == s.lower() for palindrome, [x for x in a if x not in b] labelled as a merge, arr.index(target) labelled binary search, infinite recursion in flatten.
• Math. Gives the probability of not rolling a six as 1/3. "Proves" a summation identity by listing n = 1 through 29.
• Prose. Tautological loops. "The first mistake is to make a mistake when cooking with cast iron."
• Non-Python code. JavaScript degenerates. Go invents wg.Done() as a channel. SQL emits eight identical LEFT JOINs.

---

The instruct variant

SFT on 3,111 execution-verified Python coding tasks. 276 steps, 37 minutes, val_loss 0.2739.

No chat template. Bare instruction in, code out.

Prompt:

Implement a function reverse_string(s: str) -> str that returns the reverse of the input string. Raise TypeError if the input is not a string.

Output:

def reverse_string(s: str) -> str:
    if not isinstance(s, str):
        raise TypeError("Input must be a string")
    return s.replace(" ", "").lower()

Correct signature. Correct type guard. Correct error with a sensible message. Wrong body. Terminates cleanly.

This is representative: the right shape of answer, frequently the wrong algorithm.

Prompt detail matters

Training prompts averaged ~194 tokens — backtick-quoted signatures, type annotations, explicit error conditions, edge cases. Terse prompts do markedly worse.

"Write a function that reverses a string." produces:

    def reverse(self):
        return self.replace(...)

Write specifications, not requests.

Python only

The SFT set was 3,111 Python rows. Off-distribution requests fail badly.

Asked for an HTML page directly, the model emitted import { LitElement, html } from 'lit'; eleven times and ran to the cap without terminating.

Asked for a Python function that returns HTML, it produced a well-formed function that terminated — and read "escape HTML special characters" as "strip all non-alphanumerics."

---

Limitations

• Frequently wrong algorithms. Structure is not correctness.
• Python only in the instruct variant.
• No math.
• Degenerates into repetition off-distribution, and does not terminate there.
• 2048 context.
• No multi-turn, no tool use, no chat template.
• No safety tuning of any kind.
• Not stock bitnet.cpp / llama.cpp compatible.
• No in-domain benchmark number. Golden-set evaluation is in progress; the harness that produced earlier numbers did not pass a stop token, so those numbers are uninterpretable and are withheld.

Do not use this for anything that matters. It is a research artifact and a fine-tuning starting point.

---

What broke, and what we fixed

Published because the lab publishes failures.

The trainer used the wrong EOS token. sft_train.py resolved end-of-text via sp.eos_id(), which returns 2 (</s>). The corpus uses token 4 (<|endoftext|>). Every SFT and DPO run for roughly a month terminated responses with a token the base had never seen at a document boundary across 30B tokens, while suppressing the one it had. The wrong value printed in every training log and nobody caught it, because 2 is the correct SentencePiece EOS — just not for this corpus. Fixed by reading eot_id from the corpus manifest and asserting the piece.

A stale directory produced a false diagnosis. A runs/R008/ directory held a dense, non-ternary 16k-vocab baseline stopped at 1% of training. Checking token id 4 against that model's tokenizer produced the conclusion that the corpus separator was an apostrophe and that the model had no learnable stop token. That conclusion propagated across work sessions into a documented lesson and became the load-bearing premise of a recovery plan that included a corpus rebuild. It was false. Token 4 is a properly reserved <|endoftext|> written at document boundaries throughout the corpus; the apostrophe is 27530. The error was caught the first time anyone generated from the model and read the raw output.

The evaluation harness never stopped generation. It passed no stop token and the reference generate() had no stop logic, so every evaluation ran to the length cap regardless of what the model emitted. Earlier benchmark numbers are therefore uninterpretable and are not published.

If you fine-tune this model: the EOT is token 4. Do not use sp.eos_id().

---

Citation

@misc{ultron-r008-2026,
  title  = {Ultron 0.3B: an experimental native ternary language model},
  author = {1bitLabs},
  year   = {2026},
  note   = {305M parameters, 30B tokens, ~\$235}
}

License

Apache 2.0 — weights and code.

Training data derives from The Stack, FineWeb-Edu, open-web-math, finemath, and peS2o. Their respective licenses apply to their content.