README.md

---
license: apache-2.0
pipeline_tag: text-generation
library_name: pytorch
tags:
  - ternary
  - bitnet
  - 1.58-bit
  - code
  - base-model
  - experimental
language:
  - en
---

# 1bitLabs Ultron 0.3B — Base (Experimental Research Artifact)

**Experimental native-ternary dense base model, ~305M parameters.** Every weight
is -1, 0, or +1 from step one — not post-hoc quantized. Code-heavy pretraining
corpus, 30B tokens, 2048 context.

**Not an instruct or chat model.** The base continues text; it does not follow
requests. A Python-coding instruct variant is included and follows short
specifications, but produces frequently-incorrect algorithms.

**Not compatible with stock bitnet.cpp or llama.cpp.** The architecture uses a
two-tensor squared-ReLU FFN with no sub-norms, which does not match any BITNET
graph in llama.cpp. Reference runtime is PyTorch, included.

Intended for fine-tuning, ternary scaling study, and comparison. Not for
out-of-box assistant use.

Trained by one person on rented compute for about $235. Full recipe, costs, and
failure log below.

---

## Files

| File | What it is |
|---|---|
| `ultron_r008_305m.pt` | Base model, step 228882 |
| `ultron_r008_305m_instruct.pt` | Python instruct variant |
| `ultron.py` | Self-contained inference: model, KV cache, CLI |

Tokenizer, config, and the special-token table are bundled inside each
checkpoint. There are no other files to download.

```bash
pip install torch sentencepiece
python ultron.py -p "def fibonacci(n):"

python ultron.py --ckpt ultron_r008_305m_instruct.pt \
  -p "Implement a function \`reverse_string(s: str) -> str\` that returns the reverse of the input string. Raise TypeError if the input is not a string."
```

---

## Architecture

| | |
|---|---|
| Parameters | 305.45M |
| Layers | 26 |
| Hidden | 1024 |
| Heads | 16 query / 4 KV (GQA) |
| FFN | squared-ReLU, d_ff 3840 — **two tensors, no gate** |
| Norm | RMSNorm, pre-norm — **no sub-norms** |
| RoPE base | 10000.0 |
| Context | 2048 |
| Vocab | 32000 (SentencePiece) |
| Embeddings | tied |
| Quantization | absmean ternary, per-output-channel scale, 8-bit activations |

### Why this is not a drop-in BitNet GGUF

Ternary weights are the quantization scheme. The architecture is a separate
question, and this one differs from Microsoft's BitNet b1.58.

All three BITNET architectures in llama.cpp — `BITNET`, `BITNET_25`,
`BITNET_B158` — require `FFN_GATE`, `ATTN_SUB_NORM`, and `FFN_SUB_NORM`. Ultron
has a two-tensor squared-ReLU FFN and no sub-norms, so it has none of them.
Conversion would require a patched llama.cpp.

**Native ternary != drop-in b1.58 GGUF.** Use the PyTorch path.

---

## Training

30B tokens on a single A100-SXM4-40GB, ~14.3 days, **~$235** rented.

| Domain | Share | Source |
|---|---|---|
| Code | 65% | The Stack (10 languages) |
| General | 20% | FineWeb-Edu |
| Math | 10% | open-web-math, finemath |
| Science | 5% | peS2o |

228,882 steps at 131,072 tokens/step. MFU 14.3%. Two OOM restarts, both
recovered from checkpoints.

**Final val_loss 2.5064**, inside a pre-registered band of 2.35-2.55 with a
primary prediction of 2.42, registered before any training data existed.

### What val_loss 2.5064 does and does not mean

The validation set is **100% peS2o science** while training was 65% code. This
is an out-of-domain number and there is no in-domain generalization measurement.
That is a design mistake — a stratified holdout should have been cut at build
time and cannot be recovered retroactively.

More importantly: on peS2o-style prompts the model reproduces academic register
with real fidelity — section numbering, statistical boilerplate, citation
conventions — and says nothing of substance. Cross-entropy rewards exactly that.
**2.5064 is a real, honestly obtained, pre-registered number that is compatible
with output that is fluent, well-formatted, and empty.**

---

## Behaviour

Measured on 30 frozen probes, greedy and sampled, 256 tokens each.

### Form >> content

The base has learned the *shape* of Python — imports, class structure, dunder
conventions, docstring formats, the `__main__` guard, where a file ends — much
more strongly than what code computes.

Greedy, `def fibonacci(n):` after a partial loop body:

```python
    a, b = b, a + b
    return a
```

Correct. Then it writes `fibonacci_2`, `fibonacci_3`, `fibonacci_4` with
identical bodies until the token cap.

Sampled (temp 0.8, top-k 40), same prompt:

```python
    if a < b:
        a = b
    else:
        b = a
    return a

def main():
    fibonacci(10)

if __name__ == "__main__":
    main()
```

Wrong algorithm. Well-formed file. Terminates.

**Neither decoding mode is "the truth."** Greedy is deterministic and sometimes
exactly right on short completions, but locks into verbatim repetition. Sampling
produces better document structure and terminates more often, at some cost to
correctness. Both are shown deliberately.

### Termination

The model emits `<|endoftext|>` (token 4) at document boundaries. Teacher-forced
at 20 real corpus boundaries: **median rank 2**, top-1 in 6/20, max p 0.88, from
a 32,000-token vocabulary.

In practice: **3/30 sampled probes terminated, all three code completions. 0/24
on prose, math, or science.** Greedy: 0/30 — under argmax a rank-2 token never
wins.

Always set a length cap.

### Where the base fails

- **Synthesis from specification.** All 7 docstring-to-function probes wrong:
  `s.lower() == s.lower()` for palindrome, `[x for x in a if x not in b]`
  labelled as a merge, `arr.index(target)` labelled binary search, infinite
  recursion in flatten.
- **Math.** Gives the probability of *not* rolling a six as 1/3. "Proves" a
  summation identity by listing n = 1 through 29.
- **Prose.** Tautological loops. "The first mistake is to make a mistake when
  cooking with cast iron."
- **Non-Python code.** JavaScript degenerates. Go invents `wg.Done()` as a
  channel. SQL emits eight identical LEFT JOINs.

---

## The instruct variant

SFT on 3,111 execution-verified Python coding tasks. 276 steps, 37 minutes,
val_loss 0.2739.

**No chat template.** Bare instruction in, code out.

Prompt:

> Implement a function `reverse_string(s: str) -> str` that returns the reverse
> of the input string. Raise TypeError if the input is not a string.

Output:

```python
def reverse_string(s: str) -> str:
    if not isinstance(s, str):
        raise TypeError("Input must be a string")
    return s.replace(" ", "").lower()
```

Correct signature. Correct type guard. Correct error with a sensible message.
**Wrong body.** Terminates cleanly.

This is representative: the right *shape* of answer, frequently the wrong
algorithm.

### Prompt detail matters

Training prompts averaged ~194 tokens — backtick-quoted signatures, type
annotations, explicit error conditions, edge cases. Terse prompts do markedly
worse.

"Write a function that reverses a string." produces:

```python
    def reverse(self):
        return self.replace(...)
```

Write specifications, not requests.

### Python only

The SFT set was 3,111 Python rows. Off-distribution requests fail badly.

Asked for an HTML page directly, the model emitted
`import { LitElement, html } from 'lit';` eleven times and ran to the cap
without terminating.

Asked for a *Python function that returns HTML*, it produced a well-formed
function that terminated — and read "escape HTML special characters" as "strip
all non-alphanumerics."

---

## Limitations

- Frequently wrong algorithms. Structure is not correctness.
- Python only in the instruct variant.
- No math.
- Degenerates into repetition off-distribution, and does not terminate there.
- 2048 context.
- No multi-turn, no tool use, no chat template.
- **No safety tuning of any kind.**
- Not stock bitnet.cpp / llama.cpp compatible.
- No in-domain benchmark number. Golden-set evaluation is in progress; the
  harness that produced earlier numbers did not pass a stop token, so those
  numbers are uninterpretable and are withheld.

**Do not use this for anything that matters.** It is a research artifact and a
fine-tuning starting point.

---

## What broke, and what we fixed

Published because the lab publishes failures.

**The trainer used the wrong EOS token.** `sft_train.py` resolved end-of-text via
`sp.eos_id()`, which returns 2 (`</s>`). The corpus uses token 4
(`<|endoftext|>`). Every SFT and DPO run for roughly a month terminated responses
with a token the base had never seen at a document boundary across 30B tokens,
while suppressing the one it had. The wrong value printed in every training log
and nobody caught it, because 2 *is* the correct SentencePiece EOS — just not for
this corpus. Fixed by reading `eot_id` from the corpus manifest and asserting the
piece.

**A stale directory produced a false diagnosis.** A `runs/R008/` directory held a
*dense, non-ternary* 16k-vocab baseline stopped at 1% of training. Checking token
id 4 against that model's tokenizer produced the conclusion that the corpus
separator was an apostrophe and that the model had no learnable stop token. That
conclusion propagated across work sessions into a documented lesson and became
the load-bearing premise of a recovery plan that included a corpus rebuild. It
was false. Token 4 is a properly reserved `<|endoftext|>` written at document
boundaries throughout the corpus; the apostrophe is 27530. The error was caught
the first time anyone generated from the model and read the raw output.

**The evaluation harness never stopped generation.** It passed no stop token and
the reference `generate()` had no stop logic, so every evaluation ran to the
length cap regardless of what the model emitted. Earlier benchmark numbers are
therefore uninterpretable and are not published.

If you fine-tune this model: the EOT is **token 4**. Do not use `sp.eos_id()`.

---

## Citation

```bibtex
@misc{ultron-r008-2026,
  title  = {Ultron 0.3B: an experimental native ternary language model},
  author = {1bitLabs},
  year   = {2026},
  note   = {305M parameters, 30B tokens, ~\$235}
}
```

## License

Apache 2.0 — weights and code.

Training data derives from The Stack, FineWeb-Edu, open-web-math, finemath, and
peS2o. Their respective licenses apply to their content.