ThoxMicro-1bit-16M

A BitNet b1.58 ternary language model, 15,737,088 parameters, trained from scratch. Block weights are {-1, 0, +1} with a per-tensor absmean scale; activations are int8 per-token (W1.58A8).

This is the deeper sibling of ThoxMicro-1bit-9M — same width, same vocabulary, same token budget, 16 layers instead of 8. It exists because of a specific measured finding, described below.

Licence pending. Trained from scratch on TinyStories (CDLA-Sharing-1.0). Whether share-alike terms extend to trained weights is not settled. Treat as unlicensed for redistribution until THOX confirms.

What this is — and is not

It writes simple children's stories. That is the whole of it.

• ❌ Not an assistant. No instruction tuning, no chat template.
• ❌ Cannot answer questions or follow instructions.
• ✅ A completion model and an edge/on-device research artifact.

Why 16 layers — the finding

The 9M model's training loss was 1.6318 against its fp16 twin's 1.4118. A model that cannot fit data it has already seen 24,000 steps of is not short of data or steps — it is short of representational capacity. Neither arm overfit, and validation was flat at the end, which ruled out the alternatives.

Doubling depth was the test. It closed 25% of the ternary gap:

	8 layers	16 layers
fp16	1.5476	1.4844
ternary	1.7385	1.6271 (this model)

Ternary's penalty against fp16 fell from +0.1909 to +0.1427 nats — a 25% reduction. Every cell is a real 24,000-step run at 393.2M tokens with the same seed and data order.

Correction, 2026-08-05. This card first reported the penalty as +0.0795 nats and "58% of the gap closed". That was wrong. It compared the 16-layer ternary model against the 8-layer fp16 baseline — the only fp16 number that existed at the time — which credits ternary with an improvement that came from depth. fp16 also improves with depth (1.5476 → 1.4844), and running that control showed the real figure is +0.1427 nats / 25%, overstating by 1.8×. The control run is configs/ternary/deep-16m-fp16.yaml.

It did not reach fp16. Capacity is the right lever; 2× is not the whole answer. That is the honest result and it is why this card does not claim parity.

Why depth and not width

BitNet "Reloaded" (arXiv:2407.09527) recommends widening when you ternarize. A d_model=512 config was written first and rejected on arithmetic before any GPU was spent: widening quadruples per-layer parameters while the device budget grows linearly, landing 58% larger than the fp16 model and 2.7× slower to decode. Batch-1 decode is bandwidth-bound — every parameter is re-read on every token — so width is charged per token forever.

The paper's advice is sound in its regime. It was not written against a 16 MB flash part with a 60.7 MB/s memory bus.

Architecture

Plain Llama decoder — RMSNorm, RoPE, SwiGLU. Nothing custom, so stock tooling converts it.

Parameters	15,737,088 (13,631,488 ternary = 86.6%)
vocab_size	8,192 (own byte-level BPE)
d_model	256
n_layers	16
n_heads	8 (head_dim 32, MHA)
ffn_hidden	768
seq_len	512

Every dimension is a multiple of 256, so block quantizers pack every tensor rather than silently falling back to F16.

Per BitNet, the embedding and output head are not ternarized.

Training

Data	TinyStories, 393,216,000 tokens
Steps	24,000
Hardware	one RTX 4060 Ti, 154 min
Cloud cost	$0

Generation quality

Measured over 10 prompts against the 9M model, greedy decode:

	9M (8 layers)	16M (16 layers)
mean 4-gram repetition	0.0406	0.0332
worst prompt	0.1500	0.1250
prompts that loop (>0.05)	3/10	3/10

Both models degenerate on 3 of 10 prompts — but on different prompts, with zero overlap. Lower loss did not eliminate looping; it moved it. Do not read the perplexity improvement as "the repetition problem is fixed".

Sample, greedy:

Tom and Sara went to the park and played on the swings and slides. They had fun and forgot about the time.

But then, they saw a big dog. The dog was angry and barked loud. Tom and Sara were scared and ran to their mom

Files

file	bytes	sha256	runs on
thoxmicro-1bit-16m.TQ2_0.gguf	8,030,496	13642c72e84db9c3…	llama.cpp
thoxmicro-1bit-16m.Q8_0.gguf	17,033,504	e0c8d621ea3c9caa…	llama.cpp and Ollama

Why two. TQ2_0 is the ternary-native quant and the smaller file, but Ollama cannot load it — its vendored ggml predates the type and fails with tensor "blk.0.ffn_down.weight" size overflow.

Q8_0 is not a quality downgrade: the weights are ternary, so every value is one of three levels and Q8_0's 256 levels hold them exactly. It is a less dense container for identical numbers.

Both passed a near-lossless check against the trained weights (112 tensors, worst relative error 4.73e-04 for TQ2_0).

On-device budget

A TERN1 artifact (6,147,260 B) exists for ESP32-S3 class targets. It is not distributed here — it is not a GGUF and no general-purpose runtime reads it.

	9M	16M
TERN1 artifact	4.42 MB	6.15 MB
KV cache @ seq 512	4.19 MB	8.39 MB
total resident	8.61 MB	14.54 MB
fits 16 MB flash	✅	✅ (1.5 MB spare)

The 9M is still the right choice for tighter targets. This one fits, but with much less headroom, and its KV cache doubles. Neither supersedes the other.

Usage

llama-completion -m thoxmicro-1bit-16m.TQ2_0.gguf \
  -p "Once upon a time, there was a little girl named Lily. She" \
  -n 60 --temp 0

A completion model — llama-cli's conversation mode has no chat template to apply.

Limitations

• TinyStories domain only.
• No instruction following, no factual grounding, no safety tuning.
• Loops on roughly 3 of 10 greedy prompts. Use a repeat penalty.
• Still +0.1427 nats behind its fp16 equivalent at the same depth. Published because the measurement is the point — including when the first version of that measurement was wrong.

Provenance

Trained in thoxllm-factory (thox_ternary/), config configs/ternary/deep-16m-ternary.yaml. Success and failure thresholds were registered in that config before the run.