Braid 300M (open-v1)

A 295M-parameter tokenizer-free byte-level language model. Braid replaces most of a transformer's attention layers with cheap gated causal convolutions and spends attention only at 1/16 of positions, inside a causal U-Net over raw bytes. The decode state is constant in context length. There is no tokenizer.

This is a research preview. The win is systems cost -- attention on 1/16 of positions, bounded decode state, no tokenizer to version or align -- not quality. A matched dense transformer reaches slightly lower loss at equal data (0.019 bpb at 30M scale, three seeds; see Comparisons below). We release weights and runtime so others can build on the architecture, not because the numbers are competitive with production LMs.

Developer: Solexsis Research

Architecture

HourglassBraid: a causal U-Net over the byte sequence.

bytes (vocab 264: 256 byte values + 8 reserved)
  -> hashed byte n-gram (3,4) input embeddings, prime buckets
  -> gated causal conv stem (2 ConvGLU blocks, kernel 4)
  -> causal 4x downsample
  -> causal 4x downsample
  -> dense attention at 1/16 rate, bounded window (128 coarse positions)
  -> 4x upsample + gated U-Net skip
  -> 4x upsample + gated U-Net skip
  -> 256-byte sliding-window attention
  -> byte logits, mixed with exact suffix-copy branch (lengths 4, 8, 16; 3 votes)

Causality is structural, not masked: each pooled sequence is shifted by one position so a coarse slot only summarises completed groups. Both causality and the invariant "cached incremental decode reproduces the parallel forward" are enforced by the test suite.

295,092,931 parameters trained; 289,669,891 at inference (the three multi-token-prediction training heads are not shipped). float32 weights. Context: 8,192 bytes. Decode state: fixed size regardless of prompt length, because both attention windows are bounded.

Training

Data: 128,000,720,896 bytes (one epoch) of Solenopsisbot/braid-open-v1, an openly licensed byte mixture pinned to one Common Pile revision.
Hardware: 4 x H200, 14 h 09 min.
Recipe: 122,071 steps of 4 x 32 x 8,192 bytes. Muon optimizer on 2D hidden matrices, AdamW on embeddings/norms/conv taps. Warmup-stable-decay schedule (20% decay). Multi-token prediction aux heads (offsets 2, 3, 4; weight 0.3). Seed 1337.

Evaluation

Bits per byte on the corpus's own frozen held-out validation splits (every complete 8,192-byte context, bf16 inference):

source	scored bytes	bpb
cccc	17,915,904	0.8917
stackexchange	11,517,952	0.9655
wikimedia	7,675,904	1.0363
peS2o	6,397,952	0.8773
arxiv_papers	3,227,648	0.8685
project_gutenberg	5,775,360	1.1568
pre_1929_books	3,899,392	1.1608
stackv2_edu (code)	7,708,672	0.6161
byte-weighted aggregate	64,118,784	0.9268

Best validation bpb across training: 0.9296 (the final eval; the run was monotonic).

Caveat: these numbers are measured on this model's own corpus and are not directly comparable to other models scored on other data. The same architecture trained on a different 64 GB mixture (diverse-v1, which includes FineWeb-Edu) reads 0.9063; the gap is corpus difficulty, not the recipe.

Use of context

Scored with chunked inference past the 8K training crop, the model predicts the same bytes better the more context it is given, monotonically, out to 63,488 bytes (the last 2,048 bytes of each window scored with its context and again standalone, paired per window, 32/32 windows better at every length):

context bytes	books	mixed (open-v1 val)	code
512	-0.031	-0.032	-0.080
2,048	-0.040	-0.043	-0.120
8,192	-0.048	-0.048	-0.119
63,488	-0.058	-0.064	-0.203

(bpb, in-context minus standalone; negative is better.) On prose the gain past 512 bytes is small; on code the long history pays through the copy branch.

Comparisons

These are from controlled experiments in the research repo, not from this checkpoint. They establish what the architecture trades.

At 30M parameters on FineWeb (three paired seeds, matched data):

	final bpb (95% CI)	steady throughput	peak VRAM	gross energy
Default Braid	1.5296 +/-0.0040	465K byte/s	1.86 GiB	15.8 Wh
MHA Transformer	1.5106 +/-0.0070	152K byte/s	4.13 GiB	47.2 Wh

The transformer wins quality by 0.019 bpb. Braid trains at 3.07x the throughput, uses 2.23x less peak memory, and consumes 2.99x less energy.

At 300M and matched training FLOPs, byte-level Braid vs 8K-BPE Braid inside the same hourglass family: BPE is 1.35% better on prose, bytes are 1.49% better on code (both significant, both near the harness's resolution limit). The effects roughly cancel; neither side wins cleanly. A flat byte transformer (no hierarchy) is 6-8% behind everywhere -- attention over raw bytes is quadratic, and the hourglass is what makes tokenizer-freedom affordable.

Usage

pip install torch "braid-lite @ https://huggingface.co/Solexsis/braid-300m/resolve/main/runtime/braid_lite-0.1.0-py3-none-any.whl"
hf download Solexsis/braid-300m --local-dir braid-300m

from braid_lite import load

model = load("braid-300m")  # CUDA if available, else CPU
for text in model.generate_stream("The history of the printing press",
                                   max_new_bytes=300, temperature=0.8, top_k=50, seed=1234):
    print(text, end="", flush=True)

The repo holds config.json (the architecture, provenance, and the weights' SHA-256, verified on load) and model.safetensors. braid-lite is a standalone, torch-only, inference-only implementation of exactly this architecture: three files, Apache-2.0, with a parallel prefill and an O(1) per-byte cached decoder. It is tested against the research implementation to ~1e-5 in logits on this checkpoint. Training code is not part of this release.

Sample output (seed 1234, temperature 0.8, top-k 50, CPU), prompt "The history of the printing press":

in France is a series of historic and compiled papers which originated in the industrial world in the late 18th century. The printing industry was in a much earlier period of development and steam navigation was the new primary mode of operation. Two branches of printing were found, with the first

Fluent prose; not factual. This is a base model. It continues text; it does not chat, follow instructions, or retrieve knowledge.

Limitations

Not instruction-tuned, not aligned, no safety filtering. Output may be offensive, incorrect, or harmful.
Output can contain an invalid UTF-8 tail (the model predicts bytes, not characters; a generation may end mid-sequence).
Tested on CPU and CUDA. Apple GPU (MPS) is untested with braid-lite.
Quality is below a matched dense transformer at equal data (see Comparisons). The architecture's value is in systems cost, not perplexity.
The runtime's context limit is 8,192 bytes (the training crop); longer prompts are truncated to their last 8,192 bytes. The model itself is not the limit -- see "Use of context" above -- but the packaged runtime does not yet expose longer contexts.
Single-stream decode is ~240-280 bytes/s on the ZeroGPU demo (half an RTX Pro 6000, bf16, braid-lite eager; 1,024-byte generations, prefill included).

Licence

Apache-2.0 (weights and runtime).

Citation

@misc{braid2026,
  title   = {Braid: a tokenizer-free byte-level hourglass language model},
  author  = {{Solexsis Research}},
  year    = {2026},
  url     = {https://huggingface.co/Solexsis/braid-300m}
}

Downloads last month: 115

Safetensors

Model size

0.3B params

Tensor type

F32

Solexsis
/

braid-300m