Finnish H-Nets (195M and 501M)
Two byte-level H-Nets trained from scratch on Finnish, on a single RX 7900 XT. No tokenizer: the model reads raw UTF-8 bytes and learns its own chunk boundaries.
These are research checkpoints, not chat models. They predict the next byte and expose their segmentation; they have no instruction tuning and no safety tuning.
Results
| 195M | 501M | |
|---|---|---|
| parameters | 195.1M | 501.1M |
| layout | m4/T12/m4 |
m4/T20/m4 |
| d_model | 768 β 1024 | 1024 β 1280 |
| data | 690 MB Finnish Wikipedia | 1.53 GB (FineWeb-2 1 GB + Wikipedia 0.5 GB) |
| training | 84 000 Γ 8192 bytes | 183 000 Γ 8192 bytes |
| held-out bits/byte | 1.2540 | 1.2531 |
| mean chunk length | 3.55 bytes | 3.97 bytes |
| boundaries inside a stem | 56.8% | 73.0% |
For reference, the English/Chinese/code Cartesia checkpoint (hnet_1stage_L, 1.3B) scores 2.074 bits/byte
on the same held-out Finnish text, so training on Finnish cuts entropy by about 40% with a much smaller model.
Scaling 2.5Γ on 2.2Γ the data bought 0.0009 bits/byte and made the segmentation less word-aligned: the 501M model puts 73% of its chunk boundaries inside a stem, and its word-start and word-end boundary shares are both below what random cuts would produce. Byte-level modelling works on Finnish; the morphological story does not, at this scale. Details and the probe tool: bqv/hnets.
Files
195m/model.safetensors bf16, state_dict of HNetForCausalLM
195m/config.json architecture config, upstream format
501m/model.safetensors
501m/config.json
Usage
import json, torch
from hnet.models.config_hnet import AttnConfig, SSMConfig, HNetConfig
from hnet.models.mixer_seq import HNetForCausalLM
from hnet.utils.tokenizers import ByteTokenizer
from safetensors.torch import load_file
cfg = json.load(open("195m/config.json"))
attn = AttnConfig(**cfg.pop("attn_cfg"))
ssm = SSMConfig(**cfg.pop("ssm_cfg"))
model = HNetForCausalLM(HNetConfig(**cfg, attn_cfg=attn, ssm_cfg=ssm), device="cuda", dtype=torch.bfloat16)
model.load_state_dict(load_file("195m/model.safetensors"))
model.eval()
ids = torch.tensor(ByteTokenizer().encode(["Suomen kieli on agglutinatiivinen."], add_bos=True)[0]["input_ids"],
device="cuda").unsqueeze(0)
out = model.forward(ids)
print(out.bpred_output[0].boundary_mask)
bpred_output is a list, one entry per hierarchy stage; each carries boundary_mask and the router
probabilities. Install the trainer, probe CLI and ROCm shims from bqv/hnets.
Running in llama.cpp
There is an experimental fork that adds an hnet architecture to llama.cpp:
bqv/llama-hnet. It loads these checkpoints, reproduces the learned chunking
boundary for boundary against the PyTorch reference, and can generate text byte by byte:
git clone https://github.com/bqv/llama-hnet && cd llama-hnet
cmake -B build -G Ninja -DLLAMA_CURL=OFF -DCMAKE_BUILD_TYPE=Release
cmake --build build -j4 --target hnet-generate hnet-segment
python convert_hnet_to_gguf.py --input path/to/195m --output 195m.gguf
./build/bin/hnet-generate 195m.gguf "Suomen kieli on " 12 # Suomen kieli on kieli, joka
printf 'Suomen kieli on vaikea.' | ./build/bin/hnet-segment 195m.gguf
# |S|uom|en| kieli| o|n| va|i|ke|a|.
Quantise it if you want it small, keeping the convolution weights in float:
llama-quantize --tensor-type "conv1d.weight=f32" 501m.gguf 501m-q4.gguf Q4_0
# 956 MB -> 270 MB; 12 bytes of Finnish in about 2 s on a CPU
CPU is the reference backend and works everywhere; the fork also builds with CUDA and runs several times faster
per byte on a small card, if the model fits. The fork is not upstream; see its HNET.md for the parity numbers,
the quantisation constraints and the known gaps.
Data and licence
Trained on Finnish Wikipedia (CC BY-SA) and the Finnish subset of FineWeb-2 (ODC-By). Checkpoint weights are released under MIT, matching the upstream H-Net repository; the training data keeps its own licences, and outputs may reflect them.
H-Net is by Sukjun Hwang, Brandon Wang and Albert Gu (arXiv:2507.07955). These checkpoints are an independent training run, not an official release.