Finnish H-Nets (195M and 501M)

Two byte-level H-Nets trained from scratch on Finnish, on a single RX 7900 XT. No tokenizer: the model reads raw UTF-8 bytes and learns its own chunk boundaries.

These are research checkpoints, not chat models. They predict the next byte and expose their segmentation; they have no instruction tuning and no safety tuning.

Results

195M 501M
parameters 195.1M 501.1M
layout m4/T12/m4 m4/T20/m4
d_model 768 β†’ 1024 1024 β†’ 1280
data 690 MB Finnish Wikipedia 1.53 GB (FineWeb-2 1 GB + Wikipedia 0.5 GB)
training 84 000 Γ— 8192 bytes 183 000 Γ— 8192 bytes
held-out bits/byte 1.2540 1.2531
mean chunk length 3.55 bytes 3.97 bytes
boundaries inside a stem 56.8% 73.0%

For reference, the English/Chinese/code Cartesia checkpoint (hnet_1stage_L, 1.3B) scores 2.074 bits/byte on the same held-out Finnish text, so training on Finnish cuts entropy by about 40% with a much smaller model.

Scaling 2.5Γ— on 2.2Γ— the data bought 0.0009 bits/byte and made the segmentation less word-aligned: the 501M model puts 73% of its chunk boundaries inside a stem, and its word-start and word-end boundary shares are both below what random cuts would produce. Byte-level modelling works on Finnish; the morphological story does not, at this scale. Details and the probe tool: bqv/hnets.

Files

195m/model.safetensors   bf16, state_dict of HNetForCausalLM
195m/config.json         architecture config, upstream format
501m/model.safetensors
501m/config.json

Usage

import json, torch
from hnet.models.config_hnet import AttnConfig, SSMConfig, HNetConfig
from hnet.models.mixer_seq import HNetForCausalLM
from hnet.utils.tokenizers import ByteTokenizer
from safetensors.torch import load_file

cfg = json.load(open("195m/config.json"))
attn = AttnConfig(**cfg.pop("attn_cfg"))
ssm = SSMConfig(**cfg.pop("ssm_cfg"))
model = HNetForCausalLM(HNetConfig(**cfg, attn_cfg=attn, ssm_cfg=ssm), device="cuda", dtype=torch.bfloat16)
model.load_state_dict(load_file("195m/model.safetensors"))
model.eval()

ids = torch.tensor(ByteTokenizer().encode(["Suomen kieli on agglutinatiivinen."], add_bos=True)[0]["input_ids"],
                   device="cuda").unsqueeze(0)
out = model.forward(ids)
print(out.bpred_output[0].boundary_mask)

bpred_output is a list, one entry per hierarchy stage; each carries boundary_mask and the router probabilities. Install the trainer, probe CLI and ROCm shims from bqv/hnets.

Running in llama.cpp

There is an experimental fork that adds an hnet architecture to llama.cpp: bqv/llama-hnet. It loads these checkpoints, reproduces the learned chunking boundary for boundary against the PyTorch reference, and can generate text byte by byte:

git clone https://github.com/bqv/llama-hnet && cd llama-hnet
cmake -B build -G Ninja -DLLAMA_CURL=OFF -DCMAKE_BUILD_TYPE=Release
cmake --build build -j4 --target hnet-generate hnet-segment

python convert_hnet_to_gguf.py --input path/to/195m --output 195m.gguf

./build/bin/hnet-generate 195m.gguf "Suomen kieli on " 12   # Suomen kieli on kieli, joka
printf 'Suomen kieli on vaikea.' | ./build/bin/hnet-segment 195m.gguf
# |S|uom|en| kieli| o|n| va|i|ke|a|.

Quantise it if you want it small, keeping the convolution weights in float:

llama-quantize --tensor-type "conv1d.weight=f32" 501m.gguf 501m-q4.gguf Q4_0
# 956 MB -> 270 MB; 12 bytes of Finnish in about 2 s on a CPU

CPU is the reference backend and works everywhere; the fork also builds with CUDA and runs several times faster per byte on a small card, if the model fits. The fork is not upstream; see its HNET.md for the parity numbers, the quantisation constraints and the known gaps.

Data and licence

Trained on Finnish Wikipedia (CC BY-SA) and the Finnish subset of FineWeb-2 (ODC-By). Checkpoint weights are released under MIT, matching the upstream H-Net repository; the training data keeps its own licences, and outputs may reflect them.

H-Net is by Sukjun Hwang, Brandon Wang and Albert Gu (arXiv:2507.07955). These checkpoints are an independent training run, not an official release.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Paper for izf/hnet-finnish