Hetman-2.0B
Hetman-2.0B is an edge-native foundation model engineered for high-efficiency reasoning, extreme context handling, and low-bit compute. Developed from first principles for consumer hardware accessibility, Hetman combines 1.58-bit ternary quantization, linear recurrent attention, and sparse associative memory to deliver 256,000-token context processing within consumer GPU memory budgets.
Introduction
In conventional autoregressive language models, expanding context windows and scaling parameters to preserve factual recall inevitably incurs an unsustainable hardware penalty. At 128k–256k tokens, the KV-cache of standard transformers expands to tens of gigabytes of VRAM, pricing independent developers, researchers, and edge devices out of frontier AI.
Hetman-2.0B rethinks the core architecture of sub-3B parameter models:
- Rather than memorizing encyclopedic facts inside continuous dense weights, it delegates factual storage to a dedicated, sparse associative memory core.
- Rather than relying on quadratic Softmax attention that bloats memory with sequence length, it adopts linear recurrent memory with constant state size.
- Rather than executing energy-intensive FP16 matrix multiplications, it uses 1.58-bit ternary weights (${-1, 0, +1}$), replacing multiplications with integer additions.
The result is a model designed to run a complete 256,000-token context window locally on an accessible consumer GPU (such as an NVIDIA GeForce RTX 2060 6 GB) using under 2.00 GB of VRAM.
Key Architectural Innovations
1. 1.58-Bit Ternary Weights (BitNet b1.58 + FWHT)
Hetman-2.0B constrains linear projection weights strictly to ternary values ${-1, 0, +1}$. This replaces standard floating-point GEMM operations with lightweight integer addition and subtraction inside GPU registers. To prevent activation outliers and preserve signal variance across deep layers, activations pass through online orthogonal Fast Walsh-Hadamard Transforms (FWHT, $B=512$), bounding quantization loss to $< 1.8%$ compared to full FP32 baselines.
2. Hybrid 256k Context Engine (Gated DeltaNet + Global FlashAttention)
To break the quadratic memory barrier of standard transformers:
- 28 Gated DeltaNet Recurrent Layers: Maintain a constant $O(1)$ recurrent memory state (just 3.67 MB across all 28 layers combined), allowing infinite context streaming without memory growth.
- 6 Interleaved Global FlashAttention Anchor Layers: Positioned periodically with Grouped-Query Attention (12:2 GQA) and bounded QK-Norm ($|S_{ij}| \le 11.31$) to anchor associative recall and eliminate attention dilution.
3. Sparse Associative Memory (SLH-Core v2)
Factual knowledge retrieval is offloaded from dense FFN layers to the Speculative Latent-Hopfield Core (SLH-Core). Containing 16,777,216 discrete attractor slots across 16 semantic groups, it performs sparse Top-32 fact retrieval via 2D bilinear contractions. Micro-facts are projected into the model dimension via an isometry matrix $W_{\text{expand}}$, reducing memory traffic by 75% down to 4 KB per token. An auxiliary-loss-free PID dynamic bias controller prevents slot starvation without interfering with task gradients.
4. Native Speculative Drafting (Multi-Token Prediction)
Hetman incorporates an auxiliary 2-token speculative prediction head trained concurrently with the backbone. When served in the companion C++ runtime Rada.cpp, it enables multi-token drafting and verification with $O(1)$ recurrent shadow-state rollback, driving generation throughput above 350 tok/s on consumer GPUs.
5. Cyrillic & Code First-Class Tokenizer
The model utilizes a custom 65,536-token Byte-Level BPE tokenizer with deterministic Ukrainian apostrophe normalization (ʼ, ', ’). It achieves an exceptional fertility of $\le 1.18$ tokens/word on Ukrainian text (compared to 2.5–3.2 in standard LLaMA tokenizers), saving over 30% compute on multilingual tasks.
Model Architecture Specifications
| Parameter | Value | Hardware Alignment Rationale |
|---|---|---|
| Total Parameters (Storage) | 2,137,522,176 (~2.14B) | Fully packs within ~500 MB in 2-bit format |
| Active Parameters per Token | Sub-linear Top-32 sparse Hopfield routing | |
| Hidden Size ($d_{\text{model}}$) | 1536 | Systolic tile alignment ($12 \times 128$) |
| Intermediate Size (FFN) | 6144 ($4.0\times$) | Blockwise FWHT alignment ($12 \times 512$) |
| Total Layers | 34 | 28 Gated DeltaNet + 6 Global Attention |
| Recurrent Head Dimension | 64 | 16 heads $\times$ 64 = 1024 state dimension |
| Global Attention Heads / KV | 12 / 2 (GQA) | 6:1 KV-cache compression ratio |
| Hopfield Memory Slots | 16,777,216 | 16 groups $\times$ 1024 dual centroids $\times$ 32 trits |
| Context Window Length | 262,144 tokens (256k) | Linear recurrent $O(1)$ memory state |
| Vocabulary Size | 65,536 tokens | Byte-Level BPE with unified apostrophe handling |
Development Status & Pre-Training Roadmap
Hetman-2.0B is an independent, open-source research initiative developed from the ground up for community access.
Currently, the complete architectural codebase, mathematical verification suites, production 65k tokenizer, distributed training autopilot, and native C++ runtime are fully finished, verified, and released. The final model weights are currently not yet trained due to cloud compute budget limitations; the project is actively awaiting approval and allocation of compute grants to carry out the planned 855-billion-token pre-training curriculum on Cloud TPU v5e-128 accelerator pods.
As soon as compute resources are secured and pre-training completes, all production checkpoints (model.safetensors and single-file .rada containers) will be released directly to this repository under the Apache 2.0 license.
License & Citation
This project is licensed under the permissive Apache License 2.0.
@software{samchuk2026hetman,
author = {Samchuk, Yaroslav},
title = {Hetman-2.0B: High-Efficiency Edge-Native Foundation Model},
year = {2026},
publisher = {Hugging Face / GitHub},
url = {https://github.com/YaroslavSamchuk/Hetman-2b}
}
Repositories & Resources
- Hetman-2.0B GitHub: github.com/YaroslavSamchuk/Hetman-2b
- Rada.cpp GitHub: github.com/YaroslavSamchuk/Rada.cpp
- Hugging Face Model Hub: huggingface.co/YSamchuk/Hetman-2b
- Downloads last month
- 68