| --- |
| license: mit |
| language: |
| - en |
| - hu |
| library_name: mlx |
| tags: |
| - bitnet |
| - b1.58 |
| - ternary |
| - quantization |
| - mlx |
| - apple-silicon |
| - llm |
| - sovereign |
| - constellation |
| pipeline_tag: text-generation |
| datasets: |
| - PeetPedro/ultrawhale-dogfood |
| base_model: Qwen/Qwen2.5-0.5B |
| model_creator: peterlodri-sec |
| quant_method: ternary |
| --- |
| |
| # quantal-ternary |
|
|
| A **BitNet b1.58** ternary model — Qwen/Qwen2.5-0.5B, continued-trained and |
| quantized to **{-1, 0, +1}** weights. Exported as 168 ayeOS ternary matrices |
| (24 layers × 7 tensors) plus the runtime assets (token embeddings + RMSNorm |
| vectors). Part of the vaked constellation — the "cogito" that runs offline in |
| Rust. |
|
|
| ## Model |
|
|
| | | | |
| |---|---| |
| | Base model | `Qwen/Qwen2.5-0.5B` | |
| | Quantization | weight-quant-only ternary `{-1,0,+1}` (group size 64) | |
| | Matrices | 168 ayeOS per-layer files (`m000.json` … `m167.json`) | |
| | Runtime assets | `embeddings.f16` `[151936, 896]`, `norms.f32` `[49, 896]` | |
| | Training | continued-train, masked CE, deployed-forward QAT | |
| | Validation (masked, n=90) | **2.1469** | |
| | Checkpoint sha256 | `21294c68...8285` (best, epoch 2 / early stop) | |
|
|
| ## Training |
|
|
| - **Data**: 20,007 text samples (konstellation corpus: kompress + domain + |
| c3), stratified 90-sample held-out val. |
| - **Loss**: masked cross-entropy — pad tokens (id 0) weighted out, honest mean |
| over valid tokens. Dynamic per-batch padding bucketed to multiples of 64. |
| - **Optimizer**: AdamW, weight decay 0.1, grad clip off, lr 3e-4 → cosine → |
| 3e-5 (2% warmup), early stop patience 5 / min-delta 0.02, 40-epoch cap. |
| - **Hardware**: vast.ai RTX PRO 6000 (96 GB) — the nightly run, B12/256, |
| 20,007 samples. `MLX_CUDA_GRAPH_CACHE_SIZE=2000`. |
| - **Quantizer**: true thresholded ternary — `weight_quant` uses a per-group |
| (64) scale and a real zero state (`|w| < 0.5·scale → 0`), so ~30% of the |
| weights land on zero. The old sign-based collapse wasted the third state; |
| this one earns it. Training forward ≡ export ≡ Rust (per-group scale, same |
| dequant `(code−1)·scale`). |
| - **Forward**: deployed-forward QAT — the exact forward the Rust runner uses |
| (weight-quant-only BitLinear, per-projection RMSNorm + activation quant |
| skipped), so training ≡ inference. The Rust runner reproduces the forward |
| to **1.3e-5** (golden-logits gate, both prompts, identical argmax 71703). |
| - **Val trajectory** (nightly run): 2.38 → **2.1469** (epoch 2, best) → 2.25 |
| → 2.49 → 2.72 → 2.76 → 3.07 (overfit after epoch 2; early stop). Prior |
| runs: 1.6998 (7,000 samples, L40), 3.2862 (2,785, 3090); the old artifact |
| measured 11.34. |
|
|
| ## Layout |
|
|
| ``` |
| m000.json … m167.json 168 ternary matrices (packed codes + per-group scales) |
| index.json capsule metadata + file manifest (sha256, shapes) |
| embeddings.f16 token embedding matrix, BF16→FP16, [151936, 896] |
| norms.f32 49 RMSNorm gain vectors (24×2 + final), [49, 896] |
| ``` |
|
|
| `norms.f32` row ordering: row `2i` = layer `i` input_layernorm, row `2i+1` = |
| layer `i` post_attention_layernorm, row 48 = final `model.norm.weight`. |
| |
| ## Runtime |
| |
| Consumed by the entheai Rust ternary runner (`crates/ternary`) and the |
| `pocoo.vaked.dev/demos/quantal` live viewer. The same export tooling |
| (`export_quantal_checkpoint.py` + `export_quantal_assets.py`) produced this |
| repo from the winning checkpoint. |
| |
| > Part of the vaked constellation — sovereign, offline, fine. |
| |
