card: thresholded-ternary best 2.1469 (per-group scale, zero state)

b0ac489 verified 5 days ago

3.4 kB

license: mit
language:
  - en
  - hu
library_name: mlx
tags:
  - bitnet
  - b1.58
  - ternary
  - quantization
  - mlx
  - apple-silicon
  - llm
  - sovereign
  - constellation
pipeline_tag: text-generation
datasets:
  - PeetPedro/ultrawhale-dogfood
base_model: Qwen/Qwen2.5-0.5B
model_creator: peterlodri-sec
quant_method: ternary

quantal-ternary

A BitNet b1.58 ternary model — Qwen/Qwen2.5-0.5B, continued-trained and quantized to {-1, 0, +1} weights. Exported as 168 ayeOS ternary matrices (24 layers × 7 tensors) plus the runtime assets (token embeddings + RMSNorm vectors). Part of the vaked constellation — the "cogito" that runs offline in Rust.

Model


Base model	`Qwen/Qwen2.5-0.5B`
Quantization	weight-quant-only ternary `{-1,0,+1}` (group size 64)
Matrices	168 ayeOS per-layer files (`m000.json` … `m167.json`)
Runtime assets	`embeddings.f16` `[151936, 896]`, `norms.f32` `[49, 896]`
Training	continued-train, masked CE, deployed-forward QAT
Validation (masked, n=90)	2.1469
Checkpoint sha256	`21294c68...8285` (best, epoch 2 / early stop)

Training

Data: 20,007 text samples (konstellation corpus: kompress + domain + c3), stratified 90-sample held-out val.
Loss: masked cross-entropy — pad tokens (id 0) weighted out, honest mean over valid tokens. Dynamic per-batch padding bucketed to multiples of 64.
Optimizer: AdamW, weight decay 0.1, grad clip off, lr 3e-4 → cosine → 3e-5 (2% warmup), early stop patience 5 / min-delta 0.02, 40-epoch cap.
Hardware: vast.ai RTX PRO 6000 (96 GB) — the nightly run, B12/256, 20,007 samples. MLX_CUDA_GRAPH_CACHE_SIZE=2000.
Quantizer: true thresholded ternary — weight_quant uses a per-group (64) scale and a real zero state (|w| < 0.5·scale → 0), so ~30% of the weights land on zero. The old sign-based collapse wasted the third state; this one earns it. Training forward ≡ export ≡ Rust (per-group scale, same dequant (code−1)·scale).
Forward: deployed-forward QAT — the exact forward the Rust runner uses (weight-quant-only BitLinear, per-projection RMSNorm + activation quant skipped), so training ≡ inference. The Rust runner reproduces the forward to 1.3e-5 (golden-logits gate, both prompts, identical argmax 71703).
Val trajectory (nightly run): 2.38 → 2.1469 (epoch 2, best) → 2.25 → 2.49 → 2.72 → 2.76 → 3.07 (overfit after epoch 2; early stop). Prior runs: 1.6998 (7,000 samples, L40), 3.2862 (2,785, 3090); the old artifact measured 11.34.

Layout

m000.json … m167.json   168 ternary matrices (packed codes + per-group scales)
index.json              capsule metadata + file manifest (sha256, shapes)
embeddings.f16          token embedding matrix, BF16→FP16, [151936, 896]
norms.f32               49 RMSNorm gain vectors (24×2 + final), [49, 896]

norms.f32 row ordering: row 2i = layer i input_layernorm, row 2i+1 = layer i post_attention_layernorm, row 48 = final model.norm.weight.

Runtime

Consumed by the entheai Rust ternary runner (crates/ternary) and the pocoo.vaked.dev/demos/quantal live viewer. The same export tooling (export_quantal_checkpoint.py + export_quantal_assets.py) produced this repo from the winning checkpoint.

Part of the vaked constellation — sovereign, offline, fine.