Kthulhu-0.1-150M (RU + EN + Thinking + Code + Jev) — lgpt-rs Native Rust Model

Note: kthulhu/Kthulhu-0.1-150M is a compact (151.2M parameters) bilingual (Russian + English + Code + Chain-of-Thought) causal language model trained 100% from scratch (Gpt::init_with_seed) in pure Rust + CUDA using the lgpt-rs framework on a single NVIDIA B300 SXM6 AC GPU — with zero Python or PyTorch in the training or inference runtime.

---

1. Overview & Highlights

kthulhu/Kthulhu-0.1-150M is purpose-built for high-density bilingual (RU + EN) generation, System-2 reasoning (<think>...</think>), and System-1 calibrated judgment (Jev) at the ~150M parameter scale:

• Custom 32,768 Bilingual ByteLevel BPE Tokenizer (50/50 RU + EN):
  • Trained from scratch on balanced Russian (rus_Cyrl) and English (FineWeb-Edu + code) corpora so Cyrillic words tokenize with the same fertility and efficiency as English, while keeping the embedding table compact (32,768 × 768 = 25.2M params vs 151,936 × 768 = 116.7M params), dedicating >83% of model weights to the 16 Transformer blocks.
• Trained on 1× NVIDIA B300 SXM6 AC (lgpt-rs Pure-Rust Engine):
  • Pretraining (6,000 steps, 786.4M tokens) at 150,650 tok/s (0.870 s/step, BATCH=64, SEQ_LEN=2048).
  • Full-Parameter SFT (1,000 steps, 262.1M tokens) at 152,580 tok/s (1.718 s/step, BATCH=64, GRAD_ACCUM=2, SEQ_LEN=2048).
• Dual System-1 (Jev) + System-2 (<think>) Capabilities:
  • Supports both explicit Chain-of-Thought reasoning (<think>...</think>) and single-pass non-autoregressive System-1 calibrated judgment (Jev) outputting tri-state truth/abstention probabilities (noul), order-invariant candidate choice distributions (choice), and continuous expected rubric scores $\mathbb{E}[S] \pm \sigma_S$ (score) in 1 forward pass (0 generated tokens).
• Ready for LM Studio / llama.cpp (Q8_0 & BF16 GGUF v3):
  • Ships with both Kthulhu-0.1-150M-q8_0.gguf (155 MiB) and Kthulhu-0.1-150M-bf16.gguf (290 MiB), complete with embedded tokenizer.ggml.* vocabulary, control token types, F32 1D RMSNorm scales, and built-in ChatML tokenizer.chat_template.

---

2. Architecture (Kthulhu-0.1-150M / qwen3 GGUF arch + System-1 Jev Head)

Parameter	Value
Model Name	Kthulhu-0.1-150M (Qwen3ForCausalLM / qwen3 GGUF architecture)
Total Parameters	151,192,320 (~151.2M)
Layers (n_layer)	16
Hidden Size (n_embd)	768
Intermediate Size (SwiGLU)	2,048
Attention Heads	12 Query / 4 KV (GQA 3:1, head_dim = 64)
Attention Normalization	Per-head QK RMSNorm (norm_qk = true, eps = 1e-6)
Vocabulary Size	32,768 (`<
Context Length	8,192 (rope_theta = 1,000,000)
System-2 Reasoning	Explicit <think>...</think> chain-of-thought traces
System-1 Jev Head	Single-pass non-autoregressive calibrated judgment (noul, choice, score)

---

3. System-1 Calibrated Jev Head (LLM-as-Jev / OpenJev)

Traditional LLM-as-a-Judge pipelines sample free-form text tokens and regex-parse "Yes" / "No" or "Score: 4/5". That approach is slow (dozens of autoregressive decode steps), suffers from format/regex failures, exhibits strong positional bias when ranking options A, B, C, and produces uncalibrated discrete outputs (4 instead of 4.18 ± 0.62).

Kthulhu-0.1-150M and lgpt-rs implement a System-1 calibrated Jev head (crates/model/src/jev.rs, configured via jev_config in config.json) that extracts three structured probability distributions directly from the final normalized hidden state $\mathbf{h}_T = \text{RMSNorm}(\mathbf{x}_T)$ in a single forward pass (0 generated tokens):

3.1. The Three Jev Structured Outputs (JevOutput)

1. noul (JevNoul — Ternary Verification with Explicit Epistemic Abstention [Yes, No, Null]):

   • Extracts raw logits $\mathbf{z}{\text{noul}} = [z{\text{yes}}, z_{\text{no}}, z_{\text{null}}]$ from anchor slots ([20680 ("Yes"), 11096 ("No"), 36 ("?")]) and applies temperature-calibrated softmax: $$p_i = \frac{\exp(z_i / T_{\text{noul}})}{\sum_{j \in {\text{yes, no, null}}} \exp(z_j / T_{\text{noul}})}$$
   • Returns:
     • p_yes, p_no, p_null: Calibrated probabilities summing to 1.0.
     • log_odds_yes_no: $\ln(p_{\text{yes}} / p_{\text{no}}) = (z_{\text{yes}} - z_{\text{no}}) / T_{\text{noul}}$ for threshold-free ROC/PR routing.
     • entropy_bits: Shannon entropy $H = -\sum_i p_i \log_2 p_i \in [0, 1.585]$ bits.
     • decision: Yes, No, or Null (explicit abstention when p_null >= max(p_yes, p_no)).

2. choice (JevChoice — Categorical Slot Selection & Permutation-Invariant Candidate Scoring):

   • Single-Pass Slot Mode (0 extra passes): Reads logits for discrete option tokens A..H (choice_slots: [38, 39, 40, 41, 42, 43, 44, 45]) and returns probs, best_index, confidence, and entropy_bits.
   • Order-Invariant Candidate Mode (score_jev_choices): When free-text candidate continuations $C_1, \dots, C_K$ are passed (--choice "..."), lgpt-rs scores each continuation independently against the prompt using length-normalized conditional log-likelihood: $$s(C_k \mid \text{prompt}) = \frac{1}{|C_k|} \sum_{t=1}^{|C_k|} \ln P(c_{k,t} \mid \text{prompt}, c_{k,<t})$$ Because candidates $C_1, \dots, C_K$ never appear together inside the prompt, the resulting distribution is strictly permutation-equivariant (zero order/position bias).

3. score (JevScore — Continuous Expected Rubric Value $\mathbb{E}[S] \pm \sigma_S$):

   • Reads logits at ordinal rubric anchors $b_k \in {1.0, 2.0, 3.0, 4.0, 5.0}$ (score_slots: [[22, 1.0], [23, 2.0], [24, 3.0], [25, 4.0], [26, 5.0]]), computes calibrated bucket probabilities $p_k = \text{softmax}(z_k / T_{\text{score}})$, and returns: $$\mathbb{E}[S] = \sum_{k=1}^5 b_k , p_k \in [1.0, 5.0], \qquad \sigma_S = \sqrt{\sum_{k=1}^5 \bigl(b_k - \mathbb{E}[S]\bigr)^2 p_k}$$
   • Provides a smooth, differentiable scalar reward / quality signal $\mathbb{E}[S]$ alongside epistemic spread $\sigma_S$ and entropy_bits.

3.2. Example Jev Output (lgpt jev --json / POST /v1/jev)

{
  "noul": {
    "decision": "No",
    "p_yes": 0.3988,
    "p_no": 0.5794,
    "p_null": 0.0218,
    "log_odds_yes_no": -0.3735,
    "entropy_bits": 1.1241
  },
  "choice": {
    "best_index": 1,
    "confidence": 0.6142,
    "entropy_bits": 0.9620,
    "probs": [0.3858, 0.6142]
  },
  "score": {
    "expected": 1.8082,
    "std_dev": 1.1640,
    "entropy_bits": 1.8415,
    "buckets": [1.0, 2.0, 3.0, 4.0, 5.0],
    "probs": [0.5812, 0.1944, 0.1021, 0.0800, 0.0423]
  }
}

---

4. Deep Dive: Burn / CubeCL / CubeK & Custom CUDA Kernel Optimizations (With Numbers)

Out of the box, stock Burn 0.22 / CubeCL 0.11 / CubeK 0.3 achieved only 2,214 tok/s (1,850 ms/step, 7.3% of PyTorch) when training a ~160M parameter transformer at seq_len = 1024, batch = 4 (bf16) on an RTX 3080 Ti. Through systematic profiling and kernel engineering across vendor/burn, vendor/cubek, vendor/cubecl, and crates/kernels, we achieved a 17.3× end-to-end training speedup (38,259 tok/s peak / 37,130 tok/s median = 125.4% of PyTorch efficient SDPA and 162.5% of PyTorch cuDNN SDPA) while cutting peak VRAM by 34%, plus 234%–351% of PyTorch in autoregressive token decoding.

4.1. Key Engine Optimizations Behind the Speedup

1. Single-Pass Tensor-Core FlashAttention-2 Forward & Backward (78.7×–102.8× Kernel Speedup)

   • The Problem: In stock burn-cubecl / cubek-attention, the default accelerated routine exceeded static shared-memory limits on consumer GPUs and fell back to an $O(S^2)$ 15-kernel chain materializing the full [B, H, S, S] matrix in global VRAM, while the stock backward pass took 111.0 ms per call.
   • The Solution: lgpt-rs integrates a proprietary single-pass Tensor-Core FlashAttention-2 forward and backward engine in crates/kernels that maintains attention scores on-chip with zero $O(T^2)$ intermediate buffers.
   • The Numbers (b=4, h=12, s=1024, d=64, bf16, causal):
     • Stock cubek-attention backward: 111.0 ms
     • lgpt-rs SIMT tiled backward: 16.19 ms (6.86× faster)
     • lgpt-rs Tensor-Core backward: 1.41 ms median / 1.08 ms min (78.7× median / 102.8× peak speedup).

2. FusedAdamW + Device-Side Gradient Norm Clipping & Zero-Copy Transposed Gradients (30.5× Optimizer Speedup)

   • The Problem: Standard Burn optimizer steps launch separate elementwise kernels for L2 norm squaring, host synchronization, gradient scaling, weight decay, momentum, variance, and parameter update — taking 195.18 ms/step on ~160M parameters.
   • The Solution: lgpt-rs fuses gradient norm accumulation, clipping, weight decay, and moment updates on device with native support for transposed 2D gradients.
   • The Numbers:
     • Optimizer step time (opt_ms) dropped from 195.18 ms $\to$ 6.4–7.2 ms (27.3×–30.5× faster).

3. Zero-Scratch Online Cross-Entropy with Recompute (0 Scratch Buffers & 64-Bit Indexing)

   • The Problem: Materializing [B·T, V] f32 logits and softmax probabilities consumes gigabytes of VRAM (>2.5 GiB at B·T = 8,192).
   • The Solution: lgpt-rs computes online softmax loss and native-precision backward gradients on the fly with 0 f32 scratch buffers, native prompt masking (ignore_index = -100), and 64-bit indexing support (FastDivmod PR #1859) for large-batch datacenter runs.

4. Fused Normalization (RMSNorm / LayerNorm), SwiGLU, Exact erf GELU, and Precomputed GPU RoPE

   • Fused Norms & Residuals: High-occupancy normalization forward and backward passes with fused residual stream addition.
   • Fused SwiGLU: Single-pass gated MLP forward and backward execution eliminating intermediate activation round-trips.
   • Exact erf gelu_backward (PR #6114): Fixed upstream Burn's ActivationOps::gelu_backward (reducing max gradient error vs PyTorch from 3.67e-4 to 1.19e-7 machine epsilon) and fused multi-kernel activation chains into single vectorized passes.
   • Precomputed GPU RoPE: Precomputes rotary tables on GPU once at model initialization (0 PCIe transfers per step) and fuses QKV unpacking, QK-normalization, and rotary embedding.

5. Dual Dense GEMM Engine (cuBLAS + Pure-Rust Native Tensor-Core / GEMV) & Weight-Only Quantization

   • Selectable via --gemm-backend cublas|native (LGPT_GEMM=cublas|native), combining stream-bound vendor Tensor-Core GEMM with a pure-Rust native Tensor-Core GEMM and low-batch GEMV decode fast-path, plus weight-only quantized matmul (Int8, Int4, Nf4, Fp4, NvFp4, HadamardFp4).

---

4.2. Step-by-Step Training Acceleration Table (RTX 3080 Ti, b=4, s=1024, bf16)

Optimization Stage	Median tok/s	Peak tok/s	step_ms	fwd / bwd / opt (ms)	Pool In-Use / Reserved (MiB)	Speedup vs Stock Burn	% of PyTorch (eff 30,520 / cuDNN 23,550)
0. Stock Burn 0.22 / CubeK Baseline	2,214	2,364	1,850.0	—	3,984 / 6,623	1.00×	7.3% / 9.4%
1. Phase A: Tiled WMMA 16×16×16 Flash Bwd + LSE	13,782	13,915	297.2	—	3,984 / 6,623	6.22×	45.2% / 58.5%
2. Phase B + C.1: Stream cuBLAS + Fused Online CE	14,346	14,358	285.5	—	3,582 / 5,923	6.48×	47.0% / 60.9%
3. Phase C.2: + Fused RMSNorm/LN, GELU/SiLU, RoPE	19,827	19,839	206.6	118.3 / 89.3 / 195.2	2,633 / 5,059	8.96×	65.0% / 84.2%
4. Phase C.3: + FusedAdamW (195 ms → 8.7 ms)	23,036	23,115	177.8	113.2 / 83.4 / 8.7	2,633 / 5,059	10.40×	75.5% / 97.8%
5. Phase C.4: + Single-Pass WMMA Flash Fwd+LSE	34,295	34,603	119.4	43.2 / 78.7 / 7.9	2,633 / 4,530	15.49×	112.4% / 145.6%
6. Phase F: + 8-Warp LSE-Cached CE, 2D Norms, SwiGLU, grad_norm_sq	37,130	38,259	107.1	39.2 / 69.4 / 8.8	2,633 / 4,683	16.77× (17.28× peak)	121.7% / 157.7% (125.4% / 162.5% peak)
7. Kthulhu-0.1-150M (RTX 3080 Ti, b=4, s=1024)	34,294	34,468	115.5	39.7 / 69.2 / 6.4	2,902 / 4,831	—	—
8. Kthulhu-0.1-150M (NVIDIA B300 SXM6 AC, b=64, s=2048)	150,650	152,580	870.0	—	82,646 (B=64, S=2048)	—	—

4.3. Training & Autoregressive Decode vs PyTorch (RTX 3080 Ti, bf16)

Benchmark Workload	lgpt-rs (Pure Rust + CUDA)	PyTorch efficient SDPA	PyTorch cuDNN SDPA	lgpt-rs vs PyTorch (eff / cuDNN)	VRAM In-Use (lgpt-rs vs PyTorch)
Train pythia-160m (s=1024, b=4)	37,130 tok/s (peak 38,259)	30,520 tok/s	23,550 tok/s	121.7% / 157.7%	2,633 MiB vs 3,103–3,663 MiB (-15%..-28%)
Train pythia-160m (s=2048, b=4)	30,022 tok/s (peak 30,246)	29,432 tok/s	23,310 tok/s	102.0% / 128.8%	4,337 MiB vs 5,500–6,620 MiB (-21%..-34%)
Decode pythia-160m (B=1, prompt=128, new=128)	104.5 tok/s (prefill 9.0 ms)	44.7 tok/s (20.6 ms)	37.6 tok/s (25.4 ms)	233.7% / 278.3%	330 MiB vs 349 MiB
Decode pythia-160m (B=8, prompt=128, new=128)	553.5 tok/s (prefill 8.9 ms)	332.3 tok/s (22.6 ms)	281.9 tok/s (29.4 ms)	166.6% / 196.4%	480 MiB vs 500 MiB
Decode Qwen2.5-0.5B (B=1, prompt=128, new=128)	42.1 tok/s (prefill 21.8 ms)	15.1 tok/s (71.6 ms)	12.0 tok/s (87.4 ms)	279.2% / 350.6%	1,349 MiB vs 1,257 MiB
Decode Qwen2.5-0.5B (B=8, prompt=128, new=128)	193.8 tok/s (prefill 29.1 ms)	92.7 tok/s (93.2 ms)	113.7 tok/s (103.6 ms)	209.1% / 170.4%	1,373 MiB vs 1,539 MiB

---

5. Training Pipeline & Metrics (NVIDIA B300 SXM6 AC)

Trained end-to-end from random initialization (seed = 1337) on 1× NVIDIA B300 SXM6 AC:

1. Stage 1 — Bilingual Pretraining (lgpt pretrain):
   • Data: Binary LGPTSHRD token shards (~192M unique tokens across 6 shards from Russian FineWeb-2 rus_Cyrl, English FineWeb-Edu, Stack-Edu Rust/Python code, and bilingual reasoning traces).
   • Schedule: 6,000 optimizer steps, seq_len = 2048, micro_batch = 64, grad_accum = 1 (131,072 tokens/step = 786.4M tokens trained), peak lr = 6e-4 cosine schedule (warmup = 300 steps), bf16 compute with fused FlashAttention-2 and fused online-softmax cross-entropy (--recompute-loss).
   • Throughput: 150,650 tok/s (0.870 s/step).
   • Loss Progression:
     • Step 1: ~10.40
     • Step 1,000: 3.92
     • Step 2,000: 3.58
     • Step 4,000: 3.3924
     • Step 6,000: 3.2702
2. Stage 2 — Full-Parameter Bilingual SFT (lgpt finetune --lora-r 0):
   • Data: sft_en_ru_grandmaster.jsonl (345 MB curated Russian + English instruction, coding, dialogue, and <think> reasoning mixture: GrandMaster-PRO-MAX, saiga_scored, s1K-1.1, Bespoke-Stratos-17k, OpenR1-Math-ru).
   • Schedule: 1,000 optimizer steps, seq_len = 2048, micro_batch = 64, grad_accum = 2 (262,144 tokens/step = 262.1M SFT tokens), hardware prompt masking (ignore_index = -100), peak lr = 5e-5.
   • Throughput: 152,580 tok/s (1.718 s/step).
   • Final SFT Loss: 2.6997 – 2.7203.

---

6. Repository Contents

File	Size	Description
model.safetensors	576.8 MiB	Full Hugging Face weights (F32/BF16)
Kthulhu-0.1-150M-q8_0.gguf	154.7 MiB	Quantized Q8_0 GGUF v3 (LM Studio / llama.cpp ready)
Kthulhu-0.1-150M-bf16.gguf	289.8 MiB	Full-precision BF16 GGUF v3 (with F32 1D RMSNorm weights for llama.cpp)
config.json	1.1 KiB	Model & jev_config architecture definition
generation_config.json	0.2 KiB	Default sampling parameters (bos=1, eos=2, pad=3)
tokenizer.json	2.8 MiB	Custom 32,768 RU+EN ByteLevel BPE tokenizer
tokenizer_config.json	1.3 KiB	ChatML template + <think> special tokens
harness_report.md	3.5 KiB	5-stage evaluation report (RU/EN generation, Jev, Embeddings, Perplexity)
harness_report.json	4.2 KiB	Machine-readable evaluation metrics (loss = 3.6977, ppl = 40.36)
bootstrap.py	19.6 KiB	Self-contained CLI runner & 5-stage evaluation harness

---

7. Quickstart

Option A — LM Studio / llama.cpp (GGUF v3)

Download Kthulhu-0.1-150M-q8_0.gguf (or Kthulhu-0.1-150M-bf16.gguf) and load it in LM Studio or llama-server:

llama-server -m Kthulhu-0.1-150M-q8_0.gguf --port 8080 -ngl 99 -c 4096

Chat completion request (/v1/chat/completions, ChatML template is embedded in the GGUF):

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "user", "content": "Привет! Объясни простыми словами, что такое нейронная сеть."}
    ],
    "max_tokens": 200,
    "temperature": 0.3,
    "repeat_penalty": 1.15
  }'

Option B — Native Rust lgpt-rs (bootstrap.py)

# 1. Run the 5-stage evaluation harness (RU/EN generation, Thinking, Jev, Embeddings, Perplexity)
python bootstrap.py kthulhu harness

# 2. Interactive bilingual terminal chat
python bootstrap.py kthulhu chat

# 3. Single-prompt generation
python bootstrap.py kthulhu gen "Привет! Объясни простыми словами, как работает нейронная сеть."

# 4. Single-pass System-1 Jev judgment (1 forward pass, no token generation)
python bootstrap.py kthulhu jev "Земля вращается вокруг Солнца." \
  --choice "Да, это астрономический факт" \
  --choice "Нет, Солнце вращается вокруг Земли"

# 5. Cross-lingual sequence embeddings & cosine similarity
python bootstrap.py kthulhu embed \
  "Обучение языковой модели на видеокарте с использованием CUDA." \
  "Training a language model on a GPU using CUDA kernels." \
  "Приготовление домашнего яблочного пирога."

# 6. Launch OpenAI-compatible HTTP server (/v1/chat/completions, /v1/embeddings, /v1/jev)
python bootstrap.py kthulhu serve --port 8080