Quazim0t0/Wheeler-DeWitt-62M

⚠️ Research artifact. At 62.9M parameters this model has essentially no factual/world knowledge. It learned the form of fluent English and of a chat turn, and it can do small‑number arithmetic, but it hallucinates confidently and does not reliably follow instructions. It is interesting for its architecture, not its accuracy. 2B Tokens Trained on a Consumer 3060 GPU.

Wheeler–DeWitt‑62M

A 62.9M‑parameter research language model whose per‑layer channel mixer is the Wheeler–DeWitt equation of canonical quantum gravity, with a fractal (Cantor‑set) RoPE frequency spectrum.

Wheeler–DeWitt‑62M replaces the usual transformer MLP with a WheelerDeWittBlock: each token’s hidden state is projected onto K = 64 minisuperspace modes which are then evolved by a leapfrog integration of the Wheeler–DeWitt wave equation (a Klein–Gordon equation on superspace under a Lorentzian DeWitt supermetric) rather than a feed‑forward flow. "Time" is the volume/scale mode (the timelike direction of the supermetric), and a Hamiltonian‑constraint auxiliary loss ⟨H²⟩ pressures every layer toward the physical surface H Ψ = 0.

This repository hosts two checkpoints:

File	What it is	Steps
wheeler-dewitt-62m-base-1M.pt	Base pretrained model	1,000,000
wheeler-dewitt-62m-sft.pt	Chat SFT (assistant‑masked)	5,000

---

Architecture

Component	Detail
Parameters	62.9M
Layers × heads × d_model	10 × 12 × 768
Context length	1024 train / 4096 RoPE positions
Vocab	16,512 (SpikeWhale byte‑level tokenizer, ChatML specials)
Channel mixer	WheelerDeWittBlock — 64 minisuperspace modes, Lorentzian DeWitt supermetric, 4 leapfrog wave steps, learnable lapse, ⟨H²⟩ constraint (weight 0.05)
Attention	MLA‑style low‑rank Q/O projections (rank 384), partial RoPE (32 rotated + 32 no‑position dims per 64‑dim head), QK‑Norm, GQA (12 query / 4 KV heads)
Positional encoding	Fractal RoPE — RoPE frequencies placed on the middle‑thirds Cantor set (γ = 1.0) instead of the geometric ladder
Attention extras	Elo/Bradley‑Terry key‑rating bias (persistent through the KV cache)
KV compression	TriAttention (trigonometric, pre‑RoPE Q/K concentration) — inference‑time, budget 2048, scorer auto‑bound to the fractal spectrum

Trait modules (all active)

• HRM — top‑of‑trunk iterative gated refinement.
• MoE — SwiGLU experts (4 routed + 1 shared, top‑2).
• MTP — 4 multi‑token‑prediction draft heads (enables self‑speculative decoding).
• JEPA — train‑time latent‑prediction auxiliary.
• RingSpecialists — 7 per‑ring memory specialists with test‑time‑writable stores.
• FractalSeed — each token’s oscillator phases seeded from its Mandelbrot orbit angles (frozen, gated).

Two fractal ideas, one model

1. Fractal RoPE spectrum (positional): the frequencies live on a Cantor set. Baked into the weights — the model was trained with it from scratch. TriAttention’s importance scorer reads the model’s actual RoPE buffer (bind_omega) so compression stays exact on the non‑geometric spectrum.
2. Fractal phase seed (content): each token’s Wheeler–DeWitt mode phases are seeded from the Mandelbrot iteration z ← z² + c.

---

Training

Base — 1,000,000 steps, from scratch. AdamW (peak lr 3e‑4, 1k warmup, cosine decay to 0.3× floor), grad‑clip 1.0, block 1024, bf16. Streamed 4‑way data blend:

Weight	Source
35%	Ultra‑FineWeb‑L3
25%	FineWeb‑Edu
25%	FineMath
15%	PretrainNew

Plus the Wheeler–DeWitt Hamiltonian‑constraint aux loss (⟨H²⟩, weight 0.05), MTP loss, JEPA loss, and a z‑loss on logits.

SFT — 5,000 steps, assistant‑masked chat. Blended: trivia_qa (general QA) + sciq (science facts) + SmolTalk (chat) + Quazim0t0/Arithmetic, rendered in ChatML with loss on assistant turns only.

The run survived two mid‑training soft‑divergences (~step 67k, ~91k) that were recovered by resuming from the last good checkpoint at a lower learning rate.

---

Evaluation

Base perplexity (step 1M, 2×1024‑token windows)

Corpus	Perplexity
WikiText‑test	14.9
FineWeb (edu)	8.9
FineMath	6.5
Nemotron Formal‑Logic	3.5

For reference, the earlier geometric‑RoPE ancestor plateaued at WikiText ≈ 41.5; the fractal‑RoPE spectrum roughly halved WikiText perplexity over the run.

SFT chat evaluation (greedy, 20 prompts each)

Category	Score	Note
QA (trivia_qa)	0 / 20	Fluent, correctly‑typed, factually wrong — capacity limit
Facts (sciq)	0 / 20	Same; several strict‑match near‑misses ("Sunlight" vs "the sun")
Arithmetic	7 / 20 (35%)	Nails small sums; multi‑digit carry unreliable

The failure profile is the expected one for the scale: format learned, knowledge absent. Arithmetic is the one category with real, method‑sensitive signal (greedy > sampling).

---

Usage

This is custom‑code (not a 🤗 Transformers AutoModel). The modeling code is included in this repo.

import torch
from huggingface_hub import snapshot_download
import sys

path = snapshot_download("Quazim0t0/Wheeler-DeWitt-62M")
sys.path.insert(0, path)
from model import QuazimotoLM, QuazimotoConfig
from spike_tokenizer import SpikeTokenizer

ck = torch.load(f"{path}/wheeler-dewitt-62m-sft.pt", map_location="cpu", weights_only=False)
cfg = QuazimotoConfig(**ck["family_config"])          # rebuilds fractal RoPE + traits
model = QuazimotoLM(cfg); model.load_state_dict(ck["model"], strict=False); model.eval()
tok = SpikeTokenizer(vocab_file=f"{path}/tokenizer.json")

# ChatML — exactly the format SFT was trained on
prompt = ("<|im_start|><|user|>\nWhat is the capital of France?<|im_end|>\n"
          "<|im_start|><|assistant|>\n")
ids = torch.tensor([tok.encode(prompt, add_special_tokens=False)])
out = model.generate(ids, n_new=40, temperature=0.7, top_k=20)
print(tok.decode(out[0, ids.shape[1]:].tolist(), skip_special_tokens=True))

A live demo (chat + a token‑by‑token Wheeler–DeWitt dynamics dashboard: the emergent‑time mode Ψ₀, the constraint residual |H|, and the K minisuperspace modes) is at Quazim0t0/Wheeler-Chat.

---

Intended use & limitations

Intended: research into physics‑inspired sequence mixers, fractal positional encodings, and small‑model behavior; architecture study; the Wheeler–DeWitt dynamics visualization.

Not intended: any factual, advisory, or production use. It will state false things fluently. It has no reliable knowledge, cannot do multi‑digit arithmetic, and does not robustly follow instructions.

Biases/safety: trained on filtered web text; standard web‑corpus biases apply. No alignment/RLHF safety tuning beyond light chat SFT.

Citation

@misc{wheeler-dewitt-62m,
  title  = {Wheeler–DeWitt‑62M: a quantum‑gravity channel mixer with a fractal RoPE spectrum},
  author = {Quazim0t0},
  year   = {2026},
  url    = {https://huggingface.co/Quazim0t0/Wheeler-DeWitt-62M}
}