MonumentalSystems/harmonic-gpt-spherical-kan-128m

Harmonic GPT Spherical KAN 128M

An experimental 128.26M-parameter causal byte language model from the Harmonic GPT geometry A/B. This revision is a deliberately benchmark-specialized derivative of the step-52,500 spherical pretraining checkpoint (validation BPB 1.1740).

Evaluation contamination: the post-training set directly transcribes 24 public prompts from evals.blah.dev, with 28 assistant targets across multi-turn cases. Scores on those prompts are training-set measurements and must not be presented as evidence of generalization. Use an earlier Hub revision for the clean base model.

The model uses a fixed QPSK + Hamming wire encoder, 18 unified blocks, seven heads, eight Funk-Hecke/SpectralKAN channels per head, bounded decayed recurrent state, and projection-free spherical Lohe mean-field coupling. Its hidden width is 1,330 (seven oscillator groups of 190 dimensions). The minimal readout exposes raw oscillator state, alignment, and order-parameter observables to a 256-byte head.

Checkpoint and data

• Parameters: 128,260,556
• Pretraining exposure: 3,440,640,000 bytes
• Context used in training: 2,048 bytes
• Optimizer: split-decay RotationalAdamW
• Dropout: 0
• Training stream: ordered document-corpus-v3
• Checkpoint precision in this repository: BF16
• Specialist updates: 1,550 full public-eval sweeps
• Specialist trainable scope: final normalization + byte head only (346,724 parameters)
• EOS objective: null byte, 4x primary-CE weight
• Public training-set audit: 18/28 greedy exact, 28/28 null-stopped, 92.07% teacher-forced byte accuracy

The demo formats chat turns as Q: ...\nA:. Byte value zero is an explicitly trained end-of-response signal and is never emitted as visible text.

Native bounded-state inference

modeling_spherical_kan.py is a standalone inference implementation. Its step() path preserves the trained SpectralKAN recurrence and spherical Lohe update while retaining O(1) state with respect to generated sequence length. Packaged BF16 logits were checked against the original checkpoint's BF16 step() path with maximum absolute delta 0.0.

Important limitations

This is a contaminated research artifact, not a reliable general assistant. It can produce plausible but incorrect, repetitive, malformed, or nonsensical text. Do not use it for medical, legal, financial, safety-critical, or production decisions.

The model is released under CC BY-NC 4.0 for research use. The training corpus is an internally assembled, filtered, globally deduplicated research mixture of curated, reference, long-form, mathematical, reasoning, scientific/code, and document-QA sources; downstream users remain responsible for reviewing source-specific terms.

Research context

The checkpoint is the spherical geometry sibling of the modern Transformer and planar Givens controls. Harmonic GPT treats benchmark scores as diagnostics of self-organizing oscillator computation rather than as claims of production-model competitiveness.