README.md

metadata

license: apache-2.0
tags:
  - grt
  - geometric-rotation-transformer
  - causal-lm
  - rotational
  - knowledge-distillation
  - qwen3
  - gguf
datasets:
  - HuggingFaceFW/fineweb-edu
  - HuggingFaceTB/cosmopedia-v2
  - wikimedia/wikipedia
  - HuggingFaceTB/finemath
  - HuggingFaceTB/smollm-corpus
language:
  - en
library_name: transformers
pipeline_tag: text-generation

GRT-30M (24-Hour Qwen 3 Distillation Preview)

A Geometric Rotation Transformer (GRTForCausalLM) distilled from Qwen 3 (sharing the 151,669-token Qwen 3 BPE vocabulary) in a single 24-hour distillation run (10.79B tokens at ~123.9k tok/s across 8x NVIDIA A100-SXM4-80GB). GRT replaces every dense linear projection (nn.Linear) in attention (Q, K, V, O) and feed-forward (expand, contract) with cascades of content-modulated 2×2 Givens rotations on SO(D) interleaved with butterfly permutations.

This is a base research checkpoint distilled from Qwen 3, not an instruction-tuned assistant. It demonstrates that a rotation-only student backbone with just 9.74M non-embedding parameters (29.31M total stored parameters with the 151,669-token Qwen 3 vocabulary):

1. Distills stably from Qwen 3 at 123,900 tok/s to val_loss = 4.4776 (val_ppl = 88.02, 27.2% top-1 token accuracy over a 151,669-token vocabulary).
2. Achieves bit-for-bit 4-way greedy generation parity across PyTorch ROCm (gfx1151), llama.cpp CPU (AVX2), llama.cpp Vulkan (RADV), and llama.cpp ROCm (HIP).
3. Runs at 3,790 tok/s prefill (Vulkan) and 87.4 tok/s decode (CPU, 16 threads) in pure FP32 on an AMD Ryzen AI Max+ 395 (Strix Halo / gfx1151), and 5,760 tok/s forward in BF16 with the fused GRT HIP kernels.
4. Beats random chance across 6 of 7 zero-shot multiple-choice benchmarks (ARC-Easy 29.50%, HellaSwag 29.20%, OpenBookQA 28.40%, CommonsenseQA 21.70%, BoolQ 51.53%, QNLI 51.50%), matching dense models (Pythia-14M / Pythia-31M) of comparable backbone capacity.

---

Model Details

Property	Value
Architecture	GRTForCausalLM (model_type: "grt", grt30m_kd)
Training paradigm	Knowledge distillation from Qwen 3
Total stored parameters	29,305,498 (29.31M)
Non-embedding backbone parameters	9,740,197 (9.74M across 12 blocks)
Dense-equivalent parameters	55,680,446 (55.68M)
hidden_size (d_model)	256
num_hidden_layers (n_blocks)	12
num_attention_heads (n_heads)	4 (head_dim = 64)
intermediate_size (d_ff)	1,024
n_qkv_rotations / n_mixer_rotations / ffn_n_rotations	11 / 1 / 21
Vocabulary	151,669 (Qwen 3 BPE, tied polar cosine head)
Context length (max_position_embeddings)	2,048
Weight files	model.safetensors (116 MB), grt-30m-f32.gguf (340 MB)

Because grt-30m shares the full 151,669-token Qwen 3 vocabulary at hidden_size = 256 for direct token-aligned distillation from Qwen 3, the tied polar embedding table (phase_angles [151669, 128] + log_magnitude [151669]) accounts for 19.57M parameters (66.8% of stored weights), leaving 9.74M stored parameters for the 12-layer rotational transformer backbone.

---

24-Hour Qwen 3 Distillation Summary

Metric	Value
Teacher family / Tokenizer	Qwen 3 (151,669-token Qwen 3 BPE vocabulary)
Student config	grt30m_kd (d_model = 256, 12 rotational blocks, 9.74M non-embedding params)
Training corpus mix	fineweb-edu (45%), cosmopedia-v2 (20%), wikipedia (15%), finemath (12%), stack-edu (8%)
Hardware	8x NVIDIA A100-SXM4-80GB (DDP + Fused CUDA Givens Kernels)
Wall-clock duration	24.0 hours (86,969s)
Tokens trained	10,789,404,672 (10.79B tokens, 20,580 steps)
Global batch size	524,288 tokens/step (micro_batch = 32, grad_accum = 1, seq_len = 2048, 8 GPUs)
Sustained throughput	123,900 tok/s (~15,490 tok/s/GPU, zero NaNs or gradient spikes)
Optimizer	PairAdam (factored second moments for 2D coupling matrices + angular step clamping)
Learning rate	Peak 1e-3, WSD schedule (checkpoint-20000 saved at 20,000 / 57,220 steps)
Final validation loss / PPL (FineWeb-Edu)	val_loss = 4.4776 / val_ppl = 88.02 (top-1 token acc = 27.20%)

---

4-Backend Exact Greedy Generation Parity

model.safetensors (transformers) and grt-30m-f32.gguf (llama.cpp) produce identical token-for-token greedy continuations (temperature = 0, 16 new tokens) across all four supported backends on AMD Strix Halo (gfx1151):

• PyTorch ROCm (gfx1151, fused HIP kernels)
• llama.cpp CPU (x86_64 AVX2)
• llama.cpp Vulkan (RADV gfx1151)
• llama.cpp ROCm (HIP gfx1151)

Prompt	Greedy continuation (16 tokens, identical on all 4 backends)
The capital of France is	The capital of France is the first to be the first to be the first to be the first to be
Once upon a time	Once upon a time, you might be wondering what exactly is the way to do with the right point
In the beginning	In the beginning of the world, the world is the most common form of the world. The

---

Benchmarks

1. Zero-Shot Multiple-Choice & Held-Out Perplexity

Evaluated zero-shot (no task fine-tuning) via teacher-forced length-normalized log-likelihood scoring on checkpoint-20000:

Benchmark	Split (n)	GRT-30M	Random Baseline	Delta vs. Random
ARC-Easy (allenai/ai2_arc)	test (2,376)	29.50%	25.00%	+4.50%
HellaSwag (Rowan/hellaswag)	val (1,000)	29.20%	25.00%	+4.20%
OpenBookQA (allenai/openbookqa)	test (500)	28.40%	25.00%	+3.40%
CommonsenseQA (tau/commonsense_qa)	val (1,221)	21.70%	20.00%	+1.70%
BoolQ (google/boolq)	val (3,270)	51.53%	50.00%	+1.53%
QNLI (nyu-mll/glue)	val (1,000)	51.50%	50.00%	+1.50%
ARC-Challenge (allenai/ai2_arc)	test (1,172)	21.08%	25.00%	-3.92%

Held-Out Language Modeling	Split / Tokens	Loss (NLL)	Perplexity (151k vocab)	Top-1 Token Acc
FineWeb-Edu (In-Domain Val)	val (step 20,580)	4.4776	88.02	27.20%
WikiText-2 (wikitext-2-raw-v1)	test (297,984 toks, T=1024)	5.0305	153.01	25.71%

2. Inference & Kernel Throughput on AMD Strix Halo (Radeon 8060S / gfx1151)

llama.cpp (grt-30m-f32.gguf, FP32)

Backend	Prefill pp128 (tok/s)	Prefill pp512 (tok/s)	Decode tg32 (tok/s)	Decode tg64 (tok/s)
Vulkan (-ngl 99)	1,947.0	3,790.1	49.0	49.9
CPU (-t 16, AVX2)	754.1	905.4	86.5	87.4
ROCm (-ngl 99)	181.9	195.0	37.7	37.5

PyTorch ROCm (gfx1151, BF16, Fused HIP Kernels)

Workload	Batch × Seq (B × T)	Latency (ms)	Throughput (tok/s)
Prefill (B=1, T=512)	1 x 512	129.36	3,957.9
Prefill (B=1, T=2048)	1 x 2048	357.89	5,722.5
Forward (B=4, T=512, Fused HIP)	4 x 512	355.58	5,759.6
Training Step (Fwd + Bwd, Fused HIP)	4 x 512	1,771.68	1,156.0

3. Circular-Pooled Sentence Encoder (MatterWaveAI/grt-30m-embedding)

We also release MatterWaveAI/grt-30m-embedding (GRTModel + SentenceTransformer), adapted from grt-30m using planar-pair circular pooling (pooling_mode="circular", averaging unit directions and pair radii separately across 128 planar (x, y) pairs). On an Intel Arc 140T GPU (xpu:0, fused oneAPI SYCL kernels), the 151,669 lexical embedding.phase_angles were frozen while training the 9,891,865 rotational backbone and log_magnitude parameters with PairAdam in two stages:

Stage	Dataset & Budget	Objective & Hyperparameters
Stage 1: AllNLI Contrastive Alignment	sentence-transformers/all-nli (150 steps, 4,800 triplets, <1% of corpus)	MultipleNegativesRankingLoss + explicit hard negatives (τ = 0.05, B = 32, PairAdam lr = 1.5e-4)
Stage 2: STS-B Rank & Metric Calibration	sentence-transformers/stsb (2 epochs, 358 steps over 5,749 pairs)	CoSENT (τ = 20.0) + 2.0 × CosineMSE (B = 32, PairAdam lr = 8e-5)

Split	Metric	GRT-30M (Pretrained Base)	GRT-30M-Embedding (PairAdam)	AdamW Ablation¹	Delta (PairAdam vs. Base)
STS-B Validation	Spearman (×100)	45.51	71.79	72.89	+26.28
STS-B Test	Spearman (×100)	28.49	62.42	63.17	+33.93
STS-B Test	Pearson (×100)	29.81	63.30	63.72	+33.49

¹ Footnote: Under the identical 508-step schedule and seed (42), standard AdamW (1× coupling LR, full diagonal second moments) reaches 72.89 Val / 63.17 Test Spearman vs. 71.79 / 62.42 for PairAdam (3× coupling LR, factored second moments), suggesting PairAdam's 3× coupling multiplier is primarily beneficial for large-batch pretraining rather than short small-batch (B = 32) fine-tuning.

---

Usage

PyTorch (transformers)

import torch
from transformers import AutoTokenizer, GRTForCausalLM

# Optional: register fused CUDA/ROCm Givens kernels if grt-kernels is installed
try:
    from grt_kernels.torch_ops import ensure_grt_ops
    ensure_grt_ops()
except ImportError:
    pass

model_id = "MatterWaveAI/grt-30m"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = GRTForCausalLM.from_pretrained(model_id, dtype=torch.float32).eval()

prompt = "The history of the city"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids
with torch.no_grad():
    output = model.generate(input_ids, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

llama.cpp (GGUF)

./build/bin/llama-completion \
  -m grt-30m-f32.gguf \
  -p "Once upon a time" \
  -n 32 --temp 0 --seed 0

Verify file integrity against SHA256SUMS:

sha256sum --ignore-missing -c SHA256SUMS

---

Architecture Overview

GRT replaces dense weight matrices (nn.Linear) in both attention and FFN blocks with cascades of learned 2×2 Givens rotations arranged in a butterfly topology. At each stage k, pairs of coordinates (x0, x1) are rotated by an angle theta_k modulated by 4-component Cl(2,0) multivector features:

feat(x0, x1) = [x0^2 + x1^2,  x0,  x1,  x0 * x1_next - x1 * x0_next]
theta_k      = alpha_k + tanh(norm(feat(x0, x1)) @ C_k)
r0           = x0 * cos(theta_k) - x1 * sin(theta_k)
r1           = x0 * sin(theta_k) + x1 * cos(theta_k)

• GRTEmbedding & GRTCosineHead — Token embeddings are stored in polar form (phase_angles + log_magnitude) and tied to a scaled cosine-similarity language modeling head (lm_head.output_scale = 40.23).
• GRTAttention — 11-stage content-modulated Givens butterfly cascades for Q, K, V, O, paired with standard scaled dot-product attention and learnable RoPE frequencies.
• GRTMixer & GRTFFN — Cross-head Givens mixing stage followed by 21-stage rotational FFN expansion/contraction and phase-preserving GRTRadialGELU.
• GRTPearlNorm — Pair-aware RMS normalization that normalizes pair radii sqrt(x_{2i}^2 + x_{2i+1}^2) while preserving phase angles.

---

Intended Use & Limitations

• Research preview. Intended for studying geometric/rotational neural architectures, kernel benchmarking, and multi-backend (PyTorch / llama.cpp / vLLM) verification.
• Base causal LM only. Not instruction-tuned, no chat template, no safety filtering, and prone to repetition under greedy decoding at small parameter scale.

---

Complete MatterWaveAI/grt-30m* Research Suite on Hugging Face

1. Base Causal LM: MatterWaveAI/grt-30m
2. Chat Fine-Tune: MatterWaveAI/grt-30m-chat
3. 4-Expert Mixture-of-Rotations (MoR): MatterWaveAI/grt-30m-mor
4. Circular-Pooled Sentence Embedding: MatterWaveAI/grt-30m-embedding
5. Cross-Encoder Reranker: MatterWaveAI/grt-30m-reranker
6. Vision-Language (GRTVL): MatterWaveAI/grt-30m-vl
7. Canonical Tokenizer & Polar Embedding Bank: MatterWaveAI/grt-30m-tokenizer

• Whitepaper: The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (GRT_Whitepaper.pdf)

---

License

Apache 2.0

---

Citation

@techreport{householter2026grt,
  title       = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
  author      = {Householter, David},
  institution = {MatterWave AI},
  year        = {2026},
  url         = {https://matterwave.ai}
}