Hugging Face's logo

MatterWaveAI
/
grt-30m

Text Generation
Transformers
Safetensors
GGUF
English
grt
geometric-rotation-transformer
causal-lm
rotational
knowledge-distillation
qwen3
Model card Files
xet
 Community
grt-30m / README.md
davidhouseholter's picture
davidhouseholter
Update README: remove GNU Free Software Statement, keep Apache 2.0 and matterwave.ai citation
1999504 verified
preview code
|
Raw History Blame Contribute Delete
14 kB
---
license: apache-2.0
tags:
- grt
- geometric-rotation-transformer
- causal-lm
- rotational
- knowledge-distillation
- qwen3
- gguf
datasets:
- HuggingFaceFW/fineweb-edu
- HuggingFaceTB/cosmopedia-v2
- wikimedia/wikipedia
- HuggingFaceTB/finemath
- HuggingFaceTB/smollm-corpus
language:
- en
library_name: transformers
pipeline_tag: text-generation
---
# GRT-30M (24-Hour Qwen 3 Distillation Preview)
A **Geometric Rotation Transformer (`GRTForCausalLM`)** distilled from **Qwen 3** (sharing the `151,669`-token Qwen 3 BPE vocabulary) in a single **24-hour distillation run** (`10.79B` tokens at `~123.9k tok/s` across `8x NVIDIA A100-SXM4-80GB`). GRT replaces every dense linear projection (`nn.Linear`) in attention (`Q, K, V, O`) and feed-forward (`expand, contract`) with cascades of content-modulated `2×2` Givens rotations on `SO(D)` interleaved with butterfly permutations.
**This is a base research checkpoint distilled from Qwen 3, not an instruction-tuned assistant.** It demonstrates that a rotation-only student backbone with just **`9.74M` non-embedding parameters** (`29.31M` total stored parameters with the 151,669-token Qwen 3 vocabulary):
1. Distills stably from Qwen 3 at **`123,900 tok/s`** to **`val_loss = 4.4776`** (`val_ppl = 88.02`, **`27.2%` top-1 token accuracy** over a `151,669`-token vocabulary).
2. Achieves **bit-for-bit 4-way greedy generation parity** across **PyTorch ROCm (`gfx1151`)**, **`llama.cpp` CPU (`AVX2`)**, **`llama.cpp` Vulkan (`RADV`)**, and **`llama.cpp` ROCm (`HIP`)**.
3. Runs at **`3,790 tok/s` prefill** (Vulkan) and **`87.4 tok/s` decode** (CPU, 16 threads) in pure `FP32` on an AMD Ryzen AI Max+ 395 (`Strix Halo` / `gfx1151`), and **`5,760 tok/s` forward** in `BF16` with the fused GRT HIP kernels.
4. Beats random chance across **6 of 7 zero-shot multiple-choice benchmarks** (`ARC-Easy 29.50%`, `HellaSwag 29.20%`, `OpenBookQA 28.40%`, `CommonsenseQA 21.70%`, `BoolQ 51.53%`, `QNLI 51.50%`), matching dense models (`Pythia-14M` / `Pythia-31M`) of comparable backbone capacity.
---
## Model Details
| Property | Value |
|---|---|
| **Architecture** | `GRTForCausalLM` (`model_type: "grt"`, `grt30m_kd`) |
| **Training paradigm** | Knowledge distillation from **Qwen 3** |
| **Total stored parameters** | `29,305,498` (`29.31M`) |
| **Non-embedding backbone parameters** | `9,740,197` (`9.74M` across 12 blocks) |
| **Dense-equivalent parameters** | `55,680,446` (`55.68M`) |
| **`hidden_size` (`d_model`)** | `256` |
| **`num_hidden_layers` (`n_blocks`)** | `12` |
| **`num_attention_heads` (`n_heads`)** | `4` (`head_dim = 64`) |
| **`intermediate_size` (`d_ff`)** | `1,024` |
| **`n_qkv_rotations` / `n_mixer_rotations` / `ffn_n_rotations`** | `11` / `1` / `21` |
| **Vocabulary** | `151,669` (Qwen 3 BPE, tied polar cosine head) |
| **Context length (`max_position_embeddings`)** | `2,048` |
| **Weight files** | `model.safetensors` (`116 MB`), `grt-30m-f32.gguf` (`340 MB`) |
Because `grt-30m` shares the full `151,669`-token Qwen 3 vocabulary at `hidden_size = 256` for direct token-aligned distillation from Qwen 3, the tied polar embedding table (`phase_angles [151669, 128]` + `log_magnitude [151669]`) accounts for **`19.57M` parameters (`66.8%` of stored weights)**, leaving **`9.74M` stored parameters** for the 12-layer rotational transformer backbone.
---
## 24-Hour Qwen 3 Distillation Summary
| Metric | Value |
|---|---|
| **Teacher family / Tokenizer** | **Qwen 3** (`151,669`-token Qwen 3 BPE vocabulary) |
| **Student config** | `grt30m_kd` (`d_model = 256`, `12` rotational blocks, `9.74M` non-embedding params) |
| **Training corpus mix** | `fineweb-edu` (`45%`), `cosmopedia-v2` (`20%`), `wikipedia` (`15%`), `finemath` (`12%`), `stack-edu` (`8%`) |
| **Hardware** | `8x NVIDIA A100-SXM4-80GB` (DDP + Fused CUDA Givens Kernels) |
| **Wall-clock duration** | `24.0 hours` (`86,969s`) |
| **Tokens trained** | `10,789,404,672` (`10.79B` tokens, `20,580` steps) |
| **Global batch size** | `524,288` tokens/step (`micro_batch = 32`, `grad_accum = 1`, `seq_len = 2048`, `8` GPUs) |
| **Sustained throughput** | `123,900 tok/s` (`~15,490 tok/s/GPU`, zero NaNs or gradient spikes) |
| **Optimizer** | `PairAdam` (factored second moments for 2D coupling matrices + angular step clamping) |
| **Learning rate** | Peak `1e-3`, WSD schedule (`checkpoint-20000` saved at `20,000 / 57,220` steps) |
| **Final validation loss / PPL (FineWeb-Edu)** | `val_loss = 4.4776` / `val_ppl = 88.02` (`top-1 token acc = 27.20%`) |
---
## 4-Backend Exact Greedy Generation Parity
`model.safetensors` (`transformers`) and `grt-30m-f32.gguf` (`llama.cpp`) produce **identical token-for-token greedy continuations** (`temperature = 0`, `16` new tokens) across all four supported backends on AMD Strix Halo (`gfx1151`):
- **PyTorch ROCm (`gfx1151`, fused HIP kernels)**
- **`llama.cpp` CPU (`x86_64 AVX2`)**
- **`llama.cpp` Vulkan (`RADV gfx1151`)**
- **`llama.cpp` ROCm (`HIP gfx1151`)**
| Prompt | Greedy continuation (`16` tokens, identical on all 4 backends) |
|---|---|
| `The capital of France is` | `The capital of France is the first to be the first to be the first to be the first to be` |
| `Once upon a time` | `Once upon a time, you might be wondering what exactly is the way to do with the right point` |
| `In the beginning` | `In the beginning of the world, the world is the most common form of the world. The` |
---
## Benchmarks
### 1. Zero-Shot Multiple-Choice & Held-Out Perplexity
Evaluated zero-shot (no task fine-tuning) via teacher-forced length-normalized log-likelihood scoring on `checkpoint-20000`:
| Benchmark | Split (`n`) | `GRT-30M` | Random Baseline | Delta vs. Random |
|---|---|---:|---:|---:|
| **ARC-Easy** (`allenai/ai2_arc`) | `test` (`2,376`) | **`29.50%`** | `25.00%` | **`+4.50%`** |
| **HellaSwag** (`Rowan/hellaswag`) | `val` (`1,000`) | **`29.20%`** | `25.00%` | **`+4.20%`** |
| **OpenBookQA** (`allenai/openbookqa`) | `test` (`500`) | **`28.40%`** | `25.00%` | **`+3.40%`** |
| **CommonsenseQA** (`tau/commonsense_qa`) | `val` (`1,221`) | **`21.70%`** | `20.00%` | **`+1.70%`** |
| **BoolQ** (`google/boolq`) | `val` (`3,270`) | **`51.53%`** | `50.00%` | **`+1.53%`** |
| **QNLI** (`nyu-mll/glue`) | `val` (`1,000`) | **`51.50%`** | `50.00%` | **`+1.50%`** |
| **ARC-Challenge** (`allenai/ai2_arc`) | `test` (`1,172`) | `21.08%` | `25.00%` | `-3.92%` |
| Held-Out Language Modeling | Split / Tokens | Loss (NLL) | Perplexity (151k vocab) | Top-1 Token Acc |
|---|---|---:|---:|---:|
| **FineWeb-Edu (In-Domain Val)** | `val` (`step 20,580`) | `4.4776` | **`88.02`** | **`27.20%`** |
| **WikiText-2 (`wikitext-2-raw-v1`)** | `test` (`297,984` toks, `T=1024`) | `5.0305` | **`153.01`** | **`25.71%`** |
### 2. Inference & Kernel Throughput on AMD Strix Halo (`Radeon 8060S` / `gfx1151`)
#### `llama.cpp` (`grt-30m-f32.gguf`, `FP32`)
| Backend | Prefill `pp128` (`tok/s`) | Prefill `pp512` (`tok/s`) | Decode `tg32` (`tok/s`) | Decode `tg64` (`tok/s`) |
|---|---:|---:|---:|---:|
| **Vulkan (`-ngl 99`)** | **1,947.0** | **3,790.1** | 49.0 | 49.9 |
| **CPU (`-t 16`, AVX2)** | 754.1 | 905.4 | **86.5** | **87.4** |
| **ROCm (`-ngl 99`)** | 181.9 | 195.0 | 37.7 | 37.5 |
#### PyTorch ROCm (`gfx1151`, `BF16`, Fused HIP Kernels)
| Workload | Batch × Seq (`B × T`) | Latency (`ms`) | Throughput (`tok/s`) |
|---|---|---:|---:|
| **Prefill (`B=1, T=512`)** | `1 x 512` | `129.36` | `3,957.9` |
| **Prefill (`B=1, T=2048`)** | `1 x 2048` | `357.89` | **`5,722.5`** |
| **Forward (`B=4, T=512`, Fused HIP)** | `4 x 512` | `355.58` | **`5,759.6`** |
| **Training Step (`Fwd + Bwd`, Fused HIP)** | `4 x 512` | `1,771.68` | **`1,156.0`** |
### 3. Circular-Pooled Sentence Encoder (`MatterWaveAI/grt-30m-embedding`)
We also release [`MatterWaveAI/grt-30m-embedding`](https://huggingface.co/MatterWaveAI/grt-30m-embedding) (`GRTModel` + `SentenceTransformer`), adapted from `grt-30m` using planar-pair **`circular` pooling** (`pooling_mode="circular"`, averaging unit directions and pair radii separately across 128 planar `(x, y)` pairs). On an **Intel Arc 140T GPU (`xpu:0`, fused oneAPI SYCL kernels)**, the `151,669` lexical `embedding.phase_angles` were frozen while training the `9,891,865` rotational backbone and `log_magnitude` parameters with `PairAdam` in two stages:
| Stage | Dataset & Budget | Objective & Hyperparameters |
|---|---|---|
| **Stage 1: AllNLI Contrastive Alignment** | `sentence-transformers/all-nli` (`150 steps`, `4,800` triplets, `<1%` of corpus) | `MultipleNegativesRankingLoss` + explicit hard negatives (`τ = 0.05`, `B = 32`, `PairAdam lr = 1.5e-4`) |
| **Stage 2: STS-B Rank & Metric Calibration** | `sentence-transformers/stsb` (`2 epochs`, `358 steps` over `5,749` pairs) | `CoSENT` (`τ = 20.0`) + `2.0 × CosineMSE` (`B = 32`, `PairAdam lr = 8e-5`) |
| Split | Metric | `GRT-30M` (Pretrained Base) | `GRT-30M-Embedding` (`PairAdam`) | `AdamW` Ablation¹ | Delta (`PairAdam` vs. Base) |
|---|---|---:|---:|---:|---:|
| **STS-B Validation** | **Spearman (`×100`)** | `45.51` | **`71.79`** | `72.89` | **`+26.28`** |
| **STS-B Test** | **Spearman (`×100`)** | `28.49` | **`62.42`** | `63.17` | **`+33.93`** |
| **STS-B Test** | **Pearson (`×100`)** | `29.81` | **`63.30`** | `63.72` | **`+33.49`** |
> ¹ *Footnote:* Under the identical `508`-step schedule and seed (`42`), standard `AdamW` (`1×` coupling LR, full diagonal second moments) reaches `72.89` Val / `63.17` Test Spearman vs. `71.79` / `62.42` for `PairAdam` (`3×` coupling LR, factored second moments), suggesting `PairAdam`'s `3×` coupling multiplier is primarily beneficial for large-batch pretraining rather than short small-batch (`B = 32`) fine-tuning.
---
## Usage
### PyTorch (`transformers`)
```python
import torch
from transformers import AutoTokenizer, GRTForCausalLM
# Optional: register fused CUDA/ROCm Givens kernels if grt-kernels is installed
try:
from grt_kernels.torch_ops import ensure_grt_ops
ensure_grt_ops()
except ImportError:
pass
model_id = "MatterWaveAI/grt-30m"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = GRTForCausalLM.from_pretrained(model_id, dtype=torch.float32).eval()
prompt = "The history of the city"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids
with torch.no_grad():
output = model.generate(input_ids, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```
### `llama.cpp` (`GGUF`)
```bash
./build/bin/llama-completion \
-m grt-30m-f32.gguf \
-p "Once upon a time" \
-n 32 --temp 0 --seed 0
```
Verify file integrity against `SHA256SUMS`:
```bash
sha256sum --ignore-missing -c SHA256SUMS
```
---
## Architecture Overview
GRT replaces dense weight matrices (`nn.Linear`) in both attention and FFN blocks with cascades of learned `2×2` Givens rotations arranged in a butterfly topology. At each stage `k`, pairs of coordinates `(x0, x1)` are rotated by an angle `theta_k` modulated by 4-component `Cl(2,0)` multivector features:
```text
feat(x0, x1) = [x0^2 + x1^2, x0, x1, x0 * x1_next - x1 * x0_next]
theta_k = alpha_k + tanh(norm(feat(x0, x1)) @ C_k)
r0 = x0 * cos(theta_k) - x1 * sin(theta_k)
r1 = x0 * sin(theta_k) + x1 * cos(theta_k)
```
- **`GRTEmbedding` & `GRTCosineHead`** — Token embeddings are stored in polar form (`phase_angles` + `log_magnitude`) and tied to a scaled cosine-similarity language modeling head (`lm_head.output_scale = 40.23`).
- **`GRTAttention`** — 11-stage content-modulated Givens butterfly cascades for `Q, K, V, O`, paired with standard scaled dot-product attention and learnable RoPE frequencies.
- **`GRTMixer` & `GRTFFN`** — Cross-head Givens mixing stage followed by 21-stage rotational FFN expansion/contraction and phase-preserving `GRTRadialGELU`.
- **`GRTPearlNorm`** — Pair-aware RMS normalization that normalizes pair radii `sqrt(x_{2i}^2 + x_{2i+1}^2)` while preserving phase angles.
---
## Intended Use & Limitations
- **Research preview.** Intended for studying geometric/rotational neural architectures, kernel benchmarking, and multi-backend (`PyTorch` / `llama.cpp` / `vLLM`) verification.
- **Base causal LM only.** Not instruction-tuned, no chat template, no safety filtering, and prone to repetition under greedy decoding at small parameter scale.
---
## Complete `MatterWaveAI/grt-30m*` Research Suite on Hugging Face
1. **Base Causal LM**: [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m)
2. **Chat Fine-Tune**: [`MatterWaveAI/grt-30m-chat`](https://huggingface.co/MatterWaveAI/grt-30m-chat)
3. **4-Expert Mixture-of-Rotations (`MoR`)**: [`MatterWaveAI/grt-30m-mor`](https://huggingface.co/MatterWaveAI/grt-30m-mor)
4. **Circular-Pooled Sentence Embedding**: [`MatterWaveAI/grt-30m-embedding`](https://huggingface.co/MatterWaveAI/grt-30m-embedding)
5. **Cross-Encoder Reranker**: [`MatterWaveAI/grt-30m-reranker`](https://huggingface.co/MatterWaveAI/grt-30m-reranker)
6. **Vision-Language (`GRTVL`)**: [`MatterWaveAI/grt-30m-vl`](https://huggingface.co/MatterWaveAI/grt-30m-vl)
7. **Canonical Tokenizer & Polar Embedding Bank**: [`MatterWaveAI/grt-30m-tokenizer`](https://huggingface.co/MatterWaveAI/grt-30m-tokenizer)
- **Whitepaper**: [**The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (`GRT_Whitepaper.pdf`)**](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/GRT_Whitepaper.pdf)
---
## License
Apache 2.0
---
## Citation
```bibtex
@techreport{householter2026grt,
title = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
author = {Householter, David},
institution = {MatterWave AI},
year = {2026},
url = {https://matterwave.ai}
}
```