MatterWaveAI
/

grt-30m-embedding

@@ -1,135 +1,173 @@
----
-license: apache-2.0
-tags:
-  - grt
-  - geometric-rotation-transformer
-  - sentence-transformers
-  - feature-extraction
-  - sentence-similarity
-  - circular-pooling
-  - rotational
-  - qwen3
-datasets:
-  - sentence-transformers/all-nli
-  - sentence-transformers/stsb
-language:
-  - en
-library_name: sentence-transformers
-pipeline_tag: sentence-similarity
-base_model: MatterWaveAI/grt-30m
----
-# GRT-30M-Embedding (Circular-Pooled Rotational Sentence Encoder)
-An **embedding-only Geometric Rotation Transformer (`GRTModel`)** contrastively fine-tuned from [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) for semantic textual similarity and dense retrieval using **planar-pair `circular` pooling** (`sentence-transformers`).
-Instead of dense linear projections (`nn.Linear`), GRT represents token embeddings in polar coordinates (`phase_angles` + `log_magnitude`) and transforms hidden states through cascades of content-modulated `2×2` Givens rotations on `SO(D)` interleaved with butterfly permutations. Because every hidden state is structured as `D / 2 = 128` planar `(x, y)` coordinate pairs, **`circular` pooling** decomposes each pair into its unit direction `(cos θ, sin θ)` and radius `r`, averages the unit directions and radii across non-padding tokens, and recombines them into a `256`-dimensional sentence embedding.
----
-## Model Details
-| Property | Value |
-|---|---|
-| **Architecture** | `GRTModel` (`model_type: "grt"`, encoder/backbone without causal LM head) |
-| **Base model** | [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) |
-| **Pooling strategy** | `circular` (`Pooling(embedding_dimension=256, pooling_mode="circular")` + `Normalize`) |
-| **Output dimensionality** | `256` (`128` planar `(x, y)` pairs, L2-normalized) |
-| **Total stored parameters** | `29,305,497` (`29.31M`, `lm_head.output_scale` stripped) |
-| **Non-embedding backbone parameters** | `9,740,197` (`9.74M` across 12 rotational blocks) |
-| **`hidden_size` (`d_model`)** | `256` |
-| **`num_hidden_layers` (`n_blocks`)** | `12` |
-| **`num_attention_heads` (`n_heads`)** | `4` (`head_dim = 64`) |
-| **`intermediate_size` (`d_ff`)** | `1,024` |
-| **`n_qkv_rotations` / `n_mixer_rotations` / `ffn_n_rotations`** | `11` / `1` / `21` |
-| **Vocabulary** | `151,669` (Qwen 3 BPE) |
-| **Max sequence length** | `512` (supports up to `2,048` positions) |
----
-## Two-Stage Contrastive Fine-Tuning Method (Intel Arc XPU)
-Starting from the pretrained [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) causal checkpoint (`lm_head.output_scale` stripped to form a headless `GRTModel` encoder), the `151,669` lexical `embedding.phase_angles` (`19.41M` parameters) were frozen to preserve the pretrained Qwen 3 polar token geometry while training the **`9,740,196` rotational backbone parameters + `151,669` token `log_magnitude` weights (`9,891,865` trainable parameters)** in `FP32` on an **Intel Arc 140T GPU (`xpu:0`, 32 GB)** using fused oneAPI DPC++ SYCL Givens and Clifford kernels (`~75s` total wall-clock time):
-| Stage | Dataset & Budget | Objective & Hyperparameters |
-|---|---|---|
-| **Parameter Freezing** | `embedding.phase_angles` frozen (`19.41M` params) | `9,891,865` trainable parameters (`12` rotational blocks + `embedding.log_magnitude`); max grad norm `1.0` |
-| **Stage 1: AllNLI Contrastive Alignment** | `sentence-transformers/all-nli` (`triplet` split) · `150 steps` (`4,800` triplets, `<1%` of corpus) | In-batch `MultipleNegativesRankingLoss` + explicit hard negatives (`τ = 0.05`), `batch_size = 32`, `max_length = 96`, `PairAdam(lr=1.5e-4, coupling_lr_mult=3.0, max_angular_step=0.05)` |
-| **Stage 2: STS-B Rank & Metric Calibration** | `sentence-transformers/stsb` (`train` split, `5,749` pairs) · `2 epochs` (`358 steps`) | `CoSENT` rank-consistency loss (`τ = 20.0`) + `2.0 × CosineMSE`, `batch_size = 32`, `max_length = 128`, `PairAdam(lr=8e-5, coupling_lr_mult=3.0, max_angular_step=0.05)` |
----
-## Evaluation Results (STS-B)
-Evaluated on the **STS Benchmark (`sentence-transformers/stsb`)** validation and test splits using `circular` pooling and cosine similarity:
-| Split | Metric | `GRT-30M` (Pretrained Base) | `GRT-30M-Embedding` (`PairAdam`) | `AdamW` Ablation¹ | Delta (`PairAdam` vs. Base) |
-|---|---|---:|---:|---:|---:|
-| **STS-B Validation** | **Spearman (`×100`)** | `45.51` | **`71.79`** | `72.89` | **`+26.28`** |
-| **STS-B Test** | **Spearman (`×100`)** | `28.49` | **`62.42`** | `63.17` | **`+33.93`** |
-| **STS-B Test** | **Pearson (`×100`)** | `29.81` | **`63.30`** | `63.72` | **`+33.49`** |
-> ¹ *Optimizer comparison note:* The published checkpoint is trained with `PairAdam` (zero weight decay and `[-π, π]` wrapping on rotation angles, `max_angular_step = 0.05`, and rank-1 factored second moments with a `3×` learning-rate multiplier on `Cl(2,0)` coupling tensors) for consistency with `GRT-30M` pretraining. In a head-to-head run on the identical `508`-step schedule and random seed (`42`), standard `AdamW` (`1×` coupling LR, full diagonal second moments) reaches `72.89` Val / `63.17` Test Spearman (`+1.10` / `+0.75` points), indicating that while `PairAdam`'s factored moments and `3×` coupling LR multiplier stabilize large-batch (`524k` tok/step) pretraining from scratch, standard diagonal moments at `1×` coupling LR are slightly gentler during short, small-batch (`B = 32`) downstream fine-tuning.
----
-## Usage
-### 1. `sentence-transformers` (Recommended)
-```python
-from sentence_transformers import SentenceTransformer
-model = SentenceTransformer("MatterWaveAI/grt-30m-embedding")
-sentences = [
-    "Plants convert sunlight into chemical energy through photosynthesis.",
-    "Chlorophyll absorbs light energy to synthesize glucose in plant leaves.",
-    "The stock market experienced volatility during the afternoon session.",
-]
-embeddings = model.encode(sentences)
-similarities = model.similarity(embeddings, embeddings)
-print(similarities)
-```
-### 2. Hugging Face `transformers` (`GRTModel` + Circular Pooling)
-```python
-import torch
-import torch.nn.functional as F
-from transformers import AutoModel, AutoTokenizer
-model_id = "MatterWaveAI/grt-30m-embedding"
-tokenizer = AutoTokenizer.from_pretrained(model_id)
-model = AutoModel.from_pretrained(model_id).eval()
-def circular_pool(token_embeddings: torch.Tensor, attention_mask: torch.Tensor) -> torch.Tensor:
-    pairs = token_embeddings.float().unflatten(-1, (-1, 2))
-    mask = attention_mask.float().unsqueeze(-1).unsqueeze(-1)
-    radii = pairs.norm(dim=-1, keepdim=True)
-    directions = pairs / torch.clamp(radii, min=1e-9)
-    denom = torch.clamp(mask.sum(dim=1), min=1e-9)
-    pooled = ((directions * mask).sum(dim=1) / denom) * ((radii * mask).sum(dim=1) / denom)
-    return F.normalize(pooled.flatten(start_dim=1), p=2, dim=-1).to(token_embeddings.dtype)
-inputs = tokenizer(
-    ["How does photosynthesis work?", "Plants convert sunlight into chemical energy."],
-    padding=True,
-    truncation=True,
-    return_tensors="pt",
-)
-with torch.no_grad():
-    outputs = model(**inputs)
-    embeddings = circular_pool(outputs.last_hidden_state, inputs["attention_mask"])
-print("Cosine similarity:", float((embeddings[0] @ embeddings[1]).item()))
-```
----
-## License
-Apache 2.0

+---
+license: apache-2.0
+tags:
+  - grt
+  - geometric-rotation-transformer
+  - sentence-transformers
+  - feature-extraction
+  - sentence-similarity
+  - circular-pooling
+  - rotational
+  - qwen3
+datasets:
+  - sentence-transformers/all-nli
+  - sentence-transformers/stsb
+language:
+  - en
+library_name: sentence-transformers
+pipeline_tag: sentence-similarity
+base_model: MatterWaveAI/grt-30m
+---
+# GRT-30M-Embedding (Circular-Pooled Rotational Sentence Encoder)
+An **embedding-only Geometric Rotation Transformer (`GRTModel`)** contrastively fine-tuned from [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) for semantic textual similarity and dense retrieval using **planar-pair `circular` pooling** (`sentence-transformers`).
+Instead of dense linear projections (`nn.Linear`), GRT represents token embeddings in polar coordinates (`phase_angles` + `log_magnitude`) and transforms hidden states through cascades of content-modulated `2×2` Givens rotations on `SO(D)` interleaved with butterfly permutations. Because every hidden state is structured as `D / 2 = 128` planar `(x, y)` coordinate pairs, **`circular` pooling** decomposes each pair into its unit direction `(cos θ, sin θ)` and radius `r`, averages the unit directions and radii across non-padding tokens, and recombines them into a `256`-dimensional sentence embedding.
+---
+## Model Details
+| Property | Value |
+|---|---|
+| **Architecture** | `GRTModel` (`model_type: "grt"`, encoder/backbone without causal LM head) |
+| **Base model** | [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) |
+| **Pooling strategy** | `circular` (`Pooling(embedding_dimension=256, pooling_mode="circular")` + `Normalize`) |
+| **Output dimensionality** | `256` (`128` planar `(x, y)` pairs, L2-normalized) |
+| **Total stored parameters** | `29,305,497` (`29.31M`, `lm_head.output_scale` stripped) |
+| **Non-embedding backbone parameters** | `9,740,197` (`9.74M` across 12 rotational blocks) |
+| **`hidden_size` (`d_model`)** | `256` |
+| **`num_hidden_layers` (`n_blocks`)** | `12` |
+| **`num_attention_heads` (`n_heads`)** | `4` (`head_dim = 64`) |
+| **`intermediate_size` (`d_ff`)** | `1,024` |
+| **`n_qkv_rotations` / `n_mixer_rotations` / `ffn_n_rotations`** | `11` / `1` / `21` |
+| **Vocabulary** | `151,669` (Qwen 3 BPE) |
+| **Max sequence length** | `512` (supports up to `2,048` positions) |
+---
+## Two-Stage Contrastive Fine-Tuning Method (Intel Arc XPU)
+Starting from the pretrained [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) causal checkpoint (`lm_head.output_scale` stripped to form a headless `GRTModel` encoder), the `151,669` lexical `embedding.phase_angles` (`19.41M` parameters) were frozen to preserve the pretrained Qwen 3 polar token geometry while training the **`9,740,196` rotational backbone parameters + `151,669` token `log_magnitude` weights (`9,891,865` trainable parameters)** in `FP32` on an **Intel Arc 140T GPU (`xpu:0`, 32 GB)** using fused oneAPI DPC++ SYCL Givens and Clifford kernels (`~75s` total wall-clock time):
+| Stage | Dataset & Budget | Objective & Hyperparameters |
+|---|---|---|
+| **Parameter Freezing** | `embedding.phase_angles` frozen (`19.41M` params) | `9,891,865` trainable parameters (`12` rotational blocks + `embedding.log_magnitude`); max grad norm `1.0` |
+| **Stage 1: AllNLI Contrastive Alignment** | `sentence-transformers/all-nli` (`triplet` split) · `150 steps` (`4,800` triplets, `<1%` of corpus) | In-batch `MultipleNegativesRankingLoss` + explicit hard negatives (`τ = 0.05`), `batch_size = 32`, `max_length = 96`, `PairAdam(lr=1.5e-4, coupling_lr_mult=3.0, max_angular_step=0.05)` |
+| **Stage 2: STS-B Rank & Metric Calibration** | `sentence-transformers/stsb` (`train` split, `5,749` pairs) · `2 epochs` (`358 steps`) | `CoSENT` rank-consistency loss (`τ = 20.0`) + `2.0 × CosineMSE`, `batch_size = 32`, `max_length = 128`, `PairAdam(lr=8e-5, coupling_lr_mult=3.0, max_angular_step=0.05)` |
+---
+## Evaluation Results (STS-B)
+Evaluated on the **STS Benchmark (`sentence-transformers/stsb`)** validation and test splits using `circular` pooling and cosine similarity:
+| Split | Metric | `GRT-30M` (Pretrained Base) | `GRT-30M-Embedding` (`PairAdam`) | `AdamW` Ablation¹ | Delta (`PairAdam` vs. Base) |
+|---|---|---:|---:|---:|---:|
+| **STS-B Validation** | **Spearman (`×100`)** | `45.51` | **`71.79`** | `72.89` | **`+26.28`** |
+| **STS-B Test** | **Spearman (`×100`)** | `28.49` | **`62.42`** | `63.17` | **`+33.93`** |
+| **STS-B Test** | **Pearson (`×100`)** | `29.81` | **`63.30`** | `63.72` | **`+33.49`** |
+> ¹ *Optimizer comparison note:* The published checkpoint is trained with `PairAdam` (zero weight decay and `[-π, π]` wrapping on rotation angles, `max_angular_step = 0.05`, and rank-1 factored second moments with a `3×` learning-rate multiplier on `Cl(2,0)` coupling tensors) for consistency with `GRT-30M` pretraining. In a head-to-head run on the identical `508`-step schedule and random seed (`42`), standard `AdamW` (`1×` coupling LR, full diagonal second moments) reaches `72.89` Val / `63.17` Test Spearman (`+1.10` / `+0.75` points), indicating that while `PairAdam`'s factored moments and `3×` coupling LR multiplier stabilize large-batch (`524k` tok/step) pretraining from scratch, standard diagonal moments at `1×` coupling LR are slightly gentler during short, small-batch (`B = 32`) downstream fine-tuning.
+---
+## Usage
+### 1. `sentence-transformers` (Recommended)
+```python
+from sentence_transformers import SentenceTransformer
+model = SentenceTransformer("MatterWaveAI/grt-30m-embedding")
+sentences = [
+    "Plants convert sunlight into chemical energy through photosynthesis.",
+    "Chlorophyll absorbs light energy to synthesize glucose in plant leaves.",
+    "The stock market experienced volatility during the afternoon session.",
+]
+embeddings = model.encode(sentences)
+similarities = model.similarity(embeddings, embeddings)
+print(similarities)
+```
+### 2. Hugging Face `transformers` (`GRTModel` + Circular Pooling)
+```python
+import torch
+import torch.nn.functional as F
+from transformers import AutoModel, AutoTokenizer
+model_id = "MatterWaveAI/grt-30m-embedding"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModel.from_pretrained(model_id).eval()
+def circular_pool(token_embeddings: torch.Tensor, attention_mask: torch.Tensor) -> torch.Tensor:
+    pairs = token_embeddings.float().unflatten(-1, (-1, 2))
+    mask = attention_mask.float().unsqueeze(-1).unsqueeze(-1)
+    radii = pairs.norm(dim=-1, keepdim=True)
+    directions = pairs / torch.clamp(radii, min=1e-9)
+    denom = torch.clamp(mask.sum(dim=1), min=1e-9)
+    pooled = ((directions * mask).sum(dim=1) / denom) * ((radii * mask).sum(dim=1) / denom)
+    return F.normalize(pooled.flatten(start_dim=1), p=2, dim=-1).to(token_embeddings.dtype)
+inputs = tokenizer(
+    ["How does photosynthesis work?", "Plants convert sunlight into chemical energy."],
+    padding=True,
+    truncation=True,
+    return_tensors="pt",
+)
+with torch.no_grad():
+    outputs = model(**inputs)
+    embeddings = circular_pool(outputs.last_hidden_state, inputs["attention_mask"])
+print("Cosine similarity:", float((embeddings[0] @ embeddings[1]).item()))
+```
+---
+## Whitepaper & Technical Reports (`matterwave.ai`)
+- **Full Whitepaper**: [**The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (`GRT_Whitepaper.pdf`)**](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/GRT_Whitepaper.pdf)
+- **Companion Technical Reports** ([`MatterWaveAI/grt-30m/tree/main/papers`](https://huggingface.co/MatterWaveAI/grt-30m/tree/main/papers)):
+  - [`rotational_embedding.pdf`](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/papers/rotational_embedding.pdf) — *RotationalEmbedding: Phase-Angle Token Representation on a Product of Circles*
+  - [`pearlnorm.pdf`](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/papers/pearlnorm.pdf) — *PearlNorm: Pair-Aware Root-Mean-Square Layer Normalization*
+  - [`rotationalmixer.pdf`](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/papers/rotationalmixer.pdf) — *RotationalMixer: Content-Dependent Butterfly-Shuffled Mixing Between Attention and FFN*
+  - [`radialgelu.pdf`](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/papers/radialgelu.pdf) — *RadialGELU: A Phase-Preserving Nonlinearity for Rotation-Based Neural Networks*
+  - [`pairadam.pdf`](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/papers/pairadam.pdf) — *PairAdam: Geometry-Aware Optimization for Rotation-Based Neural Networks*
+  - [`mor.pdf`](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/papers/mor.pdf) — *Mixture-of-Rotations: Soft Expert Routing for Rotation-Based Feed-Forward Networks*
+  - [`stokes_coupling.pdf`](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/papers/stokes_coupling.pdf) — *Stokes & Clifford Multivector Coupling for Rotational Attention*
+  - [`fermionic_grt.pdf`](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/papers/fermionic_grt.pdf) — *FermionicGRT: Real-Only Clifford-Algebra Transformers with Matchgate Simulation*
+  - [`prism_grt.pdf`](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/papers/prism_grt.pdf) — *PRISM: Probe-Reservoir Injection and State Modulation for Self-Editing Rotational Transformers*
+  - [`rotation_machine.pdf`](https://huggingface.co/MatterWaveAI/grt-30m/resolve/main/papers/rotation_machine.pdf) — *Rotation Machine: Language Modeling via Recursive Application of a Single Computational Rule*
+---
+## Free Software Statement ("Free as in Freedom") & License
+> *"Free software is a matter of liberty, not price. To understand the concept, you should think of 'free' as in 'free speech,' not as in 'free beer.'"*
+> — **Richard M. Stallman**, GNU Project / Free Software Foundation
+All Geometric Rotation Transformer (**GRT**) model weights, polar tokenizers, training recipes, and custom hardware kernels (CUDA, ROCm/HIP, oneAPI SYCL XPU, Vulkan, and CPU AVX2) are released as **free software and free scientific works** under the **Apache-2.0 License** at [`https://matterwave.ai`](https://matterwave.ai) and [`https://huggingface.co/MatterWaveAI`](https://huggingface.co/MatterWaveAI). You have the four essential freedoms:
+1. **Freedom 0**: The freedom to run the models and software for any purpose.
+2. **Freedom 1**: The freedom to study how the rotational primitives, phase angles, and Clifford couplings work, and change them to make them do what you wish.
+3. **Freedom 2**: The freedom to redistribute exact copies to help others.
+4. **Freedom 3**: The freedom to distribute copies of your modified versions, upcycled experts, and fine-tuned checkpoints to the community.
+---
+## Citation
+```bibtex
+@techreport{householter2026grt,
+  title       = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
+  author      = {Householter, David and {The MatterWave AI Team}},
+  institution = {MatterWave AI},
+  year        = {2026},
+  url         = {https://matterwave.ai}
+}
+```