Release MatterWaveAI/grt-30m-embedding

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

1_Pooling/config.json

@@ -0,0 +1,5 @@
+{
+    "embedding_dimension": 256,
+    "pooling_mode": "circular",
+    "include_prompt": true
+}

README.md

@@ -0,0 +1,121 @@
+---
+license: apache-2.0
+tags:
+  - grt
+  - geometric-rotation-transformer
+  - sentence-transformers
+  - feature-extraction
+  - sentence-similarity
+  - circular-pooling
+  - rotational
+  - qwen3
+datasets:
+  - sentence-transformers/all-nli
+  - sentence-transformers/stsb
+language:
+  - en
+library_name: sentence-transformers
+pipeline_tag: sentence-similarity
+base_model: MatterWaveAI/grt-30m
+---
+
+# GRT-30M-Embedding (Circular-Pooled Rotational Sentence Encoder)
+
+An **embedding-only Geometric Rotation Transformer (`GRTModel`)** contrastively fine-tuned from [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) for semantic textual similarity and dense retrieval using **planar-pair `circular` pooling** (`sentence-transformers`).
+
+Instead of dense linear projections (`nn.Linear`), GRT represents token embeddings in polar coordinates (`phase_angles` + `log_magnitude`) and transforms hidden states through cascades of content-modulated `2×2` Givens rotations on `SO(D)` interleaved with butterfly permutations. Because every hidden state is structured as `D / 2 = 128` planar `(x, y)` coordinate pairs, **`circular` pooling** decomposes each pair into its unit direction `(cos θ, sin θ)` and radius `r`, averages the unit directions and radii across non-padding tokens, and recombines them into a `256`-dimensional sentence embedding.
+
+---
+
+## Model Details
+
+| Property | Value |
+|---|---|
+| **Architecture** | `GRTModel` (`model_type: "grt"`, encoder/backbone without causal LM head) |
+| **Base model** | [`MatterWaveAI/grt-30m`](https://huggingface.co/MatterWaveAI/grt-30m) |
+| **Pooling strategy** | `circular` (`Pooling(embedding_dimension=256, pooling_mode="circular")` + `Normalize`) |
+| **Output dimensionality** | `256` (`128` planar `(x, y)` pairs, L2-normalized) |
+| **Total stored parameters** | `29,305,497` (`29.31M`, `lm_head.output_scale` stripped) |
+| **Non-embedding backbone parameters** | `9,740,197` (`9.74M` across 12 rotational blocks) |
+| **`hidden_size` (`d_model`)** | `256` |
+| **`num_hidden_layers` (`n_blocks`)** | `12` |
+| **`num_attention_heads` (`n_heads`)** | `4` (`head_dim = 64`) |
+| **`intermediate_size` (`d_ff`)** | `1,024` |
+| **`n_qkv_rotations` / `n_mixer_rotations` / `ffn_n_rotations`** | `11` / `1` / `21` |
+| **Vocabulary** | `151,669` (Qwen 3 BPE) |
+| **Max sequence length** | `512` (supports up to `2,048` positions) |
+
+---
+
+## Evaluation Results (STS-B)
+
+Evaluated on the **STS Benchmark (`sentence-transformers/stsb`)** validation and test splits using `circular` pooling and cosine similarity:
+
+| Split | Metric | `GRT-30M` (Pretrained Base) | `GRT-30M-Embedding` (Fine-Tuned) | Delta |
+|---|---|---:|---:|---:|
+| **STS-B Validation** | **Spearman (`×100`)** | `45.51` | **`72.89`** | **`+27.38`** |
+| **STS-B Test** | **Spearman (`×100`)** | `28.49` | **`63.17`** | **`+34.67`** |
+| **STS-B Test** | **Pearson (`×100`)** | `29.81` | **`63.72`** | **`+33.91`** |
+
+---
+
+## Usage
+
+### 1. `sentence-transformers` (Recommended)
+
+```python
+from sentence_transformers import SentenceTransformer
+
+model = SentenceTransformer("MatterWaveAI/grt-30m-embedding")
+
+sentences = [
+    "Plants convert sunlight into chemical energy through photosynthesis.",
+    "Chlorophyll absorbs light energy to synthesize glucose in plant leaves.",
+    "The stock market experienced volatility during the afternoon session.",
+]
+
+embeddings = model.encode(sentences)
+similarities = model.similarity(embeddings, embeddings)
+print(similarities)
+```
+
+### 2. Hugging Face `transformers` (`GRTModel` + Circular Pooling)
+
+```python
+import torch
+import torch.nn.functional as F
+from transformers import AutoModel, AutoTokenizer
+
+model_id = "MatterWaveAI/grt-30m-embedding"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModel.from_pretrained(model_id).eval()
+
+
+def circular_pool(token_embeddings: torch.Tensor, attention_mask: torch.Tensor) -> torch.Tensor:
+    pairs = token_embeddings.float().unflatten(-1, (-1, 2))
+    mask = attention_mask.float().unsqueeze(-1).unsqueeze(-1)
+    radii = pairs.norm(dim=-1, keepdim=True)
+    directions = pairs / torch.clamp(radii, min=1e-9)
+    denom = torch.clamp(mask.sum(dim=1), min=1e-9)
+    pooled = ((directions * mask).sum(dim=1) / denom) * ((radii * mask).sum(dim=1) / denom)
+    return F.normalize(pooled.flatten(start_dim=1), p=2, dim=-1).to(token_embeddings.dtype)
+
+
+inputs = tokenizer(
+    ["How does photosynthesis work?", "Plants convert sunlight into chemical energy."],
+    padding=True,
+    truncation=True,
+    return_tensors="pt",
+)
+with torch.no_grad():
+    outputs = model(**inputs)
+    embeddings = circular_pool(outputs.last_hidden_state, inputs["attention_mask"])
+
+print("Cosine similarity:", float((embeddings[0] @ embeddings[1]).item()))
+```
+
+---
+
+## License
+
+Apache 2.0

SHA256SUMS

@@ -0,0 +1,10 @@
+a4a8edb13245bb69ccb55a2a2c3f322221e2a16154c96516c9211274030bf815  1_Pooling/config.json
+c987cec1b71deba4a1749a939032309921872ae2a6c21378f8160edb29b210df  config.json
+b15e9a2b676eeb6b5ed466de9be3edf23422cf8ca005fde5e04aef05bc032906  config_sentence_transformers.json
+6045ee7586e1300b509298f76f7f254094a02a78cf23ed669083ab7949eb3eae  eval_results.json
+d1fedff586b749844cc395f28ad5f555dd14d500d6d0c7c0215cf43892f54688  model.safetensors
+e86db55b23831f6efca3d43309863db7d3b96ec21fbf42cffa5afc35fdcc61af  modules.json
+780c5a02bd80ef587d45f338e1284b83d9f1249d100b8be2e55893242ddc8ad0  README.md
+f40b497174aa416490cf8c6e70ef19ad68f2d20160de9ca1fa8a130e89b2ee19  sentence_bert_config.json
+76e9e7b2014f0479052f15dfe092fc5dfadfb52b57b8133a141a4d0deb89a6f7  tokenizer.json
+b0b05688d53eec97348d23f9a203772fa28f12bdc8a0f6f5401f41568ec6ca31  tokenizer_config.json

config.json

@@ -0,0 +1,36 @@
+{
+  "angle_init_scale": 0.1,
+  "architectures": [
+    "GRTModel"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 151643,
+  "coupling_init_mode": "diagonal",
+  "coupling_init_scale": 0.01,
+  "dtype": "float32",
+  "eos_token_id": 151643,
+  "ffn_n_rotations": 21,
+  "hidden_size": 256,
+  "initializer_range": 0.02,
+  "intermediate_size": 1024,
+  "learnable_residual_gates": true,
+  "learnable_rope": true,
+  "log_magnitude_init": 0.0,
+  "max_position_embeddings": 2048,
+  "model_type": "grt",
+  "n_mixer_rotations": 1,
+  "n_qkv_rotations": 11,
+  "num_attention_heads": 4,
+  "num_hidden_layers": 12,
+  "output_scale_init": 30.0,
+  "pad_token_id": 151643,
+  "peri_ln": false,
+  "phase_init_scale": 0.1,
+  "pos_freq_base": 10000.0,
+  "residual_gate_mode": "bounded_depth",
+  "residual_norm_mode": "none",
+  "tie_word_embeddings": false,
+  "transformers_version": "5.21.0.dev0",
+  "use_cache": false,
+  "vocab_size": 151669
+}

config_sentence_transformers.json

@@ -0,0 +1,13 @@
+{
+  "__version__": {
+    "sentence_transformers": "5.2.0.dev0",
+    "transformers": "5.21.0.dev0",
+    "pytorch": "2.12.0+xpu"
+  },
+  "prompts": {
+    "query": "",
+    "document": ""
+  },
+  "default_prompt_name": null,
+  "similarity_fn_name": "cosine"
+}

eval_results.json

@@ -0,0 +1,9 @@
+{
+  "stsb_dev_spearman_before": 0.4551,
+  "stsb_dev_spearman_after": 0.7289,
+  "stsb_test_spearman_before": 0.2849,
+  "stsb_test_spearman_after": 0.6317,
+  "stsb_test_pearson_before": 0.2981,
+  "stsb_test_pearson_after": 0.6372,
+  "parameters": 29305497
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d1fedff586b749844cc395f28ad5f555dd14d500d6d0c7c0215cf43892f54688
+size 121571460

modules.json

@@ -0,0 +1,20 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  },
+  {
+    "idx": 2,
+    "name": "2",
+    "path": "2_Normalize",
+    "type": "sentence_transformers.models.Normalize"
+  }
+]

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 512,
+  "do_lower_case": false
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:76e9e7b2014f0479052f15dfe092fc5dfadfb52b57b8133a141a4d0deb89a6f7
+size 11422916

tokenizer_config.json

@@ -0,0 +1,30 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 131072,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}