davidhouseholter commited on
Commit
0268177
·
verified ·
1 Parent(s): 8e7d26b

docs(model-card): replace raw LaTeX with clean Unicode and code formatting

Browse files
Files changed (2) hide show
  1. README.md +10 -7
  2. SHA256SUMS +1 -1
README.md CHANGED
@@ -22,7 +22,7 @@ pipeline_tag: text-generation
22
 
23
  # GRT-30M (24-Hour Qwen 3 Distillation Preview)
24
 
25
- A **Geometric Rotation Transformer (`GRTForCausalLM`)** distilled from **Qwen 3** (sharing the `151,669`-token Qwen 3 BPE vocabulary) in a single **24-hour distillation run** (`10.79B` tokens at `~123.9k tok/s` across `8x NVIDIA A100-SXM4-80GB`). GRT replaces every dense linear projection (`nn.Linear`) in attention (`Q, K, V, O`) and feed-forward (`expand, contract`) with cascades of content-modulated $2 \times 2$ Givens rotations on $\mathrm{SO}(D)$ interleaved with butterfly permutations.
26
 
27
  **This is a base research checkpoint distilled from Qwen 3, not an instruction-tuned assistant.** It demonstrates that a rotation-only student backbone with just **`9.74M` non-embedding parameters** (`29.31M` total stored parameters with the 151,669-token Qwen 3 vocabulary):
28
  1. Distills stably from Qwen 3 at **`123,900 tok/s`** to **`val_loss = 4.4776`** (`val_ppl = 88.02`, **`27.2%` top-1 token accuracy** over a `151,669`-token vocabulary).
@@ -123,7 +123,7 @@ Evaluated zero-shot (no task fine-tuning) via teacher-forced length-normalized l
123
 
124
  #### PyTorch ROCm (`gfx1151`, `BF16`, Fused HIP Kernels)
125
 
126
- | Workload | Batch $\times$ Seq (`B x T`) | Latency (`ms`) | Throughput (`tok/s`) |
127
  |---|---|---:|---:|
128
  | **Prefill (`B=1, T=512`)** | `1 x 512` | `129.36` | `3,957.9` |
129
  | **Prefill (`B=1, T=2048`)** | `1 x 2048` | `357.89` | **`5,722.5`** |
@@ -177,16 +177,19 @@ sha256sum --ignore-missing -c SHA256SUMS
177
 
178
  ## Architecture Overview
179
 
180
- GRT replaces dense weight matrices (`nn.Linear`) in both attention and FFN blocks with cascades of learned $2 \times 2$ Givens rotations arranged in a butterfly topology. At each stage $k$, pairs of coordinates $(x_0, x_1)$ are rotated by an angle $\theta_k$ modulated by 4-component $\mathrm{Cl}(2,0)$ multivector features:
181
 
182
- $$\mathrm{feat}(x_0, x_1) = \bigl[x_0^2 + x_1^2,\; x_0,\; x_1,\; x_0 x_{1,\text{next}} - x_1 x_{0,\text{next}}\bigr]$$
183
-
184
- $$\theta_k = \alpha_k + \tanh\!\bigl(\widehat{\mathrm{feat}}(x_0, x_1) \cdot C_k\bigr)$$
 
 
 
185
 
186
  - **`GRTEmbedding` & `GRTCosineHead`** — Token embeddings are stored in polar form (`phase_angles` + `log_magnitude`) and tied to a scaled cosine-similarity language modeling head (`lm_head.output_scale = 40.23`).
187
  - **`GRTAttention`** — 11-stage content-modulated Givens butterfly cascades for `Q, K, V, O`, paired with standard scaled dot-product attention and learnable RoPE frequencies.
188
  - **`GRTMixer` & `GRTFFN`** — Cross-head Givens mixing stage followed by 21-stage rotational FFN expansion/contraction and phase-preserving `GRTRadialGELU`.
189
- - **`GRTPearlNorm`** — Pair-aware RMS normalization that normalizes pair radii $\sqrt{x_{2i}^2 + x_{2i+1}^2}$ while preserving phase angles.
190
 
191
  ---
192
 
 
22
 
23
  # GRT-30M (24-Hour Qwen 3 Distillation Preview)
24
 
25
+ A **Geometric Rotation Transformer (`GRTForCausalLM`)** distilled from **Qwen 3** (sharing the `151,669`-token Qwen 3 BPE vocabulary) in a single **24-hour distillation run** (`10.79B` tokens at `~123.9k tok/s` across `8x NVIDIA A100-SXM4-80GB`). GRT replaces every dense linear projection (`nn.Linear`) in attention (`Q, K, V, O`) and feed-forward (`expand, contract`) with cascades of content-modulated `2×2` Givens rotations on `SO(D)` interleaved with butterfly permutations.
26
 
27
  **This is a base research checkpoint distilled from Qwen 3, not an instruction-tuned assistant.** It demonstrates that a rotation-only student backbone with just **`9.74M` non-embedding parameters** (`29.31M` total stored parameters with the 151,669-token Qwen 3 vocabulary):
28
  1. Distills stably from Qwen 3 at **`123,900 tok/s`** to **`val_loss = 4.4776`** (`val_ppl = 88.02`, **`27.2%` top-1 token accuracy** over a `151,669`-token vocabulary).
 
123
 
124
  #### PyTorch ROCm (`gfx1151`, `BF16`, Fused HIP Kernels)
125
 
126
+ | Workload | Batch × Seq (`B × T`) | Latency (`ms`) | Throughput (`tok/s`) |
127
  |---|---|---:|---:|
128
  | **Prefill (`B=1, T=512`)** | `1 x 512` | `129.36` | `3,957.9` |
129
  | **Prefill (`B=1, T=2048`)** | `1 x 2048` | `357.89` | **`5,722.5`** |
 
177
 
178
  ## Architecture Overview
179
 
180
+ GRT replaces dense weight matrices (`nn.Linear`) in both attention and FFN blocks with cascades of learned `2×2` Givens rotations arranged in a butterfly topology. At each stage `k`, pairs of coordinates `(x0, x1)` are rotated by an angle `theta_k` modulated by 4-component `Cl(2,0)` multivector features:
181
 
182
+ ```text
183
+ feat(x0, x1) = [x0^2 + x1^2, x0, x1, x0 * x1_next - x1 * x0_next]
184
+ theta_k = alpha_k + tanh(norm(feat(x0, x1)) @ C_k)
185
+ r0 = x0 * cos(theta_k) - x1 * sin(theta_k)
186
+ r1 = x0 * sin(theta_k) + x1 * cos(theta_k)
187
+ ```
188
 
189
  - **`GRTEmbedding` & `GRTCosineHead`** — Token embeddings are stored in polar form (`phase_angles` + `log_magnitude`) and tied to a scaled cosine-similarity language modeling head (`lm_head.output_scale = 40.23`).
190
  - **`GRTAttention`** — 11-stage content-modulated Givens butterfly cascades for `Q, K, V, O`, paired with standard scaled dot-product attention and learnable RoPE frequencies.
191
  - **`GRTMixer` & `GRTFFN`** — Cross-head Givens mixing stage followed by 21-stage rotational FFN expansion/contraction and phase-preserving `GRTRadialGELU`.
192
+ - **`GRTPearlNorm`** — Pair-aware RMS normalization that normalizes pair radii `sqrt(x_{2i}^2 + x_{2i+1}^2)` while preserving phase angles.
193
 
194
  ---
195
 
SHA256SUMS CHANGED
@@ -2,6 +2,6 @@
2
  5dbb8b936d1e2411f30d1ecf072a4d7c2f2df6e299b0d11c7ef5cbaf72ebd5bc generation_config.json
3
  65bda04ea57b1799e7b0c6d09102475a81c3247c2e29fdb5cbdcdf6d52a21af2 grt-30m-f32.gguf
4
  205f76f3ed59fee0ed54bb8133ea1b1a64527994c68776066057b4707f1b2456 model.safetensors
5
- c4855c6e2a1dd0d2877525588da601b924e9dde79836f7491423f88309cbd819 README.md
6
  be75606093db2094d7cd20f3c2f385c212750648bd6ea4fb2bf507a6a4c55506 tokenizer.json
7
  5f007d04324afa93876e2a484190e868136c69dfe0e1a29a1063eae583df9d32 tokenizer_config.json
 
2
  5dbb8b936d1e2411f30d1ecf072a4d7c2f2df6e299b0d11c7ef5cbaf72ebd5bc generation_config.json
3
  65bda04ea57b1799e7b0c6d09102475a81c3247c2e29fdb5cbdcdf6d52a21af2 grt-30m-f32.gguf
4
  205f76f3ed59fee0ed54bb8133ea1b1a64527994c68776066057b4707f1b2456 model.safetensors
5
+ 025b49eeb0c4dab8e792d4931140b7334191bbde8f0d9795b091017cbdfff14c README.md
6
  be75606093db2094d7cd20f3c2f385c212750648bd6ea4fb2bf507a6a4c55506 tokenizer.json
7
  5f007d04324afa93876e2a484190e868136c69dfe0e1a29a1063eae583df9d32 tokenizer_config.json