Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -1,3 +1,17 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
# Rodan-10M
|
| 2 |
|
| 3 |
A ~11M-parameter language model trained start to finish on one Apple M2 with MLX. The aim was a tiny model
|
|
@@ -46,41 +60,6 @@ It was built in two passes: a from-scratch base on 262M tokens, then a warm-star
|
|
| 46 |
115M tokens that adds LRM, raises the RoPE base from 10k to 200k, and mixes in 21% arithmetic/reasoning data
|
| 47 |
(Falcon's reasoning-in-pretraining idea). That second pass is the 11.46M v6 checkpoint.
|
| 48 |
|
| 49 |
-
```mermaid
|
| 50 |
-
flowchart TB
|
| 51 |
-
ids["token ids"]:::io --> emb["Embedding 8192x320 (tied)"]:::emb
|
| 52 |
-
emb --> blk["8 x ModernBlock"]:::core
|
| 53 |
-
blk --> fn["RMSNorm"]:::norm
|
| 54 |
-
fn --> head["tied head (x @ Wemb^T)"]:::emb
|
| 55 |
-
head --> out["logits 8192"]:::io
|
| 56 |
-
|
| 57 |
-
subgraph ModernBlock["ModernBlock (x8)"]
|
| 58 |
-
direction TB
|
| 59 |
-
x(["x"]):::res --> n1["RMSNorm"]:::norm
|
| 60 |
-
n1 --> qkv["q/k/v projection<br/>MQA: 8 q-heads, 1 kv-head, head_dim 40"]:::attn
|
| 61 |
-
qkv -->|"q, k"| qk["QK-norm to RoPE"]:::attn
|
| 62 |
-
qkv -->|"v"| vm["+ value-PLE (per-layer)<br/>+ value-residual (layer-0 v)"]:::attn
|
| 63 |
-
qk --> sdpa{{"scaled dot-product<br/>attention"}}:::attn
|
| 64 |
-
vm --> sdpa
|
| 65 |
-
sdpa --> wo["output projection"]:::attn
|
| 66 |
-
x --> a1(["+"]):::res
|
| 67 |
-
wo --> a1
|
| 68 |
-
a1 --> n2["RMSNorm"]:::norm
|
| 69 |
-
n2 --> ffn["SwiGLU FFN<br/>320 to 768 to 320"]:::ffn
|
| 70 |
-
a1 --> a2(["+"]):::res
|
| 71 |
-
ffn --> a2
|
| 72 |
-
a2 --> xo(["x out"]):::res
|
| 73 |
-
end
|
| 74 |
-
|
| 75 |
-
classDef io fill:#ffb73d,stroke:#fff,color:#0a0703,font-weight:bold
|
| 76 |
-
classDef emb fill:#e08a2b,stroke:#ffd98a,color:#0a0703
|
| 77 |
-
classDef core fill:#c4631a,stroke:#ffd98a,color:#fff
|
| 78 |
-
classDef attn fill:#1f4e6b,stroke:#5ad1ff,color:#dff4ff
|
| 79 |
-
classDef ffn fill:#5c3a0c,stroke:#ffb73d,color:#ffd98a
|
| 80 |
-
classDef norm fill:#231603,stroke:#a86d18,color:#ffd98a
|
| 81 |
-
classDef res fill:#5ad1ff,stroke:#fff,color:#0a0703,font-weight:bold
|
| 82 |
-
```
|
| 83 |
-
|
| 84 |
Pre-norm residual blocks: `x += Attn(RMSNorm(x))`, then `x += SwiGLU(RMSNorm(x))`. Layer-0's attention
|
| 85 |
values feed the value-residual mix in every later layer, and each layer also adds its own low-rank value-PLE.
|
| 86 |
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
language:
|
| 4 |
+
- en
|
| 5 |
+
library_name: mlx
|
| 6 |
+
pipeline_tag: text-generation
|
| 7 |
+
tags:
|
| 8 |
+
- rodan
|
| 9 |
+
- tiny-language-model
|
| 10 |
+
- mlx
|
| 11 |
+
- apple-silicon
|
| 12 |
+
- byte-bpe
|
| 13 |
+
---
|
| 14 |
+
|
| 15 |
# Rodan-10M
|
| 16 |
|
| 17 |
A ~11M-parameter language model trained start to finish on one Apple M2 with MLX. The aim was a tiny model
|
|
|
|
| 60 |
115M tokens that adds LRM, raises the RoPE base from 10k to 200k, and mixes in 21% arithmetic/reasoning data
|
| 61 |
(Falcon's reasoning-in-pretraining idea). That second pass is the 11.46M v6 checkpoint.
|
| 62 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 63 |
Pre-norm residual blocks: `x += Attn(RMSNorm(x))`, then `x += SwiGLU(RMSNorm(x))`. Layer-0's attention
|
| 64 |
values feed the value-residual mix in every later layer, and each layer also adds its own low-rank value-PLE.
|
| 65 |
|