Card: the planned model uses gated attention and QK-norm (now the default); 105M parameters
Browse files
README.md
CHANGED
|
@@ -32,10 +32,11 @@ research checkpoints of the 17M-parameter experiments (`checkpoints/17m/`, below
|
|
| 32 |
|
| 33 |
## Model (planned)
|
| 34 |
|
| 35 |
-
- **Size:** 0.1 billion parameters (
|
| 36 |
- **Architecture:** a hybrid of TONN and attention. Every third layer is a TONN layer: small learned circuits
|
| 37 |
(convolutions, running sums, state spaces, memories) whose layout TONN's search adjusts during training. The other
|
| 38 |
-
layers are
|
|
|
|
| 39 |
- **Built-in memory circuits** in every TONN layer:
|
| 40 |
- retrieval from a read-only library, with the model's own embeddings;
|
| 41 |
- a read-write message board shared by several copies of the model, so agents can talk.
|
|
|
|
| 32 |
|
| 33 |
## Model (planned)
|
| 34 |
|
| 35 |
+
- **Size:** 0.1 billion parameters (105M with the planned vocabulary, about 94M with this 32k tokenizer).
|
| 36 |
- **Architecture:** a hybrid of TONN and attention. Every third layer is a TONN layer: small learned circuits
|
| 37 |
(convolutions, running sums, state spaces, memories) whose layout TONN's search adjusts during training. The other
|
| 38 |
+
layers are multi-head attention with two parts from Qwen3.5: gated attention and QK-norm. Each layer is followed by
|
| 39 |
+
a gated MLP.
|
| 40 |
- **Built-in memory circuits** in every TONN layer:
|
| 41 |
- retrieval from a read-only library, with the model's own embeddings;
|
| 42 |
- a read-write message board shared by several copies of the model, so agents can talk.
|