AvnehBhatia commited on
Commit
36ce8b6
·
verified ·
1 Parent(s): ce2a9da

Card: the planned model uses gated attention and QK-norm (now the default); 105M parameters

Browse files
Files changed (1) hide show
  1. README.md +3 -2
README.md CHANGED
@@ -32,10 +32,11 @@ research checkpoints of the 17M-parameter experiments (`checkpoints/17m/`, below
32
 
33
  ## Model (planned)
34
 
35
- - **Size:** 0.1 billion parameters (101M with the planned vocabulary, about 90M with this 32k tokenizer).
36
  - **Architecture:** a hybrid of TONN and attention. Every third layer is a TONN layer: small learned circuits
37
  (convolutions, running sums, state spaces, memories) whose layout TONN's search adjusts during training. The other
38
- layers are standard multi-head attention. Each layer is followed by a gated MLP.
 
39
  - **Built-in memory circuits** in every TONN layer:
40
  - retrieval from a read-only library, with the model's own embeddings;
41
  - a read-write message board shared by several copies of the model, so agents can talk.
 
32
 
33
  ## Model (planned)
34
 
35
+ - **Size:** 0.1 billion parameters (105M with the planned vocabulary, about 94M with this 32k tokenizer).
36
  - **Architecture:** a hybrid of TONN and attention. Every third layer is a TONN layer: small learned circuits
37
  (convolutions, running sums, state spaces, memories) whose layout TONN's search adjusts during training. The other
38
+ layers are multi-head attention with two parts from Qwen3.5: gated attention and QK-norm. Each layer is followed by
39
+ a gated MLP.
40
  - **Built-in memory circuits** in every TONN layer:
41
  - retrieval from a read-only library, with the model's own embeddings;
42
  - a read-write message board shared by several copies of the model, so agents can talk.