Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -20,7 +20,7 @@ Public experimental byte-level causal language model. Training is in progress an
|
|
| 20 |
- Approximately 1B stored parameters and approximately one third active per token.
|
| 21 |
- UTF-8 bytes, vocabulary 256, no external tokenizer.
|
| 22 |
- Four local causal Transformer layers process every byte.
|
| 23 |
-
- Entropy boundaries compress the stream to about one patch per eight bytes, with capacity for 192 patches per 1024-byte sequence.
|
| 24 |
- Three independent eight-layer global routes are trained cyclically, one full batch per route.
|
| 25 |
- The previous completed global patch conditions the local prediction, preventing future-token leakage.
|
| 26 |
- Muon optimizes hidden matrices; AdamW optimizes embeddings and normalization parameters.
|
|
|
|
| 20 |
- Approximately 1B stored parameters and approximately one third active per token.
|
| 21 |
- UTF-8 bytes, vocabulary 256, no external tokenizer.
|
| 22 |
- Four local causal Transformer layers process every byte.
|
| 23 |
+
- Entropy boundaries compress the stream to about one patch per eight bytes, with capacity for 192 patches per 1024-byte sequence. Over-cap boundaries are uniformly coalesced across the sequence instead of truncating its tail.
|
| 24 |
- Three independent eight-layer global routes are trained cyclically, one full batch per route.
|
| 25 |
- The previous completed global patch conditions the local prediction, preventing future-token leakage.
|
| 26 |
- Muon optimizes hidden matrices; AdamW optimizes embeddings and normalization parameters.
|