Update README.md
Browse files
README.md
CHANGED
|
@@ -15,6 +15,7 @@ tags:
|
|
| 15 |
- litgpt
|
| 16 |
- custom-architecture
|
| 17 |
- custom_code
|
|
|
|
| 18 |
|
| 19 |
model-index:
|
| 20 |
- name: ObsidianSmall-Base
|
|
@@ -151,8 +152,16 @@ It is the first public checkpoint in this model family built entirely from **Mul
|
|
| 151 |
| Training tokens | 3,999,989,760 |
|
| 152 |
| Validation loss / PPL | 2.592 / 13.359 |
|
| 153 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 154 |
Multiscreen replaces conventional self-attention and MLP/FFN sublayers with its own sequence-mixing blocks.
|
| 155 |
|
|
|
|
|
|
|
|
|
|
| 156 |
## Benchmarks
|
| 157 |
|
| 158 |
Zero-shot task results were measured with `lm-evaluation-harness`.
|
|
|
|
| 15 |
- litgpt
|
| 16 |
- custom-architecture
|
| 17 |
- custom_code
|
| 18 |
+
|
| 19 |
|
| 20 |
model-index:
|
| 21 |
- name: ObsidianSmall-Base
|
|
|
|
| 152 |
| Training tokens | 3,999,989,760 |
|
| 153 |
| Validation loss / PPL | 2.592 / 13.359 |
|
| 154 |
|
| 155 |
+
## Architecture
|
| 156 |
+
|
| 157 |
+
This model implements and adapts the **Multiscreen** architecture introduced in
|
| 158 |
+
[Screening Is Enough](https://arxiv.org/abs/2604.01178) by Ken M. Nakanishi.
|
| 159 |
+
|
| 160 |
Multiscreen replaces conventional self-attention and MLP/FFN sublayers with its own sequence-mixing blocks.
|
| 161 |
|
| 162 |
+
The implementation may differ from the architecture and training configuration
|
| 163 |
+
described in the original paper.
|
| 164 |
+
|
| 165 |
## Benchmarks
|
| 166 |
|
| 167 |
Zero-shot task results were measured with `lm-evaluation-harness`.
|