File size: 1,109 Bytes
4ec5e47 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 | ---
language:
- en
pipeline_tag: fill-mask
library_name: transformers
datasets:
- allenai/c4
---
# SpliNet
**SpliNet: A Zero-Parameter B-Spline Transformer with Linear Complexity**
SpliNet replaces learned self-attention token mixing with a fixed order-2 single-sided cardinal B-spline operator.
This model was pretrained from scratch on exactly **2,000,000,000 C4 tokens** using the dedicated SpliNet tokenizer.
## Architecture
- Layers: 12
- Hidden size: 768
- Heads: 12
- FFN width: 3072
- Sequence length: 512
- Vocabulary: 32000
- Spline order: 2
- Spline radius: 16
- Trainable mixer parameters: 0
- Total parameters: 82,894,592
- Trainable parameters: 82,894,592
## Pretraining
- Training tokens: 2,000,000,000
- Validation tokens: 5,120,000
- Objective: masked language modeling
- Masked positions: 77/512
- Optimizer: AdamW
- Precision: BF16
- Hardware: NVIDIA A100-SXM4-80GB
## Final validation
- MLM loss: 4.614502
- MLM perplexity: 100.937575
Load with `trust_remote_code=True`.
OpenReview: https://openreview.net/forum?id=nWHnuiEF3C
GitHub: https://github.com/AngshulMajumdar/SpliNet
|