---
language:
- en
pipeline_tag: fill-mask
library_name: transformers
datasets:
- allenai/c4
---

# SpliNet

**SpliNet: A Zero-Parameter B-Spline Transformer with Linear Complexity**

SpliNet replaces learned self-attention token mixing with a fixed order-2 single-sided cardinal B-spline operator.

This model was pretrained from scratch on exactly **2,000,000,000 C4 tokens** using the dedicated SpliNet tokenizer.

## Architecture

- Layers: 12
- Hidden size: 768
- Heads: 12
- FFN width: 3072
- Sequence length: 512
- Vocabulary: 32000
- Spline order: 2
- Spline radius: 16
- Trainable mixer parameters: 0
- Total parameters: 82,894,592
- Trainable parameters: 82,894,592

## Pretraining

- Training tokens: 2,000,000,000
- Validation tokens: 5,120,000
- Objective: masked language modeling
- Masked positions: 77/512
- Optimizer: AdamW
- Precision: BF16
- Hardware: NVIDIA A100-SXM4-80GB

## Final validation

- MLM loss: 4.614502
- MLM perplexity: 100.937575

Load with `trust_remote_code=True`.

OpenReview: https://openreview.net/forum?id=nWHnuiEF3C
GitHub: https://github.com/AngshulMajumdar/SpliNet