File size: 2,168 Bytes
58f60c2 36e6031 58f60c2 50148ae 58f60c2 d156b68 58f60c2 d156b68 3bc475b d156b68 58f60c2 3c744ab 31e4e3d 4ccd8d7 58f60c2 4ccd8d7 58f60c2 8fa0a12 48c99a4 8fa0a12 48c99a4 c4167b6 58f60c2 9dc4433 58f60c2 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 | ---
license: apache-2.0
datasets:
- Salesforce/wikitext
language:
- en
metrics:
- perplexity
pipeline_tag: text-generation
tags:
- wavelet
- attention-free
- no-attention
- attentionfree
- FWHT
- walsh-hadamard
- PKM
- product-key-memory
- causal-lm
- text-generation
- sub-quadratic
---
# WaveletLM
WaveletLM is a fully causal, attention-free language model that mixes tokens through learned lifting wavelet decomposition, a Fast Walsh-Hadamard Transform, per-scale gated spectral mixing with SwiGLU activation, an inverse FWHT, and wavelet reconstruction. Combined with expanded MLPs and sparse product-key memory, this yields an architecture with no attention and O(n log n) scaling in sequence length.
Full code, training details, ablations, and documentation: [github.com/ramongougis/WaveletLM](https://github.com/ramongougis/WaveletLM)
## Results
| Dataset | Params | Perplexity | BPB |
|---------|--------|------------|-----|
| WikiText-103 | 883M | 23.8 | 1.0140 |
| PG-19 (1 epoch) | 808M | 27.4 | 1.0853 |
## How to Use
```python
import torch
from huggingface_hub import hf_hub_download
# Download the checkpoint
ckpt_path = hf_hub_download(repo_id="ragou19/WaveletLM", filename="best_model.pt")
```
Then follow the instructions in the GitHub repo to load and run:
https://github.com/ramongougis/WaveletLM
## Architecture
## Training
- Trained on a single RTX 5090 for 5 epochs
- WikiText-103: best PPL of 23.749 with mean PPL of 23.818 across 3 seeds.
- PG-19: PPL of 27.40 (single seed).
- VRAM required: 18.3 GB.
- Time to train: 16 hours 15 minutes.
## Generation
- VRAM: 5.0 GB by default, 4.5 GB with `--ptq8` enabled.
- Can set `compile:false` to save 0.5-1 GB, but it's slower.
- 28.8 tokens/s. on a 5090 by default.
- Future enhancements expected to increase speed by up to 120%.
## Logs
See <a href="https://github.com/ramongougis/WaveletLM/blob/main/runs.md">runs.md</a> for the full training history.
## License
Apache 2.0. See <a href="https://github.com/ramongougis/WaveletLM/blob/main/LICENSE">LICENSE</a>.
|