metadata
license: mit
tags:
- sparseflow
- sparse-attention
- efficient-nlp
datasets:
- gsm8k
- lighteval/MATH
- allenai/ai2_arc
- tau/commonsense_qa
- piqa
- allenai/sciq
- trivia_qa
- nq_open
- wikitext
SparseFlow v8
Efficient language model with sparse attention and persistent memory.
📊 REAL Measured Metrics
| Metric | Value |
|---|---|
| Parameters | 71,359,746 |
| Perplexity | 14.77 |
| Attention Sparsity | 87.5% |
| Channel Sparsity | 75.0% |
| Peak Memory | 3.67 GB |
🏗️ Architecture
- Sparse Token Attention: Attends to top-64 tokens per position
- Sparse Channel FFN: Activates top-128 channels
- Persistent Memory: 20,000 memory vectors
- 8 Transformer layers with 512 dim
📚 Training Data
Open source datasets only:
- GSM8K, MATH (mathematics)
- ARC, OpenBookQA, SciQ (science & reasoning)
- CommonsenseQA, PIQA (common sense)
- TriviaQA, Natural Questions (factual)
- WikiText-103 (language modeling)
👨💻 Author
Logo (Mike Amega) — Ame Web Studio