|
|
--- |
|
|
license: apache-2.0 |
|
|
tags: |
|
|
- pytorch |
|
|
- transformer |
|
|
- mamba |
|
|
- hybrid |
|
|
- matryoshka |
|
|
- nanochat |
|
|
- adaptive-compute |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
|
|
|
# 🌀 Adamba: Adaptive Mamba |
|
|
|
|
|
> **Ad**aptive **Mamba**: Elastic compute with dynamic Matryoshka scaling |
|
|
|
|
|
**Project location [unixsysdev/adamba](https://github.com/unixsysdev/adamba)** |
|
|
|
|
|
## Available Checkpoints |
|
|
|
|
|
| Variant | Parameters | Dim | Features | Status | Download | |
|
|
|---------|------------|-----|----------|--------|----------| |
|
|
| phase1_6b_base | 6.4B | 2048 | mamba_integration | ✅ | [Download](./checkpoints/phase1_6b_base.pt) | |
|
|
| phase2_6b_matryoshka | 6.4B | 2048 | matryoshka, early_exit | ⏳ | — | |
|
|
| phase3_9b_matryoshka | 9.3B | 2560 | matryoshka, early_exit | ⏳ | — | |
|
|
| phase3_20b_matryoshka | 20B | 4096 | matryoshka, early_exit | ⏳ | — | |
|
|
| sft_20b | 20B | 4096 | matryoshka, early_exit, sft | ⏳ | — | |
|
|
| rl_20b | 20B | 4096 | matryoshka, early_exit, rl_agent | ⏳ | — | |
|
|
|
|
|
## Architecture Overview |
|
|
|
|
|
Adamba combines three efficiency techniques: |
|
|
|
|
|
| Technique | Implementation | Purpose | |
|
|
|-----------|----------------|---------| |
|
|
| **Matryoshka (MRL)** | Width: 128 → 4096 per layer | Elastic compute | |
|
|
| **Early Exit** | ConfidenceGate per layer | Skip when confident | |
|
|
| **Static SSM** | Mamba at full dim | Stable memory backbone | |
|
|
|
|
|
``` |
|
|
┌─────────────────────────────────────────────────┐ |
|
|
│ PROMPT → LayerDimPredictor → [dim per layer] │ |
|
|
│ │ |
|
|
│ Attention + MLP: Dynamic (Matryoshka sliced) │ |
|
|
│ Mamba: Static (full dim) │ |
|
|
│ │ |
|
|
│ Gate > 0.95 → EXIT EARLY │ |
|
|
│ Gate < 0.50 → EXPAND remaining layers │ |
|
|
└─────────────────────────────────────────────────┘ |
|
|
``` |
|
|
|
|
|
## Training Pipeline |
|
|
|
|
|
``` |
|
|
nanochat-d32 (1.9B) |
|
|
↓ Surgery (add 32 Mamba layers) |
|
|
Phase 1: 6.4B (dim=2048) ← Mamba integration |
|
|
↓ Enable Matryoshka |
|
|
Phase 2: 6.4B (dim=2048) ← Full training |
|
|
↓ Progressive expand |
|
|
Phase 3: 9.3B → 20B (dim=4096) |
|
|
↓ Fine-tuning |
|
|
SFT: Instruction tuning |
|
|
RL: Agent capabilities |
|
|
``` |
|
|
|
|
|
## Model Details |
|
|
|
|
|
- **Base**: [karpathy/nanochat-d32](https://huggingface.co/karpathy/nanochat-d32) |
|
|
- **Architecture**: 64 blocks (32 Attention + 32 Mamba interleaved) |
|
|
- **Vocabulary**: 65,536 tokens |
|
|
- **Matryoshka Dims**: [128, 256, 512, 1024, 2048, 4096] |
|
|
|
|
|
## Usage |
|
|
|
|
|
```python |
|
|
# Coming soon - inference code |
|
|
# See: https://github.com/unixsysdev/adamba |
|
|
``` |
|
|
|
|
|
## Links |
|
|
|
|
|
- 📂 **GitHub**: [unixsysdev/adamba](https://github.com/unixsysdev/adamba) |
|
|
- 📊 **Training**: [WandB](https://wandb.ai/dalletest123/nano-fractal) |
|
|
|
|
|
## License |
|
|
|
|
|
Apache 2.0 |
|
|
|
