---
license: apache-2.0
tags:
- pytorch
- transformer
- mamba
- hybrid
- matryoshka
- nanochat
- adaptive-compute
pipeline_tag: text-generation
---

# 🌀 Adamba: Adaptive Mamba

> **Ad**aptive **Mamba**: Elastic compute with dynamic Matryoshka scaling

**Project location [unixsysdev/adamba](https://github.com/unixsysdev/adamba)**

## Available Checkpoints

| Variant | Parameters | Dim | Features | Status | Download |
|---------|------------|-----|----------|--------|----------|
| phase1_6b_base | 6.4B | 2048 | mamba_integration | ✅ | [Download](./checkpoints/phase1_6b_base.pt) |
| phase2_6b_matryoshka | 6.4B | 2048 | matryoshka, early_exit | ⏳ | — |
| phase3_9b_matryoshka | 9.3B | 2560 | matryoshka, early_exit | ⏳ | — |
| phase3_20b_matryoshka | 20B | 4096 | matryoshka, early_exit | ⏳ | — |
| sft_20b | 20B | 4096 | matryoshka, early_exit, sft | ⏳ | — |
| rl_20b | 20B | 4096 | matryoshka, early_exit, rl_agent | ⏳ | — |

## Architecture Overview

Adamba combines three efficiency techniques:

| Technique | Implementation | Purpose |
|-----------|----------------|---------|
| **Matryoshka (MRL)** | Width: 128 → 4096 per layer | Elastic compute |
| **Early Exit** | ConfidenceGate per layer | Skip when confident |
| **Static SSM** | Mamba at full dim | Stable memory backbone |

```
┌─────────────────────────────────────────────────┐
│  PROMPT → LayerDimPredictor → [dim per layer]   │
│                                                 │
│  Attention + MLP: Dynamic (Matryoshka sliced)   │
│  Mamba:           Static (full dim)             │
│                                                 │
│  Gate > 0.95 → EXIT EARLY                       │
│  Gate < 0.50 → EXPAND remaining layers          │
└─────────────────────────────────────────────────┘
```

## Training Pipeline

```
nanochat-d32 (1.9B)
    ↓ Surgery (add 32 Mamba layers)
Phase 1: 6.4B  (dim=2048)  ← Mamba integration
    ↓ Enable Matryoshka
Phase 2: 6.4B  (dim=2048)  ← Full training
    ↓ Progressive expand
Phase 3: 9.3B → 20B (dim=4096)
    ↓ Fine-tuning
SFT: Instruction tuning
RL:  Agent capabilities
```

## Model Details

- **Base**: [karpathy/nanochat-d32](https://huggingface.co/karpathy/nanochat-d32)
- **Architecture**: 64 blocks (32 Attention + 32 Mamba interleaved)
- **Vocabulary**: 65,536 tokens  
- **Matryoshka Dims**: [128, 256, 512, 1024, 2048, 4096]

## Usage

```python
# Coming soon - inference code
# See: https://github.com/unixsysdev/adamba
```

## Links

- 📂 **GitHub**: [unixsysdev/adamba](https://github.com/unixsysdev/adamba)
- 📊 **Training**: [WandB](https://wandb.ai/dalletest123/nano-fractal)

## License

Apache 2.0