---
license: apache-2.0
language:
- en
library_name: pytorch
pipeline_tag: text-generation
datasets:
- Salesforce/wikitext
tags:
- deepseek
- deepseek-v3
- mixture-of-experts
- multi-head-latent-attention
- multi-token-prediction
- research
---

# DeepSeek-V3 Compact Proxy — Checkpoints

Trained checkpoints for [deepseek-v3-compact-proxy](https://github.com/syedazeez337/deepseek-v3-compact-proxy),
a mechanism-faithful proxy for DeepSeek-V3's architecture (Multi-head Latent Attention, DeepSeekMoE, Multi-Token
Prediction) — not a reproduction of the 671B model. Every checkpoint here corresponds to a specific, documented
experiment ("gate") in that repository; each gate's specification, implementation, tests, measured results, and
interpretation live in its `experiments_GATE_*.md` file.

Checkpoints are hosted here rather than in the code repository because GitHub blocks any file over 100MB, and
several of these (Configuration B, with optimizer state) are close to 2GB each.

Every checkpoint below is also fully reproducible from the exact command recorded in its gate document — this
Hugging Face copy is a convenience, not the only record.

## Configuration A (~15M parameters, Gates I-N)

| File | Gate | Config | Doc |
|---|---|---|---|
| `compact_v3_wikitext_dense_1m.pt` | I | Dense control, MoE disabled | `experiments_GATE_I_DENSE_CONTROL_1M.md` |
| `compact_v3_wikitext_moe_1m.pt` | J | 4-expert top-1 MoE | `experiments_GATE_J_MOE_COMPARISON_1M.md` |
| `compact_v3_wikitext_moe_1m_gatek.pt` | K | Same as J, entropy-logging rerun | `experiments_GATE_K_LOAD_BALANCING.md` |
| `compact_v3_wikitext_moe_1m_gatel_u00001.pt` | L | `router_bias_update_rate=0.0001` (new default) | `experiments_GATE_L_BIAS_RATE_SWEEP.md` |
| `compact_v3_wikitext_moe_1m_gatel_u003.pt` | L | `router_bias_update_rate=0.003` | `experiments_GATE_L_BIAS_RATE_SWEEP.md` |
| `compact_v3_wikitext_moe_1m_gatel_u01.pt` | L | `router_bias_update_rate=0.01` | `experiments_GATE_L_BIAS_RATE_SWEEP.md` |
| `compact_v3_wikitext_moe_1m_gatem_topk2.pt` | M | `top_k=2` (new default) | `experiments_GATE_M_TOP2_ROUTING.md` |
| `compact_v3_wikitext_moe_1m_gaten_dense1.pt` | N | `n_dense_layers=1`, confounded first attempt (kept for the record) | `experiments_GATE_N_DENSE_LAYER_PREFIX.md` |
| `compact_v3_wikitext_moe_1m_gaten_dense1_matched.pt` | N | `n_dense_layers=1`, corrected (new default) | `experiments_GATE_N_DENSE_LAYER_PREFIX.md` |

## Configuration B (~155M parameters, Gates O-Q)

| File | Gate | Config | Doc |
|---|---|---|---|
| `compact_v3_wikitext_moe_scaleup_1m.pt` | O | Scale-up baseline, `route_scale=1.0` | `experiments_GATE_O_GPU_SCALEUP.md` |
| `compact_v3_wikitext_moe_scaleup_1m_gatep_rs075.pt` | P | `route_scale=0.75` (new default) | `experiments_GATE_P_ROUTE_SCALE.md` |
| `compact_v3_wikitext_moe_scaleup_1m_gatep_rs15.pt` | P | `route_scale=1.5` | `experiments_GATE_P_ROUTE_SCALE.md` |
| `compact_v3_wikitext_moe_scaleup_1m_gatep_rs25.pt` | P | `route_scale=2.5` (V3's own value) | `experiments_GATE_P_ROUTE_SCALE.md` |
| `compact_v3_wikitext_moe_scaleup_1m_gateq_mtp_const.pt` | Q | MTP enabled, constant weight 0.3 | `experiments_GATE_Q_MTP_ANNEALING.md` |
| `compact_v3_wikitext_moe_scaleup_1m_gateq_mtp_anneal.pt` | Q | MTP enabled, annealed 0.3->0.1 (current best, 292.54 PPL) | `experiments_GATE_Q_MTP_ANNEALING.md` |

Gate R (MLA weight absorption) is an inference-path code change validated by unit tests and benchmarks; it did
not produce a training checkpoint.

## Regenerating a checkpoint

Each gate document's "Experiment" section has the exact command. For example, Gate Q's best checkpoint:

```powershell
uv run python v3_cli.py --real-corpus --steps 1954 --checkpoint-every 250 --eval-batches 32 --generate 32 --device cuda --enable-mtp --checkpoint checkpoints/compact_v3_wikitext_moe_scaleup_1m_gateq_mtp_anneal.pt
```