py-edu-gemma-15m
Two 15M-parameter Gemma-style decoder-only models trained from scratch on Python functions, used to compare a dense feed-forward block against a top-2 mixture-of-experts block at the same per-token compute.
| dense | mixture-of-experts | |
|---|---|---|
| inference weights | dense.weights.pt |
moe.weights.pt |
| parameters, total | 13,051,400 | 15,425,056 |
| parameters per token | 13,051,400 | 13,065,760 |
| validation loss | 1.0639 | 1.0637 |
| token accuracy | 0.7494 | 0.7495 |
| HumanEval+ mini, base pass@1 (docstring) | 15/164 | 16/164 |
| HumanEval+ mini, base pass@1 (stripped) | 19/164 | 18/164 |
| training time, 4 epochs, RTX 3090 | 7h57m | 12h27m |
The MoE block is a shared FFN of width 768 plus three experts of width 384 with top-2 routing, so each token computes 768 + 2x384 = 1536, exactly the dense FFN width. The extra 2.4M parameters are storage: at this scale the two models are indistinguishable, and the MoE is 1.6x slower per step.
Files
dense.weights.pt,moe.weights.ptβ inference weights only (float32,{"model": state_dict}, plain keys, nomodule.prefix).dense.best.pt,moe.best.ptβ the exactbest.pttraining artifacts, including the AdamW optimizer state.
Usage
import torch
state = torch.load("dense.weights.pt", map_location="cpu", weights_only=True)["model"]
The model expects a prompt containing imports, the function signature, and optionally
a docstring; it completes the body. Tokenizer: byte-level BPE, 8192 tokens plus
<pad> at id 8192, <eos> at id 0.
Training
- corpus:
Avelina/python-edu-cleaned(relaxed extraction, docstrings optional), 4 epochs, 3.35M training functions - architecture: hidden 256, 8 layers, 4 query heads / 2 KV heads, head_dim 64, 512 context, sliding-window/global pattern
sssg - batch 64, AdamW at 6e-4 with linear warmup and decay, bf16, seed 42
- best checkpoint selected by validation loss
HumanEval+ mini is 164 problems at pass@1 with one greedy sample; the 95% interval at that size is roughly +/-7 problems.
License
The training data is ODC-BY. No separate license is asserted for these weights; they are research artifacts, not production models.