py-edu-gemma-15m

Two 15M-parameter Gemma-style decoder-only models trained from scratch on Python functions, used to compare a dense feed-forward block against a top-2 mixture-of-experts block at the same per-token compute.

dense mixture-of-experts
inference weights dense.weights.pt moe.weights.pt
parameters, total 13,051,400 15,425,056
parameters per token 13,051,400 13,065,760
validation loss 1.0639 1.0637
token accuracy 0.7494 0.7495
HumanEval+ mini, base pass@1 (docstring) 15/164 16/164
HumanEval+ mini, base pass@1 (stripped) 19/164 18/164
training time, 4 epochs, RTX 3090 7h57m 12h27m

The MoE block is a shared FFN of width 768 plus three experts of width 384 with top-2 routing, so each token computes 768 + 2x384 = 1536, exactly the dense FFN width. The extra 2.4M parameters are storage: at this scale the two models are indistinguishable, and the MoE is 1.6x slower per step.

Files

  • dense.weights.pt, moe.weights.pt β€” inference weights only (float32, {"model": state_dict}, plain keys, no module. prefix).
  • dense.best.pt, moe.best.pt β€” the exact best.pt training artifacts, including the AdamW optimizer state.

Usage

import torch

state = torch.load("dense.weights.pt", map_location="cpu", weights_only=True)["model"]

The model expects a prompt containing imports, the function signature, and optionally a docstring; it completes the body. Tokenizer: byte-level BPE, 8192 tokens plus <pad> at id 8192, <eos> at id 0.

Training

  • corpus: Avelina/python-edu-cleaned (relaxed extraction, docstrings optional), 4 epochs, 3.35M training functions
  • architecture: hidden 256, 8 layers, 4 query heads / 2 KV heads, head_dim 64, 512 context, sliding-window/global pattern sssg
  • batch 64, AdamW at 6e-4 with linear warmup and decay, bf16, seed 42
  • best checkpoint selected by validation loss

HumanEval+ mini is 164 problems at pass@1 with one greedy sample; the 95% interval at that size is roughly +/-7 problems.

License

The training data is ODC-BY. No separate license is asserted for these weights; they are research artifacts, not production models.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Dataset used to train Blankyy/py-edu-gemma-15m