JuliaFluxGPT-distilled
Cross-species knowledge distillation: two JuliaFluxGPT siblings — v1 (JuliaSLM fusion, val_loss=3.687) and v2 (Pythia-14m fusion, val_loss=3.856) — serve as co-teachers for a student model. The student inherits v1's perplexity advantage and v2's linguistic quality (superior grammar, coherence, and syntactic complexity).
Why Distillation?
Weight-level fusion between v1 and v2 fails catastrophically — even 50/50 alpha blend produces loss=10.3. Evolutionary per-layer search, SLERP, and Kuramoto sync all fail. The models occupy separate loss basins after being fused from different parent species (JuliaSLM vs Pythia-14m). Knowledge distillation bypasses this by operating on output distributions instead of weight matrices.
Architecture
|
|
| Parameters |
~23M |
| Embedding dim |
512 |
| Layers |
8 |
| Attention |
GQA (8 query, 2 KV heads) |
| Head dim |
64 |
| FFN |
SwiGLU (1344 inner) |
| Normalization |
RMSNorm (pre-norm) |
| Position encoding |
RoPE (base=10000) |
| Context length |
256 |
| Vocab |
2000 (BPE) |
| Weight tying |
Yes (embedding = output projection) |
Training
|
|
| Method |
Knowledge distillation (warm start) |
| Teacher 1 |
JuliaFluxGPT-fused v1 (JuliaSLM fusion, val_loss=3.687) |
| Teacher 2 |
JuliaFluxGPT-fused v2 (Pythia fusion, val_loss=3.856) |
| Student init |
v1 weights (warm start) |
| Loss |
0.35 KL(s||v1) + 0.35 KL(s||v2) + 0.30 CE(s, targets) |
| Temperature |
3.0 |
| Steps |
3000 (best at step 2600) |
| LR |
3e-4 (cosine decay) |
| Optimizer |
AdamW (weight_decay=0.01) |
| Val loss |
3.687 (beats both parents) |
Scaling Context
| Model |
Params |
d_model |
Val Loss |
Method |
| MicroJulia |
1M |
192 |
— |
Baseline |
| JuliaSLM |
5M |
256 |
3.54 |
Baseline |
| SymbioSLM |
5M |
256 |
3.48 |
Multi-organelle |
| MonarchSLM |
5M |
256 |
3.51 |
Monarch matrices |
| JuliaFluxGPT-fused (v1) |
23M |
512 |
3.698 |
JuliaSLM fusion |
| JuliaFluxGPT-fused-v2 |
23M |
512 |
3.873 |
Pythia fusion |
| JuliaFluxGPT-distilled |
23M |
512 |
3.687 |
v1+v2 distillation |
Files
| File |
Description |
juliaflux_distilled_warm_best.pt |
Best checkpoint (step 2600, val_loss=3.687) |
juliaflux_model.py |
Model definition (JuliaFluxGPT class) |
vocab.json |
BPE vocabulary (2000 tokens) |
merges.txt |
BPE merge rules |
Links
