ANLP Assignment 2, Task 1: Mixture-of-Experts FFN ablation
Decoder-only transformers (d_model 384, 8 layers) trained from scratch for vi->en and ja->en translation on belumind/en-vi-ja-curated-500k-triplets, differing only in the feed-forward layer:
- v1: dense 2-layer MLP
- v2: 4 experts, top-1
- v3: 4 experts, top-2
- v4: 1 shared + 3 routed experts, top-1
- v5: 4 experts, top-2, active parameters matched to v1
| variant | total M | active M | PPL | PPL vi | PPL ja | BLEU | BLEU vi | BLEU ja | ROUGE-L | train h | tok/s | peak MB |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| v1 dense MLP | 26.45 | 26.45 | 4.04 | 3.54 | 4.61 | 41.19 | 46.12 | 36.20 | 67.06 | 0.60 | 101,434 | 4,859 |
| v2 4E top-1 | 26.46 | 19.38 | 4.26 | 3.69 | 4.92 | 39.98 | 44.85 | 35.06 | 66.12 | 0.69 | 87,472 | 4,647 |
| v3 4E top-2 | 26.46 | 21.74 | 4.07 | 3.57 | 4.63 | 40.84 | 45.32 | 36.26 | 66.80 | 0.72 | 85,305 | 5,010 |
| v4 1S+3E top-1 | 26.46 | 21.74 | 4.18 | 3.63 | 4.81 | 40.50 | 45.16 | 35.77 | 66.47 | 0.70 | 87,321 | 4,863 |
| v5 4E top-2 (active=dense) | 35.90 | 26.46 | 4.02 | 3.52 | 4.60 | 41.49 | 46.31 | 36.64 | 67.08 | 0.76 | 80,410 | 5,460 |
<variant>/best.pt holds the best-validation weights (the ones evaluated) and
final.pt the weights after the full budget; both are {'model': state_dict, 'config': {'raw': yaml config, 'vocab_size': ...}}. The model code is in the
submission repository.