Update README.md

README.md

@@ -27,45 +27,4 @@ The weights in this repository are the checkpoints from the DRMoET paper: two DR
 | `DRMoET-290M-32E` | DRMoET, activation credit | 32 (top-6) + shared | 9 / 1024 | 16,000 | 0.999, 0.001, 1.0, L2 activation norm |
 | `FLAME-MoE-290M-32E` | FLAME-MoE baseline | 32 (top-6) + shared | 9 / 1024 | 16,000 | — |
 
-All models use sequence length 2048, global batch size 1024, learning rate 3e-4, auxiliary load-balancing loss coefficient 0.01, and the `EleutherAI/pythia-12b` tokenizer, trained on DCLM. Each DRMoET checkpoint and its baseline share the same seed (1,234 for 1.7B and 3,407 for 290M-32E).
-
-## Results (1.7B, 0-shot)
-
-| Model | ARC-C | ARC-E | HellaSwag | PIQA | WinoGrande | SciQ | ReCoRD | Avg |
-|---|---|---|---|---|---|---|---|---|
-| FLAME-MoE-1.7B | 34.81 | 69.70 | 49.42 | 75.63 | 59.04 | **90.50** | 84.67 | 66.25 |
-| **DRMoET-1.7B** | **38.31** | **71.84** | **49.90** | **76.66** | **62.51** | 90.40 | **84.79** | **67.77** |
-
-Accuracy (`acc`) from lm-evaluation-harness, final checkpoint (iteration 32,000).
-
-## Results (290M, 32 experts, 0-shot)
-
-| Model | ARC-C | ARC-E | HellaSwag | PIQA | WinoGrande | SciQ | ReCoRD | Avg |
-|---|---|---|---|---|---|---|---|---|
-| FLAME-MoE-290M-32E | 23.72 | 55.72 | 34.73 | 69.26 | 51.14 | 81.70 | 71.47 | 55.39 |
-| DRMoET-290M-32E | 23.46 | 57.03 | 34.90 | 68.50 | 50.99 | 82.10 | 71.45 | 55.49 |
-
-Accuracy (`acc`) from lm-evaluation-harness, final checkpoint (iteration 16,000), evaluated from the released (optimizer-stripped) checkpoints.
-
-## Format
-
-Each folder is a Megatron-LM distributed (`torch_dist`) checkpoint containing **model weights only**; optimizer state has been removed. Load with `--no-load-optim` (and `--no-load-rng`), as for fine-tuning or evaluation.
-
-```
-DRMoET-1.7B/
-├── latest_checkpointed_iteration.txt
-└── iter_0032000/
-```
-
-## Usage
-
-```bash
-git clone --recursive https://github.com/MAPS-research/DRMoET.git && cd DRMoET
-huggingface-cli download TonyTeng/DRMoET --include "DRMoET-1.7B/*" --local-dir weights
-```
-
-Then pass `--load weights/DRMoET-1.7B --no-load-optim --no-load-rng` with the matching model arguments from `configs/model/flame-moe.sh` and the corresponding recipe in `scripts/release/` (e.g. `flame-moe-1.7b-dro-activation.sh`). See the repository README for evaluation with lm-evaluation-harness.
-
-## Citation
-
-<!-- TODO: BibTeX -->
+All models use sequence length 2048, global batch size 1024, learning rate 3e-4, auxiliary load-balancing loss coefficient 0.01, and the `EleutherAI/pythia-12b` tokenizer, trained on DCLM. Each DRMoET checkpoint and its baseline share the same seed (1,234 for 1.7B and 3,407 for 290M-32E).