MuLoCo under non-IID data: run logs and checkpoints
Run logs and final checkpoints for the "Muon Workers Drift Further: Data Heterogeneity in Low-Communication Training with Muon" project studying whether MuLoCo (Muon as DiLoCo's inner optimizer) keeps its advantage over DiLoCo when workers train on different text domains. Code, experiment design and results: https://github.com/abhay1105/muloco-data-heterogeneity
Layout
runs/<run_id>/ holds, per run:
config.json: full run configurationmetrics.jsonl: train loss, per-domain validation loss and diagnostics over training (one JSON object per line)summary.json: final per-domain evaluation (written last, marks the run as complete)final.pt: final (outer) model weights, bf16 state dict, ~51M parameters (absent for short tuning runs)
CHECKPOINTS.sha256 lists the SHA-256 of every checkpoint (the same file is committed to the GitHub repo).
Loading a checkpoint
import json, torch
from src.model import GPT # from the GitHub repo
cfg = json.load(open("runs/<run_id>/config.json"))
model = GPT(cfg["vocab_size"], cfg["n_layer"], cfg["n_embd"], cfg["n_head"], cfg["seq_len"])
model.load_state_dict({k: v.float() for k, v in torch.load("runs/<run_id>/final.pt").items()})
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐ Ask for provider support