MuLoCo under non-IID data: run logs and checkpoints

Run logs and final checkpoints for the "Muon Workers Drift Further: Data Heterogeneity in Low-Communication Training with Muon" project studying whether MuLoCo (Muon as DiLoCo's inner optimizer) keeps its advantage over DiLoCo when workers train on different text domains. Code, experiment design and results: https://github.com/abhay1105/muloco-data-heterogeneity

Layout

runs/<run_id>/ holds, per run:

  • config.json: full run configuration
  • metrics.jsonl: train loss, per-domain validation loss and diagnostics over training (one JSON object per line)
  • summary.json: final per-domain evaluation (written last, marks the run as complete)
  • final.pt: final (outer) model weights, bf16 state dict, ~51M parameters (absent for short tuning runs)

CHECKPOINTS.sha256 lists the SHA-256 of every checkpoint (the same file is committed to the GitHub repo).

Loading a checkpoint

import json, torch
from src.model import GPT          # from the GitHub repo
cfg = json.load(open("runs/<run_id>/config.json"))
model = GPT(cfg["vocab_size"], cfg["n_layer"], cfg["n_embd"], cfg["n_head"], cfg["seq_len"])
model.load_state_dict({k: v.float() for k, v in torch.load("runs/<run_id>/final.pt").items()})
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support