MixerLoop

Canonical GDN and MixerLoop T4 checkpoints, organized by architecture, dataset, processed-token budget and training seed. Directories containing only .gitkeep are planned experiments, not available models. Historical releases and their original evaluation results are preserved unchanged under legacy/.

Loading

These are standard HF weights, but the architectures require FLA and the MixerLoop model registration code. Install the dependencies from MixerLoop commit 24b6a923 and run from that checkout (or install it as a package). Plain Transformers without these registrations does not recognize these model types.

import fla.models
import custom_models
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "ruhai-lin/MixerLoop"
run = "mixerloop-13m/climbmix-10B-s1337"
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=run)
model = AutoModelForCausalLM.from_pretrained(repo, subfolder=run)

Tokenizer and generation files are the actual save_pretrained() outputs. Training checkpoints, optimizer states and logs are not published here.

Evaluation and budgets

eval/core_eval.csv contains all 22 Karpathy CORE tasks and the Core_v2 aggregate, stored as fractions, not percentages. Evaluation uses the canonical evaluator at the commit above: official Karpathy bundle, in-memory v2 baseline corrections (CommonsenseQA 40.3%, LSAT AR 25%, language identification 25%), FP32 weights and BF16 autocast. Historical scores in legacy/ retain their original protocols and should not be assumed to be CORE v2.

First-batch runs use context 1024 and global batch 128. The 1B runs contain 7,630 optimizer steps (1,000,079,360 processed tokens); the 10B run contains 76,294 steps (10,000,007,168 processed tokens), all seed 1337. Actual parameters: MixerLoop-13m 12,896,380; MixerLoop-100m 100,447,266; GDN-100m 100,444,194. Other directories are reserved experiment labels, not claims about completed training or measured parameter counts.