AETHORIA-AI/TR-HASH-MoE-200M-130B-Checkpoints

TR-HASH MoE 200M — 70B Unique / 130B Replay Checkpoints

Status: training complete. This is the raw checkpoint backup repository, not the finished model release. It exists as a safety net and a record of the training trajectory — token-pack folders include model weights and resumable optimizer/scheduler state.

• Checkpoints are uploaded automatically at token-pack boundaries and on clean/interrupted shutdown by scripts/sync_checkpoints_to_hf.py.
• Folder names follow {tag}_{step} (token_pack_NNN_STEP, final_STEP, interrupted_STEP).
• The PIQA sweep below is exploratory, not a full validated evaluation suite. Do not treat an individual checkpoint as a finished model release.
• The architecture config is tracked at model_config.yaml in this repo (not embedded in checkpoint.pt).

Exploratory zero-shot checks (not a full evaluation)

Informal checks across the final token-pack checkpoints, via scripts/convert_to_mlx.py + scripts/eval_mlx_zero_shot.py, zero-shot causal-log-likelihood scoring, no chat template. Every row uses the same tokenizer, MLX FP16 inference path, PIQA validation split, and 1,838 examples:

Checkpoint	Tokens trained	Learning rate	PIQA acc	PIQA acc_norm
token_pack_032_132239	104.00B	3.00e-4	0.6736	0.6763
token_pack_033_136371	107.25B	2.91e-4	0.6774	0.6746
token_pack_034_140504	110.50B	2.62e-4	0.6768	0.6703
token_pack_035_144636	113.75B	2.19e-4	0.6741	0.6801
token_pack_036_148769	117.00B	1.67e-4	0.6768	0.6774
token_pack_037_152901	120.25B	1.14e-4	0.6676	0.6697
token_pack_038_157034	123.50B	7.01e-5	0.6578	0.6638
token_pack_039_161166	126.75B	4.04e-5	0.6567	0.6529
token_pack_040_165298 / final	130.00B	3.00e-5	0.6545	0.6561

token_pack_035_144636 has the highest acc_norm in this sweep. The trajectory is not monotonic, and PIQA alone must not be used as a complete model-quality or checkpoint-selection criterion.

The finished base model is published separately as AETHORIA-AI/TR-HASH-MoE-200M-130B.

See Complexity Framework for the training code.