AETHORIA-AI/TR-HASH-MoE-200M-160B-SFT

TR-HASH MoE 200M — 160B-source Full SFT v2

Full-parameter instruction SFT of AETHORIA-AI/TR-HASH-MoE-200M-160B-Refinement on the audited AETHORIA-AI/TR-HASH-MoE-200M-SFT-v2-300K. This release is not LoRA or QLoRA: all 201.2M model parameters were trainable.

The root model.safetensors is the epoch 3 / step 5,982 checkpoint, selected for the strongest full-PIQA result and lowest held-out SFT loss across the three epochs.

Results

Epoch	Step	Held-out SFT loss	SFT ppl	PIQA acc	PIQA acc_norm
1	1,994	0.990943	2.69	67.90%	68.93%
2	3,988	0.963912	2.62	67.85%	68.82%
3	5,982	0.959617	2.61	68.01%	69.10%

Released checkpoint benchmark panel

Benchmark	Split	Examples	Accuracy	Accuracy (length-normalized)	Evaluation backend
PIQA	validation	1,838	68.01%	69.10%	PyTorch FP16, custom Triton
ARC-Easy	test	2,376	57.24%	52.86%	MLX FP16
ARC-Challenge	test	1,172	27.13%	29.01%	MLX FP16
ARC Combined (micro)	test	3,548	47.29%	44.98%	Derived from both ARC test splits
HellaSwag	validation	10,042	33.21%	38.74%	MLX FP16

All benchmark evaluations use zero-shot causal continuation log-likelihood, no chat template and a maximum sequence length of 2,048. Accuracy selects the choice with the highest total continuation log-likelihood; the normalized metric selects by mean continuation log-likelihood per scored token. PIQA was evaluated from the native epoch-3 checkpoint. ARC and HellaSwag were evaluated from an FP16 MLX conversion of the same root F32 SafeTensors weights. ARC Combined is the micro-average over all 3,548 ARC-Easy and ARC-Challenge test examples, not the arithmetic mean of the two percentages. Machine-readable reports are published under reports/sft-v2-300k/.

Compact-model comparison

Model	Parameters	PIQA acc	ARC-Easy acc	ARC-Challenge acc	Combined ARC acc	HellaSwag acc
TR-HASH MoE 200M Full SFT	201.2M	68.01%	57.24%	27.13%	47.29%	33.21%
GPT-2 Small	124M	62.89%	43.81%	19.03%	≈35.63%	28.92%
OPT-125M	125M	63.00%	43.60%	19.10%	≈35.51%	29.20%
Pythia-160M	160M	62.73%	43.52%	18.77%	35.34%	—

Combined ARC is weighted by the public test-set sizes (2,376 ARC-Easy and 1,172 ARC-Challenge examples). GPT-2 Small and OPT reference scores come from the AMD-LLM lm-evaluation-harness comparison; their combined values are approximate because the published component scores are rounded. Pythia-160M uses EleutherAI's official zero-shot result. The comparison is informative rather than a claim of bit-identical evaluation runtimes: TR-HASH was scored by the repository's causal-choice evaluator, whereas the references were reported through lm-evaluation-harness.

Training recipe

Setting	Value
Method	Full-parameter supervised fine-tuning
Source	Refinement step 8,156 (about 162B prior token exposures)
Dataset	300,000 train / 3,000 held-out examples
Tokenized corpus	202,948,693 train tokens; no truncation
Supervision	Final assistant turn only; prior assistant turns masked
Epochs	3
Context	2,048 tokens
Tokenizer	TR-HASH 32,000-token vocabulary; EOS </s> (ID 0)
Optimizer	AdamW, betas 0.9 / 0.95, weight decay 0.1
LR	2e-5 peak, 3% warmup, continuous cosine decay
Precision	BF16 training
Root SafeTensors precision	float32
Kernels	Liger required; custom Triton enabled

Architecture and loading

201.2M parameters, 16 decoder layers, GQA (14 query heads / 2 KV heads), four stored deterministic token-ID-routed experts with top-2 activation, an always-on shared SwiGLU path and tied embeddings. The persisted multi-hash routing tables are part of the checkpoint.

The repository includes an autonomous Transformers adapter. Load it with:

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "AETHORIA-AI/TR-HASH-MoE-200M-160B-SFT"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)

License

The released SFT checkpoint is Apache-2.0. Source datasets retain their own licenses and terms; see the dataset manifest for the per-source audit.