TR-HASH-MOE-500M-20B

Research pretraining artifact for a 492.1M-parameter decoder-only language model with grouped-query attention, a shared dense MLP path, and deterministic token-ID hashed residual experts.

This is a base language model, not an instruction-tuned chat model. The repository provides both the final raw training checkpoint and a BF16 Safetensors export for direct base-model inference with the custom vllm-i64 runtime. For the instruction-tuned release, use TR-HASH-MOE-500M-HF.

Model summary

Component	Value
Parameters	492.1M
Transformer layers	24
Hidden size	1,024
Attention	GQA, 16 query heads / 4 KV heads
Vocabulary	32,000
Training context	2,048 tokens
Shared MLP	SwiGLU, width 4,864
Routed MLP	4 experts, width 128 each
Active experts	Top-2 with fixed 0.5 / 0.5 weights
Routing signal	Balanced, layer-specific token-ID hash
Final step	76,293
Pretraining budget	20B tokens

Each transformer block retains a shared dense MLP for contextual computation. The residual expert branch selects two experts from the token ID and layer index. No learned contextual router is evaluated at inference time. With four stored experts of width 128, each token activates two expert paths alongside the shared width-4,864 path.

Pretraining

• Corpus: FineWeb-Edu.
• Tokenizer: project 32k BPE tokenizer.
• Training tokens: 20B.
• Separate evaluation shard: approximately 40M tokens.
• Global tokens per optimization step: 262,144.
• Optimizer: AdamW.
• Precision: BF16.
• Run seed: 42.

The long run experienced host interruptions and was resumed from checkpoints. metrics.csv provides the merged trajectory, while metrics-raw-with-resumes.csv preserves the raw records around those resumes.

The plotted training curve is a 100-measurement rolling mean. Evaluation points are the recorded measurements on the disjoint evaluation shard. The chart starts after the initial warm-up to keep the later trajectory readable.

Recorded result

Measurement	Step	NLL	Perplexity
First scheduled held-out evaluation	1,000	4.8554	128.43
Last scheduled held-out evaluation	76,000	2.6615	14.32

Step 76,293 is the final optimization checkpoint. The reported held-out value is from step 76,000, the last scheduled evaluation before completion; it is not a substituted training loss.

These numbers describe one exploratory run on one data distribution and one seed. They are not a claim of superiority over a dense or learned-router baseline at this scale.

Repository contents

File	Description
checkpoint.pt	Final raw PyTorch training checkpoint at step 76,293
model.safetensors	BF16 base-model weights without optimizer state
config.json	vllm-i64 model and hash-routing configuration
tokenizer.json	Project 32k BPE tokenizer
tokenizer_config.json	Base tokenizer metadata with no chat template
metrics.csv	Clean merged metric history through the completed run
metrics-raw-with-resumes.csv	Raw metric history retaining restart boundaries

The raw checkpoint contains model weights as well as optimizer, scheduler, configuration, argument, and backend state. It is intended for exact research resumption and framework-level inspection. Use model.safetensors with vllm-i64 for inference; this custom architecture is not a stock Transformers AutoModelForCausalLM implementation.

import torch

artifact = torch.load(
    "checkpoint.pt",
    map_location="cpu",
    mmap=True,
    weights_only=False,
)

print(artifact["step"])        # 76293
print(artifact["config"])
model_state = artifact["model"]

PyTorch checkpoints use pickle internally. Only load artifacts obtained from a source you trust.

Inference and downstream use

The base checkpoint has no instruction-following guarantee and should be prompted as a text-completion model. It intentionally contains no chat template, system prompt, or User/Assistant formatting. The public SFT export and live demo are:

• Instruction-tuned model
• TR-HASH-0.5B live demo
• Training framework
• Custom vllm-i64 runtime

Limitations

This is an experimental small base model. It can repeat text, lose coherence, invent facts, reproduce undesirable patterns from web data, or fail on reasoning, arithmetic, code, multilingual prompts, safety-sensitive requests, and long-context tasks. It has not been independently audited for memorization, contamination, bias, toxicity, privacy, or downstream safety.

Do not use it for consequential decisions. Evaluate the exact checkpoint and serving stack independently before any deployment.

License

The model artifact is released under CC BY-NC 4.0. FineWeb-Edu and individual source documents retain their own applicable terms; users remain responsible for reviewing dataset provenance and compliance for their intended use.