MyLabs-LLC/axion-250m

Axion 250M

Axion 250M is a 234,914,816-parameter decoder-only causal language model trained from scratch on 5 billion tokens from FineWeb-Edu. It is an experimental base model and has not been instruction-tuned or safety-aligned.

Architecture

Property	Value
Parameters	234,914,816
Vocabulary	16,384 SentencePiece tokens
Layers	16
Hidden size	1,024
Attention heads	16
FFN size	4,096
Context length	1,024
Position encoding	RoPE, theta 10,000
Precision	BF16 training, FP32 exported weights

The implementation uses RMSNorm, causal scaled dot-product attention, GELU feed-forward layers, and untied input/output embeddings. The model was trained with a custom three-node PyTorch pipeline, with one RTX 3090 per node.

Training and Evaluation

• Training corpus: HuggingFaceFW/fineweb-edu, sample-10BT configuration
• Training tokens: 5,000,003,584 processed
• Validation split: final 10 million tokens reserved from the prepared corpus
• Final validation cross-entropy: 3.0035
• Final validation perplexity: 20.16
• Best observed validation cross-entropy: 2.9967
• Best observed validation perplexity: 20.02

The reported final score used 8,192 held-out tokens over four evaluation batches. These measurements are project-internal language-modeling metrics and are not benchmark scores.

A larger post-training evaluation over 524,288 held-out tokens produced a cross-entropy of 3.0709 and perplexity of 21.56. This larger sample is the more reliable estimate of general validation performance.

Standard Benchmarks

The full evaluation splits were run zero-shot with lm-eval 0.4.12 using BF16 inference. Normalized accuracy is reported for multiple-choice tasks where the harness provides it.

Benchmark	Metric	Score
HellaSwag	normalized accuracy	32.75%
ARC-Easy	normalized accuracy	43.60%
PIQA	normalized accuracy	62.24%
WinoGrande	accuracy	51.85%
LAMBADA OpenAI	exact-match accuracy	25.58%
LAMBADA OpenAI	perplexity	85.95
WikiText-2	word perplexity	47.32
WikiText-2	bits per byte	1.041

The exact task versions, sample counts, standard errors, and runner settings are stored in evaluation/lm_eval_0_4_12_zero_shot.json.

Usage

This release uses the native Axion PyTorch architecture. It is not compatible with transformers.AutoModelForCausalLM.

git clone https://huggingface.co/ikdataml/axion-250m
cd axion-250m
python -m pip install torch pyyaml sentencepiece safetensors
python generate.py --prompt "The universe" --max-new-tokens 80

The weights are stored in model.safetensors. The exact model and training configuration is in configs/axion_250m.yaml, and the tokenizer is in tokenizer/axion16k.model.

Limitations

This is a small experimental base model trained on web text. It can produce incorrect, repetitive, biased, unsafe, or incoherent text. It should not be used for factual, medical, legal, financial, or safety-critical decisions. Users are responsible for evaluating outputs for their application.

License

The model code and weights are released under the MIT License. FineWeb-Edu is an external training dataset and remains subject to its own terms and source content licenses.