User01110/cma-8M

CMA 8M

Benchmark-selected checkpoint from a 7.85M-parameter Channel-Mixing Attention generalist language model using the unmodified native GPT-S 4,096-token tokenizer. It has no arithmetic-specific token splitting, place embeddings, role embeddings, or inference-time equation detection. It was selected at step 30,000 for an Open SLM Leaderboard-style average of 35.13%. WikiText normalized BPB is reported separately and is not used for selection.

Loading

This is a custom Transformers architecture. trust_remote_code=True is required because stock Hugging Face model classes do not implement CMA or this model's exact rotary convention.

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "User01110/cma-8M"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, dtype="auto"
)

Architecture

• Parameters: 7,849,161, with tied input/output embeddings
• Weights: safetensors only (model.safetensors); no PyTorch .bin weights
• Runtime: PyTorch 2.5+ for native SDPA grouped-query attention
• Tokenizer: AxiomicLabs/GPT-S-5M at revision df47402
• Vocabulary: 4,096 native tokens
• Parameter allocation: 1,179,648 tied embedding parameters and 6,669,513 non-embedding parameters
• Context: 1,024 tokens
• Width/layers: 288 / 9
• Token-attention heads: 6 query, 2 KV
• CMA: chunk=24, heads=3, expansion=2
• Contiguous-half RoPE without scaling
• No task-specific model features or tokenizer transformations

Training mixture

• FineWeb-Edu 100BT shuffled: 55.00% of trained tokens
• Cosmopedia v2: 25.00% of trained tokens
• FineWeb-HQ: 10.00% of trained tokens
• FineMath 4+: 10.00% of trained tokens

FineMath-4+ supplies high-quality mathematical explanations and reasoning as ordinary causal-language-model text, with no task-specific model features or tokenizer transformations. All four training sources are streamed natural-text corpora.

Zero-shot evaluation at step 30,000

The four lm-eval tasks use normalized accuracy when supplied by lm-eval 0.4.12, with float32 weights and softmax. ArithMark uses float32 weights and its official raw continuation log-likelihood-sum rule.

Benchmark	Accuracy
HellaSwag	27.65%
ARC-Easy	34.01%
ARC-Challenge	22.78%
PIQA	56.37%
ArithMark-2	28.12%
ARC mean	28.39%
Open SLM Leaderboard-style average	35.13%

The average is (HellaSwag + mean(ARC-Easy, ARC-Challenge) + PIQA + ArithMark-2) / 4, matching the Open SLM Leaderboard formula.

WikiText-103 validation at this step: loss 3.1482, perplexity 23.30, normalized BPB 1.4234 over 358,911 scored tokens and 1,145,226 normalized UTF-8 bytes.