User01110/cma-7.8M-exp

CMA 7.8M Exp

Arithmetic-aware experimental checkpoint from a 7.87M-parameter Channel-Mixing Attention language model. This model uses atomic arithmetic tokens plus learned digit-place and equation-role embeddings; it is preserved as the specialized experiment while the canonical CMA-7.8M name is reserved for the new native-tokenizer generalist model. It was selected at step 45,000 for an Open SLM Leaderboard-style average of 42.89%. WikiText normalized BPB is reported separately and is not used for selection.

Loading

This is a custom Transformers architecture. trust_remote_code=True is required because stock Hugging Face model classes do not implement CMA, arithmetic features, or this model's exact rotary convention.

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "User01110/cma-7.8M-exp"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, dtype="auto"
)

Architecture

• Parameters: 7,871,625, with tied input/output embeddings
• Context: 1,024 tokens
• Width/layers: 288 / 9
• Token-attention heads: 6 query, 2 KV
• CMA: chunk=24, heads=3, expansion=2
• Contiguous-half RoPE without scaling
• Atomic arithmetic tokens with learned place and role embeddings

Training mixture

• Ultra-FineWeb: 35.00% of trained tokens
• FineWeb-Edu: 27.00% of trained tokens
• Cosmopedia v2: 15.00% of trained tokens
• Ultra-FineWeb-L3: 13.00% of trained tokens
• CMA arithmetic synth: 10.00% of trained tokens

The synthetic stream contains only independently generated integer arithmetic. ArithMark contexts are used solely as an exact exclusion set, so the generator cannot reproduce an official benchmark prompt. General reasoning and commonsense come from the four streamed natural-language corpora.

Zero-shot evaluation at step 45,000

The four lm-eval tasks use normalized accuracy when supplied by lm-eval 0.4.12, with float32 weights and softmax. ArithMark uses float32 weights and its official raw continuation log-likelihood-sum rule.

Benchmark	Accuracy
HellaSwag	27.94%
ARC-Easy	35.31%
ARC-Challenge	22.35%
PIQA	56.47%
ArithMark-2	58.32%
ARC mean	28.83%
Open SLM Leaderboard-style average	42.89%

The average is (HellaSwag + mean(ARC-Easy, ARC-Challenge) + PIQA + ArithMark-2) / 4, matching the Open SLM Leaderboard formula.

WikiText-103 validation at this step: loss 2.9405, perplexity 18.93, normalized BPB 1.3950 over 376,319 scored tokens and 1,144,374 normalized UTF-8 bytes.