Refresh model card

README.md

@@ -15,79 +15,180 @@ datasets:
 - HuggingFaceTB/finemath
 ---
 
+<div align="center">
+
 # CMA 8M
 
-Benchmark-selected checkpoint from a 7.85M-parameter
-Channel-Mixing Attention generalist language model using the unmodified native
-GPT-S 4,096-token tokenizer. It has no arithmetic-specific token splitting,
-place embeddings, role embeddings, or inference-time equation detection. It
-was selected at step 40,000 for an Open SLM Leaderboard-style average of
-36.21%. WikiText normalized BPB is reported separately
-and is not used for selection.
+### Channel-Mixing Attention, distilled to 7.85 million parameters.
+
+A compact generalist language model built to study channel mixing at small scale.
+
+![Parameters](https://img.shields.io/badge/parameters-7.85M-111827?style=flat-square)
+![Context](https://img.shields.io/badge/context-1%2C024-1f2937?style=flat-square)
+![Vocabulary](https://img.shields.io/badge/vocabulary-4%2C096-374151?style=flat-square)
+![Weights](https://img.shields.io/badge/weights-safetensors-f59e0b?style=flat-square)
+
+**Benchmark-selected at step 40,000 ? Open SLM Leaderboard-style average: 36.21%**
+
+</div>
+
+---
+
+## Overview
+
+CMA 8M is the benchmark-selected checkpoint of a **7,849,161-parameter**
+Channel-Mixing Attention generalist causal language model. It pairs a compact
+nine-layer architecture with the unmodified native GPT-S 4,096-token tokenizer.
+
+The model is deliberately free of task-specific shortcuts. It uses no
+arithmetic-specific token splitting, place embeddings, role embeddings,
+inference-time equation detection, or tokenizer transformations. Mathematical
+text is learned as ordinary causal-language-model text.
 
-## Loading
+| | |
+|---|---|
+| **Architecture** | Channel-Mixing Attention causal LM |
+| **Parameters** | 7,849,161, with tied input/output embeddings |
+| **Checkpoint** | Step 40,000 |
+| **Context length** | 1,024 tokens |
+| **Vocabulary** | 4,096 native tokens |
+| **Weight format** | Safetensors only |
 
-This is a custom Transformers architecture. `trust_remote_code=True` is
-required because stock Hugging Face model classes do not implement CMA or this
-model's exact rotary convention.
+## Quick start
+
+This repository contains a custom Transformers architecture. Stock Hugging Face
+model classes do not implement CMA or this model's exact rotary convention, so
+`trust_remote_code=True` is required.
 
 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
 
-repo = "User01110/cma-8M"
-tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
+repo_id = "User01110/cma-8M"
+
+tokenizer = AutoTokenizer.from_pretrained(
+    repo_id,
+    trust_remote_code=True,
+)
 model = AutoModelForCausalLM.from_pretrained(
-    repo, trust_remote_code=True, dtype="auto"
+    repo_id,
+    trust_remote_code=True,
+    dtype="auto",
 )
+model.eval()
+```
+
+Generate a continuation:
+
+```python
+import torch
+
+prompt = "Channel-mixing attention is"
+inputs = tokenizer(prompt, return_tensors="pt")
+
+with torch.inference_mode():
+    output = model.generate(
+        **inputs,
+        max_new_tokens=80,
+        do_sample=True,
+        temperature=0.8,
+        top_p=0.95,
+    )
+
+print(tokenizer.decode(output[0], skip_special_tokens=True))
 ```
 
+> `trust_remote_code=True` executes the modeling code stored in this repository.
+> Review that code before loading the model in a sensitive environment.
+
 ## Architecture
 
-- Parameters: 7,849,161, with tied input/output embeddings
-- Weights: safetensors only (`model.safetensors`); no PyTorch `.bin` weights
-- Runtime: PyTorch 2.5+ for native SDPA grouped-query attention
-- Tokenizer: AxiomicLabs/GPT-S-5M at revision df47402
-- Vocabulary: 4,096 native tokens
-- Parameter allocation: 1,179,648 tied embedding parameters and
-  6,669,513 non-embedding parameters
-- Context: 1,024 tokens
-- Width/layers: 288 / 9
-- Token-attention heads: 6 query, 2 KV
-- CMA: chunk=24, heads=3, expansion=2
-- Contiguous-half RoPE without scaling
-- No task-specific model features or tokenizer transformations
+| Component | Configuration |
+|---|---:|
+| Total parameters | **7,849,161** |
+| Tied embedding parameters | 1,179,648 |
+| Non-embedding parameters | 6,669,513 |
+| Width / layers | 288 / 9 |
+| Context length | 1,024 tokens |
+| Token-attention heads | 6 query / 2 KV |
+| CMA | chunk 24 / 3 heads / expansion 2 |
+| Positional encoding | Contiguous-half RoPE without scaling |
+| Tokenizer | `AxiomicLabs/GPT-S-5M` at revision `df47402` |
+| Vocabulary | 4,096 native tokens |
+| Runtime | PyTorch 2.5+ for native SDPA grouped-query attention |
+| Weights | `model.safetensors` only; no PyTorch `.bin` weights |
+
+Input and output embeddings are tied. The model contains no task-specific model
+features or tokenizer transformations.
 
 ## Training mixture
 
-- FineWeb-Edu 100BT shuffled: 55.00% of trained tokens
-- Cosmopedia v2: 25.00% of trained tokens
-- FineWeb-HQ: 10.00% of trained tokens
-- FineMath 4+: 10.00% of trained tokens
+All four sources are streamed natural-text corpora.
+
+| Source | Share of trained tokens |
+|---|---:|
+| FineWeb-Edu 100BT shuffled | **55.00%** |
+| Cosmopedia v2 | **25.00%** |
+| FineWeb-HQ | **10.00%** |
+| FineMath 4+ | **10.00%** |
 
 FineMath-4+ supplies high-quality mathematical explanations and reasoning as
-ordinary causal-language-model text, with no task-specific model features or
-tokenizer transformations. All four training sources are streamed natural-text
-corpora.
+ordinary causal-language-model text, without task-specific model features or
+tokenizer transformations.
+
+## Evaluation
 
-## Zero-shot evaluation at step 40,000
+### Zero-shot results at step 40,000
 
-The four lm-eval tasks use normalized accuracy when supplied by lm-eval
-0.4.12, with float32 weights and softmax. ArithMark
-uses float32 weights and its official raw continuation log-likelihood-sum rule.
+The four lm-eval tasks use normalized accuracy when supplied by `lm-eval`
+0.4.12, with float32 weights and softmax. ArithMark uses float32 weights and its
+official raw continuation log-likelihood-sum rule.
 
 | Benchmark | Accuracy |
 |---|---:|
-| HellaSwag | 28.19% |
-| ARC-Easy | 35.35% |
-| ARC-Challenge | 23.29% |
-| PIQA | 58.22% |
-| ArithMark-2 | 29.12% |
-| ARC mean | 29.32% |
-| Open SLM Leaderboard-style average | 36.21% |
-
-The average is `(HellaSwag + mean(ARC-Easy, ARC-Challenge) + PIQA +
-ArithMark-2) / 4`, matching the Open SLM Leaderboard formula.
-
-WikiText-103 validation at this step: loss 2.9903, perplexity
-19.89, normalized BPB 1.3520 over 358,911
-scored tokens and 1,145,226 normalized UTF-8 bytes.
+| HellaSwag | **28.19%** |
+| ARC-Easy | **35.35%** |
+| ARC-Challenge | **23.29%** |
+| PIQA | **58.22%** |
+| ArithMark-2 | **29.12%** |
+| ARC mean | **29.32%** |
+| **Open SLM Leaderboard-style average** | **36.21%** |
+
+The selection average is:
+
+```text
+(HellaSwag + mean(ARC-Easy, ARC-Challenge) + PIQA + ArithMark-2) / 4
+```
+
+This matches the Open SLM Leaderboard formula. The checkpoint was selected at
+step 40,000 for its **36.21%** average.
+
+### WikiText-103 validation
+
+| Metric | Value |
+|---|---:|
+| Loss | **2.9903** |
+| Perplexity | **19.89** |
+| Normalized BPB | **1.3520** |
+| Scored tokens | 358,911 |
+| Normalized UTF-8 bytes | 1,145,226 |
+
+WikiText normalized BPB is reported separately and was **not** used for
+checkpoint selection.
+
+## Intended use and limitations
+
+CMA 8M is intended for research into compact language models, Channel-Mixing
+Attention, controlled evaluation, and lightweight local experiments. It is a
+base causal language model, not an instruction-tuned chat assistant.
+
+At 7.85M parameters, its knowledge and reasoning capacity are sharply limited
+relative to larger models. Generated text may be incorrect, incoherent, biased,
+or unsafe. Verify outputs before using them downstream.
+
+---
+
+<div align="center">
+
+**Compact scale. Native tokens. Channel mixing.**
+
+</div>