README.md

---
library_name: transformers
pipeline_tag: text-generation
language:
- en
tags:
- custom_code
- causal-lm
- cma
- small-language-model
- base-model
- byte-level
- safetensors
datasets:
- HuggingFaceFW/fineweb_edu_100BT-shuffled
- mlfoundations/dclm-baseline-1.0-parquet
- HuggingFaceTB/dclm-edu
- HuggingFaceTB/smollm-corpus
- HuggingFaceTB/finemath
widget:
- text: "The process of photosynthesis"
  example_title: Science
- text: "Once upon a time, in a quiet village"
  example_title: Story
- text: "A computer program is"
  example_title: Technology
---

# CMA-1M Mini

> [!WARNING]
> **Experimental research model, generations are fully unreliable.** CMA-1M Mini
> exists only as a test bed for understanding how Channel-Mixing Attention works,
> where it helps, and where its limits and failure modes appear. Do not treat its
> output as factual, safe, coherent, or suitable for production or real-world
> decisions.

CMA-1M Mini is a **958,692-parameter base causal language model** built to test
Channel-Mixing Attention (CMA) at very small scale. It combines causal grouped-query
token attention with content-dependent mixing across each token's hidden channels.
The model uses a lossless byte-level tokenizer, tied embeddings, native BF16 weights,
and a 2,048-token context window.

| | |
|---|---|
| Parameters | 958,692 |
| Architecture | Decoder-only CMA causal LM |
| Context | 2,048 byte tokens |
| Vocabulary | 260 tokens: 256 bytes + PAD/BOS/EOS/UNK |
| Weight format | BF16 Safetensors |
| Intended interface | Plain-text completion |

> This is a pretrained base model, not a chat or instruction model. Give it ordinary
> text to continue; no chat template or role markers are required.

## Quick start

The architecture is provided as custom Transformers code, so
`trust_remote_code=True` is required. PyTorch 2.5 or newer is recommended.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "User01110/CMA-1M-Mini"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = (
    torch.bfloat16
    if device == "cuda" and torch.cuda.is_bf16_supported()
    else torch.float32
)

tokenizer = AutoTokenizer.from_pretrained(
    repo_id,
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    dtype=dtype,
).to(device).eval()

prompt = "The process of photosynthesis"
inputs = tokenizer(prompt, return_tensors="pt")
inputs = {name: tensor.to(device) for name, tensor in inputs.items()}

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=96,
        do_sample=False,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))
```

The tokenizer automatically prepends `<bos>` during normal encoding. It does not
append `<eos>` to a prompt; generation ends when the model emits EOS or reaches the
requested length. If you deliberately use `add_special_tokens=False`, prepend BOS
yourself.

## Generation options

The included generation defaults are deterministic decoding with a repetition
penalty of 1.2. Override them per request as needed.

| Goal | Recommended settings |
|---|---|
| Reproducible completion | `do_sample=False` |
| Balanced sampling | `do_sample=True, temperature=0.8, top_p=0.9, top_k=50` |
| More varied text | `do_sample=True, temperature=1.0, top_p=0.95` |
| Reduce loops | `repetition_penalty=1.1` to `1.2` |
| Beam search | `do_sample=False, num_beams=4` |
| Output length | Set `max_new_tokens`; keep prompt + output within 2,048 tokens |

Example with sampling:

```python
with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=True,
        temperature=0.8,
        top_p=0.9,
        top_k=50,
        repetition_penalty=1.15,
    )
```

For the high-level pipeline API:

```python
import torch
from transformers import pipeline

generator = pipeline(
    "text-generation",
    model="User01110/CMA-1M-Mini",
    tokenizer="User01110/CMA-1M-Mini",
    trust_remote_code=True,
    dtype="auto",
    device=0 if torch.cuda.is_available() else -1,
)

result = generator(
    "In a distant future,",
    max_new_tokens=80,
    do_sample=True,
    temperature=0.8,
    top_p=0.9,
)
print(result[0]["generated_text"])
```

To score text rather than generate it:

```python
encoded = tokenizer("CMA models text one byte at a time.", return_tensors="pt")
encoded = {name: tensor.to(device) for name, tensor in encoded.items()}

with torch.inference_mode():
    result = model(**encoded, labels=encoded["input_ids"])

print(float(result.loss))
```

## Tokenizer and context

- Every UTF-8 byte has a token, so ordinary text cannot become out-of-vocabulary.
- The four control tokens are `<pad>` (0), `<bos>` (1), `<eos>` (2), and `<unk>` (3).
- The context limit is 2,048 **byte tokens**, not 2,048 words or subword tokens.
- Non-ASCII text usually consumes multiple byte tokens per character.
- For long inputs, explicitly keep the most recent 2,048 tokens rather than relying
  on implicit truncation.
- The tokenizer has no arithmetic-specific splitting, chat template, or hidden prompt
  transformation.

## Architecture

| Component | Configuration |
|---|---|
| Hidden width / layers | 128 / 6 |
| Token attention | 4 query heads, 2 key-value heads |
| Position encoding | Contiguous-half rotary embeddings, no scaling |
| CMA layout | 8 channel slots x 16 channels |
| CMA routing | 2 heads, expansion 2, content-dependent softmax mixing |
| CMA initialization | 90% diagonal routing prior with a dense base path |
| Feed-forward gate | SiLU-gated routed values |
| Normalization | RMSNorm |
| Embeddings | Input and output weights tied |
| Attention runtime | Native PyTorch scaled-dot-product attention |

For each token, CMA projects dense values, reshapes them into channel slots, and
learns a softmax mixing matrix between those slots. A bounded learned coefficient
controls the routed difference from the dense base value, so routing enriches rather
than replaces the fallback path.

The exported generation implementation does not maintain a KV cache. This keeps the
custom model compact and straightforward, but long autoregressive generations will
recompute the active context and are slower than cached generation.

## Training data

The model was pretrained as a general causal language model on the following mixture.
Percentages describe the trained-token mixture.

| Source | Share |
|---|---:|
| [FineWeb-Edu 100BT](https://huggingface.co/datasets/HuggingFaceFW/fineweb_edu_100BT-shuffled) | 45% |
| [DCLM-Baseline 1.0](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0-parquet) | 25% |
| [DCLM-Edu](https://huggingface.co/datasets/HuggingFaceTB/dclm-edu) | 10% |
| [Cosmopedia v2](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus) | 10% |
| [FineMath 4+](https://huggingface.co/datasets/HuggingFaceTB/finemath) | 10% |

No benchmark-specific prompts, task detectors, arithmetic vocabulary, or
inference-time answer rules are built into the model.

## Evaluation

Evaluation is zero-shot. The four lm-eval tasks use normalized accuracy when
provided by lm-eval 0.4.12. ArithMark-2 uses raw continuation log-likelihood sums.
Weights are evaluated in BF16 with FP32 likelihood softmax and an automatic BOS
prefix.

| Benchmark | Accuracy |
|---|---:|
| HellaSwag | 29.35% |
| ARC-Easy | 29.29% |
| ARC-Challenge | 21.76% |
| PIQA | 54.62% |
| ArithMark-2 | 27.44% |
| **Open SLM Leaderboard-style average** | **34.23%** |

The combined score is
`(HellaSwag + mean(ARC-Easy, ARC-Challenge) + PIQA + ArithMark-2) / 4`.
This is a report-only reproduction of the leaderboard formula, not an official
leaderboard submission. WikiText-103 normalized validation BPB is **1.6974**.
Exact machine-readable results are available in
[`benchmark_results.json`](./benchmark_results.json).

## Intended use and limitations

CMA-1M Mini is intended for architecture research, educational experiments,
lightweight language-model tooling, and controlled comparisons at tiny scale.

- At fewer than one million parameters, generations are short-range and frequently
  incoherent; the model should not be treated as a knowledge source.
- It is not instruction-tuned, conversationally aligned, tool-using, or safety-tuned.
- Training data is predominantly English even though byte tokenization can represent
  any UTF-8 text.
- Outputs may reproduce biases, inaccuracies, or undesirable patterns from public
  training corpora.
- Do not use it for consequential medical, legal, financial, or safety decisions.
- Loading custom code executes files from the repository. Review the code or pin a
  trusted revision in security-sensitive environments.

## Repository contents

- `model.safetensors` — BF16 model weights
- `modeling_cma.py` — Transformers-compatible CMA implementation
- `config.json` and `generation_config.json` — architecture and decoding defaults
- `tokenizer.json` and `tokenizer_config.json` — deterministic byte tokenizer
- `benchmark_results.json` — exact evaluation metrics and protocol metadata
- `training_config.json` — reproducibility configuration