README.md

---
license: cc-by-nc-4.0
library_name: transformers
pipeline_tag: text-generation
datasets:
  - karpathy/climbmix-400b-shuffle
tags:
  - custom_code
  - mixture-of-experts
  - kimi-delta-attention
  - multi-head-latent-attention
---

# Maccy 106M (70M active)

Maccy is a compact, from-scratch base language model trained on Apple Silicon. It has
**106,017,561 total parameters** and activates approximately **70,185,753 parameters per
token** through top-2 routing across four SwiGLU experts.

This is a base completion model, not a chat or instruction-following model.

## Architecture

| Property | Value |
| --- | ---: |
| Total parameters | 106.0M |
| Active parameters per token | 70.2M |
| Layers | 12 |
| Model width | 576 |
| Sequence mixers | 9 KDA, 3 MLA |
| Channel mixers | 4-expert sparse MoE, top-2 routing |
| Context length | 1,024 tokens |
| Vocabulary | 32,768 byte-level BPE tokens |

Maccy combines Kimi Delta Attention (KDA), Multi-head Latent Attention (MLA), and a sparse
mixture of experts. Input and output embeddings are tied.

## Usage

The repository includes a portable Transformers reference implementation built for
Transformers 5.14 or newer. Because Maccy is a custom architecture, loading the model
requires `trust_remote_code=True`.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "bgub/maccy-106m-base"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.float32,
)

inputs = tokenizer("Once upon a time", return_tensors="pt")
output = model.generate(
    **inputs,
    max_new_tokens=100,
    do_sample=True,
    temperature=0.8,
    top_k=50,
    use_cache=False,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

For the optimized Apple-Silicon kernels and training code, see
[https://github.com/bgub/mokka](https://github.com/bgub/mokka).

## Training

- Training data: [Karpathy's shuffled ClimbMix repack](https://huggingface.co/datasets/karpathy/climbmix-400b-shuffle), derived from [NVIDIA Nemotron-ClimbMix](https://huggingface.co/datasets/nvidia/Nemotron-ClimbMix)
- Tokens processed: 2,120,089,600
- Optimizer steps: 64,700
- Training context: 1,024 tokens
- Effective batch: 32 sequences / 32,768 tokens per optimizer step
- Precision: bfloat16 activations with float32 master weights

The tokenizer was trained from scratch on two billion characters of the same corpus. It is
an NFC-normalized byte-level BPE with complete UTF-8 byte fallback.

NVIDIA's source dataset card designates ClimbMix for research and development under CC
BY-NC 4.0. Users are responsible for reviewing both the source-dataset terms and this
model's license before use.

## Evaluation

On the full held-out ClimbMix validation shard, Maccy reached **0.9881 bits per byte** over
20,971,520 target tokens. Treat this as an in-domain pretraining metric rather than a broad
capability benchmark.

The table below recomputes BPB for every model with the same harness and the same 256 KiB of
UTF-8 text per corpus. Each model uses its native tokenizer; all runs use float32, a common
1,024-token context, and a 512-token sliding stride. Lower is better.

| Model | Parameters | Pretraining tokens | ClimbMix validation | WikiText-103 test | enwik8 test | FineWeb-Edu sample |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| **Maccy 106M** | 106.0M / 70.2M active | 2.12B | 0.9916 | 1.3377 | 1.5251 | 1.0979 |
| NanoWhale 100M | 110.4M / 100.5M active | 2.6B | 1.2516 | 1.4063 | 1.8468 | 1.1641 |
| Pythia 70M | 70.4M / 70.4M active | 299.9B | 1.0815 | 1.2441 | 1.2157 | 1.1168 |
| GPT-2 Small | 124.4M / 124.4M active | Not disclosed | 0.9668 | 1.0499 | 1.1907 | 0.9897 |
| SmolLM2 135M | 134.5M / 134.5M active | 2T | 0.8119 | 0.9287 | 0.8482 | 0.8421 |

ClimbMix favors Maccy. FineWeb-Edu favors NanoWhale and SmolLM2 and is not claimed to be
held out from them. Possible WikiText-103 and enwik8 overlap for public reference models is
unknown. Training budgets also differ enormously, so this is a checkpoint comparison, not a
controlled architecture comparison. Full corpus hashes, model revisions, loss sums, and the
reproduction script are in the source repository under `benchmarks/results/reference-bpb-v1`.
Active counts include dense, embedding, and shared-expert weights and exclude only unselected
routed experts; active therefore equals total for dense models.

In a small blind side-by-side generation evaluation against Pythia-70M, graders preferred
Maccy in all 15 non-tied comparisons (one additional comparison was tied). Both models were
still weak in absolute terms, especially on code and mathematics.

## Limitations

- This checkpoint has not been post-trained for conversation or instruction following.
- The 1,024-token context is short by modern standards.
- Code, mathematics, factual reliability, and long-form coherence are limited.
- The portable Transformers implementation does not yet include a recurrent generation
  cache and is slower than Mokka's native Metal implementation.
- Training data may contain errors, biases, and objectionable material that the model can
  reproduce.