bgub/maccy-106m-base

Maccy 106M (70M active)

Maccy is a compact, from-scratch base language model trained on Apple Silicon. It has 106,017,561 total parameters and activates approximately 70,185,753 parameters per token through top-2 routing across four SwiGLU experts.

This is a base completion model, not a chat or instruction-following model.

Architecture

Property	Value
Total parameters	106.0M
Active parameters per token	70.2M
Layers	12
Model width	576
Sequence mixers	9 KDA, 3 MLA
Channel mixers	4-expert sparse MoE, top-2 routing
Context length	1,024 tokens
Vocabulary	32,768 byte-level BPE tokens

Maccy combines Kimi Delta Attention (KDA), Multi-head Latent Attention (MLA), and a sparse mixture of experts. Input and output embeddings are tied.

Usage

The repository includes a portable Transformers reference implementation built for Transformers 5.14 or newer. Because Maccy is a custom architecture, loading the model requires trust_remote_code=True.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "bgub/maccy-106m-base"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.float32,
)

inputs = tokenizer("Once upon a time", return_tensors="pt")
output = model.generate(
    **inputs,
    max_new_tokens=100,
    do_sample=True,
    temperature=0.8,
    top_k=50,
    use_cache=False,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))

For the optimized Apple-Silicon kernels and training code, see https://github.com/bgub/mokka.

Training

• Training data: Karpathy's shuffled ClimbMix repack, derived from NVIDIA Nemotron-ClimbMix
• Tokens processed: 2,120,089,600
• Optimizer steps: 64,700
• Training context: 1,024 tokens
• Effective batch: 32 sequences / 32,768 tokens per optimizer step
• Precision: bfloat16 activations with float32 master weights

The tokenizer was trained from scratch on two billion characters of the same corpus. It is an NFC-normalized byte-level BPE with complete UTF-8 byte fallback.

NVIDIA's source dataset card designates ClimbMix for research and development under CC BY-NC 4.0. Users are responsible for reviewing both the source-dataset terms and this model's license before use.

Evaluation

On the full held-out ClimbMix validation shard, Maccy reached 0.9881 bits per byte over 20,971,520 target tokens. Treat this as an in-domain pretraining metric rather than a broad capability benchmark.

The table below recomputes BPB for every model with the same harness and the same 256 KiB of UTF-8 text per corpus. Each model uses its native tokenizer; all runs use float32, a common 1,024-token context, and a 512-token sliding stride. Lower is better.

Model	Parameters	Pretraining tokens	ClimbMix validation	WikiText-103 test	enwik8 test	FineWeb-Edu sample
Maccy 106M	106.0M / 70.2M active	2.12B	0.9916	1.3377	1.5251	1.0979
NanoWhale 100M	110.4M / 100.5M active	2.6B	1.2516	1.4063	1.8468	1.1641
Pythia 70M	70.4M / 70.4M active	299.9B	1.0815	1.2441	1.2157	1.1168
GPT-2 Small	124.4M / 124.4M active	Not disclosed	0.9668	1.0499	1.1907	0.9897
SmolLM2 135M	134.5M / 134.5M active	2T	0.8119	0.9287	0.8482	0.8421

ClimbMix favors Maccy. FineWeb-Edu favors NanoWhale and SmolLM2 and is not claimed to be held out from them. Possible WikiText-103 and enwik8 overlap for public reference models is unknown. Training budgets also differ enormously, so this is a checkpoint comparison, not a controlled architecture comparison. Full corpus hashes, model revisions, loss sums, and the reproduction script are in the source repository under benchmarks/results/reference-bpb-v1. Active counts include dense, embedding, and shared-expert weights and exclude only unselected routed experts; active therefore equals total for dense models.

In a small blind side-by-side generation evaluation against Pythia-70M, graders preferred Maccy in all 15 non-tied comparisons (one additional comparison was tied). Both models were still weak in absolute terms, especially on code and mathematics.

Limitations

• This checkpoint has not been post-trained for conversation or instruction following.
• The 1,024-token context is short by modern standards.
• Code, mathematics, factual reliability, and long-form coherence are limited.
• The portable Transformers implementation does not yet include a recurrent generation cache and is slower than Mokka's native Metal implementation.
• Training data may contain errors, biases, and objectionable material that the model can reproduce.