---
license: mit
language:
- en
pipeline_tag: text-generation
library_name: transformers
tags:
- gift-of-gab
- small-language-model
- trained-from-scratch
- mixture-of-experts
- browser
datasets:
- gszauer/Gab100MPretrain
- HuggingFaceTB/smollm-corpus
- gszauer/Gab100MFinetune
- HuggingFaceTB/smoltalk
---

# Gab 1B

Gab 1B is a ~1 billion parameter mixture-of-experts language model trained from scratch on a Mac Studio. It is the [Gab 100M](https://huggingface.co/gszauer/Gab100M) architecture with every block's feed-forward network replaced by 32 experts, of which each token uses 2. It has ten times the parameters, but each token costs about the same compute as Gab 100M: ~108M parameters are active per token.

It is a decoder-only transformer with pre-normalization (RMSNorm), rotary position embeddings, causal self-attention, tied input/output embeddings, and a top-2 mixture of exact-GeLU experts in every block. The canonical model definition is `MiniGPT.js` (included in this repo). It is a dependency-free JavaScript implementation, and the whole training suite is matched against it.

It speaks English, holds multi-turn conversations, and supports an optional `<think>` reasoning trace inside assistant turns.

| Setting | Value |
| --- | ---: |
| Parameters | 1,028,477,184 |
| Active parameters per token | 107,752,704 |
| Transformer blocks | 13 |
| Feature dimension | 768 |
| Attention heads | 12 |
| Head dimension | 64 |
| Experts per block | 32 (top-2 routing) |
| Expert hidden dimension | 1536 (with a per-neuron bias) |
| Context length | 2048 |
| Vocabulary size | 20,000 |
| RoPE theta | 10,000 (interleaved pairs) |
| RMSNorm epsilon | 1e-5 |
| Precision | float32 weights (trained in bfloat16 mixed precision) |

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "gszauer/Gab1B"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)

messages = [{"role": "user", "content": "Hello, how are you?"}]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    # enable_thinking=True,  # seed the reply with a <think> trace
    return_tensors="pt",
    return_dict=True,
)
output = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

The Transformers port is a simple, faithful implementation that recomputes the full sequence each generation step (no KV cache), and runs the experts one at a time. Its float32 logits match the training suite's to within rounding. The browser runtime (`MiniGPT.js`) and the training suite's `inference.py` both implement KV-cached generation.

## Files

| File | What it is |
| --- | --- |
| `model.safetensors` | float32 weights with HF-style tensor names (per-head attention matrices fused, see below) |
| `gab1b.weights` | the native format: a headerless flat float32 buffer in `MiniGPT.js` `parameters()` order, loadable via `deserializeFromArrayBuffer` |
| `vocab.json` | the tokenizer: `{reserved, merges}` |
| `MiniGPT.js` | the canonical implementation: tensor library, autograd, tokenizer, model with mixture of experts, KV cache and AdamW trainer, in dependency-free JavaScript |
| `modeling_gift_of_gab.py` etc. | the Transformers port (`trust_remote_code`) |
| `convert.py` | converts `gab1b.weights` to `model.safetensors` |

## Architecture notes

**Each block's MLP is a mixture of experts.** A router, a bias-free `[768, 32]` linear layer, scores every expert for each token. The token goes to its two most probable experts, and their outputs are summed, weighted by the router probabilities renormalized over the two choices:

```text
p = softmax(x @ Wrouter)
MoE(x) = sum over the top-2 experts e of (p_e / (p_a + p_b)) * Expert_e(x)
```

Each expert is a Gab 100M-style MLP at half width: `Expert(x) = GeLU(x @ Wup + b) @ Wdown` with exact (erf) GeLU, `768 -> 1536 -> 768`. Two half-width experts cost the same as Gab 100M's one `768 -> 3072 -> 768` MLP. Inference is dropless: every token always reaches both of its experts.

**Attention is trained per head.** Each of the 12 heads owns its own `Wq`, `Wk` and `Wv` of shape `[768, 64]`, *and its own output projection* `Wo` of shape `[64, 768]`. Heads are never concatenated. Each head projects its result back up to 768 on its own, and the per-head results are summed:

```text
Attention(x) = sum_i( softmax(causal(q_i @ k_i.T / sqrt(64))) @ v_i @ Wo_i )
```

This is algebraically identical to a concat followed by one big `[768, 768]` output projection, so `model.safetensors` ships fused `[768, 768]` tensors and the port runs the standard formulation. The unfused per-head matrices are preserved in `gab1b.weights`.

**RoPE uses interleaved pairs.** Adjacent channels `(2i, 2i+1)` rotate together with `freq_i = 1/10000^(2i/64)`, unlike Llama-style split-half RoPE. There is no position embedding table, so position enters only through RoPE.

**The unembedding is tied.** `logits = RMSNorm(h) @ E.T`, with no separate output head.

**The tokenizer is byte-level BPE with sequential merge application.** Ids 0–255 are raw bytes, and every other id is a merge of two earlier ids, learned in order. Text is first split into pre-tokens (words with one leading space, digit groups of up to 3, and whitespace and punctuation runs of up to 4), and encoding applies every merge rule in sequence over each pre-token's UTF-8 bytes. Reserved tokens (`<|user|>`, `<|assistant|>`, `<|end|>`, `<|endoftext|>`, `<think>`, `</think>`, `<|pad|>`) are chained merges taught before training, so each encodes to a single id and is atomic.

## Chat format

There is no system role. Turns are concatenated directly:

```text
<|user|>QUESTION<|end|><|assistant|>ANSWER<|end|>
```

Thinking is optional and lives inside the assistant turn:

```text
<|user|>QUESTION<|end|><|assistant|><think>PRIVATE_REASONING</think>ANSWER<|end|>
```

The chat template supports `enable_thinking=True` to seed the reply with `<think>`.

## Training

Pre-trained on 31,047,744,657 tokens ([Gab100MPretrain](https://huggingface.co/datasets/gszauer/Gab100MPretrain) plus [Cosmopedia v2](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus)): 947,501 steps of 16 sequences in 514 hours. The context grew from 512 to 1024 to 2048 tokens, at 60% and 85% of training. The final validation loss is 1.270, measured on a held-out 0.02% of the corpus.

Then fine-tuned into a chat model on 941,343,647 tokens ([Gab100MFinetune](https://huggingface.co/datasets/gszauer/Gab100MFinetune) plus [SmolTalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk)), learning only from assistant replies.

AdamW with linear warmup and cosine decay: peak learning rate 3e-4 for pre-training and 3e-5 for fine-tuning, betas 0.9/0.999, weight decay 0.01 (decoupled), and global-norm gradient clipping at 1.0. Trained with MLX on a Mac Studio (M3 Ultra). Training routing uses a fixed expert capacity (capacity factor 1.25), with a load-balancing loss (coefficient 0.01) and a router z-loss (coefficient 0.001).

## Links

- Blog/site: [gabormakesgames.com](https://gabormakesgames.com)