gszauer/Gab1B

Gab 1B

Gab 1B is a ~1 billion parameter mixture-of-experts language model trained from scratch on a Mac Studio. It is the Gab 100M architecture with every block's feed-forward network replaced by 32 experts, of which each token uses 2. It has ten times the parameters, but each token costs about the same compute as Gab 100M: ~108M parameters are active per token.

It is a decoder-only transformer with pre-normalization (RMSNorm), rotary position embeddings, causal self-attention, tied input/output embeddings, and a top-2 mixture of exact-GeLU experts in every block. The canonical model definition is MiniGPT.js (included in this repo). It is a dependency-free JavaScript implementation, and the whole training suite is matched against it.

It speaks English, holds multi-turn conversations, and supports an optional <think> reasoning trace inside assistant turns.

Setting	Value
Parameters	1,028,477,184
Active parameters per token	107,752,704
Transformer blocks	13
Feature dimension	768
Attention heads	12
Head dimension	64
Experts per block	32 (top-2 routing)
Expert hidden dimension	1536 (with a per-neuron bias)
Context length	2048
Vocabulary size	20,000
RoPE theta	10,000 (interleaved pairs)
RMSNorm epsilon	1e-5
Precision	float32 weights (trained in bfloat16 mixed precision)

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "gszauer/Gab1B"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True)

messages = [{"role": "user", "content": "Hello, how are you?"}]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    # enable_thinking=True,  # seed the reply with a <think> trace
    return_tensors="pt",
    return_dict=True,
)
output = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

The Transformers port is a simple, faithful implementation that recomputes the full sequence each generation step (no KV cache), and runs the experts one at a time. Its float32 logits match the training suite's to within rounding. The browser runtime (MiniGPT.js) and the training suite's inference.py both implement KV-cached generation.

Files

File	What it is
model.safetensors	float32 weights with HF-style tensor names (per-head attention matrices fused, see below)
gab1b.weights	the native format: a headerless flat float32 buffer in MiniGPT.js parameters() order, loadable via deserializeFromArrayBuffer
vocab.json	the tokenizer: {reserved, merges}
MiniGPT.js	the canonical implementation: tensor library, autograd, tokenizer, model with mixture of experts, KV cache and AdamW trainer, in dependency-free JavaScript
modeling_gift_of_gab.py etc.	the Transformers port (trust_remote_code)
convert.py	converts gab1b.weights to model.safetensors

Architecture notes

Each block's MLP is a mixture of experts. A router, a bias-free [768, 32] linear layer, scores every expert for each token. The token goes to its two most probable experts, and their outputs are summed, weighted by the router probabilities renormalized over the two choices:

p = softmax(x @ Wrouter)
MoE(x) = sum over the top-2 experts e of (p_e / (p_a + p_b)) * Expert_e(x)

Each expert is a Gab 100M-style MLP at half width: Expert(x) = GeLU(x @ Wup + b) @ Wdown with exact (erf) GeLU, 768 -> 1536 -> 768. Two half-width experts cost the same as Gab 100M's one 768 -> 3072 -> 768 MLP. Inference is dropless: every token always reaches both of its experts.

Attention is trained per head. Each of the 12 heads owns its own Wq, Wk and Wv of shape [768, 64], and its own output projection Wo of shape [64, 768]. Heads are never concatenated. Each head projects its result back up to 768 on its own, and the per-head results are summed:

Attention(x) = sum_i( softmax(causal(q_i @ k_i.T / sqrt(64))) @ v_i @ Wo_i )

This is algebraically identical to a concat followed by one big [768, 768] output projection, so model.safetensors ships fused [768, 768] tensors and the port runs the standard formulation. The unfused per-head matrices are preserved in gab1b.weights.

RoPE uses interleaved pairs. Adjacent channels (2i, 2i+1) rotate together with freq_i = 1/10000^(2i/64), unlike Llama-style split-half RoPE. There is no position embedding table, so position enters only through RoPE.

The unembedding is tied. logits = RMSNorm(h) @ E.T, with no separate output head.

The tokenizer is byte-level BPE with sequential merge application. Ids 0–255 are raw bytes, and every other id is a merge of two earlier ids, learned in order. Text is first split into pre-tokens (words with one leading space, digit groups of up to 3, and whitespace and punctuation runs of up to 4), and encoding applies every merge rule in sequence over each pre-token's UTF-8 bytes. Reserved tokens (<|user|>, <|assistant|>, <|end|>, <|endoftext|>, <think>, </think>, <|pad|>) are chained merges taught before training, so each encodes to a single id and is atomic.

Chat format

There is no system role. Turns are concatenated directly:

<|user|>QUESTION<|end|><|assistant|>ANSWER<|end|>

Thinking is optional and lives inside the assistant turn:

<|user|>QUESTION<|end|><|assistant|><think>PRIVATE_REASONING</think>ANSWER<|end|>

The chat template supports enable_thinking=True to seed the reply with <think>.

Training

Pre-trained on 31,047,744,657 tokens (Gab100MPretrain plus Cosmopedia v2): 947,501 steps of 16 sequences in 514 hours. The context grew from 512 to 1024 to 2048 tokens, at 60% and 85% of training. The final validation loss is 1.270, measured on a held-out 0.02% of the corpus.

Then fine-tuned into a chat model on 941,343,647 tokens (Gab100MFinetune plus SmolTalk), learning only from assistant replies.

AdamW with linear warmup and cosine decay: peak learning rate 3e-4 for pre-training and 3e-5 for fine-tuning, betas 0.9/0.999, weight decay 0.01 (decoupled), and global-norm gradient clipping at 1.0. Trained with MLX on a Mac Studio (M3 Ultra). Training routing uses a fixed expert capacity (capacity factor 1.25), with a load-balancing loss (coefficient 0.01) and a router z-loss (coefficient 0.001).

Links

• Blog/site: gabormakesgames.com