README.md

---
library_name: transformers
pipeline_tag: text-generation
tags:
- custom-code
- mlx
- causal-lm
---

# Gab 100M

* [DEMO](https://giftofgab.chat/) at [giftofgab.chat](https://giftofgab.chat/)
* Pre-trained on: [https://huggingface.co/datasets/gszauer/Gab100MPretrain](https://huggingface.co/datasets/gszauer/Gab100MPretrain)
* Fine-tuned on: [https://huggingface.co/datasets/gszauer/Gab100MFinetune](https://huggingface.co/datasets/gszauer/Gab100MFinetune)

Gab 100M is a small full-parameter causal language model trained locally with
MLX and exported for Hugging Face Transformers using custom model code. Load it
with `trust_remote_code=True`.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained(".", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(".", trust_remote_code=True)

prompt = "<|user|>Explain photosynthesis in simple terms.<|end|><|assistant|>"
inputs = tok(prompt, return_tensors="pt")
out = model.generate(
    **inputs,
    max_new_tokens=300,
    eos_token_id=tok.convert_tokens_to_ids("<|end|>"),
    pad_token_id=tok.convert_tokens_to_ids("<|pad|>"),
    use_cache=False,
)
print(tok.decode(out[0], skip_special_tokens=False))
```

## Architecture

This model is a decoder-only causal transformer. It is **not** a stock Llama
model, even though several parameter names follow Llama-style naming. In
particular, the MLP is exact GeLU with `up_proj` and `down_proj`; there is no
SwiGLU gate projection.

Configuration:

- Vocabulary size: 10,000 total token ids.
- Context length: 4,096 tokens.
- Layers: 12 transformer blocks.
- Hidden size: 768.
- Attention heads: 12.
- Head dimension: 64.
- Attention projection size: `12 * 64 = 768`.
- MLP intermediate size: 3,456.
- Positional encoding: RoPE, base/theta 100,000.
- Normalization: RMSNorm with epsilon `1e-5`.
- Activation: exact GeLU.
- Dropout: 0.0.
- Biases: no attention or MLP biases.
- Embeddings: input embeddings are tied to the output projection.
- Weight dtype in this export: fp32.

### Forward Pass

Given integer token ids `input_ids` with shape `(batch, sequence)`, the model
performs:

1. Token embedding lookup:

   ```text
   h = embed_tokens[input_ids]
   ```

2. For each transformer block:

   ```text
   h = h + SelfAttention(RMSNorm(h))
   h = h + MLP(RMSNorm(h))
   ```

3. Final RMSNorm:

   ```text
   h = RMSNorm(h)
   ```

4. Tied output projection:

   ```text
   logits = h @ embed_tokens.weight.T
   ```

### RMSNorm

For a hidden vector `x`:

```text
rms = rsqrt(mean(x^2) + 1e-5)
RMSNorm(x) = weight * x * rms
```

The normalization math is done in float32 for numerical stability.

### Attention

Each block uses standard multi-head causal self-attention:

```text
q = q_proj(x)
k = k_proj(x)
v = v_proj(x)
q, k, v -> reshape to (batch, heads, sequence, head_dim)
q, k = RoPE(q, k)
attention = softmax((q @ k.T) / sqrt(head_dim) + causal_mask)
out = attention @ v
out = o_proj(out)
```

All heads are query/key/value heads; there is no grouped-query attention.

### RoPE

RoPE is applied to all 64 dimensions of each head before attention. The inverse
frequency vector is:

```text
inv_freq[i] = 1 / (100000 ** (i / 64)), for i = 0, 2, 4, ..., 62
```

For a token position `p`, compute:

```text
freqs = p * inv_freq
emb = concat(freqs, freqs)
q_rot = q * cos(emb) + rotate_half(q) * sin(emb)
k_rot = k * cos(emb) + rotate_half(k) * sin(emb)
```

Where:

```text
rotate_half([x1, x2]) = [-x2, x1]
```

with `x1` and `x2` being the first and second halves of the head dimension.

### MLP

The feed-forward network is:

```text
MLP(x) = down_proj(gelu(up_proj(x), exact=True))
```

There is no `gate_proj`.

### Weight Layout

The exported `model.safetensors` uses these parameter names:

```text
model.embed_tokens.weight
model.layers.N.input_layernorm.weight
model.layers.N.self_attn.q_proj.weight
model.layers.N.self_attn.k_proj.weight
model.layers.N.self_attn.v_proj.weight
model.layers.N.self_attn.o_proj.weight
model.layers.N.post_attention_layernorm.weight
model.layers.N.mlp.up_proj.weight
model.layers.N.mlp.down_proj.weight
model.norm.weight
```

There is no separate `lm_head.weight`; the output projection is tied to
`model.embed_tokens.weight`.

## Tokenizer

The tokenizer is a byte-level BPE tokenizer with a 10,000-token vocabulary.
It uses special tokens plus 256 byte tokens and learned BPE merges.

Important special tokens:

| Token | Meaning |
| --- | --- |
| `<|end|>` | End of a turn or generated response |
| `<|user|>` | User turn marker |
| `<|assistant|>` | Assistant turn marker |
| `<think>` | Start visible thinking trace |
| `</think>` | End visible thinking trace |
| `<|pad|>` | Padding |

## Chat Format

This model supports a simple two-role chat format. It does not require or use a
system role.

Single-turn prompt:

```text
<|user|>QUESTION<|end|><|assistant|>
```

The model should generate:

```text
ANSWER<|end|>
```

Multi-turn prompt:

```text
<|user|>QUESTION 1<|end|><|assistant|>ANSWER 1<|end|><|user|>QUESTION 2<|end|><|assistant|>
```

Thinking can be forced by opening a thinking tag after the assistant marker:

```text
<|user|>QUESTION<|end|><|assistant|><think>
```

The expected completion format is:

```text
reasoning...</think>final answer<|end|>
```

For normal non-thinking responses, omit `<think>`:

```text
<|user|>QUESTION<|end|><|assistant|>
```

## Notes

- Generation should use `<|end|>` as the EOS token.
- This export disables KV caching in `generation_config.json` because the
  included custom model implementation favors correctness and simplicity.
- The model was trained as a learning project