---
library_name: transformers
pipeline_tag: text-generation
tags:
- custom-code
- mlx
- causal-lm
---
# Gab 100M
* [DEMO](https://giftofgab.chat/) at [giftofgab.chat](https://giftofgab.chat/)
* Pre-trained on: [https://huggingface.co/datasets/gszauer/Gab100MPretrain](https://huggingface.co/datasets/gszauer/Gab100MPretrain)
* Fine-tuned on: [https://huggingface.co/datasets/gszauer/Gab100MFinetune](https://huggingface.co/datasets/gszauer/Gab100MFinetune)
Gab 100M is a small full-parameter causal language model trained locally with
MLX and exported for Hugging Face Transformers using custom model code. Load it
with `trust_remote_code=True`.
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained(".", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(".", trust_remote_code=True)
prompt = "<|user|>Explain photosynthesis in simple terms.<|end|><|assistant|>"
inputs = tok(prompt, return_tensors="pt")
out = model.generate(
**inputs,
max_new_tokens=300,
eos_token_id=tok.convert_tokens_to_ids("<|end|>"),
pad_token_id=tok.convert_tokens_to_ids("<|pad|>"),
use_cache=False,
)
print(tok.decode(out[0], skip_special_tokens=False))
```
## Architecture
This model is a decoder-only causal transformer. It is **not** a stock Llama
model, even though several parameter names follow Llama-style naming. In
particular, the MLP is exact GeLU with `up_proj` and `down_proj`; there is no
SwiGLU gate projection.
Configuration:
- Vocabulary size: 10,000 total token ids.
- Context length: 4,096 tokens.
- Layers: 12 transformer blocks.
- Hidden size: 768.
- Attention heads: 12.
- Head dimension: 64.
- Attention projection size: `12 * 64 = 768`.
- MLP intermediate size: 3,456.
- Positional encoding: RoPE, base/theta 100,000.
- Normalization: RMSNorm with epsilon `1e-5`.
- Activation: exact GeLU.
- Dropout: 0.0.
- Biases: no attention or MLP biases.
- Embeddings: input embeddings are tied to the output projection.
- Weight dtype in this export: fp32.
### Forward Pass
Given integer token ids `input_ids` with shape `(batch, sequence)`, the model
performs:
1. Token embedding lookup:
```text
h = embed_tokens[input_ids]
```
2. For each transformer block:
```text
h = h + SelfAttention(RMSNorm(h))
h = h + MLP(RMSNorm(h))
```
3. Final RMSNorm:
```text
h = RMSNorm(h)
```
4. Tied output projection:
```text
logits = h @ embed_tokens.weight.T
```
### RMSNorm
For a hidden vector `x`:
```text
rms = rsqrt(mean(x^2) + 1e-5)
RMSNorm(x) = weight * x * rms
```
The normalization math is done in float32 for numerical stability.
### Attention
Each block uses standard multi-head causal self-attention:
```text
q = q_proj(x)
k = k_proj(x)
v = v_proj(x)
q, k, v -> reshape to (batch, heads, sequence, head_dim)
q, k = RoPE(q, k)
attention = softmax((q @ k.T) / sqrt(head_dim) + causal_mask)
out = attention @ v
out = o_proj(out)
```
All heads are query/key/value heads; there is no grouped-query attention.
### RoPE
RoPE is applied to all 64 dimensions of each head before attention. The inverse
frequency vector is:
```text
inv_freq[i] = 1 / (100000 ** (i / 64)), for i = 0, 2, 4, ..., 62
```
For a token position `p`, compute:
```text
freqs = p * inv_freq
emb = concat(freqs, freqs)
q_rot = q * cos(emb) + rotate_half(q) * sin(emb)
k_rot = k * cos(emb) + rotate_half(k) * sin(emb)
```
Where:
```text
rotate_half([x1, x2]) = [-x2, x1]
```
with `x1` and `x2` being the first and second halves of the head dimension.
### MLP
The feed-forward network is:
```text
MLP(x) = down_proj(gelu(up_proj(x), exact=True))
```
There is no `gate_proj`.
### Weight Layout
The exported `model.safetensors` uses these parameter names:
```text
model.embed_tokens.weight
model.layers.N.input_layernorm.weight
model.layers.N.self_attn.q_proj.weight
model.layers.N.self_attn.k_proj.weight
model.layers.N.self_attn.v_proj.weight
model.layers.N.self_attn.o_proj.weight
model.layers.N.post_attention_layernorm.weight
model.layers.N.mlp.up_proj.weight
model.layers.N.mlp.down_proj.weight
model.norm.weight
```
There is no separate `lm_head.weight`; the output projection is tied to
`model.embed_tokens.weight`.
## Tokenizer
The tokenizer is a byte-level BPE tokenizer with a 10,000-token vocabulary.
It uses special tokens plus 256 byte tokens and learned BPE merges.
Important special tokens:
| Token | Meaning |
| --- | --- |
| `<|end|>` | End of a turn or generated response |
| `<|user|>` | User turn marker |
| `<|assistant|>` | Assistant turn marker |
| `` | Start visible thinking trace |
| `` | End visible thinking trace |
| `<|pad|>` | Padding |
## Chat Format
This model supports a simple two-role chat format. It does not require or use a
system role.
Single-turn prompt:
```text
<|user|>QUESTION<|end|><|assistant|>
```
The model should generate:
```text
ANSWER<|end|>
```
Multi-turn prompt:
```text
<|user|>QUESTION 1<|end|><|assistant|>ANSWER 1<|end|><|user|>QUESTION 2<|end|><|assistant|>
```
Thinking can be forced by opening a thinking tag after the assistant marker:
```text
<|user|>QUESTION<|end|><|assistant|>
```
The expected completion format is:
```text
reasoning...final answer<|end|>
```
For normal non-thinking responses, omit ``:
```text
<|user|>QUESTION<|end|><|assistant|>
```
## Notes
- Generation should use `<|end|>` as the EOS token.
- This export disables KV caching in `generation_config.json` because the
included custom model implementation favors correctness and simplicity.
- The model was trained as a learning project