---
license: mit
tags:
- llama
- gguf
- safetensors
- transformers
- validation
- test-suite
- scratch-trained
---

# TinyStories 2M (tinygpt1m) GGUF & HF Validation Suite with GPT-2 Style Byte-Level BPE

This repository provides an ultra-lightweight Llama-style validation model with a ~1M parameter footprint, trained on the TinyStories dataset and optimized for testing and validation. It is specifically designed and trained from scratch to aggressively stress-test custom inference engines against edge-case boundary conditions, memory alignment non-uniformity, and sparse vocabulary handling.

The network architecture deliberately avoids common "power-of-two" or "multiples of 32" configurations to thoroughly expose hidden assumptions, hardcoded boundaries, or improper optimization loops within the codebase.

---

## 📂 Repository Structure

```text
.:
hf/                  # Hugging Face / Safetensors native weight formats
└── config.json
└── generation_config.json
└── model.safetensors
└── tokenizer.json
└── tokenizer_config.json
└── special_tokens_map.json
tinygpt1m.F32.gguf   # llama.cpp / Custom engine GGUF binary format (F32)

```

---

## 📐 Model Specifications

To rigorously validate the numerical and algorithmic structural integrity of inference engines, the model enforces the following non-standard and asymmetric topology:

* **Architecture:** Llama (`LlamaForCausalLM`)
* **Total Parameters:** ~1M class footprint
* **Hidden Size (`hidden_size`):** **`234`**
* A highly irregular dimension that is neither a power of two nor a multiple of 32. Any faulty memory padding or rigid SIMD kernel alignment assumptions will immediately trigger a segmentation fault or severe numerical corruption.


* **Number of Attention Heads (`num_heads` / `num_kv_heads`):** **`9` / `3**`
* Replicates the exact **odd head count and non-power-of-two GQA layout (Group Size = 3)** found in SmolLM2. This structure completely destabilizes head-loop or key-value indexing calculations that assume even splits or rely blindly on bit-shifting optimizations.


* **Head Dimension (`head_dim`):** **`26`** ($234 \div 9$)
* Selected as an even number to satisfy the internal tensor concatenation requirements of Hugging Face's RoPE execution, while remaining a highly atypical dimension relative to standard 64 or 128 sizes.


* **Individual FFN Internal Dimension (`intermediate_size`):** **`521`** (Prime Number)
* Utilizes a prime number configuration to deny any divisible memory padding shortcuts during matrix math execution.


* **Number of Hidden Layers (`num_hidden_layers`):** 2

---

## 🔤 Tokenizer Specifications

The tokenizer contains a maliciously structured sparse token distribution layout designed specifically to pin down and validate Jules' dynamic resizing implementation for non-contiguous index structures:

* **Tokenizer Type:** GPT-2 Style Byte-Level BPE (`llama-bpe` mapping specification)
* **Base Vocabulary Size:** **`1,009`** (Prime Number)
* **Sparse Added Tokens:**
Special control tokens are explicitly assigned to IDs widely separated from the core token sequence, forcing a large gap in the array mapping.
* `id: 2000` ➔ `<|im_start|>`
* `id: 2001` ➔ `<s>` (**BOS token pushed to the very end**)
* `id: 0` ➔ `</s>` (**EOS token pulled to the very beginning**)


* **Conceptual Inversion:**
By mapping BOS to a high index (`2001`) and EOS to `0`—inverting the conventional `1` or `2` assignments—this setup aggressively breaks and exposes hardcoded token ID assumptions in custom pipeline logic.

---

## 🚀 Usage

### Hugging Face / Transformers (Python)

```python
import torch
from transformers import LlamaForCausalLM, PreTrainedTokenizerFast

model_dir = "./tinygpt1m/hf"
tokenizer = PreTrainedTokenizerFast(tokenizer_file=f"{model_dir}/tokenizer.json")
model = LlamaForCausalLM.from_pretrained(model_dir)

prompt = "Once upon"
# Prepend 2001 (BOS) manually to match the specialized training distribution
input_ids = [2001] + tokenizer.encode(prompt, add_special_tokens=False)
input_tensor = torch.tensor([input_ids])

with torch.no_grad():
    output_ids = model.generate(
        input_tensor,
        max_length=32,
        do_sample=False,
        pad_token_id=0,
        bos_token_id=2001,
        eos_token_id=0
    )

print(tokenizer.decode(output_ids[0]))

```

### llama.cpp / Custom Engine (GGUF)

```bash
# Verify behavior using llama.cpp completion binary
./llama-completion -m tinygpt1m.F32.gguf -p "Once upon" -n 20

```

---

## 📜 Acknowledgments

This validation model was successfully trained from scratch using TinyStories dataset. It acts as a permanent, zero-regression guardrail asset for evaluating attention kernels, dynamic token allocation arrays, and sampling routines across custom LLM hardware inference runtimes.