| --- |
| license: mit |
| base_model: karpathy/tinyllamas |
| tags: |
| - llama2 |
| - gguf |
| - safetensors |
| - transformers |
| - tinyllamas |
| - validation |
| - test-suite |
| --- |
| |
| # TinyStories Llama2 1M (tinybpe1m) GGUF & HF Validation Suite |
|
|
| This repository provides ultra-lightweight Llama2 model files across various formats (both **GGUF** and **Hugging Face / Safetensors**), trained on the TinyStories dataset and optimized for testing and validation. |
|
|
| ### Why this repository exists |
| When developing a custom LLM inference engine, debugging with a full-sized model is slow. This suite offers a true **1M parameter scale model** (~0.5MB to ~4MB depending on the quantization format), allowing developers to validate their loaders, serialization, quantization kernels, and inference logic step-by-step with maximum efficiency. |
|
|
| ### Difference from `tiny1m` |
| This is a **BPE-based model variant**. Unlike the standard `tiny1m` model, this model is **NOT compatible with `llama2.c`**. |
| The custom SentencePiece BPE tokenizer utilized here relies on the `byte_fallback` mechanism to handle unknown characters. Because `llama2.c`'s simplified native C loader/tokenizer cannot interpret or process `byte_fallback` routines, text generation will fail or corrupt in that environment. This suite is strictly designed and optimized for **`llama.cpp` (GGUF)** and **Hugging Face `transformers` (Python)** execution. |
|
|
| --- |
|
|
| ## 📂 Repository Structure & File Descriptions |
|
|
| ### 1. GGUF Formats (Root Directory `./`) |
| A comprehensive validation suite converted for `llama.cpp` and compatible engines. The tokenizer vocabulary and special tokens are fully embedded within each GGUF binary. Every compiled quantization variant available in the root directory is explicitly covered below: |
|
|
| | Filename(s) / Wildcard Pattern | Type | Size | Purpose / Validation Target | |
| | :--- | :--- | :--- | :--- | |
| | **`tinybpe1m.F32.gguf`** | `F32` | ~4.0 MB | **Baseline Test.** Validates GGUF parsing, tensor layout, matrix multiplication, RoPE, and Attention logic without dequantization overhead. | |
| | **`tinybpe1m.F16.gguf`**<br>**`tinybpe1m.BF16.gguf`** | `F16`<br>`BF16` | ~2.0 MB | **Half-Precision Test.** Validates 16-bit floating point loading, type casting, and inference stability. | |
| | **`tinybpe1m.Q8_0.gguf`** | `Q8_0` | ~1.1 MB | **Quantization Level 1.** Validates block-based uniform scaling with 32 elements. | |
| | **`tinybpe1m.Q4_0.gguf`**<br>**`tinybpe1m.Q4_1.gguf`** | `Q4_0`<br>`Q4_1` | ~0.7 MB | **Quantization Level 2.** Validates classic 4-bit linear quantization and bit-unpacking logic. | |
| | **`tinybpe1m.Q2_K.gguf`** | `Q2_K` | ~0.5 MB | **Standard K-Quant (2-bit).** Validates 2-bit super-block quantization parsing. | |
| | **`tinybpe1m.Q3_K_*.gguf`**<br>↳ *`tinybpe1m.Q3_K_S.gguf`*<br>↳ *`tinybpe1m.Q3_K_M.gguf`*<br>↳ *`tinybpe1m.Q3_K_L.gguf`* | `Q3_K` | ~0.6 MB | **Standard K-Quant (3-bit).** Validates Small, Medium, and Large sub-variants of 3-bit multi-block structures. | |
| | **`tinybpe1m.Q4_K_*.gguf`**<br>↳ *`tinybpe1m.Q4_K_S.gguf`*<br>↳ *`tinybpe1m.Q4_K_M.gguf`* | `Q4_K` | ~0.7 MB | **Standard K-Quant (4-bit).** Validates Small and Medium sub-variants of modern 4-bit super-block structural parsing. | |
| | **`tinybpe1m.Q5_K_*.gguf`**<br>↳ *`tinybpe1m.Q5_K_S.gguf`*<br>↳ *`tinybpe1m.Q5_K_M.gguf`* | `Q5_K` | ~0.8 MB | **Standard K-Quant (5-bit).** Validates Small and Medium sub-variants of 5-bit mixed precision super-blocks. | |
| | **`tinybpe1m.Q6_K.gguf`** | `Q6_K` | ~0.9 MB | **Standard K-Quant (6-bit).** Validates 6-bit high-fidelity super-block quantization. | |
| | **`tinybpe1m.IQ3_*.gguf`**<br>↳ *`tinybpe1m.IQ3_XXS.gguf`*<br>↳ *`tinybpe1m.IQ3_S.gguf`* | `I-Quants` | ~0.5 MB | **Importance Quants (3-bit).** Non-linear 3-bit importance quantization targeting lookup table (codebook) decoding logic. | |
| | **`tinybpe1m.IQ4_*.gguf`**<br>↳ *`tinybpe1m.IQ4_NL.gguf`*<br>↳ *`tinybpe1m.IQ4_XS.gguf`* | `I-Quants` | ~0.6 MB | **Importance Quants (4-bit).** Non-linear 4-bit importance quantization variants (Non-Linear and Extra Small). | |
| | **`tinybpe1m.TQ1_0.gguf`**<br>**`tinybpe1m.TQ2_0.gguf`** | `Ternary` | ~0.4 MB | **Experimental.** Ternary (-1, 0, 1) state quantization for cutting-edge engine testing. | |
|
|
| ### 2. Hugging Face Native Format (`./hf/`) |
| This directory contains the standard files required to load the model using the PyTorch `transformers` library: |
| * **`hf/model.safetensors`**: The raw, unquantized model weights stored securely in Safetensors format. |
| * **`hf/config.json`**: The architectural configuration file defining hyperparameters (layers, heads, dimensions). |
| * **`hf/generation_config.json`**: Default parameters optimized for text generation. |
| * **`hf/tokenizer_config.json`**: Tokenizer behavior layout enabling automatic BOS token injection and padding setup. |
| * **`hf/special_tokens_map.json`**: Architectural mappings tying token strings to exact internal special token IDs. |
| * **`hf/tokenizer.model`**: The custom 512-vocab SentencePiece tokenizer model file. |
|
|
| --- |
|
|
| ## 🚀 Usage Examples |
|
|
| ### A. Running GGUF via llama.cpp |
| To verify your local setup or test custom execution backends using the official native utilities: |
| ```bash |
| ./llama-cli -m tinybpe1m.Q4_K_M.gguf -p "Tom and Jerry are " -n 64 --temp 0.0 |
| |
| ``` |
|
|
| ### B. Loading Hugging Face Formats via Python |
|
|
| You can import the Hugging Face variant directly into Python using the `transformers` library. |
|
|
| ```python |
| import torch |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
| |
| repo_id = "shibatch/tinybpe1m" |
| |
| # The library automatically loads from the hf/ subfolder |
| tokenizer = AutoTokenizer.from_pretrained(repo_id, subfolder="hf") |
| model = AutoModelForCausalLM.from_pretrained(repo_id, subfolder="hf") |
| |
| prompt = "Tom and Jerry are " |
| inputs = tokenizer(prompt, return_tensors="pt") |
| |
| with torch.no_grad(): |
| outputs = model.generate( |
| **inputs, |
| max_new_tokens=64, |
| do_sample=False, |
| pad_token_id=tokenizer.eos_token_id |
| ) |
| |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| |
| ``` |
|
|
| --- |
|
|
| ## 📝 Model Specifications |
|
|
| The network architecture features an unshared output layer (`lm_head`) to keep memory structures consistent with standard Llama 2 definitions. Thanks to the highly optimized 512 vocabulary size, the token embedding and output layers remain extremely lightweight. |
|
|
| * **Architecture:** Llama 2 (Scaled-down variant) |
| * **Dataset:** TinyStories |
| * **Total Parameters:** ~1M (Exactly 896,256 parameters) |
| * **Vocabulary Size:** 512 (Custom SentencePiece BPE Tokenizer with `byte_fallback` enabled) |
| * **Hidden Size (`hidden_size`):** 128 |
| * **Number of Hidden Layers (`num_hidden_layers`):** 4 |
| * **Number of Attention Heads (`num_heads`):** 2 |
| * **Number of Key-Value Heads (`num_kv_heads`):** 2 |
| * **Intermediate Size (`intermediate_size`):** 352 |
| * **Max Position Embeddings (`max_position_embeddings`):** 256 |
| |
| ## 📜 Acknowledgments & License |
| |
| * **Original Implementation:** Inspired by Andrej Karpathy's `llama2.c` project. |
| * **Dataset:** TinyStories dataset. |
| * **License:** **MIT License**. You are free to use, modify, and distribute these assets for any purpose. |
| |
