README.md

---
license: mit
base_model: karpathy/tinyllamas
tags:
- llama2
- gguf
- safetensors
- transformers
- tinyllamas
- validation
- test-suite
---

# TinyStories Llama2 GGUF & HF Validation Suite

This repository provides a comprehensive collection of ultra-lightweight Llama2 models across various formats (both **GGUF** and **Hugging Face/Safetensors**), converted from Andrej Karpathy's `llama2.c` project.

### Why this repository exists?
When developing a custom LLM inference engine from scratch (C/C++, Vulkan, WebAssembly, etc.) or testing custom hardware kernels, debugging with a full-sized 7B model is slow and inefficient. This suite offers **1MB to 60MB scale models**, allowing developers to validate their loaders, serialization, quantization kernels, and inference logic step-by-step with lightning speed.

---

## 📦 Included Formats & Testing Roadmap

### 1. GGUF Formats (For Native Inference Engines)
Recommended validation order when developing a custom native GGUF engine:

| Filename | Type | Size | Purpose / Validation Target |
| :--- | :--- | :--- | :--- |
| **`stories15M.F32.gguf`** | `F32` | ~60 MB | **Baseline Test.** Validates GGUF parsing, tensor layout, matrix multiplication, RoPE, and Attention logic without any dequantization overhead. |
| **`stories15M.F16.gguf`**<br>`stories15M.BF16.gguf` | `F16`<br>`BF16` | ~30 MB | **Half-Precision Test.** Validates 16-bit floating point loading, type casting, and inference stability. |
| **`stories15M.Q8_0.gguf`** | `Q8_0` | ~16 MB | **Quantization Level 1.** Validates the simplest linear quantization logic (block-based uniform scaling with 32 elements). |
| **`stories15M.Q4_0.gguf`**<br>`stories15M.Q4_1.gguf` | `Q4_0`<br>`Q4_1` | ~10 MB | **Quantization Level 2.** Validates classic 4-bit linear quantization and bit-unpacking logic. |
| **`stories15M.Q2_K`** 〜 **`Q6_K.gguf`** | `K-Quants` | 9~15 MB | **Standard Quants.** Validates modern super-block structural parsing with mixed precision. |
| **`stories15M.IQ3_XXS`** 〜 **`IQ4_XS.gguf`** | `I-Quants` | 8~12 MB | **Advanced Quants.** Non-linear quantization targeting lookup table (codebook) decoding logic. |
| **`stories15M.TQ1_0.gguf`**<br>`stories15M.TQ2_0.gguf` | `Ternary` | 7~9 MB | **Experimental.** Ternary (-1, 0, 1) state quantization for cutting-edge engine testing. |
| **`stories260K.F32.gguf`**<br>`stories260K.F16.gguf` | `F32`<br>`F16` | ~1 MB | **Ultra-Mini Check.** Extreme low-resource baseline utilizing a tiny 512-token vocabulary. |

### 2. Hugging Face / Transformers Formats (For PyTorch Validation)
Standard Safetensors weights accompanied by standard `config.json` files for out-of-the-box usage with the Hugging Face `transformers` library. Ideal for calculating mathematical baseline answers or testing upstream conversion scripts (like `convert_hf_to_gguf.py`).

- **`hf_stories15M/`**: The 15M parameter model mapped to standard Hugging Face Llama architecture. Includes pre-bundled Llama-2 compatible tokenizer configurations.
- **`hf_stories260K/`**: The ultra-mini 260K parameter model with its custom architecture parameters intact. 

---

## 🚀 Quick Start & Usage Examples

### A. Running GGUF via llama.cpp
To verify your local setup or compare tokens using the official native utilities:

```bash
./llama-cli -m stories15M.Q4_K_M.gguf -p "One day, Timmy went to" -n 30 --temp 0.0

```

### B. Loading Hugging Face Formats via Python

You can import the Hugging Face variants directly into Python via the `transformers` library using the `subfolder` argument.

#### Example for `hf_stories15M`

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo_id = "shibatch/stories-converted"

# Load directly from the subfolder in this repository
tokenizer = AutoTokenizer.from_pretrained(repo_id, subfolder="hf_stories15M")
model = AutoModelForCausalLM.from_pretrained(repo_id, subfolder="hf_stories15M")

prompt = "One day, Timmy went to"
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(
        **inputs, 
        max_new_tokens=30, 
        do_sample=False,
        pad_token_id=tokenizer.eos_token_id
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

```

---

## 📝 Model Specifications

* **Architecture:** Llama 2 (scaled down variants)
* **Dataset:** TinyStories (focused on simple vocabulary suited for 3 to 4-year-olds)
* **Vocabulary Size:** 32,000 for 15M models, 512 for 260K models.

## 📜 Acknowledgments & License

* **Original Weights:** Trained by Andrej Karpathy ([karpathy/tinyllamas](https://huggingface.co/karpathy/tinyllamas)).
* **License:** **MIT License** (inherited from the original `llama2.c` repository). You are free to use, modify, and distribute these assets for any purpose.