README.md

---
license: mit
language: en
tags:
- llm
- pytorch
- custom-model
- causal-lm
- character-level
- math
- tiny-model
model_type: tiny-causal-llm
datasets:
- custom
pipeline_tag: text-generation
---
# TinyLLM: Character-Level Math Solver

## Model Description

**TinyLLM** is a highly compact, character-level **Causal Language Model** (based on the standard Transformer decoder architecture) trained specifically to solve single-digit math problems.

This model serves as a minimalist, educational example of how a standard LLM architecture can be trained from scratch on a very small, custom dataset.

### Key Features
* **Architecture:** Causal Transformer Decoder.
* **Task:** Character-level text generation (autoregressive).
* **Input/Output:** Solves problems formatted as `N op N` and generates the answer, e.g., `4 + 5 = 9<EOS>`.
* **Custom Code Required:** This is a custom PyTorch model and requires custom code (`model.py`, `tokenizer.py`) to be loaded by users.

---
##  How to Use (Inference)

To load and run this custom model, users must download the entire repository structure and use the provided custom code, specifically the `TinyLLM` class defined in **`model.py`** and the `CharacterTokenizer` in **`tokenizer.py`**.

### 1. Installation

First, ensure you have the required libraries installed:
```bash
pip install torch huggingface-hub
from huggingface_hub import snapshot_download
import torch
import os
import sys

# 1. Configuration: REPLACE with your repository ID
MODEL_ID = "anujbhatt4ai/tiny-math-llm" 
DEVICE = 'cuda' if torch.cuda.is_available() else 'cpu'

# 2. Download all files (code and weights)
local_path = snapshot_download(repo_id=MODEL_ID)

# 3. Import Custom Classes
# The downloaded path must be added to sys.path to allow custom imports
sys.path.append(local_path) 
from model import TinyLLM
from tokenizer import CharacterTokenizer, generate_v1_data

# 4. Setup and Load Model
def load_tiny_llm():
    # In this minimal case, we hardcode the known config values
    vocab_size = 22
    block_size = 14
    
    # Initialize the model with the exact trained parameters
    model = TinyLLM(
        vocab_size=vocab_size, 
        block_size=block_size, 
        n_embed=64, n_head=4, n_layer=4, dropout=0.1
    ).to(DEVICE)

    # Load the trained weights
    weights_path = os.path.join(local_path, "pytorch_model.bin")
    model.load_state_dict(torch.load(weights_path, map_location=DEVICE))
    model.eval()
    
    # Initialize the tokenizer
    raw_data = generate_v1_data()
    tokenizer = CharacterTokenizer(raw_data)
    
    return model, tokenizer

# Use the loaded model and tokenizer in your own generation logic
model, tokenizer = load_tiny_llm()
print("Model loaded and ready for math inference!")

**Block 4: Training Details and Repository Files**

`markdown
##  Training Details

### Architecture Configuration

The `TinyLLM` is configured with the following parameters, derived from the `config.json` and `model.py` files:

| Parameter | Value | Description |
| :--- | :--- | :--- |
| **`vocab_size`** | `22` | The size of the character vocabulary. |
| **`block_size`** | `14` | The maximum sequence length (context window). |
| **`n_embed`** | `64` | Embedding dimension. |
| **`n_head`** | `4` | Number of attention heads. |
| **`n_layer`** | `4` | Number of Transformer decoder blocks. |
| **`dropout`** | `0.1` | Dropout rate. |

### Training Hyperparameters (from `train.py`)

| Parameter | Value |
| :--- | :--- |
| **`BATCH_SIZE`** | `32` |
| **`LEARNING_RATE`** | `1e-3` (AdamW) |
| **`EPOCHS`** | `100` |
| **`DEVICE`** | `cuda` if available, else `cpu` |

### Dataset

The model was trained on an **exhaustive set of all single-digit math problems** (addition, subtraction, multiplication, and non-remainder division) where the result is also a single digit (0-9). The **`dataset.py`** file contains the logic for the essential sequence shift used for language modeling training.

---

## Repository Files

This flat repository contains all the source code needed for complete reproducibility.

| File Name | Description |
| :--- | :--- |
| **`pytorch_model.bin`** | The trained model weights. |
| **`config.json`** | Model configuration/hyperparameters. |
| **`model.py`** | **Core Logic:** Custom `TinyLLM` architecture definition. |
| **`tokenizer.py`** | **Core Logic:** Custom `CharacterTokenizer` and data generator. |
| **`dataset.py`** | Defines the `MathDataset` class and sequence shift logic. |
| **`train.py`** | The complete training script and final hyperparameters. |
| **`custom_run.py`** (or `run.py`) | Example script demonstrating how to use the model for generation. |
| **`README.md`** | This model card and documentation. |