---
library_name: transformers
license: apache-2.0
language:
- en
tags:
- tinystories
- llama
- language-model
- educational
- safetensors
datasets:
- roneneldan/TinyStories
model-index:
- name: Tiny LLaMA
  results: []
---

# Tiny LLaMA - TinyStories Edition

A small LLaMA-style causal language model trained on the TinyStories dataset.
This repository contains the Hugging Face `LlamaForCausalLM` conversion of the
local checkpoint from `/home/manojk/small_llama/llama2.c/out/ckpt.pt`.

## Model Details

- **Model Type**: Decoder-only Transformer (`LlamaForCausalLM`)
- **Parameters**: 6,270,624
- **Layers**: 6
- **Attention Heads**: 6
- **Key/Value Heads**: 6
- **Head Dimension**: 48
- **Hidden Size**: 288
- **Intermediate Size**: 768
- **Vocabulary Size**: 512
- **Training Sequence Length**: 256
- **Data Type**: float32
- **Format**: safetensors

## Training

- **Dataset**: TinyStories
- **Training Iterations**: 100
- **Initial Loss**: 6.27
- **Final Loss**: 4.81
- **Validation Loss**: 6.29 to 4.77

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("manojredhat/tiny-llama")
model = AutoModelForCausalLM.from_pretrained("manojredhat/tiny-llama")

inputs = tokenizer("Once upon a time", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=40, do_sample=False)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Tokenizer

The model uses a SentencePiece tokenizer with 512 tokens:

- `<unk>`: token ID 0
- `<s>`: token ID 1
- `</s>`: token ID 2

## Notes

This is an educational small model trained for short TinyStories-style text.
It is not intended for production use, knowledge-intensive tasks, or long-form
generation.