---
language: [hi, en]
license: mit
library_name: pytorch
pipeline_tag: text-generation
tags: [hindi, hinglish, gpt, from-scratch, semantic-parsing, safetensors]
datasets: [wikimedia/wikipedia, findnitai/english-to-hinglish]
---

# TinyGPT — a Hindi & Hinglish GPT, built from scratch

Two small models from [TinyGPT](https://github.com/TanushLukha/TinyGPT), where every part — the BPE
tokenizer, the Transformer, training, decoding, LoRA and the serving code — is written by hand in PyTorch.

| folder | model | what it does |
|---|---|---|
| `lm/` | `gpt-llama`, 13.8M parameters | continues Hindi or Hinglish text |
| `parser/` | `gpt-llama-clean` fully fine-tuned on Hinglish-TOP | turns a Hinglish request to a voice assistant into intent + slots |

**Architecture:** decoder-only Transformer, Llama recipe (RoPE, RMSNorm, SwiGLU), 6 layers,
d_model 384, 6 heads, context 256 tokens, tied embeddings;
8,192-token byte-level BPE with a Devanagari-aware pre-tokeniser.

**Training data:** 18.2M tokens — Hindi Wikipedia (`wikimedia/wikipedia`, 20231101.hi, CC BY-SA) and
Hinglish sentences (`findnitai/english-to-hinglish`, Apache-2.0). The parser: the human-annotated part of
[Hinglish-TOP](https://github.com/google-research-datasets/Hinglish-TOP-Dataset) (Apache-2.0), 2,993
requests; its base model was pretrained without Hinglish-TOP's validation and test requests.

## Scores

| | |
|---|---|
| `lm`: bits per byte, held-out text | 0.733 (Hindi 0.689, Hinglish 1.147) |
| `parser`: exact match (whole tree right), test, 6,513 requests | 41.0% |
| `parser`: intent accuracy, test | 83.9% |
| `parser`: slot F1, test | 56.8% |
| `parser`: exact match, validation | 51.0% |

## Use

```python
# pip install torch safetensors "git+https://github.com/TanushLukha/TinyGPT"   (the model's code)
from huggingface_hub import snapshot_download
from tinygpt.serve.engine import TextGenerator, Parser, GenerateParams

path = snapshot_download("tanushlukha/tinygpt")
lm = TextGenerator.from_bundle(f"{path}/lm")
print("".join(p.text for p in lm.stream("भारत की राजधानी", GenerateParams(max_new_tokens=60))
              if hasattr(p, "text")))
print(Parser.from_bundle(f"{path}/parser").parse(["kal subah 7 baje ka alarm laga do"])[0].parse)
```

Each folder holds `model.safetensors`, `config.json` (architecture, task, provenance, scores, SHA-256 of
each file — checked on load) and `tokenizer.json`.

## Limitations

A 13.8M-parameter model trained for 53 minutes on a laptop: it writes fluent Hindi
but has learned the language, not the facts — names, dates and places in its text are invented
([what it knows](https://github.com/TanushLukha/TinyGPT/blob/main/docs/results/m6_facts.md)) — and greedy
decoding repeats itself. The parser gets fewer than half of
unseen requests exactly right; its most common mistake is a slot missing, extra or mislabelled, and about
one mistake in five puts words in a slot that the request doesn't contain. For learning and demos, not for
products.