language:
- hi
- en
license: mit
library_name: pytorch
pipeline_tag: text-generation
tags:
- hindi
- hinglish
- gpt
- from-scratch
- semantic-parsing
- safetensors
datasets:
- wikimedia/wikipedia
- findnitai/english-to-hinglish
TinyGPT — a Hindi & Hinglish GPT, built from scratch
Two small models from TinyGPT, where every part — the BPE tokenizer, the Transformer, training, decoding, LoRA and the serving code — is written by hand in PyTorch.
| folder | model | what it does |
|---|---|---|
lm/ |
gpt-llama, 13.8M parameters |
continues Hindi or Hinglish text |
parser/ |
gpt-llama-clean fully fine-tuned on Hinglish-TOP |
turns a Hinglish request to a voice assistant into intent + slots |
Architecture: decoder-only Transformer, Llama recipe (RoPE, RMSNorm, SwiGLU), 6 layers, d_model 384, 6 heads, context 256 tokens, tied embeddings; 8,192-token byte-level BPE with a Devanagari-aware pre-tokeniser.
Training data: 18.2M tokens — Hindi Wikipedia (wikimedia/wikipedia, 20231101.hi, CC BY-SA) and
Hinglish sentences (findnitai/english-to-hinglish, Apache-2.0). The parser: the human-annotated part of
Hinglish-TOP (Apache-2.0), 2,993
requests; its base model was pretrained without Hinglish-TOP's validation and test requests.
Scores
lm: bits per byte, held-out text |
0.733 (Hindi 0.689, Hinglish 1.147) |
parser: exact match (whole tree right), test, 6,513 requests |
41.0% |
parser: intent accuracy, test |
83.9% |
parser: slot F1, test |
56.8% |
parser: exact match, validation |
51.0% |
Use
# pip install torch safetensors "git+https://github.com/TanushLukha/TinyGPT" (the model's code)
from huggingface_hub import snapshot_download
from tinygpt.serve.engine import TextGenerator, Parser, GenerateParams
path = snapshot_download("tanushlukha/tinygpt")
lm = TextGenerator.from_bundle(f"{path}/lm")
print("".join(p.text for p in lm.stream("भारत की राजधानी", GenerateParams(max_new_tokens=60))
if hasattr(p, "text")))
print(Parser.from_bundle(f"{path}/parser").parse(["kal subah 7 baje ka alarm laga do"])[0].parse)
Each folder holds model.safetensors, config.json (architecture, task, provenance, scores, SHA-256 of
each file — checked on load) and tokenizer.json.
Limitations
A 13.8M-parameter model trained for 53 minutes on a laptop: it writes fluent Hindi but has learned the language, not the facts — names, dates and places in its text are invented (what it knows) — and greedy decoding repeats itself. The parser gets fewer than half of unseen requests exactly right; its most common mistake is a slot missing, extra or mislabelled, and about one mistake in five puts words in a slot that the request doesn't contain. For learning and demos, not for products.