README.md

metadata

license: mit
title: superlillm
sdk: gradio
emoji: 📚
colorFrom: yellow
colorTo: red
sdk_version: 6.19.0

SuperLilLM

SuperLilLM is a tiny conversational language model trained from scratch on 1000 local input/output examples.

What is inside

• data/superlillm_dataset.json - a simple list of 1000 { "input": "...", "output": "..." } examples.
• superlillm/model.py - a small decoder-only Transformer written in PyTorch.
• train.py - builds the tokenizer, does a short next-token pretraining pass, then does SFT with loss focused on assistant replies.
• chat.py - command-line testing.
• app.py and ui/index.html - a local browser UI.

Dataset notes

The data follows a simple SFT pattern: clean prompt/response pairs, repeated intent coverage with varied wording, short direct answers, and one consistent chat template:

User: ...
Assistant: ...

The trainer first learns the whole chat text as normal language modeling, then runs supervised fine-tuning where the prompt tokens are masked and the loss is applied to the assistant reply. That makes the tiny model spend its limited capacity on answering instead of memorizing how to copy the user prompt.

Run

cd "/Users/savvy/Documents/GPT Language Models/SuperLilLM"
python3 build_dataset.py
python3 train.py
python3 chat.py "hey whats up"
python3 app.py

Then open http://127.0.0.1:7860.