README.md

---
license: cc-by-4.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
base_model: roneneldan/TinyStories-Instruct-3M
datasets:
- allenai/soda
- roneneldan/TinyStoriesInstruct
tags:
- gpt-neo
- tinystories
- tiny
- embedded
- esp32
- microcontroller
- chat
- small-talk
---

# TinyTalk — human-like small talk on a microcontroller

**TinyTalk** is a ~8.3M-parameter (≈1.6M non-embedding) **GPT-Neo** chatbot built
to do one thing: hold short, friendly, human-sounding small-talk conversations
on **low-end hardware that can't run a normal LLM** — think an **ESP32-S3**.

It is the model embedded in the [Cardputer AI](https://github.com/rezor/cardputer_ai)
firmware, where it runs **fully offline on the device** in ~2 MB of flash after
Q4_0 quantization. This repository hosts the full-precision PyTorch /
`safetensors` weights so the model can be used, fine-tuned, or re-quantized on
its own.

## What it's for

Most chat models assume a datacenter GPU. TinyTalk asks the opposite question:
*how small can a model be and still feel like talking to someone?* It trades
away knowledge, reasoning, and long context to fit on a microcontroller, keeping
only the ability to make warm, coherent small talk:

```
User: hey, how are you?
Bot: I am good! I played outside today. It was so much fun!
User: nice! what did you play?
Bot: I played with my ball. Do you want to play too?
```

Good fits: an offline conversational toy or companion on an ESP32 / handheld;
a teaching example of an end-to-end on-device LLM; a tiny base to fine-tune for
embedded chat. **Not** a fit: anything needing facts, instructions, reasoning,
or safety guarantees.

## What it is, technically

- **Architecture:** GPT-Neo (`GPTNeoForCausalLM`) — 8 layers, hidden size 128,
  16 heads, alternating global/local attention (window 256), learned position
  embeddings, tied input/output embeddings, GPT-2 byte-level BPE tokenizer
  (vocab 50257).
- **Base:** [`roneneldan/TinyStories-Instruct-3M`](https://huggingface.co/roneneldan/TinyStories-Instruct-3M).
- **Fine-tune:** ~70K filtered, simple-English dialogues from
  [`allenai/SODA`](https://huggingface.co/datasets/allenai/soda), reformatted as
  `User:`/`Bot:` turns, mixed with a slice of
  [`TinyStoriesInstruct`](https://huggingface.co/datasets/roneneldan/TinyStoriesInstruct).
  Loss is masked to the bot replies / story bodies, so the model never trains on
  producing the user's turns.

## Prompt format

Trained on this exact format, with `<|endoftext|>` (token 50256) between
exchanges:

```
User: <message>
Bot: <reply><|endoftext|>
User: <message>
Bot:
```

Feed `User: <message>\nBot: ` and generate until `<|endoftext|>`.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("TheREZOR/TinyTalk")
model = AutoModelForCausalLM.from_pretrained("TheREZOR/TinyTalk")

prompt = "User: hi, what's your name?\nBot:"
ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(
    ids, max_new_tokens=40, do_sample=True, temperature=0.7, top_k=40,
    eos_token_id=tok.eos_token_id,
)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))
```

## Honest limitations

- **Kindergarten English only.** Short, simple sentences.
- **No world knowledge.** Factual questions get friendly confabulation.
- **Short memory.** Trained/served with a tiny context (~80 tokens on device,
  256 max). Not instruction-following, not safe for any production use.
- A toy/educational model — interesting because it fits on a microcontroller,
  not because it is good.

## License & attribution

Released under **CC BY 4.0**, the binding term inherited from the SODA training
data. You must retain the following attributions:

- **Base model:** TinyStories-Instruct-3M — Ronen Eldan & Yuanzhi Li,
  *TinyStories: How Small Can Language Models Be and Still Speak Coherent
  English?* (arXiv:2305.07759). Published without an explicit license tag; the
  TinyStories dataset family is CDLA-Sharing-1.0, which places no restriction on
  trained models.
- **Fine-tune data:** SODA (CC BY 4.0) — Kim et al., *SODA: Million-scale
  Dialogue Distillation with Social Commonsense Contextualization*
  (arXiv:2212.10465); and TinyStoriesInstruct (CDLA-Sharing-1.0).
- **Tokenizer:** GPT-2 byte-level BPE — OpenAI GPT-2 (MIT).

See `NOTICE.md` for the full provenance.