README.md

---
license: cc-by-nc-sa-4.0
base_model: roneneldan/TinyStories-Instruct-8M
datasets:
- allenai/soda
- roneneldan/TinyStoriesInstruct
- li2017dailydialog/daily_dialog
- allenai/sciq
language:
- en
pipeline_tag: text-generation
tags:
- gpt_neo
- chat
- conversational
- tiny
- esp32
- cardputer
---

# TinyTalk 2

A 8M-parameter chatbot that runs **fully offline on an ESP32-S3 microcontroller**
(M5Stack Cardputer, 512 KB SRAM, no PSRAM) at ~5 tokens/s — and, of course, on
anything bigger. The successor to [TinyTalk](https://huggingface.co/TheREZOR/TinyTalk).

## What it's for

Small talk with multi-turn memory, TinyStories-style story writing, simple
kindergarten Q&A (colors, animal sounds, opposites, baby animals), and —
new in TinyTalk 2 — **graceful ignorance**: questions beyond a tiny model get a
friendly "I don't know" instead of confabulation.

It was built as the brain of the
[cardputer-ai firmware](https://github.com/therezor/cardputer-ai), where it runs
Q4_0-quantized with an int4 KV cache and a hand-written ESP32-S3 PIE SIMD kernel.

## What's new vs TinyTalk 1

| | TinyTalk 1 (3M) | TinyTalk 2 (8M) |
|---|---|---|
| Base model | TinyStories-Instruct-3M | TinyStories-Instruct-8M |
| Held-out masked val loss | 1.838 | **1.486** |
| Kindergarten facts answered (8-prompt battery) | 1/8 | **7/8** |
| "I don't know" on impossible questions | 6/8 | **7/8** |
| Training dialogues | ~70K (SODA prefix filter, 47% yield) | ~124K (SODA *window* filter 85% yield + DailyDialog) |
| Extra skills | — | templated kindergarten QA + SciQ-question deflection pairs |

## What it is technically

GPT-Neo architecture: 8 layers, hidden size 256, 16 heads, alternating
global/local attention (window 256), GPT-2 byte-level BPE, tied embeddings.
Fine-tuned for 2 epochs (~47M chars) with **masked loss** — loss only on bot
replies, story bodies and EOS, so it learns to answer and to stop, never to
imitate users.

## Prompt format

```
User: <message>
Bot: <reply><|endoftext|>
User: <message>
Bot:
```

Story mode: `Summary: <what the story is about>\nStory:`

A `chat_template` is embedded, so `tokenizer.apply_chat_template()` produces
this format automatically.

## Usage

```python
from transformers import GPTNeoForCausalLM, GPT2TokenizerFast

model = GPTNeoForCausalLM.from_pretrained("TheREZOR/TinyTalk-2")
tok = GPT2TokenizerFast.from_pretrained("TheREZOR/TinyTalk-2")

msgs = [{"role": "user", "content": "what sound does a dog make?"}]
ids = tok(tok.apply_chat_template(msgs, tokenize=False), return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=40, do_sample=True,
                     temperature=0.8, top_p=0.9,
                     eos_token_id=tok.eos_token_id,
                     pad_token_id=tok.eos_token_id)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
# " A dog says woof!"
```

GGUF builds for **llama.cpp / Ollama** are at
[TheREZOR/TinyTalk-2-GGUF](https://huggingface.co/TheREZOR/TinyTalk-2-GGUF):

```
ollama run hf.co/TheREZOR/TinyTalk-2-GGUF
```

## Honest limitations

This is a toy/educational model. Kindergarten English; no world knowledge
beyond ~150 hand-written nursery facts; context trained to 256 tokens;
anything outside its lane gets a (trained) polite deflection — usually.
Do not use it for anything that matters.

## License & attribution

**CC BY-NC-SA 4.0 (non-commercial).** TinyTalk 1 was CC BY 4.0; TinyTalk 2
additionally trains on [DailyDialog](https://huggingface.co/datasets/li2017dailydialog/daily_dialog)
(CC BY-NC-SA 4.0) and question texts from
[SciQ](https://huggingface.co/datasets/allenai/sciq) (CC BY-NC 3.0), so the
most restrictive license is inherited.

- Base: [roneneldan/TinyStories-Instruct-8M](https://huggingface.co/roneneldan/TinyStories-Instruct-8M)
  (Eldan & Li, *TinyStories*, arXiv:2305.07759)
- [allenai/soda](https://huggingface.co/datasets/allenai/soda) (CC BY 4.0),
  Kim et al., arXiv:2212.10465
- [roneneldan/TinyStoriesInstruct](https://huggingface.co/datasets/roneneldan/TinyStoriesInstruct)
  (CDLA-Sharing-1.0)
- DailyDialog: Li et al., arXiv:1710.03957 · SciQ: Welbl et al., arXiv:1707.06209