TheREZOR/TinyTalk-2

TinyTalk 2

A 8M-parameter chatbot that runs fully offline on an ESP32-S3 microcontroller (M5Stack Cardputer, 512 KB SRAM, no PSRAM) at ~5 tokens/s — and, of course, on anything bigger. The successor to TinyTalk.

What it's for

Small talk with multi-turn memory, TinyStories-style story writing, simple kindergarten Q&A (colors, animal sounds, opposites, baby animals), and — new in TinyTalk 2 — graceful ignorance: questions beyond a tiny model get a friendly "I don't know" instead of confabulation.

It was built as the brain of the cardputer-ai firmware, where it runs Q4_0-quantized with an int4 KV cache and a hand-written ESP32-S3 PIE SIMD kernel.

What's new vs TinyTalk 1

	TinyTalk 1 (3M)	TinyTalk 2 (8M)
Base model	TinyStories-Instruct-3M	TinyStories-Instruct-8M
Held-out masked val loss	1.838	1.486
Kindergarten facts answered (8-prompt battery)	1/8	7/8
"I don't know" on impossible questions	6/8	7/8
Training dialogues	~70K (SODA prefix filter, 47% yield)	~124K (SODA window filter 85% yield + DailyDialog)
Extra skills	—	templated kindergarten QA + SciQ-question deflection pairs

What it is technically

GPT-Neo architecture: 8 layers, hidden size 256, 16 heads, alternating global/local attention (window 256), GPT-2 byte-level BPE, tied embeddings. Fine-tuned for 2 epochs (~47M chars) with masked loss — loss only on bot replies, story bodies and EOS, so it learns to answer and to stop, never to imitate users.

Prompt format

User: <message>
Bot: <reply><|endoftext|>
User: <message>
Bot:

Story mode: Summary: <what the story is about>\nStory:

A chat_template is embedded, so tokenizer.apply_chat_template() produces this format automatically.

Usage

from transformers import GPTNeoForCausalLM, GPT2TokenizerFast

model = GPTNeoForCausalLM.from_pretrained("TheREZOR/TinyTalk-2")
tok = GPT2TokenizerFast.from_pretrained("TheREZOR/TinyTalk-2")

msgs = [{"role": "user", "content": "what sound does a dog make?"}]
ids = tok(tok.apply_chat_template(msgs, tokenize=False), return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=40, do_sample=True,
                     temperature=0.8, top_p=0.9,
                     eos_token_id=tok.eos_token_id,
                     pad_token_id=tok.eos_token_id)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
# " A dog says woof!"

GGUF builds for llama.cpp / Ollama are at TheREZOR/TinyTalk-2-GGUF:

ollama run hf.co/TheREZOR/TinyTalk-2-GGUF

Honest limitations

This is a toy/educational model. Kindergarten English; no world knowledge beyond ~150 hand-written nursery facts; context trained to 256 tokens; anything outside its lane gets a (trained) polite deflection — usually. Do not use it for anything that matters.

License & attribution

CC BY-NC-SA 4.0 (non-commercial). TinyTalk 1 was CC BY 4.0; TinyTalk 2 additionally trains on DailyDialog (CC BY-NC-SA 4.0) and question texts from SciQ (CC BY-NC 3.0), so the most restrictive license is inherited.

• Base: roneneldan/TinyStories-Instruct-8M (Eldan & Li, TinyStories, arXiv:2305.07759)
• allenai/soda (CC BY 4.0), Kim et al., arXiv:2212.10465
• roneneldan/TinyStoriesInstruct (CDLA-Sharing-1.0)
• DailyDialog: Li et al., arXiv:1710.03957 · SciQ: Welbl et al., arXiv:1707.06209