kitchenbot-base

A ~6.85M GPT-2-style language model trained from scratch on cooking recipes over a weekend — part of a hands-on experiment to learn the full pretrain → chat-SFT loop on a single rented GPU.

Chat fine-tune	bychwa/kitchenbot-chat (LoRA adapter)
Training code	github.com/bychwa/kitchenbot
Logs	wandb · kitchenbot

Motivation

I wanted a real end-to-end run I could finish in a weekend: niche data, custom tokenizer, pretrain a small causal LM, then LoRA-tune it for Q&A. Keeping the model tiny was intentional — fit the whole loop on one RTX 3090 pod, focus on process, and ship working artifacts.

Model details

Architecture	GPT2LMHeadModel
Parameters	~6.85M
Layers / emb / heads	6 / 256 / 8
Context	256 tokens
Vocab	8000 (ByteLevel BPE, trained on the recipe corpus)
Special tokens	<|pad|>, <|unk|>, <|bos|>, <|eos|>, <|user|>, <|assistant|>

Weights are ~26 MB (safetensors).

Training data

Source: idoyaaran/mise-recipes (streamed from the Hub).

Each example was formatted roughly as:

<|bos|>{title}
Ingredients: {ingredient names}
Steps: {joined steps}<|eos|>

The weekend run used ~10k recipes (MAX_SAMPLES=10000 in the training script).

Hardware (RunPod)

Spec	Value
GPU	1× NVIDIA GeForce RTX 3090 (24 GB)
CUDA	13.0
Host	Linux (RunPod container)
Python	3.12
Stack	PyTorch 2.5.1+cu121, Transformers 5.14, Datasets, Tokenizers, Accelerate, W&B

Approximate cost: a few dollars at ~$0.25–0.40/hr for a short pretrain + SFT session.

Training procedure

Causal language modeling (next-token prediction) with Hugging Face Trainer.

Hyperparameter	Value
Learning rate	3e-4
Warmup steps	100
Batch size	16
Grad accumulation	4 (effective batch 64)
Epochs	1
Max length	256
Precision	fp16
Optimizer	AdamW
LR schedule	linear
Seed	42

Results (train)

Metric	Value
Steps	157
Train runtime	~24 s (this config on 3090)
train_loss	≈ 5.88
Last logged step loss	≈ 4.19

These are training metrics, not a held-out perplexity suite. Loss trended down; the model learns recipe-ish continuations, not general knowledge.

Intended use

• Recipe-style text continuation in the cooking domain
• Base weights for the LoRA chat adapter kitchenbot-chat
• Teaching / portfolio example of a from-scratch SLM pipeline

Not intended as a general assistant, medical/nutrition advice source, or production kitchen system.

Limitations

• Tiny capacity → invents ingredients/steps and mixes dishes
• 256-token context truncates long recipes
• English cooking text bias from the source dataset
• No safety / factuality filtering

How to use

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "bychwa/kitchenbot-base"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

prompt = "<|bos|>Simple Tomato Sauce\nIngredients: tomatoes, garlic, olive oil\nSteps:"
inputs = tok(prompt, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=80, do_sample=True, temperature=0.8)
print(tok.decode(out[0], skip_special_tokens=False))

For Q&A chat, load this base plus the LoRA adapter — see bychwa/kitchenbot-chat.

Reproduce

Full scripts (uv setup, corpus → tokenizer → pretrain → SFT → CLI):

https://github.com/bychwa/kitchenbot

export HF_USER=bychwa
export WANDB_PROJECT=kitchenbot
python scripts/03_pretrain_base.py

License

Apache-2.0 for the model code/weights packaging in this card’s training setup. Respect the license/terms of idoyaaran/mise-recipes for the underlying text.