Flatbot-micro-4M

Try it online: https://chat.flatseek.io

Flatbot-micro-4M is the flagship demonstration model for Flatseek.

It is a compact conversational language model trained entirely from scratch using the Flatseek ecosystem. Rather than competing with large language models, Flatbot demonstrates the complete lifecycle of building an AI model—from dataset creation and tokenizer training to transformer training, checkpoint export, and inference.

Powered by a 4.1M-parameter language model in just 20 MB, Flatbot was trained from scratch on approximately 2,500 conversational examples, built with FlatBuild, and served by FlatRun.

Experimental model: Expect occasional hallucinations, factual errors, and inconsistent responses. Flatbot is intended for learning, experimentation, and showcasing the Flatseek ecosystem rather than production use.

Architecture

Component	Details
Architecture	Decoder-only Transformer
Position Encoding	RoPE
Normalization	RMSNorm
Feed Forward	SwiGLU
Attention	Grouped Query Attention (8 Query Heads / 4 KV Heads)
Weight Tying	Yes
Context Length	384
Parameters	~4.1M

Model Configuration

vocab_size      = 512
hidden_size     = 256
num_layers      = 6
num_heads       = 8
num_kv_heads    = 4
head_dim        = 32
ffn_dim         = 768
context_length  = 384
rope_theta      = 10000

Training

The model was trained completely from random initialization using FlatBuild.

Dataset

Property	Value
Conversations	~2,500
Multi-turn conversations	~55%
Train Split	95%
Validation Split	5%
Context Length	384 tokens

The dataset contains general-purpose conversations including greetings, introductions, recommendations, explanations, simple reasoning, follow-up questions, and everyday dialogue.

Training Configuration

Hyperparameter	Value
Optimizer	AdamW
Learning Rate	1e-3
Scheduler	Cosine
Warmup	20 steps
Epochs	12
Batch Size	4
Gradient Accumulation	2
Precision	FP32

Features

• Trained entirely from scratch
• Custom tokenizer
• Native chat template
• RoPE positional embeddings
• RMSNorm
• SwiGLU
• Grouped Query Attention (GQA)
• Weight-tied embeddings
• SafeTensors export
• GGUF export
• Native FlatRun compatibility

Usage

FlatBuild Training

pip install flatbuild
flatbuild train config/demo_chat.yml
flatbuild export output/demo_chat/*/checkpoint/final --format gguf --output flatbot-micro-4M

LM Studio

lms import flatbot-micro-4M/model.gguf

FlatRun Inference

pip install flatrun
flatrun chat --model flatbot-micro-4M.gguf --temp 0.2

Detected format: gguf
Building tokenizer from GGUF metadata (flatbot-micro-4M.gguf) ...
Tokenizer vocab: 516
Chat template: {% if messages and messages[0]['role'] != 'system' %}{{ 'You...
Loaded model in 0.01 s; layers=6

Chat mode (max_new=128/turn, history=True). Type your message; Ctrl-D (EOF) or 'exit' to quit.

You: Who are you?
Assistant:
Sure — I'm Flatbot — I'm Flatbot — an assistant trained on a sm a sm chat assistant.
  (30 tokens, 0.9s, 35.0 tok/s)

Purpose

Flatbot-micro-4M demonstrates the complete Flatseek pipeline:

1. Build a dataset
2. Train a tokenizer
3. Configure a transformer
4. Train from scratch
5. Export checkpoints
6. Convert to GGUF
7. Run inference with FlatRun

The entire workflow is reproducible on consumer hardware and is designed for education, experimentation, and research.

Limitations

Flatbot-micro-4M is a compact language model trained on a relatively small dataset. Compared with modern large language models, it has limited factual knowledge, reasoning ability, and context capacity. Its primary purpose is to demonstrate an end-to-end open-source training and inference pipeline, not to serve as a production-grade AI assistant.