README.md

---
license: apache-2.0
language:
- en
library_name: flatbuild
tags:
- language-model
- conversational
- flatseek
- flatbuild
- flatrun
- trained-from-scratch
- rope
- rmsnorm
- swiglu
- gqa
pipeline_tag: text-generation
---

# Flatbot-micro-4M

**Try it online:** https://chat.flatseek.io

Flatbot-micro-4M is the flagship demonstration model for **Flatseek**.

It is a compact conversational language model trained entirely from scratch using the Flatseek ecosystem. Rather than competing with large language models, Flatbot demonstrates the complete lifecycle of building an AI model—from dataset creation and tokenizer training to transformer training, checkpoint export, and inference.

Powered by a **4.1M-parameter** language model in just **20 MB**, Flatbot was trained from scratch on approximately **2,500 conversational examples**, built with **FlatBuild**, and served by **FlatRun**.

> **Experimental model:** Expect occasional hallucinations, factual errors, and inconsistent responses. Flatbot is intended for learning, experimentation, and showcasing the Flatseek ecosystem rather than production use.

## Architecture

| Component | Details |
|---|---|
| Architecture | Decoder-only Transformer |
| Position Encoding | RoPE |
| Normalization | RMSNorm |
| Feed Forward | SwiGLU |
| Attention | Grouped Query Attention (8 Query Heads / 4 KV Heads) |
| Weight Tying | Yes |
| Context Length | 384 |
| Parameters | ~4.1M |

## Model Configuration

```text
vocab_size      = 512
hidden_size     = 256
num_layers      = 6
num_heads       = 8
num_kv_heads    = 4
head_dim        = 32
ffn_dim         = 768
context_length  = 384
rope_theta      = 10000
```

## Training

The model was trained completely from random initialization using FlatBuild.

### Dataset

| Property | Value |
|---|---:|
| Conversations | ~2,500 |
| Multi-turn conversations | ~55% |
| Train Split | 95% |
| Validation Split | 5% |
| Context Length | 384 tokens |

The dataset contains general-purpose conversations including greetings, introductions, recommendations, explanations, simple reasoning, follow-up questions, and everyday dialogue.

## Training Configuration

| Hyperparameter | Value |
|---|---:|
| Optimizer | AdamW |
| Learning Rate | 1e-3 |
| Scheduler | Cosine |
| Warmup | 20 steps |
| Epochs | 12 |
| Batch Size | 4 |
| Gradient Accumulation | 2 |
| Precision | FP32 |

## Features

- Trained entirely from scratch
- Custom tokenizer
- Native chat template
- RoPE positional embeddings
- RMSNorm
- SwiGLU
- Grouped Query Attention (GQA)
- Weight-tied embeddings
- SafeTensors export
- GGUF export
- Native FlatRun compatibility

## Usage

### FlatBuild Training

```bash
pip install flatbuild
flatbuild train config/demo_chat.yml
flatbuild export output/demo_chat/*/checkpoint/final --format gguf --output flatbot-micro-4M
```

### LM Studio

```bash
lms import flatbot-micro-4M/model.gguf
```

### FlatRun Inference

```bash
pip install flatrun
flatrun chat --model flatbot-micro-4M.gguf --temp 0.2
```

```text
Detected format: gguf
Building tokenizer from GGUF metadata (flatbot-micro-4M.gguf) ...
Tokenizer vocab: 516
Chat template: {% if messages and messages[0]['role'] != 'system' %}{{ 'You...
Loaded model in 0.01 s; layers=6

Chat mode (max_new=128/turn, history=True). Type your message; Ctrl-D (EOF) or 'exit' to quit.

You: Who are you?
Assistant:
Sure — I'm Flatbot — I'm Flatbot — an assistant trained on a sm a sm chat assistant.
  (30 tokens, 0.9s, 35.0 tok/s)
```

## Purpose

Flatbot-micro-4M demonstrates the complete Flatseek pipeline:

1. Build a dataset
2. Train a tokenizer
3. Configure a transformer
4. Train from scratch
5. Export checkpoints
6. Convert to GGUF
7. Run inference with FlatRun

The entire workflow is reproducible on consumer hardware and is designed for education, experimentation, and research.

## Limitations

Flatbot-micro-4M is a compact language model trained on a relatively small dataset. Compared with modern large language models, it has limited factual knowledge, reasoning ability, and context capacity. Its primary purpose is to demonstrate an end-to-end open-source training and inference pipeline, not to serve as a production-grade AI assistant.