Update README.md

README.md

@@ -1,3 +1,168 @@
 ---
 license: apache-2.0
+language:
+- en
+tags:
+- language-model
+- transformer
+- rope
+- swiglu
+- gqa
+- muon
+- from-scratch
+- tiny
+- small
+- decoder-only
+datasets:
+- epfml/FineWeb-HQ
+- HuggingFaceTB/cosmopedia
+- HuggingFaceTB/finemath
+- bigcode/python-stack-v1-functions-filtered
+- wikimedia/wikipedia
+pipeline_tag: text-generation
 ---
+
+# İvme-Conversate-22M-Base
+
+**İvme** (Turkish: *acceleration*) is a series of stupidly small language models built to punch above their weight. This is the first release: a 22M parameter decoder-only base model trained from scratch on a dense, quality-filtered corpus.
+
+The goal is not production deployment. The goal is to see how well a sub-25M model can perform when every decision — architecture, data mix, optimizer, training schedule — is made deliberately.
+
+---
+
+## Model Details
+
+| Parameter | Value |
+|---|---|
+| Architecture | Decoder-only transformer |
+| Parameters | 22,028,160 |
+| Layers | 10 |
+| Hidden dim | 384 |
+| FFN dim | 1024 (SwiGLU) |
+| Attention heads | 6 query / 2 KV (GQA) |
+| Context length | 1024 tokens |
+| Vocab size | 16,384 (custom BPE) |
+| Positional encoding | RoPE (θ=10,000) |
+| Normalization | RMSNorm (pre-norm) |
+| Embeddings | Tied input/output |
+| Biases | None |
+
+---
+
+## Benchmarks
+
+All benchmarks run via [EleutherAI lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness), 0-shot. WikiText-2 uses byte_perplexity for tokenizer-independent comparison.
+
+| Benchmark | Score | Notes |
+|---|---|---|
+| WikiText-2 (byte_perplexity) ↓ | **2.96** | Lower is better |
+| BLiMP ↑ | **61.40%** | Average over 67 subtasks; random baseline 50% |
+| ARC-Easy ↑ | **30.85%** | acc_norm, 0-shot |
+
+---
+
+## Training
+
+### Data Mix (~1.57B tokens, Chinchilla-optimal)
+
+Data is ordered in ascending quality for curriculum learning — the model sees noisier web text first and the densest material last.
+
+| Source | Tokens | Share |
+|---|---|---|
+| epfml/FineWeb-HQ (score > 0.8) | ~710M | 45% |
+| bigcode/python-stack-v1-functions-filtered | ~160M | 10% |
+| HuggingFaceTB/finemath (finemath-4plus) | ~235M | 15% |
+| HuggingFaceTB/cosmopedia (stanford + wikihow) | ~395M | 25% |
+| wikimedia/wikipedia (EN, 20231101) | ~80M | 5% |
+
+### Hyperparameters
+
+| Setting | Value |
+|---|---|
+| Optimizer | Muon (body weights) + AdamW (embeddings, norms) |
+| Muon lr | 0.02 |
+| AdamW lr | 3e-4 |
+| LR schedule | Warmup-Stable-Decay (WSD) |
+| Warmup steps | 100 |
+| Decay fraction | 20% of training |
+| Weight decay | 0.1 |
+| Gradient clipping | 1.0 |
+| Effective batch | ~1.05M tokens/step |
+| Total steps | 1,447 |
+| Precision | bfloat16 |
+| Attention | Flash Attention 2 (HF Kernels) |
+| Final weights | EMA (β=0.999) of training trajectory |
+
+### Hardware
+
+Trained on a single NVIDIA RTX PRO 6000 Blackwell (96GB) in approximately **20 minutes**.
+
+---
+
+## Tokenizer
+
+Custom BPE tokenizer trained from scratch on a balanced sample of the pretraining corpus. Vocab size 16,384 with ByteLevel pre-tokenization.
+
+Special tokens: `<|pad|>`, `<|bos|>`, `<|eos|>`, `<|unk|>`, `<|user|>`, `<|assistant|>`, `<|system|>`
+
+---
+
+## Usage
+
+```python
+import torch
+from tokenizers import Tokenizer
+
+# Load with custom code (not a standard HF AutoModel — see model.py)
+from model import IvmeConfig, IvmeConversate
+
+tokenizer = Tokenizer.from_file("ivme_tokenizer.json")
+ckpt = torch.load("ivme_base_ema.pt", map_location="cuda", weights_only=False)
+cfg = ckpt["cfg"]
+cfg.attn_backend = "sdpa"  # or "kernels" for HF Kernels flash-attn
+model = IvmeConversate(cfg).cuda()
+model.load_state_dict(ckpt["model"])
+model.eval()
+
+prompt = "The theory of relativity states that"
+ids = torch.tensor([tokenizer.encode(prompt).ids], device="cuda")
+out = model.generate(ids, max_new_tokens=100, temperature=0.8, top_k=40)
+print(tokenizer.decode(out[0].tolist()))
+```
+
+---
+
+## Limitations
+
+- Base model only — not instruction tuned, will not follow instructions or answer questions
+- English only (v1)
+- Limited factual knowledge due to Chinchilla-optimal training (1.57B tokens)
+- Repetition at higher temperatures without `repetition_penalty`
+- 1024 token context window
+
+---
+
+## What's Next
+
+- **İvme-Conversate-22M-Instruct** — SFT on smol-smoltalk for instruction following
+- **İvme-Conversate-v2** — extended training (~15B tokens), reordered curriculum
+- **Turkish support** — v2 will add EN+TR with a dedicated bilingual tokenizer
+- **İvme-Classify** — encoder-only series for classification tasks
+
+---
+
+## Citation
+
+```bibtex
+@misc{ivme-conversate-22m,
+  author       = {IvmeLabs},
+  title        = {İvme-Conversate-22M-Base},
+  year         = {2026},
+  publisher    = {Hugging Face},
+  url          = {https://huggingface.co/IvmeLabs/Ivme-Conversate-22M-Base}
+}
+```
+
+---
+
+*Built by IvmeLabs. Small models, deliberate choices.*