Update README.md

README.md

@@ -0,0 +1,195 @@
+---
+license: apache-2.0
+language:
+- en
+- es
+- id
+- it
+tags:
+- byte-level
+- tiny-lm
+- micro-model
+- research
+- from-scratch
+- rope
+- swiglu
+- muon-optimizer
+- pytorch
+- custom-code
+pipeline_tag: text-generation
+---
+
+<div align="center">
+<img src="MicroGen.png" alt="NoTokenLM MicroGen" width="100%">
+
+# NoTokenLM · MicroGen series
+
+**Tokenizer-free, byte-level language models, trained entirely from scratch**
+
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/#https://huggingface.co/omurberaisik/NoTokenLM-MicroGen/blob/main/NoTokenLM_MicroGen_Use.ipynb)
+
+</div>
+
+---
+
+## What is this?
+
+MicroGen is a research series of tiny, from-scratch language models — ranging from about 1,900 to 500,000 parameters — trained directly on **raw UTF-8 bytes**, with no tokenizer at all. This is a research project, not a product: the goal is to find out how far a model can be pushed toward real language behavior while staying this small, and to document that process honestly at every step.
+
+Each version builds on the previous one by changing roughly one thing at a time: first depth and attention, then optimizer, then data quality, then multilinguality. Read together, the series works as a practical log of small-scale language model training — including the mistakes.
+
+**All models live in this single repository.** Each is a separate checkpoint; the model picker below lets you choose which one to load and use.
+
+> This is a research artifact, not an assistant. None of these models are instruction-tuned or aligned in any way — they are raw next-byte predictors. See the honest limitations section before you draw any conclusions from their output.
+
+---
+
+## Models
+
+| Model | Parameters | Architecture summary | Context | Languages |
+|---|---:|---|---:|---|
+| [2.5](#microgen-25) | ~1.9K | Single layer, single attention head | 96 bytes | EN |
+| [2.6](#microgen-26) | ~15K | 3 layers, RoPE + SwiGLU, multi-head attention | 128 bytes | EN |
+| [2.7](#microgen-27) | ~60K | 4 layers, Muon optimizer + Flash Attention | 128 bytes | EN |
+| [3.5](#microgen-35) | ~345K | 5 layers, WSD scheduler, bf16, FineWeb-Edu | 256 bytes | EN |
+| [3.6](#microgen-36) | ~500K | 6 layers, trained with a focus on meaning/topic coherence | 256 bytes | EN |
+| [3.7](#microgen-37) | ~500K | Same architecture as 3.6, multilingual data | 256 bytes | EN, ES, ID, IT |
+
+All models: byte-level input (vocab size 256, no tokenizer), causal decoder-only, trained from random initialization — no pretrained weights were used anywhere in this series.
+
+**Through 3.6, every model in this series is single-language (English).** 3.7 is the first and only multilingual model in the series.
+
+**Data budgets were targets, not guarantees.** With the exception of 2.5 (which did complete its 20GB target), none of the other models finished the amount of data they were originally budgeted for — training was stopped earlier for practical reasons (time/compute), so treat the "target data" figures below as an upper bound the model was moving toward, not a completed run.
+
+---
+
+## MicroGen 2.5
+
+**~1,925 parameters.** The smallest model in the series — a single attention head, a single layer, `d_model=5`. Targeted 20GB of FineWeb data in streaming mode, and is the only model in the series that actually completed its full data target. This model exists purely to probe whether a model at this scale can produce anything beyond noise — it is not expected to produce coherent text.
+
+**Example outputs** *(temperature=0.5)*:
+
+```
+Prompt 1: 
+Prompt 2: 
+Prompt 3: 
+```
+
+---
+
+## MicroGen 2.6
+
+**~14,952 parameters** (8x larger than 2.5). The first model in the series to use RoPE (rotary position embeddings) and SwiGLU activations; 3 layers, 4-head attention. Targeted 30GB of FineWeb but did not complete that budget before training was stopped. Fixes an initialization bug found in 2.5, where the default PyTorch init combined with tied embeddings caused logits to blow up.
+
+**Example outputs** *(temperature=0.5)*:
+
+```
+Prompt 1: 
+Prompt 2: 
+Prompt 3: 
+```
+
+---
+
+## MicroGen 2.7
+
+**~59,952 parameters.** The last and largest model in the 2.x line. 4 layers, a Muon optimizer (Newton-Schulz orthogonalization for matrix parameters) combined with AdamW, and PyTorch's native Flash Attention (SDPA). Targeted 35GB of FineWeb, again not fully completed. **This is the point in the series where the model starts producing actual, real words** rather than word-shaped noise — output is still far from consistently coherent, but recognizable English words start appearing reliably from here on.
+
+**Example outputs** *(temperature=0.5)*:
+
+```
+Prompt 1: 
+Prompt 2: 
+Prompt 3: 
+```
+
+---
+
+## MicroGen 3.5
+
+**~344,768 parameters** (5.75x larger than 2.7). 5 layers, 8-head attention, context extended to 256 bytes, data source switched to the education-filtered **FineWeb-Edu**. The WSD (warmup-stable-decay) learning rate schedule was made data-budget-based rather than step-based, after 2.7's fixed step estimate turned out to be off by roughly 53%. bf16 mixed precision was introduced. Targeted 60GB of FineWeb-Edu; did not complete that budget. This is the point where grammar becomes noticeably real — still weak, but genuinely present.
+
+**Example outputs** *(temperature=0.5)*:
+
+```
+Prompt 1: 
+Prompt 2: 
+Prompt 3: 
+```
+
+---
+
+## MicroGen 3.6
+
+**~500,560 parameters.** 3.5 showed real syntactic structure (correct conjunctions and subject-verb agreement, e.g. "While X are Y") but no semantic consistency (subject/object mismatches like "cards = employees"). 3.6's goal was to keep that grammatical foundation and build topic and meaning coherence on top of it. 6 layers, `d_model=80` (head_dim=10 — RoPE requires an even head_dim, which is why an earlier d_model=88 attempt with head_dim=11 failed). Data moved to a chained multi-source stream: a small but high-quality `Ultra-FineWeb-EDU` subset first, falling back automatically to the much larger `Ultra-FineWeb`, then to `FineWeb-Edu` as a last resort. Training did not complete its full intended data budget.
+
+**Example outputs** *(temperature=0.5)*:
+
+```
+Prompt 1: 
+Prompt 2: 
+Prompt 3: 
+```
+
+---
+
+## MicroGen 3.7
+
+**~500,560 parameters — identical architecture and parameter count to 3.6.** The only variable changed was data: Spanish, Indonesian, and Italian were added alongside English. Mixture ratio is intentionally uneven — **EN 55% / ES 15% / ID 15% / IT 15%** — because the model was still making meaning-level mistakes in English alone, and an even split across four languages risked learning none of them well.
+
+**3.7 is stronger than 3.6 in English too**, not just multilingual — the added data and continued training moved English quality forward as well, on top of adding three new languages. In Spanish, Indonesian, and Italian, quality is clearly behind English: the model reliably picks the right language (it doesn't answer in the wrong one) but can drop or garble words within a sentence. Training did not complete its full intended data budget.
+
+**Example outputs** *(temperature=0.5)*:
+
+```
+[EN] Prompt 1: 
+[EN] Prompt 2: 
+[ES] Prompt: 
+[ID] Prompt: 
+[IT] Prompt: 
+```
+
+---
+
+## Honest limitations
+
+- **These are not chat assistants.** None of them are instruction-tuned, none have seen RLHF. They are raw, from-scratch pretrained models that continue text — nothing more.
+- **No version achieves full coherence.** Even the largest/newest models (3.6, 3.7) can lose the topic, drop subject-object agreement, or produce nonsense words after a few sentences.
+- **None of the models finished their targeted training data budget, except 2.5.** Every model from 2.6 onward was stopped before reaching its original data target for practical reasons; the parameter counts and architectures are final, but none of them (aside from 2.5) represent a fully "completed" training run in the sense of hitting their original data goal.
+- **Evaluation here is qualitative, not a fixed benchmark suite.** As the number of models in the series grew, the amount of testing time available per model shrank — this is a deliberate scope tradeoff, not a hidden result.
+- **Byte-level means the model was never taught what a "word" is** — everything is a UTF-8 byte sequence to it. That's both an advantage (no tokenizer bias) and a disadvantage (learning word boundaries is the model's own job, and that's hard at this scale).
+- **These models are small enough to be noisy** — the same prompt at the same temperature can produce very different quality output run to run.
+
+## More models are coming
+
+This series is actively ongoing. As new MicroGen versions are trained, they will be added to this repository and this README will be updated to reflect them.
+
+---
+
+## How to use these models
+
+Every model in this series is published as a real `transformers`-compatible checkpoint — a `config.json` + `model.safetensors` pair per version, each in its own subfolder of this repo, sharing a small amount of custom model code (`configuration_notokengen.py` / `modeling_notokengen.py`) at the repo root. That means you can load any of them directly with `trust_remote_code=True`, no separate download step or manual architecture code required:
+
+```python
+from transformers import AutoModelForCausalLM
+
+model = AutoModelForCausalLM.from_pretrained(
+    "omurberaisik/NoTokenLM-MicroGen",
+    subfolder="3.6",          # or "2.5", "2.6", "2.7", "3.5", "3.7"
+    trust_remote_code=True,
+)
+
+# no tokenizer — these models read and write raw UTF-8 bytes
+output = model.generate_bytes("The ", n_new_bytes=200, temperature=0.5)
+print(output)
+```
+
+Or use the model picker below to jump straight to a ready-to-run notebook — it loads whichever model you pick and runs generation for you:
+
+[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/#https://huggingface.co/omurberaisik/NoTokenLM-MicroGen/blob/main/NoTokenLM_MicroGen_Use.ipynb)
+
+> Note: because this single repository hosts six separate checkpoints (one per subfolder) rather than one model at the repo root, Hugging Face's default "Use this model" widget doesn't correctly represent it — use the snippet or the notebook above instead.
+
+## License
+
+Apache-2.0.