Update README.md

README.md

@@ -1,14 +1,189 @@
 ---
 license: apache-2.0
+language:
+- en
+tags:
+- language-model
+- transformer
+- rope
+- swiglu
+- muon
+- from-scratch
+- tiny
+- small
+- decoder-only
 datasets:
 - HuggingFaceFW/fineweb-edu
 - HuggingFaceTB/smollm-corpus
 - SimpleStories/SimpleStories
 - HuggingFaceTB/finemath
 - mlfoundations/dclm-baseline-1.0
-language:
-- en
+pipeline_tag: text-generation
 ---
 
-the readme is going to come just wait a Little more🤧
+# İvme-Conversate-v2-Base (Codenamed Apple 2)
+
+![Conversate-v2 Logo](https://cdn-uploads.huggingface.co/production/uploads/670562d6ac129959c16f84d4/9oWY_UmFyQ297XmaZEK7G.png)
+
+**İvme** (Turkish: *acceleration*) is a series of stupidly small language models built to punch above their weight. This is the second release: a 24M parameter decoder-only base model trained from scratch, this time with a much heavier training diet.
+
+v1 produced grammatically correct sentences that did not really say anything when strung together, fluent without being about anything. v2 keeps almost the exact same architecture as v1 on purpose, and instead fixes the two things that actually mattered: far more training data, and a data mix weighted toward material that teaches a model to stay on topic across sentences.
+
+---
+
+## Model Details
+
+| Parameter | Value |
+|---|---|
+| Architecture | Decoder-only transformer, dense (no loops, no exotic recurrence) |
+| Parameters | 23,846,784 |
+| Layers | 10 |
+| Hidden dim | 384 |
+| FFN dim | SwiGLU |
+| Attention heads | 6, full attention (no GQA) |
+| Context length | 1024 tokens |
+| Vocab size | 16,000 (custom BPE) |
+| Positional encoding | RoPE (θ=10,000) |
+| Normalization | RMSNorm (pre-norm) |
+| Embeddings | Tied input/output |
+| Biases | None |
+
+Nearly every setting above matches v1 on purpose. The point of v2 was to isolate the improvement to data and training, not to a bigger model.
+
+---
+
+## Benchmarks
+
+Benchmarks were run with a custom harness this time. We won't put İvme-Conversate-v2 to leaderboards for now until we verify our benchmark produces real results.
+
+| Benchmark | v1 | v2 |
+|---|---|---|
+| WikiText-2 (byte perplexity) ↓ | 2.96 | **2.50** |
+| BLiMP (macro-average) ↑ | 61.40% | **73.19%** |
+| ARC-Easy ↑ | 30.85% | **43.14%** |
+
+Every metric improved, and the ARC-Easy jump in particular is bigger than you would expect from data and training changes alone at this parameter count.
+
+### BLiMP paradigm breakdown
+
+Strong on core agreement paradigms:
+
+| Paradigm | Accuracy |
+|---|---|
+| existential_there_quantifiers_1 | 99.10% |
+| anaphor_number_agreement | 98.30% |
+| sentential_negation_npi_licensor_present | 98.10% |
+| determiner_noun_agreement_1 | 96.60% |
+| determiner_noun_agreement_2 | 95.40% |
+| anaphor_gender_agreement | 90.70% |
+
+Weaker on long-distance dependencies and island constraints, a known hard case for small models:
+
+| Paradigm | Accuracy |
+|---|---|
+| existential_there_quantifiers_2 | 18.90% |
+| left_branch_island_echo_question | 27.80% |
+| sentential_subject_island | 33.80% |
+| only_npi_scope | 42.60% |
+| complex_NP_island | 50.30% |
+
+This pattern, strong local agreement paired with weaker long-distance syntax, is typical for models at this scale.
+
+---
+
+## Does it actually make more sense now?
+
+None of the benchmarks above directly test whether the model's writing holds together as connected text, which was v1's real problem. Sample output, EMA weights, temperature 0.8, top_k 50:
+
+**Prompt:** "Once upon a time, there was a"
+
+> Once upon a time, there was a man named Peter who lived in a beautiful country called Ireland. His father was a wise old man who was known for his knowledge of geometry. One day, a young girl named Alice came to visit Peter in Ireland. As they walked together, Alice met a boy named Leo, who had big shoes and a big hat. Alice asked him, "What are you doing with that, dear?" Peter told Alice about a time when he was a boy. He was still trying to understand how things worked. One day, a clever fox came to Peter and asked him how he could have fun. Peter smiled and said, "That's because he learned to ride an imaginary plane!" Alice watched as Peter explained how he learned how to climb trees and follow a river. "You've come to know that, Peter," she said, and Emmanuel smiled. They sat down together and talked about their favorite things, like building a bridge to reach a friend and helping a neighbor.
+
+There is an actual through line here. Peter gets introduced with a name and a place, Alice arrives and has a real interaction with him, a fox shows up mid story and gets a response, and the ending calls back to the "learning" theme set up in the first sentence. It is not perfectly logical throughout, but it tracks entities and builds toward something across sentences, which is exactly what v1 could not do.
+
+---
+
+## Training
+
+### Data Mix (~12.85B tokens, roughly 8 to 9x more than v1's 1.57B)
+
+v1 was trained Chinchilla-optimal. v2 deliberately overtrains well past that point, since the model is small and cheap to run regardless.
+
+| Source | Share |
+|---|---|
+| HuggingFaceFW/fineweb-edu | 50% |
+| HuggingFaceTB/smollm-corpus (cosmopedia-v2) | 27% |
+| mlfoundations/dclm-baseline-1.0 | 8% |
+| SimpleStories/SimpleStories | 5% |
+| HuggingFaceTB/finemath (finemath-3plus) | 5% |
+
+Python-Edu was originally planned as a fifth source, but the actual code text lives behind a gated dataset with no practical way to align it against the sampled subset at this scale, so it was dropped and its share redistributed across the rest.
+
+### Hyperparameters
+
+| Setting | Value |
+|---|---|
+| Optimizer | Muon (body weights) + AdamW (embeddings, norms) |
+| Muon lr | 0.02 |
+| AdamW lr | 3e-4 |
+| LR schedule | Warmup-Stable-Decay (WSD) |
+| Weight decay | 0.1 |
+| Gradient clipping | 1.0 |
+| Batch size | 192 sequences x 1024 tokens (196,608 tokens/step) |
+| Total steps | 65,376 |
+| Precision | bfloat16 |
+| Attention | PyTorch scaled_dot_product_attention (Flash Attention backend) |
+| Compilation | torch.compile, roughly 2x throughput over eager |
+| Final weights | EMA (β=0.999) of training trajectory |
+
+### Hardware
+
+Trained on a single NVIDIA RTX PRO 6000 Blackwell (96GB) in approximately **4.75 hours**.
+
+---
+
+## Tokenizer
+
+Custom byte-level BPE tokenizer trained from scratch on a sample of the pretraining mix. Vocab size 16,000.
+
+---
+
+## Limitations
+
+- Base model only, not instruction tuned, will not follow instructions or answer questions
+- English only
+- 1024 token context window
+- Weaker on long-distance syntactic dependencies than on local agreement, see BLiMP breakdown above
+- No code data in the training mix (Python-Edu was dropped, see above)
+
+---
+
+## What's Next
+
+Still on the table for a future version: distillation from a larger teacher model, and a return to the more experimental İvmetron architecture once time allows for the kind of patient debugging a genuinely novel design needs.
+
+You can check our other upcoming models on our organization card!
+
+---
+
+## Citation
+
+```bibtex
+@misc{ivme-conversate-v2-24m,
+  author       = {IvmeLabs},
+  title        = {İvme-Conversate-v2-Base},
+  year         = {2026},
+  publisher    = {Hugging Face},
+  url          = {https://huggingface.co/IvmeLabs/Ivme-Conversate-v2-24M-Base}
+}
+```
+
+---
+
+## Credits
+
+The apple photo by <a href="https://unsplash.com/@cenali?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Matheus Cenali</a> on <a href="https://unsplash.com/photos/bunch-of-red-apples-wXuzS9xR49M?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText">Unsplash</a>
+
+---
 
+*Built by IvmeLabs. Small models, deliberate choices.*