Upload README.md with huggingface_hub

README.md

@@ -7,26 +7,55 @@ sdk: static
 pinned: false
 ---
 # ASTERIZER
-**Building AI from the model to the device** — self-hosted language models, a memory layer, application framework, and on-device AI, engineered in-house with 100% owned IP.
+
+**Building AI from the model to the device.** In-house LLMs trained from scratch (100% owned IP), a multilingual tokenizer, a memory layer and on-device inference — no third-party weights.
 
 **Models (trained from scratch)**
 - [LUNA-100M](https://huggingface.co/ASTERIZER/LUNA-100M) — 4.5B-token pretrain + RAG/MCP SFT
 - [LUNA-300M](https://huggingface.co/ASTERIZER/LUNA-300M) — 4.5B-token pretrain (2.7x scale-up)
-- [LUNA-1B](https://huggingface.co/ASTERIZER/LUNA-1B-Instruct) — 32B-token continued pretraining + instruction tuning (11x scale-up)
+- [Ezaris-1B](https://huggingface.co/ASTERIZER/Ezaris-Instruct) — 32B-token continued pretraining + SFT/Instruct (11x scale-up)
 
-**Scaling benchmarks — LUNA-100M -> 300M -> 1B** *(latest base checkpoints, held-out multilingual eval on RTX 4060 Ti, higher is better)*
+**Scaling — LUNA-100M → LUNA-300M → Ezaris-1B** *(latest base checkpoints, held-out multilingual eval, RTX 4060 Ti, higher is better)*
 
-| metric | LUNA-100M | LUNA-300M | LUNA-1B |
+| metric | LUNA-100M | LUNA-300M | Ezaris-1B |
 |---|---|---|---|
 | Parameters | 109M | 303M | 1.21B |
-| Next-token accuracy (top-1) | 32.5% | 32.7% | **40.5%** |
-| Next-token accuracy (top-5) | 56.0% | 57.6% | **60.3%** |
-| Word-prediction (LAMBADA-style) | 17.9% | 19.0% | -* |
-| Throughput (RTX 4060 Ti) | 40.4K tok/s | 14.6K tok/s | 6.8K tok/s |
+| Next-token acc (top-1) | 32.5% | 32.7% | **40.5%** |
+| Next-token acc (top-5) | 56.0% | 57.6% | **60.3%** |
+| Throughput | 40.4K tok/s | 14.6K tok/s | 6.8K tok/s |
+
+*LAMBADA-style exact-match is not cross-tokenizer comparable (the 128K tokenizer merges word+punctuation into single tokens), so it is omitted here; next-token columns are the consistent family comparison.
+
+**vs top open same-tier models** *(same held-out corpus, each model with its own tokenizer, RTX 4060 Ti, bf16)*
+
+| model | train tokens | params | next-tok top-1 | bits/byte (lower=better) | LAMBADA | tok/s |
+|---|---|---|---|---|---|---|
+| **Ezaris-1B (ours)** | 32B CPT | 1.21B | 40.5% | 0.98 | 12.9% | 6.8K |
+| SmolLM2-1.7B | 11T | 1.71B | 64.2% | 0.70 | 35.1% | 8.9K |
+| Qwen2.5-1.5B | ~18T | 1.54B | 58.7% | 0.69 | 25.7% | 6.1K |
+| TinyLlama-1.1B | 3T | 1.10B | 59.3% | 0.78 | 35.1% | 4.7K |
+
+Honest notes:
+- Ezaris-1B is a 32B-token continued-pretraining base (~1/100th of the reference models' training tokens). The quality gap (bits/byte 0.98 vs 0.69–0.78) is real and is the focus of our ongoing CPT/SFT program.
+- Per-position next-token favors fine tokenizers (TinyLlama's 32K = many easy positions; Ezaris's 128K = fewer, harder positions with 4x the vocab classes). bits/byte is the tokenizer-independent metric.
+- Where Ezaris wins by design: ~2x fewer tokens per text, ~2x faster end-to-end per text, native Indic coverage, and code next-token accuracy 66% (beats TinyLlama's 60%).
+
+**Tokenizer — multilingual BPE (tokens per 1,000 bytes, lower is better)**
+
+| tokenizer | English | Kannada | Telugu | Tamil | Hindi | Code | ALL |
+|---|---|---|---|---|---|---|---|
+| GPT-NeoX 50K (baseline) | 202 | 640 | 571 | 511 | 386 | 268 | 319 |
+| LUNA 50K (100M/300M) | 202 | 640 | 571 | 511 | 386 | 268 | 319 |
+| **Ezaris 128K** | 219 | **102** | **98** | **90** | 127 | 276 | 178 |
+| Ezaris 64K | 235 | 118 | 113 | 101 | 147 | 305 | 193 |
+| Ezaris 32K | 260 | 139 | 131 | 119 | 174 | 338 | 217 |
+| Sarvam-1 | 241 | 114 | 114 | 103 | **101** | 323 | 193 |
+| OpenAI o200k (GPT-4o) | **200** | 145 | 137 | 130 | 122 | **222** | **176** |
+| Llama-3 128K | 202 | 640 | 639 | 495 | 199 | 218 | 301 |
 
-*LUNA-1B uses a 128K multilingual tokenizer that merges word+punctuation into single tokens, so LAMBADA exact-match is not cross-tokenizer comparable; the next-token columns are the consistent comparison (and a larger vocab makes exact-match strictly harder).
+Strengths: Ezaris-128K cuts Kannada/Telugu/Tamil tokens ~84% vs GPT-NeoX, uses 10–14% fewer tokens than Sarvam-1 on Kannada/Telugu/Tamil, halves token count vs SmolLM2/TinyLlama, and ties OpenAI o200k on the mixed corpus.
+Limitations (honest): ~9% behind o200k on English, ~24% behind o200k on code, ~25% behind Sarvam-1 on Hindi — Ezaris deliberately trades some EN/code density for large Indic gains.
 
-**Tokenizer (multilingual BPE)**
-- [LUNA-1B-Tokenizer](https://huggingface.co/ASTERIZER/LUNA-Tokenizer-1B) — 128K / 64K / 8K vocab
-- **2.8x better compression** than GPT-NeoX, **67% fewer tokens** on Indic text (Kannada 7x, Telugu 6x, Tamil 5x)
-- Beats `cl100k_base` on Indic; competitive with Sarvam (34/55 head-to-head wins)
+**Programs**
+- [LUNA-100M](https://huggingface.co/collections/ASTERIZER/luna-100m-program-6a9115383c18e52460bf67c9) · [LUNA-300M](https://huggingface.co/collections/ASTERIZER/luna-300m-program-6a91195ab75671026f83ac94) · [Ezaris](https://huggingface.co/collections/ASTERIZER/luna-1b-program-6a9134d895b9571643e9b1cb)
+- Tokenizers: [LUNA (100M/300M)](https://huggingface.co/ASTERIZER/LUNA-Tokenizer) · [Ezaris (128K/64K/32K)](https://huggingface.co/datasets/ASTERIZER/Ezaris-Tokenizer)