Upload README.md with huggingface_hub

README.md

@@ -7,110 +7,86 @@ library_name: pytorch
 tags:
 - text-generation
 - bilingual
-- character-level
+- BPE
 - transformer
 - reasoning
 - custom-architecture
-- gqa
+- GQA
 - flash-attention
 datasets:
 - eyanchao/echoic-data
 ---
 
-# Echoic-Lite
+# Echoic
 
-**Bilingual (Chinese/English) character-level language model family — custom Transformer trained from scratch by [eyanchao](https://huggingface.co/eyanchao).**
+**Bilingual (Chinese/English) language model — custom Transformer architecture trained from scratch by [eyanchao](https://huggingface.co/eyanchao).**
 
-No HuggingFace Transformers dependency — pure PyTorch with SDPA, GQA, QK-Norm, Z-loss.
+No HuggingFace Transformers dependency. Pure PyTorch with SDPA, GQA, QK-Norm, Z-loss.
 
-## Models
+## Current Model
 
-| Version | Params | Dim | Layers | Heads | KV Heads | Status |
-|---------|--------|-----|--------|-------|----------|--------|
-| **v29** | **~1.0B** | 1408 | 32 | 32 | 8 (GQA) | 🏃 Training |
-| **v28** | ~480M | 1024 | 28 | 32 | — | ✅ Deployed |
-| v27 | ~200M | 768 | 20 | 24 | — | 📦 Archived |
-| v26 | ~60M | 512 | 14 | 16 | — | 📦 Archived |
+| Version | Params | Dim | Layers | Heads | KV Heads | Tokenizer | Status |
+|---------|--------|-----|--------|-------|----------|-----------|--------|
+| **v30** | **~1.1B** | 1408 | 32 | 32 | 8 (GQA) | BPE 32k | 🏃 Training |
+| ~~v28~~ | ~~480M~~ | ~~1024~~ | ~~28~~ | ~~32~~ | — | ~~char-level~~ | ❌ Retired |
+| ~~v29~~ | ~~1.0B~~ | ~~1408~~ | ~~32~~ | ~~32~~ | 8 (GQA) | ~~char-level~~ | ❌ Retired |
 
-## Architecture (v29)
+## Architecture (v30)
 
 | Technique | Description |
 |-----------|-------------|
-| **GQA** | Grouped Query Attention (32Q / 8KV, ~30% VRAM saved) |
-| **QK-Norm** | RMSNorm on Q/K projections — training stability |
-| **Z-loss** | Logit L2 regularization — prevents explosion |
+| **BPE Tokenizer** | SentencePiece 32k vocab — compresses Chinese 3:1 |
+| **GQA** | Grouped Query Attention (32Q / 8KV heads) |
+| **QK-Norm** | RMSNorm on Q/K projections |
+| **Z-loss** | Logit L2 regularization |
 | **SDPA** | PyTorch fused scaled_dot_product_attention |
-| **RoPE + SwiGLU + RMSNorm** | Llama-style architecture |
+| **RoPE + SwiGLU + RMSNorm** | Llama-style pre-norm architecture |
+| **seq_len 512** | Long context for CoT reasoning |
 | **Gradient checkpointing** | Per-layer activation recompute |
-| **torch.compile** | JIT with inductor disk cache |
+| **torch.compile** | JIT with TF32 precision + inductor cache |
 | **Layer-wise LR decay** | Higher layers learn slower (0.8×) |
-| **Dynamic LR** | Auto-half on loss spikes |
+| **Dynamic LR** | Auto-half on consecutive loss spikes |
 
-## Training Data Mix
+## Training
 
-| Task | Weight | Sources |
-|------|--------|---------|
-| Math | 40% | mathinstruct, school_math, math_large, math_sft |
-| Code | 20% | code_120k, code_large |
-| Encyclopedia | 25% | zh-wiki (5 partitions) |
-| Stories | 15% | TinyStories ZH/EN, story_instruct, story_punct |
-| Identity | SFT | Custom dialogs × 50 |
+- **Platform**: Modal (serverless A100 40GB)
+- **Data**: [eyanchao/echoic-data](https://huggingface.co/datasets/eyanchao/echoic-data) — 18 files, multi-task mixture
+  - Math 40% | Code 20% | Encyclopedia 25% | Stories 15% | Identity SFT
+- **Checkpointing**: Volume + HF Hub dual backup, auto-resume
 
-## Quick Start
+## Usage
 
 ```python
-import torch, os
+import torch, sentencepiece as spm
 from huggingface_hub import hf_hub_download
 
-REPO = "eyanchao/echoic-lite"
-
-# Tokenizer
-tok = torch.load(
-    hf_hub_download(REPO, "bilingual-tokenizer.pt"),
-    map_location="cpu", weights_only=False
-)
-stoi, itos = tok.stoi, tok.itos
-
-# v28 480M (dim=1024, layers=28, heads=32)
-ckpt = torch.load(
-    hf_hub_download(REPO, "v28-best-e15.pt"),
-    map_location="cpu", weights_only=False
-)
-# See echoic_modal.py for full EchoicLM class definition
-
-def encode(s): return [stoi.get(ord(c), 0) for c in s]
-def decode(ids): return "".join(itos.get(k, "?") for k in ids)
-
-prompt = "<|user|>你是谁<|assistant|>"
-x = torch.tensor([encode(prompt)], dtype=torch.long)
-out = model.generate(x, max_new_tokens=100, temperature=0.7, top_k=40)
-print(decode(out[0].tolist()))
-```
+# Load BPE tokenizer
+sp = spm.SentencePieceProcessor()
+sp.load(hf_hub_download("eyanchao/echoic-lite", "bpe_tokenizer.model"))
 
-## Inference API
+# Model: EchoicLM_1B (dim=1408, layers=32, heads=32, kv_heads=8, seq_len=512)
+# See echoic_v30_bpe.ipynb for full class definition
 
-```
-POST https://eyanchao--echoic-generate.modal.run
-Body: {"prompt": "从前有一个", "max_tokens": 200}
+def generate(prompt, model, max_tokens=200, temp=0.8, top_k=50):
+    x = torch.tensor([sp.encode(prompt)], dtype=torch.long)
+    out = model.generate(x, max_new_tokens=max_tokens, temperature=temp, top_k=top_k)
+    return sp.decode(out[0].tolist())
 ```
 
 ## Files
 
 | File | Description |
 |------|-------------|
-| `v28-best-e15.pt` | v28 best (480M, deployed) |
-| `v28-final.pt` | v28 final weights |
-| `v29-best-e*.pt` | v29 checkpoints (1B, training) |
-| `bilingual-tokenizer.pt` | Tokenizer |
-
-## Data
-
-[eyanchao/echoic-data](https://huggingface.co/datasets/eyanchao/echoic-data)
+| `bpe_tokenizer.model` | BPE SentencePiece model (32k vocab) |
+| `bpe_tokenizer.vocab` | BPE vocabulary |
+| `v30-best-e*.pt` | Training checkpoints |
+| `v30-final.pt` | Final weights (after training) |
 
 ## Evolution
 
 ```
-v26 60M → v27 200M → v28 480M → v29 1B → ...
+v26 60M → v27 200M → v28 480M → v29 1B → v30 BPE 1B → ...
+                                                ↑ current
 ```
 
-Weight expansion from v26→v28. v29 trained from scratch with GQA architecture.
+v26-v29 were character-level (98 vocab). v30 switches to BPE 32k vocab with seq_len 512 for serious reasoning capability.