File size: 3,081 Bytes
3c4ef52 5246d30 3c4ef52 ae8dad4 c129370 3c4ef52 ae8dad4 d1a4488 c129370 d1a4488 3c4ef52 d1a4488 3c4ef52 c129370 3c4ef52 c129370 3c4ef52 c129370 3c4ef52 c129370 3c4ef52 c129370 3c4ef52 c129370 ae8dad4 d1a4488 c129370 d1a4488 c129370 d1a4488 c129370 d1a4488 c129370 ae8dad4 c129370 ae8dad4 c129370 ae8dad4 c129370 ae8dad4 d1a4488 c129370 d1a4488 f6cccaa c129370 3c4ef52 c129370 3c4ef52 c129370 d1a4488 ae8dad4 d1a4488 ae8dad4 d1a4488 c129370 ae8dad4 d1a4488 ae8dad4 c129370 ae8dad4 3c4ef52 c129370 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 | ---
language:
- en
- zh
license: mit
library_name: pytorch
tags:
- text-generation
- bilingual
- BPE
- transformer
- reasoning
- custom-architecture
- GQA
- flash-attention
datasets:
- eyanchao/echoic-data
---
# Echoic
**Bilingual (Chinese/English) language model — custom Transformer architecture trained from scratch by [eyanchao](https://huggingface.co/eyanchao).**
No HuggingFace Transformers dependency. Pure PyTorch with SDPA, GQA, QK-Norm, Z-loss.
## Current Model
| Version | Params | Dim | Layers | Heads | KV Heads | Tokenizer | Status |
|---------|--------|-----|--------|-------|----------|-----------|--------|
| **v30** | **~1.1B** | 1408 | 32 | 32 | 8 (GQA) | BPE 32k | 🏃 Training |
| ~~v28~~ | ~~480M~~ | ~~1024~~ | ~~28~~ | ~~32~~ | — | ~~char-level~~ | ❌ Retired |
| ~~v29~~ | ~~1.0B~~ | ~~1408~~ | ~~32~~ | ~~32~~ | 8 (GQA) | ~~char-level~~ | ❌ Retired |
## Architecture (v30)
| Technique | Description |
|-----------|-------------|
| **BPE Tokenizer** | SentencePiece 32k vocab — compresses Chinese 3:1 |
| **GQA** | Grouped Query Attention (32Q / 8KV heads) |
| **QK-Norm** | RMSNorm on Q/K projections |
| **Z-loss** | Logit L2 regularization |
| **SDPA** | PyTorch fused scaled_dot_product_attention |
| **RoPE + SwiGLU + RMSNorm** | Llama-style pre-norm architecture |
| **seq_len 512** | Long context for CoT reasoning |
| **Gradient checkpointing** | Per-layer activation recompute |
| **torch.compile** | JIT with TF32 precision + inductor cache |
| **Layer-wise LR decay** | Higher layers learn slower (0.8×) |
| **Dynamic LR** | Auto-half on consecutive loss spikes |
## Training
- **Platform**: Modal (serverless A100 40GB)
- **Data**: [eyanchao/echoic-data](https://huggingface.co/datasets/eyanchao/echoic-data) — 18 files, multi-task mixture
- Math 40% | Code 20% | Encyclopedia 25% | Stories 15% | Identity SFT
- **Checkpointing**: Volume + HF Hub dual backup, auto-resume
## Usage
```python
import torch, sentencepiece as spm
from huggingface_hub import hf_hub_download
# Load BPE tokenizer
sp = spm.SentencePieceProcessor()
sp.load(hf_hub_download("eyanchao/echoic-lite", "bpe_tokenizer.model"))
# Model: EchoicLM_1B (dim=1408, layers=32, heads=32, kv_heads=8, seq_len=512)
# See echoic_v30_bpe.ipynb for full class definition
def generate(prompt, model, max_tokens=200, temp=0.8, top_k=50):
x = torch.tensor([sp.encode(prompt)], dtype=torch.long)
out = model.generate(x, max_new_tokens=max_tokens, temperature=temp, top_k=top_k)
return sp.decode(out[0].tolist())
```
## Files
| File | Description |
|------|-------------|
| `bpe_tokenizer.model` | BPE SentencePiece model (32k vocab) |
| `bpe_tokenizer.vocab` | BPE vocabulary |
| `v30-best-e*.pt` | Training checkpoints |
| `v30-final.pt` | Final weights (after training) |
## Evolution
```
v26 60M → v27 200M → v28 480M → v29 1B → v30 BPE 1B → ...
↑ current
```
v26-v29 were character-level (98 vocab). v30 switches to BPE 32k vocab with seq_len 512 for serious reasoning capability.
|