---
language:
- en
- zh
license: mit
library_name: pytorch
tags:
- text-generation
- bilingual
- BPE
- transformer
- reasoning
- custom-architecture
- GQA
- flash-attention
datasets:
- eyanchao/echoic-data
---

# Echoic

**Bilingual (Chinese/English) language model — custom Transformer architecture trained from scratch by [eyanchao](https://huggingface.co/eyanchao).**

No HuggingFace Transformers dependency. Pure PyTorch with SDPA, GQA, QK-Norm, Z-loss.

## Current Model

| Version | Params | Dim | Layers | Heads | KV Heads | Tokenizer | Status |
|---------|--------|-----|--------|-------|----------|-----------|--------|
| **v30** | **~1.1B** | 1408 | 32 | 32 | 8 (GQA) | BPE 32k | 🏃 Training |
| ~~v28~~ | ~~480M~~ | ~~1024~~ | ~~28~~ | ~~32~~ | — | ~~char-level~~ | ❌ Retired |
| ~~v29~~ | ~~1.0B~~ | ~~1408~~ | ~~32~~ | ~~32~~ | 8 (GQA) | ~~char-level~~ | ❌ Retired |

## Architecture (v30)

| Technique | Description |
|-----------|-------------|
| **BPE Tokenizer** | SentencePiece 32k vocab — compresses Chinese 3:1 |
| **GQA** | Grouped Query Attention (32Q / 8KV heads) |
| **QK-Norm** | RMSNorm on Q/K projections |
| **Z-loss** | Logit L2 regularization |
| **SDPA** | PyTorch fused scaled_dot_product_attention |
| **RoPE + SwiGLU + RMSNorm** | Llama-style pre-norm architecture |
| **seq_len 512** | Long context for CoT reasoning |
| **Gradient checkpointing** | Per-layer activation recompute |
| **torch.compile** | JIT with TF32 precision + inductor cache |
| **Layer-wise LR decay** | Higher layers learn slower (0.8×) |
| **Dynamic LR** | Auto-half on consecutive loss spikes |

## Training

- **Platform**: Modal (serverless A100 40GB)
- **Data**: [eyanchao/echoic-data](https://huggingface.co/datasets/eyanchao/echoic-data) — 18 files, multi-task mixture
  - Math 40% | Code 20% | Encyclopedia 25% | Stories 15% | Identity SFT
- **Checkpointing**: Volume + HF Hub dual backup, auto-resume

## Usage

```python
import torch, sentencepiece as spm
from huggingface_hub import hf_hub_download

# Load BPE tokenizer
sp = spm.SentencePieceProcessor()
sp.load(hf_hub_download("eyanchao/echoic-lite", "bpe_tokenizer.model"))

# Model: EchoicLM_1B (dim=1408, layers=32, heads=32, kv_heads=8, seq_len=512)
# See echoic_v30_bpe.ipynb for full class definition

def generate(prompt, model, max_tokens=200, temp=0.8, top_k=50):
    x = torch.tensor([sp.encode(prompt)], dtype=torch.long)
    out = model.generate(x, max_new_tokens=max_tokens, temperature=temp, top_k=top_k)
    return sp.decode(out[0].tolist())
```

## Files

| File | Description |
|------|-------------|
| `bpe_tokenizer.model` | BPE SentencePiece model (32k vocab) |
| `bpe_tokenizer.vocab` | BPE vocabulary |
| `v30-best-e*.pt` | Training checkpoints |
| `v30-final.pt` | Final weights (after training) |

## Evolution

```
v26 60M → v27 200M → v28 480M → v29 1B → v30 BPE 1B → ...
                                                ↑ current
```

v26-v29 were character-level (98 vocab). v30 switches to BPE 32k vocab with seq_len 512 for serious reasoning capability.