| --- |
| tags: |
| - alpha |
| - alpha-helios |
| - native-checkpoint |
| - text-generation |
| pipeline_tag: text-generation |
| --- |
| |
| # Alpha yi base knowledge model |
|
|
| Alpha yi is a 63.2M-parameter, decoder-only research base model trained from |
| scratch with the Alpha/Helios stack. This repository freezes the terminal |
| pretraining checkpoint at optimizer step 1,180,000, before chat fine-tuning. |
|
|
| This is an exact **Alpha native checkpoint**, not a standard Transformers |
| export. It includes model weights, AdamW state, RNG state, model configuration, |
| and the embedded 12,288-token byte-BPE tokenizer. The native format is retained |
| because this GPT-form Alpha architecture uses runtime details that an ordinary |
| GPT-2 conversion would not reproduce exactly. |
|
|
| ## Model |
|
|
| - Architecture: GPT-form causal decoder |
| - Parameters: 63,210,496 |
| - Layers / width / heads: 16 / 512 / 8 |
| - Context: 512 tokens |
| - Vocabulary: 12,288 byte-BPE tokens |
| - Position encoding: learned absolute positions |
| - Normalization / activation: LayerNorm / GELU |
| - Embeddings: untied input and output embeddings |
| - Training stack: Alpha native CUDA backend on an NVIDIA RTX 3070 |
|
|
| ## Pretraining |
|
|
| The model was trained on `concordance-v10.txt`, a frozen 60,418,346-token |
| knowledge-text corpus. The run performed 1,180,000 optimizer steps with batch 1 |
| and context 512: 604,160,000 token positions, approximately ten corpus passes |
| and 9.56 training tokens per parameter. |
|
|
| - AdamW, learning rate 6e-4 to 6e-5 |
| - 2,000 warmup steps |
| - Weight decay 0.1; gradient clipping at 1.0 |
| - Terminal training loss: 0.5848 (training batch, not validation loss) |
| - Source tree commit: `d130af1fde31c343f4618655b4103dcdb4f5178a` |
|
|
| Checkpoint SHA-256: |
|
|
| ```text |
| 6b46769eae17a48378ebc58472242d3a3bbd62d04130ba1e80b70b9da734bc05 alpha-yi-base-step-1180000.alph |
| ``` |
|
|
| ## Use with Alpha |
|
|
| With a compatible Alpha checkout and built CLI: |
|
|
| ```bash |
| hf download ajaxdavis/alpha-yi-base-knowledge \ |
| alpha-yi-base-step-1180000.alph --local-dir . |
| |
| node apps/cli/dist/main.js sample \ |
| --checkpoint=alpha-yi-base-step-1180000.alph \ |
| --prompt="The capital of France is" \ |
| --steps=40 --temp=0.7 --topk=40 |
| ``` |
|
|
| ## Evaluation status and limitations |
|
|
| The checkpoint was hash-verified and independently reloaded in a fresh process; |
| tokenizer restoration and text generation both work. A clean validation metric |
| is not reported: the native full-forward evaluation path exhausted the RTX 3070 |
| allocator after training, and Vulkan was unavailable in this pod. The terminal |
| training loss must not be interpreted as held-out perplexity. |
|
|
| Raw generations are often incoherent and sometimes form inaccurate or malformed |
| claims. This is a small, repeatedly exposed base model, not an instruction model |
| or a reliable factual system. It should not be used for high-stakes decisions. |
| Chat fine-tuning is tracked separately so this repository remains an immutable |
| base-model artifact. |
|
|
