File size: 2,868 Bytes
429fb1c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 | ---
tags:
- alpha
- alpha-helios
- native-checkpoint
- text-generation
pipeline_tag: text-generation
---
# Alpha yi base knowledge model
Alpha yi is a 63.2M-parameter, decoder-only research base model trained from
scratch with the Alpha/Helios stack. This repository freezes the terminal
pretraining checkpoint at optimizer step 1,180,000, before chat fine-tuning.
This is an exact **Alpha native checkpoint**, not a standard Transformers
export. It includes model weights, AdamW state, RNG state, model configuration,
and the embedded 12,288-token byte-BPE tokenizer. The native format is retained
because this GPT-form Alpha architecture uses runtime details that an ordinary
GPT-2 conversion would not reproduce exactly.
## Model
- Architecture: GPT-form causal decoder
- Parameters: 63,210,496
- Layers / width / heads: 16 / 512 / 8
- Context: 512 tokens
- Vocabulary: 12,288 byte-BPE tokens
- Position encoding: learned absolute positions
- Normalization / activation: LayerNorm / GELU
- Embeddings: untied input and output embeddings
- Training stack: Alpha native CUDA backend on an NVIDIA RTX 3070
## Pretraining
The model was trained on `concordance-v10.txt`, a frozen 60,418,346-token
knowledge-text corpus. The run performed 1,180,000 optimizer steps with batch 1
and context 512: 604,160,000 token positions, approximately ten corpus passes
and 9.56 training tokens per parameter.
- AdamW, learning rate 6e-4 to 6e-5
- 2,000 warmup steps
- Weight decay 0.1; gradient clipping at 1.0
- Terminal training loss: 0.5848 (training batch, not validation loss)
- Source tree commit: `d130af1fde31c343f4618655b4103dcdb4f5178a`
Checkpoint SHA-256:
```text
6b46769eae17a48378ebc58472242d3a3bbd62d04130ba1e80b70b9da734bc05 alpha-yi-base-step-1180000.alph
```
## Use with Alpha
With a compatible Alpha checkout and built CLI:
```bash
hf download ajaxdavis/alpha-yi-base-knowledge \
alpha-yi-base-step-1180000.alph --local-dir .
node apps/cli/dist/main.js sample \
--checkpoint=alpha-yi-base-step-1180000.alph \
--prompt="The capital of France is" \
--steps=40 --temp=0.7 --topk=40
```
## Evaluation status and limitations
The checkpoint was hash-verified and independently reloaded in a fresh process;
tokenizer restoration and text generation both work. A clean validation metric
is not reported: the native full-forward evaluation path exhausted the RTX 3070
allocator after training, and Vulkan was unavailable in this pod. The terminal
training loss must not be interpreted as held-out perplexity.
Raw generations are often incoherent and sometimes form inaccurate or malformed
claims. This is a small, repeatedly exposed base model, not an instruction model
or a reliable factual system. It should not be used for high-stakes decisions.
Chat fine-tuning is tracked separately so this repository remains an immutable
base-model artifact.
|