---
tags:
- alpha
- alpha-helios
- native-checkpoint
- text-generation
pipeline_tag: text-generation
---

# Alpha yi base knowledge model

Alpha yi is a 63.2M-parameter, decoder-only research base model trained from
scratch with the Alpha/Helios stack. This repository freezes the terminal
pretraining checkpoint at optimizer step 1,180,000, before chat fine-tuning.

This is an exact **Alpha native checkpoint**, not a standard Transformers
export. It includes model weights, AdamW state, RNG state, model configuration,
and the embedded 12,288-token byte-BPE tokenizer. The native format is retained
because this GPT-form Alpha architecture uses runtime details that an ordinary
GPT-2 conversion would not reproduce exactly.

## Model

- Architecture: GPT-form causal decoder
- Parameters: 63,210,496
- Layers / width / heads: 16 / 512 / 8
- Context: 512 tokens
- Vocabulary: 12,288 byte-BPE tokens
- Position encoding: learned absolute positions
- Normalization / activation: LayerNorm / GELU
- Embeddings: untied input and output embeddings
- Training stack: Alpha native CUDA backend on an NVIDIA RTX 3070

## Pretraining

The model was trained on `concordance-v10.txt`, a frozen 60,418,346-token
knowledge-text corpus. The run performed 1,180,000 optimizer steps with batch 1
and context 512: 604,160,000 token positions, approximately ten corpus passes
and 9.56 training tokens per parameter.

- AdamW, learning rate 6e-4 to 6e-5
- 2,000 warmup steps
- Weight decay 0.1; gradient clipping at 1.0
- Terminal training loss: 0.5848 (training batch, not validation loss)
- Source tree commit: `d130af1fde31c343f4618655b4103dcdb4f5178a`

Checkpoint SHA-256:

```text
6b46769eae17a48378ebc58472242d3a3bbd62d04130ba1e80b70b9da734bc05  alpha-yi-base-step-1180000.alph
```

## Use with Alpha

With a compatible Alpha checkout and built CLI:

```bash
hf download ajaxdavis/alpha-yi-base-knowledge \
  alpha-yi-base-step-1180000.alph --local-dir .

node apps/cli/dist/main.js sample \
  --checkpoint=alpha-yi-base-step-1180000.alph \
  --prompt="The capital of France is" \
  --steps=40 --temp=0.7 --topk=40
```

## Evaluation status and limitations

The checkpoint was hash-verified and independently reloaded in a fresh process;
tokenizer restoration and text generation both work. A clean validation metric
is not reported: the native full-forward evaluation path exhausted the RTX 3070
allocator after training, and Vulkan was unavailable in this pod. The terminal
training loss must not be interpreted as held-out perplexity.

Raw generations are often incoherent and sometimes form inaccurate or malformed
claims. This is a small, repeatedly exposed base model, not an instruction model
or a reliable factual system. It should not be used for high-stakes decisions.
Chat fine-tuning is tracked separately so this repository remains an immutable
base-model artifact.