--- language: - en license: apache-2.0 library_name: transformers pipeline_tag: text-generation tags: - llama - alpha - vulkan - from-scratch - experimental datasets: - HuggingFaceFW/finepdfs_edu_50BT-dclm_30BT-fineweb_edu_20BT-shuffled --- # Alpha 60M Base Alpha 60M Base is a 57,688,576-parameter causal language model trained from scratch with [Alpha](https://github.com/thomasdavis/alpha2), a TypeScript tensor/autograd stack and hand-written Vulkan compute backend. Every training step ran through Alpha and Helios on an RTX 3090; PyTorch and Transformers were used only after training to verify and package the result. This is the base checkpoint, not an assistant. It can continue English text, but its frozen chat evaluation is intentionally poor. Use `ajaxdavis/alpha-60m-chat` for the instruction-tuned sibling after that repository is published. ## Architecture | Property | Value | |---|---:| | Parameters | 57,688,576 | | Layers | 16 | | Hidden width | 512 | | Attention heads / KV heads | 8 / 8 | | Head width | 64 | | SwiGLU intermediate width | 1,408 | | Context length | 1,024 | | Vocabulary | 12,288-token Alpha byte-level BPE | | Positions / normalization | RoPE (`theta=10000`) / RMSNorm (`eps=1e-5`) | | Embeddings | Input/output weights tied | | Export dtype | float32 | The export is a standard `LlamaForCausalLM` repository: no custom code and no `trust_remote_code=True`. ## Use ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "ajaxdavis/alpha-60m-base" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.float32) inputs = tokenizer("Hello", return_tensors="pt") output = model.generate(**inputs, max_new_tokens=64, do_sample=False) print(tokenizer.decode(output[0], skip_special_tokens=True)) ``` The tokenizer includes `<|user|>`, `<|assistant|>`, and `<|end_of_text|>` plus a chat template so the base and chat exports have the same interface. Their presence does not make this base checkpoint an instruction-following model. ## Training - Exact Alpha source: `e561f66c7a88a5294e9cb74a4fc3afd6be167d4f`. - 61,036 steps, batch 16, block 1,024: exactly 1,000,013,824 training tokens. - AdamW: `lr=1e-3` cosine-decayed to `1e-4`, 610 warmup steps, betas `(0.9, 0.95)`, `eps=1e-8`, weight decay `0.1`, gradient clip `1.0`. - Full float32 training on one NVIDIA RTX 3090 through Helios Vulkan, with BDA and device-generated commands enabled, cooperative-matrix kernels disabled, and CPU fallback forbidden. - Median post-warmup throughput: 3,882.35 tokens/second. The sealed iteration timings total 71.7355 GPU-hours, or $15.78 at the pod's $0.22/hour rate. That figure excludes evaluation, checkpoint, interrupted-pod, storage, and operational overhead; energy was not independently metered. - Terminal train loss: 2.9975; last-100 mean: 3.1011; final-three held-out validation mean: 3.1368. - All 61,036 metric rows and all 57,688,576 terminal parameter values were finite; allocator telemetry reported zero free-range overflow. The native terminal checkpoint SHA-256 is `08e14fa9604bf1b46ebcd5df37933c84d2496c1d05d9e4b32ebad98792cc6049`. ## Data Pretraining used the first three hash-pinned shards of [HuggingFaceFW/finepdfs_edu_50BT-dclm_30BT-fineweb_edu_20BT-shuffled](https://huggingface.co/datasets/HuggingFaceFW/finepdfs_edu_50BT-dclm_30BT-fineweb_edu_20BT-shuffled), a globally shuffled 50% FinePDFs-Edu / 30% DCLM / 20% FineWeb-Edu mixture. The upstream dataset is released under ODC-By 1.0 and remains subject to its source and Common Crawl terms. Alpha trained on one billion tokens from a 1.389-billion-token training split; the remainder was not repeated to reach the target. The model weights are released under Apache-2.0. That does not replace or relax the upstream data licenses and terms. ## Verification and frozen evaluation The exact exported `model.safetensors` has SHA-256 `d0aa2ccd171a7748a209921bbc8fbaf14a7fcb23a0e9cea050a8c9d47fbccbd9`. Stock Transformers loaded it as `LlamaForCausalLM` with 57,688,576 parameters. Against Alpha's own float32 CPU reference forward, the export achieved 2/2 top-1 agreement, exact tokenizer parity, and maximum absolute logit difference `6.771e-05` (threshold `1e-3`). Both plain-text and message-list `pipeline("text-generation")` cold loads completed without custom code. Greedy frozen evaluation used 100 chat prompts at up to 128 generated tokens and 200 closed-book QA items at up to 64 generated tokens: | Metric | Base result | |---|---:| | Chat structural pass | 0 / 100 | | EOS termination | 0 / 100 | | User-role leaks | 0 / 100 | | Degenerate repetition loops | 99 / 100 | | Mean / maximum 4-gram repeat rate | 0.81256 / 0.98400 | | Closed-book exact match | 0 / 200 | | Answer contained | 1 / 200 | | Mean token F1 | 0.000238 | These failures are reported because this checkpoint is a small base model, not because the suite was optional. The frozen chat and QA inputs were finalized before flagship training and were excluded from the training corpora by an exact 13-gram overlap audit. ## Limitations - This is a research artifact at only 58M parameters and one billion pretraining tokens. It has weak factual recall, reasoning, long-form stability, and multilingual ability. - It is not safety-aligned and must not be relied on for medical, legal, financial, security, or other consequential advice. - The upstream web mixture can contain errors, bias, personal information, unsafe content, and copyrighted text. The model may reproduce or transform such material. - Contexts above 1,024 tokens are unsupported even if a tokenizer API reports a larger generic limit. - Greedy output can loop badly, as the frozen result demonstrates. Apply downstream safeguards if you experiment with the base checkpoint. ## Reproducibility The training contract, manifests, failure-closed analyzers, RunPod bootstrap, NVIDIA parity tests, and HF exporter are in the [Alpha repository](https://github.com/thomasdavis/alpha2). The base training source is `e561f66`; the terminal-evaluation/export tooling source is `c333bf247fbe87b85d01f3d34789b46615dd1034`.