| --- |
| language: en |
| license: apache-2.0 |
| library_name: pytorch |
| pipeline_tag: text-generation |
| tags: |
| - experimental |
| - from-scratch |
| --- |
| # Slayer149 Balanced |
|
|
| Experimental 149,333,081-parameter English causal model trained **from scratch**. |
| 26 layers, hidden width 576, FFN 2496, 9 query heads / 3 KV heads, tied 24,576-token |
| byte-level BPE vocabulary, QK normalization and value residuals. Architecture |
| informed by Qwen3 and AltSlate's Apache-2.0 Jugnu recipe; no Jugnu weights reused. |
| This is a separate experiment from SlayerLab/Slayer149, which remains available. |
|
|
| Training may still be in progress. See `reports/status.json`, training metrics and |
| checkpoint manifests. No leaderboard rank or winning result is claimed. |
| The final full GLINT-derived evaluation is `reports/evaluation.json`, when present. |
| Competitor scores may use different evaluation conventions. |
|
|
| `tokenizer.json` preserves Unicode/whitespace and uses EOS ID 0. Tokenizer selection |
| and dataset provenance are recorded in reports. Training sources are pinned |
| FineWeb-Edu, DCLM-edu, Cosmopedia-v2 and FineMath, with document-disjoint validation |
| and exact-overlap filtering against the benchmark suite. Filtering is not a |
| semantic contamination guarantee. No training text or credentials are uploaded. |
|
|
| `checkpoints/<step>/training-state.pt` contains model and both optimizer states. |
| Restore with the matching tokenizer, config, source files and training code. |
| Final safetensors appear under the final checkpoint and at repository root. |
| Use the supplied custom PyTorch loader; this is not yet an AutoModel package. |
|
|
| ```python |
| from load_model import load_model |
| model, tokenizer = load_model("/path/to/downloaded/repository", device="cuda") |
| ``` |
|
|
| Goal: have a durable checkpoint and evaluation report by 15:00 Warsaw time on |
| 2026-10-01, within the user's confirmed free GPU reservation. This is not a |
| promise of model quality or benchmark rank. |
|
|
