| --- |
| license: cc-by-nc-nd-4.0 |
| language: |
| - en |
| pipeline_tag: text-generation |
| tags: |
| - text-generation |
| - causal-lm |
| - pretraining |
| - small-language-model |
| --- |
| |
| # Model Card for LUNA-300M |
|
|
| **LUNA-300M** is a scaled-up, English-first causal language model (≈303M parameters) developed by **ASTERIZER**, fully pretrained from scratch on the same 4.5B-token deduplicated corpus as its smaller sibling [LUNA-100M](https://huggingface.co/ASTERIZER/LUNA-100M). |
|
|
| > ✅ **Status: TRAINED (pretraining complete, step 31,052).** Instruction tuning and downstream adapters are planned on top of this checkpoint. |
|
|
| --- |
|
|
| ## Table of Contents |
|
|
| - [Model Details](#model-details) |
| - [Program & Ecosystem](#program--ecosystem) |
| - [Uses](#uses) |
| - [Training Details](#training-details) |
| - [Training Data](#training-data) |
| - [Checkpoints](#checkpoints) |
| - [Limitations](#limitations) |
| - [Citation](#citation) |
| - [Additional Information](#additional-information) |
|
|
| --- |
|
|
| ## Model Details |
|
|
| | Property | Value | |
| |---|---| |
| | **Developed by** | [ASTERIZER](https://huggingface.co/ASTERIZER) | |
| | **Model type** | Decoder-only causal LM (GPT/Pythia-style) | |
| | **Parameters** | ≈303M (20 layers · 1024 hidden · 16 heads) | |
| | **Context window** | 1,024 tokens | |
| | **Vocabulary** | 50,304 (EleutherAI/Pythia-160m tokenizer, 50,254 + EOS padding) | |
| | **Language** | English | |
| | **License** | other (see [Additional Information](#additional-information)) | |
| | **Model card date** | 2026-08-28 | |
|
|
| ### Position in the LUNA family |
|
|
| | Model | Params | Layers | Hidden | Status | |
| |---|---|---|---:|---| |
| | [LUNA-100M](https://huggingface.co/ASTERIZER/LUNA-100M) | ≈100M | 10 | 768 | **Trained** (pretrain + SFT) | |
| | **LUNA-300M** | ≈303M | 20 | 1024 | **Trained** (pretrain) | |
|
|
| --- |
|
|
| ## Program & Ecosystem |
|
|
| | Asset | Type | Description | |
| |---|---|---| |
| | **[LUNA-300M](https://huggingface.co/ASTERIZER/LUNA-300M)** | Model | This repository — pretrained checkpoints | |
| | **[LUNA-100M](https://huggingface.co/ASTERIZER/LUNA-100M)** | Model | Smaller sibling (trained, SFT-enabled) | |
| | **[LUNA_PreTrain](https://huggingface.co/datasets/ASTERIZER/LUNA_PreTrain)** | Dataset | Pretraining corpus: 4.5B deduplicated tokens, 270 litdata chunks | |
| | **[LUNA](https://huggingface.co/spaces/ASTERIZER/LUNA)** | Workspace | Training codebase incl. `train_300m.py` + `train_config_300m.yaml` | |
| | **[LUNA-300M Program](https://huggingface.co/collections/ASTERIZER/luna-300m-program-6a91195ab75671026f83ac94)** | Collection | One link for the 300M program | |
|
|
| --- |
|
|
| ## Uses |
|
|
| ### Direct use |
|
|
| - **General English language modeling** at 3× the capacity of LUNA-100M — better next-token prediction and reasoning headroom for retrieval-heavy workloads. |
| - **Base for further training:** SFT, LoRA adapters, RAG/MCP instruction tuning (mirroring the 100M recipe). |
|
|
| ### Out-of-scope use |
|
|
| - Not instruction-tuned yet — raw base model; prompt accordingly or fine-tune first. |
| - Not for multilingual or code tasks; not a frontier knowledge model. |
|
|
| --- |
|
|
| ## Training Details |
|
|
| ### Training procedure |
|
|
| | Stage | Corpus | Tokens | Steps | |
| |---|---|---|---| |
| | **Pretrain** | [`LUNA_PreTrain`](https://huggingface.co/datasets/ASTERIZER/LUNA_PreTrain) (270 chunks) | 4,515,286,950 | **31,052** | |
| | SFT / adapters (planned) | RAG + MCP SFT corpus | 10M | — | |
|
|
| ### Hyperparameters (`config/train_config_300m.yaml`) |
|
|
| - **Optimizer:** AdamW (β=0.9/0.95, ε=1e-8), weight decay 0.1, grad-clip 1.0 |
| - **LR:** 3e-4 (scaled down for larger model), cosine → 3e-5, 500-step warmup |
| - **Batch:** global batch 120 (auto-tuned micro-batch via VRAM probe) |
| - **Precision:** bf16 (fp16 fallback); `torch.compile` when available |
|
|
| --- |
|
|
| ## Training Data |
|
|
| Same source stack as LUNA-100M, documented in the [LUNA_PreTrain dataset card](https://huggingface.co/datasets/ASTERIZER/LUNA_PreTrain): |
|
|
| | Source | Tokens | |
| |---|---:| |
| | General web (cleaned legacy corpus) | ~2.94B | |
| | Wikipedia (new) | ~365M | |
| | FineWeb-Edu (score ≥ 3.5) | ~445M | |
| | OpenWebText (no overlap) | ~445M | |
| | Wiki+FineWeb legacy subsets | ~772M | |
| | **Total (post-dedup)** | **4,515,286,950** | |
|
|
| --- |
|
|
| ## Checkpoints |
|
|
| | Path | Description | |
| |---|---| |
| | `pretrained/final/lit_model.pth` | **Recommended** — final fp32 weights (1.2 GB) | |
| | `pretrained/step-00031052/lit_model.pth` | Final logged step | |
| | `pretrained/step-00031000/lit_model.pth` | Milestone checkpoint | |
| | `pretrained/latest.pt` | Last weights + optimizer state (3.6 GB) | |
| | `pretrained/final/model_config.json` | Architecture config (20L/1024E/16H, vocab 50304) | |
| | `tokenizer/` | Pythia-160m tokenizer | |
|
|
| --- |
|
|
| ## Limitations |
|
|
| - **Data-constrained:** 4.5B tokens is a modest amount for 303M parameters — knowledge breadth reflects the source mix; further data would improve coverage. |
| - **Base model only:** not instruction-tuned; raw completions require prompting or fine-tuning. |
| - **English-only**; no multilingual or code training. |
|
|
| --- |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{asterizer_luna_300m, |
| author = {ASTERIZER}, |
| title = {LUNA-300M: A scaled-up English causal language model}, |
| year = {2026}, |
| howpublished = {https://huggingface.co/ASTERIZER/LUNA-300M} |
| } |
| ``` |
|
|
| --- |
|
|
| ## Additional Information |
|
|
| - **Program collection:** [LUNA-300M Program](https://huggingface.co/collections/ASTERIZER/luna-300m-program-6a91195ab75671026f83ac94) |
| - **Training code:** [ASTERIZER/LUNA](https://huggingface.co/spaces/ASTERIZER/LUNA) workspace (`train_300m.py`, `train_config_300m.yaml`) |
| - **Contact / issues:** [ASTERIZER](https://huggingface.co/ASTERIZER) organization. |
| - **License:** weights license to be finalized; training-data upstream terms apply to dataset redistribution. |
|
|
