---
license: cc-by-nc-nd-4.0
language:
- en
pipeline_tag: text-generation
tags:
- text-generation
- causal-lm
- pretraining
- small-language-model
---

# Model Card for LUNA-300M

**LUNA-300M** is a scaled-up, English-first causal language model (≈303M parameters) developed by **ASTERIZER**, fully pretrained from scratch on the same 4.5B-token deduplicated corpus as its smaller sibling [LUNA-100M](https://huggingface.co/ASTERIZER/LUNA-100M).

> ✅ **Status: TRAINED (pretraining complete, step 31,052).** Instruction tuning and downstream adapters are planned on top of this checkpoint.

---

## Table of Contents

- [Model Details](#model-details)
- [Program & Ecosystem](#program--ecosystem)
- [Uses](#uses)
- [Training Details](#training-details)
- [Training Data](#training-data)
- [Checkpoints](#checkpoints)
- [Limitations](#limitations)
- [Citation](#citation)
- [Additional Information](#additional-information)

---

## Model Details

| Property | Value |
|---|---|
| **Developed by** | [ASTERIZER](https://huggingface.co/ASTERIZER) |
| **Model type** | Decoder-only causal LM (GPT/Pythia-style) |
| **Parameters** | ≈303M (20 layers · 1024 hidden · 16 heads) |
| **Context window** | 1,024 tokens |
| **Vocabulary** | 50,304 (EleutherAI/Pythia-160m tokenizer, 50,254 + EOS padding) |
| **Language** | English |
| **License** | other (see [Additional Information](#additional-information)) |
| **Model card date** | 2026-08-28 |

### Position in the LUNA family

| Model | Params | Layers | Hidden | Status |
|---|---|---|---:|---|
| [LUNA-100M](https://huggingface.co/ASTERIZER/LUNA-100M) | ≈100M | 10 | 768 | **Trained** (pretrain + SFT) |
| **LUNA-300M** | ≈303M | 20 | 1024 | **Trained** (pretrain) |

---

## Program & Ecosystem

| Asset | Type | Description |
|---|---|---|
| **[LUNA-300M](https://huggingface.co/ASTERIZER/LUNA-300M)** | Model | This repository — pretrained checkpoints |
| **[LUNA-100M](https://huggingface.co/ASTERIZER/LUNA-100M)** | Model | Smaller sibling (trained, SFT-enabled) |
| **[LUNA_PreTrain](https://huggingface.co/datasets/ASTERIZER/LUNA_PreTrain)** | Dataset | Pretraining corpus: 4.5B deduplicated tokens, 270 litdata chunks |
| **[LUNA](https://huggingface.co/spaces/ASTERIZER/LUNA)** | Workspace | Training codebase incl. `train_300m.py` + `train_config_300m.yaml` |
| **[LUNA-300M Program](https://huggingface.co/collections/ASTERIZER/luna-300m-program-6a91195ab75671026f83ac94)** | Collection | One link for the 300M program |

---

## Uses

### Direct use

- **General English language modeling** at 3× the capacity of LUNA-100M — better next-token prediction and reasoning headroom for retrieval-heavy workloads.
- **Base for further training:** SFT, LoRA adapters, RAG/MCP instruction tuning (mirroring the 100M recipe).

### Out-of-scope use

- Not instruction-tuned yet — raw base model; prompt accordingly or fine-tune first.
- Not for multilingual or code tasks; not a frontier knowledge model.

---

## Training Details

### Training procedure

| Stage | Corpus | Tokens | Steps |
|---|---|---|---|
| **Pretrain** | [`LUNA_PreTrain`](https://huggingface.co/datasets/ASTERIZER/LUNA_PreTrain) (270 chunks) | 4,515,286,950 | **31,052** |
| SFT / adapters (planned) | RAG + MCP SFT corpus | 10M | — |

### Hyperparameters (`config/train_config_300m.yaml`)

- **Optimizer:** AdamW (β=0.9/0.95, ε=1e-8), weight decay 0.1, grad-clip 1.0
- **LR:** 3e-4 (scaled down for larger model), cosine → 3e-5, 500-step warmup
- **Batch:** global batch 120 (auto-tuned micro-batch via VRAM probe)
- **Precision:** bf16 (fp16 fallback); `torch.compile` when available

---

## Training Data

Same source stack as LUNA-100M, documented in the [LUNA_PreTrain dataset card](https://huggingface.co/datasets/ASTERIZER/LUNA_PreTrain):

| Source | Tokens |
|---|---:|
| General web (cleaned legacy corpus) | ~2.94B |
| Wikipedia (new) | ~365M |
| FineWeb-Edu (score ≥ 3.5) | ~445M |
| OpenWebText (no overlap) | ~445M |
| Wiki+FineWeb legacy subsets | ~772M |
| **Total (post-dedup)** | **4,515,286,950** |

---

## Checkpoints

| Path | Description |
|---|---|
| `pretrained/final/lit_model.pth` | **Recommended** — final fp32 weights (1.2 GB) |
| `pretrained/step-00031052/lit_model.pth` | Final logged step |
| `pretrained/step-00031000/lit_model.pth` | Milestone checkpoint |
| `pretrained/latest.pt` | Last weights + optimizer state (3.6 GB) |
| `pretrained/final/model_config.json` | Architecture config (20L/1024E/16H, vocab 50304) |
| `tokenizer/` | Pythia-160m tokenizer |

---

## Limitations

- **Data-constrained:** 4.5B tokens is a modest amount for 303M parameters — knowledge breadth reflects the source mix; further data would improve coverage.
- **Base model only:** not instruction-tuned; raw completions require prompting or fine-tuning.
- **English-only**; no multilingual or code training.

---

## Citation

```bibtex
@misc{asterizer_luna_300m,
  author       = {ASTERIZER},
  title        = {LUNA-300M: A scaled-up English causal language model},
  year         = {2026},
  howpublished = {https://huggingface.co/ASTERIZER/LUNA-300M}
}
```

---

## Additional Information

- **Program collection:** [LUNA-300M Program](https://huggingface.co/collections/ASTERIZER/luna-300m-program-6a91195ab75671026f83ac94)
- **Training code:** [ASTERIZER/LUNA](https://huggingface.co/spaces/ASTERIZER/LUNA) workspace (`train_300m.py`, `train_config_300m.yaml`)
- **Contact / issues:** [ASTERIZER](https://huggingface.co/ASTERIZER) organization.
- **License:** weights license to be finalized; training-data upstream terms apply to dataset redistribution.