Model Card for LUNA-300M

LUNA-300M is a scaled-up, English-first causal language model (≈303M parameters) developed by ASTERIZER, fully pretrained from scratch on the same 4.5B-token deduplicated corpus as its smaller sibling LUNA-100M.

✅ Status: TRAINED (pretraining complete, step 31,052). Instruction tuning and downstream adapters are planned on top of this checkpoint.

---

Table of Contents

• Model Details
• Program & Ecosystem
• Uses
• Training Details
• Training Data
• Checkpoints
• Limitations
• Citation
• Additional Information

---

Model Details

Property	Value
Developed by	ASTERIZER
Model type	Decoder-only causal LM (GPT/Pythia-style)
Parameters	≈303M (20 layers · 1024 hidden · 16 heads)
Context window	1,024 tokens
Vocabulary	50,304 (EleutherAI/Pythia-160m tokenizer, 50,254 + EOS padding)
Language	English
License	other (see Additional Information)
Model card date	2026-08-28

Position in the LUNA family

Model	Params	Layers	Hidden	Status
LUNA-100M	≈100M	10	768	Trained (pretrain + SFT)
LUNA-300M	≈303M	20	1024	Trained (pretrain)

---

Program & Ecosystem

Asset	Type	Description
LUNA-300M	Model	This repository — pretrained checkpoints
LUNA-100M	Model	Smaller sibling (trained, SFT-enabled)
LUNA_PreTrain	Dataset	Pretraining corpus: 4.5B deduplicated tokens, 270 litdata chunks
LUNA	Workspace	Training codebase incl. train_300m.py + train_config_300m.yaml
LUNA-300M Program	Collection	One link for the 300M program

---

Uses

Direct use

• General English language modeling at 3× the capacity of LUNA-100M — better next-token prediction and reasoning headroom for retrieval-heavy workloads.
• Base for further training: SFT, LoRA adapters, RAG/MCP instruction tuning (mirroring the 100M recipe).

Out-of-scope use

• Not instruction-tuned yet — raw base model; prompt accordingly or fine-tune first.
• Not for multilingual or code tasks; not a frontier knowledge model.

---

Training Details

Training procedure

Stage	Corpus	Tokens	Steps
Pretrain	LUNA_PreTrain (270 chunks)	4,515,286,950	31,052
SFT / adapters (planned)	RAG + MCP SFT corpus	10M	—

Hyperparameters (config/train_config_300m.yaml)

• Optimizer: AdamW (β=0.9/0.95, ε=1e-8), weight decay 0.1, grad-clip 1.0
• LR: 3e-4 (scaled down for larger model), cosine → 3e-5, 500-step warmup
• Batch: global batch 120 (auto-tuned micro-batch via VRAM probe)
• Precision: bf16 (fp16 fallback); torch.compile when available

---

Training Data

Same source stack as LUNA-100M, documented in the LUNA_PreTrain dataset card:

Source	Tokens
General web (cleaned legacy corpus)	~2.94B
Wikipedia (new)	~365M
FineWeb-Edu (score ≥ 3.5)	~445M
OpenWebText (no overlap)	~445M
Wiki+FineWeb legacy subsets	~772M
Total (post-dedup)	4,515,286,950

---

Checkpoints

Path	Description
pretrained/final/lit_model.pth	Recommended — final fp32 weights (1.2 GB)
pretrained/step-00031052/lit_model.pth	Final logged step
pretrained/step-00031000/lit_model.pth	Milestone checkpoint
pretrained/latest.pt	Last weights + optimizer state (3.6 GB)
pretrained/final/model_config.json	Architecture config (20L/1024E/16H, vocab 50304)
tokenizer/	Pythia-160m tokenizer

---

Limitations

• Data-constrained: 4.5B tokens is a modest amount for 303M parameters — knowledge breadth reflects the source mix; further data would improve coverage.
• Base model only: not instruction-tuned; raw completions require prompting or fine-tuning.
• English-only; no multilingual or code training.

---

Citation

@misc{asterizer_luna_300m,
  author       = {ASTERIZER},
  title        = {LUNA-300M: A scaled-up English causal language model},
  year         = {2026},
  howpublished = {https://huggingface.co/ASTERIZER/LUNA-300M}
}

---

Additional Information

• Program collection: LUNA-300M Program
• Training code: ASTERIZER/LUNA workspace (train_300m.py, train_config_300m.yaml)
• Contact / issues: ASTERIZER organization.
• License: weights license to be finalized; training-data upstream terms apply to dataset redistribution.