Hugging Face's logo

ASTERIZER
/
LUNA-300M

Text Generation
English
causal-lm
pretraining
small-language-model
Model card Files
xet
 Community
LUNA-300M / README.md
REZIRETSA's picture
REZIRETSA
Apply CC BY-NC-ND 4.0 license
7fdc968 verified
preview code
|
Raw
History Blame Contribute Delete
5.63 kB
metadata 
license: cc-by-nc-nd-4.0
language:
  - en
pipeline_tag: text-generation
tags:
  - text-generation
  - causal-lm
  - pretraining
  - small-language-model

Model Card for LUNA-300M

LUNA-300M is a scaled-up, English-first causal language model (≈303M parameters) developed by ASTERIZER, fully pretrained from scratch on the same 4.5B-token deduplicated corpus as its smaller sibling LUNA-100M.

✅ Status: TRAINED (pretraining complete, step 31,052). Instruction tuning and downstream adapters are planned on top of this checkpoint.

Table of Contents

Model Details

Property Value
Developed by ASTERIZER
Model type Decoder-only causal LM (GPT/Pythia-style)
Parameters ≈303M (20 layers · 1024 hidden · 16 heads)
Context window 1,024 tokens
Vocabulary 50,304 (EleutherAI/Pythia-160m tokenizer, 50,254 + EOS padding)
Language English
License other (see Additional Information)
Model card date 2026-08-28

Position in the LUNA family

Model Params Layers Hidden Status
LUNA-100M ≈100M 10 768 Trained (pretrain + SFT)
LUNA-300M ≈303M 20 1024 Trained (pretrain)

Program & Ecosystem

Asset Type Description
LUNA-300M Model This repository — pretrained checkpoints
LUNA-100M Model Smaller sibling (trained, SFT-enabled)
LUNA_PreTrain Dataset Pretraining corpus: 4.5B deduplicated tokens, 270 litdata chunks
LUNA Workspace Training codebase incl. train_300m.py + train_config_300m.yaml
LUNA-300M Program Collection One link for the 300M program

Uses

Direct use

  • General English language modeling at 3× the capacity of LUNA-100M — better next-token prediction and reasoning headroom for retrieval-heavy workloads.
  • Base for further training: SFT, LoRA adapters, RAG/MCP instruction tuning (mirroring the 100M recipe).

Out-of-scope use

  • Not instruction-tuned yet — raw base model; prompt accordingly or fine-tune first.
  • Not for multilingual or code tasks; not a frontier knowledge model.

Training Details

Training procedure

Stage Corpus Tokens Steps
Pretrain LUNA_PreTrain (270 chunks) 4,515,286,950 31,052
SFT / adapters (planned) RAG + MCP SFT corpus 10M —

Hyperparameters (config/train_config_300m.yaml)

  • Optimizer: AdamW (β=0.9/0.95, ε=1e-8), weight decay 0.1, grad-clip 1.0
  • LR: 3e-4 (scaled down for larger model), cosine → 3e-5, 500-step warmup
  • Batch: global batch 120 (auto-tuned micro-batch via VRAM probe)
  • Precision: bf16 (fp16 fallback); torch.compile when available

Training Data

Same source stack as LUNA-100M, documented in the LUNA_PreTrain dataset card:

Source Tokens
General web (cleaned legacy corpus) ~2.94B
Wikipedia (new) ~365M
FineWeb-Edu (score ≥ 3.5) ~445M
OpenWebText (no overlap) ~445M
Wiki+FineWeb legacy subsets ~772M
Total (post-dedup) 4,515,286,950

Checkpoints

Path Description
pretrained/final/lit_model.pth Recommended — final fp32 weights (1.2 GB)
pretrained/step-00031052/lit_model.pth Final logged step
pretrained/step-00031000/lit_model.pth Milestone checkpoint
pretrained/latest.pt Last weights + optimizer state (3.6 GB)
pretrained/final/model_config.json Architecture config (20L/1024E/16H, vocab 50304)
tokenizer/ Pythia-160m tokenizer

Limitations

  • Data-constrained: 4.5B tokens is a modest amount for 303M parameters — knowledge breadth reflects the source mix; further data would improve coverage.
  • Base model only: not instruction-tuned; raw completions require prompting or fine-tuning.
  • English-only; no multilingual or code training.

Citation 

@misc{asterizer_luna_300m,
  author       = {ASTERIZER},
  title        = {LUNA-300M: A scaled-up English causal language model},
  year         = {2026},
  howpublished = {https://huggingface.co/ASTERIZER/LUNA-300M}
}

Additional Information

  • Program collection: LUNA-300M Program
  • Training code: ASTERIZER/LUNA workspace (train_300m.py, train_config_300m.yaml)
  • Contact / issues: ASTERIZER organization.
  • License: weights license to be finalized; training-data upstream terms apply to dataset redistribution.