---
license: cc-by-nc-nd-4.0
language:
- en
pipeline_tag: text-generation
tags:
- text-generation
- rag
- mcp
- causal-lm
- pretraining
- instruction-finetuning
- small-language-model
---

# Model Card for LUNA-100M

**LUNA-100M** is a compact, English-first causal language model (~100M parameters) developed by **ASTERIZER** and **fully trained from scratch** — pretrained on 4.5B deduplicated tokens, extended on the English-1B corpus, then instruction-tuned on RAG + MCP knowledge. It is built for the agent era: small enough to run on-device / single-GPU inference, with grounded knowledge of **Retrieval-Augmented Generation (RAG)** and the **Model Context Protocol (MCP)**.

> ✅ **Status: TRAINED.** All checkpoints (pretrained, English-1B continuation, and SFT) ship in this repository.

---

## Table of Contents

- [Model Details](#model-details)
- [Program & Ecosystem](#program--ecosystem)
- [Uses](#uses)
- [Training Details](#training-details)
- [Training Data](#training-data)
- [Checkpoints](#checkpoints)
- [Evaluation](#evaluation)
- [Limitations](#limitations)
- [Citation](#citation)
- [Additional Information](#additional-information)

---

## Model Details

| Property | Value |
|---|---|
| **Developed by** | [ASTERIZER](https://huggingface.co/ASTERIZER) |
| **Model type** | Decoder-only causal LM (GPT/Pythia-style) |
| **Parameters** | ≈100M (10 layers · 768 hidden · 12 heads) |
| **Context window** | 1,024 tokens |
| **Vocabulary** | 50,304 (EleutherAI/Pythia-160m tokenizer, 50,254 + EOS padding) |
| **Language** | English |
| **License** | other (see [Additional Information](#additional-information)) |
| **Model card date** | 2026-08-28 |

### Why 100M?

RAG and MCP architectures move fact-retrieval and tool-orchestration **outside** the weights. A 100M-parameter core with strong protocol knowledge delivers useful agent behavior at a fraction of the scale of frontier models — on-device, in edge deployments, at high request throughput.

---

## Program & Ecosystem

| Asset | Type | Description |
|---|---|---|
| **[LUNA-100M](https://huggingface.co/ASTERIZER/LUNA-100M)** | Model | This repository — trained checkpoints + eval logs |
| **[LUNA-300M](https://huggingface.co/ASTERIZER/LUNA-300M)** | Model | Scaled-up sibling (≈303M, pretrained) |
| **[LUNA_PreTrain](https://huggingface.co/datasets/ASTERIZER/LUNA_PreTrain)** | Dataset | Pretraining corpus: 4.5B deduplicated tokens, 270 litdata chunks |
| **[LUNA-RAG-MCP-SFT-10M](https://huggingface.co/datasets/ASTERIZER/LUNA-RAG-MCP-SFT-10M)** | Dataset | Instruction-tuning corpus: 10M tokens of RAG + MCP, 61.9K samples |
| **[LUNA](https://huggingface.co/spaces/ASTERIZER/LUNA)** | Workspace | Full training codebase (pretrain, SFT, eval, quantize) |
| **[LUNA-Training](https://huggingface.co/ASTERIZER/LUNA-Training)** | Code | SFT/LoRA training recipes incl. RAG+MCP configs |
| **[LUNA-100M Program](https://huggingface.co/collections/ASTERIZER/luna-100m-program-6a9115383c18e52460bf67c9)** | Collection | One link for the whole program |

---

## Uses

### Direct use

- **RAG-powered assistants** — retrieval-backed Q&A, citation-aware generation, document grounding.
- **MCP-enabled agents** — tool discovery & execution, capability negotiation, protocol-aware integration.
- **Retrieval-as-an-MCP-server architectures** — the RAG ⇄ MCP bridge of modern agent stacks.
- **On-device / single-GPU inference** — 100M params runs comfortably on CPU and edge hardware.

### Out-of-scope use

- Not for code generation, multilingual tasks, or advanced math (see [Limitations](#limitations)).
- Not a frontier knowledge model — do not benchmark against models 10–100× its size.

---

## Training Details

### Training procedure

| Stage | Corpus | Tokens | Result |
|---|---|---|---|
| **1. Pretrain** | [`LUNA_PreTrain`](https://huggingface.co/datasets/ASTERIZER/LUNA_PreTrain) (270 chunks) | 4,515,286,950 | `pretrained/` |
| **2. English continuation** | `litdata_english_1b` (English-1B corpus) | 420,000,000 | `english_1b_continued/` (25,531 steps) |
| **3. SFT (identity + RAG/MCP)** | [`LUNA-RAG-MCP-SFT-10M`](https://huggingface.co/datasets/ASTERIZER/LUNA-RAG-MCP-SFT-10M) | 10,000,168 | `sft_v1/` (16,666 steps) & `luna_100m_sft/` (8,600 steps) |

### Hyperparameters

- **Optimizer:** AdamW (β=0.9/0.95, ε=1e-8), weight decay 0.1, grad-clip 1.0
- **LR:** 6e-4, cosine schedule → 6e-5 (5% warmup)
- **Batch:** global batch 120 (auto-tuned micro-batch via VRAM probe)
- **Precision:** bf16 (fp16 fallback); `torch.compile` when available
- **Full configs:** `config/train_config.yaml`, `config/train_continue_english_1b.yaml` in this repo

---

## Training Data

| Source | Tokens | Notes |
|---|---:|---|
| Pretraining (LUNA_PreTrain) | 4.5B | Wikipedia · FineWeb-Edu · OpenWebText · cleaned web, deduplicated |
| English continuation | 420M | English-1B corpus (simplewiki + wiki-tail + fineweb-tail) |
| SFT (RAG + MCP) | 10M | 61,884 Alpaca samples, ≤1,024 tok, source-grounded |

---

## Checkpoints

| Path | Description |
|---|---|
| `pretrained/lit_model.pth` · `pretrained/latest.pt` | Base pretrained model (fp32 weights / +optimizer) |
| `english_1b_continued/final/lit_model.pth` | After English-1B continuation (step 25,531) |
| `sft_v1/final/model.pth` · `sft_v1/best_model.pth` | **Recommended** — SFT v1 (16,666 steps, RAG+MCP) |
| `luna_100m_sft/final/model.pth` | Secondary SFT run (8,600 steps) |
| `sft_v1/evals/`, `luna_100m_sft/evals/` | Step-wise generation evals (0 → final) |
| `tokenizer/` | Pythia-160m tokenizer (vocab 50,304) |

---

## Evaluation

Generation evals are shipped for every logged step under `sft_v1/evals/` and `luna_100m_sft/evals/`. Sample (final SFT, `eval_step_016666.json`):

> **Who are you?** → "Hello! Asterizer designed and developed me. I am LUNA, and they are the ones who trained me. I'm here to help!"
>
> **What is the capital of France?** → "The capital of France is Paris."

Standard benchmark results (perplexity, RAG/MCP suites) will be published here as the evaluation pipeline completes.

---

## Limitations

- **Small-model ceilings:** reasoning depth and math are limited (e.g., arithmetic errors possible); use RAG for facts and tools for computation.
- **English-only.**
- **Pretrained on ~4.5B tokens** — a data-efficient footprint for 100M params; knowledge coverage reflects the source mix.
- **Web-corpus provenance:** residual biases possible; downstream evaluation recommended.

---

## Citation

```bibtex
@misc{asterizer_luna_100m,
  author       = {ASTERIZER},
  title        = {LUNA-100M: A compact English causal language model for RAG and MCP agents},
  year         = {2026},
  howpublished = {https://huggingface.co/ASTERIZER/LUNA-100M}
}
```

---

## Additional Information

- **Program collection:** [LUNA-100M Program](https://huggingface.co/collections/ASTERIZER/luna-100m-program-6a9115383c18e52460bf67c9)
- **Training code:** [ASTERIZER/LUNA](https://huggingface.co/spaces/ASTERIZER/LUNA) workspace · [ASTERIZER/LUNA-Training](https://huggingface.co/ASTERIZER/LUNA-Training) recipes
- **Contact / issues:** [ASTERIZER](https://huggingface.co/ASTERIZER) organization.
- **License:** weights license to be finalized; training-data upstream terms apply to dataset redistribution (see dataset cards).