| --- |
| license: cc-by-nc-nd-4.0 |
| language: |
| - en |
| pipeline_tag: text-generation |
| tags: |
| - text-generation |
| - rag |
| - mcp |
| - causal-lm |
| - pretraining |
| - instruction-finetuning |
| - small-language-model |
| --- |
| |
| # Model Card for LUNA-100M |
|
|
| **LUNA-100M** is a compact, English-first causal language model (~100M parameters) developed by **ASTERIZER** and **fully trained from scratch** — pretrained on 4.5B deduplicated tokens, extended on the English-1B corpus, then instruction-tuned on RAG + MCP knowledge. It is built for the agent era: small enough to run on-device / single-GPU inference, with grounded knowledge of **Retrieval-Augmented Generation (RAG)** and the **Model Context Protocol (MCP)**. |
|
|
| > ✅ **Status: TRAINED.** All checkpoints (pretrained, English-1B continuation, and SFT) ship in this repository. |
|
|
| --- |
|
|
| ## Table of Contents |
|
|
| - [Model Details](#model-details) |
| - [Program & Ecosystem](#program--ecosystem) |
| - [Uses](#uses) |
| - [Training Details](#training-details) |
| - [Training Data](#training-data) |
| - [Checkpoints](#checkpoints) |
| - [Evaluation](#evaluation) |
| - [Limitations](#limitations) |
| - [Citation](#citation) |
| - [Additional Information](#additional-information) |
|
|
| --- |
|
|
| ## Model Details |
|
|
| | Property | Value | |
| |---|---| |
| | **Developed by** | [ASTERIZER](https://huggingface.co/ASTERIZER) | |
| | **Model type** | Decoder-only causal LM (GPT/Pythia-style) | |
| | **Parameters** | ≈100M (10 layers · 768 hidden · 12 heads) | |
| | **Context window** | 1,024 tokens | |
| | **Vocabulary** | 50,304 (EleutherAI/Pythia-160m tokenizer, 50,254 + EOS padding) | |
| | **Language** | English | |
| | **License** | other (see [Additional Information](#additional-information)) | |
| | **Model card date** | 2026-08-28 | |
|
|
| ### Why 100M? |
|
|
| RAG and MCP architectures move fact-retrieval and tool-orchestration **outside** the weights. A 100M-parameter core with strong protocol knowledge delivers useful agent behavior at a fraction of the scale of frontier models — on-device, in edge deployments, at high request throughput. |
|
|
| --- |
|
|
| ## Program & Ecosystem |
|
|
| | Asset | Type | Description | |
| |---|---|---| |
| | **[LUNA-100M](https://huggingface.co/ASTERIZER/LUNA-100M)** | Model | This repository — trained checkpoints + eval logs | |
| | **[LUNA-300M](https://huggingface.co/ASTERIZER/LUNA-300M)** | Model | Scaled-up sibling (≈303M, pretrained) | |
| | **[LUNA_PreTrain](https://huggingface.co/datasets/ASTERIZER/LUNA_PreTrain)** | Dataset | Pretraining corpus: 4.5B deduplicated tokens, 270 litdata chunks | |
| | **[LUNA-RAG-MCP-SFT-10M](https://huggingface.co/datasets/ASTERIZER/LUNA-RAG-MCP-SFT-10M)** | Dataset | Instruction-tuning corpus: 10M tokens of RAG + MCP, 61.9K samples | |
| | **[LUNA](https://huggingface.co/spaces/ASTERIZER/LUNA)** | Workspace | Full training codebase (pretrain, SFT, eval, quantize) | |
| | **[LUNA-Training](https://huggingface.co/ASTERIZER/LUNA-Training)** | Code | SFT/LoRA training recipes incl. RAG+MCP configs | |
| | **[LUNA-100M Program](https://huggingface.co/collections/ASTERIZER/luna-100m-program-6a9115383c18e52460bf67c9)** | Collection | One link for the whole program | |
|
|
| --- |
|
|
| ## Uses |
|
|
| ### Direct use |
|
|
| - **RAG-powered assistants** — retrieval-backed Q&A, citation-aware generation, document grounding. |
| - **MCP-enabled agents** — tool discovery & execution, capability negotiation, protocol-aware integration. |
| - **Retrieval-as-an-MCP-server architectures** — the RAG ⇄ MCP bridge of modern agent stacks. |
| - **On-device / single-GPU inference** — 100M params runs comfortably on CPU and edge hardware. |
|
|
| ### Out-of-scope use |
|
|
| - Not for code generation, multilingual tasks, or advanced math (see [Limitations](#limitations)). |
| - Not a frontier knowledge model — do not benchmark against models 10–100× its size. |
|
|
| --- |
|
|
| ## Training Details |
|
|
| ### Training procedure |
|
|
| | Stage | Corpus | Tokens | Result | |
| |---|---|---|---| |
| | **1. Pretrain** | [`LUNA_PreTrain`](https://huggingface.co/datasets/ASTERIZER/LUNA_PreTrain) (270 chunks) | 4,515,286,950 | `pretrained/` | |
| | **2. English continuation** | `litdata_english_1b` (English-1B corpus) | 420,000,000 | `english_1b_continued/` (25,531 steps) | |
| | **3. SFT (identity + RAG/MCP)** | [`LUNA-RAG-MCP-SFT-10M`](https://huggingface.co/datasets/ASTERIZER/LUNA-RAG-MCP-SFT-10M) | 10,000,168 | `sft_v1/` (16,666 steps) & `luna_100m_sft/` (8,600 steps) | |
|
|
| ### Hyperparameters |
|
|
| - **Optimizer:** AdamW (β=0.9/0.95, ε=1e-8), weight decay 0.1, grad-clip 1.0 |
| - **LR:** 6e-4, cosine schedule → 6e-5 (5% warmup) |
| - **Batch:** global batch 120 (auto-tuned micro-batch via VRAM probe) |
| - **Precision:** bf16 (fp16 fallback); `torch.compile` when available |
| - **Full configs:** `config/train_config.yaml`, `config/train_continue_english_1b.yaml` in this repo |
|
|
| --- |
|
|
| ## Training Data |
|
|
| | Source | Tokens | Notes | |
| |---|---:|---| |
| | Pretraining (LUNA_PreTrain) | 4.5B | Wikipedia · FineWeb-Edu · OpenWebText · cleaned web, deduplicated | |
| | English continuation | 420M | English-1B corpus (simplewiki + wiki-tail + fineweb-tail) | |
| | SFT (RAG + MCP) | 10M | 61,884 Alpaca samples, ≤1,024 tok, source-grounded | |
| |
| --- |
| |
| ## Checkpoints |
| |
| | Path | Description | |
| |---|---| |
| | `pretrained/lit_model.pth` · `pretrained/latest.pt` | Base pretrained model (fp32 weights / +optimizer) | |
| | `english_1b_continued/final/lit_model.pth` | After English-1B continuation (step 25,531) | |
| | `sft_v1/final/model.pth` · `sft_v1/best_model.pth` | **Recommended** — SFT v1 (16,666 steps, RAG+MCP) | |
| | `luna_100m_sft/final/model.pth` | Secondary SFT run (8,600 steps) | |
| | `sft_v1/evals/`, `luna_100m_sft/evals/` | Step-wise generation evals (0 → final) | |
| | `tokenizer/` | Pythia-160m tokenizer (vocab 50,304) | |
|
|
| --- |
|
|
| ## Evaluation |
|
|
| Generation evals are shipped for every logged step under `sft_v1/evals/` and `luna_100m_sft/evals/`. Sample (final SFT, `eval_step_016666.json`): |
|
|
| > **Who are you?** → "Hello! Asterizer designed and developed me. I am LUNA, and they are the ones who trained me. I'm here to help!" |
| > |
| > **What is the capital of France?** → "The capital of France is Paris." |
|
|
| Standard benchmark results (perplexity, RAG/MCP suites) will be published here as the evaluation pipeline completes. |
|
|
| --- |
|
|
| ## Limitations |
|
|
| - **Small-model ceilings:** reasoning depth and math are limited (e.g., arithmetic errors possible); use RAG for facts and tools for computation. |
| - **English-only.** |
| - **Pretrained on ~4.5B tokens** — a data-efficient footprint for 100M params; knowledge coverage reflects the source mix. |
| - **Web-corpus provenance:** residual biases possible; downstream evaluation recommended. |
|
|
| --- |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{asterizer_luna_100m, |
| author = {ASTERIZER}, |
| title = {LUNA-100M: A compact English causal language model for RAG and MCP agents}, |
| year = {2026}, |
| howpublished = {https://huggingface.co/ASTERIZER/LUNA-100M} |
| } |
| ``` |
|
|
| --- |
|
|
| ## Additional Information |
|
|
| - **Program collection:** [LUNA-100M Program](https://huggingface.co/collections/ASTERIZER/luna-100m-program-6a9115383c18e52460bf67c9) |
| - **Training code:** [ASTERIZER/LUNA](https://huggingface.co/spaces/ASTERIZER/LUNA) workspace · [ASTERIZER/LUNA-Training](https://huggingface.co/ASTERIZER/LUNA-Training) recipes |
| - **Contact / issues:** [ASTERIZER](https://huggingface.co/ASTERIZER) organization. |
| - **License:** weights license to be finalized; training-data upstream terms apply to dataset redistribution (see dataset cards). |
|
|
