File size: 7,341 Bytes
4b82da2 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 | ---
license: cc-by-nc-nd-4.0
language:
- en
pipeline_tag: text-generation
tags:
- text-generation
- rag
- mcp
- causal-lm
- pretraining
- instruction-finetuning
- small-language-model
---
# Model Card for LUNA-100M
**LUNA-100M** is a compact, English-first causal language model (~100M parameters) developed by **ASTERIZER** and **fully trained from scratch** β pretrained on 4.5B deduplicated tokens, extended on the English-1B corpus, then instruction-tuned on RAG + MCP knowledge. It is built for the agent era: small enough to run on-device / single-GPU inference, with grounded knowledge of **Retrieval-Augmented Generation (RAG)** and the **Model Context Protocol (MCP)**.
> β
**Status: TRAINED.** All checkpoints (pretrained, English-1B continuation, and SFT) ship in this repository.
---
## Table of Contents
- [Model Details](#model-details)
- [Program & Ecosystem](#program--ecosystem)
- [Uses](#uses)
- [Training Details](#training-details)
- [Training Data](#training-data)
- [Checkpoints](#checkpoints)
- [Evaluation](#evaluation)
- [Limitations](#limitations)
- [Citation](#citation)
- [Additional Information](#additional-information)
---
## Model Details
| Property | Value |
|---|---|
| **Developed by** | [ASTERIZER](https://huggingface.co/ASTERIZER) |
| **Model type** | Decoder-only causal LM (GPT/Pythia-style) |
| **Parameters** | β100M (10 layers Β· 768 hidden Β· 12 heads) |
| **Context window** | 1,024 tokens |
| **Vocabulary** | 50,304 (EleutherAI/Pythia-160m tokenizer, 50,254 + EOS padding) |
| **Language** | English |
| **License** | other (see [Additional Information](#additional-information)) |
| **Model card date** | 2026-08-28 |
### Why 100M?
RAG and MCP architectures move fact-retrieval and tool-orchestration **outside** the weights. A 100M-parameter core with strong protocol knowledge delivers useful agent behavior at a fraction of the scale of frontier models β on-device, in edge deployments, at high request throughput.
---
## Program & Ecosystem
| Asset | Type | Description |
|---|---|---|
| **[LUNA-100M](https://huggingface.co/ASTERIZER/LUNA-100M)** | Model | This repository β trained checkpoints + eval logs |
| **[LUNA-300M](https://huggingface.co/ASTERIZER/LUNA-300M)** | Model | Scaled-up sibling (β303M, pretrained) |
| **[LUNA_PreTrain](https://huggingface.co/datasets/ASTERIZER/LUNA_PreTrain)** | Dataset | Pretraining corpus: 4.5B deduplicated tokens, 270 litdata chunks |
| **[LUNA-RAG-MCP-SFT-10M](https://huggingface.co/datasets/ASTERIZER/LUNA-RAG-MCP-SFT-10M)** | Dataset | Instruction-tuning corpus: 10M tokens of RAG + MCP, 61.9K samples |
| **[LUNA](https://huggingface.co/spaces/ASTERIZER/LUNA)** | Workspace | Full training codebase (pretrain, SFT, eval, quantize) |
| **[LUNA-Training](https://huggingface.co/ASTERIZER/LUNA-Training)** | Code | SFT/LoRA training recipes incl. RAG+MCP configs |
| **[LUNA-100M Program](https://huggingface.co/collections/ASTERIZER/luna-100m-program-6a9115383c18e52460bf67c9)** | Collection | One link for the whole program |
---
## Uses
### Direct use
- **RAG-powered assistants** β retrieval-backed Q&A, citation-aware generation, document grounding.
- **MCP-enabled agents** β tool discovery & execution, capability negotiation, protocol-aware integration.
- **Retrieval-as-an-MCP-server architectures** β the RAG β MCP bridge of modern agent stacks.
- **On-device / single-GPU inference** β 100M params runs comfortably on CPU and edge hardware.
### Out-of-scope use
- Not for code generation, multilingual tasks, or advanced math (see [Limitations](#limitations)).
- Not a frontier knowledge model β do not benchmark against models 10β100Γ its size.
---
## Training Details
### Training procedure
| Stage | Corpus | Tokens | Result |
|---|---|---|---|
| **1. Pretrain** | [`LUNA_PreTrain`](https://huggingface.co/datasets/ASTERIZER/LUNA_PreTrain) (270 chunks) | 4,515,286,950 | `pretrained/` |
| **2. English continuation** | `litdata_english_1b` (English-1B corpus) | 420,000,000 | `english_1b_continued/` (25,531 steps) |
| **3. SFT (identity + RAG/MCP)** | [`LUNA-RAG-MCP-SFT-10M`](https://huggingface.co/datasets/ASTERIZER/LUNA-RAG-MCP-SFT-10M) | 10,000,168 | `sft_v1/` (16,666 steps) & `luna_100m_sft/` (8,600 steps) |
### Hyperparameters
- **Optimizer:** AdamW (Ξ²=0.9/0.95, Ξ΅=1e-8), weight decay 0.1, grad-clip 1.0
- **LR:** 6e-4, cosine schedule β 6e-5 (5% warmup)
- **Batch:** global batch 120 (auto-tuned micro-batch via VRAM probe)
- **Precision:** bf16 (fp16 fallback); `torch.compile` when available
- **Full configs:** `config/train_config.yaml`, `config/train_continue_english_1b.yaml` in this repo
---
## Training Data
| Source | Tokens | Notes |
|---|---:|---|
| Pretraining (LUNA_PreTrain) | 4.5B | Wikipedia Β· FineWeb-Edu Β· OpenWebText Β· cleaned web, deduplicated |
| English continuation | 420M | English-1B corpus (simplewiki + wiki-tail + fineweb-tail) |
| SFT (RAG + MCP) | 10M | 61,884 Alpaca samples, β€1,024 tok, source-grounded |
---
## Checkpoints
| Path | Description |
|---|---|
| `pretrained/lit_model.pth` Β· `pretrained/latest.pt` | Base pretrained model (fp32 weights / +optimizer) |
| `english_1b_continued/final/lit_model.pth` | After English-1B continuation (step 25,531) |
| `sft_v1/final/model.pth` Β· `sft_v1/best_model.pth` | **Recommended** β SFT v1 (16,666 steps, RAG+MCP) |
| `luna_100m_sft/final/model.pth` | Secondary SFT run (8,600 steps) |
| `sft_v1/evals/`, `luna_100m_sft/evals/` | Step-wise generation evals (0 β final) |
| `tokenizer/` | Pythia-160m tokenizer (vocab 50,304) |
---
## Evaluation
Generation evals are shipped for every logged step under `sft_v1/evals/` and `luna_100m_sft/evals/`. Sample (final SFT, `eval_step_016666.json`):
> **Who are you?** β "Hello! Asterizer designed and developed me. I am LUNA, and they are the ones who trained me. I'm here to help!"
>
> **What is the capital of France?** β "The capital of France is Paris."
Standard benchmark results (perplexity, RAG/MCP suites) will be published here as the evaluation pipeline completes.
---
## Limitations
- **Small-model ceilings:** reasoning depth and math are limited (e.g., arithmetic errors possible); use RAG for facts and tools for computation.
- **English-only.**
- **Pretrained on ~4.5B tokens** β a data-efficient footprint for 100M params; knowledge coverage reflects the source mix.
- **Web-corpus provenance:** residual biases possible; downstream evaluation recommended.
---
## Citation
```bibtex
@misc{asterizer_luna_100m,
author = {ASTERIZER},
title = {LUNA-100M: A compact English causal language model for RAG and MCP agents},
year = {2026},
howpublished = {https://huggingface.co/ASTERIZER/LUNA-100M}
}
```
---
## Additional Information
- **Program collection:** [LUNA-100M Program](https://huggingface.co/collections/ASTERIZER/luna-100m-program-6a9115383c18e52460bf67c9)
- **Training code:** [ASTERIZER/LUNA](https://huggingface.co/spaces/ASTERIZER/LUNA) workspace Β· [ASTERIZER/LUNA-Training](https://huggingface.co/ASTERIZER/LUNA-Training) recipes
- **Contact / issues:** [ASTERIZER](https://huggingface.co/ASTERIZER) organization.
- **License:** weights license to be finalized; training-data upstream terms apply to dataset redistribution (see dataset cards).
|