ASTERIZER
/

LUNA-Training

@@ -1,311 +1,38 @@
-# LUNA - 100M Parameter LLM from Scratch
-Custom ~100M parameter GPT model (Pythia-like architecture) pretrained on 4.5B tokens of clean English text.
-## Quick Start (RunPod / Cloud GPU)
-### 1. Clone & Install (one command)
-```bash
-git clone https://huggingface.co/spaces/ASTERIZER/LUNA /workspace/LUNA && \
-cd /workspace/LUNA && \
-pip install -q -r requirements.txt
-```
-### 2. Get Dataset + Train (one command)
-The dataset (~4.5B tokens) is hosted as a zip at [ASTERIZER/Luna_Dataset](https://huggingface.co/datasets/ASTERIZER/Luna_Dataset). The script downloads, extracts, and starts training automatically.
-**From HuggingFace (recommended):**
-```bash
-bash setup_and_train.sh huggingface ASTERIZER/Luna_Dataset
-```
-**From Google Drive:**
-```bash
-bash setup_and_train.sh gdrive YOUR_GDRIVE_FOLDER_ID
-```
-**Smoke test (10M tokens only):**
-```bash
-bash setup_and_train.sh huggingface ASTERIZER/Luna_Dataset 10000000
-```
-That's it. The script auto-detects your GPU, VRAM, RAM, CPU cores and configures everything for maximum utilization.
----
-## How It Works
-### Auto vs Manual Config
-All hyperparameters live in `train_config.yaml`:
-```yaml
-auto_config: true   # auto-detect everything from hardware
-auto_config: false  # use exact values below, no overrides
-```
-When `auto_config: true` (default), the trainer:
-- **Probes VRAM** via binary search to find max micro_batch_size (82% safety)
-- **Sets grad_accum** to hit the target global_batch_size
-- **Picks precision** (bf16 on Ampere+, fp16 otherwise)
-- **Scales workers** to half your CPU cores, capped by RAM
-- **Enables torch.compile** if Triton is available (Linux)
-When `auto_config: false`, every value in the YAML is used exactly as-is.
-### CLI Overrides
-Any config value can be overridden from the command line:
-```bash
-python train.py --config train_config.yaml --data_path /data/litdata --max_tokens 100000000
-```
-Priority: CLI args > train_config.yaml > auto-detection
 ---
-## Dataset
-- **4,515,286,950 tokens** (4.5B) in 270 binary chunks
-- Sources: Wikipedia, FineWeb-Edu, OpenWebText (deduplicated, cleaned)
-- Format: LitData binary (int32, block_size=1025, TokensLoader)
-- Tokenizer: EleutherAI/pythia-160m (50,254 vocab)
-## Model Architecture
-| Parameter | Value |
-|-----------|-------|
-| Layers | 10 |
-| Hidden dim | 768 |
-| Attention heads | 12 |
-| Vocab size | 50,304 (padded) |
-| Context length | 1,024 |
-| Total params | ~109M (70M unique, tied embeddings) |
-| Rotary % | 25% |
-## File Structure
-```
-LUNA/
-  train.py              # Main training script (config-driven, auto-detects hardware)
-  train_config.yaml     # All hyperparameters (auto_config: true/false)
-  fetch_data.py         # Downloads dataset from HuggingFace / GDrive
-  setup_and_train.sh    # One-command cloud entrypoint
-  benchmark_runpod.py   # Local benchmark + RunPod cost calculator
-  requirements.txt      # Python dependencies
-  Base/
-    checkpoints/EleutherAI/pythia-160m/   # Tokenizer files
-    configs/             # Legacy litgpt YAML configs (reference only)
-    scripts/             # Data preprocessing scripts
-```
-## Estimated Training Times (RunPod)
-| GPU | $/hr | tok/s | Hours | Cost USD | Cost INR |
-|-----|------|-------|-------|----------|----------|
-| RTX A5000 | $0.16 | ~6,400 | ~196h | ~$31 | ~2,700 |
-| RTX 3090 | $0.22 | ~7,600 | ~165h | ~$36 | ~3,100 |
-| RTX 4090 | $0.34 | ~10,000 | ~125h | ~$42 | ~3,600 |
-| RTX 5090 | $0.69 | ~16,000 | ~78h | ~$54 | ~4,600 |
-| H100 NVL | $2.59 | ~43,000 | ~29h | ~$75 | ~6,400 |
-## Resume Training
-Training auto-saves `latest.pt` every save_interval steps. If interrupted, just re-run the same command -- it picks up where it left off.
----
-## Verified Configs (What Worked)
-These are the exact configurations that produced the current LUNA 100M model.
-Do NOT change them unless you know what you're doing — they are proven and validated.
 ---
-### 1. Pretraining — 4.5 Billion Tokens
-The pretraining ran in two phases on an RTX 4060 Ti 16GB.
-**Phase 1: Bulk pretraining on 3B general web tokens**
-| Parameter | Value |
-|-----------|-------|
-| Dataset | `litdata_3b` — deduplicated, quality-filtered (score ≥ 0.96) general web |
-| Total tokens | 3,000,000,000 (3B) |
-| Precision | bf16-mixed |
-| Global batch size | 120 (micro_batch=12 × grad_accum=10) |
-| Sequence length | 1024 |
-| Optimizer | AdamW (lr=6e-4, min_lr=6e-5, weight_decay=0.1, betas=[0.9, 0.95]) |
-| LR schedule | Cosine decay with 500-step warmup |
-| Gradient clip | max_norm=1.0 |
-| Checkpoints | Every 1000 steps |
-| Seed | 1337 |
-| Tokenizer | EleutherAI/pythia-160m (vocab 50,254) |
-**Phase 2: Continued pretraining on clean English (Wikipedia + FineWeb-Edu)**
-| Parameter | Value |
-|-----------|-------|
-| Dataset | `litdata_english` — ultra-clean Wikipedia + FineWeb-Edu |
-| Total tokens | 150,000,000 (150M) — ~3 epochs over ~50M unique tokens |
-| Init weights | Phase 1 checkpoint (`custom-100m-3b-full/final_raw`) |
-| Precision | bf16-mixed |
-| Global batch size | 120 (micro_batch=12 × grad_accum=10) |
-| Sequence length | 1024 |
-| Optimizer | AdamW (lr=1e-4, min_lr=1e-5, weight_decay=0.1, betas=[0.9, 0.95]) |
-| LR schedule | Cosine decay with 200-step warmup |
-| Gradient clip | max_norm=1.0 |
-| Checkpoints | Every 500 steps |
-**Final combined dataset used for the production run:**
-| Parameter | Value |
-|-----------|-------|
-| Dataset | `litdata_pretrain_final` — all sources merged |
-| Total tokens | 4,515,286,950 (~4.5B) in 270 chunks |
-| Sources | Wikipedia, FineWeb-Edu, OpenWebText (deduplicated, cleaned pure English) |
-| Format | LitData binary (int32, block_size=1025, EOS=0) |
-| Config file | `train_config.yaml` |
-| Precision | bf16 |
-| Global batch size | 120 (micro_batch=12 × grad_accum=10) |
-| Sequence length | 1024 |
-| Optimizer | AdamW (lr=6e-4, min_lr=6e-5, weight_decay=0.1, betas=[0.9, 0.95]) |
-| LR schedule | Cosine with 500-step warmup (5% of total steps when auto) |
-| Gradient clip | max_norm=1.0 |
-| torch.compile | true (Linux/cloud with Triton) |
-| auto_config | true (probes VRAM, CPU, RAM at runtime) |
----
-### 2. SFT Fine-Tuning — ~145 Million Tokens
-Supervised fine-tuning on the pretrained LUNA 100M checkpoint.
-| Parameter | Value |
-|-----------|-------|
-| Dataset | `Base/Datasets/sft_clean/` — 574,996 train + 5,808 val samples |
-| Format | Alpaca JSON (instruction / input / output) |
-| Estimated tokens | ~145M total (574,996 samples × ~250 tokens avg × 2 epochs) |
-| Epochs | 2 |
-| Config file | `sft_config.yaml` |
-**Model (frozen architecture — matches pretrain exactly):**
-| Parameter | Value |
-|-----------|-------|
-| vocab_size | 50,304 (padded to 128 multiple) |
-| seq_len | 1024 |
-| n_layer | 10 |
-| n_embd | 768 |
-| n_head | 12 |
-| Rotary % | 25% |
-| Total params | 109,513,728 |
-**Training hyperparameters:**
-| Parameter | Value |
-|-----------|-------|
-| Optimizer | AdamW (lr=1.5e-5, min_lr=1e-6, weight_decay=0.01, betas=[0.9, 0.95]) |
-| Precision | bf16 |
-| Global batch size | 64 (micro_batch=8 × grad_accum=8) |
-| LR warmup | 200 steps |
-| Gradient clip | max_norm=1.0 |
-| Save interval | Every 500 steps |
-| Eval interval | Every 500 steps (runs val loss + eval prompts) |
-| DataLoader | 4 workers, pin_memory=true |
-| torch.compile | false |
-**Prompt format (used during training — must be matched at inference):**
-```
-### Instruction:
-{instruction}
-### Response:
-```
-With optional input field:
-```
-### Instruction:
-{instruction}
-### Input:
-{input}
-### Response:
-```
-**Loss masking:** Only the response tokens (after `### Response:\n`) contribute to the loss.
-The prompt tokens are masked out (loss_mask=0). EOS token (id=0) is appended to every response.
----
-### 3. SFT Inference / Chat — Loaded Configs
-These are the exact generation parameters loaded when running `chat.py` or `validate_sft.py`.
-They match the training eval config from `sft_train.py`.
-```bash
-python chat.py --ckpt "Base\out\sft\model.pth"
-```
-**Model loading:**
-| Parameter | Value |
-|-----------|-------|
-| Checkpoint | `Base/out/sft/model.pth` (419 MB, raw state_dict, 154 keys) |
-| Checkpoint format | Raw `state_dict` — NOT wrapped in `{"model": ...}` dict |
-| Tokenizer | `Base/checkpoints/EleutherAI/pythia-160m` (vocab 50,254) |
-| EOS token ID | 0 (pythia tokenizer — NOT 50276) |
-| Device | auto (CUDA if available, else CPU) |
-| Precision | float32 at inference (weights loaded as-is from bf16-trained ckpt) |
-**Generation parameters:**
-| Parameter | Value | Why |
-|-----------|-------|-----|
-| temperature | 0.7 | Balanced creativity vs coherence |
-| top_k | 40 | Matches training eval (NOT 50) |
-| top_p | 0.9 | Nucleus sampling cutoff |
-| repetition_penalty | 1.0 | No penalty — matches training (NOT 1.1) |
-| max_new_tokens | 150 | Matches training eval (NOT 256) |
-**Prompt template (must match training exactly):**
-```python
-def format_prompt(instruction, context=""):
-    if instruction and context:
-        return f"### Instruction:\n{instruction}\n\n### Input:\n{context}\n\n### Response:\n"
-    else:
-        return f"### Instruction:\n{instruction}\n\n### Response:\n"
-```
-**Critical notes:**
-- There is NO Alpaca preamble text (e.g., "Below is an instruction...") — the model was never trained with one
-- EOS token is id=0 (pythia), not 50276 (GPT-NeoX) — using the wrong EOS causes the model to never stop
-- Generation stops when EOS is produced OR max_new_tokens is reached
-- For longer responses in chat, you can override: `--max_new 512`
-- For less repetition in production, add: `--rep_pen 1.05`
-**Validation results with these configs (100 complex examples):**
-| Metric | Value |
-|--------|-------|
-| Overall Grade | A |
-| Avg Loss (CE) | 1.9167 |
-| Avg Perplexity | 7.45 |
-| Token Accuracy | 58.6% |
-| BLEU-1 | 0.589 |
-| BLEU-2 | 0.219 |
-| Empty responses | 0/100 |
-| Repetitive responses | 5/100 |
----
-## License
-Private / ASTERIZER 2026

 ---
+license: other
+language:
+- en
+tags:
+- luna
+- sft
+- lora
+- rag
+- mcp
+- training
 ---
+# LUNA-Training — SFT & LoRA Recipes for LUNA-100M
+This repository is the **supervised fine-tuning workspace** for the [LUNA-100M](https://huggingface.co/ASTERIZER/LUNA-100M) model. It contains the SFT/LoRA training code, configuration files, and the RAG + MCP dataset build pipeline used to produce the trained `sft_v1` / `luna_100m_sft` checkpoints.
+## Contents
+| Path | Description |
+|---|---|
+| `train.py` | Base pretraining trainer (shared with the [LUNA workspace](https://huggingface.co/spaces/ASTERIZER/LUNA)) |
+| `sft_train.py` | Full supervised fine-tuning trainer |
+| `lora_sft_train.py` | LoRA SFT trainer (RAG+MCP adapter runs) |
+| `lora_chat.py` / `chat.py` / `chat_full_sft.py` / `generate.py` | Inference / chat entry points |
+| `validate_sft.py` / `validate_and_quantize.py` | Eval + quantization (incl. GGUF export) |
+| `rag_mcp_lora_config.yaml` / `rag_mcp_full_sft_config.yaml` | RAG + MCP SFT recipes |
+| `sft_config.yaml` / `train_config.yaml` | Base SFT / pretrain configs |
+| `push_*_to_hf.py` | Upload helpers (code / dataset / model / LoRA) |
+| `Base/Datasets/rag_mcp_sft/` | RAG + MCP SFT dataset builder + reports |
+| `Base/checkpoints/EleutherAI/pythia-160m/` | LUNA tokenizer (vocab 50,254) |
+## Related
+- **Model:** [ASTERIZER/LUNA-100M](https://huggingface.co/ASTERIZER/LUNA-100M)
+- **SFT dataset:** [ASTERIZER/LUNA-RAG-MCP-SFT-10M](https://huggingface.co/datasets/ASTERIZER/LUNA-RAG-MCP-SFT-10M)
+- **Pretrain corpus:** [ASTERIZER/LUNA_PreTrain](https://huggingface.co/datasets/ASTERIZER/LUNA_PreTrain)
+- **Program collection:** [LUNA-100M Program](https://huggingface.co/collections/ASTERIZER/luna-100m-program-6a9115383c18e52460bf67c9)