README.md

# LUNA - 100M Parameter LLM from Scratch

Custom ~100M parameter GPT model (Pythia-like architecture) pretrained on 4.5B tokens of clean English text.

## Quick Start (RunPod / Cloud GPU)

### 1. Clone & Install (one command)

```bash
git clone https://huggingface.co/spaces/ASTERIZER/LUNA /workspace/LUNA && \
cd /workspace/LUNA && \
pip install -q -r requirements.txt
```

### 2. Get Dataset + Train (one command)

The dataset (~4.5B tokens) is hosted as a zip at [ASTERIZER/Luna_Dataset](https://huggingface.co/datasets/ASTERIZER/Luna_Dataset). The script downloads, extracts, and starts training automatically.

**From HuggingFace (recommended):**
```bash
bash setup_and_train.sh huggingface ASTERIZER/Luna_Dataset
```

**From Google Drive:**
```bash
bash setup_and_train.sh gdrive YOUR_GDRIVE_FOLDER_ID
```

**Smoke test (10M tokens only):**
```bash
bash setup_and_train.sh huggingface ASTERIZER/Luna_Dataset 10000000
```

That's it. The script auto-detects your GPU, VRAM, RAM, CPU cores and configures everything for maximum utilization.

---

## How It Works

### Auto vs Manual Config

All hyperparameters live in `train_config.yaml`:

```yaml
auto_config: true   # auto-detect everything from hardware
auto_config: false  # use exact values below, no overrides
```

When `auto_config: true` (default), the trainer:
- **Probes VRAM** via binary search to find max micro_batch_size (82% safety)
- **Sets grad_accum** to hit the target global_batch_size
- **Picks precision** (bf16 on Ampere+, fp16 otherwise)
- **Scales workers** to half your CPU cores, capped by RAM
- **Enables torch.compile** if Triton is available (Linux)

When `auto_config: false`, every value in the YAML is used exactly as-is.

### CLI Overrides

Any config value can be overridden from the command line:

```bash
python train.py --config train_config.yaml --data_path /data/litdata --max_tokens 100000000
```

Priority: CLI args > train_config.yaml > auto-detection

---

## Dataset

- **4,515,286,950 tokens** (4.5B) in 270 binary chunks
- Sources: Wikipedia, FineWeb-Edu, OpenWebText (deduplicated, cleaned)
- Format: LitData binary (int32, block_size=1025, TokensLoader)
- Tokenizer: EleutherAI/pythia-160m (50,254 vocab)

## Model Architecture

| Parameter | Value |
|-----------|-------|
| Layers | 10 |
| Hidden dim | 768 |
| Attention heads | 12 |
| Vocab size | 50,304 (padded) |
| Context length | 1,024 |
| Total params | ~109M (70M unique, tied embeddings) |
| Rotary % | 25% |

## File Structure

```
LUNA/
  train.py              # Main training script (config-driven, auto-detects hardware)
  train_config.yaml     # All hyperparameters (auto_config: true/false)
  fetch_data.py         # Downloads dataset from HuggingFace / GDrive
  setup_and_train.sh    # One-command cloud entrypoint
  benchmark_runpod.py   # Local benchmark + RunPod cost calculator
  requirements.txt      # Python dependencies
  Base/
    checkpoints/EleutherAI/pythia-160m/   # Tokenizer files
    configs/             # Legacy litgpt YAML configs (reference only)
    scripts/             # Data preprocessing scripts
```

## Estimated Training Times (RunPod)

| GPU | $/hr | tok/s | Hours | Cost USD | Cost INR |
|-----|------|-------|-------|----------|----------|
| RTX A5000 | $0.16 | ~6,400 | ~196h | ~$31 | ~2,700 |
| RTX 3090 | $0.22 | ~7,600 | ~165h | ~$36 | ~3,100 |
| RTX 4090 | $0.34 | ~10,000 | ~125h | ~$42 | ~3,600 |
| RTX 5090 | $0.69 | ~16,000 | ~78h | ~$54 | ~4,600 |
| H100 NVL | $2.59 | ~43,000 | ~29h | ~$75 | ~6,400 |

## Resume Training

Training auto-saves `latest.pt` every save_interval steps. If interrupted, just re-run the same command -- it picks up where it left off.

---

## Verified Configs (What Worked)

These are the exact configurations that produced the current LUNA 100M model.
Do NOT change them unless you know what you're doing — they are proven and validated.

---

### 1. Pretraining — 4.5 Billion Tokens

The pretraining ran in two phases on an RTX 4060 Ti 16GB.

**Phase 1: Bulk pretraining on 3B general web tokens**

| Parameter | Value |
|-----------|-------|
| Dataset | `litdata_3b` — deduplicated, quality-filtered (score ≥ 0.96) general web |
| Total tokens | 3,000,000,000 (3B) |
| Precision | bf16-mixed |
| Global batch size | 120 (micro_batch=12 × grad_accum=10) |
| Sequence length | 1024 |
| Optimizer | AdamW (lr=6e-4, min_lr=6e-5, weight_decay=0.1, betas=[0.9, 0.95]) |
| LR schedule | Cosine decay with 500-step warmup |
| Gradient clip | max_norm=1.0 |
| Checkpoints | Every 1000 steps |
| Seed | 1337 |
| Tokenizer | EleutherAI/pythia-160m (vocab 50,254) |

**Phase 2: Continued pretraining on clean English (Wikipedia + FineWeb-Edu)**

| Parameter | Value |
|-----------|-------|
| Dataset | `litdata_english` — ultra-clean Wikipedia + FineWeb-Edu |
| Total tokens | 150,000,000 (150M) — ~3 epochs over ~50M unique tokens |
| Init weights | Phase 1 checkpoint (`custom-100m-3b-full/final_raw`) |
| Precision | bf16-mixed |
| Global batch size | 120 (micro_batch=12 × grad_accum=10) |
| Sequence length | 1024 |
| Optimizer | AdamW (lr=1e-4, min_lr=1e-5, weight_decay=0.1, betas=[0.9, 0.95]) |
| LR schedule | Cosine decay with 200-step warmup |
| Gradient clip | max_norm=1.0 |
| Checkpoints | Every 500 steps |

**Final combined dataset used for the production run:**

| Parameter | Value |
|-----------|-------|
| Dataset | `litdata_pretrain_final` — all sources merged |
| Total tokens | 4,515,286,950 (~4.5B) in 270 chunks |
| Sources | Wikipedia, FineWeb-Edu, OpenWebText (deduplicated, cleaned pure English) |
| Format | LitData binary (int32, block_size=1025, EOS=0) |
| Config file | `train_config.yaml` |
| Precision | bf16 |
| Global batch size | 120 (micro_batch=12 × grad_accum=10) |
| Sequence length | 1024 |
| Optimizer | AdamW (lr=6e-4, min_lr=6e-5, weight_decay=0.1, betas=[0.9, 0.95]) |
| LR schedule | Cosine with 500-step warmup (5% of total steps when auto) |
| Gradient clip | max_norm=1.0 |
| torch.compile | true (Linux/cloud with Triton) |
| auto_config | true (probes VRAM, CPU, RAM at runtime) |

---

### 2. SFT Fine-Tuning — ~145 Million Tokens

Supervised fine-tuning on the pretrained LUNA 100M checkpoint.

| Parameter | Value |
|-----------|-------|
| Dataset | `Base/Datasets/sft_clean/` — 574,996 train + 5,808 val samples |
| Format | Alpaca JSON (instruction / input / output) |
| Estimated tokens | ~145M total (574,996 samples × ~250 tokens avg × 2 epochs) |
| Epochs | 2 |
| Config file | `sft_config.yaml` |

**Model (frozen architecture — matches pretrain exactly):**

| Parameter | Value |
|-----------|-------|
| vocab_size | 50,304 (padded to 128 multiple) |
| seq_len | 1024 |
| n_layer | 10 |
| n_embd | 768 |
| n_head | 12 |
| Rotary % | 25% |
| Total params | 109,513,728 |

**Training hyperparameters:**

| Parameter | Value |
|-----------|-------|
| Optimizer | AdamW (lr=1.5e-5, min_lr=1e-6, weight_decay=0.01, betas=[0.9, 0.95]) |
| Precision | bf16 |
| Global batch size | 64 (micro_batch=8 × grad_accum=8) |
| LR warmup | 200 steps |
| Gradient clip | max_norm=1.0 |
| Save interval | Every 500 steps |
| Eval interval | Every 500 steps (runs val loss + eval prompts) |
| DataLoader | 4 workers, pin_memory=true |
| torch.compile | false |

**Prompt format (used during training — must be matched at inference):**

```
### Instruction:
{instruction}

### Response:
```

With optional input field:

```
### Instruction:
{instruction}

### Input:
{input}

### Response:
```

**Loss masking:** Only the response tokens (after `### Response:\n`) contribute to the loss.
The prompt tokens are masked out (loss_mask=0). EOS token (id=0) is appended to every response.

---

### 3. SFT Inference / Chat — Loaded Configs

These are the exact generation parameters loaded when running `chat.py` or `validate_sft.py`.
They match the training eval config from `sft_train.py`.

```bash
python chat.py --ckpt "Base\out\sft\model.pth"
```

**Model loading:**

| Parameter | Value |
|-----------|-------|
| Checkpoint | `Base/out/sft/model.pth` (419 MB, raw state_dict, 154 keys) |
| Checkpoint format | Raw `state_dict` — NOT wrapped in `{"model": ...}` dict |
| Tokenizer | `Base/checkpoints/EleutherAI/pythia-160m` (vocab 50,254) |
| EOS token ID | 0 (pythia tokenizer — NOT 50276) |
| Device | auto (CUDA if available, else CPU) |
| Precision | float32 at inference (weights loaded as-is from bf16-trained ckpt) |

**Generation parameters:**

| Parameter | Value | Why |
|-----------|-------|-----|
| temperature | 0.7 | Balanced creativity vs coherence |
| top_k | 40 | Matches training eval (NOT 50) |
| top_p | 0.9 | Nucleus sampling cutoff |
| repetition_penalty | 1.0 | No penalty — matches training (NOT 1.1) |
| max_new_tokens | 150 | Matches training eval (NOT 256) |

**Prompt template (must match training exactly):**

```python
def format_prompt(instruction, context=""):
    if instruction and context:
        return f"### Instruction:\n{instruction}\n\n### Input:\n{context}\n\n### Response:\n"
    else:
        return f"### Instruction:\n{instruction}\n\n### Response:\n"
```

**Critical notes:**
- There is NO Alpaca preamble text (e.g., "Below is an instruction...") — the model was never trained with one
- EOS token is id=0 (pythia), not 50276 (GPT-NeoX) — using the wrong EOS causes the model to never stop
- Generation stops when EOS is produced OR max_new_tokens is reached
- For longer responses in chat, you can override: `--max_new 512`
- For less repetition in production, add: `--rep_pen 1.05`

**Validation results with these configs (100 complex examples):**

| Metric | Value |
|--------|-------|
| Overall Grade | A |
| Avg Loss (CE) | 1.9167 |
| Avg Perplexity | 7.45 |
| Token Accuracy | 58.6% |
| BLEU-1 | 0.589 |
| BLEU-2 | 0.219 |
| Empty responses | 0/100 |
| Repetitive responses | 5/100 |

---

## License

Private / ASTERIZER 2026