---
language: en
tags:
- custom-gpt
- pytorch
- instruction-tuned
- from-scratch
datasets:
- HuggingFaceFW/fineweb-edu
- HuggingFaceH4/ultrachat_200k
license: mit
---

# Luna-0.1b-Instruct

This is a **124 Million parameter** language model trained from scratch. Structurally identical to the original OpenAI GPT-2 Small, this model represents a complete end-to-end LLM training pipeline built independently.

## 🧠 Training Details

The model was trained in two distinct phases to achieve "Compute-Optimal" performance for its size:

### 1. Base Pretraining
- **Dataset:** [Fineweb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) (High-quality educational text).
- **Tokens:** ~2.6 Billion tokens.

### 2. Supervised Fine-Tuning (SFT)
- **Dataset:** [UltraChat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k) (Instruction / Q&A pairs).
- **Epochs:** 1 Epoch (~5,790 steps).
- **Final Train Loss:** 2.24
- **Best Validation Loss:** 2.10

## 📉 Training Loss
Here is the training and validation loss curve during the 1-epoch Supervised Fine-Tuning phase:

![Training Loss](training_loss.png)

## 🏆 Benchmarks

After the 1-epoch Supervised Fine-Tuning, the text-only model was evaluated on three major benchmarks to test its English comprehension and world knowledge.

| Benchmark | Score | What it means |
| :--- | :--- | :--- |
| **WikiText-2 (Perplexity)** | 81.49 | The model successfully learned standard English grammar, syntax, and punctuation structure. (Lower is better). |
| **SciQ (Accuracy)** | 35.20% | The model can accurately retrieve basic scientific facts (biology, chemistry) above the 25% random-chance baseline. |
| **MMLU (Accuracy)** | 23.09% | Expected for this size. The model is too small to memorize college-level law and physics, effectively acting as random chance (~25%). |

## 💻 How to Load and Run

Because this model uses a custom `model.py` architecture script (included in this repository), you don't load it using the standard `transformers` library pipeline. Instead, download the files from this repo and use the provided PyTorch script.

```python
import torch
import tiktoken
import json
from model import GPTModel
from safetensors.torch import load_file

# 1. Load Config
with open("config.json") as f:
    cfg = json.load(f)

# 2. Instantiate Model
model = GPTModel(cfg)

# 3. Load Safetensors
state_dict = load_file("model.safetensors")
model.load_state_dict(state_dict, strict=False)
model.cuda()
model.eval()

# 4. Tokenizer
tokenizer = tiktoken.get_encoding("gpt2")
eot_token_id = tokenizer.encode("<|endoftext|>", allowed_special={"<|endoftext|>"})[0]

# 5. Inference
prompt = (
    "Below is an instruction that describes a task. "
    "Write a response that appropriately completes the request.\n\n"
    "### Instruction:\nWhat is the capital of France?\n\n### Response:\n"
)

input_ids = torch.tensor(tokenizer.encode(prompt)).unsqueeze(0).cuda()
generated = []

with torch.no_grad():
    for _ in range(100):
        logits = model(input_ids)
        next_token_logits = logits[:, -1, :]
        
        # Repetition Penalty
        penalty = 1.2
        for token_id in set(generated):
            if next_token_logits[0, token_id] < 0:
                next_token_logits[0, token_id] *= penalty
            else:
                next_token_logits[0, token_id] /= penalty
                
        next_token_id = torch.argmax(next_token_logits, dim=-1).unsqueeze(0)
        
        if next_token_id.item() == eot_token_id:
            break
            
        generated.append(next_token_id.item())
        input_ids = torch.cat([input_ids, next_token_id], dim=-1)

print(tokenizer.decode(generated))
```

## 📝 Sample Output

When running inference with a repetition penalty of `1.2`, the model generates highly coherent text and follows instructions surprisingly well for its size:

**Prompt:**
> How can I stay motivated to exercise?

**Output:**
> 1. Set realistic goals and stick to them. This will help you feel more confident in your fitness level, which can lead to better results.
> 2. Practice mindfulness meditation or yoga regularly. Mindfulness meditation helps reduce stress levels and improve overall well-being.
> 3. Take breaks throughout the day to recharge and focus on your breath.
> 4. Exercise regularly. Regular physical activity can help boost energy levels and increase muscle mass.
> 5. Get enough sleep each night. Sleep is essential for maintaining good health and reducing stress levels.
> 6. Seek professional advice from a healthcare provider if you have any concerns about your fitness level.