README.md

# Ursa_Minor_Smashed

A GPT-2 (124M parameters) model trained from scratch on the FineWeb-edu dataset using a modern training recipe. This model reproduces the GPT-2 small architecture with updated training techniques including Flash Attention, mixed precision training, and improved optimization strategies.

## Model Details

### Model Description
- **Developed by:** Kaileh57
- **Model type:** GPT-2 (Decoder-only Transformer)
- **Language(s):** English
- **License:** MIT
- **Finetuned from model:** Trained from scratch

### Model Architecture

| Parameter | Value |
|-----------|-------|
| Parameters | 124M |
| Layers | 12 |
| Hidden Size | 768 |
| Attention Heads | 12 |
| Context Length | 1024 |
| Vocabulary Size | 50,304 |
| Activation Function | GELU (tanh approximation) |
| Position Embeddings | Learned |
| Layer Norm | Pre-normalization |
| Attention Type | Multi-head with Flash Attention |
| Weight Tying | Token embeddings tied with output projection |

### Training Details

- **Dataset:** FineWeb-edu (10B token sample)
- **Training Regime:** Mixed precision (bfloat16)
- **Optimizer:** AdamW (β₁=0.9, β₂=0.95, ε=1e-8)
- **Learning Rate:** 1.5e-3 with cosine decay
- **Batch Size:** 524,288 tokens
- **Training Steps:** 19,073 (1 epoch)
- **Warmup Steps:** 715
- **Weight Decay:** 0.1
- **Gradient Clipping:** 1.0
- **Hardware:** NVIDIA RTX A6000 (48GB)
- **Training Time:** ~2 days

### Performance

| Benchmark | Score |
|-----------|-------|
| HellaSwag | 32.4% |
| Final Loss | 2.85 |
| Perplexity | ~17.3 |

*Note: Scores are from single epoch training. Multi-epoch training reaches ~35% on HellaSwag.*

## Uses

### Direct Use
This model can be used for:
- Text generation
- Research on efficient training methods
- Educational purposes to understand GPT architectures
- Fine-tuning for specific downstream tasks

### Out-of-Scope Use
- Production applications requiring high reliability
- Generation of factual information (model may hallucinate)
- Any use case requiring larger context than 1024 tokens

## Quick Start

### Installation

#### CPU Installation (Default)
```bash
# Clone the repository
git clone https://github.com/Kaileh57/Ursa_Minor_Smashed.git
cd Ursa_Minor_Smashed

# Set up CPU environment
pip install -r requirements.txt
# or use the setup script: ./setup.sh
```

#### CUDA Installation (For GPU Acceleration)
If you have a CUDA-capable GPU and want to use GPU acceleration:

**Linux/macOS:**
```bash
# Use the automated CUDA setup script
chmod +x setup-cuda.sh
./setup-cuda.sh
```

**Windows:**
```batch
# Use the Windows CUDA setup script
setup-cuda.bat
```

**Manual CUDA Installation:**
```bash
# Create separate CUDA environment
python -m venv venv-cuda
source venv-cuda/bin/activate  # On Windows: venv-cuda\Scripts\activate

# Install CUDA requirements
pip install -r requirements-cuda.txt
```

**CUDA Requirements:**
- NVIDIA GPU with CUDA Compute Capability 3.5 or higher
- NVIDIA drivers installed
- CUDA 11.8 or 12.1 toolkit (optional, PyTorch includes CUDA runtime)
- At least 4GB GPU memory recommended

### Basic Usage

#### Command Line Interface

**CUDA Version (GPU):**
```bash
# Basic text generation
python inference_cuda.py --prompt "Hello, I'm a language model" --max-tokens 50

# Creative writing
python inference_cuda.py --prompt "Once upon a time" --max-tokens 200 --temperature 0.9

# More focused output
python inference_cuda.py --prompt "The key to machine learning is" --max-tokens 100 --temperature 0.7 --top-k 50
```

**CPU Version:**
```bash
# Basic text generation
python inference_cpu.py --prompt "Hello, I'm a language model" --max-tokens 50

# Creative writing
python inference_cpu.py --prompt "Once upon a time" --max-tokens 120 --temperature 0.9

# More focused output
python inference_cpu.py --prompt "The key to machine learning is" --max-tokens 80 --temperature 0.7 --top-k 30
```

#### Python Interface

**CUDA Version:**
```python
from inference_cuda import generate_direct, load_model_direct

# Load model once (requires CUDA)
model = load_model_direct("model_optimized.pt")

# Generate text with CUDA optimizations
result = generate_direct(
    model, 
    "Hello, I'm a language model", 
    max_new_tokens=100,  # Higher tokens for GPU
    temperature=0.8,
    top_k=50            # Higher top_k for better quality
)
print(result)
```

**CPU Version:**
```python
from inference_cpu import generate_direct, load_model_direct

# Load model once (CPU only)
model = load_model_direct("model_optimized.pt")

# Generate text with CPU optimizations
result = generate_direct(
    model, 
    "Hello, I'm a language model", 
    max_new_tokens=80,   # Lower tokens for CPU efficiency
    temperature=0.8,
    top_k=30            # Lower top_k for CPU efficiency
)
print(result)
```

#### Chat Interface

**CUDA Version:**
```bash
# Start CUDA-optimized chat
python chat_cuda.py
```

**CPU Version:**
```bash
# Start CPU-optimized chat
python chat_cpu.py
```

## Training Procedure

The model was trained using a modern GPT training recipe including:
- Flash Attention for efficient attention computation
- Mixed precision training with bfloat16
- Gradient accumulation to achieve large batch sizes
- TF32 for faster matrix multiplications
- Optimized vocabulary size (50,304) for better GPU utilization

### Training Hyperparameters
- **Learning rate schedule:** Cosine decay from 1.5e-3 to 1.5e-4
- **Gradient accumulation steps:** Dynamically calculated
- **Mixed precision:** bfloat16 with PyTorch autocast

## Evaluation

### Testing Data
Evaluated on:
- FineWeb-edu validation set
- HellaSwag benchmark (10,042 examples)

### Metrics
- Cross-entropy loss on validation set
- Accuracy on HellaSwag commonsense reasoning

## Technical Specifications

### Compute Infrastructure
- **Hardware:** 1x NVIDIA RTX A6000 (48GB VRAM)
- **Software:** PyTorch 2.0+, CUDA 12.1, Flash Attention 2

### Model Initialization
- Weights initialized with σ=0.02 (scaled by 1/√(2×n_layers) for residual projections)
- Embeddings initialized with σ=0.02
- Biases initialized to zero
- LayerNorm weights initialized to 1.0, biases to 0.0

## Citation

If you use this model, please cite:

```bibtex
@misc{ursa-minor-smashed,
  author = {Kaileh57},
  title = {Ursa Minor Smashed: Efficient GPT-2 Training},
  year = {2024},
  url = {https://github.com/Kaileh57/Ursa_Minor_Smashed}
}
```

## Available Tools

### Core Scripts
- **`inference_cuda.py`** - CUDA-optimized inference script
- **`inference_cpu.py`** - CPU-optimized inference script
- **`chat_cuda.py`** - CUDA-optimized chat interface
- **`chat_cpu.py`** - CPU-optimized chat interface
- **`benchmark_cuda.py`** - CUDA performance benchmarking tool
- **`benchmark_cpu.py`** - CPU performance benchmarking tool
- **`convert_to_gguf.py`** - Convert to GGUF format for llama.cpp

### Examples
- **`examples/basic_usage_cuda.py`** - CUDA usage examples
- **`examples/basic_usage_cpu.py`** - CPU usage examples

### Parameters

#### Generation Parameters
- **`temperature`** (0.1-1.0): Controls randomness (lower = more focused)
- **`top_k`** (1-100): Limit to top-k most likely tokens
- **`top_p`** (0.1-1.0): Nucleus sampling threshold
- **`repetition_penalty`** (1.0-2.0): Reduce repetitive output
- **`max_tokens`**: Maximum tokens to generate

#### Recommended Settings
- **Creative writing**: temp=0.8-0.9, top_p=0.9, top_k=50
- **Factual content**: temp=0.3-0.5, top_p=0.8, top_k=20
- **Code generation**: temp=0.2-0.4, top_p=0.7, top_k=10

## Performance

**CUDA Performance (GPU)**:
- **Inference Speed**: ~50-150+ tokens/sec (depends on GPU)
- **Memory Usage**: ~2-4GB VRAM
- **Features**: CUDA autocast, torch.compile optimization
- **Latency**: ~10-20ms per token

**CPU Performance**:
- **Inference Speed**: ~15-25 tokens/sec
- **Memory Usage**: ~2-3GB RAM
- **Features**: Multi-threading, CPU-optimized parameters
- **Latency**: ~40-65ms per token

**General**:
- **Context Length**: 1024 tokens maximum
- **Model Size**: 124M parameters

## Acknowledgments
- Based on Andrej Karpathy's nanoGPT implementation
- Trained on HuggingFace's FineWeb-edu dataset
- Uses OpenAI's GPT-2 tokenizer