README.md

---
license: mit
datasets:
- lmsys/lmsys-chat-1m
language:
- en
base_model:
- AGofficial/AgGPT-15
---
# 🤖 AgGPT-21

![AgGPT-21 Banner](banner.png)

A powerful and lightweight GPT-style language model built with PyTorch, featuring word-level tokenization and GRU-based architecture.

## ✨ Features

- **🧠 Intelligent Architecture**: GRU-based neural network with embedding layers
- **📚 Multi-File Training**: Trains on multiple corpus files automatically
- **⚡ Optimized Performance**: Supports GPU (CUDA), Apple Silicon (MPS), and CPU
- **🎛️ Flexible Generation**: Configurable temperature, top-k, and top-p sampling
- **💬 Interactive Chat**: Beautiful command-line chat interface
- **🔄 Early Stopping**: Prevents overfitting with validation-based early stopping
- **📊 Progress Tracking**: Real-time training progress with tqdm

## 🚀 Quick Start

### Prerequisites

```bash
pip install torch tqdm
```

### Training the Model

1. **Prepare your training data**: Place your text files in the `training_corpora/` folder
2. **Start training**:
   ```bash
   python AgGPT21.py
   ```

The model will automatically:
- Load all `.txt` files from `training_corpora/`
- Build vocabulary from your data
- Train with validation split and early stopping
- Save the trained model as `AgGPT21.pt`

### Interactive Chat

Once trained, start chatting with your model:

```bash
python chat.py
```

## 📁 Project Structure

```
AgGPT-21-2/
├── banner.png              # Project banner image
├── AgGPT21.py              # Main training script
├── chat.py                 # Interactive chat interface
├── README.md               # This file
├── AgGPT21.pt              # Trained model (generated after training)
└── training_corpora/       # Training data folder
    ├── corpora_000.txt     # Training file 1
    ├── corpora_001.txt     # Training file 2
    ├── ...                 # More training files
    └── corpora_041.txt     # Training file N
```

## ⚙️ Configuration

### Model Hyperparameters

| Parameter | Default | Description |
|-----------|---------|-------------|
| `SEQ_LEN` | 64 | Sequence length for training |
| `EMBED_SIZE` | 128 | Embedding dimension |
| `HIDDEN_SIZE` | 128 | GRU hidden dimension |
| `NUM_LAYERS` | 1 | Number of GRU layers |
| `DROPOUT` | 0.2 | Dropout rate |

### Training Parameters

| Parameter | Default | Description |
|-----------|---------|-------------|
| `BATCH_SIZE` | 8 | Training batch size |
| `EPOCHS` | 6 | Maximum training epochs |
| `LR` | 2e-3 | Learning rate |
| `WEIGHT_DECAY` | 1e-4 | L2 regularization |
| `CLIP_NORM` | 1.0 | Gradient clipping |

### Generation Settings

| Parameter | Default | Description |
|-----------|---------|-------------|
| `TEMPERATURE` | 0.9 | Sampling temperature (0.1-2.0) |
| `TOP_K` | 50 | Top-k sampling limit |
| `TOP_P` | 0.9 | Nucleus sampling threshold |
| `GENERATE_LENGTH` | 200 | Default generation length |

## 🎮 Chat Commands

In the interactive chat mode, you can use these commands:

- **Basic Chat**: Just type your message
- **`quit`/`exit`/`bye`**: End the conversation
- **`help`**: Show available commands
- **`clear`**: Clear the screen
- **`model`**: Display model information
- **`temp X`**: Set temperature (e.g., `temp 0.8`)
- **`length X`**: Set response length (e.g., `length 150`)

## 🧪 Example Usage

### Training Example

```python
# Train the model (automatic multi-file loading)
python AgGPT21.py
```

Output:
```
Found 42 training files
Reading corpora_000.txt...
Reading corpora_001.txt...
...
Total words loaded: 2,847,392
Vocabulary size: 30,000
Tokens used: 1,000,000 | device=mps
Model params: 4,099,200
Train batches per epoch: 1,562 | Val batches: 79
Epochs: 100%|████████████| 6/6 [05:23<00:00, 53.92s/it, train=2.1847, val=2.3456]
Saved AgGPT21.pt
```

### Chat Example

```
👤 You: Tell me about artificial intelligence

🤖 AgGPT-21: Artificial intelligence is a fascinating field that focuses on creating systems capable of performing tasks that typically require human intelligence. These systems can learn from data, recognize patterns, make decisions, and solve complex problems. AI has applications in many areas including natural language processing, computer vision, robotics, and machine learning...
```

## 🔧 Advanced Usage

### Custom Vocabulary Size

```python
MAX_VOCAB = 50000  # Increase vocabulary size
```

### Training on Subset of Data

```python
DATA_PERCENT = 0.5  # Use only 50% of available data
MAX_TOKENS = 500_000  # Limit to 500k tokens
```

### Multi-GPU Training

```python
# The model automatically detects and uses available accelerators:
# - CUDA (NVIDIA GPUs)
# - MPS (Apple Silicon)
# - CPU (fallback)
```

## 📊 Model Architecture

```
Input → Embedding → Dropout → GRU → Dropout → [Projection] → Linear → Output
   ↑        ↓                    ↓                              ↓
 Token    Vector              Hidden                        Logits
  IDs   (128-dim)            States                     (Vocab-size)
```

**Key Features:**
- **Weight Tying**: Output layer shares weights with embedding layer
- **Gradient Clipping**: Prevents exploding gradients
- **Mixed Precision**: Automatic FP16 on supported devices
- **Early Stopping**: Validation-based training termination

## 🎯 Performance Tips

1. **GPU Acceleration**: Use CUDA or MPS for faster training
2. **Batch Size**: Increase if you have more memory
3. **Sequence Length**: Longer sequences capture more context
4. **Vocabulary**: Smaller vocab = faster training, larger vocab = better coverage
5. **Data Quality**: Clean, relevant training data improves results

## 🐛 Troubleshooting

### Common Issues

**"No .txt files found"**
- Ensure your training files are in `training_corpora/` with `.txt` extension

**"CUDA out of memory"**
- Reduce `BATCH_SIZE` or `SEQ_LEN`
- Use `DATA_PERCENT < 1.0` to train on less data

**"Model file not found"**
- Train the model first with `python AgGPT21.py`
- Ensure `AgGPT21.pt` exists in the project directory

## 📈 Training Data Format

Your training files should be plain text. The model will automatically:
- Convert to lowercase
- Split on whitespace
- Handle special tokens like `<pad>`, `<eos>`, etc.
- Build vocabulary from all files combined

Example format:
```
user: how are you today
<pad>
ai: I'm doing well, thank you for asking! How are you?
<eos>
```

## 🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your improvements
4. Test thoroughly
5. Submit a pull request

## 📄 License

This project is open source. Feel free to use, modify, and distribute as needed.

## 🙋‍♂️ Support

If you encounter issues or have questions:

1. Check the troubleshooting section
2. Review your training data format
3. Ensure all dependencies are installed
4. Verify your PyTorch installation supports your hardware

---

**Made with ❤️ for the AI community**

*AgGPT-21 - Where conversation meets intelligence.*