AGofficial/AgGPT21

🤖 AgGPT-21

A powerful and lightweight GPT-style language model built with PyTorch, featuring word-level tokenization and GRU-based architecture.

✨ Features

• 🧠 Intelligent Architecture: GRU-based neural network with embedding layers
• 📚 Multi-File Training: Trains on multiple corpus files automatically
• ⚡ Optimized Performance: Supports GPU (CUDA), Apple Silicon (MPS), and CPU
• 🎛️ Flexible Generation: Configurable temperature, top-k, and top-p sampling
• 💬 Interactive Chat: Beautiful command-line chat interface
• 🔄 Early Stopping: Prevents overfitting with validation-based early stopping
• 📊 Progress Tracking: Real-time training progress with tqdm

🚀 Quick Start

Prerequisites

pip install torch tqdm

Training the Model

1. Prepare your training data: Place your text files in the training_corpora/ folder
2. Start training:

   python AgGPT21.py
   

The model will automatically:

• Load all .txt files from training_corpora/
• Build vocabulary from your data
• Train with validation split and early stopping
• Save the trained model as AgGPT21.pt

Interactive Chat

Once trained, start chatting with your model:

python chat.py

📁 Project Structure

AgGPT-21-2/
├── banner.png              # Project banner image
├── AgGPT21.py              # Main training script
├── chat.py                 # Interactive chat interface
├── README.md               # This file
├── AgGPT21.pt              # Trained model (generated after training)
└── training_corpora/       # Training data folder
    ├── corpora_000.txt     # Training file 1
    ├── corpora_001.txt     # Training file 2
    ├── ...                 # More training files
    └── corpora_041.txt     # Training file N

⚙️ Configuration

Model Hyperparameters

Parameter	Default	Description
SEQ_LEN	64	Sequence length for training
EMBED_SIZE	128	Embedding dimension
HIDDEN_SIZE	128	GRU hidden dimension
NUM_LAYERS	1	Number of GRU layers
DROPOUT	0.2	Dropout rate

Training Parameters

Parameter	Default	Description
BATCH_SIZE	8	Training batch size
EPOCHS	6	Maximum training epochs
LR	2e-3	Learning rate
WEIGHT_DECAY	1e-4	L2 regularization
CLIP_NORM	1.0	Gradient clipping

Generation Settings

Parameter	Default	Description
TEMPERATURE	0.9	Sampling temperature (0.1-2.0)
TOP_K	50	Top-k sampling limit
TOP_P	0.9	Nucleus sampling threshold
GENERATE_LENGTH	200	Default generation length

🎮 Chat Commands

In the interactive chat mode, you can use these commands:

• Basic Chat: Just type your message
• quit/exit/bye: End the conversation
• help: Show available commands
• clear: Clear the screen
• model: Display model information
• temp X: Set temperature (e.g., temp 0.8)
• length X: Set response length (e.g., length 150)

🧪 Example Usage

Training Example

# Train the model (automatic multi-file loading)
python AgGPT21.py

Output:

Found 42 training files
Reading corpora_000.txt...
Reading corpora_001.txt...
...
Total words loaded: 2,847,392
Vocabulary size: 30,000
Tokens used: 1,000,000 | device=mps
Model params: 4,099,200
Train batches per epoch: 1,562 | Val batches: 79
Epochs: 100%|████████████| 6/6 [05:23<00:00, 53.92s/it, train=2.1847, val=2.3456]
Saved AgGPT21.pt

Chat Example

👤 You: Tell me about artificial intelligence

🤖 AgGPT-21: Artificial intelligence is a fascinating field that focuses on creating systems capable of performing tasks that typically require human intelligence. These systems can learn from data, recognize patterns, make decisions, and solve complex problems. AI has applications in many areas including natural language processing, computer vision, robotics, and machine learning...

🔧 Advanced Usage

Custom Vocabulary Size

MAX_VOCAB = 50000  # Increase vocabulary size

Training on Subset of Data

DATA_PERCENT = 0.5  # Use only 50% of available data
MAX_TOKENS = 500_000  # Limit to 500k tokens

Multi-GPU Training

# The model automatically detects and uses available accelerators:
# - CUDA (NVIDIA GPUs)
# - MPS (Apple Silicon)
# - CPU (fallback)

📊 Model Architecture

Input → Embedding → Dropout → GRU → Dropout → [Projection] → Linear → Output
   ↑        ↓                    ↓                              ↓
 Token    Vector              Hidden                        Logits
  IDs   (128-dim)            States                     (Vocab-size)

Key Features:

• Weight Tying: Output layer shares weights with embedding layer
• Gradient Clipping: Prevents exploding gradients
• Mixed Precision: Automatic FP16 on supported devices
• Early Stopping: Validation-based training termination

🎯 Performance Tips

1. GPU Acceleration: Use CUDA or MPS for faster training
2. Batch Size: Increase if you have more memory
3. Sequence Length: Longer sequences capture more context
4. Vocabulary: Smaller vocab = faster training, larger vocab = better coverage
5. Data Quality: Clean, relevant training data improves results

🐛 Troubleshooting

Common Issues

"No .txt files found"

• Ensure your training files are in training_corpora/ with .txt extension

"CUDA out of memory"

• Reduce BATCH_SIZE or SEQ_LEN
• Use DATA_PERCENT < 1.0 to train on less data

"Model file not found"

• Train the model first with python AgGPT21.py
• Ensure AgGPT21.pt exists in the project directory

📈 Training Data Format

Your training files should be plain text. The model will automatically:

• Convert to lowercase
• Split on whitespace
• Handle special tokens like <pad>, <eos>, etc.
• Build vocabulary from all files combined

Example format:

user: how are you today
<pad>
ai: I'm doing well, thank you for asking! How are you?
<eos>

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your improvements
4. Test thoroughly
5. Submit a pull request

📄 License

This project is open source. Feel free to use, modify, and distribute as needed.

🙋‍♂️ Support

If you encounter issues or have questions:

1. Check the troubleshooting section
2. Review your training data format
3. Ensure all dependencies are installed
4. Verify your PyTorch installation supports your hardware

---

Made with ❤️ for the AI community

AgGPT-21 - Where conversation meets intelligence.