metadata
language:
- en
- ar
- code
license: mit
tags:
- pytorch
- transformer
- text-generation
- multi-corpus
- custom-gpt
metrics:
- perplexity
Modular GPT Multi-Corpus (17.45M Parameters)
This is a custom PyTorch-native Modular GPT model trained on a balanced multi-corpus spanning English (TinyStories), Arabic (Wikipedia/Literature), and Python Code.
Architecture & Hyperparameters
- Model Parameters: 17,451,520 (17.45 Million)
- Vocabulary Size: 628 characters (shared multilingual/code character vocab)
- Embedding Dimension ($d_{\text{model}}$): 512
- Attention: Multi-Head Attention (8 heads)
- Positional Encoding: Sinusoidal
- Feed-Forward: SwiGLU (Hidden Dimension: 2048)
- Normalization: RMSNorm
- Layers: 4 Transformer blocks
Training Optimizations
- Distributed Data Parallel (DDP)
- Balanced Corpus Sampler: Equal 1/3 sampling weight across English, Arabic, and Code to prevent language imbalance
- Automatic Mixed Precision (AMP)
- Cosine Learning Rate Schedule with Warmup (Max LR: 5e-4, Min LR: 1e-5)
- Gradient Clipping: 1.0 norm
Usage with zahidgpt Python Library
pip install git+https://github.com/Zahid-coder-17/transformers
from zahidgpt import generate
# Generate Code
print(generate("def fibonacci(", model_type="multicorpus"))
# Generate English
print(generate("Once upon a time", model_type="multicorpus"))
# Generate Arabic
print(generate("مرحبا", model_type="multicorpus"))
Quick CLI Usage
python generate.py --prompt "def train_model(" --max_tokens 200