Alpha Transformer
A custom Encoder-Decoder Transformer model trained from scratch for English → French translation.
Alpha is a 51M parameter Transformer implemented entirely in PyTorch, including:
- Multi-Head Attention
- Rotary Positional Embeddings (RoPE)
- Encoder-Decoder architecture
- Cross Attention
- Custom tokenizer
- Mixed precision training
- Checkpointing system
The model was trained on the OPUS-100 English-French translation dataset.
Model Overview
| Feature | Value |
|---|---|
| Architecture | Encoder-Decoder Transformer |
| Parameters | 51.26M |
| Framework | PyTorch |
| Task | English → French Translation |
| Dataset | OPUS-100 |
| Training Samples | ~1M sentences |
| Embedding Dimension | 256 |
| Attention Heads | 8 |
| Feed Forward Dimension | 1024 |
| Encoder Layers | 3 |
| Decoder Layers | 3 |
| Maximum Sequence Length | 64 tokens |
| Positional Encoding | RoPE |
Performance
Evaluation on the OPUS-100 test set:
| Metric | Result |
|---|---|
| Token Accuracy | 62.8% |
The model was trained from scratch without using pretrained weights.
Architecture
Encoder
Each encoder block contains:
Input Embeddings | v LayerNorm | Multi-Head Self Attention + RoPE | Residual Connection | LayerNorm | Feed Forward Network | Residual Connection
Decoder
Each decoder block contains:
Target Embeddings | LayerNorm | Masked Self Attention + RoPE | Residual Connection | LayerNorm | Cross Attention | Residual Connection | LayerNorm | Feed Forward Network | Residual Connection
Files
Alpha/ │ ├── config.json │ └── Model architecture configuration │ ├── model.safetensors │ └── Trained model weights │ ├── src_vocab.json │ └── English vocabulary │ ├── tgt_vocab.json │ └── French vocabulary │ ├── tokenizer.py │ └── Custom tokenizer implementation │ ├── main.py │ └── Transformer architecture │ ├── inference.py │ └── Translation script │ └── README.md
Installation
Clone the repository:
git clone https://github.com/MehdiSkilll24/Alpha.git
cd Alpha
Install dependencies:
pip install torch safetensors
Usage
Run inference:
python inference.py
Example:
Input:
> hello, how are you?
Output:
> bonjour, comment allez-vous ?
Loading the Model Manually
import torch
import json
from safetensors.torch import load_file
from main import Transformer
with open("config.json") as f:
config = json.load(f)
weights = load_file("model.safetensors")
model = Transformer(
src_vocab_size=config["src_vocab_size"],
tgt_vocab_size=config["tgt_vocab_size"],
d_model=config["d_model"],
num_heads=config["num_heads"],
d_ff=config["d_ff"]
)
model.load_state_dict(weights)
model.eval()
Training
The model was trained using:
AdamW optimizer
Mixed precision training (FP16)
ReduceLROnPlateau scheduler
Custom PyTorch training loop
Training configuration:
Learning Rate: 3e-4
Batch Size: 32
Epochs: 18
Dataset
Training data:
OPUS-100 English-French translation dataset.
The dataset contains parallel English/French sentence pairs collected from various sources.
Limitations
This model is designed as an educational and experimental Transformer implementation.
Limitations:
Vocabulary is based on a custom word-level tokenizer
Limited context length
No pretrained initialization
Smaller dataset compared to production translation systems
Token accuracy does not directly represent BLEU score
Future Improvements
Possible improvements:
Replace word tokenizer with BPE/SentencePiece
Increase model depth
Train on larger datasets
Add beam search decoding
Evaluate with BLEU / COMET
Implement KV-cache for faster inference
Motivation
Alpha was built to understand and implement the Transformer architecture from first principles.
Rather than using existing libraries, the project implements the core components manually:
Attention calculations
Positional embeddings
Encoder/decoder blocks
Training pipeline
Model serialization
The goal is to better understand how modern language models are built internally.
Source Code
GitHub:
https://github.com/MehdiSkilll24/Alpha
Model weights:
https://huggingface.co/MehdiSkilll24/Alpha
License
MIT License
One suggestion before you upload: add **3-5 real translation examples** from your model. That will make the HF page look 10x more alive. A model card with "62.8% accuracy" is okay; a model card with:
English:
"The weather is beautiful today."
Alpha:
"Le temps est magnifique aujourd'hui."