Amharic Skip-gram Word Embeddings from Scratch
A lightweight, pure-Python implementation of a Skip-gram word embedding model trained on a small Amharic corpus. Built entirely from first principles without external numerical or machine learning libraries (no NumPy, PyTorch, or TensorFlow).
Overview
This project implements a Skip-gram neural network to learn vector representations (word embeddings) for Amharic words. The primary goal is educational: to understand and verify the core mathematical foundations of word embeddings, including:
- Forward propagation and softmax distribution calculation
- Cross-entropy loss formulation
- Analytical gradient derivation for input and output weight matrices
- Stochastic Gradient Descent (SGD) optimization from scratch
The implementation relies strictly on Python standard library modules (math, random, lists, and loops).
Model Architecture & Training Parameters
| Hyperparameter | Value |
|---|---|
| Language | Amharic (am) |
| Architecture | Skip-gram with Softmax |
| Vocabulary Size | 271 unique tokens |
| Training Pairs | 1,976 |
| Embedding Dimension | 10 |
| Context Window Size | 2 |
| Learning Rate | 0.05 |
| Maximum Epochs | 50 |
| Random Seed | 42 |
| Best Checkpoint | Epoch 49 |
The model optimizes two matrices: an input embedding matrix $E$ and an output context weight matrix $U$. For a given center word, it predicts surrounding context words via a softmax probability distribution over the vocabulary.
Training & Validation Results
| Metric | Result |
|---|---|
| Initial Mean Training Loss | 5.602120 |
| Best Mean Training Loss | 2.798836 |
| Best Epoch | 49 |
| Epochs Executed | 50 |
| Max Absolute Gradient-Check Error | 0.00000000 |
Note: The best mean training loss achieved a ~50% reduction from the initial loss. Epoch 49 was selected via early checkpointing as epoch 50 yielded a slightly higher loss value. These metrics validate implementation correctness on the training split and do not reflect generalization performance on unseen text.
Training Loss Curve
The following plot illustrates the trajectory of the mean training loss recorded before training and after each epoch. The best model checkpoint was saved at epoch 49: