|
|
--- |
|
|
language: |
|
|
- en |
|
|
- he |
|
|
--- |
|
|
# Bilingual Language Model for Next Token Prediction |
|
|
|
|
|
## Overview |
|
|
This project focuses on building a neural network-based language model for next token prediction using two languages: **English** and **Hebrew**. The model is implemented using an LSTM (Long Short-Term Memory) architecture, designed to predict the next word in a sequence based on the training data provided. The project leverages Recurrent Neural Networks (RNNs) and evaluates the model using the **perplexity** metric to measure the quality of the predictions. |
|
|
|
|
|
The final model and checkpoints are provided, along with training history including perplexity and loss values. |
|
|
|
|
|
## Model Architecture |
|
|
- **Embedding Layer**: Converts tokenized words into dense vector representations. |
|
|
- **LSTM Layer**: Consists of 128 units to capture long-term dependencies in the sequence data. |
|
|
- **Dense Output Layer**: Outputs a probability distribution over the vocabulary to predict the next word. |
|
|
- **Total Vocabulary Size**: The model is trained on a corpus of size `[total_words]` (combining both English and Hebrew datasets). |
|
|
|
|
|
## Dataset |
|
|
The model is trained using a combination of English and Hebrew text datasets. The input sequences are tokenized and padded to ensure consistent input length for training the model. |
|
|
|
|
|
## Training |
|
|
The model was trained with the following parameters: |
|
|
- **Optimizer**: Adam |
|
|
- **Loss Function**: Categorical Crossentropy |
|
|
- **Batch Size**: 64 |
|
|
- **Epochs**: 20 |
|
|
- **Validation Split**: 20% |
|
|
|
|
|
## Evaluation Metric: Perplexity |
|
|
Perplexity is used to measure the model's performance, with lower perplexity indicating better generalization to unseen data. The final perplexity scores are: |
|
|
- **Final Training Perplexity**: `[Final Training Perplexity]` |
|
|
- **Final Validation Perplexity**: `[Final Validation Perplexity]` |
|
|
|
|
|
## Checkpoints |
|
|
A checkpoint mechanism is used to save the model at its best-performing stage based on validation loss. The best model checkpoint (`best_model.keras`) is included, which can be loaded for inference. |
|
|
|
|
|
## Results |
|
|
The model demonstrates competitive performance in predicting next tokens for both English and Hebrew, achieving satisfactory perplexity scores on both training and validation datasets. |
|
|
|
|
|
## How to Use |
|
|
To use this model, follow these steps: |
|
|
|
|
|
1. **Clone the repository**: |
|
|
```bash |
|
|
git clone https://huggingface.co/username/model-name |
|
|
cd model-name |
|
|
|
