|
|
--- |
|
|
license: mit |
|
|
language: |
|
|
- it |
|
|
- en |
|
|
pipeline_tag: translation |
|
|
--- |
|
|
|
|
|
# OratioAI |
|
|
Sequecne to Sequence anguage translation, implimenting the methodes outlined in *'attention is all you need'* |
|
|
|
|
|
1. Input Tokenization: |
|
|
The source and target sentences are tokenized using custom WordPiece tokenizers. Tokens are mapped to embeddings via the InputEmbeddings module, scaled by the model dimension. |
|
|
2. Positional Encoding: |
|
|
Positional information is added to token embeddings using a fixed sinusoidal encoding strategy. |
|
|
3. Encoding Phase: |
|
|
The encoder processes the source sequence, transforming token embeddings into contextual representations using stacked EncoderBlock modules. |
|
|
4. Decoding Phase: |
|
|
The decoder autoregressively generates target tokens by attending to both previous tokens and encoder outputs. Cross-attention layers align source and target sequences effectively. |
|
|
5. Projection: |
|
|
Final decoder outputs are projected into the target vocabulary space for token prediction. |
|
|
6. Output Generation: |
|
|
Decoding is performed using a beam search or greedy approach to produce the final translated sentence. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| Resource | Description | |
|
|
|-----------------------------------|----------------------------------------------------------| |
|
|
| [Training Space](https://huggingface.co/spaces/torinriley/OratioAI) | Hugging Face Space for training and testing the model. | |
|
|
| [GitHub Source Code](https://github.com/torinriley/OratioAI) | Source code repository for the translation project. | |
|
|
| [Attention Is All You Need](https://arxiv.org/pdf/1706.03762) | Original paper on the transformer architecture published from google | |
|
|
|
|
|
| Dataset | Description | |
|
|
|-----------------------------------|----------------------------------------------------------| |
|
|
| [Dataset](https://opus.nlpl.eu/Europarl/en&it/v8/Europarl) | Dataset Used for main model training. | |
|
|
|
|
|
|
|
|
|
