| --- |
| license: apache-2.0 |
| pipeline_tag: translation |
| tags: |
| - SimulMT |
| - Mamba-2 |
| - Cross-attention |
| - en-ru |
| datasets: |
| - OldestSalt/translation_enru |
| language: |
| - en |
| - ru |
| --- |
| |
| # Bimba |
|
|
| Bimba is *almost* linear SimulMT model trained with wait-k policy (k = 3, 5, 7, 9, 11) on en-ru translation dataset. |
|
|
| ## Architecture |
|
|
| The model has encoder-decoder architecture, where self-attention blocks are Mamba-2 blocks instead. It means that encoder is linear, but cross-attention's input is all outputs of encoder, and this means that complexity of Bimba is O(S * T), which is not *exactly* linear |
|
|
|  |
|
|
| Bimba was developed and trained as a part of master's thesis, and I hope that I will continue research in the Linear SimulMT field. |
|
|
| ## Using |
|
|
| To download Bimba you can clone the GitHub repository and use the HybridMamba2MT class: |
|
|
| ```python |
| from model_classes import HybridMamba2MT |
| from transformers import AutoTokenizer |
| |
| tokenizer = AutoTokenizer.from_pretrained("OldestSalt/Bimba") |
| model = HybridMamba2MT.from_pretrained("OldestSalt/Bimba") |
| ``` |
|
|
| ### Translation |
|
|
| Maybe someday I will write here an example of simultaneous translation. |
|
|
| ## Tokenizer |
|
|
| This model was distilled from [NLLB-200-1.3B](https://huggingface.co/facebook/nllb-200-1.3B), so Bimba uses its' tokenizer. |
|
|
| - Code: https://github.com/OldestSalt/LinearSimultMT |
| - Paper: Soon (I hope) |
