---
license: apache-2.0
pipeline_tag: translation
tags:
- SimulMT
- Mamba-2
- Cross-attention
- en-ru
datasets:
- OldestSalt/translation_enru
language:
- en
- ru
---

# Bimba

Bimba is *almost* linear SimulMT model trained with wait-k policy (k = 3, 5, 7, 9, 11) on en-ru translation dataset. 

## Architecture

The model has encoder-decoder architecture, where self-attention blocks are Mamba-2 blocks instead. It means that encoder is linear, but cross-attention's input is all outputs of encoder, and this means that complexity of Bimba is O(S * T), which is not *exactly* linear

![Bimba inference](https://huggingface.co/OldestSalt/Bimba/resolve/main/bimba.png)

Bimba was developed and trained as a part of master's thesis, and I hope that I will continue research in the Linear SimulMT field.

## Using

To download Bimba you can clone the GitHub repository and use the HybridMamba2MT class:

```python
from model_classes import HybridMamba2MT
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("OldestSalt/Bimba")
model = HybridMamba2MT.from_pretrained("OldestSalt/Bimba")
```

### Translation

Maybe someday I will write here an example of simultaneous translation.

## Tokenizer

This model was distilled from [NLLB-200-1.3B](https://huggingface.co/facebook/nllb-200-1.3B), so Bimba uses its' tokenizer.

- Code: https://github.com/OldestSalt/LinearSimultMT
- Paper: Soon (I hope)