Update README.md
Browse files
README.md
CHANGED
|
@@ -17,11 +17,33 @@ language:
|
|
| 17 |
|
| 18 |
Bimba is *almost* linear SimulMT model trained with wait-k policy (k = 3, 5, 7, 9, 11) on en-ru translation dataset.
|
| 19 |
|
|
|
|
|
|
|
| 20 |
The model has encoder-decoder architecture, where self-attention blocks are Mamba-2 blocks instead. It means that encoder is linear, but cross-attention's input is all outputs of encoder, and this means that complexity of Bimba is O(S * T), which is not *exactly* linear
|
| 21 |
|
| 22 |

|
| 23 |
|
| 24 |
-
Bimba was developed as a part of master's thesis, and I hope that I
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
|
| 26 |
- Code: https://github.com/OldestSalt/LinearSimultMT
|
| 27 |
-
- Paper: Soon
|
|
|
|
| 17 |
|
| 18 |
Bimba is *almost* linear SimulMT model trained with wait-k policy (k = 3, 5, 7, 9, 11) on en-ru translation dataset.
|
| 19 |
|
| 20 |
+
## Architecture
|
| 21 |
+
|
| 22 |
The model has encoder-decoder architecture, where self-attention blocks are Mamba-2 blocks instead. It means that encoder is linear, but cross-attention's input is all outputs of encoder, and this means that complexity of Bimba is O(S * T), which is not *exactly* linear
|
| 23 |
|
| 24 |

|
| 25 |
|
| 26 |
+
Bimba was developed and trained as a part of master's thesis, and I hope that I will continue research in the Linear SimulMT field.
|
| 27 |
+
|
| 28 |
+
## Using
|
| 29 |
+
|
| 30 |
+
To download Bimba you can clone GitHub repository and use the HybridMamba2MT class:
|
| 31 |
+
|
| 32 |
+
```python
|
| 33 |
+
from model_classes import HybridMamba2MT
|
| 34 |
+
from transformers import AutoTokenizer
|
| 35 |
+
|
| 36 |
+
tokenizer = AutoTokenizer.from_pretrained("OldestSalt/Bimba")
|
| 37 |
+
model = HybridMamba2MT.from_pretrained("OldestSalt/Bimba")
|
| 38 |
+
```
|
| 39 |
+
|
| 40 |
+
### Translation
|
| 41 |
+
|
| 42 |
+
Maybe someday I will write here an example of simultaneous translation
|
| 43 |
+
|
| 44 |
+
## Tokenizer
|
| 45 |
+
|
| 46 |
+
This model was distilled from [NLLB-200-1.3B](https://huggingface.co/facebook/nllb-200-1.3B), so Bimba uses its'
|
| 47 |
|
| 48 |
- Code: https://github.com/OldestSalt/LinearSimultMT
|
| 49 |
+
- Paper: Soon (I hope)
|