README.md

metadata

license: mit
language:
  - en
  - ru
  - he
  - de
  - ar

This is a BPE tokenizer that is used in Prome models starting from the second version