README.md

---
license: other
language:
  - en
tags:
  - tokenizer
license-name: mixed
---

## Tokenzier

Tokenizer collection for different models, used in [cydonia](https://github.com/clearloop/cydonia).

## LICENSE

[LICENSE](LICENSE)