File size: 241 Bytes
2f6b10b
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19


.. _tokenizer:

Tokenizer models
================

.. needtable::
   :types: tokenizer
   :style: datatable
   :columns: ID;language;license
   :class: table-striped
   :colwidths: 50;25;25

.. toctree::
   :hidden:

   Japanese/index.rst