Hugging Face's logo

2ira
/
Byte-lingua-code

Model card Files
xet
 Community
Byte-lingua-code
File size: 1,003 Bytes
72c0672
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
# Normalizers

<tokenizerslangcontent>
<python>
## BertNormalizer

[[autodoc]] tokenizers.normalizers.BertNormalizer

## Lowercase

[[autodoc]] tokenizers.normalizers.Lowercase

## NFC

[[autodoc]] tokenizers.normalizers.NFC

## NFD

[[autodoc]] tokenizers.normalizers.NFD

## NFKC

[[autodoc]] tokenizers.normalizers.NFKC

## NFKD

[[autodoc]] tokenizers.normalizers.NFKD

## Nmt

[[autodoc]] tokenizers.normalizers.Nmt

## Normalizer

[[autodoc]] tokenizers.normalizers.Normalizer

## Precompiled

[[autodoc]] tokenizers.normalizers.Precompiled

## Replace

[[autodoc]] tokenizers.normalizers.Replace

## Sequence

[[autodoc]] tokenizers.normalizers.Sequence

## Strip

[[autodoc]] tokenizers.normalizers.Strip

## StripAccents

[[autodoc]] tokenizers.normalizers.StripAccents
</python>
<rust>
The Rust API Reference is available directly on the [Docs.rs](https://docs.rs/tokenizers/latest/tokenizers/) website.
</rust>
<node>
The node API has not been documented yet.
</node>
</tokenizerslangcontent>