Commit ·
32f4398
1
Parent(s): bd4de08
Sentencepiece tokenizers trimmed down to unique. (#1)
Browse files- Sentencepiece tokenizers trimmed down to unique. (5afbbd88536ec38bbba4c7492554399d5065e755)
Co-authored-by: Silver <silveroxides@users.noreply.huggingface.co>
- spiece.model +3 -0
- spiece_tokenizer.safetensors +3 -0
spiece.model
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:328852fb0ed85b3ad0b352bc7e4e0e5a3cb9e0598825458400f033136621db2f
|
| 3 |
+
size 1592643
|
spiece_tokenizer.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:76838aa32029b589d0e1363e2cff9ea9219c05b22332dc3d4ed340fc8c8bf11c
|
| 3 |
+
size 1592747
|