Upload folder using huggingface_hub
Browse files
README.md
CHANGED
|
@@ -25,7 +25,6 @@ A **Byte-Level BPE** tokenizer trained on **spa_Latn** data from Fineweb-2-HQ.
|
|
| 25 |
| Normalizer | NFC |
|
| 26 |
| Special Tokens | `<s>`, `</s>`, `<pad>`, `<unk>` |
|
| 27 |
| Training Shards | 2 |
|
| 28 |
-
| Data Source | `/scratch/gsa/data/flexitok//spa_Latn/` |
|
| 29 |
|
| 30 |
## Usage
|
| 31 |
|
|
|
|
| 25 |
| Normalizer | NFC |
|
| 26 |
| Special Tokens | `<s>`, `</s>`, `<pad>`, `<unk>` |
|
| 27 |
| Training Shards | 2 |
|
|
|
|
| 28 |
|
| 29 |
## Usage
|
| 30 |
|