| This is a test sentence for the tokenizer. | |
| It includes some punctuation: Hello, world! | |
| And maybe a Greek word if the model supports it: κόσμος. | |
| <BEGIN> Special tokens test <END> | |
| This is a test sentence for the tokenizer. | |
| It includes some punctuation: Hello, world! | |
| And maybe a Greek word if the model supports it: κόσμος. | |
| <BEGIN> Special tokens test <END> | |