raygx
/

BERT_Nepali_Tokenizer

Model card Files Files and versions

raygx commited on Jul 11, 2023

Commit

01d678a

·

1 Parent(s): 0ce72c0

Update README.md

Files changed (1) hide show

README.md +3 -1

README.md CHANGED Viewed

@@ -6,4 +6,6 @@
   - Model: Tokenizer(WordPiece(unk_token="[UNK]"))
   - Normalizer: normalizers.Sequence([NFD(),Strip()])
   - Pre-processor: pre_tokenizers.Sequence([Whitespace(),Digits(individual_digits=True), Punctuation()])
-  - Post-processor: BertProcessing

   - Model: Tokenizer(WordPiece(unk_token="[UNK]"))
   - Normalizer: normalizers.Sequence([NFD(),Strip()])
   - Pre-processor: pre_tokenizers.Sequence([Whitespace(),Digits(individual_digits=True), Punctuation()])
+  - Post-processor: BertProcessing
+[Code is available here.](https://www.kaggle.com/code/reganmaharjan/nepali-tokenizers-4-transformers/)