Spaces:

dnzblgn
/

Tokenizers

Build error

dnzblgn commited on Apr 28, 2025

Commit

a915aa6

verified ·

1 Parent(s): 103b27c

Update app.py

Files changed (1) hide show

app.py CHANGED Viewed

@@ -6,7 +6,6 @@ tokenizers = {
     "RoBERTa Tokenizer": AutoTokenizer.from_pretrained("roberta-base"),
     "DistilGPT-2 Tokenizer": AutoTokenizer.from_pretrained("distilgpt2"),
     "bert-base-german-cased Tokenizer": AutoTokenizer.from_pretrained("bert-base-german-cased"),
-    "GPT-4 Tokenizer":GPT2TokenizerFast.from_pretrained('Xenova/gpt-4')
 }
 # Fancy token visualization function

     "RoBERTa Tokenizer": AutoTokenizer.from_pretrained("roberta-base"),
     "DistilGPT-2 Tokenizer": AutoTokenizer.from_pretrained("distilgpt2"),
     "bert-base-german-cased Tokenizer": AutoTokenizer.from_pretrained("bert-base-german-cased"),
 }
 # Fancy token visualization function