Delete tokenizer.json

by rabeya100X - opened 2 days ago

base: refs/heads/main

←

from: refs/pr/5

Discussion Files changed

-32150

rabeya100X

2 days ago

Removing stale tokenizer.json (32K vocab) as discussed in transformers PR #48973 so AutoTokenizer relies on vocab.txt (100K vocab) via tokenizer_config.json.

Delete tokenizer.json32bcf44a

rabeya100X changed pull request status to closed 2 days ago

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment