Upload tokenizer.json
#28
by ArthurZ HF Staff - opened
No description provided.
ArthurZ changed pull request status to merged
Per IANA registry, iw was deprecated as the code for Hebrew in 1989 and the preferred code is he
Original PR was merged into whisper here - https://github.com/openai/whisper/pull/401
HuggingFace transformers PR here - https://github.com/huggingface/transformers/pull/21310
The correct subtag:
%%
Type: language
Subtag: he
Description: Hebrew
Added: 2005-10-16
Suppress-Script: Hebr
%%
And the deprecation
%%
Type: language
Subtag: iw
Description: Hebrew
Added: 2005-10-16
Deprecated: 1989-01-01
Preferred-Value: he
Suppress-Script: Hebr
%%