Hugging Face's logo

stabilityai
/
arcade100k

Model card Files
xet
 Community
arcade100k

Commit History

  1. Update README.md

    hassanzay
    Verified signature
    627be45

  2. Update README.md

    hassanzay
    Verified signature
    e921c0e

  3. Update README.md

    hassanzay
    Verified signature
    611d7a4

  1. Update README.md

    jon-tow
    Verified signature
    b6c6300

  1. fix: add missing `errors`

    jon-tow
    b5b7b34

  1. feat: add pickle support

    jon-tow
    cd5298a

  2. fix: make `eos_token`/`pad_token` overridable

    jon-tow
    dfe1f6f

  1. feat: expose tiktoken tokenizer merge ranks and special tokens

    jon-tow
    ee2a4ae

  1. fix: check surface form against special tokens in `_add_tokens`

    jon-tow
    9ada655

  1. fix: update license

    jon-tow
    7e4138f

  1. fix: store `cl100k_base` mergeable ranks in `arcade100k.tiktoken`

    jon-tow
    1587587

  2. Merge branch 'main' of hf.co:jon-tow/stablelm-tokenizer-v1 into main

    jon-tow
    d8301d9

  3. fix: re-ordering special tokens

    jon-tow
    f221007

  4. Create tokenizer_config.json

    jon-tow
    af51a49

  5. Delete tokenizer_config.json

    jon-tow
    beeb2b0

  6. Delete special_tokens_map.json

    jon-tow
    83aaca7

  7. fix: pass in `vocab_file` to tiktoken loader

    jon-tow
    a06afc6

  8. Upload tokenizer

    jon-tow
    f3e546a

  1. fix: remove `print` debug statements

    jon-tow
    3c66e0d

  1. Update tokenization_arcade100k.py

    jon-tow
    8593ce9

  2. Update README.md

    jon-tow
    be6493b

  3. fix: create final list of special tokens

    jon-tow
    9b47601

  1. init: first commit

    jon-tow
    aceea5b

  1. initial commit

    jon-tow
    99cb326