Commit History

  1. tokenizer_config.json: use <|im_end|> as eos_token.

    7123753
  2. removed tokenizer.json

    08b6807

  1. tokenizer.json added.

    91725c1

  1. added tiktoken.model

    a3d8e05
  2. updated README

    681fdbf
  3. initial commit

    5387bb0

  1. initial commit

    Verified signature
    5fdb435