tokenizer: expose .vocab property for fast-tokenizer-style callers

tokenizer: fix decode() to handle torch tensor input via .tolist()

Update tokenizer.py

Fix tokenizer: EOS bug + decode skip_special_tokens=True empty string (#1)

Initial release · sourced from hf-carbon/carbon-3B-longctx-32k-from-mix2decay-429k @ step-453000