jmcinern

/

Qomhra

Like 0

Hi,

There is no vocab.json file in this repo, meaning that when I try to load the tokeniser it throws this error:

ValueError Traceback (most recent call last)
/tmp/ipykernel_1001/3800241054.py in <cell line: 0>()
4 model_id = "jmcinern/Qomhra"
5
----> 6 tokenizer = AutoTokenizer.from_pretrained(model_id)
7 model = AutoModelForCausalLM.from_pretrained(
8 model_id,

3 frames
/usr/local/lib/python3.13/dist-packages/transformers/models/qwen2/tokenization_qwen2.py in init(self, vocab, merges, unk_token, bos_token, eos_token, pad_token, add_prefix_space, **kwargs)
60 self._merges = merges or []
61 self._tokenizer = Tokenizer(
---> 62 BPE(
63 vocab=self._vocab,
64 merges=self._merges,

ValueError: vocab and merges must be both be from memory or both filenames

Should we just use the Qwen 3-8B tokeniser vocab instead? Were any changes made to Qwen vocab/tokeniser before finetuning into Qomhra, apart from the additional_tokens.json? And/or, could you upload the vocab.json file you used to make loading the model directly from here more straightforward, rather than having to patch in the Qwen one?

Thanks!
Jacquie