config.json vocab_size (128128) doesn't match weights (128000) in most intermediate checkpoints
#1
by athrvkk - opened
Hi, thanks for releasing the intermediate checkpoints! I guess there is an issue with most of them , every revision from 0000180000 through 0001200000 (52 checkpoints) has vocab_size: 128128 in config.json, but embed_tokens.weight and lm_head.weight in model.safetensors are [128000, 896]. Revisions 0000020000–0000160000 are consistent (128128) and load fine.
Repro (transformers 4.57.3):
AutoModelForCausalLM.from_pretrained("trillionlabs/0.5B-Intermediate-Checkpoints", revision="0000200000")
# RuntimeError: size mismatch for weight: copying a param with shape torch.Size([128000, 896]) from checkpoint, the shape in current model is torch.Size([128128, 896]).
Could the configs be updated to vocab_size: 128000, or the weights padded back to 128128? Also, are token IDs ≥ 128000 ever used with these later checkpoints? The tokenizer still has 128128 entries.
Thanks!