1M mean parameter ? actually parameter is 3.7M
#9
by
johnodin99
- opened
model = AutoModelForCausalLM.from_pretrained('roneneldan/TinyStories-1M')
total_params = sum(p.numel() for p in model.parameters())
print(f"Total number of parameters in the model: {total_params:}")
Total number of parameters in the model: 3,745,984
I assume the 1M is non embedding and unembedding parameters... So check the num params without the token embedding table and LM_head.