Declare the dtype at the top level of config.json

#1
by qgallouedec HF Staff - opened

config.json declares its dtype only in text_config, and from_pretrained(..., dtype="auto") (the transformers default) only reads the top-level one: without it, transformers takes the dtype of the first floating-point tensor of the checkpoint, here a float32 one, so the model loads in float32, about 105 GB instead of 53 GB, more than one H100. This adds "dtype": "bfloat16" at the top level, where transformers writes it when it saves a model.

aurick changed pull request status to merged

Sign up or log in to comment