Why ValueError: `temperature` (=0.0) has to be a strictly positive float ???
generation_config = GenerationConfig(
max_length=1024, temperature=0.00, top_p=0.95, repetition_penalty=1.1,
do_sample=True, use_cache=True,
eos_token_id=tokenizer.eos_token_id, pad_token_id=tokenizer.pad_token_id,
transformers_version="4.33.1"
)
ValueError: temperature (=0.0) has to be a strictly positive float, otherwise your next token scores will be invalid. If you're looking for greedy decoding strategies, set do_sample=False.
I don't understand why the temperature here cannot be set to 0. Could you tell me what other parameters can be adjusted?
Indeed, the temperature parameter has to be a strictly positive float. The reason behind this is that temperature is used to adjust the probabilities of the next tokens during sampling. Specifically, it scales the logits (the raw prediction values) before they are turned into probabilities.
When the temperature is set to a value close to zero (but not zero), it makes the model's sampling almost deterministic, choosing the most probable token with very high likelihood. However, when the temperature is exactly zero, it would effectively mean dividing by zero or multiplying by infinity, leading to invalid probabilities.
If you're looking to have greedy decoding (always choosing the most probable next token), you can set do_sample=False and not worry about the temperature. However, if you want to use the do_sample=True option and still want the generation to be almost deterministic, set the temperature to a very small value like 0.001.