Warning about some weights were not initialized
#12
by yusuf-ackan - opened
I am using lm_eval from lm-evaluation-harness to get responses.
lm_eval \
--model hf \
--model_args pretrained=allenai/OLMo-1B,revision=step738000-tokens3095B \
--tasks mmlu \
--num_fewshot 0 \
--batch_size auto \
--output_path mmlu/olmo.jsonl \
--log_samples \
--trust_remote_code
Then, it shows the following warning:
Some weights of OLMoForCausalLM were not initialized from the model checkpoint at allenai/OLMo-1B and are newly initialized: ['model.transformer.ff_out.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Is this behavior expected for the above lm_eval parameters? What can be the cause of this warning? Because I am getting a lower accuracy score for MMLU than I expected.
Could you share your installed versions of transformers and ai2_olmo? I suspect that you need to update the latter.
Hi @yusuf-ackan - I just tried this (on Ubuntu 24.04 with Python 3.12.3):
$ python -m venv .venv
$ source .venv/bin/activate
$ python -m pip install lm_eval
$ python -m pip install ai2_olmo
$ <copy-paste your lm_eval command>
The evaluation succeeds and nothing is logged about "init".
Could you please share more about your setup?
Hi, previously, ai2_olmo=0.2.5 but after the update to 0.3.0, the issue was resolved. Thank you
yusuf-ackan changed discussion status to closed