Discrepancy between tokens and features
Dear @shruthib ,
Thank you for sharing your amazing work!
I checked your instruction and followed to execute your sample codes, but unfortunately, I encountered the following error in Use-case 1 and 2:
Traceback (most recent call last):
File "/home/jangbi/MAIRA2/maira.py", line 59, in <module>
output_decoding = model.generate(
File "/root/anaconda3/envs/maira/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/root/anaconda3/envs/maira/lib/python3.10/site-packages/transformers/generation/utils.py", line 2210, in generate
result = self._sample(
File "/root/anaconda3/envs/maira/lib/python3.10/site-packages/transformers/generation/utils.py", line 3201, in _sample
outputs = self(**model_inputs, return_dict=True)
File "/root/anaconda3/envs/maira/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in *wrapped*call_impl
return self._call_impl(*args, **kwargs)
File "/root/anaconda3/envs/maira/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in *call*impl
return forward_call(*args, **kwargs)
File "/root/anaconda3/envs/maira/lib/python3.10/site-packages/transformers/models/llava/modeling_llava.py", line 524, in forward
raise ValueError(
ValueError: Image features and image tokens do not match: tokens: 2738, features 1369
I'm using the following environment:
Package Version
beautifulsoup4 4.12.3
certifi 2024.8.30
charset-normalizer 3.4.0
filelock 3.16.1
fsspec 2024.9.0
google 3.0.0
huggingface-hub 0.25.2
idna 3.10
Jinja2 3.1.3
MarkupSafe 2.1.5
mpmath 1.3.0
networkx 3.2.1
numpy 1.26.4
packaging 24.1
pillow 10.2.0
pip 24.2
protobuf 5.28.2
PyYAML 6.0.2
regex 2024.9.11
requests 2.32.3
safetensors 0.4.5
sentencepiece 0.2.0
setuptools 75.1.0
soupsieve 2.6
sympy 1.13.1
tokenizers 0.20.1
torch 2.1.0+cu118
torchaudio 2.1.0+cu118
torchvision 0.16.0+cu118
tqdm 4.66.5
transformers 4.46.0.dev0
triton 2.1.0
typing_extensions 4.12.2
urllib3 2.2.3
wheel 0.44.0
Have you encountered this error before, or could you please check if there might be a problem with my environment? Also, would you be able to share the environment details in which your suggested sample code is supposed to run?
I'm looking forward to your reply.
Best regards,
Jangbi
Hi @jangbi , thank you for raising this issue!
It appears that after we released MAIRA-2, the forward pass in modeling_llava has been updated in the transformers library. Specifically, there is a check in the forward pass that doesn't seem to be compatible with multi-image input introduced in https://github.com/huggingface/transformers/pull/33608.
We'll work on fixing this issue. In the meantime, could you please try installing a version of transformers before this commit? I have tested commit d7950bff82b18c823193d17d72188c5e46d06c83 but in principle any commit before the PR I mentioned above should work. To install a specific commit, you can use something similar to pip install git+https://github.com/huggingface/transformers.git@d7950bff82b18c823193d17d72188c5e46d06c83.
If you do not want to downgrade the version of transformers, you can manually specify num_additional_image_tokens=1 when creating the processor. This is because the default value for LLaVA is 0, while the DINO visual encoder in Maira2 has a CLS token.
AutoProcessor.from_pretrained(
processor_path,
trust_remote_code=True,
use_fast=True,
num_additional_image_tokens=1, # <CLS>
)
