transformers_version 5.0
config.json says "transformers_version": "4.55.0"
https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.6/blob/main/config.json#L40
But in https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.6#paddleocr-vl-16-09b-usage-with-transformers it says version 5.0!
And in reality it fails with ver. 5.0
I used version 4.57.6 with this monkey patching:
try:
from transformers import masking_utils
_original_create_causal_mask = masking_utils.create_causal_mask
def _patched_create_causal_mask(*args, **kwargs):
if "inputs_embeds" in kwargs and "input_embeds" not in kwargs:
kwargs["input_embeds"] = kwargs.pop("inputs_embeds")
return _original_create_causal_mask(*args, **kwargs)
masking_utils.create_causal_mask = _patched_create_causal_mask
print("Applied create_causal_mask compatibility patch")
except Exception as e:
print(f"Warning: Could not apply create_causal_mask patch: {e}")
xd, init model must be use
model = (
AutoModelForCausalLM.from_pretrained(
model_path,
dtype=torch.bfloat16 if DEVICE == "cuda" else torch.float32,
# attn_implementation=ATTN_IMPLEMENTATION,
trust_remote_code=True,
)
.to(DEVICE)
.eval()
)