amd

/

Instella-3B

Like 43

Follow

AMD 3.29k

Running PEFT/LoRA training with amd/Instella-3B, trust_remote_code=True, use_cache=True, and no supplied past_key_values fails on the first forward pass:

File ".../modeling_instella.py", line 999, in forward
    next_cache = next_cache.to_legacy_cache()
AttributeError: 'DynamicCache' object has no attribute 'to_legacy_cache'

The failure has been observed in both eager and torch.compile execution with Transformers 5.8.1, 5.14.1, and 5.17.0.

Affected model revision: 74a5c79b1fa5e421879e8e58ffb2fde79fa92d7d.

Apparent cause

The legacy-cache compatibility logic in InstellaModel.forward treats an omitted cache as a request to return the legacy tuple format:

return_legacy_cache = False
if use_cache and not isinstance(past_key_values, Cache):
    return_legacy_cache = True
    if past_key_values is None:
        past_key_values = DynamicCache()
    else:
        past_key_values = DynamicCache.from_legacy_cache(past_key_values)

With past_key_values=None, this creates a DynamicCache but marks it for conversion back to the legacy format. At the end of the forward pass, the model calls:

if return_legacy_cache:
    next_cache = next_cache.to_legacy_cache()

That method is absent in the affected Transformers versions. The tuple-input branch also references the unavailable DynamicCache.from_legacy_cache() helper.

Expected behavior

A forward pass with use_cache=True and past_key_values=None should complete and return a usable cache.

Requested change

Could you update the model code to use the supported Cache/DynamicCache API throughout, including returning a cache object when no initial cache is supplied? If legacy tuple support is retained for older Transformers versions, it should be handled without calling unavailable helpers.

Please also validate reusing the returned cache in a subsequent decoding step, in addition to the initial forward pass.