allow Output subclasses in contrastive search
Hi,
I am trying to use contrastive search on a custom multimodal model and I found out that it was overriding the model outputs to have the default type (like CausalLMOutputWithPast for instance). This can be an issue for models that rely on extra attributes in the output class. I think we could simply replace the attributes we want in Outputs rather than recreating the object from the default class. We achieve the same thing but it's a bit cleaner in my opinion. Happy to discuss if I'm missing something!
Also happy to add assertions to check that outputs inherit from the correct class if you want.
Hi @jood-canva !
It is a known problem and there was another GH issue reporting it. Contrastive search is currently not maintained actively and we only fix outstanding bugs, if any. The diff looks reasonable to me, did you verify that it works with a few major VLM families?
It should be fine looking at code though I think I ma not sure it covers all possible cases for VLMs 🤔