Resize embeddings layer
#2
by Thalesian - opened
I really want to adapt the T5Gemma family to a project I am working on to translate cuneiform tablets - but to do it I have to resize the embeddings layer to introduce token scripts. While this is easy to do for the traditional T5 family of models, I don't see how to do it here. Any guidance would be much appreciated. I can technically do it - but I just end up re-initiating a bunch of layers that erase the models ability to function.
Hi @Thalesian , apologies for the delayed response!
If you still facing this issue, it's mostly because this particular model uses untied embeddings.
You can try manually resize and copy weights for each matrices.
If not already, you can try to use the newer T5Gemma 2.
Thank you!