Base model weights
#4
by Ollie - opened
Thanks so much for the great model!
@lorenlugosch do you know if there are any plans to release the weights of the base model mentioned in the paper?
Thanks!
Glad it's useful for you Ollie!
We don't plan to release the 275M-param model weights.
If you need a smaller version of the model, you could try distilling the 1B-param model or just using a subset of the layers (I ran some experiments applying the output layer to intermediate hidden activations that suggest that this should work pretty well with a bit of finetuning).
lorenlugosch changed discussion status to closed