Can you PLSSS do this with the Qwen3.5 series/release the datasets and training code
I am working on training that model and changing its architecture a bit, and it would be absolutely amazing if the base is something like this but smaller.
Can you pls release the training data that you used? Or make it for the 4B/9B model yourself?
Thanks in advance.
There's the 8B version of Pepe with long context (1M) and there's a snippet of the dataset publicly available.
The creation of this model wasn't straight forward (as mention in its card).
What I'd recommend if you want to distill some 'spice' out of it, is to create a dataset of it answering general questions, for example you could use vLLM and this for the question part.
I'll consider releasing something like that myself one day.
It will be very nice to see more models with interesting persona in the OS ecosystem :)