Best model I've tried so far
I've tested like 10 or 15 different models, but this is the first one that generates the most naturally sounding and coherent texts.
I think its power is in its ability go generate casual language and not the pompous overly-prosaic / overly-philosophical tone that the most famous models seem to generate when asked to create stories or chats.
Am I correct in my understanding that it has a 4K context?
I've tested like 10 or 15 different models, but this is the first one that generates the most naturally sounding and coherent texts.
I think its power is in its ability go generate casual language and not the pompous overly-prosaic / overly-philosophical tone that the most famous models seem to generate when asked to create stories or chats.Am I correct in my understanding that it has a 4K context?
Hi! Yes, it's based on Llama 2 so it have 4k context
Ah, interesting - and sorry for being a noob (just starting out in this field) but - is there any chance for a similar model with a bigger context? e.g. is the L3 one expected to have a bigger context? or is there one with a different base and a larger context but similar outputs?
The thing is - I'm trying to write a long story so maybe there's some technique I'm missing regarding how to continue generating it once the maximum context has been reached ? Is there some sort of a "rolling" mechanism I can employ where the initial prompt is maintained and only middle prompts are lost along the way as it generates more content?
Ah, interesting - and sorry for being a noob (just starting out in this field) but - is there any chance for a similar model with a bigger context? e.g. is the L3 one expected to have a bigger context? or is there one with a different base and a larger context but similar outputs?
Sadly this is an old model from when I was merging more than finetuning, this model come from a big merge, and not a dataset that I could train on Llama3 or other LLM with bigger context. So I can't recreate it.
It was more a big experimentation of when I tried frankenmerge to have bigger model back then.