Any future upgrades?
Hello and many thanks for this finetune. I think this is the best choice for my rp case with mid hardware (sorry for the language mistakes, it's not my native).
But is there any chance for another MoE models for WorldSim finetune in the future? I believe that your WorldSim experiment deserved to stay alive and move on. New generations of llm networks works better and faster with low hardware cost like this qwen, but I believe there will be more soon enough.
What is my (and not only my) problem? I have 12 GB of video RAM and 32 GB of RAM. Dense models >20b are too slow for me. But too small models are not so smart and can't hold attention to many things. Modern models are much better at this with the same cost, but there is not so much rp choice yet. So MoE rp finetunes like your WorldSim has demand.
I've tried some other llms, but they haven't produced such good results. Your WorldSim perfectly keeps attention to even small details and physics of interactions. I also using minimum amount of system promt so as not to overload the llm attention resource. It's worth it. Minimum rules -> maximum results. The text of the generations is not perfect (not in English), but this is a small price to pay for the advantages. Simulation is a priority.
P. S. I have a broken randomly generated character that was accidentally created on Mistral NeMo and lived silently for a while as background character on Mistral Small without any info. He just didn't tell me anything about himself. When switching to WorldSim, his plot began to evolve, and his strange behaviour became the rationale for the appearance of dissociative flashbacks in PTSD. It's fucking scary, but awesome.
Have you taken a look at the semi-unofficial sequel?
https://huggingface.co/Gryphe/Pantheon-Reasoning-26B-A4B-1.1
The WorldSim dataset was quite literally cobbled together from all sorts of data I had lying around and is in need of a serious quality pass, so it wasn't included for that particular version but Gemma's writing is pretty darn decent!
Not yet. I tried a simple abliterated Gemma a little earlier. And yes, it writes very well. But there was a problem with the Gemma: it didn't act out the characters at all. They were all like Gemini in their answers. I'll try this one tomorrow and it will take some time (maybe until the end of the week). Many thanks! I really appreciare your work. You said that the WorldSim is not perfect and it's true, but it's still the best what I found for my emergent rp and I'm using it. The depth of the characters and the simulation of the world come first, the writing is more secondary to me.
Upd. Also readed the notes on the Gemma page. Well, my opinion about reasoning in rp - it's a killer feature and it MUST be. But it must work right. For example, tested allura's Qwen 35B A3B Anko. The reasoning block was used to correct some generation logic errors. So the efficiency is quite low. WS was much better in comparison.