Another question: How did you train this model?
Hello! Sorry for the strike of questions, but I'm having trouble trying to finetune the base model. I managed to teach it how to use EOS tokens, I added <|im_end|> and the default <|endoftext|> as EOS tokens like in the original Qwen3 IT models, but my problem is that, yes it responds to my prompt but after that it spits out foreign letters and some random chaotic sentences that aren't relevant to the conversation.
Ex:
<|im_start|>user
Hi there!<|im_end|>
<|im_start|>assistant
Nice to meet you! onlgnoNAOIWGnon
sag
wagajhwa
sa
The fitness gram pacer test. . .<|im_end|><|endoftext|>
I have no Idea why this is happening, it's being trained on responses only on NoRobots dataset. I searched through the internet and found ComfyUI's docs on instruction tuning Qwen3 0.6B Base, and so I copied their data preparation since they used ChatML but on Alpaca. I saw that their results were much better at 500 rows only compared to mine which had 10k rows on 1 epoch! Theirs didn't have the foreign letters and irrelevant sentences, just pure response. I don't know if this was a 'scam' and like they made it seem good but I doubt they would do that, and I haven't really tested to see for myself.
Btw, I'm using unsloth lora only at rank 32, alpha 64. rslora is enabled and dropout is at 0.05 for lora. For training I used
weight decay 0.0001
learning rate 3e-6
lr scheduler is set at cosine
warmup ratio at 0.03
num train epochs at 1, but I have tried 2 and it was the same result.
If possible, could you give me a colab or pieces of code like how you prepped your data or even the full tuning? If it's alright only ofcourse and only if you used unsloth but I see axolotl tag so full finetuning? It's alright I'll just try to learn off of it and possibly copy some logic over. Thanks!!