Request to add a note regarding prompt tokenization
Dear Circlestone Labs,
I would like to suggest adding a brief note to the Model Card regarding prompt tokenization.
Qwen3-0.6B's vocabulary contains tokens consisting of a comma immediately followed by a single alphabetical character (for example, ",a" → token ID 15012). This also applies to periods.
See the vocabulary file:
https://huggingface.co/Qwen/Qwen3-0.6B/blob/main/vocab.json
As a result, prompts such as
masterpiece, best quality, score_7, safe
and
masterpiece,best quality,score_7,safe
may be tokenized into substantially different token sequences, even though many users would consider them equivalent. This behavior is easy to overlook because the prompts appear visually similar.
Because many image-generation prompting conventions originated from CLIP-based models, users may not expect spacing around commas to affect tokenization in this way. Consequently, users may unintentionally obtain different results depending on whether spaces are included after commas.
It may therefore be helpful to note that, when using comma-separated tags, including a space after each comma helps maintain predictable and consistent tokenization.
Thank you for considering this suggestion.
Did you see any difference between the two syntax examples? which one is better??
Yes, there is a difference in the results. I would recommend using the prompting style with spaces.
While both styles might function similarly if the training data were perfectly balanced on both, I’ve observed that certain tags seem to lose some of their effectiveness when spaces are omitted. This suggests that the model may have been primarily trained with the space-after-comma format in mind.