Qwen3-32B-KoTokenizer

Qwen3-32B with 3,682 Korean colloquial tokens added to the vocabulary.

Qwen3-32B's BPE tokenizer over-segments common Korean endings and particles (어미/조사) into 2-4 sub-tokens. This model adds them as single tokens, trained via QLoRA to be natively used during generation.

What was done

	Before	After
했잖아	했 + 잖 + 아 (3 tokens)	했잖아 (1 token)
봤는데	봤 + 는데 (2 tokens)	봤는데 (1 token)
죄송하지만	죄 + 송 + 하지만 (3 tokens)	죄송하지만 (1 token)
Vocab size	151,669	155,351 (+3,682)

The 3,682 tokens were extracted from HyperCLOVA's Korean-optimized vocabulary — specifically endings (어미) and particles (조사) that Qwen's BPE consistently fragments.

Training

• Method: QLoRA (r=64, alpha=128) on Colab A100
• Key technique: Old embedding freeze — gradient hook zeros out gradients for the original 151K token embeddings, forcing the optimizer to only update the 3,682 new token rows
• Data: ~77K Korean samples filtered for high new-token density (≥5 target tokens per sample), sourced from KoAlpaca, alpaca-gpt4-korean, KULLM-v2
• Epochs: 1 (with high-density data + freeze, convergence is fast)
• New token initialization: Mean pooling of constituent sub-token embeddings

Training curve

Step	Loss	Accuracy
50	1.649	65.6%
500	1.186	70.9%
1000	1.140	71.8%

Results

New token adoption rate: 92.9% — when the model generates text containing a string that matches a new token, it uses the single new token ID 92.9% of the time (vs. falling back to the old fragmented sub-tokens).

Prompt	Adoption	New tokens used
어제 친구를 만났는데 걔가 갑자기...	4/4 = 100%	났지만, 달라고, 했지만, 갔습니다
솔직히 그건 좀 아닌 것 같거든?	1/1 = 100%	좋아하는
한국의 경제 성장에 대해 설명해주세요	1/1 = 100%	때문
이거 진짜 맛있거든? 너도 한번 먹어봐	3/3 = 100%	보세요, 있으며, 하다고
Write a Python function...	0/0 = N/A	(no Korean tokens expected)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

model_id = "2264K/Qwen3-32B-KoTokenizer"

# NF4 quantization (fits in 24GB VRAM)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

# Verify new tokens work
print(tokenizer.encode("했잖아", add_special_tokens=False))
# [155305]  ← single token (was 3 tokens before)

# Generate
messages = [{"role": "user", "content": "어제 친구를 만났는데 걔가 갑자기 이상한 얘기를 하더라고."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.7, top_p=0.9)

print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Important notes

• This is a merged model (not an adapter). Load it directly like any HuggingFace model.
• The tokenizer is included. No need to load the base Qwen3-32B tokenizer separately.
• The model's generation style is unchanged from Qwen3-32B — this modification only affects tokenization efficiency, not the model's personality or capabilities.
• English and code generation are unaffected (0 new tokens in English outputs, as expected).

Files

• model-*.safetensors — merged model weights (bf16)
• tokenizer.json, tokenizer_config.json — expanded tokenizer
• token_expansion_metadata.json — metadata for all 3,682 added tokens (token strings, IDs, source sub-token IDs used for mean pooling init)

License

Apache 2.0 (same as Qwen3-32B)