RakshithFury/qwen3-4b-codemixed-kan-en

Qwen3 4B Kanglish Normalizer LoRA

A Kannada-English Kanglish normalizer fine-tuned from unsloth/Qwen3-4B on my local 5070Ti alongside a Ryzen 7 9700x. Trained for only 2.5 hours on 22000 examples.

This model converts informal Kannada written in Roman script, Kannada script, English, or mixed Kannada-English form into structured JSON containing:

• Kannada-script rendering
• Natural normalized Kannada
• Formal Kannada
• English translation
• Token-level language tags

Example

Input

ivattu office ge baralla swalpa fever ide

Output

{
  "english_translation": "I cannot come to the office today because I have a slight fever.",
  "formal_kannada": "ಇಂದು ಸ್ವಲ್ಪ ಜ್ವರ ಇರುವುದರಿಂದ ಕಚೇರಿಗೆ ಬರಲು ಸಾಧ್ಯವಿಲ್ಲ.",
  "kannada_script": "ಇವತ್ತು ಆಫೀಸಿಗೆ ಬರಲ್ಲ, ಸ್ವಲ್ಪ ಜ್ವರ ಇದೆ.",
  "normalized_kannada": "ಇವತ್ತು ಆಫೀಸಿಗೆ ಬರಲ್ಲ, ಸ್ವಲ್ಪ ಜ್ವರ ಇದೆ.",
  "tokens": [
    {"lang": "kn_roman", "text": "ivattu"},
    {"lang": "en", "text": "office"},
    {"lang": "kn_roman", "text": "ge"},
    {"lang": "kn_roman", "text": "baralla"},
    {"lang": "kn_roman", "text": "swalpa"},
    {"lang": "en", "text": "fever"},
    {"lang": "kn_roman", "text": "ide"}
  ]
}

More Examples

Input - "nanu ellige barabeku"

FT model - 
Output (English) - "I need to be where"
Output (Kannada) - "ನನು ಎಲ್ಲಿಗೆ ಬರಬೇಕು"

Default Model -
Output (English) - "I am here with a barbecue"
Output (Kannada) - ನ"ಾನು ಇಲ್ಲಿ ಬರ್ಬೆಕ್ಯೂ ಇದೆ"

Intended Use

This model is designed for:

• Romanized Kannada to Kannada-script normalization
• Kannada-English code-mixed text cleanup
• Kanglish search-query normalization
• WhatsApp-style Kannada preprocessing
• Kannada dataset cleaning
• Kannada RAG query normalization
• Kannada chatbot input normalization
• Bilingual Kannada-English NLP pipelines

It is not intended to be a general-purpose chatbot.

Output Schema

The model is trained to return JSON only:

{
  "kannada_script": "string",
  "normalized_kannada": "string",
  "formal_kannada": "string",
  "english_translation": "string",
  "tokens": [
    {
      "text": "string",
      "lang": "kn_roman | kn | en | name | number | punctuation | other"
    }
  ]
}

Model Details

• Base model: unsloth/Qwen3-4B
• Fine-tuning method: QLoRA
• Training framework: Unsloth + TRL SFTTrainer
• Adapter type: LoRA
• Primary languages: Kannada, English
• Task: Structured Kanglish normalization
• Context length used during training: 2048 tokens
• Number of examples passed: 22000 examples, 1 epoch

Training Data

The training data was built from three sources:

1. Aksharantar Kannada Transliteration Data

Used for Kannada word-level romanization and transliteration supervision.

This teaches the model mappings such as:

ivattu -> ಇವತ್ತು
nange -> ನನಗೆ
madbeku -> ಮಾಡಬೇಕು

2. Samanantar Kannada-English Parallel Data

Used to create synthetic sentence-level Kanglish examples.

The pipeline converts Kannada sentences into romanized/noisy Kanglish inputs and uses the original Kannada and English sentence pair as supervision.

3. Manual Kanglish Gold Examples

A hand-written JSONL file of high-quality Kanglish examples was added to teach realistic chat-style Kannada, code-mixing, formal rewriting, and practical phrase usage.

Example categories include:

• Office messages
• Travel
• Health
• Payments
• Government forms
• Education
• Family messages
• ML/dev terminology
• Search queries
• Daily conversation

Dataset Construction

The preprocessing pipeline creates examples of this form:

{
  "source": "manual_gold",
  "messages": [
    {
      "role": "system",
      "content": "You are a Kannada-English Kanglish normalizer..."
    },
    {
      "role": "user",
      "content": "nange ee application submit madbeku"
    },
    {
      "role": "assistant",
      "content": "{\"kannada_script\":\"ನನಗೆ ಈ ಅಪ್ಲಿಕೇಶನ್ ಸಲ್ಲಿಸಬೇಕು.\",...}"
    }
  ]
}

The final generated dataset used for the first run contained:

• Train rows: 137979
• Eval rows: 2000

Training Configuration

Initial training configuration:

model_name: unsloth/Qwen3-4B
max_seq_length: 2048
load_in_4bit: true
method: qlora
lora_r: 32
lora_alpha: 64
lora_dropout: 0
target_modules:
  - q_proj
  - k_proj
  - v_proj
  - o_proj
  - gate_proj
  - up_proj
  - down_proj
per_device_train_batch_size: 2
gradient_accumulation_steps: 2
effective_batch_size: 4
learning_rate: 2e-4
lr_scheduler_type: cosine
warmup_ratio: 0.03
optimizer: adamw_8bit
epochs: 1
bf16: true
packing: false
gradient_checkpointing: true
power drawn: approx 230W for 2hrs
Max temperature: 63C

Usage

Install dependencies:

pip install unsloth transformers peft trl accelerate bitsandbytes

Load the model:

import torch
from unsloth import FastLanguageModel

model_name = "RakshithFury/qwen3-4b-kanglish-normalizer-lora"

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=model_name,
    max_seq_length=2048,
    dtype=None,
    load_in_4bit=True,
)

FastLanguageModel.for_inference(model)

Run inference:

SYSTEM_PROMPT = """You are a Kannada-English Kanglish normalizer.
Given informal Kannada written in Roman script, Kannada script, English, or mixed form, return strict JSON with:
- kannada_script: the best Kannada-script rendering of the input
- normalized_kannada: natural Kannada preserving the input's register
- formal_kannada: polite/formal Kannada
- english_translation: faithful English meaning
- tokens: token-level language tags using kn_roman, kn, en, name, number, punctuation, other
Do not add facts. Preserve names, numbers, dates, URLs, and technical terms.
Return JSON only."""

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": "ivattu office ge baralla swalpa fever ide"},
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    do_sample=False,
)

response = tokenizer.decode(
    outputs[0][inputs["input_ids"].shape[1]:],
    skip_special_tokens=True,
)

print(response)

Recommended Decoding

For deterministic normalization, use greedy decoding:

do_sample=False
temperature=None
top_p=None

For production use, validate that the output parses as JSON. If parsing fails, retry once with the same prompt and lower max_new_tokens, or use a JSON repair step.

Limitations

The model may struggle with:

• Rare dialect-specific Roman spellings
• Highly ambiguous Romanized Kannada
• Long passages with multiple speakers
• Complex wordings and slangs

Safety and Responsible Use

This model is a text normalization model. It should not be used as the sole authority for:

• Legal document interpretation
• Medical advice
• Financial decisions
• Government benefit eligibility
• Identity verification
• High-stakes translation

For user-facing systems, show the normalized output for review when accuracy matters.

License

This model card lists cc-by-nc-4.0 because the current training pipeline uses Samanantar, whose Hugging Face dataset card lists a non-commercial Creative Commons license.

If you retrain without non-commercial data, update the license accordingly.

Citation

If you use this model or dataset, please cite the upstream resources used in the pipeline:

@misc{kanglish_normalizer_qwen3_4b_lora,
  title = {Qwen3 4B Kanglish Normalizer LoRA},
  author = {Rakshith Rao},
  year = {2026},
  publisher = {Hugging Face},
  note = {Kannada-English Kanglish normalization model}
}