muzaffercky/Sorjin2-12B

⚠️ A base model for text completion, not yet evaluated

This is google/gemma-4-12B with the Sorjîn2-LoRA adapter merged in. It continues text and has no chat template, so it does not answer questions like a chat model.

• Not evaluated: Its text completion output has not been evaluated yet
• Related issue: The same adapter merged into gemma-4-12B-it (Sorjîn2-12B-Instruct) answers Kurmanji questions in Turkish. Whether this model also drifts into Turkish has not been tested
• Intended use: A starting point for fine-tuning on Kurmanji instruction data

Sorjîn2-12B

A Kurdish Kurmanji (Latin script) base model: google/gemma-4-12B with the Sorjin2-LoRA adapter merged into its weights. The adapter was trained by continual pretraining on 585M tokens of mostly Kurmanji text, on this same base model.

It is the second version of Sorjîn, after Sorjin1.1-7B-Instruct, which was built on Qwen2.5-7B.

How It Was Made

The adapter was added to the model it was trained on:

Sorjin2-12B = gemma-4-12B + Sorjin2-LoRA

All 328 adapted modules, the attention and MLP projections of the 48 text layers, were merged, and the model was saved in bfloat16. The vision and audio parts are unchanged from gemma-4-12B. The tokenizer and processor files are the unchanged ones of gemma-4-12B, which has no chat template.

Unlike the adapter, this model loads without PEFT, and it can be fine-tuned like any other Gemma 4 model.

Models

Model	Description
muzaffercky/Sorjin2-12B	This model
muzaffercky/Sorjin2-LoRA	The adapter on its own
muzaffercky/Sorjin2-12B-Instruct	google/gemma-4-12B-it with the adapter merged in. Answers Kurmanji questions in Turkish, kept for reference only

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "muzaffercky/Sorjin2-12B"
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="cuda")
tokenizer = AutoTokenizer.from_pretrained(model_id)

inputs = tokenizer("Kurdistan welatekî", return_tensors="pt").to(model.device)
output_ids = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

In bfloat16 the model needs about 24 GB of GPU memory.

Training Data of the Adapter

One epoch over 585M tokens, counted with the Gemma 4 tokenizer:

Dataset	Part used	Documents	Tokens	Role
HuggingFaceFW/fineweb-2	kmr_Latn train	444,677	454.6M	Main Kurmanji corpus
muzaffercky/kurdish-theses-yok	Kurmanji theses, 97% of theses	398	42.7M	Academic register
HuggingFaceFW/fineweb-edu	sample-10BT	84,667	87.8M	English replay against forgetting, 15% of the mix

LoRA rank 64, alpha 128, learning rate 1e-4 with a cosine schedule, 17,865 steps of 16 sequences of 2,048 tokens, bfloat16, one H100 for about 66 hours. The full settings are on the Sorjin2-LoRA page.

Evaluation

Eval loss (cross-entropy per token) on held-out text, measured on the same 200 sequences per source before and after training:

Eval set	gemma-4-12B	gemma-4-12B + Sorjin2-LoRA
fineweb-2 kmr_Latn test split	2.274	1.620
Kurmanji theses (8 held-out theses)	2.347	1.862
fineweb-edu (English)	2.213	2.159

These losses were measured with the adapter loaded on top of gemma-4-12B during training, not on this merged model. Merging adds the same weights, so this model should match them up to the rounding of the merged weights to bfloat16, but that was not measured separately. Benchmark results (FLORES-200, Belebele) have not been run yet.

Limitations

• Not instruction-tuned: it continues text and does not answer questions in chat form. It has no chat template.
• Its text completion output has not been evaluated beyond the losses above. Check its output before relying on it.
• Kurmanji in Latin script only. Sorani and other Kurdish varieties were not trained.
• Most of the Kurmanji training text is web text. The model reproduces its style, including inconsistent spelling and Turkish-influenced phrasing, and may produce wrong or biased content.
• The theses eval set has only 8 theses, so its loss is a rough number.
• Image and audio input are unchanged from gemma-4-12B and were not tested with Kurmanji.
• Use is subject to the license and terms of the base model, google/gemma-4-12B.