vinnipukh/hukum1

hüküm1

A Turkish decision model: it reads a state (text or JSON) and a set of typed questions, and returns a calibrated probability distribution for every question in one forward pass, without generating text.

• choice: pick one of the given options
• noul: probability that the answer to a yes/no question is yes
• score: a level on an ordered scale

Base: YTÜ Turkish-Gemma-4b-T1-Scout (4.3B) with a LoRA (r=64) and an Unsloth Clef decision head (4 layers, width 512), trained for one epoch on a general decision mix of ~102k rows / ~205k decisions in Turkish and English. Its Turkish NLP follow-up is hüküm1nlp.

Usage

Requires unsloth >= 2026.10.3 (decision models with the Clef head).

from unsloth import FastDecisionModel

model, tokenizer = FastDecisionModel.from_pretrained("vinnipukh/hukum1", subfolder="merged", max_seq_length=6144)
FastDecisionModel.for_inference(model)

state = "Kargo üç gün gecikti ama ürün sağlam geldi, satıcı da hemen dönüş yaptı."
questions = {
    "duygu": {"type": "choice", "instructions": "Müşterinin genel duygusu nedir?",
              "criteria": {"olumlu": None, "olumsuz": None, "karışık": "hem olumlu hem olumsuz"}},
    "sikayet": {"type": "noul", "instructions": "Bu mesaj bir şikâyet mi?"},
}
print(FastDecisionModel.predict(model, tokenizer, state, questions))
# {"duygu": {"answer": ..., "probabilities": {...}, ...}, "sikayet": {"noul": ..., ...}}

All questions of a request are read in one prefill; up to 6 questions per request were used in training and states up to 6,144 tokens. Probabilities are temperature-calibrated per question type (stored in unsloth_decision_config.json).

Files

path
merged/	LoRA merged into the base + Clef head (bf16, ~10 GB). Use this for inference.
lora/	LoRA adapter + Clef head, to load on top of the base model
last-checkpoint/	final training checkpoint (step 6357, with optimizer state)
steps/step-N/	intermediate checkpoints (steps 1000, 2000, 3000, 4080)
results.json, results_extra.json, results_bench.json	evaluation results below

Training

data	general decision mix, 101,916 rows: tasksource-jev-typed-decisions (en + 193 tr), bev-decision, jev-bench train, tasksource procedural / synthetic, n4ze3m synthetic, LocalLLaMA typed-decisions (en); cisimcik turkish-decisions (×2), ahmetege/turkish_jev_noul, SemEval-2016 restaurant ABSA relabelled (tr). 101,701 trained, 193 calibration holdout
recipe	LoRA r=64, α=64 on all attention and MLP projections; Clef head LR 3e-4, LoRA LR 1e-4 cosine, 3% warmup; batch 8 × grad. accum. 2; max length 6,144; 1 epoch = 6,357 steps; bf16
compute	one A100 40GB, ~6.7 hours in three resumed parts
software	unsloth 2026.10.3, transformers 5.17.0, peft 0.21.1, trl 1.13.0, torch 2.11

The training data is not distributed. It contains 3,208 rows with items of public benchmarks' evaluation splits. The most important ones for anyone evaluating hüküm1: all 1,800 Belebele (tur_Latn) items, the 285 MMLU few-shot dev questions, 57 XCOPA validation rows, and ~440 GLUE MNLI dev/test premises. hüküm1's scores on those sets are not clean. None of them is in the results below (the XCOPA training rows come from the validation split, not from the test items we report).

Evaluation

Full results for all six models, task by task, with accuracy, macro-F1 and ECE: hüküm benchmarks.

Accuracy (%) per question; probabilities at the calibrated temperatures.

Decision sets

set	questions	accuracy	ECE
calibration holdout	193 rows	81.0	3.5
cisimcik holdout (tr)	1,946	84.5	5.0
turkish_noul test (tr)	1,900	88.2	2.5
jev-bench-tr test (tr)	3,000	69.1	2.8
typed-decisions test (en)	2,000	77.4	11.9
HakemBench v1.0, human-gold subset	260	67.7
HakemBench v1.0, mean of 7 tracks	4,275	78.0
MMLU-Pro-TR, fixed 1,400-question subset	1,400	27.6
Tachyon100	100	85.0

The train splits of cisimcik, turkish_noul, jev-bench and typed-decisions are in the training data; their test rows are not.

hukum-bench (29 closed-form Turkish benchmarks recast as decisions; not released): mean 69.3.

group	accuracy per task
NLI	XNLI-tr (full test) 74.4 · TrGLUE MNLI-m 81.8 · MNLI-mm 88.3 · RTE 94.9
pairs / classification	QNLI 89.8 · QQP 92.0 · MRPC 59.4 · SST-2 85.7 · CoLA 51.3 · News-Cat 92.8 · TrClaim relevance 71.3 · TrClaim check-worthiness 54.7 · X-FACT 36.1
commonsense / procedural	XCOPA 89.6 · Global PIQA non-parallel 75.0 · parallel 44.7 · PLU goal 76.1 · next event 71.0 · step 77.8 · ordering 61.8
knowledge	TR-MMLU 54.6 · TurkishMMLU 48.8 · Global-MMLU-tr 50.0 · INCLUDE-tr 56.2 · EXAMS-tr 58.8
language	TurBLiMP 65.1 · TokSuite-tr 88.4 · bilmece 56.8 · circumflex 61.4

The decision format shows every option to the model, so these numbers are not directly comparable with log-likelihood results from lm-evaluation-harness.

Against other Turkish decision models (same questions, each model with its own inference code and calibration). Knowledge = mean of TR-MMLU, TurkishMMLU, Global-MMLU-tr, INCLUDE-tr, EXAMS-tr and MMLU-Pro-TR.

	hukum-bench mean (29)	knowledge (6)	HakemBench 7 tracks	HakemBench human-gold
hüküm1	69.3	49.3	78.0	67.7
hüküm1nlp	69.5	47.7	79.1	63.1
Karar-4B	70.3	56.2	78.8	66.9
ufakzeka-karar	47.9	23.7	75.8	64.2
laya-tr-karar	51.0	22.9	49.5	41.5
saturday-labs/karar	51.1	26.2	52.1	47.3

The human-gold subset has 260 questions (±6 points). hüküm1's training data contains question sets derived from the TrGLUE train splits, which helps it on TrGLUE-style items.

hüküm1 is strongest on inference and yes/no reading (NLI, XCOPA, RTE) and weakest on world knowledge (MMLU-style tests), where the 4B base is the limit. Speed: ~38 questions/s on one A100 (Clef head, one prefill per request).

Limitations

• Single training run (one seed). Sets of 100-400 questions carry ±3-5 points of sampling noise.
• Knowledge is bounded by the 4B base; decision training does not add facts.
• Some training labels are silver (LLM-panel labels in cisimcik, LLM-assisted parts of TrGLUE).
• The model answers the questions it is asked; it does not abstain or explain.

License

The weights are a derivative of Gemma and are distributed under the Gemma Terms of Use. The training data includes sources under non-commercial and share-alike licenses (listed under Training); we recommend research and other non-commercial use.