Limitations — Avaria Pygmalion-TR

Adapter scope, not standalone model

This repo contains a QLoRA adapter only (~167 MB). It will not run on its own. You must load Trendyol/Trendyol-LLM-8B-T1 as the base and attach this adapter via PEFT. We did not merge weights; the inference path is base × LoRA.

Train/eval contamination on the headline number

Trained on bezir/gsm8k-tr (Turkish translation of GSM8K).
Headline benchmarked on malhajar/gsm8k_tr-v0.2 (a different Turkish translation of the same source GSM8K problems).
Both derive from the same English GSM8K test set, so the lift on the headline benchmark partly reflects in-distribution memorization, not generalization.
Treat n=500 result as an upper-bound on this problem family.

Not a leaderboard claim

We have not run any of:

OpenLLM Turkish leaderboard (lm-evaluation-harness with gsm8k_tr, mmlu_tr, arc_tr, hellaswag_tr, winogrande_tr, truthfulqa_tr)
Cetvel / TurkBench
Multi-seed self-consistency / pass@k estimates

The plan to run those is in the project's reports/official_benchmarks/next_benchmark_plan.md.

Inference precision caveat

All accuracy numbers come from 4-bit NF4 inference matching training-time quantization. bf16 inference may shift results by roughly ±1–3 pp on reasoning tasks. We have not benchmarked bf16-only inference.

Mild over-specialization

A 30-prompt forgetting check across 7 categories produced 28/30 OK, but 2/26 non-math prompts (one coding, one abstract logic) showed math-format leakage (#### / Adım N markers in answers that did not call for them). Mitigation: use a clear non-math system prompt (e.g. "Cevabını düz metin olarak ver, matematik formatı kullanma.") for non-math production use.

Single-GPU artifact

Training and benchmarking ran on a single NVIDIA RTX 5070 Ti (16 GB, Blackwell sm_120). 16-bit (non-quantized) LoRA training was infeasible on this hardware (forced CPU offload → ~325 s/step). QLoRA with paged_adamw_8bit was the only practical path; if you train on a larger GPU, consider non-quantized LoRA at higher rank.

Greedy decoding only

All evaluations used do_sample=False, num_beams=1. No sampling, no beam search, no self-consistency. Real-world output diversity not measured.

Heuristic answer extraction

Benchmark accuracy depends on a regex chain (#### N → Cevap: N → last number in text) with Turkish thousand/decimal normalization. Some near-miss answers (e.g. correct reasoning, wrong final-number formatting) may be miscounted.

Translation artifacts

The training corpus is a Turkish translation of an English math benchmark. Some answers contain awkward Turkish phrasing or arithmetic conventions that differ from native Turkish style. The adapter inherits and can amplify these artifacts.

Not safety-tuned

No RLHF, no DPO, no safety fine-tuning was applied. Inherits all behavioral properties of the base Trendyol/Trendyol-LLM-8B-T1 outside the math domain.

What we did not change

Base model weights are untouched.
No tokenizer modifications.
No additional special tokens.
No system-prompt baked into the adapter; system prompt is set at inference time.