Turkish Cyber Controls Mini Model (Gemma 3 1B LoRA adapter)

Status: experimental (July 2026). Evaluated, but not recommended for use: on the domain benchmark it scores close to chance, and on general Turkish questions it is slightly below its base model (results below).

A small LoRA adapter on Gemma 3 1B IT for Turkish questions about cybersecurity controls (NIST SP 800-53 Rev. 5). This repository holds the adapter weights and the tokenizer files only, not a merged model.

Repository contents

Field	Value
Adapter type	LoRA (PEFT), rank 8, alpha 8, attention and MLP projections
Base model in `adapter_config.json`	`unsloth/gemma-3-1b-it-unsloth-bnb-4bit` (a 4-bit build of `google/gemma-3-1b-it`)
Files	`adapter_model.safetensors`, `adapter_config.json`, tokenizer files, `chat_template.jinja`
Evaluated build	local Ollama model `turkish-cyber-controls:latest`, parent `gemma3:1b-it-fp16`, GGUF F16, 999.89M parameters, digest `c9b4cbe412587cdacc6aff186e4114e30c674f41091d65c70dec59f7c034b72e`

The digest, not the mutable latest tag, identifies the build used for the results below.

Usage

from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

model_id = "logicBombExe/gemma_based_mini_turkish_cyber_security_controls_assistant"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoPeftModelForCausalLM.from_pretrained(model_id)  # loads the base model named in adapter_config.json

The base model named in adapter_config.json is a bitsandbytes 4-bit build, so loading it needs bitsandbytes and, in practice, a CUDA GPU. Access to Gemma models requires accepting Google's Gemma terms on Hugging Face.

Domain result: Turkish Cyber Security Controls Benchmark

Evaluated on 2026-07-28 with version v0.1.0 of the Turkish Cyber Security Controls Benchmark: 100 Turkish scenarios across all 20 NIST SP 800-53 Rev. 5 control families, covering control selection, audit-evidence selection and audit judgment. Settings: Ollama 0.32.5, temperature 0, seed 42, num_predict=16, thinking disabled.

Prompt configuration	Strict exact-match	Format compliance	Option-recovered diagnostic
`control_id_only` (primary)	1/100 (1.00%)	5.00%	30/100 (30.00%)
`official_titles` (diagnostic)	3/100 (3.00%)	6.00%	45/100 (45.00%)

Strict exact-match is the headline metric and accepts only a bare A, B, C or D. The model usually returned a control ID or extra text, so 95 of the 100 primary outputs failed the required format.

Option-recovered accuracy separates control knowledge from format compliance: it accepts an unambiguous leading option letter, or maps an exact emitted control ID back to a displayed option, without looking at the gold answer. A uniform random choice scores 25% because the answer letters are balanced. The primary recovered result is only modestly above chance and strongly affected by response bias (78 of 100 recovered primary predictions were C). Giving the official English control titles raised the diagnostic score to 45%; that configuration is reported separately and never combined with the primary score.

Against the matched base model (gemma3:1b-it-fp16, same 100 questions), the adapter's option-recovered score was +8 pp on control_id_only (30% vs 22%, McNemar p = 0.096) and −5 pp on official_titles (45% vs 50%, p = 0.227). Neither difference is statistically conclusive, so the benchmark does not show an effect of the adapter.

Detailed results (in the benchmark repository):

The benchmark and its answer key are public, and contamination for this model has not been assessed. If the benchmark was used during training, prompt development or checkpoint selection, these scores are not an independent test.

General Turkish result: MMLU-style questions

The adapter build and its exact FP16 base model were both evaluated on all 6,200 questions of Ali Bayram's Turkish MMLU-style set on 2026-07-23.

Model	Correct	Accuracy	Invalid responses	Throughput
`turkish-cyber-controls:latest`	1,843 / 6,200	29.73%	67	1.85 q/s
`gemma3:1b-it-fp16`	1,909 / 6,200	30.79%	11	1.92 q/s

The adapter was 1.06 percentage points below the base model (paired 95% interval −2.01 to −0.12 pp; exact two-sided McNemar p = 0.0298). The strict result is sensitive to answer format: the adapter produced 56 more invalid or ambiguous responses.

The evaluation follows the published olcum.py (same Turkish instruction prompt, seed 42, 42-token generation limit, thinking disabled, Modelfile temperature, overall and per-section scores for all 62 sections), with two changes: results are never uploaded automatically, and a strict deterministic answer parser replaces the sentence-embedding fallback. All raw responses were kept for audit. The evaluator scripts and per-run reports for this comparison are not published in this repository.

What the evaluations measure

The Turkish MMLU-style set has 62 general exam sections (100 questions each) and no cybersecurity section, so it measures broad Turkish multiple-choice knowledge, not control quality. The NIST benchmark measures Turkish control selection and audit reasoning directly, but it is a 100-question expert-review draft with a public answer key. Neither is a certification test, a production security assessment or a substitute for expert review.

Do not compare the two scores directly: the MMLU-style set has five choices and its own parser, while the NIST benchmark has four choices and counts anything other than a bare letter as wrong for its headline score.

Training provenance

The training card is incomplete. The training data version, filtering, prompt template, optimiser settings, training duration, hardware and any overlap with the evaluation data are not yet documented here.