---
license: apache-2.0
base_model: google/gemma-4-12B-it
language:
- cs
- en
tags:
- gemma4
- czech
- stem
- physics
- chemistry
- biology
- mathematics
- programming
- webdev
- lora
pipeline_tag: text-generation
---

# KucLab Hertz 0.6

A Czech/English STEM + programming assistant built by [KucLab](https://kuclab.org) on top of **google/gemma-4-12B-it**. This release recovers and surpasses the STEM accuracy lost in Hertz 0.5, while keeping 0.5's direct/witty personality.

## What this is

Hertz 0.6 is a **LoRA fine-tune** (r=16, merged into the base weights). Unlike 0.3-0.5, most of the training data was **not self-distilled from a teacher model this round** — the bulk (1994 rows) is a fresh, externally-generated corpus, checked for validity (0 malformed rows, 0 duplicate instructions/outputs, 0 special tokens, 0 identity leaks) before use. It's combined with 606 rows carried forward from Hertz 0.5's corpus (itself the product of three prior training generations), 309 CS↔EN scientific-terminology rows built deterministically from a fixed term list, 8 answer-first formatting examples, and 15 hand-written identity rows.

- **Base:** google/gemma-4-12B-it (11.95B params, Apache 2.0)
- **Method:** QLoRA, r=16 / alpha=32, merged to bf16 then quantized
- **Context:** 262144 tokens natively. We tried extending this via YaRN rope scaling and hit an architecture wall: Gemma-4 stores RoPE settings in a nested `rope_parameters` structure (separate config per attention type), and `rope_scaling` is a legacy alias for that same field — writing one clobbers the other and breaks GGUF export. Context stays at the native 262144; a real extension would need a different technique than YaRN on this architecture.
- **Training data:** 2932 rows total. See the honest development story below — this number is smaller than a first attempt that scored worse.
- **Format available:** GGUF (q4_k_m, ~7.4GB) for `llama.cpp`/Ollama, plus the raw LoRA adapter.

## Quickstart (Ollama)

**Important:** `ollama pull hf.co/...` alone does NOT apply this model's system prompt (identity + personality) — Ollama only fetches the raw GGUF from Hugging Face, it does not read a repo's `Modelfile`. Without the system prompt, the model falls back to identifying as a generic Gemma model. Use `ollama create` with the Modelfile below instead — it pulls the weights AND applies the system prompt in one step:

```bash
curl -O https://huggingface.co/KucLab/kuclab-hertz-0.6/resolve/main/Modelfile
ollama create kuclab-hertz-0.6 -f Modelfile
ollama run kuclab-hertz-0.6
```

(The Modelfile's `FROM` line points at `hf.co/KucLab/kuclab-hertz-0.6:Q4_K_M`, so this pulls the same GGUF automatically — no separate download needed.)

## The honest development story

We're publishing this because it's the kind of thing that usually gets edited out of a release note, and we think it's more useful left in.

**Attempt 1** used 2944 rows including 927 terminology rows (three phrasings per term, many of them bare one-word answers) and LoRA rank 32. Result: MMLU-Pro STEM 77.1%, but Czech terminology **dropped to 71.4%** (from 0.5's 75.7%), with the specifically-targeted EN→CS direction collapsing to 59.2%. Diagnosis: the bare-lookup rows taught the model to recall the 309 training pairs by rote rather than how Czech scientific terms are formed, so it fell apart on the benchmark's held-out quarter.

**Attempt 2** fixed that (definitions only, no bare lookups, rank down to 16) but Czech terminology got *worse still* — 69.4%. Diagnosis: 0.6 was built entirely from a single fresh batch of data with nothing reused from earlier releases. 0.3→0.4→0.5 had each carried forward and re-reinforced the same Czech STEM vocabulary across three training generations; throwing that away for "everything fresh" lost more than 309 new terminology rows could replace in one round.

**Attempt 3 (this release)** added 606 rows back from Hertz 0.5's corpus alongside the fresh data, restoring some of that cumulative reinforcement. Also, a GGUF export crashed with `KeyError: 'full_attention'` on the first attempt at YaRN context extension — the root cause (rope_scaling clobbering Gemma-4's nested rope_parameters) is described above and is now guarded against in the export code rather than silently corrupting the config.

Net result: MMLU-Pro STEM ended at 79.2%, Czech terminology at 73.8% — real improvement over 0.5 on STEM, still short of 0.5's terminology peak. We're not aware of a way to have fully matched both without another full iteration cycle, which we didn't have time for before this release's deadline.

## Benchmarks

Same prompts, same grading code, same Ollama Q4_K_M quantization, cold, identical corrected methodology throughout (see Hertz 0.5's card for the timeout bug we found and fixed there — this release inherits that fix).

**MMLU-Pro STEM** (240 held-out questions, this project's own curated subset)

| | base | Hertz 0.4 | Hertz 0.5 | **Hertz 0.6** |
|---|---|---|---|---|
| Biology | 86.7% | 76.7% | 78.3% | **91.7%** |
| Chemistry | 61.7% | 45.0% | 53.3% | **61.7%** |
| Math | 83.3% | 76.7% | 78.3% | **90.0%** |
| Physics | 71.7% | 56.7% | 65.0% | **73.3%** |
| **Total** | **75.8%** | **63.7%** | **68.8%** | **79.2%** |

Hertz 0.6 beats the base model by +3.4pp and every prior Hertz release on this benchmark. It also resolved far more answers cleanly: only 18/240 (7.5%) needed the fallback answer-only re-ask, down from 46/240 (19%) in Hertz 0.5 — a direct result of training on answer-first format examples, not a benchmark-harness artifact.

**Czech terminology benchmark** (206 held-out CS↔EN scientific terms)

| | Hertz 0.3 | Hertz 0.5 | **Hertz 0.6** |
|---|---|---|---|
| CS→EN | 79.6% | 81.6% | **82.5%** |
| EN→CS | 51.5% | 69.9% | **65.0%** |
| **Total** | **65.5%** | **75.7%** | **73.8%** |

Below Hertz 0.5's peak, above Hertz 0.3. See the development story above for why.

## Honest status

- ✅ **MMLU-Pro STEM: 79.2%, beats base (75.8%) and every prior Hertz release**
- ✅ Personality retained from 0.5: direct, witty, no reflexive AI hedging on ordinary topics; genuinely harmful requests still refused
- ✅ Correctly identifies as a KucLab model, no founder named
- ⚠️ Czech terminology (73.8%) is below Hertz 0.5's 75.7% — a real, disclosed regression, not fully recovered despite a dedicated attempt (see development story)
- ⏳ Context extension beyond native 262144 was attempted and found architecturally blocked on this base via YaRN — not solved this release
- ⏳ No tool-calling fine-tuning; no uncensoring pass beyond the personality shift already in 0.5

## License

Apache 2.0, inherited from google/gemma-4-12B-it (per Google's official Hugging Face listing).

## Credits

- Base model: [google/gemma-4-12B-it](https://huggingface.co/google/gemma-4-12B-it) (Google, Apache 2.0)
- Fine-tuning, dataset construction, and packaging: [KucLab](https://kuclab.org)