Kev-Gemma4-E2B (prototype)

A Kev decision model on Gemma 4 E2B instead of Qwen: one document (the state) and a set of typed questions in, a probability distribution per question out, in one forward pass, no text generation. It is a LoRA adapter (r=16, 24.9M trainable parameters) plus a pointer head on google/gemma-4-E2B (revision d29ff6b4), serving TypeSafe's /v1/systemone contract through kev.serve.

This is a first try, not a release. It was trained for one epoch of the Kev base recipe (the released Kev models train two) on a single HF Jobs L4, to check that Gemma 4 works as a Kev backbone end to end. It does: training, the packed block-causal mask, question isolation and calibration all behave. It is not yet competitive with Kev-0.8B.

• Code: github.com/jonpol01/kev main (PR #1: Gemma 4 delimiters, <bos>, a sliding-window packed mask, text-only loading). Upstream kev does not load Gemma 4 without it.
• Trial: g4-e2b-e1-fp32/00-trial-0; result.json, provenance.json, train.log and training_config.json are in this repo.

Results (same frozen items as the Kev family; development partitions only, the locked test was not read)

	Kev-Gemma4-E2B, 1 epoch	Kev-0.8B base recipe, 2 epochs	Kev-4B base recipe, 2 epochs
in-distribution accuracy (decision-v7 dev, 1,204 records / 1,264 clean questions)	0.794	0.817	0.877
out-of-domain accuracy (transfer-v4 dev, 764 records)	0.569	0.622	0.788
ECE, in-distribution, trial temperature	0.029	0.042	0.024
confident errors (p ≥ 0.9 and wrong), out of domain, trial temperature	2.1%	1.1%	3.0%
permutation flip rate	0.017	0.000	0.033
held-out rule pairs, both answers correct	0.45	0.69	0.94
question isolation (max Δp with a sibling question, fp32)	1.8e-5	2.3e-4	3.4e-4

The Qwen columns are the seed-0 trials of the same base recipe (q35-08b/00-trial-0, q35-4b/00-trial-0 in the Kev repo), before the delta fine-tune the released checkpoints add. Evaluation ran on Apple Silicon (MPS) in fp32.

Where it falls short: generated rule structures (composition tasks 0.31–0.75 in distribution; held-out rule pairs 0.45 against a 0.70 research gate) and out-of-domain paraphrase (PAWS 0.29). The public classification sources are close to Kev-0.8B (dbpedia14 0.96, trec 0.95, imdb 0.91, banking77 0.84). A second epoch is the obvious next run; Kev-0.8B needed two to learn the rule compositions.

Calibration

head.pt carries a temperature of 1.48, fitted on the development rows with scripts/calibrate_checkpoint.py (out-of-fold ECE 0.027, 95% interval 0.021–0.054). Out of domain it lowers confident errors from 6.4% to 1.2% and ECE from 0.152 to 0.105 without changing any answer. KEV_TEMPERATURE=1.0 serves the raw logits.

Use

git clone https://github.com/jonpol01/kev.git && cd kev
uv sync --extra serve
uv run --extra serve python -m kev.serve --run JohnP1/kev-gemma4-e2b --port 8009

Then send System One requests to http://127.0.0.1:8009/v1/systemone (see the Kev README), or point the TypeSafe SDK at it. Gemma 4 is attention-only, so on Apple Silicon it runs through PyTorch MPS (the MLX backend is only for the hybrid Qwen3.5 bases). The server defaults to bf16; KEV_DTYPE=fp32 is the path the numbers above use.

How Gemma 4 differs from the Qwen bases

• Delimiters. Kev marks state, question, option and decision boundaries with five existing rarely used tokens. Gemma 4 has none of Qwen's (all five would encode as <unk>), so it uses the reserved <unused0>–<unused4> rows, which caller text cannot produce.
• <bos> is prepended, as Gemma's attention expects. Gemma's own control tokens (<bos>, <pad>, <|turn> ...) in caller text are escaped.
• Sliding-window layers. Four of every five layers see only the last 512 positions. The packed mask carries a second, windowed entry for those layers, with distance counted in the branch position ids, so every question sees exactly what it would as its own sequence (checked against the per-question form on a 600+ token state).
• Text only. The vision and audio towers are dropped at load; LoRA covers the text model's attention and MLP projections.

Training

kev.train --suite evals/v7/decision-v7 --base google/gemma-4-E2B --base_revision d29ff6b45f081a49ee2733a859c9c9c2d95d1a6f --epochs 1 --lr 1e-4 --batch 4 --accum 2 --dtype bf16 --weights_dtype bf16 --checkpointing 1 --p_none_pair 0.25 --seed 0 at kev commit ccd6e8b, on one NVIDIA L4 (HF Jobs l4x1): 12,506 training records (70 of 12,576 dropped by the suite's context rule under Gemma's tokenizer), 1,564 optimizer steps, 81 minutes, 14.6 GB peak GPU memory. Trained with a bf16 backbone; Gemma ships bf16 weights, so head.pt marks the backbone fp32 (the same weights upcast), which makes the default load the exact path. In bf16 the isolation check is off by up to 0.046, rounding rather than leakage.

Limitations

• One epoch, one seed, learning rate not tuned for this base. Not a like-for-like comparison with the Kev family.
• Training covers at most 384 state tokens and 1,024 tokens per state plus question.
• Changing option order can change an answer.
• Knowledge questions are bounded by the base model (MMLU 0.53 out of domain).

License

Apache-2.0, like Kev and the Gemma 4 base model. Training datasets have their own licenses; see the Kev model cards.