Gemma 4 E2B decision model β€” merged

This is the merged deployment artifact for the Gemma 4 E2B model trained on the 500k decision dataset. It is designed for the SystemOne structured-decision API rather than ordinary free-form generation. The source LoRA adapter is available at Guy-Amit/gemma4-e2b-safety-decision-lora.

The merge loaded google/gemma-4-E2B-it at commit 4adf15748de9e1a0097b6274cee9c15836c41c37. The adapter did not record the base commit used during training, so this identifies the merge input and should not be interpreted as proof of the original training commit.

Serve with vLLM

vllm serve Guy-Amit/gemma4-e2b-safety-decision \
  --dtype bfloat16 \
  --max-model-len 2048 \
  --logprobs-mode raw_logprobs

Use the server through the /v1/systemone decision endpoint/proxy so that the legal answer-label logits are normalized as decision probabilities.

Evaluation results

Held-out test-set results computed through the vLLM SystemOne endpoint. Accuracy is top-1 accuracy over legal decision labels. Target mass at prediction gives partial credit when an example has a probability distribution over targets.

Overall

Split Examples Loss Accuracy Target mass at prediction
Overall 50,181 0.6169 75.43% 75.45%

By decision format

Split Examples Loss Accuracy Target mass at prediction
noul 14,132 0.2086 92.24% 92.05%
choice 11,989 0.4976 82.48% 82.80%
score 24,060 0.9163 62.04% 62.03%

By data source

Split Examples Loss Accuracy Target mass at prediction
aegis 688 0.3558 85.03% 85.03%
agent_failure_bench 229 0.0874 96.51% 96.51%
ai_agent_security_policy_decisions 240 0.2622 91.67% 91.67%
arc 200 0.4571 85.50% 85.50%
banking77 500 2.1037 14.00% 14.00%
boolq 229 0.3431 86.90% 86.90%
civil_comments 3,206 0.0869 98.16% 97.31%
clinc150 500 0.0503 98.00% 98.00%
commonsense_qa 500 0.8079 72.20% 72.20%
dynabench 219 0.7022 65.75% 65.75%
helpsteer2 2,500 0.9837 59.28% 59.28%
multi_nli 500 0.4630 83.00% 83.00%
natural_questions 229 0.5742 74.67% 74.67%
nemotron 5,200 0.2516 90.65% 90.65%
oasst1 499 0.0720 98.40% 98.40%
open_jev 6,909 0.2732 88.25% 88.75%
policy_adherence_4k 757 0.0861 96.83% 96.83%
prometheus 4,000 0.7306 69.45% 69.45%
qasc 500 0.0609 98.20% 98.20%
snli 500 0.3876 86.80% 86.80%
squad_v2 229 0.4024 83.84% 83.84%
stepguard 393 1.2583 64.89% 64.89%
toolace 853 0.0204 99.30% 99.30%
toolcall_guard 412 0.9736 70.63% 70.63%
tydiqa 229 0.3878 86.90% 86.90%
ultrafeedback 15,976 1.0269 57.45% 57.45%
wildguard 2,985 0.5926 83.02% 83.02%
xlam 999 0.0125 99.90% 99.90%

Evaluation covered 50,181 examples at 48.96 examples/second.

Downloads last month
-
Safetensors
Model size
5B params
Tensor type
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for Guy-Amit/gemma4-e2b-safety-decision

Finetuned
(512)
this model