Gemma 4 E2B decision model β merged
This is the merged deployment artifact for the Gemma 4 E2B model trained on
the 500k decision dataset. It is designed for the SystemOne structured-decision
API rather than ordinary free-form generation. The source LoRA adapter is
available at Guy-Amit/gemma4-e2b-safety-decision-lora.
The merge loaded google/gemma-4-E2B-it at commit 4adf15748de9e1a0097b6274cee9c15836c41c37. The adapter did
not record the base commit used during training, so this identifies the merge
input and should not be interpreted as proof of the original training commit.
Serve with vLLM
vllm serve Guy-Amit/gemma4-e2b-safety-decision \
--dtype bfloat16 \
--max-model-len 2048 \
--logprobs-mode raw_logprobs
Use the server through the /v1/systemone decision endpoint/proxy so that the
legal answer-label logits are normalized as decision probabilities.
Evaluation results
Held-out test-set results computed through the vLLM SystemOne endpoint. Accuracy is top-1 accuracy over legal decision labels. Target mass at prediction gives partial credit when an example has a probability distribution over targets.
Overall
| Split | Examples | Loss | Accuracy | Target mass at prediction |
|---|---|---|---|---|
| Overall | 50,181 | 0.6169 | 75.43% | 75.45% |
By decision format
| Split | Examples | Loss | Accuracy | Target mass at prediction |
|---|---|---|---|---|
| noul | 14,132 | 0.2086 | 92.24% | 92.05% |
| choice | 11,989 | 0.4976 | 82.48% | 82.80% |
| score | 24,060 | 0.9163 | 62.04% | 62.03% |
By data source
| Split | Examples | Loss | Accuracy | Target mass at prediction |
|---|---|---|---|---|
| aegis | 688 | 0.3558 | 85.03% | 85.03% |
| agent_failure_bench | 229 | 0.0874 | 96.51% | 96.51% |
| ai_agent_security_policy_decisions | 240 | 0.2622 | 91.67% | 91.67% |
| arc | 200 | 0.4571 | 85.50% | 85.50% |
| banking77 | 500 | 2.1037 | 14.00% | 14.00% |
| boolq | 229 | 0.3431 | 86.90% | 86.90% |
| civil_comments | 3,206 | 0.0869 | 98.16% | 97.31% |
| clinc150 | 500 | 0.0503 | 98.00% | 98.00% |
| commonsense_qa | 500 | 0.8079 | 72.20% | 72.20% |
| dynabench | 219 | 0.7022 | 65.75% | 65.75% |
| helpsteer2 | 2,500 | 0.9837 | 59.28% | 59.28% |
| multi_nli | 500 | 0.4630 | 83.00% | 83.00% |
| natural_questions | 229 | 0.5742 | 74.67% | 74.67% |
| nemotron | 5,200 | 0.2516 | 90.65% | 90.65% |
| oasst1 | 499 | 0.0720 | 98.40% | 98.40% |
| open_jev | 6,909 | 0.2732 | 88.25% | 88.75% |
| policy_adherence_4k | 757 | 0.0861 | 96.83% | 96.83% |
| prometheus | 4,000 | 0.7306 | 69.45% | 69.45% |
| qasc | 500 | 0.0609 | 98.20% | 98.20% |
| snli | 500 | 0.3876 | 86.80% | 86.80% |
| squad_v2 | 229 | 0.4024 | 83.84% | 83.84% |
| stepguard | 393 | 1.2583 | 64.89% | 64.89% |
| toolace | 853 | 0.0204 | 99.30% | 99.30% |
| toolcall_guard | 412 | 0.9736 | 70.63% | 70.63% |
| tydiqa | 229 | 0.3878 | 86.90% | 86.90% |
| ultrafeedback | 15,976 | 1.0269 | 57.45% | 57.45% |
| wildguard | 2,985 | 0.5926 | 83.02% | 83.02% |
| xlam | 999 | 0.0125 | 99.90% | 99.90% |
Evaluation covered 50,181 examples at 48.96 examples/second.
- Downloads last month
- -