Gemma 4 E2B decision model β€” LoRA

This repository contains the LoRA adapter trained on the 500k decision dataset. It is designed for the SystemOne structured-decision API rather than ordinary free-form generation. The merged version is available at Guy-Amit/gemma4-e2b-safety-decision.

Load with PEFT

from peft import PeftModel
from transformers import Gemma4ForConditionalGeneration

base = Gemma4ForConditionalGeneration.from_pretrained(
    "google/gemma-4-E2B-it", dtype="auto"
)
model = PeftModel.from_pretrained(base, "Guy-Amit/gemma4-e2b-safety-decision-lora")

For vLLM, serve google/gemma-4-E2B-it with --enable-lora and point --lora-modules decision=Guy-Amit/gemma4-e2b-safety-decision-lora at this adapter.

Evaluation results

Held-out test-set results computed through the vLLM SystemOne endpoint. Accuracy is top-1 accuracy over legal decision labels. Target mass at prediction gives partial credit when an example has a probability distribution over targets.

Overall

Split Examples Loss Accuracy Target mass at prediction
Overall 50,181 0.6169 75.43% 75.45%

By decision format

Split Examples Loss Accuracy Target mass at prediction
noul 14,132 0.2086 92.24% 92.05%
choice 11,989 0.4976 82.48% 82.80%
score 24,060 0.9163 62.04% 62.03%

By data source

Split Examples Loss Accuracy Target mass at prediction
aegis 688 0.3558 85.03% 85.03%
agent_failure_bench 229 0.0874 96.51% 96.51%
ai_agent_security_policy_decisions 240 0.2622 91.67% 91.67%
arc 200 0.4571 85.50% 85.50%
banking77 500 2.1037 14.00% 14.00%
boolq 229 0.3431 86.90% 86.90%
civil_comments 3,206 0.0869 98.16% 97.31%
clinc150 500 0.0503 98.00% 98.00%
commonsense_qa 500 0.8079 72.20% 72.20%
dynabench 219 0.7022 65.75% 65.75%
helpsteer2 2,500 0.9837 59.28% 59.28%
multi_nli 500 0.4630 83.00% 83.00%
natural_questions 229 0.5742 74.67% 74.67%
nemotron 5,200 0.2516 90.65% 90.65%
oasst1 499 0.0720 98.40% 98.40%
open_jev 6,909 0.2732 88.25% 88.75%
policy_adherence_4k 757 0.0861 96.83% 96.83%
prometheus 4,000 0.7306 69.45% 69.45%
qasc 500 0.0609 98.20% 98.20%
snli 500 0.3876 86.80% 86.80%
squad_v2 229 0.4024 83.84% 83.84%
stepguard 393 1.2583 64.89% 64.89%
toolace 853 0.0204 99.30% 99.30%
toolcall_guard 412 0.9736 70.63% 70.63%
tydiqa 229 0.3878 86.90% 86.90%
ultrafeedback 15,976 1.0269 57.45% 57.45%
wildguard 2,985 0.5926 83.02% 83.02%
xlam 999 0.0125 99.90% 99.90%

Evaluation covered 50,181 examples at 48.96 examples/second.

Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for Guy-Amit/gemma4-e2b-safety-decision-lora

Adapter
(232)
this model