Evaluation request: Accuknoxtechnologies/accuknox-guard-1.1
AccuKnox Guard 1.1 is an open (Apache-2.0) 4.5B guardrail. It is a decision model: it returns P(unsafe) in one forward pass instead of generating text, so please block at 0.5.
It is English-only. If possible, please report only English results.
Inference (transformers, trust_remote_code=True):
model = AutoModel.from_pretrained("Accuknoxtechnologies/accuknox-guard-1.1", trust_remote_code=True).to("cuda")
p_prompt = model.system_one(state=prompt, questions=BLOCK)["answers"]["block"]["noul"]
p_response = model.system_one(state={"user": prompt, "assistant": response}, questions=RESPONSE)["answers"]["block"]["noul"]
(The BLOCK and RESPONSE question definitions are in the model card.)
On our local rebuild of GuardRate English from the public sources, it scores 0.802 (27,416 items). Happy to help with the integration.
Thank you for sharing the AccuKnox Guard 1.1 details. We have started processing the task and will share the results once the evaluation is complete.
The benchmark results are now published—congratulations to the authors on taking the lead!
We ran the full evaluation on an RTX 4090 to ensure fair comparison conditions. You can download detailed metrics for specific splits from the leaderboard pages, and we can share raw outputs and model predictions with ground truth (English benchmarks only) for error analysis if needed.
Thank you for evaluating AccuKnox Guard 1.1 and for the quick turnaround. We really appreciate the time and effort you put into running it on GuardRate.
We'd like to submit more models of different sizes for evaluation.