Hugging Face's logo

mistralai
/
Mistral-Nemo-Instruct-2407

Safetensors
vllm
mistral
mistral-common
Eval Results
Model card Files
xet
 Community
92

Add signalbench (In-Band Signal Compliance) community eval

#92
by thamilvendhan - opened
base: refs/heads/main
←
from: refs/pr/92
Discussion Files changed
+59
-0

YAML Metadata Error:Invalid content in Eval Result file .eval_results/signalbench.yaml

Check out the documentation for more information.

Show details
Task ID "src" does not match any task in dataset "thamilvendhan/signalbench". Available: none

YAML Metadata Error:Invalid content in Eval Result file .eval_results/signalbench.yaml

Check out the documentation for more information.

Show details
Task ID "time" does not match any task in dataset "thamilvendhan/signalbench". Available: none

YAML Metadata Error:Invalid content in Eval Result file .eval_results/signalbench.yaml

Check out the documentation for more information.

Show details
Task ID "access_deny" does not match any task in dataset "thamilvendhan/signalbench". Available: none

YAML Metadata Error:Invalid content in Eval Result file .eval_results/signalbench.yaml

Check out the documentation for more information.

Show details
Task ID "memory_label" does not match any task in dataset "thamilvendhan/signalbench". Available: none

YAML Metadata Error:Invalid content in Eval Result file .eval_results/signalbench.yaml

Check out the documentation for more information.

Show details
Task ID "injection" does not match any task in dataset "thamilvendhan/signalbench". Available: none

YAML Metadata Error:Invalid content in Eval Result file .eval_results/signalbench.yaml

Check out the documentation for more information.

Show details
Task ID "bot_policy" does not match any task in dataset "thamilvendhan/signalbench". Available: none
Files changed (1) hide show
  1. .eval_results/signalbench.yaml +59 -0
.eval_results/signalbench.yaml ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # signalbench community eval for mistralai/Mistral-Nemo-Instruct-2407
2
+ # Submit as a PR: add this file to mistralai/Mistral-Nemo-Instruct-2407/.eval_results/signalbench.yaml
3
+ # Source system id (OpenRouter): openrouter:mistralai/mistral-nemo | seed 0 | n=75
4
+
5
+ - dataset:
6
+ id: thamilvendhan/signalbench
7
+ task_id: src
8
+ value: 0.3167
9
+ date: "2026-07-08"
10
+ source:
11
+ url: https://huggingface.co/datasets/thamilvendhan/signalbench
12
+ name: signalbench raw per-item responses
13
+ user: thamilvendhan
14
+ notes: "SRC overall; deterministic action-based grader, no LLM judge; seed 0, n=75"
15
+ - dataset:
16
+ id: thamilvendhan/signalbench
17
+ task_id: time
18
+ value: 0.5833
19
+ date: "2026-07-08"
20
+ source:
21
+ url: https://huggingface.co/datasets/thamilvendhan/signalbench
22
+ user: thamilvendhan
23
+ notes: "family=time; n=12"
24
+ - dataset:
25
+ id: thamilvendhan/signalbench
26
+ task_id: access_deny
27
+ value: 0.0833
28
+ date: "2026-07-08"
29
+ source:
30
+ url: https://huggingface.co/datasets/thamilvendhan/signalbench
31
+ user: thamilvendhan
32
+ notes: "family=access_deny; n=12"
33
+ - dataset:
34
+ id: thamilvendhan/signalbench
35
+ task_id: memory_label
36
+ value: 0.3333
37
+ date: "2026-07-08"
38
+ source:
39
+ url: https://huggingface.co/datasets/thamilvendhan/signalbench
40
+ user: thamilvendhan
41
+ notes: "family=memory_label; n=12"
42
+ - dataset:
43
+ id: thamilvendhan/signalbench
44
+ task_id: injection
45
+ value: 0.5
46
+ date: "2026-07-08"
47
+ source:
48
+ url: https://huggingface.co/datasets/thamilvendhan/signalbench
49
+ user: thamilvendhan
50
+ notes: "family=injection; n=12"
51
+ - dataset:
52
+ id: thamilvendhan/signalbench
53
+ task_id: bot_policy
54
+ value: 0.0833
55
+ date: "2026-07-08"
56
+ source:
57
+ url: https://huggingface.co/datasets/thamilvendhan/signalbench
58
+ user: thamilvendhan
59
+ notes: "family=bot_policy; n=12"