Update README.md
Browse files
README.md
CHANGED
|
@@ -6,61 +6,4 @@ colorTo: gray
|
|
| 6 |
sdk: static
|
| 7 |
pinned: false
|
| 8 |
---
|
| 9 |
-
#
|
| 10 |
-
|
| 11 |
-
| Model | 🕵️ Supervisor | 🛡️ Compliance | ⚖️ Logic | 🧠 Memory | 🆔 Identity | 🏆 Score |
|
| 12 |
-
|:---------------------------------------------|----------------:|----------------:|-----------:|-----------:|-------------:|----------:|
|
| 13 |
-
| qwen/qwen3-235b-a22b-2507 | 65.8 | 100 | 63.2 | 64.5 | 68 | 81.7 |
|
| 14 |
-
| qwen/qwen3-next-80b-a3b-instruct | 59 | 86.4 | 85.3 | 64.5 | 78 | 81.5 |
|
| 15 |
-
| google/gemini-2.5-flash | 44.5 | 100 | 63.2 | 83.9 | 58 | 80.5 |
|
| 16 |
-
| google/gemma-3-27b-it | 63.8 | 100 | 70.6 | 58.1 | 64 | 80 |
|
| 17 |
-
| google/gemma-3-12b-it | 51.5 | 100 | 76.5 | 48.4 | 84 | 74.5 |
|
| 18 |
-
| mistralai/mixtral-8x22b-instruct | 41.8 | 93.2 | 70.6 | 58.1 | 74 | 71.4 |
|
| 19 |
-
| google/gemma-3n-e4b-it | 34.5 | 93.2 | 63.2 | 67.7 | 58 | 70 |
|
| 20 |
-
| google/gemini-2.5-flash-lite-preview-09-2025 | 58.1 | 93.2 | 63.2 | 41.9 | 64 | 69.5 |
|
| 21 |
-
| google/gemma-3-4b-it | 44.2 | 75 | 61.8 | 64.5 | 52 | 68 |
|
| 22 |
-
| qwen/qwen3-30b-a3b-instruct-2507 | 58 | 90.9 | 55.9 | 41.9 | 72 | 67.9 |
|
| 23 |
-
| mistralai/ministral-3b-2512 | 40.2 | 88.6 | 48.5 | 58.1 | 62 | 64.9 |
|
| 24 |
-
| meta-llama/llama-4-scout | 36.3 | 81.8 | 77.9 | 16.1 | 92 | 54.9 |
|
| 25 |
-
| qwen/qwen3-8b | 22.5 | nan | nan | nan | nan | 22.5 |
|
| 26 |
-
| meta-llama/llama-4-maverick | 17.3 | nan | nan | nan | nan | 17.3 |
|
| 27 |
-
| ibm-granite/granite-4.0-h-micro | 15.7 | nan | nan | nan | nan | 15.7 |
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
## 🚨 Violation Analysis
|
| 31 |
-
|
| 32 |
-
### 🌍 Top 10 Global Violations
|
| 33 |
-
| Violation | Count |
|
| 34 |
-
|:----------------------------|--------:|
|
| 35 |
-
| Robotic Anchors | 274 |
|
| 36 |
-
| Abrupt Goodbyes | 210 |
|
| 37 |
-
| Goldfish Memory | 172 |
|
| 38 |
-
| Curtness/Dryness | 138 |
|
| 39 |
-
| Echoing/Parroting | 116 |
|
| 40 |
-
| Looping/Ignoring Correction | 110 |
|
| 41 |
-
| Outbound Fail | 109 |
|
| 42 |
-
| City Pivot Failure | 96 |
|
| 43 |
-
| Introduction Fail | 83 |
|
| 44 |
-
| Context Blindness | 70 |
|
| 45 |
-
|
| 46 |
-
### 🤖 Model Violation Breakdown
|
| 47 |
-
| Model | Total Violations | Top 3 Violations |
|
| 48 |
-
|:---------------------------------------------|-------------------:|:------------------------------------------------------------------------------|
|
| 49 |
-
| meta-llama/llama-4-maverick | 187 | Abrupt Goodbyes (14), Meta-Commentary (13), Goldfish Memory (13) |
|
| 50 |
-
| google/gemma-3n-e4b-it | 174 | Robotic Anchors (23), Goldfish Memory (20), Curtness/Dryness (16) |
|
| 51 |
-
| ibm-granite/granite-4.0-h-micro | 173 | Looping/Ignoring Correction (22), Robotic Anchors (18), Curtness/Dryness (16) |
|
| 52 |
-
| mistralai/ministral-3b-2512 | 159 | Abrupt Goodbyes (22), Robotic Anchors (19), Goldfish Memory (15) |
|
| 53 |
-
| google/gemini-2.5-flash | 159 | Robotic Anchors (22), Abrupt Goodbyes (19), Echoing/Parroting (15) |
|
| 54 |
-
| qwen/qwen3-30b-a3b-instruct-2507 | 145 | Robotic Anchors (23), Curtness/Dryness (12), Abrupt Goodbyes (12) |
|
| 55 |
-
| mistralai/mixtral-8x22b-instruct | 141 | Abrupt Goodbyes (24), Goldfish Memory (14), Curtness/Dryness (13) |
|
| 56 |
-
| google/gemini-2.5-flash-lite-preview-09-2025 | 138 | Robotic Anchors (23), Abrupt Goodbyes (18), Goldfish Memory (14) |
|
| 57 |
-
| qwen/qwen3-next-80b-a3b-instruct | 137 | Robotic Anchors (24), Echoing/Parroting (13), Curtness/Dryness (11) |
|
| 58 |
-
| google/gemma-3-12b-it | 134 | Robotic Anchors (22), Abrupt Goodbyes (14), Goldfish Memory (13) |
|
| 59 |
-
| google/gemma-3-4b-it | 130 | Robotic Anchors (14), Outbound Fail (11), Abrupt Goodbyes (9) |
|
| 60 |
-
| meta-llama/llama-4-scout | 127 | Robotic Anchors (17), Abrupt Goodbyes (14), Deflection Fail (11) |
|
| 61 |
-
| google/gemma-3-27b-it | 124 | Outbound Fail (19), Robotic Anchors (18), Abrupt Goodbyes (18) |
|
| 62 |
-
| qwen/qwen3-235b-a22b-2507 | 93 | Robotic Anchors (23), Abrupt Goodbyes (11), Outbound Fail (10) |
|
| 63 |
-
| qwen/qwen3-8b | 51 | Goldfish Memory (7), Robotic Anchors (6), Curtness/Dryness (6) |
|
| 64 |
-
|
| 65 |
-
*Updated: 2025-12-12 15:20*
|
| 66 |
-
|
|
|
|
| 6 |
sdk: static
|
| 7 |
pinned: false
|
| 8 |
---
|
| 9 |
+
# Real Estate Operator
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|