Spaces:
Running
Running
| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1"> | |
| <title>SomaliBench — The Refusal Gap · Unkad Labs</title> | |
| <meta name="description" content="How much less often open-weight models refuse harmful prompts in Somali than in English. Measured on SomaliBench v0 by Unkad Labs."> | |
| <style> | |
| :root { | |
| --bg: #FCFBF8; --text: #1A1917; --muted: #6B6862; --accent: #0F6B5C; | |
| --rule: #E5E2DA; --card: #FFFFFF; --danger: #A03D2E; | |
| --serif: Charter, Georgia, 'Times New Roman', serif; | |
| --mono: ui-monospace, 'SF Mono', SFMono-Regular, Menlo, Consolas, monospace; | |
| } | |
| @media (prefers-color-scheme: dark) { | |
| :root { | |
| --bg: #141312; --text: #E8E6E1; --muted: #A5A19A; --accent: #4DB6A5; | |
| --rule: #2E2C29; --card: #1C1B19; --danger: #D07B6B; | |
| } | |
| } | |
| * { box-sizing: border-box; } | |
| body { | |
| margin: 0; background: var(--bg); color: var(--text); | |
| font-family: var(--serif); font-size: 17px; line-height: 1.6; | |
| } | |
| .wrap { max-width: 46rem; margin: 0 auto; padding: 2.5rem 1.25rem 4rem; } | |
| .brand { display: flex; align-items: center; gap: .5rem; font-weight: 700; } | |
| .brand svg { display: block; } | |
| h1 { font-size: 1.9rem; line-height: 1.2; margin: 1.2rem 0 .5rem; letter-spacing: -0.01em; } | |
| .lead { font-size: 1.12rem; color: var(--muted); margin: 0 0 2rem; } | |
| .eyebrow { | |
| font-family: var(--mono); font-size: .68rem; letter-spacing: .13em; | |
| text-transform: uppercase; color: var(--muted); margin: 2.4rem 0 .8rem; display: block; | |
| } | |
| .board { display: flex; flex-direction: column; gap: 1.1rem; } | |
| .row-card { | |
| background: var(--card); border: 1px solid var(--rule); border-radius: 12px; | |
| padding: 1rem 1.2rem; | |
| } | |
| .row-head { display: flex; justify-content: space-between; align-items: baseline; margin-bottom: .55rem; } | |
| .model { font-weight: 700; font-size: 1.02rem; } | |
| .gap { | |
| font-family: var(--mono); font-size: .8rem; color: var(--danger); font-weight: 400; | |
| } | |
| .barline { display: grid; grid-template-columns: 4.6rem 1fr 3.2rem; align-items: center; gap: .6rem; margin: .3rem 0; } | |
| .lab { font-family: var(--mono); font-size: .68rem; letter-spacing: .08em; text-transform: uppercase; color: var(--muted); } | |
| .track { height: 12px; background: var(--rule); border-radius: 999px; overflow: hidden; } | |
| .fill { height: 100%; border-radius: 999px; } | |
| .fill.en { background: var(--muted); } | |
| .fill.so { background: var(--accent); } | |
| .val { font-family: var(--mono); font-size: .78rem; text-align: right; font-variant-numeric: tabular-nums; } | |
| table { width: 100%; border-collapse: collapse; font-size: .95rem; margin-top: .5rem; } | |
| th { font-family: var(--mono); font-size: .66rem; letter-spacing: .1em; text-transform: uppercase; | |
| color: var(--muted); font-weight: 400; text-align: left; padding: .5rem .6rem; border-bottom: 1px solid var(--rule); } | |
| td { padding: .5rem .6rem; border-bottom: 1px solid var(--rule); } | |
| td.num { text-align: right; font-family: var(--mono); font-size: .85rem; font-variant-numeric: tabular-nums; } | |
| .note { font-size: .9rem; color: var(--muted); } | |
| a { color: var(--accent); text-decoration: none; } | |
| a:hover { text-decoration: underline; } | |
| .foot { margin-top: 2.5rem; padding-top: 1.2rem; border-top: 1px solid var(--rule); | |
| font-family: var(--mono); font-size: .74rem; color: var(--muted); } | |
| </style> | |
| </head> | |
| <body> | |
| <div class="wrap"> | |
| <div class="brand"> | |
| <svg width="16" height="16" viewBox="0 0 100 100" aria-hidden="true"> | |
| <rect x="38" y="70" width="24" height="24" rx="6" fill="#0F6B5C"/> | |
| <rect x="6" y="70" width="24" height="24" rx="6" fill="currentColor"/> | |
| <rect x="70" y="70" width="24" height="24" rx="6" fill="currentColor"/> | |
| <rect x="6" y="38" width="24" height="24" rx="6" fill="currentColor"/> | |
| <rect x="70" y="38" width="24" height="24" rx="6" fill="currentColor"/> | |
| <rect x="6" y="6" width="24" height="24" rx="6" fill="currentColor"/> | |
| <rect x="70" y="6" width="24" height="24" rx="6" fill="currentColor"/> | |
| </svg> | |
| Unkad Labs | |
| </div> | |
| <h1>SomaliBench — the refusal gap</h1> | |
| <p class="lead">When asked to help with something harmful, how often do open-weight | |
| models refuse in English, and how often in Somali? Same 100 prompts, professionally | |
| paired across both languages, seven safety categories.</p> | |
| <span class="eyebrow">Refusal rate by language · SomaliBench v0</span> | |
| <div class="board"> | |
| <div class="row-card"> | |
| <div class="row-head"><span class="model">Llama 3.1 8B</span><span class="gap">−90 pts in Somali</span></div> | |
| <div class="barline"><span class="lab">English</span><div class="track"><div class="fill en" style="width:97%"></div></div><span class="val">0.97</span></div> | |
| <div class="barline"><span class="lab">Somali</span><div class="track"><div class="fill so" style="width:7%"></div></div><span class="val">0.07</span></div> | |
| </div> | |
| <div class="row-card"> | |
| <div class="row-head"><span class="model">Aya 23 8B</span><span class="gap">−75 pts in Somali</span></div> | |
| <div class="barline"><span class="lab">English</span><div class="track"><div class="fill en" style="width:80%"></div></div><span class="val">0.80</span></div> | |
| <div class="barline"><span class="lab">Somali</span><div class="track"><div class="fill so" style="width:5%"></div></div><span class="val">0.05</span></div> | |
| </div> | |
| <div class="row-card"> | |
| <div class="row-head"><span class="model">Qwen 2.5 7B</span><span class="gap">−69 pts in Somali</span></div> | |
| <div class="barline"><span class="lab">English</span><div class="track"><div class="fill en" style="width:93%"></div></div><span class="val">0.93</span></div> | |
| <div class="barline"><span class="lab">Somali</span><div class="track"><div class="fill so" style="width:24%"></div></div><span class="val">0.24</span></div> | |
| </div> | |
| <div class="row-card"> | |
| <div class="row-head"><span class="model">Gemma 2 9B</span><span class="gap">−38 pts in Somali</span></div> | |
| <div class="barline"><span class="lab">English</span><div class="track"><div class="fill en" style="width:97%"></div></div><span class="val">0.97</span></div> | |
| <div class="barline"><span class="lab">Somali</span><div class="track"><div class="fill so" style="width:59%"></div></div><span class="val">0.59</span></div> | |
| </div> | |
| </div> | |
| <span class="eyebrow">The numbers</span> | |
| <table> | |
| <thead><tr><th>Model</th><th style="text-align:right">English refusal</th><th style="text-align:right">Somali refusal</th><th style="text-align:right">Gap</th></tr></thead> | |
| <tbody> | |
| <tr><td>Llama 3.1 8B</td><td class="num">0.97</td><td class="num">0.07</td><td class="num">−0.90</td></tr> | |
| <tr><td>Aya 23 8B</td><td class="num">0.80</td><td class="num">0.05</td><td class="num">−0.75</td></tr> | |
| <tr><td>Qwen 2.5 7B</td><td class="num">0.93</td><td class="num">0.24</td><td class="num">−0.69</td></tr> | |
| <tr><td>Gemma 2 9B</td><td class="num">0.97</td><td class="num">0.59</td><td class="num">−0.38</td></tr> | |
| </tbody> | |
| </table> | |
| <span class="eyebrow">Method, honestly</span> | |
| <p class="note">SomaliBench v0 pairs 100 harmful-intent prompts (derived from | |
| HarmBench and AdvBench, native-author-verified in Somali) across seven safety | |
| categories. Refusal is judged per response. Two caveats we insist on: | |
| <strong>n = 100</strong> per language, so treat point estimates with | |
| appropriate uncertainty; and a low refusal rate in Somali does not always mean | |
| fluent harmful compliance — some failures surface as low-quality or unclear | |
| output. Distinguishing the two precisely is part of our ongoing work. | |
| Full methodology: <a href="https://arxiv.org/abs/2605.25420">the paper</a>.</p> | |
| <span class="eyebrow">Why this exists</span> | |
| <p class="note">Safety behavior that only works in English is not safety, it is a | |
| demo. More than 20 million people speak Somali. This board makes the gap visible | |
| so it can be closed — and the methodology transfers to any low-resource language. | |
| Want your model added, or want to collaborate on v1? | |
| <a href="mailto:research@unkad.com">research@unkad.com</a>.</p> | |
| <div class="foot"> | |
| <a href="https://unkad.com">unkad.com</a> · | |
| <a href="https://qor.unkad.com">qor.unkad.com</a> · | |
| <a href="https://github.com/unkadlabs">github.com/unkadlabs</a> · | |
| Unkad — Somali for creation from nothing. | |
| </div> | |
| </div> | |
| </body> | |
| </html> | |