Spaces:
Build error
Build error
File size: 5,687 Bytes
801a2fb | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 | [
{
"title": "Indirect abuse: baseline misses, A and B catch",
"language": "English",
"text": "I didn't know the right wing loved islam",
"true_label": "harmful",
"sarcasm_score": "0.75",
"source": "cad",
"models": [
{
"name": "Baseline",
"prob": "0.45",
"decision": "non-harmful (MISS)"
},
{
"name": "Mechanism A",
"prob": "--",
"decision": "HARMFUL"
},
{
"name": "Mechanism B",
"prob": "0.61",
"decision": "HARMFUL"
},
{
"name": "Mechanism D",
"prob": "0.60",
"decision": "HARMFUL"
}
],
"story": "CAD is where the mechanisms' recall gains concentrate (baseline recall 0.389 -> B 0.593, p<1e-5): contextual, indirect abuse benefits most from the auxiliary non-literality signal."
},
{
"title": "Tunisian dialect: the worry that turned out backwards",
"language": "Arabic",
"text": "اتقوا انفسكم ياتوانسه",
"true_label": "harmful",
"sarcasm_score": "0.21",
"source": "thsab",
"models": [
{
"name": "Baseline",
"prob": "0.26",
"decision": "non-harmful (MISS)"
},
{
"name": "Mechanism A",
"prob": "--",
"decision": "HARMFUL"
},
{
"name": "Mechanism B",
"prob": "0.80",
"decision": "HARMFUL"
},
{
"name": "Mechanism D",
"prob": "0.18",
"decision": "non-harmful (MISS)"
}
],
"story": "Despite zero Tunisian data in the sarcasm layer's training pool, A and B are significantly effective on T-HSAB specifically (McNemar p=0.015 / p=0.029) -- the suspected weak point was verified to be a strength."
},
{
"title": "Sarcastic camouflage: high sarcasm score, hostile intent",
"language": "Arabic",
"text": "يلعن صباحك نزعتلي نهاري",
"true_label": "harmful",
"sarcasm_score": "0.85",
"source": "lhsab",
"models": [
{
"name": "Baseline",
"prob": "0.03",
"decision": "non-harmful (MISS)"
},
{
"name": "Mechanism A",
"prob": "--",
"decision": "HARMFUL"
},
{
"name": "Mechanism B",
"prob": "0.98",
"decision": "HARMFUL"
},
{
"name": "Mechanism D",
"prob": "0.93",
"decision": "HARMFUL"
}
],
"story": "The motivating case of the thesis: superficially non-hostile or joking wording carrying real hostility, recovered when the sarcasm signal is available."
},
{
"title": "The honest trade-off: a false positive the mechanisms introduce",
"language": "English",
"text": "@user “ lets go gays ”",
"true_label": "non-harmful",
"sarcasm_score": "0.48",
"source": "hatexplain",
"models": [
{
"name": "Baseline",
"prob": "0.43",
"decision": "non-harmful"
},
{
"name": "Mechanism A",
"prob": "--",
"decision": "HARMFUL (false positive)"
},
{
"name": "Mechanism B",
"prob": "0.77",
"decision": "HARMFUL (false positive)"
},
{
"name": "Mechanism D",
"prob": "0.51",
"decision": "HARMFUL (false positive)"
}
],
"story": "Every mechanism trades precision for recall. Showing a failure case demonstrates the evaluation was honest: the operating point shifts toward recall; the mechanisms do not dominate the baseline on both axes."
},
{
"title": "Mechanism D's cross-lingual cost: Arabic harmful D misses",
"language": "Arabic",
"text": "صحيح انك انسان وضيع",
"true_label": "harmful",
"sarcasm_score": "0.83",
"source": "lhsab",
"models": [
{
"name": "Baseline",
"prob": "0.99",
"decision": "HARMFUL"
},
{
"name": "Mechanism A",
"prob": "--",
"decision": "HARMFUL"
},
{
"name": "Mechanism B",
"prob": "0.98",
"decision": "HARMFUL"
},
{
"name": "Mechanism D",
"prob": "0.05",
"decision": "non-harmful (MISS)"
}
],
"story": "Mechanism D helps English but degrades Arabic (recall 0.8155 -> 0.7729, p=0.0009) -- a cost the constant-score ablation showed is independent of score values, and which generalized to French in zero-shot transfer (recall collapse to 0.15-0.18)."
},
{
"title": "Mechanism D2: what reasoning injection looks like (pilot)",
"language": "Arabic",
"text": "صحيح انك انسان وضيع",
"true_label": "harmful",
"sarcasm_score": "0.83",
"source": "lhsab",
"models": [
{
"name": "Matched baseline",
"prob": "--",
"decision": "non-harmful (MISS)"
},
{
"name": "Matched D (score)",
"prob": "--",
"decision": "HARMFUL"
},
{
"name": "D2 (reasoning)",
"prob": "--",
"decision": "HARMFUL"
}
],
"story": "Generated reasoning for this example: “Literal meaning: It is true that you are a worthless person. Implied meaning: The speaker is insulting the addressee's worthiness and dignity. Speaker intention: To demean and criticize the addressee's value and status. Indirect language: No -- the statement is made directly without any implied irony or sarcasm.” — D2 significantly beats its matched baseline (p=0.0019, replicated with a second generator, p=0.0038) but does not exceed bare-score injection: the channel, not the content, is the binding constraint."
}
] |