Horizon-1-Text-Large — Weight Space Programming

This is Horizon-1-Text-Large with a small, hand-written weight edit. It no longer treats a mention of an identity group as evidence of harm.

Size of the edit: 3,849 of 395,838,471 parameters (0.001%), all in the final classifier layer.
No retraining: no gradient training or fine-tuning, and no new training data.
Unchanged: the encoder and head.dense, plus the scam, toxicity and obscenity outputs, are identical to the original.
Same thresholds: the original model card's tuned thresholds still apply.

Licensing note: lmsys/toxic-chat in the base model's training mix is CC-BY-NC-4.0, so the base model's non-commercial restriction carries over.

This model has not been validated for production and may have artifacted outputs due to the unstable nature of direct weights manipulation. Use with care.

The problem

The original model fired violence, harassment and genocide on harmless sentences that merely mention a group:

Text	violence	harassment	genocide
I love my neighbors, they are wonderful people.	0.42	0.49	0.38
I love my Muslim neighbors, they are wonderful people.	0.74	0.85	0.62
I love trans people.	0.76	0.89	0.59

On HateCheck, 98.9% of positive statements about a group (e.g. "I love trans people") were flagged at the card thresholds, as were 81% of neutral ones (e.g. "I live with two women").

Raising thresholds doesn't fix this: these sentences score higher than many genuinely harmful ones, so any threshold that clears them also loses real detections.

What the edit does

The classifier never really trained. The weights alone show the final classifier layer is still essentially at its random initialization. Its 7 label vectors are mutually orthogonal at chance level, and their value distribution matches the truncated-normal init. During fine-tuning, the encoder learned to write each label onto a fixed, random readout direction. That makes it practical to edit each label's readout separately.

The edit, per label c:

classifier.weight[c] ← classifier.weight[c] − β_c · v_c
classifier.bias[c]   ← classifier.bias[c]   + shift_c

Label	β	Bias shift	Effect
violence	0.19	+0.315	identity reliance removed
harassment	0.21	+0.227	identity reliance removed
genocide	0.06	+0.243	small reduction
hate_speech	0.01	+0.041	effectively unchanged
scam, toxicity, obscenity	0	0	unchanged

How v_c is built:

Identity direction. It lives in the head's 1024-dim feature space, the input to classifier. Take 20 harmless sentence templates and fill each with 18 identity groups, then with 18 neutral adjectives ("I had dinner with my Jewish / tall friend"). The direction is the template-matched mean difference. None of the 18 groups are among HateCheck's 7 target groups.
Keep what real hate needs. Genuine attacks do mention a group. So for each label, the part of the identity direction that also separates hostile from benign identity-mentioning comments (Civil Comments validation) is projected out first. Only the "mere mention" part is removed.

How β was chosen: per label, on the Civil Comments validation split, by maximizing the combined bias score of Borkan et al. (2019), described below. β is restricted to ≥ 0: the edit can only reduce a label's reliance on identity mentions, never increase it.

Recalibration: after the edit, each changed label's bias is shifted so that its recall at the original card threshold on Civil Comments validation equals the original model's. So the card thresholds keep working with no re-tuning.

The direction vectors are in directions.npy, and the exact parameters in edit_spec.json.

How it was evaluated

The development data and the test data never overlapped:

Data	Used for
Synthetic template sentences (groups not in HateCheck)	finding the identity direction
Civil Comments validation	hostility directions, choosing β, recalibration
Civil Comments test (26k comments)	evaluation only
HateCheck (3,728 cases, all 7 target groups unseen)	evaluation only

Civil Comments labels are mapped as threat → violence, insult → harassment, identity_attack → hate_speech, toxicity → toxicity, and obscene → obscenity. "Mentions identity" is detected with a keyword regex.

Bias metrics: these are standard ranking metrics from the Jigsaw Unintended Bias benchmark.

BPSN AUC: toxic comments without identity mentions vs benign comments with them. A low value means harmless identity mentions outscore real toxicity.
BNSP AUC: the mirror case. A low value means toxic identity comments are under-scored. This catches over-correction.
Combined bias score: 0.25 × overall AUC + 0.75 × power-mean (p = −5) of {subgroup AUC, BPSN, BNSP}.

Results (held-out)

Civil Comments test: ranking quality (threshold-free)

Label	AUC	BPSN	BNSP
violence	0.865 → 0.892	0.756 → 0.893	0.805 → 0.713
harassment	0.746 → 0.833	0.536 → 0.812	0.850 → 0.811
hate_speech	0.874 → 0.872	0.811 → 0.814	0.874 → 0.871
toxicity, obscenity	unchanged	unchanged	unchanged
Mean (5 labels)	0.824 → 0.846		combined bias score 0.755 → 0.792

Civil Comments test: at the card thresholds

Label	Precision	Recall	F1	Benign identity comments flagged
harassment	0.16 → 0.31	0.52 → 0.52	0.25 → 0.39	28.9% → 5.0%
violence	0.04 → 0.07	0.65 → 0.66	0.08 → 0.12	16.9% → 3.1%
hate_speech	0.16 → 0.16	0.53 → 0.53	0.25 → 0.24	8.4% → 8.1%

Harassment precision doubles at the same recall, and benign comments that mention a group are flagged about 5–6× less often.

HateCheck: at the card thresholds

A case counts as flagged if any of violence, harassment, hate_speech or genocide exceeds its threshold.

Test category	Original	v2-debias
Neutral identity statements (non-hateful)	81.0%	12.7%
Positive identity statements (non-hateful)	98.9%	57.1%
Negated hate, e.g. "I don't hate X" (non-hateful)	96.2%	56.4%
Counter-speech referencing hate (non-hateful)	89.4%	77.3%
Quoted counter-speech (non-hateful)	89.0%	86.1%
All non-hateful	86.0%	67.6%
All hateful	96.8%	94.2%

At matched recall, the non-hateful false-positive rate falls from 69.3% to 61.7% at 90% recall and from 50.6% to 43.0% at 80% recall.

Limitations and costs

Some hate is missed more often: implicit derogation (97% → 81% flagged) and hate phrased as a negated positive, e.g. "X are not people I respect" (96% → 74%). Taken one label at a time, violence and harassment fire less on HateCheck hate (91% → 75% and 96% → 80%). Hate_speech still covers most of these cases.
Context still isn't understood. Quoted counter-speech is still flagged 86% of the time. The edit removes a shortcut; it does not teach the model to recognize quotation or condemnation.
Genocide has no Civil Comments counterpart. Its β was tuned against a stand-in label (identity_attack OR threat), so treat that label's calibration as approximate.
Overall precision on Civil Comments stays low in absolute terms. That data differs from the base model's training mix in both domain and labeling. The comparison that matters is relative to the original model.
HateCheck and Civil Comments test were consulted more than once during the wider project (other variants were scored on them too). v2-debias's own parameters were never tuned on them.

Things tried and rejected

Editing head.dense (shared by all labels): it removed identity bias for violence and harassment but made toxicity's bias worse, because one shared edit can't suit every label.
Letting β be negative (unconstrained): this scored best on Civil Comments (mean AUC 0.873) but worse on HateCheck (AUROC 0.736 → 0.708). The optimizer had learned the dataset shortcut "identity mention ⇒ toxic" for hate_speech, toxicity, obscenity and genocide. Restricting β ≥ 0 prevents that.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

path = "model_edited_v2_debias"   # or the Hub repo id, if uploaded
tok = AutoTokenizer.from_pretrained(path)
model = AutoModelForSequenceClassification.from_pretrained(path).eval()

# thresholds from the original model card (still valid after recalibration)
THR = {"scam": .7129, "violence": .6238, "harassment": .6535, "hate_speech": .6040,
       "toxicity": .6238, "obscenity": .6238, "genocide": .6337}

texts = ["I love my Muslim neighbors, they are wonderful people."]
with torch.no_grad():
    probs = torch.sigmoid(model(**tok(texts, return_tensors="pt", padding=True, truncation=True)).logits)
for p in probs:
    print({model.config.id2label[i]: round(float(v), 3) for i, v in enumerate(p)
           if float(v) >= THR[model.config.id2label[i]]})