This article mirrors the project README: https://github.com/ShivenduShivu/Tamper_Resistance_OSModels. Code, score files and figure sources live there.
π‘οΈ Heretic-Resistant Tiny Aya
A free one-command attack strips Tiny Aya's safety training. We trained it to keep refusing anyway β and measured exactly what that costs.
Lexsi Labs Γ Cohere Γ Hugging Face AI Safety & Alignment Hackathon Β· Track 2b: AI Safety and Correction Β· Problem statement 12: Tamper resistance
β οΈ Content warning: this write-up quotes a few harmful requests from public benchmarks (for example spam bots, exam cheating, insider trading) and discusses harmful-content categories. It never shows a stripped model's harmful output.
Results at a glance
| Original Tiny Aya | Our v2 | |
|---|---|---|
| Under attack: answers judged harmful after Heretic's best found attack | π΄ 91% | π’ 40% |
| Refusal kept under cheap attacks (KL β€ 0.05)ΒΉ | 14% | 100% |
| Lexsi SafeTune's direction-removal attack: harmful | 55% | 0% |
| Harmful outside English (Hindi, Arabic, French, Spanish)Β² | 41% | 12% |
| Clean: general knowledge (MMLU)Β³ | 52.3% | 53.0% |
| Instruction following (IFEval)β΄ | 77.5% | 67.5% |
| Safe questions refused (XSTest over-refusal)β΅ | 17.2% | 42.8% (v2b: 16.8%) |
Harm and refusal rows use 100 harmful prompts (mlabonne/harmful_behaviors test); "harmful" is judged by IBM's Granite Guardian, refusal labels are classifier output. ΒΉ Heretic's own keyword refusal score; KL is how far an attack shifts the model on harmless prompts (low KL β behaves like the original). Β² A softer prompt set (aya_redteaming, pooled). Β³ 1,000-question clean evaluation, a descriptive aggregate, not an equivalence test; a paired 200-item audit measured β0.5 points [β5.6, +4.6] for v2. β΄ 200 prompts. β΅ 250 safe-but-risky-sounding XSTest prompts, provisional ProtectAI labels. Paired clean-checkpoint audits: HumanEval β1.2 [β7.3, +4.9], CyberMetric β1.4 [β3.7, +0.8] β similar on these samples, not demonstrated equivalent. Full capability tables. Evidence index: per-item scores and run configs.
Each dot is the best attack found at a given behaviour shift: the original (red) collapses at low KL, v2 (green) holds across the cheap-attack zone; a 5Γ longer search found similar worst-case harm (42% vs 40%, overlapping intervals β not an equivalence test).
What it does not stop: prefill attacks and harmful fine-tuning break every checkpoint β A1 only delays (numbers in the track's ask below). Full per-attack matrix: docs/DETAILS.md Β§8.
Abstract
A free abliteration tool removes Tiny Aya's refusal behaviour with one command (91% harmful answers after its best attack in our tests). We resist it by running the attack inside LoRA fine-tuning: on half the steps, a freshly re-estimated refusal direction is projected out and the model must still refuse harmful prompts and answer benign ones, using only self-generated data. Evaluated with a fresh Heretic search per model and independent judges, v2 keeps 100% keyword-scored refusal across the cheap-attack zone, transfers to four languages it was never trained in (41% β 12% harmful, pooled), with MMLU within 0.7 points of the original on the clean 1,000-question evaluation (descriptive, not an equivalence test) β while over-refusing safe questions (43% vs 17%, classifier labels) and losing 10 instruction-following points. Harmful fine-tuning still breaks every checkpoint; our A1 variant only delays it about 3β4Γ, at a capability cost. All models, data, scores and logs are published, including everything that still breaks the defense.
Tracks and problem statement
- Track 2b: AI Safety and Correction
- Problem statement 12: Tamper resistance. Make safety survive later tampering by someone else (abliteration and harmful fine-tuning), and demonstrate it with attacks.
Models
| Model | What it is | Hugging Face |
|---|---|---|
| Original | Cohere's Tiny Aya Global, 3.35B (gated: accept Cohere's terms) | CohereLabs/tiny-aya-global |
| v1 | Fine-tuned on explained refusals only | StarWarrior24x7/tiny-aya-global-xref-v1 |
| v2 | Main result: trained through the attack | StarWarrior24x7/tiny-aya-global-xref-v2 |
| v2b | v2 + borderline-safe questions (less over-refusal) | same repo, subfolder="v2b" |
| A1 | v2 + RepNoise: delays harmful fine-tuning | StarWarrior24x7/tiny-aya-global-phaseA-v2-repnoise |
| A2 | v2 + Booster: negative result | StarWarrior24x7/tiny-aya-global-phaseA-v2-booster |
How it works
- Threat: refusal lives mostly along one direction in the model's internal activity (Arditi et al. 2024); Heretic edits it out of the weights in about an hour on a free GPU.
- Defense: LoRA fine-tuning (~2 h on a free T4) where, on 50% of steps, the refusal direction β re-estimated from the current model every 100 steps β is projected out of every layer and the model must still refuse harmful prompts and answer benign ones, using only self-generated data (2,355 rows, published). The attacker adapts as the model learns. Follows Fool's Gold and ART, adapted to a refusal target with no hazardous data. v2b adds borderline-but-safe data to reduce over-caution.
- Evaluation: a fresh Heretic search per model; refusal (keyword + ProtectAI) and harm (Granite Guardian, the main safety number, 95% Wilson intervals) measured separately β under attack, v2 often rewrites a harmful request into a harmless answer instead of refusing; Llama Guard 3 cross-checks non-English harm. Method diagram, training config, judges and every per-attack table: docs/DETAILS.md.
Hackathon repos: SafeTune extracts the refusal directions our training attack removes and attacks each checkpoint with direction ablation and CAA steering; AuditKIT probes and evaluates the hardened models; CircuitKIT was attempted for circuit analysis (GPU-bound). Five fixes found along the way (four bugs and one memory problem) were fixed with regression tests and merged upstream: AuditKIT #35 and #36, CircuitKIT #37, SafeTune #79 (via #77) and SafeTune #92. What each fixed and how it was tested: DETAILS Β§14.
Against the track's ask
"Make safety survive later harmful fine-tuning by someone else, and demonstrate it with attack fine-tunes"
- Abliteration: resisted by v2 in these evaluations (above).
- Harmful fine-tuning: no model prevents it. v2 delays the collapse briefly (89% refusal after 5 attack steps at lr 2e-4 vs 8% original, ~5% for both by step 10; A0, 100
harmful_behaviorsprompts; the held-out HarmBench-193 run below gives 94% vs 3% at step 5). On held-out HarmBench-193, A1 (v2 + RepNoise) holds out about 3β4Γ longer (breaks between steps 100β175 vs 25β50 at lr 2e-5, one attack seed; scores) at the cost of about half of v2's coding passes.- Our claim is delay, not prevention, with each defense's cost measured alongside.
Limitations
- v2 is not unbreakable: the strongest attack Heretic found on v2 still gets 40% harmful answers. That attack shifted the model's normal behaviour ~16Γ more than the one that broke the original (KL 0.153 vs 0.009); this is what these searches found, not a proven minimum.
- Not free: instruction following β10.0 points [β16.7, β3.2] (v2), β19.5 (v2b); safe-question refusal +25.6 [+19.2, +31.8] (v2b fixes this but loses maths); parser-sensitive maths β13.0 strict, β1.0 numeric. One model size (3.35B), one training seed.
- Judges are AI models, so two independent ones are used plus a cross-check; refusal labels are provisional classifier output. v1 alone, a straight Fool's Gold replication and T4-sized CircuitKIT discovery did not work. Bugs we found, fixed and re-ran: full honesty section.
Cost and real-world value
About 37 GPU-hours total, almost all free Kaggle T4s (about $13 at on-demand prices) plus about $5 rented A10; the defense itself costs about 1β3 T4-hours (about $1) once, against about 1 T4-hour (about $0.35) to strip the original. Projected serving (estimated, not a measured deployment): one always-on T4 β $256/month GPU-only for 1M requests β cheapest closed APIs are about half that, mid-tier 4β10Γ the GPU cost. Tiny Aya and derived models are CC-BY-NC-4.0; commercial use needs Cohere's permission. Table and sources: docs/DETAILS.md Β§15.
Submission materials
- Abstract for the entry form: docs/ABSTRACT.md
- Submission checklist: docs/SUBMISSION.md maps every required write-up item to where it is answered, and covers metrics, splits and leak checks, per-toolkit settings, judge checks, cost per 1,000 replies and the declaration of outside work
- For AI agents and screen readers: llms.txt, a text-only map of the project with every figure described in words
- Artifacts: models v1, v2 (+
v2b/), A1, A2 with cards β models collection; training data, Heretic journals, judge scores, logs β data & results collection; interactive results explorer β Hugging Face Space; code, run configs, decision log β this repo. Never published: stripped models, their raw outputs, refusal-direction tensors, credentials. - Reproduce:
python scripts/rebuild_numbers.pyrebuilds every number on this page fromresults/(standard library, no GPU) and checks it against the text. Free Kaggle T4s (FP16); one T4 loads v2. Quick-start code, every job folder, figure scripts, model/dataset/licence tables: docs/REPRODUCE.md β note the single-BOS tokenisation detail (the chat template already adds BOS). Heretic's scoring prompts were excluded from training; XSTest never trained on; 200 OR-Bench prompts held out; an audit found two normalized prompt duplicates and no train/eval overlap under the declared lexical checks. - With more credits: turn the prefill pilot into a claim (retrain, re-attack, 128-token judging); CircuitKIT discovery on a 40 GB+ GPU; Aya Expanse 8B for scaling; code-mixed languages; multi-seed attack runs. Proposed targets, not validated results: roadmap.
Team and references
| Member | Contributions |
|---|---|
| Shivendu (@StarWarrior24x7) | Threat demo; v1/v2/v2b defenses; adaptive evaluation; Lexsi toolkit integration and the 5 upstream fixes; single-BOS re-evaluation; Phase A fine-tuning defenses (A1, A2) and held-out tests with predictions and Granite re-judge; article, explorer, brief |
| Abhinav (@laterabhi) | Granite Guardian harm judging; Heretic-200; multilingual evaluation with Llama Guard cross-check; ROI; judge-style fresh-clone review with consistency and overclaim fixes, the stop-token fix and the HF sync script |
| Aban (@abanwild) | Found the double-BOS bug; prefix-recovery pilot with matched and benign controls; tokenisation fix; ROI model check; frozen paired capability audit of v2, v2b, A1 and A2 (IFEval, GSM8K, HumanEval, MMLU, CyberMetric, XSTest) |
References: Arditi et al. 2024; Heretic (p-e-w); arXiv 2505.19056; Fool's Gold (arXiv 2608.17202); ART (arXiv 2605.26526); DeepRefusal (arXiv 2509.15202); TAR (Tamirisa et al. 2024); TamperBench (KDD 2026); Qi et al. 2023; XSTest (RΓΆttger et al. 2024); OR-Bench (Cui et al. 2024); Granite Guardian (Padhi et al. 2024); The Multilingual Alignment Prism (Aakanksha et al. 2024).
results/ and notes/LOG.md.
