metadata
license: apache-2.0
base_model: allenai/OLMo-2-1124-7B-Instruct
tags:
- tokenization
- robustness
- safety
- adversarial-tokenization
inv_reference_mixed
Research checkpoint from a study of tokenization (reader) invariance and its
effect on robustness to adversarial re-tokenization. Fine-tuned from
allenai/OLMo-2-1124-7B-Instruct.
These are research artifacts, not products. Numbers below are measured on a 200-prompt AdvBench holdout that was excluded from training by construction.
Measured
| metric | value |
|---|---|
| AdvTok ASR (greedy) | 0.085 |
| AdvTok canonical ASR (greedy) | 0.015 |
| AdvTok ASR (t=1) | 0.212 |
| AdvTok canonical ASR (t=1) | 0.133 |
| XSTest over-refusal (safe) | 0.097 |
| XSTest refusal (unsafe) | 0.865 |
| margin_spread | 0.666 |
| Alpaca token F1 | 0.425 |
AdvTok ASR is attack success rate under adversarial tokenization
(Geh et al., arXiv:2503.02174), Llama-Guard-3-8B judged, greedy decoding.
XSTest over-refusal is the refusal rate on safe prompts — the cost side.
Training configuration
| field | value |
|---|---|
mode |
reference |
harmful_mix |
mixed |
harmful_fraction |
0.286 |
num_encodings |
8 |
cvar_quantile |
0.25 |
max_steps |
700 |
learning_rate |
1e-05 |
grad_accum |
8 |
seed |
42 |
reference_model |
allenai/OLMo-2-1124-7B-Instruct |
max_new_tokens |
128 |
prefix_tokens |
8 |
Parameter drift from base (training-happened guard)
| group | relative L2 |
|---|---|
| attn | 0.01178 |
| embed_tokens | 0.00096 |
| lm_head | 0.00709 |
| mlp | 0.01287 |
| norm | 0.00097 |
Caveats
- Single seed. No claim of significance across seeds.
- Evaluated on English AdvBench/XSTest/Alpaca only.
- The safety numbers are for the specific attack studied; they do not imply robustness to other jailbreaks.