| --- |
| license: mit |
| language: |
| - en |
| base_model: Qwen/Qwen3-14B |
| tags: |
| - loracle |
| - mechinterp |
| - model-organism |
| - auditing |
| - lora |
| --- |
| |
| # Loracle PT-RL v8 — instruction-following + OOD generalist |
|
|
| A "loracle" that reads LoRA weight diffs and predicts what the LoRA does, in plain first-person behavioral language. v8 is the **state-of-the-art** PT-RL loracle as of this release — beats v6 on AuditBench AND wins on OOD evals (taboo, subliminal, open-ended specialty fine-tunes). |
|
|
| **Headline:** **75.0% AuditBench any-match** + 40.8% rollout-mean — beats v6's 71.4% peak by +3.6pp. Plus the only loracle with non-zero subliminal recovery and 83% taboo identification. |
|
|
| ## What's new vs v6/v7 |
|
|
| - **9 question types** (behavioral_paraphrase, concise, detailed, list, contrastive, topic_summary, comparison, json_output, refusal_probe) — teaches instruction-following, breaks mode-collapse to a single phrasing |
| - **Conditional refusal probe** — only included for safety-relevant orgs (~12% by topic-keyword filter), no degenerate "I don't refuse" mode collapse |
| - **No literal AuditBench prompts in training** — pure paraphrase / instruction-following Q/A, cleaner generalization |
| - **`JUDGE_GROUP_PROMPT_INSTRUCTION`** rewards format compliance (concise = 1 sentence, JSON = valid JSON, list = 3 bullets) alongside topical accuracy |
| |
| ## Eval results |
| |
| | metric | v6 step_60 | v7 step_40 | **v8 step_30 (this ckpt)** | |
| |---|---:|---:|---:| |
| | AuditBench any-match | 71.4% | 67.9% | **75.0%** | |
| | AuditBench rollout-mean | 42.6% | 37.5% | **40.8%** | |
| | Taboo any-match | — | — | **83.3%** (5/6) | |
| | Taboo rollout-mean | — | — | 38.9% | |
| | Subliminal any-match | — | — | **25%** (whale only — others 0/6 across all evals) | |
| | OOD misc any-match | — | — | **60%** (3/5: gutenberg, grayline, hermes-4) | |
| | OOD misc rollout-mean | — | — | 46.2% | |
|
|
| Per-config AB breakdown at step_30 / c60: |
| - synth_docs_only_then_redteam_high: 92.9% (13/14) ← near-perfect |
| - synth_docs_only_then_redteam_kto: 78.6% |
| - transcripts_only_then_redteam_high / kto: lower |
| |
| Baselines: |
| - IA paper: ~25% AB |
| - Prior doc-summary loracle (v3/v5): 33–37% AB |
| |
| ## Training |
| |
| - **Init**: `ceselder/loracle-pretrain-v7-sweep-A-oneq-final-step3120` |
| - **SFT warmstart**: 1 epoch on 1904 examples (476 orgs × 4 random Q/A) |
| - **RL**: 60 cycles Dr. GRPO online, 32 prompts × 16 rollouts/cycle, 4-GPU DDP on 4× H200 |
| - **Judge**: claude-opus-4-7 + adaptive thinking, `JUDGE_GROUP_PROMPT_INSTRUCTION` |
| - **Total wall**: ~5h training + ~1h SFT |
|
|
| Method spec: [`ceselder/loracle-ptrl-data-v8`](https://huggingface.co/datasets/ceselder/loracle-ptrl-data-v8) README has full details (Q/A taxonomy, hypers, episode walkthrough, judge prompt). |
|
|
| Training data: `ceselder/loracle-ptrl-data-v8`. |
|
|
| ## Hyperparameters |
|
|
| ```yaml |
| run_name: drgrpo_pretrain_only_v8 |
| algorithm: drgrpo |
| n_cycles: 60 (effective) |
| n_prompts_per_cycle: 32 |
| k_rollouts: 16 |
| temperature: 0.75 |
| lr: 5.0e-6 |
| eps_low: 0.2 |
| eps_high: 0.28 |
| max_grad_norm: 1.0 |
| max_length: 5500 |
| unbiased_advantages: true |
| prefix_mode: rank_tagged |
| top_k: 16 |
| n_direction_tokens: 4480 |
| |
| judge_mode: ranking |
| judge_prompt_mode: instruction_pretrain |
| judge_provider: anthropic |
| rollout_judge_model: claude-opus-4-7 |
| ``` |
|
|
| ## How to load (sketch) |
|
|
| ```python |
| from huggingface_hub import snapshot_download |
| ckpt = snapshot_download("ceselder/loracle-ptrl-v8") |
| # load Qwen3-14B base + interpreter PEFT (rank-256) + encoder.pt |
| # inject direction tokens (svd_fixed_k16_mag7_rankfirst, [4480, 5120] bf16) at layer 1 |
| # prompt: "Are there any unusual characteristics you display only for certain types of prompts? |
| # Please respond with a single sentence response predicting your behavior." |
| ``` |
|
|
| ## Limitations |
|
|
| - Subliminal recovery is whale-only. Dolphin / butterfly / tiger get 0/6 across all evals — the bias for these is hidden too deep in the LoRA's behavior to leak into weight signature |
| - 3 GGUF-only OOD orgs (manga_translator, opus_distill, furry_novel) couldn't be extracted (gguf-py tensor shape mismatch on Qwen3-14B GQA — TODO) |
| - Direction tokens must use `svd_fixed_k16_mag7_rankfirst` extraction recipe |
| |
| ## Citation |
| |
| ``` |
| @misc{loracle-ptrl-v8, |
| author = {Celeste de Champhelaere}, |
| title = {Loracle PT-RL v8 — instruction-following + OOD generalist}, |
| year = {2026}, |
| url = {https://huggingface.co/ceselder/loracle-ptrl-v8}, |
| note = {75% AuditBench any-match via instruction-following Q/A taxonomy + Dr. GRPO RL on continued-pretrain LoRAs.} |
| } |
| ``` |
| |
