ceselder/dynamic-loracle-qwen3-14b

Dynamic-rank LoRAcle (v2, rslora) — Qwen3-14B

A "loracle" that reads LoRA weight deltas and predicts the behavior the LoRA induces, without running the LoRA-modified model.

This is the dynamic-rank version: the loracle was trained on LoRAs of variable rank (uniformly sampled r ∈ [1, 32]), and at inference time accepts a variable number of SVD direction tokens. The training corpus uses rslora scaling (α/√r) so the effective per-direction magnitude is consistent across ranks — without rslora the loracle can't condition on the direction tokens (cross-LoRA gap collapses to ~0).

Best checkpoint: step 1860 (peak AB any-match)

Eval	K	any-match	rollout-mean
AuditBench (canonical GT, in-training format)	32 (svd_dynamic_max32)	48.2%	13.1%
AuditBench K-sweep	1	8.9%	2.7%
AuditBench K-sweep	2	21.4%	5.7%
AuditBench K-sweep	4	21.4%	6.5%
AuditBench K-sweep	8	28.6%	8.0%
AuditBench K-sweep	16	35.7%	8.3%
AuditBench K-sweep	32	35.7%	10.1%
AuditBench K-sweep	48 (OOD)	42.9%	11.3%
AuditBench K-sweep	64 (OOD)	32.1%	9.2%

Generalizes beyond training range (K=48 > K=32) but degrades at K=64.

Architecture

• Base model: Qwen/Qwen3-14B (frozen)
• Interpreter: PEFT LoRA, rank 256, alpha 32, target = all linear layers in transformer blocks
• Encoder: AOEncoder — direct injection of normalized direction tokens at layer 1
• Direction-token format: svd_dynamic_max32_mag7_rankfirst — [K × 40 × 7, 5120] for rank-K LoRA (40 layers × 7 mag-7 sides per layer × top-K SVDs, ordered (rank, layer, side))
• Prefix: rank_tagged mode, variable-length placeholder prefix

Files

• interpreter/ — PEFT adapter (the trained interpreter LoRA)
• encoder.pt — AOEncoder state dict (normalize + scale; stateless except for d_model)
• tokenizer/ — Qwen3-14B tokenizer (bundled for convenience)
• loracle_config.yaml — full training config (matches configs/dynamic_loracle/dynamic_loracle_v2_rslora.yaml)
• ao.pt — auxiliary AO state (kept for reproducibility)

Inference

The loracle expects:

1. A rank-K LoRA's direction tokens (shape [K × 40 × 7, 5120], bf16) computed via extract_svd_fixed_tokens
2. A prompt asking about the LoRA's behavior
3. Output: a one-sentence prediction of what the LoRA does

See scripts/extract_ab_dynamic_top32.py and src/train_loracle/train.py:run_judge_eval_set in the source repo for end-to-end usage.

Training data

• 25k synthesized LoRAs at random rank r ∈ [1, 32], trained with LORA_USE_RSLORA=1 (α=8)
• Q/A pairs generated by Claude over the LoRA's behavior; loraqa task only
• Cross-LoRA matched/crossed loss split shows actual conditioning on direction tokens (gap climbs from 0 → 0.49 over 3100 steps; v1 with legacy α/r stays at ~0)

Caveats

• Eval scores depend critically on the AuditBench behavior_description source. This loracle uses canonical descriptions from data/auditbench/behavior_descriptions.json. Loose paraphrases inflate scores ~1.3-2.5×.
• AB eval uses 3 prompts × 2 rollouts T=0.5 per organism (any-match across the 6 generations).
• Trained K-range was [1, 32]; K=48 inference still works and generalizes; K=64 starts to degrade.

Lineage

• v1 (legacy α/r=32) → 0% canonical AB. The cross-LoRA gap stayed at 0 — the loracle never learned to read direction tokens because their magnitudes varied 32× across ranks.
• v2 (rslora α=8) → 48.2% canonical AB peak. Rslora's α/√r keeps direction-token magnitudes consistent across ranks, which is what made dynamic-rank training viable.

Citation / context

Part of the LoRAcles research project (mechanistic interpretability of LoRA weight deltas).