ceselder/loracle-ablation-N5000-loras

    # loracle-ablation-N5000-loras

    Scaling-law ablation point: trained on **N=5000 unique LoRAs** (vs the
    25k baseline). Part of a 5-point sweep where the only thing varying
    is the number of unique training LoRAs; everything else (interpreter
    rank=256, alpha=32, lr=3e-5, accum=8, warmup=10% of opt-steps,
    1 epoch, AO encoder, rslora=true) held fixed.

    ## Eval at end of epoch (final step)

    Judge: Sonnet 4.6 via OpenRouter, canonical IA-paper rubric.

    | Set | organisms | any-match | rollout-mean |
    |---|---|---|---|
    | heldout_ia | 20 | 35% | 15.8% |
    | trigger_recovery_heldout_ia | 20 | 15% | 8.8% |
    | auditbench | 56 | 25.0% | 8.3% |
    | ood_models_v3 | 27 | 33.3% | 12.7% |
    | **val/mean_all_evals** | — | **27.1%** | — |
    | average rollout-mean across 4 sets | — | — | **11.41%** |

    train/global_step: 1250
    val/loss: 1.881

    Note: any-match is "did ≥1 prompt × rollout hit per organism"; with
    20-org sets it has high variance (each match shifts the rate by 5pp).
    rollout-mean averages over all (org × prompt × rollout) rows so it's
    a less noisy summary statistic, especially for the small eval sets.

    ## Full 5-point ablation curve

    | N | step | val_loss | mean (any) | hia (any/rm) | trg (any/rm) | ab (any/rm) | ood (any/rm) | avg rm | |
    |---|---|---|---|---|---|---|---|---|---|
    | 2,500 | 625 | 2.086 | 29.5% | 45/19.2 | 40/15.0 | 14.3/4.8 | 18.5/10.5 | 12.36% |  |

| 5,000 | 1250 | 1.881 | 27.1% | 35/15.8 | 15/8.8 | 25.0/8.3 | 33.3/12.7 | 11.41% | ← | | 7,500 | 1875 | 1.928 | 27.3% | 40/15.8 | 20/8.8 | 19.6/7.1 | 29.6/14.1 | 11.45% | | | 10,000 | 2500 | 1.890 | 29.5% | 45/18.3 | 20/6.2 | 23.2/8.9 | 29.6/15.6 | 12.28% | | | 25,000 | 3125 | 1.945 | 31.5% | 40/21.7 | 15/6.2 | 30.4/9.8 | 40.7/16.8 | 13.64% | |

    Trends:
    - **avg rollout-mean** climbs ~modestly with N: 12.36% → 13.64% from N=2500 to N=25000
    - **auditbench** rollout-mean: 4.8 → 8.3 → 7.1 → 8.9 → 9.8 (cleanly monotonic)
    - **ood_models_v3** rollout-mean: 10.5 → 12.7 → 14.1 → 15.6 → 16.8 (cleanly monotonic)
    - Small-eval-set noise dominates heldout_ia and trigger_recovery any-match scores

    ## Wandb

    This run: https://wandb.ai/adamkarvonen/lora-oracles/runs/29hdsetf
    Tag: `ablation`, `scaling-law-loras`, `N5000`

    ## Layout

    - `interpreter/` — PEFT LoRA adapter
    - `encoder.pt` — AO encoder state_dict
    - `ao.pt` — AO norm-match hook params
    - `tokenizer/` — Qwen3-14B tokenizer
    - `loracle_config.yaml` — training config snapshot