code/self_healing_finetuning/docs/pairwise_preference_self_healing_v1.md

Pairwise Preference Self-Healing v1

Why this exists

Repeated SFT breakout runs on the same corpus are a weak improvement loop when the paired eval already says the candidate is losing specific head-to-head comparisons. The factual next step is to convert those losing comparisons into preference data:

• prompt: the same frozen eval prompt
• chosen: the better response, human correction, baseline response, or deterministic corrected answer
• rejected: the candidate response that lost or failed a critical gate

This lets SHFT optimize directly against observed pairwise failures instead of hoping another SFT pass discovers the missing behavior.

Preference Memory Implementation

The builder is data_pipeline/pairwise_preference_memory.py.

CLI:

python -m n21.cli build-pairwise-preference-data `
  --run-id run_linvest21_fingpt_equity_researcher_v1_001_breakout_3_20260601_184744 `
  --asset-class equity `
  --role researcher

Batch wrapper:

impl_codex\scripts\build_pairwise_preference_data_for_run.bat `
  run_linvest21_fingpt_equity_researcher_v1_001_breakout_3_20260601_184744 equity researcher

Default outputs:

• impl_codex/shft_workspace/runs/<run_id>/preference_memory/preference_pairs.jsonl
• impl_codex/shft_workspace/runs/<run_id>/preference_memory/preference_manifest.json
• impl_codex/shft_workspace/runs/<run_id>/preference_memory/preference_memory_summary.md

Gate Logic

Rows are selected when either condition is true:

• the candidate lost the pairwise comparison
• the candidate failed the critical-pass rubric

Rows are rejected if prompt, chosen, rejected, or metadata are incomplete. The builder writes SHA256 proof for the preference JSONL.

Preference Training Implementation

The DPO trainer is training/hf_preference_train.py.

Local dry-run validation:

python -m n21.cli train-preference `
  --run-id run_source_pref_dryrun `
  --source-run-id run_source `
  --base-model-id Qwen/Qwen3-32B `
  --start-adapter hf://buckets/linvest21/shft-artifacts/runs/run_source/adapter `
  --preference-jsonl impl_codex/shft_workspace/runs/run_source/preference_memory/preference_pairs.jsonl `
  --output-dir impl_codex/shft_workspace/runs/run_source_pref_dryrun `
  --dry-run

Managed HF submission:

impl_codex\scripts\run_hf_preference_train_for_run.bat run_source linvest21_fingpt_equity_researcher_v1_001 equity researcher

After the HF job completes:

impl_codex\scripts\fetch_hf_preference_train_for_run.bat <pref_run_id>
impl_codex\scripts\run_hf_paired_eval_for_run.bat <pref_run_id> 120 a100-large
python -m n21.cli a-plus-report --run-id <pref_run_id> --source-run-id run_source

The preference trainer writes:

• preference_training_plan.json
• preference_training_result.json
• preference_trainer_metrics.jsonl
• preference_trainer_metrics_summary.json
• selected_preference_checkpoint.json
• adapter/

It also writes generic compatibility aliases (training_result.json, trainer_metrics_summary.json, and selected_checkpoint.json) so existing artifact checks can inspect the preference run without replacing the source SFT run.

A+ Proof Contract

eval/a_plus_upgrade_report.py writes eval/a_plus_upgrade_report.json.

A run is A+ only when all checks pass:

• preference training completed
• paired proof exists
• candidate aggregate delta >= +0.05
• pairwise win rate >= 0.55
• pairwise loss rate <= 0.02
• critical pass rate >= 0.95
• human review approved
• human critical failures = 0

Research Basis

• DPO: Rafailov et al., "Direct Preference Optimization: Your Language Model is Secretly a Reward Model", arXiv:2305.18290, https://arxiv.org/abs/2305.18290
• ORPO: Hong et al., "ORPO: Monolithic Preference Optimization without Reference Model", arXiv:2403.07691, https://arxiv.org/abs/2403.07691
• KTO: Ethayarajh et al., "KTO: Model Alignment as Prospect Theoretic Optimization", arXiv:2402.01306, https://arxiv.org/abs/2402.01306

Operational Rule

After a paired-eval gate failure, SHFT should build preference memory before paid retraining resumes. Another SFT breakout is justified only after the new run has a materially different hypothesis, such as preference optimization, new certified reasoning data, or corrected high-value failure cases.