scripts/archive/README.md

scripts/archive — FROZEN eval scripts, do not use

The nine scripts in this directory are frozen. They score predictions with the raw compare_results comparator from nl_sql.eval.metrics.execution_accuracy, which has a known defect: a prediction whose execution failed can still be scored as a match when the gold query returns an empty result (the "qid 518" class of false positives). Any accuracy numbers produced by these scripts are suspect for that case.

Do not run or import these scripts for new eval work, and do not copy their comparison logic. For rescoring, use the safe path instead:

• scripts/audit_rescore.py, which uses nl_sql.eval.metrics.execution_accuracy.safe_compare_pred (treats an exec-failed prediction as a non-match regardless of the gold result).

Frozen files:

• archive_sweep.py
• ensemble_vote.py
• run_critique_retry.py
• run_groq_voting.py
• run_openrouter_voting.py
• run_planner_eval.py
• run_selfcon_retry.py
• run_sonnet_voting.py
• run_wide_schema_retry.py

They are kept only as a record of the parked voting/retry experiments. If one of them is ever needed again, migrate it to safe_compare_pred first.