program-structure-certificates / GITHUB_README.md
system's picture
system HF Staff
Mirror GitHub f31f2ceadd54
fd74a4d verified
|
Raw History Blame Contribute Delete
3.43 kB

When Is Program Structure Required? Certificate-Grounded Calibration of Program-Induction Difficulty

Wenzhang Du 路 Independent Researcher, China 路 arXiv preprint, November 2026

Paper 路 arXiv:2611.24975 路 Technical report 路 涓枃浼磋 路 Interactive Space

A generator can use two string-replacement rules even when the displayed examples admit a one-rule solution. This artifact follows a matched pair of specifications through complete one-rule enumeration, executable two-rule witnesses, and the programs returned by language models.

Reproduction guide 路 Result map 路 Data provenance 路 Frozen artifact v1.0.0

Follow one archived task pair

From the repository root with Python 3.11 or newer:

python -B verify_manifest.py
python -B replay_fixture.py

The fixture checks the six-example pair in Figure 1. The control admits the single replacement from exa to azy. The separation example excludes that rule, and the shared two-rule reference satisfies both specifications. The command reports certified minima of one and two, then executes four saved Qwen and Olmo programs on the displayed, witness, uniform, and rule-active inputs. It uses the included archive and the Python standard library.

Three-panel example showing how changing one input-output example excludes the one-rule control solution.

Results at a glance

Across both models and the three-to-six base-example settings, the certified separation tasks reduce displayed-task completion by 23.53 to 45.83 percentage points and increase exact success on the wholly unshown witness pair by 8.40 to 13.45 points. Agreement on strings activating both reference rules ranges from 1.93% to 5.78% in the separation conditions. The archived tables distinguish these evaluation targets. Primary panel 路 Broader-input panel

Replay the panel

Create an environment with NumPy, then run the full CPU check from the frozen tag:

python -m pip install -r requirements.txt
python -B reproduce.py --out replay

This reads 1,920 saved outputs, checks 960 certificates and 7,680 additional reference labels, and regenerates the three comparison CSVs. The result map marks which historical analyses are available as aggregate tables and protocols. Full instructions 路 Verification report

Citation

Cite Wenzhang Du, When Is Program Structure Required? Certificate-Grounded Calibration of Program-Induction Difficulty, arXiv:2611.24975 (2026). The public release lists the software and data licenses, the frozen revision and the single offline download. CITATION.cff 路 Version record