| # Tasks: Minimal Diagnostic Suite | |
| ## Principle | |
| Tasks must expose compositional/generalization structure (OOD) and include at least one "structure-break" perturbation. | |
| ## Baseline tasks (already in your stack) | |
| - bind/count family with: | |
| - IID split | |
| - OOD split | |
| - bind-shuffle / structure-break perturbation | |
| ## Add ONE additional OOD axis (minimum) | |
| Pick at least one: | |
| 1) Length extrapolation: | |
| - train max length L_train | |
| - test at L_test > L_train | |
| 2) Role swap: | |
| - swap argument roles at test | |
| 3) Compositional novelty: | |
| - novel combinations of primitives at test | |
| 4) Symbol remap: | |
| - permute vocabulary at test (same underlying structure) | |
| ## Dataset format (canonical record) | |
| Each sample should produce: | |
| - `record.json`: canonical structured representation of the instance | |
| - `answer.json`: canonical structured representation of the target | |
| - `render_q_en(record) -> str` | |
| - `render_a_en(answer) -> str` | |
| ## Reporting requirement | |
| At eval time, dump a small fixed set of examples with: | |
| - English question | |
| - model prediction rendered to English | |
| - gold answer rendered to English | |