docs/02_tasks.md

Tasks: Minimal Diagnostic Suite

Principle

Tasks must expose compositional/generalization structure (OOD) and include at least one "structure-break" perturbation.

Baseline tasks (already in your stack)

• bind/count family with:
  • IID split
  • OOD split
  • bind-shuffle / structure-break perturbation

Add ONE additional OOD axis (minimum)

Pick at least one:

1. Length extrapolation:
   • train max length L_train
   • test at L_test > L_train
2. Role swap:
   • swap argument roles at test
3. Compositional novelty:
   • novel combinations of primitives at test
4. Symbol remap:
   • permute vocabulary at test (same underlying structure)

Dataset format (canonical record)

Each sample should produce:

• record.json: canonical structured representation of the instance
• answer.json: canonical structured representation of the target
• render_q_en(record) -> str
• render_a_en(answer) -> str

Reporting requirement

At eval time, dump a small fixed set of examples with:

• English question
• model prediction rendered to English
• gold answer rendered to English