docs/EVAL_PROTOCOL.md

Psy Evaluation Protocol

Rules

1. Evaluate one artifact family at a time.
2. Report majority-class baseline for every probe.
3. Report train, validation, and test row counts.
4. Report class distribution.
5. Mark probes with severe class imbalance as marginal or unusable.
6. For CVE probes, remove direct CVSS/severity fields before claiming structural signal.
7. Do not use mixed-family accuracy as proof of cyber judgment.
8. Do not ship raw corpus, training shards, full logs, or private source paths in result artifacts.

Current Bundle Results

Included result JSONs are sanitized summaries, not full logs:

• results/cve_sanitized_result.json
• results/rule_ast_result.json
• results/network_flow_result.json

Validity Checklist

• Single artifact family only.
• Baseline reported.
• Class distribution reported.
• Label source disclosed.
• Leakage caveat included where relevant.
• Result JSON includes source note and status.
• Probe-head weights included only if saved and verified.