Disclosure to the organizers — PhysicalAI AV OOD Reasoning Challenge 2026

From: Simforge (HF user michaelvu1207) — leaderboard entries under organization simforge
Date: 2026-08-11
To: challenge organizers (@WenyanCong and the NVIDIA AV Research Group), via
nvidia/PhysicalAI-AV-OOD-Reasoning-Challenge-2026 discussions and
nvidia/PhysicalAI-OOD-Leaderboard discussions.

We are participating in both tracks. While doing public-data due diligence on our own
submissions we found two issues that let a participant score without solving the task.
We are reporting them rather than exploiting them, and we are happy to help verify or
re-run anything. We have not attempted to bypass any access control; everything below
came from public file listings, public files, or a gate we were approved through normally.

Issue 1 — Track 2 ground truth is public and can be submitted verbatim

Track 2 is scored on the 349 val events. The ground-truth Chain-of-Causation strings for
exactly those events are published in the parent dataset:

nvidia/PhysicalAI-Autonomous-Vehicles -> reasoning/ood_reasoning.parquet,
rows with split == "val" (290 clips / 349 events), field events[].coc.

Anyone with dataset access can join that parquet to the Track-2 submission keys and upload
the gold strings as their prediction. This is not hypothetical:

• We submitted exactly that file to Track 2 under the deliberately self-identifying name
  SIMFORGE-LEAKED-WARNING (coverage 349/349) precisely so the issue would be visible
  on the public board rather than quietly exploited.

Suggested remedies: score Track 2 on a held-out split whose labels are not in the public
parquet; or keep the val-based board but mark it explicitly non-competitive/diagnostic; or
require a rollout-provenance/inference artifact for prize-eligible Track-2 entries.
As a sanity check, the exact-gold submission also gives you AlpaJudge's ceiling: a
non-1.0 score there tells participants (and you) how much of the residual gap is judge noise
rather than reasoning error.

Issue 2 — nvidia/cascade contains human causal annotations on 8.5% of the hidden Track-1 test events

nvidia/cascade ("CASCADE: Causal Spatio-Temporal Analysis of Driving Environments",
2,066 PAI-AV clips, human-annotated and human-reviewed) is gated but auto-approved — the
gate granted our request immediately, so effectively any participant can obtain it. Its own
README describes it as a "structured complement to the Chain-of-Causation labels released
alongside the parent Physical AI dataset."

Its per-clip filenames (data/batch_00001/<session>__<clip_uuid>.json) are listable through
the public repo API without access, so the overlap can be computed by anyone:

• 15 of the 214 hidden Track-1 test clips are annotated in CASCADE,
• covering 24 of the 284 hidden test events (8.5%),
• plus 107 of the 1,740 released train/val CoC clips.

Contact: HF user michaelvu1207 / organization simforge on the leaderboard.