PhysTax

An AI scientist meets a universe it has never seen.

It designs its own experiments, watches them only through Meta's SAM 3.1, seals a numeric prediction, and is graded by what actually happens. This is one real run.

Sealed · verify

The prediction was hashed with SHA-256 and written to a hash-chained ledger before the test was rendered. Anyone can recompute the hash from the committed prediction in the repository's run folder.

    Then we pointed it at the real world.

    Record a few seconds with a webcam, tap the object once, and the same protocol runs on real footage.

    SAM 3.1, seen before sealing sealed predictionwhat happened

    The law the scientist sealed, fitted to the first half (pixels, seconds, y up):

    Sealed · verify

    Hashed and written to the run's ledger before the second half of the track was read.

    Try it with your own object →

    The agent rarely fails to see. It fails to choose.

    Does the agent's law carry the physics?

    A fresh forecaster predicts each episode's tests three ways: from the description alone, from the agent's sealed law, and from the true law fitted to the same measurements. The gap between them is what the agent's reasoning contributed, and what it left on the table.

    SAM 3.1, measured as an instrument.

    Every result below compares SAM's tracks with exact simulator positions, frame by frame.

    Position error inside the frame as the camera gets worse. The classical tracker collapses at the worst setting; SAM never lost an object that was in view.
    Box-prompt confidence tracks error: confident frames are several times more accurate. Word-prompt confidence does not; it rises with the frame number.
    Same objects, same frames, different words. Specific nouns are found or missed depending on how the scene is drawn; shape words and a box on the first frame always work.
    A textbook least-squares fit assumes independent errors. SAM's errors are correlated from frame to frame, so its 80% intervals hold the truth far less often. Correcting for the correlation recovers part of the gap.

    On real lab footage, the laws hold up and one assumption does not.

    The period shrinks as the swing decays, exactly as the full pendulum equation predicts. The small-angle law (dashed) says it should not change.
    Which friction law slows the swing? Quadratic air drag fits; the exponential decay used to derive the benchmark's damping values does not.

    An open problem from Meta's own research.

    Fit an ellipsoid through random points in dimensions. Meta's paper proves it works below and fails above, and leaves the edge itself open. We measured that edge, sealed predictions for larger dimensions before computing them, then graded them.

    Where the transition sits, . Dots: measured centres with 95% intervals (filled: calibration, open: sealed targets). Dashed: the asymptotic . Blue: the sealed conjecture , falsified.
    Every sealed predictor at the target dimensions. The Brier score grades the whole predicted transition curve against every trial; lower is better.

    Check it yourself.