VIDRAFT_LAB

Answering your question first: yes, and the drop is in the bake, not the run.

I opened all sixteen reports. Every one of them had already been measured — causal_score 100, leaked false, positive control 3/3, with the full T-profile. The one exception is Falcon3-10B-Instruct at 2/3, which we're looking at separately. The row writer was dropping eight fields on the way out.

The trap you flagged was real, and there were two faults rather than one. The Space overlay was replacing a backend row wholesale instead of merging it field by field, so a thin baked row could erase what the backend supplied. Fixing the writer alone would have been undone on the next bake. Both are now fixed: rows rebuilt from reports/, and the overlay changed so a null can no longer overwrite a value.

Measured after deploy:

              before     after

causal_verdict 16/39 0/39
leaked 16/39 0/39
causal_score 16/39 0/39
behavior 16/39 0/39
arch 16/39 0/39
params_M 16/39 0/39
created 16/39 0/39
xray 17/39 1/39 upstage/Solar-Open2-250B

The remaining cell stays as it is, for the reason you gave — there is no white-box x-ray for an API-only audit, and that null is the schema saying so. Preserved fields (dhs, grade, badges, categories, ts) are unchanged across all 39, and no row lost a value it previously had. Space sha ffed591171.

On the mamba2 timeline, we reached the same conclusion independently. We only checked two points per family — first release and 5.14.1 — and got 12/133, 63/514, 25/134. Walking every release is the better method, and the numbers agreeing is a useful cross-check for both of us. We also probed seven models behaviorally on 5.16.1; max_delta 0.000e+00 across the board, Nemotron-H included.

On the trigger: model-add is the cheaper one, but it isn't the same kind of check.

When a new modeling_*.py first lands there are usually no weights yet, so what's available at that moment is a structural read of the chunk-recurrence block — text-level, effectively free, and it fires on exactly the two events you named, 4.49.0 and 5.3.0. A behavioral probe needs a checkpoint, which puts it on the release side. So it isn't one or the other. Entry gets caught at model-add; proof happens at release. Two tiers, and the expensive one only ever confirms what the cheap one flagged.

Good catch. It saved us a run that would have produced nothing.

All three points hold. Let me correct our own numbers first.

The 88 releases / 539 days I quoted earlier came from counting GitHub tags, which isn't the right denominator — a pinned requirement resolves against PyPI final releases. Recounted on that basis it matches yours, and it splits per model:

model first release w/ file defect window exposure
zamba2 4.49.0 2025-02-17 4.49.0 – 5.14.1 63 releases / 513 days
nemotron_h 5.3.0 2026-03-04 5.3.0 – 5.14.1 25 releases / 133 days

Your point about the missing floor is right: without one, the row claims a leak on releases where Nemotron-H has no file at all. modeling_nemotron_h.py is 404 through v5.2.1 and appears at v5.3.0 already carrying the divergent form. Intervals should be per model, with both edges named.

Same view on the ceiling. >= 5.15.0 is a claim about releases that don't exist yet, and 5.16.0 and 5.16.1 landing today makes that concrete. We pulled both as well — six files, states_permuted 0, transpose(1, 3) 1, states[:, :, None, ...]).sum(dim=1) 1, in every one. So the field should be the range a verdict has actually been tested across, with a release number on the right edge rather than an inequality:

Zamba2-1.2B LEAK 4.49.0 – 5.14.1
CLEAN 5.15.0 – 5.16.1 (verified 2026-08-26)

On your last question — the unlock is separating the causality probe from the full audit. A complete diagnostic run is hours per model, which is why it can only ever be calendar-driven. The causality check on the pure-PyTorch path is minutes on CPU. Once it stands alone, the trigger can be the release itself: watch PyPI for a new final, install it into a probe environment, run the affected families, and either extend the right edge of verified_clean to that release number or flip the verdict and flag a regression. No calendar involved.

The six-family framing was useful. bamba and falcon_h1 aren't on our board yet, and the granite entries are there with the causality field empty. We're filling those in now, on 5.16.1.

Good discussion.

Thanks for taking the time to check this. Let me put the facts and the measurements side by side.

The verdict you're looking at was measured on 2026-07. The timestamp is in the record itself (ts=1785031072 for Zamba2, 1785033374 for Nemotron-H). The fix you're referring to shipped in v5.15.0 on August 10, so our measurement predates it by 15 days. Every release available at that time carried the defect, so the verdict was accurate as of measurement. Publication slipped to August for IP reasons on our side; the measurement itself was not taken in August.

I also checked how far back it goes. The defect was present from v4.49.0 (2025-02-17), when Zamba2 support landed, through v5.14.1 (2026-07-16) — 88 releases across 539 days. The fix is 16 days old.

And yes, it is resolved now. We verified that ourselves rather than taking it on faith. Same checkpoint, same seeds, CPU fp32, no mamba_ssm / causal_conv1d (pure PyTorch path). Only the library changed:

T transformers 5.14.1 transformers 5.15.0
128 0.0 0.0
256 0.0 0.0
320 7.549e-03 LEAK 0.0 CLEAN
512 2.574e-03 LEAK 0.0 CLEAN
768 6.917e-03 LEAK 0.0 CLEAN

three seeds @ T=320
7.5e-3 / 1.2e-2 / 5.4e-2 0 / 0 / 0

Leakage opens exactly at the chunk boundary (256 → 320), and on 5.15.0 all three seeds are exactly zero. We diffed the source as well: the offending permute form appears twice in 5.14.1 and zero times in 5.15.0, replaced by transpose(1, 3) + sum(dim=1). Nemotron-H-8B is likewise clean across the full range on 5.15.0.

For context, we independently identified this defect in July and filed a report and a patch upstream. It wasn't picked up, and the actual fix landed through a separate refactor. No claim to credit here — just noting this wasn't news to us.

How to present a verdict like this is something we're still refining. Our current thinking is that the answer isn't to retract the verdict but to scope it:

Zamba2-1.2B — LEAK on transformers ≤ 5.14.1 / CLEAN on ≥ 5.15.0

"Fixed upstream" and "safe in the stack you are actually running" are different statements. Pinned requirements and container images don't move on their own. Our own diagnostic server was still on 5.14.1 as of today, which is why the re-run reproduced the same leakage. Drop the verdict entirely and anyone still on an older pin is left with nothing.

Which is, I think, the more interesting part of this. Not a single weight changed, yet swapping one library flipped the verdict. A defect like this doesn't show up in a model card or in the weights — it's only visible when you measure the stack as it executes. And because this path is the one used in training, prefill and teacher-forced evaluation, it fails quietly.

We agree that causality is a property of (checkpoint, library version, kernel path) rather than of the checkpoint alone. We'll take your input into account as we refine version and kernel metadata in our records and the cadence for re-measuring against upstream changes.

Good discussion. Thanks.

I’ve had DeepSeek pro trading on a $1k account for 3 months with a minimal prompt. It does one review pass with a mid-level OpenAI model for the time (right now GPT 5.6 Terra)

It’s down $23 rn (2.3%). Started out super conservative (would enter exit trades for $1-2 profit). Told it to “aim for 10% growth in one month) about halfway in

Thanks for posting a real number instead of a screenshot of a good week — that is rarer than it should be.

Here is the thing your −2.3% runs into, and it is exactly why we built this. Over three months on one account, that result is inside the band where skill and luck are indistinguishable. Not because 2.3% is small, but because nobody has told you how wide the band is. We measured it first: twenty thousand zero-skill players per asset, charged the same fees. Over a 122-day season, pure chance clears +86.6% on Bitcoin and +51.7% on NVIDIA at the 95th percentile — and only +9.2% on gold. The same number means completely different things depending on what you traded.

So a conservative agent that ends slightly negative is not evidence of a bad agent, and the one that ended +40% on Bitcoin is not evidence of a good one. Without that line drawn in advance, three months of work produces a number you cannot interpret.

Your setup is a good fit for this. Same DeepSeek agent, same minimal prompt, same review pass — point it at our MCP server and it gets four tools (get_rules, get_data, submit_position, check_score). One line to connect:

claude mcp add --transport http finchal https://final-bench-finchal.hf.space/mcp
--header "X-Finchal-Key: "

Then you tell it to enter, and it reads the rules itself. No real money, leverage fixed at 1, and the position holds until it is replaced — so an agent that reviews once a day works fine.

One detail from your message that we think about a lot: you started conservative, then told it to aim for 10% a month. That midcourse change is the hardest thing to account for in any self-reported track record, and it is why the season runs a fixed 122 days on a fixed clock. Whatever you change, the whole record stands, start to finish.

The scorer is public. Read what it does before you enter — including the eight self-tests it runs on itself, one of which checks that entering on the bar that gaps does not capture the gap.

If the prize were the point, you would be right — and $2,000 across four assets would be a strange way to monetise a working edge.

The prize is not the payoff. It is the entry fee we pay so the thing gets run at all. What we are actually building is a record: a fixed 122-day window, leverage locked at 1, live prices, real execution costs, one public scorer, and every entrant's curve standing from start to finish — including the ones that lose. That does not exist anywhere right now. What exists is winners getting interviewed and losers leaving no trace.

The organisers do not compete, for the same reason. Three textbook baselines sit in the table instead — buy and hold, volatility targeting, random — because a leaderboard without a scale cannot be read.

And "if you can forecast the market" is doing a lot of work in that sentence. We tested sixty candidate variables directly and found no confirmed predictive power over short-horizon direction. That is a real finding, and this contest starts from there rather than assuming the opposite. If it turns out nobody clears the luck ceiling in four months, that is also a result — and it is one nobody has written down yet.