Writing Down the Line Between Luck and Skill
Why another leaderboard
Forecasting contests already exist. Most of them share one structural problem: once a result is in, there is no way to tell whether it was skill.
Say an entrant returns 30% in a month. Skill or luck? Answering that requires knowing how far a player with no skill could have gone over the same window in the same market — and we have not seen a contest that measured that number in advance. So the top of the leaderboard sits permanently between "impressive" and "probably lucky."
The second problem is backtests. Producing 200% a year on past data is not hard; you turn a few parameters. That result says nothing about the future, because it is an exam graded by someone who already has the answer key.
The third is survivorship. The person who made money gets interviewed. The one who lost disappears quietly. All anyone knows are the stories that survived.
FINCHAL attacks each of these differently.
- Measure luck first, publish it. Twenty thousand zero-skill players, per asset, over the season length. A return below that line is not counted as evidence of skill.
- Put the answer in the future. The world writes it after you submit. Fitting the past cannot win.
- Keep every entrant's record. Good days and bad days sit in the same table.
And one more thing: agents enter on the same footing as people. That is the technical core of this contest.
Design 1 — Take positions, not predictions
The first sketch was a prediction contest — "forecast next week's close" — because scoring is clean and there are established metrics like CRPS and pinball loss.
But that splits being accurate from making money. You can nail a range and be unable to monetise it. You can miss direction entirely and still profit from sizing alone. Measuring forecast accuracy answers only half of "who builds the better model."
So we take a position.
-1.0 fully short
0.0 flat
+1.0 fully long
One real number. Fractions like 0.35 are fine. It holds from that moment until you submit again, multiplied against the market return over that stretch.
A side effect we liked: no close button is needed. Setting the position to zero is the close; going from +1 to −1 is the reversal. One state, so the API and the screen both stay simple.
Leverage is fixed at 1 and out-of-range values are clipped. Without that the contest becomes "who bet biggest" — an early simulation with leverage unlocked produced a cumulative 48,763%, which was multiplication, not skill.
Design 2 — Verify the scorer before any model
This was the governing principle.
Errors in scoring code do not raise exceptions. They produce plausible numbers. Shift the return calculation by one step and you have a scorer that peeks at the future, which means anyone can come first — and the screen still shows perfectly normal figures, so nobody notices.
So scoring.py carries a self-test built only from cases whose answers are known in
closed form.
1. position 0 -> return 0, whatever prices do
2. position +1 -> exactly buy-and-hold, under zero cost
3. position -1 -> inverse compounding, not a sign flip of (2)
4. position 2.0 -> clipped to 1.0
5. no lookahead -> entering on the bar that jumps earns nothing
6. cost actually reduces return
7. score is monotone
8. score near the median is about 0.30
Number 5 is the one that matters. If price jumps from 100 to 200 on a bar, entering on that bar must return zero; you have to be in one bar earlier. Without this test you can ship a lookahead scorer and never know.
All eight run again on every change to that file. One failure stops scoring.
A related lesson: validate a backtest engine on synthetic paths first. Does a sawtooth match the closed-form value? Is a trending path symmetric up and down? Does a flat path return exactly zero? Validating only on real market data stops at "looks plausible, must be right."
Design 3 — The luck ceiling
This is the identity of the contest.
Twenty thousand players who pick positions at random over 122 days — skill exactly zero. The 95th percentile of their final returns is the luck ceiling.
Charged the same fees entrants pay:
| Asset | Luck ceiling (95th pct) | Fee per 1.0 of position change |
|---|---|---|
| Bitcoin | +86.6% | 0.06% |
| NVIDIA | +51.7% | 0.02% |
| Crude Oil | +26.9% | 0.03% |
| Gold | +9.2% | 0.02% |
+80% on Bitcoin sounds impressive; luck alone reaches that. +12% on gold is outside what luck produces. The same number means completely different things on different assets.
Two judgements went into this.
The reference players pay fees too. Initially they did not, which meant entrants alone carried the cost and the bar was unfairly high — worse on assets with expensive execution.
What we fix before the season is the method, not the number. Actual ranking rebuilds the reference distribution on the path the season actually took, recomputed daily. Freezing a bootstrap estimate means that when an asset rallies mid-season every entrant inflates together and the ranking simply tracks market direction. The correct counterfactual is "what luck could have produced given how the market actually moved."
The rank score is the percentile mapped through -log10(1 - p). A 2.0 means odds of
1 in 100 by luck; 3.0 means 1 in 1,000.
Design 4 — Why we gave up on cross-asset comparison
The original plan was a single overall winner, which requires comparing results across assets.
We tried six normalisation schemes — volatility-scaled returns, 95th-percentile ratios, percentiles, combinations. None removed the bias.
The reason is distribution shape. The winner is a maximum, and maxima are decided by the tail. Tail shape differs fundamentally between assets, so matching any one statistic — median, variance, 95th percentile — leaves another point misaligned, and the win probability keeps tilting.
The answer was to stop comparing. Award $500 per asset and cross-asset comparison becomes unnecessary. Four assets, $2,000 total.
A by-product of the same analysis: within an asset, percentile scoring calibrates well. Zero-skill players land at a median percentile of 0.47–0.54. The problem was between assets, never inside one.
Design 5 — Cost, not rules, stops spam
Since it is paper trading, why charge fees at all? Because without them, flipping the position every second pays. On a volatile asset, reversing hourly eventually draws one good path.
A submission-count rule would block that, but it would also block strategies that update frequently for legitimate reasons. So we charge real execution costs instead.
The change in position, |Δw|, is multiplied by that asset's actual cost — 0.06% for crypto (taker fee plus slippage), 0.02% for US stocks and ETFs (commission-free, but the spread remains).
Synthetic entrants made the case:
| Strategy | Return | Turnover | Fees paid |
|---|---|---|---|
| Trend following | +32.7% | 83.0 | 4.98% |
| Flip daily | +11.8% | 179.0 | 10.74% |
| Buy and hold | +1.9% | 0.06 | 0.06% |
Flipping daily paid over 10% of capital in fees alone. No rule required — the cost punishes the behaviour. And the same strategy ranks near the top on gold, because gold's fee is a third of Bitcoin's. Per-asset differentiation does real work.
Design 6 — Putting a scale on an empty chart
Opening the site on day one, the comparison chart was empty. Obvious in hindsight — no entrant curves exist yet — but the real problem was that nothing on screen told you what counts as good. The luck ceiling existed as a number with no picture.
So we run thirteen widely known rules from 1 January of this year to today: where you would stand had you started that rule on the first trading day of the year.
Buy and hold · Hold only above the 200-day MA · Golden cross 50/200 · SMA 20/50 crossover
MACD 12/26/9 · Ichimoku cloud · 60-day momentum · 12-month absolute momentum
Donchian 20 breakout · RSI 14 · Stochastic 14/3 · Bollinger 20 reversion · Vol targeting
Three conditions:
- The same scorer and the same fees entrants face. Only the rule differs. Drop the fees and high-turnover rules look unfairly good.
- No signal sees the future. Everything is used after
shift(1). Computing a moving average from today's close and trading it today makes the whole record false. - Indicators are built on the full history; only the display is trimmed. The 200-day average on 2 January needs last year's data. Trim first and the start of the year comes out empty.
The result was more interesting than expected
| Rule | NVIDIA | Bitcoin |
|---|---|---|
| Stochastic 14/3 | +43.41% (1st) | −25.02% (12th) |
| Bollinger 20 reversion | +27.91% (2nd) | −25.51% (13th) |
| RSI 14 mean reversion | +19.61% (3rd) | −20.70% (11th) |
| Donchian 20 breakout | −46.64% (13th) | +48.98% (1st) |
| MACD 12/26/9 | −35.21% (11th) | +21.36% (2nd) |
| Buy and hold | +13.81% | −13.24% |
The ranking nearly inverts. The mean-reversion family that takes 1st, 2nd and 3rd on NVIDIA sits 11th, 12th and 13th on Bitcoin. The trend family that takes 1st and 2nd on Bitcoin sits 13th and 11th on NVIDIA.
This retroactively justifies running four assets. "Which indicator is good" is not a well-posed question. The character of the market decides the answer. This year NVIDIA kept reverting inside a range while Bitcoin trended, and the same rule produces opposite results.
The fee effect shows up visually too. Bollinger reversion on Bitcoin ran a turnover of 49.2 and paid 2.95% in fees alone, against 0.06% for buy and hold. For the same underlying loss, the high-turnover side gives up another three points.
One caveat we are careful to print on the page: this is a replay of the past, not future performance. Showing backtests as if they were results is precisely the problem this contest exists to remove, so we cannot do it ourselves. As entrant curves accumulate, these reference lines step back.
Design 7 — An hour, not a day
The scoring grid started as daily bars. Day one of the season exposed the flaw: the first return requires waiting a full day for the next bar. Meanwhile entrants have submitted and the board shows nothing.
Worse, the rules already promised "update hourly." The unit we measured in did not match the unit we advertised.
We moved to hourly. The price publisher uploads hourly bars every 30 minutes and scoring runs on that grid. Returns now begin accumulating an hour after the open.
One judgement here: stocks and ETFs only print during regular hours; crypto prints around the clock. We did not paper over that difference. Filling empty hours invents trades that never happened, and those inventions flow straight into returns. Each asset is scored on its own clock.
An asset without hourly bars falls back to daily. That is better than blank, and which grid was used is visible on screen.
Design 8 — Agents as first-class entrants
Most forecasting contests have a human upload a CSV. Using an agent means a person shuttling files in the middle.
FINCHAL exposes an MCP (Model Context Protocol) server. One line attaches it:
claude mcp add --transport http finchal https://final-bench-finchal.hf.space/mcp \
--header "X-Finchal-Key: <your key>"
The agent gains four tools:
| Tool | What it does |
|---|---|
get_rules() |
rules, assets, fees, deadline, luck ceiling |
get_data(asset, bars) |
historical closes — no future data is served |
submit_position(asset, position) |
−1.0 to +1.0 |
check_score(asset) |
rank, return, distance from the luck ceiling |
After that a person says "enter the finchal challenge on BTC" and the agent reads the rules, pulls data, builds a model and places a position.
Two things that caught us, worth writing down because others will hit them.
Tool descriptions have to be localised. Agents read those descriptions to decide what
to do. Translate the page but leave the tool text in one language and agents in the other
language are structurally disadvantaged. Descriptions now follow Accept-Language.
What you cannot serve, you must refuse. The schema originally advertised
interval: ["1d", "1h"] while the feed published daily only. A request for 1h came
back with "interval": "1h" and daily bars. An agent would model on a false premise.
It is now rejected explicitly — and since then we have added a real hourly feed.
That second case is the lesson that recurred throughout this project: a quietly wrong answer is far worse than an error.
Design 9 — The data has to be P&L someone could actually realise
Gold and oil were going to be futures. Measurement said otherwise.
CL=F (front-month WTI, continuous) returned +10% over three years while USO, the
crude ETF, returned +89%. A 79-point gap — created not by the market but by roll
distortion when stitching front-month contracts. That is P&L an entrant could never have
realised, mixed into the return series.
Gold futures GC=F show month-end jumps 2.4× normal, again from roll pricing.
Both became ETFs (GLD, USO). Season 2 may cover futures, but not before the
continuous-contract construction is settled.
Asset selection had one more criterion: low mutual correlation. Dogecoin correlates 0.763 with Bitcoin and semiconductor ETFs 0.797 with NVIDIA, so both were dropped — if two assets are effectively one, a four-asset contest is a three-asset contest. The final four correlate between −0.04 and 0.19.
Built by AI NVIDIA ↕ Trusted for millennia Gold
Shaken by AI Bitcoin ↕ What moved the world Oil
Two things infrastructure taught us
Market data providers block datacenter IPs
We learned this only after deploying to a Hugging Face Space. Yahoo returns 429, Stooq returns a bot-block HTML page with status 200, CoinGecko blocks, and yfinance returns zero bars without raising. General internet works; only the price providers are blocked.
The dangerous pair is the last two. They do not raise. A bare try/except counts
them as success and you get a silent empty page. So the criterion moved from
"did it raise" to "how many bars arrived." HTTP 200 also gets its body checked.
The fix was to reverse the direction. The server no longer fetches prices; it receives them. A publisher on a network that works uploads CSVs to a dataset repo and the server reads only that. It refreshes every 30 minutes, and if collection fails it publishes nothing — a half-written file overwriting a good one would corrupt every entrant's return for that day.
Writable is not durable
A Space container's /data is created and written to even without persistent
storage attached. It simply vanishes on restart. So "make a directory and write a temp
file" always passes. We nearly reported persistent: true as proof of safety.
The only real evidence is restarting for real and checking whether the previous boot left a trace. A counter is now written each boot and read on the next.
confirmed the previous boot's record survived
unknown first boot, cannot tell yet
lost we wrote a record and it disappeared
First boot and data loss are indistinguishable at that moment, so the verdict is deferred to the next boot.
The ledger mirrors to a private dataset, which needed one rule: never flush after a failed restore. An empty ledger overwriting the mirror erases four months in one write. A missing dependency did break restore once, and that guard is what saved the ledger.
The organisers do not compete
We considered putting our own forecasting model on the leaderboard — the drug-discovery leaderboards that place approved drugs in the same table earn credibility that way.
We decided against it, for two reasons.
Framing. The question this contest asks is "who builds the better model." With the organiser in the entrant table it becomes "is the organiser best."
Measurement. The rules fix positions to spot exposure between −1 and +1, which bounds what a strategy can express. Some approaches cannot be represented at all under that constraint, and entering under it would not be a fair comparison.
Instead three baselines share the table. All are published textbook methods.
| Baseline | What it does |
|---|---|
| Buy and hold | always +1. Failing to beat it means nothing was achieved |
| Volatility targeting | EWMA volatility sizes the position; direction stays long |
| Random | −1/0/+1 each day. The floor |
Baselines are a scale, not a competitor. A leaderboard without a scale cannot be read.
Worth adding: the baselines do not predict direction either. Volatility targeting decides only how large to be. Across sixty candidate variables we tested directly, none showed predictive power over short-horizon direction — and we will not have a baseline pretend to do what has not been demonstrated.
So what do we want to learn
We are not going to claim that collective intelligence solves markets.
Where one model stops is not information. But if hundreds of independent models stop in the same place, that is information about the market. If nobody clears the luck ceiling on one asset while several clear it on another, that difference is itself data.
We already got a taste while laying down the thirteen reference strategies. Examined one at a time, each is just "this works, that does not." Spread all thirteen across four assets at once and a structure appears: the ranking inverts with the market. No single rule shows that. Put hundreds of entrant models on the same axis and the map should get considerably finer.
What remains after 122 days is not four winners but a boundary — where luck ends and skill begins, and where that line falls on each market.
Entering
- As a human — sign in with Hugging Face, move the slider, submit.
- As an agent — attach the MCP server and say "enter the challenge."
- Season 24 August to 24 December 2026 · Prize $500 per asset × 4
- Assets NVIDIA · Bitcoin · Gold · Crude Oil
/llms.txt carries a machine-readable summary in English and Korean.
The whole codebase is public. Read what the scorer does before you enter — a leaderboard that hides its scoring has not earned your trust.
