Spaces:

FineEnvs
/

ScottWuMath

Running

App Files Files Community

ScottWuMath / README.md

AdithyaSK HF Staff

Rename to Scott Wu Math; add eval results; mobile layout fixes

4de8125 verified 15 days ago

preview code

Raw History Blame Contribute Delete

8.59 kB

---
title: Scott Wu Math
emoji: 🃏
colorFrom: green
colorTo: gray
sdk: docker
app_port: 7860
pinned: false
license: apache-2.0
short_description: Scott Wu's card game as an OpenEnv RL environment
tags:
  - openenv
  - reinforcement-learning
  - math
  - reasoning
---
# Scott Wu Math

Deal some playing cards, name a number, and make that number from the cards with + − × ÷, using every
card exactly once. People and language models play the same environment: the page at `/` drives the
same OpenEnv session over `/ws` that an agent does.

Named after the game Scott Wu plays in the clip below, as a fan tribute. It is not affiliated with or
endorsed by Scott Wu or Cognition.

## Where it comes from

In a clip filmed at a pub, Cognition's CEO Scott Wu explains the game he plays with a deck of cards.
It starts as the 24 game: four cards, make 24. "Once that got too easy", you take eight cards and make
the current year. Dealt K Q J 10 7 9 5 5, he says it is quite easy because 13 − 12 is 1, 11 − 10 is 1,
1 + 1 + 7 is 9, and 9 × 9 × 5 × 5 is exactly 2025. Then someone names a four-digit number, 6,843, and
with nine cards (6 4 Q 10 10 8 J 7 3) he goes: 6 + 4 is 10, 12 − 10 is 2, 2 × 10 × 10 is 200, plus 8 is
208, times 11 is 2,288, minus 7 is 2,281, times 3 is 6,843.

Both hands are in the `video` split, as he played them.

## Rules

- Use every card exactly once. A = 1, J = 11, Q = 12, K = 13.
- Combine with + − × ÷ and parentheses. Division is exact, so fractions along the way are allowed, as in the 24 game.
- No joining cards into longer numbers, no powers, no unary minus.
- One graded answer per deal.

## Levels

| Level | Cards | Target | Notes |
|---|---|---|---|
| `24` | 4 | 24 | The classic 24 game. About a third of random deals have no solution; those are redealt. |
| `three-digit` | 6 | 100–999 | A step between 24 and the year. |
| `year` | 8 | 1900–2100 | The play page and a bare `reset()` use the current year, as in the video. |
| `four-digit` | 9 | 1000–9999 | "Give me your favourite four-digit number". |
| custom | 4–9 | any up to 1,000,000 | `reset(target=6843)`; card count by size unless `n_cards` is given. |

Cards come from one 52-card deck, so a value appears at most four times. Every generated deal is checked
with the solver and redealt until it has a solution, so a task is never impossible.

## Episode

```
reset(...)  ->  observation: cards, target, prompt          (done = false)
step(CardAction(expression="..."))  ->  reward, result, a reference solution   (done = true)
```

`reset` arguments, all optional:

| Argument | Meaning |
|---|---|
| `split`, `index` | A fixed task from `train` (4,000), `test` (200) or `video` (2). Without `index`, a random one. |
| `level` | A fresh deal at that level. |
| `target`, `n_cards` | A fresh deal for your own number. |
| `cards` + `target` | Your own hand, as values 1–13. It may have no solution. |
| `seed` | Makes a fresh deal reproducible. |
| `reward_mode` | `shaped` (default) or `binary`. |

The observation carries `prompt` and `system_prompt`, the exact text a model reads. The answer goes in
`CardAction.expression`; an empty expression gives up. After the step, the observation adds `result`
(the grader's report), `solution` and `solution_steps` (one way to make the target, one combination
per line), and `elapsed_s`. The reference solution is never shown before the episode ends.

## Reward

| Answer | Reward |
|---|---|
| Uses every card once and equals the target | 1.0 |
| Uses every card once, within 10% of the target (shaped mode) | `0.3 × c²`, `c = 1 − |value − target| / (0.1 × target)` |
| Anything else: no answer, parse error, division by zero, a missing or extra card | 0 |

The grader parses the expression with Python's `ast`, allows only non-negative integers, + − × ÷ and
parentheses, compares the multiset of numbers to the cards, and evaluates with exact fractions. × ÷ −
and the letters A J Q K are accepted; anything after `=` is ignored. Partial credit keeps a GRPO group of
near misses from all scoring 0, and is capped far below an exact answer, so getting close never pays
more than getting it right.

## Use it

```python
from scott_wu_math import CardAction, ScottWuMathEnv
with ScottWuMathEnv(base_url="https://fineenvs-scottwumath.hf.space").sync() as env:
    obs = env.reset(split="video", index=1).observation
    print(obs.prompt)
    result = env.step(CardAction(expression="(((12 - 10) * (6 + 4) * 10 + 8) * 11 - 7) * 3"))
    print(result.reward, result.observation.result["check"]["value"])   # 1.0 6843
```

Over plain WebSocket (what the play page does): connect to `/ws`, send
`{"type": "reset", "data": {"level": "four-digit"}}`, then `{"type": "step", "data": {"expression": "..."}}`.

Task discovery follows OpenEnv's Task API: `GET /scott_wu_math/splits`, `POST /scott_wu_math/tasks`
with `{"split": "test"}`, and so on. Task listings leave out the reference solutions.

For text-only RL (no server), `scott_wu_math.game.reward.grade(expression, card_values, target)` and
`extract_answer(completion)` are the same grader as a plain function, and `game.prompts.messages(cards, target)`
builds the same prompt.

## The solver

Every expression tree splits the cards at its root into two groups, and the smaller group has at most
half the cards. The solver enumerates every value a group of up to four cards can make, then asks the
remaining cards for the one number that completes the target (`target = v + rest` needs
`rest = target − v`, and so on for the six ways to place `v`), recursing until four cards remain, where
the answer is a table lookup. That is complete for up to nine cards, and finds a solution for an
eight- or nine-card deal in a few milliseconds. A first pass keeps every intermediate value a whole
non-negative number, the way people play; a second pass allows fractions.

## Run locally

```bash
pip install -e .
uvicorn scott_wu_math.server.app:app --port 8012     # play at http://localhost:8012
python scripts/build_tasks.py                          # rebuild data/tasks (deterministic)
pytest
```

## Evaluations

`eval/rollout.py` in the [FineEnvs repository](https://github.com/adithya-s-k/FineEnvs/tree/main/12-scott-wu-math)
runs OpenAI, Anthropic and open models on Hugging Face Inference Providers against a running server, in
parallel, one OpenEnv session per episode.

### Results (2026-10-23)

Twelve models on the first five `test` deals of each level (20 deals, one sample each) and on the two
`video` hands (three samples each), in three modes:

- **Reasoning**: the model's own reasoning at high effort, up to 32k output tokens.
- **No reasoning**: reasoning switched off, or as low as the API allows (`*`), up to 32k tokens, so the
  model may still work it out in its visible reply.
- **At once**: no reasoning, a system line asking for the answer tag alone, and 300 output tokens: the
  mental-arithmetic version of the game.

| Model | Reasoning | No reasoning | At once | Video hands: reasoning / none / at once |
|---|---|---|---|---|
| GPT-6.1 Sol | 20/20 | 20/20* | 14/20* | 6/6 / 6/6 / 3/6 |
| Claude Opus 5.5 | 20/20 | 19/20* | 8/20* | 6/6 / 6/6 / 5/6 |
| GLM-5.3 | 17/20 | 19/20 | 5/20 | 6/6 / 5/6 / 0/6 |
| Qwen3.8 2.4T | 20/20 | 19/20 | 4/20 | 6/6 / 5/6 / 0/6 |
| Claude Sonnet 5.5 | 20/20 | 15/20 | 4/20 | 6/6 / 4/6 / 0/6 |
| Claude Haiku 5.5 | 20/20 | 11/20 | 3/20 | 6/6 / 5/6 / 0/6 |
| Kimi K2.6 | 19/20 | 20/20 | 2/20 | 6/6 / 6/6 / 0/6 |
| DeepSeek V4 Pro | 20/20 | 10/20 | 2/20 | 6/6 / 3/6 / 0/6 |
| gpt-oss-120b | 16/20 | 6/20* | 2/20* | 6/6 / 1/6 / 0/6 |
| GPT-6 Luna | 20/20 | 2/20 | 2/20 | 5/6 / 0/6 / 0/6 |
| Qwen3.8 27B | 18/20 | 20/20 | 1/20 | 6/6 / 6/6 / 0/6 |
| Gemma 4 31B | 19/20 | 18/20 | 1/20 | 5/6 / 5/6 / 0/6 |

`*` lowest effort the API allows: GPT-6.1 Sol and gpt-oss-120b at `low`, Claude Opus 5.5 at `low` (its
thinking cannot be disabled). Open models ran on Hugging Face Inference Providers (Qwen3.8 2.4T and Kimi
K2.6 on DeepInfra; DeepSeek V4 Pro, GLM-5.3 and Qwen3.8 27B on Novita; gpt-oss-120b on Cerebras; Gemma 4
31B on DeepInfra). Gemma and Kimi each lost one or two episodes to provider errors, counted as failures.

With reasoning the game is close to solved: every model makes 6,843 on the first try. Answering at once
separates them: only GPT-6.1 Sol and Claude Opus 5.5, which still reason briefly at their lowest effort,
get far; most other models start working out loud and run out of room. That gap, and the no-reasoning
column for the smaller models, is where there is something to train.

	---
	title: Scott Wu Math
	emoji: 🃏
	colorFrom: green
	colorTo: gray
	sdk: docker
	app_port: 7860
	pinned: false
	license: apache-2.0
	short_description: Scott Wu's card game as an OpenEnv RL environment
	tags:
	- openenv
	- reinforcement-learning
	- math
	- reasoning
	---

	# Scott Wu Math

	Deal some playing cards, name a number, and make that number from the cards with + − × ÷, using every
	card exactly once. People and language models play the same environment: the page at `/` drives the
	same OpenEnv session over `/ws` that an agent does.

	Named after the game Scott Wu plays in the clip below, as a fan tribute. It is not affiliated with or
	endorsed by Scott Wu or Cognition.

	## Where it comes from

	In a clip filmed at a pub, Cognition's CEO Scott Wu explains the game he plays with a deck of cards.
	It starts as the 24 game: four cards, make 24. "Once that got too easy", you take eight cards and make
	the current year. Dealt K Q J 10 7 9 5 5, he says it is quite easy because 13 − 12 is 1, 11 − 10 is 1,
	1 + 1 + 7 is 9, and 9 × 9 × 5 × 5 is exactly 2025. Then someone names a four-digit number, 6,843, and
	with nine cards (6 4 Q 10 10 8 J 7 3) he goes: 6 + 4 is 10, 12 − 10 is 2, 2 × 10 × 10 is 200, plus 8 is
	208, times 11 is 2,288, minus 7 is 2,281, times 3 is 6,843.

	Both hands are in the `video` split, as he played them.

	## Rules

	- Use every card exactly once. A = 1, J = 11, Q = 12, K = 13.
	- Combine with + − × ÷ and parentheses. Division is exact, so fractions along the way are allowed, as in the 24 game.
	- No joining cards into longer numbers, no powers, no unary minus.
	- One graded answer per deal.

	## Levels

	\| Level \| Cards \| Target \| Notes \|
	\|---\|---\|---\|---\|
	\| `24` \| 4 \| 24 \| The classic 24 game. About a third of random deals have no solution; those are redealt. \|
	\| `three-digit` \| 6 \| 100–999 \| A step between 24 and the year. \|
	\| `year` \| 8 \| 1900–2100 \| The play page and a bare `reset()` use the current year, as in the video. \|
	\| `four-digit` \| 9 \| 1000–9999 \| "Give me your favourite four-digit number". \|
	\| custom \| 4–9 \| any up to 1,000,000 \| `reset(target=6843)`; card count by size unless `n_cards` is given. \|

	Cards come from one 52-card deck, so a value appears at most four times. Every generated deal is checked
	with the solver and redealt until it has a solution, so a task is never impossible.

	## Episode

	```
	reset(...) -> observation: cards, target, prompt (done = false)
	step(CardAction(expression="...")) -> reward, result, a reference solution (done = true)
	```

	`reset` arguments, all optional:

	\| Argument \| Meaning \|
	\|---\|---\|
	\| `split`, `index` \| A fixed task from `train` (4,000), `test` (200) or `video` (2). Without `index`, a random one. \|
	\| `level` \| A fresh deal at that level. \|
	\| `target`, `n_cards` \| A fresh deal for your own number. \|
	\| `cards` + `target` \| Your own hand, as values 1–13. It may have no solution. \|
	\| `seed` \| Makes a fresh deal reproducible. \|
	\| `reward_mode` \| `shaped` (default) or `binary`. \|

	The observation carries `prompt` and `system_prompt`, the exact text a model reads. The answer goes in
	`CardAction.expression`; an empty expression gives up. After the step, the observation adds `result`
	(the grader's report), `solution` and `solution_steps` (one way to make the target, one combination
	per line), and `elapsed_s`. The reference solution is never shown before the episode ends.

	## Reward

	\| Answer \| Reward \|
	\|---\|---\|
	\| Uses every card once and equals the target \| 1.0 \|
	\| Uses every card once, within 10% of the target (shaped mode) \| `0.3 × c²`, `c = 1 − \|value − target\| / (0.1 × target)` \|
	\| Anything else: no answer, parse error, division by zero, a missing or extra card \| 0 \|

	The grader parses the expression with Python's `ast`, allows only non-negative integers, + − × ÷ and
	parentheses, compares the multiset of numbers to the cards, and evaluates with exact fractions. × ÷ −
	and the letters A J Q K are accepted; anything after `=` is ignored. Partial credit keeps a GRPO group of
	near misses from all scoring 0, and is capped far below an exact answer, so getting close never pays
	more than getting it right.

	## Use it

	```python
	from scott_wu_math import CardAction, ScottWuMathEnv

	with ScottWuMathEnv(base_url="https://fineenvs-scottwumath.hf.space").sync() as env:
	obs = env.reset(split="video", index=1).observation
	print(obs.prompt)
	result = env.step(CardAction(expression="(((12 - 10) * (6 + 4) * 10 + 8) * 11 - 7) * 3"))
	print(result.reward, result.observation.result["check"]["value"]) # 1.0 6843
	```

	Over plain WebSocket (what the play page does): connect to `/ws`, send
	`{"type": "reset", "data": {"level": "four-digit"}}`, then `{"type": "step", "data": {"expression": "..."}}`.

	Task discovery follows OpenEnv's Task API: `GET /scott_wu_math/splits`, `POST /scott_wu_math/tasks`
	with `{"split": "test"}`, and so on. Task listings leave out the reference solutions.

	For text-only RL (no server), `scott_wu_math.game.reward.grade(expression, card_values, target)` and
	`extract_answer(completion)` are the same grader as a plain function, and `game.prompts.messages(cards, target)`
	builds the same prompt.

	## The solver

	Every expression tree splits the cards at its root into two groups, and the smaller group has at most
	half the cards. The solver enumerates every value a group of up to four cards can make, then asks the
	remaining cards for the one number that completes the target (`target = v + rest` needs
	`rest = target − v`, and so on for the six ways to place `v`), recursing until four cards remain, where
	the answer is a table lookup. That is complete for up to nine cards, and finds a solution for an
	eight- or nine-card deal in a few milliseconds. A first pass keeps every intermediate value a whole
	non-negative number, the way people play; a second pass allows fractions.

	## Run locally

	```bash
	pip install -e .
	uvicorn scott_wu_math.server.app:app --port 8012 # play at http://localhost:8012
	python scripts/build_tasks.py # rebuild data/tasks (deterministic)
	pytest
	```

	## Evaluations

	`eval/rollout.py` in the [FineEnvs repository](https://github.com/adithya-s-k/FineEnvs/tree/main/12-scott-wu-math)
	runs OpenAI, Anthropic and open models on Hugging Face Inference Providers against a running server, in
	parallel, one OpenEnv session per episode.

	### Results (2026-10-23)

	Twelve models on the first five `test` deals of each level (20 deals, one sample each) and on the two
	`video` hands (three samples each), in three modes:

	- Reasoning: the model's own reasoning at high effort, up to 32k output tokens.
	- No reasoning: reasoning switched off, or as low as the API allows (`*`), up to 32k tokens, so the
	model may still work it out in its visible reply.
	- At once: no reasoning, a system line asking for the answer tag alone, and 300 output tokens: the
	mental-arithmetic version of the game.

	\| Model \| Reasoning \| No reasoning \| At once \| Video hands: reasoning / none / at once \|
	\|---\|---\|---\|---\|---\|
	\| GPT-6.1 Sol \| 20/20 \| 20/20* \| 14/20* \| 6/6 / 6/6 / 3/6 \|
	\| Claude Opus 5.5 \| 20/20 \| 19/20* \| 8/20* \| 6/6 / 6/6 / 5/6 \|
	\| GLM-5.3 \| 17/20 \| 19/20 \| 5/20 \| 6/6 / 5/6 / 0/6 \|
	\| Qwen3.8 2.4T \| 20/20 \| 19/20 \| 4/20 \| 6/6 / 5/6 / 0/6 \|
	\| Claude Sonnet 5.5 \| 20/20 \| 15/20 \| 4/20 \| 6/6 / 4/6 / 0/6 \|
	\| Claude Haiku 5.5 \| 20/20 \| 11/20 \| 3/20 \| 6/6 / 5/6 / 0/6 \|
	\| Kimi K2.6 \| 19/20 \| 20/20 \| 2/20 \| 6/6 / 6/6 / 0/6 \|
	\| DeepSeek V4 Pro \| 20/20 \| 10/20 \| 2/20 \| 6/6 / 3/6 / 0/6 \|
	\| gpt-oss-120b \| 16/20 \| 6/20* \| 2/20* \| 6/6 / 1/6 / 0/6 \|
	\| GPT-6 Luna \| 20/20 \| 2/20 \| 2/20 \| 5/6 / 0/6 / 0/6 \|
	\| Qwen3.8 27B \| 18/20 \| 20/20 \| 1/20 \| 6/6 / 6/6 / 0/6 \|
	\| Gemma 4 31B \| 19/20 \| 18/20 \| 1/20 \| 5/6 / 5/6 / 0/6 \|

	`*` lowest effort the API allows: GPT-6.1 Sol and gpt-oss-120b at `low`, Claude Opus 5.5 at `low` (its
	thinking cannot be disabled). Open models ran on Hugging Face Inference Providers (Qwen3.8 2.4T and Kimi
	K2.6 on DeepInfra; DeepSeek V4 Pro, GLM-5.3 and Qwen3.8 27B on Novita; gpt-oss-120b on Cerebras; Gemma 4
	31B on DeepInfra). Gemma and Kimi each lost one or two episodes to provider errors, counted as failures.

	With reasoning the game is close to solved: every model makes 6,843 on the first try. Answering at once
	separates them: only GPT-6.1 Sol and Claude Opus 5.5, which still reason briefly at their lowest effort,
	get far; most other models start working out loud and run out of room. That gap, and the no-reasoning
	column for the smaller models, is where there is something to train.