Buckets:

Mercity
/

Desicion-Model-Custom

Files

xet

Mercity/Desicion-Model-Custom / run1 /logs /report.log

Pranav2748

8 days ago

download

raw

5.52 kB

# Reasoning decision model: results

## Hypotheses

| # | Result | Numbers | Confirmed |
|---|---|---|---|
| H1 | joint readout vs gold-only, in-distribution accuracy | +0.1 pts [-2.0, +2.2] | no |
| H2 | disagreement AUROC, joint vs others (need +0.03) | vs gold_only +0.012 [-0.002, +0.023]; vs base +0.011 [-0.003, +0.024] | no |
| H3 | joint reasoner vs base reasoner, all eval | +0.0 pts [-0.8, +0.7] | yes |
| H4 | gap recovered at f = 0.3 (need 80%) | 52.7% of 69.5 → 84.7; learned router 51.3% | no |

## Readout

Accuracy (%) after per-type temperatures fitted on the calibration set; ECE uses 15 bins.

| Readout | Procedural | tasksource | OpenDecisions | LocalLLaMA | LocalLLaMA (panel agrees) | OD Capability | OD Capability (raw) | In-dist ECE | In-dist NLL |
|---|---|---|---|---|---|---|---|---|---|
| base | 46.8 | 60.4 | 66.5 | 63.0 | 74.1 | 61.8 | 55.2 | 0.042 | 1.027 |
| gold_only | 72.4 | 65.8 | 72.1 | 67.5 | 76.7 | 66.0 | 65.1 | 0.032 | 0.774 |
| votes_only | 71.7 | 66.3 | 71.9 | 66.3 | 75.1 | 66.4 | 62.3 | 0.041 | 0.780 |
| joint | 71.9 | 66.4 | 72.2 | 66.5 | 76.3 | 66.7 | 64.6 | 0.031 | 0.757 |

Temperatures (choice / yes-no / score): base 1.71 / 2.41 / 1.28; gold_only 1.18 / 0.87 / 1.02; votes_only 1.60 / 1.08 / 1.48; joint 1.28 / 1.18 / 1.05

## Reasoning (one sample per item)

| Reasoner | Procedural | tasksource | OpenDecisions | LocalLLaMA | All | Median tokens | p90 tokens | Looped | Letter = readout after chain |
|---|---|---|---|---|---|---|---|---|---|
| base | 95.1 | 73.3 | 94.9 | 71.0 | 83.9 | 1626 | 5737 | 1.0% | 98.9% |
| joint | 97.1 | 74.3 | 94.1 | 71.0 | 83.9 | 1412 | 5497 | 1.7% | 98.3% |

## Does the readout agree with its reasoning?

Agreement = the readout's answer equals the same model's thinking-mode answer (the base reasoner for the readout-only ablations). The thinking call never sees the readout's answer, so this is also the faithfulness check of explanation on demand.

| Readout | Agreement | Procedural | tasksource | OpenDecisions | LocalLLaMA | Disagreement AUROC | Fixable AUROC |
|---|---|---|---|---|---|---|---|
| base | 66.8 | 46.8 | 67.9 | 67.8 | 70.6 | 0.778 | 0.580 |
| gold_only | 71.6 | 71.6 | 72.8 | 73.3 | 69.0 | 0.777 | 0.489 |
| votes_only | 73.1 | 71.4 | 76.2 | 72.9 | 72.7 | 0.803 | 0.562 |
| joint | 74.4 | 72.2 | 75.8 | 74.1 | 75.0 | 0.789 | 0.526 |

## Routing

Route the lowest-margin share f of items to thinking mode (all eval sets pooled).

| Readout | Router | f=0 | 0.1 | 0.2 | 0.3 | 0.5 | 1.0 | tokens at 0.3 | random at 0.3 |
|---|---|---|---|---|---|---|---|---|---|
| base | margin | 62.7 | 66.9 | 70.8 | 74.4 | 79.3 | 84.3 | 1095 | 69.2 |
| base | learned | 62.7 | 67.3 | 71.1 | 74.5 | 79.5 | 84.3 | 1138 | 69.2 |
| gold_only | margin | 69.8 | 73.1 | 75.3 | 77.5 | 80.4 | 84.4 | 1134 | 74.2 |
| gold_only | learned | 69.8 | 72.7 | 75.1 | 76.8 | 80.4 | 84.4 | 1181 | 74.2 |
| votes_only | margin | 69.2 | 72.6 | 75.7 | 78.1 | 81.7 | 84.4 | 1177 | 73.8 |
| votes_only | learned | 69.2 | 72.9 | 75.5 | 78.2 | 81.7 | 84.4 | 1204 | 73.8 |
| joint | margin | 69.5 | 72.5 | 75.4 | 77.5 | 80.9 | 84.7 | 1244 | 74.0 |
| joint | learned | 69.5 | 73.0 | 75.0 | 77.3 | 81.1 | 84.7 | 1275 | 74.0 |

## The deployable system

Readout by default; items with the lowest readout margin go to thinking mode, whose probabilities are the readout after the chain (temperatures fitted on calibration chains only). All eval sets; OpenDecisions Capability with the official scorer.

| Model | Mode | Routed | Procedural | tasksource | OpenDecisions | LocalLLaMA | All | OD Capability | Mean generated tokens |
|---|---|---|---|---|---|---|---|---|---|
| base | readout only | 0.0% | 46.8 | 60.4 | 66.5 | 63.0 | 62.7 | 61.8 | 0 |
| base | routed | 10.0% | 58.6 | 64.3 | 71.1 | 65.0 | 66.9 | 67.2 | 367 |
| base | routed | 20.0% | 67.3 | 66.1 | 76.0 | 67.1 | 70.9 | 72.3 | 730 |
| base | routed | 30.0% | 79.3 | 68.1 | 80.4 | 68.1 | 74.5 | 76.5 | 1095 |
| base | routed | 50.0% | 90.9 | 71.3 | 87.2 | 69.6 | 79.5 | 82.8 | 1718 |
| base | thinking always | 100.0% | 96.0 | 74.0 | 95.4 | 71.7 | 84.5 | 84.4 | 2823 |
| joint | readout only | 0.0% | 71.9 | 66.4 | 72.2 | 66.5 | 69.5 | 66.7 | 0 |
| joint | routed | 10.0% | 77.8 | 68.3 | 75.9 | 68.1 | 72.5 | 69.8 | 429 |
| joint | routed | 20.0% | 82.1 | 70.1 | 79.9 | 69.8 | 75.4 | 73.6 | 829 |
| joint | routed | 30.0% | 86.5 | 71.5 | 82.9 | 70.4 | 77.5 | 77.1 | 1244 |
| joint | routed | 50.0% | 92.0 | 73.8 | 87.8 | 71.9 | 80.9 | 83.2 | 1946 |
| joint | thinking always | 100.0% | 97.5 | 74.5 | 94.6 | 72.5 | 84.7 | 88.1 | 2831 |

Post-chain temperatures (choice / yes-no / score): base 4.77 / 3.49 / 4.02; joint 4.08 / 2.94 / 3.65

## Latency (batch size 1)

| Model | Readout median | Readout p90 | Reasoning median | Reasoning p90 |
|---|---|---|---|---|
| base | 43 ms | 76 ms | 10.1 s | 41.8 s |
| joint | 41 ms | 74 ms | 11.6 s | 55.6 s |

## Run

- Throughput probe: 1957 generated tokens/s; chain tokens median 1878, p90 6414, max 32471; parse rate 99.2%.
- Generation: 3169 items, 12676 chains, 41.0M tokens; 2818 items with a correct chain; 154 chains looped to the context limit.
- Training joint: 2673 reasoner + 2996 readout examples, 8.7M tokens, 207/207 steps, 46 min (3137 tokens/s).
- Training gold_only: 0 reasoner + 3169 readout examples, 1.7M tokens, 37/37 steps, 11 min (2521 tokens/s).
- Training votes_only: 0 reasoner + 3169 readout examples, 1.7M tokens, 37/37 steps, 7 min (3898 tokens/s).
- Single seed (17); see results/DEVIATIONS.md for every change to the plan.

Xet Storage Details

Size:: 5.52 kB
Xet hash:: 8d60dfb8f96098e948dbb8ce4d4a364d0eb3a2537f03213c079a3d199053fb38

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.

	# Reasoning decision model: results

	## Hypotheses

	\| # \| Result \| Numbers \| Confirmed \|
	\|---\|---\|---\|---\|
	\| H1 \| joint readout vs gold-only, in-distribution accuracy \| +0.1 pts [-2.0, +2.2] \| no \|
	\| H2 \| disagreement AUROC, joint vs others (need +0.03) \| vs gold_only +0.012 [-0.002, +0.023]; vs base +0.011 [-0.003, +0.024] \| no \|
	\| H3 \| joint reasoner vs base reasoner, all eval \| +0.0 pts [-0.8, +0.7] \| yes \|
	\| H4 \| gap recovered at f = 0.3 (need 80%) \| 52.7% of 69.5 → 84.7; learned router 51.3% \| no \|

	## Readout

	Accuracy (%) after per-type temperatures fitted on the calibration set; ECE uses 15 bins.

	\| Readout \| Procedural \| tasksource \| OpenDecisions \| LocalLLaMA \| LocalLLaMA (panel agrees) \| OD Capability \| OD Capability (raw) \| In-dist ECE \| In-dist NLL \|
	\|---\|---\|---\|---\|---\|---\|---\|---\|---\|---\|
	\| base \| 46.8 \| 60.4 \| 66.5 \| 63.0 \| 74.1 \| 61.8 \| 55.2 \| 0.042 \| 1.027 \|
	\| gold_only \| 72.4 \| 65.8 \| 72.1 \| 67.5 \| 76.7 \| 66.0 \| 65.1 \| 0.032 \| 0.774 \|
	\| votes_only \| 71.7 \| 66.3 \| 71.9 \| 66.3 \| 75.1 \| 66.4 \| 62.3 \| 0.041 \| 0.780 \|
	\| joint \| 71.9 \| 66.4 \| 72.2 \| 66.5 \| 76.3 \| 66.7 \| 64.6 \| 0.031 \| 0.757 \|

	Temperatures (choice / yes-no / score): base 1.71 / 2.41 / 1.28; gold_only 1.18 / 0.87 / 1.02; votes_only 1.60 / 1.08 / 1.48; joint 1.28 / 1.18 / 1.05

	## Reasoning (one sample per item)

	\| Reasoner \| Procedural \| tasksource \| OpenDecisions \| LocalLLaMA \| All \| Median tokens \| p90 tokens \| Looped \| Letter = readout after chain \|
	\|---\|---\|---\|---\|---\|---\|---\|---\|---\|---\|
	\| base \| 95.1 \| 73.3 \| 94.9 \| 71.0 \| 83.9 \| 1626 \| 5737 \| 1.0% \| 98.9% \|
	\| joint \| 97.1 \| 74.3 \| 94.1 \| 71.0 \| 83.9 \| 1412 \| 5497 \| 1.7% \| 98.3% \|

	## Does the readout agree with its reasoning?

	Agreement = the readout's answer equals the same model's thinking-mode answer (the base reasoner for the readout-only ablations). The thinking call never sees the readout's answer, so this is also the faithfulness check of explanation on demand.

	\| Readout \| Agreement \| Procedural \| tasksource \| OpenDecisions \| LocalLLaMA \| Disagreement AUROC \| Fixable AUROC \|
	\|---\|---\|---\|---\|---\|---\|---\|---\|
	\| base \| 66.8 \| 46.8 \| 67.9 \| 67.8 \| 70.6 \| 0.778 \| 0.580 \|
	\| gold_only \| 71.6 \| 71.6 \| 72.8 \| 73.3 \| 69.0 \| 0.777 \| 0.489 \|
	\| votes_only \| 73.1 \| 71.4 \| 76.2 \| 72.9 \| 72.7 \| 0.803 \| 0.562 \|
	\| joint \| 74.4 \| 72.2 \| 75.8 \| 74.1 \| 75.0 \| 0.789 \| 0.526 \|

	## Routing

	Route the lowest-margin share f of items to thinking mode (all eval sets pooled).

	\| Readout \| Router \| f=0 \| 0.1 \| 0.2 \| 0.3 \| 0.5 \| 1.0 \| tokens at 0.3 \| random at 0.3 \|
	\|---\|---\|---\|---\|---\|---\|---\|---\|---\|---\|
	\| base \| margin \| 62.7 \| 66.9 \| 70.8 \| 74.4 \| 79.3 \| 84.3 \| 1095 \| 69.2 \|
	\| base \| learned \| 62.7 \| 67.3 \| 71.1 \| 74.5 \| 79.5 \| 84.3 \| 1138 \| 69.2 \|
	\| gold_only \| margin \| 69.8 \| 73.1 \| 75.3 \| 77.5 \| 80.4 \| 84.4 \| 1134 \| 74.2 \|
	\| gold_only \| learned \| 69.8 \| 72.7 \| 75.1 \| 76.8 \| 80.4 \| 84.4 \| 1181 \| 74.2 \|
	\| votes_only \| margin \| 69.2 \| 72.6 \| 75.7 \| 78.1 \| 81.7 \| 84.4 \| 1177 \| 73.8 \|
	\| votes_only \| learned \| 69.2 \| 72.9 \| 75.5 \| 78.2 \| 81.7 \| 84.4 \| 1204 \| 73.8 \|
	\| joint \| margin \| 69.5 \| 72.5 \| 75.4 \| 77.5 \| 80.9 \| 84.7 \| 1244 \| 74.0 \|
	\| joint \| learned \| 69.5 \| 73.0 \| 75.0 \| 77.3 \| 81.1 \| 84.7 \| 1275 \| 74.0 \|

	## The deployable system

	Readout by default; items with the lowest readout margin go to thinking mode, whose probabilities are the readout after the chain (temperatures fitted on calibration chains only). All eval sets; OpenDecisions Capability with the official scorer.

	\| Model \| Mode \| Routed \| Procedural \| tasksource \| OpenDecisions \| LocalLLaMA \| All \| OD Capability \| Mean generated tokens \|
	\|---\|---\|---\|---\|---\|---\|---\|---\|---\|---\|
	\| base \| readout only \| 0.0% \| 46.8 \| 60.4 \| 66.5 \| 63.0 \| 62.7 \| 61.8 \| 0 \|
	\| base \| routed \| 10.0% \| 58.6 \| 64.3 \| 71.1 \| 65.0 \| 66.9 \| 67.2 \| 367 \|
	\| base \| routed \| 20.0% \| 67.3 \| 66.1 \| 76.0 \| 67.1 \| 70.9 \| 72.3 \| 730 \|
	\| base \| routed \| 30.0% \| 79.3 \| 68.1 \| 80.4 \| 68.1 \| 74.5 \| 76.5 \| 1095 \|
	\| base \| routed \| 50.0% \| 90.9 \| 71.3 \| 87.2 \| 69.6 \| 79.5 \| 82.8 \| 1718 \|
	\| base \| thinking always \| 100.0% \| 96.0 \| 74.0 \| 95.4 \| 71.7 \| 84.5 \| 84.4 \| 2823 \|
	\| joint \| readout only \| 0.0% \| 71.9 \| 66.4 \| 72.2 \| 66.5 \| 69.5 \| 66.7 \| 0 \|
	\| joint \| routed \| 10.0% \| 77.8 \| 68.3 \| 75.9 \| 68.1 \| 72.5 \| 69.8 \| 429 \|
	\| joint \| routed \| 20.0% \| 82.1 \| 70.1 \| 79.9 \| 69.8 \| 75.4 \| 73.6 \| 829 \|
	\| joint \| routed \| 30.0% \| 86.5 \| 71.5 \| 82.9 \| 70.4 \| 77.5 \| 77.1 \| 1244 \|
	\| joint \| routed \| 50.0% \| 92.0 \| 73.8 \| 87.8 \| 71.9 \| 80.9 \| 83.2 \| 1946 \|
	\| joint \| thinking always \| 100.0% \| 97.5 \| 74.5 \| 94.6 \| 72.5 \| 84.7 \| 88.1 \| 2831 \|

	Post-chain temperatures (choice / yes-no / score): base 4.77 / 3.49 / 4.02; joint 4.08 / 2.94 / 3.65

	## Latency (batch size 1)

	\| Model \| Readout median \| Readout p90 \| Reasoning median \| Reasoning p90 \|
	\|---\|---\|---\|---\|---\|
	\| base \| 43 ms \| 76 ms \| 10.1 s \| 41.8 s \|
	\| joint \| 41 ms \| 74 ms \| 11.6 s \| 55.6 s \|

	## Run

	- Throughput probe: 1957 generated tokens/s; chain tokens median 1878, p90 6414, max 32471; parse rate 99.2%.
	- Generation: 3169 items, 12676 chains, 41.0M tokens; 2818 items with a correct chain; 154 chains looped to the context limit.
	- Training joint: 2673 reasoner + 2996 readout examples, 8.7M tokens, 207/207 steps, 46 min (3137 tokens/s).
	- Training gold_only: 0 reasoner + 3169 readout examples, 1.7M tokens, 37/37 steps, 11 min (2521 tokens/s).
	- Training votes_only: 0 reasoner + 3169 readout examples, 1.7M tokens, 37/37 steps, 7 min (3898 tokens/s).
	- Single seed (17); see results/DEVIATIONS.md for every change to the plan.