analysis/extra_analysis.md

Extra analyses for paper revision

Data: 575/124/124 train/valid/test; train vocab 39, test vocab 35, unseen test tools 1. Gold tools/sample: mean 2.97, median 3.0, range 1-8. Top-3 train tools: Bash, Read, Edit. Head-only test examples: 44/124; examples with tail tools: 80/124.

Static frequency floors by k

k	predicted tools	Jaccard	Exact	Precision	Recall	Top-1
1	Bash	0.318	0.121	0.750	0.318	0.750
2	Bash, Read	0.431	0.089	0.677	0.503	0.750
3	Bash, Read, Edit	0.474	0.113	0.599	0.631	0.750
4	Bash, Read, Edit, Write	0.406	0.016	0.478	0.658	0.750
5	Bash, Read, Edit, Write, StructuredOutput	0.383	0.000	0.419	0.715	0.750
6	Bash, Read, Edit, Write, StructuredOutput, mcp__ccd_session__mark_chapter	0.350	0.000	0.370	0.741	0.750
7	Bash, Read, Edit, Write, StructuredOutput, mcp__ccd_session__mark_chapter, run_shell_command	0.315	0.000	0.332	0.808	0.750
8	Bash, Read, Edit, Write, StructuredOutput, mcp__ccd_session__mark_chapter, run_shell_command, replace	0.281	0.000	0.295	0.822	0.750

Additional non-parametric baselines

Method	Jaccard [95% CI]	Δ vs floor [95% CI]	Top-1	Exact	Pred tools/sample	Missing	Extra
floor top3	0.474 [0.420, 0.528]	—	0.750	0.113	3.00	1.17	1.20
tfidf 1nn	0.500 [0.445, 0.557]	+0.026 [-0.031, +0.084]	0.766	0.153	2.81	1.15	0.99
tfidf 3nn top3vote	0.511 [0.458, 0.563]	+0.036 [-0.011, +0.085]	0.790	0.161	2.85	1.13	1.02
tfidf 5nn top3vote	0.512 [0.459, 0.565]	+0.038 [-0.005, +0.082]	0.790	0.177	2.94	1.10	1.06
tfidf 3nn	0.505 [0.451, 0.559]	+0.031 [-0.027, +0.088]	0.790	0.169	2.74	1.13	0.90
train labelset oracle	0.944 [0.921, 0.966]	+0.470 [+0.420, +0.521]	0.935	0.815	3.00	0.10	0.14
data ceiling	0.996 [0.988, 1.000]	+0.522 [+0.468, +0.575]	1.000	0.992	2.96	0.01	0.00
diffusion lora	0.447 [0.393, 0.502]	-0.027 [-0.068, +0.013]	0.750	0.105	2.88	1.27	1.18
ar lora	0.493 [0.432, 0.553]	+0.018 [-0.039, +0.078]	0.782	0.226	1.75	1.57	0.35

Stratified Jaccard

By freq

bucket	n	floor	diffusion LoRA	AR LoRA	TF-IDF 1NN
head_only	44	0.652	0.540	0.769	0.559
one_tail	48	0.438	0.489	0.415	0.541
multi_tail	32	0.285	0.257	0.229	0.360

By card

bucket	n	floor	diffusion LoRA	AR LoRA	TF-IDF 1NN
1-2	50	0.308	0.298	0.533	0.398
3	35	0.663	0.670	0.576	0.661
4+	39	0.517	0.438	0.366	0.487

By length

bucket	n	floor	diffusion LoRA	AR LoRA	TF-IDF 1NN
short	42	0.246	0.202	0.201	0.269
medium	41	0.563	0.517	0.589	0.526
long	41	0.619	0.629	0.695	0.712

Top test tools and recall

tool	test count	floor recall	diffusion LoRA	AR LoRA	TF-IDF 1NN
Bash	93	1.00	0.85	0.92	0.91
Read	75	1.00	0.89	0.73	0.84
Edit	55	1.00	0.84	0.55	0.45
StructuredOutput	23	0.00	0.30	0.00	0.91
mcp__ccd_session__mark_chapter	15	0.00	0.07	0.07	0.20
Write	14	0.00	0.29	0.07	0.36
run_shell_command	13	0.00	0.31	0.00	0.62
read_file	9	0.00	0.00	0.00	0.33
Agent	7	0.00	0.00	0.00	0.00
ToolSearch	7	0.00	0.00	0.00	0.29
replace	5	0.00	0.20	0.00	0.60
mcp__playwright__browser_navigate	5	0.00	0.00	0.00	0.20
grep_search	4	0.00	0.00	0.00	0.25
mcp__playwright__browser_take_screenshot	4	0.00	0.00	0.00	0.25