analysis/extra_analysis.md

# Extra analyses for paper revision

Data: 575/124/124 train/valid/test; train vocab 39, test vocab 35, unseen test tools 1.
Gold tools/sample: mean 2.97, median 3.0, range 1-8.
Top-3 train tools: Bash, Read, Edit. Head-only test examples: 44/124; examples with tail tools: 80/124.

## Static frequency floors by k
| k | predicted tools | Jaccard | Exact | Precision | Recall | Top-1 |
|---:|---|---:|---:|---:|---:|---:|
| 1 | Bash | 0.318 | 0.121 | 0.750 | 0.318 | 0.750 |
| 2 | Bash, Read | 0.431 | 0.089 | 0.677 | 0.503 | 0.750 |
| 3 | Bash, Read, Edit | 0.474 | 0.113 | 0.599 | 0.631 | 0.750 |
| 4 | Bash, Read, Edit, Write | 0.406 | 0.016 | 0.478 | 0.658 | 0.750 |
| 5 | Bash, Read, Edit, Write, StructuredOutput | 0.383 | 0.000 | 0.419 | 0.715 | 0.750 |
| 6 | Bash, Read, Edit, Write, StructuredOutput, mcp__ccd_session__mark_chapter | 0.350 | 0.000 | 0.370 | 0.741 | 0.750 |
| 7 | Bash, Read, Edit, Write, StructuredOutput, mcp__ccd_session__mark_chapter, run_shell_command | 0.315 | 0.000 | 0.332 | 0.808 | 0.750 |
| 8 | Bash, Read, Edit, Write, StructuredOutput, mcp__ccd_session__mark_chapter, run_shell_command, replace | 0.281 | 0.000 | 0.295 | 0.822 | 0.750 |

## Additional non-parametric baselines
| Method | Jaccard [95% CI] | Δ vs floor [95% CI] | Top-1 | Exact | Pred tools/sample | Missing | Extra |
|---|---:|---:|---:|---:|---:|---:|---:|
| floor top3 | 0.474 [0.420, 0.528] | — | 0.750 | 0.113 | 3.00 | 1.17 | 1.20 |
| tfidf 1nn | 0.500 [0.445, 0.557] | +0.026 [-0.031, +0.084] | 0.766 | 0.153 | 2.81 | 1.15 | 0.99 |
| tfidf 3nn top3vote | 0.511 [0.458, 0.563] | +0.036 [-0.011, +0.085] | 0.790 | 0.161 | 2.85 | 1.13 | 1.02 |
| tfidf 5nn top3vote | 0.512 [0.459, 0.565] | +0.038 [-0.005, +0.082] | 0.790 | 0.177 | 2.94 | 1.10 | 1.06 |
| tfidf 3nn | 0.505 [0.451, 0.559] | +0.031 [-0.027, +0.088] | 0.790 | 0.169 | 2.74 | 1.13 | 0.90 |
| train labelset oracle | 0.944 [0.921, 0.966] | +0.470 [+0.420, +0.521] | 0.935 | 0.815 | 3.00 | 0.10 | 0.14 |
| data ceiling | 0.996 [0.988, 1.000] | +0.522 [+0.468, +0.575] | 1.000 | 0.992 | 2.96 | 0.01 | 0.00 |
| diffusion lora | 0.447 [0.393, 0.502] | -0.027 [-0.068, +0.013] | 0.750 | 0.105 | 2.88 | 1.27 | 1.18 |
| ar lora | 0.493 [0.432, 0.553] | +0.018 [-0.039, +0.078] | 0.782 | 0.226 | 1.75 | 1.57 | 0.35 |

## Stratified Jaccard

### By freq
| bucket | n | floor | diffusion LoRA | AR LoRA | TF-IDF 1NN |
|---|---:|---:|---:|---:|---:|
| head_only | 44 | 0.652 | 0.540 | 0.769 | 0.559 |
| one_tail | 48 | 0.438 | 0.489 | 0.415 | 0.541 |
| multi_tail | 32 | 0.285 | 0.257 | 0.229 | 0.360 |

### By card
| bucket | n | floor | diffusion LoRA | AR LoRA | TF-IDF 1NN |
|---|---:|---:|---:|---:|---:|
| 1-2 | 50 | 0.308 | 0.298 | 0.533 | 0.398 |
| 3 | 35 | 0.663 | 0.670 | 0.576 | 0.661 |
| 4+ | 39 | 0.517 | 0.438 | 0.366 | 0.487 |

### By length
| bucket | n | floor | diffusion LoRA | AR LoRA | TF-IDF 1NN |
|---|---:|---:|---:|---:|---:|
| short | 42 | 0.246 | 0.202 | 0.201 | 0.269 |
| medium | 41 | 0.563 | 0.517 | 0.589 | 0.526 |
| long | 41 | 0.619 | 0.629 | 0.695 | 0.712 |

## Top test tools and recall
| tool | test count | floor recall | diffusion LoRA | AR LoRA | TF-IDF 1NN |
|---|---:|---:|---:|---:|---:|
| Bash | 93 | 1.00 | 0.85 | 0.92 | 0.91 |
| Read | 75 | 1.00 | 0.89 | 0.73 | 0.84 |
| Edit | 55 | 1.00 | 0.84 | 0.55 | 0.45 |
| StructuredOutput | 23 | 0.00 | 0.30 | 0.00 | 0.91 |
| mcp__ccd_session__mark_chapter | 15 | 0.00 | 0.07 | 0.07 | 0.20 |
| Write | 14 | 0.00 | 0.29 | 0.07 | 0.36 |
| run_shell_command | 13 | 0.00 | 0.31 | 0.00 | 0.62 |
| read_file | 9 | 0.00 | 0.00 | 0.00 | 0.33 |
| Agent | 7 | 0.00 | 0.00 | 0.00 | 0.00 |
| ToolSearch | 7 | 0.00 | 0.00 | 0.00 | 0.29 |
| replace | 5 | 0.00 | 0.20 | 0.00 | 0.60 |
| mcp__playwright__browser_navigate | 5 | 0.00 | 0.00 | 0.00 | 0.20 |
| grep_search | 4 | 0.00 | 0.00 | 0.00 | 0.25 |
| mcp__playwright__browser_take_screenshot | 4 | 0.00 | 0.00 | 0.00 | 0.25 |