docs/_generated/stats_report.md

Statistics report

Headline intervals: fastapi

config	metric	mean	95 percent interval (primary)	naive SE	clustered SE	n_clusters	design effect
custom-anthropic+0bc9cd53	p_at_5	0.791	[0.664, 0.917] (clustered)	0.0566	0.0645	n_clusters=12	design effect=1.30
custom-anthropic+0bc9cd53	r_at_5	0.841	[0.710, 0.971] (clustered)	0.0662	0.0666	n_clusters=12	design effect=1.01
custom-openai+470d79fa	p_at_5	0.718	[0.610, 0.827] (clustered)	0.0667	0.0554	n_clusters=12	design effect=0.69
custom-openai+470d79fa	r_at_5	0.833	[0.715, 0.951] (clustered)	0.0719	0.0602	n_clusters=12	design effect=0.70
langchain-anthropic+claude-haiku-4-5-20251001	p_at_5	0.760	[0.645, 0.875] (clustered)	0.0606	0.0589	n_clusters=12	design effect=0.94
langchain-anthropic+claude-haiku-4-5-20251001	r_at_5	0.841	[0.710, 0.971] (clustered)	0.0662	0.0666	n_clusters=12	design effect=1.01
langchain-openai+gpt-4o-mini	p_at_5	0.627	[0.484, 0.770] (clustered)	0.0683	0.0729	n_clusters=12	design effect=1.14
langchain-openai+gpt-4o-mini	r_at_5	0.836	[0.702, 0.971] (clustered)	0.0716	0.0685	n_clusters=12	design effect=0.91

Citation accuracy zero-failure bound: fastapi

• custom-anthropic+0bc9cd53: 0 failures in 21 included questions (included n=21, excluded 1 citation-free questions; cited in all epochs: 21, in some epochs: 0). Exact Clopper-Pearson 95 percent upper bound on the per-question failure rate: 0.133 (rule of three approximation 3/n = 0.143).
• custom-openai+470d79fa: 0 failures in 19 included questions (included n=19, excluded 3 citation-free questions; cited in all epochs: 19, in some epochs: 0). Exact Clopper-Pearson 95 percent upper bound on the per-question failure rate: 0.146 (rule of three approximation 3/n = 0.158).
• langchain-anthropic+claude-haiku-4-5-20251001: 0 failures in 20 included questions (included n=20, excluded 2 citation-free questions; cited in all epochs: 20, in some epochs: 0). Exact Clopper-Pearson 95 percent upper bound on the per-question failure rate: 0.139 (rule of three approximation 3/n = 0.150).
• langchain-openai+gpt-4o-mini: 0 failures in 20 included questions (included n=20, excluded 2 citation-free questions; cited in all epochs: 19, in some epochs: 1). Exact Clopper-Pearson 95 percent upper bound on the per-question failure rate: 0.139 (rule of three approximation 3/n = 0.150).

Framework equivalence (TOST): fastapi

• custom-anthropic+0bc9cd53 vs langchain-anthropic+claude-haiku-4-5-20251001, p_at_5: mean diff +0.031, 90 percent CI [-0.013, +0.076], n=12: equivalent within plus or minus 0.10; the data support equivalence down to plus or minus 0.076.
• custom-anthropic+0bc9cd53 vs langchain-anthropic+claude-haiku-4-5-20251001, r_at_5: mean diff +0.000, 90 percent CI [+0.000, +0.000], n=12: equivalent within plus or minus 0.10; the data support equivalence down to plus or minus 0.000.
• custom-anthropic+0bc9cd53 vs langchain-openai+gpt-4o-mini, p_at_5: mean diff +0.164, 90 percent CI [+0.031, +0.317], n=12: equivalence not established at plus or minus 0.10; the data support only plus or minus 0.317.
• custom-anthropic+0bc9cd53 vs langchain-openai+gpt-4o-mini, r_at_5: mean diff +0.005, 90 percent CI [-0.076, +0.095], n=12: equivalent within plus or minus 0.10; the data support equivalence down to plus or minus 0.095.
• custom-openai+470d79fa vs langchain-anthropic+claude-haiku-4-5-20251001, p_at_5: mean diff -0.042, 90 percent CI [-0.105, +0.016], n=12: equivalence not established at plus or minus 0.10; the data support only plus or minus 0.105.
• custom-openai+470d79fa vs langchain-anthropic+claude-haiku-4-5-20251001, r_at_5: mean diff -0.008, 90 percent CI [-0.100, +0.068], n=12: equivalence not established at plus or minus 0.10; the data support only plus or minus 0.100.
• custom-openai+470d79fa vs langchain-openai+gpt-4o-mini, p_at_5: mean diff +0.091, 90 percent CI [-0.010, +0.202], n=12: equivalence not established at plus or minus 0.10; the data support only plus or minus 0.202.
• custom-openai+470d79fa vs langchain-openai+gpt-4o-mini, r_at_5: mean diff -0.003, 90 percent CI [-0.043, +0.032], n=12: equivalent within plus or minus 0.10; the data support equivalence down to plus or minus 0.043.

Variance decomposition and power: fastapi

• p_at_5 variance: between-question 0.06051, within-question 0.00036, ICC 0.99 (22 questions x 5 epochs).
• Error budget preview: the interval above is the statistical term only; template sensitivity and judge bias are systematic terms, scoped for v3.2.
• Minimum detectable p_at_5 difference at 80 percent power: 0.110 (normal approximation 0.136).

Refusal reliability (pass^k): fastapi

config	k	single-run	pass^k	95 percent interval	n_questions
custom-anthropic+0bc9cd53	5	0.560	0.400	[0.053, 0.853]	5
custom-openai+470d79fa	5	0.640	0.600	[0.147, 0.947]	5
langchain-anthropic+claude-haiku-4-5-20251001	5	0.200	0.200	[0.005, 0.716]	5
langchain-openai+gpt-4o-mini	5	0.760	0.600	[0.147, 0.947]	5

Headline intervals: k8s

config	metric	mean	95 percent interval (primary)	naive SE	clustered SE	n_clusters	design effect
custom-anthropic-k8s+0bc9cd53	p_at_5	0.823	[0.736, 0.910] (question-level)	0.0444	0.0510	n_clusters=6	design effect=1.32
custom-anthropic-k8s+0bc9cd53	r_at_5	0.943	[0.881, 1.000] (question-level)	0.0319	0.0537	n_clusters=6	design effect=2.82
custom-openai-k8s+470d79fa	p_at_5	0.814	[0.734, 0.894] (question-level)	0.0408	0.0504	n_clusters=6	design effect=1.53
custom-openai-k8s+470d79fa	r_at_5	0.957	[0.898, 1.000] (question-level)	0.0300	0.0413	n_clusters=6	design effect=1.89

ceiling-censored: k8s custom-anthropic-k8s+0bc9cd53 r_at_5: the normal-approximation interval extends past the [0,1] proportion bound and is reported clamped to it; the bound sits on the ceiling rather than estimating beyond it. correlation-sensitivity caution: k8s custom-anthropic-k8s+0bc9cd53 r_at_5: clustered SE 0.0537 exceeds 1.5x the question-level SE 0.0319; the question-level headline interval likely understates uncertainty. ceiling-censored: k8s custom-openai-k8s+470d79fa r_at_5: the normal-approximation interval extends past the [0,1] proportion bound and is reported clamped to it; the bound sits on the ceiling rather than estimating beyond it.

Citation accuracy zero-failure bound: k8s

• custom-anthropic-k8s+0bc9cd53: 0 failures in 22 included questions (included n=22, excluded 1 citation-free questions; cited in all epochs: 20, in some epochs: 2). Exact Clopper-Pearson 95 percent upper bound on the per-question failure rate: 0.127 (rule of three approximation 3/n = 0.136).
• custom-openai-k8s+470d79fa: 0 failures in 22 included questions (included n=22, excluded 1 citation-free questions; cited in all epochs: 21, in some epochs: 1). Exact Clopper-Pearson 95 percent upper bound on the per-question failure rate: 0.127 (rule of three approximation 3/n = 0.136).

Framework equivalence (TOST): k8s

Variance decomposition and power: k8s

• p_at_5 variance: between-question 0.03249, within-question 0.00209, ICC 0.94 (23 questions x 5 epochs).
• Error budget preview: the interval above is the statistical term only; template sensitivity and judge bias are systematic terms, scoped for v3.2.
• Minimum detectable p_at_5 difference at 80 percent power: 0.119 (normal approximation 0.110).

Refusal reliability (pass^k): k8s

config	k	single-run	pass^k	95 percent interval	n_questions
custom-anthropic-k8s+0bc9cd53	5	0.800	0.500	[0.013, 0.987]	2
custom-openai-k8s+470d79fa	5	1.000	1.000	[0.158, 1.000]	2

Methods appendix

• Estimators: cluster bootstrap over cluster_id (10000 replicates); paired bootstrap on per-question epoch-mean differences; TOST at margin 0.10 absolute, alpha 0.05 per one-sided test, no multiplicity adjustment across P@5 and R@5 (pre-registered, design spec section 2, frozen 2026-06-11 before any WP5 data existed).
• Zero-failure bounding: a question succeeds only if zero hallucinated citations occurred across all epochs and all citations; collapsing epochs bounds the any-of-k failure rate, which also bounds the per-answer rate.
• Seed: 20260611. No wall-clock values appear in this report.
• Input table fastapi: 1878 rows, content hash c28484520117.
• Input table k8s: 925 rows, content hash 986cda71bb39.

README values

• fastapi_custom_anthropic_p_at_5_mean = 0.791
• fastapi_custom_anthropic_p_at_5_ci = [0.664, 0.917]
• fastapi_custom_anthropic_r_at_5_mean = 0.841
• fastapi_custom_anthropic_r_at_5_ci = [0.710, 0.971]
• fastapi_custom_openai_p_at_5_mean = 0.718
• fastapi_custom_openai_p_at_5_ci = [0.610, 0.827]
• fastapi_custom_openai_r_at_5_mean = 0.833
• fastapi_custom_openai_r_at_5_ci = [0.715, 0.951]
• fastapi_langchain_anthropic_p_at_5_mean = 0.760
• fastapi_langchain_anthropic_p_at_5_ci = [0.645, 0.875]
• fastapi_langchain_anthropic_r_at_5_mean = 0.841
• fastapi_langchain_anthropic_r_at_5_ci = [0.710, 0.971]
• fastapi_langchain_openai_p_at_5_mean = 0.627
• fastapi_langchain_openai_p_at_5_ci = [0.484, 0.770]
• fastapi_langchain_openai_r_at_5_mean = 0.836
• fastapi_langchain_openai_r_at_5_ci = [0.702, 0.971]
• fastapi_custom_anthropic_citation_n = 21
• fastapi_custom_anthropic_citation_upper = 0.133
• fastapi_custom_anthropic_citation_rule_of_three = 0.143
• fastapi_custom_openai_citation_n = 19
• fastapi_custom_openai_citation_upper = 0.146
• fastapi_custom_openai_citation_rule_of_three = 0.158
• fastapi_langchain_anthropic_citation_n = 20
• fastapi_langchain_anthropic_citation_upper = 0.139
• fastapi_langchain_anthropic_citation_rule_of_three = 0.150
• fastapi_langchain_openai_citation_n = 20
• fastapi_langchain_openai_citation_upper = 0.139
• fastapi_langchain_openai_citation_rule_of_three = 0.150
• fastapi_custom_anthropic_vs_langchain_anthropic_p_at_5_tost = equivalent
• fastapi_custom_anthropic_vs_langchain_anthropic_p_at_5_support = 0.076
• fastapi_custom_anthropic_vs_langchain_anthropic_p_at_5_diff = +0.031
• fastapi_custom_anthropic_vs_langchain_anthropic_p_at_5_ci90 = [-0.013, +0.076]
• fastapi_custom_anthropic_vs_langchain_anthropic_p_at_5_ci95 = [-0.021, +0.086]
• fastapi_custom_anthropic_vs_langchain_anthropic_r_at_5_tost = equivalent
• fastapi_custom_anthropic_vs_langchain_anthropic_r_at_5_support = 0.000
• fastapi_custom_anthropic_vs_langchain_anthropic_r_at_5_diff = +0.000
• fastapi_custom_anthropic_vs_langchain_anthropic_r_at_5_ci90 = [+0.000, +0.000]
• fastapi_custom_anthropic_vs_langchain_anthropic_r_at_5_ci95 = [+0.000, +0.000]
• fastapi_custom_anthropic_vs_langchain_openai_p_at_5_tost = not established
• fastapi_custom_anthropic_vs_langchain_openai_p_at_5_support = 0.317
• fastapi_custom_anthropic_vs_langchain_openai_p_at_5_diff = +0.164
• fastapi_custom_anthropic_vs_langchain_openai_p_at_5_ci90 = [+0.031, +0.317]
• fastapi_custom_anthropic_vs_langchain_openai_p_at_5_ci95 = [+0.016, +0.353]
• fastapi_custom_anthropic_vs_langchain_openai_r_at_5_tost = equivalent
• fastapi_custom_anthropic_vs_langchain_openai_r_at_5_support = 0.095
• fastapi_custom_anthropic_vs_langchain_openai_r_at_5_diff = +0.005
• fastapi_custom_anthropic_vs_langchain_openai_r_at_5_ci90 = [-0.076, +0.095]
• fastapi_custom_anthropic_vs_langchain_openai_r_at_5_ci95 = [-0.086, +0.111]
• fastapi_custom_openai_vs_langchain_anthropic_p_at_5_tost = not established
• fastapi_custom_openai_vs_langchain_anthropic_p_at_5_support = 0.105
• fastapi_custom_openai_vs_langchain_anthropic_p_at_5_diff = -0.042
• fastapi_custom_openai_vs_langchain_anthropic_p_at_5_ci90 = [-0.105, +0.016]
• fastapi_custom_openai_vs_langchain_anthropic_p_at_5_ci95 = [-0.118, +0.023]
• fastapi_custom_openai_vs_langchain_anthropic_r_at_5_tost = not established
• fastapi_custom_openai_vs_langchain_anthropic_r_at_5_support = 0.100
• fastapi_custom_openai_vs_langchain_anthropic_r_at_5_diff = -0.008
• fastapi_custom_openai_vs_langchain_anthropic_r_at_5_ci90 = [-0.100, +0.068]
• fastapi_custom_openai_vs_langchain_anthropic_r_at_5_ci95 = [-0.114, +0.076]
• fastapi_custom_openai_vs_langchain_openai_p_at_5_tost = not established
• fastapi_custom_openai_vs_langchain_openai_p_at_5_support = 0.202
• fastapi_custom_openai_vs_langchain_openai_p_at_5_diff = +0.091
• fastapi_custom_openai_vs_langchain_openai_p_at_5_ci90 = [-0.010, +0.202]
• fastapi_custom_openai_vs_langchain_openai_p_at_5_ci95 = [-0.026, +0.227]
• fastapi_custom_openai_vs_langchain_openai_r_at_5_tost = equivalent
• fastapi_custom_openai_vs_langchain_openai_r_at_5_support = 0.043
• fastapi_custom_openai_vs_langchain_openai_r_at_5_diff = -0.003
• fastapi_custom_openai_vs_langchain_openai_r_at_5_ci90 = [-0.043, +0.032]
• fastapi_custom_openai_vs_langchain_openai_r_at_5_ci95 = [-0.055, +0.040]
• fastapi_significant_pairs_95 = custom_anthropic vs langchain_openai p_at_5
• fastapi_significant_pairs_95_count = 1
• fastapi_icc_p_at_5 = 0.99
• fastapi_between_question_var_p_at_5 = 0.06051
• fastapi_within_question_var_p_at_5 = 0.00036
• fastapi_within_question_pct = 0.6
• fastapi_mde_p_at_5_80 = 0.110
• fastapi_mde_p_at_5_80_normal = 0.136
• k8s_custom_anthropic_k8s_p_at_5_mean = 0.823
• k8s_custom_anthropic_k8s_p_at_5_ci = [0.736, 0.910]
• k8s_custom_anthropic_k8s_r_at_5_mean = 0.943
• k8s_custom_anthropic_k8s_r_at_5_ci = [0.881, 1.000]
• k8s_custom_openai_k8s_p_at_5_mean = 0.814
• k8s_custom_openai_k8s_p_at_5_ci = [0.734, 0.894]
• k8s_custom_openai_k8s_r_at_5_mean = 0.957
• k8s_custom_openai_k8s_r_at_5_ci = [0.898, 1.000]
• k8s_custom_anthropic_k8s_citation_n = 22
• k8s_custom_anthropic_k8s_citation_upper = 0.127
• k8s_custom_anthropic_k8s_citation_rule_of_three = 0.136
• k8s_custom_openai_k8s_citation_n = 22
• k8s_custom_openai_k8s_citation_upper = 0.127
• k8s_custom_openai_k8s_citation_rule_of_three = 0.136
• k8s_significant_pairs_95 = none
• k8s_significant_pairs_95_count = 0
• k8s_icc_p_at_5 = 0.94
• k8s_between_question_var_p_at_5 = 0.03249
• k8s_within_question_var_p_at_5 = 0.00209
• k8s_within_question_pct = 6.0
• k8s_mde_p_at_5_80 = 0.119
• k8s_mde_p_at_5_80_normal = 0.110