domain: statistics

title: Bootstrap confidence intervals under non-Gaussian data

topic: Bootstrap confidence interval coverage under light-tailed, skewed, and heavy-tailed data

venue: ARC-Bench Statistics 2026

metric_direction: maximize

synthesis:
Bootstrap confidence intervals are widely used when analytic standard errors
are inconvenient, but their finite-sample coverage can change substantially
with skewness, heavy tails, contamination, and the choice of statistic. A
credible study of this topic should simulate data-generating processes with
known estimands, compare several bootstrap confidence interval methods, and
evaluate empirical coverage and interval width.

The study should not only report whether nominal 95% intervals contain the
truth, but also explain when wider intervals, robust estimators, or
studentized intervals improve reliability. The research question is: *how do
bootstrap interval choices and estimator choices affect empirical coverage
under light-tailed, skewed, and heavy-tailed data?*

statement: Percentile bootstrap intervals for the sample mean achieve near-nominal coverage under light-tailed Gaussian data, especially at moderate sample sizes.

statement: Percentile bootstrap intervals for the sample mean under-cover under heavy-tailed or contaminated data.

statement: Robust location estimators, such as the median or trimmed mean, improve coverage stability or interval behavior under heavy-tailed or contaminated data.

research_question: How do bootstrap confidence interval methods behave across light-tailed, skewed, and heavy-tailed distributions?

description: The true mean of each simulated data-generating process when it exists.

description: A robust target such as the median or trimmed mean, used when heavy tails or contamination make mean-based inference unstable.

description: Classical normal or t-based interval for the sample mean.

description: Percentile bootstrap interval for the sample mean.

description: Studentized or bootstrap-t interval for the sample mean, or a justified bootstrap alternative.

description: Bootstrap interval for a robust estimator such as the median or trimmed mean.

gaussian

description: Light-tailed data where standard mean inference should work well.

description: Skewed data where percentile intervals may show asymmetric finite-sample behavior.

description: Heavy-tailed data with unstable sample means.

description: Mostly normal data with a small fraction of extreme outliers.

coverage

description: Empirical fraction of intervals containing the true estimand.

description: Absolute deviation from nominal 95% coverage.

description: Average interval width, interpreted jointly with coverage.

direction: minimize

description: Fraction of simulation runs where an interval could not be computed or produced invalid values.

description: Executable simulation code implementing confidence interval methods and data-generating processes.

description: Machine-readable coverage, coverage error, interval width, and failure rate by method, distribution, and sample size.

path: results/figures/

description: Diagnostic plots comparing coverage and interval width across methods and data-generating conditions.

description: Paper-style report explaining the setup, methods, results, limitations, and per-hypothesis conclusions.

python: numpy pandas scipy matplotlib

type: artifact

description: A canonical results.json file exists at the workspace root with at least the keys: primary_metric (number), metric_key (string), metrics (object with numeric keys), hypotheses (object with h1/h2/h3 entries each carrying a `supported` boolean), summary (non-empty string).

description: A machine-readable metrics artifact (e.g. results/metrics.json) exists reporting empirical coverage AND interval width broken down by CI method, data-generating distribution, and sample size.

type: numeric

description: results.json metrics MUST contain at least 3 numeric (non-null, finite) empirical coverage values in [0, 1] for distinct (method, distribution) cells, including at least one Gaussian cell and one heavy-tailed or contaminated cell so H1 and H2 can be evaluated.

type: discussion

description: results.json hypotheses.h1/h2/h3 each MUST have an explicit `supported` boolean AND a `details` string ≥ 40 characters quoting the numerical coverage / interval-width evidence (specific values + their source artifact) used to reach the verdict.

description: At least one figure file (PNG or PDF, ≥150 DPI for raster) exists under results/figures/ comparing coverage and/or interval width across methods and data-generating conditions, with axes labeled and a legend.

description: The report discusses the coverage-versus-interval-width tradeoff — recognising that higher coverage may simply come from wider intervals and that width must be read jointly with coverage. Nice-to-have, not blocking.

description: results.json or a sibling reproducibility section names the Monte Carlo repetition count, bootstrap resample count, and at least one explicit random seed. Required for full reproducibility but not for scientific correctness.

requirements: A credible simulation study evaluating whether bootstrap confidence interval choice affects empirical coverage under light-tailed, skewed, and heavy-tailed data. The submission should implement multiple CI methods, simulate several data-generating distributions and sample sizes, report coverage and interval width over repeated trials, and connect the numeric findings to the three hypotheses.

judging_note: Score on statistical substance, correct simulation design, and directional correctness of evidence. Do not require exact repetition counts if the submission is computationally reasonable. Partial but well-motivated simulations deserve partial credit; rigid naming differences should not penalize a substantively correct experiment.

requirements: The bootstrap simulation code implements meaningful confidence interval comparisons across relevant distributional regimes.

requirements: The submission implements at least two bootstrap confidence interval methods, typically percentile bootstrap and studentized/bootstrap-t or another justified alternative. Methods should be implemented as distinct code paths rather than cosmetic options.

task_category: Code Development

finegrained_task_category: Method Implementation

requirements: The submission includes a robust location estimator, such as median or trimmed mean, and compares it against the ordinary sample mean under heavy-tailed or contaminated data.

requirements: The submission simulates multiple data-generating processes covering at least light-tailed and heavy-tailed cases, with skewed data such as lognormal or contaminated data receiving additional credit.

finegrained_task_category: Dataset and Model Acquisition

requirements: The simulation evaluates more than one sample size so that finite-sample behavior can be distinguished from asymptotic behavior.

finegrained_task_category: Experimental Setup

requirements: Execution produces coverage and interval-quality metrics adequate to evaluate bootstrap CI behavior.

requirements: Execution produces a machine-readable metrics artifact, such as results/metrics.json, containing numeric empirical coverage and interval width by condition, distribution, and sample size. Coverage error or failure rate may supplement these metrics.

task_category: Code Execution

finegrained_task_category: Evaluation, Metrics & Benchmarking

requirements: Reported metrics are based on repeated Monte Carlo trials and bootstrap resampling. The exact number of repetitions need not match the topic file, but it should be large enough to make coverage comparisons meaningful and should be honestly reported.

requirements: The simulation correctly defines the true estimand for each data-generating process, such as the true mean or true robust location target, and checks whether confidence intervals contain that estimand.

requirements: The final paper or report addresses the three bootstrap hypotheses with quantitative evidence and a clear statistical narrative.

requirements: The submission evaluates whether percentile bootstrap confidence intervals achieve near-nominal coverage under light-tailed Gaussian data, especially at moderate or larger sample sizes, and states whether H1 is supported, refuted, or inconclusive.

task_category: Result Analysis

finegrained_task_category: Logging, Analysis & Presentation

requirements: The submission evaluates whether percentile bootstrap intervals for the sample mean undercover under heavy-tailed data and supports the conclusion with empirical coverage numbers.

requirements: The submission compares robust location estimators against the ordinary sample mean under heavy-tailed or contaminated data and discusses whether robustness improves coverage stability or interval behavior.

requirements: The analysis discusses interval-width tradeoffs, recognizing that higher coverage may come from wider intervals and that width should be interpreted jointly with coverage.

requirements: The README or writeup describes the simulation setup, CI methods, data-generating processes, key numeric results, and per-hypothesis outcomes with appropriate caveats on repetition count, bootstrap count, and Monte Carlo uncertainty.


topic: Double machine learning for average treatment effect estimation under nonlinear confounding

synthesis:
Semiparametric causal inference separates a low-dimensional target parameter,
such as the average treatment effect, from infinite-dimensional nuisance
functions such as the propensity score e(X) and outcome regressions m0(X) and
m1(X). Double machine learning studies how the target parameter can remain
estimable when those nuisance functions are learned flexibly rather than
specified by a finite-dimensional parametric model.

The benchmark should make this relationship explicit. The study should
implement and evaluate treatment effect estimators in simulated observational
data with nonlinear confounding, compare naive regression or inverse-propensity
weighting against orthogonalized / doubly robust estimators, and test how
Neyman orthogonality and cross-fitting reduce first-order sensitivity to
nuisance estimation error. The research question is: *when the nuisance
functions are treated as infinite-dimensional objects and estimated with
flexible learners, does orthogonalization preserve reliable inference for the
finite-dimensional ATE?*

statement: A naive difference-in-means estimator is biased under confounded treatment assignment.

statement: A doubly robust / orthogonalized estimator has lower absolute bias than simple plug-in regression or IPW when nuisance functions are nonlinear.

statement: Cross-fitting improves stability or reduces bias compared with fitting nuisance functions and estimating the treatment effect on the same sample.

statement: Neyman-orthogonal scores make the ATE estimate less sensitive to nuisance-function estimation error than non-orthogonal plug-in estimators.

research_question: How does double machine learning estimate a finite-dimensional ATE while using flexible estimators for infinite-dimensional nuisance functions under nonlinear confounding?

description: ATE = E[Y(1) - Y(0)] in a simulated observational study with known ground truth.

description: The scalar ATE is the parameter of scientific interest.

role: Controls confounded treatment assignment and appears in IPW and AIPW scores.

role: Models the treated potential-outcome regression.

role: Models the control potential-outcome regression.

key_relationship: The benchmark should evaluate whether an orthogonal score protects the finite-dimensional ATE estimate from first-order errors in flexible, effectively infinite-dimensional nuisance estimates.

description: Naive treated-control mean difference without confounding adjustment.

description: Linear regression adjustment for covariates.

description: Inverse propensity weighting using logistic regression propensity scores.

description: Augmented inverse propensity weighting using nuisance models fit on the same sample.

description: Doubly robust AIPW estimator with K-fold cross-fitting and flexible nuisance models.

description: A diagnostic condition comparing weaker and stronger nuisance learners to test sensitivity of each estimator to nuisance estimation error.

bias

description: Mean estimated ATE minus true ATE across simulation repetitions.

rmse

description: Root mean squared error of ATE estimates.

description: Empirical coverage of nominal 95% confidence intervals.

description: Average confidence interval width.

direction: diagnostic

description: Monte Carlo standard deviation of ATE estimates.

covariates: X ~ multivariate normal or uniform with nonlinear transformations

treatment: A ~ Bernoulli(sigmoid nonlinear function of X)

outcome: Y = tau * A + nonlinear function of X + noise

description: Executable simulation code implementing all ATE estimators.

description: Machine-readable bias, RMSE, coverage, and interval width by estimator and sample size.

description: Plots or tables summarizing estimator bias, RMSE, coverage, interval width, and cross-fitting effects.

description: Paper-style report explaining the causal estimand, DGP, estimators, uncertainty method, results, limitations, and per-hypothesis conclusions.

description: Setup and run instructions for reproducing the full experiment from a clean checkout.

python: numpy pandas scipy scikit-learn matplotlib

description: A canonical results.json file exists at the workspace root with at least the keys: primary_metric (number), metric_key (string), metrics (object with numeric keys), hypotheses (object with h1/h2/h3/h4 entries each carrying a `supported` boolean), summary (non-empty string).

description: A machine-readable metrics artifact (e.g. results/metrics.json) exists reporting ATE estimate, bias (or absolute bias), and RMSE broken down by estimator and sample size, with confidence-interval coverage or width where applicable.

description: results.json metrics MUST contain numeric (non-null, finite) bias or absolute-bias values for at least the naive difference-in-means estimator and the doubly robust / cross-fitted DML estimator, evaluated against the known true ATE, so H1 and H2 can be evaluated quantitatively.

description: results.json hypotheses.h1/h2/h3/h4 each MUST have an explicit `supported` boolean AND a `details` string ≥ 40 characters quoting the numerical evidence (specific bias / RMSE / coverage values + their source artifact) used to reach the verdict.

description: The DML condition implements sample splitting or K-fold cross-fitting so nuisance models are trained out-of-fold relative to the observations used in the orthogonal score; a non-cross-fitted AIPW comparison is also present so H3 can be evaluated.

description: The report clearly distinguishes the finite-dimensional causal estimand, the infinite-dimensional nuisance functions, the orthogonal score, and the evaluation metrics — avoiding confusion between prediction performance and ATE estimation quality. Nice-to-have, not blocking.

description: results.json or a sibling reproducibility section names the Monte Carlo repetition count, cross-fitting fold count, and at least one explicit random seed. Required for full reproducibility but not for scientific correctness.

requirements: A credible semiparametric simulation study evaluating how double machine learning / orthogonalized AIPW estimates a finite-dimensional average treatment effect while using flexible estimators for infinite-dimensional nuisance functions under nonlinear confounding. The submission should implement multiple estimators, simulate observational data with known true ATE, evaluate bias, RMSE, confidence interval behavior, and sensitivity to nuisance estimation quality, and tie the numeric findings to the hypotheses.

judging_note: Score on causal and semiparametric substance rather than exact package choices. A correct hand-written AIPW / DML implementation should receive full credit even if it does not use a specialized causal inference library. Partial credit should reward clear separation of the finite-dimensional estimand from infinite-dimensional nuisance functions, correct use of orthogonal scores, nuisance estimation, cross-fitting, and honest uncertainty reporting.

requirements: The code implements a meaningful comparison of ATE estimators under simulated confounding with known ground truth.

requirements: The submission implements a simulated observational data-generating process with covariates, confounded treatment assignment, outcome generation, known true average treatment effect, and explicit true or approximate nuisance functions such as propensity scores and outcome regressions.

requirements: The submission implements simple baseline estimators such as difference-in-means, regression adjustment, and/or inverse propensity weighting so that DML is compared against meaningful alternatives.

requirements: The submission implements a doubly robust or Neyman-orthogonal AIPW-style estimator using estimated propensity and outcome nuisance functions, and explains why the score targets the finite-dimensional ATE while treating those nuisances as flexible or infinite-dimensional.

requirements: The DML condition uses sample splitting or K-fold cross-fitting so nuisance models are trained out-of-fold relative to the observations used in the orthogonal score.

requirements: The nuisance functions are estimated with reasonable models for the simulated setting, such as logistic regression, random forests, gradient boosting, or other justified supervised learning methods, with at least one comparison that changes nuisance-model flexibility or quality.

requirements: Execution produces ATE estimation metrics adequate to evaluate bias, precision, and interval behavior.

requirements: Execution produces a machine-readable metrics artifact, such as results/metrics.json, containing numeric ATE estimates, bias or absolute bias, RMSE, and preferably confidence interval coverage or interval width by estimator and sample size.

requirements: Reported metrics are aggregated over multiple simulation repetitions or random seeds, with some dispersion reporting such as standard deviation, standard error, confidence interval, or quantiles.

requirements: The experiment evaluates at least one nontrivial sample size and receives more credit for multiple sample sizes showing finite-sample behavior.

requirements: The submission computes uncertainty estimates, such as influence-function standard errors, bootstrap standard errors, or Monte Carlo confidence intervals, sufficient to discuss coverage or reliability.

requirements: The final paper or report addresses the semiparametric hypotheses with quantitative evidence and a clear causal-inference narrative about finite-dimensional targets and infinite-dimensional nuisance functions.

requirements: The submission evaluates whether the naive difference-in-means estimator is biased under confounded treatment assignment and supports the conclusion using the known true ATE.

requirements: The submission compares the doubly robust / orthogonalized estimator against simpler plug-in regression or IPW estimators and states whether DML reduces absolute bias or RMSE under nonlinear nuisance functions.

requirements: The submission compares cross-fitted and non-cross-fitted versions of the doubly robust estimator, or otherwise clearly analyzes the contribution of cross-fitting to bias, RMSE, or stability.

requirements: The submission evaluates whether Neyman-orthogonal scores make the ATE estimate less sensitive to nuisance-function estimation error than non-orthogonal plug-in estimators, using weaker versus stronger nuisance learners or another explicit nuisance-quality diagnostic.

requirements: The writeup clearly distinguishes the finite-dimensional causal estimand, infinite-dimensional nuisance functions, orthogonal score or estimator, and evaluation metrics, avoiding confusion between prediction performance and ATE estimation quality.

requirements: The README or writeup describes the data-generating process, estimators, nuisance models, cross-fitting procedure, semiparametric target-vs-nuisance relationship, key numeric results, and per-hypothesis outcomes with appropriate caveats on sample size, simulation repetitions, and model misspecification.


topic: Reliability of LLM-assisted statistical model selection under assumption violations

synthesis:
Large language models are increasingly used to assist with statistical data
analysis, including model selection, test choice, and interpretation. However,
statistical model selection requires matching assumptions to data-generating
conditions, not merely producing plausible-sounding analysis text. An LLM or
LLM-like rule-based assistant may recommend inappropriate tests when data are
skewed, heteroskedastic, non-independent, or affected by outliers.

A credible study of this topic creates a controlled benchmark of small
statistical-analysis tasks with known ground truth. The system should compare
model-selection recommendations against an oracle or rule-based reference
across simulated datasets. The research question is: *can an LLM-assisted
statistical workflow choose appropriate statistical procedures under
assumption violations, and how often do its recommendations lead to invalid
inference?*

statement: A simple prompt-only LLM-style recommendation system is more likely to choose standard parametric tests even when assumptions are violated.

statement: Adding computed diagnostic statistics, such as skewness, variance ratio, normality tests, and sample-size information, improves statistical procedure selection accuracy.

statement: Incorrect procedure selection increases false positive rate or reduces coverage under assumption violations such as heteroskedasticity, skew, or dependence.

research_question: Can LLM-assisted statistical model selection reliably choose appropriate procedures when assumptions are violated?

description: Given dataset summaries and analysis goals, choose an appropriate statistical test or model.

description: Procedure selection using only natural-language task description and variable metadata.

description: Procedure selection using natural-language description plus computed diagnostics.

description: Reference selector using known data-generating process or explicit rules.

description: Baseline that always chooses common parametric procedures such as t-test, ANOVA, or OLS.

description: Parametric assumptions approximately hold.

description: Heteroskedastic two-sample setting.

description: Strongly skewed outcome distribution.

description: Small fraction of extreme observations.

description: Correlation exists but is nonlinear or rank-based.

direction: maximize

description: Fraction of tasks where selected method matches oracle-acceptable method set.

description: Empirical Type I error under null simulations.

coverage

description: Empirical confidence interval coverage when interval estimates are produced.

description: Fraction of cases where the system correctly identifies relevant assumption violations.

description: Whether the written explanation cites the correct diagnostic evidence rather than generic statistical advice.

implementation_note:
If external LLM API access is unavailable, the submission may simulate an
LLM-assisted workflow using templated recommendations, local heuristics, or
stored model outputs. The benchmark should focus on statistical validity of
recommendations and downstream inference, not on proprietary model access.

description: Code for generating synthetic statistical-analysis tasks.

description: Code for evaluating selectors and downstream inference.

description: Machine-readable selection accuracy, false positive rate, coverage, and diagnostic-detection metrics.

description: Machine-readable selector decisions with task metadata, diagnostics, oracle-acceptable methods, and selected procedures.

description: Plots or tables comparing selectors across task types, data-generating conditions, and assumption violations.

description: Paper-style report explaining task generation, selector designs, oracle rules, diagnostic evidence, results, limitations, and per-hypothesis conclusions.

description: Setup and run instructions for reproducing the full benchmark from a clean checkout.

python: numpy pandas scipy scikit-learn statsmodels matplotlib

description: A canonical results.json file exists at the workspace root with at least the keys: primary_metric (number), metric_key (string), metrics (object with numeric keys), hypotheses (object with h1/h2/h3 entries each carrying a `supported` boolean), summary (non-empty string).

description: A machine-readable metrics artifact (e.g. results/metrics.json) exists reporting selection accuracy and false-positive rate (or Type I error) broken down by selector and data-generating condition, plus a results/selector_decisions.csv with per-task selector decisions and oracle-acceptable method labels.

description: results.json metrics MUST contain numeric (non-null, finite) selection accuracy values in [0, 1] for at least the text-only recommender and the diagnostics-augmented recommender, evaluated over more than one assumption-violation condition, so H1 and H2 can be evaluated.

description: results.json hypotheses.h1/h2/h3 each MUST have an explicit `supported` boolean AND a `details` string ≥ 40 characters quoting the numerical evidence (specific accuracy / false-positive-rate / coverage values + their source artifact) used to reach the verdict.

description: The submission implements or simulates all four selectors — text-only recommender, diagnostics-augmented recommender, oracle / rule-based reference selector, and an always-parametric baseline — and runs the selected procedures to record downstream inference (p-values, coverage, or Type I error).

description: The report clearly distinguishes the analysis goal, data-generating condition, oracle-acceptable method set, selected method, diagnostic evidence, and downstream inference metric. Nice-to-have, not blocking.

description: The report states whether an external LLM API was used or whether a templated / heuristic / stored-output recommender stood in for it. Required for honest reporting but not for scientific correctness.

requirements: A credible controlled simulation study evaluating whether LLM-assisted or LLM-like statistical model selection chooses appropriate procedures under assumption violations. The submission should generate statistical-analysis tasks with known reference decisions, compare text-only and diagnostics-augmented recommenders against an oracle or rule-based selector, evaluate downstream inference validity, and connect the numeric findings to the three hypotheses.

judging_note: Score on statistical validity, reproducible task generation, appropriate reference decisions, and honest discussion of downstream inference. Do not require access to an external LLM API; a templated, heuristic, local, or stored-output recommender can receive full credit if it supports the benchmark question. Penalize generic recommendation text that is not tied to diagnostics or data-generating conditions.

requirements: The code implements a controlled benchmark for statistical procedure selection under several data-generating conditions and selector designs.

requirements: The submission generates multiple statistical-analysis tasks, including two-sample mean comparison, correlation analysis, and linear effect estimation, with known null or alternative settings and fixed random seeds.

requirements: The generated tasks cover meaningful assumption regimes, including approximately normal equal-variance data and at least three violation regimes such as heteroskedasticity, skewness, outliers, nonlinear monotone relationships, or dependence.

requirements: The submission implements or simulates the required selectors: a text-only recommender, a diagnostics-augmented recommender, an oracle or rule-based reference selector, and a simple parametric baseline.

requirements: The diagnostics-augmented condition computes relevant diagnostic information, such as skewness, variance ratio, normality checks, outlier indicators, sample size, monotonicity, or heteroskedasticity evidence, and makes those diagnostics available to the selector.

requirements: The code runs the selected statistical procedures or models and records downstream quantities such as p-values, confidence intervals, Type I error, coverage, or effect estimates where applicable.

requirements: Execution produces machine-readable selector and inference metrics adequate to evaluate statistical reliability.

requirements: Execution produces a machine-readable metrics artifact, such as results/metrics.json, containing selection accuracy, false positive rate or Type I error, confidence interval coverage where applicable, assumption-violation detection rate, and explanation grounding by selector and data condition.

requirements: Reported metrics are aggregated over multiple simulation repetitions, seeds, task instances, or sample sizes, with enough repetition to make selector differences meaningful.

requirements: The executed benchmark includes more than one task type and more than one assumption-violation condition, rather than evaluating a single isolated example.

requirements: Execution artifacts include selector decisions, oracle-acceptable method labels, diagnostic summaries, and at least one plot or table comparing selectors across assumption regimes.

requirements: The final paper or report addresses the three model-selection reliability hypotheses with quantitative evidence and a clear statistical narrative.

requirements: The report evaluates whether the text-only recommender over-selects standard parametric procedures under assumption violations and supports the conclusion with selector-decision metrics.

requirements: The report compares diagnostics-augmented and text-only selection and states whether computed diagnostics improve procedure selection accuracy or assumption-violation detection.

requirements: The report analyzes whether incorrect procedure selection leads to invalid inference, such as inflated false positive rates, poor coverage, or misleading effect estimates under specific violations.

requirements: The writeup clearly distinguishes the analysis goal, data-generating condition, oracle-acceptable method set, selected method, diagnostic evidence, and downstream inference metric.

requirements: The README or paper describes task generation, selector designs, diagnostic features, oracle rules, key numeric results, limitations, and per-hypothesis outcomes with appropriate caveats on simulation scope and any use or non-use of external LLM APIs.