Bandit algorithm robustness under stationary and drifting reward regimes

Comparing multi-armed bandit algorithms (epsilon-greedy, UCB1, Thompson sampling, Exp3) on synthetic Bernoulli and Gaussian arms under stationary and drift regimes

Multi-armed bandit algorithms encode different assumptions about reward
generation and non-stationarity. Epsilon-greedy is simple and adaptable with
persistent exploration; UCB1 is optimism-driven and often strong in
stationary stochastic settings; Thompson sampling can be highly sample
efficient when model assumptions are matched; Exp3 is designed for
adversarial settings and may trade off stochastic efficiency for robustness.
In small CPU-constrained studies, these differences are often discussed
qualitatively but not tested with matched synthetic environments.

A meaningful benchmark should evaluate both Bernoulli and Gaussian reward
arms, because posterior/model assumptions and noise scale can materially
change outcomes. It should also include stationary and drifting regimes:
methods tuned for fixed means can degrade when the best arm changes over
time. Regret, not just final reward, is the right primary metric because it
captures online learning efficiency throughout the horizon.

A credible experiment here implements the four algorithms with consistent
action/reward interfaces, runs repeated simulations over multiple seeds, and
reports cumulative regret trajectories and endpoint statistics. To keep the
study CPU-friendly, horizons and arm counts should be modest (e.g., 5-10 arms,
500-2000 steps), with vectorized numpy updates where possible.

The key analytical value is comparative: identify where UCB1/Thompson excel
in stationary stochastic settings, and whether epsilon-greedy or Exp3 is more
resilient under drift. The objective is not reproducing a single paper, but
testing algorithm-environment fit under controlled synthetic shifts.

*How do epsilon-greedy, UCB1, Thompson sampling, and Exp3 trade off cumulative regret across Bernoulli vs Gaussian arms and stationary vs drifting reward regimes?*

In stationary Bernoulli environments, either UCB1 or Thompson sampling achieves at least 10% lower mean cumulative regret than epsilon-greedy at horizon T on at least 2 of 3 datasets (averaged over >=20 seeds).

In drifting environments, epsilon-greedy or Exp3 achieves lower mean cumulative regret than UCB1 on at least 2 of 3 datasets at horizon T (averaged over >=20 seeds).

Across all evaluated datasets, no single algorithm is best (lowest mean cumulative regret) on every dataset, indicating environment-dependent performance.

How do epsilon-greedy, UCB1, Thompson sampling, and Exp3 compare in cumulative regret across stationary versus drifting synthetic bandit tasks with Bernoulli and Gaussian rewards?

Epsilon-greedy with constant epsilon=0.1 and sample-mean value estimates.

UCB1 with exploration bonus sqrt(2 log t / n_a).

Thompson sampling using Beta-Bernoulli updates for Bernoulli arms and Gaussian posterior sampling (known variance assumption) for Gaussian arms.

Exp3 with gamma=0.07 and importance-weighted reward estimates.

epsilon_greedy_eps0.1 as simple stochastic baseline

Mean cumulative regret at final horizon T relative to oracle best arm per round, averaged over seeds.

Area under per-round regret curve over time (lower is better).

Fraction of rounds selecting the current optimal arm (for drift: time-varying optimum).

synthetic: 10 Bernoulli arms with fixed means sampled in [0.05, 0.95] and sorted gap >=0.05

synthetic: 10 Bernoulli arms with piecewise-constant means; best arm switches at predefined changepoints

synthetic: 8 Gaussian arms (sigma=1) with linearly drifting means and periodic rank reversals

A credible experiment comparing bandit algorithms (epsilon-greedy, UCB1, Thompson sampling, Exp3, or equivalents) under stationary and drifting synthetic regimes: algorithms are implemented with a common interface, experiments cover multiple environments with repeated seeds, and results address H1/H2/H3 directionally using cumulative regret.

Score on scientific substance and directional correctness of evidence, not on exact numeric thresholds. Well-motivated algorithm variants (e.g., UCB-V, linear Thompson) should be credited when they test the same scientific question.

The bandit algorithms and synthetic environments are implemented correctly.

The submission implements multiple distinct algorithm code paths — typically including epsilon-greedy, UCB1, Thompson sampling, and/or Exp3 — with per-round action selection and update logic that are not identical wrappers.

The submission defines multiple synthetic bandit environments including at least one stationary and one drifting regime, with reproducible seed control and oracle best-arm rewards per round for regret computation.

If Thompson sampling and/or Exp3 are included, implementation uses sampling-based posterior decisions (Thompson) and probability-weighted action selection with importance-weighted updates (Exp3), rather than greedy mean selection.

Execution produces regret metrics across algorithms and datasets.

Execution runs multiple seeds per (algorithm, dataset) cell for multiple environments and logs final cumulative-regret values with mean and dispersion. Honest small-seed runs with variance reported are preferable to a single run.

A machine-readable results artifact is produced containing dataset-wise metrics for each implemented algorithm, including cumulative regret.

Findings address H1/H2/H3 directionally and summarize implications.

The submission compares stationary-regime cumulative regret between epsilon-greedy and {UCB1, Thompson} and conveys whether the principled algorithms are meaningfully better — judge directionally against H1.

The submission evaluates drifting-regime cumulative regret and conveys whether more exploratory algorithms (epsilon-greedy or Exp3) outperform UCB1 on most drifting datasets (H2).

The submission conveys whether any single algorithm dominates across all environments or whether winners are mixed (H3), with supporting tables or summaries.

The README or writeup describes setup, reports key cumulative-regret results per environment, conveys per-hypothesis outcomes (supported / refuted / inconclusive), and notes limitations (horizon length, hyperparameter sensitivity, synthetic-only scope). No strict word-count requirement.


Topic-model comparison on small 20newsgroups subsets: LDA vs NMF vs LSA

Comparing topic models (LDA, NMF, LSA) on a small subset of 20newsgroups by topic coherence (c_v) and document-cluster adjusted rand score

Topic modeling methods are often compared using either intrinsic quality
measures (such as topic coherence) or extrinsic utility (such as how well
document representations align with known labels), but the two views can
disagree. Latent Dirichlet Allocation (LDA), Non-negative Matrix
Factorization (NMF), and Latent Semantic Analysis (LSA/SVD) each induce
different assumptions over term-document structure and may therefore rank
differently depending on metric choice.

On CPU-constrained benchmarks, a practical study should use a compact text
corpus and a controlled preprocessing pipeline so that runtime stays short
while still yielding meaningful distinctions. A small subset of
20newsgroups is ideal because it has accessible labels for external
clustering evaluation and enough lexical diversity for coherence analysis.
Using multiple subset difficulties (well-separated vs more confusable
categories) helps test robustness of conclusions.

A credible experiment compares LDA, NMF, and LSA under matched topic counts
and vectorization settings, then reports both c_v-like coherence and
adjusted rand index (ARI) from document-cluster assignments against true
newsgroup labels. Because exact c_v implementations are not native in
sklearn, an explicit approximation based on sliding-window co-occurrence
and normalized PMI should be documented and applied consistently across
methods. Repeated runs for stochastic models are needed to avoid
over-interpreting single-seed noise.

The core question is whether better intrinsic coherence implies better
label alignment on small real-world corpora, and which model offers the
best trade-off under strict CPU budgets.

*Do LDA, NMF, and LSA produce different rankings on coherence versus ARI on small 20newsgroups subsets, and does NMF provide the strongest coherence-structure trade-off under CPU limits?*

NMF achieves higher mean coherence_cv than LSA on at least 2 of 3 evaluated 20newsgroups subsets when using the same number of topics and preprocessing pipeline.

Across the three methods (LDA, NMF, LSA), the method with the highest coherence_cv is also the highest-ARI method on no more than 1 of the 3 subsets, indicating weak metric agreement.

LDA achieves mean ARI at least 0.03 higher than LSA on at least 2 of 3 subsets when document clusters are obtained by argmax topic assignment.

How do LDA, NMF, and LSA compare on coherence_cv versus document-cluster ARI over small 20newsgroups subsets, and do intrinsic/extrinsic rankings diverge?

LatentDirichletAllocation with n_components=4, max_iter=20, learning_method='batch', CountVectorizer features, repeated over 3 seeds.

NMF with n_components=4, init='nndsvda', solver='cd', max_iter=300 on TF-IDF features, repeated over 3 seeds.

TruncatedSVD (LSA) with n_components=4 on TF-IDF features; document-topic embeddings clustered via argmax after non-negative shift or via KMeans(k=4) with fixed seed.

NMF sensitivity condition with n_components=6 to test topic-count robustness for coherence and ARI trends.

lsa_k4 as a linear-algebra baseline without probabilistic topic assumptions

lda_k4 as a probabilistic-topic baseline

Approximate c_v coherence computed from top words per topic using sliding-window co-occurrence and NPMI aggregation, averaged across topics and seeds.

ari

Adjusted Rand Index between predicted document clusters (topic argmax or equivalent) and true newsgroup labels.

Wall-clock training + inference time per condition/dataset cell on single CPU core.

A credible experiment comparing LDA, NMF, and LSA topic models on small 20newsgroups subsets: conditions are implemented, execution reports coherence and ARI on multiple subsets with repeated seeds, and results address H1/H2/H3 directionally.

Score on scientific substance and directional correctness of evidence, not on exact numeric thresholds. Alternative topic-model implementations or coherence approximations that test the same scientific question should be credited.

Topic-model conditions and dataset pipeline are implemented correctly.

The submission implements LDA, NMF, and LSA (or equivalents such as a truncated SVD for LSA) as distinct modeling code paths with a matched topic-count setting.

The submission loads multiple 20newsgroups subsets (or comparable document corpora) with consistent text preprocessing/vectorization documented in code.

The code computes a coherence-style metric (e.g., c_v or a documented approximation) and ARI from document-cluster assignments for each method.

Execution produces comparable numeric outputs for all implemented methods.

Execution evaluates the core methods on multiple datasets and writes per-method, per-dataset coherence and ARI values to a machine-readable artifact.

Execution repeats stochastic methods with multiple seeds per dataset, reports dispersion, and logs a wall-clock timing measure per condition. Honest small-seed runs with variance reported are preferable to a single run.

Quantitative analysis addresses H1/H2/H3 directionally.

The submission compares NMF vs LSA on coherence for each evaluated dataset and conveys whether NMF tends to yield better coherence — judge directionally against H1.

The submission checks ranking agreement between coherence and ARI across methods per dataset and conveys whether coherence-ranking and ARI-ranking tend to diverge (H2).

The submission reports LDA vs LSA ARI deltas per dataset and conveys whether LDA tends to achieve meaningfully higher document-cluster ARI (H3).

The README or writeup describes methods and preprocessing, reports coherence/ARI/runtime results, conveys per-hypothesis outcomes (supported / refuted / inconclusive), and notes limitations (subset choice, coherence approximation validity, seed count, clustering-assignment assumptions). No strict word-count requirement.


Post-hoc calibration methods for sklearn classifiers on tabular benchmarks

Evaluating probability calibration methods (Platt scaling, isotonic regression, temperature scaling) applied to scikit-learn classifiers (RandomForest, GBM, SVM-RBF) on UCI benchmarks

Many widely used scikit-learn classifiers optimize discrimination rather
than probability quality. Random forests, gradient boosting models, and
RBF-kernel SVMs can achieve strong accuracy while output probabilities are
miscalibrated, especially on small or moderately imbalanced tabular data.
This matters in decision settings where thresholds, risk ranking, or cost-
sensitive actions depend on reliable confidence estimates.

Post-hoc calibration methods are attractive because they can be layered onto
existing models without retraining the base learner. Platt scaling fits a
sigmoid map, isotonic regression fits a non-parametric monotone map, and
temperature scaling applies a single-parameter logit rescaling (implemented
for binary tasks via optimization on held-out logits/probabilities). These
methods differ in flexibility and overfitting risk, so their relative value
may depend on classifier family and dataset size.

A credible CPU-scale study should compare uncalibrated outputs versus at
least three calibrators across multiple sklearn tabular datasets, using a
proper train/calibration/test protocol and repeated seeds. Because this is a
calibration-centric question, expected calibration error (ECE) and log loss
should be primary metrics, with accuracy used as a guardrail to ensure that
calibration does not degrade classification utility.

The goal is not to reproduce a single paper result but to test whether any
one calibrator is consistently superior across heterogeneous models and
datasets under tight compute constraints. The key uncertainty is whether
method ranking is stable enough to justify a default choice.

*Which post-hoc calibration method (Platt, isotonic, temperature) most reliably improves ECE across RF/GBM/SVM-RBF on small tabular sklearn benchmarks without materially harming accuracy?*

At least one post-hoc calibrator (Platt, isotonic, or temperature) reduces ECE by at least 10% relative to the uncalibrated model for at least 2 of 3 classifiers on at least 2 of 3 datasets, averaged over >=3 seeds.

Isotonic regression achieves lower mean ECE than Platt scaling on at least 2 of 3 datasets when calibration-set size is >=150 samples.

Applying post-hoc calibration changes test accuracy by no more than 1.0 absolute percentage point (mean over seeds) for each classifier-dataset pair.

Which post-hoc calibration method (Platt, isotonic, temperature) most consistently improves probability calibration for RF/GBM/SVM-RBF on sklearn tabular datasets while preserving accuracy?

uncalibrated

Base classifier probabilities/decision scores used directly with no calibration.

Sigmoid calibration fit on a held-out calibration split (CalibratedClassifierCV with method='sigmoid' or equivalent).

Isotonic calibration fit on a held-out calibration split (CalibratedClassifierCV with method='isotonic' or equivalent).

Single temperature parameter optimized on calibration split to minimize NLL; applied to logits/scores before probability mapping.

uncalibrated outputs are the primary baseline

platt_scaling serves as a classic parametric calibration baseline

ece

Expected Calibration Error (10-15 bins) on the held-out test split, averaged over seeds.

Negative log-likelihood on test probabilities.

Classification accuracy on held-out test split to verify discrimination is preserved.

sklearn.datasets.load_digits with target transformed to binary (e.g., digit<5 vs >=5)

A credible experiment studying post-hoc probability calibration methods (Platt, isotonic, temperature, or equivalents) for sklearn classifiers: calibration conditions are implemented, execution covers multiple datasets/classifiers with repeated seeds, and results address H1/H2/H3 directionally.

Score on scientific substance and directional correctness of evidence, not on exact numeric thresholds. Well-motivated calibrator variants or alternative base classifiers that preserve the scientific question should be credited.

Calibration methods and classifier setup are implemented correctly.

The submission implements multiple calibration conditions — typically including uncalibrated, Platt, isotonic, and temperature scaling — as distinct code paths applied post-hoc to the same base model outputs.

The experiment includes multiple target classifiers (e.g., RandomForest, gradient boosting, SVM-RBF, or equivalents) with consistent train/calibration/test handling.

The submission uses multiple datasets (sklearn built-ins or comparable) and an explicit calibration split separate from train and test.

Execution reports calibration-focused metrics across conditions.

Execution outputs numeric ECE, log loss, and test accuracy (or equivalents) for each implemented condition on at least one dataset-classifier pair in a machine-readable artifact.

Metrics are aggregated over multiple random seeds per evaluated cell with a dispersion measure. Honest small-seed runs with variance reported are preferable to a single run.

Results quantitatively address H1/H2/H3 directionally with a clear narrative.

The submission computes relative ECE change versus uncalibrated and conveys whether at least one calibrator yields a meaningful ECE reduction across most classifier/dataset pairs — judge directionally against H1.

The submission compares isotonic vs Platt mean ECE per dataset and conveys the relative calibration quality plus any calibration-set size considerations (H2).

The submission reports accuracy deltas between calibrated and uncalibrated outputs per classifier-dataset pair and conveys whether accuracy changes are small (H3).

The README or report conveys per-hypothesis outcomes (supported / refuted / inconclusive), cites key ECE/log-loss/accuracy numbers, and discusses limitations (dataset scope, binning sensitivity, seed count, calibration split size). No strict word-count requirement.


Comparing graph- and pseudo-label-based semi-supervised learning on small tabular datasets

Comparing semi-supervised learning strategies (LabelPropagation, LabelSpreading, SelfTrainingClassifier) on small UCI datasets with 10-20% labeled data

Semi-supervised learning is attractive when labels are scarce but unlabeled
examples are cheap. In sklearn, LabelPropagation and LabelSpreading provide
graph-based transductive approaches, while SelfTrainingClassifier offers a
pseudo-labeling wrapper around a standard supervised base learner. These
methods rely on different assumptions: graph smoothness over neighborhoods
versus confidence-thresholded iterative self-labeling. On small tabular
datasets, their relative behavior can change quickly as the labeled fraction
drops from moderate (20%) to very low (10%).

A useful CPU-scale study should compare these methods under the same splits,
feature preprocessing, and random seeds, with explicit control of labeled
fraction. Because the key claim of semi-supervision is label efficiency, the
experiment should include a supervised-only baseline trained on the same
labeled subset and evaluated on a fixed held-out test set. This makes gains
attributable to unlabeled-data usage rather than favorable splits.

The study should report not only test accuracy but also balanced accuracy and
macro-F1, since small tabular datasets can have class imbalance and accuracy
alone may hide minority-class degradation. Repeating runs across several seeds
is necessary because which points are labeled strongly affects outcomes at low
label rates.

Practically, the benchmark should stay lightweight: sklearn datasets only,
no downloads, and model choices that complete within minutes on a single CPU
core. The focus is not SOTA performance but robust relative comparisons across
label regimes and datasets.

*How do LabelPropagation, LabelSpreading, and SelfTrainingClassifier compare in label-efficiency versus a supervised-only baseline when only 10–20% of training labels are available on small sklearn tabular datasets?*

At 10% labeled data, at least one semi-supervised method (LabelPropagation, LabelSpreading, or SelfTrainingClassifier) achieves test accuracy at least 3 percentage points higher than the supervised-only baseline on at least 2 of 3 datasets, averaged over ≥5 seeds.

Across datasets at 10% labeled data, LabelSpreading with RBF kernel attains mean test accuracy greater than or equal to LabelPropagation with RBF kernel in at least 2 of 3 datasets (seed-averaged).

For SelfTrainingClassifier, increasing labeled fraction from 10% to 20% improves mean test accuracy by at least 1 absolute percentage point on at least 2 of 3 datasets.

How do graph-based (LabelPropagation/LabelSpreading) and pseudo-labeling (SelfTrainingClassifier) methods compare to supervised-only training under 10% and 20% labeled-data regimes on small tabular benchmarks?

LogisticRegression trained only on the labeled subset of the training split; unlabeled points ignored.

LabelPropagation with RBF kernel; unlabeled training labels set to -1 and transductive predictions used for test inference.

LabelSpreading with RBF kernel under the same masked-label protocol.

SelfTrainingClassifier wrapping LogisticRegression with confidence threshold (e.g., 0.8), fit on mixed labeled/unlabeled training data.

supervised_logreg_labeled_only is the primary baseline

10% labeled regime serves as a lower-label baseline versus 20% within each method

Classification accuracy on held-out test split, averaged over seeds.

Balanced accuracy on held-out test split, averaged over seeds.

Macro-averaged F1 score on held-out test split, averaged over seeds.

A credible experiment comparing semi-supervised methods (LabelPropagation, LabelSpreading, SelfTrainingClassifier, or equivalents) against a supervised-only baseline under limited-label regimes: methods are implemented correctly, runs cover multiple datasets with repeated seeds, and results address H1/H2/H3 directionally.

Score on scientific substance and directional correctness of evidence, not on exact numeric thresholds. Well-motivated alternative semi-supervised methods that test the same scientific question should be credited.

Semi-supervised and baseline conditions are implemented correctly with explicit label masking.

The submission implements multiple distinct conditions — typically including at least one graph-based method (LabelPropagation or LabelSpreading), a self-training method, and a supervised-only baseline — as separate code paths.

Label masking for semi-supervised runs is explicit and correct: only a defined labeled fraction retains class labels and the remaining training labels are set to the unlabeled marker.

The submission uses multiple datasets (sklearn built-ins or comparable), with a held-out test split and consistent preprocessing across conditions.

Execution covers multiple label fractions and produces benchmark metrics.

Execution outputs numeric test accuracy and at least one additional metric (e.g., balanced accuracy or macro-F1) for each implemented (condition, dataset, label-fraction) cell in a machine-readable artifact.

Each reported cell is aggregated over multiple random seeds with a dispersion measure. Honest small-seed runs with variance reported are preferable to a single run.

Results address H1/H2/H3 directionally with quantitative comparisons.

The submission compares each semi-supervised method versus the supervised-only baseline at a low labeled fraction and conveys whether any method provides a meaningful gain on most datasets — judge directionally against H1.

The submission reports a direct LabelSpreading vs LabelPropagation comparison per dataset and conveys whether LabelSpreading is at least competitive on most datasets (H2).

The submission reports self-training accuracy at both low and higher labeled fractions and conveys whether the gain with more labels is meaningful (H3).

The README or writeup describes setup and methods, reports per-dataset metrics, conveys per-hypothesis outcomes (supported / refuted / inconclusive), and discusses limitations (dataset scope, seed count variance, sensitivity to masking/hyperparameters). No strict word-count requirement.


Classical forecaster robustness on synthetic seasonal time series

Benchmarking classical time-series forecasters (AR, ARIMA, SARIMAX, ETS, Theta) on synthetic seasonal series with varying noise, trend, and seasonality amplitudes

Classical univariate forecasting methods remain widely used because they are
interpretable and lightweight, yet their comparative behavior can flip under
different data-generating regimes. AR models often perform well when dynamics
are mostly autoregressive and weakly seasonal; ARIMA can absorb trend and
differencing structure; SARIMAX can explicitly represent seasonal lag effects;
ETS captures error-trend-seasonal decomposition; and Theta is often a strong
low-variance baseline. In small CPU-constrained settings, practitioners need
practical guidance about which method is robust to changing signal-to-noise
and seasonality strength.

A synthetic benchmark is appropriate here because we can precisely control
trend slope, seasonal amplitude, and observation noise while keeping compute
modest. By generating multiple families of monthly-like series with known
periodicity (e.g., period 12), we can evaluate whether model rankings are
stable or regime-dependent. This avoids overfitting conclusions to one
real-world dataset and enables direct stress testing under high-noise versus
high-seasonality conditions.

A credible study should compare at least four of {AR, ARIMA, SARIMAX, ETS,
Theta} on 2–3 synthetic dataset families, using rolling-origin or holdout
forecasts with a fixed horizon. It should report scale-robust error metrics
(especially sMAPE), include at least one baseline (e.g., seasonal naive), and
aggregate over multiple random seeds. The analysis should explicitly test
whether one method is consistently best overall or whether the best choice
depends on data regime.

The key outcome is actionable selection logic: if seasonal structure is strong
and noise is low, do seasonal/state-space methods dominate; and if noise is
high, do simpler methods become competitive? These are concrete decisions an
autonomous forecasting pipeline can apply when only limited data and CPU are
available.

*How do AR, ARIMA, SARIMAX, ETS, and Theta compare in sMAPE robustness across synthetic seasonal series with controlled trend, seasonality amplitude, and noise levels?*

SARIMAX or ETS achieves the lowest mean sMAPE on at least 2 of 3 synthetic dataset families when seasonality amplitude is high (amplitude >= 8) and noise is low (sigma <= 1.0), averaged over >=5 seeds.

Under high-noise settings (sigma >= 3.0), the sMAPE gap between the best advanced method (ARIMA/SARIMAX/ETS/Theta) and a seasonal_naive baseline is <= 5 percentage points on at least 2 of 3 dataset families.

No single method among {AR, ARIMA, SARIMAX, ETS, Theta} ranks first in mean sMAPE on all evaluated dataset families, indicating regime-dependent winners.

Which classical forecasting method is most robust in sMAPE across controlled synthetic regimes of trend, seasonality amplitude, and noise?

Autoregressive model with lag order selected from {3,6,12} by AIC on training split.

Non-seasonal ARIMA with (p,d,q) searched over small grid p,q in [0,2], d in [0,1], selected by AIC.

Seasonal ARIMA/SARIMAX with seasonal period 12 and small grid over (p,d,q)x(P,D,Q), selected by AIC.

Exponential smoothing with additive trend/seasonality (period 12), damped trend optional by AIC.

theta

ThetaModel forecast with default theta decomposition for univariate series.

seasonal_naive (forecast y_t = y_{t-12}) as a simple seasonal baseline

naive_last (random walk) as a non-seasonal baseline

smape

Symmetric Mean Absolute Percentage Error on forecast horizon, averaged over seeds and series instances.

mae

Mean Absolute Error on forecast horizon.

Per-method wall-clock fit+forecast runtime in seconds.

Synthetic monthly series: length 180, period 12, trend slope in [0.0,0.2], season amplitude in [8,12], gaussian noise sigma in [0.5,1.0].

Synthetic monthly series: length 180, period 12, trend slope in [0.1,0.4], season amplitude in [4,8], gaussian noise sigma in [1.5,2.5].

Synthetic monthly series: length 180, period 12, trend slope in [0.0,0.3], season amplitude in [1,4], gaussian noise sigma in [3.0,4.0].

A credible experiment benchmarking classical forecasters (AR, ARIMA, SARIMAX, ETS, Theta, or equivalents) on synthetic seasonal series: model conditions are implemented, execution covers multiple synthetic regimes with repeated seeds, and results address H1/H2/H3 directionally using sMAPE-centered analysis.

Score on scientific substance and directional correctness of evidence, not on exact numeric thresholds. Alternative forecaster implementations or substitutes that test the same scientific question should be credited.

The forecasting conditions and synthetic data generation pipeline are implemented correctly.

The submission implements multiple distinct forecasting methods — typically AR/ARIMA, a seasonal model (SARIMAX or ETS), and Theta or equivalents — as separate code paths, and includes at least one explicit naive/seasonal-naive baseline.

The submission generates multiple synthetic seasonal dataset families with controllable seasonality, trend, and noise, each with a train/test split suitable for forecasting.

Forecasting setup uses a fixed holdout horizon (or rolling-origin equivalent) and computes comparable forecasts for every (method, dataset, seed) cell.

Benchmark execution logs required metrics with repeated trials.

Execution produces a structured metrics artifact containing numeric sMAPE and MAE (or equivalents) for each implemented method on at least one dataset family.

Each reported method-dataset metric is aggregated over multiple random seeds with dispersion statistics. Honest small-seed runs with variance reported are preferable to a single run.

The run logs per-method wall-clock timing and completes within a CPU-only budget.

Results analysis addresses H1/H2/H3 directionally with quantitative comparisons.

The submission isolates high-seasonality low-noise results and conveys whether SARIMAX or ETS tends to attain the best mean sMAPE — judge directionally against H1.

The submission compares the best advanced method against seasonal-naive under high-noise settings and conveys whether the sMAPE gap collapses to a small margin (H2).

The submission ranks methods by mean sMAPE per dataset family and conveys whether no single method dominates across all families (H3).

The README or writeup reports key metric tables, conveys per-hypothesis outcomes (supported / refuted / inconclusive), and discusses limitations (synthetic realism, grid-search scope, seed count, horizon sensitivity). No strict word-count requirement.


PC vs GES vs NOTEARS-linear for recovering small Gaussian linear-SEM DAGs

Comparing causal structure learning algorithms (PC, GES, NOTEARS-linear) on small synthetic linear-SEM DAGs of 5-15 nodes with Gaussian noise

Causal structure learning methods often claim asymptotic guarantees, but in
practical settings researchers usually face small graphs, finite samples, and
imperfect hyperparameter choices. For Gaussian linear structural equation
models (SEMs), three widely discussed paradigms are constraint-based (PC),
score-based (GES), and continuous optimization (NOTEARS-linear). Even when
data are generated from the model class these methods assume, finite-sample
behavior can differ sharply in edge recovery and orientation errors.

A compact CPU-scale benchmark can isolate these differences by generating
synthetic DAGs with known ground truth over 5-15 nodes, sampling linear-Gaussian
observations, and evaluating structural recovery against the true adjacency.
This avoids data-download overhead and lets the study vary sample size and
graph density in a controlled way. The key metric is Structural Hamming
Distance (SHD), which penalizes missing, extra, and wrongly oriented edges.

A credible study should implement all three algorithms with transparent
assumptions: PC via partial-correlation CI tests, GES via greedy score search
(e.g., BIC), and NOTEARS-linear via a differentiable acyclicity penalty and
L1 regularization. Multiple random seeds and at least two synthetic regimes
(sparser/easier vs denser/harder) are needed to reduce variance and avoid
over-interpreting one-off runs.

Because these algorithms expose different tuning knobs (significance level,
regularization strength, score penalties), the comparison should include one
clear default setting per method plus modest tuning on validation seeds. The
resulting analysis should report SHD and at least one precision/recall-style
edge metric, then map outcomes back to explicit hypotheses about relative
ranking and sample-efficiency.

*On small Gaussian linear-SEM DAGs, which of PC, GES, and NOTEARS-linear most reliably minimizes structural recovery error under realistic finite-sample budgets?*

At n_samples=1000, NOTEARS-linear achieves lower mean SHD than PC on at least 2 of 3 synthetic DAG regimes (averaged over >=5 random DAG/data seeds per regime).

At n_samples=300, GES achieves mean SHD <= PC in all evaluated regimes, and strictly lower SHD in at least 1 regime.

The best method at n_samples=1000 reduces mean SHD by at least 20% relative to the worst method in at least 2 of 3 regimes.

For 5-15 node Gaussian linear-SEM DAGs, how do PC, GES, and NOTEARS-linear compare in SHD and edge recovery across sparse/medium/dense synthetic regimes and finite sample sizes?

Constraint-based PC using Gaussian partial-correlation conditional independence tests with significance level alpha (default 0.01, optional small grid).

Score-based greedy equivalence search using Gaussian/BIC score with forward-backward edge operations.

Continuous optimization NOTEARS-linear with squared loss, acyclicity constraint, and L1 sparsity penalty tuned on a small lambda grid.

PC variant selecting alpha from {0.005, 0.01, 0.05} by lowest validation-seed SHD, then evaluated on held-out seeds.

pc_partial_corr is the classical constraint-based baseline

ges_bic is the classical score-based baseline

shd

Structural Hamming Distance between learned and true DAG adjacency (lower is better), averaged over seeds.

F1 score on directed edge existence/orientation against ground truth adjacency.

Per-run wall-clock time in seconds to assess CPU practicality.

Synthetic 8-node DAGs generated via Erdos-Renyi with expected degree ~1.5; linear weights sampled from [-2,-0.5] U [0.5,2], Gaussian noise.

Synthetic 12-node DAGs with expected degree ~2.5 using same linear-Gaussian SEM generator.

Synthetic 15-node DAGs with expected degree ~3.5 using same linear-Gaussian SEM generator.

A credible experiment comparing causal structure learning methods (PC, GES, NOTEARS-linear, or equivalents) on small synthetic Gaussian linear-SEM DAGs: methods are implemented, runs cover multiple synthetic regimes with multiple seeds, and results address H1/H2/H3 directionally using SHD-centered evidence.

Score on scientific substance and directional correctness of evidence, not on exact numeric thresholds. Alternative structure-learning methods or linear-SEM variants that test the same scientific question should be credited.

Core causal structure learning conditions and synthetic data generation are implemented correctly.

The submission implements distinct code paths for multiple structure learning methods — typically PC, GES, and NOTEARS-linear or equivalents — rather than reusing one estimator under different names.

A reproducible synthetic linear-Gaussian SEM DAG generator is implemented with moderate node counts, including ground-truth adjacency export and sample generation.

At least one tunable hyperparameter is exposed and used for the constraint-based method (e.g., PC's alpha) and the continuous-optimization method (e.g., NOTEARS' sparsity weight), with documented default or grid values.

The benchmark produces comparable metrics across methods/regimes.

Execution covers multiple synthetic regimes and reports results for the primary methods at one or more sample sizes.

A machine-readable results artifact includes numeric SHD for each (method, regime) and at least one additional metric such as edge-F1 or runtime.

Each reported cell is averaged over multiple random seeds with a dispersion measure. Honest small-seed runs with variance reported are preferable to a single run.

Results analysis addresses H1/H2/H3 directionally.

The submission compares NOTEARS-linear vs PC at a reasonable sample size and conveys whether NOTEARS achieves meaningfully lower SHD on most regimes — judge directionally against H1.

The submission conveys qualitative verdicts for H2 and H3 using SHD aggregates, including whether larger sample sizes or denser-graph regimes yield meaningful SHD improvements.

The README or writeup describes setup, reports key SHD/edge-F1/runtime findings, conveys per-hypothesis outcomes (supported / refuted / inconclusive), and notes limitations (synthetic-only scope, small-node regime, hyperparameter sensitivity, finite seeds). No strict word-count requirement.


Active learning query strategies for logistic regression on small tabular pools

Evaluating active learning query strategies (uncertainty sampling, margin sampling, query-by-committee, expected-error reduction) for logistic regression on small UCI pools

Pool-based active learning is attractive when labels are expensive but a
modest unlabeled pool is available. For linear probabilistic models such as
logistic regression, classic query policies include uncertainty sampling,
margin sampling, query-by-committee (QBC), and expected-error reduction
style lookahead approximations. These methods are often taught as broadly
useful, but on small tabular datasets their gains can be inconsistent due to
model misspecification, class imbalance, and high variance from tiny initial
labeled sets.

A useful CPU-bounded benchmark should compare these query strategies under a
fixed annotation budget and shared learner, rather than mixing in model
architecture changes. The core signal is label efficiency: how quickly test
performance improves as labels are acquired. Because final accuracy alone can
hide early-budget differences, the study should include both
accuracy-at-budget and a budget-curve summary metric such as area under the
learning curve.

The experiment should therefore run multiple seeds, use at least two
sklearn-resident tabular classification datasets, and enforce identical
initialization, batch size, and stopping budget across strategies. A random
query baseline is essential to determine whether sophisticated policies
provide real value beyond chance selection. A passive full-data reference is
also useful for contextualizing attainable performance ceilings under the
same logistic-regression family.

Practical constraints matter: expected-error reduction can be expensive if
naively recomputing retrains for every candidate. A tractable variant can
evaluate a capped candidate subset per round and use one-step hypothetical
label outcomes, preserving the spirit of expected future loss minimization
while staying within single-core runtime limits.

*Do uncertainty, margin, QBC, and approximate expected-error reduction deliver better label efficiency than random sampling for logistic regression on small tabular pools under a fixed annotation budget?*

At 30% labeling budget, at least 2 of {uncertainty_sampling, margin_sampling, qbc, expected_error_reduction} achieve higher mean test accuracy than random_sampling by ≥1.5 percentage points, averaged over ≥5 seeds.

expected_error_reduction attains the highest mean area_under_learning_curve (AULC) on at least 1 of 3 datasets, and its AULC is not lower than random_sampling on any evaluated dataset.

qbc outperforms uncertainty_sampling in mean test accuracy at early budget (10% labels) on at least 2 of 3 datasets.

Which active learning query strategy provides the best label efficiency for logistic regression on small tabular pool-based classification tasks under fixed budgets?

Baseline pool-based active learning that queries unlabeled points uniformly at random each round.

Queries samples with highest predictive entropy from current logistic regression model.

Queries samples with smallest top-2 class probability margin (binary: |p-0.5|).

qbc

Query-by-committee with 5 bootstrapped logistic regression committee members; selects by vote entropy/disagreement.

Approximate one-step expected-error reduction over a capped candidate subset per round using hypothetical labels and expected future log-loss reduction.

random_sampling is the primary active-learning baseline

passive_full_data logistic regression reference trained once on all labels for context

Test accuracy at 30% labeled budget, averaged over seeds.

aulc

Area under the test-accuracy-vs-labeled-fraction curve from initial seed set to 30% budget.

Test accuracy when 10% of pool labels have been acquired.

A credible experiment studying active-learning query strategies (random, uncertainty, margin, QBC, expected error reduction, or equivalents) for logistic regression: strategies are implemented as distinct code paths, execution covers multiple datasets with repeated seeds under a fixed budget, and results address H1/H2/H3 directionally.

Score on scientific substance and directional correctness of evidence, not on exact numeric thresholds. Alternative acquisition functions or base classifiers that preserve the scientific question should be credited.

Active-learning strategies and shared logistic-regression pipeline are implemented correctly.

The submission implements multiple distinct query conditions — typically random plus several of {uncertainty, margin, QBC, expected error reduction} — with genuinely different acquisition logic.

There is a pool-based active-learning loop with an initial labeled seed set, iterative querying, model retraining/update, and stopping at a defined label budget.

The submission uses multiple datasets (sklearn built-ins or comparable) with consistent train/pool/test handling for all strategies.

Execution emits budget-aware performance metrics.

Execution produces a machine-readable metrics artifact containing numeric accuracy-at-budget and area-under-learning-curve (or equivalents) for each implemented strategy on at least one dataset.

Each reported (strategy, dataset) result is aggregated over multiple random seeds, with a dispersion measure. Honest small-seed runs with variance reported are preferable to a single run.

The run logs performance across multiple budget checkpoints so early-budget accuracy and AULC are computable from recorded traces.

Quantitative analysis addresses H1/H2/H3 directionally.

The submission compares non-random strategies to random sampling at the final budget and conveys whether informative strategies meaningfully outperform random — judge directionally against H1.

The submission reports per-dataset AULC rankings and conveys whether expected-error-reduction (or a comparable principled strategy) is at least competitive, never falling below random (H2).

The submission compares QBC vs uncertainty sampling at an early-budget checkpoint per dataset and conveys a qualitative verdict (H3).

The README or report describes setup, reports key numeric outcomes, conveys per-hypothesis outcomes (supported / refuted / inconclusive), and notes limitations (runtime approximations, seed count, dataset scope, strategy sensitivity). No strict word-count requirement.


Pointwise vs pairwise vs listwise learning-to-rank on synthetic query-document benchmarks

Comparing learning-to-rank approaches (pointwise linear, pairwise RankNet-lite, listwise ListMLE-lite) on synthetic query-document pairs with known relevance grades

Learning-to-rank (LTR) objectives are often grouped into pointwise, pairwise,
and listwise families, each optimizing a different surrogate of retrieval
quality. Pointwise methods reduce ranking to regression/classification on
individual query-document pairs; pairwise methods optimize relative order
between document pairs; listwise methods directly shape permutations or top-k
emphasis. In production IR systems, metric alignment (especially with NDCG)
is often cited as a reason to prefer pairwise/listwise objectives, but this
claim is difficult to test cleanly on public collections with many confounds.

A compact synthetic benchmark with known latent relevance structure allows a
controlled comparison. If query-specific utility functions generate graded
relevance labels (0-4), we can produce query groups with explicit train/test
splits and evaluate whether objectives aligned with ranking structure obtain
better NDCG@10 than a strong linear pointwise baseline. Because the data are
synthetic, we can vary noise and query heterogeneity while keeping runtime low
enough for single-core CPU execution.

A credible study should implement three distinct training paradigms in the
same feature space: (1) pointwise linear regression on grades, (2) a
RankNet-lite pairwise logistic objective over within-query document pairs,
and (3) a ListMLE-lite listwise objective using per-query permutations sorted
by true grades. Evaluation should report NDCG@10 as primary, plus at least one
secondary ranking metric and a sanity metric such as training loss. Results
should be averaged across multiple seeds and at least two synthetic dataset
regimes.

The key scientific goal is not reproducing a specific paper, but testing
whether increasing objective-level ranking awareness yields measurable gains
in top-k ranking quality under controlled conditions and limited compute.

*Do pairwise and listwise lightweight objectives consistently outperform a pointwise linear baseline on NDCG@10 across synthetic query-document datasets with graded relevance?*

ListMLE-lite achieves higher mean NDCG@10 than pointwise linear regression on at least 2 of 3 synthetic datasets, with an absolute improvement of at least 0.03 on each winning dataset (averaged over >=3 seeds).

At least one of {RankNet-lite, ListMLE-lite} outperforms pointwise linear regression in mean NDCG@10 on all evaluated datasets.

The best ranking-aware method (max of RankNet-lite and ListMLE-lite per dataset) improves mean MAP@10 over pointwise linear by at least 0.02 on at least 2 of 3 datasets.

Do pairwise/listwise lightweight ranking objectives provide consistent top-k retrieval gains over a pointwise linear baseline on synthetic grouped query-document data?

Linear model trained pointwise on query-document feature vectors to predict graded relevance via squared error; ranking by predicted score within each query.

Pairwise logistic ranking objective on within-query document pairs using a linear scorer; optimized with mini-batch gradient descent in numpy.

Listwise ListMLE-style objective with a linear scorer, optimizing likelihood of ground-truth relevance-sorted document permutations per query.

Pointwise linear baseline with L2 regularization tuned over a small grid to ensure a competitive non-ranking-aware baseline.

pointwise_linear is the primary baseline

pointwise_ridge_tuned is a strengthened linear baseline

Mean NDCG@10 across test queries; primary metric.

Mean Average Precision@10 across test queries using relevance>=1 as relevant.

Fraction of correctly ordered document pairs within test queries (based on graded relevance order).

Synthetic generator: 80 queries x 25 docs/query, 20 dense features, low noise, relevance grades 0-4 from latent linear utility.

Synthetic generator: 100 queries x 30 docs/query, 25 features, higher Gaussian noise and feature corruption, grades 0-4.

Synthetic generator: 90 queries x 20 docs/query, 40 features with sparsity mask, query-specific weight drift, grades 0-4.

A credible experiment comparing pointwise, pairwise (RankNet-style), and listwise (ListMLE-style) ranking approaches on synthetic query-document datasets: methods are implemented as distinct objectives, execution reports ranking metrics on multiple datasets with multiple seeds, and results address H1/H2/H3 directionally.

Score on scientific substance and directional correctness of evidence, not on exact numeric thresholds. Well-motivated ranking-objective substitutes (e.g., LambdaRank-style, softrank) that test the same scientific question should be credited.

The ranking methods and synthetic grouped datasets are implemented correctly.

The submission implements multiple distinct conditions — typically a pointwise baseline plus at least one pairwise and one listwise method — with genuinely different objective computations, not merely different hyperparameters of one loss.

A synthetic query-document generator is implemented with grouped queries, per-query document lists, and graded relevance labels, and multiple dataset regimes are instantiated.

The experimental setup includes query-level train/test splitting (no leakage of documents from the same query across splits) and a shared scoring-function interface across methods.

Execution logs ranking metrics per method and dataset.

Execution produces a metrics artifact containing numeric NDCG@k and MAP@k (or equivalents) for each implemented condition on at least one dataset.

Reported metrics are averaged over multiple random seeds per (condition, dataset) cell with a dispersion measure. Honest small-seed runs with variance reported are preferable to a single run.

At least one non-trivial hyperparameter search or documented default is executed (e.g., learning rate/regularization) and the chosen value is logged.

Results analysis addresses H1/H2/H3 directionally.

The submission compares the listwise method vs the pointwise baseline in mean NDCG@k per dataset and conveys whether listwise meaningfully improves ranking quality — judge directionally against H1.

The submission conveys whether at least one ranking-aware method beats the pointwise baseline on NDCG across all datasets (H2) and whether the best ranking-aware method yields a meaningful MAP improvement on most datasets (H3).

The README or writeup describes methods, datasets, and metric outcomes; conveys per-hypothesis outcomes (supported / refuted / inconclusive); and discusses limitations (synthetic-data realism, scorer capacity, seed count, metric sensitivity). No strict word-count requirement.


Online binary classification under concept drift: SGD, PA, NB, and FTRL

Benchmarking online learning algorithms (SGD logistic, Passive-Aggressive, online Naive Bayes, Follow-the-Regularized-Leader) on streaming binary classification with concept drift

In streaming classification, data arrive sequentially and model updates must
be cheap and immediate. Under concept drift, the relationship between
features and labels changes over time, so static train/test evaluation can
be misleading. Prequential (test-then-train) evaluation better reflects
deployment: each incoming point is first predicted, then used for an update.
This setting creates practical trade-offs among stability, adaptability, and
computational cost.

Several lightweight online learners in sklearn-style workflows represent
different adaptation biases. SGD logistic regression can track drift via
gradient updates but may require careful learning-rate choices. Passive-
Aggressive updates can react strongly to mistakes and often adapt quickly to
abrupt shifts. Online Naive Bayes is extremely cheap and robust but may lag
when feature dependence structure changes. A simple FTRL-style proximal
logistic update (implemented with diagonal accumulators) offers adaptive
per-feature learning rates and implicit regularization.

A credible CPU-only benchmark should compare these methods on at least one
synthetic abrupt-drift stream and one real sklearn dataset converted to a
stream with induced drift (e.g., blockwise class-prior or feature transform
shift). It should report prequential accuracy as the primary metric, plus at
least one drift-sensitive secondary metric such as post-drift recovery and
cumulative log loss. Multi-seed runs are important because stream order and
drift-point randomness can materially change outcomes.

The study should also verify that drift actually hurts models by comparing a
no-drift control stream against drifted streams, and should discuss which
algorithms recover fastest after drift while maintaining overall accuracy.
This makes the benchmark more diagnostic than a single aggregate score.

*Which lightweight online learner provides the best trade-off between overall prequential accuracy and adaptation speed after concept drift in binary streams?*

On drifted streams, Passive-Aggressive or FTRL achieves higher prequential_accuracy than online Naive Bayes by at least 0.02 absolute on at least 2 of 3 datasets, averaged over >=5 seeds.

For at least 2 of 3 drifted datasets, the best post-drift recovery accuracy in a fixed window of 200 samples after each drift point is achieved by either Passive-Aggressive or FTRL.

For SGD logistic and Passive-Aggressive, prequential_accuracy on a drifted version of each stream is at least 0.03 lower than on its matched no-drift control, on at least 2 of 3 datasets.

Which online method among SGD logistic, Passive-Aggressive, online Naive Bayes, and FTRL-style logistic best balances prequential accuracy and post-drift recovery on binary data streams with concept drift?

SGDClassifier with log_loss and partial_fit in prequential test-then-train loop.

PassiveAggressiveClassifier with partial_fit in the same loop.

GaussianNB updated incrementally via partial_fit.

Custom numpy FTRL-proximal logistic regression with diagonal accumulator and L1/L2 regularization.

online_naive_bayes as a simple low-cost baseline

sgd_logistic as a standard linear online-learning baseline

Mean test-then-train accuracy over the full stream, averaged over 5 seeds.

Accuracy in the first 200 samples after each drift point, averaged across drift events and seeds.

Cumulative mean log loss computed prequentially when probabilistic outputs are available (or clipped score-to-prob mapping).

numpy synthesis (piecewise make_classification with coefficient/sign flips at fixed indices)

sklearn.datasets.load_breast_cancer transformed into a repeated/shuffled stream with blockwise feature scaling shift

numpy synthesis with sparse informative features and class-prior shift across blocks

A credible experiment studying online learners (SGD logistic, Passive-Aggressive, online Naive Bayes, FTRL-style logistic, or equivalents) under concept drift: streaming prequential setup is implemented, execution covers multiple datasets with multiple seeds, and results address H1/H2/H3 directionally.

Score on scientific substance and directional correctness of evidence, not on exact numeric thresholds. Well-motivated online-learner substitutes or drift-stream variants that test the same scientific question should be credited.

Online-learning conditions and drifted streaming setup are implemented correctly.

The submission implements multiple distinct online methods, each as a real incremental update path (e.g., partial_fit or equivalent), not batch retraining.

A prequential test-then-train loop is implemented: each sample (or mini-batch) is predicted before being used for model update, with ordered stream processing.

The submission uses multiple streams including at least one drifted stream and one no-drift or matched-control stream, generated from sklearn datasets or numpy synthesis.

Execution reports prequential metrics for each condition.

Execution produces a metrics artifact containing numeric prequential accuracy (or equivalent) for each implemented method on at least one dataset.

Execution includes a post-drift recovery measure (e.g., fixed-window post-drift accuracy) with a documented computation referencing known drift points.

Each reported (dataset, method) metric is averaged over multiple random seeds, with a dispersion measure. Honest small-seed runs with variance reported are preferable to a single run.

Quantitative analysis addresses H1/H2/H3 directionally.

The submission compares prequential accuracy of margin-based online methods (Passive-Aggressive, FTRL) against online Naive Bayes across drifted datasets and conveys whether margin-based methods are meaningfully better — judge directionally against H1.

The submission reports post-drift recovery comparisons and conveys whether Passive-Aggressive or FTRL tends to lead on most datasets (H2).

The submission compares drifted vs no-drift control performance for the implemented methods and conveys whether drift produces a meaningful accuracy degradation (H3).

The README or writeup describes setup, key metrics, conveys per-hypothesis outcomes (supported / refuted / inconclusive), and notes limitations (stream realism, drift design choices, seed count, metric assumptions). No strict word-count requirement.


Reservoir computing vs MLP vs GP for short-horizon Lorenz-63 forecasting

Testing whether a random-feature echo state network (reservoir computing) predicts Lorenz-63 short-horizon trajectories more accurately than a matched-parameter MLP and a Gaussian Process at small training-data sizes (N=200-2000 points)

Reservoir computing (a.k.a. echo state networks, ESN) is a lightweight
approach for learning dynamical systems: a fixed random recurrent network
projects the input into a high-dimensional reservoir state, and only a
linear readout layer is trained. For chaotic systems, ESNs have repeatedly
been reported to match or outperform trainable recurrent nets on
short-horizon trajectory prediction, despite training only a ridge
regression on the readout.

A credible CPU-scale study of this claim must compare (i) a reservoir
network with a random sparse internal matrix and fixed spectral radius,
(ii) a parameter-matched fully-connected MLP that reads a fixed window of
past states, and (iii) a Gaussian Process regressor with an RBF kernel on
the same windowed input. The target signal is Lorenz-63 integrated at
dt=0.02 with the classical (σ=10, ρ=28, β=8/3) parameters. Training-set
sizes N ∈ {200, 500, 1000, 2000} are small enough that the standard
reservoir-computing folklore — "ESNs win when data is scarce" — can
actually be tested rather than assumed.

The key dynamics-aware metric is the *valid prediction time* (VPT):
the first time step at which the normalized prediction error exceeds a
threshold (commonly 0.4 of the attractor standard deviation). Unlike
per-step RMSE, VPT tracks how long a model's forecast remains useful on
the chaotic attractor. Reporting both RMSE and VPT reveals whether any
observed "superiority" is numerical or dynamical.

The research question is: *does a small reservoir (N_res ≤ 500 units) beat
a matched-parameter MLP and a Gaussian Process on valid prediction time
for Lorenz-63, and at which training-set size does the gap appear?*

An echo-state network with spectral radius ≈ 0.9 and ≤ 500 reservoir units achieves a longer mean Valid Prediction Time (VPT ≥ 1.2×) than a parameter-matched MLP on Lorenz-63 at training-set sizes N ≤ 500.

The Gaussian Process regressor is competitive with (VPT within ±20% of) the ESN at N=200 but is decisively beaten (ESN VPT at least 2× the GP's) at N=2000, reflecting the GP's O(N^3) training bottleneck and the ESN's ability to exploit more data without re-inverting a Gram matrix.

Per-step RMSE at one-step-ahead prediction is NOT a reliable proxy for VPT: at least one condition exists where model A has lower one-step RMSE than model B but a shorter VPT, showing that per-step error understates the divergence on chaotic attractors.

On Lorenz-63 short-horizon forecasting, does a small echo-state network achieve a longer valid prediction time than a parameter-matched MLP and a Gaussian Process regressor, and how does the ranking depend on training-set size?

Echo-state network with 300 reservoir units, spectral radius 0.9, input scaling 1.0, leak rate 0.3, ridge regression readout (alpha=1e-5). Trained on N=200 consecutive Lorenz-63 samples.

Same ESN configuration, N=500 training samples.

Same ESN configuration, N=1000 training samples.

Same ESN configuration, N=2000 training samples.

Fully-connected MLP with 2 hidden layers whose total parameter count is within ±10% of the ESN readout parameter count. Reads a 5-step window of past (x,y,z) states as input. Adam optimizer, early stopping on a 10% validation split.

Gaussian Process regressor with an RBF kernel and WhiteNoise term, hyperparameters optimised by marginal likelihood maximisation. Reads the same 5-step window as the MLP. At N=2000 the GP may time out or run out of memory; record the failure.

Persistence baseline: predicted next state = previous state. Establishes a floor for VPT.

Linear autoregressive baseline: one-step-ahead linear regression on the 5-step window. Establishes whether the non-linear methods' gains are real.

First forecast step (in units of integration dt) at which the normalized L2 error exceeds 0.4 × attractor std. Averaged over 10 random initial conditions on an independent test trajectory.

Root-mean-square error of one-step-ahead prediction on the test trajectory.

Wall-clock time to fit the model, including hyperparameter optimisation where applicable.

Number of trainable parameters. Must be reported to demonstrate that the MLP is parameter-matched to the ESN.

synthetic — integrate dxdt=σ(y-x), dy/dt=x(ρ-z)-y, dz/dt=xy-βz at dt=0.02 with σ=10, ρ=28, β=8/3. Use scipy.integrate.solve_ivp or a hand-rolled RK4.

independent Lorenz-63 trajectory (different IC, 10× longer warmup) with 10 evaluation windows for VPT averaging.

A credible CPU-scale study of reservoir computing (echo-state network), a matched-parameter MLP, and a Gaussian Process on Lorenz-63 short-horizon forecasting: Lorenz-63 is integrated, the three model families are implemented, multiple training-set sizes are swept, valid-prediction-time is reported, and results address H1/H2/H3 directionally.

Score on scientific substance and directional correctness of evidence, not on exact numeric thresholds. Reasonable substitutes (e.g., alternative chaotic system, library-based ESN when clearly documented, Matern-kernel GP) that test the same scientific question should be credited. A GP running out of memory at large N is a valid reported outcome.

The three model families and the dynamical system are implemented.

The submission integrates the Lorenz-63 system with classical parameters at a small time step using scipy.integrate or a hand-rolled Runge-Kutta integrator, and uses separate train/test trajectories.

An echo-state network is implemented with a sparse random reservoir scaled to a target spectral radius below ~1.1, driven by the input sequence with a leak-rate update and a linear readout fit by ridge regression. A from-scratch implementation is preferred; a clearly documented library-based ESN that exposes the same mechanics is acceptable.

An MLP is implemented that reads a fixed window of past states and predicts the next state, with its total parameter count reported and roughly matched to the ESN readout parameter count.

A Gaussian Process regressor with an RBF or Matern kernel is fit on the same windowed input, with marginal-likelihood hyperparameter optimisation enabled.

Execution produces per-method, per-N quantitative results.

The experiment sweeps multiple training-set sizes for each of the three methods and reports numeric results per cell.

A valid-prediction-time (VPT) metric is computed per method per N — the first step at which the normalised prediction error exceeds a threshold — averaged over multiple independent initial conditions or seeds.

A one-step-ahead RMSE (or equivalent pointwise error) is computed per method per N on the test trajectory and reported in the metrics file.

The writeup addresses H1/H2/H3 directionally and contextualises numeric findings.

The submission reports VPT for the ESN vs MLP at small training sizes and conveys whether the ESN meaningfully outperforms the MLP in the small-N regime — judge directionally against H1.

The submission compares ESN and GP VPT at both small-N and larger-N and conveys whether the GP is competitive at small N and falls behind (or becomes infeasible) at large N (H2). Reporting a GP memory/time failure at large N is a valid outcome.

The submission examines whether one-step RMSE ranking tracks VPT ranking across method-N cells and conveys whether at least one rank inversion is observed (H3).

The README or writeup describes the Lorenz-63 setup, the three methods, the N-sweep, the VPT and RMSE numbers, conveys per-hypothesis outcomes (supported / refuted / inconclusive), and notes limitations (single dynamical system, no Lyapunov-time normalisation, GP memory cliff). No strict word-count requirement.


Reproducing the Randall-Sundrum KK-graviton resonance pattern in $e^+e^- \to \mu^+\mu^-$

Reproducing warped extra dimension Kaluza-Klein graviton resonance from arXiv:hep-ph/9909255

Reproducing the cross-section vs $\sqrt{s}$ scan of $e^+e^- \to \mu^+\mu^-$
in the Randall-Sundrum warped extra dimension scenario as a benchmark of
the agent's ability to assemble a full Lagrangian -> MC -> analysis
pipeline. The model adds a tower of massive spin-2 KK gravitons
$h^{(n)}_{\alpha\beta}$ ($n=1,\ldots,5$) coupled to the SM
energy-momentum tensor with strength $1/\Lambda_\pi$, on top of the
massless zero-mode coupled with $1/\bar M_{Pl}$. The diagnostic
observable is $\sigma(e^+e^- \to \mu^+\mu^-)$ scanned over
$\sqrt{s}\in\{200,300,\ldots,1200\}$ GeV, which exhibits sharp
resonance peaks at the KK masses $m_n = \{600, 1098, 1592, 2086, 2580\}$
GeV where they sit in the scan window, and an off-resonance continuum
enhancement above SM Drell-Yan elsewhere.

A credible study (a) implements the spin-2 graviton-stress-tensor
Lagrangian (FeynRules + UFO or an equivalent analytic propagator),
(b) generates parton-level events at each of the 11 energy points with
proper handling of the unitary-gauge graviton propagator,
(c) extracts $\sigma(\sqrt{s})$ from the MadGraph banner for each run,
(d) plots $\sigma$ vs $\sqrt{s}$ on a log-y axis over [50, 1500] GeV with
range $[400, 5\times 10^6]$ fb, and (e) compares the resonance positions
and continuum heights against the published Figure 2. Research question:
*to what extent does an autonomous HEP pipeline reproduce the RS
KK-graviton resonance spectrum imprinted on dilepton production at a
500 GeV-1 TeV $e^+e^-$ collider?*

The first KK resonance $m_1 = 600$ GeV produces a peak in $\sigma(e^+e^- \to \mu^+\mu^-)$ at $\sqrt{s} = 600$ GeV that lies within 5% of the input mass.

Off-resonance cross sections at $\sqrt{s}=200$ and $\sqrt{s}=400$ GeV agree with the published Figure 2 values within 30%, demonstrating correct continuum (zero-mode + virtual KK) interference.

At least 2 of the 5 KK resonances within the [50,1500] GeV scan window ($m_1=600$, $m_2=1098$ GeV) are visible as $\geq 1$-decade enhancements above the SM Drell-Yan baseline at $\sqrt{s}=m_n$.

Does the agent reproduce the Randall-Sundrum KK-graviton resonance pattern in $e^+e^- \to \mu^+\mu^-$ across the $\sqrt{s}=200$-$1200$ GeV scan, matching the published positions and continuum heights of Figure 2 of arXiv:hep-ph/9909255?

Scan $\sqrt{s} \in \{200, 300, 400, 500, 600, 700, 800, 900, 1000, 1100, 1200\}$ GeV with $E^+ = E^- = \sqrt{s}/2$, 100 events per energy point. KK masses fixed at $m_n = \{600, 1098, 1592, 2086, 2580\}$ GeV; $\Lambda_\pi = 522$ GeV; $\bar M_{Pl} = 2.4 \times 10^{18}$ GeV.

Same energy scan with all KK gravitons decoupled ($1/\Lambda_\pi \to 0$) to obtain the pure SM $\gamma^*/Z$ Drell-Yan baseline for resonance-significance comparison.

Pure SM $e^+e^- \to \mu^+\mu^-$ (Drell-Yan via $\gamma^*/Z$) at the same energy points

Total $\sigma(e^+e^- \to \mu^+\mu^-)$ in pb at each of the 11 $\sqrt{s}$ points.

Reconstructed $\sqrt{s}$ at which the first KK resonance peaks; target = $600 \pm 30$ GeV.

Ratio $\sigma(\sqrt{s}=m_1) / \sigma(\sqrt{s}=m_1 - 200\,\mathrm{GeV})$; should exceed $\sim 10$.

$e^+e^- \to \mu^+\mu^-$ in the RS warped-graviton model.

A canonical results.json file exists at the workspace root with at least the keys: primary_metric (number), metric_key (string), metrics (object with numeric keys), hypotheses (object with h1/h2/h3 entries each carrying a `supported` boolean), summary (non-empty string).

results.json metrics MUST contain at least 3 numeric (non-null, finite) values directly relevant to the headline physics observable named in the experiment_design.metrics list above — these are the numbers the paper will report in its Results section.

results.json hypotheses.h1/h2/h3 each MUST have an explicit `supported` boolean AND a `details` string ≥ 40 characters quoting the numerical evidence (specific values + their source artifact) used to reach the verdict.

At least one publication-quality figure file (PDF or PNG, ≥150 DPI for raster) exists under figures/ or output/figures/ with axes labeled in physical units (GeV / pb / fb / dimensionless) and a legend if multiple series are plotted.  The figure must directly support a hypothesis verdict.

The BSM Lagrangian is implemented either as a FeynRules .fr file (models/*.fr) with a matching UFO directory (models/*_UFO/ containing at least particles.py, parameters.py, couplings.py, vertices.py), OR as analytic Python code that explicitly computes the cross sections from the Lagrangian terms.  A pure SM baseline with no BSM piece is NOT sufficient.

results.json metrics MUST report a numeric `peak_position_gev` (or equivalent first-KK-resonance position) within ±5% of 600 GeV — i.e. in the interval [570, 630] GeV — after the RS scan over √s ∈ {200..1200} GeV.

The summary or structured_results section provides a one-paragraph mechanistic interpretation of WHY the headline observable comes out the way it does (which interference / propagator structure / cut effect drives the result).  Nice-to-have, not blocking proceed.

results.json or a sibling reproducibility section names: (a) the MadGraph5_aMC@NLO version, (b) the PDF set used (if applicable), (c) at least one explicit random seed.  Required for full reproducibility but not for scientific correctness.

A credible HEP collider study reproducing the published reference figure for RS warped extra dimension KK-graviton resonance in $e^+e^- \to \mu^+\mu^-$. The agent must (a) implement the BSM Lagrangian, (b) generate Monte Carlo events for the specified process with appropriate PDFs and generation cuts, (c) run the necessary detector / shower simulation if required, (d) reconstruct the diagnostic kinematic distribution, and (e) compare against the published reference figure with quantitative residuals tied to H1, H2, H3.

Physics topics are scored on whether the agent (i) assembles a working HEP MC pipeline AND (ii) reproduces the published numbers within stated tolerance. Partial credit is allowed; the agent may use analytic shortcuts in place of MadGraph for cases where it is overkill, as long as the answer matches the published reference within the per-leaf tolerance. Quantitative result leaves are scored on a graded scale: 100% if relative error < 3%, 67% if < 10%, 33% if < 30%, 0% otherwise.

Code-development bucket: BSM model + process card + PDF/cuts + full pipeline are wired correctly.

BSM Lagrangian implemented (FeynRules .fr file + UFO output, or equivalent analytic Python module): Randall-Sundrum spin-2 KK graviton tower coupled to SM stress-energy tensor with $1/\Lambda_\pi$ (massive modes) and $1/\bar M_{Pl}$ (zero mode); five massive modes $m_n \in \{600, 1098, 1592, 2086, 2580\}$ GeV with $\Lambda_\pi = 522$ GeV.

MadGraph generate-process card or analytic process module matches the paper's process string: $e^+e^- \to \mu^+\mu^-$ at energies $\sqrt{s} \in \{200,\ldots,1200\}$ GeV with KK-graviton tower.

PDF set and generation-level cuts match the paper exactly: Lepton-collider beams $E^\pm = \sqrt{s}/2$, no proton PDF; widths of all KK gravitons computed automatically (two-body decays only); 100 events per energy point.

Full HEP chain wired together (model -> events -> [shower -> detector] -> reconstruction -> analysis -> figure). For analytic shortcuts the chain may be simpler but must still terminate in the diagnostic figure.

Execution-validity bucket: MC sample passes nevents threshold, physics-validity gates clear, MC stats are sufficient.

MC sample(s) generated successfully with at least the requested number of events per cell (or a documented partial count if compute-limited), with no fatal MadGraph or downstream errors.

Physics-validity gate: no negative cross sections; gauge invariance / unitarity sanity checks pass; mass spectrum and decay widths printed by MadGraph match the input parameter card; no obviously unphysical kinematics in the generated events.

MC statistics are sufficient: relative MC error in the signal region is < 10% OR at least 3 random seeds are run with std reported across them.

Results bucket: quantitative comparison of three hypothesis-driven observables to the published figure, plus a reproduced figure artifact and a per-hypothesis writeup.

Quantitative test of H1 — First KK resonance peaks within 5% of input $m_1=600$ GeV (i.e. peak in $[570, 630]$ GeV) — score 100% if rel. err. < 3%, 67% if < 10%, 33% if < 30%, 0% otherwise.

Quantitative test of H2 — Off-resonance cross sections at $\sqrt{s} \in \{200, 400\}$ GeV agree with the published Figure 2 within $\pm 30\%$. Use the same graded relative-error scale as H1.

Quantitative test of H3 — At least 2 of the 5 KK resonances within the [50,1500] GeV scan window appear as $\geq 1$-decade enhancements above the SM Drell-Yan baseline. Use the same graded relative-error scale as H1.

A reproduced figure exists in artifacts (PDF or PNG) with axes, units, ranges, and legend matching the published reference: Reproduced figure shows $\sigma$ vs $\sqrt{s}$ with x-axis $\sqrt{s}$ in GeV (linear, [50,1500]), y-axis $\sigma$ in fb (log, $[400, 5\times 10^6]$), blue line, all resonances visible.

Writeup discusses the outcome of each hypothesis (supported / refuted / inconclusive) tied to numeric residuals, identifies the dominant systematic uncertainty (PDF set, scale variation, k-factor, Delphes card, statistical MC error), and states explicitly any analytic shortcuts taken in place of full MC.

Reproducibility bucket (new for physics topics; absent in T-rubrics): the artifacts contain everything needed to rerun the analysis from scratch.

UFO model directory (or analytic model Python module) is checked into the artifacts and is loadable by MadGraph (or directly importable for analytic models).

MadGraph run_card / param_card, Pythia8 settings, Delphes card, and MadAnalysis5 (or equivalent analysis) cards are saved alongside the run with explicit parameter values per cell.

RNG seeds are documented; rerunning the run from saved seeds should give matching central values within the reported MC error. For analytic shortcuts, the seed requirement is replaced by an explicit list of input numerical parameters.


Reproducing $pp \to \mu^\pm N$ cross section vs heavy-Majorana-neutrino mass at $\sqrt{s} = 7, 8, 14$ TeV

Reproducing heavy Majorana neutrino production $pp \to \mu^\pm N$ at LHC 7/8/14 TeV from arXiv:1308.2209

Reproducing the LHC sensitivity to a single heavy Majorana neutrino $N$
in the muon channel as a benchmark of the agent's ability to assemble a
hadron-collider Lagrangian -> MC -> analysis pipeline. The BSM extension
adds a right-handed singlet $N$ mixing with $\nu_\mu$ via the
dimensionless real parameter $V_{\mu N}$, so the relevant interaction is
$\mathcal{L}_N \supset -(g_2/\sqrt{2})\,V_{\mu N}\,(\bar\mu\gamma^\mu P_L
N)\,W^-_\mu + (\text{NC term}) + \text{h.c.}$ The diagnostic observable
is the $W^*$-mediated cross section $\sigma(pp \to \mu^\pm N)$ scanned
over $m_N \in \{100, 200, \ldots, 1000\}$ GeV with $|V_{\mu N}|=1$, at
three LHC energies $\sqrt{s} \in \{7, 8, 14\}$ TeV with the CTEQ6L PDF.

A credible study (a) implements the heavy-Majorana-neutrino Lagrangian
(FeynRules .fr or analytic UFO modification), (b) sets up MadGraph runs
for the 30 (mass, energy) cells with the CTEQ6L PDF, (c) extracts the
cross section per cell, and (d) plots three side-by-side panels of
$\sigma(m_N)$ on a log-y axis over $[100,1000]$ GeV vs $[0.1, 4\times
10^4]$ fb. Research question: *does the agent reproduce the steep
PDF-driven fall-off of $\sigma(pp \to \mu^\pm N)$ with $m_N$ and the
correct ordering between $\sqrt{s}=7$, 8, and 14 TeV predicted by
arXiv:1308.2209 Fig. 3?*

At $m_N = 200$ GeV and $\sqrt{s}=14$ TeV, $\sigma(pp \to \mu^\pm N)$ falls within $\pm 30\%$ of the published reference value.

Cross-section ordering $\sigma(14\,\mathrm{TeV}) > \sigma(8\,\mathrm{TeV}) > \sigma(7\,\mathrm{TeV})$ holds at every mass point, with the 14/7 ratio at $m_N = 500$ GeV in $[5, 30]$.

$\sigma(m_N)$ falls by at least 2 orders of magnitude between $m_N = 100$ GeV and $m_N = 1000$ GeV at every LHC energy.

Does the agent reproduce the cross-section curves for $pp \to \mu^\pm N$ as a function of the heavy-Majorana-neutrino mass at the 7, 8, and 14 TeV LHC, matching Figure 3 of arXiv:1308.2209?

$pp$ collisions at $\sqrt{s}=7$ TeV with CTEQ6L PDF, mass scan $m_N \in \{100, 200, 300, 400, 500, 600, 700, 800, 900, 1000\}$ GeV, $|V_{\mu N}|=1$.

Same setup at $\sqrt{s}=8$ TeV (LHC Run 1 final dataset energy).

Same setup at $\sqrt{s}=14$ TeV (HL-LHC design).

Charged-current SM $pp \to \mu \nu_\mu$ (W production) cross check at the same energies

Total $\sigma(pp \to \mu^\pm N)$ in fb at each (mass, energy) cell.

$\sigma(14\,\mathrm{TeV})/\sigma(7\,\mathrm{TeV})$ at $m_N = 500$ GeV.

$\sigma(m_N=100)/\sigma(m_N=1000)$ at each $\sqrt{s}$.

$pp \to \mu^\pm N$ via $W^*$ at the LHC (CTEQ6L PDF). Three sub-samples for $\sqrt{s} \in \{7,8,14\}$ TeV.

results.json metrics MUST report a numeric `sigma_pp_to_muN_at_mN200_14TeV_fb` (or equivalent) within ±30% of the published reference value at m_N=200 GeV, √s=14 TeV.

A credible HEP collider study reproducing the published reference figure for Heavy Majorana neutrino $pp \to \mu^\pm N$ at LHC 7/8/14 TeV. The agent must (a) implement the BSM Lagrangian, (b) generate Monte Carlo events for the specified process with appropriate PDFs and generation cuts, (c) run the necessary detector / shower simulation if required, (d) reconstruct the diagnostic kinematic distribution, and (e) compare against the published reference figure with quantitative residuals tied to H1, H2, H3.

Code-development bucket: BSM model + process card + PDF/cuts + full pipeline are wired correctly.

BSM Lagrangian implemented (FeynRules .fr file + UFO output, or equivalent analytic Python module): BSM single right-handed Majorana neutrino $N$ mixing with the muon flavor: $\mathcal{L}_N \supset -(g_2/\sqrt 2) V_{\mu N} (\bar\mu \gamma^\mu P_L N) W^-_\mu - (g_2/2 c_W) V_{\mu N} (\bar\nu_\mu \gamma^\mu P_L N) Z_\mu + \mathrm{h.c.}$ at $|V_{\mu N}| = 1$.

MadGraph generate-process card or analytic process module matches the paper's process string: $pp \to \mu^\pm N$ via $W^*$ at the LHC, mass scan $m_N \in \{100,\ldots,1000\}$ GeV at three energies.

PDF set and generation-level cuts match the paper exactly: CTEQ6L PDF for all $\sqrt{s}$ values; 30 (mass, energy) cells (10 masses x 3 energies).

Full HEP chain wired together (model -> events -> [shower -> detector] -> reconstruction -> analysis -> figure). For analytic shortcuts the chain may be simpler but must still terminate in the diagnostic figure.

Execution-validity bucket: MC sample passes nevents threshold, physics-validity gates clear, MC stats are sufficient.

MC sample(s) generated successfully with at least the requested number of events per cell (or a documented partial count if compute-limited), with no fatal MadGraph or downstream errors.

Physics-validity gate: no negative cross sections; gauge invariance / unitarity sanity checks pass; mass spectrum and decay widths printed by MadGraph match the input parameter card; no obviously unphysical kinematics in the generated events.

MC statistics are sufficient: relative MC error in the signal region is < 10% OR at least 3 random seeds are run with std reported across them.

Results bucket: quantitative comparison of three hypothesis-driven observables to the published figure, plus a reproduced figure artifact and a per-hypothesis writeup.

Quantitative test of H1 — $\sigma(pp \to \mu^\pm N)$ at $m_N=200$ GeV, $\sqrt{s}=14$ TeV agrees with reference within $\pm 30\%$.

Quantitative test of H2 — Energy ordering $\sigma(14) > \sigma(8) > \sigma(7)$ TeV holds at every mass; 14/7 ratio at $m_N=500$ GeV in $[5, 30]$. Use the same graded relative-error scale as H1.

Quantitative test of H3 — $\sigma(m_N)$ falls $\geq 2$ orders of magnitude from $m_N=100$ to $m_N=1000$ GeV at every $\sqrt{s}$. Use the same graded relative-error scale as H1.

A reproduced figure exists in artifacts (PDF or PNG) with axes, units, ranges, and legend matching the published reference: Three-panel figure (one per $\sqrt{s}$), 1:1 aspect, x-axis $m_N \in [100,1000]$ GeV linear, y-axis $\sigma \in [0.1, 4\times10^4]$ fb log.

Writeup discusses the outcome of each hypothesis (supported / refuted / inconclusive) tied to numeric residuals, identifies the dominant systematic uncertainty (PDF set, scale variation, k-factor, Delphes card, statistical MC error), and states explicitly any analytic shortcuts taken in place of full MC.

Reproducibility bucket (new for physics topics; absent in T-rubrics): the artifacts contain everything needed to rerun the analysis from scratch.

UFO model directory (or analytic model Python module) is checked into the artifacts and is loadable by MadGraph (or directly importable for analytic models).

MadGraph run_card / param_card, Pythia8 settings, Delphes card, and MadAnalysis5 (or equivalent analysis) cards are saved alongside the run with explicit parameter values per cell.

RNG seeds are documented; rerunning the run from saved seeds should give matching central values within the reported MC error. For analytic shortcuts, the seed requirement is replaced by an explicit list of input numerical parameters.


Reproducing 8 TeV LHC dilepton exclusion contours for a B-L Z' boson with kinetic mixing

Reproducing 8 TeV LHC dilepton exclusion contours for B-L Z' with kinetic mixing from arXiv:1605.02910

high-energy-physics bsm-phenomenology z-prime statistical-recast

Reproducing the LHC Run 1 sensitivity to a B-L gauge boson Z' with
kinetic mixing as a benchmark of the agent's ability to combine MC
parameter scans with an analytic statistical recast. The model has a
Z' coupling to SM fermions via a left/right combination
$C_{f,L/R}(\tilde g, g_1', M_{Z'})$ where $g_1'$ is the B-L gauge
coupling, $\tilde g$ is the kinetic mixing, and the production cross
section is the exact quadratic form
$\sigma(pp \to Z') = A g_1'^2 + B g_1' \tilde g + C \tilde g^2$ — so
three runs per mass point determine $(A,B,C)$ for the entire $(\tilde
g, g_1')$ plane. The diagnostic observable is the three-panel exclusion
contour at $\mathrm{Sig}\equiv 2(\sqrt{S+B}-\sqrt B) = 2$ for
$M_{Z'} \in \{2.0, 2.5, 3.0\}$ TeV at $\mathcal{L} = 20$ fb$^{-1}$.

A credible study (a) implements the kinetic-mixing B-L Lagrangian and
$C_{f,L/R}$ couplings, (b) runs the 9 signal MadGraph runs (3 mass
points x 3 coupling points) plus 3 SM Drell-Yan background runs with
$m_{\ell\ell} > 0.9 M_{Z'}$ generator cut, (c) computes the analytic
$Z' \to e^+e^- + \mu^+\mu^-$ branching ratio including 3 degenerate
heavy Majorana neutrinos $m_{\nu_h}=95$ GeV, (d) applies the published
NNLO k-factors $k\in\{1.35,1.40,1.50\}$ and acceptance
$\epsilon_\mathrm{acc}=0.6$, (e) solves the $\mathrm{Sig}=2$ equation
on a 2D grid in $(\tilde g, g_1')$, and (f) plots three side-by-side
contours. Research question: *does the agent reproduce the three-panel
$(\tilde g, g_1')$ exclusion contours at $M_{Z'}=2.0, 2.5, 3.0$ TeV from
Figure 1 of arXiv:1605.02910?*

For the $M_{Z'}=2$ TeV panel, the exclusion contour crosses the $\tilde g = 0$ axis at $g_1' \in [0.10, 0.20]$ (matching the published value within 30%).

The $(A, B, C)$ quadratic coefficients extracted at $M_{Z'}=2$ TeV satisfy $A > 0$ and $|B| \le 4\sqrt{AC}$ (positivity of the cross section over the full parameter plane).

The maximum reach in $g_1'$ at $\tilde g = 0$ degrades by at least a factor of 2 going from $M_{Z'}=2$ TeV to $M_{Z'}=3$ TeV, reflecting the PDF suppression at high mass.

Does the agent reproduce the three-panel $(\tilde g, g_1')$ 2$\sigma$ exclusion contours at $M_{Z'}=2.0, 2.5, 3.0$ TeV from the LHC Run 1 dilepton search at $\sqrt{s}=8$ TeV with $\mathcal{L}=20$ fb$^{-1}$ (Figure 1 of arXiv:1605.02910)?

$M_{Z'}=2$ TeV: 3 signal runs at $(g_1', \tilde g) \in \{(0.10, 0), (0.10, -0.10), (0.15, -0.05)\}$.

$M_{Z'}=2.5$ TeV: 3 signal runs at the same three $(g_1', \tilde g)$ points.

$M_{Z'}=3$ TeV: 3 signal runs at $(g_1', \tilde g) \in \{(0.30, 0), (0.30, -0.30), (0.50, -0.60)\}$.

SM Drell-Yan $pp \to \ell^+\ell^-$ at $\sqrt{s}=8$ TeV with generator-level cut $m_{\ell\ell} > 0.9 M_{Z'}$, one per mass point

Value of $g_1'$ at which the $\mathrm{Sig}=2$ contour crosses $\tilde g = 0$, per mass point.

$(A, B, C)$ extracted from the 3 signal runs per mass; positive-definite check.

Analytic $\mathrm{BR}(Z' \to e^+e^-+\mu^+\mu^-)$ at the three benchmark masses.

$pp \to Z'$ with kinetic-mixing B-L Z' at the LHC, dilepton final state. Background: SM Drell-Yan with $m_{\ell\ell}$ cut.

A canonical results.json file exists at the workspace root with at least the keys: primary_metric (number), metric_key (string), metrics (object with numeric keys), hypotheses (object with h1/h2/h3 entries each carrying a `supported` boolean), summary (non-empty string).

results.json metrics MUST contain at least 3 numeric (non-null, finite) values directly relevant to the headline physics observable named in the experiment_design.metrics list above — these are the numbers the paper will report in its Results section.

results.json hypotheses.h1/h2/h3 each MUST have an explicit `supported` boolean AND a `details` string ≥ 40 characters quoting the numerical evidence (specific values + their source artifact) used to reach the verdict.

At least one publication-quality figure file (PDF or PNG, ≥150 DPI for raster) exists under figures/ or output/figures/ with axes labeled in physical units (GeV / pb / fb / dimensionless) and a legend if multiple series are plotted.  The figure must directly support a hypothesis verdict.

The BSM Lagrangian is implemented either as a FeynRules .fr file (models/*.fr) with a matching UFO directory (models/*_UFO/ containing at least particles.py, parameters.py, couplings.py, vertices.py), OR as analytic Python code that explicitly computes the cross sections from the Lagrangian terms.  A pure SM baseline with no BSM piece is NOT sufficient.

results.json metrics MUST report the g1' coordinate where the 2σ exclusion contour at M_Z'=2 TeV crosses the g̃=0 axis, in the interval [0.07, 0.26] (i.e. published [0.10, 0.20] ±30%).

The summary or structured_results section provides a one-paragraph mechanistic interpretation of WHY the headline observable comes out the way it does (which interference / propagator structure / cut effect drives the result).  Nice-to-have, not blocking proceed.

results.json or a sibling reproducibility section names: (a) the MadGraph5_aMC@NLO version, (b) the PDF set used (if applicable), (c) at least one explicit random seed.  Required for full reproducibility but not for scientific correctness.

A credible HEP collider study reproducing the published reference figure for B-L $Z'$ kinetic mixing, three-panel exclusion in $(\tilde g, g_1')$. The agent must (a) implement the BSM Lagrangian, (b) generate Monte Carlo events for the specified process with appropriate PDFs and generation cuts, (c) run the necessary detector / shower simulation if required, (d) reconstruct the diagnostic kinematic distribution, and (e) compare against the published reference figure with quantitative residuals tied to H1, H2, H3.

Code-development bucket: BSM model + process card + PDF/cuts + full pipeline are wired correctly.

BSM Lagrangian implemented (FeynRules .fr file + UFO output, or equivalent analytic Python module): Kinetic-mixing B-L $Z'$ with chiral couplings $C_{f,L/R}(\tilde g, g_1', M_{Z'})$ encoding $g_1'$ (B-L gauge), $\tilde g$ (kinetic mixing), and 3 heavy Majorana neutrinos $\nu_h$ at $m_{\nu_h}=95$ GeV.

MadGraph generate-process card or analytic process module matches the paper's process string: $pp \to Z'$ at $\sqrt{s}=8$ TeV with kinetic mixing; Drell-Yan dilepton search recast.

PDF set and generation-level cuts match the paper exactly: 9 signal runs (3 mass x 3 coupling points), 3 SM Drell-Yan background runs with generator cut $m_{\ell\ell} > 0.9 M_{Z'}$ at $\sqrt{s}=8$ TeV; mass-dependent NNLO k-factors $\{1.35, 1.40, 1.50\}$ and $\epsilon_\mathrm{acc}=0.6$.

Full HEP chain wired together (model -> events -> [shower -> detector] -> reconstruction -> analysis -> figure). For analytic shortcuts the chain may be simpler but must still terminate in the diagnostic figure.

Execution-validity bucket: MC sample passes nevents threshold, physics-validity gates clear, MC stats are sufficient.

MC sample(s) generated successfully with at least the requested number of events per cell (or a documented partial count if compute-limited), with no fatal MadGraph or downstream errors.

Physics-validity gate: no negative cross sections; gauge invariance / unitarity sanity checks pass; mass spectrum and decay widths printed by MadGraph match the input parameter card; no obviously unphysical kinematics in the generated events.

MC statistics are sufficient: relative MC error in the signal region is < 10% OR at least 3 random seeds are run with std reported across them.

Results bucket: quantitative comparison of three hypothesis-driven observables to the published figure, plus a reproduced figure artifact and a per-hypothesis writeup.

Quantitative test of H1 — $M_{Z'}=2$ TeV exclusion contour crosses the $\tilde g=0$ axis at $g_1' \in [0.10, 0.20]$ (within 30% of published).

Quantitative test of H2 — Quadratic coefficients $(A,B,C)$ at $M_{Z'}=2$ TeV satisfy $A>0$ and $|B| \le 4\sqrt{AC}$ (positive-definite cross section everywhere in plane). Use the same graded relative-error scale as H1.

Quantitative test of H3 — Maximum reach in $g_1'$ at $\tilde g=0$ degrades by $\geq 2\times$ from $M_{Z'}=2$ TeV to $M_{Z'}=3$ TeV. Use the same graded relative-error scale as H1.

A reproduced figure exists in artifacts (PDF or PNG) with axes, units, ranges, and legend matching the published reference: Three side-by-side panels (one per $M_{Z'}$), 1:1 aspect, exclusion contour at $\mathrm{Sig}\equiv 2(\sqrt{S+B}-\sqrt B)=2$ in the $(\tilde g, g_1')$ plane.

Writeup discusses the outcome of each hypothesis (supported / refuted / inconclusive) tied to numeric residuals, identifies the dominant systematic uncertainty (PDF set, scale variation, k-factor, Delphes card, statistical MC error), and states explicitly any analytic shortcuts taken in place of full MC.

Reproducibility bucket (new for physics topics; absent in T-rubrics): the artifacts contain everything needed to rerun the analysis from scratch.

UFO model directory (or analytic model Python module) is checked into the artifacts and is loadable by MadGraph (or directly importable for analytic models).

MadGraph run_card / param_card, Pythia8 settings, Delphes card, and MadAnalysis5 (or equivalent analysis) cards are saved alongside the run with explicit parameter values per cell.

RNG seeds are documented; rerunning the run from saved seeds should give matching central values within the reported MC error. For analytic shortcuts, the seed requirement is replaced by an explicit list of input numerical parameters.