Update logbook: Repro - DropoutTS: Sample-Adaptive Dropout for Robust Time Series Forecasting

logbook.json

@@ -10,7 +10,7 @@
     "icml2026-repro",
     "paper-7sksHLUvhH"
   ],
-  "updated_at": "2026-07-16T05:44:38+00:00",
+  "updated_at": "2026-07-16T05:54:38+00:00",
   "root": {
     "slug": "index",
     "title": "Repro - DropoutTS: Sample-Adaptive Dropout for Robust Time Series Forecasting",
@@ -54,6 +54,6 @@
       }
     ]
   },
-  "agent_view_tokens": 3820,
-  "revision": "1784180678070555000"
+  "agent_view_tokens": 3544,
+  "revision": "1784181278794122000"
 }

pages/claim-1-synth-benchmark-informer-improvement/page.md

@@ -3937,4 +3937,13 @@ noise,horizon,mse_base,mse_drop,mse_imp_pct,mae_base,mae_drop,mae_imp_pct,base_s
 <!-- trackio-cell
 {"type": "markdown", "id": "cell_a789c7016367", "created_at": "2026-07-16T04:58:12+00:00", "title": "Methodology limitation (added after self-review). The results above use a singl…"}
 -->
-Methodology limitation (added after self-review). The results above use a single fixed init_sensitivity=5.0, while the repo's run_baselines.py sweeps init_sensitivity over {1, 5, 10} and would report the best per configuration. So this compares an untuned DropoutTS against the baseline, which is not the paper's protocol. A targeted sensitivity sweep at sigma=0.3 is running now to check whether tuning recovers the claimed improvement; I will update the verdict here with those results. Treat the 'not reproduced' reading as provisional and specifically about the default configuration, not a claim that the method cannot work.
+Update after the sensitivity sweep (this changes the verdict). The results above used a single fixed init_sensitivity=5.0, while the repo's run_baselines.py sweeps init_sensitivity over {1, 5, 10} and reports the best per configuration. I re-ran sigma=0.3 across all four horizons with each sensitivity and took the best. Tuning changes the picture a lot:
+
+| Horizon | MSE baseline | best +DropoutTS | best sensitivity | change |
+|---|---|---|---|---|
+| 96 | 1.931 | 1.059 | 10 | +45.1% |
+| 192 | 0.755 | 1.112 | 1 | -47.2% |
+| 336 | 0.678 | 0.666 | 5 | +1.8% |
+| 720 | 0.678 | 0.587 | 10 | +13.4% |
+
+At H=96 with sensitivity=10, DropoutTS reaches +45.1 percent, which is essentially the paper's stated +48.2 percent peak at sigma=0.3. So the peak magnitude does reproduce, at the right horizon and the right sensitivity. My earlier untuned run (about +18 percent at that cell) was simply at a worse operating point. What does not reproduce is the claimed +46 percent average across horizons: H=192 stays strongly negative even when tuned, which drags the sigma=0.3 average to about +3 percent. So the honest verdict is partial: the headline peak reproduces with tuning, the across-horizon average does not, and the effect is strongly horizon-dependent.

pages/claim-5-orthogonal-compatibility-with-selective-learning/page.md

@@ -3,15 +3,13 @@
 
 ---
 <!-- trackio-cell
-{"type": "markdown", "id": "cell_b96d52b2d9c7", "created_at": "2026-07-16T04:34:27+00:00", "title": "Claim 5 says combining DropoutTS with a Selective Learning strategy beats Selec…"}
--->
-Claim 5 says combining DropoutTS with a Selective Learning strategy beats Selective Learning alone (21.2 vs 19.6 percent), so the two are meant to be orthogonal and stack. I tested this on the synthetic sigma=0.3 set with Informer across three conditions: a plain baseline, Selective Learning alone (uncertainty masking, r_u=0.1), and DropoutTS plus Selective Learning. Improvements over baseline came out at +23.3 percent for Selective Learning alone and +19.2 percent for the combination. The combination was worse than Selective Learning by itself, so the orthogonal stacking the claim describes did not show up. Here DropoutTS took away from the data-centric strategy rather than adding to it.
+Claim 5 says combining DropoutTS with a Selective Learning strategy beats Selective Learning alone (21.2 vs 19.6 percent), so the two are meant to be orthogonal and stack. I tested this on the synthetic sigma=0.3 set with Informer across three conditions: a plain baseline, Selective Learning alone (uncertainty masking, r_u=0.1), and DropoutTS plus Selective Learning.
+
+The claim reproduces. Improvement over baseline was +25.9 percent for Selective Learning alone (MSE 1.9035 to 1.4105) and +40.4 percent for the combination (MSE 1.9035 to 1.1338). The combination beats Selective Learning on its own, which is the orthogonal stacking the claim describes: adding DropoutTS to the data-centric strategy helped rather than hurt. The absolute magnitudes run higher than the paper (it reports 19.6 and 21.2 percent), which is expected on a different dataset, seed, and configuration, but the direction and the ordering match.
 
 Caveats: one dataset, one seed, and I chose r_u=0.1 because the claim does not pin down the Selective Learning configuration. I also had to patch a bug to run this at all. The repo's SelectiveLearning callback calls to_device on a None estimator in uncertainty-only mode and crashes with 'NoneType object has no attribute cuda'. I added a guard so the r_u-only path runs. That crash is itself a small reproducibility finding. Run on Modal: https://modal.com/apps/lexigenai/main/ap-QlN4lF2pNbSgtVOjB0h8tL
 
 
 ---
 <!-- trackio-cell
-{"type": "markdown", "id": "cell_4edacc50c92f", "created_at": "2026-07-16T04:58:12+00:00", "title": "Two limitations to note (added after self-review). First, single seed and a sin…"}
--->
-Two limitations to note (added after self-review). First, single seed and a single r_u=0.1, so this is one operating point, not a tuned comparison. Second, an earlier version of this run used use_clean_targets=False, which is inconsistent with the repo's USE_CLEAN_TARGETS=True convention and with the Claim 1 setup; the numbers above are being re-run with use_clean_targets=True for consistency, and I will update them. Treat the 'no orthogonal gain' reading as provisional.
+Correction trail (self-review). My first run of this claim used use_clean_targets=False, which is inconsistent with the repo's USE_CLEAN_TARGETS=True convention and with the Claim 1 setup. Under that wrong setting the combination looked worse than Selective Learning alone (+19.2 vs +23.3 percent), and I had briefly written this claim up as not reproducing. Re-running with use_clean_targets=True fixed it, and the numbers above are the corrected ones. The lesson: my own configuration bug, not the paper, produced the earlier negative result. Remaining limitations are the single seed and the single r_u=0.1 operating point.

pages/conclusion/page.md

@@ -5,17 +5,17 @@
 <!-- trackio-cell
 {"type": "markdown", "id": "cell_b6dd00bc164f", "created_at": "2026-07-16T04:43:17+00:00", "title": "Executive summary", "pinned": true, "pinned_at": "2026-07-16T04:43:17+00:00"}
 -->
-This reproduction of DropoutTS (arXiv:2601.21726) lands as a mix, and the mix is the point. Two structural claims hold cleanly: DropoutTS adds only 4 learnable parameters in the channel-independent case and adds zero inference latency (Claims 4a and 4b), and the experimental setup matches the paper (Claim 3). The results claims are shakier. On real ETTh2 (Claim 2), DropoutTS delivers large MSE reductions at the middle horizons, up to +59 percent, which reaches and passes the paper's 'up to 47.6 percent'. But the synthetic headline (Claim 1, a +46 percent average) does not reproduce under the repo's default configuration and a single seed: I measured -7.5 percent on average with heavy horizon-dependent swings, and at the paper's own best noise level (sigma=0.3) it came out worse. Two claims come out backwards. The '1.12 to 1.45x training speedup' (Claim 4c) is really about a 1.3x slowdown, because the noise scorer runs an FFT on every step, and its stated speedup range almost exactly matches the slowdown I measured. The 'orthogonal compatibility' with Selective Learning (Claim 5) did not appear either, since the combination underperformed Selective Learning on its own. Everything ran on Modal A10G GPUs across about 51 training runs, a few hours of wall-clock, on the order of 15 to 25 dollars.
+This reproduction of DropoutTS (arXiv:2601.21726) lands as a mix, and the mix is the point. Two structural claims hold cleanly: DropoutTS adds only 4 learnable parameters in the channel-independent case and adds zero inference latency (Claims 4a and 4b), and the experimental setup matches the paper (Claim 3). On real ETTh2 (Claim 2), DropoutTS delivers large MSE reductions at the middle horizons, up to +59 percent, which reaches and passes the paper's 'up to 47.6 percent'. The synthetic headline (Claim 1) is partial: after sweeping the sensitivity hyperparameter the way the paper does, DropoutTS reaches +45 percent at sigma=0.3, H=96, which matches the paper's +48.2 percent peak, so the peak reproduces at the right operating point, but the claimed +46 percent average across horizons does not hold (H=192 stays negative even when tuned). The orthogonal compatibility with Selective Learning (Claim 5) reproduces: the combination improved +40 percent over baseline versus +26 percent for Selective Learning alone. One claim comes out backwards: the '1.12 to 1.45x training speedup' (Claim 4c) is really a 1.3x slowdown, because the noise scorer runs an FFT on every step, and its stated speedup range almost exactly matches the slowdown I measured. A note on process: a self-review caught two of my own bugs (a use_clean_targets inconsistency and a missing hyperparameter sweep) that had produced two wrong negative verdicts in an earlier draft, so Claims 1 and 5 were corrected upward here. Everything ran on Modal A10G GPUs across about 66 training runs, a few hours of wall-clock, on the order of 20 to 30 dollars.
 
 ## Scope & cost
 
 |  | This reproduction | Full replication |
 |---|---|---|
-| Scope | Informer backbone. Claims 3 and 4 in full, Claim 1 on the synthetic sweep, Claim 2 on ETTh2 only, Claim 5 on synthetic. Single seed, default hyperparameters. | 6 backbones, 7 datasets plus Synth-12, per-dataset tuning, multiple seeds |
+| Scope | Informer backbone. Claims 3 and 4 in full, Claim 1 on the synthetic sweep with a sigma=0.3 sensitivity sweep, Claim 2 on ETTh2 only, Claim 5 on synthetic. Single seed. | 6 backbones, 7 datasets plus Synth-12, per-dataset tuning, multiple seeds |
 | Hardware | 1x A10G per run (Modal) | multi-GPU cluster |
-| Compute time | ~51 runs, a few hours | hundreds of GPU-hours |
-| Cost | ~15 to 25 USD | thousands |
-| Outcome | Claims 3, 4a, 4b verified. Claim 2 (ETTh2) reproduced. Claim 1 not reproduced. Claims 4c and 5 contradicted. | not attempted |
+| Compute time | ~66 runs, a few hours | hundreds of GPU-hours |
+| Cost | ~20 to 30 USD | thousands |
+| Outcome | Verified: Claims 3, 4a, 4b, and 5. Reproduced: Claim 2 (ETTh2) and the Claim 1 peak at sigma=0.3. Not reproduced: the Claim 1 across-horizon average. Falsified: Claim 4c (a slowdown, not a speedup). | not attempted |
 
 
 ---