Upload via Autoresearch export_data

README.md

@@ -1,3 +1,86 @@
 ---
-license: apache-2.0
+license: cc-by-nc-4.0
+library_name: pytorch
+pipeline_tag: time-series-forecasting
+tags:
+  - time-series
+  - forecasting
+  - foundation-model
+  - zero-shot
+  - probabilistic-forecasting
+  - gift-eval
 ---
+
+# Modular-TS-2.4M
+
+**A 2.4M-parameter time-series foundation model, pretrained from scratch by [Modular Computing](https://huggingface.co/Modularcomputing).**
+
+Modular-TS-2.4M is a small, fast, zero-shot probabilistic forecaster: give it any univariate or multivariate series and it returns 21 quantiles for every future step, with no fine-tuning. It is built for the GPU-poor: it runs comfortably on a CPU or a phone-class device.
+
+## Results (GIFT-Eval, full 97 configurations, zero-shot)
+
+Geometric mean relative to Seasonal Naive (lower is better).
+
+| Model | Params | MASE | CRPS |
+|---|---|---|---|
+| **Modular-TS-2.4M (ours)** | **2.41M** | **0.745** | **0.511** |
+| IBM TTM-R3 (PT) | ~1–5M | 0.724 | 0.520 |
+| TimeSSM-2.5M | 2.5M | 0.754 | 0.513 |
+| Seasonal Naive | – | 1.000 | 1.000 |
+
+- Best CRPS (probabilistic accuracy) of any open model at 2.5M parameters or fewer that we evaluated.
+- Beats TimeSSM-2.5M on both metrics and TTM-R3 on CRPS; TTM-R3 is still ahead on MASE.
+- Our numbers are strictly zero-shot: the model never sees a test dataset's own training split at inference time. (TTM-R3-PT uses each dataset's train split for backtest ensembling.)
+- Evaluated with the official GIFT-Eval harness at leaderboard settings; the same harness reproduces published leaderboard scores for reference models exactly.
+
+## Model
+
+| | |
+|---|---|
+| Architecture | Decoder-style patch transformer (Timestral-style), d_model 128, 2 heads, MLP 320, 8 layers alternating time and variate attention, patch 32 |
+| Parameters | 2,407,680 (embed 107,392 / trunk 1,804,672 / head 495,616) |
+| Context | up to 8,192 steps |
+| Output | 21 monotone quantiles per step (plus mean) |
+
+The architecture code adapts layers from Mistral AI's Apache-2.0 `timestral` inference package. **No Timestral weights were loaded, fine-tuned, or distilled at any point.** Every weight in this repo was trained from random initialisation.
+
+## Training data
+
+- **Synthetic (majority):** a GPU-vectorised generator with Gaussian-process kernels, regime-switching Ornstein–Uhlenbeck processes, waveform and load-curve templates, steps, growth curves, intermittent counts, multivariate lead-lag mixing, covariates and missingness, followed by an augmentation cascade.
+- **Real (~40% in the final stage):** GiftEvalPretrain, resolution-sampled with TSMixup, after a contamination filter that removes any dataset overlapping the GIFT-Eval test sets.
+
+## Usage
+
+```python
+from huggingface_hub import snapshot_download
+path = snapshot_download("Modularcomputing/Modular-TS-2.4M")
+# path/export/model.safetensors + path/export/config.json
+# Load with the bundled modular_ts package (see modular_ts/README.md),
+# which wraps the Apache-2.0 timestral forecaster around these weights.
+```
+
+Files:
+- `export/model.safetensors`, `export/config.json`: inference weights and config
+- `modular_ts/`: training, synthetic data generation and evaluation code
+- `results/all_results.csv`: per-configuration GIFT-Eval scores
+
+## Licence
+
+- **Weights:** CC BY-NC 4.0 (research and non-commercial use), because the real-data portion of training mixes datasets under several licences.
+- **Code (`modular_ts/`):** Apache-2.0, including the adapted `timestral` components (see `modular_ts/NOTICE`).
+
+## Limitations
+
+- Trails TTM-R3 on point accuracy (MASE); weakest on some long-horizon, high-frequency energy and weather series.
+- A small model: it captures common seasonal and trend structure well but can miss rare regime changes.
+
+## Citation
+
+```bibtex
+@misc{modularts2026,
+  title  = {Modular-TS-2.4M: a small time-series foundation model trained from scratch},
+  author = {Akuegbo, Chima Emmanuel and Modular Computing},
+  year   = {2026},
+  url    = {https://huggingface.co/Modularcomputing/Modular-TS-2.4M}
+}
+```

export/config.json

@@ -0,0 +1,55 @@
+{
+  "patch_size": 32,
+  "d_model": 128,
+  "num_heads": 2,
+  "mlp_hidden_dim": 320,
+  "block_schedule": [
+    "time",
+    "time",
+    "time",
+    "time_variate",
+    "time",
+    "time",
+    "time",
+    "time_variate"
+  ],
+  "quantile_knots": [
+    0.01,
+    0.05,
+    0.1,
+    0.15,
+    0.2,
+    0.25,
+    0.3,
+    0.35,
+    0.4,
+    0.45,
+    0.5,
+    0.55,
+    0.6,
+    0.65,
+    0.7,
+    0.75,
+    0.8,
+    0.85,
+    0.9,
+    0.95,
+    0.99
+  ],
+  "block_size": 32,
+  "context_length": 8192,
+  "rope_theta": 10000.0,
+  "rope_partial_factor": 0.5,
+  "norm_eps": 1e-06,
+  "attn_bias": false,
+  "mlp_bias": false,
+  "time_attn_gate": "sdpa_sigmoid",
+  "variate_attn_gate": "sdpa_sigmoid",
+  "minimum_scale": 1e-06,
+  "use_asinh": true,
+  "stabilize_with_global": false,
+  "covariate_enabled": true,
+  "split_context_horizon_attn": true,
+  "attend_missing_patches": true,
+  "quantile_width_factor": 3.0
+}

export/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:04d4c57329c8fe0a67716bff5207adea92c632d69810d33e877d64c9299bf4dc
+size 9638720

modular_ts/NOTICE

@@ -0,0 +1,6 @@
+Modular-TS — Apache-2.0.
+This package imports and adapts code from the `timestral` inference package
+(https://github.com/mistralai/timestral, Copyright Mistral AI, Apache-2.0; see TIMESTRAL_LICENSE):
+layers, masks, scaler, and the forecaster used at evaluation time.
+No Timestral model weights are loaded, fine-tuned or distilled anywhere in this code.
+All weights are trained from scratch.

modular_ts/TIMESTRAL_LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

modular_ts/distill.py

@@ -0,0 +1,177 @@
+"""Stage B: fine-tune a Stage-A checkpoint with ground-truth pinball + teacher-ensemble pinball (TimesFM-2.5 + Chronos-2).
+
+python -m modular_ts.distill --init runs/run2_8k/ckpt.pt --teach teach/synth,teach/gep --lam 0.5 --minutes 45 --out runs/run3
+
+Batches: with prob --p_distill a teacher batch (V=1, T=C+F windows, min-context split s=C/32;
+synthetic rows: GT pinball + teacher pinball; GiftEvalPretrain rows: teacher pinball ONLY, their
+future is filled with the teacher-ensemble median just for the normalisation frame and never used
+as a target), else a fresh synthetic Stage-A batch (GT pinball only) to avoid forgetting.
+Teacher ensemble = mean of the two teachers' quantiles (Vincentisation) at 0.1..0.9.
+"""
+
+from __future__ import annotations
+
+import argparse
+import glob
+import json
+import math
+import os
+import time
+
+import numpy as np
+import torch
+
+from modular_ts.model import ModularTS, count_params, make_config
+from modular_ts.synth import SynthStream
+from modular_ts.train import export, lr_at
+
+
+def load_teach(dirs, dev, max_per_dir=0):
+    ctx, fut, q, src = [], [], [], []
+    for si, d in enumerate(dirs):
+        files = sorted(glob.glob(os.path.join(d, "shard_*.npz")))
+        n = 0
+        for f in files:
+            z = np.load(f)
+            qe = 0.5 * (z["q_tfm"].astype(np.float32) + z["q_cr2"].astype(np.float32))
+            qe = np.sort(qe, -1)
+            ctx.append(z["ctx"]); fut.append(z["fut"]); q.append(qe); src.append(np.full(len(qe), si))
+            n += len(qe)
+            if max_per_dir and n >= max_per_dir:
+                break
+        print(f"[teach] {d}: {n} windows from {len(files)} shards", flush=True)
+    c = torch.from_numpy(np.concatenate(ctx)).to(dev)
+    f = torch.from_numpy(np.concatenate(fut)).to(dev)
+    qq = torch.from_numpy(np.concatenate(q)).to(dev)
+    s = torch.from_numpy(np.concatenate(src)).to(dev)
+    return c, f, qq, s
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--init", required=True, help="Stage-A ckpt.pt (model weights only are used)")
+    ap.add_argument("--teach", required=True, help="comma list of teacher shard dirs; dirs containing 'gep' are teacher-only")
+    ap.add_argument("--lam", type=float, default=0.5)
+    ap.add_argument("--p_distill", type=float, default=0.6)
+    ap.add_argument("--lr", type=float, default=3e-4)
+    ap.add_argument("--warmup", type=int, default=200)
+    ap.add_argument("--minutes", type=float, default=45)
+    ap.add_argument("--tokens_per_batch", type=int, default=262144)
+    ap.add_argument("--ctx", type=int, default=0, help="model context_length for export (0 = from init ckpt)")
+    ap.add_argument("--compile", action="store_true")
+    ap.add_argument("--log_every", type=int, default=100)
+    ap.add_argument("--seed", type=int, default=7)
+    ap.add_argument("--out", required=True)
+    args = ap.parse_args()
+    os.makedirs(args.out, exist_ok=True)
+    dev = torch.device("cuda")
+    torch.manual_seed(args.seed)
+    torch.backends.cuda.matmul.allow_tf32 = True
+    st = torch.load(args.init, map_location=dev)
+    ctx_len = args.ctx or st["args"]["ctx"]
+    cfg = make_config(st["args"].get("size", "2.5M"), context_length=ctx_len)
+    model = ModularTS(cfg).to(dev)
+    model.load_state_dict(st["model"])
+    print("params", count_params(model), "init", args.init, "step", st["step"], flush=True)
+
+    dirs = [d for d in args.teach.split(",") if d]
+    tc, tf, tq, tsrc = load_teach(dirs, dev)
+    teacher_only = torch.tensor(["gep" in d for d in dirs], device=dev)[tsrc]
+    C, F, H = tc.shape[1], tf.shape[1], tq.shape[1]
+    T = C + F
+    P = cfg.patch_size
+    Bd = max(1, args.tokens_per_batch // (T // P))
+    nT = tc.shape[0]
+    print(f"[teach] total {nT} windows (teacher-only {int(teacher_only.sum())}), C={C} F={F} H={H}, rows/batch {Bd}", flush=True)
+    # hold out 2% of synthetic teacher windows for a GT/KD validation number
+    g = torch.Generator(device=dev); g.manual_seed(args.seed)
+    perm = torch.randperm(nT, generator=g, device=dev)
+    syn_idx = perm[~teacher_only[perm]]
+    val_idx = syn_idx[:min(4096, max(1, len(syn_idx) // 50))]
+    trmask = torch.ones(nT, dtype=torch.bool, device=dev); trmask[val_idx] = False
+    tr_idx = torch.nonzero(trmask).squeeze(1)
+
+    def distill_batch(idx):
+        c, f, q, to = tc[idx], tf[idx], tq[idx], teacher_only[idx]
+        med = q[:, :, 4]
+        fill = torch.full_like(f, float("nan"))
+        fill[:, :H] = med
+        f = torch.where(to[:, None], fill, f)
+        vals = torch.cat([c, f], 1)
+        obs = torch.isfinite(vals)
+        vals = torch.where(obs, vals, torch.zeros_like(vals))
+        teach = torch.zeros(len(idx), T, q.shape[-1], device=dev)
+        teach[:, C:C + H] = q
+        tobs = torch.zeros(len(idx), T, dtype=torch.bool, device=dev)
+        tobs[:, C:C + H] = True
+        R = len(idx)
+        return dict(values=vals[:, None], observed=obs[:, None], roles=torch.zeros(R, 1, dtype=torch.long, device=dev),
+                    s=torch.full((R,), C // P, dtype=torch.long, device=dev), teacher=teach, teacher_obs=tobs,
+                    gt_row=(~to).float())
+
+    synth = SynthStream(4096, args.tokens_per_batch, dev, seed=args.seed + 99)
+
+    decay, no_decay = [], []
+    for n, p in model.named_parameters():
+        (decay if p.dim() >= 2 else no_decay).append(p)
+    opt = torch.optim.AdamW([{"params": decay, "weight_decay": 0.05}, {"params": no_decay, "weight_decay": 0.0}],
+                            lr=args.lr, betas=(0.9, 0.95), eps=1e-8, fused=True)
+    fwd = torch.compile(model.forward_train, dynamic=False) if args.compile else model.forward_train
+    amp = torch.autocast(device_type="cuda", dtype=torch.bfloat16)
+    lam = args.lam
+
+    def evaluate():
+        model.eval()
+        out = {"val_gt": 0.0, "val_kd": 0.0}
+        with torch.no_grad(), amp:
+            for i in range(0, len(val_idx), Bd):
+                b = distill_batch(val_idx[i:i + Bd])
+                l, s_ = model.forward_train(b["values"], b["observed"], b["roles"], b["s"], teacher=b["teacher"],
+                                            teacher_obs=b["teacher_obs"], gt_row=b["gt_row"], lam=lam)
+                out["val_gt"] += float(s_["pinball"]); out["val_kd"] += float(s_["kd"])
+        n = math.ceil(len(val_idx) / Bd)
+        model.train()
+        return {k: v / n for k, v in out.items()}
+
+    log = open(os.path.join(args.out, "log.jsonl"), "a")
+    rng = np.random.default_rng(args.seed)
+    t0 = time.time(); step = 0
+    rec0 = {"step": 0, **evaluate()}
+    print(json.dumps(rec0), flush=True); log.write(json.dumps(rec0) + "\n")
+    model.train()
+    while (time.time() - t0) / 60 < args.minutes:
+        frac = (time.time() - t0) / 60 / args.minutes
+        lr = lr_at(step, max(step + 1, int(step / max(frac, 1e-6))) if step > args.warmup else 10**8, args.warmup, args.lr)
+        for gp in opt.param_groups:
+            gp["lr"] = lr
+        is_d = rng.random() < args.p_distill
+        with amp:
+            if is_d:
+                idx = tr_idx[torch.randint(0, len(tr_idx), (Bd,), device=dev)]
+                b = distill_batch(idx)
+                loss, s_ = fwd(b["values"], b["observed"], b["roles"], b["s"], False, b["teacher"], b["teacher_obs"], b["gt_row"], lam)
+            else:
+                b = synth.next_batch()
+                loss, s_ = fwd(b["values"], b["observed"], b["roles"], b["s"])
+        if not torch.isfinite(loss):
+            opt.zero_grad(set_to_none=True); step += 1
+            continue
+        loss.backward()
+        gn = torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
+        opt.step(); opt.zero_grad(set_to_none=True)
+        step += 1
+        if step % args.log_every == 0:
+            rec = {"step": step, "lr": lr, "loss": float(loss), "kind": "distill" if is_d else "synth",
+                   "kd": float(s_.get("kd", float("nan"))), "gnorm": float(gn), "min": (time.time() - t0) / 60}
+            if step % (args.log_every * 10) == 0:
+                rec.update(evaluate())
+            print(json.dumps(rec), flush=True); log.write(json.dumps(rec) + "\n"); log.flush()
+    torch.save({"model": model.state_dict(), "step": step, "args": {**st["args"], "distill": vars(args)}}, os.path.join(args.out, "ckpt.pt"))
+    export(model, os.path.join(args.out, "export"))
+    fin = {"steps": step, **evaluate(), "minutes": (time.time() - t0) / 60, "lam": lam}
+    print("FINAL", json.dumps(fin), flush=True)
+    json.dump(fin, open(os.path.join(args.out, "final.json"), "w"))
+
+
+if __name__ == "__main__":
+    main()

modular_ts/eval_gift.py

@@ -0,0 +1,261 @@
+"""GIFT-Eval harness for Modular-TS (official gift-eval Dataset + gluonts evaluate_model).
+
+Adapted from gift-eval/notebooks/timestral.ipynb (Apache-2.0). Our weights load into
+the Apache-2.0 `timestral.TimestralForecaster` (block-AR decoding, advancing refine frame,
+clampsym TTA). Metrics, settings and the leaderboard aggregate match the official board:
+evaluate_model(axis=None, mask_invalid_label=True, allow_nan_forecast=False), 9
+quantiles 0.1..0.9 for CRPS (mean_weighted_sum_quantile_loss), then the geometric mean
+over configs of (model / Seasonal_Naive) for MASE and CRPS.
+
+    python -m modular_ts.eval_gift --export runs/smoke/export --mode quick --out runs/smoke/gift_quick
+    python -m modular_ts.eval_gift --export ... --mode full   # all 97 configs -> all_results.csv
+
+Requires: pip install --no-deps git+https://github.com/SalesforceAIResearch/gift-eval
+          pip install "gluonts~=0.15.1" python-dotenv toolz orjson datasets pyarrow pandas
+Data: Salesforce/GiftEval is downloaded (only the needed folders) to $GIFT_EVAL.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import logging
+import math
+import os
+import time
+import warnings
+from pathlib import Path
+
+import numpy as np
+import pandas as pd
+import torch
+
+import modular_ts.model  # noqa: F401  (puts the vendored timestral package on sys.path)
+
+ROOT = Path(__file__).resolve().parent.parent
+LB_DIR = ROOT / "gift-eval" / "results"
+PROPS = ROOT / "gift-eval" / "notebooks" / "dataset_properties.json"
+
+SHORT_DATASETS = "m4_yearly m4_quarterly m4_monthly m4_weekly m4_daily m4_hourly electricity/15T electricity/H electricity/D electricity/W solar/10T solar/H solar/D solar/W hospital covid_deaths us_births/D us_births/M us_births/W saugeenday/D saugeenday/M saugeenday/W temperature_rain_with_missing kdd_cup_2018_with_missing/H kdd_cup_2018_with_missing/D car_parts_with_missing restaurant hierarchical_sales/D hierarchical_sales/W LOOP_SEATTLE/5T LOOP_SEATTLE/H LOOP_SEATTLE/D SZ_TAXI/15T SZ_TAXI/H M_DENSE/H M_DENSE/D ett1/15T ett1/H ett1/D ett1/W ett2/15T ett2/H ett2/D ett2/W jena_weather/10T jena_weather/H jena_weather/D bitbrains_fast_storage/5T bitbrains_fast_storage/H bitbrains_rnd/5T bitbrains_rnd/H bizitobs_application bizitobs_service bizitobs_l2c/5T bizitobs_l2c/H"
+MED_LONG_DATASETS = "electricity/15T electricity/H solar/10T solar/H kdd_cup_2018_with_missing/H LOOP_SEATTLE/5T LOOP_SEATTLE/H SZ_TAXI/15T M_DENSE/H ett1/15T ett1/H ett2/15T ett2/H jena_weather/10T jena_weather/H bitbrains_fast_storage/5T bitbrains_rnd/5T bizitobs_application bizitobs_service bizitobs_l2c/5T bizitobs_l2c/H"
+PRETTY = {"saugeenday": "saugeen", "temperature_rain_with_missing": "temperature_rain",
+          "kdd_cup_2018_with_missing": "kdd_cup_2018", "car_parts_with_missing": "car_parts"}
+# Quick subset: small, domain-diverse configs (12 short + 2 long). Used for smoke checks only.
+QUICK = [("m4_hourly", "short"), ("m4_weekly", "short"), ("us_births/D", "short"), ("saugeenday/D", "short"),
+         ("hospital", "short"), ("covid_deaths", "short"), ("ett1/H", "short"), ("jena_weather/H", "short"),
+         ("bizitobs_service", "short"), ("car_parts_with_missing", "short"), ("electricity/D", "short"),
+         ("LOOP_SEATTLE/D", "short"), ("ett2/H", "long"), ("bizitobs_l2c/H", "long")]
+QUANTILE_LEVELS = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]
+METRIC_COLUMNS = ["MSE[mean]", "MSE[0.5]", "MAE[0.5]", "MASE[0.5]", "MAPE[0.5]", "sMAPE[0.5]", "MSIS",
+                  "RMSE[mean]", "NRMSE[mean]", "ND[0.5]", "mean_weighted_sum_quantile_loss"]
+COMPARE = ["Seasonal_Naive", "TimeSSM-2.5M", "Timestral-5M", "TTM-R3-PT", "TinyCast"]
+
+
+def all_configs(properties):
+    short, ml = SHORT_DATASETS.split(), MED_LONG_DATASETS.split()
+    out = []
+    for name in dict.fromkeys(short + ml):
+        for term in ["short", "medium", "long"]:
+            if term != "short" and name not in ml:
+                continue
+            out.append((name, term, config_key(name, term, properties)))
+    assert len(out) == 97
+    return out
+
+
+def config_key(name, term, properties):
+    if "/" in name:
+        k, f = name.split("/")
+        k = PRETTY.get(k.lower(), k.lower())
+    else:
+        k = PRETTY.get(name.lower(), name.lower())
+        f = properties[k]["frequency"]
+    return f"{k}/{f}/{term}"
+
+
+def ensure_data(names, root):
+    from huggingface_hub import snapshot_download
+    tops = sorted({n.split("/")[0] for n in names})
+    need = [t for t in tops if not (Path(root) / t).exists()]
+    if need:
+        snapshot_download("Salesforce/GiftEval", repo_type="dataset", local_dir=root,
+                          allow_patterns=[f"{t}/**" for t in need])
+
+
+class Predictor:
+    def __init__(self, fc, prediction_length, use_past_covariates, max_rows=1024, max_tokens=2**24):
+        self.fc, self.h, self.cov = fc, prediction_length, use_past_covariates
+        self.max_rows, self.max_tokens = max_rows, max_tokens
+        lv = list(fc.quantile_levels)
+        self.qi = [next(i for i, l in enumerate(lv) if abs(l - q) < 1e-9) for q in QUANTILE_LEVELS]
+        self.keys = [str(q) for q in QUANTILE_LEVELS] + ["mean"]
+
+    def predict(self, test_input):
+        from gluonts.model.forecast import QuantileForecast
+        entries = list(test_input)
+        mc = self.fc.max_context
+        tg, cv = [], []
+        for e in entries:
+            t = np.asarray(e["target"], dtype=np.float32)
+            tg.append((t[None] if t.ndim == 1 else t)[:, -mc:])
+            c = e.get("past_feat_dynamic_real") if self.cov else None
+            if c is not None:
+                c = np.asarray(c, dtype=np.float32)
+                c = (c[None] if c.ndim == 1 else c)[:, -mc:]
+            cv.append(c)
+        nrows = tg[0].shape[0] + (cv[0].shape[0] if cv[0] is not None else 0)
+        outs = [None] * len(entries)
+        order = sorted(range(len(entries)), key=lambda i: tg[i].shape[-1])
+        s = 0
+        while s < len(order):
+            e_ = s + 1
+            while e_ < len(order):
+                L = tg[order[e_]].shape[-1] + self.h
+                if (e_ + 1 - s) * nrows > self.max_rows or (e_ + 1 - s) * nrows * L > self.max_tokens:
+                    break
+                e_ += 1
+            self._batch(order[s:e_], tg, cv, outs)
+            s = e_
+        res = []
+        for e, q in zip(entries, outs):
+            arr = np.concatenate([q, q[..., QUANTILE_LEVELS.index(0.5), None]], -1).transpose(2, 1, 0)
+            if np.asarray(e["target"]).ndim == 1:
+                arr = arr[..., 0]
+            res.append(QuantileForecast(forecast_arrays=arr, forecast_keys=self.keys,
+                                        start_date=e["start"] + np.asarray(e["target"]).shape[-1], item_id=e.get("item_id")))
+        return res
+
+    def _batch(self, idx, tg, cv, outs):
+        try:
+            L = max(tg[i].shape[-1] for i in idx)
+            ctx = np.stack([_lpad(tg[i], L) for i in idx])
+            past = np.stack([_lpad(cv[i], L) for i in idx]) if cv[idx[0]] is not None else None
+            f = self.fc.forecast(ctx, self.h, past_covariates=past)
+            q = f.quantiles[..., self.qi].float().cpu().numpy()
+        except torch.cuda.OutOfMemoryError:
+            torch.cuda.empty_cache()
+            if len(idx) == 1:
+                raise
+            hlf = len(idx) // 2
+            self._batch(idx[:hlf], tg, cv, outs)
+            self._batch(idx[hlf:], tg, cv, outs)
+            return
+        for j, i in enumerate(idx):
+            outs[i] = q[j]
+
+
+def _lpad(x, L):
+    p = L - x.shape[-1]
+    return x if p == 0 else np.concatenate([np.full((x.shape[0], p), np.nan, np.float32), x], -1)
+
+
+def evaluate_config(fc, name, term):
+    from gift_eval.data import Dataset
+    from gluonts.ev.metrics import MAE, MAPE, MASE, MSE, MSIS, ND, NRMSE, RMSE, SMAPE, MeanWeightedSumQuantileLoss
+    from gluonts.model import evaluate_model
+    from gluonts.time_feature import get_seasonality
+    metrics = [MSE(forecast_type="mean"), MSE(forecast_type=0.5), MAE(), MASE(), MAPE(), SMAPE(), MSIS(), RMSE(),
+               NRMSE(), ND(), MeanWeightedSumQuantileLoss(quantile_levels=QUANTILE_LEVELS)]
+    ds = Dataset(name=name, term=term, to_univariate=False)
+    pred = Predictor(fc, ds.prediction_length, bool(ds.past_feat_dynamic_real_dim), max_rows=int(os.environ.get("MAX_ROWS", "1024")))
+    res = evaluate_model(pred, test_data=ds.test_data, metrics=metrics, batch_size=512, axis=None,
+                         mask_invalid_label=True, allow_nan_forecast=False, seasonality=get_seasonality(ds.freq))
+    return {k: float(res[k].iloc[0]) for k in res.columns}
+
+
+def gmean(x):
+    return float(math.exp(np.log(np.asarray(x, dtype=float)).mean()))
+
+
+def aggregate(per_cfg: dict[str, dict]) -> dict:
+    keys = sorted(per_cfg)
+    sn = pd.read_csv(LB_DIR / "Seasonal_Naive" / "all_results.csv").set_index("dataset")
+    out = {}
+    m = [per_cfg[k]["MASE[0.5]"] / sn.loc[k, "eval_metrics/MASE[0.5]"] for k in keys]
+    c = [per_cfg[k]["mean_weighted_sum_quantile_loss"] / sn.loc[k, "eval_metrics/mean_weighted_sum_quantile_loss"] for k in keys]
+    out["ours"] = {"MASE": gmean(m), "CRPS": gmean(c)}
+    for name in COMPARE:
+        p = LB_DIR / name / "all_results.csv"
+        if not p.exists():
+            continue
+        df = pd.read_csv(p).set_index("dataset")
+        if not all(k in df.index for k in keys):
+            continue
+        out[name] = {
+            "MASE": gmean([df.loc[k, "eval_metrics/MASE[0.5]"] / sn.loc[k, "eval_metrics/MASE[0.5]"] for k in keys]),
+            "CRPS": gmean([df.loc[k, "eval_metrics/mean_weighted_sum_quantile_loss"] / sn.loc[k, "eval_metrics/mean_weighted_sum_quantile_loss"] for k in keys]),
+        }
+    out["n_configs"] = len(keys)
+    return out
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--export", required=True)
+    ap.add_argument("--mode", default="quick", choices=["quick", "full"])
+    ap.add_argument("--configs", default="", help="override: 'name:term,name:term'")
+    ap.add_argument("--out", required=True)
+    ap.add_argument("--gift_eval_dir", default=os.environ.get("GIFT_EVAL", str(ROOT / "data" / "gift_eval")))
+    ap.add_argument("--max_context", type=int, default=0, help="0 = model's trained context_length")
+    ap.add_argument("--model_name", default="Modular-TS-2.5M")
+    ap.add_argument("--tta", default="", help="v3 test-time ensemble spec, e.g. 'c8192,c2048,d4' (modular_ts.tta)")
+    ap.add_argument("--tta_alpha", type=float, default=0.5)
+    ap.add_argument("--tta_backtest", action="store_true")
+    ap.add_argument("--tta_tau", type=float, default=0.5)
+    ap.add_argument("--tta_bt_windows", type=int, default=1)
+    ap.add_argument("--tta_prior", type=float, default=0.0)
+    ap.add_argument("--device", default="cuda" if torch.cuda.is_available() else "cpu")
+    args = ap.parse_args()
+
+    logging.getLogger("gluonts.model.forecast").setLevel(logging.ERROR)
+    warnings.filterwarnings("ignore")
+    os.environ["GIFT_EVAL"] = args.gift_eval_dir
+    if os.environ.get("EVAL_MEM_FRAC") and torch.cuda.is_available():  # share the GPU with a training job
+        torch.cuda.set_per_process_memory_fraction(float(os.environ["EVAL_MEM_FRAC"]))
+    props = json.load(open(PROPS))
+    if args.configs:
+        cfgs = [tuple(x.split(":")) for x in args.configs.split(",")]
+    elif args.mode == "quick":
+        cfgs = QUICK
+    else:
+        cfgs = [(n, t) for n, t, _ in all_configs(props)]
+    ensure_data([n for n, _ in cfgs], args.gift_eval_dir)
+
+    from timestral import TimestralForecaster
+    fc = TimestralForecaster.from_pretrained(args.export, device=args.device,
+                                             max_context=args.max_context or None)
+    if args.tta:
+        from modular_ts.tta import EnsembleForecaster
+        fc = EnsembleForecaster(fc, args.tta, alpha=args.tta_alpha, backtest=args.tta_backtest, tau=args.tta_tau, bt_windows=args.tta_bt_windows, prior=args.tta_prior)
+    out = Path(args.out)
+    (out / "per_config").mkdir(parents=True, exist_ok=True)
+    per = {}
+    for name, term in cfgs:
+        key = config_key(name, term, props)
+        p = out / "per_config" / (key.replace("/", "__") + ".json")
+        if p.exists():
+            per[key] = json.loads(p.read_text())["metrics"]
+            continue
+        t = time.time()
+        try:
+            met = evaluate_config(fc, name, term)
+        except Exception as e:  # keep going; a failed config is reported, not silently dropped
+            print(f"{key}: Error {e!r}"[:400], flush=True)
+            continue
+        p.write_text(json.dumps({"dataset": key, "metrics": met, "seconds": time.time() - t}, indent=1))
+        per[key] = met
+        print(f"{key}: MASE={met['MASE[0.5]']:.4f} CRPS={met['mean_weighted_sum_quantile_loss']:.4f} ({time.time()-t:.0f}s)", flush=True)
+    agg = aggregate(per)
+    json.dump(agg, open(out / "summary.json", "w"), indent=1)
+    print("SUMMARY", json.dumps(agg), flush=True)
+    if args.mode == "full":
+        rows = []
+        for k, met in sorted(per.items()):
+            ds_key = k.split("/")[0]
+            rows.append({"dataset": k, "model": args.model_name, **{f"eval_metrics/{c}": met[c] for c in METRIC_COLUMNS},
+                         "domain": props[ds_key]["domain"], "num_variates": props[ds_key]["num_variates"]})
+        pd.DataFrame(rows).to_csv(out / "all_results.csv", index=False)
+
+
+if __name__ == "__main__":
+    main()

modular_ts/model.py

@@ -0,0 +1,220 @@
+"""Modular-TS training model: a Timestral-style network trained from scratch with DHF.
+
+The network *is* `timestral.model.TimestralModel` (Apache-2.0 architecture code, random
+init), so parameter names match the open inference package and a trained checkpoint
+exports to `config.json` + `model.safetensors` that `timestral.TimestralForecaster`
+loads unchanged. We add `forward_train`, the Decoupled-Horizon-Forecasting training pass
+(our reconstruction; Mistral has not published it):
+
+- batch layout (B groups, V rows per group, T = N * patch steps); rows are target,
+  past-only covariate, known-future covariate, or padding;
+- Enc = all N patch embeddings (causal per patch, past-only scaling + asinh);
+  Dec = N copies of the learned forecast query; per group a min-context s, decoder
+  blocks of K patches anchored at s, s+K, ...; Dec token in block b reads Enc[< b] plus
+  the causal Dec chain of its own block (timestral.masks.build_two_stream_time_mask);
+- variate attention per patch slot over the V rows of a group with the covariate
+  firewall (covariates never read targets; known-future rows never read causal rows);
+- target of Dec token p = true patch p normalized in the causal frame of patches < p
+  (the "advancing refine frame" the inference code decodes in), asinh-squashed;
+- loss = pinball over the 21 quantile knots on observed target values of active patches.
+
+NEVER load Timestral weights into this model (Modified-MIT licence, see NOTICE).
+"""
+
+from __future__ import annotations
+
+import math
+import os
+import sys
+
+import torch
+import torch.nn.functional as F
+from torch import Tensor
+
+_HERE = os.path.dirname(os.path.abspath(__file__))
+_TS_SRC = os.path.join(os.path.dirname(_HERE), "timestral", "src")
+if os.path.isdir(_TS_SRC) and _TS_SRC not in sys.path:
+    sys.path.insert(0, _TS_SRC)
+
+from timestral.config import TimestralConfig  # noqa: E402
+from timestral.layers import Attention, _apply_rotary  # noqa: E402
+from timestral.masks import build_two_stream_time_mask, compute_enc_key_validity  # noqa: E402
+from timestral.model import ROLE_KNOWN_COV, ROLE_PAST_COV, ROLE_TARGET, TimestralModel  # noqa: E402
+from timestral.scaler import apply_affine_asinh_masked, causal_patch_loc_scale  # noqa: E402
+
+ROLE_PAD = -1
+QUANTILES_21 = (0.01, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5,
+                0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95, 0.99)
+
+
+def make_config(size: str = "2.5M", context_length: int = 4096) -> TimestralConfig:
+    presets = {
+        # d_model, heads, mlp, layers-groups
+        "2.5M": (128, 2, 320, 2),
+        "5M": (192, 3, 512, 2),
+        "28M": (384, 6, 1024, 3),
+        "tiny": (64, 1, 128, 2),  # CPU tests only
+    }
+    d, h, mlp, k = presets[size]
+    return TimestralConfig(
+        patch_size=32, d_model=d, num_heads=h, mlp_hidden_dim=mlp,
+        block_schedule=("time", "time", "time", "time_variate") * k,
+        quantile_knots=QUANTILES_21, block_size=32, context_length=context_length,
+        rope_theta=1e4, rope_partial_factor=0.5, norm_eps=1e-6, attn_bias=False, mlp_bias=False,
+        time_attn_gate="sdpa_sigmoid", variate_attn_gate="sdpa_sigmoid", minimum_scale=1e-6,
+        use_asinh=True, stabilize_with_global=False, covariate_enabled=True,
+        split_context_horizon_attn=True, attend_missing_patches=True, quantile_width_factor=3.0,
+    )
+
+
+def count_params(model: torch.nn.Module) -> dict:
+    tot = sum(p.numel() for p in model.parameters())
+    emb = sum(p.numel() for n, p in model.named_parameters() if n.startswith("representation"))
+    head = sum(p.numel() for n, p in model.named_parameters() if n.startswith("head"))
+    return {"total": tot, "embed": emb, "trunk": tot - emb - head, "head": head}
+
+
+def _variate_attention(attn: Attention, x: Tensor, mask: Tensor, B: int, V: int) -> Tensor:
+    """Variate attention per (group, slot) over V rows. x: (B*V, S, D); mask: (B*S, 1, V, V)."""
+    R, S, D = x.shape
+    inp = x.view(B, V, S, D).permute(0, 2, 1, 3).reshape(B * S, V, D)
+    q, k, v = attn._qkv(inp)  # (B*S, V, H, hd)
+    out = F.scaled_dot_product_attention(q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2), attn_mask=mask)
+    out = out.transpose(1, 2).reshape(B * S, V, D)
+    if attn._gate is not None:
+        out = out * attn._gate(inp).sigmoid()
+    out = attn.wO(out)
+    return out.view(B, S, V, D).permute(0, 2, 1, 3).reshape(R, S, D)
+
+
+def _time_attention(attn: Attention, x: Tensor, mask: Tensor, cos: Tensor, sin: Tensor) -> Tensor:
+    q, k, v = attn._qkv(x)
+    q, k = _apply_rotary(q, k, cos, sin)
+    out = F.scaled_dot_product_attention(q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2), attn_mask=mask)
+    out = out.transpose(1, 2).reshape(x.shape[0], x.shape[1], -1)
+    if attn._gate is not None:
+        out = out * attn._gate(x).sigmoid()
+    return attn.wO(out)
+
+
+class ModularTS(TimestralModel):
+    """TimestralModel + DHF training pass (identical parameter names)."""
+
+    def __init__(self, config: TimestralConfig) -> None:
+        super().__init__(config)
+        self.register_buffer("q_levels", torch.tensor(config.quantile_knots, dtype=torch.float32), persistent=False)
+        self._init_weights()
+
+    def _init_weights(self) -> None:
+        for name, p in self.named_parameters():
+            if p.dim() == 2:
+                torch.nn.init.normal_(p, std=0.02)
+                if name.endswith("wO.weight") or name.endswith("w3.weight"):
+                    p.data.mul_(1.0 / math.sqrt(2 * len(self.config.block_schedule)))
+            elif name.endswith(".bias"):
+                torch.nn.init.zeros_(p)
+        torch.nn.init.normal_(self.representation.forecast_query, std=0.02)
+
+    # ------------------------------------------------------------------ masks
+    def build_masks(self, observed: Tensor, roles: Tensor, s: Tensor):
+        """observed (B,V,T) bool, roles (B,V) long, s (B,) long -> masks/positions/key validity."""
+        c = self.config
+        B, V, T = observed.shape
+        P, N, R = c.patch_size, T // c.patch_size, B * V
+        dev = observed.device
+        obs_pp = observed.view(R, N, P).any(-1)
+        not_pad = (roles != ROLE_PAD).view(R)
+        key_valid = compute_enc_key_validity(obs_pp, c.attend_missing_patches) & not_pad[:, None]
+        known = (roles == ROLE_KNOWN_COV).view(R)
+        target = (roles == ROLE_TARGET).view(R)
+        s_rows = s.repeat_interleave(V)
+        time_mask = build_two_stream_time_mask(s_rows, N, c.block_size, key_valid, bidirectional_rows=known)
+        # variate mask (B*2N, 1, V, V)
+        slot_valid = torch.cat([key_valid, not_pad[:, None].expand(R, N)], dim=1).view(B, V, 2 * N)
+        keyok = slot_valid.permute(0, 2, 1)[:, :, None, :]  # (B, 2N, 1, Vk)
+        tg = target.view(B, V)
+        kn = known.view(B, V)
+        forbid = ((~tg)[:, :, None] & tg[:, None, :]) | (kn[:, :, None] & (~kn)[:, None, :])  # (B, Vq, Vk)
+        vmask = keyok & ~forbid[:, None]
+        eye = torch.eye(V, dtype=torch.bool, device=dev)
+        vmask = (vmask | eye).reshape(B * 2 * N, 1, V, V)
+        # positions anchored at each group's first observed patch
+        grp_obs = obs_pp.view(B, V, N).any(1)  # (B, N)
+        first = torch.where(grp_obs.any(-1), grp_obs.float().argmax(-1), torch.zeros(B, device=dev)).long()
+        p = torch.arange(N, device=dev)
+        pos = (p[None] - first[:, None]).repeat_interleave(V, 0)
+        positions = torch.cat([pos, pos], dim=1)
+        return time_mask, vmask, positions, obs_pp, target, known
+
+    # ---------------------------------------------------------------- forward
+    def forward_train(self, values: Tensor, observed: Tensor, roles: Tensor, s: Tensor, return_preds: bool = False,
+                      teacher: Tensor | None = None, teacher_obs: Tensor | None = None, gt_row: Tensor | None = None,
+                      lam: float = 0.5, teacher_q_idx: tuple = (2, 4, 6, 8, 10, 12, 14, 16, 18)):
+        """values (B,V,T) float (anything where unobserved), observed (B,V,T) bool,
+        roles (B,V) long in {-1,0,1,2}, s (B,) min-context patches. Returns (loss, stats)."""
+        c = self.config
+        B, V, T = values.shape
+        P, N, R = c.patch_size, T // c.patch_size, B * V
+        x = torch.where(observed, values, torch.zeros_like(values)).float().view(R, T)
+        obs = observed.view(R, T)
+
+        with torch.no_grad():
+            time_mask, vmask, positions, obs_pp, target_rows, known_rows = self.build_masks(observed, roles, s)
+            loc, scale = causal_patch_loc_scale(x[:, None], obs[:, None], P, c.minimum_scale, False)
+            loc, scale = loc[:, 0], scale[:, 0]
+            scaled = apply_affine_asinh_masked(x, loc, scale, obs, use_asinh=c.use_asinh)
+            scaled, _, _ = self._known_cov_scaling(x, obs, scaled, loc, scale, known_rows)
+            # DHF targets in the frame of patches < p
+            loc_pp = loc.view(R, N, P)[..., 0]
+            sc_pp = scale.view(R, N, P)[..., 0]
+            f_loc = torch.cat([loc_pp[:, :1], loc_pp[:, :-1]], dim=1)
+            f_sc = torch.cat([sc_pp[:, :1], sc_pp[:, :-1]], dim=1)
+            y = torch.asinh((x.view(R, N, P) - f_loc[..., None]) / f_sc[..., None].clamp_min(1e-6))
+            cnt_before = torch.cumsum(obs.view(R, N, P).sum(-1), 1) - obs.view(R, N, P).sum(-1)
+            p_idx = torch.arange(N, device=x.device)
+            active = p_idx[None] >= s.repeat_interleave(V)[:, None]
+            w = (obs.view(R, N, P) & target_rows[:, None, None] & active[..., None] & (cnt_before >= 2)[..., None]).float()
+            if gt_row is not None:  # rows whose ground truth may not be used (teacher-only inputs)
+                w = w * gt_row.view(R, 1, 1).float()
+            y = torch.where(w > 0, y, torch.zeros_like(y))
+            if teacher is not None:
+                # teacher (R, T, Qt) raw-space quantiles (anything where ~teacher_obs) -> same advancing frame + asinh
+                Qt = teacher.shape[-1]
+                tv = torch.where(teacher_obs.view(R, T, 1), teacher.view(R, T, Qt).float(), torch.zeros(1, device=x.device))
+                yt = torch.asinh((tv.view(R, N, P, Qt) - f_loc[..., None, None]) / f_sc[..., None, None].clamp_min(1e-6))
+                wt = (teacher_obs.view(R, N, P) & target_rows[:, None, None] & active[..., None] & (cnt_before >= 2)[..., None]).float()
+                yt = torch.where(wt[..., None] > 0, yt, torch.zeros_like(yt))
+
+        x_enc = self.representation(scaled, obs)
+        x_dec = self.representation.forecast_tokens(R, N)
+        h = torch.cat([x_enc, x_dec.to(x_enc.dtype)], dim=1)
+        cos, sin = self.transformer.rope(h, positions)
+        for layer in self.transformer.layers:
+            for norm, attn in zip(layer.attn_norms, layer.attns):
+                if attn.is_time:
+                    h = h + _time_attention(attn, norm(h), time_mask, cos, sin)
+                else:
+                    h = h + _variate_attention(attn, norm(h), vmask, B, V)
+            h = h + layer.mlp(layer.mlp_norm(h))
+        h = self.transformer.out_norm(h)
+        preds = self.head(h[:, N:]).view(R, N, P, -1).float()  # (R,N,P,Q) normalized, monotone
+
+        if return_preds:
+            return preds, f_loc, f_sc
+        err = y[..., None] - preds
+        tau = self.q_levels.view(1, 1, 1, -1)
+        pin = torch.maximum(tau * err, (tau - 1) * err).mean(-1)
+        denom = w.sum().clamp_min(1.0)
+        loss = (pin * w).sum() / denom
+        with torch.no_grad():
+            mae = ((y - preds[..., c.median_index]).abs() * w).sum() / denom
+        stats = {"pinball": loss.detach(), "mae_med": mae, "n_tgt": denom}
+        if teacher is not None:
+            pq = preds[..., list(teacher_q_idx)]
+            tq = self.q_levels[list(teacher_q_idx)].view(1, 1, 1, -1)
+            et = yt - pq
+            kd = torch.maximum(tq * et, (tq - 1) * et).mean(-1)
+            kd_loss = (kd * wt).sum() / wt.sum().clamp_min(1.0)
+            loss = (1 - lam) * loss + lam * kd_loss
+            stats["kd"] = kd_loss.detach()
+        return loss, stats

modular_ts/realdata.py

@@ -0,0 +1,363 @@
+"""Optional real-data source: Salesforce/GiftEvalPretrain (OFF by default).
+
+GiftEvalPretrain's card says Apache-2.0 but "for research purposes only"; the user has
+not chosen between synthetic-only and synthetic+GiftEvalPretrain, so train.py keeps this
+behind `--data synthetic+gifteval` (default `--data synthetic`).
+
+Contamination filter: every GiftEvalPretrain sub-dataset whose normalised name matches
+(or is an explicit alias of) a dataset in the GIFT-Eval test pool, fev-bench tasks or the
+TIME benchmark is dropped. The resolved include/exclude lists are written next to each
+checkpoint (data_manifest.json) for the model card's contamination audit.
+
+Data layout on the Hub: one folder per sub-dataset holding HF `save_to_disk` arrow
+shards (`data-0000k-of-0000n.arrow`) with columns item_id, start, freq, target
+(1-D or 2-D list). We download shards lazily, memory-map them, and crop random windows
+on a background thread.
+"""
+
+from __future__ import annotations
+
+import queue
+import math
+import random
+import re
+import threading
+
+import numpy as np
+import torch
+
+REPO = "Salesforce/GiftEvalPretrain"
+
+GIFT_EVAL_TEST = [
+    "m4_yearly", "m4_quarterly", "m4_monthly", "m4_weekly", "m4_daily", "m4_hourly", "electricity", "solar",
+    "hospital", "covid_deaths", "us_births", "saugeenday", "temperature_rain_with_missing",
+    "kdd_cup_2018_with_missing", "car_parts_with_missing", "restaurant", "hierarchical_sales", "LOOP_SEATTLE",
+    "SZ_TAXI", "M_DENSE", "ett1", "ett2", "jena_weather", "bitbrains_fast_storage", "bitbrains_rnd",
+    "bizitobs_application", "bizitobs_service", "bizitobs_l2c",
+]
+FEV_BENCH = [
+    "ETT", "LOOP_SEATTLE", "M_DENSE", "SZ_TAXI", "australian_tourism", "bizitobs_l2c", "boomlet", "ecdc_ili",
+    "entsoe", "epf_be", "epf_de", "epf_fr", "epf_np", "epf_pjm", "ercot", "favorita_stores", "favorita_transactions",
+    "fred_md_2025", "fred_qd_2025", "gvar", "hermes", "hierarchical_sales", "hospital", "hospital_admissions",
+    "jena_weather", "kdd_cup_2022", "m5", "proenfo_gfc12", "proenfo_gfc14", "proenfo_gfc17", "redset", "restaurant",
+    "rohlik_orders", "rohlik_sales", "rossmann", "solar", "solar_with_weather", "uci_air_quality", "uk_covid_nation",
+    "uk_covid_utla", "us_consumption", "walmart", "world_co2_emissions", "world_life_expectancy", "world_tourism",
+]
+TIME_BENCH = [
+    "Australia_Solar", "Auto_Production_SF", "CPHL", "Coastal_T_S", "Commodity_Import", "Commodity_Production",
+    "Crypto", "ECDC_COVID", "EWELD_Load", "Finland_Traffic", "Global_Influenza", "Global_Price", "Housing_Inventory",
+    "JOLTS", "Job_Claims", "MetroPT-3", "NE_China_Wind", "Oil_Price", "Online_Retail_2_UCI", "OpenElectricity_NEM",
+    "Port_Activity", "SG_Carpark", "SG_PM25", "SG_Weather", "Smart_Manufacturing", "Supply_Chain_Customer",
+    "Supply_Chain_Location", "US_Labor", "US_Term_Structure", "Uncertainty_1M", "Vehicle_Sales", "Vehicle_Supply",
+    "WUI_Global", "Water_Quality_Darwin", "azure2019", "current_velocity", "epf_electricity_price",
+]
+# GiftEvalPretrain names that are the same source as a benchmark dataset under another name.
+ALIASES_EXCLUDE = {
+    "favorita_sales": "fev favorita_stores", "favorita_transactions": "fev favorita_transactions",
+    "m5": "fev m5", "kdd2022": "fev kdd_cup_2022", "gfc12_load": "fev proenfo_gfc12",
+    "gfc14_load": "fev proenfo_gfc14", "gfc17_load": "fev proenfo_gfc17", "fred_md": "fev fred_md_2025",
+    "solar_power": "GIFT-Eval solar (conservative)", "ecdc_ili": "fev ecdc_ili",
+}
+
+
+def _norm(s: str) -> str:
+    s = s.lower()
+    s = re.sub(r"_with_missing$", "", s)
+    s = re.sub(r"_(\d+[a-z]+|daily|hourly|weekly|monthly|yearly|quarterly)$", "", s)
+    return re.sub(r"[^a-z0-9]", "", s)
+
+
+def list_pretrain_datasets() -> list[str]:
+    from huggingface_hub import HfApi
+    tree = HfApi().list_repo_tree(REPO, repo_type="dataset")
+    return sorted(t.path for t in tree if "." not in t.path)
+
+
+def filter_datasets(names: list[str]) -> tuple[list[str], dict[str, str]]:
+    bench = {}
+    for pool, lst in (("GIFT-Eval", GIFT_EVAL_TEST), ("fev-bench", FEV_BENCH), ("TIME", TIME_BENCH)):
+        for b in lst:
+            bench[_norm(b)] = f"{pool} {b}"
+    keep, dropped = [], {}
+    for n in names:
+        k = _norm(n)
+        hit = ALIASES_EXCLUDE.get(n) or bench.get(k)
+        if hit is None:
+            hit = next((v for bk, v in bench.items() if len(bk) >= 5 and (bk in k or k in bk)), None)
+        if hit:
+            dropped[n] = hit
+        else:
+            keep.append(n)
+    return keep, dropped
+
+
+class GiftEvalPretrainStream:
+    """Random fixed-length windows from GiftEvalPretrain, batched as (B, V, T) like SynthStream.
+
+    Univariate series give V=1 groups; multivariate series give up to `max_v` variates.
+    `datasets` is optional; used only to memory-map arrow shards.
+    """
+
+    def __init__(self, T: int, tokens_per_batch: int, device="cpu", seed=0, datasets_subset: list[str] | None = None,
+                 max_shards_per_dataset: int = 2, max_v: int = 16, patch=32, min_ctx=4, cache_dir=None, prefetch=4):
+        from datasets import Dataset as HFDataset
+        from huggingface_hub import HfApi, hf_hub_download
+
+        names = datasets_subset or list_pretrain_datasets()
+        self.keep, self.dropped = filter_datasets(names)
+        self.T, self.N, self.tok, self.patch = T, T // patch, tokens_per_batch, patch
+        self.dev, self.min_ctx, self.max_v = torch.device(device), min_ctx, max_v
+        self.rng = random.Random(seed)
+        api = HfApi()
+        self.tables = []
+        for name in self.keep:
+            files = sorted(f for f in api.list_repo_files(REPO, repo_type="dataset") if f.startswith(name + "/") and f.endswith(".arrow"))
+            for f in files[:max_shards_per_dataset]:
+                p = hf_hub_download(REPO, f, repo_type="dataset", cache_dir=cache_dir)
+                self.tables.append((name, HFDataset.from_file(p)))
+        assert self.tables, "no GiftEvalPretrain shards left after filtering"
+        self.q: queue.Queue = queue.Queue(maxsize=prefetch)
+        threading.Thread(target=self._worker, daemon=True).start()
+
+    def manifest(self) -> dict:
+        return {"source": REPO, "included": self.keep, "excluded": self.dropped}
+
+    def _window(self):
+        name, tab = self.rng.choice(self.tables)
+        row = tab[self.rng.randrange(len(tab))]
+        x = np.asarray(row["target"], dtype=np.float32)
+        x = x[None] if x.ndim == 1 else x
+        if x.shape[0] > self.max_v:
+            x = x[self.rng.sample(range(x.shape[0]), self.max_v)]
+        L = x.shape[1]
+        if L >= self.T:
+            st = self.rng.randrange(L - self.T + 1)
+            w = x[:, st : st + self.T]
+        else:  # left-pad with NaN (unobserved)
+            w = np.concatenate([np.full((x.shape[0], self.T - L), np.nan, np.float32), x], 1)
+        return w
+
+    def _make(self):
+        # bucket by V: draw one window, then fill the batch with windows of the same V
+        first = self._window()
+        V = first.shape[0]
+        Bg = max(1, self.tok // (V * self.N))
+        ws = [first]
+        tries = 0
+        while len(ws) < Bg and tries < 20 * Bg:
+            w = self._window()
+            tries += 1
+            if w.shape[0] == V:
+                ws.append(w)
+        arr = np.stack(ws)
+        obs = np.isfinite(arr)
+        vals = np.nan_to_num(arr, nan=0.0, posinf=0.0, neginf=0.0)
+        return vals, obs
+
+    def _worker(self):
+        while True:
+            self.q.put(self._make())
+
+    def next_batch(self, V=None) -> dict:
+        vals, obs = self.q.get()
+        Bg, V, _ = vals.shape
+        s = torch.randint(self.min_ctx, self.N - 1, (Bg,))
+        return {
+            "values": torch.from_numpy(vals).to(self.dev, non_blocking=True),
+            "observed": torch.from_numpy(obs).to(self.dev, non_blocking=True),
+            "roles": torch.zeros(Bg, V, dtype=torch.long, device=self.dev),
+            "s": s.to(self.dev),
+        }
+
+
+class MixedStream:
+    """Draw from `real` with probability `real_frac`, else from `synth`."""
+
+    def __init__(self, synth, real, real_frac: float, seed=0):
+        self.synth, self.real, self.p = synth, real, real_frac
+        self.rng = random.Random(seed)
+
+    def next_batch(self, V=None):
+        return self.real.next_batch() if self.rng.random() < self.p else self.synth.next_batch(V)
+
+
+# ---------------------------------------------------------------------------------------------
+# Fast GPU-resident stream (used for run1). The thread/row-based stream above cannot keep up
+# with ~12+ steps/s x 1024 windows; this one loads a size-capped selection of shards once into
+# one flat float32 tensor on the GPU and crops whole batches with a single vectorised gather.
+# ---------------------------------------------------------------------------------------------
+
+def _arrow_channels(path):
+    """Return (values float32 1-D, ch_off int64, ch_len int64, row_first_ch, row_nvar)."""
+    import pyarrow as pa
+    try:
+        tab = pa.ipc.open_stream(path).read_all()
+    except Exception:
+        tab = pa.ipc.open_file(path).read_all()
+    col = tab.column("target").combine_chunks()
+    t = col.type
+    if pa.types.is_list(t) and not (pa.types.is_list(t.value_type) or pa.types.is_fixed_size_list(t.value_type)):
+        inner, nvar = col, np.ones(len(col), np.int64)
+    elif pa.types.is_fixed_size_list(t):
+        inner, nvar = col.flatten(), np.full(len(col), t.list_size, np.int64)
+    else:  # list<list<float>>
+        inner = col.flatten()
+        off = np.asarray(col.offsets)
+        nvar = (off[1:] - off[:-1]).astype(np.int64)
+    off = np.asarray(inner.offsets).astype(np.int64)
+    vals = inner.values.to_numpy(zero_copy_only=False).astype(np.float32, copy=False)
+    base = off[0]
+    vals = vals[base:off[-1]]
+    ch_off = off[:-1] - base
+    ch_len = off[1:] - off[:-1]
+    row_first = np.concatenate([[0], np.cumsum(nvar)[:-1]]).astype(np.int64)
+    return vals, ch_off, ch_len, row_first, nvar
+
+
+class FastGiftEvalPretrain:
+    def __init__(self, T, tokens_per_batch, device, selection: dict, seed=0, patch=32, cache_dir=None,
+                 mv_frac=0.3, max_v=16, short_ctx_p=0.3, ds_temp=0.5, workers=16):
+        from concurrent.futures import ThreadPoolExecutor
+        from huggingface_hub import hf_hub_download
+        names = sorted(selection)
+        self.keep, self.dropped = filter_datasets(names)
+        assert set(self.keep) == set(names), f"selection contains filtered datasets: {set(names) - set(self.keep)}"
+        self.T, self.P, self.N, self.tok = T, patch, T // patch, tokens_per_batch
+        self.dev, self.max_v, self.mv_frac, self.short_ctx_p = torch.device(device), max_v, mv_frac, short_ctx_p
+        self.g = torch.Generator(device=self.dev); self.g.manual_seed(seed + 777)
+        jobs = [(n, f) for n in names for f in selection[n]]
+        with ThreadPoolExecutor(workers) as ex:
+            paths = list(ex.map(lambda nf: hf_hub_download(REPO, nf[1], repo_type="dataset", cache_dir=cache_dir), jobs))
+        print(f"[gep] downloaded {len(paths)} shards", flush=True)
+        with ThreadPoolExecutor(8) as ex:
+            parsed = list(ex.map(_arrow_channels, paths))
+        vals_all, ch_off_all, ch_len_all, ch_ds, mv_rows = [], [], [], [], []  # mv_rows: (ds, first_ch_global, nvar)
+        base, nch = 0, 0
+        ds_idx = {n: i for i, n in enumerate(names)}
+        self.ds_points = np.zeros(len(names))
+        for (n, _), (v, co, cl, rf, nv) in zip(jobs, parsed):
+            keep = cl >= 8
+            d = ds_idx[n]
+            vals_all.append(v)
+            ch_off_all.append(co + base); ch_len_all.append(cl); ch_ds.append(np.full(len(cl), d, np.int64))
+            m = (nv >= 2)
+            if m.any():
+                mv_rows.append(np.stack([np.full(m.sum(), d), rf[m] + nch, nv[m]], 1))
+            self.ds_points[d] += cl[keep].sum()
+            base += len(v); nch += len(cl)
+        flat = np.concatenate(vals_all)
+        ch_off, ch_len, ch_ds = map(np.concatenate, (ch_off_all, ch_len_all, ch_ds))
+        good = ch_len >= 8
+        print(f"[gep] {len(names)} datasets, {nch} channels, {len(flat)/1e9:.2f}B points", flush=True)
+        self.flat = torch.from_numpy(flat).to(self.dev)
+        del flat, vals_all
+        # univariate channel pool grouped by dataset
+        order = np.argsort(ch_ds[good], kind="stable")
+        gi = np.nonzero(good)[0][order]
+        self.u_off = torch.from_numpy(ch_off[gi]).to(self.dev)
+        self.u_len = torch.from_numpy(ch_len[gi]).to(self.dev)
+        cnt = np.bincount(ch_ds[gi], minlength=len(names))
+        self.u_start = torch.from_numpy(np.concatenate([[0], np.cumsum(cnt)[:-1]])).to(self.dev)
+        self.u_cnt = torch.from_numpy(cnt).to(self.dev)
+        w = np.where(cnt > 0, self.ds_points ** ds_temp, 0.0)
+        self.u_w = torch.tensor(w / w.sum(), dtype=torch.float, device=self.dev)
+        self.weights = {n: float(x) for n, x in zip(names, w / w.sum())}
+        # multivariate rows
+        self.ch_off_t = torch.from_numpy(ch_off).to(self.dev)
+        self.ch_len_t = torch.from_numpy(ch_len).to(self.dev)
+        self.mv = None
+        if mv_rows:
+            mvr = np.concatenate(mv_rows)
+            mvr = mvr[ch_len[mvr[:, 1]] >= 8]
+            self.mv = mvr
+            self.mv_by_v = {}
+            for V in (2, 4, 8, 16):
+                sub = mvr[mvr[:, 2] >= V]
+                if len(sub) == 0:
+                    continue
+                dcnt = np.bincount(sub[:, 0], minlength=len(names))
+                o = np.argsort(sub[:, 0], kind="stable"); sub = sub[o]
+                wv = np.where(dcnt > 0, self.ds_points ** ds_temp, 0.0)
+                self.mv_by_v[V] = dict(first=torch.from_numpy(sub[:, 1]).to(self.dev), nvar=torch.from_numpy(sub[:, 2]).to(self.dev),
+                                       start=torch.from_numpy(np.concatenate([[0], np.cumsum(dcnt)[:-1]])).to(self.dev),
+                                       cnt=torch.from_numpy(dcnt).to(self.dev), w=torch.tensor(wv / wv.sum(), dtype=torch.float, device=self.dev))
+        self.rng = random.Random(seed)
+        self.ar = torch.arange(T, device=self.dev)
+
+    def manifest(self):
+        try:
+            excl = filter_datasets(list_pretrain_datasets())[1]
+        except Exception as e:  # offline
+            excl = f"unavailable: {e!r}"
+        return {"source": REPO, "selection": "size-capped shard selection (gep_selection.json)", "included": self.keep,
+                "excluded_by_contamination_filter": excl, "sampling_weights": self.weights, "points_per_dataset": {n: int(x) for n, x in zip(sorted(self.weights), self.ds_points)}}
+
+    def _rand(self, n):
+        return torch.rand(n, generator=self.g, device=self.dev)
+
+    def _pick(self, w, start, cnt, n):
+        d = torch.multinomial(w, n, replacement=True, generator=self.g)
+        return start[d] + (self._rand(n) * cnt[d]).long().clamp_max(cnt[d] - 1)
+
+    def _crop(self, off, L):
+        """off, L: (R,) -> values (R,T), observed (R,T). Same start for rows of one group handled by caller."""
+        T = self.T
+        st = torch.where(L >= T, (self._rand(L.shape[0]) * (L - T + 1)).long(), L - T)
+        return st
+
+    def _gather(self, off, L, st, keep_len):
+        pos = st[:, None] + self.ar[None]
+        valid = (pos >= 0) & (pos < L[:, None]) & (self.ar[None] >= (self.T - keep_len)[:, None])
+        idx = (off[:, None] + pos.clamp(min=0)).clamp_max(self.flat.numel() - 1)
+        v = self.flat[idx]
+        obs = valid & torch.isfinite(v)
+        return torch.where(obs, v, torch.zeros_like(v)), obs
+
+    def _s(self, obs_bv):
+        # obs_bv (B,V,T): first observed patch per group -> min-context split after it
+        B = obs_bv.shape[0]
+        any_p = obs_bv.view(B, -1, self.N, self.P).any(-1).any(1)  # (B,N)
+        fs = torch.where(any_p.any(1), any_p.float().argmax(1), torch.zeros(B, dtype=torch.long, device=self.dev))
+        span = (self.N - 2 - fs).clamp_min(1)
+        s = fs + 1 + (self._rand(B) * span).long()
+        return s.clamp(1, self.N - 2)
+
+    def _keep_len(self, n):
+        k = torch.full((n,), self.T, device=self.dev, dtype=torch.long)
+        short = self._rand(n) < self.short_ctx_p
+        kl = torch.exp(math.log(64) + self._rand(n) * (math.log(self.T) - math.log(64))).long()
+        return torch.where(short, kl, k)
+
+    @torch.no_grad()
+    def next_batch(self, V=None):
+        if self.mv is not None and self.rng.random() < self.mv_frac:
+            V = self.rng.choice(sorted(self.mv_by_v))
+            m = self.mv_by_v[V]
+            Bg = max(1, self.tok // (V * self.N))
+            r = self._pick(m["w"], m["start"], m["cnt"], Bg)
+            first, nvar = m["first"][r], m["nvar"][r]
+            j0 = (self._rand(Bg) * (nvar - V + 1)).long()
+            ch = first[:, None] + j0[:, None] + torch.arange(V, device=self.dev)[None]  # (Bg,V)
+            off, L = self.ch_off_t[ch], self.ch_len_t[ch]
+            Lmin = L.min(1).values
+            st = self._crop(None, Lmin)
+            kl = self._keep_len(Bg)
+            v, o = self._gather(off.reshape(-1), L.reshape(-1), st.repeat_interleave(V), kl.repeat_interleave(V))
+            v, o = v.view(Bg, V, self.T), o.view(Bg, V, self.T)
+        else:
+            V = 1
+            Bg = max(1, self.tok // self.N)
+            c = self._pick(self.u_w, self.u_start, self.u_cnt, Bg)
+            off, L = self.u_off[c], self.u_len[c]
+            st = self._crop(off, L)
+            v, o = self._gather(off, L, st, self._keep_len(Bg))
+            v, o = v[:, None], o[:, None]
+        return {"values": v, "observed": o, "roles": torch.zeros(Bg, V, dtype=torch.long, device=self.dev), "s": self._s(o)}
+
+
+if __name__ == "__main__":
+    names = list_pretrain_datasets()
+    keep, dropped = filter_datasets(names)
+    print(f"{len(names)} sub-datasets; keep {len(keep)}; drop {len(dropped)}")
+    for k, v in dropped.items():
+        print(f"  DROP {k:40s} <- {v}")

modular_ts/realdata2.py

@@ -0,0 +1,81 @@
+"""GiftEvalPretrain stream v2 (Modular-TS v3): resolution + mixup augmentation on real data.
+
+Adds to FastGiftEvalPretrain (run1):
+  * TTM-style "diverse resolution sampling" (Ekambaram et al., arXiv 2401.03955): with prob
+    `ds_p` a univariate batch is drawn at a coarser resolution, mean-pooling k in {2,4,8}
+    consecutive raw steps (NaN-aware), creating new frequencies from the same corpus;
+  * TSMixup (Chronos, arXiv 2403.07815): with prob `mix_p` rows are replaced by a convex
+    combination of 2-3 standardised real series (Dirichlet weights), rescaled;
+  * flatter dataset temperature (ds_temp 0.3 instead of 0.5) so small datasets are seen more.
+Contamination filter and GPU-resident layout are inherited unchanged.
+"""
+
+from __future__ import annotations
+
+import torch
+
+from modular_ts.realdata import FastGiftEvalPretrain
+
+
+class FastGiftEvalPretrain2(FastGiftEvalPretrain):
+    def __init__(self, *a, ds_p=0.3, ds_factors=(2, 4, 8), mix_p=0.2, **k):
+        k.setdefault("ds_temp", 0.3)
+        super().__init__(*a, **k)
+        self.ds_p, self.ds_factors, self.mix_p = ds_p, ds_factors, mix_p
+
+    def _gather_k(self, off, L, kf):
+        """Gather T*kf raw steps ending at a random point and mean-pool by kf."""
+        T, R = self.T, off.shape[0]
+        TK = T * kf
+        st = torch.where(L >= TK, (self._rand(R) * (L - TK + 1)).long(), L - TK)
+        ar = torch.arange(TK, device=self.dev)
+        pos = st[:, None] + ar[None]
+        valid = (pos >= 0) & (pos < L[:, None])
+        idx = (off[:, None] + pos.clamp(min=0)).clamp_max(self.flat.numel() - 1)
+        v = self.flat[idx]
+        ok = valid & torch.isfinite(v)
+        v = torch.where(ok, v, torch.zeros_like(v)).view(R, T, kf)
+        c = ok.view(R, T, kf).sum(-1)
+        out = v.sum(-1) / c.clamp_min(1)
+        obs = c > 0
+        return torch.where(obs, out, torch.zeros_like(out)), obs
+
+    def _mixup(self, v, o):
+        R, T = v.shape
+        both = o & o.roll(1, 0) & o.roll(2, 0)
+        def zs(x, m):
+            n = m.sum(-1, keepdim=True).clamp_min(1)
+            mu = (x * m).sum(-1, keepdim=True) / n
+            sd = (((x - mu) ** 2) * m).sum(-1, keepdim=True).div(n).sqrt().clamp_min(1e-6)
+            return (x - mu) / sd, sd, mu
+        z0, sd0, mu0 = zs(v, o)
+        z1, _, _ = zs(v.roll(1, 0), o.roll(1, 0))
+        z2, _, _ = zs(v.roll(2, 0), o.roll(2, 0))
+        w = torch.distributions.Dirichlet(torch.full((3,), 1.5, device=self.dev)).sample((R,))
+        w[:, 2] *= (self._rand(R) < 0.5).float()
+        w = w / w.sum(-1, keepdim=True)
+        m = w[:, :1] * z0 + w[:, 1:2] * z1 + w[:, 2:3] * z2
+        m = m * sd0 + mu0
+        sel = (self._rand(R) < self.mix_p)[:, None]
+        return torch.where(sel, torch.where(both, m, 0.0), v), torch.where(sel, both, o)
+
+    @torch.no_grad()
+    def next_batch(self, V=None):
+        if (self.mv is not None and self.rng.random() < self.mv_frac) or self.rng.random() >= self.ds_p:
+            b = super().next_batch(V)
+            if b["values"].shape[1] == 1 and self.mix_p > 0:
+                v, o = self._mixup(b["values"][:, 0], b["observed"][:, 0])
+                b["values"], b["observed"] = v[:, None], o[:, None]
+                b["s"] = self._s(b["observed"])
+            return b
+        kf = self.rng.choice(self.ds_factors)
+        Bg = max(1, self.tok // self.N)
+        c = self._pick(self.u_w, self.u_start, self.u_cnt, Bg)
+        off, L = self.u_off[c], self.u_len[c]
+        v, o = self._gather_k(off, L, kf)
+        o = o & (torch.arange(self.T, device=self.dev)[None] >= (self.T - self._keep_len(Bg))[:, None])
+        v = torch.where(o, v, torch.zeros_like(v))
+        if self.mix_p > 0:
+            v, o = self._mixup(v, o)
+        v, o = v[:, None], o[:, None]
+        return {"values": v, "observed": o, "roles": torch.zeros(Bg, 1, dtype=torch.long, device=self.dev), "s": self._s(o)}

modular_ts/synth.py

@@ -0,0 +1,296 @@
+"""Streamed, fully vectorised synthetic time-series generator (torch; runs on GPU).
+
+Univariate bases (each switched on per series with its own probability, summed or
+multiplied):
+  * KernelSynth-style GP components via random Fourier features: RBF (log-uniform
+    length-scale), rational-quadratic (scale-mixture of RBFs), periodic kernels at
+    frequency-typical periods with random harmonics, linear (trend), white noise;
+  * TSI-style trends: linear / quadratic / piecewise-linear with change points / damped;
+  * AR(1) / random walk (FFT convolution with phi^k kernels);
+  * heteroscedastic noise (log-GP volatility, GARCH-like clustering), student-t tails;
+  * level shifts, spikes / outliers;
+  * intermittent (Croston-like zero-inflated count) and bursty (Poisson bursts with
+    exponential decay) observability-style signals;
+  * non-negativity transforms, random offset / scale over many orders of magnitude.
+Multivariatizers (Chronos-2 / CauKer style): latent univariate drivers mixed into each
+row by a random sparse matrix through random nonlinearities with lead-lag delays, plus
+idiosyncratic noise; optional covariate tasks where some rows are past-only or
+known-future covariates (incl. 0/1 event / calendar indicators) and targets respond to
+them through random lagged nonlinear effects.
+Masks: iid missing points, contiguous gaps, late series starts (left padding).
+
+`SynthStream(...).next_batch()` returns dict(values, observed, roles, s) ready for
+`ModularTS.forward_train`.
+"""
+
+from __future__ import annotations
+
+import math
+
+import torch
+from torch import Tensor
+
+PERIOD_BANK = torch.tensor([4, 5, 7, 12, 24, 48, 52, 60, 96, 144, 168, 288, 336, 365, 672, 1008, 1440, 2016], dtype=torch.float32)
+
+
+def _std(x: Tensor) -> Tensor:
+    return x.std(-1, keepdim=True).clamp_min(1e-6)
+
+
+def _z(x: Tensor) -> Tensor:
+    return (x - x.mean(-1, keepdim=True)) / _std(x)
+
+
+class SynthGen:
+    def __init__(self, T: int, device: str | torch.device = "cpu", seed: int = 0, n_rff: int = 32):
+        self.T = T
+        self.dev = torch.device(device)
+        self.g = torch.Generator(device=self.dev)
+        self.g.manual_seed(seed)
+        self.n_rff = n_rff
+        self.t = torch.arange(T, device=self.dev, dtype=torch.float32)
+
+    # -- random helpers
+    def U(self, *shape, lo=0.0, hi=1.0):
+        return lo + (hi - lo) * torch.rand(*shape, generator=self.g, device=self.dev)
+
+    def N(self, *shape):
+        return torch.randn(*shape, generator=self.g, device=self.dev)
+
+    def B(self, p, *shape):
+        return (self.U(*shape) < p).float()
+
+    def logU(self, *shape, lo, hi):
+        return torch.exp(self.U(*shape, lo=math.log(lo), hi=math.log(hi)))
+
+    # -- components (n, T)
+    def rff(self, n, lengthscale: Tensor) -> Tensor:
+        M = self.n_rff
+        w = self.N(n, M, 1) / lengthscale.view(n, 1, 1)
+        b = self.U(n, M, 1, hi=2 * math.pi)
+        a = self.N(n, M, 1)
+        return (a * torch.cos(w * self.t + b)).sum(1) * math.sqrt(2.0 / M)
+
+    def rbf(self, n):
+        return self.rff(n, self.logU(n, lo=self.T / 400, hi=self.T / 1.5))
+
+    def rq(self, n):  # scale mixture: sum of RBFs at several length-scales
+        out = 0
+        for _ in range(3):
+            out = out + self.U(n, 1) * self.rff(n, self.logU(n, lo=self.T / 1000, hi=self.T / 4))
+        return out
+
+    def periodic(self, n):
+        k = len(PERIOD_BANK)
+        idx = torch.randint(0, k, (n,), generator=self.g, device=self.dev)
+        per = PERIOD_BANK.to(self.dev)[idx]
+        rnd = self.logU(n, lo=3.0, hi=max(4.0, self.T / 3))
+        per = torch.where(self.U(n) < 0.25, rnd, per)
+        per = per * self.U(n, lo=0.97, hi=1.03) * torch.where(self.U(n) < 0.1, 1.0 + 0.2 * self.N(n), torch.ones(n, device=self.dev))
+        H = 6
+        h = torch.arange(1, H + 1, device=self.dev, dtype=torch.float32).view(1, H, 1)
+        decay = self.U(n, 1, 1, lo=0.3, hi=1.5)
+        amp = self.N(n, H, 1) / h.pow(decay) * self.B(0.8, n, H, 1).clamp_min((h == 1).float())
+        ph = self.U(n, H, 1, hi=2 * math.pi)
+        sig = (amp * torch.sin(2 * math.pi * h * self.t / per.view(n, 1, 1) + ph)).sum(1)
+        # optional slowly varying amplitude (multiplicative modulation)
+        mod = 1 + 0.3 * self.B(0.3, n, 1) * torch.tanh(self.rbf(n))
+        return sig * mod
+
+    def trend(self, n):
+        tt = self.t / self.T
+        lin = self.N(n, 1) * tt
+        quad = self.N(n, 1) * 0.5 * tt**2 * self.B(0.3, n, 1)
+        # piecewise linear with up to 4 change points
+        K = 4
+        cps = self.U(n, K, 1)
+        slopes = self.N(n, K, 1) * self.B(0.5, n, K, 1)
+        pw = (slopes * torch.relu(tt.view(1, 1, -1) - cps)).sum(1)
+        damp = self.N(n, 1) * (1 - torch.exp(-tt / self.U(n, 1, lo=0.05, hi=0.5))) * self.B(0.2, n, 1)
+        return lin + quad + pw + damp
+
+    def ar(self, n):
+        phi = torch.where(self.U(n) < 0.3, torch.ones(n, device=self.dev), self.U(n, lo=0.3, hi=0.995))
+        e = self.N(n, self.T)
+        L = min(self.T, 1024)
+        k = phi.view(n, 1) ** torch.arange(L, device=self.dev, dtype=torch.float32)
+        nfft = 1 << (self.T + L - 1).bit_length()
+        y = torch.fft.irfft(torch.fft.rfft(e, nfft) * torch.fft.rfft(k, nfft), nfft)[:, : self.T]
+        rw = torch.cumsum(e, -1)
+        return torch.where((phi == 1).view(n, 1), rw, y)
+
+    def noise(self, n):
+        z = self.N(n, self.T)
+        # heavy tails: scale mixture of normals (generator-friendly stand-in for student-t)
+        heavy = self.B(0.3, n, 1)
+        tz = z * (1 + heavy * (self.N(n, self.T).abs() ** 2) * 0.5)
+        vol = torch.where(self.U(n, 1) < 0.25, torch.exp(0.7 * self.rbf(n).clamp(-3, 3)), torch.ones(n, 1, device=self.dev))
+        return tz * vol
+
+    def level_shifts(self, n):
+        K = 3
+        loc = self.U(n, K, 1) * self.T
+        mag = self.N(n, K, 1) * self.B(0.6, n, K, 1)
+        return (mag * (self.t.view(1, 1, -1) > loc).float()).sum(1)
+
+    def spikes(self, n):
+        p = self.logU(n, 1, lo=1e-3, hi=2e-2)
+        return (self.U(n, self.T) < p).float() * self.N(n, self.T) * self.U(n, 1, lo=3, hi=10)
+
+    def bursts(self, n):
+        rate = self.logU(n, 1, lo=1e-3, hi=2e-2)
+        ev = (self.U(n, self.T) < rate).float() * self.logU(n, self.T, lo=1.0, hi=20.0)
+        tau = self.logU(n, lo=2.0, hi=60.0)
+        L = min(self.T, 512)
+        k = torch.exp(-torch.arange(L, device=self.dev).float().view(1, -1) / tau.view(n, 1))
+        nfft = 1 << (self.T + L - 1).bit_length()
+        return torch.fft.irfft(torch.fft.rfft(ev, nfft) * torch.fft.rfft(k, nfft), nfft)[:, : self.T]
+
+    # -- univariate
+    def univariate(self, n) -> Tensor:
+        z = torch.zeros(n, self.T, device=self.dev)
+        on = lambda p: self.B(p, n, 1)  # noqa: E731
+        w = lambda: self.logU(n, 1, lo=0.1, hi=2.0)  # noqa: E731
+        trend = on(0.6) * w() * _z(self.trend(n))
+        seas = on(0.7) * w() * _z(self.periodic(n)) + on(0.25) * w() * _z(self.periodic(n))
+        smooth = on(0.5) * w() * _z(self.rbf(n)) + on(0.25) * w() * _z(self.rq(n))
+        arp = on(0.4) * w() * _z(self.ar(n))
+        sig = trend + seas + smooth + arp + z
+        mult = on(0.25)
+        sig = torch.where(mult > 0, (1 + 0.4 * torch.tanh(trend + smooth)) * seas + trend + arp, sig)
+        sig = sig + on(0.15) * w() * self.level_shifts(n)
+        noise_lvl = self.logU(n, 1, lo=0.01, hi=1.0)
+        sig = sig + noise_lvl * self.noise(n)
+        sig = sig + on(0.12) * self.spikes(n)
+        # intermittent / bursty regimes (non-negative count-like)
+        inter = on(0.10)
+        lam = torch.exp(0.8 * _z(sig).clamp(-4, 4) + self.U(n, 1, lo=-3, hi=1.5))
+        p0 = self.U(n, 1, lo=0.2, hi=0.9)
+        cnt = torch.poisson(lam, generator=self.g) * (self.U(n, self.T) > p0).float()
+        sig = torch.where(inter > 0, cnt, sig)
+        burst = on(0.07)
+        sig = torch.where(burst > 0, self.U(n, 1, lo=0, hi=2) + 0.1 * self.noise(n).abs() + self.bursts(n), sig)
+        # non-negativity transforms on some continuous series
+        nn_ = on(0.2) * (1 - inter) * (1 - burst)
+        sig = torch.where(nn_ > 0, torch.nn.functional.softplus(sig * self.U(n, 1, lo=0.5, hi=3)), sig)
+        return sig
+
+    def affine(self, x: Tensor) -> Tensor:
+        n = x.shape[0]
+        scale = self.logU(n, 1, lo=1e-2, hi=1e4)
+        off = self.N(n, 1) * scale * self.logU(n, 1, lo=0.01, hi=20.0) * self.B(0.6, n, 1)
+        sgn = torch.where(self.U(n, 1) < 0.5, -1.0, 1.0)
+        pos = (x.min(-1, keepdim=True).values >= 0).float()  # keep count-like series non-negative
+        return torch.where(pos > 0, x * scale, sgn * x * scale + off)
+
+    # -- multivariate group batch
+    def group_batch(self, Bg: int, V: int, cov_prob: float = 0.35):
+        """Returns values (Bg,V,T), roles (Bg,V)."""
+        T = self.T
+        R = Bg * V
+        if V == 1:
+            vals = self.affine(self.univariate(Bg)).view(Bg, 1, T)
+            return vals, torch.zeros(Bg, 1, dtype=torch.long, device=self.dev)
+        K = max(1, V // 2)
+        lat = _z(self.univariate(Bg * K)).view(Bg, K, T)
+        # lead-lag delays per (row, latent)
+        lag = torch.randint(0, 48, (Bg, V, K), generator=self.g, device=self.dev)
+        lag = lag * self.B(0.5, Bg, V, K).long()
+        idx = (self.t.long().view(1, 1, 1, T) - lag[..., None]).clamp_min(0)
+        shifted = torch.gather(lat[:, None].expand(Bg, V, K, T), 3, idx)
+        W = self.N(Bg, V, K, 1) * self.B(0.6, Bg, V, K, 1)
+        W[:, :, 0] = W[:, :, 0] + 0.5
+        mixed = (W * shifted).sum(2)
+        f = torch.randint(0, 5, (Bg, V, 1), generator=self.g, device=self.dev)
+        nl = torch.stack([mixed, torch.tanh(mixed), mixed.abs(), torch.relu(mixed), mixed + 0.3 * mixed**2], 0)
+        mixed = torch.gather(nl, 0, f[None].expand(1, Bg, V, T))[0]
+        own = _z(self.univariate(R)).view(Bg, V, T)
+        a = self.U(Bg, V, 1, lo=0.0, hi=1.0)
+        indep = self.B(0.15, Bg, 1, 1)
+        rows = torch.where(indep > 0, own, (1 - a) * _z(mixed) + a * own)
+        roles = torch.zeros(Bg, V, dtype=torch.long, device=self.dev)
+
+        # covariate tasks
+        cov_g = self.U(Bg) < cov_prob
+        if cov_g.any():
+            n_cov = torch.randint(1, V, (Bg,), generator=self.g, device=self.dev)
+            ridx = torch.arange(V, device=self.dev).view(1, V)
+            is_cov = (ridx >= (V - n_cov).view(Bg, 1)) & cov_g.view(Bg, 1)
+            known = is_cov & (self.U(Bg, V) < 0.6)
+            # some covariates are 0/1 event indicators (holiday / promo / calendar)
+            ev_rate = self.logU(Bg, V, 1, lo=0.005, hi=0.15)
+            per = PERIOD_BANK.to(self.dev)[torch.randint(0, len(PERIOD_BANK), (Bg, V, 1), generator=self.g, device=self.dev)]
+            cal = ((self.t.view(1, 1, T) % per) < torch.clamp(per * self.U(Bg, V, 1, lo=0.05, hi=0.3), min=1)).float()
+            evt = (self.U(Bg, V, T) < ev_rate).float()
+            ind = torch.where(self.U(Bg, V, 1) < 0.5, cal, evt)
+            binary = is_cov & (self.U(Bg, V) < 0.4)
+            cov_vals = torch.where(binary[..., None], ind, rows)
+            # target response to covariates: lagged nonlinear effects
+            beta = self.N(Bg, V, 1) * self.U(Bg, 1, 1, lo=0.5, hi=2.5)
+            clag = torch.randint(0, 8, (Bg, V, 1), generator=self.g, device=self.dev) * (~known[..., None]).long()
+            cidx = (self.t.long().view(1, 1, T) - clag).clamp_min(0)
+            cl = torch.gather(cov_vals, 2, cidx)
+            eff_in = torch.where(binary[..., None], cl, _z(cl))
+            eff = (beta * torch.where(self.U(Bg, V, 1) < 0.5, eff_in, torch.tanh(eff_in)) * is_cov[..., None].float()).sum(1, keepdim=True)
+            tgt_mask = (~is_cov)[..., None].float()
+            rows = torch.where(is_cov[..., None], cov_vals, rows + tgt_mask * eff * cov_g.view(Bg, 1, 1).float())
+            roles = torch.where(is_cov, torch.where(known, 2, 1), roles)
+            bin_rows = binary.view(R)
+        else:
+            bin_rows = torch.zeros(R, dtype=torch.bool, device=self.dev)
+        flat = rows.reshape(R, T)
+        aff = self.affine(flat)
+        flat = torch.where(bin_rows[:, None], flat, aff)
+        return flat.view(Bg, V, T), roles
+
+    def masks(self, vals: Tensor, roles: Tensor) -> Tensor:
+        Bg, V, T = vals.shape
+        obs = torch.ones(Bg, V, T, dtype=torch.bool, device=self.dev)
+        # iid missing
+        pm = self.U(Bg, V, 1, hi=0.15) * self.B(0.2, Bg, V, 1)
+        obs &= self.U(Bg, V, T) >= pm
+        # contiguous gaps
+        gap = self.B(0.15, Bg, V, 1) > 0
+        g0 = self.U(Bg, V, 1) * T
+        gl = self.logU(Bg, V, 1, lo=8, hi=T / 4)
+        in_gap = (self.t.view(1, 1, T) >= g0) & (self.t.view(1, 1, T) < g0 + gl)
+        obs &= ~(gap & in_gap)
+        # late start (left padding), per group
+        late = self.B(0.25, Bg, 1, 1) > 0
+        st = (self.U(Bg, 1, 1) * T * 0.8)
+        obs &= ~(late & (self.t.view(1, 1, T) < st))
+        return obs
+
+
+class SynthStream:
+    """Endless batches of roughly `tokens_per_batch` encoder patch tokens."""
+
+    V_CHOICES = (1, 2, 4, 8, 16)
+    V_PROBS = (0.45, 0.1, 0.15, 0.15, 0.15)
+
+    def __init__(self, T: int, tokens_per_batch: int, device="cpu", seed=0, patch=32, block=32, min_ctx=4, chunk=8):
+        self.gen = SynthGen(T, device, seed)
+        self.chunk = chunk  # batches generated per call (amortises kernel-launch overhead on GPU)
+        self.cache: dict[int, list] = {}
+        self.T, self.patch, self.N = T, patch, T // patch
+        self.tok = tokens_per_batch
+        self.min_ctx = min_ctx
+        self.dev = torch.device(device)
+        self.rng = torch.Generator().manual_seed(seed + 1)
+
+    def next_batch(self, V: int | None = None) -> dict:
+        if V is None:
+            i = torch.multinomial(torch.tensor(self.V_PROBS), 1, generator=self.rng).item()
+            V = self.V_CHOICES[i]
+        Bg = max(1, self.tok // (V * self.N))
+        if not self.cache.get(V):
+            vals, roles = self.gen.group_batch(Bg * self.chunk, V)
+            obs = self.gen.masks(vals, roles)
+            vals = torch.nan_to_num(vals, nan=0.0, posinf=0.0, neginf=0.0)
+            s = torch.randint(self.min_ctx, self.N - 1, (Bg * self.chunk,), generator=self.gen.g, device=self.dev)
+            self.cache[V] = [
+                {"values": vals[i * Bg:(i + 1) * Bg], "observed": obs[i * Bg:(i + 1) * Bg],
+                 "roles": roles[i * Bg:(i + 1) * Bg], "s": s[i * Bg:(i + 1) * Bg]}
+                for i in range(self.chunk)
+            ]
+        return self.cache[V].pop()

modular_ts/synth2.py

@@ -0,0 +1,206 @@
+"""Synthetic prior v2 (Modular-TS v3): new generator families + augmentation cascade.
+
+Motivated by the run2/run3 GIFT-Eval diagnosis (v3/INSIGHTS.md) and TempoPFN's
+leave-one-out prior ablation (arXiv 2510.25502, Table 3: removing the regime-switching
+OU/SDE prior costs the most, then CauKer, spikes, stochastic rhythms, step functions,
+sawtooth; the augmentation cascade is worth ~5% CRPS):
+
+New univariate families (all vectorised torch, run on the GPU):
+  * ou_regime      - Ornstein-Uhlenbeck with Markov regime switching of mean / vol / speed
+  * waveform       - non-sinusoidal periodic waves: sawtooth, square (random duty), triangle,
+                     spike trains, with optional slow period drift (phase-drifting cycles)
+  * template       - "load-curve" seasonality: a random smooth+spiky daily profile tiled at a
+                     bank period, optionally modulated by a weekly profile (24x7, 96x7, 288x7)
+                     and multiplicative trend; near-deterministic, low-noise (m4_hourly-like)
+  * steps          - piecewise-constant levels with Poisson change points (+ small noise)
+  * growth         - integrated / cumulative processes: cumsum of a positive rate (covid-like),
+                     logistic / Gompertz growth with noise, compound-interest trends
+Augmentations (applied after generation, each with its own probability):
+  time reversal, integration (cumsum), differencing, quantisation to a random grid,
+  censoring (clip at a random quantile), random smoothing conv filter, sample-and-hold
+  resampling artefacts, amplitude modulation, TSMixup of two series.
+
+`SynthGen2.univariate` returns a mixture of the v1 prior (`SynthGen.univariate`) and the new
+families; everything downstream (multivariatizers, covariates, masks) is inherited.
+"""
+
+from __future__ import annotations
+
+import math
+
+import torch
+from torch import Tensor
+
+from modular_ts.synth import PERIOD_BANK, SynthGen, SynthStream, _z
+
+
+def _fft_filter(e: Tensor, phi: Tensor, L: int) -> Tensor:
+    """y_t = sum_k phi^k e_{t-k} for each row (AR(1) filter via FFT), phi (n,)."""
+    n, T = e.shape
+    L = min(T, L)
+    k = phi.view(n, 1) ** torch.arange(L, device=e.device, dtype=torch.float32)
+    nfft = 1 << (T + L - 1).bit_length()
+    return torch.fft.irfft(torch.fft.rfft(e, nfft) * torch.fft.rfft(k, nfft), nfft)[:, :T]
+
+
+class SynthGen2(SynthGen):
+    def __init__(self, *a, p_new: float = 0.55, p_aug: float = 0.5, **k):
+        super().__init__(*a, **k)
+        self.p_new, self.p_aug = p_new, p_aug
+
+    # ------------------------------------------------------------------ families
+    def _regimes(self, n, rate_lo=1e-4, rate_hi=3e-3):
+        """Piecewise-constant regime index field (n,T) in [0,1) via Poisson change points."""
+        T = self.T
+        rate = self.logU(n, 1, lo=rate_lo, hi=rate_hi)
+        jumps = (self.U(n, T) < rate).float()
+        seg = torch.cumsum(jumps, -1).long() % 64  # segment id
+        table = self.U(n, 64)
+        return torch.gather(table, 1, seg)
+
+    def ou_regime(self, n):
+        r = self._regimes(n)
+        mu = (r - 0.5) * self.U(n, 1, lo=0.0, hi=6.0)
+        vol = torch.exp(self.U(n, 1, lo=-1.0, hi=1.0) * (self._regimes(n) - 0.5) * 2)
+        theta = self.logU(n, lo=1e-3, hi=0.3)
+        e = theta.view(n, 1) * mu + 0.3 * vol * self.N(n, self.T)
+        return _fft_filter(e, 1 - theta, 2048)
+
+    def _period(self, n):
+        idx = torch.randint(0, len(PERIOD_BANK), (n,), generator=self.g, device=self.dev)
+        per = PERIOD_BANK.to(self.dev)[idx]
+        rnd = self.logU(n, lo=3.0, hi=max(4.0, self.T / 4))
+        return torch.where(self.U(n) < 0.3, rnd, per)
+
+    def _phase(self, n, per):
+        """Cycle phase in [0,1) with optional slow period drift (phase-drifting cycles)."""
+        drift = torch.where(self.U(n) < 0.3, self.N(n) * 0.02, torch.zeros(n, device=self.dev))
+        inst = 1.0 / per.view(n, 1) * (1 + drift.view(n, 1) * torch.tanh(self.rbf(n)))
+        return torch.frac(torch.cumsum(inst.expand(n, self.T), -1) + self.U(n, 1))
+
+    def waveform(self, n):
+        per = self._period(n)
+        ph = self._phase(n, per)
+        kind = torch.randint(0, 4, (n, 1), generator=self.g, device=self.dev)
+        duty = self.U(n, 1, lo=0.1, hi=0.9)
+        saw = ph * 2 - 1
+        sq = torch.where(ph < duty, 1.0, -1.0)
+        tri = 1 - 4 * (ph - 0.5).abs()
+        spk = torch.exp(-((ph - 0.5) ** 2) / (2 * self.logU(n, 1, lo=1e-4, hi=1e-2)))
+        w = torch.where(kind == 0, saw, torch.where(kind == 1, sq, torch.where(kind == 2, tri, spk)))
+        w = torch.where(self.U(n, 1) < 0.5, -w, w)
+        return w
+
+    def template(self, n):
+        """Tiled random daily profile (smooth + bumps) x optional weekly profile."""
+        T = self.T
+        per = PERIOD_BANK.to(self.dev)[torch.randint(4, len(PERIOD_BANK), (n,), generator=self.g, device=self.dev)]
+        per = torch.where(self.U(n) < 0.6, torch.tensor([24.0, 96.0, 288.0, 48.0, 12.0, 7.0], device=self.dev)[
+            torch.randint(0, 6, (n,), generator=self.g, device=self.dev)], per)
+        ph = self._phase(n, per)  # (n,T) in [0,1)
+        # profile = sum of K random bumps on the circle + low-order Fourier terms
+        K = 4
+        c = self.U(n, K, 1)
+        wdt = self.logU(n, K, 1, lo=0.02, hi=0.25)
+        a = self.N(n, K, 1).abs() * torch.where(self.U(n, K, 1) < 0.2, -1.0, 1.0)
+        d = torch.minimum((ph[:, None] - c).abs(), 1 - (ph[:, None] - c).abs())
+        prof = (a * torch.exp(-0.5 * (d / wdt) ** 2)).sum(1)
+        h = torch.arange(1, 4, device=self.dev, dtype=torch.float32).view(1, 3, 1)
+        prof = prof + (0.3 * self.N(n, 3, 1) / h * torch.sin(2 * math.pi * h * ph[:, None] + self.U(n, 3, 1, hi=6.3))).sum(1)
+        # weekly modulation for intraday periods
+        wk = self.U(n, 7, lo=0.5, hi=1.2)
+        wk = torch.where(self.U(n, 1) < 0.5, wk, torch.ones_like(wk))
+        day = (torch.cumsum((ph[:, 1:] < ph[:, :-1]).float(), -1)).long() % 7
+        day = torch.cat([torch.zeros(n, 1, dtype=torch.long, device=self.dev), day], -1)
+        mod = torch.gather(wk, 1, day)
+        sig = prof * mod
+        # slowly varying level and tiny noise (near-deterministic series)
+        sig = sig + self.B(0.5, n, 1) * 0.3 * _z(self.trend(n)) + self.logU(n, 1, lo=1e-3, hi=0.2) * self.N(n, T)
+        return sig
+
+    def steps(self, n):
+        r = self._regimes(n, 2e-4, 1e-2)
+        lv = torch.where(self.U(n, 1) < 0.5, torch.round(r * self.U(n, 1, lo=2, hi=20)), r)
+        return lv + self.logU(n, 1, lo=1e-3, hi=0.1) * self.N(n, self.T)
+
+    def growth(self, n):
+        T = self.T
+        tt = self.t / T
+        kind = torch.randint(0, 3, (n, 1), generator=self.g, device=self.dev)
+        # cumulative counts: cumsum of positive, slowly varying rate (+ noise)
+        rate = torch.exp(0.8 * _z(self.rbf(n)).clamp(-3, 3)) * (1 + 0.3 * self.N(n, T).abs() * self.B(0.5, n, 1))
+        cum = torch.cumsum(rate, -1)
+        # logistic / gompertz
+        t0 = self.U(n, 1, lo=-0.2, hi=1.2)
+        k = self.logU(n, 1, lo=2.0, hi=30.0)
+        logi = torch.sigmoid(k * (tt - t0))
+        gomp = torch.exp(-torch.exp(-k * (tt - t0)))
+        sig = torch.where(kind == 0, cum, torch.where(kind == 1, logi, gomp))
+        sig = sig * (1 + self.logU(n, 1, lo=1e-3, hi=0.05) * self.N(n, T))
+        return sig
+
+    # ------------------------------------------------------------------ augmentations
+    def augment(self, x: Tensor) -> Tensor:
+        n, T = x.shape
+        on = lambda p: self.U(n, 1) < p * self.p_aug * 2  # noqa: E731
+        # time reversal
+        x = torch.where(on(0.15), x.flip(-1), x)
+        # integration / differencing
+        xi = torch.cumsum(x - x.mean(-1, keepdim=True), -1)
+        x = torch.where(on(0.06), xi, x)
+        xd = torch.cat([x[:, :1] * 0, x[:, 1:] - x[:, :-1]], -1)
+        x = torch.where(on(0.04), xd, x)
+        # amplitude modulation
+        x = torch.where(on(0.1), x * (1 + 0.5 * torch.tanh(self.rbf(n))), x)
+        # random smoothing conv filter (moving average of random width via cumsum trick)
+        w = torch.randint(2, 16, (n, 1), generator=self.g, device=self.dev)
+        cs = torch.cumsum(torch.nn.functional.pad(x, (1, 0)), -1)
+        idx = torch.arange(1, T + 1, device=self.dev).view(1, T)
+        lo = (idx - w).clamp_min(0)
+        ma = (cs[:, 1:] - torch.gather(cs, 1, lo)) / (idx - lo).float()
+        x = torch.where(on(0.1), ma, x)
+        # sample-and-hold resampling artefacts
+        k = torch.randint(2, 8, (n, 1), generator=self.g, device=self.dev)
+        hold = torch.gather(x, 1, (torch.arange(T, device=self.dev).view(1, T) // k) * k)
+        x = torch.where(on(0.05), hold, x)
+        # TSMixup with a rolled partner
+        lam = self.U(n, 1, lo=0.2, hi=0.8)
+        xm = lam * _z(x) + (1 - lam) * _z(x.roll(1, 0))
+        x = torch.where(on(0.1), xm, x)
+        # censoring at a random quantile (one side)
+        qv = torch.quantile(x[:, :: max(1, T // 512)], 0.8 + 0.19 * torch.rand(1, device=self.dev).item(), dim=-1, keepdim=True)
+        x = torch.where(on(0.05), torch.minimum(x, qv), x)
+        return x
+
+    def quantize(self, x: Tensor) -> Tensor:
+        """Round some series to a coarse grid (integer-valued / low-resolution sensors)."""
+        n = x.shape[0]
+        step = x.std(-1, keepdim=True).clamp_min(1e-6) * self.logU(n, 1, lo=0.02, hi=0.5)
+        q = torch.round(x / step) * step
+        return torch.where(self.U(n, 1) < 0.08, q, x)
+
+    # ------------------------------------------------------------------ mixture
+    def univariate(self, n) -> Tensor:
+        base = super().univariate(n)
+        fam = torch.randint(0, 5, (n, 1), generator=self.g, device=self.dev)
+        new = torch.zeros(n, self.T, device=self.dev)
+        gens = (self.ou_regime, self.waveform, self.template, self.steps, self.growth)
+        for i, g in enumerate(gens):
+            new = torch.where(fam == i, _z(g(n)), new)
+        # optional combination of the new family with a v1 component (additive)
+        comb = self.U(n, 1) < 0.35
+        new = torch.where(comb, new + self.logU(n, 1, lo=0.05, hi=1.0) * _z(base), new)
+        use_new = self.U(n, 1) < self.p_new
+        x = torch.where(use_new, new, base)
+        x = self.augment(x)
+        x = torch.nan_to_num(x, nan=0.0, posinf=0.0, neginf=0.0)
+        return x
+
+    def affine(self, x: Tensor) -> Tensor:
+        return self.quantize(super().affine(x))
+
+
+class SynthStream2(SynthStream):
+    def __init__(self, T, tokens_per_batch, device="cpu", seed=0, patch=32, block=32, min_ctx=4, chunk=8, p_new=0.55):
+        super().__init__(T, tokens_per_batch, device, seed, patch, block, min_ctx, chunk)
+        self.gen = SynthGen2(T, device, seed, p_new=p_new)

modular_ts/teacher_gen.py

@@ -0,0 +1,139 @@
+"""Stage B: cache teacher quantile forecasts (TimesFM-2.5-200M + Chronos-2) for distillation.
+
+Run in the teacher venv (~/tvenv, system torch):
+  ~/tvenv/bin/python -m modular_ts.teacher_gen --source synth --n 200000 --out teach/synth
+  ~/tvenv/bin/python -m modular_ts.teacher_gen --source gep   --n 100000 --out teach/gep --gep_selection gep_selection.json
+
+Each shard teach/<src>/shard_XXXX.npz holds
+  ctx  (n, C) float32   context, NaN = unobserved (left-padded for short contexts)
+  fut  (n, F) float32   ground-truth continuation (synthetic only; NaN for GEP = teacher-only input)
+  q_tfm, q_cr2 (n, H, 9) float32  teacher quantiles 0.1..0.9 for the first H future steps
+Teacher weights: google/timesfm-2.5-200m-pytorch (Apache-2.0), amazon/chronos-2 (Apache-2.0).
+GiftEvalPretrain inputs use the same GIFT-Eval/fev/TIME leakage filter as run1 (realdata.filter_datasets).
+NEVER uses Timestral weights.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import os
+import time
+
+import numpy as np
+import torch
+
+Q9 = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]
+
+
+def interp_nans(x: np.ndarray) -> np.ndarray:
+    m = np.isfinite(x)
+    if m.all():
+        return x
+    idx = np.arange(len(x))
+    return np.interp(idx, idx[m], x[m]).astype(np.float32)
+
+
+class Teachers:
+    def __init__(self, C, H, dev="cuda", tfm_bs=256, cr_bs=512):
+        import timesfm
+        from chronos import Chronos2Pipeline
+        torch.set_float32_matmul_precision("high")
+        self.H = H
+        self.cr = Chronos2Pipeline.from_pretrained("amazon/chronos-2", device_map=dev)
+        self.cr_bs = cr_bs
+        self.tfm = timesfm.TimesFM_2p5_200M_torch.from_pretrained("google/timesfm-2.5-200m-pytorch")
+        self.tfm.compile(timesfm.ForecastConfig(max_context=C, max_horizon=H, normalize_inputs=True, per_core_batch_size=tfm_bs,
+                                                use_continuous_quantile_head=True, force_flip_invariance=True,
+                                                infer_is_positive=True, fix_quantile_crossing=True))
+
+    def __call__(self, ctx: np.ndarray):
+        """ctx (n, C) with NaN -> q_tfm, q_cr2 (n, H, 9)."""
+        seqs = []
+        for row in ctx:
+            f = np.isfinite(row)
+            first = int(np.argmax(f)) if f.any() else len(row) - 1
+            seqs.append(row[first:])
+        with torch.no_grad():
+            qs, _ = self.cr.predict_quantiles([torch.from_numpy(s) for s in seqs], prediction_length=self.H,
+                                              quantile_levels=Q9, batch_size=self.cr_bs)
+        q_cr2 = torch.stack([q[0] for q in qs]).float().cpu().numpy()
+        _, qt = self.tfm.forecast(horizon=self.H, inputs=[interp_nans(s) for s in seqs])
+        q_tfm = np.asarray(qt)[:, :self.H, 1:10].astype(np.float32)
+        return q_tfm, q_cr2
+
+
+def synth_windows(gen, n, C, F, rng):
+    vals, roles = gen.group_batch(n, 1)
+    v = vals[:, 0].float().cpu().numpy()
+    T = C + F
+    # context length: 50% full C, else log-uniform in [48, C]; 15% of series get iid missing values
+    clen = np.where(rng.random(n) < 0.5, C, np.exp(rng.uniform(math.log(48), math.log(C), n))).astype(int)
+    obs = np.ones((n, T), bool)
+    obs[np.arange(T)[None] < (C - clen)[:, None]] = False
+    miss = rng.random(n) < 0.15
+    pm = rng.uniform(0, 0.1, n)[:, None]
+    obs &= ~(miss[:, None] & (rng.random((n, T)) < pm))
+    v = np.where(obs, v, np.nan).astype(np.float32)
+    return v[:, :C], v[:, C:]
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--source", choices=["synth", "gep"], required=True)
+    ap.add_argument("--n", type=int, default=100000)
+    ap.add_argument("--shard", type=int, default=8192)
+    ap.add_argument("--C", type=int, default=2048)
+    ap.add_argument("--F", type=int, default=1024)
+    ap.add_argument("--H", type=int, default=512)
+    ap.add_argument("--out", required=True)
+    ap.add_argument("--seed", type=int, default=123)
+    ap.add_argument("--gep_selection", default="gep_selection.json")
+    ap.add_argument("--max_minutes", type=float, default=0)
+    args = ap.parse_args()
+    os.makedirs(args.out, exist_ok=True)
+    dev = torch.device("cuda")
+    rng = np.random.default_rng(args.seed)
+    T = args.C + args.F
+    if args.source == "synth":
+        from modular_ts.synth import SynthGen
+        gen = SynthGen(T, dev, seed=args.seed)
+    else:
+        from modular_ts.realdata import FastGiftEvalPretrain
+        gep = FastGiftEvalPretrain(T, args.shard * 2 * (T // 32), dev, json.load(open(args.gep_selection)), seed=args.seed,
+                                   mv_frac=0.0, short_ctx_p=0.3)
+        json.dump(gep.manifest(), open(os.path.join(args.out, "gep_manifest.json"), "w"), indent=1)
+    tea = Teachers(args.C, args.H)
+    t0, done, k = time.time(), 0, 0
+    while done < args.n:
+        if args.max_minutes and (time.time() - t0) / 60 > args.max_minutes:
+            print("time limit", flush=True)
+            break
+        p = os.path.join(args.out, f"shard_{k:04d}.npz")
+        if os.path.exists(p):
+            done += args.shard; k += 1
+            continue
+        if args.source == "synth":
+            gen = SynthGen(T, dev, seed=args.seed * 1000 + k)  # per-shard seed -> resumable without duplicates
+            rng = np.random.default_rng(args.seed * 1000 + k)
+            ctx, fut = synth_windows(gen, args.shard, args.C, args.F, rng)
+        else:
+            gep.g.manual_seed(args.seed * 1000 + k); gep.rng.seed(args.seed * 1000 + k)
+            b = gep.next_batch()
+            v = torch.where(b["observed"][:, 0], b["values"][:, 0], torch.full_like(b["values"][:, 0], float("nan"))).cpu().numpy()
+            ok = np.isfinite(v[:, :args.C]).sum(1) >= 48
+            v = v[ok][:args.shard]
+            ctx, fut = v[:, :args.C], np.full((len(v), args.F), np.nan, np.float32)
+        ts = time.time()
+        q_tfm, q_cr2 = tea(ctx)
+        good = np.isfinite(q_tfm).all((1, 2)) & np.isfinite(q_cr2).all((1, 2))
+        np.savez(p, ctx=ctx[good], fut=fut[good], q_tfm=q_tfm[good].astype(np.float32), q_cr2=q_cr2[good].astype(np.float32))
+        done += int(good.sum()); k += 1
+        print(json.dumps({"shard": k, "n": int(good.sum()), "dropped": int((~good).sum()), "teach_s": round(time.time() - ts, 1),
+                          "done": done, "min": round((time.time() - t0) / 60, 2)}), flush=True)
+    print("TEACH_DONE", done, flush=True)
+
+
+if __name__ == "__main__":
+    main()

modular_ts/train.py

@@ -0,0 +1,225 @@
+"""Train Modular-TS from scratch with DHF + pinball loss.
+
+python -m modular_ts.train --size 2.5M --ctx 4096 --tokens_per_batch 65536 --steps 20000 \
+    --lr 2e-3 --warmup 1000 --out runs/smoke --max_minutes 35 [--compile] [--data synthetic]
+
+Logs JSONL to <out>/log.jsonl (step, loss, val, lr, enc tokens/s, grad norm),
+checkpoints to <out>/ckpt.pt and an inference export (config.json + model.safetensors,
+loadable by timestral.TimestralForecaster) to <out>/export/.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import os
+import time
+
+import torch
+
+from modular_ts.model import ModularTS, count_params, make_config
+from modular_ts.synth import SynthStream
+
+
+def export(model: ModularTS, out_dir: str) -> None:
+    from safetensors.torch import save_file
+    os.makedirs(out_dir, exist_ok=True)
+    sd = {k: v.detach().float().cpu().contiguous() for k, v in model.state_dict().items()}
+    save_file(sd, os.path.join(out_dir, "model.safetensors"))
+    model.config.save(os.path.join(out_dir, "config.json"))
+
+
+def lr_at(step, total, warmup, peak, final_frac=0.05):
+    if step < warmup:
+        return peak * (step + 1) / warmup
+    prog = min(1.0, (step - warmup) / max(1, total - warmup))
+    return peak * (final_frac + (1 - final_frac) * 0.5 * (1 + math.cos(math.pi * prog)))
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--size", default="2.5M")
+    ap.add_argument("--ctx", type=int, default=4096)
+    ap.add_argument("--tokens_per_batch", type=int, default=65536, help="encoder patch tokens per step")
+    ap.add_argument("--steps", type=int, default=20000, help="schedule length (cosine horizon)")
+    ap.add_argument("--max_minutes", type=float, default=0, help="stop early after this wall time (0 = off)")
+    ap.add_argument("--lr", type=float, default=2e-3)
+    ap.add_argument("--warmup", type=int, default=1000)
+    ap.add_argument("--wd", type=float, default=0.05)
+    ap.add_argument("--clip", type=float, default=1.0)
+    ap.add_argument("--data", default="synthetic", choices=["synthetic", "synthetic+gifteval"])
+    ap.add_argument("--real_frac", type=float, default=0.5)
+    ap.add_argument("--gifteval_subset", default="", help="comma list of GiftEvalPretrain sub-datasets (default all kept)")
+    ap.add_argument("--gep_selection", default="", help="json {dataset: [shard files]} -> fast GPU-resident GiftEvalPretrain stream")
+    ap.add_argument("--synth", default="v1", choices=["v1", "v2"], help="v2 = modular_ts.synth2 prior (v3)")
+    ap.add_argument("--p_new", type=float, default=0.55, help="synth v2: share of new generator families")
+    ap.add_argument("--real_v2", action="store_true", help="v3 real stream: resolution sampling + TSMixup + ds_temp 0.3")
+    ap.add_argument("--min_lr_frac", type=float, default=0.05)
+    ap.add_argument("--time_schedule", action="store_true", help="cosine progress = elapsed / max_minutes (smoke runs)")
+    ap.add_argument("--compile", action="store_true")
+    ap.add_argument("--bf16", action="store_true", default=True)
+    ap.add_argument("--no_bf16", dest="bf16", action="store_false")
+    ap.add_argument("--log_every", type=int, default=25)
+    ap.add_argument("--val_every", type=int, default=250)
+    ap.add_argument("--ckpt_every", type=int, default=1000)
+    ap.add_argument("--out", default="runs/dev")
+    ap.add_argument("--seed", type=int, default=0)
+    ap.add_argument("--resume", action="store_true")
+    ap.add_argument("--resume_from", default="", help="ckpt to resume model+opt from (default <out>/ckpt.pt)")
+    ap.add_argument("--reset_schedule", action="store_true", help="on resume, restart warmup+cosine at the resumed step")
+    ap.add_argument("--device", default="cuda" if torch.cuda.is_available() else "cpu")
+    args = ap.parse_args()
+
+    os.makedirs(args.out, exist_ok=True)
+    torch.manual_seed(args.seed)
+    dev = torch.device(args.device)
+    if dev.type == "cuda":
+        torch.backends.cuda.matmul.allow_tf32 = True
+        torch.backends.cudnn.allow_tf32 = True
+
+    cfg = make_config(args.size, context_length=args.ctx)
+    model = ModularTS(cfg).to(dev)
+    pc = count_params(model)
+    print("params", pc, flush=True)
+
+    if args.synth == "v2":
+        from modular_ts.synth2 import SynthStream2
+        synth = SynthStream2(args.ctx, args.tokens_per_batch, dev, seed=args.seed, p_new=args.p_new)
+    else:
+        synth = SynthStream(args.ctx, args.tokens_per_batch, dev, seed=args.seed)
+    stream = synth
+    manifest = {"data": args.data, "synth_version": args.synth, "real_v2": args.real_v2, "synthetic": "modular_ts.synth (KernelSynth-style GP + TSI + AR + intermittent/bursty + multivariatizers + covariate tasks)"}
+    if args.data == "synthetic+gifteval":
+        from modular_ts.realdata import GiftEvalPretrainStream, MixedStream
+        sub = [s for s in args.gifteval_subset.split(",") if s] or None
+        if args.gep_selection:
+            from modular_ts.realdata import FastGiftEvalPretrain
+            if args.real_v2:
+                from modular_ts.realdata2 import FastGiftEvalPretrain2 as FastGiftEvalPretrain
+            real = FastGiftEvalPretrain(args.ctx, args.tokens_per_batch, dev, json.load(open(args.gep_selection)), seed=args.seed)
+        else:
+            real = GiftEvalPretrainStream(args.ctx, args.tokens_per_batch, dev, seed=args.seed, datasets_subset=sub)
+        stream = MixedStream(synth, real, args.real_frac, seed=args.seed)
+        manifest.update(real.manifest())
+    json.dump(manifest, open(os.path.join(args.out, "data_manifest.json"), "w"), indent=1)
+
+    # fixed held-out validation batches from a different seed (one per V bucket)
+    vgen = SynthStream(args.ctx, args.tokens_per_batch // 2, dev, seed=10_000 + args.seed, chunk=1)
+    val_batches = [vgen.next_batch(V) for V in (1, 4, 16)]
+
+    decay, no_decay = [], []
+    for n, p in model.named_parameters():
+        (decay if p.dim() >= 2 else no_decay).append(p)
+    opt = torch.optim.AdamW([{"params": decay, "weight_decay": args.wd}, {"params": no_decay, "weight_decay": 0.0}],
+                            lr=args.lr, betas=(0.9, 0.95), eps=1e-8, fused=dev.type == "cuda")
+    step = 0
+    ck = os.path.join(args.out, "ckpt.pt")
+    rck = args.resume_from or ck
+    step0 = 0
+    prev_elapsed = 0.0
+    if args.resume and os.path.exists(rck):
+        st = torch.load(rck, map_location=dev)
+        model.load_state_dict(st["model"])
+        opt.load_state_dict(st["opt"])
+        step = st["step"]
+        if args.reset_schedule:
+            step0 = step
+        elif args.time_schedule and "elapsed_min" in st:
+            prev_elapsed = st["elapsed_min"]  # continue the same time-based schedule after a crash
+        print("resumed at", step, "from", rck, flush=True)
+
+    fwd = model.forward_train
+    if args.compile:
+        try:
+            fwd = torch.compile(model.forward_train, dynamic=False)
+        except Exception as e:  # pragma: no cover
+            print("compile failed, eager:", e, flush=True)
+
+    amp = torch.autocast(device_type=dev.type, dtype=torch.bfloat16, enabled=args.bf16 and dev.type == "cuda")
+
+    def evaluate():
+        model.eval()
+        tot = 0.0
+        with torch.no_grad(), amp:
+            for b in val_batches:
+                l, _ = model.forward_train(b["values"], b["observed"], b["roles"], b["s"])
+                tot += l.item()
+        model.train()
+        return tot / len(val_batches)
+
+    log = open(os.path.join(args.out, "log.jsonl"), "a")
+    t0 = time.time() - prev_elapsed * 60
+    t_last, tok_acc, gen_t = time.time(), 0, 0.0
+    ema = None
+    model.train()
+    while step < args.steps:
+        if args.max_minutes and (time.time() - t0) / 60 > args.max_minutes:
+            print("time limit reached", flush=True)
+            break
+        prof = step % args.log_every == 0  # measure data-gen share only on log steps (sync is costly)
+        if prof and dev.type == "cuda":
+            torch.cuda.synchronize()
+        tg = time.time()
+        b = stream.next_batch()
+        if prof and dev.type == "cuda":
+            torch.cuda.synchronize()
+        if prof:
+            gen_t += (time.time() - tg) * args.log_every
+        ss = step - step0
+        if args.time_schedule and args.max_minutes:
+            frac = (time.time() - t0) / 60 / args.max_minutes
+            lr = lr_at(ss, max(ss + 1, int(ss / max(frac, 1e-6))) if ss > args.warmup else args.steps, args.warmup, args.lr, args.min_lr_frac)
+        else:
+            lr = lr_at(ss, args.steps, args.warmup, args.lr, args.min_lr_frac)
+        for g in opt.param_groups:
+            g["lr"] = lr
+        try:
+            with amp:
+                loss, st = fwd(b["values"], b["observed"], b["roles"], b["s"])
+        except Exception as e:
+            if fwd is model.forward_train:
+                raise
+            print("compiled forward failed, falling back to eager:", repr(e)[:500], flush=True)
+            fwd = model.forward_train
+            with amp:
+                loss, st = fwd(b["values"], b["observed"], b["roles"], b["s"])
+        if not torch.isfinite(loss):
+            print(f"non-finite loss at step {step}; skipping batch", flush=True)
+            opt.zero_grad(set_to_none=True)
+            step += 1
+            continue
+        loss.backward()
+        gn = torch.nn.utils.clip_grad_norm_(model.parameters(), args.clip)
+        opt.step()
+        opt.zero_grad(set_to_none=True)
+        tok_acc += b["values"].shape[0] * b["values"].shape[1] * (args.ctx // cfg.patch_size)
+        step += 1
+        lv = loss.item() if step % args.log_every == 0 or step == 1 else None
+        if lv is not None:
+            ema = lv if ema is None else 0.9 * ema + 0.1 * lv
+            if dev.type == "cuda":
+                torch.cuda.synchronize()
+            dt = time.time() - t_last
+            rec = {"step": step, "loss": lv, "loss_ema": ema, "lr": lr, "gnorm": float(gn),
+                   "enc_tok_s": tok_acc / dt, "gen_frac": gen_t / dt, "elapsed_min": (time.time() - t0) / 60}
+            if step % args.val_every == 0:
+                rec["val"] = evaluate()
+            if dev.type == "cuda":
+                rec["mem_gb"] = torch.cuda.max_memory_allocated() / 1e9
+            print(json.dumps(rec), flush=True)
+            log.write(json.dumps(rec) + "\n")
+            log.flush()
+            t_last, tok_acc, gen_t = time.time(), 0, 0.0
+        if step % args.ckpt_every == 0:
+            torch.save({"model": model.state_dict(), "opt": opt.state_dict(), "step": step, "args": vars(args), "elapsed_min": (time.time() - t0) / 60}, ck)
+            export(model, os.path.join(args.out, "export"))
+    torch.save({"model": model.state_dict(), "opt": opt.state_dict(), "step": step, "args": vars(args), "elapsed_min": (time.time() - t0) / 60}, ck)
+    export(model, os.path.join(args.out, "export"))
+    fin = {"final_step": step, "val": evaluate(), "params": pc, "minutes": (time.time() - t0) / 60}
+    print("FINAL", json.dumps(fin), flush=True)
+    json.dump(fin, open(os.path.join(args.out, "final.json"), "w"))
+
+
+if __name__ == "__main__":
+    main()

modular_ts/tta.py

@@ -0,0 +1,147 @@
+"""Test-time ensembling for Modular-TS (v3): context-length ensembles, temporal-hierarchy
+(multi-resolution) reconciliation and in-context backtest weighting. Inference only.
+
+Members (spec string, comma separated):
+  cN   - the base forecaster with max_context N (e.g. c8192, c2048, c512)
+  dK   - downsampled member: context mean-pooled by K (aligned to the forecast origin, so it
+         sees up to K x the history at the same token budget), forecast ceil(H/K) coarse
+         steps, step-upsampled back. Used as a *level corrector* for the fine forecasts
+         (temporal hierarchies / THieF, Athanasopoulos et al. 2017): fine + a*(coarse - pool(fine))
+         instead of being averaged in directly, so intra-period shape comes from full res.
+Combination of the c-members: quantile averaging (Vincentisation), uniform or, with
+backtest=True, per-series softmax weights from each member's scaled MAE on the last H
+observed steps of the context (a zero-shot, in-context model selection - it only uses the
+series' own history).
+
+    fc = EnsembleForecaster(TimestralForecaster.from_pretrained(exp), "c8192,c2048,d4", alpha=0.5)
+"""
+
+from __future__ import annotations
+
+import math
+from dataclasses import dataclass
+
+import numpy as np
+import torch
+
+
+@dataclass
+class _F:
+    quantiles: torch.Tensor
+
+
+def _pool(x: np.ndarray | None, k: int) -> np.ndarray | None:
+    """(B,V,T) NaN-aware mean pool by k aligned to the end."""
+    if x is None:
+        return None
+    B, V, T = x.shape
+    n = T // k
+    if n == 0:
+        return None
+    y = x[..., T - n * k:].reshape(B, V, n, k)
+    with np.errstate(all="ignore"):
+        cnt = np.isfinite(y).sum(-1)
+        s = np.nansum(y, -1)
+        out = np.where(cnt > 0, s / np.maximum(cnt, 1), np.nan)
+    return out.astype(np.float32)
+
+
+class EnsembleForecaster:
+    def __init__(self, fc, spec: str = "c8192", alpha: float = 0.5, backtest: bool = False, tau: float = 0.5, bt_windows: int = 1, prior: float = 0.0,
+                 min_coarse_ctx: int = 256, min_extra: float = 1.5, min_ratio: float = 8.0):
+        self.fc = fc
+        self.members = [m.strip() for m in spec.split(",") if m.strip()]
+        cl = fc.model.config.context_length  # never exceed the trained context (RoPE does not extrapolate)
+        self.ctxs = [min(int(m[1:]), cl) for m in self.members if m[0] == "c"] or [fc.max_context]
+        self.ks = [int(m[1:]) for m in self.members if m[0] == "d"]
+        self.alpha, self.backtest, self.tau = alpha, backtest, tau
+        self.bt_windows, self.prior = bt_windows, prior  # prior: logit bonus for the first (longest) member
+        self.min_coarse_ctx, self.min_extra, self.min_ratio = min_coarse_ctx, min_extra, min_ratio  # member ctx >= min_ratio*H
+        self.base_ctx = fc.max_context
+        self.max_context = max(self.ctxs + [self.base_ctx * k for k in self.ks] + [self.base_ctx])
+        self.quantile_levels = fc.quantile_levels
+        self.mi = list(fc.quantile_levels).index(0.5) if 0.5 in list(fc.quantile_levels) else len(fc.quantile_levels) // 2
+
+    def _one(self, ctx, h, past, mc):
+        self.fc.max_context = mc
+        try:
+            return self.fc.forecast(ctx, h, past_covariates=past).quantiles.float().cpu()
+        finally:
+            self.fc.max_context = self.base_ctx
+
+    def _members(self, ctx, h, past):
+        T = ctx.shape[-1]
+        outs, used = [], set()
+        for i, c in enumerate(self.ctxs):
+            if i > 0 and c < self.min_ratio * h:  # a short context cannot support a long horizon
+                continue
+            eff = min(c, T)
+            if eff in used:  # a shorter series makes several members identical
+                continue
+            used.add(eff)
+            outs.append(self._one(ctx[..., -c:], h, None if past is None else past[..., -c:], c))
+        return outs
+
+    def forecast(self, context, prediction_length, past_covariates=None):
+        ctx = np.asarray(context, np.float32)
+        if ctx.ndim == 2:
+            ctx = ctx[:, None]
+        past = None if past_covariates is None else np.asarray(past_covariates, np.float32)
+        h = prediction_length
+        T = ctx.shape[-1]
+        outs = self._members(ctx, h, past)
+        if len(outs) == 1:
+            q = outs[0]
+        elif self.backtest and T - h >= 2 * 32:
+            E, ok = 0.0, True
+            nw = 0
+            for wdw in range(1, self.bt_windows + 1):
+                if T - wdw * h < 2 * 32:
+                    break
+                cut = T - wdw * h
+                bt = self._members(ctx[..., :cut], h, None if past is None else past[..., :cut])
+                if len(bt) != len(outs):
+                    ok = False
+                    break
+                y = torch.from_numpy(ctx[..., cut:cut + h])
+                hist = torch.from_numpy(ctx[..., :cut])
+                dd = hist[..., 1:] - hist[..., :-1]
+                fin = torch.isfinite(dd)
+                scale = (torch.where(fin, dd.abs(), torch.zeros_like(dd)).sum(-1) / fin.sum(-1).clamp_min(1)).clamp_min(1e-8)
+                errs = []
+                for b in bt:
+                    e = (b[..., self.mi] - y).abs()
+                    fe = torch.isfinite(e)
+                    e = torch.where(fe, e, torch.zeros_like(e)).sum(-1) / fe.sum(-1).clamp_min(1) / scale
+                    errs.append(e.mean(-1))
+                E = E + torch.stack(errs, 0)
+                nw += 1
+            if not ok or nw == 0:
+                q = torch.stack(outs).mean(0)
+            else:
+                E = E / nw
+                logit = -E / (self.tau * E.min(0, keepdim=True).values.clamp_min(1e-3))
+                logit[0] = logit[0] + self.prior
+                w = torch.softmax(logit, 0)
+                q = (torch.stack(outs) * w[:, :, None, None, None]).sum(0)
+        else:
+            q = torch.stack(outs).mean(0)
+        # temporal-hierarchy level correction from downsampled members
+        for k in self.ks:
+            hk = math.ceil(h / k)
+            pooled = _pool(ctx, k)
+            if pooled is None or pooled.shape[-1] < self.min_coarse_ctx:
+                continue
+            # only worth it when the coarse view sees materially more history than the fine one
+            if T < self.min_extra * self.base_ctx and hk < 2:
+                continue
+            pp = _pool(past, k) if past is not None else None
+            cq = self._one(pooled[..., -self.base_ctx:], hk, None if pp is None else pp[..., -self.base_ctx:], self.base_ctx)
+            cmed = cq[..., self.mi]  # (B,V,hk)
+            fmed = q[..., self.mi]
+            pad = hk * k - h
+            fm = torch.cat([fmed, fmed[..., -1:].expand(*fmed.shape[:-1], pad)], -1) if pad else fmed
+            fpool = fm.view(*fm.shape[:-1], hk, k).mean(-1)
+            delta = (cmed - fpool).repeat_interleave(k, -1)[..., :h]
+            q = q + self.alpha / len(self.ks) * delta[..., None]
+        return _F(q)

results/all_results.csv

@@ -0,0 +1,98 @@
+dataset,model,eval_metrics/MSE[mean],eval_metrics/MSE[0.5],eval_metrics/MAE[0.5],eval_metrics/MASE[0.5],eval_metrics/MAPE[0.5],eval_metrics/sMAPE[0.5],eval_metrics/MSIS,eval_metrics/RMSE[mean],eval_metrics/NRMSE[mean],eval_metrics/ND[0.5],eval_metrics/mean_weighted_sum_quantile_loss,domain,num_variates
+bitbrains_fast_storage/5T/long,Modular-TS-2.4M-v3_M1,3678060.0939310347,3678060.0939310347,375.4961381185165,0.9167681594477102,4.714564867919659,0.7251918487783172,15.841790583442892,1917.826919701315,5.068243351659421,0.9923240653485829,0.6959523690328288,Web/CloudOps,2
+bitbrains_fast_storage/5T/medium,Modular-TS-2.4M-v3_M1,2794097.8588460106,2794097.8588460106,301.5700941306518,1.248188037721358,4.705945084704349,0.7298914125066623,23.65806409550792,1671.5555207189532,5.078647577538354,0.9162532796732381,0.674956798323308,Web/CloudOps,2
+bitbrains_fast_storage/5T/short,Modular-TS-2.4M-v3_M1,1789367.908427151,1789367.908427151,170.94836712944672,0.7464365081982969,2.2107092294123354,0.6847539590206893,14.157463028441592,1337.6725714565396,4.199961010964384,0.5367355899732063,0.4188247283434475,Web/CloudOps,2
+bitbrains_fast_storage/H/short,Modular-TS-2.4M-v3_M1,2857251.909621366,2857251.909621366,324.96432151762076,1.0518938446957118,3.765954716264035,0.5135839053460005,19.937172636928338,1690.340767307399,4.818020494788251,0.9262539195814864,0.6779741560340291,Web/CloudOps,2
+bitbrains_rnd/5T/long,Modular-TS-2.4M-v3_M1,2335002.1897918363,2335002.1897918363,202.03773498891655,3.340738878079062,2.732445501293894,0.6418363842517235,116.04497928950197,1528.0713955152214,5.853697246156131,0.7739610442256082,0.6664847615792028,Web/CloudOps,2
+bitbrains_rnd/5T/medium,Modular-TS-2.4M-v3_M1,2197458.6406868207,2197458.6406868207,166.13444570183873,4.413797604303608,1.2794170885289742,0.6578795845955763,160.08923169435977,1482.3827578216162,6.237140002367876,0.699012310815642,0.6454088859518597,Web/CloudOps,2
+bitbrains_rnd/5T/short,Modular-TS-2.4M-v3_M1,1717739.5983574833,1717739.5983574833,129.86860276201207,1.7188269009296444,1.114256719613909,0.6338544360460594,53.535910841778204,1310.6256514953013,5.362701787992878,0.5313848294067788,0.4247043445566765,Web/CloudOps,2
+bitbrains_rnd/H/short,Modular-TS-2.4M-v3_M1,1760008.1728826563,1760008.1728826563,156.42778973615788,5.9169358495513436,1.6377402633165954,0.5359674461057023,204.34510912470066,1326.6529964096326,5.806082855626521,0.684604573003369,0.6252948443035078,Web/CloudOps,2
+bizitobs_application/10S/long,Modular-TS-2.4M-v3_M1,5275426.702222222,5275426.702222222,1418.387638888889,3.0462642085000406,0.04374827490912543,0.044643965827094186,34.158310869955976,2296.8297068398915,0.08857954110209905,0.054701541774522924,0.04655703725886792,Web/CloudOps,2
+bizitobs_application/10S/medium,Modular-TS-2.4M-v3_M1,3028799.36,3028799.36,970.73,2.2233368341100945,0.031767988204956056,0.03231649398803711,19.42404117346401,1740.344609552947,0.06833821067690701,0.0381177100709006,0.03148191823098797,Web/CloudOps,2
+bizitobs_application/10S/short,Modular-TS-2.4M-v3_M1,629297.9911111111,629297.9911111111,429.53465277777775,1.232171827879832,0.017635783089531794,0.01781905280219184,8.829611381439575,793.2830460252577,0.03059375624223889,0.016565434658564034,0.012766240040952294,Web/CloudOps,2
+bizitobs_l2c/5T/long,Modular-TS-2.4M-v3_M1,92.45116071428572,92.45116071428572,6.359086061507936,0.6284250899642787,0.5096323568723897,0.7791392328354297,3.653899407257271,9.615152662037442,0.5398764472943504,0.35705317550293086,0.2669678027002204,Web/CloudOps,7
+bizitobs_l2c/5T/medium,Modular-TS-2.4M-v3_M1,80.05903486394558,80.05903486394558,5.660879703443878,0.5515346011572464,0.38841309866420504,0.6572395147049294,3.432537186556386,8.947571450619748,0.47077383803909884,0.297845519242355,0.228644277283759,Web/CloudOps,7
+bizitobs_l2c/5T/short,Modular-TS-2.4M-v3_M1,19.891878255208333,19.891878255208333,2.5970302036830355,0.26936388067215716,0.14014765634067433,0.16574777941924232,2.293561869201875,4.460031194420991,0.15317552406028445,0.08919252917942418,0.07143140322408012,Web/CloudOps,7
+bizitobs_l2c/H/long,Modular-TS-2.4M-v3_M1,80.8273685515873,80.8273685515873,5.249859716021826,0.5630390129701736,0.6261864244101014,0.7163555186958651,5.3501780426093,8.990404248507812,0.5491598394034193,0.32067658348283234,0.26036159148559485,Web/CloudOps,7
+bizitobs_l2c/H/medium,Modular-TS-2.4M-v3_M1,75.64733072916667,75.64733072916667,5.013981701078869,0.5068527865795754,0.5128123874838597,0.6639396817558852,3.3392014091576043,8.697547397350972,0.5266497057254618,0.3036042078011892,0.23915307887691264,Web/CloudOps,7
+bizitobs_l2c/H/short,Modular-TS-2.4M-v3_M1,57.78172278025794,57.78172278025794,4.74778578016493,0.47055571124351386,0.441387441804369,0.5357121708376895,2.600709930494657,7.601428995936089,0.40973653327113746,0.255917839569219,0.19489947674121633,Web/CloudOps,7
+bizitobs_service/10S/long,Modular-TS-2.4M-v3_M1,130062.9671957672,130062.9671957672,75.6047619047619,1.3220944818431355,0.08974954917948082,0.09647402162905093,17.90537812216811,360.64243676495863,0.2671754466043876,0.0560104246425329,0.051381319986138645,Web/CloudOps,2
+bizitobs_service/10S/medium,Modular-TS-2.4M-v3_M1,37549.83365079365,37549.83365079365,44.102624007936505,1.0659872190913706,0.07613135443793403,0.07608071947854662,8.761180060093817,193.77779452453692,0.14584171713131502,0.03319267014621623,0.02467642624887996,Web/CloudOps,2
+bizitobs_service/10S/short,Modular-TS-2.4M-v3_M1,3464.6414814814816,3464.6414814814816,21.6134126984127,0.8154586392523805,0.07128501932457011,0.05466276300016534,6.416263373524299,58.861205233001144,0.04360626258204015,0.01601190709042046,0.012609605005872552,Web/CloudOps,2
+car_parts/M/short,Modular-TS-2.4M-v3_M1,1.305600452790527,1.305600452790527,0.5138089770543491,0.9264228697044318,0.7229986347697147,1.815765007228728,13.205239812801091,1.1426287467023255,2.7399065232330053,1.232061220225577,1.0199670264999154,Sales,1
+covid_deaths/D/short,Modular-TS-2.4M-v3_M1,390569.1909774436,390569.1909774436,124.35499686716793,36.53481413039805,0.10018785763625955,0.1363243285045092,572.0192411858611,624.9553511871417,0.2351424922666607,0.04678917275228266,0.03960264190873637,Healthcare,1
+electricity/15T/long,Modular-TS-2.4M-v3_M1,387027.38936486485,387027.38936486485,61.74539771021021,0.8999086727862419,0.13413104448677174,0.12631141205794552,7.4110272826156764,622.1152540846952,0.9814631068825673,0.0974109370803379,0.07878154973566881,Energy,1
+electricity/15T/medium,Modular-TS-2.4M-v3_M1,324140.55405292794,324140.55405292794,57.261868524774776,0.8602740891433444,0.12796559968104762,0.1250541946200342,7.40904239422722,569.3334295937029,0.9810188500772348,0.0986679676504098,0.08014002920089768,Energy,1
+electricity/15T/short,Modular-TS-2.4M-v3_M1,159011.70030686937,159011.70030686937,58.58624731665259,1.0991763686633387,0.17856490707210804,0.16560820949532176,12.54439727462825,398.76271178091537,0.7860399099717378,0.11548504212115139,0.09610354583362528,Energy,1
+electricity/D/short,Modular-TS-2.4M-v3_M1,1324453454.931027,1324453454.931027,4420.498882882883,1.4642161417491881,0.5259458230883358,0.0996492496888516,11.876414361582455,36393.04129817989,0.5980585584955973,0.07264348060574302,0.05708177686933791,Energy,1
+electricity/H/long,Modular-TS-2.4M-v3_M1,7372469.746162162,7372469.746162162,268.9585945945946,1.1839789805436962,0.294873560923189,0.13517404831015878,9.636234992963349,2715.229225343997,1.0708856186022901,0.10607719166485649,0.0846881171621528,Energy,1
+electricity/H/medium,Modular-TS-2.4M-v3_M1,5435094.20972973,5435094.20972973,246.52309403153154,1.1023261199038497,0.24864662046620367,0.1266396394450642,8.89635907075197,2331.328850619262,0.9099944376457132,0.09622608336027569,0.07738372993314419,Energy,1
+electricity/H/short,Modular-TS-2.4M-v3_M1,1876482.3957207208,1876482.3957207208,198.68760152730857,1.048592430136885,0.2125099229835893,0.12904541010787124,10.67673108397427,1369.847581200449,0.6466658874640627,0.09379473740952059,0.07583874279304725,Energy,1
+electricity/W/short,Modular-TS-2.4M-v3_M1,63847587411.94955,63847587411.94955,35648.821621621624,1.8002616835096779,0.26463992427529304,0.107129864744331,12.682172214599442,252680.80143127128,0.5765342039285258,0.08133884679086692,0.06238429748467943,Energy,1
+ett1/15T/long,Modular-TS-2.4M-v3_M1,8.64753410218254,8.64753410218254,1.6084292844742063,1.0242327428538225,0.7725774866299528,0.3994036380828373,7.362441077699103,2.940668988883744,0.560494416846029,0.3065682119433832,0.23851112316934234,Energy,7
+ett1/15T/medium,Modular-TS-2.4M-v3_M1,8.946388268849207,8.946388268849207,1.612916976686508,1.0142896203616822,0.7179076932443985,0.4008093067956349,6.9522545201894985,2.99105136513053,0.5700972801996723,0.3074235341848529,0.23753058892474854,Energy,7
+ett1/15T/short,Modular-TS-2.4M-v3_M1,4.493948800223214,4.493948800223214,1.0467429024832589,0.6744832518361066,0.4831348848598496,0.23317115420386905,4.85612060712067,2.1198935822873786,0.40074808391760935,0.19787795766233757,0.15605220939196554,Energy,7
+ett1/D/short,Modular-TS-2.4M-v3_M1,41473.67619047619,41473.67619047619,136.99885912698412,1.6938340222853872,1.5330467587425596,0.47015250069754466,11.039966279309825,203.65086837643534,0.5348957609590255,0.3598320477956078,0.27754000750616026,Energy,7
+ett1/H/long,Modular-TS-2.4M-v3_M1,190.66746031746032,190.66746031746032,8.587733548280424,1.5527086794374731,5316620674772.501,0.5386861488301918,8.817916161074745,13.808238856474794,0.6612466522675325,0.41124795988760454,0.316052518207568,Energy,7
+ett1/H/medium,Modular-TS-2.4M-v3_M1,162.93642113095237,162.93642113095237,7.93248523530506,1.426481905997391,4927664332637.245,0.4986232212611607,8.261870012206636,12.764655151274255,0.6113967232377264,0.37994723888118587,0.2912179401657243,Energy,7
+ett1/H/short,Modular-TS-2.4M-v3_M1,101.40430617559524,101.40430617559524,5.343412853422619,0.8934073410569346,0.5035806565057663,0.2850081307547433,5.960154567403204,10.06997051513038,0.4700067218968723,0.24939893867666016,0.1950639093326618,Energy,7
+ett1/W/short,Modular-TS-2.4M-v3_M1,1106313.857142857,1106313.857142857,817.4416155133929,1.4995751755326385,0.6967250960213798,0.5054152693067279,9.828438170708507,1051.8145545403224,0.41854211569731004,0.3252795293040285,0.25851552338781847,Energy,7
+ett2/15T/long,Modular-TS-2.4M-v3_M1,12.21656746031746,12.21656746031746,2.289798332093254,0.9465275728595461,0.14278422674529917,0.17554123651413692,7.384640025056532,3.4952206597463142,0.18626662345647246,0.1220274898885009,0.09823390820536648,Energy,7
+ett2/15T/medium,Modular-TS-2.4M-v3_M1,11.67773933531746,11.67773933531746,2.1916136532738095,0.9034327604282267,0.1354037787471785,0.166792728484623,6.875634428994166,3.417270743637012,0.1821125316019545,0.11679505097284643,0.09285252044934728,Energy,7
+ett2/15T/short,Modular-TS-2.4M-v3_M1,7.374314081101191,7.374314081101191,1.7099256243024554,0.7340474423392392,0.10527994779395827,0.12591985974993025,5.222691349888079,2.7155688319578997,0.1289177129693672,0.08117625237061474,0.06418560197946685,Energy,7
+ett2/D/short,Modular-TS-2.4M-v3_M1,105054.83174603175,105054.83174603175,196.08004712301587,1.2984840916817926,0.4009605286613343,0.13548792278955854,12.586524599283038,324.12163109862286,0.18550334007335306,0.11222177162249401,0.09302339248607955,Energy,7
+ett2/H/long,Modular-TS-2.4M-v3_M1,250.87258597883599,250.87258597883599,9.74873511904762,1.0779024421450007,0.1888666196931598,0.17896481468563988,7.821690076035867,15.83895785646379,0.21690233728199415,0.13350142427467618,0.10684645595914224,Energy,7
+ett2/H/medium,Modular-TS-2.4M-v3_M1,232.85926339285714,232.85926339285714,9.49784923735119,1.0546525783918868,0.184112399827706,0.17202008565266927,7.232294896354205,15.259726845289766,0.20563461951715362,0.12798961829102862,0.10264518121230139,Energy,7
+ett2/H/short,Modular-TS-2.4M-v3_M1,110.36870349702382,110.36870349702382,6.621943591889881,0.7615127738607653,0.12385460601547361,0.1129866463797433,5.21876475954453,10.505651026805708,0.13324088597847747,0.08398466966317847,0.06666944395920799,Energy,7
+ett2/W/short,Modular-TS-2.4M-v3_M1,3541620.5714285714,3541620.5714285714,1345.2218191964287,0.9547622085461569,0.1449977159500122,0.15665956905909947,11.340939022920598,1881.919384944151,0.15786415525532294,0.11284346599404156,0.09697388059895506,Energy,7
+hierarchical_sales/D/short,Modular-TS-2.4M-v3_M1,28.936732637453492,28.936732637453492,2.3071339525630425,0.7453639144930956,0.6165533933772638,1.0357622986776354,6.798132751789339,5.379287372640868,1.6503057939121941,0.7078031466790955,0.5757825059383115,Sales,1
+hierarchical_sales/W/short,Modular-TS-2.4M-v3_M1,477.0483315677966,477.0483315677966,9.167869438559322,0.7432259541889759,0.5635383778173443,0.4685462046477754,6.674547896652912,21.84143611504968,1.002751015482967,0.42090137026274516,0.35890123597072937,Sales,1
+hospital/M/short,Modular-TS-2.4M-v3_M1,3125.2115384615386,3125.2115384615386,18.713556945349847,0.767503522036095,0.192932274796454,0.1733501290093811,5.795029146932404,55.90359146299581,0.20294301861673855,0.06793455726470443,0.0545404261615041,Healthcare,1
+jena_weather/10T/long,Modular-TS-2.4M-v3_M1,1443.2055555555555,1443.2055555555555,10.116731770833333,0.7593568382098755,1.100752820766555,0.49222818745996527,5.903263138358273,37.98954534547045,0.23246454209202339,0.06190601643657213,0.0515347532084138,Nature,21
+jena_weather/10T/medium,Modular-TS-2.4M-v3_M1,1446.1549783549783,1446.1549783549783,9.601532061688312,0.6357975527659445,0.9295913873316204,0.4859170127441035,5.517174917876551,38.02834440723101,0.23337927368603792,0.058924431598551034,0.04895998333926589,Nature,21
+jena_weather/10T/short,Modular-TS-2.4M-v3_M1,752.6180555555555,752.6180555555555,5.232061089409722,0.27035257312197475,0.3470003990971884,0.42719930557880287,2.2899825126672466,27.433885170634426,0.17039882350965418,0.03249765932243417,0.027317755948203546,Nature,21
+jena_weather/D/short,Modular-TS-2.4M-v3_M1,358.2728670634921,358.2728670634921,9.786906312003968,1.1468402426850592,0.7556325844701777,0.4527134486607143,7.451680589994401,18.92809729115666,0.11397808223330899,0.05893317195492466,0.0474915151842114,Nature,21
+jena_weather/H/long,Modular-TS-2.4M-v3_M1,1242.4757936507935,1242.4757936507935,10.863038401124339,0.8130128803640841,2.96361049556809,0.5416168209643959,6.04634515756844,35.24877010125025,0.2122366701561103,0.06540753312555465,0.054081654231405804,Nature,21
+jena_weather/H/medium,Modular-TS-2.4M-v3_M1,1334.2667658730159,1334.2667658730159,10.666871279761905,0.7757337503072408,3.033535544209986,0.6149863686895949,5.766084326924138,36.52761648223185,0.22351894657938667,0.06527247221044566,0.05414278224434971,Nature,21
+jena_weather/H/short,Modular-TS-2.4M-v3_M1,1118.265873015873,1118.265873015873,8.77562395572264,0.5575230640846863,2.1529343567204986,0.5692911840960002,4.7894401276209715,33.440482547593014,0.2049880216277734,0.053794014206363795,0.04447337600489246,Nature,21
+kdd_cup_2018/D/short,Modular-TS-2.4M-v3_M1,2937.0604625889355,2937.0604625889355,21.22017127214035,1.1948516372972833,0.511777291233548,0.46432526976086136,9.314079710765023,54.19465345021532,1.2140178020228125,0.47535437623965665,0.3776508562461224,Nature,1
+kdd_cup_2018/H/long,Modular-TS-2.4M-v3_M1,4117.359196395615,4117.359196395615,23.66399017941364,1.003633075334812,1.0376693110834994,0.6020821082508269,7.688290315191491,64.16665174680392,1.5056089083022801,0.5552528213672531,0.44193315910112674,Nature,1
+kdd_cup_2018/H/medium,Modular-TS-2.4M-v3_M1,5299.258908786141,5299.258908786141,25.391270108120917,1.0468693908626807,0.9595902537350729,0.5642321082769655,8.73280114304173,72.79600887951304,1.5237522924414542,0.531485264518917,0.42880623103540333,Nature,1
+kdd_cup_2018/H/short,Modular-TS-2.4M-v3_M1,4584.175843487953,4584.175843487953,22.749892904784264,0.95232434344199,0.890807433928544,0.5088257350694734,7.662817227431168,67.70654210257642,1.4172205403315912,0.4761964577390122,0.3829445880060442,Nature,1
+loop_seattle/5T/long,Modular-TS-2.4M-v3_M1,77.26814915147345,77.26814915147345,5.110872122606352,0.80926892145504,0.1596513534141727,0.11667130228713378,5.974691681224069,8.790230324142449,0.15541307062057544,0.0903612648171242,0.07131325128347951,Transport,1
+loop_seattle/5T/medium,Modular-TS-2.4M-v3_M1,65.26931017801857,65.26931017801857,4.754036780830753,0.7570855931174018,0.15041783068079206,0.10974240849261682,5.513218410195204,8.078942392294834,0.1437409880385142,0.08458408426080426,0.06666107906952709,Transport,1
+loop_seattle/5T/short,Modular-TS-2.4M-v3_M1,42.28828225780444,42.28828225780444,3.693087041045859,0.584577183016778,0.09555870071899288,0.07878280734130104,4.261568654004953,6.502944122303715,0.11151113514682257,0.06332828952513796,0.04999500278718841,Transport,1
+loop_seattle/D/short,Modular-TS-2.4M-v3_M1,19.247128160474716,19.247128160474716,3.069865941853715,0.9222919478266817,0.05698735145592468,0.056664226520172216,6.831595807621177,4.387154905001044,0.07840432343320114,0.05486260855008722,0.04409556422971058,Transport,1
+loop_seattle/H/long,Modular-TS-2.4M-v3_M1,61.99619453044376,61.99619453044376,4.348210206505848,0.9057515030505758,0.1377619010786195,0.10102539482418516,7.543673818871382,7.8737662227452345,0.1407147465779697,0.07770833930892147,0.062404691134790226,Transport,1
+loop_seattle/H/medium,Modular-TS-2.4M-v3_M1,65.77274735552116,65.77274735552116,4.492230331688597,0.921160909128047,0.12503000895182292,0.10257727246043359,7.348116805943949,8.11003991084638,0.14334404367348244,0.07939966608496138,0.06444749087666797,Transport,1
+loop_seattle/H/short,Modular-TS-2.4M-v3_M1,61.149788382285045,61.149788382285045,4.268878590541999,0.8667326508127984,0.12220229219881393,0.09891576271249822,7.2416823477751615,7.81983301498728,0.13841899749822176,0.07556349270015796,0.061376456341542145,Transport,1
+m4_daily/D/short,Modular-TS-2.4M-v3_M1,392397.3660481936,392397.3660481936,181.16333851895976,3.3476633916432093,0.03983545669163223,0.03301116227579889,26.376863236475945,626.4162881408765,0.09676400471083228,0.02798472912303381,0.02220331743107636,Econ/Fin,1
+m4_hourly/H/short,Modular-TS-2.4M-v3_M1,1324520.2962962964,1324520.2962962964,237.69789150563608,1.081845233732063,0.11952784441519475,0.10734349249257749,8.872149782593194,1150.8780544854856,0.15712025744314306,0.032451008828867055,0.02570204011312775,Econ/Fin,1
+m4_monthly/M/short,Modular-TS-2.4M-v3_M1,1934938.9624814815,1934938.9624814815,585.6214935619213,1.0030714222778911,0.16685686432873761,0.1375792439955252,8.349537856486776,1391.0208346683673,0.2891018823308095,0.12171225038659843,0.09736162749611771,Econ/Fin,1
+m4_quarterly/Q/short,Modular-TS-2.4M-v3_M1,1874165.072,1874165.072,603.2542565104167,1.316495197534114,0.12405512468020122,0.10885652160644531,9.885131678659691,1369.001487216139,0.22914674122762205,0.10097413940152244,0.08072033966550607,Econ/Fin,1
+m4_weekly/W/short,Modular-TS-2.4M-v3_M1,335642.3844011142,335642.3844011142,290.3625991000643,2.4103909647970867,0.07060589247810424,0.07127483360563532,19.266476231809886,579.346514964157,0.10554795669707111,0.05289956570837176,0.04221483984337579,Econ/Fin,1
+m4_yearly/A/short,Modular-TS-2.4M-v3_M1,3688188.9997388353,3688188.9997388353,936.5003173877717,3.5548652369615543,0.17382724042770525,0.1558134202350262,29.47083697666009,1920.465828839148,0.3079555980318765,0.15017216706871636,0.12110404592475733,Econ/Fin,1
+m_dense/D/short,Modular-TS-2.4M-v3_M1,10159.522962962963,10159.522962962963,53.3965625,0.7801702443914319,0.13779236247046422,0.1134519788953993,7.483199837382785,100.79445898938573,0.17449011796416458,0.09243734807373728,0.07591254278413982,Transport,1
+m_dense/H/long,Modular-TS-2.4M-v3_M1,36196.61432098765,36196.61432098765,87.6086574074074,0.7551546647160053,0.3013599794595034,0.21297898553485448,7.125273549664825,190.2540783294478,0.3293579342551789,0.1516635368867529,0.1240570907774525,Transport,1
+m_dense/H/medium,Modular-TS-2.4M-v3_M1,37816.72888888889,37816.72888888889,88.91314236111111,0.7605924991518306,0.2978922554395164,0.21514732452903898,7.291474351876949,194.46523825323868,0.33834494648820845,0.15469763472633177,0.1258787633837597,Transport,1
+m_dense/H/short,Modular-TS-2.4M-v3_M1,48915.275555555556,48915.275555555556,100.25403645833333,0.8735990399490393,0.3199904790093199,0.23692531760679472,8.501058392911478,221.16798040303112,0.39229688570665994,0.17782567897241194,0.14547168121836587,Transport,1
+restaurant/D/short,Modular-TS-2.4M-v3_M1,150.43012748669096,150.43012748669096,7.4691381806294945,0.7131750289869679,0.7115707333330414,0.4073099264791725,4.815310569254752,12.264996024731968,0.5644635347684277,0.3437470448918538,0.26815136235708237,Sales,1
+saugeen/D/short,Modular-TS-2.4M-v3_M1,1255.1364583333334,1255.1364583333334,13.298326009114584,2.9542435988005167,0.29903676350911457,0.33566294352213544,25.569303454472077,35.42790507965908,1.147518286774784,0.43073594796643183,0.3473857615991493,Nature,1
+saugeen/M/short,Modular-TS-2.4M-v3_M1,398.76102120535717,398.76102120535717,12.247690836588541,0.7174716361510609,0.342522212437221,0.35552256447928293,4.534491556471845,19.969001507470452,0.5995293316977024,0.3677124216427659,0.28603135309760525,Nature,1
+saugeen/W/short,Modular-TS-2.4M-v3_M1,1000.85078125,1000.85078125,15.224264526367188,1.2349741504575213,0.3950361728668213,0.4176785469055176,11.806592395561532,31.636225774418794,0.9549657813238388,0.45955708408994433,0.37746652189469726,Nature,1
+solar/10T/long,Modular-TS-2.4M-v3_M1,23.032447219687754,23.032447219687754,1.9713294753966444,0.8621459578505782,3.446850755508605,1.3579222325089273,4.592986812658656,4.799213187563953,1.0372731537285507,0.4260713292089982,0.3338359879421979,Energy,1
+solar/10T/medium,Modular-TS-2.4M-v3_M1,23.399660611590356,23.399660611590356,1.9647820587259457,0.8598613118615777,3.9067165151966625,1.3391397518249897,4.916194768083654,4.837319568892504,1.0734204229639002,0.43599294164316454,0.3459019924979136,Energy,1
+solar/10T/short,Modular-TS-2.4M-v3_M1,33.723401740229626,33.723401740229626,2.723277234451034,1.1968402326041616,6.204456295299063,1.5001876465957038,5.100924566936104,5.8071853543889596,1.6893095907126325,0.7922010525891977,0.6068997591975502,Energy,1
+solar/D/short,Modular-TS-2.4M-v3_M1,135535.72749391728,135535.72749391728,250.01791362530415,0.9752235465681917,1.1072755094282238,0.4214751380493461,6.18232538588262,368.15177236286297,0.5318539169598063,0.3611907279932196,0.2825565340308264,Energy,1
+solar/H/long,Modular-TS-2.4M-v3_M1,778.3379562043796,778.3379562043796,13.486013534063261,1.0291823076641544,5.149863372380082,1.276069171517705,5.4384003716373455,27.898708862676415,0.9672990802210861,0.4675846667872307,0.3518338960682997,Energy,1
+solar/H/medium,Modular-TS-2.4M-v3_M1,736.2314476885645,736.2314476885645,12.329110591545012,0.9441481311907173,4.591132831562747,1.2722219049250862,5.789483995718961,27.13358523469695,0.9774150702516566,0.44412333979240187,0.3478506380884181,Energy,1
+solar/H/short,Modular-TS-2.4M-v3_M1,715.8867962847355,715.8867962847355,11.702336005905646,0.9044365748556973,4.6407299556635095,1.1701262443745015,6.501452797930489,26.756060926166533,0.9862373044853525,0.4313519972351118,0.3508833002839567,Energy,1
+solar/W/short,Modular-TS-2.4M-v3_M1,2389976.7591240876,2389976.7591240876,1241.9268932481752,1.3624889021851072,0.2861955183265853,0.2382669936131387,8.87072004666349,1545.954966719305,0.3155680594890949,0.2535083286166676,0.18192297054488138,Energy,1
+sz_taxi/15T/long,Modular-TS-2.4M-v3_M1,15.928574608262108,15.928574608262108,2.7806888243411683,0.5244470608982269,6650318795664.556,0.41094389912749285,4.537647377087957,3.9910618396940567,0.3694055745349831,0.25737560429218964,0.20908997649672162,Transport,1
+sz_taxi/15T/medium,Modular-TS-2.4M-v3_M1,16.384969284188035,16.384969284188035,2.7931724091880343,0.5450982003341567,9861828586658.79,0.4125929351545807,4.257375272013037,4.047835135499967,0.37671862381263804,0.25995131492201445,0.20716505142307973,Transport,1
+sz_taxi/15T/short,Modular-TS-2.4M-v3_M1,16.7533195970696,16.7533195970696,2.753968011765253,0.5500745426844621,1067602223315.3171,0.4005391792791085,3.952657790711455,4.093081919174059,0.3827206648290862,0.2575077873333941,0.20260036043551058,Transport,1
+sz_taxi/H/short,Modular-TS-2.4M-v3_M1,7.632657042935363,7.632657042935363,1.927211402827858,0.5869074094720954,1.2931420741937099,0.305347410022703,4.3569741325911115,2.7627263785860814,0.2573650106397508,0.1795316347787159,0.14225309101194755,Transport,1
+temperature_rain/D/short,Modular-TS-2.4M-v3_M1,189.0777322179368,189.0777322179368,5.9719531694435695,1.3929113280636496,31.342278410802678,1.4597783411489158,18.62782191179909,13.75055388767801,1.6188037505079726,0.7030567835682812,0.5648490663882909,Nature,1
+us_births/D/short,Modular-TS-2.4M-v3_M1,208488.02666666667,208488.02666666667,290.93140625,0.42816387846273435,0.0278005854288737,0.027437305450439452,4.140300088215134,456.60489119879855,0.04280300341490426,0.027272458563743076,0.02229785791913427,Healthcare,1
+us_births/M/short,Modular-TS-2.4M-v3_M1,88368842.66666667,88368842.66666667,7829.018229166667,0.8869402107460148,0.023900558551152546,0.02429335316022237,6.003885858475542,9400.47034284278,0.029197796793272567,0.024316877242219714,0.018599029498638203,Healthcare,1
+us_births/W/short,Modular-TS-2.4M-v3_M1,4173123.4285714286,4173123.4285714286,1518.942103794643,1.3805942500939652,0.02031937667301723,0.020534809146608626,9.69547943307617,2042.8224172872758,0.027731114006895916,0.020619490120016678,0.0164939195793787,Healthcare,1