--- license: apache-2.0 library_name: timejepa pipeline_tag: time-series-forecasting tags: - time-series - jepa - energy-based-model - zero-shot --- # TimeJEPA-EBM A Joint Embedding Predictive Architecture pretrained on 15.85 billion observations of public time series and read as an energy-based model: given a context and a candidate future, it returns a scalar energy, low when the future is plausible for that context. It generates nothing. It judges. It was trained by one person on three RTX 3090s, outside of any lab or company. ## What it is The encoder embeds a window of a series, patch by patch; the predictor, from the embeddings of a context, produces the embeddings of the patches that follow. Pretraining pulls the predicted embeddings toward the embeddings of the true continuation. That objective makes the frozen model an energy function over (context, future) pairs: E(x, y) = 1 - cos(pred(enc(x)), enc(y)) The candidate is encoded on its own, with the context's normalization statistics (the training convention of these checkpoints), and compared with what the predictor expected. Nothing is trained for the judging task: the released checkpoint is the pretraining checkpoint, as saved: `epoch01_valloss0.6420`, 1.1M parameters, one epoch of the corpus and the start of a second. ## What it does Measured on frozen checkpoints, no downstream training. **It ranks the true future above bootstrapped alternatives.** For a context, 34 candidate futures are built: 32 block bootstraps of the series' own past, the seasonal naive, and the truth. The normalized rank of the truth under E, averaged over instances, is 0.50 for a random judge. ![Rank of the true future among 34 candidates](assets/judge_rank_probe.png) | Checkpoint | Rank of the true future | |---|---| | chance | 0.500 | | **TimeJEPA-EBM** | **0.219** | | the same weights after finetuning as a forecaster | 0.349 | On the released checkpoint (six GIFT-Eval configurations, 100 instances each, measured on 2026-10-12) the truth lands in the top 20 % of the candidates 63 % of the time, and the energy follows the distance to the truth (correlation with each candidate's mean absolute error 0.35). The judge is strongest on hourly electricity (rank 0.036) and weakest on 10-minute solar (0.477). Finetuning the same weights into a forecaster degrades the judge to 0.349 (measured on the finetuned checkpoint of the same run); the forecaster and the judge are two checkpoints. **It turns a point forecaster into a probabilistic one, with no training.** The proposer forecasts; its forecast is recentered and surrounded by bootstrapped seasonal innovations into a pool of candidates; the judge scores the pool; the weighted candidates are read out as quantiles. | Proposer | Benchmark | Proposer alone | + TimeJEPA-EBM | Checkpoint | |---|---|---|---|---| | TTM-R3 | Nixtla long-horizon, weighted quantile loss vs seasonal naive, ettm1 / ettm2 / etth1 / etth2 / weather / exchange | 0.88 / 0.81 / 0.85 / 0.95 / 0.63 / 0.88 | **0.77 / 0.72 / 0.71 / 0.78 / 0.64 / 0.78** | released | | TTM-R3 | GIFT-Eval, 97 configurations, 150 instances each, CRPS of the collapsed point forecast | 0.7258 | **0.6732** | released | Six datasets of six beaten or tied on Nixtla, replicated with a judge from an earlier pretraining lineage. On GIFT-Eval the gain is paid with a 3.8 % increase of the median's error (MASE) and the 80 % interval covers 0.62 of the outcomes against 0.80 nominal. These GIFT numbers are paired comparisons on the same windows, not leaderboard numbers: the harness caps each configuration at 150 instances and uses the local seasonal naive. **It knows the arrow of time.** On a simulated thermostat with a known dynamics, the energy separates a future from its time reversal with an AUC of 0.808, and an inverted response to an action at 0.935 (tiny, mixed corpus, earlier lineage). Given a deliberately biased proposer, the judge moves its mass to the alternative center kept in the pool: a verifier that can fire its proposer. ## What it does not do - **It is not a forecaster.** Read alone, with bootstrapped candidates and no proposer, a judge of an earlier lineage beats the seasonal naive on 5 of 6 Nixtla datasets and ties the finetuned forecaster on one. Its value is in judging someone else's proposals. - **It does not see continuity.** A future shifted to another level, with the right shape, is accepted (AUC 0.556 against a time jump, chance level). Continuity has to be enforced outside the energy. - **It is a classifier, not a regularizer.** Used as a penalty inside a planner, it does not reduce constraint violations. - **It does not select the sampling rate.** Choosing the resampling rate of a context by energy is worse than choosing it by backtest. - **It does not improve its own forecaster's fan.** Reweighting the calibrated quantiles of the finetuned TimeJEPA with its judge dilutes them. - **Its energy has no minimum at the truth**: the truth ranks well among alternatives, but gradient descent on E from a candidate does not converge to it. ## Where it applies - **Ensembling and calibration of an existing forecaster**, with no retraining: any point forecaster becomes a quantile forecaster through the propose-judge-reweight readout. Measured with TTM-R3; the readout is proposer-agnostic. - **Plausibility scoring of observed or simulated trajectories**: ranking scenarios, flagging a continuation that does not fit its context, checking the direction of time or the sign of a response in a simulator. Measured on the thermostat simulator and on the bootstrap probe. - **Candidate selection in planning**, as a ranker over proposals, not as a term in the objective. ## Usage ```bash pip install git+https://github.com/IUseAMouse/TimeJEPA.git ``` ```python import numpy as np import torch from timejepa.hub import from_pretrained from timejepa.judge import energy judge = from_pretrained("IUseAMouse/TimeJEPA-EBM") context = np.asarray(history[-1024:], dtype=np.float32) # raw values candidates = np.stack([future_a, future_b, future_c]).astype(np.float32) # [n, horizon] with torch.no_grad(): _, e = energy(judge, context, candidates) # [n], lower is more plausible ranking = np.argsort(e) ``` The propose-judge-reweight readout, with its candidate pool and temperature calibration, is `scripts/evaluate_energy.py` in the repository; the rank probe is `scripts/probe_energy.py`. ## Model Encoder: a RoPE transformer over patches of 16 steps. Predictor: learned queries, one per future patch, cross-attending to the context embeddings. Target embeddings during pretraining come from a slow moving-average copy of the encoder, and a variance regularizer keeps the embedding space from collapsing. Input normalization: robust arcsinh scaling then instance normalization from the context, applied to the candidates with the context's statistics. d_model 128, 3 encoder and 2 predictor layers, d_ff 512; 1.1M parameters without the moving-average copy. ## Files and license - Weights: `model.safetensors` with `config.json` (read by `timejepa.hub.from_pretrained`) and the original pretraining checkpoint `epoch01_valloss0.6420.ckpt`. - Code: [TimeJEPA](https://github.com/IUseAMouse/TimeJEPA), MIT. Weights: Apache-2.0 (the `LICENSE` file next to them). ## Citation ```bibtex @misc{vincent2026timejepaebm, author = {Vincent, Yvann}, title = {TimeJEPA-EBM: a pretrained joint-embedding predictive architecture as an energy-based judge of time series futures}, year = {2026}, url = {https://github.com/IUseAMouse/TimeJEPA} } ```