| --- |
| language: en |
| license: apache-2.0 |
| library_name: pytorch |
| pipeline_tag: time-series-forecasting |
| tags: |
| - time-series |
| - forecasting |
| - probabilistic-forecasting |
| - quantile-regression |
| - bitnet |
| - ternary |
| - quantization |
| datasets: |
| - Salesforce/lotsa_data |
| metrics: |
| - mase |
| --- |
|
|
| # TriChronos-50M
|
|
|
| **TriChronos-50M** is a ~50M-parameter, encoder-only Transformer for **probabilistic**
|
| time-series forecasting. Weights are trained with **1.58-bit ternary quantisation**
|
| (BitNet-style `{-1, 0, +1}`), and the model outputs **21 quantiles** per future step
|
| rather than a single point forecast.
|
|
|
| It was trained from scratch on a strict compute budget (single NVIDIA L40S, ~$15) as a
|
| study in *how far a small, quantised model can go* on general time-series forecasting —
|
| not as a state-of-the-art benchmark entry.
|
|
|
| > **TL;DR** — On the datasets it does well, it does *genuinely* well: it **beats the
|
| > naïve baseline on Weather (MASE 0.83) and M3-Monthly (0.81)**. Performance is strongly
|
| > **frequency-dependent**: solid on monthly/high-frequency series, weak on quarterly, and
|
| > poor on yearly (which have very few observations). Read the per-frequency breakdown
|
| > below rather than the headline aggregate.
|
|
|
| ---
|
|
|
| ## Highlights
|
|
|
| | | Dataset | MASE | Meaning |
|
| |---|---|---:|---|
|
| | 🟢 | **Weather** | **0.83** | Beats naïve — strongest result |
|
| | 🟢 | **M3-Monthly** | **0.81** | Beats naïve |
|
| | 🟡 | Traffic | 2.13 | Moderate |
|
| | 🟡 | M1-Monthly | 1.66 | Moderate |
|
| | 🔴 | Quarterly / Yearly | 4–43 | Weak → poor (few observations, long horizons) |
|
|
|
| *MASE < 1 = better than the naïve baseline; lower is better.*
|
|
|
| ---
|
|
|
| ## Architecture
|
|
|
| | Property | Value |
|
| |---|---|
|
| | Parameters | **50,081,016** (~50M) |
|
| | Type | Encoder-only Transformer |
|
| | d_model | 768 |
|
| | Layers | 6 |
|
| | Heads | 12 |
|
| | FFN dim | 2304 |
|
| | Patch size | 8 timesteps |
|
| | Forecast horizon | 24 timesteps |
|
| | Weight precision | 1.58-bit ternary (`{-1, 0, +1}`, BitLinear) in attention + FFN |
|
| | Activation precision | 8-bit per-token |
|
| | Training precision | BF16 autocast |
|
| | Output | 21 quantiles (τ = 0.025, 0.05, 0.10 … 0.90, 0.95, 0.975) |
|
|
|
| Each encoder block applies **temporal self-attention**, then **cross-series ("group")
|
| attention** over the batch, then a BitLinear FFN. The input series is split into
|
| non-overlapping 8-step patches; patch embeddings and the quantile head stay in full
|
| precision.
|
|
|
| ---
|
|
|
| ## Training
|
|
|
| - **Data:** [Salesforce/lotsa_data](https://huggingface.co/datasets/Salesforce/lotsa_data),
|
| streamed per-subset (Bronze→Silver→Gold pipeline: asinh z-score normalisation → 8-step patches).
|
| - **Hardware / budget:** 1× NVIDIA L40S, ~$15 total compute.
|
| - **Steps:** ~105k (single session; cosine LR annealed toward 10% of peak).
|
| - **Optimiser:** AdamW, `lr=3e-4`, `wd=1e-2`, β=(0.9, 0.95), 2k-step warmup.
|
| - **Loss:** pinball / quantile loss over all 21 quantiles.
|
|
|
| ---
|
|
|
| ## Evaluation
|
|
|
| Zero-shot MASE on Monash Time Series Forecasting datasets (via the Parquet mirror
|
| `autogluon/chronos_datasets`). **Two protocols are reported for honesty:**
|
|
|
| - **Full** — up to a few hundred series per dataset.
|
| - **Diagnostic (10-series)** — a fixed small sample per dataset; faster, but higher variance.
|
|
|
| > ⚠️ **Metric caveat.** MASE here is computed in the model's *normalised (asinh z-score)*
|
| > space, for both the model and the naïve baseline. The model-vs-baseline comparison is
|
| > therefore fair and internally consistent, but these numbers are **not directly
|
| > comparable to published Monash leaderboards**, which report MASE on raw values with
|
| > per-dataset seasonal-naïve denominators.
|
|
|
| ### Per-dataset MASE
|
|
|
| | Dataset | Full | Diagnostic (10-series) | Read |
|
| |---|---:|---:|---|
|
| | **Weather** | **0.868** | **0.831** | 🟢 beats naïve |
|
| | **M3 Monthly** | **0.808** | 1.012 | 🟢 beats naïve (full) |
|
| | M1 Monthly | 1.656 | 2.168 | 🟡 moderate |
|
| | Traffic | 2.133 | 1.865 | 🟡 moderate |
|
| | M4 Monthly | 2.413 | 2.283 | 🟡 moderate |
|
| | Electricity Hourly | 3.377 | 1.833 | 🟡 moderate |
|
| | Tourism Monthly | 3.236 | 2.567 | 🟠 weak |
|
| | Tourism Quarterly | 4.062 | 2.911 | 🟠 weak |
|
| | M4 Quarterly | 4.086 | 1.501 | 🟠 weak (full) |
|
| | M3 Quarterly | 4.478 | — | 🟠 weak |
|
| | M1 Quarterly | 4.906 | 3.735 | 🔴 weak |
|
| | M4 Yearly | 14.007 | — | 🔴 poor (long-horizon, few obs.) |
|
| | M1 Yearly | 42.972 | — | 🔴 poor (**only 3 series**) |
|
|
|
| ### Aggregates
|
|
|
| | Slice | Mean MASE | Note |
|
| |---|---:|---|
|
| | Full benchmark (all 13) | **6.846** | inflated by yearly outliers |
|
| | Excluding yearly | **2.91** | fairer central estimate |
|
| | **High-frequency subset** (monthly + hourly) | **≈ 2.07** | where the model is designed to work |
|
| | Diagnostic 10-series | **2.071** | small-sample sanity check |
|
|
|
| **How to read this:** the full-benchmark mean is dominated by two low-frequency datasets
|
| (M1-Yearly = 42.97 across just **3 series**, M4-Yearly = 14.01). Those series have very
|
| few observations and a long forecast horizon — intrinsically hard, and statistically
|
| noisy at this sample size. The high-frequency subset (~2.07) is the number that best
|
| reflects the model's actual behaviour.
|
|
|
| ### Observed pattern: frequency dependence
|
|
|
| ```
|
| Monthly → good (M3 0.81, M1 1.66, M4 2.41)
|
| Hourly → moderate (Traffic 2.13, Electricity 3.38)
|
| Quarterly → weak (~4.0–4.9)
|
| Yearly → poor (14–43, small N)
|
| ```
|
|
|
| This is consistent with a model that has learned **local, high-frequency temporal
|
| structure** but degrades when a series is short and the horizon is long relative to the
|
| available history — a data/generalisation limitation more than an obvious capacity ceiling.
|
|
|
| ---
|
|
|
| ## Intended use & limitations
|
|
|
| **Intended:** research on small / quantised time-series foundation models; probabilistic
|
| forecasting on **monthly and higher-frequency** univariate series; a lightweight baseline.
|
|
|
| **Not recommended (as-is):** yearly or very short series; long-horizon forecasting far
|
| beyond 24 steps; any setting needing calibrated leaderboard-grade MASE without re-running
|
| evaluation on raw values.
|
|
|
| **Known limitations**
|
| - Frequency-dependent quality (above).
|
| - MASE reported in normalised space (above) — recompute on raw values for cross-paper comparison.
|
| - Trained ~105k steps on a single small budget; not converged to SOTA.
|
| - The forecast head mean-pools patch representations before projecting the horizon, which
|
| can flatten fine temporal detail on long horizons.
|
|
|
| ---
|
|
|
| ## Usage
|
|
|
| ```python
|
| import torch
|
| from model import TriChronos # from this repo
|
|
|
| model = TriChronos() # d_model=768, n_layers=6, n_heads=12, ffn_dim=2304
|
| model.load_state_dict(torch.load("model_state.pt", map_location="cpu"))
|
| model.eval()
|
|
|
| # patches: (batch, n_patches, patch_size=8) — asinh z-scored, most-recent-last
|
| patches = torch.randn(1, 64, 8)
|
| with torch.no_grad():
|
| quantiles = model(patches) # (1, 24, 21) → horizon × quantile levels
|
| median = quantiles[..., 9] # τ = 0.50
|
| ```
|
|
|
| Preprocessing (asinh z-score → 8-step patches) and the quantile levels are defined in
|
| `data_pipeline.py` / `model.py`, both included in this repo.
|
|
|
| ## Reproducing the evaluation
|
|
|
| ```bash
|
| python evaluate.py --checkpoint model_state.pt --max-series 200
|
| ```
|
|
|
| ## Citation
|
|
|
| ```bibtex
|
| @misc{trichronos2026,
|
| title = {TriChronos-50M: Ternary-Quantised Probabilistic Time-Series Forecasting},
|
| year = {2026},
|
| url = {https://huggingface.co/iravikr/trichronos-50M}
|
| }
|
| ```
|
|
|
| ## License
|
|
|
| Apache 2.0 |
