amaye15/toto-gguf

toto-rs

Pure Rust converter and inference engine for Datadog Toto-2.0.

Pre-converted GGUF files are available at amaye15/toto-gguf. Produces GGUF v3 files and runs native forecasting — no Python required.

Build

cargo build --release

Convert

Downloads the model from HuggingFace and writes a GGUF file:

# F16 (recommended — ~2× compression vs F32, ~0.07 MAE on models ≥313m)
./target/release/toto-rs convert --model Datadog/Toto-2.0-2.5B --dtype f16 --output gguf/toto-2.5b-f16.gguf

# Q8_0 (smallest — useful when memory is the bottleneck, model ≥313m)
./target/release/toto-rs convert --model Datadog/Toto-2.0-2.5B --dtype q8 --output gguf/toto-2.5b-q8.gguf

# F32 (full precision)
./target/release/toto-rs convert --model Datadog/Toto-2.0-2.5B --dtype f32 --output gguf/toto-2.5b-f32.gguf

Available models: Datadog/Toto-2.0-{4m,22m,313m,1B,2.5B}

To convert all five sizes in all three dtypes at once:

./scripts/convert_all.sh

HuggingFace token (optional for public models):

HF_TOKEN=hf_... ./scripts/convert_all.sh

Inspect tensors

Print all tensor names and shapes from a .safetensors checkpoint:

./target/release/toto-rs inspect-tensors models/model.safetensors

Infer

Run univariate quantile forecasting from stdin JSON:

echo '{"context": [1.0, 1.2, 1.5, 1.3, 1.8, 2.0, 1.9, 2.1], "horizon": 64}' \
  | ./target/release/toto-rs infer \
      --gguf gguf/toto-2.5b-f16.gguf \
      --config models/toto-2.5b/config.json

Output is JSON in an OpenAI-compatible forecast format:

{
  "id": "forecast-000001932b7a1234",
  "object": "forecast",
  "created": 1749686400,
  "model": "toto",
  "choices": [{
    "index": 0,
    "forecast": {
      "point": [2.1, 2.3, 2.5, "..."],
      "quantiles": {
        "0.10": [1.8, 2.0, 2.2, "..."],
        "0.50": [2.1, 2.3, 2.5, "..."],
        "0.90": [2.4, 2.6, 2.8, "..."]
      }
    },
    "finish_reason": "stop"
  }],
  "usage": {"context_length": 8, "forecast_length": 64}
}

point is the median (q0.5) forecast; all 9 quantiles (q0.10–q0.90) are included.

Batch inference — pass multiple series as a nested array to get one Choice per series:

echo '{"context": [[1.0, 1.2, 1.5], [2.0, 2.2, 2.5]], "horizon": 64}' \
  | ./target/release/toto-rs infer \
      --gguf gguf/toto-2.5b-f16.gguf \
      --config models/toto-2.5b/config.json

Multivariate inference — Toto's variate-aware architecture natively handles multiple co-occurring time series. Pass a 3D context array [batch][variate][time] to get a variates array in each choice:

echo '{
  "context": [
    [[1.0, 1.2, 1.5, 1.3, 1.8, 2.0, 1.9, 2.1],
     [0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 1.1, 1.2]]
  ],
  "horizon": 64
}' \
  | ./target/release/toto-rs infer \
      --gguf gguf/toto-2.5b-f16.gguf \
      --config models/toto-2.5b/config.json

Multivariate output (one variates entry per variate):

{
  "choices": [{
    "index": 0,
    "forecast": {
      "variates": [
        {
          "point": [2.1, 2.3, "..."],
          "quantiles": {"0.10": [...], "0.50": [...], "0.90": [...]}
        },
        {
          "point": [1.3, 1.4, "..."],
          "quantiles": {"0.10": [...], "0.50": [...], "0.90": [...]}
        }
      ]
    },
    "finish_reason": "stop"
  }]
}

Python bindings

Install with maturin inside a virtual environment:

python -m venv .venv && source .venv/bin/activate
pip install maturin
maturin develop --features python

import toto_rs

model = toto_rs.Toto("gguf/toto-2.5b-f16.gguf", "models/toto-2.5b/config.json")

result = model.forecast([1.0, 1.2, 1.5, 1.3, 1.8, 2.0], horizon=64)
fc     = result["choices"][0]["forecast"]
point  = fc["point"]           # median forecast
q10    = fc["quantiles"]["0.10"]  # 10th-percentile
q90    = fc["quantiles"]["0.90"]  # 90th-percentile

# Batch — one Choice per series
result = model.forecast([[1.0, 1.2, 1.5], [2.0, 2.2, 2.5]], horizon=64)

forecast returns a Python dict in the same OpenAI-compatible format as the CLI.

Architecture notes

Toto-2.0 is an encoder-decoder time series foundation model:

• Input: Multivariate time series; context is instance-normalized, patched (patch_size=32), and projected to d_model
• Encoder: Multi-layer causal transformer with xPos RoPE (extends standard RoPE with per-dimension exponential decay for long-range stability)
• Output: Last hidden states decoded through a ResidualBlock to 9-quantile forecasts (q0.1–q0.9) for each variate
• Models: Five sizes — 4m, 22m, 313m, 1B, 2.5B — all using the same architecture

Benchmark

Measured on Apple M-series (CPU), 512-step context → 64-step forecast, 3 runs averaged.

Model	dtype	Size (GB)	Time (s)	MAE vs F32
toto-4m	f32	0.02	0.01	—
	f16	0.01	0.02	0.176
	q8	0.00	0.01	3.242
toto-22m	f32	0.09	0.04	—
	f16	0.04	0.04	0.071
	q8	0.02	0.04	5.030
toto-313m	f32	1.25	0.32	—
	f16	0.63	0.36	0.074
	q8	0.33	0.39	0.316
toto-1b	f32	4.16	1.20	—
	f16	2.08	1.19	0.074
	q8	1.11	0.87	0.334
toto-2.5b	f32	9.82	5.40	—
	f16	4.91	3.04	0.066
	q8	2.61	3.49	0.243