Spaces:
Running
Running
| TITLE = """ TSFM Realworld Bench Leaderboard """ | |
| INTRODUCTION_TEXT = """ | |
| **TSFM Realworld Bench** evaluates time series foundation models on **live TS-Bench | |
| real-world data** with **zero-shot API inference** via [TSFM.ai](https://tsfm.ai/). | |
| Following [GIFT-Eval](https://huggingface.co/spaces/Salesforce/GIFT-Eval), the leaderboard | |
| reports **absolute metric values** and **per-dataset ranks**. The Overall tab also reports | |
| the latest overall snapshot plus full 24-hour, 7-day, and 30-day metric tables and a | |
| dataset-balanced **pairwise historical ranking** over all shared releases through the | |
| current cutoff. The | |
| **GIFT-style Aggregates** tab provides Seasonal-Naive-normalized MSE, CRPS, | |
| and mean CRPS rank grouped by actual prediction length, domain, and frequency. Each | |
| subsequent domain tab retains the original absolute per-dataset results. | |
| """ | |
| LLM_BENCHMARKS_TEXT = """ | |
| ## How to participate (Submit a New Model) | |
| Community models are evaluated through owner-operated HTTPS endpoints. Copy the | |
| portable FastAPI template in `examples/community_endpoint/`, replace its | |
| `forecast_one` function, and deploy it on your own inference server or existing | |
| cloud service. A paid Hugging Face Space is optional and a Static Space cannot | |
| run the forecasting API. | |
| The endpoint receives only causal history, prediction length, frequency, and | |
| requested quantiles. It never receives future labels. Servers behind NAT can use | |
| a stable named HTTPS tunnel; temporary Quick Tunnel URLs are not accepted. | |
| Before submitting, run `scripts/validate_external_model_endpoint.py` against the | |
| public `/forecast` URL and generate `community_model.yaml` with | |
| `scripts/generate_community_model_metadata.py`. Submit the public model card, | |
| endpoint-code URL, endpoint URL, metadata, and successful validator receipt | |
| through the community model request form. Accepted entries are evaluated only | |
| on future live releases. | |
| ## Metrics | |
| - **MSE** — Mean Squared Error on the mean forecast (absolute) | |
| - **RMSE** — Root Mean Squared Error on the mean forecast | |
| - **MAPE** — Mean Absolute Percentage Error, reported only away from zero | |
| - **CRPS** — quantile approximation of the Continuous Ranked Probability Score | |
| - **RTG** — normalized real-time MSE gain over causal Seasonal-Naive (higher is better) | |
| - **Stability** — standard deviation of release-level MSE (lower is better) | |
| - **Improvement** — Kendall trend statistic over release-level MSE (more negative is better) | |
| - **Pairwise Win Rate** — dataset-balanced MSE/CRPS wins over shared future releases; | |
| official ranks require sufficient shared releases, datasets, time span, opponents, | |
| and membership in the main comparison component | |
| - **RankScore** — Elo-style aggregate from per-dataset MSE and CRPS ranks (higher is better) | |
| - **MSE_Rank** / **CRPS_Rank** — per-dataset rank (lower is better) | |
| - **Grouped MSE / CRPS** — geometric mean after per-configuration normalization against | |
| Seasonal-Naive (lower is better; 1.0 equals the baseline) | |
| - **Grouped Rank** — mean per-configuration CRPS rank (lower is better) | |
| """ | |
| CITATION_BUTTON_LABEL = "Copy citation" | |
| CITATION_BUTTON_TEXT = r""" | |
| @misc{tsfm_realworld_bench, | |
| title={TSFM Realworld Bench: A Benchmark for Time Series Foundation Models}, | |
| author={TSFM Realworld Bench Team}, | |
| year={2026}, | |
| howpublished={\url{https://huggingface.co/spaces/CityMindDev/TSFM-Realworld-Bench}} | |
| } | |
| """ | |