| TITLE = """ |
| <div class="page-heading"> |
| <h1>Leaderboard Overview</h1> |
| <p class="page-subtitle"> |
| See how leading models perform on <strong>DFL-Bench</strong>, a benchmark for |
| decision-focused learning. Submit your predictions and compare against the leaderboard. Learn more in the |
| <em>About</em> tab. |
| </p> |
| </div> |
| """ |
|
|
| INTRODUCTION_TEXT = r""" |
| ## What is DFL-Bench? |
| |
| **DFL-Bench** evaluates **Decision-Focused Learning (DFL)**, an end-to-end training |
| paradigm in which a prediction model is optimized directly for the *decision quality* |
| it enables on a downstream optimization problem. A model maps features $x$ to a |
| predicted intermediate variable $\hat{y}$, an optimizer turns $\hat{y}$ into a |
| decision $z(\hat{y})$, and the loss we ultimately care about is the cost of that |
| decision under the **true** $y^*$. |
| |
| ## Getting started |
| |
| 1. **Read the Dataset tab** to understand the file schemas. The two tasks use |
| different layouts — open and skim a few JSON entries before writing code. |
| 2. **Open `start_notebook.ipynb`** in this Space (or in the |
| [repo](https://huggingface.co/spaces/GT-KOALA/DFL-Bench/tree/main)) |
| and run it. It loads both datasets, defines trivial |
| *mean-of-context* baselines, builds the optimization layers with |
| [`cvxpylayers`](https://github.com/cvxgrp/cvxpylayers), and writes |
| valid submission JSON files. |
| |
| ## Tasks |
| |
| In every task you predict a demand-like quantity $y \in \mathbb{R}^d$ from |
| context $x$ and then choose a decision $z$ that minimizes a task-specific cost |
| under a feasibility constraint. Models are scored on the true $y^*$ that the |
| benchmark withholds. |
| |
| ### ⚡ Power Scheduling |
| |
| Schedule electricity for the next 24 hours given temperature forecasts and the |
| last 4 days of (temperature, load). The asymmetric loss penalizes |
| under-scheduling much more than over-scheduling, and the schedule must respect |
| an hourly ramp limit: |
| |
| $$ |
| \begin{aligned} |
| \min_{z\in\mathbb{R}^{24}} \quad & \sum_{t=1}^{24} \big[\gamma_{\text{under}}\,(y^*_t - z_t)_+ + \gamma_{\text{over}}\,(z_t - y^*_t)_+\big] \\ |
| \text{s.t.} \quad & |z_{t+1} - z_t| \le c_{\text{ramp}}, \quad t = 1,\dots,23 |
| \end{aligned} |
| $$ |
| |
| with $\gamma_{\text{under}} = 50, \gamma_{\text{over}} = 0.5, c_{\text{ramp}} = 0.4$. |
| $y^*$ is the true day-4 load (24 hourly values). |
| |
| ### 📦 Newsvendor (M5 subset) |
| |
| For each test instance, decide an order quantity $z_i$ for every |
| item-store pair $i$, given 13 days of context (sales + price) and the target |
| day's posted price. Underordering costs $b$ per unit (backorder); overordering |
| costs $h$ per unit (holding). A shared daily budget caps total inventory. The problem can be formulated as follows: |
| |
| $$ |
| \begin{aligned} |
| \min_{z\in\mathbb{R}_+^{n}} \quad & \sum_{i=1}^{n} \big[b\,(y^*_i - z_i)_+ + h\,(z_i - y^*_i)_+\big] \\ |
| \text{s.t.} \quad & \sum_{i=1}^{n} z_i \le B |
| \end{aligned} |
| $$ |
| |
| with $h = 1, b = 9, B = 25000, n = 8230$ item-store pairs (FOODS_3 across |
| 3 states × 10 stores). $y^*_i$ is the true day-13 sales for item $i$. |
| |
| ## Metrics |
| |
| | Metric | Definition | Direction | |
| |--------|------------|-----------| |
| | **Regret** | The task-specific regret evaluated at the submitted $z$ and the true $y^*$ (hidden) $f(z, y^*) - f(z^*, y^*)$ | Lower is better | |
| | **Constraint Violation** | Total amount by which $z$ violates the task's feasibility constraint | Lower is better | |
| |
| Each metric is reported as **mean ± std** across the test instances. The |
| leaderboard is ranked by mean **Regret** (ascending). A submission that |
| violates the constraint is still scored, but the violation column makes it |
| obvious — treat any non-zero violation as a bug in your decision step. |
| |
| ## Submission format |
| |
| See the **Dataset** tab for the exact JSON schemas. In short: |
| |
| - **Power**: `{"instance_id": [z_0, ..., z_23], ...}` for all 219 test instances. |
| - **Newsvendor**: `{"instance_id": {"FOODS_3_001_CA_1": z, ...}, ...}` for all 26 |
| test instances; the item-id set must match the full 8230-item universe — the |
| grader rejects partial submissions. |
| |
| `start_notebook.ipynb` writes both formats correctly. If you change item ordering |
| or skip items, your submission will fail the grader's `assert`. |
| """ |
|
|
| DATASET_DESCRIPTION_TEXT = """ |
| # Power Scheduling Dataset |
| |
| Download the dataset [here](https://huggingface.co/spaces/GT-KOALA/DFL-Bench/tree/main/tasks/power_scheduling). |
| |
| Every day has **24 hourly values** for two quantities: |
| |
| | Key | Meaning | Length | |
| | ------ | ---------------------- | ------ | |
| | `temp` | Temperature | 24 | |
| | `load` | Electricity load | 24 | |
| |
| ## `train.json` |
| |
| The first 1000 days in the dataset, one entry per day. |
| |
| ```json |
| { |
| "0": { "temp": [t0, ..., t23], "load": [l0, ..., l23] }, |
| "1": { "temp": [...], "load": [...] }, |
| ... |
| "999": { "temp": [...], "load": [...] } |
| } |
| ``` |
| |
| - **Key**: sequential day id (`"0"`-`"999"`). |
| - **Value**: 24 hourly `temp` and 24 hourly `load` values. |
| |
| ## `test.json` |
| |
| Test inputs. Each instance is a 5-day window where `day_0`-`day_3` provide full context and `day_4` provides only `temp` (the `load` is withheld for prediction). |
| |
| ```json |
| { |
| "0": { |
| "day_0": { "temp": [......], "load": [......] }, |
| "day_1": { "temp": [......], "load": [......] }, |
| "day_2": { "temp": [......], "load": [......] }, |
| "day_3": { "temp": [......], "load": [......] }, |
| "day_4": { "temp": [......] } |
| }, |
| ... |
| } |
| ``` |
| |
| - **Key**: `test_instance_id` (`"0"`, `"1"`, ...). |
| - **Value**: `day_0`-`day_4`, where `day_0`-`day_3` each carry `temp` and `load`, and `day_4` carries only `temp`. |
| |
| ## Submission Format |
| |
| Upload a **JSON** file mapping each `test_instance_id` to the chosen |
| 24-hour decision `z` (the scheduled load for `day_4`): |
| |
| ```json |
| { |
| "0": [z0, ..., z23], |
| "1": [z0, ..., z23], |
| ... |
| } |
| ``` |
| |
| # Newsvendor Dataset |
| |
| Download the dataset [here](https://www.dropbox.com/scl/fo/sv0xn8vkeejic05g675he/AANKfJb0eyCBC9xeu7FeFUQ?rlkey=2yxjuzadk36pvj9smu8vk0vlm&st=xx4ur2gs&dl=0). |
| |
| Built from a subset of the M5 sales data. Each **item** is an item-store pair identified by `id` (e.g. `FOODS_3_001_CA_1`). For a given `id` and day, two quantities are recorded: |
| |
| | Key | Meaning | |
| | ------- | --------------------- | |
| | `sale` | Units sold that day | |
| | `price` | That day's sell price | |
| |
| Days are split by time into training and testing periods. |
| |
| ## `train.json` |
| |
| Contains two sections: static per-item metadata and a per-date time series over the training days. |
| |
| ```json |
| { |
| "static": { |
| "FOODS_3_001_CA_1": { "item_id": "FOODS_3_001", "store_id": "CA_1", "state_id": "CA" }, |
| ... |
| }, |
| "time": { |
| "2011-01-29": { |
| "FOODS_3_001_CA_1": { "sale": 3, "price": 2.0 }, |
| ... |
| }, |
| ... |
| } |
| } |
| ``` |
| |
| - **`static`**: keyed by `id`, giving `item_id`, `store_id`, and `state_id`. |
| - **`time`**: keyed by date, then by `id`, giving that day's `sale` and `price`. |
| |
| ## `test.json` |
| |
| Test inputs. Each instance is a 14-day window where the first 13 days provide full context (`sale` + `price`) and the last day provides only `price` (the `sale` is withheld for prediction). |
| |
| ```json |
| { |
| "0": { |
| "<date day_0>": { "FOODS_3_001_CA_1": { "sale": .., "price": .. }, ... }, |
| ... |
| "<date day_12>": { "FOODS_3_001_CA_1": { "sale": .., "price": .. }, ... }, |
| "<date day_13>": { "FOODS_3_001_CA_1": { "price": .. }, ... } |
| }, |
| ... |
| } |
| ``` |
| |
| - **Key**: `test_instance_id` (`"0"`, `"1"`, ...). |
| - **Value**: 14 dates. Days 0-12 carry `sale` and `price`; day 13 carries only `price`. |
| |
| ## Submission Format |
| |
| Upload a **JSON** file mapping each `test_instance_id` to the chosen order quantity `z` per `id`: |
| |
| ```json |
| { |
| "0": { "FOODS_3_001_CA_1": 4.0, "FOODS_3_002_CA_1": 1.0, ... }, |
| ... |
| } |
| ``` |
| """ |
|
|
