Leaderboard Overview
See how leading models perform on DFL-Bench, a benchmark for
decision-focused learning. Submit your predictions and compare against the leaderboard. Learn more in the
About tab.
"""
INTRODUCTION_TEXT = r"""
## What is DFL-Bench?
**DFL-Bench** evaluates **Decision-Focused Learning (DFL)**, an end-to-end training
paradigm in which a prediction model is optimized directly for the *decision quality*
it enables on a downstream optimization problem. A model maps features $x$ to a
predicted intermediate variable $\hat{y}$, an optimizer turns $\hat{y}$ into a
decision $z(\hat{y})$, and the loss we ultimately care about is the cost of that
decision under the **true** $y^*$.
## Getting started
1. **Read the Dataset tab** to understand the file schemas. The two tasks use
different layouts — open and skim a few JSON entries before writing code.
2. **Open `start_notebook.ipynb`** in this Space (or in the
[repo](https://huggingface.co/spaces/GT-KOALA/DFL-Bench/tree/main))
and run it. It loads both datasets, defines trivial
*mean-of-context* baselines, builds the optimization layers with
[`cvxpylayers`](https://github.com/cvxgrp/cvxpylayers), and writes
valid submission JSON files.
## Tasks
In every task you predict a demand-like quantity $y \in \mathbb{R}^d$ from
context $x$ and then choose a decision $z$ that minimizes a task-specific cost
under a feasibility constraint. Models are scored on the true $y^*$ that the
benchmark withholds.
### ⚡ Power Scheduling
Schedule electricity for the next 24 hours given temperature forecasts and the
last 4 days of (temperature, load). The asymmetric loss penalizes
under-scheduling much more than over-scheduling, and the schedule must respect
an hourly ramp limit:
$$
\begin{aligned}
\min_{z\in\mathbb{R}^{24}} \quad & \sum_{t=1}^{24} \big[\gamma_{\text{under}}\,(y^*_t - z_t)_+ + \gamma_{\text{over}}\,(z_t - y^*_t)_+\big] \\
\text{s.t.} \quad & |z_{t+1} - z_t| \le c_{\text{ramp}}, \quad t = 1,\dots,23
\end{aligned}
$$
with $\gamma_{\text{under}} = 50, \gamma_{\text{over}} = 0.5, c_{\text{ramp}} = 0.4$.
$y^*$ is the true day-4 load (24 hourly values).
### 📦 Newsvendor (M5 subset)
For each test instance, decide an order quantity $z_i$ for every
item-store pair $i$, given 13 days of context (sales + price) and the target
day's posted price. Underordering costs $b$ per unit (backorder); overordering
costs $h$ per unit (holding). A shared daily budget caps total inventory. The problem can be formulated as follows:
$$
\begin{aligned}
\min_{z\in\mathbb{R}_+^{n}} \quad & \sum_{i=1}^{n} \big[b\,(y^*_i - z_i)_+ + h\,(z_i - y^*_i)_+\big] \\
\text{s.t.} \quad & \sum_{i=1}^{n} z_i \le B
\end{aligned}
$$
with $h = 1, b = 9, B = 25000, n = 8230$ item-store pairs (FOODS_3 across
3 states × 10 stores). $y^*_i$ is the true day-13 sales for item $i$.
## Metrics
| Metric | Definition | Direction |
|--------|------------|-----------|
| **Regret** | The task-specific regret evaluated at the submitted $z$ and the true $y^*$ (hidden) $f(z, y^*) - f(z^*, y^*)$ | Lower is better |
| **Constraint Violation** | Total amount by which $z$ violates the task's feasibility constraint | Lower is better |
Each metric is reported as **mean ± std** across the test instances. The
leaderboard is ranked by mean **Regret** (ascending). A submission that
violates the constraint is still scored, but the violation column makes it
obvious — treat any non-zero violation as a bug in your decision step.
## Submission format
See the **Dataset** tab for the exact JSON schemas. In short:
- **Power**: `{"instance_id": [z_0, ..., z_23], ...}` for all 219 test instances.
- **Newsvendor**: `{"instance_id": {"FOODS_3_001_CA_1": z, ...}, ...}` for all 26
test instances; the item-id set must match the full 8230-item universe — the
grader rejects partial submissions.
`start_notebook.ipynb` writes both formats correctly. If you change item ordering
or skip items, your submission will fail the grader's `assert`.
"""
DATASET_DESCRIPTION_TEXT = """
# Power Scheduling Dataset
Download the dataset [here](https://huggingface.co/spaces/GT-KOALA/DFL-Bench/tree/main/tasks/power_scheduling).
Every day has **24 hourly values** for two quantities:
| Key | Meaning | Length |
| ------ | ---------------------- | ------ |
| `temp` | Temperature | 24 |
| `load` | Electricity load | 24 |
## `train.json`
The first 1000 days in the dataset, one entry per day.
```json
{
"0": { "temp": [t0, ..., t23], "load": [l0, ..., l23] },
"1": { "temp": [...], "load": [...] },
...
"999": { "temp": [...], "load": [...] }
}
```
- **Key**: sequential day id (`"0"`-`"999"`).
- **Value**: 24 hourly `temp` and 24 hourly `load` values.
## `test.json`
Test inputs. Each instance is a 5-day window where `day_0`-`day_3` provide full context and `day_4` provides only `temp` (the `load` is withheld for prediction).
```json
{
"0": {
"day_0": { "temp": [......], "load": [......] },
"day_1": { "temp": [......], "load": [......] },
"day_2": { "temp": [......], "load": [......] },
"day_3": { "temp": [......], "load": [......] },
"day_4": { "temp": [......] }
},
...
}
```
- **Key**: `test_instance_id` (`"0"`, `"1"`, ...).
- **Value**: `day_0`-`day_4`, where `day_0`-`day_3` each carry `temp` and `load`, and `day_4` carries only `temp`.
## Submission Format
Upload a **JSON** file mapping each `test_instance_id` to the chosen
24-hour decision `z` (the scheduled load for `day_4`):
```json
{
"0": [z0, ..., z23],
"1": [z0, ..., z23],
...
}
```
# Newsvendor Dataset
Download the dataset [here](https://www.dropbox.com/scl/fo/sv0xn8vkeejic05g675he/AANKfJb0eyCBC9xeu7FeFUQ?rlkey=2yxjuzadk36pvj9smu8vk0vlm&st=xx4ur2gs&dl=0).
Built from a subset of the M5 sales data. Each **item** is an item-store pair identified by `id` (e.g. `FOODS_3_001_CA_1`). For a given `id` and day, two quantities are recorded:
| Key | Meaning |
| ------- | --------------------- |
| `sale` | Units sold that day |
| `price` | That day's sell price |
Days are split by time into training and testing periods.
## `train.json`
Contains two sections: static per-item metadata and a per-date time series over the training days.
```json
{
"static": {
"FOODS_3_001_CA_1": { "item_id": "FOODS_3_001", "store_id": "CA_1", "state_id": "CA" },
...
},
"time": {
"2011-01-29": {
"FOODS_3_001_CA_1": { "sale": 3, "price": 2.0 },
...
},
...
}
}
```
- **`static`**: keyed by `id`, giving `item_id`, `store_id`, and `state_id`.
- **`time`**: keyed by date, then by `id`, giving that day's `sale` and `price`.
## `test.json`
Test inputs. Each instance is a 14-day window where the first 13 days provide full context (`sale` + `price`) and the last day provides only `price` (the `sale` is withheld for prediction).
```json
{
"0": {
"