officeqa/README.md

---
license: cc-by-sa-4.0
extra_gated_prompt: "By accessing this dataset, you agree not to use the answer keys to train models evaluated on OfficeQA or to artificially inflate benchmark scores."
extra_gated_fields:
  Name: text
  Organization: text
  Intended use: text
  I agree to the terms above: checkbox
task_categories:
- question-answering
- text-generation
- text-retrieval
language:
- en
size_categories:
- 1K<n<10K
pretty_name: OfficeQA
configs:
- config_name: officeqa_pro
  data_files:
  - split: train
    path: officeqa_pro.csv
- config_name: officeqa_full
  data_files:
  - split: train
    path: officeqa_full.csv
---

# OfficeQA

## Dataset Summary

**OfficeQA** is a grounded reasoning benchmark by Databricks for evaluating model and agent performance on end-to-end reasoning over real-world documents.

The benchmark consists of question–answer pairs that require reasoning over historical **U.S. Treasury Bulletin** documents (1939–2025), which contain dense financial tables, charts, and narrative text. OfficeQA is designed to test retrieval, tool use, and multi-step reasoning in document-grounded settings.

Two question sets are available:

- **OfficeQA Pro** (N=133) — the default benchmark for evaluating frontier models. Contains the hardest, most discriminative questions.
- **OfficeQA Full** (N=246) — includes all questions, adding easier items useful for hillclimbing and studying model behavior at different difficulty levels.

Key facts:
- **Source documents:** U.S. Treasury Bulletins (697 issues, 1939–2025)
- **Primary use cases:** RAG, agent evaluation, document reasoning benchmarks
- **Dataset license:** CC-BY-SA 4.0
- **Code license:** Apache 2.0

---

## Getting Started

### Load the benchmark questions

```python
from datasets import load_dataset
# Authenticate first (dataset is gated)
# huggingface_hub.login() or set HF_TOKEN env var

# Pro subset — default for evaluating frontier models
dataset = load_dataset("databricks/officeqa", data_files="officeqa_pro.csv", split="train")

# Full benchmark — includes easier questions for hillclimbing
dataset = load_dataset("databricks/officeqa", data_files="officeqa_full.csv", split="train")
```

### Download the corpus

```python
from huggingface_hub import snapshot_download
# Download transformed text — recommended for LLM/RAG workflows (~460MB)
local_dir = snapshot_download(
    repo_id="databricks/officeqa",
    repo_type="dataset",
    allow_patterns="treasury_bulletins_parsed/transformed/*.txt",
)
```

### Score answers using reward.py (from GitHub)

```bash
git clone https://github.com/databricks/officeqa
```

```python
from reward import score_answer

score = score_answer(ground_truth="123.45", prediction="123.45", tolerance=0.0)
```

---

## Supported Tasks and Leaderboards

- Question Answering
- Grounded / Retrieval-Augmented Generation
- Agentic reasoning over documents

This dataset is intended for **benchmarking**, not for model pretraining.

---

## Languages

- English (`en`)

---

## Dataset Structure

The dataset has two main components:

### 1. Benchmark Dataset

| File | Contents |
|------|----------|
| `officeqa_full.csv` | All 246 questions with answers |
| `officeqa_pro.csv` | 133 curated questions with answers |

**Schema:**

| Column | Description |
|--------|-------------|
| `uid` | Unique question identifier |
| `question` | Question text |
| `answer` | Ground-truth answer *(answer files only)* |
| `source_docs` | Source URLs from the FRASER archive |
| `source_files` | Corresponding Treasury Bulletin filenames |
| `difficulty` | `easy` or `hard` |

---

### 2. Treasury Bulletin Corpus

The Treasury Bulletin corpus is provided in **three formats**, all available via Git LFS in this Hugging Face repository.

#### a) Original PDFs
697 PDFs (1939–2025), ~4GB total, available via LFS in this repo.

```python
from huggingface_hub import snapshot_download
# Download all PDFs (requires dataset access)
local_dir = snapshot_download(
    repo_id="databricks/officeqa",
    repo_type="dataset",
    allow_patterns="treasury_bulletin_pdfs/*",
)
```

#### b) Parsed JSON Documents
697 JSON files (~730MB total) with layout structure, tables, bounding boxes, and metadata. More lossless than transformed text; useful for experimenting with different table representations (e.g. Markdown vs HTML).

```python
from huggingface_hub import snapshot_download
local_dir = snapshot_download(
    repo_id="databricks/officeqa",
    repo_type="dataset",
    allow_patterns="treasury_bulletins_parsed/jsons/*.json",
)
```

#### c) Transformed Text Documents
697 plain-text files (~460MB total) with tables converted to Markdown. Recommended for LLM and RAG workflows.

```python
from huggingface_hub import snapshot_download
local_dir = snapshot_download(
    repo_id="databricks/officeqa",
    repo_type="dataset",
    allow_patterns="treasury_bulletins_parsed/transformed/*.txt",
)
```

To download the full corpus at once:
```python
from huggingface_hub import snapshot_download
local_dir = snapshot_download(
    repo_id="databricks/officeqa",
    repo_type="dataset",
)
```

---

## Mapping Questions to Source Documents

Each question references the Treasury Bulletin file(s) required to answer it via the `source_files` column.

### Filename convention
```
treasury_bulletin_{YEAR}_{MONTH_NUM}.{ext}
```

### Month mapping
```
january   → 01    july      → 07
february  → 02    august    → 08
march     → 03    september → 09
april     → 04    october   → 10
may       → 05    november  → 11
june      → 06    december  → 12
```

---

## Evaluation

The [GitHub repository](https://github.com/databricks/officeqa) includes a reference scoring function (`reward.py`) for evaluating predictions against ground-truth answers.

```bash
# Get the scoring code
git clone https://github.com/databricks/officeqa
```

```python
from reward import score_answer

score = score_answer(
    ground_truth="123.45",
    prediction="123.40",
    tolerance=0.01
)
```

---

## Limitations

- Requires access to external documents for full performance
- Focused on financial and government reporting domains
- Not designed for conversational QA without retrieval

---

## License

- **Dataset:** CC-BY-SA 4.0
- **Code and scripts:** Apache 2.0

---

## Citation

```bibtex
@dataset{officeqa,
  title = {OfficeQA: A Grounded Reasoning Benchmark},
  author = {Databricks},
  year = {2025},
  license = {CC-BY-SA-4.0}
}
```

## Contact
This dataset was created and is maintained by the Databricks research team. For questions, open an issue on the [GitHub repository](https://github.com/databricks/officeqa).