Spaces:
Running
title: FinanceBench Viewer
emoji: 🏦
colorFrom: green
colorTo: gray
sdk: static
pinned: false
license: cc-by-nc-4.0
FinanceBench Viewer
Static viewer for the open-source FinanceBench financial-QA dataset (HF dataset).
Tabs:
- 📚 Corpus — every document in the corpus (368 SEC filings / earnings PDFs).
A searchable dropdown selects a document by title (
doc_name); the panel shows its metadata and lazily loads extracted six-page text chunks with chunk navigation. Direct bundled-PDF and official-source links remain available. This avoids the Hugging Face Xet PDF iframe CSP that broke inline rendering. - ❓ Eval — the 150 annotated questions. Each shows the question, gold answer,
justification, reasoning type, question type, company, the source
doc_name(click to jump to it in the Corpus tab), and every evidence span (page number, extracted evidence text, and the full-page extract). - E2E Structures v3 — the five constructed scaffold shapes for each question's supporting filing chunks.
- E2E+Rawtext Structures — the same five generated shapes reused by the overlay run, with raw filing documents mounted as its sixth fallback source.
- c1 Closed-book, c2 With-docs, c6 Agentic-DCI, Naive-search, and E2E v3, and E2E v3 + rawtext — full-run records with search, correctness/missing filters, qid navigation, judge output, failure/stop/token details, and expandable agent trajectories when present.
- Compare — prejoined, side-by-side predictions for all six retained runs, with filters for disagreements, any missing response, and questions where only E2E is correct.
Question selection follows the same qid across Eval, run, Compare, and E2E Structures tabs. Agent trajectories pair each tool call with its result, collapse long outputs, and keep E2E supporting filings closed until requested.
Full-run results
Scores use the full 150-question denominator. Missing or unanswered responses remain visible in the viewer and count as incorrect.
| Run | Correct / 150 | Score | Answered | Missing |
|---|---|---|---|---|
| c1 Closed-book | 56 / 150 | 37.33% | 134 | 16 |
| c2 With-docs | 113 / 150 | 75.33% | 137 | 13 |
| c6 Agentic-DCI | 122 / 150 | 81.33% | 150 | 0 |
| Naive-search | 102 / 150 | 68.00% | 124 | 26 |
| E2E v3 | 121 / 150 | 80.67% | 149 | 1 |
| E2E v3 + rawtext | 123 / 150 | 82.00% | 150 | 0 |
The c6 row is the corrected rawtext rerun from AML job
gifted_helmet_6vkx7rqq1f; all 150 questions have responses.
Data
| File | Contents |
|---|---|
financebench_open_source.jsonl |
150 annotated questions (Eval tab) |
financebench_document_information.jsonl |
360 per-document metadata rows |
corpus_index.json |
368 corpus docs = PDF listing ⋈ metadata (built by scripts/build_data.py) |
corpus_text/ |
368 compressed filing-text shards covering 9,174 chunks |
pdfs/*.pdf |
368 source PDFs, ~674 MB, git-LFS tracked |
e2e_structures_v3/ |
150 compressed per-question E2E v3 structure shards |
runs/manifest.json |
Run registry, scores, coverage, colors, and provenance |
runs/<slot>/index.json |
Search/filter index for one run |
runs/<slot>/records/<qid>.json |
Normalized per-question run record |
compare/index.json |
Comparison summary and qid/filter index |
compare/records/<qid>.json |
One prejoined shard containing all six runs |
corpus_index.json is regenerated from the two source files + the pdfs/
listing with:
python scripts/build_data.py
python3 scripts/build_corpus_text.py
python3 scripts/build_e2e_structures.py \
--dataset financebench \
--unified /mnt/ramdisk/blobstore/timchen0618/data/eval/financebench/unified/test_with_chunks.unified.jsonl \
--scaffolds-dir /home/azureuser/projects/information-scaffolds/outputs/e2e_runs/new-datasets-full-20260711/financebench/named-outputs/scaffolds_dir
Run/compare shards are regenerated with:
python3 scripts/build_runs.py
scripts/build_runs.py streams the JSONL inputs, joins each run to every eval
qid, validates the exact scores and coverage above, removes bulky repeated
prompts, caps retained judge/event strings, and limits serialized trajectories
to roughly 256 KiB per record with an explicit truncation event.
Run provenance
Gold answers and metadata come from financebench_open_source.jsonl
(financebench_id → qid, answer → gold).
| Run | Responses | Judges |
|---|---|---|
| c1 | /mnt/tmp/financebench-c1-predictions.jsonl/predictions |
/mnt/tmp/financebench-c1-evaluated.jsonl/evaluated |
| c2 | /mnt/tmp/financebench-c2-predictions.jsonl/predictions |
/mnt/tmp/financebench-c2-evaluated.jsonl/evaluated |
| c6 | /tmp/baseline-pipeline-runs/gifted_helmet_6vkx7rqq1f/c6_predictions/predictions |
/tmp/baseline-pipeline-runs/gifted_helmet_6vkx7rqq1f/c6_evaluated/evaluated |
| Naive-search | /mnt/tmp/financebench-naive-predictions.jsonl/predictions |
/mnt/tmp/financebench-naive-evaluated.jsonl/evaluated |
| E2E v3 | /home/azureuser/projects/information-scaffolds/outputs/e2e_runs/new-datasets-full-20260711/financebench/named-outputs/predictions/predictions |
/home/azureuser/projects/information-scaffolds/outputs/e2e_runs/new-datasets-full-20260711/judges/financebench/named-outputs/judged/judged |
| E2E v3 + rawtext | AML silly_salt_1vr6cy7bw8 predictions |
Its canonical FinanceBench evaluation |
Local dev
python -m http.server 8000 # then open http://localhost:8000/
Serves statically — no build step. Push to the HF Space remote to deploy.
Source: patronus-ai/financebench (150-example open-source sample; the full FinanceBench has 10,231 questions). Dataset license: CC-BY-NC-4.0.