Spaces:
Running
title: HERB Viewer
emoji: 🏢
colorFrom: blue
colorTo: indigo
sdk: static
pinned: false
license: cc-by-nc-4.0
🏢 HERB Viewer
Static viewer for HERB — Benchmarking Deep Search over Heterogeneous Enterprise Data: multi-hop, source-aware RAG over diverse enterprise artifacts (Slack, meeting transcripts/chats, documents, URLs, GitHub PRs).
- Paper: https://arxiv.org/abs/2506.23139
- Code: https://github.com/SalesforceAIResearch/HERB
- Dataset: https://huggingface.co/datasets/Salesforce/HERB
Tabs:
- 📚 Corpus — the 38,600 enterprise artifacts across 30 products. Pick a product and an artifact type (Slack / Documents / Meeting transcripts / Meeting chats / URLs / Pull requests), then a dropdown selects an artifact by a constructed title. Each artifact renders in its native shape (Slack message + thread, document markdown, meeting transcript, PR summary + reviews, URL), with employee/customer ids resolved to names.
- ❓ Eval — the 1,514 questions (815 answerable + 699 unanswerable). Filter by product, answerable/unanswerable, and question type (person / content / company / pr / url), plus whether the question requires computation/comparison. Each shows the ground-truth answer and evidence-citation chips (the artifact ids that support it) that jump straight to that artifact in the Corpus tab.
- E2E Structures v3 — the five constructed scaffold shapes from the canonical combined-corpus E2E v3 run. Supporting artifacts are mapped back to the packed extraction bundle that produced each structure.
- E2E+Rawtext Structures — the same cached five-shape scaffold used by the overlay run, with raw HERB documents mounted as a sixth fallback source.
- c1 Closed-book, c2 With-docs, c6 Agentic-DCI, Naive-search, E2E raw, E2E combined, and E2E v3 + rawtext — normalized full-run responses and judge outcomes. These tabs intentionally include only the 815 answerable questions and make missing/unanswered predictions explicit. Search and filter by product, question type, computation/comparison category, and correctness status; agentic tabs also expose compact, expandable event traces.
- ⚖ Compare — the same 815 canonical qids prejoined across all seven runs, with score/coverage cards and side-by-side predictions. Filters include disagreement, any missing response, and questions where only E2E combined is correct.
Question selection follows normalized Product_aN / Product#aN ids across
Eval, run, Compare, and E2E Structures tabs. Agent traces pair tool calls with
their results, collapse long outputs, render gold answers as compact pills, and
keep E2E evidence citations closed until requested.
Full-run results
Every run uses the canonical type-aware judge_score: normalized set F1 for
person/company/PR/URL questions and the normalized 0-100 reference score for
content questions. Missing or unanswered questions receive zero. Coverage is
the number with a non-empty model response.
| Run | Perfect / 815 | Type-aware score | Answered / 815 | Coverage |
|---|---|---|---|---|
| c1 Closed-book | 0 | 4.25% | 815 | 100.00% |
| c2 With-docs | 385 | 67.96% | 815 | 100.00% |
| c6 Agentic-DCI | 386 | 75.64% | 792 | 97.18% |
| Naive-search | 178 | 45.96% | 547 | 67.12% |
| E2E raw | 284 | 67.87% | 798 | 97.91% |
| E2E combined | 336 | 73.91% | 813 | 99.75% |
| E2E v3 + rawtext | 378 | 77.63% | 812 | 99.63% |
Computation / comparison breakdown
The split marks questions that ask for a numeric maximum/minimum or comparison, or require arithmetic aggregation over retrieved documents. In this HERB release, every positive template is a maximum/highest/most aggregation; there are no separate minimum, numeric-comparison, or arithmetic-only templates.
| Bracket | Questions | Share |
|---|---|---|
| Computation / comparison | 130 | 15.95% |
| Other | 685 | 84.05% |
Every score is the mean canonical type-aware judge score over the bracket denominator. Coverage is shown in parentheses.
| Run | Computation / comparison | Other | Score gap |
|---|---|---|---|
| c1 Closed-book | 0.00% (100.00%) | 5.05% (100.00%) | -5.05 pp |
| c2 With-docs | 50.10% (100.00%) | 71.35% (100.00%) | -21.25 pp |
| c6 Agentic-DCI | 79.50% (96.92%) | 74.91% (97.23%) | +4.59 pp |
| Naive-search | 18.21% (20.00%) | 51.22% (76.06%) | -33.01 pp |
| E2E raw | 71.64% (96.15%) | 67.16% (98.25%) | +4.48 pp |
| E2E combined | 72.83% (100.00%) | 74.12% (99.71%) | -1.29 pp |
| E2E v3 + rawtext | 78.05% (99.23%) | 77.55% (99.71%) | +0.50 pp |
Data
| File | Contents |
|---|---|
eval.json |
1,514 derived question records (Eval tab) |
computation_comparison_report.json |
Bracket percentages and canonical per-run performance |
products.json |
30 products with per-type artifact counts (drives the filters) |
employees.json |
530 eid_… → {name, role, org} (from metadata/employee.json) |
customers.json |
120 customer profiles (from metadata/customers_data.json) |
runs/manifest.json |
Seven-run registry with scores and coverage |
runs/<slot>/index.json |
Search/filter metadata for all 815 answerable qids |
runs/<slot>/records/<qid>.json |
Normalized response, judge result, usage, and compact events |
compare/index.json |
Compare filters and seven-run summaries |
compare/records/<qid>.json |
All seven normalized outcomes prejoined by canonical qid |
e2e_structures_v3/ |
815 compressed per-question combined E2E v3 structure shards |
Derived files are regenerated with:
python scripts/build_data.py # reads the HERB products/ + metadata/
python scripts/build_runs.py # reads the response/judge JSONLs listed below
python scripts/analyze_computation_comparison.py
python3 scripts/build_e2e_structures.py \
--dataset herb \
--unified /mnt/ramdisk/blobstore/timchen0618/data/eval/herb/unified/test_with_chunks.unified.jsonl \
--scaffolds-dir /home/azureuser/projects/information-scaffolds/outputs/e2e_runs/new-datasets-full-20260711/herb_combined/named-outputs/scaffolds_dir \
--bundle-manifest /mnt/tmp/information-scaffolds-e2e-new-datasets/herb_combined/manifest.jsonl
build_runs.py streams the large JSONLs, overlays the c1/c2 higher-token
recovery responses on their original full-run files, filters combined files to
dataset == "herb", and joins every run against the answerable rows in
eval.json. Eval gids such as Product#aN are normalized exactly once to the
run qid Product_aN. Repeated system/user prompts are discarded; individual
strings are capped at 8,192 characters and compact event payloads near 256 KiB
per record.
Run provenance:
- c1/c2 responses: original full-run rows under
information-scaffolds/outputs/herb_phantom_full/, overlaid by their higher-token rows underinformation-scaffolds/outputs/herb_phantom_recover/ - c6/naive responses:
information-scaffolds/outputs/herb_phantom_full/ - c1/c2/c6/naive type-aware judges:
information-scaffolds/outputs/herb_type_aware_judge_20260715/ - E2E raw and combined responses/judges:
responses under
information-scaffolds/outputs/e2e_runs/new-datasets-full-20260711/and type-aware judges underinformation-scaffolds/outputs/herb_type_aware_judge_20260715/ - E2E v3 + rawtext: AML
kind_hand_jy66s6xth0predictions and canonical type-aware evaluation
The six rejudge jobs were jolly_soccer_y3q5z6y4lk (c1),
ivory_mango_jgkcpvqbl7 plus recovery polite_shoe_f4zz9xccjv (c2),
honest_truck_yfv0jfb1d3 (c6), ivory_tray_rmj660nn66 (naive),
bubbly_lychee_qgzkwj17n6 (E2E raw), and bubbly_match_5xbl28h7h2
(E2E combined). The earlier E2E tabs are distinct: raw uses the raw-corpus
run, while combined uses the combined-structures run.
Artifact content is not bundled. The 30 product JSONs (~30 MB) already live
on the Salesforce/HERB dataset, which serves them with permissive CORS, so the
viewer fetch()es each product on demand (cached) and renders its artifacts
client-side — the Space only bundles the small derived files above.
Local dev
python -m http.server 8000 # then open http://localhost:8000/
Serves statically — no build step. Push to the HF Space remote to deploy.
Dataset license: CC-BY-NC-4.0. HERB was generated using GPT-4o.