README.md

---
license: cc-by-sa-4.0
task_categories:
- question-answering
language:
- en
tags:
- Factuality
- Knowledge
- Wikipedia
- QA
pretty_name: wikiprofile
size_categories:
- 1K<n<10K
---

# WikiProfile

WikiProfile is a factual knowledge benchmark for evaluating how well language models encode and recall factual knowledge. It comprises 2,150 facts, each paired with 10 questions, for a total of 21,500 question instances.
Each fact is grounded in the first paragraph (summary) of an English Wikipedia page and is defined as a proposition between two entities, a subject and an object (e.g., "Oasis played their first gig at the Boardwalk club" → subject: Oasis, object: Boardwalk). The ten questions per fact span three formats: text completion (encoding), closed-book question answering (recall), and multiple-choice.
Facts are extracted from 2,125 Wikipedia pages spanning 9 topic categories and 19 entity types.

## Dataset Details

- **Dataset Name:** WikiProfile
- **Format:** CSV
- **Size:** ~5.22 MB
- **License:** [CC-BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/) (Required due to Wikipedia content source)
- **Paper:** [Empty Shelves or Lost Keys? Recall Is the Bottleneck for Parametric Factuality](https://arxiv.org/abs/2602.14080)

### Dataset Sources

- **Source Text:** English Wikipedia page summaries.
- **Popularity Metrics:** Wikimedia Analytics page view counts (aggregated over 2023–2025).
- **Generation Pipeline:** Questions, entity annotations, and multiple-choice distractors were generated using Gemini-2.5-Pro, followed by Google Search–grounded filtering and final manual validation (removing <2% of facts).

## Dataset Structure

The dataset consists of a single CSV file. Each row represents a single fact (2,150 rows total) with 21 columns containing different question formats and metadata.

### Data Fields

| Column | Type | Description |
| :--- | :--- | :--- |
| `index` | int | Row index (0-based) |
| `fact_id` | string | Unique fact identifier (format: `{page_id}__{entity_index}`) |
| `page_id` | string | Wikipedia page ID |
| `page_title` | string | Wikipedia page title |
| `item_id` | string | Wikidata item ID (Q-number) |
| `gbc` | int | Page view count (popularity metric, aggregated over 2023–2025) |
| `creation_date` | string | Wikipedia page creation date (YYYY-MM-DD) |
| `category` | string | Topic category (one of 9: Arts, Science, People, Society, Everyday Life, Technology, History, Geography, Philosophy & Religion) |
| `summary` | string | Wikipedia page first paragraph — the source text from which the fact is extracted |
| `object_index` | int | Character index of the object entity in the source text |
| `object` | string | Object entity — the gold answer for encoding and direct questions |
| `object_type` | string | Entity type of the object (one of 19 types, e.g., PERSON, LOCATION, DATE, CLASSIFICATION) |
| `subject` | string | Subject entity |
| `subject_type` | string | Entity type of the subject (one of 19 types) |
| `completion` | string | Proposition-completion text (left context up to the object entity, used as input for the encoding completion task) |
| `direct` | string | Direct closed-book question (high-verbatim phrasing) |
| `direct_answer` | string | Gold answer for the direct question (same as object) |
| `direct_natural` | string | Natural (conversational) rephrasing of the direct question |
| `direct_choices` | string | Comma-separated multiple-choice options (4 options including the correct answer) |
| `reverse` | string | Reverse closed-book question (subject and object roles swapped) |
| `reverse_answer` | string | Gold answer for the reverse question (same as subject) |
| `reverse_natural` | string | Natural (conversational) rephrasing of the reverse question |
| `reverse_choices` | string | Comma-separated multiple-choice options (4 options including the correct answer) |
| `contextual` | string | Contextual question, pappending the direct question to the left context (used for the encoding contextual task), answer is `direct_answer` |

## Dataset Creation

### Curation Rationale

WikiProfile was created to profile factual knowledge in LLMs at the level of individual facts, distinguishing between whether a fact is encoded in the model parameters and whether it can be accessed (recalled) under different conditions (verbatim vs. natural phrasing, direct vs. reverse questions, and with/without thinking time).

### Annotations

Annotations (entity selection, question generation, distractors) were automatically generated using Gemini-2.5-Pro.
- **Validation:** Multiple verification stages including Google Search-grounded filtering, followed by manual validation of the final set (which removed <2% of candidate facts).

## Additional Information

### Licensing Information

The dataset is licensed under **Creative Commons Attribution-ShareAlike 4.0 International (CC-BY-SA 4.0)**. This is compatible with Wikipedia's CC-BY-SA 3.0 license, which applies to the source text used in the summaries.

### Citation Information

If you use this dataset, please cite the associated paper:

```bibtex
@article{calderon2026empty,
  author       = {Nitay Calderon and
                  Eyal Ben{-}David and
                  Zorik Gekhman and
                  Eran Ofek and
                  Gal Yona},
  title        = {Empty Shelves or Lost Keys? Recall Is the Bottleneck for Parametric
                  Factuality},
  journal      = {CoRR},
  volume       = {abs/2602.14080},
  year         = {2026},
  url          = {https://doi.org/10.48550/arXiv.2602.14080},
  doi          = {10.48550/ARXIV.2602.14080},
  eprinttype   = {arXiv},
  eprint       = {2602.14080},
  timestamp    = {Fri, 27 Mar 2026 13:17:01 +0100},
  biburl       = {https://dblp.org/rec/journals/corr/abs-2602-14080.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}
```