tceron/open-endedness-classifier

Model Card for Model ID

This model classifies user interaction queries between Analytical and Factoid.

Label IDs:

0: "Analytical"
1: "Factoid"

Model Details

All details in paper:

@inproceedings{ceron-etal-2026-wildseek,
    title = "{W}ild{SEEK}: Evaluating Language Models for Information-Seeking",
    author = "Ceron, Tanise  and
      Baumann, Joachim  and
      Bassignana, Elisa  and
      Cabuk, Berat  and
      Hovy, Dirk  and
      Nozza, Debora",
    editor = "Lee, Hung-Yi  and
      Dev, Sunipa  and
      Li, Jessy  and
      Elliott, Desmond",
    booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
    month = oct,
    year = "2026",
    address = "Budapest, Hungary",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.emnlp-main.763/",
    doi = "10.18653/2026.emnlp-main.763",
    pages = "17690--17718",
    ISBN = "979-8-89176-459-0",
    abstract = "Language models are increasingly mediating information access to end users, urging a systematic evaluation of their responses for a fair and reliable information ecosystem. Existing evaluations, however, are often topic-specific or synthetic, limiting their ability to capture the complexity of ``in the wild'' information-seeking queries and the risks present in model responses. To address this gap, we introduce WildSEEK, a manually annotated dataset of 3k information-seeking queries from real user interactions, and an evaluation framework for LLM-generated responses. WildSEEK includes annotations for risk-sensitive domains (e.g. health and financial information), and distinguishes factoid queries from analytical queries which seek responses beyond facts. We train classifiers on WildSEEK to analyze more than 1.8M realistic user queries. We find that over a third of information-seeking queries are high-risk and more often analytical. Our findings show that LLM responses fail more often in four criteria: sycophantic behavior, overreliance, a default US-centric perspective, and poor handling of vulnerable populations {---} with failure rates being mostly higher for analytical queries. By providing methods to monitor the reliability, safety, and fairness of LLM behavior, our dataset and evaluation framework offer an empirical foundation for the broader question of how these systems should behave as they take on a growing role in information access."
}

Other classifiers from this study are:

• tceron/info-seek-classifier — information seeking, content creation, coding, not english, no request
• tceron/high-risk-classifier — Other, Economic and Financial, Health, Politics, Judicial and Legal, Moral Values and Religion, Security