README.md

---
license: other
license_name: mixed
license_link: https://huggingface.co/kiyam/lost-in-decoding-pag/blob/main/LICENSE.md
library_name: transformers
tags:
  - generative-retrieval
  - information-retrieval
  - t5
  - msmarco
  - robustness
  - reproducibility
---

# Lost in Decoding? — PAG Checkpoint and Evaluation Artifacts

This repository hosts the model checkpoint, document identifiers, and
evaluation artifacts used in:

> **Lost in Decoding? Reproducing and Stress-Testing the Look-Ahead Prior in
> Generative Retrieval**
> Proceedings of the 49th International ACM SIGIR Conference on Research and
> Development in Information Retrieval, 2026, pp. 2994–3005.
> [Paper](https://doi.org/10.1145/3805712.3808567) ·
> [arXiv](https://arxiv.org/abs/2604.23396) ·
> [Code](https://github.com/kidist-amde/lost-in-decoding)

This work is an inference-time reproducibility and robustness study of
**Planning Ahead in Generative Retrieval (PAG)**. It does not introduce a newly
trained checkpoint.

The model checkpoint and document identifiers are unmodified artifacts
originally released by the PAG authors, Hansi Zeng, Chen Luo, and Hamed Zamani.
They are mirrored here to support reproducibility of the exact system evaluated
in our study.

The perturbed-query evaluation sets and associated metadata were created for
the *Lost in Decoding* study.

## Repository contents and provenance

| Artifact | Original creator | Description |
|---|---|---|
| `pytorch_model.bin`, `config.json`, and tokenizer files | Zeng, Luo, and Zamani (2024) | Original PAG checkpoint, `lexical_ripor_direct_lng_knp_seq2seq_1`, mirrored without modification. |
| `identifiers/rq_docids.json` | Zeng, Luo, and Zamani (2024) | Sequential residual-quantization document identifiers, originally released as `aq_smtid/docid_to_tokenids.json`. |
| `identifiers/set_docids.json` | Zeng, Luo, and Zamani (2024) | Set-based lexical document identifiers, originally released as `top_bow/docid_to_tokenids.json`. |
| `lost_in_decoding_artifacts/msmarco-dev/` | *Lost in Decoding* authors | Query variations for the MS MARCO Passage Ranking Dev set. |
| `lost_in_decoding_artifacts/trec-dl-2019/` | *Lost in Decoding* authors | Query variations for the TREC Deep Learning 2019 Passage Ranking topics. |
| `lost_in_decoding_artifacts/trec-dl-2020/` | *Lost in Decoding* authors | Query variations for the TREC Deep Learning 2020 Passage Ranking topics. |
| `lost_in_decoding_artifacts/metadata/` | *Lost in Decoding* authors | Perturbation types, generation settings, and random seeds. |
| `data_download/` | *Lost in Decoding* authors | Scripts for downloading the original MS MARCO collection, official queries, and qrels from their upstream providers. |

The query-variation files are derived evaluation artifacts created from the
official MS MARCO Dev queries and official TREC Deep Learning 2019/2020
topics — see [`LICENSE.md`](LICENSE.md).

We do **not** redistribute the full MS MARCO passage collection or the
original MS MARCO/TREC-DL queries and qrels in this repository (their
redistribution rights are not clearly granted — see *Source data* below).
Use `data_download/` to fetch them directly from Microsoft/NIST:

```bash
git clone https://huggingface.co/kiyam/lost-in-decoding-pag
cd lost-in-decoding-pag
bash data_download/prepare_evaluation_data.sh
```

Multilingual (mMARCO) queries used for RQ3 are **not** re-hosted here either;
they are third-party data. See the *Source data* section below.

## Model

- **Architecture:** `T5ForLexicalSemanticGeneration`, based on T5-base.
- **Weight file:** `pytorch_model.bin`
- **Hosted at:** [https://huggingface.co/kiyam/lost-in-decoding-pag/blob/main/pytorch_model.bin](https://huggingface.co/kiyam/lost-in-decoding-pag/blob/main/pytorch_model.bin)
- **Serialization format:** PyTorch checkpoint (original `transformers` 4.17-era `pytorch_model.bin`).
- **SafeTensors conversion:** none.
- **Retrieval procedure:** PAG first performs simultaneous lexical scoring over
  set-based document identifiers. The resulting document-level planning scores
  guide trie-constrained autoregressive decoding of sequential document
  identifiers.
- **Training collection:** MS MARCO Passage Ranking.
- **Important:** The custom `T5ForLexicalSemanticGeneration` class is not part
  of the standard Transformers library — it is defined in the
  [Lost-in-Decoding repository](https://github.com/kidist-amde/lost-in-decoding/blob/main/t5_pretrainer/modeling/t5_generative_retriever.py).

## Files

- Hugging Face model repository:
  [https://huggingface.co/kiyam/lost-in-decoding-pag](https://huggingface.co/kiyam/lost-in-decoding-pag)
- Weight file:
  [`pytorch_model.bin`](https://huggingface.co/kiyam/lost-in-decoding-pag/blob/main/pytorch_model.bin)
  ([direct download](https://huggingface.co/kiyam/lost-in-decoding-pag/resolve/main/pytorch_model.bin))

Use the **repo ID** (`kiyam/lost-in-decoding-pag`), not the `/blob/main/...`
page URL, for `from_pretrained()` and programmatic downloads.

## Usage

```python
from transformers import AutoConfig, AutoTokenizer
from t5_pretrainer.modeling.t5_generative_retriever import (
    T5ForLexicalSemanticGeneration,
)

repo_id = "kiyam/lost-in-decoding-pag"

config = AutoConfig.from_pretrained(repo_id)
tokenizer = AutoTokenizer.from_pretrained(repo_id)

model = T5ForLexicalSemanticGeneration.from_pretrained(
    repo_id,
    config=config,
)
```

`T5ForLexicalSemanticGeneration` comes from the
[Lost-in-Decoding repository](https://github.com/kidist-amde/lost-in-decoding)
(`t5_pretrainer/modeling/t5_generative_retriever.py`) — install/clone that
repository so the class is importable. `from_pretrained` resolves `repo_id`
against the Hub and downloads `config.json` + `pytorch_model.bin`
automatically; it does not accept the `/blob/main/...` file page URL.

See the [Lost-in-Decoding repository](https://github.com/kidist-amde/lost-in-decoding)
for the full two-stage constrained-decoding pipeline (lexical planning →
sequential constrained beam search) required to actually run retrieval with
this checkpoint.

## Document identifiers (`identifiers/`)

Both files map `docid (str) -> list[int]` (token ID sequences forming each
document's identifier). See [`identifiers/README.md`](identifiers/README.md)
for details.

## Evaluation artifacts (`lost_in_decoding_artifacts/`)

Query variations (MS MARCO Dev, TREC-DL 2019, TREC-DL 2020) used in the RQ2
robustness evaluation (Table 6 in the paper, corrected — see the main
repository README for the correction notice). See
[`lost_in_decoding_artifacts/README.md`](lost_in_decoding_artifacts/README.md)
for format and generation details.

## Source data

- **MS MARCO** passage ranking dataset (queries, qrels, collection): [microsoft/MSMARCO](https://microsoft.github.io/msmarco/) — non-commercial research use only, supplied without granting a license or other intellectual-property rights ([Terms and Conditions](https://microsoft.github.io/msmarco/Notice.html)). Not redistributed here; fetch via `data_download/download_msmarco.sh`.
- **TREC Deep Learning 2019 / 2020** query sets and qrels: [NIST TREC-DL](https://trec.nist.gov/data/deep2019.html). Not redistributed here; fetch via `data_download/download_trec_dl_2019.sh` / `download_trec_dl_2020.sh`.
- **mMARCO** (multilingual queries, RQ3): [unicamp-dl/mmarco](https://huggingface.co/datasets/unicamp-dl/mmarco) — not redistributed here; download via `cross_lingual/scripts/download_mmarco.sh` in the code repository.

## Citation

If you use the perturbed-query artifacts or the RQ2/RQ3 evaluation results, please cite:

```bibtex
@inproceedings{mekonnen2026lost,
  title     = {Lost in Decoding? Reproducing and Stress-Testing the Look-Ahead Prior in Generative Retrieval},
  author    = {Mekonnen, Kidist Amde and Li, Yongkang and Tang, Yubao and Lupart, Simon and de Rijke, Maarten},
  booktitle = {Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval},
  pages     = {2994--3005},
  year      = {2026},
  doi       = {10.1145/3805712.3808567}
}
```

If you use the PAG checkpoint or document identifiers, please also cite the
original PAG paper:

```bibtex
@inproceedings{zeng2024planning,
  title     = {Planning Ahead in Generative Retrieval: Guiding Autoregressive Generation through Simultaneous Decoding},
  author    = {Zeng, Hansi and Luo, Chen and Zamani, Hamed},
  booktitle = {Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval},
  pages     = {469--480},
  year      = {2024},
  doi       = {10.1145/3626772.3657746}
}
```

- PAG paper: [arXiv:2404.14600](https://arxiv.org/abs/2404.14600)
- PAG upstream repository: [github.com/HansiZeng/PAG](https://github.com/HansiZeng/PAG/tree/main)

## License

See [`LICENSE.md`](LICENSE.md). This repository mixes materials with
different provenance and licensing: artifacts created for the *Lost in
Decoding* study are released under Apache License 2.0, while the PAG
checkpoint and document-identifier files are third-party artifacts mirrored
here without modification. The upstream PAG repository does not publish an
explicit license file, so no specific license terms are asserted for those
files beyond attribution to their original authors.