Split perturbations by eval split (msmarco-dev/trec-dl-2019/trec-dl-2020), add TSV format with original+perturbed query text, add data_download scripts for MS MARCO/TREC-DL originals

Browse files

Files changed (26) hide show

LICENSE.md +21 -5
README.md +26 -8
data_download/checksums.txt +21 -0
data_download/download_msmarco.sh +32 -0
data_download/download_trec_dl_2019.sh +24 -0
data_download/download_trec_dl_2020.sh +24 -0
data_download/prepare_evaluation_data.sh +19 -0
lost_in_decoding_artifacts/README.md +62 -30
lost_in_decoding_artifacts/metadata/generation_config.json +25 -0
lost_in_decoding_artifacts/metadata/seeds.json +7 -0
lost_in_decoding_artifacts/msmarco-dev/misspelling.tsv +0 -0
lost_in_decoding_artifacts/msmarco-dev/naturality.tsv +0 -0
lost_in_decoding_artifacts/msmarco-dev/original_qids.txt +6980 -0
lost_in_decoding_artifacts/msmarco-dev/paraphrase.tsv +0 -0
lost_in_decoding_artifacts/msmarco-dev/reordering.tsv +0 -0
lost_in_decoding_artifacts/msmarco-dev/synonym.tsv +0 -0
lost_in_decoding_artifacts/trec-dl-2019/misspelling.tsv +216 -0
lost_in_decoding_artifacts/trec-dl-2019/naturality.tsv +44 -0
lost_in_decoding_artifacts/trec-dl-2019/paraphrase.tsv +216 -0
lost_in_decoding_artifacts/trec-dl-2019/reordering.tsv +216 -0
lost_in_decoding_artifacts/trec-dl-2019/synonym.tsv +216 -0
lost_in_decoding_artifacts/trec-dl-2020/misspelling.tsv +271 -0
lost_in_decoding_artifacts/trec-dl-2020/naturality.tsv +55 -0
lost_in_decoding_artifacts/trec-dl-2020/paraphrase.tsv +271 -0
lost_in_decoding_artifacts/trec-dl-2020/reordering.tsv +271 -0
lost_in_decoding_artifacts/trec-dl-2020/synonym.tsv +271 -0

LICENSE.md CHANGED Viewed

@@ -4,9 +4,16 @@ This repository contains materials with different provenance.
 ## Lost in Decoding artifacts
-Original query perturbations, metadata, documentation, and other artifacts
-created specifically for the Lost in Decoding study are released under the
-Apache License 2.0 unless otherwise indicated.
 ## PAG checkpoint and document identifiers
@@ -24,5 +31,14 @@ https://github.com/HansiZeng/PAG
 ## Source datasets
 MS MARCO, TREC Deep Learning, and mMARCO data remain subject to their respective
-original licences and terms of use. The document collection and third-party
-multilingual query datasets are not redistributed in this repository.

 ## Lost in Decoding artifacts
+Generation code, metadata, documentation, and other artifacts created
+specifically for the Lost in Decoding study are released under the Apache
+License 2.0 unless otherwise indicated.
+The query-variation TSV files under `lost_in_decoding_artifacts/` are derived
+evaluation artifacts created from the official MS MARCO Dev queries and
+official TREC Deep Learning 2019/2020 topics. The Apache License 2.0 applies
+to our generation code and metadata; it does not grant unrestricted rights
+over the underlying query text, which remains subject to the terms of the
+original MS MARCO / TREC-DL releases (see "Source datasets" below).
 ## PAG checkpoint and document identifiers
 ## Source datasets
 MS MARCO, TREC Deep Learning, and mMARCO data remain subject to their respective
+original licences and terms of use. The full document collection and
+third-party multilingual query datasets are not redistributed in this
+repository; see `data_download/` for scripts that fetch them directly from
+their official sources.
+Per Microsoft's MS MARCO terms
+(https://microsoft.github.io/msmarco/Notice.html), MS MARCO data is
+restricted to non-commercial research use and is supplied without granting a
+license or other intellectual-property rights; Microsoft also notes it may
+not own the underlying rights in the passages. Users of the query-variation
+files in this repository should independently review those terms.

README.md CHANGED Viewed

@@ -44,11 +44,29 @@ the *Lost in Decoding* study.
 | `pytorch_model.bin`, `config.json`, and tokenizer files | Zeng, Luo, and Zamani (2024) | Original PAG checkpoint, `lexical_ripor_direct_lng_knp_seq2seq_1`, mirrored without modification. |
 | `identifiers/rq_docids.json` | Zeng, Luo, and Zamani (2024) | Sequential residual-quantization document identifiers, originally released as `aq_smtid/docid_to_tokenids.json`. |
 | `identifiers/set_docids.json` | Zeng, Luo, and Zamani (2024) | Set-based lexical document identifiers, originally released as `top_bow/docid_to_tokenids.json`. |
-| `lost_in_decoding_artifacts/perturbed_queries/` | *Lost in Decoding* authors | Perturbed MS MARCO Dev queries used for the RQ2 robustness experiments. |
 | `lost_in_decoding_artifacts/metadata/` | *Lost in Decoding* authors | Perturbation types, generation settings, and random seeds. |
-Multilingual (mMARCO) queries used for RQ3 are **not** re-hosted here; they are
-third-party data. See the *Source data* section below.
 ## Model
@@ -116,16 +134,16 @@ for details.
 ## Evaluation artifacts (`lost_in_decoding_artifacts/`)
-Perturbed MS MARCO Dev queries used in the RQ2 robustness evaluation
-(Table 6 in the paper, corrected — see the main repository README for the
-correction notice). See
 [`lost_in_decoding_artifacts/README.md`](lost_in_decoding_artifacts/README.md)
 for format and generation details.
 ## Source data
-- **MS MARCO** passage ranking dataset (queries, qrels, collection): [microsoft/MSMARCO](https://microsoft.github.io/msmarco/) — used under its original terms.
-- **TREC Deep Learning 2019 / 2020** query sets and qrels: [NIST TREC-DL](https://microsoft.github.io/msmarco/TREC-Deep-Learning.html).
 - **mMARCO** (multilingual queries, RQ3): [unicamp-dl/mmarco](https://huggingface.co/datasets/unicamp-dl/mmarco) — not redistributed here; download via `cross_lingual/scripts/download_mmarco.sh` in the code repository.
 ## Citation

 | `pytorch_model.bin`, `config.json`, and tokenizer files | Zeng, Luo, and Zamani (2024) | Original PAG checkpoint, `lexical_ripor_direct_lng_knp_seq2seq_1`, mirrored without modification. |
 | `identifiers/rq_docids.json` | Zeng, Luo, and Zamani (2024) | Sequential residual-quantization document identifiers, originally released as `aq_smtid/docid_to_tokenids.json`. |
 | `identifiers/set_docids.json` | Zeng, Luo, and Zamani (2024) | Set-based lexical document identifiers, originally released as `top_bow/docid_to_tokenids.json`. |
+| `lost_in_decoding_artifacts/msmarco-dev/` | *Lost in Decoding* authors | Query variations for the MS MARCO Passage Ranking Dev set. |
+| `lost_in_decoding_artifacts/trec-dl-2019/` | *Lost in Decoding* authors | Query variations for the TREC Deep Learning 2019 Passage Ranking topics. |
+| `lost_in_decoding_artifacts/trec-dl-2020/` | *Lost in Decoding* authors | Query variations for the TREC Deep Learning 2020 Passage Ranking topics. |
 | `lost_in_decoding_artifacts/metadata/` | *Lost in Decoding* authors | Perturbation types, generation settings, and random seeds. |
+| `data_download/` | *Lost in Decoding* authors | Scripts for downloading the original MS MARCO collection, official queries, and qrels from their upstream providers. |
+The query-variation files are derived evaluation artifacts created from the
+official MS MARCO Dev queries and official TREC Deep Learning 2019/2020
+topics — see [`LICENSE.md`](LICENSE.md).
+We do **not** redistribute the full MS MARCO passage collection or the
+original MS MARCO/TREC-DL queries and qrels in this repository (their
+redistribution rights are not clearly granted — see *Source data* below).
+Use `data_download/` to fetch them directly from Microsoft/NIST:
+```bash
+git clone https://huggingface.co/kiyam/lost-in-decoding-pag
+cd lost-in-decoding-pag
+bash data_download/prepare_evaluation_data.sh
+```
+Multilingual (mMARCO) queries used for RQ3 are **not** re-hosted here either;
+they are third-party data. See the *Source data* section below.
 ## Model
 ## Evaluation artifacts (`lost_in_decoding_artifacts/`)
+Query variations (MS MARCO Dev, TREC-DL 2019, TREC-DL 2020) used in the RQ2
+robustness evaluation (Table 6 in the paper, corrected — see the main
+repository README for the correction notice). See
 [`lost_in_decoding_artifacts/README.md`](lost_in_decoding_artifacts/README.md)
 for format and generation details.
 ## Source data
+- **MS MARCO** passage ranking dataset (queries, qrels, collection): [microsoft/MSMARCO](https://microsoft.github.io/msmarco/) — non-commercial research use only, supplied without granting a license or other intellectual-property rights ([Terms and Conditions](https://microsoft.github.io/msmarco/Notice.html)). Not redistributed here; fetch via `data_download/download_msmarco.sh`.
+- **TREC Deep Learning 2019 / 2020** query sets and qrels: [NIST TREC-DL](https://trec.nist.gov/data/deep2019.html). Not redistributed here; fetch via `data_download/download_trec_dl_2019.sh` / `download_trec_dl_2020.sh`.
 - **mMARCO** (multilingual queries, RQ3): [unicamp-dl/mmarco](https://huggingface.co/datasets/unicamp-dl/mmarco) — not redistributed here; download via `cross_lingual/scripts/download_mmarco.sh` in the code repository.
 ## Citation

data_download/checksums.txt ADDED Viewed

	@@ -0,0 +1,21 @@

+# SHA-256 checksums for files downloaded by download_msmarco.sh,
+# download_trec_dl_2019.sh, and download_trec_dl_2020.sh (as fetched, before
+# gunzip/tar extraction). Each script verifies its own downloads inline
+# using these same values; this file is a standalone reference copy.
+#
+# Verified 2026-08-02 against:
+#   https://msmarco.z22.web.core.windows.net/msmarcoranking/ (Microsoft)
+#   https://trec.nist.gov/data/deep/ (NIST)
+# download_msmarco.sh
+70667529e474322327d6441c8f7b621f2a805a7f6220897d9f80e5a8294bd62e  collection.tar.gz
+05e4c62c9c8520cd695725340d4c990627fd1a92eb8a71b88e3c746031ba6de8  queries.tar.gz
+a1befe0a99974b166b9e09408ccccae471fda5df4953fd6dd86ce4caac691752  qrels.dev.tsv
+# download_trec_dl_2019.sh
+8a1f10d550732e4cd91d7fc49846a3784de4040972f583e69285a88f3c5fee92  2019qrels-pass.txt
+d66f01dfa8a3e60fe375ba645efc004f97ff4dd2a424536b8d5810c16182a3b5  msmarco-test2019-queries.tsv.gz
+# download_trec_dl_2020.sh
+60d4c34561f9687f8f73e0a752ba80ab80159b3e9c1bedd0a351f747ed6f5684  2020qrels-pass.txt
+0c749b29fd8ca0ccbf129a1bdd283e4081d6c49e96d6bf6290ba554d5d71f7f1  msmarco-test2020-queries.tsv.gz

data_download/download_msmarco.sh ADDED Viewed

	@@ -0,0 +1,32 @@

+#!/usr/bin/env bash
+# Download the official MS MARCO Passage Ranking collection and Dev
+# queries/qrels from Microsoft's release. Subject to the MS MARCO terms:
+# https://microsoft.github.io/msmarco/Notice.html
+set -euo pipefail
+OUT_DIR="${1:-./msmarco}"
+BASE_URL="https://msmarco.z22.web.core.windows.net/msmarcoranking"
+mkdir -p "$OUT_DIR"
+cd "$OUT_DIR"
+echo "Downloading MS MARCO passage collection (~1GB compressed)..."
+curl -L -o collection.tar.gz "$BASE_URL/collection.tar.gz"
+echo "Downloading MS MARCO Dev queries..."
+curl -L -o queries.tar.gz "$BASE_URL/queries.tar.gz"
+echo "Downloading MS MARCO Dev qrels..."
+curl -L -o qrels.dev.tsv "$BASE_URL/qrels.dev.tsv"
+echo "Verifying checksums (see ../checksums.txt)..."
+sha256sum -c - <<'EOF'
+70667529e474322327d6441c8f7b621f2a805a7f6220897d9f80e5a8294bd62e  collection.tar.gz
+05e4c62c9c8520cd695725340d4c990627fd1a92eb8a71b88e3c746031ba6de8  queries.tar.gz
+a1befe0a99974b166b9e09408ccccae471fda5df4953fd6dd86ce4caac691752  qrels.dev.tsv
+EOF
+tar -xzf collection.tar.gz
+tar -xzf queries.tar.gz
+echo "Done. Files written to: $OUT_DIR"

data_download/download_trec_dl_2019.sh ADDED Viewed

	@@ -0,0 +1,24 @@

+#!/usr/bin/env bash
+# Download the official TREC Deep Learning 2019 passage-ranking topics and
+# qrels, as published by NIST: https://trec.nist.gov/data/deep2019.html
+set -euo pipefail
+OUT_DIR="${1:-./trec-dl-2019}"
+mkdir -p "$OUT_DIR"
+cd "$OUT_DIR"
+echo "Downloading TREC-DL 2019 passage-ranking qrels..."
+curl -L -o qrels-pass.txt "https://trec.nist.gov/data/deep/2019qrels-pass.txt"
+echo "Downloading TREC-DL 2019 test queries..."
+curl -L -o queries.tsv.gz "https://msmarco.z22.web.core.windows.net/msmarcoranking/msmarco-test2019-queries.tsv.gz"
+echo "Verifying checksums (see ../checksums.txt)..."
+sha256sum -c - <<'EOF'
+8a1f10d550732e4cd91d7fc49846a3784de4040972f583e69285a88f3c5fee92  qrels-pass.txt
+d66f01dfa8a3e60fe375ba645efc004f97ff4dd2a424536b8d5810c16182a3b5  queries.tsv.gz
+EOF
+gunzip -f queries.tsv.gz
+echo "Done. Files written to: $OUT_DIR"

data_download/download_trec_dl_2020.sh ADDED Viewed

	@@ -0,0 +1,24 @@

+#!/usr/bin/env bash
+# Download the official TREC Deep Learning 2020 passage-ranking topics and
+# qrels, as published by NIST: https://trec.nist.gov/data/deep2020.html
+set -euo pipefail
+OUT_DIR="${1:-./trec-dl-2020}"
+mkdir -p "$OUT_DIR"
+cd "$OUT_DIR"
+echo "Downloading TREC-DL 2020 passage-ranking qrels..."
+curl -L -o qrels-pass.txt "https://trec.nist.gov/data/deep/2020qrels-pass.txt"
+echo "Downloading TREC-DL 2020 test queries..."
+curl -L -o queries.tsv.gz "https://msmarco.z22.web.core.windows.net/msmarcoranking/msmarco-test2020-queries.tsv.gz"
+echo "Verifying checksums (see ../checksums.txt)..."
+sha256sum -c - <<'EOF'
+60d4c34561f9687f8f73e0a752ba80ab80159b3e9c1bedd0a351f747ed6f5684  qrels-pass.txt
+0c749b29fd8ca0ccbf129a1bdd283e4081d6c49e96d6bf6290ba554d5d71f7f1  queries.tsv.gz
+EOF
+gunzip -f queries.tsv.gz
+echo "Done. Files written to: $OUT_DIR"

data_download/prepare_evaluation_data.sh ADDED Viewed

	@@ -0,0 +1,19 @@

+#!/usr/bin/env bash
+# Download all original (non-redistributed) evaluation data used by the
+# Lost-in-Decoding / PAG reproduction pipeline: MS MARCO passage collection
+# + Dev queries/qrels, and the TREC-DL 2019/2020 topics + qrels.
+#
+# Each of these datasets is subject to its own upstream terms of use
+# (see the main README.md "Source data" section) and is not redistributed
+# in this repository.
+set -euo pipefail
+OUT_DIR="${1:-.}"
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+bash "$SCRIPT_DIR/download_msmarco.sh" "$OUT_DIR/msmarco"
+bash "$SCRIPT_DIR/download_trec_dl_2019.sh" "$OUT_DIR/trec-dl-2019"
+bash "$SCRIPT_DIR/download_trec_dl_2020.sh" "$OUT_DIR/trec-dl-2020"
+echo ""
+echo "All evaluation data downloaded and checksum-verified under: $OUT_DIR"

lost_in_decoding_artifacts/README.md CHANGED Viewed

@@ -1,44 +1,76 @@
 # Lost in Decoding — Evaluation Artifacts
-**Provenance:** created for this paper (*Lost in Decoding? Reproducing and
-Stress-Testing the Look-Ahead Prior in Generative Retrieval*, SIGIR 2026,
 [DOI: 10.1145/3805712.3808567](https://doi.org/10.1145/3805712.3808567) ·
 [arXiv:2604.23396](https://arxiv.org/abs/2604.23396)), not part of the
 original PAG release.
-## `perturbed_queries/`
-Perturbed MS MARCO Dev query sets used for the RQ2 robustness evaluation
-(query-perturbation sensitivity of PAG's look-ahead prior).
-- **Split:** MS MARCO Dev (6980 queries).
-- **Attack methods:** `mispelling`, `ordering` (word reordering), `synonym`
-  (synonym replacement), `paraphrase`, `naturality` (naturality
-  normalization). Implemented in
-  [`robustness/query_variations/penha/`](https://github.com/kidist-amde/lost-in-decoding/tree/main/robustness/query_variations/penha)
-  in the code repository.
-- **Seeds:** `1999`, `5`, `27`, `2016`, `2026`.
-- **File naming:** `msmarco_test_attacked_queries_seed_{seed}_attack_method_{attack_method}.json`
-- **Format:** JSON object, `{qid (str): perturbed_query_text (str)}`.
-- **Note (`naturality`):** this transformation is a deterministic
-  style-normalization rule rather than a randomized perturbation, so the five
-  seed files for `naturality` are identical to each other by construction.
-**TREC-DL 2019 / TREC-DL 2020 perturbed queries are not included as static
-files.** They are produced by the same code path
-(`robustness/scripts/generate_query_variations_sub.sh`) using the same seeds
-and attack methods, and were consumed directly during RQ2 evaluation without
-being cached to disk. Regenerate them with:
-```bash
-bash robustness/scripts/generate_query_variations_sub.sh
 ```
 ## `metadata/`
-`perturbed_queries_metadata.json` documents the generation settings (split,
-seeds, attack methods, file naming convention) for the released perturbed
-query sets.
 ## Citation

 # Lost in Decoding — Evaluation Artifacts
+**Provenance:** derived evaluation artifacts created from the official MS
+MARCO Dev queries and official TREC Deep Learning 2019/2020 topics, for this
+paper (*Lost in Decoding? Reproducing and Stress-Testing the Look-Ahead Prior
+in Generative Retrieval*, SIGIR 2026,
 [DOI: 10.1145/3805712.3808567](https://doi.org/10.1145/3805712.3808567) ·
 [arXiv:2604.23396](https://arxiv.org/abs/2604.23396)), not part of the
 original PAG release.
+The generation code and metadata here are Apache-2.0-licensed. The perturbed
+query **text** is a derived artifact of the original MS MARCO / TREC-DL query
+text and remains subject to the same provenance constraints as that source
+text — see [`../LICENSE.md`](../LICENSE.md).
+## `msmarco-dev/`, `trec-dl-2019/`, `trec-dl-2020/`
+Query variations (misspelling, reordering, synonym replacement, paraphrasing,
+naturality normalization) used for the RQ2 robustness evaluation, one
+directory per evaluation split:
+| Directory | Split | Queries |
+|---|---|---|
+| `msmarco-dev/` | MS MARCO Passage Ranking Dev | 6,980 |
+| `trec-dl-2019/` | TREC Deep Learning 2019 passage-ranking topics | 43 |
+| `trec-dl-2020/` | TREC Deep Learning 2020 passage-ranking topics | 54 |
+Each directory contains one TSV file per perturbation type:
+`misspelling.tsv`, `reordering.tsv`, `synonym.tsv`, `paraphrase.tsv`,
+`naturality.tsv`.
+**Columns:** `qid`, `original_query`, `perturbed_query`, `perturbation`, `seed`
+```text
+qid    original_query    perturbed_query    perturbation    seed
+1037798    who is robert gray    who is robrt gray    misspelling    42
 ```
+- **Seeded perturbations** (`misspelling`, `reordering`, `synonym`,
+  `paraphrase`): five seeds — `1999`, `5`, `27`, `2016`, `2026` — stacked as
+  separate rows per `qid` within the same file.
+- **`naturality`:** a deterministic style-normalization rule rather than a
+  randomized perturbation, so it produces identical output for every seed.
+  Represented as a single row per `qid` with `seed=deterministic` rather than
+  five duplicate rows.
+`msmarco-dev/original_qids.txt` lists the 6,980 Dev query IDs covered by
+these files (a subset of the full MS MARCO Dev qrel set).
+Implemented in
+[`robustness/query_variations/penha/`](https://github.com/kidist-amde/lost-in-decoding/tree/main/robustness/query_variations/penha)
+in the code repository.
 ## `metadata/`
+- `generation_config.json`: perturbation types, split sizes, file format, and provenance.
+- `seeds.json`: seed values and which perturbations use them.
+## `data_download/`
+The original MS MARCO passage collection, MS MARCO Dev queries/qrels, and
+TREC-DL 2019/2020 topics/qrels are **not** redistributed in this repository.
+Scripts to fetch them directly from their official sources (Microsoft,
+NIST) are provided at the repository root:
+```bash
+git clone https://huggingface.co/kiyam/lost-in-decoding-pag
+cd lost-in-decoding-pag
+bash data_download/prepare_evaluation_data.sh
+```
+See [`../data_download/`](../data_download/) for individual per-dataset
+scripts and checksums.
 ## Citation

lost_in_decoding_artifacts/metadata/generation_config.json ADDED Viewed

	@@ -0,0 +1,25 @@

+{
+  "description": "Generation settings for the query-variation TSV files under msmarco-dev/, trec-dl-2019/, and trec-dl-2020/, released alongside 'Lost in Decoding? Reproducing and Stress-Testing the Look-Ahead Prior in Generative Retrieval' (SIGIR 2026, DOI: 10.1145/3805712.3808567).",
+  "splits": {
+    "msmarco-dev": {"num_queries": 6980, "source": "MS MARCO Passage Ranking Dev queries"},
+    "trec-dl-2019": {"num_queries": 43, "source": "TREC Deep Learning 2019 passage-ranking topics"},
+    "trec-dl-2020": {"num_queries": 54, "source": "TREC Deep Learning 2020 passage-ranking topics"}
+  },
+  "perturbation_types": [
+    {"name": "misspelling", "seeded": true, "description": "Character-level misspelling injection."},
+    {"name": "reordering", "seeded": true, "description": "Word-order shuffling."},
+    {"name": "synonym", "seeded": true, "description": "Adversarial synonym replacement."},
+    {"name": "paraphrase", "seeded": true, "description": "Back-translation and seq2seq paraphrasing."},
+    {"name": "naturality", "seeded": false, "description": "Deterministic style-normalization (stop-word removal); identical output regardless of seed."}
+  ],
+  "file_format": {
+    "extension": "tsv",
+    "columns": ["qid", "original_query", "perturbed_query", "perturbation", "seed"],
+    "seeded_attacks": "one row per (qid, seed) pair, five seeds stacked in a single file",
+    "naturality": "one row per qid; seed column is the literal string 'deterministic'"
+  },
+  "generation_code": "robustness/query_variations (penha transformations) in the Lost-in-Decoding repository: https://github.com/kidist-amde/lost-in-decoding/tree/main/robustness/query_variations",
+  "provenance": "Derived evaluation artifacts created from the official MS MARCO Dev queries and official TREC Deep Learning 2019/2020 topics for the Lost in Decoding study.",
+  "source_paper": "Lost in Decoding? Reproducing and Stress-Testing the Look-Ahead Prior in Generative Retrieval (DOI: 10.1145/3805712.3808567, arXiv:2604.23396)",
+  "source_repository": "https://github.com/kidist-amde/lost-in-decoding"
+}

lost_in_decoding_artifacts/metadata/seeds.json ADDED Viewed

	@@ -0,0 +1,7 @@

+{
+  "seeded_perturbations": ["misspelling", "reordering", "synonym", "paraphrase"],
+  "seeds": [1999, 5, 27, 2016, 2026],
+  "unseeded_perturbations": {
+    "naturality": "Deterministic transformation (stop-word removal). Produces identical output regardless of seed; represented in the TSV files with seed='deterministic' rather than being duplicated five times."
+  }
+}

lost_in_decoding_artifacts/msmarco-dev/misspelling.tsv ADDED Viewed