yjoonjang commited on
Commit
e3a7ffa
·
verified ·
1 Parent(s): 9d03c9c

Show usage unfolded, fold the per-dataset tables

Browse files
Files changed (1) hide show
  1. README.md +10 -10
README.md CHANGED
@@ -51,8 +51,7 @@ pip install "sentence-transformers>=5.6.1" torch
51
 
52
  The prompt template ships with the model. The default instruction is *"Given a web search query, retrieve relevant passages that answer the query"*, the same instruction used in training.
53
 
54
- <details>
55
- <summary>Sentence Transformers</summary>
56
 
57
  ```python
58
  import torch
@@ -79,10 +78,7 @@ pairs = [(query, document) for document in documents]
79
  scores = model.predict(pairs, batch_size=8)
80
  ```
81
 
82
- </details>
83
-
84
- <details>
85
- <summary>Transformers</summary>
86
 
87
  ```python
88
  import torch
@@ -130,8 +126,6 @@ for index in torch.argsort(scores, descending=True).tolist():
130
  print(f"{scores[index]:.4f}\t{documents[index]}")
131
  ```
132
 
133
- </details>
134
-
135
  ## Korean Reranking Evaluation
136
 
137
  Scores are **nDCG@10**, and throughput is pairs per second (**PPS**).
@@ -167,7 +161,8 @@ PPS is measured on a single NVIDIA RTX A6000 48GB per model and averaged over th
167
 
168
  `jinaai/jina-reranker-v3` and `jinaai/jina-reranker-v3.5` are listwise rerankers that cannot be compared under the same 8,192-token condition on MLDR, so they are excluded from the 9-subset table and their MLDR cells are left blank below.
169
 
170
- ### Per-dataset NDCG@10
 
171
 
172
  | Model | Params | Ko-StrategyQA | AutoRAGRetrieval | PublicHealthQA | BelebeleRetrieval | MIRACLRetrieval | MrTidyRetrieval | MultiLongDocRetrieval | SQuADKorV1Retrieval | LawIRKo |
173
  |---|---|---|---|---|---|---|---|---|---|---|
@@ -191,7 +186,10 @@ PPS is measured on a single NVIDIA RTX A6000 48GB per model and averaged over th
191
  | Dongjin-kr/ko-reranker | 560M | 0.8468 | 0.9014 | 0.6976 | 0.9655 | 0.8017 | 0.7772 | 0.3721 | 0.9785 | 0.7343 |
192
  | telepix/PIXIE-Spell-Reranker-Preview-0.6B | 596M | 0.8329 | 0.9794 | 0.8211 | 0.9689 | 0.8449 | 0.7650 | 0.1829 | 0.9850 | 0.6042 |
193
 
194
- ### Per-dataset PPS
 
 
 
195
 
196
  | Model | Params | Ko-StrategyQA | AutoRAGRetrieval | PublicHealthQA | BelebeleRetrieval | MIRACLRetrieval | MrTidyRetrieval | MultiLongDocRetrieval | SQuADKorV1Retrieval | LawIRKo |
197
  |---|---|---|---|---|---|---|---|---|---|---|
@@ -217,6 +215,8 @@ PPS is measured on a single NVIDIA RTX A6000 48GB per model and averaged over th
217
 
218
  Batch size starts at 8 and doubles until an out-of-memory error. Samples are repeated to fill complete batches. After warmup, three full passes are timed with CUDA events. Throughput is the total number of processed pairs divided by the accumulated GPU forward time. The highest-throughput successful batch is reported. Tokenization, data loading, and CPU preprocessing are excluded. The batch-search approach is informed by the [Ettin reranker speed benchmark](https://huggingface.co/blog/ettin-reranker#speed).
219
 
 
 
220
  ## Citation
221
 
222
  ```bibtex
 
51
 
52
  The prompt template ships with the model. The default instruction is *"Given a web search query, retrieve relevant passages that answer the query"*, the same instruction used in training.
53
 
54
+ ### Sentence Transformers
 
55
 
56
  ```python
57
  import torch
 
78
  scores = model.predict(pairs, batch_size=8)
79
  ```
80
 
81
+ ### Transformers
 
 
 
82
 
83
  ```python
84
  import torch
 
126
  print(f"{scores[index]:.4f}\t{documents[index]}")
127
  ```
128
 
 
 
129
  ## Korean Reranking Evaluation
130
 
131
  Scores are **nDCG@10**, and throughput is pairs per second (**PPS**).
 
161
 
162
  `jinaai/jina-reranker-v3` and `jinaai/jina-reranker-v3.5` are listwise rerankers that cannot be compared under the same 8,192-token condition on MLDR, so they are excluded from the 9-subset table and their MLDR cells are left blank below.
163
 
164
+ <details>
165
+ <summary>Per-dataset NDCG@10</summary>
166
 
167
  | Model | Params | Ko-StrategyQA | AutoRAGRetrieval | PublicHealthQA | BelebeleRetrieval | MIRACLRetrieval | MrTidyRetrieval | MultiLongDocRetrieval | SQuADKorV1Retrieval | LawIRKo |
168
  |---|---|---|---|---|---|---|---|---|---|---|
 
186
  | Dongjin-kr/ko-reranker | 560M | 0.8468 | 0.9014 | 0.6976 | 0.9655 | 0.8017 | 0.7772 | 0.3721 | 0.9785 | 0.7343 |
187
  | telepix/PIXIE-Spell-Reranker-Preview-0.6B | 596M | 0.8329 | 0.9794 | 0.8211 | 0.9689 | 0.8449 | 0.7650 | 0.1829 | 0.9850 | 0.6042 |
188
 
189
+ </details>
190
+
191
+ <details>
192
+ <summary>Per-dataset PPS</summary>
193
 
194
  | Model | Params | Ko-StrategyQA | AutoRAGRetrieval | PublicHealthQA | BelebeleRetrieval | MIRACLRetrieval | MrTidyRetrieval | MultiLongDocRetrieval | SQuADKorV1Retrieval | LawIRKo |
195
  |---|---|---|---|---|---|---|---|---|---|---|
 
215
 
216
  Batch size starts at 8 and doubles until an out-of-memory error. Samples are repeated to fill complete batches. After warmup, three full passes are timed with CUDA events. Throughput is the total number of processed pairs divided by the accumulated GPU forward time. The highest-throughput successful batch is reported. Tokenization, data loading, and CPU preprocessing are excluded. The batch-search approach is informed by the [Ettin reranker speed benchmark](https://huggingface.co/blog/ettin-reranker#speed).
217
 
218
+ </details>
219
+
220
  ## Citation
221
 
222
  ```bibtex