File size: 3,734 Bytes
8e09265 3761d38 8e09265 3761d38 8e09265 3761d38 05f427f 3761d38 05f427f 3761d38 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 | ---
library_name: sentence-transformers
pipeline_tag: sentence-similarity
license: mit
base_model: BAAI/bge-large-en-v1.5
tags:
- sentence-transformers
- information-retrieval
- biomedical
- metasyn
datasets:
- THUIR/MetaSyn
---
# MA-Retriever
MA-Retriever is the dense retriever released with MetaSyn. It fine-tunes
`BAAI/bge-large-en-v1.5` for retrieving articles linked to the published
included-study lists of systematic reviews, scoping reviews, and
meta-analyses.
## Resources
- Paper: [arXiv:2606.17041](https://arxiv.org/abs/2606.17041)
- Dataset: [THUIR/MetaSyn](https://huggingface.co/datasets/THUIR/MetaSyn)
- Code and evaluator: [THUIR/MetaSyn](https://github.com/THUIR/MetaSyn)
## Use
```python
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("BFTree/MA-Retriever")
def encode_protocol(research_question, population, intervention_or_exposure,
comparison, outcome):
fields = [
("Research Question", research_question),
("Population", population),
("Intervention or Exposure", intervention_or_exposure),
("Comparison", comparison),
("Outcome", outcome),
]
protocol = ". ".join(f"{name}: {value}" for name, value in fields if value)
query = "Represent this sentence for searching relevant passages: " + protocol
return model.encode(query, normalize_embeddings=True)
def encode_article(title, abstract):
document = f"Title: {title}. Abstract: {abstract}"
return model.encode(document, normalize_embeddings=True)
```
The protocol query concatenates the research question, Population,
Intervention or Exposure, Comparison, and Outcome. It does not use the
source-review title. Documents use title and abstract. Retrieval uses cosine
similarity over normalized embeddings.
## Training
We used 40 reviews sampled from the training split to select the epoch count.
Recall@20 is the primary metric, with Recall@100 and Recall@200 used as
tie-breakers. The released checkpoint was retrained for one epoch using the
full training split with Multiple Negatives Ranking Loss, batch size 64,
learning rate 2e-5, maximum sequence length 512, and seed 718. After excluding
articles linked to a test review, 334 of the 336 training reviews contribute
positive pairs.
There are 5,585 constructed training pairs. The sentence-transformers training
loader used 5,568 examples in complete batches during the epoch; this accounts
for the smaller sample count shown by automatically generated trainer metadata.
## Test retrieval
All metrics are macro-averaged over the 86 held-out reviews after removing the
source review itself from each ranking.
| Metric | K=5 | K=10 | K=20 | K=50 | K=100 | K=200 |
|:--|--:|--:|--:|--:|--:|--:|
| Recall@K | 24.0% | 38.8% | 53.5% | 75.3% | 84.2% | 91.7% |
| Precision@K | 47.9% | 42.0% | 34.0% | 22.6% | 14.0% | 8.4% |
The source-review split is disjoint, and every article linked to a test review
is excluded as a positive retriever-training example.
## Limitations
The model is trained on reviews from Nature Portfolio and a PubMed-centered
corpus. Performance may not transfer unchanged to other databases, languages,
or domains. A high-recall pool still requires protocol-based screening; model
scores should not be treated as inclusion decisions.
## Citation
```bibtex
@misc{metasyn2026,
title = {Benchmarking {LLM} Agents on Meta-Analysis Articles from {Nature} Portfolio},
author = {Anzhe Xie and Weihang Su and Yujia Zhou and Yiqun Liu and Min Zhang and Qingyao Ai},
year = {2026},
eprint = {2606.17041},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2606.17041}
}
```
|