README.md

---
library_name: sentence-transformers
pipeline_tag: sentence-similarity
license: mit
base_model: BAAI/bge-large-en-v1.5
tags:
- sentence-transformers
- information-retrieval
- biomedical
- metasyn
datasets:
- THUIR/MetaSyn
---

# MA-Retriever

MA-Retriever is the dense retriever released with MetaSyn. It fine-tunes
`BAAI/bge-large-en-v1.5` for retrieving articles linked to the published
included-study lists of systematic reviews, scoping reviews, and
meta-analyses.

## Resources

- Paper: [arXiv:2606.17041](https://arxiv.org/abs/2606.17041)
- Dataset: [THUIR/MetaSyn](https://huggingface.co/datasets/THUIR/MetaSyn)
- Code and evaluator: [THUIR/MetaSyn](https://github.com/THUIR/MetaSyn)

## Use

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BFTree/MA-Retriever")

def encode_protocol(research_question, population, intervention_or_exposure,
                    comparison, outcome):
    fields = [
        ("Research Question", research_question),
        ("Population", population),
        ("Intervention or Exposure", intervention_or_exposure),
        ("Comparison", comparison),
        ("Outcome", outcome),
    ]
    protocol = ". ".join(f"{name}: {value}" for name, value in fields if value)
    query = "Represent this sentence for searching relevant passages: " + protocol
    return model.encode(query, normalize_embeddings=True)

def encode_article(title, abstract):
    document = f"Title: {title}. Abstract: {abstract}"
    return model.encode(document, normalize_embeddings=True)
```

The protocol query concatenates the research question, Population,
Intervention or Exposure, Comparison, and Outcome. It does not use the
source-review title. Documents use title and abstract. Retrieval uses cosine
similarity over normalized embeddings.

## Training

We used 40 reviews sampled from the training split to select the epoch count.
Recall@20 is the primary metric, with Recall@100 and Recall@200 used as
tie-breakers. The released checkpoint was retrained for one epoch using the
full training split with Multiple Negatives Ranking Loss, batch size 64,
learning rate 2e-5, maximum sequence length 512, and seed 718. After excluding
articles linked to a test review, 334 of the 336 training reviews contribute
positive pairs.

There are 5,585 constructed training pairs. The sentence-transformers training
loader used 5,568 examples in complete batches during the epoch; this accounts
for the smaller sample count shown by automatically generated trainer metadata.

## Test retrieval

All metrics are macro-averaged over the 86 held-out reviews after removing the
source review itself from each ranking.

| Metric | K=5 | K=10 | K=20 | K=50 | K=100 | K=200 |
|:--|--:|--:|--:|--:|--:|--:|
| Recall@K | 24.0% | 38.8% | 53.5% | 75.3% | 84.2% | 91.7% |
| Precision@K | 47.9% | 42.0% | 34.0% | 22.6% | 14.0% | 8.4% |

The source-review split is disjoint, and every article linked to a test review
is excluded as a positive retriever-training example.

## Limitations

The model is trained on reviews from Nature Portfolio and a PubMed-centered
corpus. Performance may not transfer unchanged to other databases, languages,
or domains. A high-recall pool still requires protocol-based screening; model
scores should not be treated as inclusion decisions.

## Citation

```bibtex
@misc{metasyn2026,
  title         = {Benchmarking {LLM} Agents on Meta-Analysis Articles from {Nature} Portfolio},
  author        = {Anzhe Xie and Weihang Su and Yujia Zhou and Yiqun Liu and Min Zhang and Qingyao Ai},
  year          = {2026},
  eprint        = {2606.17041},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2606.17041}
}
```