r76941156/rare-disease-embedding-model

License

This model is released under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license. It may be used, shared, and adapted for non-commercial research purposes only.

Qwen3-Based Rare Disease Biomedical Embedding Model

This repository provides a fine-tuned embedding model designed for biomedical and rare disease text representation. The repository provides two variants based on Qwen3-Embedding-8B, each fine-tuned using 1,320 disease summaries generated by either Claude Sonnet 4 or OpenAI o3 from the National Organization for Rare Disorders (NORD) disease catalog. The source code for model training and downstream ranking is available in the GEN-KnowRD GitHub repository.

We provide the Claude Sonnet 4 and OpenAI o3 knowledge fine-tuned versions in this repository. Researchers are welcome to further fine-tune or reproduce the training workflow using disease summaries generated by other LLMs or alternative rare disease knowledge sources.

Intended use

This model is intended for research use in rare disease phenotype representation, disease-profile retrieval, and embedding-based candidate ranking.

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("r76941156/rare-disease-embedding-model")

# To use the OpenAI o3 knowledge fine-tuned variant instead:
# model = SentenceTransformer(
#     "r76941156/rare-disease-embedding-model",
#     revision="o3"
# )

texts = [
    "progressive dyspnea and pulmonary fibrosis",
    "idiopathic pulmonary fibrosis with cough and exertional shortness of breath",
    "seizures, developmental delay, and skin lesions suggestive of tuberous sclerosis",
    "recurrent infections and low immunoglobulin levels"
]

embeddings = model.encode(texts, normalize_embeddings=True)

similarities = model.similarity(embeddings, embeddings)
print("Embedding shape:", embeddings.shape)
print("Similarity matrix shape:", similarities.shape)
print(similarities)

Performance

Compared with the original Qwen3-Embedding-8B model, the fine-tuned model improved Recall@1 by 0.59 to 6.65 percentage points on the combined benchmark, depending on the disease knowledge source. Performance varied across benchmark subsets.

Benchmark	Knowledge source	Original Recall@1 (%)	Fine-tuned Recall@1 (%)	Difference (percentage points)
PMC (free-text case reports)	Claude Sonnet 4	80.06	80.63	+0.57
PMC (free-text case reports)	DeepSeek R1	73.60	80.26	+6.66
PMC (free-text case reports)	Gemini 2.5 Pro	77.29	80.65	+3.36
PMC (free-text case reports)	OpenAI o3	79.89	81.77	+1.88
Non-PMC (HPO term-based profiles)	Claude Sonnet 4	27.14	28.01	+0.87
Non-PMC (HPO term-based profiles)	DeepSeek R1	22.50	29.03	+6.53
Non-PMC (HPO term-based profiles)	Gemini 2.5 Pro	28.30	26.85	−1.45
Non-PMC (HPO term-based profiles)	OpenAI o3	29.46	30.33	+0.87
Combined	Claude Sonnet 4	76.14	76.73	+0.59
Combined	DeepSeek R1	69.81	76.46	+6.65
Combined	Gemini 2.5 Pro	73.66	76.66	+3.00
Combined	OpenAI o3	76.15	77.95	+1.81

Citation

Yan C, Su WC, Xin Y, Grabowska ME, Kerchberger VE, Borza VA, Wang J, Wang L, Li R, Lynn J, Dickson AL. Reframing AI for Rare Disease Recognition. Research Square. 2026 Apr 2:rs-3.