README.md

---

license: apache-2.0
language:
- en
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- radiology
- medical
- retrieval
- embeddings
- healthcare
- clinical
base_model: zzxslp/RadBERT-RoBERTa-4m
pipeline_tag: sentence-similarity
library_name: sentence-transformers
datasets:
- radiology-education-corpus
metrics:
- mrr
- ndcg
model-index:
- name: RadLITE-Encoder
  results:
  - task:
      type: retrieval
      name: Information Retrieval
    dataset:
      name: RadLIT-9 (Radiology Retrieval Benchmark)
      type: radiology-retrieval
    metrics:
    - type: mrr
      value: 0.829
      name: MRR (with full pipeline)
    - type: ndcg@10
      value: 0.863
      name: nDCG@10
    - type: recall@10
      value: 0.90
      name: Recall@10
  - task:
      type: semantic-similarity
      name: Semantic Similarity
    dataset:
      name: Radiology Similarity Evaluation
      type: radiology-similarity
    metrics:
    - type: spearman_cosine
      value: 0.8454
      name: Spearman Correlation
    - type: pearson_cosine
      value: 0.8504
      name: Pearson Correlation
---


# RadLITE-Encoder

**Radiology Late Interaction Transformer Enhanced - Bi-Encoder Component**

A domain-specialized sentence transformer for radiology and medical imaging content. This model encodes radiology text (reports, articles, educational content) into 768-dimensional dense vectors optimized for semantic search and retrieval.

> **Recommended:** For optimal retrieval performance, use this encoder with [RadLITE-Reranker](https://huggingface.co/matulichpt/RadLITE-Reranker) in a two-stage pipeline. The bi-encoder provides fast candidate retrieval, while the cross-encoder reranker delivers precision. This combination achieves **MRR 0.829** on radiology benchmarks.

## Model Description

| Property | Value |
|----------|-------|
| **Model Type** | Sentence Transformer (Bi-Encoder) |
| **Base Model** | [RadBERT-RoBERTa-4m](https://huggingface.co/zzxslp/RadBERT-RoBERTa-4m) |
| **Domain** | Radiology / Medical Imaging |
| **Vector Dimensions** | 768 |
| **Max Sequence Length** | 512 tokens |
| **Similarity Function** | Cosine Similarity |
| **License** | Apache 2.0 |

### Why RadLITE-Encoder?

Standard embedding models (BGE, E5, OpenAI) are trained on general web text and struggle with radiology-specific terminology:

- **Anatomical terms**: "hepatic flexure", "foramen magnum", "costophrenic angle"
- **Imaging sequences**: "T2 FLAIR", "DWI/ADC mismatch", "post-gadolinium"
- **Pathology descriptions**: "ground-glass opacity", "cortical ribbon sign", "double duct sign"
- **Abbreviations**: "HCC", "RCC", "NSCLC", "BI-RADS"

RadLITE-Encoder is fine-tuned on millions of radiology documents to understand this specialized vocabulary.

## Performance

### RadLIT-9 Benchmark (Radiology Retrieval)

| Model | MRR | nDCG@10 | Notes |
|-------|-----|---------|-------|
| **RadLITE-Encoder** | **0.829** | **0.863** | Full pipeline with reranker |
| RadLITE-Encoder (standalone) | 0.78 | 0.81 | Bi-encoder only |
| BGE-large-en-v1.5 | 0.72 | 0.76 | General-purpose |
| RadBERT (baseline) | 0.45 | 0.52 | No retrieval training |

### Subspecialty Performance

| Subspecialty | MRR | Notes |
|--------------|-----|-------|
| Physics/Nuclear Medicine | 0.936 | Excellent |
| Pediatric Radiology | 0.931 | Excellent |
| Thoracic Imaging | 0.913 | Excellent |
| Cardiac Imaging | 0.862 | Good |
| Neuroradiology | 0.860 | Good |
| Gastrointestinal | 0.800 | Good |
| Breast Imaging | 0.722 | Moderate |
| Musculoskeletal | 0.695 | Moderate |
| Genitourinary | 0.694 | Moderate |

## Quick Start

### Installation

```bash

pip install sentence-transformers>=2.2.0

```

### Basic Usage

```python

from sentence_transformers import SentenceTransformer



# Load the model

model = SentenceTransformer("matulichpt/RadLITE-Encoder")



# Encode radiology text

documents = [

    "Hepatocellular carcinoma typically shows arterial enhancement with washout on portal venous phase.",

    "Ground-glass opacities in the bilateral lower lobes, concerning for viral pneumonia.",

    "No acute intracranial abnormality. Age-appropriate cerebral volume loss.",

]



queries = [

    "HCC imaging characteristics on CT",

    "COVID-19 chest CT findings",

]



# Generate embeddings

doc_embeddings = model.encode(documents, normalize_embeddings=True)

query_embeddings = model.encode(queries, normalize_embeddings=True)



# Compute similarities

similarities = query_embeddings @ doc_embeddings.T

print(similarities)

# Query 1 (HCC) will score highest with Document 1

# Query 2 (COVID) will score highest with Document 2

```

### Semantic Search over Your Corpus

```python

from sentence_transformers import SentenceTransformer, util

import torch



# Load model

model = SentenceTransformer("matulichpt/RadLITE-Encoder")



# Your radiology corpus (articles, reports, educational content)

corpus = [

    {"id": "doc1", "text": "Pancoast tumor: apical lung mass with rib destruction..."},

    {"id": "doc2", "text": "Hepatic hemangioma shows peripheral nodular enhancement..."},

    {"id": "doc3", "text": "Acoustic neuroma appears as enhancing CP angle mass..."},

    # ... your documents

]



# Pre-compute corpus embeddings (do this once, save for reuse)

corpus_texts = [doc["text"] for doc in corpus]

corpus_embeddings = model.encode(corpus_texts, normalize_embeddings=True, show_progress_bar=True)



# Save embeddings for later

torch.save(corpus_embeddings, "corpus_embeddings.pt")



# Search function

def search(query: str, top_k: int = 10):

    query_embedding = model.encode(query, normalize_embeddings=True)

    scores = util.cos_sim(query_embedding, corpus_embeddings)[0]

    top_results = torch.topk(scores, k=min(top_k, len(corpus)))



    results = []

    for score, idx in zip(top_results.values, top_results.indices):

        results.append({

            "document": corpus[idx],

            "score": float(score)

        })

    return results



# Example search

results = search("superior sulcus tumor with Horner syndrome")

for r in results[:3]:

    print(f"Score: {r['score']:.3f} - {r['document']['text'][:100]}...")

```

### Integration with FAISS (Large-Scale)

```python

import faiss

import numpy as np

from sentence_transformers import SentenceTransformer



model = SentenceTransformer("matulichpt/RadLITE-Encoder")



# Encode your corpus

corpus_embeddings = model.encode(corpus_texts, normalize_embeddings=True)

corpus_embeddings = np.array(corpus_embeddings).astype('float32')



# Build FAISS index

dimension = 768

index = faiss.IndexFlatIP(dimension)  # Inner product = cosine for normalized vectors

index.add(corpus_embeddings)



# Save index

faiss.write_index(index, "radiology_index.faiss")



# Search

def faiss_search(query: str, top_k: int = 10):

    query_embedding = model.encode(query, normalize_embeddings=True)

    query_embedding = np.array([query_embedding]).astype('float32')

    scores, indices = index.search(query_embedding, top_k)

    return [(int(idx), float(score)) for idx, score in zip(indices[0], scores[0])]

```

## Best Practices

### 1. Normalize Embeddings

Always use `normalize_embeddings=True` for retrieval tasks. This enables efficient cosine similarity via dot product.

### 2. Chunk Long Documents

The model has a 512 token limit. For long articles:

```python

def chunk_text(text: str, max_length: int = 400, overlap: int = 50):

    """Chunk text with overlap for better retrieval."""

    words = text.split()

    chunks = []

    for i in range(0, len(words), max_length - overlap):

        chunk = " ".join(words[i:i + max_length])

        chunks.append(chunk)

    return chunks

```

### 3. Batch Processing

For large corpora, use batching:

```python

embeddings = model.encode(

    texts,

    batch_size=32,

    normalize_embeddings=True,

    show_progress_bar=True

)

```

### 4. GPU Acceleration

```python

model = SentenceTransformer("matulichpt/RadLITE-Encoder", device="cuda")

```

## Two-Stage Retrieval (Recommended)

For best results, combine RadLITE-Encoder with the [RadLITE-Reranker](https://huggingface.co/matulichpt/RadLITE-Reranker):

```python

from sentence_transformers import SentenceTransformer, CrossEncoder



# Stage 1: Fast bi-encoder retrieval

encoder = SentenceTransformer("matulichpt/RadLITE-Encoder")

# Stage 2: Precise cross-encoder reranking

reranker = CrossEncoder("matulichpt/RadLITE-Reranker", max_length=512)



def two_stage_search(query: str, corpus: list, top_k: int = 10):

    # Stage 1: Get top candidates (fast)

    query_emb = encoder.encode(query, normalize_embeddings=True)

    corpus_embs = encoder.encode(corpus, normalize_embeddings=True)

    scores = query_emb @ corpus_embs.T

    top_indices = scores.argsort()[-50:][::-1]  # Top 50 candidates



    # Stage 2: Rerank with cross-encoder (precise)

    candidates = [corpus[i] for i in top_indices]

    pairs = [[query, doc] for doc in candidates]

    rerank_scores = reranker.predict(pairs)



    # Apply temperature calibration (recommended: 1.5)

    rerank_scores = rerank_scores / 1.5



    # Sort by reranked scores

    reranked = sorted(zip(top_indices, rerank_scores), key=lambda x: x[1], reverse=True)

    return reranked[:top_k]

```

## Architecture

```

Input Text

    |

    v

[RadBERT Tokenizer] --> tokens (max 512)

    |

    v

[RoBERTa Encoder] --> 12 layers, 768 hidden

    |

    v

[Mean Pooling] --> aggregate token embeddings

    |

    v

768-dim embedding vector

```

## Training Details

- **Base Model**: RadBERT-RoBERTa-4m (pre-trained on 4.42M VA radiology reports)
- **Fine-tuning**: Contrastive learning on radiology education corpus
- **Training Samples**: 6.7M query-document pairs
- **Loss Function**: Multiple Negatives Ranking Loss
- **Epochs**: 2 (8,400 steps)
- **Final Spearman**: 0.8454

## Limitations

- **English only**: Trained on English radiology text
- **Domain-specific**: May underperform on non-radiology medical content
- **Subspecialty variance**: GU/MSK content has lower performance than Physics/Neuro
- **512 token limit**: Long documents require chunking

## Citation

If you use RadLITE in your work, please cite both RadLITE and the underlying RadBERT model:

```bibtex

@software{radlite_2026,

    title = {RadLITE: Calibrated Multi-Stage Retrieval for Radiology Education},

    author = {Grai Team},

    year = {2026},

    month = {January},

    url = {https://huggingface.co/matulichpt/RadLITE-Encoder},

    note = {MRR 0.829 on RadLIT-9 benchmark}

}



@article{yan2022radbert,

    title = {RadBERT: Adapting Transformer-based Language Models to Radiology},

    author = {Yan, An and McAuley, Julian and Lu, Xing and Du, Jiang and Chang, Eric Y and Gentili, Amilcare and Hsu, Chun-Nan},

    journal = {Radiology: Artificial Intelligence},

    volume = {4},

    number = {4},

    pages = {e210258},

    year = {2022},

    publisher = {Radiological Society of North America},

    doi = {10.1148/ryai.210258}

}

```

## Related Models

- [RadLITE-Reranker](https://huggingface.co/matulichpt/RadLITE-Reranker) - Cross-encoder for reranking
- [RadBERT-RoBERTa-4m](https://huggingface.co/zzxslp/RadBERT-RoBERTa-4m) - Base model

## License

Apache 2.0 - Free for commercial and research use.