---
license: llama3.1
datasets:
- biodatlab/ec-raft-dataset
language:
- en
pipeline_tag: text-generation
---

# EC-RAFT: Automated Generation of Clinical Trial Eligibility Criteria

## Model Description

**EC-RAFT** is a fine-tuned Retrieval-Augmented Fine-Tuning (RAFT) model based on **LLaMA-3.1-8B-Instruct** architecture.  
It is designed to automatically generate **structured, high-quality clinical trial eligibility criteria (EC)** directly from trial titles and descriptions.  

EC-RAFT integrates **domain-specific retrieval** with **synthesized intermediate reasoning** steps, enabling it to produce **clinically relevant** and **contextually appropriate** EC sets.

## Fine-tuning Details

- **Original Model:** LLaMA-3.1-8B-Instruct
- **Datasets used for fine-tuning:**
  - ClinicalTrials.gov (267,347 trials, 2000–2024) [biodatlab/ec-raft-dataset](https://huggingface.co/datasets/biodatlab/ec-raft-dataset)
  - Retrieval corpus constructed using **SciNCL model**
  - Intermediate reasoning steps **R** generated using **Gemini-1.5-flash-002**
  - Fine-tuning method:
    - **Retrieval-Augmented Fine-Tuning (RAFT)**
    - **Low-Rank Adaptation (LoRA)**

## Model Performance

Evaluated on a held-out ClinicalTrials.gov test split:

| Metric                            | Score   |
|-----------------------------------|---------|
| **BERTScore** (semantic similarity) | **86.23** |
| **Precision** (LLM-guided evaluation) | **78.84%** |
| **Recall** (LLM-guided evaluation)    | **75.89%** |
| **Mean LLM-as-a-Judge Score** (0–3)   | **1.7150** |
| **Mean Pair-BERTScore**               | **67.76** |

- **Outperforms zero-shot LLaMA-3.1 and Gemini-1.5-flash baselines**
- **Outperforms fine-tuned LLaMA and Meditron baselines**
- **Clinically validated:** LLM-as-a-Judge scores highly correlated with human physician evaluation

## Intended Use

- Assist **researchers**, **trial designers**, and **sponsors** in drafting clinical trial eligibility criteria.
- **Automate** EC generation to reduce manual effort and improve consistency.
- Support **clinical trial design** transparency and quality.
- Enable integration with **trial registry platforms**, **clinical trial matching systems**, and **EC recommendation tools**.

## Limitations

- Requires **human validation** of generated EC before clinical use.
- Trained on **public ClinicalTrials.gov data** — may not generalize well to:
  - Rare or novel diseases
  - Specialized or non-standard trial designs
  - Non-public trial data
- Optimized for **English-language clinical trials**.
- As with any LLM-based system, risks include hallucination, subtle errors, and domain shifts.
- Evaluation metrics (BERTScore, LLM-as-a-Judge) are proxies — not full substitutes for domain expert review.

## Acknowledgments

This model was developed using resources provided by:

- **RAVIS Technology** for feedback and collaboration.
- **Faculty of Medicine Ramathibodi Hospital**
- **NSTDA Supercomputer Center (ThaiSC), Project \#pv814001**

We also acknowledge the contributions of the broader open-source community whose tools and prior works on **RAFT**, **SciNCL**, **LoRA**, **LLaMA-3**, and **biomedical NLP** made this project possible.