mieraci22/ehr-omop-rag-assistant

EHR-to-OMOP Mapping Assistant

Michael Ieraci | DS 5002-001, How to Train Your LLM | University of Virginia

1. Introduction

Converting electronic health record data into the Observational Medical Outcomes Partnership Common Data Model (OMOP CDM) requires analysts to translate local tables, fields, codes, and timestamps into a shared structure. The work matters because inconsistent mappings can undermine every analysis built on the converted data, yet a general language model can confuse related fields or invent a plausible schema name when it lacks the current OMOP definitions. This project addresses a narrow part of that problem: Visit and Medication mappings into visit_occurrence and drug_exposure. The assistant retrieves field-level OMOP guidance, uses the top document's verified metadata for the proposed target, and asks FLAN-T5-large to explain the recommendation. On ten held-out synthetic mappings, the final system reached 100% target accuracy compared with 50% for the same reader without retrieval; the deployed MiniLM retriever also reached 73.8% hit@1 on RAGBench and 67.8% hit@1 on MIRAGE.

2. Data

The repository uses public and synthetic material only; it contains no PHI, patient records, client schemas, proprietary ETL notes, or confidential implementation details. The retrieval corpus in data/omop_corpus.json reformats the public OHDSI OMOP CDM documentation into ten field-level JSON chunks, five for Visit and five for Medication. Each chunk stores a document ID, domain, target table and field, definition, source link, and synthetic aliases that connect source-style names such as discharge_dttm, admin_route, and qty_value to an OMOP definition. The evaluation file in data/synthetic_mapping_rows.json contains 50 manually checked, non-patient source-field descriptions built from ten mapping specifications with five wording variants each. Variants one through four form a 40-row development set; variant five from every specification forms a fixed ten-row test set. This grouped split keeps the exact test wording out of the retrieval index while guaranteeing that all ten target fields are tested once. Each test case includes the expected document ID and exact OMOP table-field target.

3. Methodology

Each corpus record is already a self-contained field definition, so one record is one chunk; splitting it further would separate the definition from its target metadata. The retriever uses sentence-transformers/all-MiniLM-L6-v2 at revision 1110a243 to create 384-dimensional mean-pooled embeddings with a maximum input length of 256 tokens. It applies cosine similarity, filters candidates to the selected Visit or Medication domain, and returns k=3 chunks. Before embedding the query, transparent rules expand common source cues such as qty, IV, discharge, and RxNorm. I compared MiniLM-cosine, MiniLM-Euclidean, and word TF-IDF-cosine on the same test prompts; all reached 100% expected-document hit@1 after normalization, so I retained MiniLM-cosine for its learned representation and ability to handle wording beyond exact token overlap. The reader is google/flan-t5-large at revision 0613663d, run deterministically with two beams, no sampling, and at most 96 new tokens. The top chunk's metadata locks the table and field, while FLAN-T5 writes only the rationale; field-specific code supplies transformation and validation guidance. This avoids allowing fluent generated text to override the retrieved schema evidence.

4. Evaluation

The ten-case OMOP split is the main end-to-end test because it measures the task the application actually performs. RAGBench PubMedQA tests whether the retriever ranks relevant biomedical evidence first, RAGTruth tests a context-aware unsupported-claim detector, and MIRAGE tests medical supporting-document retrieval. I used FLAN-T5-large without retrieval as the base condition because it isolates the value of the RAG evidence and schema constraint. flan-t5-small (80M parameters) is a practical lower-bound comparison that tests whether a very small reader can use the retrieved context. flan-t5-base (250M) is the intermediate comparison that tests whether a smaller, less expensive reader approaches FLAN-T5-large (780M). Keeping all three readers in the same instruction-tuned family makes differences easier to attribute to model capacity. The full published benchmark splits were used without row sampling.

System	Held-out OMOP target accuracy (10)	RAGBench hit@1 (2,450)	RAGTruth hallucination F1 (2,700)	MIRAGE hit@1 (7,560)
Final: MiniLM + constrained FLAN-T5-large	100.0%	73.8%	0.632	67.8%
Base: FLAN-T5-large, no retrieval	50.0%	0.0%	0.518	0.0%
Comparison: MiniLM + FLAN-T5-small	30.0%	73.8%	0.632	67.8%
Comparison: MiniLM + FLAN-T5-base	60.0%	73.8%	0.632	67.8%

The public component scores repeat across the three RAG reader rows because those conditions share the same MiniLM retriever and groundedness policy; the reader comparison affects the OMOP target-selection test. The final system improves that test from the closed-book base model's 50% to 100% by locking the structured target to the top retrieved metadata instead of asking the reader to invent it. MiniLM also achieved 75.5% recall@3 on RAGBench and 91.3% recall@3 on MIRAGE. These results support a narrow, auditable mapping assistant, not a claim of general medical question-answering performance. Reproducible outputs are stored in evaluation/final_results.json, evaluation/reader_comparison_results.json, evaluation/dense_benchmark_results.json, and evaluation/ragtruth_results.json.

5. Usage and Intended Uses

This is a custom pipeline repository, not a standalone model checkpoint. It loads the linked MiniLM retriever and FLAN-T5 reader with Transformers. The intended user is an OMOP ETL analyst reviewing an approved source data dictionary. The Single field tab accepts a domain, source system label, source table, source field, description, and non-PHI sample values. It returns a proposed OMOP target, rationale, transformation guidance, validation checks, and linked evidence. The Batch fields tab accepts pasted, edited, or uploaded source-field metadata for up to 50 fields and returns a compact table containing each proposed target and its highest-ranked evidence chunk. Batch mode performs retrieval only rather than generating a separate rationale for every row; an analyst can then use Single field for detailed review of important or uncertain mappings. The batch limit is an application safeguard, not a model requirement.

Appropriate uses are early mapping discovery, data-dictionary triage, documentation review, ETL design discussion, and finding fields that need vocabulary or data-quality analysis. A practical workflow is therefore bulk triage followed by field-level adjudication. The system maps schema metadata, not patient records, and it treats every field independently. Its output is an analyst-review recommendation, not executable ETL or a clinical decision. It must not receive PHI, confidential schemas, or unrestricted notes.

Run the Gradio application locally with:

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python app.py

The two Hugging Face models can be loaded directly with transformers, after which the repository pipeline can be called:

from transformers import AutoModel, AutoModelForSeq2SeqLM, AutoTokenizer
from rag_pipeline import EHROMOPRAG

embedding_id = "sentence-transformers/all-MiniLM-L6-v2"
reader_id = "google/flan-t5-large"
embedding_revision = "1110a243fdf4706b3f48f1d95db1a4f5529b4d41"
reader_revision = "0613663d0d48ea86ba8cb3d7a44f0f65dc596a2a"

embedding_tokenizer = AutoTokenizer.from_pretrained(
    embedding_id, revision=embedding_revision
)
embedding_model = AutoModel.from_pretrained(
    embedding_id, revision=embedding_revision
)
reader_tokenizer = AutoTokenizer.from_pretrained(reader_id, revision=reader_revision)
reader_model = AutoModelForSeq2SeqLM.from_pretrained(
    reader_id, revision=reader_revision
)

rag = EHROMOPRAG(
    embedding_model_id=embedding_id,
    reader_model_id=reader_id,
    embedding_revision=embedding_revision,
    reader_revision=reader_revision,
    embedding_tokenizer=embedding_tokenizer,
    embedding_model=embedding_model,
    reader_tokenizer=reader_tokenizer,
    reader_model=reader_model,
)
result = rag.map_field(
    domain="Medication",
    source_system="Synthetic Epic-like",
    source_table="med_admin",
    source_field="admin_route",
    description="Route used for medication administration",
    sample_values="oral, IV, subcutaneous",
)

print(result.to_markdown())

6. Prompt Format

The reader receives only the highest-ranked target and its supporting evidence. It is explicitly prohibited from proposing a different field:

Write two concise sentences explaining this EHR-to-OMOP mapping.
Use only the evidence. Do not propose a different table or field.

Source field: {source_field}
Description: {description}
Sample values: {sample_values}
Required target: {target_table}.{target_field}
Evidence: {retrieved_omop_definition}

This prompt is shorter than the eight-shot prompts tested in Check-in 3. Those experiments showed that additional examples increased context length without consistently improving field selection. In the final pipeline, retrieval provides the task-specific evidence and the structured target is constrained before generation.

7. Expected Output Format

The Single field interface renders a fixed seven-part response:

{
  "target_domain": "Medication",
  "target_table": "drug_exposure",
  "target_field": "route_concept_id",
  "mapping_rationale": "The administration route aligns with the OMOP route definition.",
  "transformation_logic": "Map the local route to a standard OMOP route concept and preserve the source value.",
  "validation_checks": [
    "Measure null, unmapped, and ambiguous values.",
    "Review standard vocabulary coverage.",
    "Preserve source-system lineage."
  ],
  "retrieved_document_ids": [
    "drug_route",
    "drug_concept",
    "drug_source"
  ]
}

The Gradio application's Single field tab presents the same content as readable Markdown and a separate evidence table. Its Batch fields tab returns row number, source metadata, proposed OMOP table-field target, top document ID, and retrieval similarity. Target table and field names in both modes always come from corpus metadata rather than unconstrained generated text.

8. Limitations

The corpus covers only ten fields across two OMOP tables; it does not cover Person, Condition, Procedure, Measurement, Observation, Provider, Care Site, cost, payer, or episode-level mappings. The system also does not search Athena vocabularies, execute SQL, inspect source distributions, infer table relationships, or validate referential integrity. Batch mode considers each source field independently, so it cannot infer joins or resolve mappings that depend on combinations of columns. Its synthetic dataset cannot reproduce the naming variation, null patterns, local code systems, and historical quirks found in production EHR platforms, and the 100% result comes from only ten held-out prompts rather than a production-sized sample. Query-normalization rules may not transfer to a broader schema without revision. RAG reduces unsupported answers but does not eliminate them: a relevant-looking chunk may still be wrong for the local business meaning, and a generated rationale can omit a caveat. Every recommendation therefore requires analyst review against the official OMOP CDM documentation, approved source metadata, data profiling, and organizational governance. No real PHI or confidential metadata should be entered into this demonstration.

Reproducing the Evaluation

The stored results can be regenerated from the public data and model repositories. The dense benchmark run embeds every query and candidate document in the selected published splits and may take several minutes on CPU.

pip install -r requirements-evaluation.txt
python evaluation/compare_readers.py
python evaluation/evaluate_dense_benchmarks.py
python evaluation/evaluate_ragtruth.py
python -m unittest -v tests/test_pipeline.py

References

• Chung, H. W., Hou, L., Longpre, S., et al. (2022). Scaling instruction-finetuned language models.
• Friel, R., Belyi, M., & Sanyal, A. (2024). RAGBench: Explainable benchmark for retrieval-augmented generation systems.
• Lewis, P., Perez, E., Piktus, A., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks.
• Niu, C., Wu, Y., Zhu, J., et al. (2024). RAGTruth: A hallucination corpus for developing trustworthy retrieval-augmented language models.
• Observational Health Data Sciences and Informatics. (n.d.). OMOP Common Data Model.
• Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks.
• Xiong, G., Jin, Q., Lu, Z., & Zhang, A. (2024). Benchmarking retrieval-augmented generation for medicine.