+🧬 BioReason
Incentivizing Multimodal Biological Reasoning
within a DNA-LLM Model
+
+
+
+ 
+ +## Updates [Jun 10, 2025] +- We are integrating vLLM to improve the speed and efficiency of the GRPO pipeline. We expect this to be pushed by end of week. +- Checkpoints along with the custom DNA-LLM model class will be released on HuggingFace by end of week. +- More training results with GRPO will be shared soon. + +
+ +## Abstract + +Unlocking deep, interpretable biological reasoning from complex genomic data is a major AI challenge hindering scientific discovery. Current DNA foundation models, despite strong sequence representation, struggle with multi-step reasoning and lack inherent transparent, biologically intuitive explanations. We introduce BioReason, a pioneering architecture that, for the first time, deeply integrates a DNA foundation model with a large language model (LLM). This novel connection enables the LLM to directly process and reason with genomic information as a fundamental input, fostering a new form of multimodal biological understanding. BioReason's sophisticated multi-step reasoning is developed through supervised fine-tuning and targeted reinforcement learning, guiding the system to generate logical, biologically coherent deductions. On biological reasoning benchmarks including KEGG-based disease pathway prediction—where accuracy improves from 88% to 97%—and variant effect prediction, BioReason demonstrates an average 15% performance gain over strong single-modality baselines. + +
+ +## Key Contributions + +• **Novel multimodal architecture**: The first successful integration of a DNA foundation model with an LLM, establishing a new methodology for AI-driven biological studies. + +• **Advanced reasoning methodology**: A systematic training approach combining supervised fine-tuning and reinforcement learning that incentivizes multi-step biological reasoning. + +• **New biological reasoning benchmarks**: Development and curation of novel benchmarks for evaluating biological reasoning capabilities, including an annotated reasoning dataset for gene pathway and disease prediction from KEGG. + +• **Empirical performance improvements**: Demonstration that BioReason outperforms both DNA foundation models and LLMs used independently or in simple combination, with average performance gains of 15%+ over baseline. + +• **Interpretable reasoning traces**: A mechanism for generating step-by-step biological reasoning traces that provide interpretable predictions, enhancing scientific insight and hypothesis generation. + +
+ +## Datasets + +The datasets used to train and evaluate BioReason can be found on our [HuggingFace collection](https://huggingface.co/collections/wanglab/bioreason-683cd17172a037a31d208f70) with detailed download and usage instructions. + +
+ +## Checkpoints + +We will release the checkpoints soon! + +
+ +## Installation + +### Prerequisites +- Python 3.11+ +- CUDA/GPU for best performance + +### Installation Steps +```bash +# Clone the repository +git clone https://github.com/bowang-lab/BioReason.git +cd BioReason + +# Install package +pip install -e . +``` + +
+ +## Results + +### KEGG-Derived Biological Reasoning Task +Performance comparison on 290 test datapoints for multi-step mechanistic reasoning: + +| Model | Accuracy | F1-Score | Precision | Recall | +|-------|----------|----------|-----------|---------| +| [DNA] NT - 500M | 86.55 | 69.76 | 73.23 | 66.61 | +| [DNA] Evo2 - 1B | 88.28 | 72.43 | 75.23 | 69.83 | +| [LLM] Qwen3 - 1B | 85.17 | 65.71 | 71.39 | 64.19 | +| [LLM] Qwen3 - 4B | 93.48 | 85.44 | 88.31 | 86.72 | +| [DNA-LLM] NT + Qwen3 - 1B | 88.42 | 72.13 | 75.42 | 71.91 | +| [DNA-LLM] NT + Qwen3 - 1B (+RL) | 89.66 | 74.11 | 78.82 | 72.96 | +| [DNA-LLM] NT + Qwen3 - 4B | 96.90 | **89.03** | **90.99** | **89.38** | +| [DNA-LLM] Evo2 + Qwen3 - 1B | 90.42 | 75.62 | 77.42 | 73.91 | +| [DNA-LLM] Evo2 + Qwen3 - 4B | **97.24** | 86.30 | 86.75 | 87.25 | + +### Variant Effect Prediction Benchmarks +Performance on pathogenic/benign classification: + +| Model | Variant Effect - Coding | | Variant Effect - Non-SNV | | +|-------|------------|----------|------------|----------| +| | Accuracy | F1-Score | Accuracy | F1-Score | +| [DNA] NT - 500M | 60.91 | 45.20 | 67.93 | 65.97 | +| [DNA] Evo2 - 1B | 70.07 | 49.19 | 76.17 | 66.51 | +| [LLM] Qwen3 - 1B | 46.55 | 34.82 | 70.67 | 76.21 | +| [LLM] Qwen3 - 4B | 48.99 | 39.58 | 61.86 | 67.60 | +| [DNA-LLM] NT + Qwen3 - 1B | 55.58 | 54.50 | 72.82 | 76.93 | +| [DNA-LLM] NT + Qwen3 - 4B | 60.94 | 55.66 | 65.59 | 73.00 | +| [DNA-LLM] Evo2 + Qwen3 - 1B | 72.83 | 68.90 | **88.20** | **89.91** | +| [DNA-LLM] Evo2 + Qwen3 - 4B | **80.21** | **80.00** | 83.85 | 85.02 | + +
+ +## Citation + +If you find this work useful, please cite our paper: + +```bibtex +@misc{fallahpour2025bioreasonincentivizingmultimodalbiological, + title={BioReason: Incentivizing Multimodal Biological Reasoning within a DNA-LLM Model}, + author={Adibvafa Fallahpour and Andrew Magnuson and Purav Gupta and Shihao Ma and Jack Naimer and Arnav Shah and Haonan Duan and Omar Ibrahim and Hani Goodarzi and Chris J. Maddison and Bo Wang}, + year={2025}, + eprint={2505.23579}, + archivePrefix={arXiv}, + primaryClass={cs.LG}, + url={https://arxiv.org/abs/2505.23579}, +} +``` + +
+ +## Authors + +- **Adibvafa Fallahpour**¹²³⁵ * (adibvafa.fallahpour@mail.utoronto.ca) +- **Andrew Magnuson**¹² * +- **Purav Gupta**¹² * +- **Shihao Ma**¹²³ +- **Jack Naimer**¹²³ +- **Arnav Shah**¹²³ +- **Haonan Duan**¹² +- **Omar Ibrahim**³ +- **Hani Goodarzi**†⁴⁶ +- **Chris J. Maddison**†¹²⁷ +- **Bo Wang**†¹²³ + +¹ University of Toronto ² Vector Institute ³ University Health Network (UHN)
+⁴ Arc Institute ⁵ Cohere ⁶ University of California, San Francisco ⁷ Google DeepMind + +
+* Equal contribution
+† Equal advising + +--- + +
+Made with ❤️ at University of Toronto, Vector Institute, and University Health Network +
diff --git a/BioReason-main/bioreason.egg-info/PKG-INFO b/BioReason-main/bioreason.egg-info/PKG-INFO new file mode 100644 index 0000000000000000000000000000000000000000..1b0f156459433672bd6d5f2976f0fe642559e568 --- /dev/null +++ b/BioReason-main/bioreason.egg-info/PKG-INFO @@ -0,0 +1,181 @@ +Metadata-Version: 2.4 +Name: bioreason +Version: 0.1.0 +Summary: Bio-related Reasoning with Language Models +License: UNKNOWN +Platform: UNKNOWN +Classifier: Programming Language :: Python :: 3 +Classifier: Programming Language :: Python :: 3.11 +Classifier: License :: OSI Approved :: MIT License +Classifier: Operating System :: OS Independent +Requires-Python: >=3.11 +Description-Content-Type: text/markdown +License-File: LICENSE +Requires-Dist: torch +Requires-Dist: torchvision +Requires-Dist: transformers +Requires-Dist: accelerate +Requires-Dist: qwen-vl-utils +Requires-Dist: jupyter +Requires-Dist: datasets +Requires-Dist: peft +Requires-Dist: pytorch_lightning +Requires-Dist: wandb +Requires-Dist: trl[vllm] +Requires-Dist: bitsandbytes +Requires-Dist: deepspeed +Provides-Extra: dev +Requires-Dist: pytest; extra == "dev" +Requires-Dist: black; extra == "dev" +Requires-Dist: isort; extra == "dev" +Requires-Dist: mypy; extra == "dev" +Dynamic: license-file + +
+🧬 BioReason
Incentivizing Multimodal Biological Reasoning
within a DNA-LLM Model
+
+
+
+
+ +## Updates [Jun 10, 2025] +- We are integrating vLLM to improve the speed and efficiency of the GRPO pipeline. We expect this to be pushed by end of week. +- Checkpoints along with the custom DNA-LLM model class will be released on HuggingFace by end of week. +- More training results with GRPO will be shared soon. + +
+ +## Abstract + +Unlocking deep, interpretable biological reasoning from complex genomic data is a major AI challenge hindering scientific discovery. Current DNA foundation models, despite strong sequence representation, struggle with multi-step reasoning and lack inherent transparent, biologically intuitive explanations. We introduce BioReason, a pioneering architecture that, for the first time, deeply integrates a DNA foundation model with a large language model (LLM). This novel connection enables the LLM to directly process and reason with genomic information as a fundamental input, fostering a new form of multimodal biological understanding. BioReason's sophisticated multi-step reasoning is developed through supervised fine-tuning and targeted reinforcement learning, guiding the system to generate logical, biologically coherent deductions. On biological reasoning benchmarks including KEGG-based disease pathway prediction—where accuracy improves from 88% to 97%—and variant effect prediction, BioReason demonstrates an average 15% performance gain over strong single-modality baselines. + +
+ +## Key Contributions + +• **Novel multimodal architecture**: The first successful integration of a DNA foundation model with an LLM, establishing a new methodology for AI-driven biological studies. + +• **Advanced reasoning methodology**: A systematic training approach combining supervised fine-tuning and reinforcement learning that incentivizes multi-step biological reasoning. + +• **New biological reasoning benchmarks**: Development and curation of novel benchmarks for evaluating biological reasoning capabilities, including an annotated reasoning dataset for gene pathway and disease prediction from KEGG. + +• **Empirical performance improvements**: Demonstration that BioReason outperforms both DNA foundation models and LLMs used independently or in simple combination, with average performance gains of 15%+ over baseline. + +• **Interpretable reasoning traces**: A mechanism for generating step-by-step biological reasoning traces that provide interpretable predictions, enhancing scientific insight and hypothesis generation. + +
+ +## Datasets + +The datasets used to train and evaluate BioReason can be found on our [HuggingFace collection](https://huggingface.co/collections/wanglab/bioreason-683cd17172a037a31d208f70) with detailed download and usage instructions. + +
+ +## Checkpoints + +We will release the checkpoints soon! + +
+ +## Installation + +### Prerequisites +- Python 3.11+ +- CUDA/GPU for best performance + +### Installation Steps +```bash +# Clone the repository +git clone https://github.com/bowang-lab/BioReason.git +cd BioReason + +# Install package +pip install -e . +``` + +
+ +## Results + +### KEGG-Derived Biological Reasoning Task +Performance comparison on 290 test datapoints for multi-step mechanistic reasoning: + +| Model | Accuracy | F1-Score | Precision | Recall | +|-------|----------|----------|-----------|---------| +| [DNA] NT - 500M | 86.55 | 69.76 | 73.23 | 66.61 | +| [DNA] Evo2 - 1B | 88.28 | 72.43 | 75.23 | 69.83 | +| [LLM] Qwen3 - 1B | 85.17 | 65.71 | 71.39 | 64.19 | +| [LLM] Qwen3 - 4B | 93.48 | 85.44 | 88.31 | 86.72 | +| [DNA-LLM] NT + Qwen3 - 1B | 88.42 | 72.13 | 75.42 | 71.91 | +| [DNA-LLM] NT + Qwen3 - 1B (+RL) | 89.66 | 74.11 | 78.82 | 72.96 | +| [DNA-LLM] NT + Qwen3 - 4B | 96.90 | **89.03** | **90.99** | **89.38** | +| [DNA-LLM] Evo2 + Qwen3 - 1B | 90.42 | 75.62 | 77.42 | 73.91 | +| [DNA-LLM] Evo2 + Qwen3 - 4B | **97.24** | 86.30 | 86.75 | 87.25 | + +### Variant Effect Prediction Benchmarks +Performance on pathogenic/benign classification: + +| Model | Variant Effect - Coding | | Variant Effect - Non-SNV | | +|-------|------------|----------|------------|----------| +| | Accuracy | F1-Score | Accuracy | F1-Score | +| [DNA] NT - 500M | 60.91 | 45.20 | 67.93 | 65.97 | +| [DNA] Evo2 - 1B | 70.07 | 49.19 | 76.17 | 66.51 | +| [LLM] Qwen3 - 1B | 46.55 | 34.82 | 70.67 | 76.21 | +| [LLM] Qwen3 - 4B | 48.99 | 39.58 | 61.86 | 67.60 | +| [DNA-LLM] NT + Qwen3 - 1B | 55.58 | 54.50 | 72.82 | 76.93 | +| [DNA-LLM] NT + Qwen3 - 4B | 60.94 | 55.66 | 65.59 | 73.00 | +| [DNA-LLM] Evo2 + Qwen3 - 1B | 72.83 | 68.90 | **88.20** | **89.91** | +| [DNA-LLM] Evo2 + Qwen3 - 4B | **80.21** | **80.00** | 83.85 | 85.02 | + +
+ +## Citation + +If you find this work useful, please cite our paper: + +```bibtex +@misc{fallahpour2025bioreasonincentivizingmultimodalbiological, + title={BioReason: Incentivizing Multimodal Biological Reasoning within a DNA-LLM Model}, + author={Adibvafa Fallahpour and Andrew Magnuson and Purav Gupta and Shihao Ma and Jack Naimer and Arnav Shah and Haonan Duan and Omar Ibrahim and Hani Goodarzi and Chris J. Maddison and Bo Wang}, + year={2025}, + eprint={2505.23579}, + archivePrefix={arXiv}, + primaryClass={cs.LG}, + url={https://arxiv.org/abs/2505.23579}, +} +``` + +
+ +## Authors + +- **Adibvafa Fallahpour**¹²³⁵ * (adibvafa.fallahpour@mail.utoronto.ca) +- **Andrew Magnuson**¹² * +- **Purav Gupta**¹² * +- **Shihao Ma**¹²³ +- **Jack Naimer**¹²³ +- **Arnav Shah**¹²³ +- **Haonan Duan**¹² +- **Omar Ibrahim**³ +- **Hani Goodarzi**†⁴⁶ +- **Chris J. Maddison**†¹²⁷ +- **Bo Wang**†¹²³ + +¹ University of Toronto ² Vector Institute ³ University Health Network (UHN)
+⁴ Arc Institute ⁵ Cohere ⁶ University of California, San Francisco ⁷ Google DeepMind + +
+* Equal contribution
+† Equal advising + +--- + +
+Made with ❤️ at University of Toronto, Vector Institute, and University Health Network +
diff --git a/BioReason-main/bioreason.egg-info/SOURCES.txt b/BioReason-main/bioreason.egg-info/SOURCES.txt new file mode 100644 index 0000000000000000000000000000000000000000..391fc97522cd46e17ef8ed932a77e26cd6373052 --- /dev/null +++ b/BioReason-main/bioreason.egg-info/SOURCES.txt @@ -0,0 +1,9 @@ +LICENSE +README.md +pyproject.toml +bioreason/__init__.py +bioreason.egg-info/PKG-INFO +bioreason.egg-info/SOURCES.txt +bioreason.egg-info/dependency_links.txt +bioreason.egg-info/requires.txt +bioreason.egg-info/top_level.txt \ No newline at end of file diff --git a/BioReason-main/bioreason.egg-info/dependency_links.txt b/BioReason-main/bioreason.egg-info/dependency_links.txt new file mode 100644 index 0000000000000000000000000000000000000000..8b137891791fe96927ad78e64b0aad7bded08bdc --- /dev/null +++ b/BioReason-main/bioreason.egg-info/dependency_links.txt @@ -0,0 +1 @@ + diff --git a/BioReason-main/bioreason.egg-info/requires.txt b/BioReason-main/bioreason.egg-info/requires.txt new file mode 100644 index 0000000000000000000000000000000000000000..5dee50c1b476cc5e7f85f7b8e996c57311f348d7 --- /dev/null +++ b/BioReason-main/bioreason.egg-info/requires.txt @@ -0,0 +1,19 @@ +torch +torchvision +transformers +accelerate +qwen-vl-utils +jupyter +datasets +peft +pytorch_lightning +wandb +trl[vllm] +bitsandbytes +deepspeed + +[dev] +pytest +black +isort +mypy diff --git a/BioReason-main/bioreason.egg-info/top_level.txt b/BioReason-main/bioreason.egg-info/top_level.txt new file mode 100644 index 0000000000000000000000000000000000000000..7b9995054ed2dec4763dced82d34176d2c514f30 --- /dev/null +++ b/BioReason-main/bioreason.egg-info/top_level.txt @@ -0,0 +1 @@ +bioreason diff --git a/BioReason-main/bioreason/__init__.py b/BioReason-main/bioreason/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 diff --git a/BioReason-main/bioreason/dataset/__init__.py b/BioReason-main/bioreason/dataset/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..a318a901b205faa943fd38d702d1001effb1cdd0 --- /dev/null +++ b/BioReason-main/bioreason/dataset/__init__.py @@ -0,0 +1,11 @@ +from .kegg import KEGGDataset, split_kegg_dataset +from .utils import torch_to_hf_dataset, truncate_dna +from .variant_effect import get_format_variant_effect_function + +__all__ = [ + "KEGGDataset", + "split_kegg_dataset", + "torch_to_hf_dataset", + "truncate_dna", + "get_format_variant_effect_function", +] diff --git a/BioReason-main/bioreason/dataset/kegg.py b/BioReason-main/bioreason/dataset/kegg.py new file mode 100644 index 0000000000000000000000000000000000000000..d721e79c9d06bfa3b713d0c0223ebc482ec43fee --- /dev/null +++ b/BioReason-main/bioreason/dataset/kegg.py @@ -0,0 +1,382 @@ +import json +import os +import random +import sys +import torch +from torch.utils.data import Dataset, DataLoader +from typing import Any, Dict, List, Tuple + +from bioreason.dataset.utils import torch_to_hf_dataset +from bioreason.models.dl.processing_dl import DLProcessor +from trl.data_utils import maybe_apply_chat_template + + +class KEGGDataset(Dataset): + """Dataset for KEGG data.""" + + def __init__(self, data_dir: str): + """ + Initialize the dataset by loading all JSON files from the given directory. + + Args: + data_dir: Path to the directory containing JSON files + """ + self.data_dir = data_dir + self.data = [] + + # Load all JSON files + json_files = sorted([f for f in os.listdir(data_dir) if f.endswith(".json")]) + + # Process each file + for filename in json_files: + file_path = os.path.join(data_dir, filename) + kegg_id = filename.split("_")[1] + + with open(file_path, "r", encoding="utf-8") as f: + item = json.load(f) + item["kegg_id"] = kegg_id + processed_item = self._process_item(item) + self.data.append(processed_item) + + def _process_item(self, item: Dict[str, Any]) -> Dict[str, Any]: + """ + Process a single data item to format fields as required. + + Args: + item: Original data item from JSON + + Returns: + Processed data item + """ + # Extract question as is + question = item.get("question", "") + + # Convert answer to lowercase and strip whitespace + answer = item.get("answer", "").lower().strip() + + # Combine reasoning steps into a single paragraph with newlines + reasoning_steps = item.get("reasoning", {}).get("reasoning_steps", []) + reasoning = "\n".join(reasoning_steps) + + # Convert sequences to uppercase and strip whitespace + reference_sequence = item.get("reference_sequence", "").upper().strip() + variant_sequence = item.get("variant_sequence", "").upper().strip() + + return { + "question": question, + "answer": answer, + "reasoning": reasoning, + "reference_sequence": reference_sequence, + "variant_sequence": variant_sequence, + } + + def __len__(self) -> int: + """Return the number of items in the dataset.""" + return len(self.data) + + def __getitem__(self, idx: int) -> Dict[str, Any]: + """Return a specific item from the dataset.""" + return self.data[idx] + + +def split_kegg_dataset( + dataset: KEGGDataset, + train_ratio: float = 0.8, + val_ratio: float = 0.1, + test_ratio: float = 0.1, + seed: int = 42, +) -> Tuple[KEGGDataset, KEGGDataset, KEGGDataset]: + """ + Split a KEGG dataset into train, validation, and test sets. + + Args: + dataset: The dataset to split + train_ratio: Proportion of data for training + val_ratio: Proportion of data for validation + test_ratio: Proportion of data for testing + batch_size: Batch size for the dataloaders + seed: Random seed for reproducibility + + Returns: + Tuple of (train_dataset, val_dataset, test_dataset) + """ + # Calculate the size of each split + dataset_size = len(dataset) + train_size = int(train_ratio * dataset_size) + val_size = int(val_ratio * dataset_size) + test_size = dataset_size - train_size - val_size + assert train_ratio + val_ratio + test_ratio == 1.0, "Ratios must sum to 1" + + # Set the random seed + torch.manual_seed(seed) + random.seed(seed) + + # Split the dataset + train_dataset, val_dataset, test_dataset = torch.utils.data.random_split( + dataset, [train_size, val_size, test_size] + ) + + return train_dataset, val_dataset, test_dataset + + +def create_kegg_dataloader( + data_dir: str, + batch_size: int = 2, + shuffle: bool = True, + num_workers: int = 2, + pin_memory: bool = True, +) -> DataLoader: + """ + Create a DataLoader for the KEGG dataset. + + Args: + data_dir: Path to the directory containing JSON files + batch_size: Batch size for the dataloader + shuffle: Whether to shuffle the data + num_workers: Number of worker processes for loading data + pin_memory: Whether to pin memory for faster data transfer + + Returns: + DataLoader for the KEGG dataset + """ + dataset = KEGGDataset(data_dir) + return DataLoader( + dataset, + batch_size=batch_size, + shuffle=shuffle, + num_workers=num_workers, + pin_memory=pin_memory, + ) + + +def get_format_kegg_function(model_name: str) -> Any: + """ + Get the appropriate format function for a given model name. + """ + if model_name.lower() == "llm": + return format_kegg_for_llm + elif model_name.lower() == "dna-llm": + return format_kegg_for_dna_llm + else: + raise ValueError(f"Unsupported model name: {model_name}") + + +def format_kegg_for_dna_llm(example: Dict[str, Any]) -> Dict[str, Any]: + """ + Format a KEGG example into the required chat format for DNA-LLM. + """ + return { + "prompt": [ + { + "role": "user", + "content": [ + *({"type": "dna", "text": None} for _ in range(2)), + {"type": "text", "text": example["question"].strip()}, + ], + }, + { + "role": "assistant", + "reasoning_content": example["reasoning"].strip(), + "content": [ + {"type": "text", "text": f"Answer: {example['answer'].strip()}"}, + ], + }, + ], + "dna_sequences": [ + example["reference_sequence"], + example["variant_sequence"], + ], + "answer": example["answer"], + } + + +def format_kegg_for_llm(example: Dict[str, Any]) -> Dict[str, Any]: + """ + Format a KEGG example into the required chat format for LLM. + """ + question = f"Reference sequence: {example['reference_sequence']}\nVariant sequence: {example['variant_sequence']}\nQuestion: {example['question']}" + return { + "prompt": [ + { + "role": "user", + "content": [ + *({"type": "dna", "text": None} for _ in range(2)), + {"type": "text", "text": question.strip()}, + ], + }, + { + "role": "assistant", + "reasoning_content": example["reasoning"].strip(), + "content": [ + {"type": "text", "text": f"Answer: {example['answer'].strip()}"}, + ], + }, + ], + "dna_sequences": [ + "", + "", + ], + "answer": example["answer"], + } + + +def qwen_dna_collate_fn( + examples: List[Dict], + processor: DLProcessor, + max_length_text: int, + max_length_dna: int, + return_answer_in_batch: bool = False, +) -> Dict: + """ + Custom collate function for Qwen DNA models. + + Creates a batch with proper labels for supervised fine-tuning where only + the assistant responses contribute to the loss calculation. + """ + prompts_text = [ + maybe_apply_chat_template(example, processor)["prompt"] for example in examples + ] + batch_dna_sequences = [example["dna_sequences"] for example in examples] + + batch = processor( + text=prompts_text, + batch_dna_sequences=batch_dna_sequences, + return_tensors="pt", + padding=True, + padding_side="left", + add_special_tokens=False, + max_length_text=max_length_text, + max_length_dna=max_length_dna, + ) + + # Create labels tensor filled with -100 (ignored in loss calculation) + labels = torch.full_like(batch["input_ids"], -100) + + # Get token IDs for special markers + assistant_start_marker = "<|im_start|>assistant\n" + im_end_marker = "<|im_end|>" + + assistant_start_token_ids = processor.tokenizer.encode( + assistant_start_marker, add_special_tokens=False + ) + im_end_token_ids = processor.tokenizer.encode( + im_end_marker, add_special_tokens=False + ) + + # Convert token arrays to tensors for faster comparison + assistant_marker_tensor = torch.tensor( + assistant_start_token_ids, device=batch["input_ids"].device + ) + im_end_marker_tensor = torch.tensor( + im_end_token_ids, device=batch["input_ids"].device + ) + + # Get dimensions for easier reference + assistant_marker_len = len(assistant_start_token_ids) + im_end_marker_len = len(im_end_token_ids) + + # For each sequence in the batch + for i in range(batch["input_ids"].shape[0]): + input_ids = batch["input_ids"][i] + seq_len = input_ids.size(0) + + # Track assistant sections + assistant_sections = [] + + # Find all assistant start markers + start_positions = [] + for pos in range(seq_len - assistant_marker_len + 1): + if torch.all( + input_ids[pos : pos + assistant_marker_len] == assistant_marker_tensor + ): + start_positions.append( + pos + assistant_marker_len + ) # Store position after marker + + # Find all end markers + end_positions = [] + for pos in range(seq_len - im_end_marker_len + 1): + if torch.all( + input_ids[pos : pos + im_end_marker_len] == im_end_marker_tensor + ): + end_positions.append(pos) # Store position at start of end marker + + # Match start and end markers to create sections + for start_pos in start_positions: + # Find the next end marker after this start position + valid_ends = [pos for pos in end_positions if pos > start_pos] + if valid_ends: + end_pos = min(valid_ends) # Take the first end marker after start + # Only include content between markers (not the markers themselves) + if start_pos < end_pos: + assistant_sections.append((start_pos, end_pos)) + else: + # If no end marker, assume the section runs to the end of the sequence + assistant_sections.append((start_pos, seq_len)) + + # Set labels for all identified assistant sections + for start_pos, end_pos in assistant_sections: + if start_pos < end_pos and start_pos < seq_len: + end_pos = min(end_pos, seq_len) # Safety check + labels[i, start_pos:end_pos] = input_ids[start_pos:end_pos] + + # Also mask padding tokens + labels[batch["input_ids"] == processor.tokenizer.pad_token_id] = -100 + + # Add labels to batch + batch["labels"] = labels + + # Add answer to batch + if return_answer_in_batch: + batch["answer"] = [example["answer"].strip() for example in examples] + + return batch + + +def dna_collate_fn( + batch: List[Dict[str, Any]], + dna_tokenizer: Any, + label2id: Dict[str, int], + max_length: int = 2048, +) -> Dict[str, Any]: + """ + Custom collate function for DNA models. + """ + ref_sequences = [item["reference_sequence"] for item in batch] + alt_sequences = [item["variant_sequence"] for item in batch] + + # Tokenize DNA sequences separately + tokenized_ref = dna_tokenizer( + ref_sequences, + padding=True, + truncation=True, + max_length=max_length, + return_tensors="pt", + ) + + tokenized_alt = dna_tokenizer( + alt_sequences, + padding=True, + truncation=True, + max_length=max_length, + return_tensors="pt", + ) + + # Get labels + labels = [] + for item in batch: + label = label2id[item["answer"]] + labels.append(label) + + # Create labels tensor + labels_tensor = torch.tensor(labels, dtype=torch.long) + + tokenized_batch = { + "ref_ids": tokenized_ref.input_ids, + "ref_attention_mask": tokenized_ref.attention_mask, + "alt_ids": tokenized_alt.input_ids, + "alt_attention_mask": tokenized_alt.attention_mask, + "labels": labels_tensor, + } + + return tokenized_batch diff --git a/BioReason-main/bioreason/dataset/utils.py b/BioReason-main/bioreason/dataset/utils.py new file mode 100644 index 0000000000000000000000000000000000000000..25488fe729ff5b34be0488ddc5fdf2dbd6cb9993 --- /dev/null +++ b/BioReason-main/bioreason/dataset/utils.py @@ -0,0 +1,59 @@ +from datasets import Dataset as HFDataset +from torch.utils.data import Dataset as TorchDataset +from typing import Dict, Any, Union, List + + +def truncate_dna( + example: Dict[str, Any], truncate_dna_per_side: int = 1024 +) -> Dict[str, Any]: + """ + Truncate DNA sequences by removing a specified number of base pairs from both ends. + If the sequence is too short, it will return the middle portion. + """ + for key in ["reference_sequence", "variant_sequence"]: + sequence = example[key] + seq_len = len(sequence) + + if seq_len > 2 * truncate_dna_per_side + 8: + example[key] = sequence[truncate_dna_per_side:-truncate_dna_per_side] + + return example + + +def torch_to_hf_dataset(torch_dataset: TorchDataset) -> HFDataset: + """ + Convert a PyTorch Dataset to a Hugging Face Dataset. + + This function takes a PyTorch Dataset and converts it to a Hugging Face Dataset + by extracting all items and organizing them into a dictionary structure that + can be used to create a Hugging Face Dataset. + + Args: + torch_dataset: A PyTorch Dataset object to be converted + + Returns: + A Hugging Face Dataset containing the same data as the input PyTorch Dataset + """ + # Get first item to determine structure + if len(torch_dataset) == 0: + return HFDataset.from_dict({}) + + first_item = torch_dataset[0] + + # Initialize dictionary based on first item's keys + data_dict = ( + {k: [] for k in first_item.keys()} + if isinstance(first_item, dict) + else {"data": []} + ) + + # Populate dictionary + for i in range(len(torch_dataset)): + item = torch_dataset[i] + if isinstance(item, dict): + for k in data_dict: + data_dict[k].append(item[k]) + else: + data_dict["data"].append(item) + + return HFDataset.from_dict(data_dict) diff --git a/BioReason-main/bioreason/dataset/variant_effect.py b/BioReason-main/bioreason/dataset/variant_effect.py new file mode 100644 index 0000000000000000000000000000000000000000..f36b4a29943b9026cadfd2916fa4dc0e70f1722c --- /dev/null +++ b/BioReason-main/bioreason/dataset/variant_effect.py @@ -0,0 +1,98 @@ +import json +import os +import random +import sys +import torch +from torch.utils.data import Dataset, DataLoader +from typing import Any, Dict, List, Tuple + +from bioreason.dataset.utils import torch_to_hf_dataset +from bioreason.models.dl.processing_dl import DLProcessor +from trl.data_utils import maybe_apply_chat_template + + +def get_format_variant_effect_function(model_name: str) -> Any: + """ + Get the appropriate format function for a given model name. + """ + if model_name.lower() == "llm": + return format_variant_effect_for_llm + elif model_name.lower() == "dna-llm": + return format_variant_effect_for_dna_llm + else: + raise ValueError(f"Unsupported model name: {model_name}") + + +def clean_variant_effect_example(example: Dict[str, Any]) -> Dict[str, Any]: + """ + Clean a variant effect example. + """ + example['answer'] = example['answer'].split(";")[0].strip().lower() + return example + + +def clean_variant_effect_non_snv_example(example: Dict[str, Any]) -> Dict[str, Any]: + """ + Clean a variant effect non-SNV example. + """ + example['answer'] = example['answer'].replace("[", "").replace("]", "").replace("'", "").replace("_", " ").strip() + return example + + +def format_variant_effect_for_dna_llm(example: Dict[str, Any]) -> Dict[str, Any]: + """ + Format a VEP example into the required chat format for DNA-LLM. + """ + return { + "prompt": [ + { + "role": "user", + "content": [ + *({"type": "dna", "text": None} for _ in range(2)), + {"type": "text", "text": example["question"].strip()}, + ], + }, + { + "role": "assistant", + "reasoning_content": f"Answer: {example['answer'].strip()}", + "content": [ + {"type": "text", "text": f"Answer: {example['answer'].strip()}"}, + ], + }, + ], + "dna_sequences": [ + example["reference_sequence"], + example["variant_sequence"], + ], + "answer": example["answer"].strip(), + } + + +def format_variant_effect_for_llm(example: Dict[str, Any]) -> Dict[str, Any]: + """ + Format a VEP example into the required chat format for LLM. + """ + question = f"Reference sequence: {example['reference_sequence']}\nVariant sequence: {example['variant_sequence']}\nQuestion: {example['question']}" + return { + "prompt": [ + { + "role": "user", + "content": [ + *({"type": "dna", "text": None} for _ in range(2)), + {"type": "text", "text": question.strip()}, + ], + }, + { + "role": "assistant", + "reasoning_content": f"Answer: {example['answer'].strip()}", + "content": [ + {"type": "text", "text": f"Answer: {example['answer'].strip()}"}, + ], + }, + ], + "dna_sequences": [ + "", + "", + ], + "answer": example["answer"].strip(), + } \ No newline at end of file diff --git a/BioReason-main/bioreason/dna_modules/__init__.py b/BioReason-main/bioreason/dna_modules/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..0b2b166dfc42880b3737646adb004e43873633d9 --- /dev/null +++ b/BioReason-main/bioreason/dna_modules/__init__.py @@ -0,0 +1,4 @@ +from .dna_module import DNABaseModule +from .nucleotide_module import NucleotideDNAModule + +__all__ = ["DNABaseModule", "NucleotideDNAModule"] \ No newline at end of file diff --git a/BioReason-main/bioreason/dna_modules/dna_module.py b/BioReason-main/bioreason/dna_modules/dna_module.py new file mode 100644 index 0000000000000000000000000000000000000000..679d92745fec46687e73d99e5ade6f50a54c4811 --- /dev/null +++ b/BioReason-main/bioreason/dna_modules/dna_module.py @@ -0,0 +1,49 @@ +from abc import ABC, abstractmethod +from typing import Dict, Any, Union +import torch + +class DNABaseModule(ABC): + def __init__(self): + super().__init__() + + @abstractmethod + def get_dnallm_key(self): + pass + + @abstractmethod + def get_model_class(self, model_id: str, model_init_kwargs: dict): + pass + + def post_model_init(self, model, processing_class): + pass + + def is_embeds_input(self): + return False + + @abstractmethod + def get_processing_class(self): + pass + + @abstractmethod + def get_dnallm_modules_keywords(self): + pass + + @abstractmethod + def get_custom_multimodal_keywords(self): + pass + + @abstractmethod + def get_non_generate_params(self): + pass + + @abstractmethod + def get_custom_processing_keywords(self): + pass + + @abstractmethod + def prepare_prompt(self, processing_class, inputs: dict[str, Union[torch.Tensor, Any]]): + pass + + @abstractmethod + def prepare_model_inputs(self, processing_class, prompts_text, images, return_tensors, padding, padding_side, add_special_tokens): + pass \ No newline at end of file diff --git a/BioReason-main/bioreason/dna_modules/nucleotide_module.py b/BioReason-main/bioreason/dna_modules/nucleotide_module.py new file mode 100644 index 0000000000000000000000000000000000000000..ef40652ddbbfd817435460ae2d9033e823cd5096 --- /dev/null +++ b/BioReason-main/bioreason/dna_modules/nucleotide_module.py @@ -0,0 +1,263 @@ +from transformers import ( + Qwen2_5_VLForConditionalGeneration, + Qwen2VLForConditionalGeneration, + AutoProcessor, +) +from typing import Dict, Any, Union, List, Optional, Callable, Type +from trl.data_utils import maybe_apply_chat_template +from trl import SFTTrainer +import torch + +from bioreason.dna_modules.dna_module import DNABaseModule +from bioreason.models.dna_llm import DNALLMModel +from bioreason.models.dl.processing_dl import DLProcessor + + +class NucleotideDNAModule(DNABaseModule): + """ + DNA module implementation for NucleotideTransformer-based models. + + This module provides the interface between DNA-LLM models and the training + infrastructure, handling model loading, processing setup, and reward functions. + """ + + def __init__(self): + """Initialize the NucleotideDNAModule.""" + super().__init__() + + def get_dnallm_key(self) -> str: + """ + Get the key identifier for this DNA-LLM implementation. + + Returns: + String identifier for this module type + """ + return "qwen" + + def get_model_class(self, model_id: str, model_init_kwargs: Dict[str, Any]) -> Type: + """ + Return the appropriate model class based on model ID. + + Args: + model_id: Identifier for the model + model_init_kwargs: Initialization arguments for the model + + Returns: + The model class to instantiate + + Raises: + ValueError: If the model is not supported + """ + if "DNALLM" in model_id: + model_cls = DNALLMModel + else: + raise ValueError(f"Unsupported model: {model_id}") + return model_cls + + def post_model_init(self, model: Any, processing_class: Any) -> None: + """ + Perform any post-initialization setup on the model. + + Args: + model: The initialized model + processing_class: The processor for the model + """ + # No post-init needed for this implementation + pass + + def get_processing_class(self) -> Type: + """ + Get the processing class to use with this DNA-LLM model. + + Returns: + The processing class + """ + return DLProcessor + + def get_dnallm_modules_keywords(self) -> List[str]: + """ + Get keywords to identify DNA-specific modules in the model. + + Used to exclude DNA modules from LoRA adaptation during training. + + Returns: + List of keywords that identify DNA modules + """ + return ["dna"] + + def get_custom_multimodal_keywords(self) -> List[str]: + """ + Get keywords for multimodal inputs that should be passed to the model. + + Returns: + List of input keywords for multimodal processing + """ + return ["dna_tokenized", "batch_idx_map"] + + def get_non_generate_params(self) -> List[str]: + """ + Get parameter names that should be excluded from generation. + + Returns: + List of parameter names to exclude from generation calls + """ + return [] + + def get_custom_processing_keywords(self) -> List[tuple]: + """ + Get custom processing keywords for the processor. + + Returns: + List of (component, parameter) tuples for custom processing + """ + return [("dna_tokenizer", "max_length")] + + def prepare_prompt( + self, processing_class: Any, inputs: List[Dict[str, Union[torch.Tensor, Any]]] + ) -> List[str]: + """ + Prepare prompts from input examples. + + Args: + processing_class: The processor to use + inputs: List of input examples + + Returns: + List of prepared prompts + """ + prompts_text = [ + maybe_apply_chat_template(example, processing_class)["prompt"] + for example in inputs + ] + return prompts_text + + def prepare_model_inputs( + self, + processing_class: Any, + model: Any, + prompts_text: List[str], + batch_dna_sequences: List[List[str]], + return_tensors: str = "pt", + padding: bool = True, + padding_side: str = "left", + add_special_tokens: bool = False, + ) -> Dict[str, Any]: + """ + Prepare inputs for the model. + + Args: + processing_class: The processor to use + model: The model to prepare inputs for + prompts_text: List of text prompts + batch_dna_sequences: List of lists of DNA sequences + return_tensors: Return format for tensors + padding: Whether to pad inputs + padding_side: Side to pad on + add_special_tokens: Whether to add special tokens + + Returns: + Processed inputs for the model + """ + # Handle DataParallel wrapped models by accessing the module attribute if needed + max_length_text = model.max_length_text if not hasattr(model, 'module') else model.module.max_length_text + max_length_dna = model.max_length_dna if not hasattr(model, 'module') else model.module.max_length_dna + + prompt_inputs = processing_class( + text=prompts_text, + batch_dna_sequences=batch_dna_sequences, + return_tensors=return_tensors, + padding=padding, + padding_side=padding_side, + add_special_tokens=add_special_tokens, + max_length_text=max_length_text, + max_length_dna=max_length_dna, + ) + + return prompt_inputs + + def is_embeds_input(self) -> bool: + """ + Whether the model uses embeddings as input (instead of token IDs). + + Returns: + Boolean indicating if the model takes embedding inputs + """ + return True + + @staticmethod + def get_question_template() -> str: + """ + Get the template for formatting questions. + + Returns: + String template for questions + """ + return "{Question}" + + @staticmethod + def format_reward_rec(completions: List[Dict[str, Any]], **kwargs) -> List[float]: + """ + Check if the Qwen model output matches a specific format. + + Args: + completions: List of model completions + **kwargs: Additional arguments + + Returns: + List of reward scores (1.0 for match, 0.0 for no match) + """ + import re + import os + from datetime import datetime + + # Pattern to match the expected output format + pattern = r"\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " CHROM POS REF ALT LABEL SOURCE \\\n",
+ "0 chr1 976215 A G Pathogenic ClinVar \n",
+ "1 chr1 1050449 G A Pathogenic ClinVar \n",
+ "2 chr1 1050575 G C Pathogenic ClinVar \n",
+ "3 chr1 1213738 G A Pathogenic ClinVar \n",
+ "4 chr1 1232279 A G Pathogenic ClinVar \n",
+ "... ... ... .. .. ... ... \n",
+ "22249 chrY 2787412 C T Pathogenic ClinVar \n",
+ "22250 chrY 2787426 C G Pathogenic ClinVar \n",
+ "22251 chrY 2787515 C A Pathogenic ClinVar \n",
+ "22252 chrY 2787551 C T Pathogenic ClinVar \n",
+ "22253 chrY 7063898 A T Pathogenic ClinVar \n",
+ "\n",
+ " CONSEQUENCE ID \\\n",
+ "0 missense_variant 1320032 \n",
+ "1 missense_variant 1284257 \n",
+ "2 missense_variant 18241 \n",
+ "3 missense_variant 96692 \n",
+ "4 initiatior_codon_variant,missense_variant 60484 \n",
+ "... ... ... \n",
+ "22249 missense_variant 9747 \n",
+ "22250 missense_variant 9739 \n",
+ "22251 missense_variant 492908 \n",
+ "22252 missense_variant 9754 \n",
+ "22253 missense_variant 625467 \n",
+ "\n",
+ " REVIEW_STATUS GENE split \\\n",
+ "0 no_assertion_criteria_provided NaN train \n",
+ "1 no_assertion_criteria_provided NaN train \n",
+ "2 no_assertion_criteria_provided NaN train \n",
+ "3 no_assertion_criteria_provided NaN train \n",
+ "4 criteria_provided,_multiple_submitters,_no_con... NaN train \n",
+ "... ... ... ... \n",
+ "22249 no_assertion_criteria_provided NaN train \n",
+ "22250 criteria_provided,_single_submitter NaN train \n",
+ "22251 no_assertion_criteria_provided NaN train \n",
+ "22252 no_assertion_criteria_provided NaN train \n",
+ "22253 no_assertion_criteria_provided NaN train \n",
+ "\n",
+ " INT_LABEL \n",
+ "0 1 \n",
+ "1 1 \n",
+ "2 1 \n",
+ "3 1 \n",
+ "4 1 \n",
+ "... ... \n",
+ "22249 1 \n",
+ "22250 1 \n",
+ "22251 1 \n",
+ "22252 1 \n",
+ "22253 1 \n",
+ "\n",
+ "[22254 rows x 12 columns]"
+ ]
+ },
+ "execution_count": 17,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "clinvar_raw\n",
+ "\n",
+ "# Preview the cleaned dataset\n",
+ "if clinvar_raw is not None:\n",
+ " print(f\"📊 Dataset shape: {clinvar_raw.shape}\")\n",
+ " print(f\"📋 Column names: {list(clinvar_raw.columns)}\")\n",
+ " print(\"\\n🔍 First few rows:\")\n",
+ " display(clinvar_raw.head())\n",
+ " \n",
+ " # Check for any null values\n",
+ " null_counts = clinvar_raw.isnull().sum()\n",
+ " if null_counts.sum() > 0:\n",
+ " print(\"\\n⚠️ Null values found:\")\n",
+ " print(null_counts[null_counts > 0])\n",
+ "else:\n",
+ " print(\"❌ No data to display\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "e380634b-0c22-4d1e-8520-6fc5728e7de5",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Add new columns for gene information\n",
+ "if clinvar_raw is not None:\n",
+ " clinvar_raw['GENE_ID'] = \"\"\n",
+ " clinvar_raw['GENE'] = \"\"\n",
+ " print(\"✅ Added GENE_ID and GENE columns\")\n",
+ " print(f\"📊 Updated dataset shape: {clinvar_raw.shape}\")\n",
+ "else:\n",
+ " print(\"⚠️ Cannot add columns - data not loaded\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "92b159f5-694d-4ee4-9616-1ebf00f71904",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "| \n", + " | CHROM | \n", + "POS | \n", + "REF | \n", + "ALT | \n", + "LABEL | \n", + "SOURCE | \n", + "CONSEQUENCE | \n", + "ID | \n", + "REVIEW_STATUS | \n", + "GENE | \n", + "split | \n", + "INT_LABEL | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "chr1 | \n", + "976215 | \n", + "A | \n", + "G | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "1320032 | \n", + "no_assertion_criteria_provided | \n", + "NaN | \n", + "train | \n", + "1 | \n", + "
| 1 | \n", + "chr1 | \n", + "1050449 | \n", + "G | \n", + "A | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "1284257 | \n", + "no_assertion_criteria_provided | \n", + "NaN | \n", + "train | \n", + "1 | \n", + "
| 2 | \n", + "chr1 | \n", + "1050575 | \n", + "G | \n", + "C | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "18241 | \n", + "no_assertion_criteria_provided | \n", + "NaN | \n", + "train | \n", + "1 | \n", + "
| 3 | \n", + "chr1 | \n", + "1213738 | \n", + "G | \n", + "A | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "96692 | \n", + "no_assertion_criteria_provided | \n", + "NaN | \n", + "train | \n", + "1 | \n", + "
| 4 | \n", + "chr1 | \n", + "1232279 | \n", + "A | \n", + "G | \n", + "Pathogenic | \n", + "ClinVar | \n", + "initiatior_codon_variant,missense_variant | \n", + "60484 | \n", + "criteria_provided,_multiple_submitters,_no_con... | \n", + "NaN | \n", + "train | \n", + "1 | \n", + "
| ... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "
| 22249 | \n", + "chrY | \n", + "2787412 | \n", + "C | \n", + "T | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "9747 | \n", + "no_assertion_criteria_provided | \n", + "NaN | \n", + "train | \n", + "1 | \n", + "
| 22250 | \n", + "chrY | \n", + "2787426 | \n", + "C | \n", + "G | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "9739 | \n", + "criteria_provided,_single_submitter | \n", + "NaN | \n", + "train | \n", + "1 | \n", + "
| 22251 | \n", + "chrY | \n", + "2787515 | \n", + "C | \n", + "A | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "492908 | \n", + "no_assertion_criteria_provided | \n", + "NaN | \n", + "train | \n", + "1 | \n", + "
| 22252 | \n", + "chrY | \n", + "2787551 | \n", + "C | \n", + "T | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "9754 | \n", + "no_assertion_criteria_provided | \n", + "NaN | \n", + "train | \n", + "1 | \n", + "
| 22253 | \n", + "chrY | \n", + "7063898 | \n", + "A | \n", + "T | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "625467 | \n", + "no_assertion_criteria_provided | \n", + "NaN | \n", + "train | \n", + "1 | \n", + "
22254 rows × 12 columns
\n", + "\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " CHROM POS REF ALT LABEL SOURCE \\\n",
+ "0 chr1 976215 A G Pathogenic ClinVar \n",
+ "1 chr1 1050449 G A Pathogenic ClinVar \n",
+ "2 chr1 1050575 G C Pathogenic ClinVar \n",
+ "3 chr1 1213738 G A Pathogenic ClinVar \n",
+ "4 chr1 1232279 A G Pathogenic ClinVar \n",
+ "... ... ... .. .. ... ... \n",
+ "22249 chrY 2787412 C T Pathogenic ClinVar \n",
+ "22250 chrY 2787426 C G Pathogenic ClinVar \n",
+ "22251 chrY 2787515 C A Pathogenic ClinVar \n",
+ "22252 chrY 2787551 C T Pathogenic ClinVar \n",
+ "22253 chrY 7063898 A T Pathogenic ClinVar \n",
+ "\n",
+ " CONSEQUENCE ID \\\n",
+ "0 missense_variant 1320032 \n",
+ "1 missense_variant 1284257 \n",
+ "2 missense_variant 18241 \n",
+ "3 missense_variant 96692 \n",
+ "4 initiatior_codon_variant,missense_variant 60484 \n",
+ "... ... ... \n",
+ "22249 missense_variant 9747 \n",
+ "22250 missense_variant 9739 \n",
+ "22251 missense_variant 492908 \n",
+ "22252 missense_variant 9754 \n",
+ "22253 missense_variant 625467 \n",
+ "\n",
+ " REVIEW_STATUS GENE split \\\n",
+ "0 no_assertion_criteria_provided train \n",
+ "1 no_assertion_criteria_provided train \n",
+ "2 no_assertion_criteria_provided train \n",
+ "3 no_assertion_criteria_provided train \n",
+ "4 criteria_provided,_multiple_submitters,_no_con... train \n",
+ "... ... ... ... \n",
+ "22249 no_assertion_criteria_provided train \n",
+ "22250 criteria_provided,_single_submitter train \n",
+ "22251 no_assertion_criteria_provided train \n",
+ "22252 no_assertion_criteria_provided train \n",
+ "22253 no_assertion_criteria_provided train \n",
+ "\n",
+ " INT_LABEL GENE_ID \n",
+ "0 1 \n",
+ "1 1 \n",
+ "2 1 \n",
+ "3 1 \n",
+ "4 1 \n",
+ "... ... ... \n",
+ "22249 1 \n",
+ "22250 1 \n",
+ "22251 1 \n",
+ "22252 1 \n",
+ "22253 1 \n",
+ "\n",
+ "[22254 rows x 13 columns]"
+ ]
+ },
+ "execution_count": 34,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "clinvar_raw\n",
+ "\n",
+ "# Display updated dataset with new columns\n",
+ "if clinvar_raw is not None:\n",
+ " print(f\"📊 Dataset with new columns: {clinvar_raw.shape}\")\n",
+ " print(f\"📋 All columns: {list(clinvar_raw.columns)}\")\n",
+ " display(clinvar_raw.head())\n",
+ "else:\n",
+ " print(\"❌ No data to display\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "f36db716-392a-46a8-a404-d78165a4623c",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import pandas as pd\n",
+ "import xml.etree.ElementTree as ET\n",
+ "import os\n",
+ "\n",
+ "# Parse ClinVar XML files to extract gene information\n",
+ "# This processes each ClinVar ID and extracts gene symbols and IDs from XML records\n",
+ "\n",
+ "if clinvar_raw is not None:\n",
+ " # Load list of ClinVar IDs\n",
+ " try:\n",
+ " with open(\"Clinvar_ID.txt\", \"r\") as f:\n",
+ " clinvar_ids = [line.strip() for line in f if line.strip()]\n",
+ " \n",
+ " print(f\"📋 Processing {len(clinvar_ids)} ClinVar IDs\")\n",
+ " \n",
+ " processed_count = 0\n",
+ " error_count = 0\n",
+ " \n",
+ " # Process each ClinVar ID\n",
+ " for i, clinvar_id in enumerate(clinvar_ids):\n",
+ " if i % 100 == 0: # Progress indicator\n",
+ " print(f\"📊 Processing ID {i+1}/{len(clinvar_ids)}...\")\n",
+ " \n",
+ " try:\n",
+ " id_int = int(clinvar_id)\n",
+ " xml_path = f'data/{clinvar_id}.xml'\n",
+ " \n",
+ " # Check if XML file exists\n",
+ " if not os.path.exists(xml_path):\n",
+ " print(f\"⚠️ XML file not found: {xml_path}\")\n",
+ " continue\n",
+ " \n",
+ " # Parse XML file\n",
+ " with open(xml_path, 'r', encoding='utf-8') as file:\n",
+ " tree = ET.parse(file)\n",
+ " root = tree.getroot()\n",
+ " \n",
+ " # Check for error in XML\n",
+ " error_element = root.find(\".//error\")\n",
+ " if error_element is not None:\n",
+ " # Remove entries with errors\n",
+ " clinvar_raw = clinvar_raw[clinvar_raw[\"ID\"] != id_int]\n",
+ " error_count += 1\n",
+ " continue\n",
+ " \n",
+ " # Extract gene information\n",
+ " gene_names = []\n",
+ " gene_ids = []\n",
+ " \n",
+ " for gene in root.findall(\".//genes/gene\"):\n",
+ " symbol = gene.findtext(\"symbol\")\n",
+ " gene_id_data = gene.findtext(\"GeneID\")\n",
+ " \n",
+ " if symbol:\n",
+ " gene_names.append(symbol)\n",
+ " if gene_id_data:\n",
+ " gene_ids.append(gene_id_data)\n",
+ " \n",
+ " # Join multiple entries with commas\n",
+ " gene_name_str = \", \".join(gene_names) if gene_names else \"\"\n",
+ " gene_id_str = \", \".join(gene_ids) if gene_ids else \"\"\n",
+ " \n",
+ " # Update DataFrame\n",
+ " mask = clinvar_raw[\"ID\"] == id_int\n",
+ " if mask.any():\n",
+ " clinvar_raw.loc[mask, \"GENE\"] = gene_name_str\n",
+ " clinvar_raw.loc[mask, \"GENE_ID\"] = gene_id_str\n",
+ " processed_count += 1\n",
+ " \n",
+ " except ET.ParseError as e:\n",
+ " print(f\"⚠️ XML parsing error for {clinvar_id}: {e}\")\n",
+ " error_count += 1\n",
+ " except ValueError as e:\n",
+ " print(f\"⚠️ Invalid ClinVar ID {clinvar_id}: {e}\")\n",
+ " error_count += 1\n",
+ " except Exception as e:\n",
+ " print(f\"⚠️ Unexpected error processing {clinvar_id}: {e}\")\n",
+ " error_count += 1\n",
+ " \n",
+ " print(f\"\\n✅ Processing complete:\")\n",
+ " print(f\" 📊 Successfully processed: {processed_count}\")\n",
+ " print(f\" ❌ Errors encountered: {error_count}\")\n",
+ " print(f\" 📋 Final dataset shape: {clinvar_raw.shape}\")\n",
+ " \n",
+ " except FileNotFoundError:\n",
+ " print(\"❌ Error: Clinvar_ID.txt not found\")\n",
+ " print(\"Please run the ID extraction step first\")\n",
+ " except Exception as e:\n",
+ " print(f\"❌ Error during XML processing: {e}\")\n",
+ "else:\n",
+ " print(\"⚠️ Cannot process XML files - ClinVar data not loaded\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "ae0c9d8b-1b12-40a4-82ec-c3452e9dda90",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "| \n", + " | CHROM | \n", + "POS | \n", + "REF | \n", + "ALT | \n", + "LABEL | \n", + "SOURCE | \n", + "CONSEQUENCE | \n", + "ID | \n", + "REVIEW_STATUS | \n", + "GENE | \n", + "split | \n", + "INT_LABEL | \n", + "GENE_ID | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "chr1 | \n", + "976215 | \n", + "A | \n", + "G | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "1320032 | \n", + "no_assertion_criteria_provided | \n", + "\n", + " | train | \n", + "1 | \n", + "\n", + " |
| 1 | \n", + "chr1 | \n", + "1050449 | \n", + "G | \n", + "A | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "1284257 | \n", + "no_assertion_criteria_provided | \n", + "\n", + " | train | \n", + "1 | \n", + "\n", + " |
| 2 | \n", + "chr1 | \n", + "1050575 | \n", + "G | \n", + "C | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "18241 | \n", + "no_assertion_criteria_provided | \n", + "\n", + " | train | \n", + "1 | \n", + "\n", + " |
| 3 | \n", + "chr1 | \n", + "1213738 | \n", + "G | \n", + "A | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "96692 | \n", + "no_assertion_criteria_provided | \n", + "\n", + " | train | \n", + "1 | \n", + "\n", + " |
| 4 | \n", + "chr1 | \n", + "1232279 | \n", + "A | \n", + "G | \n", + "Pathogenic | \n", + "ClinVar | \n", + "initiatior_codon_variant,missense_variant | \n", + "60484 | \n", + "criteria_provided,_multiple_submitters,_no_con... | \n", + "\n", + " | train | \n", + "1 | \n", + "\n", + " |
| ... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "
| 22249 | \n", + "chrY | \n", + "2787412 | \n", + "C | \n", + "T | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "9747 | \n", + "no_assertion_criteria_provided | \n", + "\n", + " | train | \n", + "1 | \n", + "\n", + " |
| 22250 | \n", + "chrY | \n", + "2787426 | \n", + "C | \n", + "G | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "9739 | \n", + "criteria_provided,_single_submitter | \n", + "\n", + " | train | \n", + "1 | \n", + "\n", + " |
| 22251 | \n", + "chrY | \n", + "2787515 | \n", + "C | \n", + "A | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "492908 | \n", + "no_assertion_criteria_provided | \n", + "\n", + " | train | \n", + "1 | \n", + "\n", + " |
| 22252 | \n", + "chrY | \n", + "2787551 | \n", + "C | \n", + "T | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "9754 | \n", + "no_assertion_criteria_provided | \n", + "\n", + " | train | \n", + "1 | \n", + "\n", + " |
| 22253 | \n", + "chrY | \n", + "7063898 | \n", + "A | \n", + "T | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "625467 | \n", + "no_assertion_criteria_provided | \n", + "\n", + " | train | \n", + "1 | \n", + "\n", + " |
22254 rows × 13 columns
\n", + "\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " CHROM POS REF ALT LABEL SOURCE \\\n",
+ "0 chr1 976215 A G Pathogenic ClinVar \n",
+ "1 chr1 1050449 G A Pathogenic ClinVar \n",
+ "2 chr1 1050575 G C Pathogenic ClinVar \n",
+ "3 chr1 1213738 G A Pathogenic ClinVar \n",
+ "4 chr1 1232279 A G Pathogenic ClinVar \n",
+ "... ... ... .. .. ... ... \n",
+ "22249 chrY 2787412 C T Pathogenic ClinVar \n",
+ "22250 chrY 2787426 C G Pathogenic ClinVar \n",
+ "22251 chrY 2787515 C A Pathogenic ClinVar \n",
+ "22252 chrY 2787551 C T Pathogenic ClinVar \n",
+ "22253 chrY 7063898 A T Pathogenic ClinVar \n",
+ "\n",
+ " CONSEQUENCE ID \\\n",
+ "0 missense_variant 1320032 \n",
+ "1 missense_variant 1284257 \n",
+ "2 missense_variant 18241 \n",
+ "3 missense_variant 96692 \n",
+ "4 initiatior_codon_variant,missense_variant 60484 \n",
+ "... ... ... \n",
+ "22249 missense_variant 9747 \n",
+ "22250 missense_variant 9739 \n",
+ "22251 missense_variant 492908 \n",
+ "22252 missense_variant 9754 \n",
+ "22253 missense_variant 625467 \n",
+ "\n",
+ " REVIEW_STATUS GENE \\\n",
+ "0 no_assertion_criteria_provided PERM1 \n",
+ "1 no_assertion_criteria_provided AGRN \n",
+ "2 no_assertion_criteria_provided AGRN \n",
+ "3 no_assertion_criteria_provided TNFRSF4 \n",
+ "4 criteria_provided,_multiple_submitters,_no_con... B3GALT6 \n",
+ "... ... ... \n",
+ "22249 no_assertion_criteria_provided SRY \n",
+ "22250 criteria_provided,_single_submitter SRY \n",
+ "22251 no_assertion_criteria_provided SRY \n",
+ "22252 no_assertion_criteria_provided SRY \n",
+ "22253 no_assertion_criteria_provided LOC126057105, TBL1Y \n",
+ "\n",
+ " split INT_LABEL GENE_ID \n",
+ "0 train 1 84808 \n",
+ "1 train 1 375790 \n",
+ "2 train 1 375790 \n",
+ "3 train 1 7293 \n",
+ "4 train 1 126792 \n",
+ "... ... ... ... \n",
+ "22249 train 1 6736 \n",
+ "22250 train 1 6736 \n",
+ "22251 train 1 6736 \n",
+ "22252 train 1 6736 \n",
+ "22253 train 1 126057105, 90665 \n",
+ "\n",
+ "[22150 rows x 13 columns]"
+ ]
+ },
+ "execution_count": 39,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "clinvar_raw\n",
+ "\n",
+ "# Display the dataset with extracted gene information\n",
+ "if clinvar_raw is not None:\n",
+ " print(f\"📊 Dataset after gene extraction: {clinvar_raw.shape}\")\n",
+ " \n",
+ " # Show statistics\n",
+ " gene_filled = (clinvar_raw['GENE'] != '').sum()\n",
+ " gene_id_filled = (clinvar_raw['GENE_ID'] != '').sum()\n",
+ " \n",
+ " print(f\"📋 Entries with gene names: {gene_filled} ({gene_filled/len(clinvar_raw)*100:.1f}%)\")\n",
+ " print(f\"📋 Entries with gene IDs: {gene_id_filled} ({gene_id_filled/len(clinvar_raw)*100:.1f}%)\")\n",
+ " \n",
+ " # Show sample data\n",
+ " display(clinvar_raw.head(10))\n",
+ "else:\n",
+ " print(\"❌ No data to display\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "b76910bd-aa86-4943-a0f2-dcf9756ad81d",
+ "metadata": {},
+ "source": [
+ "## Disease/Phenotype Information Extraction\n",
+ "\n",
+ "This section extracts disease and phenotype information from the ClinVar XML records. Each variant may be associated with multiple diseases, so the data is expanded to create one row per variant-disease combination.\n",
+ "\n",
+ "### Putting in the Disease Name"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "54ccd972-5804-4d63-9012-5531034d2b60",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Extract disease/phenotype information from ClinVar XML files\n",
+ "# This creates multiple rows for variants associated with multiple diseases\n",
+ "\n",
+ "if clinvar_raw is not None:\n",
+ " try:\n",
+ " # Load ClinVar IDs\n",
+ " with open(\"Clinvar_ID.txt\", \"r\") as f:\n",
+ " clinvar_ids = [line.strip() for line in f if line.strip()]\n",
+ " \n",
+ " print(f\"📋 Processing {len(clinvar_ids)} ClinVar IDs for disease extraction\")\n",
+ " \n",
+ " # Ensure ID column is integer type\n",
+ " clinvar_raw[\"ID\"] = clinvar_raw[\"ID\"].astype(int)\n",
+ " \n",
+ " # Create new DataFrame to store expanded data\n",
+ " clinvar_data = pd.DataFrame(columns=clinvar_raw.columns.tolist() + [\"Disease\"])\n",
+ " \n",
+ " processed_count = 0\n",
+ " disease_count = 0\n",
+ " \n",
+ " # Process each ClinVar ID\n",
+ " for i, clinvar_id in enumerate(clinvar_ids):\n",
+ " if i % 100 == 0: # Progress indicator\n",
+ " print(f\"📊 Processing disease info {i+1}/{len(clinvar_ids)}...\")\n",
+ " \n",
+ " try:\n",
+ " id_int = int(clinvar_id)\n",
+ " xml_path = f\"data/{clinvar_id}.xml\"\n",
+ " \n",
+ " if not os.path.exists(xml_path):\n",
+ " continue\n",
+ " \n",
+ " # Parse XML\n",
+ " tree = ET.parse(xml_path)\n",
+ " root = tree.getroot()\n",
+ " \n",
+ " # Extract all trait names (diseases/phenotypes)\n",
+ " trait_names = []\n",
+ " for trait in root.findall(\".//trait\"):\n",
+ " trait_name = trait.findtext(\"trait_name\")\n",
+ " if trait_name:\n",
+ " trait_names.append(trait_name)\n",
+ " \n",
+ " # Filter out 'not provided' if other traits exist\n",
+ " filtered_traits = [t for t in trait_names if t.lower() != \"not provided\"]\n",
+ " if not filtered_traits and \"not provided\" in [t.lower() for t in trait_names]:\n",
+ " filtered_traits = [\"not provided\"]\n",
+ " \n",
+ " # If no traits found, use empty string\n",
+ " if not filtered_traits:\n",
+ " filtered_traits = [\"\"]\n",
+ " \n",
+ " # Create one row for each disease/trait\n",
+ " base_row = clinvar_raw[clinvar_raw[\"ID\"] == id_int]\n",
+ " if not base_row.empty:\n",
+ " for disease_name in filtered_traits:\n",
+ " new_row = base_row.copy()\n",
+ " new_row[\"Disease\"] = disease_name\n",
+ " clinvar_data = pd.concat([clinvar_data, new_row], ignore_index=True)\n",
+ " disease_count += 1\n",
+ " processed_count += 1\n",
+ " \n",
+ " except ET.ParseError as e:\n",
+ " print(f\"⚠️ XML parsing error for {clinvar_id}: {e}\")\n",
+ " except Exception as e:\n",
+ " print(f\"⚠️ Error processing {clinvar_id}: {e}\")\n",
+ " \n",
+ " print(f\"\\n✅ Disease extraction complete:\")\n",
+ " print(f\" 📊 Variants processed: {processed_count}\")\n",
+ " print(f\" 🔬 Total variant-disease pairs: {disease_count}\")\n",
+ " print(f\" 📋 Final dataset shape: {clinvar_data.shape}\")\n",
+ " \n",
+ " # Save intermediate results\n",
+ " clinvar_data.to_csv(\"clinvar_with_disease.csv\", sep='\\t', index=False)\n",
+ " print(\"💾 Saved results to clinvar_with_disease.csv\")\n",
+ " \n",
+ " except FileNotFoundError:\n",
+ " print(\"❌ Error: Required files not found\")\n",
+ " print(\"Please ensure Clinvar_ID.txt exists and XML files are downloaded\")\n",
+ " clinvar_data = None\n",
+ " except Exception as e:\n",
+ " print(f\"❌ Error during disease extraction: {e}\")\n",
+ " clinvar_data = None\n",
+ "else:\n",
+ " print(\"⚠️ Cannot extract diseases - ClinVar data not loaded\")\n",
+ " clinvar_data = None"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "277445cd-72b9-44a4-a257-49cd3202e501",
+ "metadata": {
+ "scrolled": true
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "| \n", + " | CHROM | \n", + "POS | \n", + "REF | \n", + "ALT | \n", + "LABEL | \n", + "SOURCE | \n", + "CONSEQUENCE | \n", + "ID | \n", + "REVIEW_STATUS | \n", + "GENE | \n", + "split | \n", + "INT_LABEL | \n", + "GENE_ID | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "chr1 | \n", + "976215 | \n", + "A | \n", + "G | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "1320032 | \n", + "no_assertion_criteria_provided | \n", + "PERM1 | \n", + "train | \n", + "1 | \n", + "84808 | \n", + "
| 1 | \n", + "chr1 | \n", + "1050449 | \n", + "G | \n", + "A | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "1284257 | \n", + "no_assertion_criteria_provided | \n", + "AGRN | \n", + "train | \n", + "1 | \n", + "375790 | \n", + "
| 2 | \n", + "chr1 | \n", + "1050575 | \n", + "G | \n", + "C | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "18241 | \n", + "no_assertion_criteria_provided | \n", + "AGRN | \n", + "train | \n", + "1 | \n", + "375790 | \n", + "
| 3 | \n", + "chr1 | \n", + "1213738 | \n", + "G | \n", + "A | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "96692 | \n", + "no_assertion_criteria_provided | \n", + "TNFRSF4 | \n", + "train | \n", + "1 | \n", + "7293 | \n", + "
| 4 | \n", + "chr1 | \n", + "1232279 | \n", + "A | \n", + "G | \n", + "Pathogenic | \n", + "ClinVar | \n", + "initiatior_codon_variant,missense_variant | \n", + "60484 | \n", + "criteria_provided,_multiple_submitters,_no_con... | \n", + "B3GALT6 | \n", + "train | \n", + "1 | \n", + "126792 | \n", + "
| ... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "
| 22249 | \n", + "chrY | \n", + "2787412 | \n", + "C | \n", + "T | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "9747 | \n", + "no_assertion_criteria_provided | \n", + "SRY | \n", + "train | \n", + "1 | \n", + "6736 | \n", + "
| 22250 | \n", + "chrY | \n", + "2787426 | \n", + "C | \n", + "G | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "9739 | \n", + "criteria_provided,_single_submitter | \n", + "SRY | \n", + "train | \n", + "1 | \n", + "6736 | \n", + "
| 22251 | \n", + "chrY | \n", + "2787515 | \n", + "C | \n", + "A | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "492908 | \n", + "no_assertion_criteria_provided | \n", + "SRY | \n", + "train | \n", + "1 | \n", + "6736 | \n", + "
| 22252 | \n", + "chrY | \n", + "2787551 | \n", + "C | \n", + "T | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "9754 | \n", + "no_assertion_criteria_provided | \n", + "SRY | \n", + "train | \n", + "1 | \n", + "6736 | \n", + "
| 22253 | \n", + "chrY | \n", + "7063898 | \n", + "A | \n", + "T | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "625467 | \n", + "no_assertion_criteria_provided | \n", + "LOC126057105, TBL1Y | \n", + "train | \n", + "1 | \n", + "126057105, 90665 | \n", + "
22150 rows × 13 columns
\n", + "\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " CHROM POS REF ALT LABEL SOURCE CONSEQUENCE ID \\\n",
+ "0 chr1 976215 A G Pathogenic ClinVar missense_variant 1320032 \n",
+ "1 chr1 976215 A G Pathogenic ClinVar missense_variant 1320032 \n",
+ "2 chr1 1050449 G A Pathogenic ClinVar missense_variant 1284257 \n",
+ "3 chr1 1050575 G C Pathogenic ClinVar missense_variant 18241 \n",
+ "4 chr1 1213738 G A Pathogenic ClinVar missense_variant 96692 \n",
+ "... ... ... .. .. ... ... ... ... \n",
+ "32680 chrY 2787412 C T Pathogenic ClinVar missense_variant 9747 \n",
+ "32681 chrY 2787426 C G Pathogenic ClinVar missense_variant 9739 \n",
+ "32682 chrY 2787515 C A Pathogenic ClinVar missense_variant 492908 \n",
+ "32683 chrY 2787551 C T Pathogenic ClinVar missense_variant 9754 \n",
+ "32684 chrY 7063898 A T Pathogenic ClinVar missense_variant 625467 \n",
+ "\n",
+ " REVIEW_STATUS GENE split \\\n",
+ "0 no_assertion_criteria_provided PERM1 train \n",
+ "1 no_assertion_criteria_provided PERM1 train \n",
+ "2 no_assertion_criteria_provided AGRN train \n",
+ "3 no_assertion_criteria_provided AGRN train \n",
+ "4 no_assertion_criteria_provided TNFRSF4 train \n",
+ "... ... ... ... \n",
+ "32680 no_assertion_criteria_provided SRY train \n",
+ "32681 criteria_provided,_single_submitter SRY train \n",
+ "32682 no_assertion_criteria_provided SRY train \n",
+ "32683 no_assertion_criteria_provided SRY train \n",
+ "32684 no_assertion_criteria_provided LOC126057105, TBL1Y train \n",
+ "\n",
+ " INT_LABEL GENE_ID \\\n",
+ "0 1 84808 \n",
+ "1 1 84808 \n",
+ "2 1 375790 \n",
+ "3 1 375790 \n",
+ "4 1 7293 \n",
+ "... ... ... \n",
+ "32680 1 6736 \n",
+ "32681 1 6736 \n",
+ "32682 1 6736 \n",
+ "32683 1 6736 \n",
+ "32684 1 126057105, 90665 \n",
+ "\n",
+ " Disease \n",
+ "0 Renal tubular epithelial cell apoptosis \n",
+ "1 Neutrophil inclusion bodies \n",
+ "2 Congenital myasthenic syndrome 8 \n",
+ "3 Congenital myasthenic syndrome 8 \n",
+ "4 Combined immunodeficiency due to OX40 deficiency \n",
+ "... ... \n",
+ "32680 46,XY sex reversal 1 \n",
+ "32681 not provided \n",
+ "32682 46,XY sex reversal 1 \n",
+ "32683 46,XY sex reversal 1 \n",
+ "32684 Deafness, Y-linked 2 \n",
+ "\n",
+ "[32685 rows x 14 columns]"
+ ]
+ },
+ "execution_count": 51,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "clinvar_data\n",
+ "\n",
+ "# Display the dataset with disease information\n",
+ "if 'clinvar_data' in locals() and clinvar_data is not None:\n",
+ " print(f\"📊 Dataset with diseases: {clinvar_data.shape}\")\n",
+ " \n",
+ " # Show disease statistics\n",
+ " disease_counts = clinvar_data['Disease'].value_counts()\n",
+ " print(f\"\\n🔬 Disease distribution (top 10):\")\n",
+ " print(disease_counts.head(10))\n",
+ " \n",
+ " # Show sample data\n",
+ " print(\"\\n🔍 Sample data:\")\n",
+ " display(clinvar_data.head())\n",
+ "else:\n",
+ " print(\"❌ No disease data to display\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c6b1c6dc-33ed-4f57-a385-29816f4c9984",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "np.int64(2749)"
+ ]
+ },
+ "execution_count": 53,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Count entries with 'not provided' disease information\n",
+ "if 'clinvar_data' in locals() and clinvar_data is not None:\n",
+ " not_provided_count = (clinvar_data[\"Disease\"] == \"not provided\").sum()\n",
+ " total_count = len(clinvar_data)\n",
+ " \n",
+ " print(f\"📊 Entries with 'not provided' disease: {not_provided_count}\")\n",
+ " print(f\"📊 Total entries: {total_count}\")\n",
+ " print(f\"📊 Percentage: {not_provided_count/total_count*100:.1f}%\")\n",
+ "else:\n",
+ " print(\"❌ Cannot calculate statistics - data not available\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8a7513ee-96b2-4c7d-8678-0195eb826aa5",
+ "metadata": {},
+ "source": [
+ "## Gene ID to Gene Name Mapping\n",
+ "\n",
+ "This section converts gene IDs to human-readable gene names using NCBI Entrez utilities.\n",
+ "\n",
+ "**Prerequisites**: NCBI Entrez Direct tools must be installed:\n",
+ "- macOS: `brew install brewsci/bio/edirect`\n",
+ "- Linux: Follow NCBI EDirect installation guide\n",
+ "\n",
+ "The process:\n",
+ "1. Extract unique gene IDs from the dataset\n",
+ "2. Use `esummary` to fetch gene descriptions from NCBI\n",
+ "3. Create a mapping dictionary\n",
+ "4. Apply the mapping to add gene names to the dataset"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "ee0d3632-d11e-4429-bb50-5eb9ba55d424",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "#!/usr/bin/env python3\n",
+ "\n",
+ "import os\n",
+ "import pandas as pd\n",
+ "\n",
+ "# Extract unique gene IDs and create mapping file\n",
+ "# This prepares the gene ID list for NCBI lookup\n",
+ "\n",
+ "if 'clinvar_data' in locals() and clinvar_data is not None:\n",
+ " # Extract all unique gene IDs\n",
+ " all_gene_ids = set()\n",
+ " \n",
+ " for gene_id_str in clinvar_data['GENE_ID'].dropna():\n",
+ " if gene_id_str.strip(): # Skip empty strings\n",
+ " # Split comma-separated IDs\n",
+ " ids = [gid.strip() for gid in gene_id_str.split(',') if gid.strip()]\n",
+ " all_gene_ids.update(ids)\n",
+ " \n",
+ " # Save unique gene IDs to file\n",
+ " with open(\"gene_id.txt\", 'w') as f:\n",
+ " for gene_id in sorted(all_gene_ids):\n",
+ " f.write(f\"{gene_id}\\n\")\n",
+ " \n",
+ " print(f\"✅ Extracted {len(all_gene_ids)} unique gene IDs to gene_id.txt\")\n",
+ " \n",
+ " # Create the shell script for NCBI lookup\n",
+ " script_content = '''#!/bin/bash\n",
+ "\n",
+ "input_file=\"gene_id.txt\"\n",
+ "output_file=\"gene_id_to_name.json\"\n",
+ "\n",
+ "# Check if input file exists\n",
+ "if [ ! -f \"$input_file\" ]; then\n",
+ " echo \"❌ Error: $input_file not found\"\n",
+ " exit 1\n",
+ "fi\n",
+ "\n",
+ "# Check if EDirect tools are available\n",
+ "if ! command -v esummary &> /dev/null; then\n",
+ " echo \"❌ Error: NCBI EDirect tools not found\"\n",
+ " echo \"Please install: brew install brewsci/bio/edirect (macOS)\"\n",
+ " exit 1\n",
+ "fi\n",
+ "\n",
+ "echo \"🚀 Starting gene ID to name mapping...\"\n",
+ "\n",
+ "# Start JSON object\n",
+ "echo \"{\" > \"$output_file\"\n",
+ "\n",
+ "first_entry=true\n",
+ "total_lines=$(wc -l < \"$input_file\")\n",
+ "current_line=0\n",
+ "\n",
+ "while IFS= read -r gene_id; do\n",
+ " # Skip empty lines\n",
+ " [[ -z \"$gene_id\" ]] && continue\n",
+ " \n",
+ " current_line=$((current_line + 1))\n",
+ " \n",
+ " # Progress indicator\n",
+ " if (( current_line % 50 == 0 )); then\n",
+ " echo \"📊 Processing $current_line/$total_lines gene IDs...\"\n",
+ " fi\n",
+ " \n",
+ " # Fetch gene description using Entrez Direct\n",
+ " description=$(esummary -db gene -id \"$gene_id\" 2>/dev/null | xtract -pattern DocumentSummary -element Description)\n",
+ " \n",
+ " # Handle empty description\n",
+ " if [ -z \"$description\" ]; then\n",
+ " description=\"Unknown\"\n",
+ " fi\n",
+ " \n",
+ " # JSON escape quotes and other special characters\n",
+ " description=$(printf '%s' \"$description\" | sed 's/\"/\\\\\"/g')\n",
+ " \n",
+ " # Add comma if not the first entry\n",
+ " if [ \"$first_entry\" = true ]; then\n",
+ " first_entry=false\n",
+ " else\n",
+ " echo \",\" >> \"$output_file\"\n",
+ " fi\n",
+ " \n",
+ " # Append key-value pair\n",
+ " echo \" \\\"$gene_id\\\": \\\"$description\\\"\" >> \"$output_file\"\n",
+ " \n",
+ "done < \"$input_file\"\n",
+ "\n",
+ "# Close JSON object\n",
+ "echo \"\" >> \"$output_file\"\n",
+ "echo \"}\" >> \"$output_file\"\n",
+ "\n",
+ "echo \"✅ Gene ID to name mapping completed\"\n",
+ "echo \"💾 Results saved to $output_file\"\n",
+ "'''\n",
+ " \n",
+ " # Write the script\n",
+ " with open(\"gene_mapping.sh\", 'w') as f:\n",
+ " f.write(script_content)\n",
+ " \n",
+ " # Make executable\n",
+ " os.chmod(\"gene_mapping.sh\", 0o755)\n",
+ " \n",
+ " print(\"✅ Created gene_mapping.sh script\")\n",
+ " print(\"\\n🚀 To run the gene mapping:\")\n",
+ " print(\" ./gene_mapping.sh\")\n",
+ " print(\"\\n⚠️ Note: This requires NCBI EDirect tools to be installed\")\n",
+ " \n",
+ "else:\n",
+ " print(\"⚠️ Cannot create gene mapping - data not available\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "1957ef57-1af8-46a1-8d1b-147f6b423619",
+ "metadata": {},
+ "source": [
+ "## Apply Gene Name Mapping\n",
+ "\n",
+ "Load the gene ID to name mapping and apply it to the dataset to add human-readable gene names.\n",
+ "\n",
+ "Read json and add it to the clinvar_data df"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b39be718-c0ae-4aae-b1d8-d0c872947ec2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "\n",
+ "# Load gene ID to name mapping and apply to dataset\n",
+ "\n",
+ "if 'clinvar_data' in locals() and clinvar_data is not None:\n",
+ " try:\n",
+ " # Load gene ID → name mapping\n",
+ " with open(\"gene_id_to_name.json\", \"r\") as f:\n",
+ " gene_id_dict = json.load(f)\n",
+ " \n",
+ " print(f\"✅ Loaded mapping for {len(gene_id_dict)} gene IDs\")\n",
+ " \n",
+ " # Function to convert gene IDs to gene names\n",
+ " def get_gene_names(gene_id_str):\n",
+ " if pd.isna(gene_id_str) or not gene_id_str.strip():\n",
+ " return \"\"\n",
+ " \n",
+ " gene_ids = [gid.strip() for gid in gene_id_str.split(\",\") if gid.strip()]\n",
+ " gene_names = []\n",
+ " \n",
+ " for gid in gene_ids:\n",
+ " gene_name = gene_id_dict.get(gid, f\"Unknown_ID_{gid}\")\n",
+ " gene_names.append(gene_name)\n",
+ " \n",
+ " return \" | \".join(gene_names)\n",
+ " \n",
+ " # Apply mapping to create gene names column\n",
+ " print(\"📊 Applying gene name mapping...\")\n",
+ " clinvar_data[\"GENE_Name\"] = clinvar_data[\"GENE_ID\"].apply(get_gene_names)\n",
+ " \n",
+ " # Statistics\n",
+ " mapped_count = (clinvar_data[\"GENE_Name\"] != \"\").sum()\n",
+ " print(f\"✅ Gene names mapped for {mapped_count} entries ({mapped_count/len(clinvar_data)*100:.1f}%)\")\n",
+ " \n",
+ " # Show sample mappings\n",
+ " sample_data = clinvar_data[clinvar_data[\"GENE_Name\"] != \"\"][[\"GENE_ID\", \"GENE_Name\"]].head()\n",
+ " if not sample_data.empty:\n",
+ " print(\"\\n🔍 Sample gene ID to name mappings:\")\n",
+ " for _, row in sample_data.iterrows():\n",
+ " print(f\" {row['GENE_ID']} → {row['GENE_Name'][:100]}{'...' if len(row['GENE_Name']) > 100 else ''}\")\n",
+ " \n",
+ " except FileNotFoundError:\n",
+ " print(\"❌ Error: gene_id_to_name.json not found\")\n",
+ " print(\"Please run the gene mapping script first: ./gene_mapping.sh\")\n",
+ " # Create empty column as fallback\n",
+ " clinvar_data[\"GENE_Name\"] = \"\"\n",
+ " except json.JSONDecodeError as e:\n",
+ " print(f\"❌ Error parsing JSON mapping file: {e}\")\n",
+ " clinvar_data[\"GENE_Name\"] = \"\"\n",
+ " except Exception as e:\n",
+ " print(f\"❌ Error applying gene mapping: {e}\")\n",
+ " clinvar_data[\"GENE_Name\"] = \"\"\n",
+ "else:\n",
+ " print(\"⚠️ Cannot apply gene mapping - data not available\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "4b7a44c2-7823-47c1-b268-22a1815ffd09",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "| \n", + " | CHROM | \n", + "POS | \n", + "REF | \n", + "ALT | \n", + "LABEL | \n", + "SOURCE | \n", + "CONSEQUENCE | \n", + "ID | \n", + "REVIEW_STATUS | \n", + "GENE | \n", + "split | \n", + "INT_LABEL | \n", + "GENE_ID | \n", + "Disease | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "chr1 | \n", + "976215 | \n", + "A | \n", + "G | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "1320032 | \n", + "no_assertion_criteria_provided | \n", + "PERM1 | \n", + "train | \n", + "1 | \n", + "84808 | \n", + "Renal tubular epithelial cell apoptosis | \n", + "
| 1 | \n", + "chr1 | \n", + "976215 | \n", + "A | \n", + "G | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "1320032 | \n", + "no_assertion_criteria_provided | \n", + "PERM1 | \n", + "train | \n", + "1 | \n", + "84808 | \n", + "Neutrophil inclusion bodies | \n", + "
| 2 | \n", + "chr1 | \n", + "1050449 | \n", + "G | \n", + "A | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "1284257 | \n", + "no_assertion_criteria_provided | \n", + "AGRN | \n", + "train | \n", + "1 | \n", + "375790 | \n", + "Congenital myasthenic syndrome 8 | \n", + "
| 3 | \n", + "chr1 | \n", + "1050575 | \n", + "G | \n", + "C | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "18241 | \n", + "no_assertion_criteria_provided | \n", + "AGRN | \n", + "train | \n", + "1 | \n", + "375790 | \n", + "Congenital myasthenic syndrome 8 | \n", + "
| 4 | \n", + "chr1 | \n", + "1213738 | \n", + "G | \n", + "A | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "96692 | \n", + "no_assertion_criteria_provided | \n", + "TNFRSF4 | \n", + "train | \n", + "1 | \n", + "7293 | \n", + "Combined immunodeficiency due to OX40 deficiency | \n", + "
| ... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "
| 32680 | \n", + "chrY | \n", + "2787412 | \n", + "C | \n", + "T | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "9747 | \n", + "no_assertion_criteria_provided | \n", + "SRY | \n", + "train | \n", + "1 | \n", + "6736 | \n", + "46,XY sex reversal 1 | \n", + "
| 32681 | \n", + "chrY | \n", + "2787426 | \n", + "C | \n", + "G | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "9739 | \n", + "criteria_provided,_single_submitter | \n", + "SRY | \n", + "train | \n", + "1 | \n", + "6736 | \n", + "not provided | \n", + "
| 32682 | \n", + "chrY | \n", + "2787515 | \n", + "C | \n", + "A | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "492908 | \n", + "no_assertion_criteria_provided | \n", + "SRY | \n", + "train | \n", + "1 | \n", + "6736 | \n", + "46,XY sex reversal 1 | \n", + "
| 32683 | \n", + "chrY | \n", + "2787551 | \n", + "C | \n", + "T | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "9754 | \n", + "no_assertion_criteria_provided | \n", + "SRY | \n", + "train | \n", + "1 | \n", + "6736 | \n", + "46,XY sex reversal 1 | \n", + "
| 32684 | \n", + "chrY | \n", + "7063898 | \n", + "A | \n", + "T | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "625467 | \n", + "no_assertion_criteria_provided | \n", + "LOC126057105, TBL1Y | \n", + "train | \n", + "1 | \n", + "126057105, 90665 | \n", + "Deafness, Y-linked 2 | \n", + "
32685 rows × 14 columns
\n", + "\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " CHROM POS REF ALT LABEL SOURCE CONSEQUENCE ID \\\n",
+ "0 chr1 976215 A G Pathogenic ClinVar missense_variant 1320032 \n",
+ "1 chr1 976215 A G Pathogenic ClinVar missense_variant 1320032 \n",
+ "2 chr1 1050449 G A Pathogenic ClinVar missense_variant 1284257 \n",
+ "3 chr1 1050575 G C Pathogenic ClinVar missense_variant 18241 \n",
+ "4 chr1 1213738 G A Pathogenic ClinVar missense_variant 96692 \n",
+ "... ... ... .. .. ... ... ... ... \n",
+ "32680 chrY 2787412 C T Pathogenic ClinVar missense_variant 9747 \n",
+ "32681 chrY 2787426 C G Pathogenic ClinVar missense_variant 9739 \n",
+ "32682 chrY 2787515 C A Pathogenic ClinVar missense_variant 492908 \n",
+ "32683 chrY 2787551 C T Pathogenic ClinVar missense_variant 9754 \n",
+ "32684 chrY 7063898 A T Pathogenic ClinVar missense_variant 625467 \n",
+ "\n",
+ " REVIEW_STATUS GENE split \\\n",
+ "0 no_assertion_criteria_provided PERM1 train \n",
+ "1 no_assertion_criteria_provided PERM1 train \n",
+ "2 no_assertion_criteria_provided AGRN train \n",
+ "3 no_assertion_criteria_provided AGRN train \n",
+ "4 no_assertion_criteria_provided TNFRSF4 train \n",
+ "... ... ... ... \n",
+ "32680 no_assertion_criteria_provided SRY train \n",
+ "32681 criteria_provided,_single_submitter SRY train \n",
+ "32682 no_assertion_criteria_provided SRY train \n",
+ "32683 no_assertion_criteria_provided SRY train \n",
+ "32684 no_assertion_criteria_provided LOC126057105, TBL1Y train \n",
+ "\n",
+ " INT_LABEL GENE_ID \\\n",
+ "0 1 84808 \n",
+ "1 1 84808 \n",
+ "2 1 375790 \n",
+ "3 1 375790 \n",
+ "4 1 7293 \n",
+ "... ... ... \n",
+ "32680 1 6736 \n",
+ "32681 1 6736 \n",
+ "32682 1 6736 \n",
+ "32683 1 6736 \n",
+ "32684 1 126057105, 90665 \n",
+ "\n",
+ " Disease \\\n",
+ "0 Renal tubular epithelial cell apoptosis \n",
+ "1 Neutrophil inclusion bodies \n",
+ "2 Congenital myasthenic syndrome 8 \n",
+ "3 Congenital myasthenic syndrome 8 \n",
+ "4 Combined immunodeficiency due to OX40 deficiency \n",
+ "... ... \n",
+ "32680 46,XY sex reversal 1 \n",
+ "32681 not provided \n",
+ "32682 46,XY sex reversal 1 \n",
+ "32683 46,XY sex reversal 1 \n",
+ "32684 Deafness, Y-linked 2 \n",
+ "\n",
+ " GENE_Name \n",
+ "0 PPARGC1 and ESRR induced regulator, muscle 1 \n",
+ "1 PPARGC1 and ESRR induced regulator, muscle 1 \n",
+ "2 agrin \n",
+ "3 agrin \n",
+ "4 TNF receptor superfamily member 4 \n",
+ "... ... \n",
+ "32680 sex determining region Y \n",
+ "32681 sex determining region Y \n",
+ "32682 sex determining region Y \n",
+ "32683 sex determining region Y \n",
+ "32684 P300/CBP strongly-dependent group 1 enhancer G... \n",
+ "\n",
+ "[32685 rows x 15 columns]"
+ ]
+ },
+ "execution_count": 59,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Display final dataset with all extracted information\n",
+ "if 'clinvar_data' in locals() and clinvar_data is not None:\n",
+ " print(f\"📊 Final dataset shape: {clinvar_data.shape}\")\n",
+ " print(f\"📋 Columns: {list(clinvar_data.columns)}\")\n",
+ " \n",
+ " # Data completeness statistics\n",
+ " print(\"\\n📈 Data Completeness:\")\n",
+ " for col in ['GENE', 'GENE_ID', 'GENE_Name', 'Disease']:\n",
+ " if col in clinvar_data.columns:\n",
+ " filled_count = (clinvar_data[col] != '').sum()\n",
+ " print(f\" {col}: {filled_count}/{len(clinvar_data)} ({filled_count/len(clinvar_data)*100:.1f}%)\")\n",
+ " \n",
+ " # Sample data\n",
+ " print(\"\\n🔍 Sample data:\")\n",
+ " display(clinvar_data.head())\n",
+ " \n",
+ " # Memory usage\n",
+ " memory_mb = clinvar_data.memory_usage(deep=True).sum() / 1024 / 1024\n",
+ " print(f\"\\n💾 Dataset memory usage: {memory_mb:.1f} MB\")\n",
+ "else:\n",
+ " print(\"❌ No final data to display\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c545ae83-5cd1-4e29-87fd-69389bdb153f",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "'P300/CBP strongly-dependent group 1 enhancer GRCh37_chrY:6931456-6932655| transducin beta like 1 Y-linked'"
+ ]
+ },
+ "execution_count": 60,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Show example of gene name mapping\n",
+ "if 'clinvar_data' in locals() and clinvar_data is not None and len(clinvar_data) > 32684:\n",
+ " example_gene_name = clinvar_data.iloc[32684]['GENE_Name']\n",
+ " example_gene_id = clinvar_data.iloc[32684]['GENE_ID']\n",
+ " \n",
+ " print(f\"🔍 Example gene mapping for row 32684:\")\n",
+ " print(f\" Gene ID: {example_gene_id}\")\n",
+ " print(f\" Gene Name: {example_gene_name}\")\n",
+ "else:\n",
+ " # Show any available example\n",
+ " if 'clinvar_data' in locals() and clinvar_data is not None and not clinvar_data.empty:\n",
+ " # Find first row with gene name data\n",
+ " example_row = clinvar_data[clinvar_data['GENE_Name'] != ''].iloc[0] if (clinvar_data['GENE_Name'] != '').any() else clinvar_data.iloc[0]\n",
+ " \n",
+ " print(f\"🔍 Example gene mapping:\")\n",
+ " print(f\" Gene ID: {example_row.get('GENE_ID', 'N/A')}\")\n",
+ " print(f\" Gene Name: {example_row.get('GENE_Name', 'N/A')}\")\n",
+ " else:\n",
+ " print(\"❌ No data available for example\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "a214c29d-a4f1-4af6-a914-e6b4a14a1c49",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import os\n",
+ "\n",
+ "# Save the final processed dataset\n",
+ "if 'clinvar_data' in locals() and clinvar_data is not None:\n",
+ " output_file = \"clinvar_with_disease.csv\"\n",
+ " \n",
+ " try:\n",
+ " clinvar_data.to_csv(output_file, index=False)\n",
+ " \n",
+ " print(f\"✅ Final dataset saved to {output_file}\")\n",
+ " print(f\"📊 Saved {len(clinvar_data)} records with {len(clinvar_data.columns)} columns\")\n",
+ " \n",
+ " # File size\n",
+ " file_size = os.path.getsize(output_file) / 1024 / 1024\n",
+ " print(f\"💾 File size: {file_size:.1f} MB\")\n",
+ " \n",
+ " # Summary of what was accomplished\n",
+ " print(\"\\n🎯 Processing Summary:\")\n",
+ " print(f\" ✓ Extracted ClinVar coding variants\")\n",
+ " print(f\" ✓ Parsed XML records for gene information\")\n",
+ " print(f\" ✓ Mapped diseases/phenotypes\")\n",
+ " print(f\" ✓ Added human-readable gene names\")\n",
+ " print(f\" ✓ Created comprehensive dataset\")\n",
+ " \n",
+ " except Exception as e:\n",
+ " print(f\"❌ Error saving dataset: {e}\")\n",
+ "else:\n",
+ " print(\"⚠️ No data available to save\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b6c4c1f4-4b87-4624-8f8a-c568e40b2e63",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import os\n",
+ "import shutil\n",
+ "\n",
+ "# Optional: Clean up temporary XML data directory\n",
+ "# Uncomment the following lines if you want to remove the XML files to save space\n",
+ "\n",
+ "if os.path.exists(\"data\") and os.path.isdir(\"data\"):\n",
+ " # Count files before cleanup\n",
+ " xml_files = [f for f in os.listdir(\"data\") if f.endswith('.xml')]\n",
+ " \n",
+ " print(f\"🗂️ Found {len(xml_files)} XML files in data directory\")\n",
+ " \n",
+ " # Uncomment to actually remove the directory\n",
+ " # shutil.rmtree(\"data\")\n",
+ " # print(\"🗑️ Removed temporary XML data directory\")\n",
+ " \n",
+ " print(\"ℹ️ XML files preserved. Uncomment the cleanup code to remove them.\")\n",
+ "else:\n",
+ " print(\"ℹ️ No XML data directory found to clean up\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c08beea6-6ff7-4900-a8b8-8a719db36189",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "## Processing Complete ✅\n",
+ "\n",
+ "The ClinVar coding variants have been successfully processed with the following enhancements:\n",
+ "\n",
+ "### Generated Files:\n",
+ "- `clinvar_coding_raw.csv` - Raw ClinVar entries extracted from VEP data\n",
+ "- `Clinvar_ID.txt` - List of ClinVar IDs for processing\n",
+ "- `gene_id.txt` - Unique gene IDs for name mapping\n",
+ "- `gene_id_to_name.json` - Gene ID to name mapping dictionary\n",
+ "- `clinvar_with_disease.csv` - **Final comprehensive dataset**\n",
+ "\n",
+ "### Dataset Features:\n",
+ "- **Variant Information**: Genomic coordinates, alleles, and annotations\n",
+ "- **Gene Data**: Symbols, IDs, and human-readable names\n",
+ "- **Disease/Phenotype**: Associated conditions and clinical significance\n",
+ "- **Expanded Format**: One row per variant-disease combination\n",
+ "\n",
+ "### Next Steps:\n",
+ "1. **Quality Control**: Review the data for completeness and accuracy\n",
+ "2. **Analysis**: Use the dataset for downstream genetic analysis\n",
+ "3. **Integration**: Combine with other datasets as needed\n",
+ "4. **Documentation**: Update metadata and create data dictionary\n",
+ "\n",
+ "### File Cleanup:\n",
+ "- XML files in `data/` directory can be removed to save space\n",
+ "- Intermediate files can be archived or removed as needed"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.12.9"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/BioReason-main/data/Clinvar_SNV_Non_SNV.ipynb b/BioReason-main/data/Clinvar_SNV_Non_SNV.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..62657133d3efd260415bab1cfbf958e9551843e3
--- /dev/null
+++ b/BioReason-main/data/Clinvar_SNV_Non_SNV.ipynb
@@ -0,0 +1,3425 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# ClinVar SNV and Non-SNV Processing Pipeline\n",
+ "\n",
+ "This notebook processes ClinVar genetic variants to create machine learning datasets for variant effect prediction. See `Clinvar_SNV_Non_SNV_README.md` for detailed documentation.\n",
+ "\n",
+ "## Quick Start\n",
+ "\n",
+ "1. Update file paths in the configuration section\n",
+ "2. Ensure all dependencies are installed\n",
+ "3. Run cells in order\n",
+ "4. Monitor progress and memory usage\n",
+ "\n",
+ "**⚠️ Important**: This pipeline requires significant computational resources and storage space."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Configuration\n",
+ "\n",
+ "Update these paths for your environment:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Configuration - Update these paths for your environment\n",
+ "import os\n",
+ "from pathlib import Path\n",
+ "\n",
+ "# File paths (update these for your system)\n",
+ "CONFIG = {\n",
+ " # Input data\n",
+ " 'clinvar_vcf': 'data/clinvar_grch38.vcf.gz',\n",
+ " 'reference_genome': 'data/reference/GRCh38.fa',\n",
+ " 'hgnc_mapping': 'data/hgnc_complete_set.txt',\n",
+ " \n",
+ " # VEP configuration\n",
+ " 'vep_root': '/path/to/vep',\n",
+ " 'vep_cache': '/path/to/vep/cache',\n",
+ " \n",
+ " # Output paths\n",
+ " 'output_dir': 'output',\n",
+ " 'temp_dir': 'temp',\n",
+ " \n",
+ " # Processing parameters\n",
+ " 'window_size': 4096,\n",
+ " 'max_variant_size': 64,\n",
+ " 'num_threads': 8,\n",
+ " 'batch_size': 100000\n",
+ "}\n",
+ "\n",
+ "SCRATCH_DIR = '/your/scratch/directory' # Update this to your scratch directory\n",
+ "\n",
+ "# Create output directories\n",
+ "for dir_path in [CONFIG['output_dir'], CONFIG['temp_dir']]:\n",
+ " os.makedirs(dir_path, exist_ok=True)\n",
+ " \n",
+ "print(\"Configuration loaded. Please verify all paths are correct:\")\n",
+ "for key, value in CONFIG.items():\n",
+ " if 'path' in key or 'dir' in key:\n",
+ " exists = os.path.exists(value) if not key.endswith('dir') else True\n",
+ " status = \"✅\" if exists else \"❌\"\n",
+ " print(f\" {status} {key}: {value}\")\n",
+ " \n",
+ "print(\"\\n📝 Update CONFIG dictionary above with your actual file paths\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# ClinVar SNV and Non-SNV Variant Processing Pipeline\n",
+ "\n",
+ "This notebook processes ClinVar genetic variants (both SNVs and non-SNVs) to create a comprehensive machine learning dataset for variant effect prediction. The pipeline includes:\n",
+ "\n",
+ "## Overview\n",
+ "\n",
+ "1. **Data Processing**: Download and process ClinVar VCF data using VEP (Variant Effect Predictor)\n",
+ "2. **Sequence Window Extraction**: Generate 4096bp genomic windows centered on variants\n",
+ "3. **Feature Engineering**: Extract pathogenicity, disease associations, and gene information\n",
+ "4. **Dataset Creation**: Build training/test datasets with disjoint disease splits\n",
+ "5. **Quality Control**: Comprehensive statistics and validation\n",
+ "\n",
+ "## Key Features\n",
+ "\n",
+ "- **Genomic Windows**: 4096bp sequences with centered mutations\n",
+ "- **Variant Types**: Both SNVs and structural variants (insertions, deletions, etc.)\n",
+ "- **Clinical Annotations**: Pathogenicity classification and disease associations\n",
+ "- **Gene Mapping**: Integration with HGNC gene nomenclature\n",
+ "- **Disjoint Splits**: Train/test splits ensuring no disease overlap\n",
+ "\n",
+ "## Requirements\n",
+ "\n",
+ "- **Computational Resources**: High-memory system (recommended for large datasets)\n",
+ "- **Software Dependencies**: VEP, Python libraries (pandas, pysam, pyarrow, hgvs)\n",
+ "- **Reference Data**: GRCh38 genome assembly, HGNC gene mapping\n",
+ "- **Storage**: Sufficient space for intermediate files (~100GB+)\n",
+ "\n",
+ "## Output\n",
+ "\n",
+ "Final datasets suitable for:\n",
+ "- Variant effect prediction models\n",
+ "- Pathogenicity classification\n",
+ "- Disease association studies\n",
+ "- Genomic language model training"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Initial Setup (For HPC/Cluster Environments)\n",
+ "\n",
+ "**Note**: This section contains setup instructions for high-performance computing environments. Adapt paths and module loading commands for your specific system.\n",
+ "\n",
+ "### Prerequisites Installation\n",
+ "If running on a cluster, you may need to download Python wheels and reference data:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Download required Python packages and reference data\n",
+ "# Adjust paths and module loading for your specific environment\n",
+ "\n",
+ "# Example for cluster environments:\n",
+ "# module load python gcc arrow postgresql\n",
+ "\n",
+ "# Create directory for Python wheels (adjust path as needed)\n",
+ "# mkdir -p /path/to/your/pywheels\n",
+ "# pip download hgvs -d /path/to/your/pywheels\n",
+ "\n",
+ "# Download HGNC gene mapping data\n",
+ "# wget -O hgnc_complete_set.txt \"https://storage.googleapis.com/public-download-files/hgnc/tsv/tsv/hgnc_complete_set.txt\"\n",
+ "\n",
+ "print(\"Setup instructions provided above. Adjust paths for your environment.\")\n",
+ "print(\"Required data:\")\n",
+ "print(\"- HGNC complete gene set\")\n",
+ "print(\"- Python packages: hgvs, pandas, pyarrow, pysam, tqdm\")\n",
+ "print(\"- VEP installation with cache\")\n",
+ "print(\"- GRCh38 reference genome\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Environment Setup\n",
+ "\n",
+ "**For cluster/HPC environments**: Configure virtual environment and load required modules.\n",
+ "**For local environments**: Ensure all dependencies are installed."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Environment setup for cluster/HPC systems\n",
+ "# Adjust module loading and paths for your specific environment\n",
+ "\n",
+ "# Example cluster setup:\n",
+ "\"\"\"\n",
+ "# Create virtual environment\n",
+ "python -m venv /tmp/clinvar_env\n",
+ "\n",
+ "# Load required modules (adjust for your system)\n",
+ "module load python gcc arrow postgresql\n",
+ "module load perl samtools tabix bcftools mariadb\n",
+ "\n",
+ "# Activate virtual environment\n",
+ "source /tmp/clinvar_env/bin/activate\n",
+ "\n",
+ "# Install packages\n",
+ "pip install notebook pandas pyarrow pysam hgvs tqdm networkx\n",
+ "\n",
+ "# Start Jupyter (for remote access)\n",
+ "jupyter notebook --no-browser --ip=$(hostname -f) --port=8888\n",
+ "\"\"\"\n",
+ "\n",
+ "# For local environments, ensure these packages are installed:\n",
+ "required_packages = [\n",
+ " 'pandas>=1.3.0',\n",
+ " 'pyarrow>=5.0.0', \n",
+ " 'pysam>=0.19.0',\n",
+ " 'hgvs>=1.5.0',\n",
+ " 'tqdm>=4.60.0',\n",
+ " 'networkx>=2.6.0'\n",
+ "]\n",
+ "\n",
+ "print(\"Required packages:\")\n",
+ "for pkg in required_packages:\n",
+ " print(f\" - {pkg}\")\n",
+ " \n",
+ "print(\"\\nFor VEP processing, also required:\")\n",
+ "print(\" - VEP (Ensembl Variant Effect Predictor)\")\n",
+ "print(\" - BCFtools, SAMtools, Tabix\")\n",
+ "print(\" - Reference genome and VEP cache files\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now code"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "/localscratch/naimerja.43836119.0/clinvar_env/bin/python\n"
+ ]
+ }
+ ],
+ "source": [
+ "!which python\n",
+ "# Verify Python environment and core dependencies\n",
+ "import sys\n",
+ "import subprocess\n",
+ "\n",
+ "print(f\"Python executable: {sys.executable}\")\n",
+ "print(f\"Python version: {sys.version}\")\n",
+ "\n",
+ "# Check for required packages\n",
+ "try:\n",
+ " import pandas as pd\n",
+ " import pyarrow as pa\n",
+ " import pysam\n",
+ " import hgvs\n",
+ " import tqdm\n",
+ " import networkx as nx\n",
+ " \n",
+ " print(\"\\n✅ Core dependencies available:\")\n",
+ " print(f\" - pandas: {pd.__version__}\")\n",
+ " print(f\" - pyarrow: {pa.__version__}\")\n",
+ " print(f\" - pysam: {pysam.__version__}\")\n",
+ " print(f\" - hgvs: {hgvs.__version__}\")\n",
+ " print(f\" - networkx: {nx.__version__}\")\n",
+ " \n",
+ "except ImportError as e:\n",
+ " print(f\"❌ Missing dependency: {e}\")\n",
+ " print(\"Please install required packages first\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Install required packages\n",
+ "# Adjust installation method based on your environment\n",
+ "\n",
+ "# For environments with pre-downloaded wheels:\n",
+ "# !pip install --no-index --find-links /path/to/pywheels hgvs\n",
+ "# !pip install --no-index tqdm pandas pyarrow\n",
+ "\n",
+ "# For standard environments:\n",
+ "# !pip install hgvs tqdm pandas pyarrow pysam networkx\n",
+ "\n",
+ "print(\"Package installation commands provided above.\")\n",
+ "print(\"Choose the appropriate method for your environment:\")\n",
+ "print(\" - Standard: pip install | \n", + " | CHROM | \n", + "POS | \n", + "REF | \n", + "ALT | \n", + "LABEL | \n", + "SOURCE | \n", + "CONSEQUENCE | \n", + "ID | \n", + "REVIEW_STATUS | \n", + "GENE | \n", + "split | \n", + "INT_LABEL | \n", + "GENE_ID | \n", + "Disease | \n", + "GENE_Name | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "chr1 | \n", + "976215 | \n", + "A | \n", + "G | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "1320032 | \n", + "no_assertion_criteria_provided | \n", + "PERM1 | \n", + "train | \n", + "1 | \n", + "84808 | \n", + "Renal tubular epithelial cell apoptosis | \n", + "PPARGC1 and ESRR induced regulator, muscle 1 | \n", + "
| 1 | \n", + "chr1 | \n", + "976215 | \n", + "A | \n", + "G | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "1320032 | \n", + "no_assertion_criteria_provided | \n", + "PERM1 | \n", + "train | \n", + "1 | \n", + "84808 | \n", + "Neutrophil inclusion bodies | \n", + "PPARGC1 and ESRR induced regulator, muscle 1 | \n", + "
| 2 | \n", + "chr1 | \n", + "1050449 | \n", + "G | \n", + "A | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "1284257 | \n", + "no_assertion_criteria_provided | \n", + "AGRN | \n", + "train | \n", + "1 | \n", + "375790 | \n", + "Congenital myasthenic syndrome 8 | \n", + "agrin | \n", + "
| 3 | \n", + "chr1 | \n", + "1050575 | \n", + "G | \n", + "C | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "18241 | \n", + "no_assertion_criteria_provided | \n", + "AGRN | \n", + "train | \n", + "1 | \n", + "375790 | \n", + "Congenital myasthenic syndrome 8 | \n", + "agrin | \n", + "
| 4 | \n", + "chr1 | \n", + "1213738 | \n", + "G | \n", + "A | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "96692 | \n", + "no_assertion_criteria_provided | \n", + "TNFRSF4 | \n", + "train | \n", + "1 | \n", + "7293 | \n", + "Combined immunodeficiency due to OX40 deficiency | \n", + "TNF receptor superfamily member 4 | \n", + "
| ... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "... | \n", + "
| 32680 | \n", + "chrY | \n", + "2787412 | \n", + "C | \n", + "T | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "9747 | \n", + "no_assertion_criteria_provided | \n", + "SRY | \n", + "train | \n", + "1 | \n", + "6736 | \n", + "46,XY sex reversal 1 | \n", + "sex determining region Y | \n", + "
| 32681 | \n", + "chrY | \n", + "2787426 | \n", + "C | \n", + "G | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "9739 | \n", + "criteria_provided,_single_submitter | \n", + "SRY | \n", + "train | \n", + "1 | \n", + "6736 | \n", + "not provided | \n", + "sex determining region Y | \n", + "
| 32682 | \n", + "chrY | \n", + "2787515 | \n", + "C | \n", + "A | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "492908 | \n", + "no_assertion_criteria_provided | \n", + "SRY | \n", + "train | \n", + "1 | \n", + "6736 | \n", + "46,XY sex reversal 1 | \n", + "sex determining region Y | \n", + "
| 32683 | \n", + "chrY | \n", + "2787551 | \n", + "C | \n", + "T | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "9754 | \n", + "no_assertion_criteria_provided | \n", + "SRY | \n", + "train | \n", + "1 | \n", + "6736 | \n", + "46,XY sex reversal 1 | \n", + "sex determining region Y | \n", + "
| 32684 | \n", + "chrY | \n", + "7063898 | \n", + "A | \n", + "T | \n", + "Pathogenic | \n", + "ClinVar | \n", + "missense_variant | \n", + "625467 | \n", + "no_assertion_criteria_provided | \n", + "LOC126057105, TBL1Y | \n", + "train | \n", + "1 | \n", + "126057105, 90665 | \n", + "Deafness, Y-linked 2 | \n", + "P300/CBP strongly-dependent group 1 enhancer G... | \n", + "
32685 rows × 15 columns
\n", + "