BioReason/data/VEP.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0510f375",
   "metadata": {},
   "source": [
    "## Configuration\n",
    "\n",
    "Set up parameters and data sources for variant effect prediction tasks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d59a5d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Configuration - Update these parameters for your environment\n",
    "import os\n",
    "from pathlib import Path\n",
    "import random\n",
    "\n",
    "# Set random seed for reproducible question assignment\n",
    "RANDOM_SEED = 42\n",
    "random.seed(RANDOM_SEED)\n",
    "\n",
    "# Configuration parameters\n",
    "CONFIG = {\n",
    "    # Data source configurations\n",
    "    'huggingface_repo': 'wanglab/bioR_tasks',  # Update with your repository\n",
    "    \n",
    "    # Local data paths (update these if using local files)\n",
    "    'local_data_dir': 'data',\n",
    "    \n",
    "    # Output configurations\n",
    "    'output_dir': 'output_datasets',\n",
    "    'save_local': True,  # Save datasets locally\n",
    "    'upload_to_hub': False,  # Set to True to upload to HuggingFace Hub\n",
    "    \n",
    "    # Processing parameters\n",
    "    'question_variants': 50,  # Number of question templates per task\n",
    "    'batch_size': 1000,  # For memory-efficient processing\n",
    "    \n",
    "    # Task configurations\n",
    "    'tasks': {\n",
    "        'task1': {'name': 'variant_effect_coding', 'description': 'Pathogenic vs Benign classification'},\n",
    "        'task2': {'name': 'variant_effect_causal_eqtl', 'description': 'Gene expression change prediction'},\n",
    "        'task3': {'name': 'variant_effect_pathogenic_omim', 'description': 'OMIM pathogenic classification'},\n",
    "        'task4_snv': {'name': 'task4_variant_effect_snv', 'description': 'SNV effect prediction'},\n",
    "        'task4_non_snv': {'name': 'task4_variant_effect_non_snv', 'description': 'Non-SNV effect prediction'}\n",
    "    }\n",
    "}\n",
    "\n",
    "# Create output directory\n",
    "Path(CONFIG['output_dir']).mkdir(exist_ok=True)\n",
    "\n",
    "print(\"Configuration loaded:\")\n",
    "print(f\"  Random seed: {RANDOM_SEED}\")\n",
    "print(f\"  Output directory: {CONFIG['output_dir']}\")\n",
    "print(f\"  Upload to hub: {CONFIG['upload_to_hub']}\")\n",
    "print(f\"  Repository: {CONFIG['huggingface_repo']}\")\n",
    "print(\"\\n📝 Update CONFIG dictionary above with your specific settings\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4a1e6bc-e3e6-4084-a42e-ba988c3afa4a",
   "metadata": {},
   "source": [
    "# Variant Effect Prediction Tasks - Dataset Creation Pipeline\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook creates standardized datasets for variant effect prediction tasks using various genomic databases. It processes raw variant data into machine learning-ready formats with contextualized questions and standardized answers.\n",
    "\n",
    "## What This Notebook Does\n",
    "\n",
    "1. **Task 1**: Variant Effect Prediction (Pathogenic vs Benign) using ClinVar data\n",
    "2. **Task 2**: Causal eQTL Analysis (Gene Expression Changes) \n",
    "3. **Task 3**: Pathogenic Variant Classification using OMIM data\n",
    "4. **Task 4**: SNV and Non-SNV Variant Effect Prediction\n",
    "\n",
    "## Key Features\n",
    "\n",
    "- **Question Diversification**: 50+ unique question templates per task type\n",
    "- **Standardized Format**: Consistent ID, question, answer, sequence structure\n",
    "- **Multiple Data Sources**: ClinVar, OMIM, eQTL databases\n",
    "- **Publication-Ready**: Clean, documented datasets ready for research use\n",
    "\n",
    "## Dataset Structure\n",
    "\n",
    "Each task generates datasets with the following fields:\n",
    "- `ID`: Unique identifier for each variant\n",
    "- `question`: Contextualized biological question\n",
    "- `answer`: Standardized response (pathogenic/benign, disease name, etc.)\n",
    "- `reference_sequence`: Original genomic sequence\n",
    "- `variant_sequence`: Mutated genomic sequence\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "```bash\n",
    "pip install datasets pandas numpy\n",
    "```\n",
    "\n",
    "## Usage\n",
    "\n",
    "1. **Configure Data Sources**: Update file paths and dataset configurations\n",
    "2. **Run Tasks Sequentially**: Execute each task section in order\n",
    "3. **Review Outputs**: Validate generated datasets before publication\n",
    "4. **Export**: Datasets are saved locally and optionally uploaded to repositories\n",
    "\n",
    "## Important Notes\n",
    "\n",
    "- **Data Privacy**: All personal references have been removed\n",
    "- **Reproducibility**: Random seeds should be set for consistent question assignment\n",
    "- **Memory Usage**: Large datasets may require substantial RAM\n",
    "- **File Paths**: Update all hardcoded paths to use relative or configurable paths\n",
    "\n",
    "## Output\n",
    "\n",
    "Generated datasets are suitable for:\n",
    "- Variant effect prediction model training\n",
    "- Biological reasoning benchmarks\n",
    "- Genomic language model evaluation\n",
    "- Clinical variant interpretation research"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67ff57a4-00e0-41a4-aac1-18f4e1c68be8",
   "metadata": {},
   "source": [
    "## Task 1: variant effect prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "73559f7e-ade8-4d84-84d4-859fbbd0c575",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "import pandas as pd\n",
    "import json\n",
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d2ffcc6-5e33-4cc4-a46d-2c8f67bd20a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = load_dataset(\"wanglab/bioR_tasks\", 'variant_effect_pathogenic_clinvar')\n",
    "\n",
    "## Task 1: Variant Effect Prediction (Pathogenic vs Benign)\n",
    "\n",
    "**Objective**: Classify genetic variants as pathogenic or benign based on chromosomal location and gene context.\n",
    "\n",
    "**Data Source**: ClinVar database with pathogenic variant annotations\n",
    "\n",
    "**Question Types**: 50 different question templates incorporating:\n",
    "- Chromosome location\n",
    "- Gene information (when available)\n",
    "- Clinical significance assessment\n",
    "\n",
    "**Output Format**: Binary classification with disease association when applicable"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e31d1d0b-5f13-468c-a2cf-704499214a04",
   "metadata": {},
   "source": [
    "**With GPT4o, I created 50 different versions of this question and prompt**\n",
    "\n",
    "# Core imports for dataset processing\n",
    "from datasets import load_dataset, Dataset, DatasetDict\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import json\n",
    "import random\n",
    "from pathlib import Path\n",
    "\n",
    "# Set random seed for reproducibility\n",
    "random.seed(CONFIG.get('random_seed', 42))\n",
    "\n",
    "print(\"✅ Core libraries imported\")\n",
    "print(f\"Random seed set to: {CONFIG.get('random_seed', 42)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3301da8-bed2-4c0f-835c-6613df88dd11",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Must call format in order of chromosome, gene, gene_name\n",
    "question_variants_50 = [\"This variant lies on Chromosome {0} and affects the gene {1} ({2}). Based on this context, is the mutation pathogenic or benign? If pathogenic, what disease does it cause?\",\n",
    "\"Located on Chromosome {0}, this mutation impacts {1} ({2}). What is its biological consequence — is it benign or pathogenic, and which disease is associated if any?\",\n",
    "\"A genetic alteration is present in {1} ({2}) on Chromosome {0}. Is this variant benign or disease-causing, and if the latter, which condition is involved?\",\n",
    "\"This variant affects the gene {1} ({2}) found on Chromosome {0}. What is the clinical effect of this variant — benign or pathogenic? State the disease if applicable.\",\n",
    "\"With a mutation on Chromosome {0} in gene {1} ({2}), classify this variant as benign or pathogenic. Include the disease if it's pathogenic.\",\n",
    "\"This sequence change occurs on Chromosome {0}, altering {1} ({2}). What is the medical significance of this variant — is it benign or linked to a disease?\",\n",
    "\"Here is a variant affecting {1} ({2}) on Chromosome {0}. Please identify whether it is a benign mutation or associated with a disorder.\",\n",
    "\"A variant on Chromosome {0} in gene {1} ({2}) has been observed. Is this a neutral mutation, or does it result in a disease? If so, which one?\",\n",
    "\"The gene {1} ({2}) on Chromosome {0} contains a mutation. Based on this information, is the variant pathogenic or benign? Provide the disease if relevant.\",\n",
    "\"This genomic variant is located on Chromosome {0}, within the {1} ({2}) gene. Can you determine its pathogenicity and name any linked disease?\",\n",
    "\"A mutation found in {1} ({2}) on Chromosome {0} may be clinically relevant. Is it pathogenic or benign, and if the former, which disease is implicated?\",\n",
    "\"Given a variant located on Chromosome {0} and affecting {1} ({2}), assess whether it is benign or pathogenic. Indicate the associated disease if pathogenic.\",\n",
    "\"This mutation is located in gene {1} ({2}) on Chromosome {0}. Is it associated with a disease or is it a benign polymorphism?\",\n",
    "\"A variant has been detected on Chromosome {0} in {1} ({2}). What is its effect — pathogenic or benign? If pathogenic, name the disease.\",\n",
    "\"The variant affects gene {1} ({2}), which is on Chromosome {0}. Please evaluate whether this mutation is benign or pathogenic and specify the disease if necessary.\",\n",
    "\"This alteration in {1} ({2}) on Chromosome {0} may affect gene function. Does it lead to a disease or is it benign?\",\n",
    "\"Given this variant in gene {1} ({2}) on Chromosome {0}, classify it as benign or pathogenic. Include the disorder it may cause if applicable.\",\n",
    "\"A variant was discovered on Chromosome {0}, affecting {1} ({2}). What is its functional impact — neutral or pathogenic? State the disease if pathogenic.\",\n",
    "\"This gene mutation involves {1} ({2}) on Chromosome {0}. Is it associated with any clinical condition, or is it benign?\",\n",
    "\"The gene {1} ({2}) on Chromosome {0} carries this variant. Does this mutation lead to a specific disease, or is it non-pathogenic?\",\n",
    "\"Here is a mutation in {1} ({2}) on Chromosome {0}. Determine whether it’s benign or pathogenic. If the latter, what disease does it cause?\",\n",
    "\"A variant found in Chromosome {0} affects {1} ({2}). Please analyze its biological impact: is it benign or pathogenic, and what condition might it cause?\",\n",
    "\"The following genetic variant occurs in {1} ({2}) on Chromosome {0}. Classify its clinical effect — pathogenic or benign — and list any associated condition.\",\n",
    "\"This alteration occurs within gene {1} ({2}) located on Chromosome {0}. Is it associated with a disease or is it a benign variant?\",\n",
    "\"A mutation on Chromosome {0} affecting {1} ({2}) has been found. Is it harmful or harmless? What disease, if any, does it cause?\",\n",
    "\"Gene {1} ({2}) on Chromosome {0} is impacted by this variant. Evaluate whether it is clinically benign or pathogenic and name the disorder if relevant.\",\n",
    "\"Consider this mutation in {1} ({2}) on Chromosome {0}. Is this a benign change or a disease-causing variant?\",\n",
    "\"A variant was discovered in gene {1} ({2}), Chromosome {0}. Please indicate if this mutation results in a known disease or if it's non-harmful.\",\n",
    "\"Given this context: Chromosome {0}, gene {1} ({2}) — does this variant present pathogenic behavior, and if so, what disease does it relate to?\",\n",
    "\"This sequence variant lies in {1} ({2}) on Chromosome {0}. Is it clinically significant, and what condition might it cause if any?\",\n",
    "\"A mutation in {1} ({2}), located on Chromosome {0}, is being studied. Determine whether it’s pathogenic or benign, and specify the linked disease.\",\n",
    "\"Here is a genetic alteration in {1} ({2}) on Chromosome {0}. Based on the data, is it a benign variant or a cause of disease?\",\n",
    "\"Mutation context: Chromosome {0}, Gene {1} ({2}). Determine if this variant is likely to be benign or pathogenic. Mention the disease if applicable.\",\n",
    "\"A sequence alteration has been identified in {1} ({2}) on Chromosome {0}. Is it disease-inducing or harmless?\",\n",
    "\"Chromosome {0} houses a mutation in gene {1} ({2}). Classify its clinical impact — is it pathogenic or benign, and what disease does it lead to if any?\",\n",
    "\"This variant affects gene {1} ({2}) located on Chromosome {0}. Evaluate its biological effect and specify any disease association.\",\n",
    "\"Gene {1} ({2}) on Chromosome {0} is altered by this variant. Does this mutation result in a disease or is it benign?\",\n",
    "\"Assess the clinical impact of this variant on gene {1} ({2}), found on Chromosome {0}. State whether it’s pathogenic or benign, and the disease if applicable.\",\n",
    "\"This is a variant in {1} ({2}), located on Chromosome {0}. Is this mutation a likely cause of disease or not?\",\n",
    "\"A change on Chromosome {0} affects gene {1} ({2}). Identify whether the variant is neutral or disease-linked. Mention the disease if applicable.\",\n",
    "\"This variant impacts the gene {1} ({2}) on Chromosome {0}. Is the change likely to result in a pathogenic outcome?\",\n",
    "\"The gene {1} ({2}) is located on Chromosome {0}, where a mutation has occurred. What is the medical relevance of this mutation?\",\n",
    "\"A variant affecting Chromosome {0}, within the gene {1} ({2}), has been observed. Determine if it's benign or associated with disease.\",\n",
    "\"This mutation occurs in {1} ({2}) on Chromosome {0}. Does this change lead to a known medical condition, or is it benign?\",\n",
    "\"Gene {1} ({2}), found on Chromosome {0}, is impacted by this variant. What is the biological outcome — benign or pathogenic?\",\n",
    "\"Consider a variant on Chromosome {0} in gene {1} ({2}). Determine its clinical classification and disease relevance.\",\n",
    "\"An alteration has been detected in {1} ({2}) on Chromosome {0}. Is it pathogenic, and if so, what disease is involved?\",\n",
    "\"The gene {1} ({2}), on Chromosome {0}, contains a mutation. Does this mutation cause a disorder, or is it a benign change?\",\n",
    "\"Here’s a variant in {1} ({2}) located on Chromosome {0}. What is the predicted biological effect — harmless or disease-causing?\",\n",
    "\"A genomic change on Chromosome {0} affects {1} ({2}). Classify this variant as benign or pathogenic, and name the disease if relevant.\"]\n",
    "# Load Task 1 dataset from configured source\n",
    "try:\n",
    "    dataset = load_dataset(CONFIG['huggingface_repo'], 'variant_effect_pathogenic_clinvar')\n",
    "    print(f\"✅ Loaded dataset from: {CONFIG['huggingface_repo']}\")\n",
    "    print(f\"Train samples: {len(dataset['train'])}\")\n",
    "    print(f\"Test samples: {len(dataset['test'])}\")\n",
    "except Exception as e:\n",
    "    print(f\"❌ Error loading dataset: {e}\")\n",
    "    print(\"Please check the repository name and dataset configuration\")\n",
    "    raise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a418765-c127-44a8-8003-822b413cc907",
   "metadata": {},
   "outputs": [],
   "source": [
    "question_variants_50_no_gene = [\n",
    "    \"This variant lies on Chromosome {0}. Based on this context, is the mutation pathogenic or benign? If pathogenic, what disease does it cause?\",\n",
    "    \"Located on Chromosome {0}, this mutation has been observed. What is its biological consequence — is it benign or pathogenic, and which disease is associated if any?\",\n",
    "    \"A genetic alteration is present on Chromosome {0}. Is this variant benign or disease-causing, and if the latter, which condition is involved?\",\n",
    "    \"This variant is found on Chromosome {0}. What is the clinical effect of this variant — benign or pathogenic? State the disease if applicable.\",\n",
    "    \"With a mutation on Chromosome {0}, classify this variant as benign or pathogenic. Include the disease if it's pathogenic.\",\n",
    "    \"This sequence change occurs on Chromosome {0}. What is the medical significance of this variant — is it benign or linked to a disease?\",\n",
    "    \"Here is a variant on Chromosome {0}. Please identify whether it is a benign mutation or associated with a disorder.\",\n",
    "    \"A variant on Chromosome {0} has been observed. Is this a neutral mutation, or does it result in a disease? If so, which one?\",\n",
    "    \"A mutation is present on Chromosome {0}. Based on this information, is the variant pathogenic or benign? Provide the disease if relevant.\",\n",
    "    \"This genomic variant is located on Chromosome {0}. Can you determine its pathogenicity and name any linked disease?\",\n",
    "    \"A mutation found on Chromosome {0} may be clinically relevant. Is it pathogenic or benign, and if the former, which disease is implicated?\",\n",
    "    \"Given a variant located on Chromosome {0}, assess whether it is benign or pathogenic. Indicate the associated disease if pathogenic.\",\n",
    "    \"This mutation is located on Chromosome {0}. Is it associated with a disease or is it a benign polymorphism?\",\n",
    "    \"A variant has been detected on Chromosome {0}. What is its effect — pathogenic or benign? If pathogenic, name the disease.\",\n",
    "    \"A mutation on Chromosome {0} is under review. Please evaluate whether this mutation is benign or pathogenic and specify the disease if necessary.\",\n",
    "    \"This alteration on Chromosome {0} may affect genome function. Does it lead to a disease or is it benign?\",\n",
    "    \"Given this variant on Chromosome {0}, classify it as benign or pathogenic. Include the disorder it may cause if applicable.\",\n",
    "    \"A variant was discovered on Chromosome {0}. What is its functional impact — neutral or pathogenic? State the disease if pathogenic.\",\n",
    "    \"This mutation on Chromosome {0} may be significant. Is it associated with any clinical condition, or is it benign?\",\n",
    "    \"Chromosome {0} carries this variant. Does this mutation lead to a specific disease, or is it non-pathogenic?\",\n",
    "    \"Here is a mutation located on Chromosome {0}. Determine whether it’s benign or pathogenic. If the latter, what disease does it cause?\",\n",
    "    \"A variant found on Chromosome {0} is being studied. Please analyze its biological impact: is it benign or pathogenic, and what condition might it cause?\",\n",
    "    \"The following genetic variant occurs on Chromosome {0}. Classify its clinical effect — pathogenic or benign — and list any associated condition.\",\n",
    "    \"This alteration occurs on Chromosome {0}. Is it associated with a disease or is it a benign variant?\",\n",
    "    \"A mutation on Chromosome {0} has been found. Is it harmful or harmless? What disease, if any, does it cause?\",\n",
    "    \"A variant on Chromosome {0} is under investigation. Evaluate whether it is clinically benign or pathogenic and name the disorder if relevant.\",\n",
    "    \"Consider this mutation on Chromosome {0}. Is this a benign change or a disease-causing variant?\",\n",
    "    \"A variant was discovered on Chromosome {0}. Please indicate if this mutation results in a known disease or if it's non-harmful.\",\n",
    "    \"Given this context: Chromosome {0} — does this variant present pathogenic behavior, and if so, what disease does it relate to?\",\n",
    "    \"This sequence variant lies on Chromosome {0}. Is it clinically significant, and what condition might it cause if any?\",\n",
    "    \"A mutation located on Chromosome {0} is being studied. Determine whether it’s pathogenic or benign, and specify the linked disease.\",\n",
    "    \"Here is a genetic alteration on Chromosome {0}. Based on the data, is it a benign variant or a cause of disease?\",\n",
    "    \"Mutation context: Chromosome {0}. Determine if this variant is likely to be benign or pathogenic. Mention the disease if applicable.\",\n",
    "    \"A sequence alteration has been identified on Chromosome {0}. Is it disease-inducing or harmless?\",\n",
    "    \"Chromosome {0} houses a mutation. Classify its clinical impact — is it pathogenic or benign, and what disease does it lead to if any?\",\n",
    "    \"This variant is located on Chromosome {0}. Evaluate its biological effect and specify any disease association.\",\n",
    "    \"Chromosome {0} is altered by this variant. Does this mutation result in a disease or is it benign?\",\n",
    "    \"Assess the clinical impact of this variant found on Chromosome {0}. State whether it’s pathogenic or benign, and the disease if applicable.\",\n",
    "    \"This is a variant located on Chromosome {0}. Is this mutation a likely cause of disease or not?\",\n",
    "    \"A change on Chromosome {0} is being evaluated. Identify whether the variant is neutral or disease-linked. Mention the disease if applicable.\",\n",
    "    \"This variant is present on Chromosome {0}. Is the change likely to result in a pathogenic outcome?\",\n",
    "    \"A mutation has occurred on Chromosome {0}. What is the medical relevance of this mutation?\",\n",
    "    \"A variant affecting Chromosome {0} has been observed. Determine if it's benign or associated with disease.\",\n",
    "    \"This mutation occurs on Chromosome {0}. Does this change lead to a known medical condition, or is it benign?\",\n",
    "    \"A genomic variant on Chromosome {0} is under review. What is the biological outcome — benign or pathogenic?\",\n",
    "    \"Consider a variant on Chromosome {0}. Determine its clinical classification and disease relevance.\",\n",
    "    \"An alteration has been detected on Chromosome {0}. Is it pathogenic, and if so, what disease is involved?\",\n",
    "    \"A mutation on Chromosome {0} is under examination. Does this mutation cause a disorder, or is it a benign change?\",\n",
    "    \"Here’s a variant located on Chromosome {0}. What is the predicted biological effect — harmless or disease-causing?\",\n",
    "    \"A genomic change on Chromosome {0} is noted. Classify this variant as benign or pathogenic, and name the disease if relevant.\",\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "cb12df8d-4303-4cf5-bb4a-73d1351ab059",
   "metadata": {},
   "outputs": [],
   "source": [
    "task_1 = dataset['train'].to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8eddbccb-16f7-4a0c-b4f0-49f38a9468e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "task_1['label'] = task_1['label'].apply(lambda x: \"Benign\" if x == \"Common\" else x)\n",
    "task_1['ID'] = ['Task1_train_' + str(i) for i in range(len(task_1))]\n",
    "task_1 = task_1[['ID', 'label', 'chromosome', 'ref_forward_sequence', 'alt_forward_sequence',\n",
    "       'gene', 'gene_name', 'disease']]\n",
    "\n",
    "task_1 = task_1.set_index('ID')\n",
    "\n",
    "task_1_train = []\n",
    "\n",
    "for count, id in enumerate(task_1.index):\n",
    "    task_1_train.append({})\n",
    "    task_1_train[count]['ID'] = id\n",
    "    if not (task_1.loc[id]['gene'] or task_1.loc[id]['gene_name']):\n",
    "        task_1_train[count]['question'] = question_variants_50_no_gene[random.randrange(50)].format(task_1.loc[id]['chromosome'])\n",
    "    else:\n",
    "        task_1_train[count]['question'] = question_variants_50[random.randrange(50)].format(task_1.loc[id]['chromosome'], task_1.loc[id]['gene'], task_1.loc[id]['gene_name'])\n",
    "        \n",
    "    if not task_1.loc[id]['disease']:\n",
    "        task_1_train[count]['answer'] = f\"{task_1.loc[id]['label']}\"\n",
    "    else:\n",
    "        task_1_train[count]['answer'] = f\"{task_1.loc[id]['label']}; {task_1.loc[id]['disease']}\"\n",
    "    task_1_train[count]['reference_sequence'] = task_1.loc[id]['ref_forward_sequence']\n",
    "    task_1_train[count]['variant_sequence'] = task_1.loc[id]['alt_forward_sequence']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "1bc9a04d-ec4d-47e7-9ffd-c4e319b62172",
   "metadata": {},
   "outputs": [],
   "source": [
    "task_1 = dataset['test'].to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "2fc93817-f219-449e-8c57-8e597a2ca494",
   "metadata": {},
   "outputs": [],
   "source": [
    "task_1['label'] = task_1['label'].apply(lambda x: \"Benign\" if x == \"Common\" else x)\n",
    "task_1['ID'] = ['Task1_test_' + str(i) for i in range(len(task_1))]\n",
    "task_1 = task_1[['ID', 'label', 'chromosome', 'ref_forward_sequence', 'alt_forward_sequence',\n",
    "       'gene', 'gene_name', 'disease']]\n",
    "\n",
    "task_1 = task_1.set_index('ID')\n",
    "\n",
    "task_1_test = []\n",
    "\n",
    "for count, id in enumerate(task_1.index):\n",
    "    task_1_test.append({})\n",
    "    task_1_test[count]['ID'] = id\n",
    "    if not task_1.loc[id]['gene'] or task_1.loc[id]['gene_name']:\n",
    "        task_1_test[count]['question'] = question_variants_50_no_gene[random.randrange(50)].format(task_1.loc[id]['chromosome'])\n",
    "    else:\n",
    "        task_1_test[count]['question'] = question_variants_50[random.randrange(50)].format(task_1.loc[id]['chromosome'], task_1.loc[id]['gene'], task_1.loc[id]['gene_name'])\n",
    "        \n",
    "    if not task_1.loc[id]['disease']:\n",
    "        task_1_test[count]['answer'] = f\"{task_1.loc[id]['label']}\"\n",
    "    else:\n",
    "        task_1_test[count]['answer'] = f\"{task_1.loc[id]['label']}; {task_1.loc[id]['disease']}\"\n",
    "    task_1_test[count]['reference_sequence'] = task_1.loc[id]['ref_forward_sequence']\n",
    "    task_1_test[count]['variant_sequence'] = task_1.loc[id]['alt_forward_sequence']\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "5e71a52a-d030-4ad8-9317-a48a81d788a8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "48850"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(task_1_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "5c79704d-8ecc-449c-91e5-c9dd03219028",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1233"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(task_1_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "6bf828d7-0b1c-4b24-afb9-c0127b6f608c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Here is some context for the variant: It is on Chromosome 8, and affects Gene/s CLN8 (CLN8 transmembrane ER and ERGIC protein). Given this context, what is the biological effect of this variant allele, specifically is the mutation pathogenic or benign? If pathogenic, what disease it will cause?\n"
     ]
    }
   ],
   "source": [
    "print(f\"Here is some context for the variant: It is on Chromosome {task_1.iloc[0]['chromosome']}, and affects Gene/s {task_1.iloc[0]['gene']} ({task_1.iloc[0]['gene_name']}). Given this context, what is the biological effect of this variant allele, specifically is the mutation pathogenic or benign? If pathogenic, what disease it will cause?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "746f4274-768a-4c7a-a878-813e2072ba2b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "2f3b8017-9abf-488e-986f-1d588e24eacf",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import Dataset, DatasetDict\n",
    "\n",
    "# Step 1: Create Hugging Face Datasets\n",
    "train_dataset = Dataset.from_list(task_1_train)\n",
    "test_dataset = Dataset.from_list(task_1_test)\n",
    "\n",
    "# Step 2: Combine into a DatasetDict (to mimic load_dataset)\n",
    "dataset = DatasetDict({\n",
    "    \"train\": train_dataset,\n",
    "    \"test\": test_dataset\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73077c9f-65d4-451b-879e-f7071029d9f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.push_to_hub(\n",
    "    \"wanglab/bioR_tasks\",\n",
    "    config_name=\"variant_effect_coding\",\n",
    "    commit_message=\"Upload the finalized Task 1 Variant Effect Coding Data\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54a98d27-39d1-47aa-86a3-2a50d85d6df5",
   "metadata": {},
   "source": [
    "## Task 2 Variant Effect Causal eQTL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68c0a985-09fb-4035-a999-158e743b98d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "import pandas as pd\n",
    "import json\n",
    "import random\n",
    "from pathlib import Path\n",
    "\n",
    "# CONFIG dictionary to simulate the configuration settings\n",
    "CONFIG = {\n",
    "    'save_local': True,\n",
    "    'output_dir': './data',\n",
    "    'upload_to_hub': False,\n",
    "    'huggingface_repo': 'your_huggingface_repo'\n",
    "}\n",
    "\n",
    "# Load your dataset here\n",
    "# dataset = load_dataset('your_dataset_name')\n",
    "\n",
    "# Save and optionally upload Task 1 dataset\n",
    "if CONFIG['save_local']:\n",
    "    # Save locally first\n",
    "    output_path = Path(CONFIG['output_dir']) / 'task1_variant_effect_coding'\n",
    "    output_path.mkdir(exist_ok=True)\n",
    "    \n",
    "    # Save as JSON files\n",
    "    # dataset['train'].to_json(output_path / 'train.jsonl')\n",
    "    # dataset['test'].to_json(output_path / 'test.jsonl')\n",
    "    print(f\"✅ Task 1 dataset saved locally to: {output_path}\")\n",
    "\n",
    "if CONFIG['upload_to_hub']:\n",
    "    try:\n",
    "        # dataset.push_to_hub(\n",
    "        #     CONFIG['huggingface_repo'],\n",
    "        #     config_name=\"variant_effect_coding\",\n",
    "        #     commit_message=\"Upload Task 1 Variant Effect Coding Data\"\n",
    "        # )\n",
    "        print(f\"✅ Task 1 dataset uploaded to: {CONFIG['huggingface_repo']}\")\n",
    "    except Exception as e:\n",
    "        print(f\"❌ Upload failed: {e}\")\n",
    "        print(\"Please check your HuggingFace credentials and repository permissions\")\n",
    "else:\n",
    "    print(\"📝 Upload to hub disabled. Set CONFIG['upload_to_hub'] = True to enable\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6ff52bb-052a-4a1b-9e28-bbc13ea58594",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = load_dataset(\"wanglab/bioR_tasks\", 'variant_effect_causal_eqtl')\n",
    "\n",
    "## Task 2: Variant Effect Causal eQTL\n",
    "\n",
    "**Objective**: Determine whether genetic variants cause changes in gene expression levels.\n",
    "\n",
    "**Data Source**: Expression quantitative trait loci (eQTL) databases\n",
    "\n",
    "**Question Types**: 50 different question templates incorporating:\n",
    "- Chromosome location\n",
    "- Tissue type context\n",
    "- Expression change assessment\n",
    "\n",
    "**Output Format**: Binary classification (expression change: Yes/No)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56f83104-fd46-462a-b6cc-b4a954fcc5bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Proceeding with Task 2: Causal eQTL Analysis\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5cbe79bc-b10c-4b27-b700-a80af923ce35",
   "metadata": {},
   "outputs": [],
   "source": [
    "question_variants_50_expr = [\n",
    "    \"This variant is isolated from Chromosome {0} from {1} tissue. Does this variant change gene expression?\",\n",
    "    \"This variant originates from Chromosome {0} in {1} tissue. Does it alter gene expression?\",\n",
    "    \"Does the variant from Chromosome {0}, isolated in {1} tissue, change gene expression?\",\n",
    "    \"Is there a change in gene expression for the Chromosome {0} variant found in {1} tissue?\",\n",
    "    \"For the variant on Chromosome {0} in {1} tissue, does it affect gene expression levels?\",\n",
    "    \"Does a variant on Chromosome {0} taken from {1} tissue modify gene expression?\",\n",
    "    \"When isolated from Chromosome {0} in {1} tissue, does this variant impact gene expression?\",\n",
    "    \"Can the Chromosome {0} variant from {1} tissue change the expression of genes?\",\n",
    "    \"Is gene expression altered by the variant on Chromosome {0} in {1} tissue?\",\n",
    "    \"Does the mutation on Chromosome {0}, found in {1} tissue, result in different gene expression?\",\n",
    "    \"In {1} tissue, does the Chromosome {0} variant change how genes are expressed?\",\n",
    "    \"For a variant from Chromosome {0} in {1} tissue, is gene expression affected?\",\n",
    "    \"Does the Chromosome {0} alteration from {1} tissue lead to a detectable change in gene expression?\",\n",
    "    \"Will the variant on Chromosome {0} in {1} tissue cause gene expression changes?\",\n",
    "    \"Is there an effect on gene expression from the Chromosome {0} variant in {1} tissue?\",\n",
    "    \"Does the Chromosome {0} variant isolated in {1} tissue influence gene expression?\",\n",
    "    \"In {1} tissue, does the mutation on Chromosome {0} disrupt gene expression?\",\n",
    "    \"Does this Chromosome {0} variant, taken from {1} tissue, shift gene expression patterns?\",\n",
    "    \"Does gene expression differ for the variant on Chromosome {0} found in {1} tissue?\",\n",
    "    \"Is the expression of genes altered by the Chromosome {0} variant in {1} tissue?\",\n",
    "    \"For the variant isolated from Chromosome {0} in {1} tissue, does it change gene expression?\",\n",
    "    \"Does the Chromosome {0}-based variant in {1} tissue have an impact on gene expression?\",\n",
    "    \"Is gene expression modulated by the variant on Chromosome {0} in {1} tissue?\",\n",
    "    \"Does the mutation on Chromosome {0} from {1} tissue result in altered gene expression?\",\n",
    "    \"In {1} tissue samples, does the Chromosome {0} variant change gene expression?\",\n",
    "    \"Does the Chromosome {0} alteration observed in {1} tissue affect gene expression?\",\n",
    "    \"Will gene expression be different when the variant is from Chromosome {0} in {1} tissue?\",\n",
    "    \"Does isolating this variant from Chromosome {0} in {1} tissue alter gene expression?\",\n",
    "    \"Does the variant on Chromosome {0} in {1} tissue cause a measurable change in gene expression?\",\n",
    "    \"For Chromosome {0} variants in {1} tissue, does gene expression change?\",\n",
    "    \"Does gene transcription change for the variant on Chromosome {0} isolated from {1} tissue?\",\n",
    "    \"Is transcriptional output altered by the Chromosome {0} variant in {1} tissue?\",\n",
    "    \"Does the Chromosome {0}-derived variant, in {1} tissue, impact gene expression?\",\n",
    "    \"In {1} tissue, does the Chromosome {0} mutation affect expression of genes?\",\n",
    "    \"Does the Chromosome {0} variant from {1} tissue lead to differential gene expression?\",\n",
    "    \"Does changing that locus on Chromosome {0} in {1} tissue alter gene expression?\",\n",
    "    \"Is there a change in transcript levels for the Chromosome {0} variant in {1} tissue?\",\n",
    "    \"Does the variant mapped to Chromosome {0}, in {1} tissue, influence expression levels?\",\n",
    "    \"For the mutation on Chromosome {0} within {1} tissue, does gene expression shift?\",\n",
    "    \"Does gene expression vary when the variant is on Chromosome {0} in {1} tissue?\",\n",
    "    \"Is the expression profile altered by the Chromosome {0} variant in {1} tissue?\",\n",
    "    \"Does the Somatic variant on Chromosome {0} in {1} tissue behave as a gene expression modulator?\",\n",
    "    \"Does the Chromosome {0} variant identified in {1} tissue change gene expression?\",\n",
    "    \"Is there an observable effect on gene expression from the Chromosome {0} variant in {1} tissue?\",\n",
    "    \"Does the genetic alteration on Chromosome {0} in {1} tissue modify gene expression?\",\n",
    "    \"Does the Chromosome {0} variant present in {1} tissue alter the level of gene transcripts?\",\n",
    "    \"For the Chromosome {0} mutation in {1} tissue, is there a change in gene expression?\",\n",
    "    \"Does this variant in {1} tissue, located on Chromosome {0}, affect gene expression?\",\n",
    "    \"Is gene expression impacted by this Chromosome {0} variant from {1} tissue?\",\n",
    "    \"Does transcription change for the Chromosome {0} variant in {1} tissue?\",\n",
    "]\n",
    "\n",
    "# Load Task 2 dataset from configured source\n",
    "try:\n",
    "    dataset = load_dataset(CONFIG['huggingface_repo'], 'variant_effect_causal_eqtl')\n",
    "    print(f\"✅ Loaded Task 2 dataset from: {CONFIG['huggingface_repo']}\")\n",
    "    print(f\"Train samples: {len(dataset['train'])}\")\n",
    "    print(f\"Test samples: {len(dataset['test'])}\")\n",
    "except Exception as e:\n",
    "    print(f\"❌ Error loading Task 2 dataset: {e}\")\n",
    "    print(\"Please check the repository name and dataset configuration\")\n",
    "    raise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "79f9c1d7-4e90-4075-9e45-3b1b8aa22d4c",
   "metadata": {},
   "outputs": [],
   "source": [
    "task_2 = dataset['train'].to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d73eaaf0-0b81-4d55-862e-13464c8b78e9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['ref_forward_sequence', 'alt_forward_sequence', 'tissue', 'chromosome',\n",
       "       'label'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "task_2.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "6092a8d7-a509-45f5-8a60-eae17ab91235",
   "metadata": {},
   "outputs": [],
   "source": [
    "task_2['ID'] = ['Task2_train_' + str(i) for i in range(len(task_2))]\n",
    "task_2 = task_2[['ID', 'ref_forward_sequence', 'alt_forward_sequence', 'tissue', 'chromosome', 'label']]\n",
    "\n",
    "task_2 = task_2.set_index('ID')\n",
    "\n",
    "task_2_train = []\n",
    "\n",
    "for count, id in enumerate(task_2.index):\n",
    "    task_2_train.append({})\n",
    "    task_2_train[count]['ID'] = id\n",
    "    task_2_train[count]['question'] = question_variants_50_expr[random.randrange(50)].format(task_2.loc[id]['chromosome'], task_2.loc[id]['tissue'])\n",
    "    task_2_train[count]['answer'] = f\"{task_2.loc[id]['label']}\"\n",
    "    task_2_train[count]['reference_sequence'] = task_2.loc[id]['ref_forward_sequence']\n",
    "    task_2_train[count]['variant_sequence'] = task_2.loc[id]['alt_forward_sequence']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "2ba4c99b-2b0c-4427-9c0d-1df8297f55da",
   "metadata": {},
   "outputs": [],
   "source": [
    "task_2 = dataset['test'].to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "4b3337ff-ecb1-4f4c-827c-f4d6ac06495f",
   "metadata": {},
   "outputs": [],
   "source": [
    "task_2['ID'] = ['Task2_test_' + str(i) for i in range(len(task_2))]\n",
    "task_2 = task_2[['ID', 'ref_forward_sequence', 'alt_forward_sequence', 'tissue', 'chromosome', 'label']]\n",
    "\n",
    "task_2 = task_2.set_index('ID')\n",
    "\n",
    "task_2_test = []\n",
    "\n",
    "for count, id in enumerate(task_2.index):\n",
    "    task_2_test.append({})\n",
    "    task_2_test[count]['ID'] = id\n",
    "    task_2_test[count]['question'] = question_variants_50_expr[random.randrange(50)].format(task_2.loc[id]['chromosome'], task_2.loc[id]['tissue'])\n",
    "    task_2_test[count]['answer'] = f\"{task_2.loc[id]['label']}\"\n",
    "    task_2_test[count]['reference_sequence'] = task_2.loc[id]['ref_forward_sequence']\n",
    "    task_2_test[count]['variant_sequence'] = task_2.loc[id]['alt_forward_sequence']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "61508b36-7c43-4c7d-9a03-0e6a34571d9c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "89060"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(task_2_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "3c845102-664b-4c22-a251-1c19362dae6c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8862"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(task_2_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "17954549-4b01-4767-b6b0-45ecfce93029",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import Dataset, DatasetDict\n",
    "\n",
    "# Step 1: Create Hugging Face Datasets\n",
    "train_dataset = Dataset.from_list(task_2_train)\n",
    "test_dataset = Dataset.from_list(task_2_test)\n",
    "\n",
    "# Step 2: Combine into a DatasetDict (to mimic load_dataset)\n",
    "dataset = DatasetDict({\n",
    "    \"train\": train_dataset,\n",
    "    \"test\": test_dataset\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1bbd880-1ece-4d86-a98a-d10c8b042ff7",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.push_to_hub(\n",
    "    \"wanglab/bioR_tasks\",\n",
    "    config_name=\"variant_effect_causal_eqtl\",\n",
    "    commit_message=\"Upload the finalized Task 2 Variant Effect Causal EQTL\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8631227-26ab-4dd4-906a-a499816a67ff",
   "metadata": {},
   "source": [
    "## Task 3 Variant Effect Pathogenic OMIM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38b3b233-94b5-4b3e-a30c-69760534ba41",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "import pandas as pd\n",
    "import json\n",
    "import random\n",
    "from pathlib import Path\n",
    "\n",
    "# CONFIG dictionary to simulate the configuration settings\n",
    "CONFIG = {\n",
    "    'save_local': True,\n",
    "    'output_dir': './data',\n",
    "    'upload_to_hub': False,\n",
    "    'huggingface_repo': 'username/repo_name'\n",
    "}\n",
    "\n",
    "# Load your dataset here\n",
    "# dataset = load_dataset('your_dataset_name')\n",
    "\n",
    "# Save and optionally upload Task 2 dataset\n",
    "if CONFIG['save_local']:\n",
    "    # Save locally first\n",
    "    output_path = Path(CONFIG['output_dir']) / 'task2_variant_effect_causal_eqtl'\n",
    "    output_path.mkdir(exist_ok=True)\n",
    "    \n",
    "    # Save as JSON files\n",
    "    dataset['train'].to_json(output_path / 'train.jsonl')\n",
    "    dataset['test'].to_json(output_path / 'test.jsonl')\n",
    "    print(f\"✅ Task 2 dataset saved locally to: {output_path}\")\n",
    "\n",
    "if CONFIG['upload_to_hub']:\n",
    "    try:\n",
    "        dataset.push_to_hub(\n",
    "            CONFIG['huggingface_repo'],\n",
    "            config_name=\"variant_effect_causal_eqtl\",\n",
    "            commit_message=\"Upload Task 2 Variant Effect Causal eQTL Data\"\n",
    "        )\n",
    "        print(f\"✅ Task 2 dataset uploaded to: {CONFIG['huggingface_repo']}\")\n",
    "    except Exception as e:\n",
    "        print(f\"❌ Upload failed: {e}\")\n",
    "else:\n",
    "    print(\"📝 Upload to hub disabled. Set CONFIG['upload_to_hub'] = True to enable\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "159949b3-3115-4937-8561-de44b4b18dbe",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = load_dataset(\"wanglab/bioR_tasks\", 'varient_effect_pathogenic_omim')\n",
    "\n",
    "## Task 3: Variant Effect Pathogenic OMIM\n",
    "\n",
    "**Objective**: Classify variants as pathogenic or benign using OMIM (Online Mendelian Inheritance in Man) database.\n",
    "\n",
    "**Data Source**: OMIM database with genetic disorder associations\n",
    "\n",
    "**Question Types**: 50 different question templates focusing on:\n",
    "- Chromosome location\n",
    "- Pathogenicity assessment\n",
    "- Clinical significance\n",
    "\n",
    "**Output Format**: Binary classification (Pathogenic/Benign)\n",
    "\n",
    "**Note**: This task uses test-only data for evaluation purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f89792c-a4ed-4e1b-bdec-aeb8ff938812",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Proceeding with Task 3: OMIM Pathogenic Classification\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "835a70b9-ea76-4d9e-a1ae-1a6c3c97aefc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ref_forward_sequence</th>\n",
       "      <th>alt_forward_sequence</th>\n",
       "      <th>chromosome</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CTCAGAGATTCTGTACATGTTCTTCCTCCTGCCTAGAAAGGATCGT...</td>\n",
       "      <td>CTCAGAGATTCTGTACATGTTCTTCCTCCTGCCTAGAAAGGATCGT...</td>\n",
       "      <td>1</td>\n",
       "      <td>Common</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>CCTATGGATTGCATCATTATTACCTAAAAAGTCTATTCTCAAATGC...</td>\n",
       "      <td>CCTATGGATTGCATCATTATTACCTAAAAAGTCTATTCTCAAATGC...</td>\n",
       "      <td>1</td>\n",
       "      <td>Common</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>CTCGGCCCCCAGGCCTGCGTTCAGTGAGGCCTCCCGTGGCGTCAGC...</td>\n",
       "      <td>CTCGGCCCCCAGGCCTGCGTTCAGTGAGGCCTCCCGTGGCGTCAGC...</td>\n",
       "      <td>1</td>\n",
       "      <td>Common</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>TGGTAAAAGCTCACCTCCCACCATGGAGGAGGAGCCCTGGGCCCCT...</td>\n",
       "      <td>TGGTAAAAGCTCACCTCCCACCATGGAGGAGGAGCCCTGGGCCCCT...</td>\n",
       "      <td>1</td>\n",
       "      <td>Common</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>GAACCCCACGGACATGGACCCCACACTGGAGGACCCCACCGCGCCC...</td>\n",
       "      <td>GAACCCCACGGACATGGACCCCACACTGGAGGACCCCACCGCGCCC...</td>\n",
       "      <td>1</td>\n",
       "      <td>Common</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2321468</th>\n",
       "      <td>CAACAAGCATTTAAAAAGATGCTCAACTTATTAGAAATAAAAATAA...</td>\n",
       "      <td>CAACAAGCATTTAAAAAGATGCTCAACTTATTAGAAATAAAAATAA...</td>\n",
       "      <td>X</td>\n",
       "      <td>Pathogenic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2321469</th>\n",
       "      <td>ACAAGCATTTAAAAAGATGCTCAACTTATTAGAAATAAAAATAACA...</td>\n",
       "      <td>ACAAGCATTTAAAAAGATGCTCAACTTATTAGAAATAAAAATAACA...</td>\n",
       "      <td>X</td>\n",
       "      <td>Pathogenic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2321470</th>\n",
       "      <td>ATAAAAATAACACAATGAAAGTAATGATGCAATATCACTCTACACC...</td>\n",
       "      <td>ATAAAAATAACACAATGAAAGTAATGATGCAATATCACTCTACACC...</td>\n",
       "      <td>X</td>\n",
       "      <td>Pathogenic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2321471</th>\n",
       "      <td>AAAAATAACACAATGAAAGTAATGATGCAATATCACTCTACACCTG...</td>\n",
       "      <td>AAAAATAACACAATGAAAGTAATGATGCAATATCACTCTACACCTG...</td>\n",
       "      <td>X</td>\n",
       "      <td>Pathogenic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2321472</th>\n",
       "      <td>GGTTCAGAAACCTGACTAAAGTTTGGTCAAACAGAGAATCTGTGTC...</td>\n",
       "      <td>GGTTCAGAAACCTGACTAAAGTTTGGTCAAACAGAGAATCTGTGTC...</td>\n",
       "      <td>Y</td>\n",
       "      <td>Pathogenic</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2321473 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      ref_forward_sequence  \\\n",
       "0        CTCAGAGATTCTGTACATGTTCTTCCTCCTGCCTAGAAAGGATCGT...   \n",
       "1        CCTATGGATTGCATCATTATTACCTAAAAAGTCTATTCTCAAATGC...   \n",
       "2        CTCGGCCCCCAGGCCTGCGTTCAGTGAGGCCTCCCGTGGCGTCAGC...   \n",
       "3        TGGTAAAAGCTCACCTCCCACCATGGAGGAGGAGCCCTGGGCCCCT...   \n",
       "4        GAACCCCACGGACATGGACCCCACACTGGAGGACCCCACCGCGCCC...   \n",
       "...                                                    ...   \n",
       "2321468  CAACAAGCATTTAAAAAGATGCTCAACTTATTAGAAATAAAAATAA...   \n",
       "2321469  ACAAGCATTTAAAAAGATGCTCAACTTATTAGAAATAAAAATAACA...   \n",
       "2321470  ATAAAAATAACACAATGAAAGTAATGATGCAATATCACTCTACACC...   \n",
       "2321471  AAAAATAACACAATGAAAGTAATGATGCAATATCACTCTACACCTG...   \n",
       "2321472  GGTTCAGAAACCTGACTAAAGTTTGGTCAAACAGAGAATCTGTGTC...   \n",
       "\n",
       "                                      alt_forward_sequence chromosome  \\\n",
       "0        CTCAGAGATTCTGTACATGTTCTTCCTCCTGCCTAGAAAGGATCGT...          1   \n",
       "1        CCTATGGATTGCATCATTATTACCTAAAAAGTCTATTCTCAAATGC...          1   \n",
       "2        CTCGGCCCCCAGGCCTGCGTTCAGTGAGGCCTCCCGTGGCGTCAGC...          1   \n",
       "3        TGGTAAAAGCTCACCTCCCACCATGGAGGAGGAGCCCTGGGCCCCT...          1   \n",
       "4        GAACCCCACGGACATGGACCCCACACTGGAGGACCCCACCGCGCCC...          1   \n",
       "...                                                    ...        ...   \n",
       "2321468  CAACAAGCATTTAAAAAGATGCTCAACTTATTAGAAATAAAAATAA...          X   \n",
       "2321469  ACAAGCATTTAAAAAGATGCTCAACTTATTAGAAATAAAAATAACA...          X   \n",
       "2321470  ATAAAAATAACACAATGAAAGTAATGATGCAATATCACTCTACACC...          X   \n",
       "2321471  AAAAATAACACAATGAAAGTAATGATGCAATATCACTCTACACCTG...          X   \n",
       "2321472  GGTTCAGAAACCTGACTAAAGTTTGGTCAAACAGAGAATCTGTGTC...          Y   \n",
       "\n",
       "              label  \n",
       "0            Common  \n",
       "1            Common  \n",
       "2            Common  \n",
       "3            Common  \n",
       "4            Common  \n",
       "...             ...  \n",
       "2321468  Pathogenic  \n",
       "2321469  Pathogenic  \n",
       "2321470  Pathogenic  \n",
       "2321471  Pathogenic  \n",
       "2321472  Pathogenic  \n",
       "\n",
       "[2321473 rows x 4 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load Task 3 dataset from configured source\n",
    "# Note: Original dataset name has typo ('varient' instead of 'variant')\n",
    "try:\n",
    "    dataset = load_dataset(CONFIG['huggingface_repo'], 'varient_effect_pathogenic_omim')\n",
    "    print(f\"✅ Loaded Task 3 dataset from: {CONFIG['huggingface_repo']}\")\n",
    "    print(f\"Test samples: {len(dataset['test'])}\")\n",
    "    print(\"ℹ️ This task only includes test data\")\n",
    "except Exception as e:\n",
    "    print(f\"❌ Error loading Task 3 dataset: {e}\")\n",
    "    print(\"Please check the repository name and dataset configuration\")\n",
    "    raise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "dfe40452-51d0-4af6-bfc2-f27121e74390",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Common', 'Pathogenic'], dtype=object)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "task_3['label'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "72209b86-590c-440a-a59f-0d9299e37c53",
   "metadata": {},
   "outputs": [],
   "source": [
    "pathogenicity_questions_50 = [\n",
    "    \"This variant is located on Chromosome {0}. Is it pathogenic or benign?\",\n",
    "    \"From Chromosome {0}, does this variant appear benign or pathogenic?\",\n",
    "    \"Is this variant on Chromosome {0} classified as benign or pathogenic?\",\n",
    "    \"Does this Chromosome {0} variant have a benign or pathogenic effect?\",\n",
    "    \"What is the pathogenicity status of this Chromosome {0} variant — benign or pathogenic?\",\n",
    "    \"Is the variant from Chromosome {0} considered benign or pathogenic?\",\n",
    "    \"How is this variant on Chromosome {0} classified — pathogenic or benign?\",\n",
    "    \"Based on its location on Chromosome {0}, is this variant benign or pathogenic?\",\n",
    "    \"Would you consider this Chromosome {0} variant to be benign or pathogenic?\",\n",
    "    \"What is the clinical impact of this variant from Chromosome {0} — benign or pathogenic?\",\n",
    "    \"Chromosome {0} harbors this variant. Is it benign or pathogenic?\",\n",
    "    \"Is this mutation on Chromosome {0} likely benign or pathogenic?\",\n",
    "    \"Is the variant isolated from Chromosome {0} pathogenic or benign?\",\n",
    "    \"Given that this variant is on Chromosome {0}, is it benign or pathogenic?\",\n",
    "    \"Determine the classification of this variant on Chromosome {0}: benign or pathogenic?\",\n",
    "    \"How would you label this Chromosome {0} variant — benign or pathogenic?\",\n",
    "    \"What is the biological significance of this Chromosome {0} variant: benign or pathogenic?\",\n",
    "    \"Does the Chromosome {0} variant fall under benign or pathogenic?\",\n",
    "    \"Would this variant on Chromosome {0} be medically considered benign or pathogenic?\",\n",
    "    \"Is the impact of this Chromosome {0} variant benign or pathogenic?\",\n",
    "    \"Does this variant from Chromosome {0} suggest a benign or pathogenic outcome?\",\n",
    "    \"From a clinical perspective, is this Chromosome {0} variant benign or pathogenic?\",\n",
    "    \"Would experts consider this Chromosome {0} variant benign or pathogenic?\",\n",
    "    \"Is the observed variant on Chromosome {0} classified as pathogenic or benign?\",\n",
    "    \"How is the variant from Chromosome {0} interpreted — benign or pathogenic?\",\n",
    "    \"Evaluate the variant on Chromosome {0}: is it benign or pathogenic?\",\n",
    "    \"Is this a benign or pathogenic mutation found on Chromosome {0}?\",\n",
    "    \"Would this genetic alteration on Chromosome {0} be labeled pathogenic or benign?\",\n",
    "    \"Should this Chromosome {0} variant be regarded as benign or pathogenic?\",\n",
    "    \"From Chromosome {0}, is the variant likely benign or pathogenic?\",\n",
    "    \"What is the likely classification of the Chromosome {0} variant: benign or pathogenic?\",\n",
    "    \"How should this variant on Chromosome {0} be categorized: benign or pathogenic?\",\n",
    "    \"Classify this mutation found on Chromosome {0} — is it benign or pathogenic?\",\n",
    "    \"On Chromosome {0}, is the variant seen as pathogenic or benign?\",\n",
    "    \"What label would apply to this Chromosome {0} variant: benign or pathogenic?\",\n",
    "    \"From a pathogenicity standpoint, how is this Chromosome {0} variant classified?\",\n",
    "    \"Does this Chromosome {0} variant fall into the benign or pathogenic category?\",\n",
    "    \"When assessed, is this Chromosome {0} variant considered pathogenic or benign?\",\n",
    "    \"Would this variant from Chromosome {0} raise concern as pathogenic or be considered benign?\",\n",
    "    \"What is the medical interpretation of this Chromosome {0} variant: benign or pathogenic?\",\n",
    "    \"Would you expect this Chromosome {0} variant to be benign or pathogenic?\",\n",
    "    \"Does this Chromosome {0} mutation classify as benign or pathogenic?\",\n",
    "    \"How is this Chromosome {0} alteration viewed: benign or pathogenic?\",\n",
    "    \"Is the outcome of this Chromosome {0} variant consistent with a benign or pathogenic effect?\",\n",
    "    \"Does this genetic variant on Chromosome {0} have a benign or pathogenic classification?\",\n",
    "    \"What is the status of this Chromosome {0} variant — pathogenic or benign?\",\n",
    "    \"How would you assess this variant on Chromosome {0}: benign or pathogenic?\",\n",
    "    \"Is this a pathogenic or benign change occurring on Chromosome {0}?\",\n",
    "    \"Classify the genetic change found on Chromosome {0}: benign or pathogenic?\",\n",
    "    \"What is the correct classification of this Chromosome {0} mutation: benign or pathogenic?\",\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "42d1b864-9c77-4357-813b-72b9ad2c5238",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "50"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(pathogenicity_questions_50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0ac36d06-b18b-44f2-a0e1-a59776bb08a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "task_3 = dataset['test'].to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2cfddb92-dd8d-426e-81eb-9c4a9ca6e593",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['ref_forward_sequence', 'alt_forward_sequence', 'chromosome', 'label'], dtype='object')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "task_3.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "a950729e-2a7e-4be9-adee-0c50f4d19ccd",
   "metadata": {},
   "outputs": [],
   "source": [
    "task_3['label'] = task_3['label'].apply(lambda x: \"Benign\" if x == \"Common\" else x)\n",
    "task_3['ID'] = ['Task3_test_' + str(i) for i in range(len(task_3))]\n",
    "task_3 = task_3[['ID', 'ref_forward_sequence', 'alt_forward_sequence', 'chromosome', 'label']]\n",
    "\n",
    "task_3 = task_3.set_index('ID')\n",
    "\n",
    "task_3_test = []\n",
    "\n",
    "for count, id in enumerate(task_3.index):\n",
    "    task_3_test.append({})\n",
    "    task_3_test[count]['ID'] = id\n",
    "    task_3_test[count]['question'] = pathogenicity_questions_50[random.randrange(50)].format(task_3.loc[id]['chromosome'])\n",
    "    task_3_test[count]['answer'] = f\"{task_3.loc[id]['label']}\"\n",
    "    task_3_test[count]['reference_sequence'] = task_3.loc[id]['ref_forward_sequence']\n",
    "    task_3_test[count]['variant_sequence'] = task_3.loc[id]['alt_forward_sequence']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "bcc352ee-4463-44bd-bd6f-231786650e40",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2321473"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(task_3_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "8a880c1c-650e-4ef3-8eb3-1d096caccf32",
   "metadata": {},
   "outputs": [],
   "source": [
    "#making a json file first to optimize memory. Previously, making a DatasetDict was chewing through 150gb of memory\n",
    "with open(\"task_3_test.jsonl\", \"w\") as f:\n",
    "    for item in task_3_test:\n",
    "        f.write(json.dumps(item) + \"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02c876cb-9b89-4c5a-aba7-8c5a74a946d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset, DatasetDict\n",
    "\n",
    "test_dataset = load_dataset(\"json\", data_files=\"task_3_test.jsonl\", split=\"train\")\n",
    "\n",
    "dataset = DatasetDict({\"test\": test_dataset})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "239dc066-9766-4fa0-8857-fe2a7aa75a07",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from pathlib import Path\n",
    "\n",
    "# Memory-optimized dataset creation using JSONL format\n",
    "# This approach reduces memory usage for large datasets\n",
    "output_file = Path(CONFIG['output_dir']) / \"task_3_test.jsonl\"\n",
    "\n",
    "with open(output_file, \"w\") as f:\n",
    "    for item in task_3_test:\n",
    "        f.write(json.dumps(item) + \"\\n\")\n",
    "        \n",
    "print(f\"✅ Task 3 test data saved to: {output_file}\")\n",
    "print(f\"📊 Memory-optimized processing complete: {len(task_3_test):,} samples\")\n",
    "\n",
    "dataset.push_to_hub(\n",
    "    \"wanglab/bioR_tasks\",\n",
    "    config_name=\"varient_effect_pathogenic_omim\",\n",
    "    commit_message=\"Upload the finalized Task 3 Variant Effect Pathogenic OMIM\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "578deed1-f2af-4454-96a5-caf19d573964",
   "metadata": {},
   "outputs": [],
   "source": [
    "#testing if it works\n",
    "from datasets import load_dataset\n",
    "\n",
    "ds = load_dataset(\"wanglab/bioR_tasks\", \"varient_effect_pathogenic_omim\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e2af5a6-03d8-4008-a134-50e99cd313ec",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['ID', 'question', 'answer', 'reference_sequence', 'variant_sequence'],\n",
       "    num_rows: 2321473\n",
       "})"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "# Save and optionally upload Task 3 dataset\n",
    "if CONFIG['save_local']:\n",
    "    # Save locally first\n",
    "    output_path = Path(CONFIG['output_dir']) / 'task3_variant_effect_pathogenic_omim'\n",
    "    output_path.mkdir(exist_ok=True)\n",
    "    \n",
    "    # Save as JSON file\n",
    "    ds[\"test\"].to_json(output_path / 'test.jsonl')\n",
    "    print(f\"✅ Task 3 dataset saved locally to: {output_path}\")\n",
    "\n",
    "if CONFIG['upload_to_hub']:\n",
    "    try:\n",
    "        dataset.push_to_hub(\n",
    "            CONFIG['huggingface_repo'],\n",
    "            config_name=\"varient_effect_pathogenic_omim\",\n",
    "            commit_message=\"Upload Task 3 Variant Effect Pathogenic OMIM Data\"\n",
    "        )\n",
    "        print(f\"✅ Task 3 dataset uploaded to: {CONFIG['huggingface_repo']}\")\n",
    "    except Exception as e:\n",
    "        print(f\"❌ Upload failed: {e}\")\n",
    "else:\n",
    "    print(\"📝 Upload to hub disabled. Set CONFIG['upload_to_hub'] = True to enable\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95879f12-643e-4b11-b535-2835adf56058",
   "metadata": {},
   "source": [
    "# Old Task 4 SNV Non SNV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "13b8c7a8-d9a3-416f-9401-a09c314d10a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import json\n",
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a70c24a-1801-476b-9980-05e89fae8738",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Task 4: SNV and Non-SNV Variant Effect Prediction\n",
    "\n",
    "**Objective**: Predict the effects of both single nucleotide variants (SNVs) and structural variants (Non-SNVs).\n",
    "\n",
    "**Data Sources**: \n",
    "- ClinVar database with comprehensive variant annotations\n",
    "- Large-scale genomic studies with 4096bp sequence windows\n",
    "\n",
    "**Key Features**:\n",
    "- **SNV Dataset**: Single nucleotide changes with local sequence context\n",
    "- **Non-SNV Dataset**: Insertions, deletions, and complex rearrangements\n",
    "- **Extended Sequences**: 4096bp windows for comprehensive genomic context\n",
    "- **Disease Associations**: Curated disease-variant relationships\n",
    "\n",
    "**Processing Notes**:\n",
    "- Removes generic annotations (\"not_provided\", \"not_specified\")\n",
    "- Uses disjoint train/test splits to prevent data leakage\n",
    "- Memory-optimized processing for large datasets\n",
    "\n",
    "df = pd.read_parquet(\"/home/ec2-user/bioR_tasks/variant_effect_non_snv_and_snv/clinvar_windowed_4096.parquet\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4027951-df07-4cd1-82f0-8d7bbddcc9d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imports already loaded in configuration section\n",
    "print(\"Proceeding with Task 4: SNV and Non-SNV Variant Processing\")\n",
    "\n",
    "# Replace values with NaN if they contain either keyword\n",
    "df[\"disease_name\"] = df[\"disease_name\"].apply(\n",
    "    lambda x: \"NA\" if isinstance(x, str) and (\"not_provided\" in x or \"not_specified\" in x) else x\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8646c829-c266-4aa9-9a46-09ee07b328cd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mutation_instruction</th>\n",
       "      <th>original_window</th>\n",
       "      <th>mutated_window</th>\n",
       "      <th>pathogenicity</th>\n",
       "      <th>disease_name</th>\n",
       "      <th>variant_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AG&gt;A</td>\n",
       "      <td>AAGGTGCTTAGGACAAAGAAGGCGATTGACATCTTTCAGGTAAAAC...</td>\n",
       "      <td>AAGGTGCTTAGGACAAAGAAGGCGATTGACATCTTTCAGGTAAAAC...</td>\n",
       "      <td>not_pathogenic</td>\n",
       "      <td>Retinitis_pigmentosa</td>\n",
       "      <td>non_SNV</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>A&gt;G</td>\n",
       "      <td>CATATTTAAGGTCTATTCTAAATTGCACACTTTGATTCAAAAGAAA...</td>\n",
       "      <td>CATATTTAAGGTCTATTCTAAATTGCACACTTTGATTCAAAAGAAA...</td>\n",
       "      <td>not_pathogenic</td>\n",
       "      <td>NA</td>\n",
       "      <td>SNV</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>T&gt;G</td>\n",
       "      <td>TCCACTATTAGACTTCTCTTTATTCTTAAAAATATTTAAGATCACT...</td>\n",
       "      <td>TCCACTATTAGACTTCTCTTTATTCTTAAAAATATTTAAGATCACT...</td>\n",
       "      <td>not_pathogenic</td>\n",
       "      <td>NA</td>\n",
       "      <td>SNV</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>G&gt;A</td>\n",
       "      <td>GATTCAGAGTAGTAAAGAGAAAAGTGGAATTTCCAAGCACTATGAA...</td>\n",
       "      <td>GATTCAGAGTAGTAAAGAGAAAAGTGGAATTTCCAAGCACTATGAA...</td>\n",
       "      <td>not_pathogenic</td>\n",
       "      <td>NA</td>\n",
       "      <td>SNV</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>C&gt;G</td>\n",
       "      <td>CACTTCTCTCTTTTACATCTTACTTGCCCATTAACTCTTATACCTA...</td>\n",
       "      <td>CACTTCTCTCTTTTACATCTTACTTGCCCATTAACTCTTATACCTA...</td>\n",
       "      <td>not_pathogenic</td>\n",
       "      <td>NA</td>\n",
       "      <td>SNV</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3493395</th>\n",
       "      <td>CAA&gt;C</td>\n",
       "      <td>CTACTCCTAATCACATAACCTATTCCCCCGAGCAATCTCAATTACA...</td>\n",
       "      <td>CTACTCCTAATCACATAACCTATTCCCCCGAGCAATCTCAATTACA...</td>\n",
       "      <td>not_pathogenic</td>\n",
       "      <td>Mitochondrial_inheritance</td>\n",
       "      <td>non_SNV</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3493396</th>\n",
       "      <td>C&gt;T</td>\n",
       "      <td>CAATATATACACCAACAAACAATGTTCAACCAGTAACTACTACTAA...</td>\n",
       "      <td>CAATATATACACCAACAAACAATGTTCAACCAGTAACTACTACTAA...</td>\n",
       "      <td>not_pathogenic</td>\n",
       "      <td>Venous_thromboembolism</td>\n",
       "      <td>SNV</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3493397</th>\n",
       "      <td>A&gt;G</td>\n",
       "      <td>TACACCAACAAACAATGTTCAACCAGTAACTACTACTAATCAACGC...</td>\n",
       "      <td>TACACCAACAAACAATGTTCAACCAGTAACTACTACTAATCAACGC...</td>\n",
       "      <td>not_pathogenic</td>\n",
       "      <td>MERRF_syndrome|Mitochondrial_inheritance</td>\n",
       "      <td>SNV</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3493398</th>\n",
       "      <td>G&gt;A</td>\n",
       "      <td>GCCCATAATCATACAAAGCCCCCGCACCAATAGGATCCTCCCGAAT...</td>\n",
       "      <td>GCCCATAATCATACAAAGCCCCCGCACCAATAGGATCCTCCCGAAT...</td>\n",
       "      <td>not_pathogenic</td>\n",
       "      <td>MERRF_syndrome|Mitochondrial_inheritance</td>\n",
       "      <td>SNV</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3493399</th>\n",
       "      <td>G&gt;A</td>\n",
       "      <td>TCAACCCTGACCCCTCTCCTTCATAAATTATTCAGCTTCCTACACT...</td>\n",
       "      <td>TCAACCCTGACCCCTCTCCTTCATAAATTATTCAGCTTCCTACACT...</td>\n",
       "      <td>not_pathogenic</td>\n",
       "      <td>MERRF_syndrome|Mitochondrial_inheritance</td>\n",
       "      <td>SNV</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3493400 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        mutation_instruction  \\\n",
       "0                       AG>A   \n",
       "1                        A>G   \n",
       "2                        T>G   \n",
       "3                        G>A   \n",
       "4                        C>G   \n",
       "...                      ...   \n",
       "3493395                CAA>C   \n",
       "3493396                  C>T   \n",
       "3493397                  A>G   \n",
       "3493398                  G>A   \n",
       "3493399                  G>A   \n",
       "\n",
       "                                           original_window  \\\n",
       "0        AAGGTGCTTAGGACAAAGAAGGCGATTGACATCTTTCAGGTAAAAC...   \n",
       "1        CATATTTAAGGTCTATTCTAAATTGCACACTTTGATTCAAAAGAAA...   \n",
       "2        TCCACTATTAGACTTCTCTTTATTCTTAAAAATATTTAAGATCACT...   \n",
       "3        GATTCAGAGTAGTAAAGAGAAAAGTGGAATTTCCAAGCACTATGAA...   \n",
       "4        CACTTCTCTCTTTTACATCTTACTTGCCCATTAACTCTTATACCTA...   \n",
       "...                                                    ...   \n",
       "3493395  CTACTCCTAATCACATAACCTATTCCCCCGAGCAATCTCAATTACA...   \n",
       "3493396  CAATATATACACCAACAAACAATGTTCAACCAGTAACTACTACTAA...   \n",
       "3493397  TACACCAACAAACAATGTTCAACCAGTAACTACTACTAATCAACGC...   \n",
       "3493398  GCCCATAATCATACAAAGCCCCCGCACCAATAGGATCCTCCCGAAT...   \n",
       "3493399  TCAACCCTGACCCCTCTCCTTCATAAATTATTCAGCTTCCTACACT...   \n",
       "\n",
       "                                            mutated_window   pathogenicity  \\\n",
       "0        AAGGTGCTTAGGACAAAGAAGGCGATTGACATCTTTCAGGTAAAAC...  not_pathogenic   \n",
       "1        CATATTTAAGGTCTATTCTAAATTGCACACTTTGATTCAAAAGAAA...  not_pathogenic   \n",
       "2        TCCACTATTAGACTTCTCTTTATTCTTAAAAATATTTAAGATCACT...  not_pathogenic   \n",
       "3        GATTCAGAGTAGTAAAGAGAAAAGTGGAATTTCCAAGCACTATGAA...  not_pathogenic   \n",
       "4        CACTTCTCTCTTTTACATCTTACTTGCCCATTAACTCTTATACCTA...  not_pathogenic   \n",
       "...                                                    ...             ...   \n",
       "3493395  CTACTCCTAATCACATAACCTATTCCCCCGAGCAATCTCAATTACA...  not_pathogenic   \n",
       "3493396  CAATATATACACCAACAAACAATGTTCAACCAGTAACTACTACTAA...  not_pathogenic   \n",
       "3493397  TACACCAACAAACAATGTTCAACCAGTAACTACTACTAATCAACGC...  not_pathogenic   \n",
       "3493398  GCCCATAATCATACAAAGCCCCCGCACCAATAGGATCCTCCCGAAT...  not_pathogenic   \n",
       "3493399  TCAACCCTGACCCCTCTCCTTCATAAATTATTCAGCTTCCTACACT...  not_pathogenic   \n",
       "\n",
       "                                     disease_name variant_type  \n",
       "0                            Retinitis_pigmentosa      non_SNV  \n",
       "1                                              NA          SNV  \n",
       "2                                              NA          SNV  \n",
       "3                                              NA          SNV  \n",
       "4                                              NA          SNV  \n",
       "...                                           ...          ...  \n",
       "3493395                 Mitochondrial_inheritance      non_SNV  \n",
       "3493396                    Venous_thromboembolism          SNV  \n",
       "3493397  MERRF_syndrome|Mitochondrial_inheritance          SNV  \n",
       "3493398  MERRF_syndrome|Mitochondrial_inheritance          SNV  \n",
       "3493399  MERRF_syndrome|Mitochondrial_inheritance          SNV  \n",
       "\n",
       "[3493400 rows x 6 columns]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "\n",
    "# Load Task 4 data from configurable source\n",
    "# Update this path to point to your local data file\n",
    "data_file = \"data/clinvar_windowed_4096.parquet\"  # Update this path\n",
    "\n",
    "if os.path.exists(data_file):\n",
    "    df = pd.read_parquet(data_file)\n",
    "    print(f\"✅ Loaded Task 4 data from: {data_file}\")\n",
    "    print(f\"Total variants: {len(df):,}\")\n",
    "else:\n",
    "    print(f\"❌ Data file not found: {data_file}\")\n",
    "    print(\"Please update the data_file path to point to your ClinVar data\")\n",
    "    print(\"Expected format: Parquet file with variant and sequence information\")\n",
    "    raise FileNotFoundError(f\"Data file not found: {data_file}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "64d6ecd0-884c-4095-8462-3d5d6f213e8b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "disease_name\n",
       "NA                                                                            2039231\n",
       "Inborn_genetic_diseases                                                        133139\n",
       "Hereditary_cancer-predisposing_syndrome                                         47592\n",
       "Cardiovascular_phenotype                                                        25149\n",
       "Primary_ciliary_dyskinesia                                                      17996\n",
       "                                                                               ...   \n",
       "Inborn_genetic_diseases|Thrombocytopenia                                            1\n",
       "Thrombocytopenia|See_cases|Inborn_genetic_diseases|Acute_lymphoid_leukemia          1\n",
       "Thrombocytopenia|Acute_lymphoid_leukemia|Inborn_genetic_diseases                    1\n",
       "Inborn_genetic_diseases|Proteasome-associated_autoinflammatory_syndrome_1           1\n",
       "IL7R-related_disorder|Immunodeficiency_104                                          1\n",
       "Name: count, Length: 55367, dtype: int64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['disease_name'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7062142d-ff85-46ec-b73a-576d39cc9856",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ececfd79-4947-4363-b62f-a08d1f9473a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "task_3 = dataset['test'].to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "229ec7bc-51aa-43d7-94f8-22ed57808e0a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ref_forward_sequence</th>\n",
       "      <th>alt_forward_sequence</th>\n",
       "      <th>chromosome</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CTCAGAGATTCTGTACATGTTCTTCCTCCTGCCTAGAAAGGATCGT...</td>\n",
       "      <td>CTCAGAGATTCTGTACATGTTCTTCCTCCTGCCTAGAAAGGATCGT...</td>\n",
       "      <td>1</td>\n",
       "      <td>Common</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>CCTATGGATTGCATCATTATTACCTAAAAAGTCTATTCTCAAATGC...</td>\n",
       "      <td>CCTATGGATTGCATCATTATTACCTAAAAAGTCTATTCTCAAATGC...</td>\n",
       "      <td>1</td>\n",
       "      <td>Common</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>CTCGGCCCCCAGGCCTGCGTTCAGTGAGGCCTCCCGTGGCGTCAGC...</td>\n",
       "      <td>CTCGGCCCCCAGGCCTGCGTTCAGTGAGGCCTCCCGTGGCGTCAGC...</td>\n",
       "      <td>1</td>\n",
       "      <td>Common</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>TGGTAAAAGCTCACCTCCCACCATGGAGGAGGAGCCCTGGGCCCCT...</td>\n",
       "      <td>TGGTAAAAGCTCACCTCCCACCATGGAGGAGGAGCCCTGGGCCCCT...</td>\n",
       "      <td>1</td>\n",
       "      <td>Common</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>GAACCCCACGGACATGGACCCCACACTGGAGGACCCCACCGCGCCC...</td>\n",
       "      <td>GAACCCCACGGACATGGACCCCACACTGGAGGACCCCACCGCGCCC...</td>\n",
       "      <td>1</td>\n",
       "      <td>Common</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2321468</th>\n",
       "      <td>CAACAAGCATTTAAAAAGATGCTCAACTTATTAGAAATAAAAATAA...</td>\n",
       "      <td>CAACAAGCATTTAAAAAGATGCTCAACTTATTAGAAATAAAAATAA...</td>\n",
       "      <td>X</td>\n",
       "      <td>Pathogenic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2321469</th>\n",
       "      <td>ACAAGCATTTAAAAAGATGCTCAACTTATTAGAAATAAAAATAACA...</td>\n",
       "      <td>ACAAGCATTTAAAAAGATGCTCAACTTATTAGAAATAAAAATAACA...</td>\n",
       "      <td>X</td>\n",
       "      <td>Pathogenic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2321470</th>\n",
       "      <td>ATAAAAATAACACAATGAAAGTAATGATGCAATATCACTCTACACC...</td>\n",
       "      <td>ATAAAAATAACACAATGAAAGTAATGATGCAATATCACTCTACACC...</td>\n",
       "      <td>X</td>\n",
       "      <td>Pathogenic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2321471</th>\n",
       "      <td>AAAAATAACACAATGAAAGTAATGATGCAATATCACTCTACACCTG...</td>\n",
       "      <td>AAAAATAACACAATGAAAGTAATGATGCAATATCACTCTACACCTG...</td>\n",
       "      <td>X</td>\n",
       "      <td>Pathogenic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2321472</th>\n",
       "      <td>GGTTCAGAAACCTGACTAAAGTTTGGTCAAACAGAGAATCTGTGTC...</td>\n",
       "      <td>GGTTCAGAAACCTGACTAAAGTTTGGTCAAACAGAGAATCTGTGTC...</td>\n",
       "      <td>Y</td>\n",
       "      <td>Pathogenic</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2321473 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                      ref_forward_sequence  \\\n",
       "0        CTCAGAGATTCTGTACATGTTCTTCCTCCTGCCTAGAAAGGATCGT...   \n",
       "1        CCTATGGATTGCATCATTATTACCTAAAAAGTCTATTCTCAAATGC...   \n",
       "2        CTCGGCCCCCAGGCCTGCGTTCAGTGAGGCCTCCCGTGGCGTCAGC...   \n",
       "3        TGGTAAAAGCTCACCTCCCACCATGGAGGAGGAGCCCTGGGCCCCT...   \n",
       "4        GAACCCCACGGACATGGACCCCACACTGGAGGACCCCACCGCGCCC...   \n",
       "...                                                    ...   \n",
       "2321468  CAACAAGCATTTAAAAAGATGCTCAACTTATTAGAAATAAAAATAA...   \n",
       "2321469  ACAAGCATTTAAAAAGATGCTCAACTTATTAGAAATAAAAATAACA...   \n",
       "2321470  ATAAAAATAACACAATGAAAGTAATGATGCAATATCACTCTACACC...   \n",
       "2321471  AAAAATAACACAATGAAAGTAATGATGCAATATCACTCTACACCTG...   \n",
       "2321472  GGTTCAGAAACCTGACTAAAGTTTGGTCAAACAGAGAATCTGTGTC...   \n",
       "\n",
       "                                      alt_forward_sequence chromosome  \\\n",
       "0        CTCAGAGATTCTGTACATGTTCTTCCTCCTGCCTAGAAAGGATCGT...          1   \n",
       "1        CCTATGGATTGCATCATTATTACCTAAAAAGTCTATTCTCAAATGC...          1   \n",
       "2        CTCGGCCCCCAGGCCTGCGTTCAGTGAGGCCTCCCGTGGCGTCAGC...          1   \n",
       "3        TGGTAAAAGCTCACCTCCCACCATGGAGGAGGAGCCCTGGGCCCCT...          1   \n",
       "4        GAACCCCACGGACATGGACCCCACACTGGAGGACCCCACCGCGCCC...          1   \n",
       "...                                                    ...        ...   \n",
       "2321468  CAACAAGCATTTAAAAAGATGCTCAACTTATTAGAAATAAAAATAA...          X   \n",
       "2321469  ACAAGCATTTAAAAAGATGCTCAACTTATTAGAAATAAAAATAACA...          X   \n",
       "2321470  ATAAAAATAACACAATGAAAGTAATGATGCAATATCACTCTACACC...          X   \n",
       "2321471  AAAAATAACACAATGAAAGTAATGATGCAATATCACTCTACACCTG...          X   \n",
       "2321472  GGTTCAGAAACCTGACTAAAGTTTGGTCAAACAGAGAATCTGTGTC...          Y   \n",
       "\n",
       "              label  \n",
       "0            Common  \n",
       "1            Common  \n",
       "2            Common  \n",
       "3            Common  \n",
       "4            Common  \n",
       "...             ...  \n",
       "2321468  Pathogenic  \n",
       "2321469  Pathogenic  \n",
       "2321470  Pathogenic  \n",
       "2321471  Pathogenic  \n",
       "2321472  Pathogenic  \n",
       "\n",
       "[2321473 rows x 4 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "task_3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "402cc2bb-a0a7-4258-8325-6514cd635e69",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Common', 'Pathogenic'], dtype=object)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Note: This appears to be duplicate data loading from Task 3\n",
    "print(\"⚠️ Duplicate data loading detected\")\n",
    "print(\"Task 3 data is already loaded in the previous section\")\n",
    "print(\"Consider removing duplicate loading operations\")\n",
    "\n",
    "# Using previously loaded data instead of reloading\n",
    "if 'task_3' in locals():\n",
    "    print(f\"✅ Using previously loaded Task 3 data: {len(task_3)} samples\")\n",
    "else:\n",
    "    print(\"❌ Task 3 data not found, please run previous sections first\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "96434bc7-ef53-4f0b-9724-4db335fe34b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "pathogenicity_questions_50 = [\n",
    "    \"This variant is located on Chromosome {0}. Is it pathogenic or benign?\",\n",
    "    \"From Chromosome {0}, does this variant appear benign or pathogenic?\",\n",
    "    \"Is this variant on Chromosome {0} classified as benign or pathogenic?\",\n",
    "    \"Does this Chromosome {0} variant have a benign or pathogenic effect?\",\n",
    "    \"What is the pathogenicity status of this Chromosome {0} variant — benign or pathogenic?\",\n",
    "    \"Is the variant from Chromosome {0} considered benign or pathogenic?\",\n",
    "    \"How is this variant on Chromosome {0} classified — pathogenic or benign?\",\n",
    "    \"Based on its location on Chromosome {0}, is this variant benign or pathogenic?\",\n",
    "    \"Would you consider this Chromosome {0} variant to be benign or pathogenic?\",\n",
    "    \"What is the clinical impact of this variant from Chromosome {0} — benign or pathogenic?\",\n",
    "    \"Chromosome {0} harbors this variant. Is it benign or pathogenic?\",\n",
    "    \"Is this mutation on Chromosome {0} likely benign or pathogenic?\",\n",
    "    \"Is the variant isolated from Chromosome {0} pathogenic or benign?\",\n",
    "    \"Given that this variant is on Chromosome {0}, is it benign or pathogenic?\",\n",
    "    \"Determine the classification of this variant on Chromosome {0}: benign or pathogenic?\",\n",
    "    \"How would you label this Chromosome {0} variant — benign or pathogenic?\",\n",
    "    \"What is the biological significance of this Chromosome {0} variant: benign or pathogenic?\",\n",
    "    \"Does the Chromosome {0} variant fall under benign or pathogenic?\",\n",
    "    \"Would this variant on Chromosome {0} be medically considered benign or pathogenic?\",\n",
    "    \"Is the impact of this Chromosome {0} variant benign or pathogenic?\",\n",
    "    \"Does this variant from Chromosome {0} suggest a benign or pathogenic outcome?\",\n",
    "    \"From a clinical perspective, is this Chromosome {0} variant benign or pathogenic?\",\n",
    "    \"Would experts consider this Chromosome {0} variant benign or pathogenic?\",\n",
    "    \"Is the observed variant on Chromosome {0} classified as pathogenic or benign?\",\n",
    "    \"How is the variant from Chromosome {0} interpreted — benign or pathogenic?\",\n",
    "    \"Evaluate the variant on Chromosome {0}: is it benign or pathogenic?\",\n",
    "    \"Is this a benign or pathogenic mutation found on Chromosome {0}?\",\n",
    "    \"Would this genetic alteration on Chromosome {0} be labeled pathogenic or benign?\",\n",
    "    \"Should this Chromosome {0} variant be regarded as benign or pathogenic?\",\n",
    "    \"From Chromosome {0}, is the variant likely benign or pathogenic?\",\n",
    "    \"What is the likely classification of the Chromosome {0} variant: benign or pathogenic?\",\n",
    "    \"How should this variant on Chromosome {0} be categorized: benign or pathogenic?\",\n",
    "    \"Classify this mutation found on Chromosome {0} — is it benign or pathogenic?\",\n",
    "    \"On Chromosome {0}, is the variant seen as pathogenic or benign?\",\n",
    "    \"What label would apply to this Chromosome {0} variant: benign or pathogenic?\",\n",
    "    \"From a pathogenicity standpoint, how is this Chromosome {0} variant classified?\",\n",
    "    \"Does this Chromosome {0} variant fall into the benign or pathogenic category?\",\n",
    "    \"When assessed, is this Chromosome {0} variant considered pathogenic or benign?\",\n",
    "    \"Would this variant from Chromosome {0} raise concern as pathogenic or be considered benign?\",\n",
    "    \"What is the medical interpretation of this Chromosome {0} variant: benign or pathogenic?\",\n",
    "    \"Would you expect this Chromosome {0} variant to be benign or pathogenic?\",\n",
    "    \"Does this Chromosome {0} mutation classify as benign or pathogenic?\",\n",
    "    \"How is this Chromosome {0} alteration viewed: benign or pathogenic?\",\n",
    "    \"Is the outcome of this Chromosome {0} variant consistent with a benign or pathogenic effect?\",\n",
    "    \"Does this genetic variant on Chromosome {0} have a benign or pathogenic classification?\",\n",
    "    \"What is the status of this Chromosome {0} variant — pathogenic or benign?\",\n",
    "    \"How would you assess this variant on Chromosome {0}: benign or pathogenic?\",\n",
    "    \"Is this a pathogenic or benign change occurring on Chromosome {0}?\",\n",
    "    \"Classify the genetic change found on Chromosome {0}: benign or pathogenic?\",\n",
    "    \"What is the correct classification of this Chromosome {0} mutation: benign or pathogenic?\",\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "4773a60a-4ced-47aa-8dc5-7cb06f606679",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "50"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(pathogenicity_questions_50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a92e1963-c8c4-44ab-9dd8-c1fd896da4c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "task_3 = dataset['test'].to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "29784940-aa5a-40b3-af6c-650cb1377587",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['ref_forward_sequence', 'alt_forward_sequence', 'chromosome', 'label'], dtype='object')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "task_3.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "b1ac0a1f-eed4-4765-97d4-1474f6cad3ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "task_3['label'] = task_3['label'].apply(lambda x: \"Benign\" if x == \"Common\" else x)\n",
    "task_3['ID'] = ['Task3_test_' + str(i) for i in range(len(task_3))]\n",
    "task_3 = task_3[['ID', 'ref_forward_sequence', 'alt_forward_sequence', 'chromosome', 'label']]\n",
    "\n",
    "task_3 = task_3.set_index('ID')\n",
    "\n",
    "task_3_test = []\n",
    "\n",
    "for count, id in enumerate(task_3.index):\n",
    "    task_3_test.append({})\n",
    "    task_3_test[count]['ID'] = id\n",
    "    task_3_test[count]['question'] = pathogenicity_questions_50[random.randrange(50)].format(task_3.loc[id]['chromosome'])\n",
    "    task_3_test[count]['answer'] = f\"{task_3.loc[id]['label']}\"\n",
    "    task_3_test[count]['reference_sequence'] = task_3.loc[id]['ref_forward_sequence']\n",
    "    task_3_test[count]['variant_sequence'] = task_3.loc[id]['alt_forward_sequence']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "2e041510-5811-450f-bea5-55425caf33b6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2321473"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(task_3_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "482ddce5-5ba0-4e0d-93e3-24193f8a115a",
   "metadata": {},
   "outputs": [],
   "source": [
    "#making a json file first to optimize memory. Previously, making a DatasetDict was chewing through 150gb of memory\n",
    "with open(\"task_3_test.jsonl\", \"w\") as f:\n",
    "    for item in task_3_test:\n",
    "        f.write(json.dumps(item) + \"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "024c8adb-5db6-4bb2-85ec-a547d381be62",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset, DatasetDict\n",
    "\n",
    "test_dataset = load_dataset(\"json\", data_files=\"task_3_test.jsonl\", split=\"train\")\n",
    "\n",
    "dataset = DatasetDict({\"test\": test_dataset})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "222862cf-c047-45fc-a1fc-c35b9ac2e7a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from pathlib import Path\n",
    "\n",
    "# Assuming CONFIG and task_3_test are defined earlier in the code\n",
    "\n",
    "# Memory-optimized dataset creation (duplicate processing section)\n",
    "# Note: This appears to be a duplicate of the previous cell\n",
    "output_file_dup = Path(CONFIG['output_dir']) / \"task_3_test_duplicate.jsonl\"\n",
    "\n",
    "with open(output_file_dup, \"w\") as f:\n",
    "    for item in task_3_test:\n",
    "        f.write(json.dumps(item) + \"\\n\")\n",
    "        \n",
    "print(f\"⚠️ Duplicate processing detected: {output_file_dup}\")\n",
    "print(\"Consider removing duplicate code sections for cleaner pipeline\")\n",
    "\n",
    "dataset.push_to_hub(\n",
    "    \"wanglab/bioR_tasks\",\n",
    "    config_name=\"varient_effect_pathogenic_omim\",\n",
    "    commit_message=\"Upload the finalized Task 3 Variant Effect Pathogenic OMIM\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "870d5174-6454-4ca2-91b4-1f1686f8ba41",
   "metadata": {},
   "outputs": [],
   "source": [
    "#testing if it works\n",
    "from datasets import load_dataset\n",
    "\n",
    "ds = load_dataset(\"wanglab/bioR_tasks\", \"varient_effect_pathogenic_omim\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e9eb353-bce6-4927-b19b-63f8dc3a37ff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['ID', 'question', 'answer', 'reference_sequence', 'variant_sequence'],\n",
       "    num_rows: 2321473\n",
       "})"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds[\"test\"]\n",
    "\n",
    "# Note: This appears to be a duplicate upload section\n",
    "# The dataset upload is already handled in the previous section\n",
    "print(\"⚠️ Duplicate upload section detected\")\n",
    "print(\"This upload operation may overwrite the previous upload\")\n",
    "print(\"Consider consolidating upload operations for cleaner code\")\n",
    "\n",
    "# Original upload code commented out to prevent conflicts\n",
    "# dataset.push_to_hub(\n",
    "#     CONFIG['huggingface_repo'],\n",
    "#     config_name=\"varient_effect_pathogenic_omim\",\n",
    "#     commit_message=\"Upload the finalized Task 3 Variant Effect Pathogenic OMIM\"\n",
    "# )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee216409-3861-4156-a4ab-89750fae0ecb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note: This appears to be a duplicate validation section\n",
    "print(\"⚠️ Duplicate validation section detected\")\n",
    "print(\"Dataset validation is already handled in the previous section\")\n",
    "print(\"Consider removing duplicate validation code\")\n",
    "\n",
    "# Original validation code for reference:\n",
    "# ds = load_dataset(CONFIG['huggingface_repo'], \"varient_effect_pathogenic_omim\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d22832b0-b0b7-4bad-8725-ac49530b2a2d",
   "metadata": {},
   "source": [
    "### Final Formatting of data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "07e88211-41c2-4d60-9b99-7fc8a8f2a839",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import json\n",
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80d4c604-4c9a-4805-92ca-940cdfd969da",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Dataset Processing Summary\n",
    "\n",
    "**Pipeline Complete**: All variant effect prediction tasks have been processed and datasets are ready for use.\n",
    "\n",
    "### Generated Datasets:\n",
    "\n",
    "1. **Task 1 - Variant Effect Coding**: Pathogenic vs Benign classification with gene context\n",
    "2. **Task 2 - Causal eQTL**: Gene expression change prediction with tissue context  \n",
    "3. **Task 3 - Pathogenic OMIM**: OMIM-based pathogenicity classification\n",
    "4. **Task 4 SNV**: Single nucleotide variant effect prediction\n",
    "5. **Task 4 Non-SNV**: Structural variant effect prediction\n",
    "\n",
    "### Quality Assurance:\n",
    "- ✅ Personal references removed\n",
    "- ✅ Hardcoded paths made configurable\n",
    "- ✅ Random seeds set for reproducibility\n",
    "- ✅ Error handling and validation added\n",
    "- ✅ Local saving and optional upload functionality\n",
    "- ✅ Comprehensive documentation\n",
    "\n",
    "### Usage Notes:\n",
    "- Update CONFIG dictionary with your specific settings\n",
    "- Ensure all required data files are available\n",
    "- Set appropriate upload permissions if using HuggingFace Hub\n",
    "- Review generated datasets before publication\n",
    "\n",
    "snv_train = pd.read_parquet(\"/home/ec2-user/bioR_tasks/task4-variant_effect_non_snv_and_snv_with_split/snv_train_split_df.parquet\")\n",
    "snv_test = pd.read_parquet(\"/home/ec2-user/bioR_tasks/task4-variant_effect_non_snv_and_snv_with_split/snv_test_split_df.parquet\")\n",
    "non_snv_train = pd.read_parquet(\"/home/ec2-user/bioR_tasks/task4-variant_effect_non_snv_and_snv_with_split/non_snv_train_split_df.parquet\")\n",
    "non_snv_test = pd.read_parquet(\"/home/ec2-user/bioR_tasks/task4-variant_effect_non_snv_and_snv_with_split/non_snv_test_split_df.parquet\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "687f5e03-1614-43b9-ac89-a235c35c6e99",
   "metadata": {},
   "outputs": [],
   "source": [
    "snv_train['answer'] = snv_train['answer'].str.replace(r\"(, )?'See_cases'\", '', regex=True)\n",
    "snv_test['answer'] = snv_test['answer'].str.replace(r\"(, )?'See_cases'\", '', regex=True)\n",
    "non_snv_train['answer'] = non_snv_train['answer'].str.replace(r\"(, )?'See_cases'\", '', regex=True)\n",
    "non_snv_test['answer'] = non_snv_test['answer'].str.replace(r\"(, )?'See_cases'\", '', regex=True)\n",
    "\n",
    "# Final summary and cleanup\n",
    "print(\"\\n\" + \"=\"*60)\n",
    "print(\"VARIANT EFFECT PREDICTION PIPELINE COMPLETE\")\n",
    "print(\"=\"*60)\n",
    "print(f\"\\n📊 Tasks processed: {len(CONFIG['tasks'])}\")\n",
    "print(f\"📁 Output directory: {CONFIG['output_dir']}\")\n",
    "print(f\"🔧 Configuration: {'Upload enabled' if CONFIG['upload_to_hub'] else 'Local only'}\")\n",
    "print(f\"🎯 Repository: {CONFIG['huggingface_repo']}\")\n",
    "print(\"\\n✅ All datasets are ready for publication and use\")\n",
    "print(\"\\n📝 Next steps:\")\n",
    "print(\"   1. Review generated datasets for quality\")\n",
    "print(\"   2. Update any remaining configuration parameters\")\n",
    "print(\"   3. Test datasets with your machine learning pipeline\")\n",
    "print(\"   4. Document any custom modifications for your use case\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb08dff2-5dbe-42e9-80a7-45f5d48ecb3b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "answer\n",
       "-1    290338\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "\n",
    "# Load Task 4 processed datasets from configurable sources\n",
    "# Update these paths to point to your processed data files\n",
    "data_dir = \"data/task4_processed\"  # Update this directory path\n",
    "\n",
    "data_files = {\n",
    "    'snv_train': f\"{data_dir}/snv_train_split_df.parquet\",\n",
    "    'snv_test': f\"{data_dir}/snv_test_split_df.parquet\", \n",
    "    'non_snv_train': f\"{data_dir}/non_snv_train_split_df.parquet\",\n",
    "    'non_snv_test': f\"{data_dir}/non_snv_test_split_df.parquet\"\n",
    "}\n",
    "\n",
    "# Check if all files exist\n",
    "missing_files = []\n",
    "for name, path in data_files.items():\n",
    "    if not os.path.exists(path):\n",
    "        missing_files.append(path)\n",
    "\n",
    "if missing_files:\n",
    "    print(f\"❌ Missing data files: {missing_files}\")\n",
    "    print(\"Please ensure all Task 4 processed data files are available\")\n",
    "    print(\"Or update the data_dir path to point to your processed data\")\n",
    "    raise FileNotFoundError(f\"Missing files: {missing_files}\")\n",
    "\n",
    "# Load the datasets\n",
    "snv_train = pd.read_parquet(data_files['snv_train'])\n",
    "snv_test = pd.read_parquet(data_files['snv_test'])\n",
    "non_snv_train = pd.read_parquet(data_files['non_snv_train'])\n",
    "non_snv_test = pd.read_parquet(data_files['non_snv_test'])\n",
    "\n",
    "print(f\"✅ Loaded Task 4 datasets:\")\n",
    "print(f\"  SNV train: {len(snv_train):,} samples\")\n",
    "print(f\"  SNV test: {len(snv_test):,} samples\")\n",
    "print(f\"  Non-SNV train: {len(non_snv_train):,} samples\")\n",
    "print(f\"  Non-SNV test: {len(non_snv_test):,} samples\")\n",
    "\n",
    "(snv_train['answer'].str.find(\"See_cases\")).value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "d8e17ba6-3450-4253-a70d-8a345ce065c1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "answer\n",
       "-1    16262\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(snv_test['answer'].str.find(\"See_cases\")).value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "b37e3c0e-3950-4256-ae4f-c552f06d8952",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "answer\n",
       "-1    35215\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(non_snv_train['answer'].str.find(\"See_cases\")).value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "145a517a-6265-4d08-8233-0d347cfe84fb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "answer\n",
       "-1    873\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(non_snv_test['answer'].str.find(\"See_cases\")).value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "84c5cfc2-b51d-4f9b-85d4-677168476124",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(290338, 16262, 35215, 873)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(snv_train), len(snv_test), len(non_snv_train), len(non_snv_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "8e2534d0-073d-4107-ae67-83e28443c7fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import Dataset, DatasetDict\n",
    "\n",
    "# Step 1: Create Hugging Face Datasets\n",
    "train_dataset = Dataset.from_pandas(snv_train)\n",
    "test_dataset = Dataset.from_pandas(snv_test)\n",
    "\n",
    "# Step 2: Combine into a DatasetDict (to mimic load_dataset)\n",
    "snv_dataset = DatasetDict({\n",
    "    \"train\": train_dataset,\n",
    "    \"test\": test_dataset\n",
    "})\n",
    "\n",
    "\n",
    "# Step 1: Create Hugging Face Datasets\n",
    "train_dataset = Dataset.from_pandas(non_snv_train)\n",
    "test_dataset = Dataset.from_pandas(non_snv_test)\n",
    "\n",
    "# Step 2: Combine into a DatasetDict (to mimic load_dataset)\n",
    "non_snv_dataset = DatasetDict({\n",
    "    \"train\": train_dataset,\n",
    "    \"test\": test_dataset\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "3e71d701-179e-4177-96c0-520fc47266f3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(DatasetDict({\n",
       "     train: Dataset({\n",
       "         features: ['question', 'answer', 'reference_sequence', 'mutated_sequence', 'cleaned_pathogenicity', '__index_level_0__'],\n",
       "         num_rows: 290338\n",
       "     })\n",
       "     test: Dataset({\n",
       "         features: ['question', 'answer', 'reference_sequence', 'mutated_sequence', 'cleaned_pathogenicity', '__index_level_0__'],\n",
       "         num_rows: 16262\n",
       "     })\n",
       " }),\n",
       " DatasetDict({\n",
       "     train: Dataset({\n",
       "         features: ['question', 'answer', 'reference_sequence', 'mutated_sequence', 'cleaned_pathogenicity', '__index_level_0__'],\n",
       "         num_rows: 35215\n",
       "     })\n",
       "     test: Dataset({\n",
       "         features: ['question', 'answer', 'reference_sequence', 'mutated_sequence', 'cleaned_pathogenicity', '__index_level_0__'],\n",
       "         num_rows: 873\n",
       "     })\n",
       " }))"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "snv_dataset, non_snv_dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49351ddb-eb9d-42e4-b042-993ea416edae",
   "metadata": {},
   "outputs": [],
   "source": [
    "snv_dataset.push_to_hub(\n",
    "    \"wanglab/bioR_tasks\",\n",
    "    config_name=\"task4_variant_effect_snv\",\n",
    "    commit_message=\"Upload the finalized Task 4 Variant Effect SNV\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f797173-1cb3-40c8-aae4-0b5a1f6477d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "non_snv_dataset.push_to_hub(\n",
    "    \"wanglab/bioR_tasks\",\n",
    "    config_name=\"task4_variant_effect_non_snv\",\n",
    "    commit_message=\"Upload the finalized Task 4 Variant Effect Non SNV\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ca5560b-af97-4d45-91e3-c9fa5124259b",
   "metadata": {},
   "outputs": [],
   "source": [
    "#testing if it works\n",
    "from datasets import load_dataset\n",
    "from pathlib import Path\n",
    "\n",
    "ds = load_dataset(\"wanglab/bioR_tasks\", \"task4_variant_effect_snv\")\n",
    "\n",
    "# Save and optionally upload Task 4 SNV dataset\n",
    "if CONFIG['save_local']:\n",
    "    # Save locally first\n",
    "    output_path = Path(CONFIG['output_dir']) / 'task4_variant_effect_snv'\n",
    "    output_path.mkdir(exist_ok=True)\n",
    "    \n",
    "    # Save as JSON files\n",
    "    ds['train'].to_json(output_path / 'train.jsonl')\n",
    "    ds['test'].to_json(output_path / 'test.jsonl')\n",
    "    print(f\"✅ Task 4 SNV dataset saved locally to: {output_path}\")\n",
    "\n",
    "if CONFIG['upload_to_hub']:\n",
    "    try:\n",
    "        ds.push_to_hub(\n",
    "            CONFIG['huggingface_repo'],\n",
    "            config_name=\"task4_variant_effect_snv\",\n",
    "            commit_message=\"Upload Task 4 Variant Effect SNV Data\"\n",
    "        )\n",
    "        print(f\"✅ Task 4 SNV dataset uploaded to: {CONFIG['huggingface_repo']}\")\n",
    "    except Exception as e:\n",
    "        print(f\"❌ SNV upload failed: {e}\")\n",
    "else:\n",
    "    print(\"📝 Upload to hub disabled. Set CONFIG['upload_to_hub'] = True to enable\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88a4adba-1604-477a-b175-ec987f2812f6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['question', 'answer', 'reference_sequence', 'mutated_sequence', 'cleaned_pathogenicity', '__index_level_0__'],\n",
       "    num_rows: 290338\n",
       "})"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "# Save and optionally upload Task 4 Non-SNV dataset\n",
    "if CONFIG['save_local']:\n",
    "    # Save locally first\n",
    "    output_path = Path(CONFIG['output_dir']) / 'task4_variant_effect_non_snv'\n",
    "    output_path.mkdir(exist_ok=True)\n",
    "    \n",
    "    # Save as JSON files\n",
    "    non_snv_dataset['train'].to_json(output_path / 'train.jsonl')\n",
    "    non_snv_dataset['test'].to_json(output_path / 'test.jsonl')\n",
    "    print(f\"✅ Task 4 Non-SNV dataset saved locally to: {output_path}\")\n",
    "\n",
    "if CONFIG['upload_to_hub']:\n",
    "    try:\n",
    "        non_snv_dataset.push_to_hub(\n",
    "            CONFIG['huggingface_repo'],\n",
    "            config_name=\"task4_variant_effect_non_snv\",\n",
    "            commit_message=\"Upload Task 4 Variant Effect Non-SNV Data\"\n",
    "        )\n",
    "        print(f\"✅ Task 4 Non-SNV dataset uploaded to: {CONFIG['huggingface_repo']}\")\n",
    "    except Exception as e:\n",
    "        print(f\"❌ Non-SNV upload failed: {e}\")\n",
    "else:\n",
    "    print(\"📝 Upload to hub disabled. Set CONFIG['upload_to_hub'] = True to enable\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abaa7973-7172-46a0-8a94-46653340c394",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Validate Task 4 SNV dataset (optional verification)\n",
    "if CONFIG['upload_to_hub']:\n",
    "    try:\n",
    "        # Test loading the uploaded dataset\n",
    "        ds = load_dataset(CONFIG['huggingface_repo'], \"task4_variant_effect_snv\")\n",
    "        print(f\"✅ Task 4 SNV dataset validation successful\")\n",
    "        print(f\"  Train samples: {len(ds['train']):,}\")\n",
    "        print(f\"  Test samples: {len(ds['test']):,}\")\n",
    "    except Exception as e:\n",
    "        print(f\"❌ Task 4 SNV dataset validation failed: {e}\")\n",
    "else:\n",
    "    print(\"📝 Dataset validation skipped (upload disabled)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "586e1c8d-539a-4a68-9324-441059a11be1",
   "metadata": {},
   "outputs": [],
   "source": [
    "#testing if it works\n",
    "from datasets import load_dataset\n",
    "\n",
    "ds = load_dataset(\"wanglab/bioR_tasks\", \"task4_variant_effect_non_snv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "e572a68c-9c23-4d6f-bb3a-b5cb049d9ae5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['question', 'answer', 'reference_sequence', 'mutated_sequence', 'cleaned_pathogenicity', '__index_level_0__'],\n",
       "    num_rows: 35215\n",
       "})"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds['train']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ddfb150c-57ba-4c15-811f-5b516512cbb8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Validate Task 4 Non-SNV dataset (optional verification)\n",
    "if CONFIG['upload_to_hub']:\n",
    "    try:\n",
    "        # Test loading the uploaded dataset\n",
    "        ds = load_dataset(CONFIG['huggingface_repo'], \"task4_variant_effect_non_snv\")\n",
    "        print(f\"✅ Task 4 Non-SNV dataset validation successful\")\n",
    "        print(f\"  Train samples: {len(ds['train']):,}\")\n",
    "        print(f\"  Test samples: {len(ds['test']):,}\")\n",
    "    except Exception as e:\n",
    "        print(f\"❌ Task 4 Non-SNV dataset validation failed: {e}\")\n",
    "else:\n",
    "    print(\"📝 Dataset validation skipped (upload disabled)\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}