Model save

README.md

@@ -1,517 +1,60 @@
 ---
-library_name: transformers
+library_name: peft
 license: mit
-language:
-  - en
-tags:
-  - medical
-  - clinical-nlp
-  - biobert
-  - bio-clinicalbert
-  - cancer
-  - survival-analysis
-  - oncology
-  - pathology
-  - tcga
-  - lora
-  - peft
-  - cox-regression
-  - risk-prediction
-  - text-classification
-  - feature-extraction
-  - pytorch
-datasets:
-  - custom
 base_model: emilyalsentzer/Bio_ClinicalBERT
-pipeline_tag: feature-extraction
+tags:
+- base_model:adapter:emilyalsentzer/Bio_ClinicalBERT
+- lora
+- transformers
 model-index:
-  - name: finetunePathologicalTextUsingBioBERT
-    results:
-      - task:
-          type: feature-extraction
-          name: Survival Risk Prediction
-        metrics:
-          - type: loss
-            name: Cox PH Validation Loss
-            value: 0.5290
-          - type: loss
-            name: Cox PH Training Loss
-            value: 0.4003
----
-
-# 🧬 Fine-Tuned Bio_ClinicalBERT for Cancer Survival Prediction from Pathological Text
-
-> **A domain-adapted biomedical language model fine-tuned on 19,637 TCGA pathological text reports for cancer survival risk prediction using Cox Proportional Hazards loss with LoRA adapters — trained on NVIDIA RTX 3090 (24 GB VRAM).**
-
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-[![PyTorch](https://img.shields.io/badge/PyTorch-2.6.0+cu124-red.svg)](https://pytorch.org/)
-[![Transformers](https://img.shields.io/badge/Transformers-5.7.0-orange.svg)](https://huggingface.co/docs/transformers)
-[![PEFT](https://img.shields.io/badge/PEFT-0.19.1-green.svg)](https://huggingface.co/docs/peft)
-[![GPU](https://img.shields.io/badge/GPU-RTX_3090_24GB-76B900.svg)](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090/)
-
+- name: finetunePathologicalTextUsingBioBERT
+  results: []
 ---
 
-## Model Details
-
-### Model Description
-
-This model is a **fine-tuned version of [Bio_ClinicalBERT](https://huggingface.co/emilyalsentzer/Bio_ClinicalBERT)** (Alsentzer et al., 2019) adapted for **cancer survival risk prediction** directly from unstructured pathological text reports. The model was trained on data from **The Cancer Genome Atlas (TCGA)** spanning **24 cancer types** across **32 cohorts**.
-
-Instead of traditional hand-crafted features (stage, grade, tumor size), this model **learns survival-relevant patterns directly from raw pathological text** — capturing subtle linguistic cues such as pathologist phrasing correlating with tumor aggressiveness, specific morphological descriptions, and diagnostic uncertainty language.
-
-The model outputs:
-1. **A continuous risk score** — higher values indicate higher mortality risk (used with Cox Proportional Hazards framework)
-2. **768-dimensional embeddings** — from the `[CLS]` token, suitable for downstream multimodal survival pipelines
-
-- **Developed by:** [Dr. Kareem Kamal](https://github.com/drkareemkamal)
-- **Model type:** BERT-based encoder with LoRA adapters + linear survival risk head
-- **Language(s):** English (clinical/biomedical)
-- **License:** MIT
-- **Fine-tuned from:** [emilyalsentzer/Bio_ClinicalBERT](https://huggingface.co/emilyalsentzer/Bio_ClinicalBERT)
-- **Base architecture:** BERT-Base (cased, 12-layer, 768-hidden, 12-attention-heads, ~110M parameters)
-
-### Model Sources
-
-- **Repository:** [github.com/drkareemkamal/cancer-survival-analysis](https://github.com/drkareemkamal/cancer-survival-analysis)
-- **Base model paper:** [Publicly Available Clinical BERT Embeddings (Alsentzer et al., NAACL 2019)](https://arxiv.org/abs/1904.03323)
-- **BioBERT paper:** [BioBERT: a pre-trained biomedical language representation model (Lee et al., 2020)](https://arxiv.org/abs/1901.08746)
-
----
-
-## About Bio_ClinicalBERT (Base Model)
-
-Bio_ClinicalBERT has a unique **three-stage pre-training lineage** that makes it ideal for clinical text understanding:
-
-| Stage | Training Data | Details |
-|-------|--------------|---------|
-| **1. BERT-Base** | Wikipedia + BookCorpus | General English language understanding |
-| **2. BioBERT v1.0** | PubMed abstracts (200K) + PMC full-text (270K) | Biomedical scientific literature |
-| **3. Bio_ClinicalBERT** | MIMIC-III clinical notes (~880M words) | Real electronic health records (EHR) |
-
-**Key specifications of the base model:**
-- **Architecture:** `cased_L-12_H-768_A-12` (12 layers, 768 hidden dim, 12 attention heads)
-- **Parameters:** ~110 million
-- **Vocabulary:** 28,996 WordPiece tokens (domain-adapted)
-- **Max sequence length:** 128 tokens (original); extended to **512 tokens** in our fine-tuning
-- **Original training:** 150,000 steps on GeForce GTX TITAN X (12 GB), batch size 32, LR 5e-5
-
-This lineage means the model understands:
-- ✅ General English grammar and semantics (BERT)
-- ✅ Biomedical terminology and relationships (BioBERT)
-- ✅ Clinical shorthand, abbreviations, and report structure (MIMIC-III)
-
----
-
-## Uses
-
-### Direct Use
-
-Load the fine-tuned model to extract survival-relevant embeddings or risk scores from pathological text:
-
-```python
-from transformers import AutoTokenizer, AutoModel
-import torch
-
-# Load model and tokenizer
-tokenizer = AutoTokenizer.from_pretrained("drkareemkamal/finetunePathologicalTextUsingBioBERT")
-model = AutoModel.from_pretrained("drkareemkamal/finetunePathologicalTextUsingBioBERT")
-model.eval()
-
-# Example pathological report text
-text = """Invasive ductal carcinoma, Nottingham grade 3/3. 
-Tumor size: 2.8 cm. ER negative, PR negative, HER2 positive (3+). 
-Lymphovascular invasion present. 2 of 14 sentinel lymph nodes positive 
-for metastatic carcinoma. Margins: negative, closest margin 0.3 cm."""
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
 
-# Tokenize
-inputs = tokenizer(
-    text,
-    return_tensors="pt",
-    max_length=512,
-    truncation=True,
-    padding=True
-)
+# finetunePathologicalTextUsingBioBERT
 
-# Extract [CLS] embedding (768-dim)
-with torch.no_grad():
-    outputs = model(**inputs)
-    cls_embedding = outputs.last_hidden_state[:, 0, :]  # Shape: (1, 768)
+This model is a fine-tuned version of [emilyalsentzer/Bio_ClinicalBERT](https://huggingface.co/emilyalsentzer/Bio_ClinicalBERT) on the None dataset.
 
-print(f"Embedding shape: {cls_embedding.shape}")  # torch.Size([1, 768])
-```
+## Model description
 
-### Downstream Use
+More information needed
 
-**Survival Risk Scoring** — Use with the custom risk head for direct risk prediction:
+## Intended uses & limitations
 
-```python
-import torch.nn as nn
+More information needed
 
-# Reconstruct the risk head (trained alongside the model)
-risk_head = nn.Linear(768, 1)
-# Load risk head weights from checkpoint if available
+## Training and evaluation data
 
-risk_score = risk_head(cls_embedding)
-print(f"Risk score: {risk_score.item():.4f}")
-# Higher score → higher predicted mortality risk
-```
+More information needed
 
-**Multimodal Fusion** — Combine text embeddings with clinical, genomic, and mutation data:
+## Training procedure
 
-```python
-# Text embedding: 768-dim from this model
-# Gene expression: 50-dim from PCA of RNA-Seq FPKM values
-# Mutation features: binary mutation matrix
-# Clinical features: age, stage, grade, etc.
+### Training hyperparameters
 
-combined = torch.cat([text_emb, gene_emb, mutation_emb, clinical_emb], dim=-1)
-# Feed into downstream survival model (e.g., DeepSurv, Cox-nnet)
-```
+The following hyperparameters were used during training:
+- learning_rate: 0.0005
+- train_batch_size: 8
+- eval_batch_size: 8
+- seed: 42
+- gradient_accumulation_steps: 2
+- total_train_batch_size: 16
+- optimizer: Use adamw_8bit with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
+- lr_scheduler_type: linear
+- lr_scheduler_warmup_steps: 500
+- training_steps: 10000
+- mixed_precision_training: Native AMP
 
-### Out-of-Scope Use
+### Training results
 
-- ❌ **Not a diagnostic tool** — This model predicts survival risk, not diagnosis
-- ❌ **Not for non-cancer text** — Trained exclusively on oncological pathology reports
-- ❌ **Not for clinical deployment without regulatory approval** — Research use only
-- ❌ **Not for non-English text** — Trained on English pathology reports only
-- ❌ **Not for individual patient decisions** — Requires human clinical oversight
-
----
-
-## Training Details
-
-### Training Data
-
-| Property | Value |
-|----------|-------|
-| **Source** | [The Cancer Genome Atlas (TCGA)](https://portal.gdc.cancer.gov/) via [cBioPortal](https://www.cbioportal.org/) |
-| **Dataset file** | `merged_tcga_data_final.csv` |
-| **Total samples** | **19,637** pathological text reports with survival outcomes |
-| **Train split** | 16,691 samples (85%) |
-| **Validation split** | 2,946 samples (15%) |
-| **Cancer types** | 24 disease types across 32 TCGA cohorts |
-| **Text column** | `text` — raw pathological report content |
-| **Survival endpoint** | Overall Survival: `OS_MONTHS` (time) + `OS_STATUS` (event: LIVING/DECEASED) |
-| **Event distribution** | ~70.7% Living / ~29.3% Deceased |
-
-**Cancer type distribution in training data:**
-
-| Disease Type | Samples | Deaths | Event Rate |
-|-------------|---------|--------|------------|
-| Adenomas and Adenocarcinomas | 8,977 | 1,944 | 21.7% |
-| Squamous Cell Neoplasms | 2,764 | 1,166 | 42.2% |
-| Ductal and Lobular Neoplasms | 2,362 | 498 | 21.1% |
-| Gliomas | 1,654 | 794 | 48.0% |
-| Cystic, Mucinous and Serous | 1,078 | 382 | 35.4% |
-| Transitional Cell Papillomas | 816 | 386 | 47.3% |
-| Others (18 types) | ~1,986 | varies | varies |
-
-### Training Procedure
-
-#### Preprocessing
-
-1. **Text cleaning:** Rows with missing `text`, `OS_MONTHS`, or `OS_STATUS` dropped
-2. **Survival labels:** `OS_STATUS` mapped to binary events (`1:DECEASED` → 1.0, `0:LIVING` → 0.0)
-3. **Tokenization:** WordPiece tokenizer from Bio_ClinicalBERT, `max_length=512`, right-truncation, `max_length` padding
-4. **No text augmentation** — raw pathological reports used as-is to preserve clinical accuracy
-
-#### Fine-Tuning Method: LoRA (Low-Rank Adaptation)
-
-Instead of updating all 110M parameters, we use **LoRA adapters** via the [PEFT library](https://github.com/huggingface/peft) to efficiently fine-tune only ~0.5% of parameters:
-
-| LoRA Parameter | Value |
-|---------------|-------|
-| **Rank (r)** | 8 |
-| **Alpha (α)** | 32 |
-| **Target modules** | `query`, `value` (attention layers) |
-| **Dropout** | 0.1 |
-| **Task type** | `FEATURE_EXTRACTION` |
-| **Trainable parameters** | ~590K (~0.5% of total) |
-
-#### Loss Function: Cox Proportional Hazards (Cox PH)
-
-The model is trained with the **negative partial log-likelihood of the Cox PH model**, which:
-- Handles **right-censored data** (patients still alive at last follow-up)
-- Models **relative hazard** — ranking patients by risk, not predicting absolute survival time
-- Is the gold standard for survival analysis in clinical research
-
-```
-L(β) = -Σ [log(h_i) - log(Σ exp(h_j))] × event_i
-        i                j∈R(t_i)
-```
-
-Where `h_i` is the predicted log-hazard for patient `i`, and `R(t_i)` is the risk set at time `t_i`.
-
-#### Training Hyperparameters
-
-| Hyperparameter | Value |
-|---------------|-------|
-| **Optimizer** | AdamW |
-| **Learning rate** | 1e-4 |
-| **Batch size** | 8 |
-| **Max epochs** | 20 |
-| **Early stopping patience** | 3 epochs |
-| **Validation split** | 15% (random, seed=42) |
-| **Precision** | FP32 (full precision) |
-| **Gradient clipping** | None |
-| **Scheduler** | None (constant LR) |
-| **Weight decay** | AdamW default (0.01) |
-
-#### Training Results
-
-📈 **Weights & Biases Dashboard:** [View Full Training Run & Loss Curves](https://wandb.ai/dr-kareem-kamal/cancer-survival-analysis/runs/bd7qqvhj)
-
-The model was trained for **all 20 epochs** (early stopping was not triggered, indicating continuous improvement):
-
-| Epoch | Train Loss | Val Loss | Best? |
-|-------|-----------|----------|-------|
-| 1 | 1.1658 | 0.9934 | |
-| 2 | 1.0408 | 0.9006 | |
-| 3 | 0.9440 | 0.8677 | |
-| 4 | 0.8720 | 0.8249 | |
-| 5 | 0.8122 | 0.7941 | |
-| 6 | 0.7347 | 0.7653 | |
-| 7 | 0.7011 | 0.7099 | |
-| 8 | 0.6649 | 0.7331 | |
-| 9 | 0.6167 | 0.6881 | |
-| 10 | 0.5849 | 0.6672 | |
-| 11 | 0.5562 | 0.6481 | |
-| 12 | 0.5424 | 0.6050 | |
-| 13 | 0.5150 | 0.6253 | |
-| 14 | 0.4998 | 0.6108 | |
-| 15 | 0.4705 | 0.5765 | |
-| 16 | 0.4630 | 0.6028 | |
-| 17 | 0.4347 | 0.5442 | |
-| 18 | 0.4230 | 0.5298 | |
-| 19 | 0.4104 | 0.5605 | |
-| **20** | **0.4003** | **0.5290** | **✅** |
-
-**Key observations:**
-- Consistent downward trend in both train and validation loss over 20 epochs
-- Best validation loss: **0.5290** at epoch 20
-- Final training loss: **0.4003**
-- No signs of catastrophic overfitting — the gap between train/val loss remains reasonable
-- Model checkpoint saved at epoch 20 (~415 MB)
-
-#### Speeds, Sizes, Times
-
-| Property | Value |
-|----------|-------|
-| **Total training time** | ~4.5 hours (20 epochs on RTX 3090) |
-| **VRAM usage** | ~3.8 GB (FP32, batch_size=8) |
-| **Checkpoint size** | 415 MB (full state dict with LoRA adapters + risk head) |
-| **Embeddings output** | 162 MB CSV (19,637 samples × 768 dimensions + risk scores) |
-| **Throughput** | ~120 samples/second (inference) |
-
----
 
-## Evaluation
 
-### Metrics
-
-| Metric | Description |
-|--------|-------------|
-| **Cox PH Loss** | Primary training objective — negative partial log-likelihood |
-| **C-index (Concordance Index)** | How well the model ranks patients by survival (0.5 = random, >0.7 = strong) |
-| **Kaplan-Meier Curves** | Visual separation between predicted high-risk and low-risk groups |
-| **Risk Score Distribution** | Separation of scores between alive vs deceased patients |
-
-### Results
-
-| Metric | Value |
-|--------|-------|
-| **Best Validation Cox PH Loss** | 0.5290 |
-| **Final Training Cox PH Loss** | 0.4003 |
-| **Total epochs trained** | 20 / 20 |
-| **Embedding dimension** | 768 |
-
-### Evaluation Outputs
-
-The following evaluation artifacts are generated during training:
-
-| File | Description |
-|------|-------------|
-| `clinicalbert_training_loss.png` | Train vs Validation loss curves with best epoch marked |
-| `clinicalbert_training_results.csv` | Per-epoch numerical loss values |
-| `finetuned_text_embeddings.csv` | 768-dim embeddings + risk scores for all 19,637 samples |
-
----
-
-## Technical Specifications
-
-### Model Architecture and Objective
-
-```
-Input: Raw pathological text (up to 512 tokens)
-  │
-  ▼
-┌─────────────────────────────────────────────┐
-│     Bio_ClinicalBERT (Frozen backbone)      │
-│     12 Transformer layers, 768 hidden dim   │
-│     + LoRA adapters on query/value (r=8)    │
-│     ~110M total params, ~590K trainable     │
-└──────────────────┬──────────────────────────┘
-                   │
-                   ▼
-          [CLS] Token Embedding (768-dim)
-                   │
-            ┌──────┴──────┐
-            ▼             ▼
-       Risk Head     Embeddings
-    (Linear 768→1)  (768-dim vector)
-            │             │
-            ▼             ▼
-     Cox PH Loss    Downstream Tasks
-```
-
-### Compute Infrastructure
-
-#### Hardware
-
-| Component | Specification |
-|-----------|--------------|
-| **GPU** | NVIDIA GeForce RTX 3090 |
-| **GPU Memory** | 24,576 MiB (24 GB GDDR6X) |
-| **CUDA Compute Capability** | 8.6 (Ampere architecture) |
-| **NVIDIA Driver** | 580.126.09 |
-| **CUDA Version** | 12.4 (PyTorch) / 13.0 (driver) |
-
-#### Software
-
-| Package | Version |
-|---------|---------|
-| **Python** | 3.10+ |
-| **PyTorch** | 2.6.0+cu124 |
-| **Transformers** | 5.7.0 |
-| **PEFT** | 0.19.1 |
-| **CUDA Toolkit** | 12.4 |
-| **OS** | Linux (Ubuntu) |
-| **Package Manager** | [uv](https://github.com/astral-sh/uv) |
-| **Experiment Tracking** | [Weights & Biases](https://wandb.ai/) |
-
-### How to Reproduce
-
-```bash
-# 1. Clone the repository
-git clone https://github.com/drkareemkamal/cancer-survival-analysis.git
-cd cancer-survival-analysis
-
-# 2. Set up environment with uv
-uv venv && source .venv/bin/activate
-uv sync
-
-# 3. Configure API keys in .env
-cat > .env << 'EOF'
-HF_TOKEN="hf_your_huggingface_token"
-HF_REPO_ID="your-username/your-repo-name"
-WANDB_API_KEY="your_wandb_api_key"
-WANDB_PROJECT="cancer-survival-analysis"
-EOF
-
-# 4. Run fine-tuning (baseline strategy)
-python src/training/text_finetune.py
-
-# Model will automatically push to HuggingFace Hub on completion
-```
-
----
-
-## Fine-Tuning Strategies Available
-
-This repository implements **three fine-tuning strategies**, each with both Bio_ClinicalBERT and OpenBioLLM-8B variants:
-
-### Strategy 1: Pan-Cancer Baseline (This Model)
-Single model trained on all 19,637 samples. Maximum data, simplest approach.
-```bash
-python src/training/text_finetune.py
-```
-
-### Strategy 2: Cancer-Type Conditioning Token
-Prepends a cancer-type tag to each text to enable cancer-aware representations:
-```
-Before: "Invasive ductal carcinoma, Nottingham grade 3..."
-After:  "[DUCTAL AND LOBULAR NEOPLASMS] Invasive ductal carcinoma..."
-```
-```bash
-python src/training/text_finetune_conditioned.py
-```
-
-### Strategy 3: Hierarchical Two-Stage
-Stage 1 trains on all cancers, Stage 2 fine-tunes per cancer type (500+ samples):
-```bash
-python src/training/text_finetune_hierarchical.py
-```
-
----
-
-## Bias, Risks, and Limitations
-
-### Dataset Bias
-- **Geographic bias:** TCGA data originates from US academic medical centers, which may not represent global patient populations
-- **Demographic bias:** The cohort reflects the demographics of TCGA participants and may underrepresent certain racial/ethnic groups
-- **Institutional bias:** Pathology report styles vary by institution; model performance may degrade on reports with different formatting conventions
-
-### Clinical Limitations
-- **Not a diagnostic tool** — predicts survival risk only, not disease diagnosis
-- **Text quality dependency** — performance is directly tied to report completeness and detail
-- **No external validation** — requires independent cohort validation before any clinical consideration
-- **Censoring assumptions** — Cox PH model assumes non-informative censoring, which may not always hold
-
-### Technical Limitations
-- **Max 512 tokens** — longer reports are truncated from the right, potentially losing relevant information
-- **Single-modality** — text-only; does not incorporate imaging, genomics, or structured clinical variables (see multimodal pipeline in repository)
-- **FP32 only** — not optimized for mixed-precision inference
-
-### Recommendations
-
-- **Always pair with clinical judgment** — this model is a decision-support tool, not a replacement for clinical expertise
-- **Validate on your institution's data** before use — report styles differ across institutions
-- **Monitor for bias** — regularly audit predictions across demographics, cancer types, and institutions
-- **Regulatory compliance** — any clinical deployment requires appropriate regulatory approval (e.g., FDA, CE marking)
-
----
-
-## Citation
-
-**BibTeX:**
-
-```bibtex
-@software{kamal2026cancer_survival_biobert,
-  title={Cancer Survival Prediction from Pathological Text Reports using Fine-Tuned Bio_ClinicalBERT},
-  author={Kareem Kamal},
-  year={2026},
-  url={https://huggingface.co/drkareemkamal/finetunePathologicalTextUsingBioBERT},
-  note={Fine-tuned on TCGA pathological reports with Cox PH loss and LoRA adapters, trained on NVIDIA RTX 3090}
-}
-```
-
-**APA:**
-
-Kamal, K. (2026). *Cancer Survival Prediction from Pathological Text Reports using Fine-Tuned Bio_ClinicalBERT* [Computer software]. Hugging Face. https://huggingface.co/drkareemkamal/finetunePathologicalTextUsingBioBERT
-
----
-
-## References
-
-1. **Bio_ClinicalBERT:** Alsentzer, E., et al. (2019). *Publicly Available Clinical BERT Embeddings.* NAACL Clinical NLP Workshop. [HuggingFace](https://huggingface.co/emilyalsentzer/Bio_ClinicalBERT) | [Paper](https://arxiv.org/abs/1904.03323)
-2. **BioBERT:** Lee, J., et al. (2020). *BioBERT: a pre-trained biomedical language representation model for biomedical text mining.* Bioinformatics, 36(4), 1234–1240. [Paper](https://arxiv.org/abs/1901.08746)
-3. **TCGA:** The Cancer Genome Atlas Research Network. [GDC Data Portal](https://portal.gdc.cancer.gov/)
-4. **cBioPortal:** Cerami, E., et al. (2012). *The cBio Cancer Genomics Portal.* Cancer Discovery, 2(5), 401–404. [Website](https://www.cbioportal.org/)
-5. **Cox PH Model:** Cox, D.R. (1972). *Regression Models and Life-Tables.* Journal of the Royal Statistical Society, Series B, 34(2), 187–220.
-6. **LoRA:** Hu, E., et al. (2022). *LoRA: Low-Rank Adaptation of Large Language Models.* ICLR 2022. [Paper](https://arxiv.org/abs/2106.09685)
-7. **PEFT:** HuggingFace. *Parameter-Efficient Fine-Tuning.* [GitHub](https://github.com/huggingface/peft)
-
----
-
-## Model Card Authors
-
-- **Dr. Kareem Kamal** — [@drkareemkamal](https://github.com/drkareemkamal)
-
-## Model Card Contact
-
-- **GitHub:** [github.com/drkareemkamal](https://github.com/drkareemkamal)
-- **HuggingFace:** [huggingface.co/drkareemkamal](https://huggingface.co/drkareemkamal)
-
----
+### Framework versions
 
-*This model is for research purposes only. Always consult qualified medical professionals for clinical decisions. Not approved for clinical use.*
+- PEFT 0.19.1
+- Transformers 5.7.0
+- Pytorch 2.6.0+cu124
+- Datasets 4.8.5
+- Tokenizers 0.22.2