Update README.md

README.md

@@ -12,87 +12,307 @@ tags:
 - pharma
 - medical
 - domain-specific
+- rope
+- rmsnorm
+- swiglu
+- synthetic-data
 pipeline_tag: text-generation
+model-index:
+- name: PharmaGPT-336M
+  results:
+  - task:
+      type: text-generation
+      name: Text Generation
+    metrics:
+    - name: Validation Loss
+      type: loss
+      value: 0.3748
+    - name: Perplexity
+      type: perplexity
+      value: 1.45
+    - name: Training Loss
+      type: loss
+      value: 0.3748
 ---
 
 # PharmaGPT-336M
 
-A **336M parameter GPT** language model trained **entirely from scratch** on 200K synthetic pharmaceutical documents across 6 domains.
+A **336M parameter GPT** language model trained **entirely from scratch** on 200K synthetic pharmaceutical documents across 6 manufacturing domains.
 
-No pre-trained weights, no fine-tuning — every component built from scratch: custom BPE tokenizer, full transformer architecture, training loop.
+**No pre-trained weights. No fine-tuning. Every component built from scratch**: custom BPE tokenizer, full transformer architecture (RoPE + RMSNorm + SwiGLU), training loop, and evaluation pipeline.
 
-## Model Details
+> **Paper**: [ArXiv preprint (coming soon)]()  
+> **Blog**: [Medium article (coming soon)]()  
+> **Code**: Included in this repository
 
-| Property | Value |
-|---|---|
-| Parameters | 336,380,928 (336.4M) |
-| Architecture | Decoder-only Transformer (GPT) |
-| Embedding Dim | 1024 |
-| Attention Heads | 16 |
-| Layers | 24 |
-| Context Length | 1024 |
-| Vocabulary | 32,000 tokens (custom BPE) |
-| Positional Encoding | Rotary (RoPE) |
-| Normalization | RMSNorm |
-| Activation | SwiGLU |
-| Val Loss | 0.4985 |
-| Perplexity | 1.65 |
-
-## Pharma Domains Covered
-
-1. **Manufacturing Deviation Reports** — equipment failures, process excursions, root cause analysis, CAPA
-2. **Batch Production Records** — raw materials, process steps, yield, disposition
-3. **SOP Q&A** — cleaning validation, environmental monitoring, aseptic processing, water systems
-4. **Stability Studies** — ICH Q1A conditions, assay trending, shelf life prediction
-5. **Pharmacovigilance (ICSR)** — adverse event narratives, WHO-UMC causality assessment
-6. **Scientific Writing** — formulation development, DoE, analytical methods, results sections
-
-## Architecture
-
-Uses modern architectural choices from Llama / Mistral:
-- **RoPE** (Rotary Positional Embeddings) — encodes relative position
-- **RMSNorm** — faster, simpler alternative to LayerNorm
-- **SwiGLU** — gated activation function
-- **No bias** in linear layers
-- **Weight tying** between token embedding and output projection
-
-## Training
-
-- **Data**: 200K synthetic pharmaceutical samples (~32M tokens) generated with domain-specific templates
-- **Tokenizer**: Custom BPE (32K vocab) trained on the pharma corpus
-- **Optimizer**: AdamW (b1=0.9, b2=0.95, weight decay=0.1)
-- **Schedule**: Cosine LR decay with 500-step linear warmup
-- **Hardware**: Kaggle T4 GPU, 15K iterations across multiple sessions
-- **Precision**: float16 mixed precision with gradient checkpointing
-
-## Usage
+---
+
+## Quick Start
 
 ```python
 import torch
 from tokenizers import Tokenizer
-from model import GPT, GPTConfig
 
+# Download model files from this repo, then:
 ckpt = torch.load("best_model.pt", map_location="cpu", weights_only=False)
+
+# Reconstruct model from saved config
+from model import GPT  # model.py included in this repo
 model = GPT(ckpt["model_config"])
 model.load_state_dict(ckpt["model"])
 model.eval()
 
+# Load tokenizer
 tok = Tokenizer.from_file("tokenizer/tokenizer.json")
 
+# Generate pharmaceutical text
 prompt = "<|deviation|>\nDuring manufacturing of Batch B-NDL-2026"
 ids = torch.tensor([tok.encode(prompt).ids])
-out = model.generate(ids, max_new_tokens=200, temperature=0.8, top_k=50)
-print(tok.decode(out[0].tolist()))
+output = model.generate(ids, max_new_tokens=200, temperature=0.8, top_k=50)
+print(tok.decode(output[0].tolist()))
+```
+
+### Generation with Different Domains
+
+```python
+prompts = {
+    "deviation": "<|deviation|>\nDuring routine inspection of the tablet coating line",
+    "batch_record": "<|batch_record|>\nBATCH PRODUCTION RECORD\nProduct: Metformin HCl 500mg Tablets",
+    "sop": "<|sop|>\nSOP-ENV-205 | Environmental Monitoring Program",
+    "stability": "<|stability_study|>\nSTABILITY STUDY REPORT\nProduct: Adalimumab 40mg/0.8mL",
+    "pharmacovigilance": "<|icsr|>\nA 72-year-old female patient with history of diabetes",
+    "scientific": "<|scientific_paper|>\nObjective: To evaluate the impact of granulation",
+}
+
+for domain, prompt in prompts.items():
+    ids = torch.tensor([tok.encode(prompt).ids])
+    out = model.generate(ids, max_new_tokens=150, temperature=0.8, top_k=50)
+    print(f"\n{'='*60}\n[{domain.upper()}]\n{'='*60}")
+    print(tok.decode(out[0].tolist()))
+```
+
+---
+
+## Model Details
+
+| Property | Value |
+|---|---|
+| **Parameters** | 336,380,928 (336M) |
+| **Architecture** | Decoder-only Transformer (GPT) |
+| **Embedding Dimension** | 1024 |
+| **Attention Heads** | 16 |
+| **Layers** | 24 |
+| **Context Length** | 512 tokens |
+| **Vocabulary** | 32,000 tokens (custom BPE) |
+| **Positional Encoding** | Rotary (RoPE), base=10000 |
+| **Normalization** | RMSNorm (ε=1e-6) |
+| **Activation** | SwiGLU (FFN hidden=2752) |
+| **Bias** | None (all linear layers) |
+| **Weight Tying** | Embedding ↔ LM Head |
+| **Dropout** | 0.1 |
+
+### Architecture Highlights
+
+This model implements the same architectural innovations found in LLaMA/Mistral, all coded from scratch:
+
+- **RoPE** (Rotary Positional Embeddings) — encodes relative position through rotation of Q/K vectors
+- **RMSNorm** — faster, simpler alternative to LayerNorm (no mean subtraction)
+- **SwiGLU** — gated feed-forward network with Swish activation
+- **No bias** in any linear layer — modern simplification
+- **Weight tying** — token embedding and output projection share parameters
+- **Pre-norm** architecture — normalize before attention/FFN, not after
+
+---
+
+## Training Details
+
+| Parameter | Value |
+|---|---|
+| **Optimizer** | AdamW (β₁=0.9, β₂=0.95, wd=0.1) |
+| **Learning Rate** | 2e-4 (peak), cosine decay |
+| **Warmup** | 500 steps (linear) |
+| **Batch Size** | 8 micro × 4 grad accum = 32 effective |
+| **Iterations** | 15,000 |
+| **Precision** | float16 mixed precision |
+| **Gradient Clipping** | 1.0 (global norm) |
+| **Gradient Checkpointing** | Enabled |
+| **Hardware** | NVIDIA T4 (16GB), Kaggle free tier |
+| **Training Time** | ~9 hours |
+| **Cost** | $0 (free compute) |
+
+### Training Results
+
+| Metric | Value |
+|---|---|
+| Final Training Loss | 0.3748 |
+| Best Validation Loss | 0.3748 |
+| Validation Perplexity | 1.45 |
+| Tokens Processed | ~245M |
+
+*Note: Low perplexity reflects the structured/templated nature of synthetic training data. Real-world pharmaceutical text would yield higher perplexity.*
+
+---
+
+## Training Data: 6 Pharmaceutical Domains
+
+The model was trained on 200K synthetic documents (~32M tokens) generated across six pharmaceutical manufacturing domains:
+
+### 1. Manufacturing Deviation Reports (~33K samples)
+Equipment failures, process excursions, out-of-specification results, root cause analysis (Ishikawa, 5-Why), CAPA documentation following ICH Q10.
+
+### 2. Batch Production Records (~33K samples)
+Raw material dispensing, process step documentation, in-process controls, critical process parameters (CPPs), yield calculations, lot disposition decisions.
+
+### 3. Standard Operating Procedures (~33K samples)
+Cleaning validation, environmental monitoring, aseptic processing, water system maintenance (WFI, PW), equipment qualification — in Q&A format.
+
+### 4. Stability Studies (~33K samples)
+ICH Q1A(R2) study designs, accelerated (40°C/75% RH) and long-term (25°C/60% RH) conditions, assay trending, degradation products, shelf-life determination.
+
+### 5. Pharmacovigilance Case Reports (~33K samples)
+Individual Case Safety Reports (ICSRs), adverse event narratives, MedDRA coding, WHO-UMC causality assessment (certain/probable/possible/unlikely).
+
+### 6. Scientific Writing (~33K samples)
+Formulation development, Design of Experiments (DoE), analytical method development/validation, dissolution studies, results and discussion sections.
+
+---
+
+## Special Tokens
+
+| Token | Purpose | Example Use |
+|---|---|---|
+| `<\|deviation\|>` | Start of deviation report | Triggers investigation-style generation |
+| `<\|batch_record\|>` | Start of batch record | Triggers manufacturing record format |
+| `<\|sop\|>` | Start of SOP document | Triggers procedural/Q&A format |
+| `<\|stability_study\|>` | Start of stability study | Triggers ICH-compliant study format |
+| `<\|icsr\|>` | Start of pharmacovigilance case | Triggers adverse event narrative |
+| `<\|scientific_paper\|>` | Start of scientific writing | Triggers academic/research style |
+| `<\|end\|>` | End of document | Marks document boundary |
+
+---
+
+## Repository Contents
+
+```
+├── best_model.pt          # Full checkpoint (model weights + config + metadata)
+├── config.json            # Architecture specification (JSON)
+├── tokenizer/
+│   └── tokenizer.json     # Trained BPE tokenizer (32K vocab)
+├── model.py               # Complete model source code (GPT + all components)
+├── tokenizer.py           # Tokenizer training/loading utilities
+└── README.md              # This file
 ```
 
-## Limitations
+### Loading Without `model.py`
+
+If you want to inspect the architecture without running the custom code:
+
+```python
+import torch, json
+
+# Load config
+with open("config.json") as f:
+    config = json.load(f)
+print(config)
+# {'vocab_size': 32000, 'n_embd': 1024, 'n_head': 16, 'n_layer': 24, ...}
+
+# Load checkpoint metadata
+ckpt = torch.load("best_model.pt", map_location="cpu", weights_only=False)
+print(f"Keys: {ckpt.keys()}")
+print(f"Val loss: {ckpt.get('best_val_loss')}")
+print(f"Iteration: {ckpt.get('iter_num')}")
+```
 
-- Trained on **synthetic** data only — does not contain real pharmaceutical knowledge
-- Not suitable for clinical or regulatory decision-making
-- 336M parameters — small model, not comparable to large LLMs
-- May generate plausible-sounding but factually incorrect pharmaceutical content
-- English only
+---
+
+## Intended Use
+
+### Primary Use Cases
+- **Educational**: Understanding how modern GPT architectures work end-to-end
+- **Research baseline**: Starting point for pharmaceutical NLP research
+- **Template generation**: Generating draft pharmaceutical document structures
+- **Domain adaptation**: Fine-tuning on real pharmaceutical data for production use
+
+### Out-of-Scope Uses
+- **Clinical decision-making**: This model generates plausible but NOT factually verified content
+- **Regulatory submissions**: Generated text requires expert review and verification
+- **Production deployment without validation**: The model was trained on synthetic data only
+- **General-purpose chat**: This is a domain-specific completion model, not a chatbot
+
+---
+
+## Limitations and Risks
+
+| Limitation | Impact | Mitigation |
+|---|---|---|
+| Synthetic training data | May generate structurally correct but factually wrong content | Always verify with domain experts |
+| 336M parameters | Limited reasoning and knowledge capacity | Use as starting point, not final solution |
+| English only | Cannot process multilingual pharmaceutical docs | Extend training data for other languages |
+| No instruction tuning | Cannot follow complex instructions | Fine-tune with instruction data |
+| Context length (512) | Cannot process long documents in one pass | Chunk documents or extend context |
+
+### Ethical Considerations
+- Generated pharmaceutical content should NEVER be used for actual drug manufacturing without expert review
+- The model may reproduce biases present in the synthetic data templates
+- Not intended as a replacement for qualified pharmaceutical professionals
+
+---
+
+## Citation
+
+If you use PharmaGPT in your research, please cite:
+
+```bibtex
+@misc{chaturvedi2026pharmagpt,
+  title={PharmaGPT: A Domain-Specific Language Model for Pharmaceutical Manufacturing Intelligence Trained from Scratch on Synthetic Data},
+  author={Chaturvedi, Parth},
+  year={2026},
+  howpublished={\url{https://huggingface.co/ParthChat1802/PharmaGPT-336M}},
+}
+```
+
+---
+
+## Technical Notes for Reproducibility
+
+### Checkpoint Format
+
+The `best_model.pt` file is a PyTorch checkpoint dictionary containing:
+
+```python
+{
+    "model": OrderedDict,        # model.state_dict()
+    "model_config": GPTConfig,   # dataclass with architecture params
+    "config": dict,              # training configuration
+    "iter_num": int,             # iteration at save time
+    "best_val_loss": float,      # best validation loss achieved
+}
+```
+
+### System Requirements
+
+- **Inference**: Any machine with 2GB+ RAM and PyTorch installed
+- **Training (reproduce)**: NVIDIA GPU with 8GB+ VRAM, or Apple M-series with 16GB+ unified memory
+- **Dependencies**: `torch>=2.0`, `tokenizers>=0.13`
+
+### Reproducing Training
+
+```bash
+git clone <source-repo>
+cd gpt-from-scratch
+pip install -r requirements.txt
+
+# Generate data
+python -m data.generators.master_generator
+
+# Train tokenizer + model
+python -m src.train_pharma
+```
+
+Or use the Kaggle notebook for GPU-accelerated training (see repository).
+
+---
 
 ## License
 
-Apache 2.0
+**Apache 2.0** — Use freely for any purpose (commercial, research, educational). Attribution appreciated but not legally required beyond the license notice.