README.md

---
license: gemma
base_model: unsloth/gemma-4-E2B-it-unsloth-bnb-4bit
tags:
  - drug-discovery
  - chemistry
  - smiles
  - lora
  - unsloth
  - gemma4
  - biology
  - fine-tuned
  - healthcare
language:
  - en
library_name: peft
pipeline_tag: text-generation
---

# Gemma-Cure — Drug Discovery LoRA Adapter

**Gemma 4 E2B fine-tuned on 225K drug–target pairs for novel small-molecule generation.**

Built for the [Kaggle Gemma 4 Good Hackathon 2026](https://www.kaggle.com/competitions/gemma-4-good-hackathon) by **DLYog Lab**.

---

## Model Description

Gemma-Cure takes a protein target name, amino-acid sequence, and measured binding affinity, then generates:
1. A structured **scientific rationale** (binding pocket analysis, key residues, physicochemical reasoning)
2. A **novel SMILES string** for a drug-like small molecule

The model is designed for educational drug discovery — explaining reasoning in plain English accessible to early researchers and high school students.

### Live Demo

> **Try it now:** [Deep2Lead Platform](https://deep2lead.dlyog.com) — PathoHunt 3D game uses this model live

---

## Training Details

| Parameter | Value |
|---|---|
| Base model | `unsloth/gemma-4-E2B-it-unsloth-bnb-4bit` |
| Architecture | Gemma 4 E2B (5.2B parameters) |
| Adapter type | RS-LoRA (Rank-Stabilised LoRA) |
| LoRA rank (r) | 32 |
| LoRA alpha | 64 |
| Trainable params | 62M / 5.2B (1.2%) |
| Training dataset | 225,000 drug–target binding pairs |
| Data sources | BindingDB, ChEMBL, MOSES |
| Training framework | Unsloth + HuggingFace TRL SFTTrainer |
| Hardware | NVIDIA GB10 Grace Blackwell (122GB VRAM) |
| Training time | ~3 hours total (iterative runs) |
| Final LR | 2e-5 (cosine scheduler) |
| Batch size | 8 × 16 gradient accumulation = 128 effective |
| Quantization | 4-bit (BnB NF4) |

### Auto-Evaluation Loop

Training used a custom `EvalAndStopCallback` that evaluates drug quality every 100 steps on 3 benchmark targets and stops automatically when a pharmaceutical quality gate passes:
- **Validity gate:** ≥ 67% of generated SMILES must be chemically valid (RDKit)
- **QED gate:** Average QED (Quantitative Estimate of Drug-likeness) ≥ 0.55

---

## Evaluation Results

Evaluated on 3 benchmark drug targets using RDKit SMILES validation and QED scoring:

| Target | SMILES | QED |
|---|---|---|
| SARS-CoV-2 Main Protease | `O=C(O)c1ccc(-n2cc(Nc3ccccc3)cn2)o1` | 0.761 |
| EGFR Kinase | `CN(C)c1ccc(-n2cc(NC(=O)[C@](Cc3ccccc3)N=O)nc2OC)cn1` | 0.591 |
| BACE1 Alzheimer target | `CN(C)c1ccc(-n2cc(N[C@@H]3CC4CCN(CCc5ccccc5Cl)CC4CCC3)nc2O)[nH]1` | 0.421 |

| Metric | Base Gemma 4 | v2 baseline | **Gemma-Cure (final)** |
|---|---|---|---|
| SMILES validity | 0% | 33% | **100%** |
| Average QED | 0.000 | 0.116 | **0.591** |
| Composite score | 0.000 | 0.224 | **0.795** |

Quality gate passed at step 100 of run 2 (41 minutes of training).

---

## How to Load

### Requirements

```bash
pip install unsloth
```

### Inference

```python
from unsloth import FastModel

model, processor = FastModel.from_pretrained(
    model_name="dlyog/gemma-cure",   # downloads base + adapter automatically
    load_in_4bit=True,
    max_seq_length=2048,
)
FastModel.for_inference(model)

SYSTEM = (
    "You are Deep2Lead's drug discovery AI v2. When given a protein target or biological "
    "context, first reason about the binding pocket geometry, key residues, and desired "
    "physicochemical profile (2-3 sentences), then output novel drug-like SMILES molecules. "
    "Always label your reasoning as 'Rationale:' and your molecules as 'SMILES:'. Explain "
    "choices in plain English suitable for high school students and early researchers. "
    "Prioritize selectivity, low toxicity, and synthetic accessibility."
)

messages = [
    {"role": "system", "content": [{"type": "text", "text": SYSTEM}]},
    {"role": "user",   "content": [{"type": "text", "text": (
        "Target: EGFR Kinase\n"
        "Protein sequence (first 150 AA): MRPSGTAGAALLALLAALCPASRALEEKKVCQGTSNKLTQLGTFEDHFLSLQ"
        "RMFNNCEVVLGNLEITYVQRNYDLSFLKTIQEVAGYVLIALNTVERIPLENLQIIRGNMYYENSYALAVLSNYDANKTGLKELPMRNLQEILHGAVR\n"
        "Measured binding affinity: IC50 = 2.0 nM\n"
        "Design a small molecule drug candidate with high binding affinity to this target. "
        "First explain your structural reasoning, then provide the SMILES.\n"
        "Requirements: MW 200-500 Da, QED > 0.50, SAS <= 5.0, Lipinski Ro5 compliant."
    )}]},
]

inputs = processor.apply_chat_template(
    messages, tokenize=True, return_dict=True,
    return_tensors="pt", add_generation_prompt=True,
).to("cuda")

with __import__("torch").no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=350,
        temperature=0.9,
        top_p=0.92,
        top_k=50,
        repetition_penalty=1.3,
        do_sample=True,
    )

response = processor.decode(
    outputs[0][inputs["input_ids"].shape[1]:],
    skip_special_tokens=True
)
print(response)
```

### Expected Output Format

```
Rationale: The EGFR kinase active site contains a conserved ATP-binding hinge region
with Cys797 as a key covalent anchor point. A pyrimidine-aniline scaffold provides
optimal hinge binding geometry while maintaining MW within the 200-500 Da window
and favorable LogP for cell penetration.

SMILES: CN(C)c1ccc(-c2nc(Nc3ccccc3F)c(C#N)cn2)cc1
```

---

## Prompt Format

The model expects this exact format (use `processor.apply_chat_template`):

```
System: You are Deep2Lead's drug discovery AI v2...

User:
Target: <target name>
Protein sequence (first 150 AA): <sequence>
Measured binding affinity: <Ki/IC50/Kd value>
Design a small molecule drug candidate...
Requirements: MW 200-500 Da, QED > 0.50, SAS ≤ 5.0, Lipinski Ro5 compliant.
```

---

## Dataset

Training data: 225,000 drug–target binding pairs curated from:

| Source | Records | Description |
|---|---|---|
| [BindingDB](https://www.bindingdb.org) | ~150K | Experimental binding affinities (Ki, IC50, Kd) |
| [ChEMBL](https://www.ebi.ac.uk/chembl/) | ~50K | Bioactive molecules with target annotations |
| [MOSES](https://github.com/molecularsets/moses) | ~25K | Drug-like SMILES diversity scaffold |

Each record: `{target_name, protein_sequence_150AA, binding_affinity, smiles, rationale}`

---

## Technical Notes

- **Why RS-LoRA:** Uses α/√r scaling (vs standard α/r), stabilising gradients across different rank sizes. With r=32 and α=64, the effective scale is α/√r ≈ 11.3 vs standard α/r = 2.0 — higher signal without the instability of large α.
- **Why Unsloth:** 2× faster training, 80% less VRAM via custom CUDA kernels and gradient checkpointing optimisations. Enables 4-bit training on the full 5.2B model with only 62M trainable LoRA params.
- **repetition_penalty=1.3:** Critical for SMILES generation — without it the model loops on fragments like `c1c1c1c1...`. Matches deployment params.
- **TRANSFORMERS_OFFLINE=1:** Set during training to use local HF cache and prevent mid-training download attempts.

---

## Citation

```bibtex
@misc{gemma-cure-2026,
  title        = {Gemma-Cure: Drug Discovery LoRA Adapter for Gemma 4 E2B},
  author       = {Tarun Kumar Chawdhury},
  year         = {2026},
  howpublished = {HuggingFace Model Hub},
  url          = {https://huggingface.co/dlyog/gemma-cure},
  note         = {Fine-tuned on 225K drug-target pairs. Kaggle Gemma 4 Good Hackathon 2026.}
}
```

---

## License

This adapter is released under the [Gemma Terms of Use](https://ai.google.dev/gemma/terms). The base model weights remain subject to the original Gemma license.