---
title: Multimodal Skin Lesion Explainability
colorFrom: blue
colorTo: indigo
sdk: docker
app_port: 7860
startup_duration_timeout: 1h
license: mit
---

# 🔬 RG-DermNet: Multimodal Skin Lesion Explainability

A Gradio-based web application for multimodal skin lesion analysis with **GradCAM++ visualization**. This tool enables clinicians and researchers to understand how deep learning models make predictions on dermoscopic images by combining image data with clinical metadata.
## 📄 Abstract

Skin cancer accounts for nearly one-third of all diagnosed tumors worldwide, making early and accurate recognition critical for improving patient outcomes. In this work, we propose RG-DermNet, a multimodal deep learning framework that integrates skin lesion images with structured clinical metadata through a residual gated-attention (RG-ATT) fusion mechanism. The architecture combines CNN- and Transformer-based visual backbones with a lightweight one-hot encoding pipeline for metadata, enabling effective cross-modal interaction. The proposed model is evaluated using a patient-wise cross-validation protocol across four dermatological datasets with heterogeneous metadata. On PAD-UFES-20, using Caformer-B36 as the visual backbone, RG-DermNet achieves an accuracy of 0.75 ± 0.05, balanced accuracy of 0.78 ± 0.03, F1-score of 0.77 ± 0.04, and AUC of 0.95 ± 0.01, outperforming existing multimodal baselines under the same evaluation setting. In addition, a SHAP-based analysis provides insights into the contribution of clinical metadata to the model’s predictions, supporting both performance gains and interpretability.

## 🎯 Features

- **Dermoscopic Image Upload**: Load and analyze dermoscopic skin lesion images
- **Metadata Management**: Organize clinical information into specific metadata groups:
  - Demographics (age, gender, lesion location)
  - Clinical History
  - Symptoms (itch, growth, bleeding, elevation, etc.)
  - Lesion Geometry (diameter measurements)
- **Live Metadata Preview**: Real-time CSV generation showing exact model inputs
- **GradCAM++ Attention Maps**: Visualize where the model focuses its attention on the image
- **Multi-Model Support**: Choose between different attention mechanisms (concatenation, metadata blocks, cross-attention, etc.)
- **Prediction Summary**: Get classification results with confidence scores

## 📊 Model Information

This application includes pre-trained models using:
- **CNN Backbone**: ResNet-50
- **Attention Mechanisms**: Multiple architectures including:
  - No-metadata baseline
  - Concatenation-based fusion
  - Cross-attention modules
  - Metadata blocks

Models are trained on the **PAD-UFES-20** skin lesion dataset and classify into 6 categories:
- **NEV**: Nevus
- **BCC**: Basal Cell Carcinoma  
- **ACK**: Actinic Keratosis
- **SEK**: Seborrheic Keratosis
- **SCC**: Squamous Cell Carcinoma
- **MEL**: Melanoma

## 🚀 Quick Start

### Local Installation

**Requirements:**
- Python 3.10+
- PyTorch with CUDA support (optional but recommended)

**Setup:**
```bash
# Clone the repository
git clone <this-repo>
cd GradCAMPlusPlus_SkinLesion

# Create virtual environment (recommended)
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Run the application
python app.py
```

The application will start a local Gradio server. Open the provided URL in your browser.

### HuggingFace Spaces Deployment

This repository is configured to deploy directly to [HuggingFace Spaces](https://huggingface.co/spaces).

**To deploy your own instance:**
1. Fork this repository to your HuggingFace account
2. Create a new Space using this repository
3. The app will automatically build and launch

**Direct link to this Space:** [Coming soon - your HF Space URL]

## 📖 Usage Guide

1. **Upload an Image**: Click "Upload" and select a dermoscopic image file (PNG, JPG, etc.)

2. **Select Metadata Groups**: Check which metadata categories are relevant:
   - ✅ Demographics: Always available
   - ✅ Clinical History: Optional
   - ✅ Symptoms: Optional  
   - ✅ Lesion Geometry: Optional

3. **Fill Patient Information**:
   - Age: Patient's age in years
   - Gender: Male/Female
   - Region: Lesion location (HEAD, NECK, BACK, ARM, LEG, TORSO)
   - Diameter: Measure in two perpendicular directions

4. **Select Model**: Choose an attention mechanism from the dropdown:
   - Different models may perform differently on your image
   - Try multiple models to understand variations

5. **Generate Analysis**: Click "Generate GradCAM++" to:
   - Run inference on the image and metadata
   - Generate attention heatmap overlay
   - Display confidence scores and classification

6. **Review Results**:
   - Original Lesion: Your uploaded image
   - Attention Map: Where the model focused (red = high focus)
   - Metadata Details: View the exact CSV format sent to the model

## 📁 Project Structure

```
GradCAMPlusPlus_SkinLesion/
├── app.py                          # Entry point for HF Spaces
├── requirements.txt                 # Python dependencies
├── README.md                        # This file
├── .gitattributes                   # Git LFS configuration
├── .gitignore                       # Git ignore rules
├── data/
│   ├── weights/TO_BE_USED/         # Pre-trained model weights
│   │   ├── concatenation/
│   │   ├── metablock/
│   │   ├── no-metadata/
│   │   └── att-intramodal+residual+cross-attention-metadados/
│   └── preprocess_data/             # Encoders and scalers
│       ├── label_encoder_pad_20.pickle
│       ├── ohe_pad_20.pickle
│       └── scaler_pad_20.pickle
└── src/
    ├── main.py                      # Gradio UI and event handling
    ├── utils/
    │   ├── load_local_variables.py
    │   └── transforms.py
    └── models/
        ├── __init__.py
        ├── inference.py             # Model inference pipeline
        ├── model_loader.py          # PyTorch model loading
        ├── cam.py                   # GradCAM++ implementation
        ├── preprocessing.py         # Image & metadata preprocessing
        ├── loadImageModelClassifier.py
        ├── metadata_builder.py      # CSV generation
        ├── metadata_groups.py       # Metadata schema
        ├── metadata_schema.py       # Column definitions
        └── ... (other model architecture files)
```

## 🛠️ Technical Details

### Input Processing
- **Images**: Normalized using ImageNet statistics, resized for model input
- **Metadata**: One-hot encoded and padded to 20 dimensions

### Attention Visualization
- **GradCAM++**: Computes class activation maps using gradient averaging
- **Overlay**: Jet colormap (blue=low importance, red=high importance)

### Model Input Format
- Metadata is sent as CSV with specific column order
- Columns are enabled/disabled based on selected groups
- Empty values represented as empty strings

## ⚠️ Important Notes

- **Pre-condition**: Model weights must be present in `data/weights/TO_BE_USED/` for inference to work
- **Missing Weights**: If models fail to load, download from the original repository
- **Not for Clinical Use**: This tool is for **research and education only**. Do not use for clinical diagnosis.
- **GPU Recommended**: Faster inference with CUDA. Falls back to CPU if unavailable.

## 🔗 References & Acknowledgments
- **Paper repository**: RG-DermNet (https://github.com/wyctorfogos/rg-dermnet)
- **Dataset**: PAD-UFES-20 (Universidade Federal do Espírito Santo)
- **GradCAM++**: [Paper](https://arxiv.org/abs/1710.11063) by Chattopadhyay et al.
- **Framework**: [Gradio](https://gradio.app) for the web interface

## 📄 License

This project is licensed under the MIT License - see LICENSE file for details.

## 🤝 Contributing

Contributions are welcome! Please:
1. Fork the repository
2. Create a feature branch (`git checkout -b feature/improvement`)
3. Commit your changes (`git commit -am 'Add feature'`)
4. Push to the branch (`git push origin feature/improvement`)
5. Open a Pull Request

## 📎 Citation

@inproceedings{rocha2026rgdermnet,
  title     = {RG-DermNet: A Multimodal Attention-Based Model with Residual Block Usage for Skin Lesion Classification},
  author    = {Rocha, Wyctor F. and Bouzon, Pedro H. G. and Ramos, Lucas A. and Pacheco, Andre G. C. and Souza Jr., Luis A.},
  booktitle = {International Joint Conference on Neural Networks (IJCNN)},
  year      = {2026},
  note      = {Accepted}
}

## 📧 Support

For issues, questions, or suggestions:
- Open an issue on GitHub
- Check existing documentation
- Verify model weights are properly downloaded

---

**Last Updated**: March 2026 | **Version**: 1.0.0