README.md

metadata

title: Multimodal Skin Lesion Explainability
colorFrom: blue
colorTo: indigo
sdk: docker
app_port: 7860
startup_duration_timeout: 1h
license: mit

🔬 RG-DermNet: Multimodal Skin Lesion Explainability

A Gradio-based web application for multimodal skin lesion analysis with GradCAM++ visualization. This tool enables clinicians and researchers to understand how deep learning models make predictions on dermoscopic images by combining image data with clinical metadata.

📄 Abstract

Skin cancer accounts for nearly one-third of all diagnosed tumors worldwide, making early and accurate recognition critical for improving patient outcomes. In this work, we propose RG-DermNet, a multimodal deep learning framework that integrates skin lesion images with structured clinical metadata through a residual gated-attention (RG-ATT) fusion mechanism. The architecture combines CNN- and Transformer-based visual backbones with a lightweight one-hot encoding pipeline for metadata, enabling effective cross-modal interaction. The proposed model is evaluated using a patient-wise cross-validation protocol across four dermatological datasets with heterogeneous metadata. On PAD-UFES-20, using Caformer-B36 as the visual backbone, RG-DermNet achieves an accuracy of 0.75 ± 0.05, balanced accuracy of 0.78 ± 0.03, F1-score of 0.77 ± 0.04, and AUC of 0.95 ± 0.01, outperforming existing multimodal baselines under the same evaluation setting. In addition, a SHAP-based analysis provides insights into the contribution of clinical metadata to the model’s predictions, supporting both performance gains and interpretability.

🎯 Features

• Dermoscopic Image Upload: Load and analyze dermoscopic skin lesion images
• Metadata Management: Organize clinical information into specific metadata groups:
  • Demographics (age, gender, lesion location)
  • Clinical History
  • Symptoms (itch, growth, bleeding, elevation, etc.)
  • Lesion Geometry (diameter measurements)
• Live Metadata Preview: Real-time CSV generation showing exact model inputs
• GradCAM++ Attention Maps: Visualize where the model focuses its attention on the image
• Multi-Model Support: Choose between different attention mechanisms (concatenation, metadata blocks, cross-attention, etc.)
• Prediction Summary: Get classification results with confidence scores

📊 Model Information

This application includes pre-trained models using:

• CNN Backbone: ResNet-50
• Attention Mechanisms: Multiple architectures including:
  • No-metadata baseline
  • Concatenation-based fusion
  • Cross-attention modules
  • Metadata blocks

Models are trained on the PAD-UFES-20 skin lesion dataset and classify into 6 categories:

• NEV: Nevus
• BCC: Basal Cell Carcinoma
• ACK: Actinic Keratosis
• SEK: Seborrheic Keratosis
• SCC: Squamous Cell Carcinoma
• MEL: Melanoma

🚀 Quick Start

Local Installation

Requirements:

• Python 3.10+
• PyTorch with CUDA support (optional but recommended)

Setup:

# Clone the repository
git clone <this-repo>
cd GradCAMPlusPlus_SkinLesion

# Create virtual environment (recommended)
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Run the application
python app.py

The application will start a local Gradio server. Open the provided URL in your browser.

HuggingFace Spaces Deployment

This repository is configured to deploy directly to HuggingFace Spaces.

To deploy your own instance:

1. Fork this repository to your HuggingFace account
2. Create a new Space using this repository
3. The app will automatically build and launch

Direct link to this Space: [Coming soon - your HF Space URL]

📖 Usage Guide

1. Upload an Image: Click "Upload" and select a dermoscopic image file (PNG, JPG, etc.)

2. Select Metadata Groups: Check which metadata categories are relevant:

   • ✅ Demographics: Always available
   • ✅ Clinical History: Optional
   • ✅ Symptoms: Optional
   • ✅ Lesion Geometry: Optional

3. Fill Patient Information:

   • Age: Patient's age in years
   • Gender: Male/Female
   • Region: Lesion location (HEAD, NECK, BACK, ARM, LEG, TORSO)
   • Diameter: Measure in two perpendicular directions

4. Select Model: Choose an attention mechanism from the dropdown:

   • Different models may perform differently on your image
   • Try multiple models to understand variations

5. Generate Analysis: Click "Generate GradCAM++" to:

   • Run inference on the image and metadata
   • Generate attention heatmap overlay
   • Display confidence scores and classification

6. Review Results:

   • Original Lesion: Your uploaded image
   • Attention Map: Where the model focused (red = high focus)
   • Metadata Details: View the exact CSV format sent to the model

📁 Project Structure

GradCAMPlusPlus_SkinLesion/
├── app.py                          # Entry point for HF Spaces
├── requirements.txt                 # Python dependencies
├── README.md                        # This file
├── .gitattributes                   # Git LFS configuration
├── .gitignore                       # Git ignore rules
├── data/
│   ├── weights/TO_BE_USED/         # Pre-trained model weights
│   │   ├── concatenation/
│   │   ├── metablock/
│   │   ├── no-metadata/
│   │   └── att-intramodal+residual+cross-attention-metadados/
│   └── preprocess_data/             # Encoders and scalers
│       ├── label_encoder_pad_20.pickle
│       ├── ohe_pad_20.pickle
│       └── scaler_pad_20.pickle
└── src/
    ├── main.py                      # Gradio UI and event handling
    ├── utils/
    │   ├── load_local_variables.py
    │   └── transforms.py
    └── models/
        ├── __init__.py
        ├── inference.py             # Model inference pipeline
        ├── model_loader.py          # PyTorch model loading
        ├── cam.py                   # GradCAM++ implementation
        ├── preprocessing.py         # Image & metadata preprocessing
        ├── loadImageModelClassifier.py
        ├── metadata_builder.py      # CSV generation
        ├── metadata_groups.py       # Metadata schema
        ├── metadata_schema.py       # Column definitions
        └── ... (other model architecture files)

🛠️ Technical Details

Input Processing

• Images: Normalized using ImageNet statistics, resized for model input
• Metadata: One-hot encoded and padded to 20 dimensions

Attention Visualization

• GradCAM++: Computes class activation maps using gradient averaging
• Overlay: Jet colormap (blue=low importance, red=high importance)

Model Input Format

• Metadata is sent as CSV with specific column order
• Columns are enabled/disabled based on selected groups
• Empty values represented as empty strings

⚠️ Important Notes

• Pre-condition: Model weights must be present in data/weights/TO_BE_USED/ for inference to work
• Missing Weights: If models fail to load, download from the original repository
• Not for Clinical Use: This tool is for research and education only. Do not use for clinical diagnosis.
• GPU Recommended: Faster inference with CUDA. Falls back to CPU if unavailable.

🔗 References & Acknowledgments

• Paper repository: RG-DermNet (https://github.com/wyctorfogos/rg-dermnet)
• Dataset: PAD-UFES-20 (Universidade Federal do Espírito Santo)
• GradCAM++: Paper by Chattopadhyay et al.
• Framework: Gradio for the web interface

📄 License

This project is licensed under the MIT License - see LICENSE file for details.

🤝 Contributing

Contributions are welcome! Please:

1. Fork the repository
2. Create a feature branch (git checkout -b feature/improvement)
3. Commit your changes (git commit -am 'Add feature')
4. Push to the branch (git push origin feature/improvement)
5. Open a Pull Request

📎 Citation

@inproceedings{rocha2026rgdermnet, title = {RG-DermNet: A Multimodal Attention-Based Model with Residual Block Usage for Skin Lesion Classification}, author = {Rocha, Wyctor F. and Bouzon, Pedro H. G. and Ramos, Lucas A. and Pacheco, Andre G. C. and Souza Jr., Luis A.}, booktitle = {International Joint Conference on Neural Networks (IJCNN)}, year = {2026}, note = {Accepted} }

📧 Support

For issues, questions, or suggestions:

• Open an issue on GitHub
• Check existing documentation
• Verify model weights are properly downloaded

---

Last Updated: March 2026 | Version: 1.0.0