README.md

metadata

title: Medical X-Ray Captioning System
emoji: 🩻
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: 5.27.0
app_file: app.py
pinned: false
license: mit

Medical X-Ray Captioning System (Scratch vs. Pretrained)

Authors: Toni Ihab, Mahmoud Yasser, Micheal Amgad (Benha National University) Supervisor: Dr. Amr Nagy Academic Term: 2025

---

📖 Project Overview & The Origin Story

This project began as a series of exploratory experiments aimed at bridging the visual-linguistic gap in medical imaging by automatically generating diagnostic text reports from chest X-rays. In the early stages, development was scattered across multiple isolated attempts and reference scripts (preserved in this repo under reference-notebooks/ as scratch-attempt(1).py and scratch-attempt(2)-withGUI.py).

Instead of maintaining a fragmented codebase, we stepped back and merged these iterations into a single, cohesive pipeline. We extracted the strongest architectural components from our trials:

• A lightweight, custom-built CNN engineered specifically for raw grayscale medical textures.
• A ResNet50V2 backbone to test the benefits of transfer learning.

To eliminate bad coding practices and ensure a mathematically fair comparison between the two approaches, we standardized the downstream architecture, routing both encoders through the exact same Bahdanau (Additive) Attention mechanism and RNN Decoder.

With a clean, unified architecture successfully trained on the Indiana University dataset, our next goal was deployment. We built a local desktop application using Tkinter (app_gui.py) to allow users to upload X-rays and view the generated reports side-by-side.

⚠️ The Local Bottleneck: "Safe Guessing" & Greedy Search

It was during this local GUI deployment that we hit a major roadblock. Because medical datasets are overwhelmingly imbalanced toward "Normal" (healthy) cases, the models learned to artificially minimize loss by becoming "safe guessers."

Our Tkinter deployment severely worsened this flaw because it was hardcoded to use a Greedy Search decoding strategy. By rigidly picking only the single highest-probability token at every step, the GUI forced the model down the most common historical path. This resulted in repetitive, generic outputs (e.g., constantly predicting "no acute cardiopulmonary abnormality") and erased the nuanced text generation seen during notebook evaluation. Relying on Tkinter also restricted the entire system to local desktop environments, making it impossible to share or scale.

🚀 The Hugging Face Evolution

To prepare for web-scale deployment on Hugging Face Spaces, we abandoned the local Tkinter configuration in favor of a web-native Gradio app (app.py).

Alongside this framework migration, we implemented a parameterized Beam Search decoding pipeline. By holding multiple high-probability sequence hypotheses in parallel during text generation, Beam Search breaks the repetitive loop of the greedy algorithm, unlocking the accurate, diverse, and nuanced medical terminology the models actually learned.

---

🗂️ Repository Structure (GitHub / Local Workspace)

project-root/
├── docs/
│   ├── BNU.jpg                    # University branding image
│   ├── CODE_DOCUMENTATION.md      # Function reference & tensor shape notes
│   ├── INTEGRATION_GUIDE.md       # Environment setup & run instructions
│   ├── DLPD.tex                   # Academic report source (LaTeX)
│   ├── DLPD.pdf                   # Compiled academic report
│   ├── DLPD.aux / DLPD.log / DLPD.synctex.gz   # LaTeX build artifacts
│   ├── DLPP.pdf                   # Supplementary/presentation PDF
│   ├── pretrainedgui.png          # Screenshot: Tkinter GUI, pretrained model
│   ├── pretrainedresults.png      # Training/validation results, pretrained model
│   ├── scratchgui.png             # Screenshot: Tkinter GUI, scratch model
│   └── scratchresults.png         # Training/validation results, scratch model
│
├── checkpoints-models/
│   ├── Scratch-Attempt(1)s-Model/ # Weights from the first scratch training attempt
│   └── Scratch-Attempt(2)s-Model/ # Weights from the second scratch training attempt
│
├── pretrained_model/
│   ├── best_encoder.weights.h5    # Best-validation-loss encoder checkpoint
│   ├── best_decoder.weights.h5    # Best-validation-loss decoder checkpoint
│   ├── encoder_weights.weights.h5 # Final/latest encoder checkpoint
│   ├── decoder_weights.weights.h5 # Final/latest decoder checkpoint
│   └── tokenizer.pickle           # Fitted tokenizer for this model variant
│
├── scratch_model/                 # Same layout as pretrained_model/, for the from-scratch CNN
│
├── reference-notebooks/
│   ├── ready-attempt-withGUI.py       # Early integration test, foundational Tkinter wiring
│   ├── scratch-attempt(1).py          # First from-scratch training prototype
│   ├── scratch-attempt(1)s-gui.py     # Early desktop app running on prototype (1) weights
│   └── scratch-attempt(2)-withGUI.py  # Second prototype merging architecture + UI
│
├── model_scratch.py               # Custom CNN encoder + shared RNN decoder definitions
├── model_pretrained.py            # ResNet50V2 transfer-learning encoder wrapper
├── data_utilities.py              # Data ingestion, cleaning, tokenization, tf.data pipelines
├── app_gui.py                     # Legacy local Tkinter interface (Greedy Search only)
├── app.py                         # New Gradio web interface (Beam Search) — for HF Spaces
├── train_scratch_model.ipynb      # Training notebook for the scratch CNN (Colab GPU)
├── train_pretrained_model.ipynb   # Fine-tuning notebook for the ResNet50V2 variant
└── evaluation_utilities.ipynb     # BLEU scoring and attention heatmap visualization

File Descriptions (GitHub Workspace)

• model_scratch.py — Defines a 4-block custom CNN (Conv2D → BatchNorm → LeakyReLU → MaxPool) built from scratch to process grayscale X-rays without ImageNet priors. Also contains the shared BahdanauAttention and RNN_Decoder modules used by both model variants.
• model_pretrained.py — Implements the transfer-learning encoder. Loads ResNet50V2 with ImageNet weights, extracts spatial feature maps from an intermediate layer, projects them through custom dense layers, and feeds them into the shared attention network.
• data_utilities.py — Handles data ingestion (kagglehub), text cleaning, tokenizer fitting, padded sequence generation, and construction of tf.data pipelines with the correct preprocessing per model variant.
• app_gui.py — Legacy Tkinter GUI: local file dialogs, image rendering, and a Greedy Search inference loop. Deprecated in favor of app.py.
• app.py — Gradio web interface built for Hugging Face Spaces. Loads model weights inside the container, exposes a Beam Search width control, and runs inference through the multi-path decoding pipeline.
• train_scratch_model.ipynb — Full training loop for the scratch CNN on Colab GPU, including masked loss, backpropagation, and checkpointing.
• train_pretrained_model.ipynb — Fine-tuning loop for the ResNet50V2 variant, including layer-freeze management and low learning-rate adaptation.
• evaluation_utilities.ipynb — Computes BLEU-1 through BLEU-4 scores and renders attention heatmaps over the input X-rays.
• checkpoints-models/ — Historical weight checkpoints from the two scratch-model training attempts made before the final architecture merge.
• pretrained_model/ & scratch_model/ — Current production weight + tokenizer directories for each model variant, loaded directly by app.py.
• reference-notebooks/ — Archived early scripts and prototypes kept for historical reference.
• docs/ — Academic report (LaTeX/PDF), code documentation, integration guide, and result/GUI screenshots.

---

🚀 Hugging Face Space Production Layout

A minimized structure is uploaded to the Hugging Face Space, omitting development-only files to keep the container build fast:

huggingface-space/
├── pretrained_model/
│   ├── encoder_weights.weights.h5
│   ├── decoder_weights.weights.h5
│   └── tokenizer.pickle
├── scratch_model/
│   ├── encoder_weights.weights.h5
│   ├── decoder_weights.weights.h5
│   └── tokenizer.pickle
├── model_scratch.py                # Blueprint to reconstruct the scratch model
├── model_pretrained.py             # Blueprint to reconstruct the pretrained model
├── app.py                          # Gradio entry point
└── requirements.txt                # Locked dependency versions for the container

File Descriptions (Hugging Face Space)

• app.py — Main entry point. Launches the Gradio interface, loads and caches both models, handles image uploads, applies the selected Beam Search width, and returns the generated report.
• model_scratch.py / model_pretrained.py — Imported by app.py to rebuild the model architectures before loading saved weights.
• requirements.txt — Pinned dependencies (TensorFlow, Gradio, Pillow, NumPy, etc.) so the Hugging Face container builds reproducibly.
• pretrained_model/ & scratch_model/ — Weight and tokenizer files loaded directly by app.py at runtime.

---

🧪 Getting Started

Prerequisites

• Python 3.9+
• TensorFlow 2.15+
• Gradio 4.21+
• (Optional) kagglehub, if re-training

Local Development

# 1. Clone the repository
git clone <your-repo-url>
cd <your-repo>

# 2. Install dependencies
pip install -r requirements.txt

# 3. Run the legacy Tkinter GUI (old)
python app_gui.py

# 4. Or run the new Gradio interface locally
python app.py

Deploying to Hugging Face

1. Create a new Space at Hugging Face Spaces with the SDK set to Gradio.
2. Push or upload the contents of the huggingface-space/ layout above (weights, app.py, model definitions, requirements.txt).
3. The Space will automatically build the container and launch the app.

---

📜 License

This project is licensed under the MIT License.

🙏 Acknowledgements

Thanks to our supervisor, Dr. Amr Nagy, and Benha National University for the environment that made this research possible. For full technical detail, see docs/DLPD.pdf.