Upload 2 files
Browse files
README.md
CHANGED
|
@@ -12,37 +12,75 @@ license: apache-2.0
|
|
| 12 |
|
| 13 |
# π©Ί Skin Disease Detection β Demo
|
| 14 |
|
| 15 |
-
A demo Space
|
| 16 |
-
- **
|
| 17 |
-
prediction
|
| 18 |
-
|
| 19 |
-
- **Chat
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 23 |
|
| 24 |
## How it works
|
| 25 |
|
| 26 |
| Component | What it does | Default model |
|
| 27 |
|---|---|---|
|
| 28 |
-
| Image classifier | Predicts a skin
|
| 29 |
-
|
|
| 30 |
-
|
| 31 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 32 |
|
| 33 |
## π Deploy in 3 steps
|
| 34 |
|
| 35 |
1. **Create a new Space**
|
| 36 |
Go to [huggingface.co/new-space](https://huggingface.co/new-space) β
|
| 37 |
-
choose **Gradio** as the SDK β CPU basic hardware is fine for the classifier
|
|
|
|
|
|
|
| 38 |
|
| 39 |
2. **Upload these files**
|
| 40 |
-
Upload `app.py`, `requirements.txt`, this `README.md`,
|
| 41 |
-
folder (if you generated test images
|
| 42 |
-
either via the web UI
|
|
|
|
| 43 |
```bash
|
| 44 |
git clone https://huggingface.co/spaces/<your-username>/<your-space-name>
|
| 45 |
-
cp -r app.py requirements.txt README.md examples <your-space-name>/
|
| 46 |
cd <your-space-name>
|
| 47 |
git add . && git commit -m "Initial commit" && git push
|
| 48 |
```
|
|
@@ -79,20 +117,74 @@ Set these as Space **variables** (not secret, unless noted) to customize:
|
|
| 79 |
|
| 80 |
- `IMAGE_MODEL_ID` β any Hugging Face image-classification model compatible
|
| 81 |
with `transformers.pipeline("image-classification", ...)`.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 82 |
- `CHAT_MODEL_ID` β any chat-completion-capable model available via the HF
|
| 83 |
-
Inference
|
| 84 |
- `HF_TOKEN` *(secret)* β required for the chat tab and the LLM explanations.
|
| 85 |
|
| 86 |
-
##
|
| 87 |
-
|
| 88 |
-
-
|
| 89 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 90 |
output as a talking point, not a result.
|
| 91 |
- Performance depends heavily on image quality, lighting, and skin tone
|
| 92 |
-
representation in the training data
|
|
|
|
|
|
|
|
|
|
| 93 |
- This app must not be used as a substitute for professional medical
|
| 94 |
-
evaluation
|
| 95 |
-
|
|
|
|
| 96 |
|
| 97 |
## Local development
|
| 98 |
|
|
|
|
| 12 |
|
| 13 |
# π©Ί Skin Disease Detection β Demo
|
| 14 |
|
| 15 |
+
A demo Space with four tabs:
|
| 16 |
+
- **π· Quick Image Check** β upload a photo (or click a test image), get a
|
| 17 |
+
classifier prediction plus an LLM-generated plain-language assessment with
|
| 18 |
+
an explicit **Low / Medium / High severity** call.
|
| 19 |
+
- **π¬ Chat Assistant** β a multi-turn triage conversation. It asks about
|
| 20 |
+
onset, evolution, symptoms, triggers, and history over several exchanges
|
| 21 |
+
before giving a narrowed, severity-rated preliminary read β more targeted
|
| 22 |
+
than a single photo alone could support, without claiming false certainty.
|
| 23 |
+
- **π Model Performance** β real test-set metrics (accuracy, per-class
|
| 24 |
+
precision/recall/F1, confusion matrix) for whichever fine-tuned model(s)
|
| 25 |
+
are backing the app, pulled from `assets/`.
|
| 26 |
+
- **βΉοΈ About** β what this is, what models it uses, and its limitations.
|
| 27 |
+
|
| 28 |
+
> β οΈ **This is a research/portfolio demo, not a medical device.** It does not
|
| 29 |
+
> diagnose anyone with certainty. Always see a licensed dermatologist or
|
| 30 |
+
> doctor for real concerns.
|
| 31 |
|
| 32 |
## How it works
|
| 33 |
|
| 34 |
| Component | What it does | Default model |
|
| 35 |
|---|---|---|
|
| 36 |
+
| Image classifier | Predicts a skin condition category from a photo | [`Anwarkh1/Skin_Cancer-Image_Classification`](https://huggingface.co/Anwarkh1/Skin_Cancer-Image_Classification) (ViT, 7-class HAM10000) β recommended upgrade: train `finetune_unified_skin_model.ipynb` (24-class, see below) and point `IMAGE_MODEL_ID` at it |
|
| 37 |
+
| Second-opinion lesion model *(optional, redundant if using the unified model)* | Re-checks images the primary model flags as an ambiguous melanoma/mole label | Off by default β set `LESION_MODEL_ID` to a HAM10000-based model to enable |
|
| 38 |
+
| Chat / explanation | Multi-turn triage with enforced minimum question rounds, turns predictions into plain language | `Qwen/Qwen2.5-7B-Instruct` via the Hugging Face Inference Providers router |
|
| 39 |
+
|
| 40 |
+
All three model IDs are configurable via Space variables β see below.
|
| 41 |
+
|
| 42 |
+
## Multi-round chat behavior
|
| 43 |
+
|
| 44 |
+
The chat tab doesn't just *ask* the model to have a longer conversation β
|
| 45 |
+
`app.py` counts user turns explicitly and injects a stage-aware instruction
|
| 46 |
+
on every call (`MIN_ROUNDS_BEFORE_ASSESSMENT = 4` by default, adjustable
|
| 47 |
+
in code): before that many exchanges, the model is told to keep gathering
|
| 48 |
+
information; after, it's told enough rounds have happened and to give its
|
| 49 |
+
assessment. Red-flag symptoms override this and get addressed immediately
|
| 50 |
+
regardless of turn count.
|
| 51 |
+
|
| 52 |
+
## Severity levels
|
| 53 |
+
|
| 54 |
+
Every assessment states one of three levels explicitly:
|
| 55 |
+
- π΄ **High** β cancer-related, ambiguous-but-could-be-serious, systemic/
|
| 56 |
+
autoimmune conditions, or red-flag symptoms. Recommends seeing a doctor soon.
|
| 57 |
+
- π‘ **Medium** β likely needs a proper diagnosis and prescription
|
| 58 |
+
(infections, infestations). Recommends a doctor visit, not urgent.
|
| 59 |
+
- π’ **Low** β typically manageable with general skin care. Doctor visit
|
| 60 |
+
optional.
|
| 61 |
+
|
| 62 |
+
Tiering is **keyword-based** (see `classify_tier()` in `app.py`), so it works
|
| 63 |
+
across different classifier taxonomies without hardcoding every exact label
|
| 64 |
+
string β including correctly flagging DermNet's mixed
|
| 65 |
+
*"Melanoma Skin Cancer, Nevi and Moles"* label as High by default, since that
|
| 66 |
+
one label can't distinguish the dangerous case from the harmless one.
|
| 67 |
|
| 68 |
## π Deploy in 3 steps
|
| 69 |
|
| 70 |
1. **Create a new Space**
|
| 71 |
Go to [huggingface.co/new-space](https://huggingface.co/new-space) β
|
| 72 |
+
choose **Gradio** as the SDK β CPU basic hardware is fine for the classifier
|
| 73 |
+
(or ZeroGPU if you want faster inference β see the `@spaces.GPU` decorator
|
| 74 |
+
already applied to `diagnose_image`).
|
| 75 |
|
| 76 |
2. **Upload these files**
|
| 77 |
+
Upload `app.py`, `requirements.txt`, this `README.md`, the `examples/`
|
| 78 |
+
folder (if you generated test images), and the `assets/` folder (if you
|
| 79 |
+
exported model performance metrics β see below), either via the web UI
|
| 80 |
+
("Add file") or:
|
| 81 |
```bash
|
| 82 |
git clone https://huggingface.co/spaces/<your-username>/<your-space-name>
|
| 83 |
+
cp -r app.py requirements.txt README.md examples assets <your-space-name>/
|
| 84 |
cd <your-space-name>
|
| 85 |
git add . && git commit -m "Initial commit" && git push
|
| 86 |
```
|
|
|
|
| 117 |
|
| 118 |
- `IMAGE_MODEL_ID` β any Hugging Face image-classification model compatible
|
| 119 |
with `transformers.pipeline("image-classification", ...)`.
|
| 120 |
+
- `LESION_MODEL_ID` *(optional)* β a second, pigmented-lesion-specific model
|
| 121 |
+
(like the HAM10000 fine-tune below) used automatically as a second opinion
|
| 122 |
+
when the primary model's top prediction is an ambiguous melanoma/mole label.
|
| 123 |
+
Leave unset to disable.
|
| 124 |
- `CHAT_MODEL_ID` β any chat-completion-capable model available via the HF
|
| 125 |
+
Inference Providers router.
|
| 126 |
- `HF_TOKEN` *(secret)* β required for the chat tab and the LLM explanations.
|
| 127 |
|
| 128 |
+
## π§ Fine-tuning your own model
|
| 129 |
+
|
| 130 |
+
Three self-contained Colab notebooks are included:
|
| 131 |
+
|
| 132 |
+
### `finetune_unified_skin_model.ipynb` β β recommended: 24-class unified model
|
| 133 |
+
- **Retrained from scratch**, not resumed β fresh weights each run
|
| 134 |
+
- Merges DermNet's 22 clean categories with HAM10000's melanoma and nevi
|
| 135 |
+
as their own separate classes, **replacing DermNet's ambiguous combined
|
| 136 |
+
"Melanoma Skin Cancer, Nevi and Moles" label entirely** β melanoma finally
|
| 137 |
+
gets a clean training signal instead of being merged with benign moles
|
| 138 |
+
- Early stopping (up to 14 epochs, stops automatically once validation
|
| 139 |
+
macro-F1 plateaus) + cosine LR schedule β addresses the previous
|
| 140 |
+
DermNet-only run's validation loss not having converged at 6 epochs
|
| 141 |
+
- Melanoma threshold-tuning (now meaningful, since melanoma is unambiguous)
|
| 142 |
+
- Exports `unified_metrics.json` + `unified_confusion_matrix.png`
|
| 143 |
+
- **Honest tradeoff:** melanoma/nevi images come from HAM10000's
|
| 144 |
+
dermatoscope close-ups, not regular phone photos like the rest of the
|
| 145 |
+
dataset β a real domain gap, flagged in the notebook rather than hidden
|
| 146 |
+
|
| 147 |
+
### `finetune_skin_lesion_model.ipynb` β 7-class pigmented lesions (HAM10000 only)
|
| 148 |
+
- Useful if you want a smaller, faster, cancer-focused model specifically
|
| 149 |
+
- class-weighted loss, melanoma recall tracking, threshold tuning
|
| 150 |
+
- exports `ham10000_metrics.json` + `ham10000_confusion_matrix.png`
|
| 151 |
+
|
| 152 |
+
### `finetune_dermnet_23class.ipynb` β 23-class DermNet only (superseded by the unified notebook above, kept for reference)
|
| 153 |
+
- Same as the unified notebook's DermNet portion, but keeps DermNet's
|
| 154 |
+
original ambiguous melanoma/nevi label instead of replacing it
|
| 155 |
+
- exports `dermnet_metrics.json` + `dermnet_confusion_matrix.png`
|
| 156 |
+
|
| 157 |
+
**To use any of them:** upload the `.ipynb` to
|
| 158 |
+
[Google Colab](https://colab.research.google.com), set **Runtime β Change
|
| 159 |
+
runtime type β T4 GPU**, and run cells top to bottom. The unified notebook
|
| 160 |
+
takes roughly 60β100 minutes on a free T4 depending on when early stopping
|
| 161 |
+
kicks in; keep the tab active so the session doesn't disconnect.
|
| 162 |
+
|
| 163 |
+
**Before treating any of these as more than a research project**, see the
|
| 164 |
+
"Next steps" section at the end of each notebook β a higher benchmark score
|
| 165 |
+
is not the same as a clinically validated model.
|
| 166 |
+
|
| 167 |
+
## π Populating the Model Performance tab
|
| 168 |
+
|
| 169 |
+
After running a fine-tuning notebook, download the two files it exports
|
| 170 |
+
(`*_metrics.json` and `*_confusion_matrix.png`) from Colab's file browser and
|
| 171 |
+
place them in this Space's `assets/` folder, then redeploy. The tab picks up
|
| 172 |
+
any matching pair automatically β you can have both the HAM10000 and DermNet
|
| 173 |
+
reports side by side if you've fine-tuned both.
|
| 174 |
+
|
| 175 |
+
## Limitations & responsible use
|
| 176 |
+
|
| 177 |
+
- Neither bundled classifier has been clinically validated β treat their
|
| 178 |
output as a talking point, not a result.
|
| 179 |
- Performance depends heavily on image quality, lighting, and skin tone
|
| 180 |
+
representation in the training data; public dermatology datasets skew
|
| 181 |
+
toward lighter skin tones.
|
| 182 |
+
- DermNet's melanoma/nevi label limitation (above) means that category's
|
| 183 |
+
severity is intentionally conservative rather than precise.
|
| 184 |
- This app must not be used as a substitute for professional medical
|
| 185 |
+
evaluation. The chatbot is instructed to narrow toward likely categories
|
| 186 |
+
and state a severity level, but never to claim certainty, prescribe
|
| 187 |
+
treatment, or give dosages.
|
| 188 |
|
| 189 |
## Local development
|
| 190 |
|
app.py
CHANGED
|
@@ -36,7 +36,11 @@ CHAT_MODEL_ID = os.environ.get("CHAT_MODEL_ID", "Qwen/Qwen2.5-7B-Instruct")
|
|
| 36 |
HF_TOKEN = os.environ.get("HF_TOKEN") # add as a Space SECRET to enable the chatbot
|
| 37 |
ROUTER_URL = "https://router.huggingface.co/v1/chat/completions"
|
| 38 |
|
| 39 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
|
| 41 |
# Optional second classifier specifically for pigmented-lesion cases (HAM10000-based,
|
| 42 |
# where melanoma is its own clean label). Useful because broader taxonomies like
|
|
@@ -64,10 +68,10 @@ PROMPT_DOCTOR_KEYWORDS = [
|
|
| 64 |
# manageable/chronic conditions (acne, eczema, psoriasis, hives, hair loss,
|
| 65 |
# benign tumors, common moles, etc.) where general skin-care guidance fits.
|
| 66 |
|
| 67 |
-
|
| 68 |
-
"urgent": "
|
| 69 |
-
"prompt_doctor_visit": "needs a proper diagnosis/prescription
|
| 70 |
-
"general_care": "
|
| 71 |
}
|
| 72 |
|
| 73 |
|
|
@@ -83,6 +87,12 @@ def classify_tier(label):
|
|
| 83 |
return "general_care"
|
| 84 |
|
| 85 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 86 |
# Friendly one-line descriptions for labels we recognize exactly (both the
|
| 87 |
# original 7-class HAM10000 set and common DermNet-style labels). Purely
|
| 88 |
# cosmetic β if a label isn't here, the tier-based guidance still works fine
|
|
@@ -97,26 +107,28 @@ KNOWN_DESCRIPTIONS = {
|
|
| 97 |
"vascular lesions": "Blood-vessel-related marks such as angiomas.",
|
| 98 |
}
|
| 99 |
|
| 100 |
-
SYSTEM_PROMPT = """You are a warm, knowledgeable skin-health triage assistant. You are not a doctor and cannot diagnose anyone β but your job is to be genuinely useful, not to hide behind constant disclaimers.
|
| 101 |
|
| 102 |
-
CONVERSATION STYLE
|
| 103 |
-
|
| 104 |
-
-
|
| 105 |
-
-
|
| 106 |
-
-
|
| 107 |
-
-
|
| 108 |
-
-
|
| 109 |
-
-
|
| 110 |
-
-
|
| 111 |
-
|
|
|
|
| 112 |
|
| 113 |
GIVING A PRELIMINARY ASSESSMENT
|
| 114 |
-
Once you have
|
| 115 |
-
-
|
| 116 |
-
-
|
| 117 |
-
- **
|
|
|
|
|
|
|
| 118 |
- Never give specific medications, dosages, or treatment prescriptions β general skin-care habits are fine; treating a self- or AI-identified condition with anything beyond general care is not.
|
| 119 |
-
- You don't need to repeat a formal warning block every message. Say once, naturally, as part of your assessment β not as a bolted-on disclaimer β that this is a preliminary read from a conversation and an image classifier, and that an in-person exam is what actually confirms things. Then move on; don't repeat it every turn.
|
| 120 |
|
| 121 |
URGENT SITUATIONS
|
| 122 |
If the user describes something urgent β rapidly growing lesion, uncontrolled bleeding, signs of spreading infection, fever with a skin issue, severe pain β tell them clearly to seek in-person or emergency care promptly, regardless of how many turns you've had.
|
|
@@ -227,6 +239,7 @@ def diagnose_image(image):
|
|
| 227 |
return f"**Model error:** {e}\n\nMake sure `IMAGE_MODEL_ID` is a valid image-classification model."
|
| 228 |
|
| 229 |
lines = ["### π¬ Classifier prediction\n"] + lines_list
|
|
|
|
| 230 |
|
| 231 |
if HF_TOKEN:
|
| 232 |
context = build_classifier_context(preds)
|
|
@@ -234,8 +247,11 @@ def diagnose_image(image):
|
|
| 234 |
f"{context}\n\n"
|
| 235 |
"The user just uploaded a single image with no conversation yet. Give a short "
|
| 236 |
"preliminary assessment following your instructions: plain-language explanation of "
|
| 237 |
-
"the top finding,
|
| 238 |
-
"in-person
|
|
|
|
|
|
|
|
|
|
| 239 |
)
|
| 240 |
try:
|
| 241 |
explanation = chat_completion(
|
|
@@ -264,6 +280,9 @@ def diagnose_image(image):
|
|
| 264 |
# ---------------------------------------------------------------------------
|
| 265 |
# Chat tab β multimodal, multi-turn triage
|
| 266 |
# ---------------------------------------------------------------------------
|
|
|
|
|
|
|
|
|
|
| 267 |
def _extract_text_and_image(message):
|
| 268 |
"""Normalize gr.ChatInterface(multimodal=True) message input into (text, image_path_or_None)."""
|
| 269 |
if isinstance(message, dict):
|
|
@@ -274,6 +293,31 @@ def _extract_text_and_image(message):
|
|
| 274 |
return str(message), None
|
| 275 |
|
| 276 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 277 |
def chat_respond(message, history):
|
| 278 |
if not HF_TOKEN:
|
| 279 |
return (
|
|
@@ -307,7 +351,10 @@ def chat_respond(message, history):
|
|
| 307 |
except Exception as e:
|
| 308 |
user_content = f"(Image analysis failed: {e})\n\nUser's message: {text}"
|
| 309 |
|
| 310 |
-
|
|
|
|
|
|
|
|
|
|
| 311 |
|
| 312 |
try:
|
| 313 |
return chat_completion(messages, max_tokens=500)
|
|
@@ -315,6 +362,118 @@ def chat_respond(message, history):
|
|
| 315 |
return f"Sorry, I hit an error talking to the model: {e}"
|
| 316 |
|
| 317 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 318 |
# ---------------------------------------------------------------------------
|
| 319 |
# UI
|
| 320 |
# ---------------------------------------------------------------------------
|
|
@@ -329,6 +488,7 @@ if os.path.isdir(EXAMPLES_DIR):
|
|
| 329 |
|
| 330 |
with gr.Blocks(title="Skin Disease Detection β Demo") as demo:
|
| 331 |
gr.Markdown("# π©Ί Skin Disease Detection")
|
|
|
|
| 332 |
|
| 333 |
with gr.Tab("π· Quick Image Check"):
|
| 334 |
with gr.Row():
|
|
@@ -352,10 +512,31 @@ with gr.Blocks(title="Skin Disease Detection β Demo") as demo:
|
|
| 352 |
multimodal=True,
|
| 353 |
description=(
|
| 354 |
"Talk through what you're noticing, and optionally attach a photo (π). "
|
| 355 |
-
"The assistant will ask
|
|
|
|
|
|
|
| 356 |
),
|
| 357 |
)
|
| 358 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 359 |
gr.Markdown(
|
| 360 |
"---\nBuilt with π€ Transformers + Gradio. "
|
| 361 |
f"Image model: `{IMAGE_MODEL_ID}` Β· Chat model: `{CHAT_MODEL_ID}`"
|
|
|
|
| 36 |
HF_TOKEN = os.environ.get("HF_TOKEN") # add as a Space SECRET to enable the chatbot
|
| 37 |
ROUTER_URL = "https://router.huggingface.co/v1/chat/completions"
|
| 38 |
|
| 39 |
+
PAGE_DISCLAIMER = (
|
| 40 |
+
"This tool is a research/portfolio project, not a medical device. It hasn't "
|
| 41 |
+
"been clinically validated. Use it to get informed and organize your thoughts "
|
| 42 |
+
"before a real appointment β not as a substitute for one."
|
| 43 |
+
)
|
| 44 |
|
| 45 |
# Optional second classifier specifically for pigmented-lesion cases (HAM10000-based,
|
| 46 |
# where melanoma is its own clean label). Useful because broader taxonomies like
|
|
|
|
| 68 |
# manageable/chronic conditions (acne, eczema, psoriasis, hives, hair loss,
|
| 69 |
# benign tumors, common moles, etc.) where general skin-care guidance fits.
|
| 70 |
|
| 71 |
+
SEVERITY = {
|
| 72 |
+
"urgent": {"label": "High", "emoji": "π΄", "note": "recommend seeing a dermatologist/doctor soon"},
|
| 73 |
+
"prompt_doctor_visit": {"label": "Medium", "emoji": "π‘", "note": "needs a proper diagnosis/prescription, not an emergency"},
|
| 74 |
+
"general_care": {"label": "Low", "emoji": "π’", "note": "typically manageable with general skin care"},
|
| 75 |
}
|
| 76 |
|
| 77 |
|
|
|
|
| 87 |
return "general_care"
|
| 88 |
|
| 89 |
|
| 90 |
+
def severity_line(label):
|
| 91 |
+
tier = classify_tier(label)
|
| 92 |
+
s = SEVERITY[tier]
|
| 93 |
+
return f"{s['emoji']} **Severity: {s['label']}** β {s['note']}"
|
| 94 |
+
|
| 95 |
+
|
| 96 |
# Friendly one-line descriptions for labels we recognize exactly (both the
|
| 97 |
# original 7-class HAM10000 set and common DermNet-style labels). Purely
|
| 98 |
# cosmetic β if a label isn't here, the tier-based guidance still works fine
|
|
|
|
| 107 |
"vascular lesions": "Blood-vessel-related marks such as angiomas.",
|
| 108 |
}
|
| 109 |
|
| 110 |
+
SYSTEM_PROMPT = """You are a warm, knowledgeable skin-health triage assistant. You are not a doctor and cannot diagnose anyone with certainty β but your job is to be genuinely useful and specific, not to hide behind vague hedging or constant disclaimers.
|
| 111 |
|
| 112 |
+
CONVERSATION STYLE β GO DEEPER BEFORE ASSESSING
|
| 113 |
+
Have a real back-and-forth, one or two questions at a time. Aim to gather a fuller picture than a quick glance would give β typically 5-8 exchanges before your assessment, more if the picture is still unclear, fewer only if the user clearly just wants a fast read or red flags are already obvious. Useful ground to cover (skip anything already answered or clearly irrelevant):
|
| 114 |
+
- Onset: how long they've had it, how it started
|
| 115 |
+
- Evolution: how it's changed over time β size, shape, color, texture
|
| 116 |
+
- Symptoms: itching, bleeding, pain, crusting, oozing, discharge, fever, swelling
|
| 117 |
+
- Pattern: is it one spot or spreading, single or multiple, symmetric or not
|
| 118 |
+
- Triggers/exposures: new products, plants, insect bites, contacts, travel, sun exposure
|
| 119 |
+
- History: prior similar episodes, what's been tried already (and whether it helped), relevant personal/family medical history, allergies
|
| 120 |
+
- Location and distribution on the body
|
| 121 |
+
The goal of the extra rounds is to genuinely narrow the differential, not to interrogate β keep it conversational, and if the user gives a rich answer that covers several of these at once, don't force redundant questions.
|
| 122 |
+
If the user shared an image, you'll also receive the image classifier's findings as extra context, including an urgency tier (not shown verbatim to the user) β weave that in naturally rather than reading out raw percentages.
|
| 123 |
|
| 124 |
GIVING A PRELIMINARY ASSESSMENT
|
| 125 |
+
Once you have real depth of context, give a clear, specific preliminary assessment:
|
| 126 |
+
- Name the 1-2 most likely categories given everything discussed, in plain language, and briefly say why (which symptoms/pattern point that way). It's fine β good, even β to be specific about what you think is most likely. What you must NOT do is claim certainty: use "most consistent with" / "this pattern often points to," never "you have X" or "this is definitely X." A real diagnosis needs an in-person exam and sometimes tests (biopsy, culture, bloodwork) that no photo or conversation can replace β say this once, naturally, as part of the assessment, not as a repeated disclaimer.
|
| 127 |
+
- Always state an explicit severity level as part of the assessment: **Low**, **Medium**, or **High** β plus one line on what that means for next steps. Base it on:
|
| 128 |
+
- **High** = cancer-related or ambiguous-but-could-be-cancer findings, autoimmune/systemic/bullous conditions, cellulitis, vasculitis, or any red-flag symptoms (rapid growth, irregular/changing borders, multiple colors, asymmetry, a sore that won't heal, bleeding, spreading redness, fever) β recommend seeing a dermatologist or doctor soon, directly, without softening it into "keep an eye on it." Note explicitly if the classifier's category can't distinguish something dangerous from something benign (e.g. a label that groups melanoma with ordinary moles) β when that ambiguity exists, say so and default to High rather than assuming benign.
|
| 129 |
+
- **Medium** = things that need a proper diagnosis and often a prescription to clear up (fungal, viral, bacterial infections, STD-related, infestations like scabies) but aren't emergencies β recommend a doctor visit, explain why self-treatment usually doesn't fully work here. For anything STD-related, stay factual and non-judgmental, and point toward in-person testing rather than guessing from a photo.
|
| 130 |
+
- **Low** = typically manageable conditions (acne, eczema, psoriasis, hives, hair loss, common moles, benign growths, contact dermatitis) with no red flags β explain what it usually is, and give general skin-care guidance: gentle skincare habits, sun protection, not picking/scratching, watching for changes. Still fine to mention a doctor visit is reasonable if they're unsure, worried, or it's not improving.
|
| 131 |
- Never give specific medications, dosages, or treatment prescriptions β general skin-care habits are fine; treating a self- or AI-identified condition with anything beyond general care is not.
|
|
|
|
| 132 |
|
| 133 |
URGENT SITUATIONS
|
| 134 |
If the user describes something urgent β rapidly growing lesion, uncontrolled bleeding, signs of spreading infection, fever with a skin issue, severe pain β tell them clearly to seek in-person or emergency care promptly, regardless of how many turns you've had.
|
|
|
|
| 239 |
return f"**Model error:** {e}\n\nMake sure `IMAGE_MODEL_ID` is a valid image-classification model."
|
| 240 |
|
| 241 |
lines = ["### π¬ Classifier prediction\n"] + lines_list
|
| 242 |
+
lines.append(f"\n{severity_line(preds[0]['label'])}")
|
| 243 |
|
| 244 |
if HF_TOKEN:
|
| 245 |
context = build_classifier_context(preds)
|
|
|
|
| 247 |
f"{context}\n\n"
|
| 248 |
"The user just uploaded a single image with no conversation yet. Give a short "
|
| 249 |
"preliminary assessment following your instructions: plain-language explanation of "
|
| 250 |
+
"the top finding(s), an explicit Low/Medium/High severity call, and a brief natural "
|
| 251 |
+
"mention that an in-person exam is what actually confirms things. Note that without "
|
| 252 |
+
"any conversation history you have less context than usual β you can still give your "
|
| 253 |
+
"best read, but you may want to suggest the Chat Assistant for a fuller picture if the "
|
| 254 |
+
"case seems ambiguous. Keep it to 5-7 sentences."
|
| 255 |
)
|
| 256 |
try:
|
| 257 |
explanation = chat_completion(
|
|
|
|
| 280 |
# ---------------------------------------------------------------------------
|
| 281 |
# Chat tab β multimodal, multi-turn triage
|
| 282 |
# ---------------------------------------------------------------------------
|
| 283 |
+
MIN_ROUNDS_BEFORE_ASSESSMENT = 4 # user turns, not counting the current one
|
| 284 |
+
|
| 285 |
+
|
| 286 |
def _extract_text_and_image(message):
|
| 287 |
"""Normalize gr.ChatInterface(multimodal=True) message input into (text, image_path_or_None)."""
|
| 288 |
if isinstance(message, dict):
|
|
|
|
| 293 |
return str(message), None
|
| 294 |
|
| 295 |
|
| 296 |
+
def _count_user_turns(messages):
|
| 297 |
+
return sum(1 for m in messages if m.get("role") == "user")
|
| 298 |
+
|
| 299 |
+
|
| 300 |
+
def _stage_note(user_turn_count):
|
| 301 |
+
"""Injected each turn so multi-round behavior is enforced by code, not just
|
| 302 |
+
requested once in the system prompt β the model gets a fresh reminder of
|
| 303 |
+
what stage the conversation is in on every single call."""
|
| 304 |
+
if user_turn_count < MIN_ROUNDS_BEFORE_ASSESSMENT:
|
| 305 |
+
return (
|
| 306 |
+
f"[Internal note β not from the user: this is exchange {user_turn_count} of at least "
|
| 307 |
+
f"{MIN_ROUNDS_BEFORE_ASSESSMENT} before a full assessment, UNLESS the user has already "
|
| 308 |
+
"described clear red-flag symptoms (rapid growth, bleeding, severe pain, spreading "
|
| 309 |
+
"infection, fever) in which case address that urgently right now regardless of turn "
|
| 310 |
+
"count. Otherwise, continue asking focused follow-up questions β don't jump to a full "
|
| 311 |
+
"Low/Medium/High assessment yet.]"
|
| 312 |
+
)
|
| 313 |
+
return (
|
| 314 |
+
f"[Internal note β not from the user: this is exchange {user_turn_count}, enough rounds "
|
| 315 |
+
"have happened. If you have a reasonably clear picture, give your preliminary assessment "
|
| 316 |
+
"now with an explicit Low/Medium/High severity call. If something important is still "
|
| 317 |
+
"genuinely unclear, you may ask one more focused question first, but don't stall further.]"
|
| 318 |
+
)
|
| 319 |
+
|
| 320 |
+
|
| 321 |
def chat_respond(message, history):
|
| 322 |
if not HF_TOKEN:
|
| 323 |
return (
|
|
|
|
| 351 |
except Exception as e:
|
| 352 |
user_content = f"(Image analysis failed: {e})\n\nUser's message: {text}"
|
| 353 |
|
| 354 |
+
# current turn counts too, since the model is about to respond to it
|
| 355 |
+
current_turn_count = _count_user_turns(messages) + 1
|
| 356 |
+
stage_note = _stage_note(current_turn_count)
|
| 357 |
+
messages.append({"role": "user", "content": f"{stage_note}\n\n{user_content}"})
|
| 358 |
|
| 359 |
try:
|
| 360 |
return chat_completion(messages, max_tokens=500)
|
|
|
|
| 362 |
return f"Sorry, I hit an error talking to the model: {e}"
|
| 363 |
|
| 364 |
|
| 365 |
+
# ---------------------------------------------------------------------------
|
| 366 |
+
# Model Performance tab data loading
|
| 367 |
+
# ---------------------------------------------------------------------------
|
| 368 |
+
ASSETS_DIR = "assets"
|
| 369 |
+
|
| 370 |
+
|
| 371 |
+
def load_performance_reports():
|
| 372 |
+
"""Scan assets/ for *_metrics.json files and pair each with its confusion-matrix
|
| 373 |
+
image if present. Returns a list of dicts; empty list if nothing's been uploaded yet."""
|
| 374 |
+
import json
|
| 375 |
+
|
| 376 |
+
reports = []
|
| 377 |
+
if not os.path.isdir(ASSETS_DIR):
|
| 378 |
+
return reports
|
| 379 |
+
for fname in sorted(os.listdir(ASSETS_DIR)):
|
| 380 |
+
if not fname.endswith("_metrics.json"):
|
| 381 |
+
continue
|
| 382 |
+
prefix = fname[: -len("_metrics.json")]
|
| 383 |
+
try:
|
| 384 |
+
with open(os.path.join(ASSETS_DIR, fname)) as f:
|
| 385 |
+
data = json.load(f)
|
| 386 |
+
except Exception:
|
| 387 |
+
continue
|
| 388 |
+
image_path = None
|
| 389 |
+
for ext in (".png", ".jpg", ".jpeg"):
|
| 390 |
+
candidate = os.path.join(ASSETS_DIR, f"{prefix}_confusion_matrix{ext}")
|
| 391 |
+
if os.path.isfile(candidate):
|
| 392 |
+
image_path = candidate
|
| 393 |
+
break
|
| 394 |
+
reports.append({"data": data, "image_path": image_path})
|
| 395 |
+
return reports
|
| 396 |
+
|
| 397 |
+
|
| 398 |
+
def format_performance_markdown(report):
|
| 399 |
+
data = report["data"]
|
| 400 |
+
lines = [f"### {data.get('model_name', 'Model')}"]
|
| 401 |
+
acc = data.get("accuracy")
|
| 402 |
+
f1m = data.get("f1_macro")
|
| 403 |
+
if acc is not None:
|
| 404 |
+
lines.append(f"**Accuracy:** {acc*100:.1f}% Β· **Macro F1:** {f1m*100:.1f}%" if f1m is not None else f"**Accuracy:** {acc*100:.1f}%")
|
| 405 |
+
|
| 406 |
+
per_class = data.get("per_class", {})
|
| 407 |
+
if per_class:
|
| 408 |
+
lines.append("\n| Class | Precision | Recall | F1 | Support |")
|
| 409 |
+
lines.append("|---|---|---|---|---|")
|
| 410 |
+
for cls, m in per_class.items():
|
| 411 |
+
lines.append(
|
| 412 |
+
f"| {cls} | {m.get('precision', 0)*100:.1f}% | {m.get('recall', 0)*100:.1f}% "
|
| 413 |
+
f"| {m.get('f1', 0)*100:.1f}% | {int(m.get('support', 0))} |"
|
| 414 |
+
)
|
| 415 |
+
|
| 416 |
+
notes = data.get("notes")
|
| 417 |
+
if notes:
|
| 418 |
+
lines.append(f"\n_Note: {notes}_")
|
| 419 |
+
|
| 420 |
+
return "\n".join(lines)
|
| 421 |
+
|
| 422 |
+
|
| 423 |
+
ABOUT_MARKDOWN = f"""
|
| 424 |
+
## About this project
|
| 425 |
+
|
| 426 |
+
A research/portfolio demo exploring how far an image classifier + LLM chat
|
| 427 |
+
assistant can go toward useful skin-condition triage β while being explicit
|
| 428 |
+
about what it isn't: a validated medical device.
|
| 429 |
+
|
| 430 |
+
### What's inside
|
| 431 |
+
- **Quick Image Check** β upload a photo, get a classifier prediction plus
|
| 432 |
+
an LLM-generated plain-language assessment with an explicit Low/Medium/High
|
| 433 |
+
severity call.
|
| 434 |
+
- **Chat Assistant** β a multi-turn conversation that asks about onset,
|
| 435 |
+
evolution, symptoms, triggers, and history before giving a more targeted
|
| 436 |
+
preliminary read than a single photo alone could support.
|
| 437 |
+
- **Model Performance** β the actual test-set metrics for whichever
|
| 438 |
+
fine-tuned model(s) are currently backing this app, not just marketing
|
| 439 |
+
claims about accuracy.
|
| 440 |
+
|
| 441 |
+
### Models
|
| 442 |
+
- **Image classifier:** configurable via the `IMAGE_MODEL_ID` variable.
|
| 443 |
+
Two purpose-built options are included as fine-tuning notebooks in this
|
| 444 |
+
repo: a 7-class pigmented-lesion model (HAM10000/ISIC, ViT-Base) and a
|
| 445 |
+
23-class common-skin-condition model (DermNet, ConvNeXt-Base).
|
| 446 |
+
- **Optional second-opinion lesion model:** `LESION_MODEL_ID`, used
|
| 447 |
+
automatically when the primary model's top prediction is an ambiguous
|
| 448 |
+
mixed melanoma/mole-type label, to get a model that treats melanoma as
|
| 449 |
+
its own distinct class.
|
| 450 |
+
- **Chat model:** configurable via `CHAT_MODEL_ID`, served through Hugging
|
| 451 |
+
Face's Inference Providers router.
|
| 452 |
+
|
| 453 |
+
### Severity levels
|
| 454 |
+
- π΄ **High** β cancer-related, ambiguous-but-could-be-serious, systemic/
|
| 455 |
+
autoimmune, or red-flag symptoms present. See a doctor soon.
|
| 456 |
+
- π‘ **Medium** β likely needs a proper diagnosis and prescription
|
| 457 |
+
(infections, infestations). See a doctor, not an emergency.
|
| 458 |
+
- π’ **Low** β typically manageable with general skin care. Doctor visit
|
| 459 |
+
optional, especially if unsure or it's not improving.
|
| 460 |
+
|
| 461 |
+
### Known limitations (read this before trusting the output)
|
| 462 |
+
- **Not clinically validated.** Benchmark accuracy on a public dataset is
|
| 463 |
+
not the same as real-world diagnostic reliability.
|
| 464 |
+
- **Dataset label limitations.** The DermNet taxonomy combines melanoma
|
| 465 |
+
with benign moles under one label β the app defaults that whole category
|
| 466 |
+
to High severity rather than pretending to distinguish them.
|
| 467 |
+
- **Skin-tone representation.** Public dermatology datasets skew toward
|
| 468 |
+
lighter skin tones; performance on underrepresented skin tones is
|
| 469 |
+
unverified without a dedicated bias audit.
|
| 470 |
+
- **No clinical validation, IRB review, or regulatory clearance** has been
|
| 471 |
+
done. This should inform, not replace, professional care.
|
| 472 |
+
|
| 473 |
+
{PAGE_DISCLAIMER}
|
| 474 |
+
"""
|
| 475 |
+
|
| 476 |
+
|
| 477 |
# ---------------------------------------------------------------------------
|
| 478 |
# UI
|
| 479 |
# ---------------------------------------------------------------------------
|
|
|
|
| 488 |
|
| 489 |
with gr.Blocks(title="Skin Disease Detection β Demo") as demo:
|
| 490 |
gr.Markdown("# π©Ί Skin Disease Detection")
|
| 491 |
+
gr.Markdown(f"_{PAGE_DISCLAIMER}_")
|
| 492 |
|
| 493 |
with gr.Tab("π· Quick Image Check"):
|
| 494 |
with gr.Row():
|
|
|
|
| 512 |
multimodal=True,
|
| 513 |
description=(
|
| 514 |
"Talk through what you're noticing, and optionally attach a photo (π). "
|
| 515 |
+
"The assistant will ask several follow-up questions before giving a preliminary "
|
| 516 |
+
"read with a Low/Medium/High severity call β the more you share, the more "
|
| 517 |
+
"targeted that read can be."
|
| 518 |
),
|
| 519 |
)
|
| 520 |
|
| 521 |
+
with gr.Tab("π Model Performance"):
|
| 522 |
+
performance_reports = load_performance_reports()
|
| 523 |
+
if not performance_reports:
|
| 524 |
+
gr.Markdown(
|
| 525 |
+
"_No model performance data yet._\n\n"
|
| 526 |
+
"Run either fine-tuning notebook in this repo, then download the "
|
| 527 |
+
"`*_metrics.json` and `*_confusion_matrix.png` files it produces and place "
|
| 528 |
+
"them in this Space's `assets/` folder. This tab picks them up automatically "
|
| 529 |
+
"on the next restart."
|
| 530 |
+
)
|
| 531 |
+
else:
|
| 532 |
+
for report in performance_reports:
|
| 533 |
+
gr.Markdown(format_performance_markdown(report))
|
| 534 |
+
if report["image_path"]:
|
| 535 |
+
gr.Image(value=report["image_path"], label="Confusion matrix (test set)", show_label=True)
|
| 536 |
+
|
| 537 |
+
with gr.Tab("βΉοΈ About"):
|
| 538 |
+
gr.Markdown(ABOUT_MARKDOWN)
|
| 539 |
+
|
| 540 |
gr.Markdown(
|
| 541 |
"---\nBuilt with π€ Transformers + Gradio. "
|
| 542 |
f"Image model: `{IMAGE_MODEL_ID}` Β· Chat model: `{CHAT_MODEL_ID}`"
|