Pro-Coder commited on
Commit
b842c0f
Β·
verified Β·
1 Parent(s): eaa8185

Upload 2 files

Browse files
Files changed (2) hide show
  1. README.md +117 -25
  2. app.py +206 -25
README.md CHANGED
@@ -12,37 +12,75 @@ license: apache-2.0
12
 
13
  # 🩺 Skin Disease Detection β€” Demo
14
 
15
- A demo Space where users can:
16
- - **Upload a skin image** (or click a bundled test image) and get an AI classifier's
17
- prediction across common skin-lesion categories, plus a plain-language explanation
18
- from an LLM.
19
- - **Chat with an assistant** about skin-health topics in general terms.
20
-
21
- > ⚠️ **This is an educational demo, not a medical device.** It does not diagnose
22
- > anyone. Always see a licensed dermatologist or doctor for real concerns.
 
 
 
 
 
 
 
 
23
 
24
  ## How it works
25
 
26
  | Component | What it does | Default model |
27
  |---|---|---|
28
- | Image classifier | Predicts a skin-lesion category from a photo | [`Anwarkh1/Skin_Cancer-Image_Classification`](https://huggingface.co/Anwarkh1/Skin_Cancer-Image_Classification) (ViT, trained on a HAM10000-derived dataset) |
29
- | Chat / explanation | Turns predictions into plain language, answers general questions | `HuggingFaceH4/zephyr-7b-beta` via the Hugging Face Inference API |
30
-
31
- Both model IDs are configurable via Space variables β€” see below.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
32
 
33
  ## πŸš€ Deploy in 3 steps
34
 
35
  1. **Create a new Space**
36
  Go to [huggingface.co/new-space](https://huggingface.co/new-space) β†’
37
- choose **Gradio** as the SDK β†’ CPU basic hardware is fine for the classifier.
 
 
38
 
39
  2. **Upload these files**
40
- Upload `app.py`, `requirements.txt`, this `README.md`, and the `examples/`
41
- folder (if you generated test images β€” see below) to the Space repo,
42
- either via the web UI ("Add file") or:
 
43
  ```bash
44
  git clone https://huggingface.co/spaces/<your-username>/<your-space-name>
45
- cp -r app.py requirements.txt README.md examples <your-space-name>/
46
  cd <your-space-name>
47
  git add . && git commit -m "Initial commit" && git push
48
  ```
@@ -79,20 +117,74 @@ Set these as Space **variables** (not secret, unless noted) to customize:
79
 
80
  - `IMAGE_MODEL_ID` β€” any Hugging Face image-classification model compatible
81
  with `transformers.pipeline("image-classification", ...)`.
 
 
 
 
82
  - `CHAT_MODEL_ID` β€” any chat-completion-capable model available via the HF
83
- Inference API.
84
  - `HF_TOKEN` *(secret)* β€” required for the chat tab and the LLM explanations.
85
 
86
- ## 🩹 Limitations & responsible use
87
-
88
- - The bundled classifier was trained on a research dataset (HAM10000-derived
89
- dermatoscopic images) and has real, published accuracy limits β€” treat its
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
90
  output as a talking point, not a result.
91
  - Performance depends heavily on image quality, lighting, and skin tone
92
- representation in the training data.
 
 
 
93
  - This app must not be used as a substitute for professional medical
94
- evaluation, and the chatbot is instructed to avoid giving diagnoses,
95
- treatments, or dosages.
 
96
 
97
  ## Local development
98
 
 
12
 
13
  # 🩺 Skin Disease Detection β€” Demo
14
 
15
+ A demo Space with four tabs:
16
+ - **πŸ“· Quick Image Check** β€” upload a photo (or click a test image), get a
17
+ classifier prediction plus an LLM-generated plain-language assessment with
18
+ an explicit **Low / Medium / High severity** call.
19
+ - **πŸ’¬ Chat Assistant** β€” a multi-turn triage conversation. It asks about
20
+ onset, evolution, symptoms, triggers, and history over several exchanges
21
+ before giving a narrowed, severity-rated preliminary read β€” more targeted
22
+ than a single photo alone could support, without claiming false certainty.
23
+ - **πŸ“Š Model Performance** β€” real test-set metrics (accuracy, per-class
24
+ precision/recall/F1, confusion matrix) for whichever fine-tuned model(s)
25
+ are backing the app, pulled from `assets/`.
26
+ - **ℹ️ About** β€” what this is, what models it uses, and its limitations.
27
+
28
+ > ⚠️ **This is a research/portfolio demo, not a medical device.** It does not
29
+ > diagnose anyone with certainty. Always see a licensed dermatologist or
30
+ > doctor for real concerns.
31
 
32
  ## How it works
33
 
34
  | Component | What it does | Default model |
35
  |---|---|---|
36
+ | Image classifier | Predicts a skin condition category from a photo | [`Anwarkh1/Skin_Cancer-Image_Classification`](https://huggingface.co/Anwarkh1/Skin_Cancer-Image_Classification) (ViT, 7-class HAM10000) β€” recommended upgrade: train `finetune_unified_skin_model.ipynb` (24-class, see below) and point `IMAGE_MODEL_ID` at it |
37
+ | Second-opinion lesion model *(optional, redundant if using the unified model)* | Re-checks images the primary model flags as an ambiguous melanoma/mole label | Off by default β€” set `LESION_MODEL_ID` to a HAM10000-based model to enable |
38
+ | Chat / explanation | Multi-turn triage with enforced minimum question rounds, turns predictions into plain language | `Qwen/Qwen2.5-7B-Instruct` via the Hugging Face Inference Providers router |
39
+
40
+ All three model IDs are configurable via Space variables β€” see below.
41
+
42
+ ## Multi-round chat behavior
43
+
44
+ The chat tab doesn't just *ask* the model to have a longer conversation β€”
45
+ `app.py` counts user turns explicitly and injects a stage-aware instruction
46
+ on every call (`MIN_ROUNDS_BEFORE_ASSESSMENT = 4` by default, adjustable
47
+ in code): before that many exchanges, the model is told to keep gathering
48
+ information; after, it's told enough rounds have happened and to give its
49
+ assessment. Red-flag symptoms override this and get addressed immediately
50
+ regardless of turn count.
51
+
52
+ ## Severity levels
53
+
54
+ Every assessment states one of three levels explicitly:
55
+ - πŸ”΄ **High** β€” cancer-related, ambiguous-but-could-be-serious, systemic/
56
+ autoimmune conditions, or red-flag symptoms. Recommends seeing a doctor soon.
57
+ - 🟑 **Medium** β€” likely needs a proper diagnosis and prescription
58
+ (infections, infestations). Recommends a doctor visit, not urgent.
59
+ - 🟒 **Low** β€” typically manageable with general skin care. Doctor visit
60
+ optional.
61
+
62
+ Tiering is **keyword-based** (see `classify_tier()` in `app.py`), so it works
63
+ across different classifier taxonomies without hardcoding every exact label
64
+ string β€” including correctly flagging DermNet's mixed
65
+ *"Melanoma Skin Cancer, Nevi and Moles"* label as High by default, since that
66
+ one label can't distinguish the dangerous case from the harmless one.
67
 
68
  ## πŸš€ Deploy in 3 steps
69
 
70
  1. **Create a new Space**
71
  Go to [huggingface.co/new-space](https://huggingface.co/new-space) β†’
72
+ choose **Gradio** as the SDK β†’ CPU basic hardware is fine for the classifier
73
+ (or ZeroGPU if you want faster inference β€” see the `@spaces.GPU` decorator
74
+ already applied to `diagnose_image`).
75
 
76
  2. **Upload these files**
77
+ Upload `app.py`, `requirements.txt`, this `README.md`, the `examples/`
78
+ folder (if you generated test images), and the `assets/` folder (if you
79
+ exported model performance metrics β€” see below), either via the web UI
80
+ ("Add file") or:
81
  ```bash
82
  git clone https://huggingface.co/spaces/<your-username>/<your-space-name>
83
+ cp -r app.py requirements.txt README.md examples assets <your-space-name>/
84
  cd <your-space-name>
85
  git add . && git commit -m "Initial commit" && git push
86
  ```
 
117
 
118
  - `IMAGE_MODEL_ID` β€” any Hugging Face image-classification model compatible
119
  with `transformers.pipeline("image-classification", ...)`.
120
+ - `LESION_MODEL_ID` *(optional)* β€” a second, pigmented-lesion-specific model
121
+ (like the HAM10000 fine-tune below) used automatically as a second opinion
122
+ when the primary model's top prediction is an ambiguous melanoma/mole label.
123
+ Leave unset to disable.
124
  - `CHAT_MODEL_ID` β€” any chat-completion-capable model available via the HF
125
+ Inference Providers router.
126
  - `HF_TOKEN` *(secret)* β€” required for the chat tab and the LLM explanations.
127
 
128
+ ## 🧠 Fine-tuning your own model
129
+
130
+ Three self-contained Colab notebooks are included:
131
+
132
+ ### `finetune_unified_skin_model.ipynb` β€” ⭐ recommended: 24-class unified model
133
+ - **Retrained from scratch**, not resumed β€” fresh weights each run
134
+ - Merges DermNet's 22 clean categories with HAM10000's melanoma and nevi
135
+ as their own separate classes, **replacing DermNet's ambiguous combined
136
+ "Melanoma Skin Cancer, Nevi and Moles" label entirely** β€” melanoma finally
137
+ gets a clean training signal instead of being merged with benign moles
138
+ - Early stopping (up to 14 epochs, stops automatically once validation
139
+ macro-F1 plateaus) + cosine LR schedule β€” addresses the previous
140
+ DermNet-only run's validation loss not having converged at 6 epochs
141
+ - Melanoma threshold-tuning (now meaningful, since melanoma is unambiguous)
142
+ - Exports `unified_metrics.json` + `unified_confusion_matrix.png`
143
+ - **Honest tradeoff:** melanoma/nevi images come from HAM10000's
144
+ dermatoscope close-ups, not regular phone photos like the rest of the
145
+ dataset β€” a real domain gap, flagged in the notebook rather than hidden
146
+
147
+ ### `finetune_skin_lesion_model.ipynb` β€” 7-class pigmented lesions (HAM10000 only)
148
+ - Useful if you want a smaller, faster, cancer-focused model specifically
149
+ - class-weighted loss, melanoma recall tracking, threshold tuning
150
+ - exports `ham10000_metrics.json` + `ham10000_confusion_matrix.png`
151
+
152
+ ### `finetune_dermnet_23class.ipynb` β€” 23-class DermNet only (superseded by the unified notebook above, kept for reference)
153
+ - Same as the unified notebook's DermNet portion, but keeps DermNet's
154
+ original ambiguous melanoma/nevi label instead of replacing it
155
+ - exports `dermnet_metrics.json` + `dermnet_confusion_matrix.png`
156
+
157
+ **To use any of them:** upload the `.ipynb` to
158
+ [Google Colab](https://colab.research.google.com), set **Runtime β†’ Change
159
+ runtime type β†’ T4 GPU**, and run cells top to bottom. The unified notebook
160
+ takes roughly 60–100 minutes on a free T4 depending on when early stopping
161
+ kicks in; keep the tab active so the session doesn't disconnect.
162
+
163
+ **Before treating any of these as more than a research project**, see the
164
+ "Next steps" section at the end of each notebook β€” a higher benchmark score
165
+ is not the same as a clinically validated model.
166
+
167
+ ## πŸ“Š Populating the Model Performance tab
168
+
169
+ After running a fine-tuning notebook, download the two files it exports
170
+ (`*_metrics.json` and `*_confusion_matrix.png`) from Colab's file browser and
171
+ place them in this Space's `assets/` folder, then redeploy. The tab picks up
172
+ any matching pair automatically β€” you can have both the HAM10000 and DermNet
173
+ reports side by side if you've fine-tuned both.
174
+
175
+ ## Limitations & responsible use
176
+
177
+ - Neither bundled classifier has been clinically validated β€” treat their
178
  output as a talking point, not a result.
179
  - Performance depends heavily on image quality, lighting, and skin tone
180
+ representation in the training data; public dermatology datasets skew
181
+ toward lighter skin tones.
182
+ - DermNet's melanoma/nevi label limitation (above) means that category's
183
+ severity is intentionally conservative rather than precise.
184
  - This app must not be used as a substitute for professional medical
185
+ evaluation. The chatbot is instructed to narrow toward likely categories
186
+ and state a severity level, but never to claim certainty, prescribe
187
+ treatment, or give dosages.
188
 
189
  ## Local development
190
 
app.py CHANGED
@@ -36,7 +36,11 @@ CHAT_MODEL_ID = os.environ.get("CHAT_MODEL_ID", "Qwen/Qwen2.5-7B-Instruct")
36
  HF_TOKEN = os.environ.get("HF_TOKEN") # add as a Space SECRET to enable the chatbot
37
  ROUTER_URL = "https://router.huggingface.co/v1/chat/completions"
38
 
39
-
 
 
 
 
40
 
41
  # Optional second classifier specifically for pigmented-lesion cases (HAM10000-based,
42
  # where melanoma is its own clean label). Useful because broader taxonomies like
@@ -64,10 +68,10 @@ PROMPT_DOCTOR_KEYWORDS = [
64
  # manageable/chronic conditions (acne, eczema, psoriasis, hives, hair loss,
65
  # benign tumors, common moles, etc.) where general skin-care guidance fits.
66
 
67
- TIER_LABELS = {
68
- "urgent": "higher-risk β€” recommend seeing a dermatologist soon",
69
- "prompt_doctor_visit": "needs a proper diagnosis/prescription β€” recommend seeing a doctor, not urgent-emergency",
70
- "general_care": "typically manageable β€” general skin-care guidance fits, doctor visit optional",
71
  }
72
 
73
 
@@ -83,6 +87,12 @@ def classify_tier(label):
83
  return "general_care"
84
 
85
 
 
 
 
 
 
 
86
  # Friendly one-line descriptions for labels we recognize exactly (both the
87
  # original 7-class HAM10000 set and common DermNet-style labels). Purely
88
  # cosmetic β€” if a label isn't here, the tier-based guidance still works fine
@@ -97,26 +107,28 @@ KNOWN_DESCRIPTIONS = {
97
  "vascular lesions": "Blood-vessel-related marks such as angiomas.",
98
  }
99
 
100
- SYSTEM_PROMPT = """You are a warm, knowledgeable skin-health triage assistant. You are not a doctor and cannot diagnose anyone β€” but your job is to be genuinely useful, not to hide behind constant disclaimers.
101
 
102
- CONVERSATION STYLE
103
- - Have a real back-and-forth. Ask about one or two things at a time, not a long checklist at once.
104
- - Useful things to learn over the conversation (don't force all of them if the user has already told you, or if it's clearly unnecessary):
105
- - How long they've had it / when they first noticed it
106
- - Whether it's changed recently in size, shape, or color
107
- - Symptoms: itching, bleeding, pain, crusting, oozing, discharge, fever
108
- - Personal or family history of relevant conditions
109
- - Relevant exposure history (sun, new products, contacts, travel, bites)
110
- - Where on the body it is
111
- - If the user shared an image, you'll also receive the image classifier's findings as extra context, including an urgency tier (not shown verbatim to the user) β€” weave that in naturally rather than reading out raw percentages.
 
112
 
113
  GIVING A PRELIMINARY ASSESSMENT
114
- Once you have enough context (often after 2-5 exchanges, sooner if the user just wants a quick read, or immediately if red flags are already obvious), give a clear preliminary assessment using three tiers:
115
- - **URGENT** (cancer-related findings, or things like bullous/autoimmune/systemic conditions, cellulitis, vasculitis) OR any red-flag symptoms (rapid growth, irregular/changing borders, multiple colors, asymmetry, a sore that won't heal, bleeding, spreading redness, fever) β€” be direct: recommend seeing a dermatologist or doctor soon, and explain briefly why those signs matter. Don't soften this into vague "keep an eye on it" language. Note: some categories (like a classifier label that groups "melanoma" together with ordinary moles) genuinely can't be told apart by the model β€” when that ambiguity exists, say so plainly and default to recommending it get checked rather than assuming it's benign.
116
- - **NEEDS A DOCTOR VISIT, NOT URGENT** (infections β€” fungal, viral, bacterial, STD-related, infestations like scabies) β€” explain that these generally need a proper diagnosis and often a prescription to clear up, so a doctor visit is the right next step, but it's not an emergency. For anything STD-related, stay factual and non-judgmental, and emphasize in-person testing rather than guessing from a photo.
117
- - **TYPICALLY MANAGEABLE** (acne, eczema, psoriasis, hives, hair loss, common moles, benign growths, contact dermatitis, etc.) with no red flags β€” explain what that category usually is, and give general skin-care guidance: gentle skincare habits, sun protection, not picking or scratching, and watching for changes. It's fine to mention a dermatologist visit is a reasonable option too, especially if they're unsure, worried, or it's not improving β€” but don't invent a treatment plan or prescribe anything.
 
 
118
  - Never give specific medications, dosages, or treatment prescriptions β€” general skin-care habits are fine; treating a self- or AI-identified condition with anything beyond general care is not.
119
- - You don't need to repeat a formal warning block every message. Say once, naturally, as part of your assessment β€” not as a bolted-on disclaimer β€” that this is a preliminary read from a conversation and an image classifier, and that an in-person exam is what actually confirms things. Then move on; don't repeat it every turn.
120
 
121
  URGENT SITUATIONS
122
  If the user describes something urgent β€” rapidly growing lesion, uncontrolled bleeding, signs of spreading infection, fever with a skin issue, severe pain β€” tell them clearly to seek in-person or emergency care promptly, regardless of how many turns you've had.
@@ -227,6 +239,7 @@ def diagnose_image(image):
227
  return f"**Model error:** {e}\n\nMake sure `IMAGE_MODEL_ID` is a valid image-classification model."
228
 
229
  lines = ["### πŸ”¬ Classifier prediction\n"] + lines_list
 
230
 
231
  if HF_TOKEN:
232
  context = build_classifier_context(preds)
@@ -234,8 +247,11 @@ def diagnose_image(image):
234
  f"{context}\n\n"
235
  "The user just uploaded a single image with no conversation yet. Give a short "
236
  "preliminary assessment following your instructions: plain-language explanation of "
237
- "the top finding, urgency-tiered guidance, and a brief natural mention that an "
238
- "in-person dermatologist exam is what actually confirms things. Keep it to 4-6 sentences."
 
 
 
239
  )
240
  try:
241
  explanation = chat_completion(
@@ -264,6 +280,9 @@ def diagnose_image(image):
264
  # ---------------------------------------------------------------------------
265
  # Chat tab β€” multimodal, multi-turn triage
266
  # ---------------------------------------------------------------------------
 
 
 
267
  def _extract_text_and_image(message):
268
  """Normalize gr.ChatInterface(multimodal=True) message input into (text, image_path_or_None)."""
269
  if isinstance(message, dict):
@@ -274,6 +293,31 @@ def _extract_text_and_image(message):
274
  return str(message), None
275
 
276
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
277
  def chat_respond(message, history):
278
  if not HF_TOKEN:
279
  return (
@@ -307,7 +351,10 @@ def chat_respond(message, history):
307
  except Exception as e:
308
  user_content = f"(Image analysis failed: {e})\n\nUser's message: {text}"
309
 
310
- messages.append({"role": "user", "content": user_content})
 
 
 
311
 
312
  try:
313
  return chat_completion(messages, max_tokens=500)
@@ -315,6 +362,118 @@ def chat_respond(message, history):
315
  return f"Sorry, I hit an error talking to the model: {e}"
316
 
317
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
318
  # ---------------------------------------------------------------------------
319
  # UI
320
  # ---------------------------------------------------------------------------
@@ -329,6 +488,7 @@ if os.path.isdir(EXAMPLES_DIR):
329
 
330
  with gr.Blocks(title="Skin Disease Detection β€” Demo") as demo:
331
  gr.Markdown("# 🩺 Skin Disease Detection")
 
332
 
333
  with gr.Tab("πŸ“· Quick Image Check"):
334
  with gr.Row():
@@ -352,10 +512,31 @@ with gr.Blocks(title="Skin Disease Detection β€” Demo") as demo:
352
  multimodal=True,
353
  description=(
354
  "Talk through what you're noticing, and optionally attach a photo (πŸ“Ž). "
355
- "The assistant will ask a few follow-up questions before giving a preliminary read."
 
 
356
  ),
357
  )
358
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
359
  gr.Markdown(
360
  "---\nBuilt with πŸ€— Transformers + Gradio. "
361
  f"Image model: `{IMAGE_MODEL_ID}` Β· Chat model: `{CHAT_MODEL_ID}`"
 
36
  HF_TOKEN = os.environ.get("HF_TOKEN") # add as a Space SECRET to enable the chatbot
37
  ROUTER_URL = "https://router.huggingface.co/v1/chat/completions"
38
 
39
+ PAGE_DISCLAIMER = (
40
+ "This tool is a research/portfolio project, not a medical device. It hasn't "
41
+ "been clinically validated. Use it to get informed and organize your thoughts "
42
+ "before a real appointment β€” not as a substitute for one."
43
+ )
44
 
45
  # Optional second classifier specifically for pigmented-lesion cases (HAM10000-based,
46
  # where melanoma is its own clean label). Useful because broader taxonomies like
 
68
  # manageable/chronic conditions (acne, eczema, psoriasis, hives, hair loss,
69
  # benign tumors, common moles, etc.) where general skin-care guidance fits.
70
 
71
+ SEVERITY = {
72
+ "urgent": {"label": "High", "emoji": "πŸ”΄", "note": "recommend seeing a dermatologist/doctor soon"},
73
+ "prompt_doctor_visit": {"label": "Medium", "emoji": "🟑", "note": "needs a proper diagnosis/prescription, not an emergency"},
74
+ "general_care": {"label": "Low", "emoji": "🟒", "note": "typically manageable with general skin care"},
75
  }
76
 
77
 
 
87
  return "general_care"
88
 
89
 
90
+ def severity_line(label):
91
+ tier = classify_tier(label)
92
+ s = SEVERITY[tier]
93
+ return f"{s['emoji']} **Severity: {s['label']}** β€” {s['note']}"
94
+
95
+
96
  # Friendly one-line descriptions for labels we recognize exactly (both the
97
  # original 7-class HAM10000 set and common DermNet-style labels). Purely
98
  # cosmetic β€” if a label isn't here, the tier-based guidance still works fine
 
107
  "vascular lesions": "Blood-vessel-related marks such as angiomas.",
108
  }
109
 
110
+ SYSTEM_PROMPT = """You are a warm, knowledgeable skin-health triage assistant. You are not a doctor and cannot diagnose anyone with certainty β€” but your job is to be genuinely useful and specific, not to hide behind vague hedging or constant disclaimers.
111
 
112
+ CONVERSATION STYLE β€” GO DEEPER BEFORE ASSESSING
113
+ Have a real back-and-forth, one or two questions at a time. Aim to gather a fuller picture than a quick glance would give β€” typically 5-8 exchanges before your assessment, more if the picture is still unclear, fewer only if the user clearly just wants a fast read or red flags are already obvious. Useful ground to cover (skip anything already answered or clearly irrelevant):
114
+ - Onset: how long they've had it, how it started
115
+ - Evolution: how it's changed over time β€” size, shape, color, texture
116
+ - Symptoms: itching, bleeding, pain, crusting, oozing, discharge, fever, swelling
117
+ - Pattern: is it one spot or spreading, single or multiple, symmetric or not
118
+ - Triggers/exposures: new products, plants, insect bites, contacts, travel, sun exposure
119
+ - History: prior similar episodes, what's been tried already (and whether it helped), relevant personal/family medical history, allergies
120
+ - Location and distribution on the body
121
+ The goal of the extra rounds is to genuinely narrow the differential, not to interrogate β€” keep it conversational, and if the user gives a rich answer that covers several of these at once, don't force redundant questions.
122
+ If the user shared an image, you'll also receive the image classifier's findings as extra context, including an urgency tier (not shown verbatim to the user) β€” weave that in naturally rather than reading out raw percentages.
123
 
124
  GIVING A PRELIMINARY ASSESSMENT
125
+ Once you have real depth of context, give a clear, specific preliminary assessment:
126
+ - Name the 1-2 most likely categories given everything discussed, in plain language, and briefly say why (which symptoms/pattern point that way). It's fine β€” good, even β€” to be specific about what you think is most likely. What you must NOT do is claim certainty: use "most consistent with" / "this pattern often points to," never "you have X" or "this is definitely X." A real diagnosis needs an in-person exam and sometimes tests (biopsy, culture, bloodwork) that no photo or conversation can replace β€” say this once, naturally, as part of the assessment, not as a repeated disclaimer.
127
+ - Always state an explicit severity level as part of the assessment: **Low**, **Medium**, or **High** β€” plus one line on what that means for next steps. Base it on:
128
+ - **High** = cancer-related or ambiguous-but-could-be-cancer findings, autoimmune/systemic/bullous conditions, cellulitis, vasculitis, or any red-flag symptoms (rapid growth, irregular/changing borders, multiple colors, asymmetry, a sore that won't heal, bleeding, spreading redness, fever) β†’ recommend seeing a dermatologist or doctor soon, directly, without softening it into "keep an eye on it." Note explicitly if the classifier's category can't distinguish something dangerous from something benign (e.g. a label that groups melanoma with ordinary moles) β€” when that ambiguity exists, say so and default to High rather than assuming benign.
129
+ - **Medium** = things that need a proper diagnosis and often a prescription to clear up (fungal, viral, bacterial infections, STD-related, infestations like scabies) but aren't emergencies β†’ recommend a doctor visit, explain why self-treatment usually doesn't fully work here. For anything STD-related, stay factual and non-judgmental, and point toward in-person testing rather than guessing from a photo.
130
+ - **Low** = typically manageable conditions (acne, eczema, psoriasis, hives, hair loss, common moles, benign growths, contact dermatitis) with no red flags β†’ explain what it usually is, and give general skin-care guidance: gentle skincare habits, sun protection, not picking/scratching, watching for changes. Still fine to mention a doctor visit is reasonable if they're unsure, worried, or it's not improving.
131
  - Never give specific medications, dosages, or treatment prescriptions β€” general skin-care habits are fine; treating a self- or AI-identified condition with anything beyond general care is not.
 
132
 
133
  URGENT SITUATIONS
134
  If the user describes something urgent β€” rapidly growing lesion, uncontrolled bleeding, signs of spreading infection, fever with a skin issue, severe pain β€” tell them clearly to seek in-person or emergency care promptly, regardless of how many turns you've had.
 
239
  return f"**Model error:** {e}\n\nMake sure `IMAGE_MODEL_ID` is a valid image-classification model."
240
 
241
  lines = ["### πŸ”¬ Classifier prediction\n"] + lines_list
242
+ lines.append(f"\n{severity_line(preds[0]['label'])}")
243
 
244
  if HF_TOKEN:
245
  context = build_classifier_context(preds)
 
247
  f"{context}\n\n"
248
  "The user just uploaded a single image with no conversation yet. Give a short "
249
  "preliminary assessment following your instructions: plain-language explanation of "
250
+ "the top finding(s), an explicit Low/Medium/High severity call, and a brief natural "
251
+ "mention that an in-person exam is what actually confirms things. Note that without "
252
+ "any conversation history you have less context than usual β€” you can still give your "
253
+ "best read, but you may want to suggest the Chat Assistant for a fuller picture if the "
254
+ "case seems ambiguous. Keep it to 5-7 sentences."
255
  )
256
  try:
257
  explanation = chat_completion(
 
280
  # ---------------------------------------------------------------------------
281
  # Chat tab β€” multimodal, multi-turn triage
282
  # ---------------------------------------------------------------------------
283
+ MIN_ROUNDS_BEFORE_ASSESSMENT = 4 # user turns, not counting the current one
284
+
285
+
286
  def _extract_text_and_image(message):
287
  """Normalize gr.ChatInterface(multimodal=True) message input into (text, image_path_or_None)."""
288
  if isinstance(message, dict):
 
293
  return str(message), None
294
 
295
 
296
+ def _count_user_turns(messages):
297
+ return sum(1 for m in messages if m.get("role") == "user")
298
+
299
+
300
+ def _stage_note(user_turn_count):
301
+ """Injected each turn so multi-round behavior is enforced by code, not just
302
+ requested once in the system prompt β€” the model gets a fresh reminder of
303
+ what stage the conversation is in on every single call."""
304
+ if user_turn_count < MIN_ROUNDS_BEFORE_ASSESSMENT:
305
+ return (
306
+ f"[Internal note β€” not from the user: this is exchange {user_turn_count} of at least "
307
+ f"{MIN_ROUNDS_BEFORE_ASSESSMENT} before a full assessment, UNLESS the user has already "
308
+ "described clear red-flag symptoms (rapid growth, bleeding, severe pain, spreading "
309
+ "infection, fever) in which case address that urgently right now regardless of turn "
310
+ "count. Otherwise, continue asking focused follow-up questions β€” don't jump to a full "
311
+ "Low/Medium/High assessment yet.]"
312
+ )
313
+ return (
314
+ f"[Internal note β€” not from the user: this is exchange {user_turn_count}, enough rounds "
315
+ "have happened. If you have a reasonably clear picture, give your preliminary assessment "
316
+ "now with an explicit Low/Medium/High severity call. If something important is still "
317
+ "genuinely unclear, you may ask one more focused question first, but don't stall further.]"
318
+ )
319
+
320
+
321
  def chat_respond(message, history):
322
  if not HF_TOKEN:
323
  return (
 
351
  except Exception as e:
352
  user_content = f"(Image analysis failed: {e})\n\nUser's message: {text}"
353
 
354
+ # current turn counts too, since the model is about to respond to it
355
+ current_turn_count = _count_user_turns(messages) + 1
356
+ stage_note = _stage_note(current_turn_count)
357
+ messages.append({"role": "user", "content": f"{stage_note}\n\n{user_content}"})
358
 
359
  try:
360
  return chat_completion(messages, max_tokens=500)
 
362
  return f"Sorry, I hit an error talking to the model: {e}"
363
 
364
 
365
+ # ---------------------------------------------------------------------------
366
+ # Model Performance tab data loading
367
+ # ---------------------------------------------------------------------------
368
+ ASSETS_DIR = "assets"
369
+
370
+
371
+ def load_performance_reports():
372
+ """Scan assets/ for *_metrics.json files and pair each with its confusion-matrix
373
+ image if present. Returns a list of dicts; empty list if nothing's been uploaded yet."""
374
+ import json
375
+
376
+ reports = []
377
+ if not os.path.isdir(ASSETS_DIR):
378
+ return reports
379
+ for fname in sorted(os.listdir(ASSETS_DIR)):
380
+ if not fname.endswith("_metrics.json"):
381
+ continue
382
+ prefix = fname[: -len("_metrics.json")]
383
+ try:
384
+ with open(os.path.join(ASSETS_DIR, fname)) as f:
385
+ data = json.load(f)
386
+ except Exception:
387
+ continue
388
+ image_path = None
389
+ for ext in (".png", ".jpg", ".jpeg"):
390
+ candidate = os.path.join(ASSETS_DIR, f"{prefix}_confusion_matrix{ext}")
391
+ if os.path.isfile(candidate):
392
+ image_path = candidate
393
+ break
394
+ reports.append({"data": data, "image_path": image_path})
395
+ return reports
396
+
397
+
398
+ def format_performance_markdown(report):
399
+ data = report["data"]
400
+ lines = [f"### {data.get('model_name', 'Model')}"]
401
+ acc = data.get("accuracy")
402
+ f1m = data.get("f1_macro")
403
+ if acc is not None:
404
+ lines.append(f"**Accuracy:** {acc*100:.1f}% Β· **Macro F1:** {f1m*100:.1f}%" if f1m is not None else f"**Accuracy:** {acc*100:.1f}%")
405
+
406
+ per_class = data.get("per_class", {})
407
+ if per_class:
408
+ lines.append("\n| Class | Precision | Recall | F1 | Support |")
409
+ lines.append("|---|---|---|---|---|")
410
+ for cls, m in per_class.items():
411
+ lines.append(
412
+ f"| {cls} | {m.get('precision', 0)*100:.1f}% | {m.get('recall', 0)*100:.1f}% "
413
+ f"| {m.get('f1', 0)*100:.1f}% | {int(m.get('support', 0))} |"
414
+ )
415
+
416
+ notes = data.get("notes")
417
+ if notes:
418
+ lines.append(f"\n_Note: {notes}_")
419
+
420
+ return "\n".join(lines)
421
+
422
+
423
+ ABOUT_MARKDOWN = f"""
424
+ ## About this project
425
+
426
+ A research/portfolio demo exploring how far an image classifier + LLM chat
427
+ assistant can go toward useful skin-condition triage β€” while being explicit
428
+ about what it isn't: a validated medical device.
429
+
430
+ ### What's inside
431
+ - **Quick Image Check** β€” upload a photo, get a classifier prediction plus
432
+ an LLM-generated plain-language assessment with an explicit Low/Medium/High
433
+ severity call.
434
+ - **Chat Assistant** β€” a multi-turn conversation that asks about onset,
435
+ evolution, symptoms, triggers, and history before giving a more targeted
436
+ preliminary read than a single photo alone could support.
437
+ - **Model Performance** β€” the actual test-set metrics for whichever
438
+ fine-tuned model(s) are currently backing this app, not just marketing
439
+ claims about accuracy.
440
+
441
+ ### Models
442
+ - **Image classifier:** configurable via the `IMAGE_MODEL_ID` variable.
443
+ Two purpose-built options are included as fine-tuning notebooks in this
444
+ repo: a 7-class pigmented-lesion model (HAM10000/ISIC, ViT-Base) and a
445
+ 23-class common-skin-condition model (DermNet, ConvNeXt-Base).
446
+ - **Optional second-opinion lesion model:** `LESION_MODEL_ID`, used
447
+ automatically when the primary model's top prediction is an ambiguous
448
+ mixed melanoma/mole-type label, to get a model that treats melanoma as
449
+ its own distinct class.
450
+ - **Chat model:** configurable via `CHAT_MODEL_ID`, served through Hugging
451
+ Face's Inference Providers router.
452
+
453
+ ### Severity levels
454
+ - πŸ”΄ **High** β€” cancer-related, ambiguous-but-could-be-serious, systemic/
455
+ autoimmune, or red-flag symptoms present. See a doctor soon.
456
+ - 🟑 **Medium** β€” likely needs a proper diagnosis and prescription
457
+ (infections, infestations). See a doctor, not an emergency.
458
+ - 🟒 **Low** β€” typically manageable with general skin care. Doctor visit
459
+ optional, especially if unsure or it's not improving.
460
+
461
+ ### Known limitations (read this before trusting the output)
462
+ - **Not clinically validated.** Benchmark accuracy on a public dataset is
463
+ not the same as real-world diagnostic reliability.
464
+ - **Dataset label limitations.** The DermNet taxonomy combines melanoma
465
+ with benign moles under one label β€” the app defaults that whole category
466
+ to High severity rather than pretending to distinguish them.
467
+ - **Skin-tone representation.** Public dermatology datasets skew toward
468
+ lighter skin tones; performance on underrepresented skin tones is
469
+ unverified without a dedicated bias audit.
470
+ - **No clinical validation, IRB review, or regulatory clearance** has been
471
+ done. This should inform, not replace, professional care.
472
+
473
+ {PAGE_DISCLAIMER}
474
+ """
475
+
476
+
477
  # ---------------------------------------------------------------------------
478
  # UI
479
  # ---------------------------------------------------------------------------
 
488
 
489
  with gr.Blocks(title="Skin Disease Detection β€” Demo") as demo:
490
  gr.Markdown("# 🩺 Skin Disease Detection")
491
+ gr.Markdown(f"_{PAGE_DISCLAIMER}_")
492
 
493
  with gr.Tab("πŸ“· Quick Image Check"):
494
  with gr.Row():
 
512
  multimodal=True,
513
  description=(
514
  "Talk through what you're noticing, and optionally attach a photo (πŸ“Ž). "
515
+ "The assistant will ask several follow-up questions before giving a preliminary "
516
+ "read with a Low/Medium/High severity call β€” the more you share, the more "
517
+ "targeted that read can be."
518
  ),
519
  )
520
 
521
+ with gr.Tab("πŸ“Š Model Performance"):
522
+ performance_reports = load_performance_reports()
523
+ if not performance_reports:
524
+ gr.Markdown(
525
+ "_No model performance data yet._\n\n"
526
+ "Run either fine-tuning notebook in this repo, then download the "
527
+ "`*_metrics.json` and `*_confusion_matrix.png` files it produces and place "
528
+ "them in this Space's `assets/` folder. This tab picks them up automatically "
529
+ "on the next restart."
530
+ )
531
+ else:
532
+ for report in performance_reports:
533
+ gr.Markdown(format_performance_markdown(report))
534
+ if report["image_path"]:
535
+ gr.Image(value=report["image_path"], label="Confusion matrix (test set)", show_label=True)
536
+
537
+ with gr.Tab("ℹ️ About"):
538
+ gr.Markdown(ABOUT_MARKDOWN)
539
+
540
  gr.Markdown(
541
  "---\nBuilt with πŸ€— Transformers + Gradio. "
542
  f"Image model: `{IMAGE_MODEL_ID}` Β· Chat model: `{CHAT_MODEL_ID}`"