| --- |
| title: README |
| emoji: 🔬 |
| colorFrom: blue |
| colorTo: indigo |
| sdk: static |
| pinned: false |
| short_description: AI-powered diabetic retinopathy screening from your phone |
| --- |
| |
| <div align="center"> |
| <table border="1"> |
| <tr> |
| <td align="center" style="padding: 20px;"> |
| <h3>📢 Domain & Email Migration Notice</h3> |
| <p>From <b>May 30th, 2026</b>, Fundusnap will transition to new domains as <code>fundusnap.com</code> will not be renewed:</p> |
| <p>🌐 <b>Website:</b> <a href="https://fundusnap.faizath.com">fundusnap.faizath.com</a> (formerly <i>fundusnap.com</i>)<br> |
| ⚙️ <b>API:</b> <a href="https://fundusnap-api.faizath.com">fundusnap-api.faizath.com</a> (formerly <i>api.fundusnap.com</i>)<br> |
| 📧 <b>Email:</b> <a href="mailto:contact@fundusnap.faizath.com">contact@fundusnap.faizath.com</a> (formerly <i>contact@fundusnap.com</i>)<br> |
| 🛰️ <b>CDN:</b> <a>fundusnap-cdn.faizath.com</a> (formerly <i>cdn.fundusnap.com</i>)<br> |
| 📈 <b>Status Pages:</b> <a href="https://status.faizath.com/status/fundusnap">https://status.faizath.com/status/fundusnap</a> (formerly <i>status.fundusnap.com</i>) |
| </p> |
| </td> |
| </tr> |
| </table> |
| </div> |
| |
| <p align="center"> |
| <img src="profile/assets/logo.png"/> |
| <h1 align="center">Fundusnap</h1> |
| </p> |
|
|
| <p align="center"> |
| <b>AI-powered diabetic retinopathy screening — from your phone to a clinical second opinion.</b> |
| </p> |
|
|
| <p align="center"> |
| 🌐 <a href="https://fundusnap.faizath.com">Website</a> • |
| ⚙️ <a href="https://fundusnap-api.faizath.com">API</a> • |
| 📈 <a href="https://status.faizath.com/status/fundusnap">Status</a> |
| </p> |
|
|
| <p align="center"> |
| <a href="https://drive.google.com/file/d/1Td7bPj-vSIjByO5UIGwOhZstPwcz0tfm/preview"> |
| <img src="profile/assets/demo.gif" width="100%" alt="Watch the Fundusnap demo video"/> |
| </a> |
| </p> |
| |
| <p align="center"> |
| ▶️ <a href="https://drive.google.com/file/d/1Td7bPj-vSIjByO5UIGwOhZstPwcz0tfm/preview"><b>Watch the demo video</b></a> |
| </p> |
|
|
| --- |
|
|
| ## 🔍 What is Fundusnap? |
|
|
| **Fundusnap** is a comprehensive medical-imaging solution that helps healthcare workers and patients **detect and analyze diabetic retinopathy (DR)** from *fundus* (retinal) images. A user captures a photo of the back of the eye with the mobile app, and Fundusnap returns an AI classification of disease severity, highlights the specific retinal lesions it found, and lets the user ask follow-up questions to an AI medical assistant that explains the result in plain language. |
|
|
| It is delivered as an end-to-end product spanning a **mobile app**, a **backend API**, a **marketing/management website**, and a family of **open AI models** — a retinal lesion detector, a diabetic-retinopathy severity classifier, and a result-explanation language model — together with the **synthetic dataset** that language model was trained on. |
|
|
| ## 🩺 The Problem |
|
|
| Diabetic retinopathy is one of the leading causes of preventable blindness worldwide, and it disproportionately affects regions with limited access to specialist eye care. |
|
|
| - **Too few specialists.** Screening for DR traditionally requires an ophthalmologist to manually examine retinal images — a scarce and unevenly distributed resource, especially in rural and developing areas. |
| - **Late detection.** Early-stage DR is often asymptomatic. By the time patients notice vision problems, the disease may already be advanced and harder to treat. |
| - **High screening cost & low throughput.** Manual grading is slow and expensive, making large-scale population screening impractical. |
| - **Results are hard to understand.** Even when a patient receives a screening result, the clinical terminology is rarely accessible to non-experts, leading to poor follow-up. |
|
|
| ## 💡 How Fundusnap Solves It |
|
|
| Fundusnap brings specialist-grade screening to a smartphone and makes the result understandable to everyone: |
|
|
| 1. **Capture** — The Flutter mobile app guides users to take a high-quality fundus image (with photo and video capture support). |
| 2. **Classify** — The image is sent to the API, which runs it through **Microsoft Azure Custom Vision** to grade the severity of diabetic retinopathy, with our own [**fundusnap-v1-severitycls-rn34-22m**](https://github.com/fundusnap/fundusnap-v1-severitycls-rn34-22m) ResNet34 grader as the open, self-hostable alternative. |
| 3. **Detect** — [**fundusnap-v1-lesiondet-yolo11m-20m**](https://github.com/fundusnap/fundusnap-v1-lesiondet-yolo11m-20m), a YOLO11m detector, locates and bounds individual retinal lesions and landmarks (microaneurysms, haemorrhages, exudates, optic disc, fovea), so the result is explainable rather than a black box. |
| 4. **Explain** — An **AI medical chat assistant** interprets the findings in simple, informative language and encourages appropriate follow-up with a healthcare professional — without making a clinical diagnosis. Two interchangeable backends serve this role: Microsoft's **Phi-4** via OpenRouter, and the self-hosted [**fundusnap-v1-resultexp-clm-mediphi-3.8b-adapter**](https://github.com/fundusnap/fundusnap-v1-resultexp-clm-mediphi-3.8b-adapter) — a MediPhi-Instruct LoRA fine-tuned on our [**FundusTalk v1**](https://github.com/fundusnap/fundusnap-fundustalk-v1-chatsft-11k) dataset to answer in Indonesian or English. |
| 5. **Stay available offline** — The severity classifier also ships as an **ONNX** graph for on-device inference, acting as a fallback for poor connectivity or primary-API outages, so screening keeps working where it's needed most. |
|
|
| All medical data is handled with security and compliance in mind (JWT-based auth, encrypted transmission, and secure image storage). |
|
|
| --- |
|
|
| ## 🏆 Achievements & Competitions |
|
|
| Fundusnap was built for and submitted to three national programs in Indonesia, achieving recognition in each: |
|
|
| | Competition | Achievement | |
| | --- | --- | |
| | **elevAIte Microsoft × Biji-biji Hackathon 2025** — Tel-U Hub | 🥉 **3rd Winner** | |
| | **Digination Fest PPI Hackathon 2025** | 🏅 **Top 5 Finalist** | |
| | **Pikiran Terbaik Negeri × elevAIte 2025** | 🏅 **Top 30** | |
|
|
| ### elevAIte Microsoft × Biji-biji Hackathon 2025 — 🥉 3rd Winner |
| *Organized by Microsoft, the Biji-biji Initiative, and Telkom University, held at Tel-U Hub.* |
|
|
| **ElevAIte Indonesia** is an AI-skilling initiative by Microsoft and the Biji-biji Initiative that aims to equip **1 million Indonesian talents** with relevant AI skills for the era of digital transformation — **free of charge and with no selection barrier**. The program partners with government, industry, educational institutions, and communities to connect talent with new opportunities created by AI, such as improved productivity, creativity, and responsible innovation. It runs as a journey — from mastering AI fundamentals on Microsoft Learn and earning the **Microsoft AI-900** certification, to a **Hackathon** where participants apply their AI skills to solve real-world problems, followed by an **incubation** phase. Fundusnap was developed and submitted during this hackathon stage and placed **3rd overall**. |
|
|
| ### Digination Fest PPI Hackathon 2025 — 🏅 Top 5 Finalist |
| *Organized by the Indonesia World Students Association (Perhimpunan Pelajar Indonesia Dunia / PPI Dunia).* |
|
|
| **Digination Competition 2025**, themed **"AI for All: Bridging Innovation and People,"** is a hackathon open to active **undergraduate Indonesian students** from universities around the world. Teams of **three members from one university** submit a paper and video to advance through the stages, competing across three impact tracks — **Health, Education, and Social Business** — for prizes of IDR 10,000,000 per track. Fundusnap competed in the health track and reached the **Top 5 Finalists**. |
|
|
| ### Pikiran Terbaik Negeri × elevAIte 2025 — 🏅 Top 30 |
| *Organized by Yayasan BUMN, Microsoft, and the Biji-biji Initiative.* |
|
|
| **Pikiran Terbaik Negeri** is a grant-competition created by **Yayasan BUMN** in partnership with impact-investment organizations, media partners, and the ANGIN Foundation. The program's mission is to **identify, nurture, and develop social entrepreneurs** (*menemukan, membina, dan mengembangkan wirausaha sosial*) who create meaningful impact for Indonesian communities and environmental sustainability. Beyond grants, participants receive bootcamp training to strengthen their entrepreneurial skills and networking opportunities with financiers in the impact sector. Fundusnap was selected into the **Top 30**. |
|
|
| ### 📸 Moments |
|
|
| <table> |
| <tr> |
| <td width="50%" align="center" valign="top"> |
| <img src="profile/assets/3rd_Winner_Awarding.jpg" width="100%"/> |
| </td> |
| <td width="50%" align="center" valign="top"> |
| <img src="profile/assets/Fundusnap_Booth.jpg" width="100%"/> |
| </td> |
| </tr> |
| <tr> |
| <td align="center" valign="top"> |
| <b>🥉 3rd Winner Awarding</b><br/> |
| The Fundusnap team receiving the <b>Juara 3</b> award at the elevAIte Microsoft × Biji-biji Hackathon 2025, held at Tel-U Hub. |
| </td> |
| <td align="center" valign="top"> |
| <b>🏢 Fundusnap Booth</b><br/> |
| The team demonstrating Fundusnap to visitors and judges at the exhibition booth, held during the competition day at Telkom University. |
| </td> |
| </tr> |
| </table> |
| |
| --- |
|
|
| ## 📱 Screenshots |
|
|
| <table> |
| <tr> |
| <td><img src="profile/assets/screenshots/1.jpg" width="180"/></td> |
| <td><img src="profile/assets/screenshots/2.jpg" width="180"/></td> |
| <td><img src="profile/assets/screenshots/3.jpg" width="180"/></td> |
| <td><img src="profile/assets/screenshots/4.jpg" width="180"/></td> |
| <td><img src="profile/assets/screenshots/5.jpg" width="180"/></td> |
| </tr> |
| <tr> |
| <td><img src="profile/assets/screenshots/6.jpg" width="180"/></td> |
| <td><img src="profile/assets/screenshots/7.jpg" width="180"/></td> |
| <td><img src="profile/assets/screenshots/8.jpg" width="180"/></td> |
| <td><img src="profile/assets/screenshots/9.jpg" width="180"/></td> |
| <td><img src="profile/assets/screenshots/10.jpg" width="180"/></td> |
| </tr> |
| <tr> |
| <td><img src="profile/assets/screenshots/11.jpg" width="180"/></td> |
| <td><img src="profile/assets/screenshots/12.jpg" width="180"/></td> |
| <td><img src="profile/assets/screenshots/13.jpg" width="180"/></td> |
| <td><img src="profile/assets/screenshots/14.jpg" width="180"/></td> |
| <td><img src="profile/assets/screenshots/15.jpg" width="180"/></td> |
| </tr> |
| </table> |
| |
| --- |
|
|
| ## 🧩 Project Components |
|
|
| | Component | Repository | 🤗 Hub | Deployment | |
| | --- | --- | --- | --- | |
| | 📱 Mobile App | [fundusnap-app](https://github.com/fundusnap/fundusnap-app) | — | Android APK release | |
| | 🌐 Website | [fundusnap-web](https://github.com/fundusnap/fundusnap-web) | — | [fundusnap.faizath.com](https://fundusnap.faizath.com) | |
| | ⚙️ Backend API | [fundusnap-api](https://github.com/fundusnap/fundusnap-api) | — | [fundusnap-api.faizath.com](https://fundusnap-api.faizath.com) | |
| | 🔬 Lesion Detector | [fundusnap-v1-lesiondet-yolo11m-20m](https://github.com/fundusnap/fundusnap-v1-lesiondet-yolo11m-20m) | [model](https://huggingface.co/fundusnap/fundusnap-v1-lesiondet-yolo11m-20m) | Self-hosted FastAPI service | |
| | 🧠 Severity Classifier | [fundusnap-v1-severitycls-rn34-22m](https://github.com/fundusnap/fundusnap-v1-severitycls-rn34-22m) | [model](https://huggingface.co/fundusnap/fundusnap-v1-severitycls-rn34-22m) | ONNX · offline-capable | |
| | 💬 Result Explainer | [fundusnap-v1-resultexp-clm-mediphi-3.8b-adapter](https://github.com/fundusnap/fundusnap-v1-resultexp-clm-mediphi-3.8b-adapter) | [model](https://huggingface.co/fundusnap/fundusnap-v1-resultexp-clm-mediphi-3.8b-adapter) | Self-hosted (merged → vLLM) | |
| | 📚 FundusTalk v1 Dataset | [fundusnap-fundustalk-v1-chatsft-11k](https://github.com/fundusnap/fundusnap-fundustalk-v1-chatsft-11k) | [dataset](https://huggingface.co/datasets/fundusnap/fundusnap-fundustalk-v1-chatsft-11k) | Hugging Face dataset | |
|
|
| <br/> |
|
|
| <table> |
| <tr> |
| <td width="64" align="center" valign="top"> |
| <h1>📱</h1> |
| </td> |
| <td valign="top"> |
| <h3>Fundusnap Mobile App</h3> |
| <p>A modern cross-platform app that guides users to capture high-quality fundus images, runs AI-powered diabetic retinopathy analysis, and answers questions through an intelligent medical chatbot.</p> |
| <p> |
| <img src="https://img.shields.io/badge/Flutter-02569B?style=flat-square&logo=flutter&logoColor=white" alt="Flutter"/> |
| <img src="https://img.shields.io/badge/Dart-0175C2?style=flat-square&logo=dart&logoColor=white" alt="Dart"/> |
| <img src="https://img.shields.io/badge/Flutter_Bloc-13B9FD?style=flat-square&logo=flutter&logoColor=white" alt="Flutter Bloc"/> |
| <img src="https://img.shields.io/badge/Go_Router-0553B1?style=flat-square&logo=flutter&logoColor=white" alt="Go Router"/> |
| <img src="https://img.shields.io/badge/Dio-5C6BC0?style=flat-square&logo=dart&logoColor=white" alt="Dio"/> |
| </p> |
| <p> |
| <b>✨ Highlights:</b> Fundus photo & video capture · Encrypted on-device secure storage · On-the-go DR analysis · Conversational medical assistant |
| </p> |
| <p> |
| <a href="https://github.com/fundusnap/fundusnap-app"><img src="https://img.shields.io/badge/Repository-fundusnap--app-181717?style=flat-square&logo=github&logoColor=white" alt="Repository"/></a> |
| <img src="https://img.shields.io/badge/Deployment-Android_APK-3DDC84?style=flat-square&logo=android&logoColor=white" alt="Android APK"/> |
| </p> |
| </td> |
| </tr> |
| </table> |
| |
| <details> |
| <summary><b>Full tech stack — Mobile App</b></summary> |
|
|
| - **Framework:** Flutter (SDK `^3.8.0`) |
| - **Language:** Dart |
| - **State Management:** Flutter Bloc |
| - **Navigation:** Go Router |
| - **Key Dependencies:** |
| - Camera — `camera: ^0.11.1` |
| - Video Player — `video_player: ^2.9.5` |
| - Secure Storage — `flutter_secure_storage: ^9.2.4` |
| - Image Picker — `image_picker: ^1.1.2` |
| - HTTP Client — `dio: ^5.8.0+1` |
|
|
| </details> |
|
|
| <br/> |
|
|
| <table> |
| <tr> |
| <td width="64" align="center" valign="top"> |
| <h1>🌐</h1> |
| </td> |
| <td valign="top"> |
| <h3>Fundusnap Website</h3> |
| <p>The public-facing landing experience that introduces the product, showcases its features, and routes visitors to downloads and access links.</p> |
| <p> |
| <img src="https://img.shields.io/badge/Next.js-000000?style=flat-square&logo=nextdotjs&logoColor=white" alt="Next.js"/> |
| <img src="https://img.shields.io/badge/React-61DAFB?style=flat-square&logo=react&logoColor=black" alt="React"/> |
| <img src="https://img.shields.io/badge/TypeScript-3178C6?style=flat-square&logo=typescript&logoColor=white" alt="TypeScript"/> |
| <img src="https://img.shields.io/badge/Tailwind_CSS-06B6D4?style=flat-square&logo=tailwindcss&logoColor=white" alt="Tailwind CSS"/> |
| <img src="https://img.shields.io/badge/Cloudflare-F38020?style=flat-square&logo=cloudflare&logoColor=white" alt="Cloudflare"/> |
| </p> |
| <p> |
| <b>✨ Highlights:</b> Marketing & product showcase · Edge-hosted on Cloudflare Pages |
| </p> |
| <p> |
| <a href="https://github.com/fundusnap/fundusnap-web"><img src="https://img.shields.io/badge/Repository-fundusnap--web-181717?style=flat-square&logo=github&logoColor=white" alt="Repository"/></a> |
| <a href="https://fundusnap.faizath.com"><img src="https://img.shields.io/badge/Live-fundusnap.faizath.com-2ea44f?style=flat-square&logo=googlechrome&logoColor=white" alt="Live site"/></a> |
| </p> |
| </td> |
| </tr> |
| </table> |
| |
| <br/> |
|
|
| <table> |
| <tr> |
| <td width="64" align="center" valign="top"> |
| <h1>⚙️</h1> |
| </td> |
| <td valign="top"> |
| <h3>Fundusnap API</h3> |
| <p>The backend brain — handling authentication, image analysis & processing, AI chat interactions, and secure medical-data storage.</p> |
| <p> |
| <img src="https://img.shields.io/badge/Bun-000000?style=flat-square&logo=bun&logoColor=white" alt="Bun"/> |
| <img src="https://img.shields.io/badge/Node.js-5FA04E?style=flat-square&logo=nodedotjs&logoColor=white" alt="Node.js"/> |
| <img src="https://img.shields.io/badge/Express-000000?style=flat-square&logo=express&logoColor=white" alt="Express"/> |
| <img src="https://img.shields.io/badge/MongoDB-47A248?style=flat-square&logo=mongodb&logoColor=white" alt="MongoDB"/> |
| <img src="https://img.shields.io/badge/JWT-000000?style=flat-square&logo=jsonwebtokens&logoColor=white" alt="JWT"/> |
| <img src="https://img.shields.io/badge/Cloudflare_R2-F38020?style=flat-square&logo=cloudflare&logoColor=white" alt="Cloudflare R2"/> |
| <img src="https://img.shields.io/badge/Azure_Custom_Vision-0078D4?style=flat-square&logo=microsoftazure&logoColor=white" alt="Azure Custom Vision"/> |
| <img src="https://img.shields.io/badge/OpenRouter-6566F1?style=flat-square&logo=openai&logoColor=white" alt="OpenRouter"/> |
| </p> |
| <p> |
| <b>🤖 AI services:</b> Azure Custom Vision (DR grading) · <a href="https://github.com/fundusnap/fundusnap-v1-lesiondet-yolo11m-20m">fundusnap-v1-lesiondet-yolo11m-20m</a> (lesion detection) · medical chat via Microsoft <b>Phi-4</b> on OpenRouter <i>or</i> the self-hosted <a href="https://github.com/fundusnap/fundusnap-v1-resultexp-clm-mediphi-3.8b-adapter">MediPhi LoRA adapter</a> |
| </p> |
| <p> |
| <a href="https://github.com/fundusnap/fundusnap-api"><img src="https://img.shields.io/badge/Repository-fundusnap--api-181717?style=flat-square&logo=github&logoColor=white" alt="Repository"/></a> |
| <a href="https://fundusnap-api.faizath.com"><img src="https://img.shields.io/badge/Live-fundusnap--api.faizath.com-2ea44f?style=flat-square&logo=googlechrome&logoColor=white" alt="Live API"/></a> |
| </p> |
| </td> |
| </tr> |
| </table> |
| |
| <details> |
| <summary><b>Full tech stack — API</b></summary> |
|
|
| - **Runtime:** Bun / Node.js |
| - **Framework:** Express.js |
| - **Database:** MongoDB with Mongoose |
| - **Authentication:** JWT (access + refresh tokens) |
| - **Storage:** Cloudflare R2 (with Azure Blob Storage support) |
| - **AI Services:** |
| - Microsoft Azure Custom Vision API (DR severity grading) |
| - `fundusnap-v1-lesiondet-yolo11m-20m` — self-hosted YOLO11m lesion-detection service |
| - Medical chat, two interchangeable backends: |
| - OpenRouter API with Microsoft's **Phi-4** model |
| - Self-hosted `fundusnap-v1-resultexp-clm-mediphi-3.8b-adapter` (MediPhi-Instruct LoRA) |
| - **Email Service:** Nodemailer |
|
|
| </details> |
|
|
| <br/> |
|
|
| <table> |
| <tr> |
| <td width="64" align="center" valign="top"> |
| <h1>🔬</h1> |
| </td> |
| <td valign="top"> |
| <h3>Fundusnap Lesion Detector — <code>fundusnap-v1-lesiondet-yolo11m-20m</code></h3> |
| <p>A YOLO11m object detector that finds <i>where</i> the findings are. Given one colour fundus photograph it returns bounding boxes for twelve classes — ten pathological findings plus the optic disc and fovea as anatomical landmarks — each with a label and a confidence score, so a severity grade comes with visual evidence instead of being a black box.</p> |
| <p> |
| <img src="https://img.shields.io/badge/YOLO11m-111F68?style=flat-square&logo=yolo&logoColor=white" alt="YOLO11m"/> |
| <img src="https://img.shields.io/badge/Ultralytics-0B23A9?style=flat-square&logo=ultralytics&logoColor=white" alt="Ultralytics"/> |
| <img src="https://img.shields.io/badge/PyTorch-EE4C2C?style=flat-square&logo=pytorch&logoColor=white" alt="PyTorch"/> |
| <img src="https://img.shields.io/badge/FastAPI-009688?style=flat-square&logo=fastapi&logoColor=white" alt="FastAPI"/> |
| <img src="https://img.shields.io/badge/Params-20M-5B9BD5?style=flat-square" alt="20M parameters"/> |
| <img src="https://img.shields.io/badge/License-CC_BY--NC_4.0-EF9421?style=flat-square&logo=creativecommons&logoColor=white" alt="CC BY-NC 4.0"/> |
| </p> |
| <p> |
| <b>📊 Performance:</b> <code>0.53</code> mAP@50 · <code>0.28</code> mAP@50-95 · <code>0.54</code> precision · <code>0.53</code> recall |
| </p> |
| <p> |
| <b>✨ Highlights:</b> 12 classes (10 lesions + <code>Disc</code>/<code>Fovea</code> landmarks) · JSON and annotated-image endpoints · Dockerised FastAPI service · exports to ONNX, TorchScript, TFLite, CoreML |
| </p> |
| <p> |
| <a href="https://github.com/fundusnap/fundusnap-v1-lesiondet-yolo11m-20m"><img src="https://img.shields.io/badge/Repository-fundusnap--v1--lesiondet--yolo11m--20m-181717?style=flat-square&logo=github&logoColor=white" alt="Repository"/></a> |
| <a href="https://huggingface.co/fundusnap/fundusnap-v1-lesiondet-yolo11m-20m"><img src="https://img.shields.io/badge/%F0%9F%A4%97_Hugging_Face-Model-FFD21E?style=flat-square" alt="Hugging Face"/></a> |
| <img src="https://img.shields.io/badge/Deployment-Self--hosted_service-555555?style=flat-square&logo=docker&logoColor=white" alt="Self-hosted"/> |
| </p> |
| </td> |
| </tr> |
| </table> |
| |
| <details> |
| <summary><b>Full tech stack — Lesion Detector</b></summary> |
|
|
| - **Framework:** Ultralytics `8.3.165` / PyTorch |
| - **Base Model:** `yolo11m.pt`, COCO-pretrained — `yolo11m.yaml` scale `m`, anchor-free `Detect` head, `nc=12` |
| - **Input:** 640×640, letterboxed (coordinates returned in the original image's pixel space) |
| - **Training:** 35 epochs, batch 16 (`nbs=64`), optimizer `auto` (`lr0=0.01`, `lrf=0.01`, momentum 0.937, weight decay 0.0005), 3 warmup epochs, AMP, seed 0 deterministic |
| - **Loss weights:** box 7.5 · cls 0.5 · dfl 1.5 |
| - **Augmentation:** mosaic 1.0 (off for the last 10 epochs), `fliplr=0.5`, `scale=0.5`, `translate=0.1`, HSV (0.015/0.7/0.4), `erasing=0.4`, RandAugment |
| - **Shipped checkpoint:** epoch 27 — best by Ultralytics fitness (`0.1·mAP50 + 0.9·mAP50-95` = 0.3071), stripped of optimiser/EMA state (~40 MB, Git LFS) |
| - **Serving:** `POST /inspect/fundus-artifacts/` (JSON detections) · `POST /visualize/fundus-artifacts/` (annotated JPEG) · `GET /` (health) — `python:3.10-slim`, port 8000 |
| - **License:** CC BY-NC 4.0 (weights derive from Ultralytics YOLO11 — review Ultralytics' AGPL-3.0 terms before redistributing) |
|
|
| </details> |
|
|
| <br/> |
|
|
| <table> |
| <tr> |
| <td width="64" align="center" valign="top"> |
| <h1>🧠</h1> |
| </td> |
| <td valign="top"> |
| <h3>Fundusnap Severity Classifier — <code>fundusnap-v1-severitycls-rn34-22m</code></h3> |
| <p>A ResNet34 classifier that grades <i>how severe</i> the retinopathy is. It predicts one of the five standard ordinal ICDR grades (0 = No DR through 4 = Proliferative) with a probability for each, and ships as an ONNX graph with a dynamic batch axis — which is what keeps screening working offline or during a primary-API outage.</p> |
| <p> |
| <img src="https://img.shields.io/badge/FastAI-2EC4B6?style=flat-square&logo=fastapi&logoColor=white" alt="FastAI"/> |
| <img src="https://img.shields.io/badge/PyTorch-EE4C2C?style=flat-square&logo=pytorch&logoColor=white" alt="PyTorch"/> |
| <img src="https://img.shields.io/badge/ONNX-005CED?style=flat-square&logo=onnx&logoColor=white" alt="ONNX"/> |
| <img src="https://img.shields.io/badge/ResNet34-FF6F00?style=flat-square&logo=tensorflow&logoColor=white" alt="ResNet34"/> |
| <img src="https://img.shields.io/badge/Params-22M-5B9BD5?style=flat-square" alt="22M parameters"/> |
| <img src="https://img.shields.io/badge/License-CC_BY--NC_4.0-EF9421?style=flat-square&logo=creativecommons&logoColor=white" alt="CC BY-NC 4.0"/> |
| </p> |
| <p> |
| <b>📊 Performance:</b> <code>0.82</code> accuracy · <code>0.8153</code> macro F1 · <code>0.81</code> macro precision · <code>0.82</code> macro recall |
| </p> |
| <p> |
| <b>✨ Highlights:</b> 5 ordinal ICDR grades · ONNX opset 14 with dynamic batch · fastai checkpoint for further fine-tuning · ONNX → TensorFlow/TFLite path for on-device inference |
| </p> |
| <p> |
| <a href="https://github.com/fundusnap/fundusnap-v1-severitycls-rn34-22m"><img src="https://img.shields.io/badge/Repository-fundusnap--v1--severitycls--rn34--22m-181717?style=flat-square&logo=github&logoColor=white" alt="Repository"/></a> |
| <a href="https://huggingface.co/fundusnap/fundusnap-v1-severitycls-rn34-22m"><img src="https://img.shields.io/badge/%F0%9F%A4%97_Hugging_Face-Model-FFD21E?style=flat-square" alt="Hugging Face"/></a> |
| <img src="https://img.shields.io/badge/Deployment-ONNX_·_offline--capable-555555?style=flat-square&logo=onnx&logoColor=white" alt="ONNX / offline-capable"/> |
| </p> |
| </td> |
| </tr> |
| </table> |
| |
| <details> |
| <summary><b>Full tech stack — Severity Classifier</b></summary> |
|
|
| - **Deep Learning Framework:** FastAI / PyTorch, exported to ONNX (opset 14) |
| - **Base Model:** `resnet34`, ImageNet-pretrained (`timm/resnet34.tv_in1k`) |
| - **Head:** fastai default (`AdaptiveConcatPool2d` → BN/dropout → linear), `n_out=5` |
| - **Loss Function:** Focal Loss |
| - **Input:** `Resize(224)` centre crop, ImageNet normalisation |
| - **Training:** batch 32, `learn.fine_tune(4)` (1 frozen + 4 unfrozen epochs), LR from `lr_find()` valley, seed 3865 |
| - **Data Augmentation:** Albumentations — `ShiftScaleRotate`, `HorizontalFlip`, `RandomBrightnessContrast`, `HueSaturationValue` |
| - **Dataset:** Kaggle *resized-2015-2019-diabetic-retinopathy-detection* (EyePACS 2015 + APTOS 2019), each grade resampled to 10,000 rows for a 50,000-image balanced frame, 10% held out |
| - **Performance Metrics:** |
| - Overall Accuracy: 0.82 |
| - Macro Average F1-Score: 0.8153 |
| - Weighted Average F1-Score: 0.81 |
| - Grades 3–4 separate near-perfectly (F1 0.97–0.98); grades 0/1/2 sit at 0.65–0.76 |
| - **Deployment:** ONNX Runtime for inference (offline-capable), fastai checkpoint for fine-tuning |
| - **License:** CC BY-NC 4.0 (training data carries its own Kaggle / EyePACS / APTOS terms) |
|
|
| </details> |
|
|
| <br/> |
|
|
| <table> |
| <tr> |
| <td width="64" align="center" valign="top"> |
| <h1>💬</h1> |
| </td> |
| <td valign="top"> |
| <h3>Fundusnap Result Explainer — <code>fundusnap-v1-resultexp-clm-mediphi-3.8b-adapter</code></h3> |
| <p>A LoRA adapter over <a href="https://huggingface.co/microsoft/MediPhi-Instruct">microsoft/MediPhi-Instruct</a> that turns a prediction record into a plain-language explanation, in Indonesian or English. It is the conversational layer of the pipeline and the only model that never sees an image — it reads the severity probabilities and the lesion boxes the other two produce, and explains them without ever diagnosing.</p> |
| <p> |
| <img src="https://img.shields.io/badge/PEFT_LoRA-FFD21E?style=flat-square&logo=huggingface&logoColor=black" alt="PEFT LoRA"/> |
| <img src="https://img.shields.io/badge/MediPhi--Instruct-0078D4?style=flat-square" alt="MediPhi-Instruct"/> |
| <img src="https://img.shields.io/badge/Transformers-FFD21E?style=flat-square&logo=huggingface&logoColor=black" alt="Transformers"/> |
| <img src="https://img.shields.io/badge/FastAPI-009688?style=flat-square&logo=fastapi&logoColor=white" alt="FastAPI"/> |
| <img src="https://img.shields.io/badge/Params-3.8B_+_50M_LoRA-5B9BD5?style=flat-square" alt="3.8B + 50M LoRA"/> |
| <img src="https://img.shields.io/badge/Lang-id_|_en-5B9BD5?style=flat-square" alt="Indonesian and English"/> |
| <img src="https://img.shields.io/badge/License-CC_BY--NC_4.0-EF9421?style=flat-square&logo=creativecommons&logoColor=white" alt="CC BY-NC 4.0"/> |
| </p> |
| <p> |
| <b>📊 Performance:</b> val loss <code>1.229</code> → <code>0.691</code> · <code>0/60</code> stub replies (base: 10/60) · <code>40/41</code> Indonesian prompts answered in Indonesian (base: 35/41) |
| </p> |
| <p> |
| <b>✨ Highlights:</b> explains rather than diagnoses — every conversation routes to a clinician · bilingual with code-switching · FastAPI <code>serve.py</code> · <code>merge.py</code> fuses the adapter into a standalone checkpoint for vLLM |
| </p> |
| <p> |
| <a href="https://github.com/fundusnap/fundusnap-v1-resultexp-clm-mediphi-3.8b-adapter"><img src="https://img.shields.io/badge/Repository-fundusnap--v1--resultexp--clm--mediphi--3.8b--adapter-181717?style=flat-square&logo=github&logoColor=white" alt="Repository"/></a> |
| <a href="https://huggingface.co/fundusnap/fundusnap-v1-resultexp-clm-mediphi-3.8b-adapter"><img src="https://img.shields.io/badge/%F0%9F%A4%97_Hugging_Face-Model-FFD21E?style=flat-square" alt="Hugging Face"/></a> |
| <img src="https://img.shields.io/badge/Deployment-Self--hosted_·_vLLM-555555?style=flat-square&logo=docker&logoColor=white" alt="Self-hosted / vLLM"/> |
| </p> |
| </td> |
| </tr> |
| </table> |
| |
| <details> |
| <summary><b>Full tech stack — Result Explainer</b></summary> |
|
|
| - **Base Model:** `microsoft/MediPhi-Instruct` (Phi-3 architecture, 3.82B) |
| - **Method:** QLoRA-style supervised fine-tuning, loss on assistant turns only |
| - **LoRA config:** r=32, α=64, dropout 0.05 — targets `qkv_proj`, `o_proj`, `gate_up_proj`, `down_proj` |
| - **Trainable params:** 50,331,648 (**1.30%** of the model) |
| - **Training Data:** [FundusTalk v1](https://github.com/fundusnap/fundusnap-fundustalk-v1-chatsft-11k) — 10,201 synthetic consultations distilled from `microsoft/phi-4` (~70% Indonesian incl. code-switched, ~30% English) |
| - **Schedule:** 2 epochs, 638 steps, lr 1e-4 cosine, effective batch 32, bf16 |
| - **Hardware:** 1× A100-SXM4-40GB, ~170 min |
| - **Prompt envelope:** the exact three-system-message format the API sends (persona → severity JSON → detection JSON), byte-for-byte compatible with `JSON.stringify` |
| - **Serving:** `GET /` · `POST /chat` · `POST /prompt` (returns the envelope without generating) — or merge and serve with vLLM |
| - **Licensing:** weights CC BY-NC 4.0 · repository code MIT · base and teacher models both MIT |
|
|
| </details> |
|
|
| <br/> |
|
|
| <table> |
| <tr> |
| <td width="64" align="center" valign="top"> |
| <h1>📚</h1> |
| </td> |
| <td valign="top"> |
| <h3>FundusTalk v1 — <code>fundusnap-fundustalk-v1-chatsft-11k</code></h3> |
| <p>The synthetic SFT dataset behind the Result Explainer: 10,849 multi-turn consultations that teach a model to explain a diabetic retinopathy screening result and never to diagnose it. Fully synthetic — no patient data, images, or recorded conversations — with every prediction record procedurally generated and every rejected sample published alongside the kept ones.</p> |
| <p> |
| <img src="https://img.shields.io/badge/%F0%9F%A4%97_Datasets-FFD21E?style=flat-square" alt="Hugging Face Datasets"/> |
| <img src="https://img.shields.io/badge/Format-JSONL-000000?style=flat-square&logo=json&logoColor=white" alt="JSONL"/> |
| <img src="https://img.shields.io/badge/Conversations-10,849-5B9BD5?style=flat-square" alt="10,849 conversations"/> |
| <img src="https://img.shields.io/badge/Teacher-phi--4-0078D4?style=flat-square" alt="Teacher: phi-4"/> |
| <img src="https://img.shields.io/badge/Lang-id_|_en-5B9BD5?style=flat-square" alt="Indonesian and English"/> |
| <img src="https://img.shields.io/badge/License-CC_BY--NC_4.0-EF9421?style=flat-square&logo=creativecommons&logoColor=white" alt="CC BY-NC 4.0"/> |
| </p> |
| <p> |
| <b>📊 Measured quality:</b> <code>97.2%</code> numeric grounding · <code>100%</code> opening diversity · <code>88.8%</code> safety-refusal rate · <code>90.4%</code> keep rate after filtering |
| </p> |
| <p> |
| <b>✨ Highlights:</b> 39,288 assistant turns (mean 77.2 words) · 5 categories incl. <code>safety_refusal</code> and <code>adversarial_oos</code> · 12 patient/caregiver/clinician personas · splits 10,201 / 324 / 324 · drops straight into TRL's <code>SFTTrainer</code> |
| </p> |
| <p> |
| <a href="https://github.com/fundusnap/fundusnap-fundustalk-v1-chatsft-11k"><img src="https://img.shields.io/badge/Repository-fundusnap--fundustalk--v1--chatsft--11k-181717?style=flat-square&logo=github&logoColor=white" alt="Repository"/></a> |
| <a href="https://huggingface.co/datasets/fundusnap/fundusnap-fundustalk-v1-chatsft-11k"><img src="https://img.shields.io/badge/%F0%9F%A4%97_Hugging_Face-Dataset-FFD21E?style=flat-square" alt="Hugging Face"/></a> |
| <img src="https://img.shields.io/badge/Distribution-Hugging_Face_dataset-555555?style=flat-square" alt="Hugging Face dataset"/> |
| </p> |
| </td> |
| </tr> |
| </table> |
| |
| <details> |
| <summary><b>Full breakdown — FundusTalk v1</b></summary> |
|
|
| - **Teacher:** `microsoft/phi-4` via OpenRouter · **Intended student:** `microsoft/MediPhi-Instruct` |
| - **Scale:** 10,849 conversations · 39,288 assistant turns · 3.62 turns per conversation |
| - **Splits:** `train` 10,201 · `validation` 324 · `test` 324 — disjoint by conversation id, stratified on category, language, grade, and record profile |
| - **Configs:** `default` (filtered, 10,849) · `raw` (unfiltered teacher output, 12,000) · `scenarios` (the seeded, deterministic generation plan) |
| - **Categories:** `result_explanation` 42.2% · `safety_refusal` 16.5% · `detector_literacy` 16.3% · `general_knowledge` 14.5% · `adversarial_oos` 10.5% |
| - **Languages:** Indonesian 47.2% · English 29.3% · Indonesian–English code-switch 23.5% |
| - **Record profiles:** deliberate edge cases — `landmarks_only`, `empty_detections`, `poor_quality`, `low_confidence`, `disagreement` |
| - **Filtering:** 12,000 generated → 10,849 kept (90.4%); the safety-critical filters are `no_clinician_referral` and `diagnostic_language` |
| - **Reproducibility:** seeded scenario plan, full generation and filtering logs, complete reject list |
| - **License:** CC BY-NC 4.0 |
|
|
| </details> |
|
|
| <br/> |
|
|
| > [!IMPORTANT] |
| > **The four AI repositories above are released for research and engineering use.** None of them is a |
| > medical device, none carries regulatory clearance (FDA, CE/MDR, or otherwise), and none has been |
| > prospectively validated. Reported metrics are self-reported on the runs' own validation splits. They |
| > must never be the sole basis for a diagnosis, referral, or treatment decision — keep a qualified |
| > clinician in the loop. |
|
|
| --- |
|
|
| ## 🔒 Security & Compliance |
|
|
| The entire system is designed with security and compliance in mind: |
| - Secure authentication using JWT (short-lived access tokens + refresh tokens) |
| - Encrypted data transmission |
| - Secure storage of medical images |
| - Privacy-conscious handling of medical data |
| - Regular security updates and patches |
|
|
| ## 🚀 Getting Started |
|
|
| Each component has its own repository with detailed setup instructions. Please refer to the individual README files in each repository for specific setup and installation steps. |
|
|
| ## 📝 License |
|
|
| This project is licensed under the MIT License. |
|
|
| ## 👥 Authors |
|
|
| Fundusnap Developers <dev@fundusnap.faizath.com> |
|
|
