|
|
--- |
|
|
tags: |
|
|
- chatbot |
|
|
- evaluation |
|
|
- first-aid |
|
|
license: mit |
|
|
language: |
|
|
- en |
|
|
metrics: |
|
|
- faithfulness |
|
|
- answer_relevancy |
|
|
- context_recall |
|
|
- answer_correctness |
|
|
- semantic_similarity |
|
|
--- |
|
|
|
|
|
# 🏥 First Aid Tutor |
|
|
|
|
|
|
|
|
|
|
|
## 📘 About the Project |
|
|
First Aid Tutor is an **AI-powered chatbot** designed to evaluate the efficacy of **Retrieval-Augmented Generation (RAG)** in medical education for paramedical and medical students. It provides **first-aid guidance** based on medical literature to assess learning outcomes and response accuracy. |
|
|
|
|
|
This repository contains the **codebase** for the **First Aid Assistant** chatbot, tested on **10 first-aid-related questions**. |
|
|
|
|
|
## 📢 **Ownership and Attribution** |
|
|
> **⚠️ Disclaimer:** |
|
|
> - The **First Aid Assistant chatbot** is the property of **Mysore Medical College and Research Institute**. |
|
|
> - The uploaded **FIRST AID PDF** that forms the basis of the **Retrieval-Augmented Generation (RAG)** process is an **official course material** and the property of **Mysore Medical College and Research Institute**. |
|
|
|
|
|
--- |
|
|
|
|
|
# 🚑 First Aid Tutor - RAG Chatbot |
|
|
|
|
|
The **First Aid Tutor** is a **Retrieval-Augmented Generation (RAG)** chatbot designed to assist **medical and paramedical students** in learning **first-aid protocols**. It is strictly trained on a **verified first-aid guide** and only provides information present in the **First Aid PDF**. |
|
|
|
|
|
## 🎯 Features |
|
|
✅ **Evidence-Based Responses** - The chatbot only retrieves and generates answers based on the **First Aid PDF**. |
|
|
✅ **User-Friendly Interface** - Developed using **Gradio**, featuring a modern and intuitive UI. |
|
|
✅ **Medical Education Research** - Used for evaluating **RAG efficacy in medical education**. |
|
|
|
|
|
## 🔬 How It Works |
|
|
- Uses **TF-IDF + Cosine Similarity** for retrieving contextually relevant information. |
|
|
- Queries are answered based strictly on **verified medical content**. |
|
|
- Integrated with **Gradio UI** for an easy-to-use chatbot experience. |
|
|
|
|
|
## 📌 Live Demo |
|
|
🚀 **[Try the First Aid Tutor Chatbot](https://huggingface.co/spaces/DrSyedFaizan/First_Aid_Assistant)** |
|
|
|
|
|
## 📥 How to Run Locally |
|
|
You can run the chatbot locally using the following commands: |
|
|
|
|
|
``` |
|
|
git clone https://github.com/SYEDFAIZAN1987/First-Aid-Tutor |
|
|
cd First-Aid-Tutor |
|
|
pip install -r requirements.txt |
|
|
python raggpt.py |
|
|
``` |
|
|
|
|
|
|
|
|
# 🩺 First Aid Assistant - Model Evaluation Report |
|
|
|
|
|
This repository presents the evaluation results of the **First Aid Assistant** chatbot, which provides first aid guidance based on common emergency conditions. The model has been evaluated using the **RAGAS** framework with metrics that assess the quality of the generated answers. |
|
|
|
|
|
## 📊 **Evaluation Metrics** |
|
|
|
|
|
The chatbot was evaluated based on the following RAGAS metrics: |
|
|
|
|
|
- **Answer Relevancy:** Measures how relevant the response is to the user's question. |
|
|
- **Answer Correctness:** Compares the generated response to the ground truth to assess factual correctness. |
|
|
- **Semantic Similarity:** Evaluates how semantically similar the generated answer is to the reference answer. |
|
|
|
|
|
--- |
|
|
|
|
|
## 🚀 **Performance Summary** |
|
|
|
|
|
| **Metric** | **Average Score** | |
|
|
|:--------------------------|:-----------------:| |
|
|
| **Answer Relevancy** | **0.94** | |
|
|
| **Answer Correctness** | **0.91** | |
|
|
| **Semantic Similarity** | **0.97** | |
|
|
|
|
|
--- |
|
|
|
|
|
## 📈 **Detailed Results** |
|
|
|
|
|
Here’s a snapshot of the evaluation for some sample questions: |
|
|
|
|
|
| **Question** | **Answer Relevancy** | **Answer Correctness** | **Semantic Similarity** | |
|
|
|---------------------------------------------------------|----------------------|------------------------|-------------------------| |
|
|
| What are the first aid measures for high fever in infants? | 0.93 | 0.85 | 0.98 | |
|
|
| What are the signs and symptoms of low blood sugar? | 0.85 | 0.98 | 0.94 | |
|
|
| What does RICE stand for in first aid treatment? | 0.99 | 1.00 | 0.98 | |
|
|
| What is the treatment of snake bite? | 0.96 | 1.00 | 0.98 | |
|
|
| How do you provide first aid for choking? | 0.96 | 0.97 | 0.98 | |
|
|
|
|
|
--- |
|
|
|
|
|
## 📋 **Key Insights** |
|
|
|
|
|
- The chatbot performed exceptionally well in **semantic similarity** (average score of **0.97**), indicating that responses are closely aligned with the ground truth. |
|
|
- **Answer correctness** is strong overall but showed slight variability, suggesting room for improvement in handling complex queries. |
|
|
- The **relevancy** of responses remained consistently high, reflecting the model's ability to address user questions effectively. |
|
|
|
|
|
--- |
|
|
|
|
|
## 📝 **Evaluation Artifacts** |
|
|
|
|
|
- **RAGAS Evaluation Report:** [View Full Report](https://huggingface.co/spaces/DrSyedFaizan/FirstAidTutor_RAGAS_Evaluation) |
|
|
|
|
|
--- |
|
|
|
|
|
## 🌟 **Conclusion** |
|
|
|
|
|
The **First Aid Assistant** demonstrates reliable performance in answering first aid-related queries with high semantic accuracy and relevancy. Continuous improvement in factual correctness will further enhance its capability to provide life-saving information in emergency situations. |
|
|
|
|
|
--- |
|
|
|
|
|
|
|
|
## 📥 Download & Re-Evaluate |
|
|
You can **re-evaluate the chatbot** by running the following: |
|
|
|
|
|
``` |
|
|
git clone https://huggingface.co/DrSyedFaizan/First_Aid_Assistant_Evaluation |
|
|
cd First_Aid_Assistant_Evaluation |
|
|
python eval.py |
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
## 📌 Author |
|
|
👨⚕ **Dr. Syed Faizan** | 🏥 **Healthcare AI & Data Science** |
|
|
🔗 **Hugging Face Repo**: [First_Aid_Assistant_Evaluation](https://huggingface.co/DrSyedFaizan/First_Aid_Assistant_Evaluation) |
|
|
|
|
|
📩 **For collaborations, inquiries, or improvements, feel free to contribute or reach out!** |
|
|
|
|
|
|
|
|
|
