README.md

---
tags:
- chatbot
- evaluation
- first-aid
license: mit
language:
- en
metrics:
- faithfulness
- answer_relevancy
- context_recall
- answer_correctness
- semantic_similarity
---

# 🏥 First Aid Tutor



## 📘 About the Project
First Aid Tutor is an **AI-powered chatbot** designed to evaluate the efficacy of **Retrieval-Augmented Generation (RAG)** in medical education for paramedical and medical students. It provides **first-aid guidance** based on medical literature to assess learning outcomes and response accuracy.

This repository contains the **codebase** for the **First Aid Assistant** chatbot, tested on **10 first-aid-related questions**.

## 📢 **Ownership and Attribution**
> **⚠️ Disclaimer:**  
> - The **First Aid Assistant chatbot** is the property of **Mysore Medical College and Research Institute**.  
> - The uploaded **FIRST AID PDF** that forms the basis of the **Retrieval-Augmented Generation (RAG)** process is an **official course material** and the property of **Mysore Medical College and Research Institute**.

---

# 🚑 First Aid Tutor - RAG Chatbot

The **First Aid Tutor** is a **Retrieval-Augmented Generation (RAG)** chatbot designed to assist **medical and paramedical students** in learning **first-aid protocols**. It is strictly trained on a **verified first-aid guide** and only provides information present in the **First Aid PDF**.

## 🎯 Features
✅ **Evidence-Based Responses** - The chatbot only retrieves and generates answers based on the **First Aid PDF**.  
✅ **User-Friendly Interface** - Developed using **Gradio**, featuring a modern and intuitive UI.  
✅ **Medical Education Research** - Used for evaluating **RAG efficacy in medical education**.  

## 🔬 How It Works
- Uses **TF-IDF + Cosine Similarity** for retrieving contextually relevant information.  
- Queries are answered based strictly on **verified medical content**.  
- Integrated with **Gradio UI** for an easy-to-use chatbot experience.  

## 📌 Live Demo
🚀 **[Try the First Aid Tutor Chatbot](https://huggingface.co/spaces/DrSyedFaizan/First_Aid_Assistant)**

## 📥 How to Run Locally
You can run the chatbot locally using the following commands:

```
git clone https://github.com/SYEDFAIZAN1987/First-Aid-Tutor
cd First-Aid-Tutor
pip install -r requirements.txt
python raggpt.py
```


# 🩺 First Aid Assistant - Model Evaluation Report

This repository presents the evaluation results of the **First Aid Assistant** chatbot, which provides first aid guidance based on common emergency conditions. The model has been evaluated using the **RAGAS** framework with metrics that assess the quality of the generated answers.

## 📊 **Evaluation Metrics**

The chatbot was evaluated based on the following RAGAS metrics:

- **Answer Relevancy:** Measures how relevant the response is to the user's question.
- **Answer Correctness:** Compares the generated response to the ground truth to assess factual correctness.
- **Semantic Similarity:** Evaluates how semantically similar the generated answer is to the reference answer.

---

## 🚀 **Performance Summary**

| **Metric**               | **Average Score** |
|:--------------------------|:-----------------:|
| **Answer Relevancy**      | **0.94**          |
| **Answer Correctness**    | **0.91**          |
| **Semantic Similarity**   | **0.97**          |

---

## 📈 **Detailed Results**

Here’s a snapshot of the evaluation for some sample questions:

| **Question**                                            | **Answer Relevancy** | **Answer Correctness** | **Semantic Similarity** |
|---------------------------------------------------------|----------------------|------------------------|-------------------------|
| What are the first aid measures for high fever in infants? | 0.93                 | 0.85                   | 0.98                    |
| What are the signs and symptoms of low blood sugar?       | 0.85                 | 0.98                   | 0.94                    |
| What does RICE stand for in first aid treatment?          | 0.99                 | 1.00                   | 0.98                    |
| What is the treatment of snake bite?                      | 0.96                 | 1.00                   | 0.98                    |
| How do you provide first aid for choking?                 | 0.96                 | 0.97                   | 0.98                    |

---

## 📋 **Key Insights**

- The chatbot performed exceptionally well in **semantic similarity** (average score of **0.97**), indicating that responses are closely aligned with the ground truth.
- **Answer correctness** is strong overall but showed slight variability, suggesting room for improvement in handling complex queries.
- The **relevancy** of responses remained consistently high, reflecting the model's ability to address user questions effectively.

---

## 📝 **Evaluation Artifacts**

- **RAGAS Evaluation Report:** [View Full Report](https://huggingface.co/spaces/DrSyedFaizan/FirstAidTutor_RAGAS_Evaluation)

---

## 🌟 **Conclusion**

The **First Aid Assistant** demonstrates reliable performance in answering first aid-related queries with high semantic accuracy and relevancy. Continuous improvement in factual correctness will further enhance its capability to provide life-saving information in emergency situations.

---


## 📥 Download & Re-Evaluate
You can **re-evaluate the chatbot** by running the following:

```
git clone https://huggingface.co/DrSyedFaizan/First_Aid_Assistant_Evaluation
cd First_Aid_Assistant_Evaluation
python eval.py
```

---

## 📌 Author
👨‍⚕ **Dr. Syed Faizan** | 🏥 **Healthcare AI & Data Science**  
🔗 **Hugging Face Repo**: [First_Aid_Assistant_Evaluation](https://huggingface.co/DrSyedFaizan/First_Aid_Assistant_Evaluation)

📩 **For collaborations, inquiries, or improvements, feel free to contribute or reach out!**