Create README.md

README.md

@@ -0,0 +1,54 @@
+---
+title: "Model Card for SNOWTEAM/sft_medico-mistral"
+summary: "A specialized language model for medical applications, refined through instruction tuning."
+---
+
+# Model Card for SNOWTEAM/sft_medico-mistral
+
+## Overview
+
+SNOWTEAM/sft_medico-mistral is a specialized language model designed for medical applications, further refined through instruction tuning to enhance its ability to respond to various medical-related instructions. This tuning leverages the embedded medical knowledge within the PMC LLaMAK model, focusing on medical consulting conversations, medical rationale QA, and medical knowledge graph prompting.
+
+## Model Description
+
+**Base Model:** Medico-mistral  
+**Model type:** Transformer-based decoder-only language model  
+**Language(s) (NLP):** English  
+
+### Instruction Tuning Datasets
+
+Using open source instruction tuning datasets are composed of three main parts: (This dataset is from [https://huggingface.co/datasets/axiong/pmc_llama_instructions](https://huggingface.co/datasets/axiong/pmc_llama_instructions))
+
+1. **Medical Conversation:**
+   - We utilize diverse doctor-patient dialogues where patient questions serve as instructions and doctor responses as ground truth.
+   - Data Sources: Med-Alpaca (Han, Adams et al. 2023) and ChatDoctor (Yunxiang et al. 2023).
+   - We expand the provided instructions into various synonymous sentences using GPT-4 to improve the model's robustness to diverse instructions. Specifically, we use the following query prompt:
+     ```
+     Rewrite 10 sentences that convey similar meanings to what I’ve stated: {instruction seeds}.
+     ```
+     where `{instruction seeds}` denotes the provided instruction from ChatDoctor or MedAlpaca. During training, we randomly select one instruction from the instruction base to simulate real user inputs and avoid over-fitting on specific templates.
+
+2. **Medical Rationale QA:**
+   - We equip the model with reasoning ability using professional medical knowledge.
+   - Data Sources: Open-source medical multi-choice question-answering datasets such as USMLE (Jin, Pan et al. 2021), PubMedQA (Jin et al. 2019), and MedMCQA (Pal, Umapathi et al. 2022).
+   - To add detailed reasoning guidance, we prompt ChatGPT for causality analysis given a QA pair, treating the output as an explanation with a structured format.
+
+3. **Medical Knowledge Graph Prompting:**
+   - We exploit medical knowledge graphs like UMLS (Lindberg, Humphreys, and McCray 1993) to align with clinicians' experience.
+   - We construct QA pairs to translate common knowledge graphs, focusing on entity descriptions and entity relationships. The model is prompted to output descriptions for certain entities or predict relationships between entities.
+
+### Medical-Specific Instruction Tuning
+
+By combining the above three parts, we form a large-scale, high-quality, medical-specific instruction tuning dataset, MedCI, consisting of 202M tokens. We further tune Medico-mistral on this dataset, resulting in sft_medico-mistral.
+
+## Training Details
+
+### Training Data
+
+The training data combines diverse datasets from medical consultations, rationale QA, and knowledge graphs to ensure comprehensive medical knowledge coverage and reasoning ability.
+
+## Model Sources
+
+**Repository:** [https://huggingface.co/SNOWTEAM/sft_medico-mistral](https://huggingface.co/SNOWTEAM/sft_medico-mistral)  
+**Paper [optional]:**  
+**Demo [optional]:**