| # Llama2-7B-MIMIC-iii-Extraction-v1 |
|
|
| ## Model Description |
| This model is a fine-tuned version of **Llama-2-7b-chat-hf** designed for **Structured Clinical Information Extraction**. It has been specifically trained to process unstructured clinical notes (discharge summaries) from the **MIMIC-III** database and transform them into a structured **JSON format**. |
|
|
| The model can identify and extract key medical entities such as: |
| - Drug names |
| - Dosages |
| - Frequency of administration |
| - Indications/Reasons for treatment |
|
|
| ## Training Procedure |
| The model was fine-tuned using **QLoRA (4-bit quantization)** to ensure efficiency and high performance. |
|
|
| ### Training Hyperparameters: |
| - **Base Model:** NousResearch/Llama-2-7b-chat-hf |
| - **Method:** LoRA (Low-Rank Adaptation) |
| - **Max Sequence Length:** 2048 tokens |
| - **Learning Rate:** 2e-4 |
| - **Batch Size:** 1 (with 4 gradient accumulation steps) |
| - **Optimizer:** paged_adamw_32bit |
| - **Precision:** 4-bit (bitsandbytes) |
|
|
| ### LoRA Configuration: |
| - **r (Rank):** 16 |
| - **lora_alpha:** 32 |
| - **Target Modules:** q_proj, v_proj, k_proj, o_proj (Attention layers) |
| - **lora_dropout:** 0.05 |
|
|
| ## Intended Use |
| This model is intended for researchers and developers working on clinical natural language processing (NLP). It is designed to assist in converting medical narratives into machine-readable data. |
|
|
| ### How to use: |
| To use this model, you need to load it as a PEFT (Adapter) on top of the base Llama-2-7b-chat-hf model. |
|
|
| ```python |
| from peft import PeftModel, PeftConfig |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| base_model_name = "NousResearch/Llama-2-7b-chat-hf" |
| adapter_model_name = "maherghanem86/PharmaCompass" |
| |
| model = AutoModelForCausalLM.from_pretrained(base_model_name) |
| model = PeftModel.from_pretrained(model, adapter_model_name) |
| tokenizer = AutoTokenizer.from_pretrained(base_model_name) |
