Update README.md

README.md

@@ -11,24 +11,73 @@ tags:
 
 Notable generation Of patient Text summaries through an Efficient approach based on direct preference optimization (DPO)
 
-This modelcard aims to be a base template for new models. It has been generated using [this raw template](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/modelcard_template.md?plain=1).
 
-## Model Details
 
-### Model Description
+## Model Description
 
 - **Model type:** MistralForCausalLM
 - **Language(s) (NLP):** English
 - **License:** [CC-BY-NC-SA](https://creativecommons.org/licenses/by-nc-sa/4.0/)
 - **Finetuned from model:** [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)
 
-### Model Sources
+## Model Sources
 
 - **Paper:** [NOTE](arvix.)
 - **Demo:** [NOTE-DEMO](https://huggingface.co/spaces/jinee/note-demo)
 
 ## Usage
 
+~~~python
+from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
+model = AutoModelForCausalLM.from_pretrained("jinee/note", load_in_4bit=True, device_map="auto")
+tokenizer = AutoTokenizer.from_pretrained("jinee/note")
+tokenizer.padding_side = 'right'
+tokenizer.add_eos_token = True
+tokenizer.pad_token = tokenizer.eos_token
+tokenizer.add_eos_token, tokenizer.add_bos_token
+
+instruction = '''
+As a doctor, you need to create a discharge summary based on input data. 
+Never change the dates or numbers in the input data and use them as is. And please follow the format below for your report. 
+Also, never make up information that is not in the input data, and write a report only with information that can be identified from the input data.
+
+1. Patient information (SUBJECT_ID, HADM_ID, hospitalization and discharge date, hospitalization period, gender, date of birth, age, allergy)
+2. Diagnostic information and past history (if applicable)
+3. Surgery or procedure information
+4. Significant medication administration during hospitalization and discharge medication history
+5. Meaningful lab tests during hospitalization
+6. Summary of significant text records/notes
+7. Discharge outcomes and treatment plan
+8. Overall summary of at least 500 characters in lines including the above contents
+'''
+
+def generation(model, tokenizer, input_data):
+    pipe = pipeline('text-generation', 
+                   model = model,
+                   tokenizer = tokenizer,
+                   torch_dtype=torch.bfloat16,
+                   device_map = 'auto')
+    global instruction
+       
+    sequences = pipe(
+        f"[INST]{instruction}: {input_data} [/INST]",
+        do_sample=True,
+        max_new_tokens=1024, 
+        temperature=0.7, 
+        top_k=50, 
+        top_p=0.95,
+        early_stopping =True,
+        num_return_sequences=1,)
+
+    text = sequences[0]['generated_text']
+    start_index = text.find('[/INST]')
+    if start_index != -1:
+        summary_ = text[start_index + len('[/INST]'):]
+        return(summary_)
+    else:
+        return("'[summary_] 'is not founded.")
+
+~~~
 
 
 ## Dataset
@@ -37,9 +86,39 @@ The model has been trained on a [MIMIC-III](https://physionet.org/content/mimici
 Access to this databased requires a number of steps to obtain permission. 
 
 
+## Training and Hyper-parameters
 
+### List of LoRA config 
+based on [Parameter-Efficient Fine-Tuning (PEFT)](https://github.com/huggingface/peft)
+
+Parameter | SFT | DPO 
+:------:| :------:| :------:
+r | 16 | 16 
+lora alpha | 16 | 16 
+lora dropout | 0.05 | 0.05  
+target | q, k, v, o, gate | q, k, v, o, gate 
+
+
+### List of Training arguments
+based on [Transformer Reinforment Learning (TRL)](https://github.com/huggingface/trl)
+
+Parameter | SFT | DPO 
+:------:| :------:| :------:
+early stopping patience | 3 | 3 
+early stopping threshold | 0.0005 | 0.0005 
+train epochs | 20 | 3 
+per device train batch size | 4 | 1 
+per device eval batch size | 8 (default) | 1 
+optimizer | paged adamw 8bit | paged adamw 8bit
+lr scheduler | cosine | cosine 
+wramup ratio | 0.3 | 0.1
+gradient accumulation step | 2 | 2 
+evaluation strategy | step | step 
+eval step | 10 | 5 
+
+
+## Applicability in medicine
 
-## Clinical appli
 
 
 ## Limitations