Upload README.md

README.md

@@ -1,48 +1,41 @@
----
-license: apache-2.0
----
-# Multilingual Medicine: Model, Dataset, Benchmark, Code
+# MedJamba
 
-Covering English, Chinese, French, Hindi, Spanish, Hindi, Arabic So far
+Multilingual Medical Model Based On Jamba
+<center>
 
+![Python 3.10](https://img.shields.io/badge/Python-3.10-lightblue) ![Pytorch 2.1.2](https://img.shields.io/badge/PyTorch-2.1.2-lightblue) ![transformers](https://img.shields.io/badge/transformers-4.34.0.dev0%2B-lightblue) ![accelerate](https://img.shields.io/badge/accelerate-0.22-lightblue)
+</center>
 
-<p align="center">
-   👨🏻‍💻<a href="https://github.com/FreedomIntelligence/Apollo" target="_blank">Github</a> •📃 <a href="https://arxiv.org/abs/2403.03640" target="_blank">Paper</a> • 🌐 <a href="https://apollo.llmzoo.com/" target="_blank">Demo</a> • 🤗 <a href="https://huggingface.co/datasets/FreedomIntelligence/ApolloCorpus" target="_blank">ApolloCorpus</a> • 🤗 <a href="https://huggingface.co/datasets/FreedomIntelligence/XMedbench" target="_blank">XMedBench</a> 
-   <br>  <a href="./README_zh.md"> 中文 </a> | <a href="./README.md"> English
-</p>
-
-![Apollo](assets/apollo_medium_final.png)
 
 ## 🌈 Update
 
-* **[2024.03.07]** [Paper](https://arxiv.org/abs/2403.03640) released.
-* **[2024.02.12]** <a href="https://huggingface.co/datasets/FreedomIntelligence/ApolloCorpus" target="_blank">ApolloCorpus</a> and  <a href="https://huggingface.co/datasets/FreedomIntelligence/XMedbench" target="_blank">XMedBench</a>  is published！🎉
-* **[2024.01.23]** Apollo repo is published！🎉
-
+* **[2024.04.25]** MedJamba Model is published！🎉
+      
+   
 
 ## Results
-   🤗<a href="https://huggingface.co/FreedomIntelligence/Apollo-0.5B" target="_blank">Apollo-0.5B</a> • 🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo-1.8B" target="_blank">Apollo-1.8B</a> • 🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo-2B" target="_blank">Apollo-2B</a>  • 🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo-6B" target="_blank">Apollo-6B</a> • 🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo-7B" target="_blank">Apollo-7B</a>  🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo-34B" target="_blank">Apollo-34B</a> • 🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo-72B" target="_blank">Apollo-72B</a> 
-
-   🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo-0.5B-GGUF" target="_blank">Apollo-0.5B-GGUF</a> • 🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo-2B-GGUF" target="_blank">Apollo-2B-GGUF</a>  • 🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo-6B-GGUF" target="_blank">Apollo-6B-GGUF</a> • 🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo-7B-GGUF" target="_blank">Apollo-7B-GGUF</a>
-
-
+   🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo-0.5B" target="_blank">Apollo-0.5B</a> • 🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo-1.8B" target="_blank">Apollo-1.8B</a> • 🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo-2B" target="_blank">Apollo-2B</a>  • 🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo-6B" target="_blank">Apollo-6B</a> • 🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo-7B" target="_blank">Apollo-7B</a>  • 🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo-34B" target="_blank">Apollo-34B</a> • 🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo-72B" target="_blank">Apollo-72B</a>  
+   
+   🤗 <a href="https://huggingface.co/FreedomIntelligence/MedJamba" target="_blank">Apollo-53B (MedJamba)</a>
+
+   🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo-0.5B-GGUF" target="_blank">Apollo-0.5B-GGUF</a> • 🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo-2B-GGUF" target="_blank">Apollo-2B-GGUF</a>  • 🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo-6B-GGUF" target="_blank">Apollo-6B-GGUF</a> • 🤗 <a href="https://huggingface.co/FreedomIntelligence/Apollo-7B-GGUF" target="_blank">Apollo-7B-GGUF</a> 
+   
+   
+   
    ![Apollo](assets/result.png)
-      
-
-  
 
 
 ## Dataset & Evaluation
 
 - Dataset
-  🤗 <a href="https://huggingface.co/datasets/FreedomIntelligence/ApolloCorpus" target="_blank">ApolloCorpus</a>
-  
-  <details><summary>Click to expand</summary>
+  🤗 <a href="https://huggingface.co/datasets/FreedomIntelligence/ApolloCorpus" target="_blank">ApolloCorpus
+
+   <details><summary>Click to expand</summary>
 
     ![Apollo](assets/dataset.png)
 
-    - [Zip File](https://huggingface.co/datasets/FreedomIntelligence/ApolloCorpus/blob/main/ApolloCorpus.zip)
-    - [Data category](https://huggingface.co/datasets/FreedomIntelligence/ApolloCorpus/tree/main/train)
+    - [Zip File](https://huggingface.co/datasets/FreedomIntelligence/Medbase_data/blob/main/Medbase_data-datasets.zip)
+    - [Data category](https://huggingface.co/datasets/FreedomIntelligence/Medbase_data/tree/main/train)
        - Pretrain:
          - data item:
             - json_name: {data_source}_{language}_{data_type}.json
@@ -85,18 +78,16 @@ Covering English, Chinese, French, Hindi, Spanish, Hindi, Arabic So far
                 ],
                 ...
               ]
-            ```
+              ```
 
 
    </details>
-
-
-
+   
 - Evaluation
-  🤗 <a href="https://huggingface.co/datasets/FreedomIntelligence/XMedbench" target="_blank">XMedBench</a>
+  🤗 <a href="https://huggingface.co/datasets/FreedomIntelligence/XMedbench" target="_blank">XMedBench</a> 
 
-  <details><summary>Click to expand</summary>
-  
+   <details><summary>Click to expand</summary>
+      
      - EN:
        - [MedQA-USMLE](https://huggingface.co/datasets/GBaker/MedQA-USMLE-4-options) 
        - [MedMCQA](https://huggingface.co/datasets/medmcqa/viewer/default/test)
@@ -123,17 +114,77 @@ Covering English, Chinese, French, Hindi, Spanish, Hindi, Arabic So far
 
    </details>
 
-
+   
 ## Results reproduction
    <details><summary>Click to expand</summary>
 
-   **Waiting for Update**
-      
+   1. Download Dataset for project:
 
+      ```
+      bash 0.download_data.sh
+      ```
+    
+   2. Prepare test and dev for specific model:
 
+      
+      - Create test data for with special token, you can use ./util/check.ipynb to check models' special tokens
+        
+       ```
+       bash 1.data_process_test&dev.sh
+       ```
+    
+   3. Prepare train data for specific model (Create tokenized data in advance):
+
+    
+      - You can adjust data Training order and Training Epoch in this step
+
+       ```
+       bash 2.data_process_train.sh
+       ```
+    
+   4. Train the model
+
+    
+      - Multi Nodes refer to ./scripts/multi_node_train_*.sh
+       ```
+       pip install causal-conv1d>=1.2.0
+       pip install mamba-ssm
+       ```
+
+       Node 0: 
+       ```
+       bash ./scripts/3.multinode_train_jamba_rank0.sh
+       ```
+       ...
+       Node 4: 
+       ```
+       bash ./scripts/3.multinode_train_jamba_rank4.sh
+       ```
+
+
+   5. Evaluate your model: Generate score for benchmark
+      
+         ```
+         bash 4.eval.sh
+         ```
+
+   6. Evaluate your model: Play with your ckpts in bash
+    
+         ```
+         python ./src/evaluate/cli_demo.py --model_name='./ckpts/your/path/tfmr'
+         ```
+   
    </details>
 
+## To do 
+
+- Long Context Capability Evaluation and new Long-Med Benchmark
+
+##  Acknowledgment
 
+- [HuatuoGPT-II](https://github.com/FreedomIntelligence/HuatuoGPT-II)
+- [proxy-tuning](https://github.com/alisawuffles/proxy-tuning)
+- [Apollo](https://github.com/FreedomIntelligence/Apollo)
 
 
 ##  Citation