Update README.md

README.md

@@ -21,8 +21,76 @@ We developed a domain-speciffc large language-vision assistant (PA-LLaVA) for pa
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/663f06e01cd68975883a353e/IAeFWhH8brZYDaTJnew2N.png)
 
 
-Only the image names of the cleaned dataset are provided here, for the specific training code please visit our Github.
+### Step 1 Download the public datasets.
+Here we only provide the download link for the public dataset and expose the image id index of our cleaned dataset on Hugging Face.
 
+#### Domain Alignment Stage
+
+PubMedVision-Alignment: [FreedomIntelligence/PubMedVision · Datasets at Hugging Face](https://huggingface.co/datasets/FreedomIntelligence/PubMedVision)
+
+PMC-OA: [axiong/pmc_oa · Datasets at Hugging Face](https://huggingface.co/datasets/axiong/pmc_oa)
+
+Quilt-1M: [Quilt-1M: One Million Image-Text Pairs for Histopathology (zenodo.org)](https://zenodo.org/records/8239942)
+
+
+#### Instruction Tuning Stage
+
+PathVQA: https://drive.google.com/drive/folders/1G2C2_FUCyYQKCkSeCRRiTTsLDvOAjFj5
+
+PMC-VQA: [xmcmic/PMC-VQA · Datasets at Hugging Face](https://huggingface.co/datasets/xmcmic/PMC-VQA)
+
+
+#### Categorical dataset for zero-sample testing
+
+ICIAR 2018 BACH: https://iciar2018-challenge.grand-challenge.org/Download/
+
+OSCC: https://data.mendeley.com/datasets/ftmp4cvtmb/1 
+
+ColonPath : https://medfm2023.grand-challenge.org/datasets
+
+
+### Step 2 Data processing.
+First, use the image index of the clean dataset provided by us to extract the human pathological dataset, and then process it into the following format:
+```
+[
+	{
+		"image": ,
+		"caption": 
+	},
+]
+```
+
+Finally, run dataformate.py to get the format needed to train the model.
+```
+python dataformat.py
+```
+
+## Moddel
+Our released weights are distributed training weights that can be directly loaded for training through XTuner. If you need merged weights, they can be merged using XTuner (using the weights from the domain alignment phase as an example):
+```
+xtuner convert pth_to_hf path/pallava_domain_alignment.py ./domain_alignment_weight.pth ./domain_alignment_weight_ft
+xtuner convert merge meta-llama/Meta-Llama-3-8B-Instruct ./domain_alignment_weight_ft/llm_adapter ./domain_alignment_weight_ft/llm_merge_lora
+
+```
+
+## Training
+
+We used xtuner as a training tool, so please go to xtuner official to complete the environment configuration [https://github.com/InternLM/xtuner]. Then place the pallava folder under the xtuner_add folder into the xtuner folder.
+
+
+#### Domain Alignment
+```
+NPROC_PER_NODE=8 NNODES=2 PORT=12345 ADDR= NODE_RANK=0 xtuner train pallava_domain_alignment.py --deepspeed deepspeed_zero2 --seed 1024
+```
+
+#### Instruction Tuning
+```
+NPROC_PER_NODE=8 NNODES=2 PORT=12345 ADDR= NODE_RANK=0 xtuner train pallava_instruction_tuning.py --deepspeed deepspeed_zero2 --seed 1024
+```
+
+## Result
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/663f06e01cd68975883a353e/ng9KbUevJk5HyYONpOg2S.png)
 
 ## contact