Update README.md

README.md

@@ -7,8 +7,15 @@ datasets:
 This repo reproduced [tloen/alpaca-lora-7b](https://huggingface.co/tloen/alpaca-lora-7b)
 fit on the [Stanford Alpaca](https://github.com/tatsu-lab/stanford_alpaca) dataset.
 
-The training log is in [W&B link](https://wandb.ai/sharpbai/alpaca-lora-reproduce/runs/08ulvstd),
-4x H100 training for about 1h15min
+4x H100 training for about 1h15min, details in [W&B link](https://wandb.ai/sharpbai/alpaca-lora-reproduce/runs/08ulvstd), there is a hyperparameter of val_set_size=500
+
+4 x 4090 training for about 4h35min, details in [W&B link](https://wandb.ai/sharpbai/alpaca-lora-reproduce/runs/ws16av1u), all key hyperparameters are the same
+
+To optimize the running speed, I change these code
+
+- `load_in_8bits=False` to use 16bit finetune
+- comment `model = prepare_model_for_int8_training` to not turn some parameters to fp32 and turn off gradient checkpointing
+- for 4090 enable gradient checkpointing, add `model.gradient_checkpointing_enable()` and `model.enable_input_require_grads()`
 
 This version of the weights was trained with the following hyperparameters: