Update README.md

README.md

@@ -1,3 +1,43 @@
 ---
 license: apache-2.0
+datasets:
+- tatsu-lab/alpaca
 ---
+
+This repo contains a low-rank adapter for LLaMA-7b
+fit on the [Stanford Alpaca](https://github.com/tatsu-lab/stanford_alpaca) dataset.
+
+This version of the weights was trained with the following hyperparameters:
+
+- Epochs: 3 (load from best epoch)
+- Batch size: 32
+- Learning rate: 1e-4
+- Lora _r_: 8
+- lora_alpha : 16
+- Lora target modules: q_proj, v_proj
+
+That is:
+
+```
+python train_alpaca_lora.py \
+    --model_name_or_path  decapoda-research/llama-7b-hf  \
+    --data_path tatsu-lab/alpaca  \
+    --output_dir work_dir_lora/ \
+    --num_train_epochs 3 \
+    --per_device_train_batch_size 4 \
+    --per_device_eval_batch_size 4 \
+    --gradient_accumulation_steps 8 \
+    --evaluation_strategy "no" \
+    --save_strategy "steps" \
+    --save_steps 500 \
+    --save_total_limit 5 \
+    --learning_rate 1e-4 \
+    --weight_decay 0. \
+    --warmup_ratio 0.03 \
+    --lr_scheduler_type "cosine" \
+    --model_max_length 2048 \
+    --logging_steps 1 \
+    --fp16 True
+```
+
+Instructions for running it can be found at https://github.com/jianzhnie/open-chatgpt.