Update README.md

README.md

@@ -17,17 +17,34 @@ This lora was trained on 250k post and response pairs from 43 different fincial,
 * Training code will be released soon.
 * Dataset and tools for building the dataset will be released soon.
 
-# Training Details
+## Training Details
 
-Coming soon. 
+1 note worthy change I will mention now, is this was trained with casualLM rather than seq2seq like a number of the other instruct models have been. I can't explain why they used seq2seq for data collators, other than that's what alpaca lora originally used. Llama as a generative model was trained for casualLM so to me it makes sense to use that when fine tuning.
 
-# Usage
+* More coming soon. 
+
+### Training Hyperparams
+
+| Hyperparameter | LLaMA-7B |
+|----------------|----------|
+| Learning rate  | 2.5e-4         | 
+| Epochs         | 3              | 
+| optim          | adamw_bnb_8bit |
+| Warmup step    | 300            | 
+| LR scheduler   | polynomial     |
+| lora_r         | 32       | 
+| lora_alpha     | 64       | 
+| lora_dropout   | 0.05     |
+| lora_target_modules | ["q_proj", "v_proj"] | 
+
+
+## Usage
 
 This is a lora, and needs to be loaded with a 7B llama, such as in text-generation-webui, https://github.com/oobabooga/text-generation-webui/blob/main/docs/Using-LoRAs.md
 
 * inference code and other scripts may follow. 
 
-# Prompting
+## Prompting
 
 Editing the system prompt can have some effect on the replies. 
 ```
@@ -43,7 +60,7 @@ You are an experienced financial analyst. You are tasked with responding to user
 <|RESPONSE|>
 ```
 
-# Examples:
+## Examples:
 
 ```
 <|SYSTEM|>
@@ -93,7 +110,7 @@ Just make sure it works well enough, and leave it at that.
 <|END_RESPONSE|>
 ```
 
-# Evaluation
+## Evaluation
 
 In progress.