Update README.md

README.md

@@ -12,7 +12,6 @@ language:
 This is a DPO finetune of Mistral 7b-instruct0.2 following the article: https://towardsdatascience.com/fine-tune-a-mistral-7b-model-with-direct-preference-optimization-708042745aac
 
 
-
 ## Model Details
 
 ### Model Description
@@ -25,39 +24,48 @@ This is the model card of a 🤗 transformers model that has been pushed on the
 - **Finetuned from model: mistralai/Mistral-7B-Instruct-v0.2
 
 
-## Uses
-
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+## Instruction format
 
-[More Information Needed]
-
-### Downstream Use [optional]
+In order to leverage instruction fine-tuning, your prompt should be surrounded by `[INST]` and `[/INST]` tokens. The very first instruction should begin with a begin of sentence id. The next instructions should not. The assistant generation will be ended by the end-of-sentence token id.
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
+E.g.
+```
+text = "<s>[INST] What is your favourite condiment? [/INST]"
+"Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!</s> "
+"[INST] Do you have mayonnaise recipes? [/INST]"
+```
 
-### Out-of-Scope Use
+This format is available as a [chat template](https://huggingface.co/docs/transformers/main/chat_templating) via the `apply_chat_template()` method:
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
 
-[More Information Needed]
+device = "cuda" # the device to load the model onto
 
-## Bias, Risks, and Limitations
+model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")
+tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+messages = [
+    {"role": "user", "content": "What is your favourite condiment?"},
+    {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
+    {"role": "user", "content": "Do you have mayonnaise recipes?"}
+]
 
-[More Information Needed]
+encodeds = tokenizer.apply_chat_template(messages, return_tensors="pt")
 
-### Recommendations
+model_inputs = encodeds.to(device)
+model.to(device)
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+generated_ids = model.generate(model_inputs, max_new_tokens=1000, do_sample=True)
+decoded = tokenizer.batch_decode(generated_ids)
+print(decoded[0])
+```
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+## Model Architecture
+This instruction model is based on Mistral-7B-v0.1, a transformer model with the following architecture choices:
+- Grouped-Query Attention
+- Sliding-Window Attention
+- Byte-fallback BPE tokenizer
 
 ## How to Get Started with the Model
 
@@ -69,28 +77,54 @@ Use the code below to get started with the model.
 
 ### Training Data
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
+Intel/orca_dpo_pairs
 
 ### Training Procedure 
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+https://medium.com/towards-data-science/fine-tune-a-mistral-7b-model-with-direct-preference-optimization-708042745aac
 
 #### Preprocessing [optional]
 
-[More Information Needed]
+def chatml_format(example):
+    # Format system
+    if len(example['system']) > 0:
+        message = {"role": "user", "content": f"{example['system']}\n{example['question']}"}
+        prompt = tokenizer.apply_chat_template([message], tokenize=False)
+    else:
+        # Format instruction
+        message = {"role": "user", "content": example['question']}
+        prompt = tokenizer.apply_chat_template([message], tokenize=False, add_generation_prompt=True)
 
+    # Format chosen answer
+    chosen = example['chosen'] + tokenizer.eos_token
 
-#### Training Hyperparameters
+    # Format rejected answer
+    rejected = example['rejected'] + tokenizer.eos_token
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+    return {
+        "prompt": prompt,
+        "chosen": chosen,
+        "rejected": rejected,
+    }
 
-#### Speeds, Sizes, Times [optional]
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+#### Training Hyperparameters
 
-[More Information Needed]
+training_args = TrainingArguments(
+    per_device_train_batch_size=4,
+    gradient_accumulation_steps=4,
+    gradient_checkpointing=True,
+    learning_rate=5e-5,
+    lr_scheduler_type="cosine",
+    max_steps=200,
+    save_strategy="no",
+    logging_steps=1,
+    output_dir=new_model,
+    optim="paged_adamw_32bit",
+    warmup_steps=100,
+    bf16=True,
+    report_to="wandb",
+)
 
 ## Evaluation