Update README.md

README.md

@@ -1,58 +1,144 @@
 ---
-license: other
-library_name: transformers
-tags:
-- trl
-- sft
-- generated_from_trainer
-base_model: AlphaRandy/WhelanBot
-model-index:
-- name: WhelanBot
-  results: []
-pipeline_tag: question-answering
+inference:
+  parameters:
+    temperature: 0.5
+widget:
+- messages:
+  - role: user
+    content: Hey Bud
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
+## Instruction format
 
-# gemma-chatbot
+This format must be strictly respected, otherwise the model will generate sub-optimal outputs.
 
-This model is a fine-tuned version of [google/gemma-2b-it](https://huggingface.co/google/gemma-2b-it) on the None dataset.
+The template used to build a prompt for the Instruct model is defined as follows:
+```
+<s> [INST] Instruction [/INST] Model answer</s> [INST] Follow-up instruction [/INST]
+```
+Note that `<s>` and `</s>` are special tokens for beginning of string (BOS) and end of string (EOS) while [INST] and [/INST] are regular strings.
 
-## Model description
+As reference, here is the pseudo-code used to tokenize instructions during fine-tuning:
+```python
+def tokenize(text):
+    return tok.encode(text, add_special_tokens=False)
 
-More information needed
+[BOS_ID] + 
+tokenize("[INST]") + tokenize(USER_MESSAGE_1) + tokenize("[/INST]") +
+tokenize(BOT_MESSAGE_1) + [EOS_ID] +
+…
+tokenize("[INST]") + tokenize(USER_MESSAGE_N) + tokenize("[/INST]") +
+tokenize(BOT_MESSAGE_N) + [EOS_ID]
+```
 
-## Intended uses & limitations
+In the pseudo-code above, note that the `tokenize` method should not add a BOS or EOS token automatically, but should add a prefix space. 
 
-More information needed
+In the Transformers library, one can use [chat templates](https://huggingface.co/docs/transformers/main/en/chat_templating) which make sure the right format is applied.
 
-## Training and evaluation data
+## Run the model
 
-More information needed
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
 
-## Training procedure
+model_id = "AlphaRandy/WhelanBot"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
 
-### Training hyperparameters
+model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
 
-The following hyperparameters were used during training:
-- learning_rate: 0.0002
-- train_batch_size: 1
-- eval_batch_size: 8
-- seed: 42
-- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-- lr_scheduler_type: cosine
-- training_steps: 250
-- mixed_precision_training: Native AMP
+messages = [
+    {"role": "user", "content": "What is your favourite condiment?"},
+    {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
+    {"role": "user", "content": "Do you have mayonnaise recipes?"}
+]
 
-### Training results
+inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to("cuda")
 
+outputs = model.generate(inputs, max_new_tokens=20)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
 
+By default, transformers will load the model in full precision. Therefore you might be interested to further reduce down the memory requirements to run the model through the optimizations we offer in HF ecosystem:
 
-### Framework versions
+### In half-precision
 
-- PEFT 0.10.0
-- Transformers 4.40.0.dev0
-- Pytorch 2.2.1+cu121
-- Datasets 2.18.0
-- Tokenizers 0.15.2
+Note `float16` precision only works on GPU devices
+
+<details>
+<summary> Click to expand </summary>
+
+```diff
++ import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "AlphaRandy/WhelanBot"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+
++ model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")
+
+messages = [
+    {"role": "user", "content": "What is your favourite condiment?"},
+    {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
+    {"role": "user", "content": "Do you have mayonnaise recipes?"}
+]
+
+input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt").to("cuda")
+
+outputs = model.generate(input_ids, max_new_tokens=20)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+</details>
+
+### Lower precision using (8-bit & 4-bit) using `bitsandbytes`
+
+<details>
+<summary> Click to expand </summary>
+
+```diff
++ import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "AlphaRandy/WhelanBot"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+
++ model = AutoModelForCausalLM.from_pretrained(model_id, load_in_4bit=True, device_map="auto")
+
+text = "Hello my name is"
+messages = [
+    {"role": "user", "content": "What is your favourite condiment?"},
+    {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
+    {"role": "user", "content": "Do you have mayonnaise recipes?"}
+]
+
+input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt").to("cuda")
+
+outputs = model.generate(input_ids, max_new_tokens=20)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+</details>
+
+### Load the model with Flash Attention 2
+
+<details>
+<summary> Click to expand </summary>
+
+```diff
++ import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "AlphaRandy/WhelanBot"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+
++ model = AutoModelForCausalLM.from_pretrained(model_id, use_flash_attention_2=True, device_map="auto")
+
+messages = [
+    {"role": "user", "content": "What is your favourite condiment?"},
+    {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
+    {"role": "user", "content": "Do you have mayonnaise recipes?"}
+]
+
+input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt").to("cuda")
+
+outputs = model.generate(input_ids, max_new_tokens=20)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+</details>