Update README.md

README.md

@@ -1,205 +1,226 @@
 ---
+license: apache-2.0
 library_name: peft
+tags:
+- mistral
+datasets:
+- jondurbin/airoboros-2.2.1
+inference: false
+pipeline_tag: text-generation
 base_model: mistralai/Mistral-7B-v0.1
 ---
 
-# Model Card for Model ID
+<div align="center">
 
-<!-- Provide a quick summary of what the model is/does. -->
+<img src="./logo.png" width="150px">
 
+</div>
 
+# Mistral-7B-Instruct-v0.1
 
-## Model Details
-
-### Model Description
-
-<!-- Provide a longer summary of what this model is. -->
-
-
-
-- **Developed by:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-
-## Uses
-
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
-
-## Training Details
-
-### Training Data
-
-<!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure 
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
+The Mistral-7B-Instruct-v0.1 LLM is a pretrained generative text model with 7 billion parameters geared towards instruction-following capabilities.
 
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Data Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
+## Model Details
 
-#### Hardware
+This model was built via parameter-efficient finetuning of the [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) base model on the [jondurbin/airoboros-2.2.1](https://huggingface.co/datasets/jondurbin/airoboros-2.2.1) dataset. Finetuning was executed on 1x A100 (40 GB SXM) for roughly 3 hours.
+
+- **Developed by:** Daniel Furman
+- **Model type:** Decoder-only
+- **Language(s) (NLP):** English
+- **License:** Apache 2.0
+- **Finetuned from model:** [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1)
+
+## Model Sources 
+
+- **Repository:** [github.com/daniel-furman/sft-demos](https://github.com/daniel-furman/sft-demos/blob/main/src/sft/one_gpu/mistral/sft-mistral-7b-instruct-peft.ipynb)
+
+## Evaluation Results
+
+| Metric                | Value |
+|-----------------------|-------|
+| MMLU (5-shot)         | Coming |
+| ARC (25-shot)         | Coming |
+| HellaSwag (10-shot)   | Coming |
+| TruthfulQA (0-shot)   | Coming |
+| Avg.                  | Coming |
+
+We use Eleuther.AI's [Language Model Evaluation Harness](https://github.com/EleutherAI/lm-evaluation-harness) to run the benchmark tests above, the same version as Hugging Face's [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).
+
+## Basic Usage
+
+<details>
+
+<summary>Setup</summary>
+
+```python
+!pip install -q -U transformers peft torch accelerate bitsandbytes einops sentencepiece
+
+import torch
+from peft import PeftModel, PeftConfig
+from transformers import (
+    AutoModelForCausalLM,
+    AutoTokenizer,
+    BitsAndBytesConfig,
+)
+```
 
-[More Information Needed]
+```python
+peft_model_id = "dfurman/Mistral-7B-Instruct-v0.1"
+config = PeftConfig.from_pretrained(peft_model_id)
 
-#### Software
+tokenizer = AutoTokenizer.from_pretrained(
+    peft_model_id,
+    use_fast=True,
+    trust_remote_code=True,
+)
+bnb_config = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_quant_type="nf4",
+    bnb_4bit_compute_dtype=torch.bfloat16,
+)
+model = AutoModelForCausalLM.from_pretrained(
+    config.base_model_name_or_path,
+    quantization_config=bnb_config,
+    device_map="auto",
+    trust_remote_code=True,
+)
+model = PeftModel.from_pretrained(
+    model, 
+    peft_model_id
+)
+```
 
-[More Information Needed]
+</details>
 
-## Citation [optional]
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+```python
+messages = [
+    {"role": "user", "content": "Tell me a recipe for a mai tai."},
+]
 
-**BibTeX:**
+print("\n\n*** Prompt:")
+prompt = tokenizer.apply_chat_template(
+    messages,
+    tokenize=False,
+    add_generation_prompt=True
+)
+print(prompt)
 
-[More Information Needed]
+print("\n\n*** Generate:")
+input_ids = tokenizer(prompt, return_tensors="pt").input_ids.cuda()
+with torch.autocast("cuda", dtype=torch.bfloat16):
+    output = model.generate(
+        input_ids=input_ids,
+        max_new_tokens=1024,
+        do_sample=True,
+        temperature=0.7,
+        return_dict_in_generate=True,
+        eos_token_id=tokenizer.eos_token_id,
+        pad_token_id=tokenizer.pad_token_id,
+        repetition_penalty=1.2,
+        no_repeat_ngram_size=5,
+    )
+
+response = tokenizer.decode(
+    output["sequences"][0][len(input_ids[0]):], 
+    skip_special_tokens=True
+)
+print(response)
+```
+
+<details>
+
+<summary>Output</summary>
+
+**Prompt**: 
+```python
+coming
+```
+
+**Generation**:
+```python
+coming
+```
+
+</details>
+
+
+## Speeds, Sizes, Times 
+
+| runtime / 50 tokens (sec) | GPU             | attn | torch dtype | VRAM (GB) |
+|:-----------------------------:|:----------------------:|:---------------------:|:-------------:|:-----------------------:|
+| 3.1                        | 1x A100 (40 GB SXM)  | torch               | fp16    | 13                    |
+
+## Training
+
+It took ~3 hours to train 3 epochs on 1x A100 (40 GB SXM).
+
+### Prompt Format
+
+This model was finetuned with the following format:
+
+```python
+tokenizer.chat_template = "{{ bos_token }}{% for message in messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if message['role'] == 'user' %}{{ '[INST] ' + message['content'] + ' [/INST] ' }}{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token + ' ' }}{% else %}{{ raise_exception('Only user and assistant roles are supported!') }}{% endif %}{% endfor %}"
+```
+
+
+This format is available as a [chat template](https://huggingface.co/docs/transformers/main/chat_templating) via the `apply_chat_template()` method. Here's an illustrative example:
+
+```python
+messages = [
+    {"role": "user", "content": "What is your favourite condiment?"},
+    {"role": "assistant", "content": "Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!"},
+    {"role": "user", "content": "Do you have mayonnaise recipes?"}
+]
+prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+print(prompt)
+```
+
+<details>
+
+<summary>Output</summary>
 
-**APA:**
+```python
+coming
+```
+</details>
 
-[More Information Needed]
+### Training Hyperparameters
 
-## Glossary [optional]
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+We use the [SFTTrainer](https://huggingface.co/docs/trl/main/en/sft_trainer) from `trl` to fine-tune LLMs on instruction-following datasets.
 
-[More Information Needed]
+The following `TrainingArguments` config was used:
 
-## More Information [optional]
+- num_train_epochs = 1
+- auto_find_batch_size = True
+- gradient_accumulation_steps = 1
+- optim = "paged_adamw_32bit"
+- save_strategy = "epoch"
+- learning_rate = 3e-4
+- lr_scheduler_type = "cosine"
+- warmup_ratio = 0.03
+- logging_strategy = "steps"
+- logging_steps = 25
+- bf16 = True
 
-[More Information Needed]
+The following `bitsandbytes` quantization config was used:
 
-## Model Card Authors [optional]
+- quant_method: bitsandbytes
+- load_in_8bit: False
+- load_in_4bit: True
+- llm_int8_threshold: 6.0
+- llm_int8_skip_modules: None
+- llm_int8_enable_fp32_cpu_offload: False
+- llm_int8_has_fp16_weight: False
+- bnb_4bit_quant_type: nf4
+- bnb_4bit_use_double_quant: False
+- bnb_4bit_compute_dtype: bfloat16
 
-[More Information Needed]
 
 ## Model Card Contact
 
-[More Information Needed]
-
-
-## Training procedure
-
+dryanfurman at gmail
 
 ### Framework versions