File size: 3,839 Bytes
ef01f92 ff93b09 ef01f92 9c66e58 ef01f92 a892e9a ff93b09 e71aa94 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 | ---
language:
- en
license: cc-by-nc-4.0
tags:
- text-generation-inference
- transformers
- unsloth
- mistral
- trl
base_model: alnrg2arg/blockchainlabs_7B_merged_test2_4
datasets:
- Open-Orca/SlimOrca
---
# Uploaded model
- **Finetuned from model :** alnrg2arg/blockchainlabs_7B_merged_test2_4
This is a SFT version of the model from blockchainlab test 2.4 - alnrg2arg/blockchainlabs_7B_merged_test2_4.
The project is running to make a small LLM for a on-device purpose.
Overall pipeline for this iteration is
1.Merging to make a base model (7B)
2.Prune the model to reduce the parameter (50% sparcity)
3.For recovery phase of the pruning, the DPO is chosen.
This model which is not pruned is intended to compare with the pruned model.
DPO consists of two parts : SFT and DPO - Now this model is the intermediate format (SFT)
This model can also be compared to the DPO version of the model.
This is the code and parameters I chose for this model(SFT).
```
from transformers import TrainingArguments
from trl import SFTTrainer
from datasets import load_dataset
from unsloth import FastLanguageModel, FastMistralModel
max_seq_length = 2048 # Supports automatic RoPE Scaling, so choose any number
# Load model
model, tokenizer = FastMistralModel.from_pretrained(
model_name = "alnrg2arg/blockchainlabs_7B_merged_test2_4,
max_seq_length = max_seq_length,
dtype = None, # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
load_in_4bit = True, # Use 4bit quantization to reduce memory usage. Can be False
#device_map = "balanced"
# token = "hf_...", # use one if using gated models like meta-llama/Llama-2-7b-hf
)
model = FastMistralModel.get_peft_model(
model,
r = 16,
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",],
lora_alpha = 16,
lora_dropout = 0, # Dropout = 0 is currently optimized
bias = "none", # Bias = "none" is currently optimized
use_gradient_checkpointing = True,
random_state = 3407,
max_seq_length = max_seq_length,
)
```
The code and parameters are borrowed from https://colab.research.google.com/drive/1SKrKGV-BZoU4kv5q3g0jtE_OhRgPtrrQ?usp=sharing
Benchmark scores
| Tasks |Version|Filter|n-shot| Metric |Value | |Stderr|
|-------------|------:|------|-----:|--------|-----:|---|-----:|
|arc_challenge| 1|none | 25|acc |0.7116|± |0.0132|
| | |none | 25|acc_norm|0.7346|± |0.0129|
| Tasks |Version|Filter|n-shot| Metric |Value | |Stderr|
|---------|------:|------|-----:|--------|-----:|---|-----:|
|hellaswag| 1|none | 10|acc |0.7222|± |0.0045|
| | |none | 10|acc_norm|0.8865|± |0.0032|
| Tasks |Version|Filter|n-shot|Metric|Value | |Stderr|
|--------------|------:|------|-----:|------|-----:|---|-----:|
|truthfulqa_mc2| 2|none | 0|acc |0.7043|± | 0.015|
| Groups |Version|Filter|n-shot|Metric|Value | |Stderr|
|------------------|-------|------|-----:|------|-----:|---|-----:|
|mmlu |N/A |none | 0|acc |0.6367|± |0.1258|
| - humanities |N/A |none | 5|acc |0.5968|± |0.1122|
| - other |N/A |none | 5|acc |0.7049|± |0.1123|
| - social_sciences|N/A |none | 5|acc |0.7374|± |0.0774|
| - stem |N/A |none | 5|acc |0.5309|± |0.1373|
| Tasks |Version|Filter|n-shot|Metric|Value | |Stderr|
|----------|------:|------|-----:|------|-----:|---|-----:|
|winogrande| 1|none | 5|acc |0.8477|± |0.0101|
|Tasks|Version| Filter |n-shot| Metric |Value | |Stderr|
|-----|------:|----------|-----:|-----------|-----:|---|-----:|
|gsm8k| 2|get-answer| 5|exact_match|0.7468|± | 0.012|
Average 75.94 |