|
|
--- |
|
|
language: |
|
|
- en |
|
|
license: cc-by-nc-4.0 |
|
|
tags: |
|
|
- text-generation-inference |
|
|
- transformers |
|
|
- unsloth |
|
|
- mistral |
|
|
- trl |
|
|
base_model: alnrg2arg/blockchainlabs_7B_merged_test2_4 |
|
|
datasets: |
|
|
- Open-Orca/SlimOrca |
|
|
--- |
|
|
|
|
|
# Uploaded model |
|
|
|
|
|
- **Finetuned from model :** alnrg2arg/blockchainlabs_7B_merged_test2_4 |
|
|
|
|
|
This is a SFT version of the model from blockchainlab test 2.4 - alnrg2arg/blockchainlabs_7B_merged_test2_4. |
|
|
|
|
|
The project is running to make a small LLM for a on-device purpose. |
|
|
|
|
|
Overall pipeline for this iteration is |
|
|
|
|
|
1.Merging to make a base model (7B) |
|
|
2.Prune the model to reduce the parameter (50% sparcity) |
|
|
3.For recovery phase of the pruning, the DPO is chosen. |
|
|
|
|
|
This model which is not pruned is intended to compare with the pruned model. |
|
|
|
|
|
DPO consists of two parts : SFT and DPO - Now this model is the intermediate format (SFT) |
|
|
This model can also be compared to the DPO version of the model. |
|
|
|
|
|
|
|
|
This is the code and parameters I chose for this model(SFT). |
|
|
|
|
|
``` |
|
|
from transformers import TrainingArguments |
|
|
from trl import SFTTrainer |
|
|
from datasets import load_dataset |
|
|
from unsloth import FastLanguageModel, FastMistralModel |
|
|
|
|
|
|
|
|
max_seq_length = 2048 # Supports automatic RoPE Scaling, so choose any number |
|
|
|
|
|
# Load model |
|
|
model, tokenizer = FastMistralModel.from_pretrained( |
|
|
model_name = "alnrg2arg/blockchainlabs_7B_merged_test2_4, |
|
|
max_seq_length = max_seq_length, |
|
|
dtype = None, # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+ |
|
|
load_in_4bit = True, # Use 4bit quantization to reduce memory usage. Can be False |
|
|
#device_map = "balanced" |
|
|
# token = "hf_...", # use one if using gated models like meta-llama/Llama-2-7b-hf |
|
|
) |
|
|
|
|
|
model = FastMistralModel.get_peft_model( |
|
|
model, |
|
|
r = 16, |
|
|
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", |
|
|
"gate_proj", "up_proj", "down_proj",], |
|
|
lora_alpha = 16, |
|
|
lora_dropout = 0, # Dropout = 0 is currently optimized |
|
|
bias = "none", # Bias = "none" is currently optimized |
|
|
use_gradient_checkpointing = True, |
|
|
random_state = 3407, |
|
|
max_seq_length = max_seq_length, |
|
|
) |
|
|
``` |
|
|
|
|
|
The code and parameters are borrowed from https://colab.research.google.com/drive/1SKrKGV-BZoU4kv5q3g0jtE_OhRgPtrrQ?usp=sharing |
|
|
|
|
|
|
|
|
Benchmark scores |
|
|
|
|
|
| Tasks |Version|Filter|n-shot| Metric |Value | |Stderr| |
|
|
|-------------|------:|------|-----:|--------|-----:|---|-----:| |
|
|
|arc_challenge| 1|none | 25|acc |0.7116|± |0.0132| |
|
|
| | |none | 25|acc_norm|0.7346|± |0.0129| |
|
|
|
|
|
| Tasks |Version|Filter|n-shot| Metric |Value | |Stderr| |
|
|
|---------|------:|------|-----:|--------|-----:|---|-----:| |
|
|
|hellaswag| 1|none | 10|acc |0.7222|± |0.0045| |
|
|
| | |none | 10|acc_norm|0.8865|± |0.0032| |
|
|
|
|
|
| Tasks |Version|Filter|n-shot|Metric|Value | |Stderr| |
|
|
|--------------|------:|------|-----:|------|-----:|---|-----:| |
|
|
|truthfulqa_mc2| 2|none | 0|acc |0.7043|± | 0.015| |
|
|
|
|
|
| Groups |Version|Filter|n-shot|Metric|Value | |Stderr| |
|
|
|------------------|-------|------|-----:|------|-----:|---|-----:| |
|
|
|mmlu |N/A |none | 0|acc |0.6367|± |0.1258| |
|
|
| - humanities |N/A |none | 5|acc |0.5968|± |0.1122| |
|
|
| - other |N/A |none | 5|acc |0.7049|± |0.1123| |
|
|
| - social_sciences|N/A |none | 5|acc |0.7374|± |0.0774| |
|
|
| - stem |N/A |none | 5|acc |0.5309|± |0.1373| |
|
|
|
|
|
| Tasks |Version|Filter|n-shot|Metric|Value | |Stderr| |
|
|
|----------|------:|------|-----:|------|-----:|---|-----:| |
|
|
|winogrande| 1|none | 5|acc |0.8477|± |0.0101| |
|
|
|
|
|
|Tasks|Version| Filter |n-shot| Metric |Value | |Stderr| |
|
|
|-----|------:|----------|-----:|-----------|-----:|---|-----:| |
|
|
|gsm8k| 2|get-answer| 5|exact_match|0.7468|± | 0.012| |
|
|
|
|
|
|
|
|
|
|
|
Average 75.94 |
