File size: 4,045 Bytes
571e9a7 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 | # The Super Weight in Large Language Models
Pruning as few as a **single parameter** can destroy an LLM's ability to
generate text -- increasing perplexity by 3 orders of magnitude and reducing
zero-shot accuracy to guessing. We propose a data-free method for identifying
such parameters, termed *super weights*, using a single forward pass through
the model.
To reproduce any of the results from the paper, use the corresponding bash
scripts found in the `scripts/`. *Run these bash scripts from the root of the
repository*, such as `bash scripts/table1_superweight_importance.sh`.
```
scripts/
table1_superweight_importance.sh
figure3_how_to_identify_superweight.sh
figure4_super_activation.sh
```
This repository supports the following models.
```
allenai/OLMo-1B-0724-hf
allenai/OLMo-7B-0724-hf
mistralai/Mistral-7B-v0.1
mistralai/Mistral-7B-Instruct-v0.1
microsoft/Phi-3-mini-4k-instruct
huggyllama/llama-7B # up to 30B
```
## Ablation: Importance of SW
```
model_name=huggyllama/llama-7B
outlier_method=search_percentage_1e-6_0-32_all
for restore_and_scale_GO in 1.0 0.0
do
python evaluate.py --model hf-outlier \
--model_args pretrained=${model_name},outlier_method=${outlier_method},restore_and_scale_GO=${restore_and_scale_GO},dtype=float16 \
--tasks winogrande,arc_challenge,arc_easy,piqa,sciq,hellaswag,lambada_openai \
--device cuda:0 \
--batch_size 16 \
--output_path outputs/${model_name}/search/${outlier_method}_restore_and_scale-${restore_and_scale_GO} \
done
for outlier_method in manual_scaling_SO_0.0 manual_scaling_SO_1.0
do
python evaluate.py --model hf-outlier \
--model_args pretrained=${model_name},outlier_method=${outlier_method},dtype=float16 \
--tasks winogrande,arc_challenge,arc_easy,piqa,sciq,hellaswag,lambada_openai \
--device cuda:0 \
--batch_size 16 \
--output_path outputs/${model_name}/sensitivity/${outlier_method} \
done
```
## Sensitity of SW
```
model_name=huggyllama/llama-7B
for scale in 0.0 0.2 0.5 0.8 1.0 1.5 2.0 3.0
do
outlier_method=manual_scaling_SO_${scale}
python evaluate.py --model hf-outlier \
--model_args pretrained=${model_name},outlier_method=${outlier_method},dtype=float16 \
--tasks winogrande,arc_challenge,arc_easy,piqa,sciq,hellaswag,lambada_openai \
--device cuda:0 \
--batch_size 16 \
--output_path outputs/${model_name}/sensitivity/${outlier_method} \
done
```
## Block-wise Weight Quantization
```
model_name=huggyllama/llama-7B
# Baseline: INT4, no scale-shift
for blocksize in tensor 1048576 262144 65536 16384
do
manual_quantize=minmax_4_${blocksize}_no_0_False_False
restore_and_scale_GO=False
python evaluate.py --model hf-outlier \
--model_args pretrained=${model_name},manual_quantize=${manual_quantize},restore_and_scale_GO=${restore_and_scale_GO},trust_remote_code=True,dtype=float16 \
--tasks wikitext,winogrande,arc_challenge,arc_easy,piqa,sciq,hellaswag,lambada_openai \
--device cuda:0 \
--batch_size 4 \
--output_path outputs/${model_name}/groupwise/int4/minmax/manual_${manual_quantize}_restore_scale-${restore_and_scale_GO}_core \
--trust_remote_code \
done
for blocksize in tensor 1048576 262144 65536 16384
do
restore_and_scale_GO=1.0
for manual_quantize in clip_4_${blocksize}_z_9_False_False clip_4_${blocksize}_bp_1e-5_False_False clip_4_${blocksize}_tp_1e-6_False_False
do
python evaluate.py --model hf-outlier \
--model_args pretrained=${model_name},manual_quantize=${manual_quantize},restore_and_scale_GO=${restore_and_scale_GO},trust_remote_code=True,dtype=float16 \
--tasks winogrande,arc_challenge,arc_easy,piqa,sciq,hellaswag,lambada_openai \
--device cuda:0 \
--batch_size 4 \
--output_path outputs/${model_name}/groupwise/int4/ours/manual_${manual_quantize}_restore_scale-${restore_and_scale_GO}_core \
--trust_remote_code \
done
done
```
|