Hugging Face's logo

cosmo3769
/
starcoderbase-1b-GPTQ

Text Generation
Transformers
gpt_bigcode
4-bit precision
gptq
Model card Files
xet
 Community
cosmo3769 commited on
Commit
1ac95e3
·
verified ·
1 Parent(s): 99e9d9c

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +57 -0
README.md ADDED
@@ -0,0 +1,57 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # starcoderbase-1b-GPTQ
2
+
3
+ Quantized starcoderbase-1b model to GPTQ format (4-bit precision) using [Auto-GPTQ](https://github.com/AutoGPTQ/AutoGPTQ).
4
+
5
+ [Quantization script](https://github.com/cosmo3769/Quantized-LLMs/blob/main/notebooks/quantize-starcoderbase-1b-4bit-gptq.ipynb)
6
+
7
+ ## Benchmark
8
+
9
+ [Benchmarking script](https://github.com/cosmo3769/Quantized-LLMs/blob/main/notebooks/llmbenchmark-starcoderbase-1b-lm-eval-harness.ipynb)
10
+
11
+ ### Baseline starcoderbase-1b model (non-quantized)
12
+
13
+ | Tasks |Version|Filter|n-shot| Metric |Value | |Stderr|
14
+ |-----------------------|-------|------|------|---------------|-----:|---|-----:|
15
+ |codexglue_code2text |N/A |none |None |smoothed_bleu_4|0.8767|± |0.0592|
16
+ | - code2text_go | 1|none |None |smoothed_bleu_4|1.0054|± |0.0983|
17
+ | - code2text_java | 1|none |None |smoothed_bleu_4|1.2158|± |0.1657|
18
+ | - code2text_javascript| 1|none |None |smoothed_bleu_4|0.8560|± |0.0429|
19
+ | - code2text_php | 1|none |None |smoothed_bleu_4|0.9879|± |0.0887|
20
+ | - code2text_python | 1|none |None |smoothed_bleu_4|1.1950|± |0.2819|
21
+ | - code2text_ruby | 3|none |None |smoothed_bleu_4|0.0000|± |0.0000|
22
+
23
+ | Groups |Version|Filter|n-shot| Metric |Value | |Stderr|
24
+ |-------------------|-------|------|------|---------------|-----:|---|-----:|
25
+ |codexglue_code2text|N/A |none |None |smoothed_bleu_4|0.8767|± |0.0592|
26
+
27
+ | Tasks |Version|Filter|n-shot| Metric |Value| |Stderr|
28
+ |---------------------------------------------|------:|------|------|-----------|----:|---|-----:|
29
+ |bigbench_code_line_description_generate_until| 1|none |None |exact_match| 0|± | 0|
30
+
31
+ | Tasks |Version|Filter|n-shot|Metric|Value| |Stderr|
32
+ |----------------------------------------------|------:|------|------|------|----:|---|-----:|
33
+ |bigbench_code_line_description_multiple_choice| 0|none |None |acc | 0.15|± |0.0465|
34
+
35
+ ### Quantized starcoderbase-1b model to GPTQ format
36
+
37
+ | Tasks |Version|Filter|n-shot| Metric |Value | |Stderr|
38
+ |-----------------------|-------|------|------|---------------|-----:|---|-----:|
39
+ |codexglue_code2text |N/A |none |None |smoothed_bleu_4|0.7959|± |0.2180|
40
+ | - code2text_go | 1|none |None |smoothed_bleu_4|0.9280|± |0.0291|
41
+ | - code2text_java | 1|none |None |smoothed_bleu_4|1.2112|± |0.1703|
42
+ | - code2text_javascript| 1|none |None |smoothed_bleu_4|0.8848|± |0.0391|
43
+ | - code2text_php | 1|none |None |smoothed_bleu_4|0.6055|± |0.6055|
44
+ | - code2text_python | 1|none |None |smoothed_bleu_4|1.1460|± |1.1460|
45
+ | - code2text_ruby | 3|none |None |smoothed_bleu_4|0.0000|± |0.0000|
46
+
47
+ | Groups |Version|Filter|n-shot| Metric |Value | |Stderr|
48
+ |-------------------|-------|------|------|---------------|-----:|---|-----:|
49
+ |codexglue_code2text|N/A |none |None |smoothed_bleu_4|0.7959|± | 0.218|
50
+
51
+ | Tasks |Version|Filter|n-shot| Metric |Value| |Stderr|
52
+ |---------------------------------------------|------:|------|------|-----------|----:|---|-----:|
53
+ |bigbench_code_line_description_generate_until| 1|none |None |exact_match| 0|± | 0|
54
+
55
+ | Tasks |Version|Filter|n-shot|Metric|Value | |Stderr|
56
+ |----------------------------------------------|------:|------|------|------|-----:|---|-----:|
57
+ |bigbench_code_line_description_multiple_choice| 0|none |None |acc |0.1333|± |0.0443|