Hugging Face's logo

Yhyu13
/
Xwin-Math-7B-V1.0-QUIP-2bit

Text Generation
Transformers
PyTorch
Safetensors
llama
text-generation-inference
Model card Files
xet
 Community
3
yhyu13 commited on
Commit
62e3564
·
1 Parent(s): b119dcd

Upload

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. README.md +43 -0
  2. Xwin-Math-7B-V1.0/Ckpt-quip.log +0 -0
  3. Xwin-Math-7B-V1.0/Ckpt/0_down.pt +3 -0
  4. Xwin-Math-7B-V1.0/Ckpt/0_o.pt +3 -0
  5. Xwin-Math-7B-V1.0/Ckpt/0_qkv.pt +3 -0
  6. Xwin-Math-7B-V1.0/Ckpt/0_up.pt +3 -0
  7. Xwin-Math-7B-V1.0/Ckpt/10_down.pt +3 -0
  8. Xwin-Math-7B-V1.0/Ckpt/10_o.pt +3 -0
  9. Xwin-Math-7B-V1.0/Ckpt/10_qkv.pt +3 -0
  10. Xwin-Math-7B-V1.0/Ckpt/10_up.pt +3 -0
  11. Xwin-Math-7B-V1.0/Ckpt/11_down.pt +3 -0
  12. Xwin-Math-7B-V1.0/Ckpt/11_o.pt +3 -0
  13. Xwin-Math-7B-V1.0/Ckpt/11_qkv.pt +3 -0
  14. Xwin-Math-7B-V1.0/Ckpt/11_up.pt +3 -0
  15. Xwin-Math-7B-V1.0/Ckpt/12_down.pt +3 -0
  16. Xwin-Math-7B-V1.0/Ckpt/12_o.pt +3 -0
  17. Xwin-Math-7B-V1.0/Ckpt/12_qkv.pt +3 -0
  18. Xwin-Math-7B-V1.0/Ckpt/12_up.pt +3 -0
  19. Xwin-Math-7B-V1.0/Ckpt/13_down.pt +3 -0
  20. Xwin-Math-7B-V1.0/Ckpt/13_o.pt +3 -0
  21. Xwin-Math-7B-V1.0/Ckpt/13_qkv.pt +3 -0
  22. Xwin-Math-7B-V1.0/Ckpt/13_up.pt +3 -0
  23. Xwin-Math-7B-V1.0/Ckpt/14_down.pt +3 -0
  24. Xwin-Math-7B-V1.0/Ckpt/14_o.pt +3 -0
  25. Xwin-Math-7B-V1.0/Ckpt/14_qkv.pt +3 -0
  26. Xwin-Math-7B-V1.0/Ckpt/14_up.pt +3 -0
  27. Xwin-Math-7B-V1.0/Ckpt/15_down.pt +3 -0
  28. Xwin-Math-7B-V1.0/Ckpt/15_o.pt +3 -0
  29. Xwin-Math-7B-V1.0/Ckpt/15_qkv.pt +3 -0
  30. Xwin-Math-7B-V1.0/Ckpt/15_up.pt +3 -0
  31. Xwin-Math-7B-V1.0/Ckpt/16_down.pt +3 -0
  32. Xwin-Math-7B-V1.0/Ckpt/16_o.pt +3 -0
  33. Xwin-Math-7B-V1.0/Ckpt/16_qkv.pt +3 -0
  34. Xwin-Math-7B-V1.0/Ckpt/16_up.pt +3 -0
  35. Xwin-Math-7B-V1.0/Ckpt/17_down.pt +3 -0
  36. Xwin-Math-7B-V1.0/Ckpt/17_o.pt +3 -0
  37. Xwin-Math-7B-V1.0/Ckpt/17_qkv.pt +3 -0
  38. Xwin-Math-7B-V1.0/Ckpt/17_up.pt +3 -0
  39. Xwin-Math-7B-V1.0/Ckpt/18_down.pt +3 -0
  40. Xwin-Math-7B-V1.0/Ckpt/18_o.pt +3 -0
  41. Xwin-Math-7B-V1.0/Ckpt/18_qkv.pt +3 -0
  42. Xwin-Math-7B-V1.0/Ckpt/18_up.pt +3 -0
  43. Xwin-Math-7B-V1.0/Ckpt/19_down.pt +3 -0
  44. Xwin-Math-7B-V1.0/Ckpt/19_o.pt +3 -0
  45. Xwin-Math-7B-V1.0/Ckpt/19_qkv.pt +3 -0
  46. Xwin-Math-7B-V1.0/Ckpt/19_up.pt +3 -0
  47. Xwin-Math-7B-V1.0/Ckpt/1_down.pt +3 -0
  48. Xwin-Math-7B-V1.0/Ckpt/1_o.pt +3 -0
  49. Xwin-Math-7B-V1.0/Ckpt/1_qkv.pt +3 -0
  50. Xwin-Math-7B-V1.0/Ckpt/1_up.pt +3 -0
README.md CHANGED
@@ -1,3 +1,46 @@
1
  ---
2
  license: llama2
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: llama2
3
  ---
4
+
5
+ Experiment QUIP 2-bit E8P12 version that works in textgen-webui with quip mode loader
6
+
7
+ Generated by using scripts from https://gitee.com/yhyu13/llama_-tools
8
+
9
+ Original weight : https://huggingface.co/Xwin-LM/Xwin-Math-7B-V1.0
10
+
11
+ GPTQ 4bit : https://huggingface.co/Yhyu13/Xwin-Math-7B-V1.0-GPTQ-4bit
12
+
13
+ ---
14
+
15
+ I used `hessian_offline_llama.py` provided by QUIP repo to generate hessian specifically for the orignal model before applying Quip quantization.
16
+
17
+ It took quite a long time for hessian for all 31 layers, about 6 hours for 7B models on a single RTX3090. I am not sure if I made any error.
18
+
19
+ QUIP byproducts are also uploaded.
20
+
21
+
22
+ Perplexity calcaultead using `eval_ppl.py` provided by QUIP repo
23
+ QUIP PPL:
24
+ wikitext2 perplexity: 11.247852325439453
25
+ c4 perplexity: 16.275997161865234
26
+
27
+ Original model PPL:
28
+ wikitext2 perplexity: 6.042122840881348
29
+ c4 perplexity: 8.430611610412598
30
+
31
+ Looks like something is wrong, the quantized model is a disaster.
32
+
33
+ ---
34
+
35
+ Here is some testing done in textgen-webui, I was using Q&A from this dataset https://huggingface.co/datasets/TIGER-Lab/MathInstruct
36
+
37
+ It seems the 2 bit could hardly answer any question correctly, in compare to GPTQ 4bit version. But the https://huggingface.co/relaxml/Llama-2-13b-E8P-2Bit model made by the author of QUIP seems to work fine, just as good as GPTQ.
38
+
39
+ So in conclusion, this is a very experimental model that I made just to testify QUIP, I may made some error. But I think it is a good start.
40
+
41
+ QUIP 2 bit version:
42
+ ![Alt text](img/XMath_Quip2.png)
43
+
44
+
45
+ GPTQ 4 bit version:
46
+ ![Alt text](img/textgen-xinmath.png)
Xwin-Math-7B-V1.0/Ckpt-quip.log ADDED
File without changes
Xwin-Math-7B-V1.0/Ckpt/0_down.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7ff97612d07b6979fbffe5f9df731c1f3d1e647347fc49c92fef54e44f8931b7
3
+ size 11289347
Xwin-Math-7B-V1.0/Ckpt/0_o.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c07e86c96753080bcdd8d6bba6ebf459d5a3dbe299e9d3d2fbb107bae6db1dff
3
+ size 4204456
Xwin-Math-7B-V1.0/Ckpt/0_qkv.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3d16173c8d0a9a4cb8db07ac5d73d6c904afe52cff0159dd68d8accadcb617c5
3
+ size 12601838
Xwin-Math-7B-V1.0/Ckpt/0_up.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dac4ef43f1f5ecd51110ba27974329b1b48129495974435d5811aa654d1624eb
3
+ size 22572778
Xwin-Math-7B-V1.0/Ckpt/10_down.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2fa389dc12a2eb451e69225c88af9b78533eea0810d587930a49715a9da7e8d6
3
+ size 11289356
Xwin-Math-7B-V1.0/Ckpt/10_o.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7ad940528d592c64d02b12a2255d6f775fb021ac456470dc6b408f962b5cd895
3
+ size 4204529
Xwin-Math-7B-V1.0/Ckpt/10_qkv.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:47af3b065cfcb646fbef6553199c841c7bc1ee265137b93396f0f36ef87e861e
3
+ size 12601849
Xwin-Math-7B-V1.0/Ckpt/10_up.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a59cfc9b1524d0b76c93adf840807c81165ae330dae9a63bdb737018ac5c2b59
3
+ size 22572788
Xwin-Math-7B-V1.0/Ckpt/11_down.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:161c5862bca5738f3829dcaa72ab2babc575b6d7df83d7e370f9497b7fd4c564
3
+ size 11289356
Xwin-Math-7B-V1.0/Ckpt/11_o.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d7a308d94c23def22f4feccd8690ddd09c27a9852dec52fb2ab40d6f437b6a5c
3
+ size 4204529
Xwin-Math-7B-V1.0/Ckpt/11_qkv.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9012d3bbbd77e1c975eff400c70ece99f5901616faa4ad3131c6cdb487aee4e3
3
+ size 12601849
Xwin-Math-7B-V1.0/Ckpt/11_up.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7359995b8336a8502b49e92acb501979e9e6eb6efb41da074b394f5eda1b87ad
3
+ size 22572788
Xwin-Math-7B-V1.0/Ckpt/12_down.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:34e2fdaa6ea7547b9b186c0711442afb0765c2375ba68311aa8267ac172cbe10
3
+ size 11289356
Xwin-Math-7B-V1.0/Ckpt/12_o.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6f76e799ce60b5f2eeaf2939ca7c2d315b1d3cadcc92efd61a8ca7b687034883
3
+ size 4204529
Xwin-Math-7B-V1.0/Ckpt/12_qkv.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:39efa2f5948b8ad991aa16419a3e6c7403afcd80d32de9ac6cab478966b90525
3
+ size 12601849
Xwin-Math-7B-V1.0/Ckpt/12_up.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2ab331d3c1e62ec8a5b861ad3d561a21f278d21b2f41c786f626feab30f66dda
3
+ size 22572788
Xwin-Math-7B-V1.0/Ckpt/13_down.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:18ef2dd5689d77f0cb598ea746890cc2314f74a7744d64ea28405767db02912f
3
+ size 11289356
Xwin-Math-7B-V1.0/Ckpt/13_o.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6c9d32c790cc74b13585ac1e3921341b8f7e27d9b919b7f791c9071e5e674a78
3
+ size 4204529
Xwin-Math-7B-V1.0/Ckpt/13_qkv.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a1ba9871b1da1a8e4af1baece88d4238c79358e4998171aa4e78eb4cf3c444d6
3
+ size 12601849
Xwin-Math-7B-V1.0/Ckpt/13_up.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ee47689ee7d4ed6d1a9764b197ca2ce9d3f59ebcacd763feab16c6fe07251ff3
3
+ size 22572788
Xwin-Math-7B-V1.0/Ckpt/14_down.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3c320003cd0730b79be82b030f134d5f1b5543d65e0f62e4342b642c6a522b89
3
+ size 11289356
Xwin-Math-7B-V1.0/Ckpt/14_o.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d0b69e1f0a39dd440ca4755262c5d492e117c926afcf660ff690a4b4ec947d67
3
+ size 4204529
Xwin-Math-7B-V1.0/Ckpt/14_qkv.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:521012c46c4409a214a6b4d6d8a1b9f2d07628db74db0932cc76ce7712a3b3fc
3
+ size 12601849
Xwin-Math-7B-V1.0/Ckpt/14_up.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:87ff7ad616058ff70f72eb52ae72dbcff298c10a59c4e01bfd68f836c9d67038
3
+ size 22572788
Xwin-Math-7B-V1.0/Ckpt/15_down.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:45351f827024854fa0442253589fe469cea637e5bc82c8a970786d7f36293e4e
3
+ size 11289356
Xwin-Math-7B-V1.0/Ckpt/15_o.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:67678485a3ac27afd8f0fe748987624585a177a20ae425c23111ed4715682049
3
+ size 4204529
Xwin-Math-7B-V1.0/Ckpt/15_qkv.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b56d80f895b5b1865db23a7b873377fb5bb1e680caccc1282c63fd1cba3702a9
3
+ size 12601849
Xwin-Math-7B-V1.0/Ckpt/15_up.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3bf5fb11f99c0bfd5695b2c38a971400ddc267d9e62731ba358675633aaf5d17
3
+ size 22572788
Xwin-Math-7B-V1.0/Ckpt/16_down.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:11336d76766b8e589b9e9bd8a61c9d006712564a0e6ba2246d839334d42c9c1a
3
+ size 11289356
Xwin-Math-7B-V1.0/Ckpt/16_o.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f002438f13833647e3667e58cfefa52137c031aadb8ad7e8a2a4a2694ed6e2e8
3
+ size 4204529
Xwin-Math-7B-V1.0/Ckpt/16_qkv.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6a6e48d17a685259f402070583e76afc19041fa88a71e0ae9c3d3839c013ccaf
3
+ size 12601849
Xwin-Math-7B-V1.0/Ckpt/16_up.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:728697b7504aa47e259ce7e8582fd283b810aeb968c101e0cd2ac37cb3c71f63
3
+ size 22572788
Xwin-Math-7B-V1.0/Ckpt/17_down.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dbd176c918b9bc15f73f6cc9823287f07bd209dbcbbf14a3a5be3d58d4e68486
3
+ size 11289356
Xwin-Math-7B-V1.0/Ckpt/17_o.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:044754be62bf6c3dbcec3b3a10ae27babe5009e279e066e510469a85acb77fb8
3
+ size 4204529
Xwin-Math-7B-V1.0/Ckpt/17_qkv.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f8762f310e39709c3445a4a7616a0979a3859aeef35d23d3d3f13b2dbca4c322
3
+ size 12601849
Xwin-Math-7B-V1.0/Ckpt/17_up.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8d69d00d3cd26c03976fa65cf79e22bc10d2b7db897aefa3c657c2aa46251452
3
+ size 22572788
Xwin-Math-7B-V1.0/Ckpt/18_down.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b54b2764aa980ed515efeea6e8edcb2aa708a4d3b31b21420d72c6fdb36e807c
3
+ size 11289356
Xwin-Math-7B-V1.0/Ckpt/18_o.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f1f49a7b5698aee82fcdc7be6746ec07a13015447074d11ff6aabe49f7d551fd
3
+ size 4204529
Xwin-Math-7B-V1.0/Ckpt/18_qkv.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9b408e44c5d91e0c6914f9e545f9256b7986bcc0861c16daa5e245f5fc5efcdf
3
+ size 12601849
Xwin-Math-7B-V1.0/Ckpt/18_up.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0488540dde4e6f8dcc21a198a3e5f1ded19f0999bb1d70cb043399016245a927
3
+ size 22572788
Xwin-Math-7B-V1.0/Ckpt/19_down.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8af01c93f932c4e50ce8b0f34bfee813457751412c6fdd8c58ae97c65cfa71bf
3
+ size 11289356
Xwin-Math-7B-V1.0/Ckpt/19_o.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:88d62c46015ac8ce6a8d223059e4f9f1d0414254dac0f0cb44a1a6d538c712e5
3
+ size 4204529
Xwin-Math-7B-V1.0/Ckpt/19_qkv.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:317369e1732eb2f0d7f4b2fc6f0df3ceeca4dce8c3de7f5e65360f009cf15477
3
+ size 12601849
Xwin-Math-7B-V1.0/Ckpt/19_up.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bd1785756aa4aa48c9a087c2266df408c3488ea1e0a97e765207e99eb7abb6bf
3
+ size 22572788
Xwin-Math-7B-V1.0/Ckpt/1_down.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3503ae4a30e75cbb8cbcac061be94d41754947f9ad4ed98ed133729b53f5c8ad
3
+ size 11289347
Xwin-Math-7B-V1.0/Ckpt/1_o.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f4844dbe8f7d8350f5cfd3d98ee3cc44041c13bdd0c3c4a74058788e22829df0
3
+ size 4204456
Xwin-Math-7B-V1.0/Ckpt/1_qkv.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a681cc834ae6033d1308c279be175529483deaf4d37eb1727544abdd0754e660
3
+ size 12601838
Xwin-Math-7B-V1.0/Ckpt/1_up.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2ce222ca230864d7f1e987078fe88cf6616264b362b34604b0f20cfd26eb57c1
3
+ size 22572778