Update README.md
Browse files
README.md
CHANGED
|
@@ -67,7 +67,7 @@ We did not considered it for our score, but "if" considered those extra 5 questi
|
|
| 67 |
<hr>
|
| 68 |
|
| 69 |
|
| 70 |
-
#### Comparison Table
|
| 71 |
|
| 72 |
<table>
|
| 73 |
<thead>
|
|
@@ -112,6 +112,19 @@ We did not considered it for our score, but "if" considered those extra 5 questi
|
|
| 112 |
|
| 113 |
<hr>
|
| 114 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 115 |
#### Benchmark tool used
|
| 116 |
|
| 117 |
https://github.com/nerdskingcom/gguf-humaneval-benchmark
|
|
|
|
| 67 |
<hr>
|
| 68 |
|
| 69 |
|
| 70 |
+
#### Comparison Table with models of the same size
|
| 71 |
|
| 72 |
<table>
|
| 73 |
<thead>
|
|
|
|
| 112 |
|
| 113 |
<hr>
|
| 114 |
|
| 115 |
+
#### TIME Comparison Table with models of Bigger size
|
| 116 |
+
|
| 117 |
+
|
| 118 |
+
Nanbeige/Nanbeige4.1-3B (4 billion model, "marketed" as 3B billions), with 92% HumanEval accuracy in Python
|
| 119 |
+
- Time to finish 160 questions from HumanEval: 22263.80s --- MORE THAN SIX HOURS!!! (in a RTX 5060 ti 16gb)
|
| 120 |
+
|
| 121 |
+
Nerdsking/nerdsking-python-coder-3B-i, with 88,41% HumanEval accuracy in Python
|
| 122 |
+
- Time to finish 160 questions from HumanEval: 210.51s --- THREE MINUTES AND HALF (in a RTX 5060 ti 16gb)
|
| 123 |
+
|
| 124 |
+
<hr>
|
| 125 |
+
|
| 126 |
+
|
| 127 |
+
|
| 128 |
#### Benchmark tool used
|
| 129 |
|
| 130 |
https://github.com/nerdskingcom/gguf-humaneval-benchmark
|