Update README.md
Browse files
README.md
CHANGED
|
@@ -56,6 +56,24 @@ A demo of a website it had generated for itself can be found [here](test_website
|
|
| 56 |
|
| 57 |
***
|
| 58 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 59 |
## 🧠 Model Philosophy: The Art of the Finetune
|
| 60 |
|
| 61 |
While GRaPE Mini is not trained "from-scratch" (i.e., from random weights), it represents an extensive and highly curated instruction-tuning process. A base model possesses linguistic structure but lacks the ability to follow instructions, reason, or converse. The true "creation" of an assistant like GRaPE lies in the meticulous selection, blending, and application of high-quality datasets. This finetuning process is what transforms a raw linguistic engine into a capable and helpful agent.
|
|
|
|
| 56 |
|
| 57 |
***
|
| 58 |
|
| 59 |
+
***
|
| 60 |
+
|
| 61 |
+
## 🚀 Benchmarks
|
| 62 |
+
|
| 63 |
+
GRaPE Mini Beta is not the final model, and will improve. These are the benchmarks ran for GRaPE Mini Beta.
|
| 64 |
+
|
| 65 |
+
*(The benchmarks below were ran with the F16 weights of the model)*
|
| 66 |
+
|
| 67 |
+
| Tasks |Version| Filter |n-shot| Metric | |
|
| 68 |
+
|---------|------:|----------------|-----:|-----------|---|-----:|
|
| 69 |
+
|gsm8k* | 3|flexible-extract| 5|exact_match|↑ |28.51%|
|
| 70 |
+
| | |strict-match | 5|exact_match|↑ |14.48%|
|
| 71 |
+
|humaneval*| 1|create_test | 0|pass@1 | |20.73%|
|
| 72 |
+
|
| 73 |
+
|
| 74 |
+
|
| 75 |
+
* - These models were tested with the GPT-2 tokenizer on accident, updated benchmarks coming soon...
|
| 76 |
+
|
| 77 |
## 🧠 Model Philosophy: The Art of the Finetune
|
| 78 |
|
| 79 |
While GRaPE Mini is not trained "from-scratch" (i.e., from random weights), it represents an extensive and highly curated instruction-tuning process. A base model possesses linguistic structure but lacks the ability to follow instructions, reason, or converse. The true "creation" of an assistant like GRaPE lies in the meticulous selection, blending, and application of high-quality datasets. This finetuning process is what transforms a raw linguistic engine into a capable and helpful agent.
|