Sweaterdog commited on
Commit
f123645
·
verified ·
1 Parent(s): e45658a

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +18 -0
README.md CHANGED
@@ -56,6 +56,24 @@ A demo of a website it had generated for itself can be found [here](test_website
56
 
57
  ***
58
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
59
  ## 🧠 Model Philosophy: The Art of the Finetune
60
 
61
  While GRaPE Mini is not trained "from-scratch" (i.e., from random weights), it represents an extensive and highly curated instruction-tuning process. A base model possesses linguistic structure but lacks the ability to follow instructions, reason, or converse. The true "creation" of an assistant like GRaPE lies in the meticulous selection, blending, and application of high-quality datasets. This finetuning process is what transforms a raw linguistic engine into a capable and helpful agent.
 
56
 
57
  ***
58
 
59
+ ***
60
+
61
+ ## 🚀 Benchmarks
62
+
63
+ GRaPE Mini Beta is not the final model, and will improve. These are the benchmarks ran for GRaPE Mini Beta.
64
+
65
+ *(The benchmarks below were ran with the F16 weights of the model)*
66
+
67
+ | Tasks |Version| Filter |n-shot| Metric | |
68
+ |---------|------:|----------------|-----:|-----------|---|-----:|
69
+ |gsm8k* | 3|flexible-extract| 5|exact_match|↑ |28.51%|
70
+ | | |strict-match | 5|exact_match|↑ |14.48%|
71
+ |humaneval*| 1|create_test | 0|pass@1 | |20.73%|
72
+
73
+
74
+
75
+ * - These models were tested with the GPT-2 tokenizer on accident, updated benchmarks coming soon...
76
+
77
  ## 🧠 Model Philosophy: The Art of the Finetune
78
 
79
  While GRaPE Mini is not trained "from-scratch" (i.e., from random weights), it represents an extensive and highly curated instruction-tuning process. A base model possesses linguistic structure but lacks the ability to follow instructions, reason, or converse. The true "creation" of an assistant like GRaPE lies in the meticulous selection, blending, and application of high-quality datasets. This finetuning process is what transforms a raw linguistic engine into a capable and helpful agent.