CultriX commited on
Commit
3ade048
·
verified ·
1 Parent(s): d11bd6b

Added disclaimer

Browse files
Files changed (1) hide show
  1. README.md +19 -0
README.md CHANGED
@@ -9,6 +9,25 @@ tags:
9
  - argilla/distilabeled-Marcoro14-7B-slerp
10
  ---
11
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
12
  # MergeTrix-7B
13
 
14
  MergeTrix-7B is a merge of the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing):
 
9
  - argilla/distilabeled-Marcoro14-7B-slerp
10
  ---
11
 
12
+ # IMPORTANT NOTE | READ ME! #
13
+ This model uses udkai/Turdus which may produce inaccurate results for the Winogrande evaluation scores.
14
+ The following is a quote directly taken from that models page:
15
+ - "A less contaminated version of udkai/Garrulus and the second model to be discussed in the paper Subtle DPO-Contamination with modified Winogrande increases TruthfulQA, Hellaswag & ARC."
16
+ - "Subtle DPO-Contamination with modified Winogrande causes the average accuracy of all 5-non Winogrande metrics (e.g. including also MMLU and GSM8K) to be 0.2% higher than the underlying model."
17
+
18
+ In my understanding the Winogrande scores are only slightly influenced by the DPO-Contamination, that has the "side-effect" of increasing the scores on the other benchmarks.
19
+ Since the effect on the Winogrande scores was subtle in the udkai/Turdus benchmarking results, and this model combines it with other models (probably making this effect even less pronounced),
20
+ I still believe that this model can be of value to the community as it's overall performance is quite impressive.
21
+
22
+ However I do not want to mislead anybody or produce any unfair scores, hence this note!
23
+ The full training configuration is also fully transparant and can be found below.
24
+
25
+ Hope this model will prove useful.
26
+ There's GGUF versions available here: https://huggingface.co/CultriX/MergeTrix-7B-GGUF
27
+
28
+ Kind regards,
29
+ CultriX
30
+
31
  # MergeTrix-7B
32
 
33
  MergeTrix-7B is a merge of the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing):