| | --- |
| | tags: |
| | - merge |
| | - mergekit |
| | - lazymergekit |
| | - yleo/EmertonMonarch-7B |
| | base_model: |
| | - yleo/EmertonMonarch-7B |
| | --- |
| | |
| | # Spaetzle-v31-7b |
| |
|
| | Spaetzle-v31-7b is a merge of the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing): |
| | * [yleo/EmertonMonarch-7B](https://huggingface.co/yleo/EmertonMonarch-7B) |
| | * [cstr/spaetzle-v8-7b](https://huggingface.co/cstr/spaetzle-v8-7b) |
| |
|
| | | Model |AGIEval|GPT4All|TruthfulQA|Bigbench|Average| |
| | |--------------------------------------------------------------|------:|------:|---------:|-------:|------:| |
| | |[Spaetzle-v31-7b](https://huggingface.co/cstr/Spaetzle-v31-7b)| 46.23| 76.6| 69.58| 46.79| 59.8| |
| |
|
| | ### AGIEval |
| | | Task |Version| Metric |Value| |Stderr| |
| | |------------------------------|------:|--------|----:|---|-----:| |
| | |agieval_aqua_rat | 0|acc |28.74|± | 2.85| |
| | | | |acc_norm|27.56|± | 2.81| |
| | |agieval_logiqa_en | 0|acc |39.63|± | 1.92| |
| | | | |acc_norm|40.25|± | 1.92| |
| | |agieval_lsat_ar | 0|acc |24.35|± | 2.84| |
| | | | |acc_norm|24.35|± | 2.84| |
| | |agieval_lsat_lr | 0|acc |54.31|± | 2.21| |
| | | | |acc_norm|54.12|± | 2.21| |
| | |agieval_lsat_rc | 0|acc |65.80|± | 2.90| |
| | | | |acc_norm|66.54|± | 2.88| |
| | |agieval_sat_en | 0|acc |79.13|± | 2.84| |
| | | | |acc_norm|79.61|± | 2.81| |
| | |agieval_sat_en_without_passage| 0|acc |46.12|± | 3.48| |
| | | | |acc_norm|45.15|± | 3.48| |
| | |agieval_sat_math | 0|acc |35.00|± | 3.22| |
| | | | |acc_norm|32.27|± | 3.16| |
| |
|
| | Average: 46.23% |
| |
|
| | ### GPT4All |
| | | Task |Version| Metric |Value| |Stderr| |
| | |-------------|------:|--------|----:|---|-----:| |
| | |arc_challenge| 0|acc |64.76|± | 1.40| |
| | | | |acc_norm|66.89|± | 1.38| |
| | |arc_easy | 0|acc |86.66|± | 0.70| |
| | | | |acc_norm|82.83|± | 0.77| |
| | |boolq | 1|acc |87.80|± | 0.57| |
| | |hellaswag | 0|acc |67.43|± | 0.47| |
| | | | |acc_norm|85.85|± | 0.35| |
| | |openbookqa | 0|acc |38.00|± | 2.17| |
| | | | |acc_norm|48.80|± | 2.24| |
| | |piqa | 0|acc |83.57|± | 0.86| |
| | | | |acc_norm|84.71|± | 0.84| |
| | |winogrande | 0|acc |79.32|± | 1.14| |
| | |
| | Average: 76.6% |
| | |
| | ### TruthfulQA |
| | | Task |Version|Metric|Value| |Stderr| |
| | |-------------|------:|------|----:|---|-----:| |
| | |truthfulqa_mc| 1|mc1 |53.37|± | 1.75| |
| | | | |mc2 |69.58|± | 1.48| |
| |
|
| | Average: 69.58% |
| |
|
| | ### Bigbench |
| | | Task |Version| Metric |Value| |Stderr| |
| | |------------------------------------------------|------:|---------------------|----:|---|-----:| |
| | |bigbench_causal_judgement | 0|multiple_choice_grade|56.84|± | 3.60| |
| | |bigbench_date_understanding | 0|multiple_choice_grade|66.94|± | 2.45| |
| | |bigbench_disambiguation_qa | 0|multiple_choice_grade|44.57|± | 3.10| |
| | |bigbench_geometric_shapes | 0|multiple_choice_grade|21.17|± | 2.16| |
| | | | |exact_str_match | 0.28|± | 0.28| |
| | |bigbench_logical_deduction_five_objects | 0|multiple_choice_grade|31.80|± | 2.08| |
| | |bigbench_logical_deduction_seven_objects | 0|multiple_choice_grade|22.57|± | 1.58| |
| | |bigbench_logical_deduction_three_objects | 0|multiple_choice_grade|56.00|± | 2.87| |
| | |bigbench_movie_recommendation | 0|multiple_choice_grade|45.40|± | 2.23| |
| | |bigbench_navigate | 0|multiple_choice_grade|52.80|± | 1.58| |
| | |bigbench_reasoning_about_colored_objects | 0|multiple_choice_grade|70.65|± | 1.02| |
| | |bigbench_ruin_names | 0|multiple_choice_grade|50.67|± | 2.36| |
| | |bigbench_salient_translation_error_detection | 0|multiple_choice_grade|30.66|± | 1.46| |
| | |bigbench_snarks | 0|multiple_choice_grade|71.27|± | 3.37| |
| | |bigbench_sports_understanding | 0|multiple_choice_grade|74.34|± | 1.39| |
| | |bigbench_temporal_sequences | 0|multiple_choice_grade|49.80|± | 1.58| |
| | |bigbench_tracking_shuffled_objects_five_objects | 0|multiple_choice_grade|22.16|± | 1.18| |
| | |bigbench_tracking_shuffled_objects_seven_objects| 0|multiple_choice_grade|18.57|± | 0.93| |
| | |bigbench_tracking_shuffled_objects_three_objects| 0|multiple_choice_grade|56.00|± | 2.87| |
| | |
| | Average: 46.79% |
| | |
| | Average score: 59.8% |
| | |
| | Elapsed time: 02:09:50 |
| | |
| | ## 🧩 Configuration |
| | |
| | ```yaml |
| | models: |
| | - model: cstr/spaetzle-v8-7b |
| | # no parameters necessary for base model |
| | - model: yleo/EmertonMonarch-7B |
| | parameters: |
| | density: 0.60 |
| | weight: 0.3 |
| | merge_method: dare_ties |
| | base_model: cstr/spaetzle-v8-7b |
| | parameters: |
| | int8_mask: true |
| | dtype: bfloat16 |
| | random_seed: 0 |
| | tokenizer_source: base |
| | ``` |
| | |
| | ## 💻 Usage |
| | |
| | ```python |
| | !pip install -qU transformers accelerate |
| | |
| | from transformers import AutoTokenizer |
| | import transformers |
| | import torch |
| | |
| | model = "cstr/Spaetzle-v31-7b" |
| | messages = [{"role": "user", "content": "What is a large language model?"}] |
| | |
| | tokenizer = AutoTokenizer.from_pretrained(model) |
| | prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| | pipeline = transformers.pipeline( |
| | "text-generation", |
| | model=model, |
| | torch_dtype=torch.float16, |
| | device_map="auto", |
| | ) |
| | |
| | outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95) |
| | print(outputs[0]["generated_text"]) |
| | ``` |
