| --- |
| base_model: |
| - mistralai/Mistral-7B-v0.3 |
| - dreamgen/WizardLM-2-7B |
| - uukuguy/speechless-code-mistral-7b-v2.0 |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - mergekit |
| - merge |
| - mistral |
| - instruction-tuned |
|
|
| --- |
| # Runforge_Core-7b |
| |
| This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit). |
| |
| ## Merge Details |
| ### Merge Method |
| |
| This model was merged using the [Linear](https://arxiv.org/abs/2203.05482) merge method using `mistralai/Mistral-7B-v0.3` as a base. |
| |
| ### Models Merged |
| |
| The following models were included in the merge: |
| * `dreamgen/WizardLM-2-7B` |
| * `uukuguy/speechless-code-mistral-7b-v2.0` |
| |
| ### Configuration |
| |
| The following YAML configuration was used to produce this model: |
| |
| ```yaml |
| # Clean 3-way dense merge rebuild for runeforge_core-7b |
| models: |
| - model: mistralai/Mistral-7B-v0.3 |
| parameters: |
| weight: 0.4 |
| - model: dreamgen/WizardLM-2-7B |
| parameters: |
| weight: 0.3 |
| - model: uukuguy/speechless-code-mistral-7b-v2.0 |
| parameters: |
| weight: 0.3 |
| merge_method: linear |
| base_model: mistralai/Mistral-7B-v0.3 |
| dtype: float16 |
| out_dtype: float16 |
| ``` |
| |
| ## Evaluation |
|
|
| ### Setup |
|
|
| - Date: 2026-03-14 |
| - Runtime: local GPU inference in WSL |
| - Loader: Transformers/Unsloth with 4-bit quantization (`load_in_4bit`) |
| - Benchmarks: |
| - ARC-Challenge (multiple-choice) |
| - HellaSwag (multiple-choice) |
| - Winogrande XL (multiple-choice) |
| - TruthfulQA MC1 (multiple-choice) |
| - Metric: Accuracy per benchmark and macro average across the four tasks |
|
|
| ### Primary Comparison (200 samples per benchmark) |
|
|
| | Model | ARC | HellaSwag | Winogrande | TruthfulQA MC1 | Macro Avg | |
| |---|---:|---:|---:|---:|---:| |
| | runeforge_core-7b (this model) | 0.7650 | 0.7050 | 0.6000 | 0.5800 | **0.6625** | |
| | mistral-7b baseline | 0.7000 | 0.6000 | 0.4600 | 0.5900 | 0.5875 | |
| |
| Interpretation: runeforge_core-7b outperformed the local Mistral baseline by +0.0750 macro accuracy on this evaluation run. |
|
|
| ### Expanded Comparison (30 samples per benchmark) |
|
|
| | Model | ARC | HellaSwag | Winogrande | TruthfulQA MC1 | Macro Avg | |
| |---|---:|---:|---:|---:|---:| |
| | runeforge_core-7b (this model) | 0.8000 | 0.7333 | 0.5000 | 0.4333 | **0.6167** | |
| | mistral-7b baseline | 0.7000 | 0.6000 | 0.4667 | 0.5000 | 0.5667 | |
| | speechless-code-mistral-7b-v2.0 | 0.6000 | 0.3000 | 0.5333 | 0.6000 | 0.5083 | |
| | dreamgen/WizardLM-2-7B | 0.2000 | 0.2333 | 0.6667 | 0.7667 | 0.4667 | |
| | runeforge_mk1_merged_from_7922 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | |
| |
| Note: the expanded table uses a smaller sample size and is more variance-prone; use the 200-sample comparison as the primary signal. |
| |
| ### Coding Sanity Check (Executable) |
| |
| A separate executable coding sanity check (5 unit-tested tasks) was also run: |
| |
| | Model | Passes | Total | Pass Rate | |
| |---|---:|---:|---:| |
| | runeforge_core-7b (this model) | 5 | 5 | **1.00** | |
| | runeforge_mk1_merged_from_7922 | 0 | 5 | 0.00 | |
|
|
| ### Reproducibility Files |
|
|
| Repository-relative references (from this model folder): |
|
|
| - `../Making_Runeforge/evaluate_general_models.py` |
| - `../Making_Runeforge/evaluate_coding_exec.py` |
| - `../Making_Runeforge/eval_general_runeforge_core_200.json` |
| - `../Making_Runeforge/eval_general_mistral_base_200.json` |
| - `../Making_Runeforge/eval_general_leaderboard.json` |
| - `../Making_Runeforge/runeforge_coding_exec_eval.json` |
|
|
| ## Intended Use |
|
|
| - General-purpose assistant and instruction-following use cases. |
| - Strong performance on local multiple-choice reasoning benchmarks relative to the local Mistral baseline used in this project. |
| - Suitable as a base for additional task-specific fine-tuning where broad instruction quality is desired. |
|
|
| ## Limitations |
|
|
| - Reported metrics are from local, sampled benchmark runs (not full official leaderboard submissions). |
| - Quantized inference (`load_in_4bit`) was used for evaluation; scores may shift under different precision/runtime setups. |
| - Expanded 5-model comparison used 30 samples per benchmark and should be treated as directional. |
| - A separate merged artifact (`runeforge_mk1_merged_from_7922`) showed severe degradation (near-zero on sampled general benchmarks and 0/5 on coding executable sanity checks). |
|
|
| ## Evaluation Notes |
|
|
| - The 200-sample comparison is the primary result set for this card. |
| - The 30-sample expanded table is included for breadth across additional local peer models. |
| - All benchmark scripts and JSON outputs are listed above for reproducibility. |
|
|
