Model Stock: All we need is just a few fine-tuned models
Paper • 2403.19522 • Published • 15
How to use from
llama.cppInstall from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf QuantFactory/Llama-3-RedMagic4-8B-GGUF:# Run inference directly in the terminal:
llama-cli -hf QuantFactory/Llama-3-RedMagic4-8B-GGUF:Use pre-built binary
# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf QuantFactory/Llama-3-RedMagic4-8B-GGUF:# Run inference directly in the terminal:
./llama-cli -hf QuantFactory/Llama-3-RedMagic4-8B-GGUF:Build from source code
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf QuantFactory/Llama-3-RedMagic4-8B-GGUF:# Run inference directly in the terminal:
./build/bin/llama-cli -hf QuantFactory/Llama-3-RedMagic4-8B-GGUF:Use Docker
docker model run hf.co/QuantFactory/Llama-3-RedMagic4-8B-GGUF:Quick Links
QuantFactory/Llama-3-RedMagic4-8B-GGUF
This is quantized version of lemon07r/Llama-3-RedMagic4-8B created using llama.cpp
Original Model Card
Llama-3-RedMagic4-8B
This is a merge of pre-trained language models created using mergekit.
Merge Details
Merge Method
This model was merged using the Model Stock merge method using NousResearch/Meta-Llama-3-8B as a base.
Models Merged
The following models were included in the merge:
- flammenai/Mahou-1.2-llama3-8B
- lemon07r/Llama-3-RedMagic2-8B
- lemon07r/Lllama-3-RedElixir-8B
- nbeerbower/llama-3-spicy-abliterated-stella-8B
Configuration
The following YAML configuration was used to produce this model:
base_model: NousResearch/Meta-Llama-3-8B
dtype: bfloat16
merge_method: model_stock
slices:
- sources:
- layer_range: [0, 32]
model: lemon07r/Llama-3-RedMagic2-8B
- layer_range: [0, 32]
model: lemon07r/Lllama-3-RedElixir-8B
- layer_range: [0, 32]
model: nbeerbower/llama-3-spicy-abliterated-stella-8B
- layer_range: [0, 32]
model: flammenai/Mahou-1.2-llama3-8B
- layer_range: [0, 32]
model: NousResearch/Meta-Llama-3-8B
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 19.32 |
| IFEval (0-Shot) | 48.64 |
| BBH (3-Shot) | 19.48 |
| MATH Lvl 5 (4-Shot) | 8.31 |
| GPQA (0-shot) | 5.37 |
| MuSR (0-shot) | 4.38 |
| MMLU-PRO (5-shot) | 29.73 |
- Downloads last month
- 202
Hardware compatibility
Log In to add your hardware
2-bit
3-bit
4-bit
5-bit
6-bit
8-bit
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for QuantFactory/Llama-3-RedMagic4-8B-GGUF
Merge model
this model
Paper for QuantFactory/Llama-3-RedMagic4-8B-GGUF
Evaluation results
- strict accuracy on IFEval (0-Shot)Open LLM Leaderboard48.640
- normalized accuracy on BBH (3-Shot)Open LLM Leaderboard19.480
- exact match on MATH Lvl 5 (4-Shot)Open LLM Leaderboard8.310
- acc_norm on GPQA (0-shot)Open LLM Leaderboard5.370
- acc_norm on MuSR (0-shot)Open LLM Leaderboard4.380
- accuracy on MMLU-PRO (5-shot)test set Open LLM Leaderboard29.730
Install from brew
# Start a local OpenAI-compatible server with a web UI: llama-server -hf QuantFactory/Llama-3-RedMagic4-8B-GGUF:# Run inference directly in the terminal: llama-cli -hf QuantFactory/Llama-3-RedMagic4-8B-GGUF: