Instructions to use nold/Magicoder-S-DS-6.7B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nold/Magicoder-S-DS-6.7B-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="nold/Magicoder-S-DS-6.7B-GGUF")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("nold/Magicoder-S-DS-6.7B-GGUF", dtype="auto")

llama-cpp-python

How to use nold/Magicoder-S-DS-6.7B-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="nold/Magicoder-S-DS-6.7B-GGUF",
	filename="Magicoder-S-DS-6.7B_Q2_K.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use nold/Magicoder-S-DS-6.7B-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf nold/Magicoder-S-DS-6.7B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf nold/Magicoder-S-DS-6.7B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf nold/Magicoder-S-DS-6.7B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf nold/Magicoder-S-DS-6.7B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf nold/Magicoder-S-DS-6.7B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf nold/Magicoder-S-DS-6.7B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf nold/Magicoder-S-DS-6.7B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf nold/Magicoder-S-DS-6.7B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/nold/Magicoder-S-DS-6.7B-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use nold/Magicoder-S-DS-6.7B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nold/Magicoder-S-DS-6.7B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nold/Magicoder-S-DS-6.7B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/nold/Magicoder-S-DS-6.7B-GGUF:Q4_K_M

SGLang

How to use nold/Magicoder-S-DS-6.7B-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nold/Magicoder-S-DS-6.7B-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nold/Magicoder-S-DS-6.7B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nold/Magicoder-S-DS-6.7B-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nold/Magicoder-S-DS-6.7B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use nold/Magicoder-S-DS-6.7B-GGUF with Ollama:
```
ollama run hf.co/nold/Magicoder-S-DS-6.7B-GGUF:Q4_K_M
```

Unsloth Studio new

How to use nold/Magicoder-S-DS-6.7B-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for nold/Magicoder-S-DS-6.7B-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for nold/Magicoder-S-DS-6.7B-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for nold/Magicoder-S-DS-6.7B-GGUF to start chatting

Docker Model Runner
How to use nold/Magicoder-S-DS-6.7B-GGUF with Docker Model Runner:
```
docker model run hf.co/nold/Magicoder-S-DS-6.7B-GGUF:Q4_K_M
```

Lemonade

How to use nold/Magicoder-S-DS-6.7B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull nold/Magicoder-S-DS-6.7B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Magicoder-S-DS-6.7B-GGUF-Q4_K_M

List all available models

lemonade list

🎩 Magicoder: Source Code Is All You Need

Refer to our GitHub repo ise-uiuc/magicoder for an up-to-date introduction to the Magicoder family!

🎩Magicoder is a model family empowered by 🪄OSS-Instruct, a novel approach to enlightening LLMs with open-source code snippets for generating low-bias and high-quality instruction data for code.
🪄OSS-Instruct mitigates the inherent bias of the LLM-synthesized instruction data by empowering them with a wealth of open-source references to produce more diverse, realistic, and controllable data.

Model Details

Model Description

Developed by: Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, Lingming Zhang
License: DeepSeek
Finetuned from model: deepseek-coder-6.7b-base

Model Sources

Repository: https://github.com/ise-uiuc/magicoder
Paper: https://arxiv.org/abs/2312.02120
Demo (powered by Gradio): https://github.com/ise-uiuc/magicoder/tree/main/demo

Training Data

Magicoder-OSS-Instruct-75K: generated through OSS-Instruct using gpt-3.5-turbo-1106 and used to train both Magicoder and Magicoder-S series.
Magicoder-Evol-Instruct-110K: decontaminated and redistributed from theblackcat102/evol-codealpaca-v1, used to further finetune Magicoder series and obtain Magicoder-S models.

Uses

Direct Use

Magicoders are designed and best suited for coding tasks.

Out-of-Scope Use

Magicoders may not work well in non-coding tasks.

Bias, Risks, and Limitations

Magicoders may sometimes make errors, producing misleading contents, or struggle to manage tasks that are not related to coding.

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.

How to Get Started with the Model

Use the code below to get started with the model. Make sure you installed the transformers library.

from transformers import pipeline
import torch

MAGICODER_PROMPT = """You are an exceptionally intelligent coding assistant that consistently delivers accurate and reliable responses to user instructions.

@@ Instruction
{instruction}

@@ Response
"""

instruction = <Your code instruction here>

prompt = MAGICODER_PROMPT.format(instruction=instruction)
generator = pipeline(
    model="ise-uiuc/Magicoder-S-DS-6.7B",
    task="text-generation",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
result = generator(prompt, max_length=1024, num_return_sequences=1, temperature=0.0)
print(result[0]["generated_text"])

Technical Details

Refer to our GitHub repo: ise-uiuc/magicoder.

Citation

@misc{magicoder,
    title={Magicoder: Source Code Is All You Need}, 
    author={Yuxiang Wei and Zhe Wang and Jiawei Liu and Yifeng Ding and Lingming Zhang},
    year={2023},
    eprint={2312.02120},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

Acknowledgements

WizardCoder: Evol-Instruct
DeepSeek-Coder: Base model for Magicoder-DS
CodeLlama: Base model for Magicoder-CL
StarCoder: Data decontamination

Important Note

Magicoder models are trained on the synthetic data generated by OpenAI models. Please pay attention to OpenAI's terms of use when using the models and the datasets. Magicoders will not compete with OpenAI's commercial products.

Vanilla Quantization by nold, Original Model ise-uiuc/Magicoder-S-DS-6.7B. Created using llm-quantizer Pipeline - c21aafcd05203496bf1294e97058a17efa858c8a

Downloads last month: 9

GGUF

Model size

7B params

Architecture

llama

Hardware compatibility

2-bit

4-bit

5-bit

8-bit

Datasets used to train nold/Magicoder-S-DS-6.7B-GGUF

Papers for nold/Magicoder-S-DS-6.7B-GGUF

Magicoder: Source Code Is All You Need

Paper • 2312.02120 • Published Dec 4, 2023 • 82

StarCoder: may the source be with you!

Paper • 2305.06161 • Published May 9, 2023 • 33