Instructions to use QuantFactory/OpenMath2-Llama3.1-8B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use QuantFactory/OpenMath2-Llama3.1-8B-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="QuantFactory/OpenMath2-Llama3.1-8B-GGUF",
	filename="OpenMath2-Llama3.1-8B.Q2_K.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use QuantFactory/OpenMath2-Llama3.1-8B-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf QuantFactory/OpenMath2-Llama3.1-8B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf QuantFactory/OpenMath2-Llama3.1-8B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf QuantFactory/OpenMath2-Llama3.1-8B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf QuantFactory/OpenMath2-Llama3.1-8B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf QuantFactory/OpenMath2-Llama3.1-8B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf QuantFactory/OpenMath2-Llama3.1-8B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf QuantFactory/OpenMath2-Llama3.1-8B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf QuantFactory/OpenMath2-Llama3.1-8B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/QuantFactory/OpenMath2-Llama3.1-8B-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use QuantFactory/OpenMath2-Llama3.1-8B-GGUF with Ollama:
```
ollama run hf.co/QuantFactory/OpenMath2-Llama3.1-8B-GGUF:Q4_K_M
```

Unsloth Studio new

How to use QuantFactory/OpenMath2-Llama3.1-8B-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for QuantFactory/OpenMath2-Llama3.1-8B-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for QuantFactory/OpenMath2-Llama3.1-8B-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for QuantFactory/OpenMath2-Llama3.1-8B-GGUF to start chatting

Pi new

How to use QuantFactory/OpenMath2-Llama3.1-8B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf QuantFactory/OpenMath2-Llama3.1-8B-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "QuantFactory/OpenMath2-Llama3.1-8B-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use QuantFactory/OpenMath2-Llama3.1-8B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf QuantFactory/OpenMath2-Llama3.1-8B-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default QuantFactory/OpenMath2-Llama3.1-8B-GGUF:Q4_K_M

Run Hermes

hermes

Docker Model Runner
How to use QuantFactory/OpenMath2-Llama3.1-8B-GGUF with Docker Model Runner:
```
docker model run hf.co/QuantFactory/OpenMath2-Llama3.1-8B-GGUF:Q4_K_M
```

Lemonade

How to use QuantFactory/OpenMath2-Llama3.1-8B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull QuantFactory/OpenMath2-Llama3.1-8B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.OpenMath2-Llama3.1-8B-GGUF-Q4_K_M

List all available models

lemonade list

QuantFactory/OpenMath2-Llama3.1-8B-GGUF

This is quantized version of nvidia/OpenMath2-Llama3.1-8B created using llama.cpp

Original Model Card

OpenMath2-Llama3.1-8B

OpenMath2-Llama3.1-8B is obtained by finetuning Llama3.1-8B-Base with OpenMathInstruct-2.

The model outperforms Llama3.1-8B-Instruct on all the popular math benchmarks we evaluate on, especially on MATH by 15.9%.

Performance of Llama-3.1-8B-Instruct as it is trained on increasing proportions of OpenMathInstruct-2

Comparison of OpenMath2-Llama3.1-8B vs. Llama-3.1-8B-Instruct across MATH levels

Model	GSM8K	MATH	AMC 2023	AIME 2024	Omni-MATH
Llama3.1-8B-Instruct	84.5	51.9	9/40	2/30	12.7
OpenMath2-Llama3.1-8B (nemo \| HF)	91.7	67.8	16/40	3/30	22.0
+ majority@256	94.1	76.1	23/40	3/30	24.6
Llama3.1-70B-Instruct	95.8	67.9	19/40	6/30	19.0
OpenMath2-Llama3.1-70B (nemo \| HF)	94.9	71.9	20/40	4/30	23.1
+ majority@256	96.0	79.6	24/40	6/30	27.6

The pipeline we used to produce the data and models is fully open-sourced!

See our paper to learn more details!

How to use the models?

Our models are trained with the same "chat format" as Llama3.1-instruct models (same system/user/assistant tokens). Please note that these models have not been instruction tuned on general data and thus might not provide good answers outside of math domain.

We recommend using instructions in our repo to run inference with these models, but here is an example of how to do it through transformers api:

import transformers
import torch

model_id = "nvidia/OpenMath2-Llama3.1-8B"

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)

messages = [
    {
        "role": "user", 
        "content": "Solve the following math problem. Make sure to put the answer (and only answer) inside \\boxed{}.\n\n" + 
        "What is the minimum value of $a^2+6a-7$?"},
]

outputs = pipeline(
    messages,
    max_new_tokens=4096,
)
print(outputs[0]["generated_text"][-1]['content'])

Reproducing our results

We provide all instructions to fully reproduce our results.

Citation

If you find our work useful, please consider citing us!

@article{toshniwal2024openmath2,
  title   = {OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data},
  author  = {Shubham Toshniwal and Wei Du and Ivan Moshkov and  Branislav Kisacanin and Alexan Ayrapetyan and Igor Gitman},
  year    = {2024},
  journal = {arXiv preprint arXiv:2410.01560}
}

Terms of use

By accessing this model, you are agreeing to the LLama 3.1 terms and conditions of the license, acceptable use policy and Meta’s privacy policy

Downloads last month: 390

GGUF

Model size

8B params

Architecture

llama

Hardware compatibility

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for QuantFactory/OpenMath2-Llama3.1-8B-GGUF

Base model

meta-llama/Llama-3.1-8B

Quantized

(324)

this model

Dataset used to train QuantFactory/OpenMath2-Llama3.1-8B-GGUF

Paper for QuantFactory/OpenMath2-Llama3.1-8B-GGUF

OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data

Paper • 2410.01560 • Published Oct 2, 2024 • 6