Instructions to use RedHatAI/Llama-2-7b-gsm8k with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RedHatAI/Llama-2-7b-gsm8k with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="RedHatAI/Llama-2-7b-gsm8k")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("RedHatAI/Llama-2-7b-gsm8k")
model = AutoModelForCausalLM.from_pretrained("RedHatAI/Llama-2-7b-gsm8k")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use RedHatAI/Llama-2-7b-gsm8k with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RedHatAI/Llama-2-7b-gsm8k"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/Llama-2-7b-gsm8k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/RedHatAI/Llama-2-7b-gsm8k

SGLang

How to use RedHatAI/Llama-2-7b-gsm8k with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RedHatAI/Llama-2-7b-gsm8k" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/Llama-2-7b-gsm8k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RedHatAI/Llama-2-7b-gsm8k" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/Llama-2-7b-gsm8k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use RedHatAI/Llama-2-7b-gsm8k with Docker Model Runner:
```
docker model run hf.co/RedHatAI/Llama-2-7b-gsm8k
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Llama-2-7b-gsm8k

This repo contains a dense Llama 2 7B finetuned for arithmetic reasoning task using the GSM8k dataset.

Official model weights from Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment.

Authors: Neural Magic, Cerebras

Usage

Below we share some code snippets on how to get quickly started with running the model.

Running the model

# pip install transformers accelerate
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("neuralmagic/Llama-2-7b-gsm8k")
model = AutoModelForCausalLM.from_pretrained("neuralmagic/Llama-2-7b-gsm8k", device_map="auto")

input_text = "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?"
input_ids = tokenizer.apply_chat_template(input_text, add_generation_prompt=True, return_tensors="pt").to("cuda")

outputs = model.generate(**input_ids)
print(tokenizer.decode(outputs[0]))

Evaluation Benchmark Results

Model evaluation metrics and results.

Benchmark	Metric	Llama-2-7b-gsm8k
GSM8K	0-shot	35.5%

Model Training Details

This model was obtained by fine-tuning the dense Llama 2 7B on the GSM8k dataset. Fine-tuning was performed for 2 epochs with batch-size of 32, with linearly decaying learning-rate from initial value of 3e-5 and warm-up phase of 20 steps.