Instructions to use RedHatAI/Llama-2-7b-evol-code-alpaca-pruned_50-quantized-deepsparse with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RedHatAI/Llama-2-7b-evol-code-alpaca-pruned_50-quantized-deepsparse with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="RedHatAI/Llama-2-7b-evol-code-alpaca-pruned_50-quantized-deepsparse")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("RedHatAI/Llama-2-7b-evol-code-alpaca-pruned_50-quantized-deepsparse")
model = AutoModelForCausalLM.from_pretrained("RedHatAI/Llama-2-7b-evol-code-alpaca-pruned_50-quantized-deepsparse")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use RedHatAI/Llama-2-7b-evol-code-alpaca-pruned_50-quantized-deepsparse with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RedHatAI/Llama-2-7b-evol-code-alpaca-pruned_50-quantized-deepsparse"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/Llama-2-7b-evol-code-alpaca-pruned_50-quantized-deepsparse",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/RedHatAI/Llama-2-7b-evol-code-alpaca-pruned_50-quantized-deepsparse

SGLang

How to use RedHatAI/Llama-2-7b-evol-code-alpaca-pruned_50-quantized-deepsparse with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RedHatAI/Llama-2-7b-evol-code-alpaca-pruned_50-quantized-deepsparse" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/Llama-2-7b-evol-code-alpaca-pruned_50-quantized-deepsparse",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RedHatAI/Llama-2-7b-evol-code-alpaca-pruned_50-quantized-deepsparse" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/Llama-2-7b-evol-code-alpaca-pruned_50-quantized-deepsparse",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use RedHatAI/Llama-2-7b-evol-code-alpaca-pruned_50-quantized-deepsparse with Docker Model Runner:
```
docker model run hf.co/RedHatAI/Llama-2-7b-evol-code-alpaca-pruned_50-quantized-deepsparse
```

Abhinav Agarwalla commited on May 15, 2024

Commit

91f7aa0

1 Parent(s): b5a9b63

Updating pruned50-quant model

Browse files

Files changed (7) hide show

README.md +2 -2
config.json +4 -3
model-orig.onnx +2 -2
model.data +2 -2
model.onnx +2 -2
tokenizer.json +1 -1
tokenizer_config.json +0 -4

README.md CHANGED Viewed

@@ -50,8 +50,8 @@ Model evaluation metrics and results.
 | Benchmark                                      | Metric        | Llama-2-7b-evolcodealpaca  | Llama-2-7b-pruned50-retrained-evolcodealpaca-quant-ds |
 |------------------------------------------------|---------------|-------------|-------------------------------|
-| [HumanEval](https://arxiv.org/abs/2107.03374)  | pass@1        | 32.03       | 32.2                          |
 ## Help
-For further support, and discussions on these models and AI in general, join [Neural Magic's Slack Community](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-q1a1cnvo-YBoICSIw3L1dmQpjBeDurQ)

 | Benchmark                                      | Metric        | Llama-2-7b-evolcodealpaca  | Llama-2-7b-pruned50-retrained-evolcodealpaca-quant-ds |
 |------------------------------------------------|---------------|-------------|-------------------------------|
+| [HumanEval](https://arxiv.org/abs/2107.03374)  | pass@1        | 32.03       | 36.34                         |
 ## Help
+For further support, and discussions on these models and AI in general, join [Neural Magic's Slack Community](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-q1a1cnvo-YBoICSIw3L1dmQpjBeDurQ)

config.json CHANGED Viewed

@@ -1,5 +1,5 @@
 {
-  "_name_or_path": "/home/abhinav/src/llama-recipes/llama_7b_evol_codealpaca_sparse/sparse-sft_cerebras_50sp_llama_lr5e-4_epochs3_gradclip5.0_cosine-/combined/",
   "architectures": [
     "LlamaForCausalLM"
   ],
@@ -21,8 +21,9 @@
   "rope_scaling": null,
   "rope_theta": 10000.0,
   "tie_word_embeddings": false,
-  "torch_dtype": "float32",
-  "transformers_version": "1.7.0.20240312",
   "use_cache": true,
   "vocab_size": 32000
 }

 {
+  "_name_or_path": "neuralmagic/Llama-2-7b-pruned50-retrained-evolcodealpaca-quant-ds",
   "architectures": [
     "LlamaForCausalLM"
   ],
   "rope_scaling": null,
   "rope_theta": 10000.0,
   "tie_word_embeddings": false,
+  "tokenizer_class": "LlamaTokenizerFast",
+  "torch_dtype": "float16",
+  "transformers_version": "1.7.0.20240506",
   "use_cache": true,
   "vocab_size": 32000
 }

model-orig.onnx CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:16cb24f5b49e63173e59d9da7164a74fe968a51ddf20ef30a65dd73b87342390
-size 1220058

 version https://git-lfs.github.com/spec/v1
+oid sha256:d363691fa0eab6564db9d26c5603fd328d74a9860e2d73e962f9d000913181df
+size 1222751

model.data CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:48e44b79d03f06427edc1d451b31169c04966366fff80933784bb4d84df0f213
-size 7425272832

 version https://git-lfs.github.com/spec/v1
+oid sha256:aa7081720c95cd1692e42b5cc95b1fded0d9b7560058076bbc9168835a42372c
+size 7154772992

model.onnx CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:db93bc16d048df38acd6eaa934e5d1bb0beca2becc6023ea0ebcdfb090a697ee
-size 1205003

 version https://git-lfs.github.com/spec/v1
+oid sha256:7a6ad1f6b97d487e6966d5b7902b2aec84bf28771841d3c09d72bf73d75591e4
+size 1207696

tokenizer.json CHANGED Viewed

@@ -2,7 +2,7 @@
   "version": "1.0",
   "truncation": {
     "direction": "Right",
-    "max_length": 2048,
     "strategy": "LongestFirst",
     "stride": 0
   },

   "version": "1.0",
   "truncation": {
     "direction": "Right",
+    "max_length": 384,
     "strategy": "LongestFirst",
     "stride": 0
   },

tokenizer_config.json CHANGED Viewed

@@ -29,15 +29,11 @@
   "clean_up_tokenization_spaces": false,
   "eos_token": "</s>",
   "legacy": false,
-  "max_length": 2048,
   "model_max_length": 1000000000000000019884624838656,
   "pad_token": "</s>",
   "padding_side": "right",
   "sp_model_kwargs": {},
-  "stride": 0,
   "tokenizer_class": "LlamaTokenizer",
-  "truncation_side": "right",
-  "truncation_strategy": "longest_first",
   "unk_token": "<unk>",
   "use_default_system_prompt": false
 }

   "clean_up_tokenization_spaces": false,
   "eos_token": "</s>",
   "legacy": false,
   "model_max_length": 1000000000000000019884624838656,
   "pad_token": "</s>",
   "padding_side": "right",
   "sp_model_kwargs": {},
   "tokenizer_class": "LlamaTokenizer",
   "unk_token": "<unk>",
   "use_default_system_prompt": false
 }