Instructions to use Intel/deepseek-coder-1.3b_base_ov_int8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Intel/deepseek-coder-1.3b_base_ov_int8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Intel/deepseek-coder-1.3b_base_ov_int8")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Intel/deepseek-coder-1.3b_base_ov_int8")
model = AutoModelForCausalLM.from_pretrained("Intel/deepseek-coder-1.3b_base_ov_int8")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use Intel/deepseek-coder-1.3b_base_ov_int8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Intel/deepseek-coder-1.3b_base_ov_int8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Intel/deepseek-coder-1.3b_base_ov_int8",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/Intel/deepseek-coder-1.3b_base_ov_int8

SGLang

How to use Intel/deepseek-coder-1.3b_base_ov_int8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Intel/deepseek-coder-1.3b_base_ov_int8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Intel/deepseek-coder-1.3b_base_ov_int8",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Intel/deepseek-coder-1.3b_base_ov_int8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Intel/deepseek-coder-1.3b_base_ov_int8",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use Intel/deepseek-coder-1.3b_base_ov_int8 with Docker Model Runner:
```
docker model run hf.co/Intel/deepseek-coder-1.3b_base_ov_int8
```

kumarijy commited on May 21, 2024

Commit

ddd8ddd

verified ·

1 Parent(s): b00a3c2

Create README.md

Browse files

Files changed (1) hide show

README.md +51 -0

README.md ADDED Viewed

	@@ -0,0 +1,51 @@

+---
+license: mit
+language:
+- en
+---
+# Deepseek-coder-1.3b base Model Card
+The original source of this model is: [deepseek-ai/deepseek-coder-1.3b-base](https://huggingface.co/deepseek-ai/deepseek-coder-1.3b-base)
+This model is optimized and converted to OpenVino Intermediate Representation (IR) format using Optimum-cli.
+The model has been exported in Int8 by adding --weight-format Int8 while exporting this model from Huggingface.
+Intended to be used with:
+- [OpenVINO Code Completion - a VisualStudioCode extension for AI code completion with OpenVINO](https://marketplace.visualstudio.com/items?itemName=OpenVINO.openvino-code-completion)
+## Original Model Details
+- **Model type:** text generation model
+- **Language(s):** English
+- **License:** This model is licensed under the MIT License. The use of DeepSeek Coder model is subject to the model License. DeppSeek Coder supports commercial use. See the [LICENSE-MODEL](https://github.com/deepseek-ai/deepseek-coder/blob/main/LICENSE-MODEL) for more details.
+- **Model Summary:** deepseek-coder-1.3b-base is a 1.3B parameter model with Multi-Head Attention trained on 1 trillion tokens.
+- **Resources for more information:** [deepseek-coder-1.3b-base](https://huggingface.co/deepseek-ai/deepseek-coder-1.3b-base), [Paper](https://arxiv.org/abs/2401.14196).
+# Uses
+These models are pre-trained on a high-quality project-level code corpus and employ a fill-in-the-blank task with a 16K window to enhance code generation and infilling.
+## Direct Use
+DeepSeek-Coder models are under a permissive license that allows for both research and unrestricted commercial use. Possible tasks include
+- Code Completion
+- Code Insertion
+- Repository Level Code Completion
+### use base restrictions
+You agree not to use the Model or Derivatives of the Model:
+-	In any way that violates any applicable national or international law or regulation or infringes upon the lawful rights and interests of any third party;
+-	For military use in any way;
+-	For the purpose of exploiting, harming or attempting to exploit or harm minors in any way;
+-	To generate or disseminate verifiably false information and/or content with the purpose of harming others;
+-	To generate or disseminate inappropriate content subject to applicable regulatory requirements;
+-	To generate or disseminate personal identifiable information without due authorization or for unreasonable use;
+-	To defame, disparage or otherwise harass others;
+-	For fully automated decision making that adversely impacts an individual’s legal rights or otherwise creates or modifies a binding, enforceable obligation;
+-	For any use intended to or which has the effect of discriminating against or harming individuals or groups based on online or offline social behavior or known or predicted personal or personality characteristics;
+-	To exploit any of the vulnerabilities of a specific group of persons based on their age, social, physical or mental characteristics, in order to materially distort the behavior of a person pertaining to that group in a manner that causes or is likely to cause that person or another person physical or psychological harm;
+-	For any use intended to or which has the effect of discriminating against individuals or groups based on legally protected characteristics or categories.
+### Intel’s Human Rights Disclaimer:
+Intel is committed to respecting human rights and avoiding complicity in human rights abuses. See Intel's Global Human Rights Principles. Intel's products and software are intended only to be used in applications that do not cause or contribute to a violation of an internationally recognized human right.