Instructions to use openbmb/BitCPM-CANN-1B-unquantized with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use openbmb/BitCPM-CANN-1B-unquantized with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="openbmb/BitCPM-CANN-1B-unquantized", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("openbmb/BitCPM-CANN-1B-unquantized", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("openbmb/BitCPM-CANN-1B-unquantized", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use openbmb/BitCPM-CANN-1B-unquantized with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "openbmb/BitCPM-CANN-1B-unquantized"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "openbmb/BitCPM-CANN-1B-unquantized",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/openbmb/BitCPM-CANN-1B-unquantized

SGLang

How to use openbmb/BitCPM-CANN-1B-unquantized with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "openbmb/BitCPM-CANN-1B-unquantized" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "openbmb/BitCPM-CANN-1B-unquantized",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "openbmb/BitCPM-CANN-1B-unquantized" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "openbmb/BitCPM-CANN-1B-unquantized",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use openbmb/BitCPM-CANN-1B-unquantized with Docker Model Runner:
```
docker model run hf.co/openbmb/BitCPM-CANN-1B-unquantized
```

guanwenyu1995 commited on 3 days ago

Commit

d99c09f

verified ·

1 Parent(s): 5f9e13c

Upload config.json with huggingface_hub

Browse files

Files changed (1) hide show

config.json +139 -6

config.json CHANGED Viewed

@@ -9,7 +9,10 @@
         "AutoModelForCausalLM": "modeling_llama.LlamaForCausalLM"
     },
     "bos_token_id": 1,
-    "eos_token_id": [2,73440],
     "pad_token_id": 2,
     "hidden_act": "silu",
     "hidden_size": 2048,
@@ -24,13 +27,143 @@
     "rms_norm_eps": 1e-06,
     "rope_scaling": {
         "factor": 1.0,
-        "rope_type": "longrope",
-        "long_factor": [0.9977997200264581, 1.014658295992452, 1.0349680404997148, 1.059429246056193, 1.0888815016813513, 1.1243301355211495, 1.166977103606075, 1.2182568066927284, 1.2798772354275727, 1.3538666751582975, 1.4426259039919596, 1.5489853358570191, 1.6762658237220625, 1.8283407612492941, 2.0096956085876183, 2.225478927469756, 2.481536379650452, 2.784415934557119, 3.1413289096347365, 3.560047844772632, 4.048719380066383, 4.615569542115128, 5.2684819496549835, 6.014438591970396, 6.858830049237097, 7.804668263503327, 8.851768731513417, 9.99600492938444, 11.228766118181639, 12.536757560834843, 13.902257701387796, 15.303885189125953, 16.717837610115794, 18.119465097853947, 19.484965238406907, 20.792956681060105, 22.02571786985731, 23.16995406772833, 24.217054535738416, 25.16289275000465, 26.007284207271347, 26.753240849586767, 27.40615325712662, 27.973003419175363, 28.461674954469114, 28.880393889607006, 29.237306864684626, 29.540186419591297, 29.79624387177199, 30.01202719065413, 30.193382037992453, 30.34545697551969, 30.47273746338473, 30.579096895249787, 30.66785612408345, 30.741845563814174, 30.80346599254902, 30.85474569563567, 30.897392663720595, 30.932841297560394, 30.962293553185553, 30.986754758742034, 31.007064503249293, 31.02392307921529],
-	"short_factor": [0.9977997200264581, 1.014658295992452, 1.0349680404997148, 1.059429246056193, 1.0888815016813513, 1.1243301355211495, 1.166977103606075, 1.2182568066927284, 1.2798772354275727, 1.3538666751582975, 1.4426259039919596, 1.5489853358570191, 1.6762658237220625, 1.8283407612492941, 2.0096956085876183, 2.225478927469756, 2.481536379650452, 2.784415934557119, 3.1413289096347365, 3.560047844772632, 4.048719380066383, 4.615569542115128, 5.2684819496549835, 6.014438591970396, 6.858830049237097, 7.804668263503327, 8.851768731513417, 9.99600492938444, 11.228766118181639, 12.536757560834843, 13.902257701387796, 15.303885189125953, 16.717837610115794, 18.119465097853947, 19.484965238406907, 20.792956681060105, 22.02571786985731, 23.16995406772833, 24.217054535738416, 25.16289275000465, 26.007284207271347, 26.753240849586767, 27.40615325712662, 27.973003419175363, 28.461674954469114, 28.880393889607006, 29.237306864684626, 29.540186419591297, 29.79624387177199, 30.01202719065413, 30.193382037992453, 30.34545697551969, 30.47273746338473, 30.579096895249787, 30.66785612408345, 30.741845563814174, 30.80346599254902, 30.85474569563567, 30.897392663720595, 30.932841297560394, 30.962293553185553, 30.986754758742034, 31.007064503249293, 31.02392307921529],
-	"original_max_position_embeddings": 32768
     },
     "torch_dtype": "bfloat16",
-    "transformers_version": "4.36.0",
     "use_cache": true,
     "vocab_size": 73448
 }

         "AutoModelForCausalLM": "modeling_llama.LlamaForCausalLM"
     },
     "bos_token_id": 1,
+    "eos_token_id": [
+        2,
+        73440
+    ],
     "pad_token_id": 2,
     "hidden_act": "silu",
     "hidden_size": 2048,
     "rms_norm_eps": 1e-06,
     "rope_scaling": {
         "factor": 1.0,
+        "rope_type": "longrope",
+        "long_factor": [
+            0.9977997200264581,
+            1.014658295992452,
+            1.0349680404997148,
+            1.059429246056193,
+            1.0888815016813513,
+            1.1243301355211495,
+            1.166977103606075,
+            1.2182568066927284,
+            1.2798772354275727,
+            1.3538666751582975,
+            1.4426259039919596,
+            1.5489853358570191,
+            1.6762658237220625,
+            1.8283407612492941,
+            2.0096956085876183,
+            2.225478927469756,
+            2.481536379650452,
+            2.784415934557119,
+            3.1413289096347365,
+            3.560047844772632,
+            4.048719380066383,
+            4.615569542115128,
+            5.2684819496549835,
+            6.014438591970396,
+            6.858830049237097,
+            7.804668263503327,
+            8.851768731513417,
+            9.99600492938444,
+            11.228766118181639,
+            12.536757560834843,
+            13.902257701387796,
+            15.303885189125953,
+            16.717837610115794,
+            18.119465097853947,
+            19.484965238406907,
+            20.792956681060105,
+            22.02571786985731,
+            23.16995406772833,
+            24.217054535738416,
+            25.16289275000465,
+            26.007284207271347,
+            26.753240849586767,
+            27.40615325712662,
+            27.973003419175363,
+            28.461674954469114,
+            28.880393889607006,
+            29.237306864684626,
+            29.540186419591297,
+            29.79624387177199,
+            30.01202719065413,
+            30.193382037992453,
+            30.34545697551969,
+            30.47273746338473,
+            30.579096895249787,
+            30.66785612408345,
+            30.741845563814174,
+            30.80346599254902,
+            30.85474569563567,
+            30.897392663720595,
+            30.932841297560394,
+            30.962293553185553,
+            30.986754758742034,
+            31.007064503249293,
+            31.02392307921529
+        ],
+        "short_factor": [
+            0.9977997200264581,
+            1.014658295992452,
+            1.0349680404997148,
+            1.059429246056193,
+            1.0888815016813513,
+            1.1243301355211495,
+            1.166977103606075,
+            1.2182568066927284,
+            1.2798772354275727,
+            1.3538666751582975,
+            1.4426259039919596,
+            1.5489853358570191,
+            1.6762658237220625,
+            1.8283407612492941,
+            2.0096956085876183,
+            2.225478927469756,
+            2.481536379650452,
+            2.784415934557119,
+            3.1413289096347365,
+            3.560047844772632,
+            4.048719380066383,
+            4.615569542115128,
+            5.2684819496549835,
+            6.014438591970396,
+            6.858830049237097,
+            7.804668263503327,
+            8.851768731513417,
+            9.99600492938444,
+            11.228766118181639,
+            12.536757560834843,
+            13.902257701387796,
+            15.303885189125953,
+            16.717837610115794,
+            18.119465097853947,
+            19.484965238406907,
+            20.792956681060105,
+            22.02571786985731,
+            23.16995406772833,
+            24.217054535738416,
+            25.16289275000465,
+            26.007284207271347,
+            26.753240849586767,
+            27.40615325712662,
+            27.973003419175363,
+            28.461674954469114,
+            28.880393889607006,
+            29.237306864684626,
+            29.540186419591297,
+            29.79624387177199,
+            30.01202719065413,
+            30.193382037992453,
+            30.34545697551969,
+            30.47273746338473,
+            30.579096895249787,
+            30.66785612408345,
+            30.741845563814174,
+            30.80346599254902,
+            30.85474569563567,
+            30.897392663720595,
+            30.932841297560394,
+            30.962293553185553,
+            30.986754758742034,
+            31.007064503249293,
+            31.02392307921529
+        ],
+        "original_max_position_embeddings": 32768
     },
     "torch_dtype": "bfloat16",
+    "transformers_version": "4.46.3",
     "use_cache": true,
     "vocab_size": 73448
 }