Instructions to use ai21labs/AI21-Jamba2-Mini with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ai21labs/AI21-Jamba2-Mini with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="ai21labs/AI21-Jamba2-Mini") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("ai21labs/AI21-Jamba2-Mini") model = AutoModelForCausalLM.from_pretrained("ai21labs/AI21-Jamba2-Mini", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use ai21labs/AI21-Jamba2-Mini with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "ai21labs/AI21-Jamba2-Mini" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ai21labs/AI21-Jamba2-Mini", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/ai21labs/AI21-Jamba2-Mini
- SGLang
How to use ai21labs/AI21-Jamba2-Mini with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "ai21labs/AI21-Jamba2-Mini" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ai21labs/AI21-Jamba2-Mini", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "ai21labs/AI21-Jamba2-Mini" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "ai21labs/AI21-Jamba2-Mini", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use ai21labs/AI21-Jamba2-Mini with Docker Model Runner:
docker model run hf.co/ai21labs/AI21-Jamba2-Mini
Reorganize Safetensors to Layer-Sequential Sharding
#2
by asafgardin - opened
- model-00001-of-00021.safetensors +2 -2
- model-00002-of-00021.safetensors +2 -2
- model-00003-of-00021.safetensors +2 -2
- model-00004-of-00021.safetensors +2 -2
- model-00005-of-00021.safetensors +2 -2
- model-00006-of-00021.safetensors +2 -2
- model-00007-of-00021.safetensors +2 -2
- model-00008-of-00021.safetensors +2 -2
- model-00009-of-00021.safetensors +2 -2
- model-00010-of-00021.safetensors +2 -2
- model-00011-of-00021.safetensors +2 -2
- model-00012-of-00021.safetensors +2 -2
- model-00013-of-00021.safetensors +2 -2
- model-00014-of-00021.safetensors +2 -2
- model-00015-of-00021.safetensors +2 -2
- model-00016-of-00021.safetensors +2 -2
- model-00017-of-00021.safetensors +2 -2
- model-00018-of-00021.safetensors +2 -2
- model-00019-of-00021.safetensors +2 -2
- model-00020-of-00021.safetensors +2 -2
- model-00021-of-00021.safetensors +2 -2
- model.safetensors.index.json +0 -0
model-00001-of-00021.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4811d18e57cce4c4ac4f28a92600e6bc82e343929b3d6ec159a8d956fee5af59
|
| 3 |
+
size 4833813080
|
model-00002-of-00021.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:48f10c383ef9b52c396ecb5779454d03483336e5da51fff9ab2c72e05509647a
|
| 3 |
+
size 4884145320
|
model-00003-of-00021.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c66d92ef2d9ac22fe6c01c66ecadd8807233a2b57019e2be21fd5ab40a216db9
|
| 3 |
+
size 4874854304
|
model-00004-of-00021.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0d2e9182ff7abdb197a35fc8e4a67bcbc2ca0a9d0e2a0220c5198a8cd477f4d4
|
| 3 |
+
size 4815066464
|
model-00005-of-00021.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2fd69b8cb738e095712c00ea2600a5a5966c96c74f7812552b9b56d60c397157
|
| 3 |
+
size 4884145288
|
model-00006-of-00021.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:735bd9617c5c7e2196fb2b9dee8bf25a24f239679711b708561fe3f2f21d9841
|
| 3 |
+
size 4884145320
|
model-00007-of-00021.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:936959101cdc3a6cf153012fadcc0f1b65da95a5c72896e17e3331708698381b
|
| 3 |
+
size 4884145336
|
model-00008-of-00021.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:461b11a7926fce899e27e430314c160e1db2286a2d9f2138735d22f998f79efc
|
| 3 |
+
size 4898969664
|
model-00009-of-00021.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:77d4aa097a1cf01bca8d51745fcd8dccf2b11013ce3235d0b981b32200f5cd40
|
| 3 |
+
size 4908391840
|
model-00010-of-00021.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:df94f3d9716a9b4a59e2c6e8af1cf098d2c3a561822d3ee6f6360f47048fb8d5
|
| 3 |
+
size 4884145384
|
model-00011-of-00021.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0eb252c7e49350a707789b4e73c4a686d9ab7d051235442343e1ab6e63492fa9
|
| 3 |
+
size 4884145392
|
model-00012-of-00021.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:114416e5e53188167f80f3f9ca653874edae9c538a2dedd16d394d5ab3b1edd4
|
| 3 |
+
size 4908647240
|
model-00013-of-00021.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:dd82bd7edb1c80cdca7848a4bd45ba46a5b4140b4da23cb6acab14f302f3a702
|
| 3 |
+
size 4908005296
|
model-00014-of-00021.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ec2b2336e1a83affe50a56213c0d8f1d93f9d353b808474feeb33ae8316d9512
|
| 3 |
+
size 4874854352
|
model-00015-of-00021.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:21b0e4a286b73e52846a5588e94cfbdd62611f88a45775313ae3299cb6fe1464
|
| 3 |
+
size 4884145392
|
model-00016-of-00021.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:17d2a755fa6dce4dbd529a9bf8bd7210df876bbd8b8d42190e26252dcc5fb5c8
|
| 3 |
+
size 4884145392
|
model-00017-of-00021.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:46ba42d2f02985d83fb7463672b949788c54eb4cc8e7d37da89373115024a85c
|
| 3 |
+
size 4815066496
|
model-00018-of-00021.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2668c763fbe0af5fbe1a89561d2e75868f3f3a500c65fe8e96599d9e9b331e74
|
| 3 |
+
size 4884145352
|
model-00019-of-00021.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e92751e8071ade6fe8ea73fcbb82304d1beadedf78c4c7c9de536546b89f4ae6
|
| 3 |
+
size 4874854368
|
model-00020-of-00021.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:59a1196ff5401b4ae39e8b3b72d678c09fd027861063045a1bd548ee781ca354
|
| 3 |
+
size 4884145392
|
model-00021-of-00021.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6a14e32ce6083f0d77114579b08206527b6edc1c0a79bff53c86aac3d9525873
|
| 3 |
+
size 5586827104
|
model.safetensors.index.json
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|