Instructions to use hmellor/tiny-random-DeepseekV2ForCausalLM with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use hmellor/tiny-random-DeepseekV2ForCausalLM with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="hmellor/tiny-random-DeepseekV2ForCausalLM")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("hmellor/tiny-random-DeepseekV2ForCausalLM")
model = AutoModelForCausalLM.from_pretrained("hmellor/tiny-random-DeepseekV2ForCausalLM", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use hmellor/tiny-random-DeepseekV2ForCausalLM with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "hmellor/tiny-random-DeepseekV2ForCausalLM"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hmellor/tiny-random-DeepseekV2ForCausalLM",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/hmellor/tiny-random-DeepseekV2ForCausalLM

SGLang

How to use hmellor/tiny-random-DeepseekV2ForCausalLM with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "hmellor/tiny-random-DeepseekV2ForCausalLM" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hmellor/tiny-random-DeepseekV2ForCausalLM",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "hmellor/tiny-random-DeepseekV2ForCausalLM" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hmellor/tiny-random-DeepseekV2ForCausalLM",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use hmellor/tiny-random-DeepseekV2ForCausalLM with Docker Model Runner:
```
docker model run hf.co/hmellor/tiny-random-DeepseekV2ForCausalLM
```

hmellor HF Staff commited on 7 days ago

Commit

c0146ef

verified ·

1 Parent(s): 21169e1

Add tiny random DeepseekV2 for vLLM MLA tests

Browse files

Files changed (7) hide show

README.md +34 -0
chat_template.jinja +5 -0
config.json +57 -0
generation_config.json +9 -0
model.safetensors +3 -0
tokenizer.json +0 -0
tokenizer_config.json +14 -0

README.md ADDED Viewed

	@@ -0,0 +1,34 @@

+---
+library_name: transformers
+tags: []
+---
+# tiny-random-DeepseekV2ForCausalLM
+Randomly initialised `DeepseekV2ForCausalLM` for testing MLA (multi-head latent
+attention) and MoE code paths without loading a 29 GiB checkpoint. Used by
+vLLM's `tests/models/transformers/test_backend.py::test_mla`.
+The attention head dimensions are copied verbatim from
+[deepseek-ai/DeepSeek-V2-Lite-Chat](https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite-Chat)
+so that `kv_lora_rank + qk_rope_head_dim == 576`, the latent head size every MLA
+kernel is built around. Only the width, depth and expert counts are shrunk.
+|                     | DeepSeek-V2-Lite-Chat | this model |
+| ------------------- | --------------------- | ---------- |
+| `kv_lora_rank`      | 512                   | 512        |
+| `qk_nope_head_dim`  | 128                   | 128        |
+| `qk_rope_head_dim`  | 64                    | 64         |
+| `v_head_dim`        | 128                   | 128        |
+| `hidden_size`       | 2048                  | 256        |
+| `num_hidden_layers` | 27                    | 4          |
+| `num_attention_heads` | 16                  | 4          |
+| `n_routed_experts`  | 64                    | 8          |
+| `n_shared_experts`  | 2                     | 2          |
+| parameters          | 15.7B                 | 59.8M      |
+Layer 0 is dense and the rest are MoE (`first_k_dense_replace=1`), matching the
+real model. The tokenizer is DeepSeek-V2-Lite-Chat's, unmodified.
+The weights are random, so the outputs are meaningless. This is only useful for
+checking that two implementations agree with each other.

chat_template.jinja ADDED Viewed

	@@ -0,0 +1,5 @@

+{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{{ bos_token }}{% for message in messages %}{% if message['role'] == 'user' %}{{ 'User: ' + message['content'] + '
+' }}{% elif message['role'] == 'assistant' %}{{ 'Assistant: ' + message['content'] + eos_token }}{% elif message['role'] == 'system' %}{{ message['content'] + '
+' }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ 'Assistant:' }}{% endif %}

config.json ADDED Viewed

	@@ -0,0 +1,57 @@

+{
+  "architectures": [
+    "DeepseekV2ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 100000,
+  "dtype": "bfloat16",
+  "eos_token_id": 100001,
+  "first_k_dense_replace": 1,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 256,
+  "initializer_range": 0.02,
+  "intermediate_size": 512,
+  "kv_lora_rank": 512,
+  "max_position_embeddings": 163840,
+  "mlp_bias": false,
+  "model_type": "deepseek_v2",
+  "moe_intermediate_size": 128,
+  "moe_layer_freq": 1,
+  "n_group": 1,
+  "n_routed_experts": 8,
+  "n_shared_experts": 2,
+  "norm_topk_prob": false,
+  "num_attention_heads": 4,
+  "num_experts_per_tok": 2,
+  "num_hidden_layers": 4,
+  "num_key_value_heads": 4,
+  "pad_token_id": null,
+  "pretraining_tp": 1,
+  "q_lora_rank": null,
+  "qk_head_dim": 192,
+  "qk_nope_head_dim": 128,
+  "qk_rope_head_dim": 64,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "beta_fast": 32,
+    "beta_slow": 1,
+    "factor": 40,
+    "mscale": 0.707,
+    "mscale_all_dim": 0.707,
+    "original_max_position_embeddings": 4096,
+    "rope_theta": 10000,
+    "rope_type": "yarn",
+    "type": "yarn"
+  },
+  "routed_scaling_factor": 1.0,
+  "scoring_func": "softmax",
+  "tie_word_embeddings": false,
+  "topk_group": 1,
+  "topk_method": "greedy",
+  "transformers_version": "5.15.0.dev0",
+  "use_cache": true,
+  "v_head_dim": 128,
+  "vocab_size": 102400
+}

generation_config.json ADDED Viewed

	@@ -0,0 +1,9 @@

+{
+  "_from_model_config": true,
+  "bos_token_id": 100000,
+  "eos_token_id": 100001,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "transformers_version": "5.15.0.dev0",
+  "use_cache": true
+}

model.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:652d9531e43e7cf8338120cbe4ca21948b0bc8585eaa80e5b990a21c95fbddd3
+size 119572560

tokenizer.json ADDED Viewed

The diff for this file is too large to render. See raw diff

tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,14 @@

+{
+  "backend": "tokenizers",
+  "bos_token": "<｜begin▁of▁sentence｜>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<｜end▁of▁sentence｜>",
+  "is_local": false,
+  "legacy": true,
+  "local_files_only": false,
+  "model_max_length": 16384,
+  "pad_token": "<｜end▁of▁sentence｜>",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": null
+}