---
library_name: transformers
tags: []
---

# tiny-random-DeepseekV2ForCausalLM

Randomly initialised `DeepseekV2ForCausalLM` for testing MLA (multi-head latent
attention) and MoE code paths without loading a 29 GiB checkpoint. Used by
vLLM's `tests/models/transformers/test_backend.py::test_mla`.

The attention head dimensions are copied verbatim from
[deepseek-ai/DeepSeek-V2-Lite-Chat](https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite-Chat)
so that `kv_lora_rank + qk_rope_head_dim == 576`, the latent head size every MLA
kernel is built around. Only the width, depth and expert counts are shrunk.

|                     | DeepSeek-V2-Lite-Chat | this model |
| ------------------- | --------------------- | ---------- |
| `kv_lora_rank`      | 512                   | 512        |
| `qk_nope_head_dim`  | 128                   | 128        |
| `qk_rope_head_dim`  | 64                    | 64         |
| `v_head_dim`        | 128                   | 128        |
| `hidden_size`       | 2048                  | 256        |
| `num_hidden_layers` | 27                    | 4          |
| `num_attention_heads` | 16                  | 4          |
| `n_routed_experts`  | 64                    | 8          |
| `n_shared_experts`  | 2                     | 2          |
| parameters          | 15.7B                 | 59.8M      |

Layer 0 is dense and the rest are MoE (`first_k_dense_replace=1`), matching the
real model. The tokenizer is DeepSeek-V2-Lite-Chat's, unmodified.

The weights are random, so the outputs are meaningless. This is only useful for
checking that two implementations agree with each other.