| # Custom Pipeline Model Parallel Layout | |
| *This is an experimental feature and may be changed.* | |
| `--pipeline-model-parallel-layout` is a flexible API for defining the pipeline parallel partitioning, which is essential for balanced partitioning for an imbalanced model. For example, to partition DeepSeek-V3 (61 decoder layers + 1 mtp layer) with PP16VPP2, we can include the arguments as follows: | |
| ```bash | |
| --pipeline-model-parallel-size 16 | |
| --pipeline-model-parallel-layout "Et*3|(tt|)*29,m|L" | |
| ``` | |
| | PP \ VPP rank | 0 | 1 | | |
| |---------------|-------------------------|---------------| | |
| | 0 | embedding + 3 × decoder | 2 × decoder | | |
| | 1~13 | 2 × decoder | 2 × decoder | | |
| | 14 | 2 × decoder | mtp | | |
| | 15 | 2 × decoder | loss | | |
| In the layout string, stages are split by '|'. Replicated stages or layers can be described with multiplication. Commas can be used cosmetically. Symbol choices: | |
| * `E` = embedding layer | |
| * `t` = transformer decoder layer | |
| * `m` = MTP layer | |
| * `L` = loss calculation layer | |
| Note that it is legal to have empty stages, e.g., `E||t|L` (the second stage is empty). | |