Buckets:
16.9 MB
16 files
Updated 2 days ago
Ctrl+K
| Name | Size | Uploaded | Xet hash |
|---|---|---|---|
| README.md | 783 Bytes xet | 23ac6803 | |
| config_671B_v3.2.json | 605 Bytes xet | 7c221bdd | |
| convert.py | 3.97 kB xet | a9cb05d3 | |
| dist_writer.py | 9.51 kB xet | a761dd72 | |
| generate.py | 8.1 kB xet | 0d32d322 | |
| model_v32.py | 39.5 kB xet | 884ee251 | |
| requirements.txt | 83 Bytes xet | 2f9a6ae2 | |
| tilelang_kernel.py | 9.96 kB xet | 66761d76 |
Inference code for DeepSeek models
First convert huggingface model weight files to the format of this project.
python convert.py --hf-ckpt-path ${HF_CKPT_PATH} --save-path ${SAVE_PATH} --n-experts ${EXPERTS} --model-parallel ${MP}
Then chat with DeepSeek model at will!
torchrun --nproc-per-node ${MP} generate.py --ckpt-path ${SAVE_PATH} --config ${CONFIG} --interactive --temperature {T}
Or batch inference from file.
torchrun --nproc-per-node ${MP} generate.py --ckpt-path ${SAVE_PATH} --config ${CONFIG} --input-file ${FILE}
Or multi nodes inference.
torchrun --nnodes ${NODES} --nproc-per-node $((MP / NODES)) --node-rank $RANK --master-addr $ADDR generate.py --ckpt-path ${SAVE_PATH} --config ${CONFIG} --input-file ${FILE}
- Total size
- 16.9 MB
- Files
- 16
- Last updated
- May 25
- Pre-warmed CDN
- US EU US EU