Update README.md
Browse files
README.md
CHANGED
|
@@ -79,23 +79,9 @@ response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
|
|
| 79 |
print(response)
|
| 80 |
```
|
| 81 |
|
| 82 |
-
### vLLM Deployment
|
| 83 |
-
|
| 84 |
-
```bash
|
| 85 |
-
vllm serve trillionlabs/Tri-21B-Think-Preview \
|
| 86 |
-
--dtype bfloat16 \
|
| 87 |
-
--max-model-len 32768 \
|
| 88 |
-
--tensor-parallel-size 8 \
|
| 89 |
-
--reasoning-parser qwen3 \
|
| 90 |
-
--enable-auto-tool-choice \
|
| 91 |
-
--tool-call-parser hermes
|
| 92 |
-
```
|
| 93 |
-
|
| 94 |
-
### SGLang Deployment
|
| 95 |
|
| 96 |
-
|
| 97 |
-
python3 -m sglang.launch_server --model-path trillionlabs/Tri-21B-Think-Preview --dtype bfloat16 --context-length 32768
|
| 98 |
-
```
|
| 99 |
|
| 100 |
|
| 101 |
## Fine-tuning Notes
|
|
|
|
| 79 |
print(response)
|
| 80 |
```
|
| 81 |
|
| 82 |
+
### vLLM & SGLang Deployment
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 83 |
|
| 84 |
+
vLLM and SGLang support for Trillion Model is on the way. Stay tuned!
|
|
|
|
|
|
|
| 85 |
|
| 86 |
|
| 87 |
## Fine-tuning Notes
|