Update README.md

README.md

@@ -209,96 +209,67 @@ completion = client.chat.completions.create(
 print(completion.choices[0].message.content)
 ```
 
-### 🤗 Hugging Face Transformers
-
-Here is a code snippet to show you how to use the chat model with `transformers`:
-
-```python
-from transformers import AutoModelForCausalLM, AutoTokenizer
-
-model_name = "inclusionAI/Ling-1T"
+## Deployment
 
-model = AutoModelForCausalLM.from_pretrained(
-    model_name,
-    dtype="auto",
-    device_map="auto",
-    trust_remote_code=True,
-)
-tokenizer = AutoTokenizer.from_pretrained(model_name)
-
-prompt = "Give me a short introduction to large language models."
-messages = [
-    {"role": "system", "content": "You are Ling, an assistant created by inclusionAI"},
-    {"role": "user", "content": prompt}
-]
-text = tokenizer.apply_chat_template(
-    messages,
-    tokenize=False,
-    add_generation_prompt=True
-)
-model_inputs = tokenizer([text], return_tensors="pt", return_token_type_ids=False).to(model.device)
+### SGLang
 
-generated_ids = model.generate(
-    **model_inputs,
-    max_new_tokens=512
-)
-generated_ids = [
-    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
-]
+#### Environment Preparation
 
-response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
+We will later submit our model to the SGLang official release. Now we can prepare the environment by following these steps:
+```shell
+pip3 install -U sglang sgl-kernel
 ```
 
-### 🤖 ModelScope
+#### Run Inference
 
-If you're in mainland China, we strongly recommend you to use our model from 🤖 <a href="https://modelscope.cn/models/inclusionAI/Ling-1T">ModelScope</a>.
+Both BF16 and FP8 models are supported by SGLang now. It depends on the dtype of the model in ${MODEL_PATH}.
 
-## Deployment
+Here is the example to run Ling-1T with multiple GPU nodes, where the master node IP is ${MASTER_IP} and server port is ${PORT}:
 
-### vLLM
+- Start server:
+```bash
+# Node 0:
+python -m sglang.launch_server --model-path $MODEL_PATH --tp-size 8 --pp-size 4 --dp-size 1 --trust-remote-code --dist-init-addr $MASTER_IP:2345 --port $PORT --nnodes 4 --node-rank 0 
 
-vLLM supports offline batched inference or launching an OpenAI-Compatible API Service for online inference.
+# Node 1:
+python -m sglang.launch_server --model-path $MODEL_PATH --tp-size 8 --pp-size 4 --dp-size 1 --trust-remote-code --dist-init-addr $MASTER_IP:2345 --port $PORT --nnodes 4 --node-rank 1 
 
-#### Environment Preparation
+# Node 2:
+python -m sglang.launch_server --model-path $MODEL_PATH --tp-size 8 --pp-size 4 --dp-size 1 --trust-remote-code --dist-init-addr $MASTER_IP:2345 --port $PORT --nnodes 4 --node-rank 2 
 
-```bash
-pip install vllm==0.11.0
-```
+# Node 3:
+python -m sglang.launch_server --model-path $MODEL_PATH --tp-size 8 --pp-size 4 --dp-size 1 --trust-remote-code --dist-init-addr $MASTER_IP:2345 --port $PORT --nnodes 4 --node-rank 3
 
-#### Offline Inference:
+# This is only an example. Please adjust arguments according to your actual environment.
+```
 
-```python
-from transformers import AutoTokenizer
-from vllm import LLM, SamplingParams
+- Client:
 
-tokenizer = AutoTokenizer.from_pretrained("inclusionAI/Ling-1T")
+```shell
+curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{"model": "auto", "messages": [{"role": "user", "content": "What is the capital of France?"}]}'
+```
 
-sampling_params = SamplingParams(temperature=0.7, top_p=0.8, repetition_penalty=1.05, max_tokens=16384)
+More usage can be found [here](https://docs.sglang.ai/basic_usage/send_request.html)
 
-llm = LLM(model="inclusionAI/Ling-1T", dtype='bfloat16', trust_remote_code=True)
-prompt = "Give me a short introduction to large language models."
-messages = [
-    {"role": "system", "content": "You are Ling, an assistant created by inclusionAI"},
-    {"role": "user", "content": prompt}
-]
+### vLLM
 
-text = tokenizer.apply_chat_template(
-    messages,
-    tokenize=False,
-    add_generation_prompt=True
-)
-outputs = llm.generate([text], sampling_params)
+#### Environment Preparation
 
+```bash
+pip install vllm==0.11.0
 ```
 
-#### Online Inference:
+#### Run Inference:
+
+Here is the example to deploy the model with multiple GPU nodes, where the master node IP is ${MASTER_IP}, server port is ${PORT} and the path of model is ${MODEL_PATH}:
 
 ```bash
-vllm serve inclusionAI/Ling-1T \
-              --tensor-parallel-size 32 \
-              --pipeline-parallel-size 1 \
-              --trust-remote-code \
-              --gpu-memory-utilization 0.90
+# step 1. start ray on all nodes
+
+# step 2. start vllm server only on node 0:
+vllm serve $MODEL_PATH --port $PORT --served-model-name my_model --trust-remote-code --tensor-parallel-size 8 --pipeline-parallel-size 4 --gpu-memory-utilization 0.85
 
 # This is only an example, please adjust arguments according to your actual environment.
 ```
@@ -320,52 +291,6 @@ To handle long context in vLLM using YaRN, we need to follow these two steps:
 For detailed guidance, please refer to the vLLM [`instructions`](https://docs.vllm.ai/en/latest/).
 
 
-### SGLang
-
-#### Environment Preparation
-
-We will later submit our model to SGLang official release, now we can prepare the environment following steps:
-```shell
-pip3 install sglang==0.5.2rc0 sgl-kernel==0.3.7.post1
-```
-You can use docker image as well:
-```shell
-docker pull lmsysorg/sglang:v0.5.2rc0-cu126
-```
-Then you should apply patch to sglang installation:
-```bash
-# patch command is needed, run `yum install -y patch` if needed
-patch -d `python -c 'import sglang;import os; print(os.path.dirname(sglang.__file__))'` -p3 < inference/sglang/bailing_moe_v2.patch
-```
-
-#### Run Inference
-
-BF16 and FP8 models are supported by SGLang now, it depends on the dtype of the model in ${MODEL_PATH}. They both share the same command in the following:  
-
-- Start server:
-```bash
-python -m sglang.launch_server \
-    --model-path $MODEL_PATH \
-    --host 0.0.0.0 --port $PORT \
-    --trust-remote-code \
-    --attention-backend fa3
-
-# This is only an example, please adjust arguments according to your actual environment.
-```
-MTP is supported for base model, and not yet for chat model. You can add parameter `--speculative-algorithm NEXTN`
-to start command.
-
-- Client:
-
-```shell
-curl -s http://localhost:${PORT}/v1/chat/completions \
-  -H "Content-Type: application/json" \
-  -d '{"model": "auto", "messages": [{"role": "user", "content": "What is the capital of France?"}]}'
-```
-
-More usage can be found [here](https://docs.sglang.ai/basic_usage/send_request.html)
-
-
 
 ## Limitations & Future Plans