Update README.md

README.md

@@ -172,50 +172,65 @@ print(completion.choices[0].message.content)
 
 ## Deployment
 
-### vLLM
-
-vLLM supports offline batched inference or launching an OpenAI-Compatible API Service for online inference.
+### SGLang
 
 #### Environment Preparation
 
-```bash
-pip install vllm==0.11.0
+We will later submit our model to the SGLang official release. Now we can prepare the environment by following these steps:
+```shell
+pip3 install -U sglang sgl-kernel
 ```
 
-#### Offline Inference:
+#### Run Inference
+
+Both BF16 and FP8 models are supported by SGLang now. It depends on the dtype of the model in ${MODEL_PATH}.
 
-```python
-from transformers import AutoTokenizer
-from vllm import LLM, SamplingParams
+Here is the example to run Ling-1T with multiple GPU nodes, where the master node IP is ${MASTER_IP} and server port is ${PORT}:
 
-tokenizer = AutoTokenizer.from_pretrained("inclusionAI/Ling-1T")
+- Start server:
+```bash
+# Node 0:
+python -m sglang.launch_server --model-path $MODEL_PATH --tp-size 8 --pp-size 4 --dp-size 1 --trust-remote-code --dist-init-addr $MASTER_IP:2345 --port $PORT --nnodes 4 --node-rank 0 
 
-sampling_params = SamplingParams(temperature=0.7, top_p=0.8, repetition_penalty=1.05, max_tokens=16384)
+# Node 1:
+python -m sglang.launch_server --model-path $MODEL_PATH --tp-size 8 --pp-size 4 --dp-size 1 --trust-remote-code --dist-init-addr $MASTER_IP:2345 --port $PORT --nnodes 4 --node-rank 1 
 
-llm = LLM(model="inclusionAI/Ling-1T", dtype='bfloat16', trust_remote_code=True)
-prompt = "Give me a short introduction to large language models."
-messages = [
-    {"role": "system", "content": "You are Ling, an assistant created by inclusionAI"},
-    {"role": "user", "content": prompt}
-]
+# Node 2:
+python -m sglang.launch_server --model-path $MODEL_PATH --tp-size 8 --pp-size 4 --dp-size 1 --trust-remote-code --dist-init-addr $MASTER_IP:2345 --port $PORT --nnodes 4 --node-rank 2 
 
-text = tokenizer.apply_chat_template(
-    messages,
-    tokenize=False,
-    add_generation_prompt=True
-)
-outputs = llm.generate([text], sampling_params)
+# Node 3:
+python -m sglang.launch_server --model-path $MODEL_PATH --tp-size 8 --pp-size 4 --dp-size 1 --trust-remote-code --dist-init-addr $MASTER_IP:2345 --port $PORT --nnodes 4 --node-rank 3
+
+# This is only an example. Please adjust arguments according to your actual environment.
+```
 
+- Client:
+
+```shell
+curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{"model": "auto", "messages": [{"role": "user", "content": "What is the capital of France?"}]}'
+```
+
+More usage can be found [here](https://docs.sglang.ai/basic_usage/send_request.html)
+
+### vLLM
+
+#### Environment Preparation
+
+```bash
+pip install vllm==0.11.0
 ```
 
-#### Online Inference:
+#### Run Inference:
+
+Here is the example to deploy the model with multiple GPU nodes, where the master node IP is ${MASTER_IP}, server port is ${PORT} and the path of model is ${MODEL_PATH}:
 
 ```bash
-vllm serve inclusionAI/Ling-1T \
-              --tensor-parallel-size 32 \
-              --pipeline-parallel-size 1 \
-              --trust-remote-code \
-              --gpu-memory-utilization 0.90
+# step 1. start ray on all nodes
+
+# step 2. start vllm server only on node 0:
+vllm serve $MODEL_PATH --port $PORT --served-model-name my_model --trust-remote-code --tensor-parallel-size 8 --pipeline-parallel-size 4 --gpu-memory-utilization 0.85
 
 # This is only an example, please adjust arguments according to your actual environment.
 ```
@@ -236,51 +251,6 @@ To handle long context in vLLM using YaRN, we need to follow these two steps:
 
 For detailed guidance, please refer to the vLLM [`instructions`](https://docs.vllm.ai/en/latest/).
 
-### SGLang
-
-#### Environment Preparation
-
-We will later submit our model to SGLang official release, now we can prepare the environment following steps:
-```shell
-pip3 install -U sglang sgl-kernel
-```
-
-#### Run Inference
-
-BF16 and FP8 models are supported by SGLang now, it depends on the dtype of the model in ${MODEL_PATH}.
-
-Here is the example to run Ling-1T with multiple nodes, with master node IP is ${MASTER_IP} and port is ${PORT} :
-
-- Start server:
-```bash
-# Node 0:
-python -m sglang.launch_server --model-path $MODEL_PATH --tp-size 8 --pp-size 4 --dp-size 1 --trust-remote-code --dist-init-addr $MASTER_IP:$PORT --nnodes 4 --node-rank 0 
-
-# Node 1:
-python -m sglang.launch_server --model-path $MODEL_PATH --tp-size 8 --pp-size 4 --dp-size 1 --trust-remote-code --dist-init-addr $MASTER_IP:$PORT --nnodes 4 --node-rank 1 
-
-# Node 2:
-python -m sglang.launch_server --model-path $MODEL_PATH --tp-size 8 --pp-size 4 --dp-size 1 --trust-remote-code --dist-init-addr $MASTER_IP:$PORT --nnodes 4 --node-rank 2 
-
-# Node 3:
-python -m sglang.launch_server --model-path $MODEL_PATH --tp-size 8 --pp-size 4 --dp-size 1 --trust-remote-code --dist-init-addr $MASTER_IP:$PORT --nnodes 4 --node-rank 3
-
-# This is only an example, please adjust arguments according to your actual environment.
-```
-
-MTP is supported for base model, and not yet for chat model. You can add parameter `--speculative-algorithm NEXTN`
-to start command.
-
-- Client:
-
-```shell
-curl -s http://${MASTER_IP}:${PORT}/v1/chat/completions \
-  -H "Content-Type: application/json" \
-  -d '{"model": "auto", "messages": [{"role": "user", "content": "What is the capital of France?"}]}'
-```
-
-More usage can be found [here](https://docs.sglang.ai/basic_usage/send_request.html)
-
 
 
 ## Limitations & Future Plans