Update README.md

README.md

@@ -1,3 +1,232 @@
----
-license: mit
----
+---
+license: mit
+pipeline_tag: text-generation
+library_name: transformers
+---
+
+
+
+<p align="center">
+    <img src="https://mdn.alipayobjects.com/huamei_qa8qxu/afts/img/A*4QxcQrBlTiAAAAAAQXAAAAgAemJ7AQ/original" width="100"/>
+<p>
+
+<p align="center">🤗 <a href="https://huggingface.co/inclusionAI">Hugging Face</a>&nbsp&nbsp | &nbsp&nbsp🤖 <a href="https://modelscope.cn/organization/inclusionAI">ModelScope </a>&nbsp&nbsp | &nbsp&nbsp🐙 <a href="https://zenmux.ai/inclusionai/ling-1t?utm_source=hf_inclusionAI">Experience Now</a></p>
+
+
+## Introduction
+
+
+
+## Model Downloads
+
+You can download Ling-1T model from the following table. If you are located in mainland China, we also provide the model on ModelScope.cn to speed up the download process.
+
+<center>
+
+|       **Model**        | **Context Length** |                                                                             **Download**                                                                             |
+|:----------------------:| :----------------: |:--------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
+|     Ling-1T      |        32K -> 128K (YaRN)        |          [🤗 HuggingFace](https://huggingface.co/inclusionAI/Ling-1T) &nbsp;&nbsp; [🤖 ModelScope](https://www.modelscope.cn/models/inclusionAI/Ling-1T)          |
+
+</center>
+
+Note: If you are interested in previous version, please visit the past model collections in [Huggingface](https://huggingface.co/inclusionAI) or [ModelScope](https://modelscope.cn/organization/inclusionAI).
+
+
+## Quickstart
+
+### 🚀 Try Online
+
+You can experience Ling-1T online at: [ZenMux](https://zenmux.ai/inclusionai/ling-1t?utm_source=hf_inclusionAI)
+
+### 🔌 API Usage
+
+You can also use Ling-1T through API calls:
+
+```python   
+from openai import OpenAI
+
+# 1. Initialize the OpenAI client
+client = OpenAI(
+    # 2. Point the base URL to the ZenMux endpoint
+    base_url="https://zenmux.ai/api/v1",
+    # 3. Replace with the API Key from your ZenMux user console
+    api_key="<your ZENMUX_API_KEY>",
+)
+
+# 4. Make a request
+completion = client.chat.completions.create(
+    # 5. Specify the model to use in the format "provider/model-name"
+    model="inclusionai/ling-1t",
+    messages=[
+        {
+            "role": "user",
+            "content": "What is the meaning of life?"
+        }
+    ]
+)
+
+print(completion.choices[0].message.content)
+```
+
+### 🤗 Hugging Face Transformers
+
+Here is a code snippet to show you how to use the chat model with `transformers`:
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_name = "inclusionAI/Ling-1T"
+
+model = AutoModelForCausalLM.from_pretrained(
+    model_name,
+    dtype="auto",
+    device_map="auto",
+    trust_remote_code=True,
+)
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+
+prompt = "Give me a short introduction to large language models."
+messages = [
+    {"role": "system", "content": "You are Ling, an assistant created by inclusionAI"},
+    {"role": "user", "content": prompt}
+]
+text = tokenizer.apply_chat_template(
+    messages,
+    tokenize=False,
+    add_generation_prompt=True
+)
+model_inputs = tokenizer([text], return_tensors="pt", return_token_type_ids=False).to(model.device)
+
+generated_ids = model.generate(
+    **model_inputs,
+    max_new_tokens=512
+)
+generated_ids = [
+    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
+]
+
+response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
+```
+
+### 🤖 ModelScope
+
+If you're in mainland China, we strongly recommend you to use our model from 🤖 <a href="https://modelscope.cn/organization/inclusionAI">ModelScope</a>.
+
+## Deployment
+
+### vLLM
+
+vLLM supports offline batched inference or launching an OpenAI-Compatible API Service for online inference.
+
+#### Environment Preparation
+
+```bash
+pip install vllm==0.11.0
+```
+
+#### Offline Inference:
+
+```python
+from transformers import AutoTokenizer
+from vllm import LLM, SamplingParams
+
+tokenizer = AutoTokenizer.from_pretrained("inclusionAI/Ling-1T")
+
+sampling_params = SamplingParams(temperature=0.7, top_p=0.8, repetition_penalty=1.05, max_tokens=16384)
+
+llm = LLM(model="inclusionAI/Ling-1T", dtype='bfloat16', trust_remote_code=True)
+prompt = "Give me a short introduction to large language models."
+messages = [
+    {"role": "system", "content": "You are Ling, an assistant created by inclusionAI"},
+    {"role": "user", "content": prompt}
+]
+
+text = tokenizer.apply_chat_template(
+    messages,
+    tokenize=False,
+    add_generation_prompt=True
+)
+outputs = llm.generate([text], sampling_params)
+
+```
+
+#### Online Inference:
+
+```bash
+vllm serve inclusionAI/Ling-1T \
+              --tensor-parallel-size 2 \
+              --pipeline-parallel-size 1 \
+              --trust-remote-code \
+              --gpu-memory-utilization 0.90
+
+# This is only an example, please adjust the model sharding strategy according to your actual environment.
+```
+
+To handle long context in vLLM using YaRN, we need to follow these two steps:
+1. Add a `rope_scaling` field to the model's `config.json` file, for example:
+```json
+{
+  ...,
+  "rope_scaling": {
+    "factor": 4.0,
+    "original_max_position_embeddings": 32768,
+    "type": "yarn"
+  }
+}
+```
+2. Use an additional parameter `--max-model-len` to specify the desired maximum context length when starting the vLLM service.
+
+For detailed guidance, please refer to the vLLM [`instructions`](https://docs.vllm.ai/en/latest/).
+
+
+### SGLang
+
+#### Environment Preparation
+
+We will later submit our model to SGLang official release, now we can prepare the environment following steps:
+```shell
+pip3 install sglang==0.5.2rc0 sgl-kernel==0.3.7.post1
+```
+You can use docker image as well:
+```shell
+docker pull lmsysorg/sglang:v0.5.2rc0-cu126
+```
+Then you should apply patch to sglang installation:
+```bash
+# patch command is needed, run `yum install -y patch` if needed
+patch -d `python -c 'import sglang;import os; print(os.path.dirname(sglang.__file__))'` -p3 < inference/sglang/bailing_moe_v2.patch
+```
+
+#### Run Inference
+
+BF16 and FP8 models are supported by SGLang now, it depends on the dtype of the model in ${MODEL_PATH}. They both share the same command in the following:  
+
+- Start server:
+```bash
+python -m sglang.launch_server \
+    --model-path $MODLE_PATH \
+    --host 0.0.0.0 --port $PORT \
+    --trust-remote-code \
+    --attention-backend fa3
+
+# This is only an example, please adjust the model sharding strategy according to your actual environment.
+```
+MTP is supported for base model, and not yet for chat model. You can add parameter `--speculative-algorithm NEXTN`
+to start command.
+
+- Client:
+
+```shell
+curl -s http://localhost:${PORT}/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{"model": "auto", "messages": [{"role": "user", "content": "What is the capital of France?"}]}'
+```
+
+More usage can be found [here](https://docs.sglang.ai/basic_usage/send_request.html)
+
+
+
+## License
+
+This code repository is licensed under [the MIT License](https://github.com/inclusionAI/Ling-V2/blob/main/LICENSE).
+