update README.md

README.md

@@ -18,7 +18,7 @@ tags:
 
 ## Introduction
 
-Qwen2.5-Coder is the latest series of Code-Specific Qwen large language models (formerly known as CodeQwen). As of now, Qwen2.5-Coder has covered six mainstream model sizes, 0.5, 1.5, 3, 7, 14, 32 billion parameters, to meet the needs of different developers. All of these models follows the Apache License (except for the 3B);  Qwen2.5-Coder brings the following improvements upon CodeQwen1.5:
+Qwen2.5-Coder is the latest series of Code-Specific Qwen large language models (formerly known as CodeQwen). As of now, Qwen2.5-Coder has covered six mainstream model sizes, 0.5, 1.5, 3, 7, 14, 32 billion parameters, to meet the needs of different developers. Qwen2.5-Coder brings the following improvements upon CodeQwen1.5:
 
 - Significantly improvements in **code generation**, **code reasoning** and **code fixing**. Base on the strong Qwen2.5, we scale up the training tokens into 5.5 trillion including source code, text-code grounding, Synthetic data, etc. Qwen2.5-Coder-32B has become the current state-of-the-art open-source codeLLM, with its coding abilities matching those of GPT-4o.
 - A more comprehensive foundation for real-world applications such as **Code Agents**. Not only enhancing coding capabilities but also maintaining its strengths in mathematics and general competencies.
@@ -32,7 +32,6 @@ Qwen2.5-Coder is the latest series of Code-Specific Qwen large language models (
 - Number of Layers: 28
 - Number of Attention Heads (GQA): 12 for Q and 2 for KV
 - Context Length: Full 32,768 tokens
-  - Please refer to [this section](#processing-long-texts) for detailed instructions on how to deploy Qwen2.5 for handling long texts.
   
 **We do not recommend using base language models for conversations.** Instead, you can apply post-training, e.g., SFT, RLHF, continued pretraining, etc., or fill in the middle tasks on this model.
 
@@ -47,27 +46,6 @@ With `transformers<4.37.0`, you will encounter the following error:
 KeyError: 'qwen2'
 ```
 
-### Processing Long Texts
-
-The current `config.json` is set for context length up to 32,768 tokens.
-To handle extensive inputs exceeding 32,768 tokens, we utilize [YaRN](https://arxiv.org/abs/2309.00071), a technique for enhancing model length extrapolation, ensuring optimal performance on lengthy texts.
-
-For supported frameworks, you could add the following to `config.json` to enable YaRN:
-```json
-{
-  ...,
-  "rope_scaling": {
-    "factor": 4.0,
-    "original_max_position_embeddings": 32768,
-    "type": "yarn"
-  }
-}
-```
-
-For deployment, we recommend using vLLM. 
-Please refer to our [Documentation](https://qwen.readthedocs.io/en/latest/deployment/vllm.html) for usage if you are not familar with vLLM.
-Presently, vLLM only supports static YARN, which means the scaling factor remains constant regardless of input length, **potentially impacting performance on shorter texts**. 
-We advise adding the `rope_scaling` configuration only when processing long contexts is required.
 
 ## Evaluation & Performance
 
@@ -81,10 +59,10 @@ If you find our work helpful, feel free to give us a cite.
 
 ```
 @article{hui2024qwen2,
-  title={Qwen2. 5-Coder Technical Report},
-  author={Hui, Binyuan and Yang, Jian and Cui, Zeyu and Yang, Jiaxi and Liu, Dayiheng and Zhang, Lei and Liu, Tianyu and Zhang, Jiajun and Yu, Bowen and Dang, Kai and others},
-  journal={arXiv preprint arXiv:2409.12186},
-  year={2024}
+      title={Qwen2. 5-Coder Technical Report},
+      author={Hui, Binyuan and Yang, Jian and Cui, Zeyu and Yang, Jiaxi and Liu, Dayiheng and Zhang, Lei and Liu, Tianyu and Zhang, Jiajun and Yu, Bowen and Dang, Kai and others},
+      journal={arXiv preprint arXiv:2409.12186},
+      year={2024}
 }
 @article{qwen2,
       title={Qwen2 Technical Report}, 

config.json

@@ -17,7 +17,7 @@
   "num_key_value_heads": 2,
   "rms_norm_eps": 1e-06,
   "rope_theta": 1000000.0,
-  "sliding_window": 131072,
+  "sliding_window": 32768,
   "tie_word_embeddings": true,
   "torch_dtype": "bfloat16",
   "transformers_version": "4.44.0",