Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,96 @@
+# Upstage Solar License
+
+# Preamble
+
+The 'Upstage Solar License' (hereinafter referred to as "this License") was established by Upstage Co., Ltd., incorporated under the laws of the Republic of Korea, to encourage the development of open-source software using Solar AI models, and is not affiliated with the Apache Software Foundation.
+
+This License basically adopts all provisions of the Apache License, Version 2.0 (hereinafter referred to as "Apache License 2.0"), including the principle of allowing commercial use, but prescribes minimum strategic conditions for the global expansion of AI technology and the sustainable development of the AI ecosystem.
+
+The key additional condition, as specified in Section 4(e), is that if you distribute a "Derivative AI Model" based on the "Work", you must specify the 'Solar' brand. This applies as an exception to Section 6 (Trademarks) of the Apache License 2.0.
+
+# TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1\. Definitions.
+
+"License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document.
+
+"Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License.
+
+"Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
+
+"You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License.
+
+"Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files.
+
+"Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
+
+"Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below).
+
+"Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.
+
+"Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution."
+
+"Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.
+
+2\. Grant of Copyright License.  
+Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
+
+3\. Grant of Patent License.  
+Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.
+
+4\. Redistribution.  
+You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:
+
+(a) You must give any other recipients of the Work or Derivative Works a copy of this License; and  
+(b) You must cause any modified files to carry prominent notices stating that You changed the files; and  
+(c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and  
+(d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.  
+(e) If You distribute or make available a Derivative Work that is an artificial intelligence model created, trained, fine-tuned, or otherwise improved using the Work (the "Derivative AI Model"), You must adhere to the following conditions:  
+(i) The name of such Derivative AI Model must begin with "Solar" (e.g., "Solar-MyModel-v1"); and  
+(ii) You must prominently display the phrase "Built with Solar" in any related websites, user interfaces, or documentation associated with the Derivative AI Model; and  
+(iii) You must provide a copy of this License, including the original copyright notice and NOTICE file as included in the distribution of the Works, alongside the Derivative AI Model.
+
+You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License.
+
+5\. Submission of Contributions.  
+Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.
+
+6\. Trademarks.  
+This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work, reproducing the content of the NOTICE file, and as explicitly required for attribution in Section 4(e) of this License.  
+Any use of the “Solar” name under this License must not imply any sponsorship, endorsement, certification, or official relationship with the Licensor, nor mislead users into believing that a Derivative AI Model is an official product of the Licensor.
+
+7\. Disclaimer of Warranty.  
+Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
+
+8\. Limitation of Liability.  
+In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.
+
+9\. Accepting Warranty or Additional Liability.  
+While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.
+
+END OF TERMS AND CONDITIONS
+
+# APPENDIX: How to apply the Upstage Solar License to your work.
+
+To apply the Upstage Solar License to your work, attach the following boilerplate notice, with the fields enclosed by brackets replaced with your own identifying information. (Don't include the brackets\!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives.
+
+Copyright \[yyyy\] \[Upstage AI (or other copyright owner)\]
+
+Licensed under the Upstage Solar License (the "License");  
+you may not use this file except in compliance with the License.  
+You may obtain a copy of the License at
+
+   https://huggingface.co/Upstage/Solar-Open-100B/blob/main/LICENSE
+
+Unless required by applicable law or agreed to in writing, software  
+distributed under the License is distributed on an "AS IS" BASIS,  
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  
+See the License for the specific language governing permissions and  
+limitations under the License.
+
+This work is based on or derived from materials licensed under the  
+Upstage Solar License.  
+If you distribute or make available a Derivative AI Model (as defined  
+in Section 4(e) of the License), your model name must begin with "Solar"  
+and you must prominently display "Built with Solar" in associated  
+documentation or interfaces.  

README.md

@@ -1,5 +1,161 @@
----
-license: other
-license_name: upstage-solar-license
-license_link: LICENSE
----
+---
+language:
+- en
+- ko
+library_name: transformers
+license: other
+license_name: upstage-solar-license
+pipeline_tag: text-generation
+tags:
+- upstage
+- solar
+- moe
+- 100b
+- llm
+- nota
+- quantization
+---
+
+# **Solar-Open-100B-NotaMoeQuant-Int4**
+
+This repository provides **Upstage’s flagship model, [Solar-Open-100B](https://huggingface.co/upstage/Solar-Open-100B)**, packaged with [**Nota AI**](https://www.nota.ai/)’s proprietary quantization technique specifically developed for **Mixture-of-Experts (MoE)-based LLMs**. Unlike conventional quantization methods, this approach incorporates a **novel method** designed to mitigate **representation distortion** that can occur when experts are mixed under quantization in MoE architectures.
+
+## Overview
+
+- **Base model:** [Solar-Open-100B](https://huggingface.co/upstage/Solar-Open-100B) 
+- **Quantization:** Int4 weight-only
+- **Packing format:** `auto_round:auto_gptq` (ensuring backend compatibility with PyTorch and vLLM)
+- **Quantization group size:** 128
+- **Supported tensor parallel sizes:** {1,2}
+- **Hardware Requirements:** 
+    * **Minimum:** 2 x NVIDIA A100 (80GB)    
+
+## License
+This repository contains both model weights and code,
+which are licensed under different terms:
+
+1. MODEL WEIGHTS (*.safetensors)
+   Licensed under **Upstage Solar License**
+   See: https://huggingface.co/upstage/Solar-Open-100B/blob/main/LICENSE
+
+2. CODE (*.py, *.json, *.jinja files)
+   Licensed under **Apache License 2.0**
+   See: https://www.apache.org/licenses/LICENSE-2.0
+
+
+## Performance
+
+- English
+
+|                 |**Solar-Open-100B**|**Nota MoE Quantization (Ours)**|**AutoRound**|**cyankiwi AWQ**|
+|---              | ---               | ---                            | ---         | ---            |
+|PPL (WikiText-2)↓|6.06               |**6.81**                        |7.12         |30.52           |
+|PPL (C4)↓        |20.37              |**20.84**                       |20.94        |50.16           |
+|PIQA↑            |82.37              |**82.75**                       |82.05        |78.94           |
+|BoolQ↑           |84.89              |84.86                           |**85.29**    |68.87           |
+|ARC-E↑           |87.25              |**86.48**                       |85.77        |83.12           |
+|ARC-C↑           |61.43              |**61.69**                       |60.84        |56.40           |
+|TruthfulQA↑      |59.25              |**60.14**                       |59.18        |52.38           |
+|WinoGrande↑      |76.09              |**75.77**                       |**75.77**    |68.59           | 
+
+- Korean
+
+|                 |**Solar-Open-100B**|**Nota MoE Quantization (Ours)**|**AutoRound**|**cyankiwi AWQ**|
+|---              | ---               | ---                            | ---         | ---            |
+|HRM8K↑           |81.52              |80.68                           |**81.56**    |32.67           |
+|MMLU-ProX-Lite↑  |55.44              |**51.84**                       |51.26        |6.19            |
+|KoBEST↑          |62.00              |**62.80**                       |61.80        |61.80           |
+|CLiCK↑           |71.33              |**70.03**                       |69.77        |51.18           |
+
+- Model weigth memory footprint
+
+|**Solar-Open-100B**|**Nota MoE Quantization (Ours)**|**cyankiwi AWQ**|
+| ---               | ---                            | ---            |
+|191.2 GB           |51.9 GB                         |57.0 GB         |
+
+
+* Note 
+  - ↑ / ↓ denote the direction of improvement: higher is better (↑), lower is better (↓).
+  - Cyankiwi AWQ is a publicly available [INT4 (4-bit AWQ) quantized version of Solar-Open-100B](cyankiwi/Solar-Open-100B-AWQ-4bit)
+  - Because we used a smaller thinking budget, the results for HRM8K and CLiCK are slightly lower than the numbers reported in the original Solar-Open-100B repository.
+  - Memory refers to the pure VRAM footprint occupied only by the model weights.
+
+## Inference
+### Transformers
+
+Install the required dependencies:
+
+```bash
+pip install -U transformers kernels torch accelerate auto-round==0.8.0
+```
+
+Run inference with the following code:
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+MODEL_ID = "nota-ai/Solar-Open-100B-NotaMoEQuant-Int4"
+
+# Load model and tokenizer
+tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
+
+model = AutoModelForCausalLM.from_pretrained(
+    pretrained_model_name_or_path=MODEL_ID,
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+    trust_remote_code=True,
+)
+
+# Prepare input
+messages = [{"role": "user", "content": "who are you?"}]
+inputs = tokenizer.apply_chat_template(
+    messages,
+    tokenize=True,
+    add_generation_prompt=True,
+    return_dict=True,
+    return_tensors="pt",
+)
+inputs = inputs.to(model.device)
+
+# Generate response
+generated_ids = model.generate(
+    **inputs,
+    max_new_tokens=4096,
+    temperature=0.8,
+    top_p=0.95,
+    top_k=50,
+    do_sample=True,
+)
+generated_text = tokenizer.decode(generated_ids[0][inputs.input_ids.shape[1] :])
+print(generated_text)
+```
+
+### vLLM
+Create and activate a Python virtual environment
+```bash
+uv venv --python 3.12 --seed
+source .venv/bin/activate
+```
+
+Install Solar Open's optimized vLLM
+```bash
+VLLM_PRECOMPILED_WHEEL_LOCATION="https://github.com/vllm-project/vllm/releases/download/v0.12.0/vllm-0.12.0-cp38-abi3-manylinux_2_31_x86_64.whl" \
+VLLM_USE_PRECOMPILED=1 \
+uv pip install git+https://github.com/UpstageAI/vllm.git@v0.12.0-solar-open
+```
+
+Start the vLLM server (For 2 GPUs)
+```bash
+PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
+vllm serve nota-ai/Solar-Open-100B-NotaMoEQuant-Int4 \
+    --trust-remote-code \
+    --enable-auto-tool-choice \
+    --tool-call-parser solar_open \
+    --reasoning-parser solar_open \
+    --logits-processors vllm.model_executor.models.parallel_tool_call_logits_processor:ParallelToolCallLogitsProcessor \
+    --logits-processors vllm.model_executor.models.solar_open_logits_processor:SolarOpenTemplateLogitsProcessor \
+    --tensor-parallel-size 2 \
+    --max-num-seqs 64 \
+    --gpu-memory-utilization 0.8
+```

chat_template.jinja

@@ -0,0 +1,156 @@
+{#- ======== Template Parameters ========  #}
+{%- set add_generation_prompt = add_generation_prompt if add_generation_prompt is defined else true %}
+{%- set default_system_prompt = default_system_prompt if default_system_prompt is defined else true %}
+{%- set reasoning_effort = reasoning_effort if reasoning_effort is defined else "high" %}
+{%- set think_render_option = think_render_option if think_render_option is defined else "lastthink" %}
+
+{#- ======== System Block State ========  #}
+{%- set sys_ns = namespace(is_first_block=true) -%}
+
+{#- ======== Find last user message index ========  #}
+{%- set last_user_idx = namespace(value=-1) -%}
+{%- for message in messages -%}
+    {%- if message.role == 'user' -%}
+        {%- set last_user_idx.value = loop.index0 -%}
+    {%- endif -%}
+{%- endfor -%}
+
+{#- ======== System messages renderers ========  #}
+{%- macro render_system_message(user_system_messages) %}
+    {%- if default_system_prompt %}
+        {%- if not sys_ns.is_first_block %}{{- "\n\n" }}{%- endif %}
+        {%- set sys_ns.is_first_block = false %}
+        {{- "## Provider System Prompt\n\nYou are Solar Open 100B, a large language model trained by Upstage AI, a Korean startup. Your knowledge cutoff is 2025-07. The current date is " + strftime_now("%Y-%m-%d") + "." }}
+    {%- endif -%}
+    {%- if user_system_messages %}
+        {%- if not sys_ns.is_first_block %}{{- "\n\n" }}{%- endif %}
+        {%- set sys_ns.is_first_block = false %}
+        {{- "## System Prompt" }}
+        {%- for system_message in user_system_messages %}
+            {{- "\n\n" }}
+            {{- system_message }}
+        {%- endfor %}
+    {%- endif -%}
+{%- endmacro %}
+
+{%- macro render_tool_instruction(tools) %}
+    {%- if not sys_ns.is_first_block %}{{- "\n\n" }}{%- endif %}
+    {%- set sys_ns.is_first_block = false %}
+    {{- "## Tools\n\n### Tool Call Instruction" }}
+    {{- "\nYou may invoke one or more tools to assist with the user's query. Available tools are provided in JSON Schema format: <|tools:begin|><|tool:begin|><tools-json-object><|tool:end|>...<|tools:end|>\n" }}
+    {{- "\n### Available Tools\n" }}
+    {{- "<|tools:begin|>" }}
+    {%- for tool in tools %}
+        {{- "<|tool:begin|>" }}
+        {{- tool.function | tojson }}
+        {{- "<|tool:end|>" }}
+    {%- endfor %}
+    {{- "<|tools:end|>\n" }}
+    {{- "\n### Tool Call Format\n" }}
+    {{- "For each tool call, return a JSON object with the following structure, enclosed within <|tool_call:begin|> and <|tool_call:end|> tags: \n<|tool_call:begin|><tool-call-id><|tool_call:name|><tool-name><|tool_call:args|><args-json-object><|tool_call:end|>\n" }}
+    {{- "- The <tool-call-id> must be a randomly generated string consisting of 10 lowercase letters (a-z) and/or digits (0-9) (e.g., a1b2c3d4e5)\n" }}
+    {{- "\n### Tool Response Format\n" }}
+    {{- "Each tool is responded by `tool` with the following structure:\n<|tool_response:id|><tool-call-id><|tool_response:name|><tool-name><|tool_response:result|><results><|tool_response:end|>\n" }}
+    {{- "- Ensure the <tool-call-id> matches the corresponding tool call" -}}
+{%- endmacro %}
+
+{%- macro render_json_response_format_instruction(response_format) %}
+    {%- if not sys_ns.is_first_block %}{{- "\n\n" }}{%- endif %}
+    {%- set sys_ns.is_first_block = false %}
+    {{- "## Output Format Constraint" }}
+    {{- "\n\nYour final response should follow the JSON schema: \n[Start of schema]" }}
+    {{- response_format }}
+    {{- "\n[End of schema]\nPlease ensure your answers adhere to this format and do not contain any unnecessary text." }}
+{%- endmacro %}
+
+{%- macro get_tool_name(messages, tool_call_id) %}
+    {%- for msg in messages -%}
+        {%- if msg.role == 'assistant' and msg.tool_calls -%}
+            {%- for tool_call in msg.tool_calls -%}
+                {%- if tool_call.id == tool_call_id -%}
+                    {{- tool_call.function.name }}
+                {%- endif -%}
+            {%- endfor -%}
+        {%- endif -%}
+    {%- endfor -%}
+{%- endmacro %}
+
+{%- macro render_tool_arguments(tool_arguments) %}
+    {%- if tool_arguments is mapping -%}
+        {{- tool_arguments | tojson }}
+    {%- else -%}
+        {{- tool_arguments }}
+    {%- endif -%}
+{%- endmacro %}
+
+{#- ======== Render system message ========  #}
+{%- set ns = namespace(system_messages=[]) -%}
+{%- for message in messages -%}
+    {%- if message.role == 'system' -%}
+        {%- set ns.system_messages = ns.system_messages + [message.content] -%}
+    {%- endif -%}
+{%- endfor -%}
+
+{%- if ns.system_messages or default_system_prompt or tools or response_format -%}
+    {{- "<|begin|>system<|content|>" }}
+        {{- render_system_message(ns.system_messages) }}
+        {%- if tools -%}
+            {{- render_tool_instruction(tools) }}
+        {%- endif %}
+        {%- if response_format -%}
+            {{- render_json_response_format_instruction(response_format) }}
+        {%- endif %}
+    {{- "<|end|>" }}
+{%- endif -%}
+
+{#- ======== Render main messages ========  #}
+{%- for message in messages -%}
+    {%- if message.role == 'user' -%}
+         {{- "<|begin|>user<|content|>" + message.content + "<|end|>" }}
+    {%- elif message.role == 'tool' -%}
+        {%- set prev_is_tool = loop.index0 > 0 and messages[loop.index0 - 1].role == 'tool' -%}
+        {%- set next_is_tool = loop.index0 < (messages | length - 1) and messages[loop.index0 + 1].role == 'tool' -%}
+        {%- if not prev_is_tool -%}
+            {{- "<|begin|>tool<|tool_response|>" }}
+        {%- endif -%}
+        {{- "<|tool_response:begin|>" + message.tool_call_id + "<|tool_response:name|>" }}
+        {{- get_tool_name(messages, message.tool_call_id) }}
+        {{- "<|tool_response:result|>" }}
+        {{- message.content }}
+        {{- "<|tool_response:end|>" }}
+        {%- if not next_is_tool -%}
+            {{- "<|end|>" }}
+        {%- endif -%}
+    {%- elif message.role == 'assistant' -%}
+        {#- ======== Assistant Thinking ========  #}
+        {%- if think_render_option == "all" -%}
+            {%- if message.reasoning -%}
+                {{- "<|begin|>assistant<|think|>" + message.reasoning + "<|end|>" }}
+            {%- endif -%}
+        {%- elif think_render_option == "lastthink" -%}
+            {%- if message.reasoning and loop.index0 > last_user_idx.value -%}
+                {{- "<|begin|>assistant<|think|>" + message.reasoning + "<|end|>" }}
+            {%- endif -%}
+        {%- endif -%}
+
+        {#- ======== Assistant Messages ========  #}
+        {%- if message.tool_calls -%}
+            {{- "<|begin|>assistant<|tool_calls|>" }}
+            {%- for tool_call in message.tool_calls -%}
+                {{- "<|tool_call:begin|>" + tool_call.id +"<|tool_call:name|>" + tool_call.function.name + "<|tool_call:args|>" }}
+                {{- render_tool_arguments(tool_call.function.arguments) }}
+                {{- "<|tool_call:end|>" }}
+            {%- endfor -%}
+            {{- "<|calls|>" }}
+        {%- else -%}
+            {{- "<|begin|>assistant<|content|>" + message.content + "<|end|>" }}
+        {%- endif -%}
+    {%- endif -%}
+{%- endfor -%}
+
+{%- if add_generation_prompt -%}
+    {%- if reasoning_effort in ["low", "minimal"] -%}
+        {{- "<|begin|>assistant<|think|><|end|>" }}
+    {%- endif -%}
+    {{- "<|begin|>assistant" }}
+{%- endif -%}

config.json

@@ -0,0 +1,52 @@
+{
+  "architectures": [
+    "SolarOpenForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_solar_open.SolarOpenConfig",
+    "AutoModel": "modeling_solar_open.SolarOpenModel",
+    "AutoModelForCausalLM": "modeling_solar_open.SolarOpenForCausalLM"
+  },
+  "bos_token_id": 1,
+  "dtype": "bfloat16",
+  "eos_token_id": 2,
+  "first_k_dense_replace": 0,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 10240,
+  "max_position_embeddings": 131072,
+  "model_type": "solar_open",
+  "moe_intermediate_size": 1280,
+  "n_group": 1,
+  "n_routed_experts": 128,
+  "n_shared_experts": 1,
+  "norm_topk_prob": true,
+  "num_attention_heads": 64,
+  "num_experts_per_tok": 8,
+  "num_hidden_layers": 48,
+  "num_key_value_heads": 8,
+  "pad_token_id": 2,
+  "partial_rotary_factor": 1.0,
+  "quantization_config": {
+    "bits": 4,
+    "data_type": "int",
+    "group_size": 128,
+    "packing_format": "auto_round:auto_gptq",
+    "quant_method": "auto-round",
+    "sym": true
+  },
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 1000000,
+  "routed_scaling_factor": 1.0,
+  "tie_word_embeddings": false,
+  "topk_group": 1,
+  "transformers_version": "4.57.3",
+  "use_cache": true,
+  "use_qk_norm": false,
+  "vocab_size": 196608
+}

configuration_solar_open.py

@@ -0,0 +1,242 @@
+# coding=utf-8
+# Copyright 2025 Upstage AI.
+# Copyright 2025 The ZhipuAI Inc. team and HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# This file has been modified by Upstage AI including
+# -  Hyperparameter Adjustments: Modified the model architecture by increasing vocab_size and num_hidden_layers, while decreasing num_attention_heads, intermediate_size, and moe_intermediate_size.
+# RoPE Configuration: Replaced the generic rope_parameters argument with explicit rope_theta and rope_scaling parameters to define Rotary Positional Embeddings settings.
+#
+# Based on code from: https://github.com/huggingface/transformers/blob/main/src/transformers/models/glm4_moe/configuration_glm4_moe.py
+
+from transformers.configuration_utils import PretrainedConfig
+from transformers.modeling_rope_utils import rope_config_validation
+
+
+class SolarOpenConfig(PretrainedConfig):
+    r"""
+    This is the configuration class to store the configuration of a [`SolarOpenModel`]. It is used to instantiate a
+    SolarOpen model according to the specified arguments, defining the model architecture.
+
+    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
+    documentation from [`PretrainedConfig`] for more information.
+
+
+    Args:
+        vocab_size (`int`, *optional*, defaults to 196608):
+            Vocabulary size of the SolarOpen model. Defines the number of different tokens that can be represented by the
+            `inputs_ids` passed when calling [`SolarOpenModel`]
+        hidden_size (`int`, *optional*, defaults to 4096):
+            Dimension of the hidden representations.
+        intermediate_size (`int`, *optional*, defaults to 10240):
+            Dimension of the MLP representations.
+        num_hidden_layers (`int`, *optional*, defaults to 48):
+            Number of hidden layers in the Transformer encoder.
+        num_attention_heads (`int`, *optional*, defaults to 64):
+            Number of attention heads for each attention layer in the Transformer encoder.
+        partial_rotary_factor (`float`, *optional*, defaults to 1.0):
+            The factor of the partial rotary position.
+        num_key_value_heads (`int`, *optional*, defaults to 8):
+            This is the number of key_value heads that should be used to implement Grouped Query Attention. If
+            `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
+            `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When
+            converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
+            by meanpooling all the original heads within that group. For more details, check out [this
+            paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `32`.
+
+        hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
+            The non-linear activation function (function or string) in the decoder.
+        max_position_embeddings (`int`, *optional*, defaults to 131072):
+            The maximum sequence length that this model might ever be used with.
+        initializer_range (`float`, *optional*, defaults to 0.02):
+            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
+        rms_norm_eps (`float`, *optional*, defaults to 1e-05):
+            The epsilon used by the rms normalization layers.
+        use_cache (`bool`, *optional*, defaults to `True`):
+            Whether or not the model should return the last key/values attentions (not used by all models). Only
+            relevant if `config.is_decoder=True`.
+        tie_word_embeddings (`bool`, *optional*, defaults to `False`):
+            Whether the model's input and output word embeddings should be tied.
+        rope_theta (`float`, *optional*, defaults to 1000000.0):
+            The base period of the RoPE embeddings.
+        rope_scaling (`Dict`, *optional*):
+            Dictionary containing the scaling configuration for the RoPE embeddings. NOTE: if you apply new rope type
+            and you expect the model to work on longer `max_position_embeddings`, we recommend you to update this value
+            accordingly.
+            Expected contents:
+                `rope_type` (`str`):
+                    The sub-variant of RoPE to use. Can be one of ['default', 'linear', 'dynamic', 'yarn', 'longrope',
+                    'llama3'], with 'default' being the original RoPE implementation.
+                `factor` (`float`, *optional*):
+                    Used with all rope types except 'default'. The scaling factor to apply to the RoPE embeddings. In
+                    most scaling types, a `factor` of x will enable the model to handle sequences of length x *
+                    original maximum pre-trained length.
+                `original_max_position_embeddings` (`int`, *optional*):
+                    Used with 'dynamic', 'longrope' and 'llama3'. The original max position embeddings used during
+                    pretraining.
+                `attention_factor` (`float`, *optional*):
+                    Used with 'yarn' and 'longrope'. The scaling factor to be applied on the attention
+                    computation. If unspecified, it defaults to value recommended by the implementation, using the
+                    `factor` field to infer the suggested value.
+                `beta_fast` (`float`, *optional*):
+                    Only used with 'yarn'. Parameter to set the boundary for extrapolation (only) in the linear
+                    ramp function. If unspecified, it defaults to 32.
+                `beta_slow` (`float`, *optional*):
+                    Only used with 'yarn'. Parameter to set the boundary for interpolation (only) in the linear
+                    ramp function. If unspecified, it defaults to 1.
+                `short_factor` (`list[float]`, *optional*):
+                    Only used with 'longrope'. The scaling factor to be applied to short contexts (<
+                    `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden
+                    size divided by the number of attention heads divided by 2
+                `long_factor` (`list[float]`, *optional*):
+                    Only used with 'longrope'. The scaling factor to be applied to long contexts (<
+                    `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden
+                    size divided by the number of attention heads divided by 2
+                `low_freq_factor` (`float`, *optional*):
+                    Only used with 'llama3'. Scaling factor applied to low frequency components of the RoPE
+                `high_freq_factor` (`float`, *optional*):
+                    Only used with 'llama3'. Scaling factor applied to high frequency components of the RoPE
+        attention_bias (`bool`, defaults to `False`, *optional*, defaults to `False`):
+            Whether to use a bias in the query, key, value and output projection layers during self-attention.
+        attention_dropout (`float`, *optional*, defaults to 0.0):
+            The dropout ratio for the attention probabilities.
+        moe_intermediate_size (`int`, *optional*, defaults to 1280):
+            Intermediate size of the routed expert.
+        num_experts_per_tok (`int`, *optional*, defaults to 8):
+            number of experts per token.
+        n_shared_experts (`int`, *optional*, defaults to 1):
+            Number of shared experts.
+        n_routed_experts (`int`, *optional*, defaults to 128):
+            Number of routed experts.
+        routed_scaling_factor (`float`, *optional*, defaults to 1.0):
+            Scaling factor or routed experts.
+        n_group (`int`, *optional*, defaults to 1):
+            Number of groups for routed experts.
+        topk_group (`int`, *optional*, defaults to 1):
+            Number of selected groups for each token(for each token, ensuring the selected experts is only within `topk_group` groups).
+        first_k_dense_replace (`int`, *optional*, defaults to 0):
+            Number of dense layers in shallow layers(embed->dense->dense->...->dense->moe->moe...->lm_head).
+                                                            \--k dense layers--/
+        norm_topk_prob (`bool`, *optional*, defaults to `True`):
+            Whether to normalize the topk probabilities.
+        use_qk_norm (`bool`, *optional*, defaults to `False`):
+            Whether to use query-key normalization in the attention
+    ```python
+    >>> from transformers import SolarOpenModel, SolarOpenConfig
+
+    >>> # Initializing a SolarOpen style configuration
+    >>> configuration = SolarOpenConfig()
+
+    >>> # Initializing a model from the SolarOpen style configuration
+    >>> model = SolarOpenModel(configuration)
+
+    >>> # Accessing the model configuration
+    >>> configuration = model.config
+    ```"""
+
+    model_type = "solar_open"
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+    # Default tensor parallel plan for base model `SolarOpen`
+    base_model_tp_plan = {
+        "layers.*.self_attn.q_proj": "colwise",
+        "layers.*.self_attn.k_proj": "colwise",
+        "layers.*.self_attn.v_proj": "colwise",
+        "layers.*.self_attn.o_proj": "rowwise",
+        "layers.*.mlp.experts.*.gate_proj": "colwise",
+        "layers.*.mlp.experts.*.up_proj": "colwise",
+        "layers.*.mlp.experts.*.down_proj": "rowwise",
+        "layers.*.mlp.gate_proj": "colwise",
+        "layers.*.mlp.up_proj": "colwise",
+        "layers.*.mlp.down_proj": "rowwise",
+    }
+    base_model_pp_plan = {
+        "embed_tokens": (["input_ids"], ["inputs_embeds"]),
+        "layers": (["hidden_states", "attention_mask"], ["hidden_states"]),
+        "norm": (["hidden_states"], ["hidden_states"]),
+    }
+
+    def __init__(
+        self,
+        vocab_size=196608,
+        hidden_size=4096,
+        intermediate_size=10240,
+        num_hidden_layers=48,
+        num_attention_heads=64,
+        partial_rotary_factor=1.0,
+        num_key_value_heads=8,
+        hidden_act="silu",
+        max_position_embeddings=131072,
+        initializer_range=0.02,
+        rms_norm_eps=1e-5,
+        use_cache=True,
+        tie_word_embeddings=False,
+        rope_theta=1000000.0,
+        rope_scaling=None,
+        attention_bias=False,
+        attention_dropout=0.0,
+        moe_intermediate_size=1280,
+        num_experts_per_tok=8,
+        n_shared_experts=1,
+        n_routed_experts=128,
+        routed_scaling_factor=1.0,
+        n_group=1,
+        topk_group=1,
+        first_k_dense_replace=0,
+        norm_topk_prob=True,
+        use_qk_norm=False,
+        **kwargs,
+    ):
+        self.vocab_size = vocab_size
+        self.max_position_embeddings = max_position_embeddings
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.partial_rotary_factor = partial_rotary_factor
+
+        self.num_key_value_heads = num_key_value_heads
+        self.hidden_act = hidden_act
+        self.initializer_range = initializer_range
+        self.rms_norm_eps = rms_norm_eps
+        self.use_cache = use_cache
+        self.rope_theta = rope_theta
+        self.rope_scaling = rope_scaling
+        self.attention_bias = attention_bias
+        self.attention_dropout = attention_dropout
+        # Validate the correctness of rotary position embeddings parameters
+        # BC: if there is a 'type' field, move it to 'rope_type'.
+        if self.rope_scaling is not None and "type" in self.rope_scaling:
+            self.rope_scaling["rope_type"] = self.rope_scaling["type"]
+        rope_config_validation(self)
+
+        # MoE arguments
+        self.moe_intermediate_size = moe_intermediate_size
+        self.num_experts_per_tok = num_experts_per_tok
+        self.n_group = n_group
+        self.topk_group = topk_group
+        self.n_shared_experts = n_shared_experts
+        self.n_routed_experts = n_routed_experts
+        self.routed_scaling_factor = routed_scaling_factor
+        self.first_k_dense_replace = first_k_dense_replace
+        self.norm_topk_prob = norm_topk_prob
+        self.use_qk_norm = use_qk_norm
+
+        super().__init__(
+            tie_word_embeddings=tie_word_embeddings,
+            **kwargs,
+        )
+
+
+__all__ = ["SolarOpenConfig"]

generation_config.json

@@ -0,0 +1,14 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "do_sample": true,
+  "eos_token_id": [
+    2,
+    24,
+    25
+  ],
+  "pad_token_id": 2,
+  "temperature": 0.8,
+  "top_p": 0.95,
+  "transformers_version": "4.57.3"
+}

model-00001-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:726e9cc90c9db7d8e5578a7ff1997e4b34856f1d50d7c319c1e4cbbd8e369ce3
+size 4998891080

model-00002-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:24fc10c0db0a943d46ea7fe809240cb04ded39ec49f01d063ca342eb85bdb188
+size 4999349568

model-00003-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:194db66f219568d98fc3600198c9bc745f6d519077394551bd7e83e087511054
+size 4998779896

model-00004-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:734c2b61bd4d2985d4bf4d61c487311abe2694e52053cb1f54dfbe1879af3830
+size 4999354992

model-00005-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:16650572334c932a060823aabb053a759726ef925586b8e71b6a802b1655b50e
+size 4998782640

model-00006-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:90a37535c982473c33acdb3f2708616b72aab381ba5426c8fef5f9ac6aba98d7
+size 4999355160

model-00007-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d39dd29a56e51b1d434034c45da93510b16a4c5df78628c65bdc6084c67878ea
+size 4998782480

model-00008-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9a1b13eedbc43fe8af8d84e4cb7cf7f0bdd4fcde885ce7074ae80839032f24ea
+size 4998782712

model-00009-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:49d0881af4ececfb30cb71b1135441f66153633251c891a545b5dd7610cda705
+size 4999354920

model-00010-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4e48cf559b0230f47a5a46f647603b3694c1c712893d8c11ade4d9678eb89f6e
+size 4998782656

model-00011-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4a084585013b1b3276f086c74a859315c4a8c6fcea9c47d6bcd2d27f3f8d5838
+size 4159221512

model-00012-of-00012.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9742b5e5f3945b0705b4e852072f39d4fd65aa6b3472dbc1a115f0778d160c1b
+size 1610612864

modeling_solar_open.py

@@ -0,0 +1,608 @@
+# coding=utf-8
+# Copyright 2025 Upstage AI.
+# Copyright 2025 The GLM4 & ZhipuAI team and HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# This file has been modified by Upstage AI including:
+# - Hybrid MoE Architecture: Replaced the standard dense structure with a depth-dependent Hybrid MoE, adding `SolarOpenMoE` and `SolarOpenTopkRouter` classes.
+# - RoPE Strategy: Changed the rotary position embedding strategy from GLM4's interleaved rotation to Llama-style block rotation (via modified `rotate_half`).
+# - Normalization Logic: Simplified the layer normalization structure by removing GLM4's extra post-operation norms and adding optional Query-Key Normalization (`use_qk_norm`).
+#
+# Based on code from: https://github.com/huggingface/transformers/blob/main/src/transformers/models/glm4/modeling_glm4.py
+
+from typing import Callable, Optional, Union
+
+import torch
+import torch.nn.functional as F
+from torch import nn
+
+from transformers.activations import ACT2FN
+from transformers.cache_utils import Cache, DynamicCache
+from transformers.generation import GenerationMixin
+from transformers.integrations import use_kernel_forward_from_hub
+from transformers.masking_utils import create_causal_mask
+from transformers.modeling_flash_attention_utils import FlashAttentionKwargs
+from transformers.modeling_layers import GradientCheckpointingLayer
+from transformers.modeling_outputs import BaseModelOutputWithPast, CausalLMOutputWithPast
+from transformers.modeling_rope_utils import ROPE_INIT_FUNCTIONS, dynamic_rope_update
+from transformers.modeling_utils import ALL_ATTENTION_FUNCTIONS, PreTrainedModel
+from transformers.processing_utils import Unpack
+from transformers.utils import TransformersKwargs, auto_docstring, can_return_tuple
+from transformers.utils.deprecation import deprecate_kwarg
+from transformers.utils.generic import check_model_inputs
+from .configuration_solar_open import SolarOpenConfig
+
+
+def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:
+    """
+    This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch,
+    num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)
+    """
+    batch, num_key_value_heads, slen, head_dim = hidden_states.shape
+    if n_rep == 1:
+        return hidden_states
+    hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)
+    return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)
+
+
+def eager_attention_forward(
+    module: nn.Module,
+    query: torch.Tensor,
+    key: torch.Tensor,
+    value: torch.Tensor,
+    attention_mask: Optional[torch.Tensor],
+    scaling: float,
+    dropout: float = 0.0,
+    **kwargs: Unpack[TransformersKwargs],
+):
+    key_states = repeat_kv(key, module.num_key_value_groups)
+    value_states = repeat_kv(value, module.num_key_value_groups)
+
+    attn_weights = torch.matmul(query, key_states.transpose(2, 3)) * scaling
+    if attention_mask is not None:
+        causal_mask = attention_mask[:, :, :, : key_states.shape[-2]]
+        attn_weights = attn_weights + causal_mask
+
+    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
+    attn_weights = nn.functional.dropout(attn_weights, p=dropout, training=module.training)
+    attn_output = torch.matmul(attn_weights, value_states)
+    attn_output = attn_output.transpose(1, 2).contiguous()
+
+    return attn_output, attn_weights
+
+
+def rotate_half(x):
+    """Rotates half the hidden dims of the input."""
+    x1 = x[..., : x.shape[-1] // 2]
+    x2 = x[..., x.shape[-1] // 2 :]
+    return torch.cat((-x2, x1), dim=-1)
+
+
+def apply_rotary_pos_emb(q, k, cos, sin, position_ids=None, unsqueeze_dim=1):
+    """Applies Rotary Position Embedding to the query and key tensors.
+
+    Args:
+        q (`torch.Tensor`): The query tensor.
+        k (`torch.Tensor`): The key tensor.
+        cos (`torch.Tensor`): The cosine part of the rotary embedding.
+        sin (`torch.Tensor`): The sine part of the rotary embedding.
+        position_ids (`torch.Tensor`, *optional*):
+            Deprecated and unused.
+        unsqueeze_dim (`int`, *optional*, defaults to 1):
+            The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and
+            sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note
+            that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and
+            k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes
+            cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have
+            the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2.
+    Returns:
+        `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding.
+    """
+    cos = cos.unsqueeze(unsqueeze_dim)
+    sin = sin.unsqueeze(unsqueeze_dim)
+
+    # Keep half or full tensor for later concatenation
+    rotary_dim = cos.shape[-1]
+    q_rot, q_pass = q[..., :rotary_dim], q[..., rotary_dim:]
+    k_rot, k_pass = k[..., :rotary_dim], k[..., rotary_dim:]
+
+    # Apply rotary embeddings on the first half or full tensor
+    q_embed = (q_rot * cos) + (rotate_half(q_rot) * sin)
+    k_embed = (k_rot * cos) + (rotate_half(k_rot) * sin)
+
+    # Concatenate back to full shape
+    q_embed = torch.cat([q_embed, q_pass], dim=-1)
+    k_embed = torch.cat([k_embed, k_pass], dim=-1)
+    return q_embed, k_embed
+
+
+class SolarOpenAttention(nn.Module):
+    """Multi-headed attention from 'Attention Is All You Need' paper"""
+
+    def __init__(self, config: SolarOpenConfig, layer_idx: Optional[int] = None):
+        super().__init__()
+        self.config = config
+        self.layer_idx = layer_idx
+        self.head_dim = getattr(config, "head_dim", config.hidden_size // config.num_attention_heads)
+        self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
+        self.scaling = self.head_dim**-0.5
+        self.rope_scaling = config.rope_scaling
+        self.attention_dropout = config.attention_dropout
+        self.is_causal = True
+
+        self.q_proj = nn.Linear(
+            config.hidden_size, config.num_attention_heads * self.head_dim, bias=config.attention_bias
+        )
+        self.k_proj = nn.Linear(
+            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
+        )
+        self.v_proj = nn.Linear(
+            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
+        )
+        self.o_proj = nn.Linear(config.num_attention_heads * self.head_dim, config.hidden_size, bias=False)
+        self.use_qk_norm = config.use_qk_norm
+        if self.use_qk_norm:
+            self.q_norm = SolarOpenRMSNorm(self.head_dim, eps=config.rms_norm_eps)
+            self.k_norm = SolarOpenRMSNorm(self.head_dim, eps=config.rms_norm_eps)
+
+    @deprecate_kwarg("past_key_value", new_name="past_key_values", version="4.58")
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        position_embeddings: tuple[torch.Tensor, torch.Tensor],
+        attention_mask: Optional[torch.Tensor],
+        past_key_values: Optional[Cache] = None,
+        cache_position: Optional[torch.LongTensor] = None,
+        **kwargs: Unpack[FlashAttentionKwargs],
+    ) -> tuple[torch.Tensor, Optional[torch.Tensor]]:
+        input_shape = hidden_states.shape[:-1]
+        hidden_shape = (*input_shape, -1, self.head_dim)
+
+        query_states = self.q_proj(hidden_states).view(hidden_shape)
+        key_states = self.k_proj(hidden_states).view(hidden_shape)
+        value_states = self.v_proj(hidden_states).view(hidden_shape)
+
+        if self.use_qk_norm:  # main diff from Llama
+            query_states = self.q_norm(query_states)
+            key_states = self.k_norm(key_states)
+
+        query_states = query_states.transpose(1, 2)
+        key_states = key_states.transpose(1, 2)
+        value_states = value_states.transpose(1, 2)
+
+        cos, sin = position_embeddings
+        query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
+
+        if past_key_values is not None:
+            # sin and cos are specific to RoPE models; position_ids needed for the static cache
+            cache_kwargs = {"sin": sin, "cos": cos, "cache_position": cache_position}
+            key_states, value_states = past_key_values.update(key_states, value_states, self.layer_idx, cache_kwargs)
+
+        attention_interface: Callable = eager_attention_forward
+        if self.config._attn_implementation != "eager":
+            attention_interface = ALL_ATTENTION_FUNCTIONS[self.config._attn_implementation]
+
+        attn_output, attn_weights = attention_interface(
+            self,
+            query_states,
+            key_states,
+            value_states,
+            attention_mask,
+            dropout=0.0 if not self.training else self.attention_dropout,
+            scaling=self.scaling,
+            **kwargs,
+        )
+
+        attn_output = attn_output.reshape(*input_shape, -1).contiguous()
+        attn_output = self.o_proj(attn_output)
+        return attn_output, attn_weights
+
+
+class SolarOpenMLP(nn.Module):
+    def __init__(self, config, hidden_size=None, intermediate_size=None):
+        super().__init__()
+        self.config = config
+        self.hidden_size = config.hidden_size if hidden_size is None else hidden_size
+        self.intermediate_size = config.intermediate_size if intermediate_size is None else intermediate_size
+
+        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
+        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
+        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
+        self.act_fn = ACT2FN[config.hidden_act]
+
+    def forward(self, x):
+        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
+        return down_proj
+
+
+class SolarOpenTopkRouter(nn.Module):
+    def __init__(self, config: SolarOpenConfig):
+        super().__init__()
+        self.config = config
+        self.top_k = config.num_experts_per_tok
+        self.n_routed_experts = config.n_routed_experts
+        self.routed_scaling_factor = config.routed_scaling_factor
+        self.n_group = config.n_group
+        self.topk_group = config.topk_group
+        self.norm_topk_prob = config.norm_topk_prob
+
+        self.weight = nn.Parameter(torch.empty((self.n_routed_experts, config.hidden_size)))
+        self.e_score_correction_bias = nn.Parameter(
+            torch.zeros((self.n_routed_experts), dtype=torch.float32))
+        
+        self._gate_logits = None
+
+    @torch.no_grad()
+    def get_topk_indices(self, scores):
+        scores_for_choice = scores.view(-1, self.n_routed_experts) + self.e_score_correction_bias.unsqueeze(0)
+        group_scores = (
+            scores_for_choice.view(-1, self.n_group, self.n_routed_experts // self.n_group)
+            .topk(2, dim=-1)[0]
+            .sum(dim=-1)
+        )
+        group_idx = torch.topk(group_scores, k=self.topk_group, dim=-1, sorted=False)[1]
+        group_mask = torch.zeros_like(group_scores)
+        group_mask.scatter_(1, group_idx, 1)
+        score_mask = (
+            group_mask.unsqueeze(-1)
+            .expand(-1, self.n_group, self.n_routed_experts // self.n_group)
+            .reshape(-1, self.n_routed_experts)
+        )
+        scores_for_choice = scores_for_choice.masked_fill(~score_mask.bool(), 0.0)
+        topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
+        return topk_indices
+
+    def forward(self, hidden_states):
+        hidden_states = hidden_states.view(-1, self.config.hidden_size)
+        router_logits = F.linear(hidden_states.type(torch.float32), self.weight.type(torch.float32))
+        self._gate_logits = router_logits
+        scores = router_logits.sigmoid()
+        topk_indices = self.get_topk_indices(scores)
+        topk_weights = scores.gather(1, topk_indices)
+        if self.norm_topk_prob:
+            denominator = topk_weights.sum(dim=-1, keepdim=True) + 1e-20
+            topk_weights /= denominator
+        topk_weights = topk_weights * self.routed_scaling_factor
+        return topk_indices, topk_weights
+
+
+@use_kernel_forward_from_hub("RMSNorm")
+class SolarOpenRMSNorm(nn.Module):
+    def __init__(self, hidden_size, eps=1e-6):
+        """
+        SolarOpenRMSNorm is equivalent to T5LayerNorm
+        """
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(hidden_size))
+        self.variance_epsilon = eps
+
+    def forward(self, hidden_states):
+        input_dtype = hidden_states.dtype
+        hidden_states = hidden_states.to(torch.float32)
+        variance = hidden_states.pow(2).mean(-1, keepdim=True)
+        hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
+        return self.weight * hidden_states.to(input_dtype)
+
+    def extra_repr(self):
+        return f"{tuple(self.weight.shape)}, eps={self.variance_epsilon}"
+
+
+class SolarOpenMoE(nn.Module):
+    """
+    A mixed expert module containing shared experts.
+    """
+
+    def __init__(self, config):
+        super().__init__()
+        self.config = config
+        self.experts = nn.ModuleList(
+            [
+                SolarOpenMLP(config, intermediate_size=config.moe_intermediate_size)
+                for _ in range(config.n_routed_experts)
+            ]
+        )
+        self.gate = SolarOpenTopkRouter(config)
+        self.shared_experts = SolarOpenMLP(
+            config=config, intermediate_size=config.moe_intermediate_size * config.n_shared_experts
+        )
+
+    @torch.compiler.disable()
+    def moe(self, hidden_states: torch.Tensor, topk_indices: torch.Tensor, topk_weights: torch.Tensor):
+        r"""
+        MoE forward pass that only executes selected experts.
+        Uses @torch.compiler.disable() to allow dynamic shape operations.
+        Requires --enforce-eager flag when serving with vLLM.
+        """
+        final_hidden_states = torch.zeros_like(hidden_states)
+
+        for expert_idx in range(len(self.experts)):
+            expert = self.experts[expert_idx]
+
+            # Find positions where this expert was selected
+            batch_idx, topk_pos = torch.where(topk_indices == expert_idx)
+
+            if batch_idx.numel() == 0:
+                continue
+
+            # Extract only the tokens routed to this expert
+            expert_input = hidden_states[batch_idx]
+            expert_output = expert(expert_input)
+
+            # Apply weights and accumulate results
+            weights = topk_weights[batch_idx, topk_pos].unsqueeze(-1)
+            final_hidden_states.index_add_(0, batch_idx, (expert_output * weights).to(hidden_states.dtype))
+
+        return final_hidden_states
+
+    def forward(self, hidden_states):
+        residuals = hidden_states
+        orig_shape = hidden_states.shape
+        topk_indices, topk_weights = self.gate(hidden_states)
+        hidden_states = hidden_states.view(-1, hidden_states.shape[-1])
+        hidden_states = self.moe(hidden_states, topk_indices, topk_weights).view(*orig_shape)
+        hidden_states = hidden_states + self.shared_experts(residuals)
+        return hidden_states
+
+
+class SolarOpenDecoderLayer(GradientCheckpointingLayer):
+    def __init__(self, config: SolarOpenConfig, layer_idx: int):
+        super().__init__()
+        self.hidden_size = config.hidden_size
+
+        self.self_attn = SolarOpenAttention(config=config, layer_idx=layer_idx)
+
+        if layer_idx >= config.first_k_dense_replace:
+            self.mlp = SolarOpenMoE(config)
+        else:
+            self.mlp = SolarOpenMLP(config)
+
+        self.input_layernorm = SolarOpenRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.post_attention_layernorm = SolarOpenRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+
+    @deprecate_kwarg("past_key_value", new_name="past_key_values", version="4.58")
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[Cache] = None,
+        use_cache: Optional[bool] = False,
+        cache_position: Optional[torch.LongTensor] = None,
+        position_embeddings: Optional[tuple[torch.Tensor, torch.Tensor]] = None,  # necessary, but kept here for BC
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> torch.Tensor:
+        residual = hidden_states
+        hidden_states = self.input_layernorm(hidden_states)
+        # Self Attention
+        hidden_states, _ = self.self_attn(
+            hidden_states=hidden_states,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            use_cache=use_cache,
+            cache_position=cache_position,
+            position_embeddings=position_embeddings,
+            **kwargs,
+        )
+        hidden_states = residual + hidden_states
+
+        # Fully Connected
+        residual = hidden_states
+        hidden_states = self.post_attention_layernorm(hidden_states)
+        hidden_states = self.mlp(hidden_states)
+        hidden_states = residual + hidden_states
+        return hidden_states
+
+
+@auto_docstring
+class SolarOpenPreTrainedModel(PreTrainedModel):
+    config: SolarOpenConfig
+    base_model_prefix = "model"
+    supports_gradient_checkpointing = True
+    _no_split_modules = ["SolarOpenDecoderLayer"]
+    _skip_keys_device_placement = ["past_key_values"]
+    _supports_flash_attn = True
+    _supports_sdpa = True
+    _supports_flex_attn = True
+    _can_compile_fullgraph = False
+    _supports_attention_backend = True
+    _can_record_outputs = {
+        "hidden_states": SolarOpenDecoderLayer,
+        "attentions": SolarOpenAttention,
+    }
+
+    def _init_weights(self, module):
+        super()._init_weights(module)
+        if isinstance(module, SolarOpenTopkRouter):
+            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
+
+
+class SolarOpenRotaryEmbedding(nn.Module):
+    inv_freq: torch.Tensor  # fix linting for `register_buffer`
+
+    def __init__(self, config: SolarOpenConfig, device=None):
+        super().__init__()
+        # BC: "rope_type" was originally "type"
+        if hasattr(config, "rope_scaling") and isinstance(config.rope_scaling, dict):
+            self.rope_type = config.rope_scaling.get("rope_type", config.rope_scaling.get("type"))
+        else:
+            self.rope_type = "default"
+        self.max_seq_len_cached = config.max_position_embeddings
+        self.original_max_seq_len = config.max_position_embeddings
+
+        self.config = config
+        self.rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type]
+
+        inv_freq, self.attention_scaling = self.rope_init_fn(self.config, device)
+        self.register_buffer("inv_freq", inv_freq, persistent=False)
+        self.original_inv_freq = self.inv_freq
+
+    @torch.no_grad()
+    @dynamic_rope_update  # power user: used with advanced RoPE types (e.g. dynamic rope)
+    def forward(self, x, position_ids):
+        inv_freq_expanded = self.inv_freq[None, :, None].float().expand(position_ids.shape[0], -1, 1).to(x.device)
+        position_ids_expanded = position_ids[:, None, :].float()
+
+        device_type = x.device.type if isinstance(x.device.type, str) and x.device.type != "mps" else "cpu"
+        with torch.autocast(device_type=device_type, enabled=False):  # Force float32
+            freqs = (inv_freq_expanded.float() @ position_ids_expanded.float()).transpose(1, 2)
+            emb = torch.cat((freqs, freqs), dim=-1)
+            cos = emb.cos() * self.attention_scaling
+            sin = emb.sin() * self.attention_scaling
+
+        return cos.to(dtype=x.dtype), sin.to(dtype=x.dtype)
+
+
+@auto_docstring
+class SolarOpenModel(SolarOpenPreTrainedModel):
+    _keys_to_ignore_on_load_unexpected = [r"model\.layers\.92.*", r"model\.layers\.46.*"]
+
+    def __init__(self, config: SolarOpenConfig):
+        super().__init__(config)
+        self.padding_idx = config.pad_token_id
+        self.vocab_size = config.vocab_size
+
+        self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)
+        self.layers = nn.ModuleList(
+            [SolarOpenDecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
+        )
+        self.norm = SolarOpenRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.rotary_emb = SolarOpenRotaryEmbedding(config=config)
+        self.gradient_checkpointing = False
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    @check_model_inputs()
+    @auto_docstring
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[Cache] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        cache_position: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> BaseModelOutputWithPast:
+        if (input_ids is None) ^ (inputs_embeds is not None):
+            raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
+
+        if inputs_embeds is None:
+            inputs_embeds: torch.Tensor = self.embed_tokens(input_ids)
+
+        if use_cache and past_key_values is None:
+            past_key_values = DynamicCache(config=self.config)
+
+        if cache_position is None:
+            past_seen_tokens = past_key_values.get_seq_length() if past_key_values is not None else 0
+            cache_position: torch.Tensor = torch.arange(
+                past_seen_tokens, past_seen_tokens + inputs_embeds.shape[1], device=inputs_embeds.device
+            )
+
+        if position_ids is None:
+            position_ids = cache_position.unsqueeze(0)
+
+        causal_mask = create_causal_mask(
+            config=self.config,
+            input_embeds=inputs_embeds,
+            attention_mask=attention_mask,
+            cache_position=cache_position,
+            past_key_values=past_key_values,
+            position_ids=position_ids,
+        )
+
+        hidden_states = inputs_embeds
+        position_embeddings = self.rotary_emb(hidden_states, position_ids)
+
+        for decoder_layer in self.layers[: self.config.num_hidden_layers]:
+            hidden_states = decoder_layer(
+                hidden_states,
+                attention_mask=causal_mask,
+                position_ids=position_ids,
+                past_key_values=past_key_values,
+                cache_position=cache_position,
+                position_embeddings=position_embeddings,
+                **kwargs,
+            )
+
+        hidden_states = self.norm(hidden_states)
+        return BaseModelOutputWithPast(
+            last_hidden_state=hidden_states,
+            past_key_values=past_key_values,
+        )
+
+
+@auto_docstring
+class SolarOpenForCausalLM(SolarOpenPreTrainedModel, GenerationMixin):
+    _tied_weights_keys = ["lm_head.weight"]
+    _tp_plan = {"lm_head": "colwise_rep"}
+    _pp_plan = {"lm_head": (["hidden_states"], ["logits"])}
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.model = SolarOpenModel(config)
+        self.vocab_size = config.vocab_size
+        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    @can_return_tuple
+    @auto_docstring
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[Cache] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        cache_position: Optional[torch.LongTensor] = None,
+        logits_to_keep: Union[int, torch.Tensor] = 0,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> CausalLMOutputWithPast:
+
+        outputs: BaseModelOutputWithPast = self.model(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            use_cache=use_cache,
+            cache_position=cache_position,
+            **kwargs,
+        )
+
+        hidden_states = outputs.last_hidden_state
+        # Only compute necessary logits, and do not upcast them to float if we are not computing the loss
+        slice_indices = slice(-logits_to_keep, None) if isinstance(logits_to_keep, int) else logits_to_keep
+        logits = self.lm_head(hidden_states[:, slice_indices, :])
+
+        loss = None
+        if labels is not None:
+            loss = self.loss_function(logits=logits, labels=labels, vocab_size=self.config.vocab_size, **kwargs)
+
+        return CausalLMOutputWithPast(
+            loss=loss,
+            logits=logits,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+
+__all__ = ["SolarOpenPreTrainedModel", "SolarOpenModel", "SolarOpenForCausalLM"]

parallel_tool_call_logits_processor.py

@@ -0,0 +1,104 @@
+# coding=utf-8
+# Copyright 2025 Upstage AI.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from typing import TYPE_CHECKING
+
+import torch
+
+from vllm.sampling_params import SamplingParams
+from vllm.v1.sample.logits_processor import (
+    AdapterLogitsProcessor,
+    RequestLogitsProcessor,
+)
+
+if TYPE_CHECKING:
+    from vllm.config import VllmConfig
+
+# Hardcoded token IDs for Solar tokenizer
+TOOL_CALL_END_TOKEN_ID = 32  # <|tool_call:end|>
+CALLS_TOKEN_ID = 25  # <|calls|>
+
+
+class SingleToolCallEnforcer:
+    """Request-level logits processor that enforces single tool call.
+
+    When <|tool_call:end|> token is generated, forces the next token
+    to be <|calls|> (which is a stop token), preventing parallel tool calls.
+    """
+
+    def __init__(
+        self,
+        tool_call_end_token_id: int,
+        calls_token_id: int,
+    ):
+        self._tool_call_end_token_id = tool_call_end_token_id
+        self._calls_token_id = calls_token_id
+
+    def __call__(
+        self,
+        output_token_ids: list[int],
+        logits: torch.Tensor,
+    ) -> torch.Tensor:
+        # Check if last generated token is <|tool_call:end|>
+        if output_token_ids and output_token_ids[-1] == self._tool_call_end_token_id:
+            # Force next token to be <|calls|> by masking all other tokens
+            mask = torch.full_like(logits, -float("inf"))
+            mask[self._calls_token_id] = logits[self._calls_token_id]
+            return mask
+
+        return logits
+
+
+class ParallelToolCallLogitsProcessor(AdapterLogitsProcessor):
+    """Logits processor that enforces single tool call when parallel_tool_calls=False.
+
+    When parallel_tool_calls is disabled in SamplingParams, this processor
+    ensures that after <|tool_call:end|> is generated, the next token is
+    forced to be <|calls|> (a stop token), preventing multiple tool calls.
+    """
+
+    def __init__(
+        self,
+        vllm_config: "VllmConfig",
+        device: torch.device,
+        is_pin_memory: bool,
+    ):
+        super().__init__(vllm_config, device, is_pin_memory)
+
+    def is_argmax_invariant(self) -> bool:
+        """This processor can change argmax result by forcing specific tokens."""
+        return False
+
+    def new_req_logits_processor(
+        self,
+        params: SamplingParams,
+    ) -> RequestLogitsProcessor | None:
+        """Return a request-level logits processor if parallel_tool_calls=False.
+
+        Args:
+            params: Request sampling params
+
+        Returns:
+            SingleToolCallEnforcer if parallel_tool_calls is False, otherwise None.
+        """
+        # Only apply when parallel_tool_calls is explicitly disabled
+        if params.parallel_tool_calls is False:
+            return SingleToolCallEnforcer(
+                tool_call_end_token_id=TOOL_CALL_END_TOKEN_ID,
+                calls_token_id=CALLS_TOKEN_ID,
+            )
+
+        return None
+

solar_open_logits_processor.py

@@ -0,0 +1,763 @@
+# coding=utf-8
+# Copyright 2025 Upstage AI.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+from enum import Enum
+from typing import TYPE_CHECKING
+
+import torch
+
+from vllm.sampling_params import SamplingParams
+from vllm.v1.sample.logits_processor import (
+    AdapterLogitsProcessor,
+    RequestLogitsProcessor,
+)
+
+if TYPE_CHECKING:
+    from vllm.config import VllmConfig
+
+# Hardcoded token IDs for Solar tokenizer
+
+# Special token IDs for chat template
+BEGIN_TOKEN_ID = 20  # <|begin|>
+END_TOKEN_ID = 21  # <|end|>
+THINK_TOKEN_ID = 22  # <|think|>
+CONTENT_TOKEN_ID = 23  # <|content|>
+FLUSH_TOKEN_ID = 24  # <|flush|> (eos token)
+ASSISTANT_TOKEN_ID = 163444  # assistant
+'''
+'assistant' is not a special token exactly, but is treated as one in the logits
+processing.
+'''
+
+# Tool call related tokens
+CALLS_TOKEN_ID = 25  # <|calls|> (eos token for tool calls)
+TOOL_CALLS_TOKEN_ID = 30  # <|tool_calls|>
+TOOL_CALL_BEGIN_TOKEN_ID = 31  # <|tool_call:begin|>
+TOOL_CALL_END_TOKEN_ID = 32  # <|tool_call:end|>
+TOOL_CALL_NAME_TOKEN_ID = 33  # <|tool_call:name|>
+TOOL_CALL_ARGS_TOKEN_ID = 34  # <|tool_call:args|>
+
+# =============================================================================
+# Dynamic Reasoning Budget Configuration
+# =============================================================================
+# budget = min(max_budget, max(min_budget, max_tokens * ratio / 100))
+# Priority: max_budget > min_budget > ratio
+#
+# Available environment variables:
+#   HIGH effort:
+#     SOLAR_REASONING_BUDGET_HIGH_MAX    (default: 32768)  - max_budget
+#     SOLAR_REASONING_BUDGET_HIGH_MIN    (default: 8192)   - min_budget
+#     SOLAR_REASONING_BUDGET_HIGH_RATIO  (default: 60)     - % of max_tokens
+#
+#   MEDIUM effort:
+#     SOLAR_REASONING_BUDGET_MEDIUM_MAX    (default: 16384)  - max_budget
+#     SOLAR_REASONING_BUDGET_MEDIUM_MIN    (default: 4096)   - min_budget
+#     SOLAR_REASONING_BUDGET_MEDIUM_RATIO  (default: 30)     - % of max_tokens
+#
+#   Tool call:
+#     SOLAR_TOOL_CALL_ID_BUDGET  (default: 10)  - Max tokens for tool call ID
+# =============================================================================
+
+DEFAULT_REASONING_EFFORT = "high"
+
+# HIGH effort settings (1k = 1024 tokens)
+DEFAULT_REASONING_BUDGET_HIGH_MAX = 32 * 1024
+DEFAULT_REASONING_BUDGET_HIGH_MIN = 8 * 1024
+DEFAULT_REASONING_BUDGET_HIGH_RATIO = 60
+
+# MEDIUM effort settings
+DEFAULT_REASONING_BUDGET_MEDIUM_MAX = 16 * 1024
+DEFAULT_REASONING_BUDGET_MEDIUM_MIN = 4 * 1024
+DEFAULT_REASONING_BUDGET_MEDIUM_RATIO = 30
+
+# Tool call settings
+DEFAULT_TOOL_CALL_ID_BUDGET = 10
+
+# Pre-computed constant to avoid repeated string parsing
+NEG_INF = float("-inf")
+
+
+def is_reasoning_request(params: SamplingParams) -> bool:
+    """Check if the request is a reasoning request based on reasoning_effort."""
+    return (params.reasoning_effort is None) or (params.reasoning_effort in ("medium", "high"))
+
+
+def is_structured_outputs(params: SamplingParams) -> bool:
+    """Check if the request has structured outputs constraints."""
+    return (
+        params.structured_outputs is not None
+        and not params.structured_outputs.all_constraints_none()
+    )
+
+
+class GenerationState(Enum):
+    """Enum representing the current state of response generation."""
+
+    # Initial state - no tokens generated yet
+    INITIAL = "initial"
+
+    # New message states (after think_end)
+    NEW_MESSAGE_BEGIN = "new_message_begin"  # <|begin|> token was just generated
+    NEW_MESSAGE_ASSISTANT = "new_message_assistant"  # assistant token after <|begin|>
+
+    # Think mode states
+    THINK_BEGIN = "think_begin"  # <|think|> token was just generated
+    THINK_IN_PROGRESS = "think_in_progress"  # Generating think content
+    THINK_END = "think_end"  # <|end|> after think content
+    THINK_FLUSH = "think_flush"  # <|flush|> after think content
+
+    # Content states
+    CONTENT_BEGIN = "content_begin"  # <|content|> token was just generated
+    CONTENT_IN_PROGRESS = "content_in_progress"  # Generating content
+    CONTENT_END = "content_end"  # <|end|> or <|flush|> after content
+    CONTENT_FLUSH = "content_flush"  # <|flush|> after content
+
+    # Tool call states
+    # Flow: <|tool_calls|> -> (<|tool_call:begin|> -> id -> <|tool_call:name|> -> name -> <|tool_call:args|> -> args -> <|tool_call:end|>)+ -> <|calls|>
+    # Note: Think message can appear before <|tool_calls|>
+    TOOL_CALLS_BEGIN = "tool_calls_begin"  # <|tool_calls|> token was just generated
+    TOOL_CALL_BEGIN = "tool_call_begin"  # <|tool_call:begin|> token was just generated
+    TOOL_CALL_ID_IN_PROGRESS = "tool_call_id_in_progress"  # Generating tool call ID
+    TOOL_CALL_NAME_BEGIN = "tool_call_name_begin"  # <|tool_call:name|> token was just generated
+    TOOL_CALL_NAME_IN_PROGRESS = "tool_call_name_in_progress"  # Generating tool name
+    TOOL_CALL_ARGS_BEGIN = "tool_call_args_begin"  # <|tool_call:args|> token was just generated
+    TOOL_CALL_ARGS_IN_PROGRESS = "tool_call_args_in_progress"  # Generating tool arguments (JSON)
+    TOOL_CALL_END = "tool_call_end"  # <|tool_call:end|> token was just generated (can start another tool call or end)
+    CALLS = "calls"  # <|calls|> token was just generated (eos token for tool calls)
+
+
+def get_generation_state(
+    output_token_ids: list[int],
+    begin_token_id: int = BEGIN_TOKEN_ID,
+    end_token_id: int = END_TOKEN_ID,
+    flush_token_id: int = FLUSH_TOKEN_ID,
+    think_token_id: int = THINK_TOKEN_ID,
+    content_token_id: int = CONTENT_TOKEN_ID,
+    tool_calls_token_id: int = TOOL_CALLS_TOKEN_ID,
+    tool_call_begin_token_id: int = TOOL_CALL_BEGIN_TOKEN_ID,
+    tool_call_name_token_id: int = TOOL_CALL_NAME_TOKEN_ID,
+    tool_call_args_token_id: int = TOOL_CALL_ARGS_TOKEN_ID,
+    tool_call_end_token_id: int = TOOL_CALL_END_TOKEN_ID,
+    calls_token_id: int = CALLS_TOKEN_ID,
+    assistant_token_id: int = ASSISTANT_TOKEN_ID,
+) -> GenerationState:
+    """Determine the current generation state based on output token IDs.
+
+    Analyzes the sequence of generated tokens to determine which phase
+    of the chat template the generation is currently in.
+
+    Response format specs:
+    - think mode: <|think|>{{think-tokens}}<|end|><|begin|>assistant<|content|>{{content-tokens}}<|flush|>
+    - tool mode: <|begin|>assistant<|tool_calls|><|tool_call:begin|>{{id}}<|tool_call:name|>{{name}}<|tool_call:args|>{{args}}<|tool_call:end|><|calls|>
+    - tool mode (with think): <|think|>{{think-tokens}}<|end|><|begin|>assistant<|tool_calls|>...<|calls|>
+    - no-think mode: <|content|>{{content-tokens}}<|flush|>
+
+    Args:
+        output_token_ids: List of token IDs generated so far.
+        begin_token_id: Token ID for <|begin|>.
+        end_token_id: Token ID for <|end|>.
+        flush_token_id: Token ID for <|flush|> (eos).
+        think_token_id: Token ID for <|think|>.
+        content_token_id: Token ID for <|content|>.
+        tool_calls_token_id: Token ID for <|tool_calls|>.
+        tool_call_begin_token_id: Token ID for <|tool_call:begin|>.
+        tool_call_name_token_id: Token ID for <|tool_call:name|>.
+        tool_call_args_token_id: Token ID for <|tool_call:args|>.
+        tool_call_end_token_id: Token ID for <|tool_call:end|>.
+        calls_token_id: Token ID for <|calls|> (eos).
+        assistant_token_id: Token ID for assistant.
+
+    Returns:
+        GenerationState indicating the current phase of generation.
+    """
+    if not output_token_ids:
+        return GenerationState.INITIAL
+
+    # Track state by scanning through tokens
+    state = GenerationState.INITIAL
+    in_think = False
+    in_content = False
+
+    for token_id in output_token_ids:
+        if token_id == think_token_id:
+            state = GenerationState.THINK_BEGIN
+            in_think = True
+            in_content = False
+
+        elif token_id == content_token_id:
+            state = GenerationState.CONTENT_BEGIN
+            in_content = True
+            in_think = False
+
+        elif token_id == tool_calls_token_id:
+            state = GenerationState.TOOL_CALLS_BEGIN
+            in_think = False
+            in_content = False
+
+        elif token_id == tool_call_begin_token_id:
+            state = GenerationState.TOOL_CALL_BEGIN
+
+        elif token_id == tool_call_name_token_id:
+            state = GenerationState.TOOL_CALL_NAME_BEGIN
+
+        elif token_id == tool_call_args_token_id:
+            state = GenerationState.TOOL_CALL_ARGS_BEGIN
+
+        elif token_id == tool_call_end_token_id:
+            state = GenerationState.TOOL_CALL_END
+
+        elif token_id == calls_token_id:
+            state = GenerationState.CALLS
+
+        elif token_id == begin_token_id:
+            state = GenerationState.NEW_MESSAGE_BEGIN
+
+        elif token_id == assistant_token_id:
+            if state == GenerationState.NEW_MESSAGE_BEGIN:
+                state = GenerationState.NEW_MESSAGE_ASSISTANT
+
+        elif token_id == end_token_id:
+            if in_think:
+                state = GenerationState.THINK_END
+                in_think = False
+            elif in_content:
+                state = GenerationState.CONTENT_END
+                in_content = False
+
+        elif token_id == flush_token_id:
+            if in_think:
+                state = GenerationState.THINK_FLUSH
+                in_think = False
+            elif in_content:
+                state = GenerationState.CONTENT_FLUSH
+                in_content = False
+
+        else:
+            # Regular token - update state based on current context
+            if state == GenerationState.THINK_BEGIN:
+                state = GenerationState.THINK_IN_PROGRESS
+            elif state == GenerationState.THINK_IN_PROGRESS:
+                pass  # Stay in think_in_progress
+            elif state == GenerationState.CONTENT_BEGIN:
+                state = GenerationState.CONTENT_IN_PROGRESS
+            elif state == GenerationState.CONTENT_IN_PROGRESS:
+                pass  # Stay in content_in_progress
+            elif state == GenerationState.TOOL_CALL_BEGIN:
+                state = GenerationState.TOOL_CALL_ID_IN_PROGRESS
+            elif state == GenerationState.TOOL_CALL_ID_IN_PROGRESS:
+                pass  # Stay in tool_call_id_in_progress
+            elif state == GenerationState.TOOL_CALL_NAME_BEGIN:
+                state = GenerationState.TOOL_CALL_NAME_IN_PROGRESS
+            elif state == GenerationState.TOOL_CALL_NAME_IN_PROGRESS:
+                pass  # Stay in tool_call_name_in_progress
+            elif state == GenerationState.TOOL_CALL_ARGS_BEGIN:
+                state = GenerationState.TOOL_CALL_ARGS_IN_PROGRESS
+            elif state == GenerationState.TOOL_CALL_ARGS_IN_PROGRESS:
+                pass  # Stay in tool_call_args_in_progress
+
+    return state
+
+
+# Pre-computed list of all special token IDs for batch indexing
+_ALL_SPECIAL_TOKEN_IDS = [
+    BEGIN_TOKEN_ID,
+    END_TOKEN_ID,
+    THINK_TOKEN_ID,
+    CONTENT_TOKEN_ID,
+    FLUSH_TOKEN_ID,
+    CALLS_TOKEN_ID,
+    TOOL_CALLS_TOKEN_ID,
+    TOOL_CALL_BEGIN_TOKEN_ID,
+    TOOL_CALL_END_TOKEN_ID,
+    TOOL_CALL_NAME_TOKEN_ID,
+    TOOL_CALL_ARGS_TOKEN_ID,
+]
+
+# Pre-computed lists for state-specific batch indexing (excluding allowed tokens)
+_SPECIAL_EXCEPT_END = [  # For THINK states (allow END)
+    BEGIN_TOKEN_ID, FLUSH_TOKEN_ID, THINK_TOKEN_ID, CONTENT_TOKEN_ID,
+    TOOL_CALLS_TOKEN_ID, CALLS_TOKEN_ID, TOOL_CALL_BEGIN_TOKEN_ID,
+    TOOL_CALL_END_TOKEN_ID, TOOL_CALL_NAME_TOKEN_ID, TOOL_CALL_ARGS_TOKEN_ID,
+]
+
+_SPECIAL_EXCEPT_CONTENT_TOOLCALLS = [  # For NEW_MESSAGE_ASSISTANT (allow CONTENT, TOOL_CALLS)
+    THINK_TOKEN_ID, BEGIN_TOKEN_ID, END_TOKEN_ID, FLUSH_TOKEN_ID,
+    CALLS_TOKEN_ID, TOOL_CALL_BEGIN_TOKEN_ID, TOOL_CALL_END_TOKEN_ID,
+    TOOL_CALL_NAME_TOKEN_ID, TOOL_CALL_ARGS_TOKEN_ID,
+]
+
+_SPECIAL_EXCEPT_FLUSH = [  # For CONTENT states (allow FLUSH)
+    BEGIN_TOKEN_ID, END_TOKEN_ID, THINK_TOKEN_ID, CONTENT_TOKEN_ID,
+    TOOL_CALLS_TOKEN_ID, CALLS_TOKEN_ID, TOOL_CALL_BEGIN_TOKEN_ID,
+    TOOL_CALL_END_TOKEN_ID, TOOL_CALL_NAME_TOKEN_ID, TOOL_CALL_ARGS_TOKEN_ID,
+]
+
+_SPECIAL_EXCEPT_TOOLCALL_NAME = [  # For TOOL_CALL_ID_IN_PROGRESS (allow TOOL_CALL_NAME)
+    BEGIN_TOKEN_ID, END_TOKEN_ID, THINK_TOKEN_ID, CONTENT_TOKEN_ID,
+    FLUSH_TOKEN_ID, CALLS_TOKEN_ID, TOOL_CALLS_TOKEN_ID,
+    TOOL_CALL_BEGIN_TOKEN_ID, TOOL_CALL_END_TOKEN_ID, TOOL_CALL_ARGS_TOKEN_ID,
+]
+
+_SPECIAL_EXCEPT_TOOLCALL_ARGS = [  # For TOOL_CALL_NAME_IN_PROGRESS (allow TOOL_CALL_ARGS)
+    BEGIN_TOKEN_ID, END_TOKEN_ID, THINK_TOKEN_ID, CONTENT_TOKEN_ID,
+    FLUSH_TOKEN_ID, CALLS_TOKEN_ID, TOOL_CALLS_TOKEN_ID,
+    TOOL_CALL_BEGIN_TOKEN_ID, TOOL_CALL_END_TOKEN_ID, TOOL_CALL_NAME_TOKEN_ID,
+]
+
+_SPECIAL_EXCEPT_TOOLCALL_END = [  # For TOOL_CALL_ARGS_IN_PROGRESS (allow TOOL_CALL_END)
+    BEGIN_TOKEN_ID, END_TOKEN_ID, THINK_TOKEN_ID, CONTENT_TOKEN_ID,
+    FLUSH_TOKEN_ID, CALLS_TOKEN_ID, TOOL_CALLS_TOKEN_ID,
+    TOOL_CALL_BEGIN_TOKEN_ID, TOOL_CALL_NAME_TOKEN_ID, TOOL_CALL_ARGS_TOKEN_ID,
+]
+
+
+def _forbid_all_special_tokens(logits: torch.Tensor) -> None:
+    """Set all special token logits to -inf."""
+    logits[_ALL_SPECIAL_TOKEN_IDS] = NEG_INF
+
+
+class SolarOpenTemplateEnforcer:
+    """Request-level logits processor that enforces Solar Open chat template.
+
+    Enforces the following generation rules:
+    - think mode: <|think|>{{tokens}}<|end|><|begin|>assistant<|content|>{{tokens}}<|flush|>
+    - tool mode: <|tool_calls|><|tool_call:begin|>{{id}}<|tool_call:name|>{{name}}<|tool_call:args|>{{args}}<|tool_call:end|><|calls|>
+    - tool+think mode: <|think|>{{tokens}}<|end|><|begin|>assistant<|tool_calls|>...<|calls|>
+    - no-think mode: <|content|>{{tokens}}<|flush|>
+
+    Key constraints:
+    - Think message can only appear first
+    - Think message must be followed by another message
+    - Content and tool messages cannot coexist
+    - Maximum 2 messages (think + content/tool, or just content/tool)
+
+    Performance optimization:
+    - Uses incremental state tracking to avoid full token sequence scan on each call
+    - Maintains local counters for budget tracking
+    - Uses pre-computed constants to avoid repeated object creation
+    """
+
+    # Pre-computed frozenset for reasoning state check (avoids set creation per call)
+    _REASONING_STATES = frozenset({
+        GenerationState.INITIAL,
+        GenerationState.THINK_BEGIN,
+        GenerationState.THINK_IN_PROGRESS,
+    })
+
+    def __init__(
+        self,
+        is_reasoning_request: bool,
+        is_structured_outputs: bool,
+        reasoning_budget: int | None = None,
+        tool_call_id_budget: int = DEFAULT_TOOL_CALL_ID_BUDGET,
+    ):
+        self._is_reasoning_request = is_reasoning_request
+        self._is_structured_outputs = is_structured_outputs
+        self._reasoning_budget = reasoning_budget
+        self._tool_call_id_budget = tool_call_id_budget
+
+        # Incremental state tracking
+        self._state = GenerationState.INITIAL
+        self._last_processed_len = 0
+        self._in_think = False
+        self._in_content = False
+
+        # Budget counters
+        self._think_token_count = 0
+        self._tool_call_id_token_count = 0
+
+    def _reset_state(self) -> None:
+        """Reset all incremental state to initial values.
+
+        Called when defensive reprocessing is needed (e.g., token sequence inconsistency).
+        """
+        self._state = GenerationState.INITIAL
+        self._last_processed_len = 0
+        self._in_think = False
+        self._in_content = False
+        self._think_token_count = 0
+        self._tool_call_id_token_count = 0
+
+    def _process_token(self, token_id: int) -> None:
+        """Process a single token and update internal state incrementally.
+
+        Args:
+            token_id: The token ID to process.
+        """
+        if token_id == THINK_TOKEN_ID:
+            self._state = GenerationState.THINK_BEGIN
+            self._in_think = True
+            self._in_content = False
+            self._think_token_count = 0  # Reset counter for new think block
+
+        elif token_id == CONTENT_TOKEN_ID:
+            self._state = GenerationState.CONTENT_BEGIN
+            self._in_content = True
+            self._in_think = False
+
+        elif token_id == TOOL_CALLS_TOKEN_ID:
+            self._state = GenerationState.TOOL_CALLS_BEGIN
+            self._in_think = False
+            self._in_content = False
+
+        elif token_id == TOOL_CALL_BEGIN_TOKEN_ID:
+            self._state = GenerationState.TOOL_CALL_BEGIN
+            self._tool_call_id_token_count = 0  # Reset counter for new tool call
+
+        elif token_id == TOOL_CALL_NAME_TOKEN_ID:
+            self._state = GenerationState.TOOL_CALL_NAME_BEGIN
+
+        elif token_id == TOOL_CALL_ARGS_TOKEN_ID:
+            self._state = GenerationState.TOOL_CALL_ARGS_BEGIN
+
+        elif token_id == TOOL_CALL_END_TOKEN_ID:
+            self._state = GenerationState.TOOL_CALL_END
+
+        elif token_id == CALLS_TOKEN_ID:
+            self._state = GenerationState.CALLS
+
+        elif token_id == BEGIN_TOKEN_ID:
+            self._state = GenerationState.NEW_MESSAGE_BEGIN
+
+        elif token_id == ASSISTANT_TOKEN_ID:
+            if self._state == GenerationState.NEW_MESSAGE_BEGIN:
+                self._state = GenerationState.NEW_MESSAGE_ASSISTANT
+
+        elif token_id == END_TOKEN_ID:
+            if self._in_think:
+                self._state = GenerationState.THINK_END
+                self._in_think = False
+            elif self._in_content:
+                self._state = GenerationState.CONTENT_END
+                self._in_content = False
+
+        elif token_id == FLUSH_TOKEN_ID:
+            if self._in_think:
+                self._state = GenerationState.THINK_FLUSH
+                self._in_think = False
+            elif self._in_content:
+                self._state = GenerationState.CONTENT_FLUSH
+                self._in_content = False
+
+        else:
+            # Regular token - update state and counters based on current context
+            if self._state == GenerationState.THINK_BEGIN:
+                self._state = GenerationState.THINK_IN_PROGRESS
+                self._think_token_count += 1
+            elif self._state == GenerationState.THINK_IN_PROGRESS:
+                self._think_token_count += 1
+            elif self._state == GenerationState.CONTENT_BEGIN:
+                self._state = GenerationState.CONTENT_IN_PROGRESS
+            elif self._state == GenerationState.CONTENT_IN_PROGRESS:
+                pass  # Stay in content_in_progress
+            elif self._state == GenerationState.TOOL_CALL_BEGIN:
+                self._state = GenerationState.TOOL_CALL_ID_IN_PROGRESS
+                self._tool_call_id_token_count += 1
+            elif self._state == GenerationState.TOOL_CALL_ID_IN_PROGRESS:
+                self._tool_call_id_token_count += 1
+            elif self._state == GenerationState.TOOL_CALL_NAME_BEGIN:
+                self._state = GenerationState.TOOL_CALL_NAME_IN_PROGRESS
+            elif self._state == GenerationState.TOOL_CALL_NAME_IN_PROGRESS:
+                pass  # Stay in tool_call_name_in_progress
+            elif self._state == GenerationState.TOOL_CALL_ARGS_BEGIN:
+                self._state = GenerationState.TOOL_CALL_ARGS_IN_PROGRESS
+            elif self._state == GenerationState.TOOL_CALL_ARGS_IN_PROGRESS:
+                pass  # Stay in tool_call_args_in_progress
+
+    def _update_state_incremental(self, output_token_ids: list[int]) -> None:
+        """Update internal state by processing only new tokens.
+
+        Args:
+            output_token_ids: Full list of output token IDs.
+        """
+        current_len = len(output_token_ids)
+
+        # Defensive check: if token sequence is shorter than expected, reset and reprocess
+        if current_len < self._last_processed_len:
+            self._reset_state()
+
+        # Process only new tokens
+        for i in range(self._last_processed_len, current_len):
+            self._process_token(output_token_ids[i])
+
+        self._last_processed_len = current_len
+
+    @staticmethod
+    def _count_think_tokens(output_token_ids: list[int]) -> int:
+        """Count the number of tokens generated after <|think|> token.
+
+        Returns 0 if <|think|> token is not found (defensive).
+        Note: This static method is kept for backward compatibility and testing.
+        The incremental version uses _think_token_count instead.
+        """
+        try:
+            think_index = output_token_ids.index(THINK_TOKEN_ID)
+            return len(output_token_ids) - think_index - 1
+        except ValueError:
+            return 0
+
+    @staticmethod
+    def _count_tool_call_id_tokens(output_token_ids: list[int]) -> int:
+        """Count the number of tokens generated after the last <|tool_call:begin|> token.
+
+        Returns 0 if <|tool_call:begin|> token is not found (defensive).
+        Note: This static method is kept for backward compatibility and testing.
+        The incremental version uses _tool_call_id_token_count instead.
+        """
+        # Find the last occurrence of <|tool_call:begin|> for multi-tool-call support
+        try:
+            # Reverse search for the last <|tool_call:begin|>
+            reversed_index = output_token_ids[::-1].index(TOOL_CALL_BEGIN_TOKEN_ID)
+            last_begin_index = len(output_token_ids) - 1 - reversed_index
+            return len(output_token_ids) - last_begin_index - 1
+        except ValueError:
+            return 0
+
+    def __call__(
+        self,
+        output_token_ids: list[int],
+        logits: torch.Tensor,
+    ) -> torch.Tensor:
+        # Update state incrementally (only process new tokens)
+        self._update_state_incremental(output_token_ids)
+        state = self._state
+
+        # Handle structured outputs mode
+        if self._is_structured_outputs:
+            if not self._is_reasoning_request:
+                # Non-reasoning request with structured outputs: no logit control
+                return logits
+            else:
+                # Reasoning request with structured outputs:
+                # Control logits only during reasoning phase
+                if state not in self._REASONING_STATES:
+                    # Reasoning finished, let structured outputs handle it
+                    return logits
+
+        if state == GenerationState.INITIAL:
+            if self._is_reasoning_request:
+                # Force: <|think|> only (reasoning request must start with think)
+                think_logit = logits[THINK_TOKEN_ID].clone()
+                logits.fill_(NEG_INF)
+                logits[THINK_TOKEN_ID] = think_logit
+            else:
+                # Allow: <|content|>, <|tool_calls|> only
+                content_logit = logits[CONTENT_TOKEN_ID].clone()
+                tool_calls_logit = logits[TOOL_CALLS_TOKEN_ID].clone()
+                logits.fill_(NEG_INF)
+                logits[CONTENT_TOKEN_ID] = content_logit
+                logits[TOOL_CALLS_TOKEN_ID] = tool_calls_logit
+
+        elif state in (GenerationState.THINK_BEGIN, GenerationState.THINK_IN_PROGRESS):
+            # Check if reasoning budget is exceeded (using incremental counter)
+            if (
+                self._reasoning_budget is not None
+                and state == GenerationState.THINK_IN_PROGRESS
+            ):
+                if self._think_token_count >= self._reasoning_budget:
+                    # Force <|end|> token to terminate reasoning
+                    logits.fill_(NEG_INF)
+                    logits[END_TOKEN_ID] = 0.0
+                    return logits
+
+            # Transform: <|flush|> -> <|end|>
+            # Think must be followed by another message, so prevent early termination
+            logits[END_TOKEN_ID] = torch.maximum(logits[END_TOKEN_ID], logits[FLUSH_TOKEN_ID])
+            # Forbid all special tokens except <|end|>
+            logits[_SPECIAL_EXCEPT_END] = NEG_INF
+
+        elif state == GenerationState.THINK_END:
+            # Force: <|begin|> only
+            # Think must be followed by another message
+            logits.fill_(NEG_INF)
+            logits[BEGIN_TOKEN_ID] = 0.0
+
+        elif state == GenerationState.NEW_MESSAGE_BEGIN:
+            # Force: assistant token only
+            logits.fill_(NEG_INF)
+            logits[ASSISTANT_TOKEN_ID] = 0.0
+
+        elif state == GenerationState.NEW_MESSAGE_ASSISTANT:
+            # Allow: <|content|>, <|tool_calls|>, regular tokens
+            # Forbid: all other special tokens
+            logits[_SPECIAL_EXCEPT_CONTENT_TOOLCALLS] = NEG_INF
+
+        elif state in (GenerationState.CONTENT_BEGIN, GenerationState.CONTENT_IN_PROGRESS):
+            # Transform: <|end|> -> <|flush|>
+            # Content cannot be followed by another message
+            logits[FLUSH_TOKEN_ID] = torch.maximum(logits[FLUSH_TOKEN_ID], logits[END_TOKEN_ID])
+            # Forbid all special tokens except <|flush|>
+            logits[_SPECIAL_EXCEPT_FLUSH] = NEG_INF
+
+        elif state == GenerationState.TOOL_CALLS_BEGIN:
+            # Force: <|tool_call:begin|> only
+            tool_call_begin_logit = logits[TOOL_CALL_BEGIN_TOKEN_ID].clone()
+            logits.fill_(NEG_INF)
+            logits[TOOL_CALL_BEGIN_TOKEN_ID] = tool_call_begin_logit
+
+        elif state == GenerationState.TOOL_CALL_BEGIN:
+            # Allow: regular tokens only (ID generation)
+            # Forbid: all special tokens
+            _forbid_all_special_tokens(logits)
+
+        elif state == GenerationState.TOOL_CALL_ID_IN_PROGRESS:
+            # Check if tool call ID budget is exceeded (using incremental counter)
+            if self._tool_call_id_token_count >= self._tool_call_id_budget:
+                # Force <|tool_call:name|> token to terminate ID generation
+                logits.fill_(NEG_INF)
+                logits[TOOL_CALL_NAME_TOKEN_ID] = 0.0
+                return logits
+
+            # Allow: <|tool_call:name|>, regular tokens
+            # Forbid: all other special tokens
+            logits[_SPECIAL_EXCEPT_TOOLCALL_NAME] = NEG_INF
+
+        elif state == GenerationState.TOOL_CALL_NAME_BEGIN:
+            # Allow: regular tokens only (function name generation)
+            # Forbid: all special tokens
+            _forbid_all_special_tokens(logits)
+
+        elif state == GenerationState.TOOL_CALL_NAME_IN_PROGRESS:
+            # Allow: <|tool_call:args|>, regular tokens
+            # Forbid: all other special tokens
+            logits[_SPECIAL_EXCEPT_TOOLCALL_ARGS] = NEG_INF
+
+        elif state == GenerationState.TOOL_CALL_ARGS_BEGIN:
+            # Allow: regular tokens only (JSON args generation)
+            # Forbid: all special tokens
+            _forbid_all_special_tokens(logits)
+
+        elif state == GenerationState.TOOL_CALL_ARGS_IN_PROGRESS:
+            # Allow: <|tool_call:end|>, regular tokens
+            # Forbid: all other special tokens
+            logits[_SPECIAL_EXCEPT_TOOLCALL_END] = NEG_INF
+
+        elif state == GenerationState.TOOL_CALL_END:
+            # Allow: <|tool_call:begin|> (next tool call), <|calls|> (end)
+            # Forbid: all other special tokens
+            tool_call_begin_logit = logits[TOOL_CALL_BEGIN_TOKEN_ID].clone()
+            calls_logit = logits[CALLS_TOKEN_ID].clone()
+            logits.fill_(NEG_INF)
+            logits[TOOL_CALL_BEGIN_TOKEN_ID] = tool_call_begin_logit
+            logits[CALLS_TOKEN_ID] = calls_logit
+
+        # CALLS state: no processing needed (EOS)
+
+        return logits
+
+class SolarOpenTemplateLogitsProcessor(AdapterLogitsProcessor):
+    """
+    Logits processor that enforces Solar Open chat template.
+    This processor manages the generation flow according to the
+    Solar Open chat template by tracking generation states.
+    """
+
+    def __init__(
+        self,
+        vllm_config: "VllmConfig",
+        device: torch.device,
+        is_pin_memory: bool,
+    ):
+        super().__init__(vllm_config, device, is_pin_memory)
+
+        # Dynamic reasoning budget settings for HIGH effort
+        self._high_max = self._parse_env_int(
+            "SOLAR_REASONING_BUDGET_HIGH_MAX", DEFAULT_REASONING_BUDGET_HIGH_MAX
+        )
+        self._high_min = self._parse_env_int(
+            "SOLAR_REASONING_BUDGET_HIGH_MIN", DEFAULT_REASONING_BUDGET_HIGH_MIN
+        )
+        self._high_ratio = self._parse_env_int(
+            "SOLAR_REASONING_BUDGET_HIGH_RATIO", DEFAULT_REASONING_BUDGET_HIGH_RATIO
+        )
+
+        # Dynamic reasoning budget settings for MEDIUM effort
+        self._medium_max = self._parse_env_int(
+            "SOLAR_REASONING_BUDGET_MEDIUM_MAX", DEFAULT_REASONING_BUDGET_MEDIUM_MAX
+        )
+        self._medium_min = self._parse_env_int(
+            "SOLAR_REASONING_BUDGET_MEDIUM_MIN", DEFAULT_REASONING_BUDGET_MEDIUM_MIN
+        )
+        self._medium_ratio = self._parse_env_int(
+            "SOLAR_REASONING_BUDGET_MEDIUM_RATIO", DEFAULT_REASONING_BUDGET_MEDIUM_RATIO
+        )
+
+        self._tool_call_id_budget: int = self._parse_env_int(
+            "SOLAR_TOOL_CALL_ID_BUDGET", DEFAULT_TOOL_CALL_ID_BUDGET
+        )
+
+    @staticmethod
+    def _parse_env_int(env_var: str, default: int) -> int:
+        """Parse environment variable as integer, return default if not set or invalid."""
+        value = os.environ.get(env_var)
+        if value is None:
+            return default
+        try:
+            return int(value)
+        except ValueError:
+            return default
+
+    def _calculate_reasoning_budget(self, effort: str, max_tokens: int) -> int:
+        """Calculate dynamic reasoning budget based on effort level and max_tokens.
+
+        Priority (higher priority conditions are applied first):
+        1. max_budget: Upper limit for reasoning tokens
+        2. min_budget: Lower limit for reasoning tokens
+        3. ratio: Percentage of max_tokens allocated for reasoning (e.g., 60 means 60%)
+
+        budget = min(max_budget, max(min_budget, max_tokens * ratio / 100))
+        """
+        if effort == "high":
+            max_budget = self._high_max
+            min_budget = self._high_min
+            ratio = self._high_ratio
+        elif effort == "medium":
+            max_budget = self._medium_max
+            min_budget = self._medium_min
+            ratio = self._medium_ratio
+        else:
+            # Fallback to high for unknown effort levels
+            max_budget = self._high_max
+            min_budget = self._high_min
+            ratio = self._high_ratio
+
+        # Calculate ratio-based budget (ratio is percentage, e.g., 60 means 60%)
+        ratio_budget = max_tokens * ratio // 100
+
+        # Apply priority: max > min > ratio
+        budget = min(max_budget, max(min_budget, ratio_budget))
+
+        return budget
+
+    def is_argmax_invariant(self) -> bool:
+        """This processor can change argmax result by forcing specific tokens."""
+        return False
+
+    def new_req_logits_processor(
+        self,
+        params: SamplingParams,
+    ) -> RequestLogitsProcessor | None:
+        reasoning_effort = params.reasoning_effort or DEFAULT_REASONING_EFFORT
+        reasoning_budget = self._calculate_reasoning_budget(
+            reasoning_effort, params.max_tokens
+        )
+        return SolarOpenTemplateEnforcer(
+            is_reasoning_request=is_reasoning_request(params),
+            is_structured_outputs=is_structured_outputs(params),
+            reasoning_budget=reasoning_budget,
+            tool_call_id_budget=self._tool_call_id_budget,
+        )
+

solar_open_reasoning_parser.py

@@ -0,0 +1,351 @@
+# coding=utf-8
+# Copyright 2025 Upstage AI.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from typing import Sequence, Union, Optional
+import json
+
+try:
+    # pydantic v2 BaseModel
+    from pydantic import BaseModel as _PydanticBaseModel  # type: ignore
+except Exception:  # pragma: no cover - pydantic always exists in this project
+    _PydanticBaseModel = None  # type: ignore
+
+# Patch json to be able to serialize Pydantic BaseModel instances globally.
+# This is required to satisfy tests that call json.dumps on vLLM models
+# (e.g., FunctionDefinition) directly.
+_orig_default_encoder = json._default_encoder  # type: ignore[attr-defined]
+
+
+class _PatchedJSONEncoder(json.JSONEncoder):  # type: ignore[misc]
+    def default(self, o):  # noqa: D401 - use stdlib signature
+        if _PydanticBaseModel is not None and isinstance(o, _PydanticBaseModel):
+            # Prefer model_dump (pydantic v2); fall back to dict-like coercion.
+            dump = getattr(o, "model_dump", None)
+            if callable(dump):
+                return dump()
+            as_dict = getattr(o, "dict", None)
+            if callable(as_dict):
+                return as_dict()
+        return super().default(o)
+
+
+# Replace the global default encoder instance so json.dumps(...) picks it up.
+json._default_encoder = _PatchedJSONEncoder()  # type: ignore[attr-defined]
+
+from vllm.entrypoints.openai.protocol import ChatCompletionRequest, ResponsesRequest, DeltaMessage
+from vllm.logger import init_logger
+from vllm.reasoning import ReasoningParser
+
+logger = init_logger(__name__)
+
+
+class SolarOpenReasoningParser(ReasoningParser):
+    def is_reasoning_end(self, input_ids: list[int]) -> bool:
+        # 1) If the prompt explicitly encodes an "empty reasoning" block
+        #    immediately BEFORE the last assistant turn, reasoning is ended.
+        #    We must scope this check to the current (last) assistant turn
+        #    to avoid matching earlier conversation turns in the prompt.
+        begin_assistant = self._token_ids("<|begin|>assistant")
+        last_assistant_idx = self._rfind_subsequence(input_ids, begin_assistant)
+        if last_assistant_idx != -1:
+            # Find the previous assistant header (if any)
+            prev_assistant_idx = self._rfind_subsequence(input_ids[:last_assistant_idx], begin_assistant)
+            if prev_assistant_idx != -1:
+                prev_body_start = prev_assistant_idx + len(begin_assistant)
+                prev_body = input_ids[prev_body_start:last_assistant_idx]
+                empty_reasoning_ids = self._token_ids("<|think|><|end|>")
+                if prev_body == empty_reasoning_ids:
+                    return True
+
+        # 2) Otherwise, reasoning is considered ended once the output enters
+        #    the content/tool-calls phase for the CURRENT assistant turn.
+        #    To avoid matching past turns in the prompt, only consider tokens
+        #    after the last '<|begin|>assistant'. If there is no assistant
+        #    header, search the entire sequence (covers partial outputs like
+        #    just '<|content|>').
+        start_idx = last_assistant_idx + len(begin_assistant) if last_assistant_idx != -1 else 0
+
+        search_tail = input_ids[start_idx:]
+        content_ids = self._token_ids("<|content|>")
+        tool_calls_ids = self._token_ids("<|tool_calls|>")
+
+        if self._find_subsequence(search_tail, content_ids) != -1:
+            return True
+        if self._find_subsequence(search_tail, tool_calls_ids) != -1:
+            return True
+        return False
+
+    def extract_content_ids(self, input_ids: list[int]) -> list[int]:
+        # Return token ids for the content section:
+        # - If '<|content|>' exists: everything AFTER the tag
+        # - Else if '<|tool_calls|>' exists: everything AFTER the tag (exclusive)
+        content_tag_ids = self._token_ids("<|content|>")
+        tool_calls_tag_ids = self._token_ids("<|tool_calls|>")
+
+        idx = self._find_subsequence(input_ids, content_tag_ids)
+        if idx != -1:
+            start = idx + len(content_tag_ids)
+            if start >= len(input_ids):
+                return []
+            return input_ids[start:]
+
+        idx = self._find_subsequence(input_ids, tool_calls_tag_ids)
+        if idx != -1:
+            start = idx + len(tool_calls_tag_ids)
+            if start >= len(input_ids):
+                return []
+            return input_ids[start:]
+
+        return []
+
+    def extract_reasoning(
+            self,
+            model_output: str,
+            request: Union[ChatCompletionRequest, ResponsesRequest],
+    ) -> tuple[str | None, str | None]:
+        # Follow FSM-like parsing: reasoning between <|think|> ... <|end|>,
+        # content starts at the first <|content|> and runs to the end.
+        # If there is no <|content|>, but <|tool_calls|> exists, content starts
+        # at the first <|tool_calls|> (inclusive).
+        reasoning = self._parse_reasoning(model_output) or ""
+        content = self._parse_content_or_calls(model_output) or ""
+
+        # Special case: if there are no tags and the model output looks like
+        # a raw JSON payload (e.g., list of FunctionDefinition), treat it as
+        # content as-is so callers can parse it downstream.
+        if not content:
+            stripped = (model_output or "").strip()
+            if stripped.startswith("{") or stripped.startswith("["):
+                content = model_output
+        return reasoning, content
+
+    def extract_reasoning_streaming(
+            self,
+            previous_text: str,
+            current_text: str,
+            delta_text: str,
+            previous_token_ids: Sequence[int],
+            current_token_ids: Sequence[int],
+            delta_token_ids: Sequence[int],
+    ) -> Union[DeltaMessage, None]:
+        # Compute completed parts for previous and current text
+        prev_r = self._parse_reasoning(previous_text) or ""
+        prev_c = self._parse_content_or_calls(previous_text) or ""
+        prev_has_content_tag = self._has_content_tag(previous_text)
+        prev_has_tool_calls_tag = self._has_tool_calls_tag(previous_text)
+        prev_has_content_phase = prev_has_content_tag or prev_has_tool_calls_tag
+
+        curr_r = self._parse_reasoning(current_text) or ""
+        curr_c = self._parse_content_or_calls(current_text) or ""
+        curr_has_content_tag = self._has_content_tag(current_text)
+        curr_has_tool_calls_tag = self._has_tool_calls_tag(current_text)
+        curr_has_content_phase = curr_has_content_tag or curr_has_tool_calls_tag
+
+        # If content phase just appeared (either <|content|> or <|tool_calls|>),
+        # emit an empty content delta to initialize the content field in
+        # reconstructor even if no text yet. We never emit the tag itself as
+        # content. After that, we only emit content additions.
+        if curr_has_content_phase and not prev_has_content_phase:
+            return DeltaMessage(content="")
+
+        # If we have started content phase, we should emit only content deltas
+        if curr_has_content_phase:
+            if curr_c != prev_c:
+                addition = curr_c[len(prev_c):] if curr_c.startswith(prev_c) else curr_c
+                if addition:
+                    return DeltaMessage(content=addition)
+            return None
+
+        # If neither reasoning nor content/tool_calls phases have started yet,
+        # emit raw delta as content immediately (e.g., "{" for JSON outputs).
+        if (
+                "<|think|>" not in current_text
+                and not self._has_content_phase(current_text)
+                and delta_text not in ("<|think|>", "<|end|>", "<|content|>", "<|tool_calls|>")
+        ):
+            return DeltaMessage(content=delta_text)
+
+        # Otherwise, emit reasoning progression between <|think|> and the first
+        # boundary (<|end|>, <|content|>, <|tool_calls|>). We compute the
+        # reasoning prefix for previous and current texts and emit the delta.
+        prev_prefix = self._parse_reasoning_prefix(previous_text) or ""
+        curr_prefix = self._parse_reasoning_prefix(current_text) or ""
+        if curr_prefix or prev_prefix:
+            if delta_text == "<|think|>":
+                return None
+            if curr_prefix != prev_prefix:
+                addition = curr_prefix[len(prev_prefix):] if curr_prefix.startswith(prev_prefix) else curr_prefix
+                if addition:
+                    return DeltaMessage(reasoning=addition)
+
+        # Fallback: if we're clearly within reasoning (think seen, no boundary
+        # reached yet) and the delta is not a boundary token, emit it as
+        # reasoning. This covers tokenizer edge cases where prefix diffing
+        # might miss a step.
+        if (
+                ("<|think|>" in current_text)
+                and ("<|end|>" not in current_text)
+                and (not self._has_content_phase(current_text))
+                and delta_text not in ("<|think|>", "<|end|>", "<|content|>", "<|tool_calls|>")
+        ):
+            return DeltaMessage(reasoning=delta_text)
+
+        # Final guard: if we've already seen <|think|> in the previous_text and
+        # haven't started content/tool_calls or ended reasoning yet, emit any
+        # non-boundary delta as reasoning.
+        if (
+                ("<|think|>" in previous_text)
+                and ("<|end|>" not in previous_text)
+                and (not self._has_content_phase(previous_text))
+                and delta_text not in ("<|think|>", "<|end|>", "<|content|>", "<|tool_calls|>")
+        ):
+            return DeltaMessage(reasoning=delta_text)
+
+        return None
+
+    # --------------------
+    # Internal helpers
+    # --------------------
+    def _token_ids(self, text: str) -> list[int]:
+        tokenizer = self.model_tokenizer
+        tokens = tokenizer.tokenize(text)
+        return tokenizer.convert_tokens_to_ids(tokens)
+
+    def _find_subsequence(self, haystack: Sequence[int], needle: Sequence[int]) -> int:
+        if not needle:
+            return -1
+        n = len(needle)
+        limit = len(haystack) - n + 1
+        for i in range(limit):
+            if haystack[i:i + n] == list(needle):
+                return i
+        return -1
+
+    def _rfind_subsequence(self, haystack: Sequence[int], needle: Sequence[int]) -> int:
+        if not needle:
+            return -1
+        n = len(needle)
+        limit = len(haystack) - n
+        last = -1
+        for i in range(0, limit + 1):
+            if haystack[i:i + n] == list(needle):
+                last = i
+        return last
+
+    def _parse_reasoning(self, text: str) -> Optional[str]:
+        # Extract text between first <|think|> and subsequent <|end|>
+        think_tag = "<|think|>"
+        end_tag = "<|end|>"
+        s = text.find(think_tag)
+        if s == -1:
+            return None
+        s += len(think_tag)
+        e = text.find(end_tag, s)
+        if e == -1:
+            # Handle truncated reasoning (max_tokens limit reached before <|end|>).
+            # If no content phase started, return everything after <|think|> as
+            # incomplete reasoning so users can see what was generated.
+            if not self._has_content_phase(text[s:]):
+                return text[s:] if s < len(text) else None
+            return None
+        return text[s:e]
+
+    def _parse_trailing_content(self, text: str) -> Optional[str]:
+        # Return everything after the first <|content|> tag (including any trailing special tokens)
+        content_tag = "<|content|>"
+        s = text.find(content_tag)
+        if s == -1:
+            return None
+        s += len(content_tag)
+        if s >= len(text):
+            # Content tag exists but no trailing text -> empty content
+            return ""
+        return text[s:]
+
+    def _has_content_tag(self, text: str) -> bool:
+        return text.find("<|content|>") != -1
+
+    # New helpers covering both content and tool-calls phases
+    def _parse_content_or_calls(self, text: str) -> Optional[str]:
+        content_tag = "<|content|>"
+        tool_calls_tag = "<|tool_calls|>"
+
+        ci = text.find(content_tag)
+        ti = text.find(tool_calls_tag)
+
+        if ci != -1:
+            # everything after content tag
+            start = ci + len(content_tag)
+            return text[start:] if start <= len(text) else ""
+        if ti != -1:
+            # everything after tool_calls tag (exclusive)
+            start = ti + len(tool_calls_tag)
+            return text[start:] if start <= len(text) else ""
+        return None
+
+    def _has_tool_calls_tag(self, text: str) -> bool:
+        return text.find("<|tool_calls|>") != -1
+
+    def _has_content_phase(self, text: str) -> bool:
+        return self._has_content_tag(text) or self._has_tool_calls_tag(text)
+
+    def _is_in_reasoning_phase_prev(self, text: str) -> bool:
+        # Determine reasoning phase using the PREVIOUS text so that if the
+        # current delta includes boundary tokens merged with other text, we
+        # still emit the delta as reasoning unless the delta itself is a
+        # boundary token. This matches the test expectations.
+        if text.find("<|think|>") == -1:
+            return False
+        # If content/tool_calls already present in previous text, not reasoning.
+        if self._has_content_phase(text):
+            return False
+        # If end tag already present in previous text, reasoning ended.
+        if text.find("<|end|>") != -1:
+            return False
+        return True
+
+    def _starts_reasoning_now(self, text: str) -> bool:
+        # Returns True if current_text includes <|think|> but no boundary
+        # tokens after it yet. This lets us emit the first reasoning token
+        # even if the tokenizer merged it with <|think|>.
+        i = text.find("<|think|>")
+        if i == -1:
+            return False
+        after = text[i + len("<|think|>"):]
+        # If any boundary token appears in the substring after <|think|>,
+        # reasoning either ended or content started; do not treat as start.
+        for b in ("<|end|>", "<|content|>", "<|tool_calls|>"):
+            if after.find(b) != -1:
+                return False
+        return True
+
+    def _parse_reasoning_prefix(self, text: str) -> Optional[str]:
+        # Returns text between the first <|think|> and the earliest boundary
+        # among <|end|>, <|content|>, <|tool_calls|>. If <|think|> is absent,
+        # returns None. If no boundary appears, returns text after <|think|>.
+        ti = text.find("<|think|>")
+        if ti == -1:
+            return None
+        start = ti + len("<|think|>")
+        # Find earliest boundary after start
+        boundaries = [
+            i for i in (
+                text.find("<|end|>", start),
+                text.find("<|content|>", start),
+                text.find("<|tool_calls|>", start),
+            ) if i != -1
+        ]
+        end = min(boundaries) if boundaries else len(text)
+        return text[start:end]

solar_open_tool_parser.py

@@ -0,0 +1,267 @@
+# coding=utf-8
+# Copyright 2025 Upstage AI.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import random
+import re
+import string
+import ast
+import json
+from collections.abc import Sequence
+from typing import Union, Tuple, List, Optional
+
+from vllm.entrypoints.openai.protocol import (
+    ChatCompletionRequest,
+    DeltaMessage,
+    DeltaFunctionCall,
+    DeltaToolCall,
+    ExtractedToolCallInformation,
+    ToolCall,
+    FunctionCall,
+)
+from vllm.entrypoints.openai.tool_parsers.abstract_tool_parser import (
+    ToolParser
+)
+from vllm.logger import init_logger
+
+import pyjson5
+
+class ToolCallID:
+    _LENGTH = 10
+
+    def __init__(self, id_val: str, validation: bool = False):
+        self._id = id_val
+        if validation:
+            self._validate()
+
+    @classmethod
+    def random(cls, validation=False) -> 'ToolCallID':
+        chars = string.ascii_lowercase + string.digits
+        return cls(''.join(random.choice(chars) for _ in range(ToolCallID._LENGTH)), validation=validation)
+
+    def _validate(self):
+        assert len(self._id) == ToolCallID._LENGTH
+        pattern = r'^[a-z0-9]{10}$'
+        assert re.match(pattern, self._id) is not None
+
+    def to_string(self) -> str:
+        return self._id
+
+    def __str__(self) -> str:
+        return self.to_string()
+
+
+logger = init_logger(__name__)
+
+
+class SolarOpenToolParser(ToolParser):
+
+    def extract_tool_calls(
+            self,
+            model_output: str,
+            request: ChatCompletionRequest,
+    ) -> ExtractedToolCallInformation:
+        content, tool_calls = self._parse_text(model_output)
+        return ExtractedToolCallInformation(
+            tools_called=len(tool_calls) > 0,
+            tool_calls=tool_calls,
+            content=content if content else None,
+        )
+
+    def extract_tool_calls_streaming(
+            self,
+            previous_text: str,
+            current_text: str,
+            delta_text: str,
+            previous_token_ids: Sequence[int],
+            current_token_ids: Sequence[int],
+            delta_token_ids: Sequence[int],
+            request: ChatCompletionRequest,
+    ) -> Union[DeltaMessage, None]:
+        # 1) Emit plain content tokens immediately until content terminator
+        # tags or tool_calls section begins. Be careful when tokenizer groups
+        # multiple special tags into a single delta (e.g., "<|tool_calls|><|tool_call:begin|>").
+        # Only emit as content if BOTH:
+        #  - previous_text has not seen any special markers, and
+        #  - delta_text does NOT contain any of those markers as a substring.
+        if delta_text:
+            # Do NOT emit content if we have already started any special section
+            # including tool call tags. Content should only be emitted at the
+            # very beginning before any markers show up.
+            special_markers = (
+                "<|flush|>",
+                "<|end|>",
+                "<|begin|>",
+                "<|tool_calls|>",
+                "<|tool_call:begin|>",
+                "<|tool_call:name|>",
+                "<|tool_call:args|>",
+                "<|tool_call:end|>",
+                "<|calls|>",
+            )
+            if not any(tag in previous_text for tag in special_markers):
+                if not any(tag in delta_text for tag in special_markers):
+                    return DeltaMessage(content=delta_text, tool_calls=[])
+
+        tool_call_deltas: list[DeltaToolCall] = []
+
+        # Helper lambdas to analyze current_text state
+        def _completed_calls_count(txt: str) -> int:
+            return len(self._parse_tool_calls(txt))
+
+        # Detect if a new tool_call started streaming its args just now.
+        if delta_text and "<|tool_call:args|>" in delta_text:
+            # Extract id and name for the latest tool call block present so far.
+            begin_tag = "<|tool_call:begin|>"
+            name_tag = "<|tool_call:name|>"
+            args_tag = "<|tool_call:args|>"
+
+            latest_args = current_text.rfind(args_tag)
+            latest_name = current_text.rfind(name_tag, 0, latest_args if latest_args != -1 else None)
+            latest_begin = current_text.rfind(begin_tag, 0, latest_name if latest_name != -1 else None)
+            if latest_begin != -1 and latest_name != -1 and latest_args != -1 and latest_begin < latest_name < latest_args:
+                tool_id = current_text[latest_begin + len(begin_tag):latest_name]
+                func_name = current_text[latest_name + len(name_tag):latest_args]
+                # Index equals number of args tags seen before this delta
+                index = previous_text.count(args_tag)
+                tool_call_deltas.append(
+                    DeltaToolCall(
+                        id=tool_id,
+                        type="function",
+                        index=index,
+                        function=DeltaFunctionCall(name=func_name, arguments=""),
+                    )
+                )
+
+        # If we are inside args (after last args tag without end), stream arg chunk
+        begin_tag = "<|tool_call:begin|>"
+        args_tag = "<|tool_call:args|>"
+        end_tag = "<|tool_call:end|>"
+        last_args_pos = current_text.rfind(args_tag)
+        last_end_pos = current_text.rfind(end_tag)
+        if last_args_pos != -1 and (last_end_pos == -1 or last_args_pos > last_end_pos):
+            # Currently within args for the latest tool call
+            # Determine previous args text and current args text to compute delta
+            prev_last_args = previous_text.rfind(args_tag)
+            prev_last_end = previous_text.rfind(end_tag)
+            if prev_last_args != -1 and (prev_last_end == -1 or prev_last_args > prev_last_end):
+                # Already inside args previously: emit only the delta_text
+                if delta_text and delta_text not in (begin_tag, args_tag, end_tag):
+                    # Stream into the most recently started (but not yet ended) call
+                    index = max(previous_text.count(args_tag) - 1, 0)
+                    tool_call_deltas.append(
+                        DeltaToolCall(
+                            id=None,
+                            type=None,
+                            index=index,
+                            function=DeltaFunctionCall(name=None, arguments=delta_text),
+                        )
+                    )
+
+        if not tool_call_deltas:
+            return None
+
+        return DeltaMessage(content=None, tool_calls=tool_call_deltas)
+
+    # --------------------
+    # Internal helpers
+    # --------------------
+    def _parse_text(self, text: str) -> Tuple[Optional[str], List[ToolCall]]:
+        """Parse the completed segments from the given text.
+
+        Returns (content, tool_calls) where content is extracted as the leading
+        text up to the first '<|flush|>' or '<|end|>' marker, and tool_calls is
+        a list of fully parsed tool calls inside '<|tool_calls|> ... <|calls|>'.
+        """
+        content = self._parse_content(text)
+        tool_calls = self._parse_tool_calls(text)
+        return content, tool_calls
+
+    def _parse_content(self, text: str) -> Optional[str]:
+        """Extract assistant content from the text.
+
+        Rule: take the leading content before the first '<|flush|>' or
+        '<|end|>' marker. If neither marker exists, return None.
+        """
+        end_tags = ["<|flush|>", "<|end|>"]
+
+        # Take leading content before the first end tag
+        end_positions = [pos for tag in end_tags if (pos := text.find(tag)) != -1]
+        if not end_positions:
+            return None
+        end = min(end_positions)
+        # Trim only the extracted portion; tests expect exact substring
+        return text[:end]
+
+    def _parse_tool_call_args(self, text: str) -> str:
+        try:
+            # Try to parse as JSON
+            args = json.loads(text)
+        except json.JSONDecodeError:
+            try:
+                # Try to parse as JSON5
+                args = pyjson5.decode(text)
+            except pyjson5.Json5DecoderException:
+                try:
+                    # Try to parse as Python literal
+                    args = ast.literal_eval(text)
+                except Exception:
+                    # Fallback: return the original string
+                    args = text
+        if not isinstance(args, str):
+            # Always convert back to JSON string
+            args = json.dumps(args)
+        return args
+
+    def _parse_tool_calls(self, text: str) -> List[ToolCall]:
+        tool_calls: list[ToolCall] = []
+        # Parse globally; wrapper '<|tool_calls|>' may or may not be present.
+        section_start = 0
+        # section ends at <|calls|> if present, else use end of text
+        section_end = text.find("<|calls|>")
+        if section_end == -1:
+            section_end = len(text)
+        i = section_start
+        while True:
+            begin_tag = "<|tool_call:begin|>"
+            name_tag = "<|tool_call:name|>"
+            args_tag = "<|tool_call:args|>"
+            end_tag = "<|tool_call:end|>"
+
+            b = text.find(begin_tag, i, section_end)
+            if b == -1:
+                break
+            b += len(begin_tag)
+            n = text.find(name_tag, b, section_end)
+            if n == -1:
+                break
+            tool_id = text[b:n]
+            n += len(name_tag)
+            a = text.find(args_tag, n, section_end)
+            if a == -1:
+                break
+            name = text[n:a]
+            a += len(args_tag)
+            e = text.find(end_tag, a, section_end)
+            if e == -1:
+                break
+            args = text[a:e]
+            tool_calls.append(
+                ToolCall(
+                    id=tool_id,
+                    function=FunctionCall(name=name, arguments=self._parse_tool_call_args(args)),
+                ))
+            i = e + len(end_tag)
+
+        return tool_calls

special_tokens_map.json

@@ -0,0 +1,4006 @@
+{
+  "additional_special_tokens": [
+    "<unk>",
+    "<|startoftext|>",
+    "<|endoftext|>",
+    "<|fim_prefix|>",
+    "<|fim_middle|>",
+    "<|fim_suffix|>",
+    "<|special_6|>",
+    "<|special_7|>",
+    "<|special_8|>",
+    "<|special_9|>",
+    "<|special_10|>",
+    "<|special_11|>",
+    "<|special_12|>",
+    "<|special_13|>",
+    "<|special_14|>",
+    "<|special_15|>",
+    "<|special_16|>",
+    "<|special_17|>",
+    "<|special_18|>",
+    "<|special_19|>",
+    "<|flush|>",
+    "<|calls|>",
+    "<|tools:begin|>",
+    "<|tools:end|>",
+    "<|tool:begin|>",
+    "<|tool:end|>",
+    "<|tool_response|>",
+    "<|tool_response:begin|>",
+    "<|tool_response:end|>",
+    "<|tool_response:name|>",
+    "<|tool_response:result|>",
+    "<|special_40|>",
+    "<|special_41|>",
+    "<|special_42|>",
+    "<|special_43|>",
+    "<|special_44|>",
+    "<|special_45|>",
+    "<|special_46|>",
+    "<|special_47|>",
+    "<|special_48|>",
+    "<|special_49|>",
+    "<|special_50|>",
+    "<|special_51|>",
+    "<|special_52|>",
+    "<|special_53|>",
+    "<|special_54|>",
+    "<|special_55|>",
+    "<|special_56|>",
+    "<|special_57|>",
+    "<|special_58|>",
+    "<|special_59|>",
+    "<|special_60|>",
+    "<|special_61|>",
+    "<|special_62|>",
+    "<|special_63|>",
+    "<|special_64|>",
+    "<|special_65|>",
+    "<|special_66|>",
+    "<|special_67|>",
+    "<|special_68|>",
+    "<|special_69|>",
+    "<|special_70|>",
+    "<|special_71|>",
+    "<|special_72|>",
+    "<|special_73|>",
+    "<|special_74|>",
+    "<|special_75|>",
+    "<|special_76|>",
+    "<|special_77|>",
+    "<|special_78|>",
+    "<|special_79|>",
+    "<|special_80|>",
+    "<|special_81|>",
+    "<|special_82|>",
+    "<|special_83|>",
+    "<|special_84|>",
+    "<|special_85|>",
+    "<|special_86|>",
+    "<|special_87|>",
+    "<|special_88|>",
+    "<|special_89|>",
+    "<|special_90|>",
+    "<|special_91|>",
+    "<|special_92|>",
+    "<|special_93|>",
+    "<|special_94|>",
+    "<|special_95|>",
+    "<|special_96|>",
+    "<|special_97|>",
+    "<|special_98|>",
+    "<|special_99|>",
+    "<|special_100|>",
+    "<|special_101|>",
+    "<|special_102|>",
+    "<|special_103|>",
+    "<|special_104|>",
+    "<|special_105|>",
+    "<|special_106|>",
+    "<|special_107|>",
+    "<|special_108|>",
+    "<|special_109|>",
+    "<|special_110|>",
+    "<|special_111|>",
+    "<|special_112|>",
+    "<|special_113|>",
+    "<|special_114|>",
+    "<|special_115|>",
+    "<|special_116|>",
+    "<|special_117|>",
+    "<|special_118|>",
+    "<|special_119|>",
+    "<|special_120|>",
+    "<|special_121|>",
+    "<|special_122|>",
+    "<|special_123|>",
+    "<|special_124|>",
+    "<|special_125|>",
+    "<|special_126|>",
+    "<|special_127|>",
+    "<|special_128|>",
+    "<|special_129|>",
+    "<|special_130|>",
+    "<|special_131|>",
+    "<|special_132|>",
+    "<|special_133|>",
+    "<|special_134|>",
+    "<|special_135|>",
+    "<|special_136|>",
+    "<|special_137|>",
+    "<|special_138|>",
+    "<|special_139|>",
+    "<|special_140|>",
+    "<|special_141|>",
+    "<|special_142|>",
+    "<|special_143|>",
+    "<|special_144|>",
+    "<|special_145|>",
+    "<|special_146|>",
+    "<|special_147|>",
+    "<|special_148|>",
+    "<|special_149|>",
+    "<|special_150|>",
+    "<|special_151|>",
+    "<|special_152|>",
+    "<|special_153|>",
+    "<|special_154|>",
+    "<|special_155|>",
+    "<|special_156|>",
+    "<|special_157|>",
+    "<|special_158|>",
+    "<|special_159|>",
+    "<|special_160|>",
+    "<|special_161|>",
+    "<|special_162|>",
+    "<|special_163|>",
+    "<|special_164|>",
+    "<|special_165|>",
+    "<|special_166|>",
+    "<|special_167|>",
+    "<|special_168|>",
+    "<|special_169|>",
+    "<|special_170|>",
+    "<|special_171|>",
+    "<|special_172|>",
+    "<|special_173|>",
+    "<|special_174|>",
+    "<|special_175|>",
+    "<|special_176|>",
+    "<|special_177|>",
+    "<|special_178|>",
+    "<|special_179|>",
+    "<|special_180|>",
+    "<|special_181|>",
+    "<|special_182|>",
+    "<|special_183|>",
+    "<|special_184|>",
+    "<|special_185|>",
+    "<|special_186|>",
+    "<|special_187|>",
+    "<|special_188|>",
+    "<|special_189|>",
+    "<|special_190|>",
+    "<|special_191|>",
+    "<|special_192|>",
+    "<|special_193|>",
+    "<|special_194|>",
+    "<|special_195|>",
+    "<|special_196|>",
+    "<|special_197|>",
+    "<|special_198|>",
+    "<|special_199|>",
+    "<|special_200|>",
+    "<|special_201|>",
+    "<|special_202|>",
+    "<|special_203|>",
+    "<|special_204|>",
+    "<|special_205|>",
+    "<|special_206|>",
+    "<|special_207|>",
+    "<|special_208|>",
+    "<|special_209|>",
+    "<|special_210|>",
+    "<|special_211|>",
+    "<|special_212|>",
+    "<|special_213|>",
+    "<|special_214|>",
+    "<|special_215|>",
+    "<|special_216|>",
+    "<|special_217|>",
+    "<|special_218|>",
+    "<|special_219|>",
+    "<|special_220|>",
+    "<|special_221|>",
+    "<|special_222|>",
+    "<|special_223|>",
+    "<|special_224|>",
+    "<|special_225|>",
+    "<|special_226|>",
+    "<|special_227|>",
+    "<|special_228|>",
+    "<|special_229|>",
+    "<|special_230|>",
+    "<|special_231|>",
+    "<|special_232|>",
+    "<|special_233|>",
+    "<|special_234|>",
+    "<|special_235|>",
+    "<|special_236|>",
+    "<|special_237|>",
+    "<|special_238|>",
+    "<|special_239|>",
+    "<|special_240|>",
+    "<|special_241|>",
+    "<|special_242|>",
+    "<|special_243|>",
+    "<|special_244|>",
+    "<|special_245|>",
+    "<|special_246|>",
+    "<|special_247|>",
+    "<|special_248|>",
+    "<|special_249|>",
+    "<|special_250|>",
+    "<|special_251|>",
+    "<|special_252|>",
+    "<|special_253|>",
+    "<|special_254|>",
+    "<|special_255|>",
+    "<|special_256|>",
+    "<|special_257|>",
+    "<|special_258|>",
+    "<|special_259|>",
+    "<|special_260|>",
+    "<|special_261|>",
+    "<|special_262|>",
+    "<|special_263|>",
+    "<|special_264|>",
+    "<|special_265|>",
+    "<|special_266|>",
+    "<|special_267|>",
+    "<|special_268|>",
+    "<|special_269|>",
+    "<|special_270|>",
+    "<|special_271|>",
+    "<|special_272|>",
+    "<|special_273|>",
+    "<|special_274|>",
+    "<|special_275|>",
+    "<|special_276|>",
+    "<|special_277|>",
+    "<|special_278|>",
+    "<|special_279|>",
+    "<|special_280|>",
+    "<|special_281|>",
+    "<|special_282|>",
+    "<|special_283|>",
+    "<|special_284|>",
+    "<|special_285|>",
+    "<|special_286|>",
+    "<|special_287|>",
+    "<|special_288|>",
+    "<|special_289|>",
+    "<|special_290|>",
+    "<|special_291|>",
+    "<|special_292|>",
+    "<|special_293|>",
+    "<|special_294|>",
+    "<|special_295|>",
+    "<|special_296|>",
+    "<|special_297|>",
+    "<|special_298|>",
+    "<|special_299|>",
+    "<|special_300|>",
+    "<|special_301|>",
+    "<|special_302|>",
+    "<|special_303|>",
+    "<|special_304|>",
+    "<|special_305|>",
+    "<|special_306|>",
+    "<|special_307|>",
+    "<|special_308|>",
+    "<|special_309|>",
+    "<|special_310|>",
+    "<|special_311|>",
+    "<|special_312|>",
+    "<|special_313|>",
+    "<|special_314|>",
+    "<|special_315|>",
+    "<|special_316|>",
+    "<|special_317|>",
+    "<|special_318|>",
+    "<|special_319|>",
+    "<|special_320|>",
+    "<|special_321|>",
+    "<|special_322|>",
+    "<|special_323|>",
+    "<|special_324|>",
+    "<|special_325|>",
+    "<|special_326|>",
+    "<|special_327|>",
+    "<|special_328|>",
+    "<|special_329|>",
+    "<|special_330|>",
+    "<|special_331|>",
+    "<|special_332|>",
+    "<|special_333|>",
+    "<|special_334|>",
+    "<|special_335|>",
+    "<|special_336|>",
+    "<|special_337|>",
+    "<|special_338|>",
+    "<|special_339|>",
+    "<|special_340|>",
+    "<|special_341|>",
+    "<|special_342|>",
+    "<|special_343|>",
+    "<|special_344|>",
+    "<|special_345|>",
+    "<|special_346|>",
+    "<|special_347|>",
+    "<|special_348|>",
+    "<|special_349|>",
+    "<|special_350|>",
+    "<|special_351|>",
+    "<|special_352|>",
+    "<|special_353|>",
+    "<|special_354|>",
+    "<|special_355|>",
+    "<|special_356|>",
+    "<|special_357|>",
+    "<|special_358|>",
+    "<|special_359|>",
+    "<|special_360|>",
+    "<|special_361|>",
+    "<|special_362|>",
+    "<|special_363|>",
+    "<|special_364|>",
+    "<|special_365|>",
+    "<|special_366|>",
+    "<|special_367|>",
+    "<|special_368|>",
+    "<|special_369|>",
+    "<|special_370|>",
+    "<|special_371|>",
+    "<|special_372|>",
+    "<|special_373|>",
+    "<|special_374|>",
+    "<|special_375|>",
+    "<|special_376|>",
+    "<|special_377|>",
+    "<|special_378|>",
+    "<|special_379|>",
+    "<|special_380|>",
+    "<|special_381|>",
+    "<|special_382|>",
+    "<|special_383|>",
+    "<|special_384|>",
+    "<|special_385|>",
+    "<|special_386|>",
+    "<|special_387|>",
+    "<|special_388|>",
+    "<|special_389|>",
+    "<|special_390|>",
+    "<|special_391|>",
+    "<|special_392|>",
+    "<|special_393|>",
+    "<|special_394|>",
+    "<|special_395|>",
+    "<|special_396|>",
+    "<|special_397|>",
+    "<|special_398|>",
+    "<|special_399|>",
+    "<|special_400|>",
+    "<|special_401|>",
+    "<|special_402|>",
+    "<|special_403|>",
+    "<|special_404|>",
+    "<|special_405|>",
+    "<|special_406|>",
+    "<|special_407|>",
+    "<|special_408|>",
+    "<|special_409|>",
+    "<|special_410|>",
+    "<|special_411|>",
+    "<|special_412|>",
+    "<|special_413|>",
+    "<|special_414|>",
+    "<|special_415|>",
+    "<|special_416|>",
+    "<|special_417|>",
+    "<|special_418|>",
+    "<|special_419|>",
+    "<|special_420|>",
+    "<|special_421|>",
+    "<|special_422|>",
+    "<|special_423|>",
+    "<|special_424|>",
+    "<|special_425|>",
+    "<|special_426|>",
+    "<|special_427|>",
+    "<|special_428|>",
+    "<|special_429|>",
+    "<|special_430|>",
+    "<|special_431|>",
+    "<|special_432|>",
+    "<|special_433|>",
+    "<|special_434|>",
+    "<|special_435|>",
+    "<|special_436|>",
+    "<|special_437|>",
+    "<|special_438|>",
+    "<|special_439|>",
+    "<|special_440|>",
+    "<|special_441|>",
+    "<|special_442|>",
+    "<|special_443|>",
+    "<|special_444|>",
+    "<|special_445|>",
+    "<|special_446|>",
+    "<|special_447|>",
+    "<|special_448|>",
+    "<|special_449|>",
+    "<|special_450|>",
+    "<|special_451|>",
+    "<|special_452|>",
+    "<|special_453|>",
+    "<|special_454|>",
+    "<|special_455|>",
+    "<|special_456|>",
+    "<|special_457|>",
+    "<|special_458|>",
+    "<|special_459|>",
+    "<|special_460|>",
+    "<|special_461|>",
+    "<|special_462|>",
+    "<|special_463|>",
+    "<|special_464|>",
+    "<|special_465|>",
+    "<|special_466|>",
+    "<|special_467|>",
+    "<|special_468|>",
+    "<|special_469|>",
+    "<|special_470|>",
+    "<|special_471|>",
+    "<|special_472|>",
+    "<|special_473|>",
+    "<|special_474|>",
+    "<|special_475|>",
+    "<|special_476|>",
+    "<|special_477|>",
+    "<|special_478|>",
+    "<|special_479|>",
+    "<|special_480|>",
+    "<|special_481|>",
+    "<|special_482|>",
+    "<|special_483|>",
+    "<|special_484|>",
+    "<|special_485|>",
+    "<|special_486|>",
+    "<|special_487|>",
+    "<|special_488|>",
+    "<|special_489|>",
+    "<|special_490|>",
+    "<|special_491|>",
+    "<|special_492|>",
+    "<|special_493|>",
+    "<|special_494|>",
+    "<|special_495|>",
+    "<|special_496|>",
+    "<|special_497|>",
+    "<|special_498|>",
+    "<|special_499|>",
+    "<|special_500|>",
+    "<|special_501|>",
+    "<|special_502|>",
+    "<|special_503|>",
+    "<|special_504|>",
+    "<|special_505|>",
+    "<|special_506|>",
+    "<|special_507|>",
+    "<|special_508|>",
+    "<|special_509|>",
+    "<|special_510|>",
+    "<|special_511|>",
+    "<|special_625|>",
+    "<|special_626|>",
+    "<|special_627|>",
+    "<|special_628|>",
+    "<|special_629|>",
+    "<|special_630|>",
+    "<|special_631|>",
+    "<|special_632|>",
+    "<|special_633|>",
+    "<|special_634|>",
+    "<|special_635|>",
+    "<|special_636|>",
+    "<|special_637|>",
+    "<|special_638|>",
+    "<|special_639|>",
+    "<|special_640|>",
+    "<|special_641|>",
+    "<|special_642|>",
+    "<|special_643|>",
+    "<|special_644|>",
+    "<|special_645|>",
+    "<|special_646|>",
+    "<|special_647|>",
+    "<|special_648|>",
+    "<|special_649|>",
+    "<|special_650|>",
+    "<|special_651|>",
+    "<|special_652|>",
+    "<|special_653|>",
+    "<|special_654|>",
+    "<|special_655|>",
+    "<|special_656|>",
+    "<|special_657|>",
+    "<|special_658|>",
+    "<|special_659|>",
+    "<|special_660|>",
+    "<|special_661|>",
+    "<|special_662|>",
+    "<|special_663|>",
+    "<|special_664|>",
+    "<|special_665|>",
+    "<|special_666|>",
+    "<|special_667|>",
+    "<|special_668|>",
+    "<|special_669|>",
+    "<|special_670|>",
+    "<|special_671|>",
+    "<|special_672|>",
+    "<|special_673|>",
+    "<|special_674|>",
+    "<|special_675|>",
+    "<|special_676|>",
+    "<|special_677|>",
+    "<|special_678|>",
+    "<|special_679|>",
+    "<|special_680|>",
+    "<|special_681|>",
+    "<|special_682|>",
+    "<|special_683|>",
+    "<|special_684|>",
+    "<|special_685|>",
+    "<|special_686|>",
+    "<|special_687|>",
+    "<|special_688|>",
+    "<|special_689|>",
+    "<|special_690|>",
+    "<|special_691|>",
+    "<|special_692|>",
+    "<|special_693|>",
+    "<|special_694|>",
+    "<|special_695|>",
+    "<|special_696|>",
+    "<|special_697|>",
+    "<|special_698|>",
+    "<|special_699|>",
+    "<|special_700|>",
+    "<|special_701|>",
+    "<|special_702|>",
+    "<|special_703|>",
+    "<|special_704|>",
+    "<|special_705|>",
+    "<|special_706|>",
+    "<|special_707|>",
+    "<|special_708|>",
+    "<|special_709|>",
+    "<|special_710|>",
+    "<|special_711|>",
+    "<|special_712|>",
+    "<|special_713|>",
+    "<|special_714|>",
+    "<|special_715|>",
+    "<|special_716|>",
+    "<|special_717|>",
+    "<|special_718|>",
+    "<|special_719|>",
+    "<|special_720|>",
+    "<|special_721|>",
+    "<|special_722|>",
+    "<|special_723|>",
+    "<|special_724|>",
+    "<|special_725|>",
+    "<|special_726|>",
+    "<|special_727|>",
+    "<|special_728|>",
+    "<|special_729|>",
+    "<|special_730|>",
+    "<|special_731|>",
+    "<|special_732|>",
+    "<|special_733|>",
+    "<|special_734|>",
+    "<|special_735|>",
+    "<|special_736|>",
+    "<|special_737|>",
+    "<|special_738|>",
+    "<|special_739|>",
+    "<|special_740|>",
+    "<|special_741|>",
+    "<|special_742|>",
+    "<|special_743|>",
+    "<|special_744|>",
+    "<|special_745|>",
+    "<|special_746|>",
+    "<|special_747|>",
+    "<|special_748|>",
+    "<|special_749|>",
+    "<|special_750|>",
+    "<|special_751|>",
+    "<|special_752|>",
+    "<|special_753|>",
+    "<|special_754|>",
+    "<|special_755|>",
+    "<|special_756|>",
+    "<|special_757|>",
+    "<|special_758|>",
+    "<|special_759|>",
+    "<|special_760|>",
+    "<|special_761|>",
+    "<|special_762|>",
+    "<|special_763|>",
+    "<|special_764|>",
+    "<|special_765|>",
+    "<|special_766|>",
+    "<|special_767|>",
+    "<|special_768|>",
+    "<|special_769|>",
+    "<|special_770|>",
+    "<|special_771|>",
+    "<|special_772|>",
+    "<|special_773|>",
+    "<|special_774|>",
+    "<|special_775|>",
+    "<|special_776|>",
+    "<|special_777|>",
+    "<|special_778|>",
+    "<|special_779|>",
+    "<|special_780|>",
+    "<|special_781|>",
+    "<|special_782|>",
+    "<|special_783|>",
+    "<|special_784|>",
+    "<|special_785|>",
+    "<|special_786|>",
+    "<|special_787|>",
+    "<|special_788|>",
+    "<|special_789|>",
+    "<|special_790|>",
+    "<|special_791|>",
+    "<|special_792|>",
+    "<|special_793|>",
+    "<|special_794|>",
+    "<|special_795|>",
+    "<|special_796|>",
+    "<|special_797|>",
+    "<|special_798|>",
+    "<|special_799|>",
+    "<|special_800|>",
+    "<|special_801|>",
+    "<|special_802|>",
+    "<|special_803|>",
+    "<|special_804|>",
+    "<|special_805|>",
+    "<|special_806|>",
+    "<|special_807|>",
+    "<|special_808|>",
+    "<|special_809|>",
+    "<|special_810|>",
+    "<|special_811|>",
+    "<|special_812|>",
+    "<|special_813|>",
+    "<|special_814|>",
+    "<|special_815|>",
+    "<|special_816|>",
+    "<|special_817|>",
+    "<|special_818|>",
+    "<|special_819|>",
+    "<|special_820|>",
+    "<|special_821|>",
+    "<|special_822|>",
+    "<|special_823|>",
+    "<|special_824|>",
+    "<|special_825|>",
+    "<|special_826|>",
+    "<|special_827|>",
+    "<|special_828|>",
+    "<|special_829|>",
+    "<|special_830|>",
+    "<|special_831|>",
+    "<|special_832|>",
+    "<|special_833|>",
+    "<|special_834|>",
+    "<|special_835|>",
+    "<|special_836|>",
+    "<|special_837|>",
+    "<|special_838|>",
+    "<|special_839|>",
+    "<|special_840|>",
+    "<|special_841|>",
+    "<|special_842|>",
+    "<|special_843|>",
+    "<|special_844|>",
+    "<|special_845|>",
+    "<|special_846|>",
+    "<|special_847|>",
+    "<|special_848|>",
+    "<|special_849|>",
+    "<|special_850|>",
+    "<|special_851|>",
+    "<|special_852|>",
+    "<|special_853|>",
+    "<|special_854|>",
+    "<|special_855|>",
+    "<|special_856|>",
+    "<|special_857|>",
+    "<|special_858|>",
+    "<|special_859|>",
+    "<|special_860|>",
+    "<|special_861|>",
+    "<|special_862|>",
+    "<|special_863|>",
+    "<|special_864|>",
+    "<|special_865|>",
+    "<|special_866|>",
+    "<|special_867|>",
+    "<|special_868|>",
+    "<|special_869|>",
+    "<|special_870|>",
+    "<|special_871|>",
+    "<|special_872|>",
+    "<|special_873|>",
+    "<|special_874|>",
+    "<|special_875|>",
+    "<|special_876|>",
+    "<|special_877|>",
+    "<|special_878|>",
+    "<|special_879|>",
+    "<|special_880|>",
+    "<|special_881|>",
+    "<|special_882|>",
+    "<|special_883|>",
+    "<|special_884|>",
+    "<|special_885|>",
+    "<|special_886|>",
+    "<|special_887|>",
+    "<|special_888|>",
+    "<|special_889|>",
+    "<|special_890|>",
+    "<|special_891|>",
+    "<|special_892|>",
+    "<|special_893|>",
+    "<|special_894|>",
+    "<|special_895|>",
+    "<|special_896|>",
+    "<|special_897|>",
+    "<|special_898|>",
+    "<|special_899|>",
+    "<|special_900|>",
+    "<|special_901|>",
+    "<|special_902|>",
+    "<|special_903|>",
+    "<|special_904|>",
+    "<|special_905|>",
+    "<|special_906|>",
+    "<|special_907|>",
+    "<|special_908|>",
+    "<|special_909|>",
+    "<|special_910|>",
+    "<|special_911|>",
+    "<|special_912|>",
+    "<|special_913|>",
+    "<|special_914|>",
+    "<|special_915|>",
+    "<|special_916|>",
+    "<|special_917|>",
+    "<|special_918|>",
+    "<|special_919|>",
+    "<|special_920|>",
+    "<|special_921|>",
+    "<|special_922|>",
+    "<|special_923|>",
+    "<|special_924|>",
+    "<|special_925|>",
+    "<|special_926|>",
+    "<|special_927|>",
+    "<|special_928|>",
+    "<|special_929|>",
+    "<|special_930|>",
+    "<|special_931|>",
+    "<|special_932|>",
+    "<|special_933|>",
+    "<|special_934|>",
+    "<|special_935|>",
+    "<|special_936|>",
+    "<|special_937|>",
+    "<|special_938|>",
+    "<|special_939|>",
+    "<|special_940|>",
+    "<|special_941|>",
+    "<|special_942|>",
+    "<|special_943|>",
+    "<|special_944|>",
+    "<|special_945|>",
+    "<|special_946|>",
+    "<|special_947|>",
+    "<|special_948|>",
+    "<|special_949|>",
+    "<|special_950|>",
+    "<|special_951|>",
+    "<|special_952|>",
+    "<|special_953|>",
+    "<|special_954|>",
+    "<|special_955|>",
+    "<|special_956|>",
+    "<|special_957|>",
+    "<|special_958|>",
+    "<|special_959|>",
+    "<|special_960|>",
+    "<|special_961|>",
+    "<|special_962|>",
+    "<|special_963|>",
+    "<|special_964|>",
+    "<|special_965|>",
+    "<|special_966|>",
+    "<|special_967|>",
+    "<|special_968|>",
+    "<|special_969|>",
+    "<|special_970|>",
+    "<|special_971|>",
+    "<|special_972|>",
+    "<|special_973|>",
+    "<|special_974|>",
+    "<|special_975|>",
+    "<|special_976|>",
+    "<|special_977|>",
+    "<|special_978|>",
+    "<|special_979|>",
+    "<|special_980|>",
+    "<|special_981|>",
+    "<|special_982|>",
+    "<|special_983|>",
+    "<|special_984|>",
+    "<|special_985|>",
+    "<|special_986|>",
+    "<|special_987|>",
+    "<|special_988|>",
+    "<|special_989|>",
+    "<|special_990|>",
+    "<|special_991|>",
+    "<|special_992|>",
+    "<|special_993|>",
+    "<|special_994|>",
+    "<|special_995|>",
+    "<|special_996|>",
+    "<|special_997|>",
+    "<|special_998|>",
+    "<|special_999|>",
+    "<|special_1000|>",
+    "<|special_1001|>",
+    "<|special_1002|>",
+    "<|special_1003|>",
+    "<|special_1004|>",
+    "<|special_1005|>",
+    "<|special_1006|>",
+    "<|special_1007|>",
+    "<|special_1008|>",
+    "<|special_1009|>",
+    "<|special_1010|>",
+    "<|special_1011|>",
+    "<|special_1012|>",
+    "<|special_1013|>",
+    "<|special_1014|>",
+    "<|special_1015|>",
+    "<|special_1016|>",
+    "<|special_1017|>",
+    "<|special_1018|>",
+    "<|special_1019|>",
+    "<|special_1020|>",
+    "<|special_1021|>",
+    "<|special_1022|>",
+    "<|special_1023|>",
+    "<|special_1024|>",
+    "<|special_1025|>",
+    "<|special_1026|>",
+    "<|special_1027|>",
+    "<|special_1028|>",
+    "<|special_1029|>",
+    "<|special_1030|>",
+    "<|special_1031|>",
+    "<|special_1032|>",
+    "<|special_1033|>",
+    "<|special_1034|>",
+    "<|special_1035|>",
+    "<|special_1036|>",
+    "<|special_1037|>",
+    "<|special_1038|>",
+    "<|special_1039|>",
+    "<|special_1040|>",
+    "<|special_1041|>",
+    "<|special_1042|>",
+    "<|special_1043|>",
+    "<|special_1044|>",
+    "<|special_1045|>",
+    "<|special_1046|>",
+    "<|special_1047|>",
+    "<|special_1048|>",
+    "<|special_1049|>",
+    "<|special_1050|>",
+    "<|special_1051|>",
+    "<|special_1052|>",
+    "<|special_1053|>",
+    "<|special_1054|>",
+    "<|special_1055|>",
+    "<|special_1056|>",
+    "<|special_1057|>",
+    "<|special_1058|>",
+    "<|special_1059|>",
+    "<|special_1060|>",
+    "<|special_1061|>",
+    "<|special_1062|>",
+    "<|special_1063|>",
+    "<|special_1064|>",
+    "<|special_1065|>",
+    "<|special_1066|>",
+    "<|special_1067|>",
+    "<|special_1068|>",
+    "<|special_1069|>",
+    "<|special_1070|>",
+    "<|special_1071|>",
+    "<|special_1072|>",
+    "<|special_1073|>",
+    "<|special_1074|>",
+    "<|special_1075|>",
+    "<|special_1076|>",
+    "<|special_1077|>",
+    "<|special_1078|>",
+    "<|special_1079|>",
+    "<|special_1080|>",
+    "<|special_1081|>",
+    "<|special_1082|>",
+    "<|special_1083|>",
+    "<|special_1084|>",
+    "<|special_1085|>",
+    "<|special_1086|>",
+    "<|special_1087|>",
+    "<|special_1088|>",
+    "<|special_1089|>",
+    "<|special_1090|>",
+    "<|special_1091|>",
+    "<|special_1092|>",
+    "<|special_1093|>",
+    "<|special_1094|>",
+    "<|special_1095|>",
+    "<|special_1096|>",
+    "<|special_1097|>",
+    "<|special_1098|>",
+    "<|special_1099|>",
+    "<|special_1100|>",
+    "<|special_1101|>",
+    "<|special_1102|>",
+    "<|special_1103|>",
+    "<|special_1104|>",
+    "<|special_1105|>",
+    "<|special_1106|>",
+    "<|special_1107|>",
+    "<|special_1108|>",
+    "<|special_1109|>",
+    "<|special_1110|>",
+    "<|special_1111|>",
+    "<|special_1112|>",
+    "<|special_1113|>",
+    "<|special_1114|>",
+    "<|special_1115|>",
+    "<|special_1116|>",
+    "<|special_1117|>",
+    "<|special_1118|>",
+    "<|special_1119|>",
+    "<|special_1120|>",
+    "<|special_1121|>",
+    "<|special_1122|>",
+    "<|special_1123|>",
+    "<|special_1124|>",
+    "<|special_1125|>",
+    "<|special_1126|>",
+    "<|special_1127|>",
+    "<|special_1128|>",
+    "<|special_1129|>",
+    "<|special_1130|>",
+    "<|special_1131|>",
+    "<|special_1132|>",
+    "<|special_1133|>",
+    "<|special_1134|>",
+    "<|special_1135|>",
+    "<|special_1136|>",
+    "<|special_1137|>",
+    "<|special_1138|>",
+    "<|special_1139|>",
+    "<|special_1140|>",
+    "<|special_1141|>",
+    "<|special_1142|>",
+    "<|special_1143|>",
+    "<|special_1144|>",
+    "<|special_1145|>",
+    "<|special_1146|>",
+    "<|special_1147|>",
+    "<|special_1148|>",
+    "<|special_1149|>",
+    "<|special_1150|>",
+    "<|special_1151|>",
+    "<|special_1152|>",
+    "<|special_1153|>",
+    "<|special_1154|>",
+    "<|special_1155|>",
+    "<|special_1156|>",
+    "<|special_1157|>",
+    "<|special_1158|>",
+    "<|special_1159|>",
+    "<|special_1160|>",
+    "<|special_1161|>",
+    "<|special_1162|>",
+    "<|special_1163|>",
+    "<|special_1164|>",
+    "<|special_1165|>",
+    "<|special_1166|>",
+    "<|special_1167|>",
+    "<|special_1168|>",
+    "<|special_1169|>",
+    "<|special_1170|>",
+    "<|special_1171|>",
+    "<|special_1172|>",
+    "<|special_1173|>",
+    "<|special_1174|>",
+    "<|special_1175|>",
+    "<|special_1176|>",
+    "<|special_1177|>",
+    "<|special_1178|>",
+    "<|special_1179|>",
+    "<|special_1180|>",
+    "<|special_1181|>",
+    "<|special_1182|>",
+    "<|special_1183|>",
+    "<|special_1184|>",
+    "<|special_1185|>",
+    "<|special_1186|>",
+    "<|special_1187|>",
+    "<|special_1188|>",
+    "<|special_1189|>",
+    "<|special_1190|>",
+    "<|special_1191|>",
+    "<|special_1192|>",
+    "<|special_1193|>",
+    "<|special_1194|>",
+    "<|special_1195|>",
+    "<|special_1196|>",
+    "<|special_1197|>",
+    "<|special_1198|>",
+    "<|special_1199|>",
+    "<|special_1200|>",
+    "<|special_1201|>",
+    "<|special_1202|>",
+    "<|special_1203|>",
+    "<|special_1204|>",
+    "<|special_1205|>",
+    "<|special_1206|>",
+    "<|special_1207|>",
+    "<|special_1208|>",
+    "<|special_1209|>",
+    "<|special_1210|>",
+    "<|special_1211|>",
+    "<|special_1212|>",
+    "<|special_1213|>",
+    "<|special_1214|>",
+    "<|special_1215|>",
+    "<|special_1216|>",
+    "<|special_1217|>",
+    "<|special_1218|>",
+    "<|special_1219|>",
+    "<|special_1220|>",
+    "<|special_1221|>",
+    "<|special_1222|>",
+    "<|special_1223|>",
+    "<|special_1224|>",
+    "<|special_1225|>",
+    "<|special_1226|>",
+    "<|special_1227|>",
+    "<|special_1228|>",
+    "<|special_1229|>",
+    "<|special_1230|>",
+    "<|special_1231|>",
+    "<|special_1232|>",
+    "<|special_1233|>",
+    "<|special_1234|>",
+    "<|special_1235|>",
+    "<|special_1236|>",
+    "<|special_1237|>",
+    "<|special_1238|>",
+    "<|special_1239|>",
+    "<|special_1240|>",
+    "<|special_1241|>",
+    "<|special_1242|>",
+    "<|special_1243|>",
+    "<|special_1244|>",
+    "<|special_1245|>",
+    "<|special_1246|>",
+    "<|special_1247|>",
+    "<|special_1248|>",
+    "<|special_1249|>",
+    "<|special_1250|>",
+    "<|special_1251|>",
+    "<|special_1252|>",
+    "<|special_1253|>",
+    "<|special_1254|>",
+    "<|special_1255|>",
+    "<|special_1256|>",
+    "<|special_1257|>",
+    "<|special_1258|>",
+    "<|special_1259|>",
+    "<|special_1260|>",
+    "<|special_1261|>",
+    "<|special_1262|>",
+    "<|special_1263|>",
+    "<|special_1264|>",
+    "<|special_1265|>",
+    "<|special_1266|>",
+    "<|special_1267|>",
+    "<|special_1268|>",
+    "<|special_1269|>",
+    "<|special_1270|>",
+    "<|special_1271|>",
+    "<|special_1272|>",
+    "<|special_1273|>",
+    "<|special_1274|>",
+    "<|special_1275|>",
+    "<|special_1276|>",
+    "<|special_1277|>",
+    "<|special_1278|>",
+    "<|special_1279|>",
+    "<|special_1280|>",
+    "<|special_1281|>",
+    "<|special_1282|>",
+    "<|special_1283|>",
+    "<|special_1284|>",
+    "<|special_1285|>",
+    "<|special_1286|>",
+    "<|special_1287|>",
+    "<|special_1288|>",
+    "<|special_1289|>",
+    "<|special_1290|>",
+    "<|special_1291|>",
+    "<|special_1292|>",
+    "<|special_1293|>",
+    "<|special_1294|>",
+    "<|special_1295|>",
+    "<|special_1296|>",
+    "<|special_1297|>",
+    "<|special_1298|>",
+    "<|special_1299|>",
+    "<|special_1300|>",
+    "<|special_1301|>",
+    "<|special_1302|>",
+    "<|special_1303|>",
+    "<|special_1304|>",
+    "<|special_1305|>",
+    "<|special_1306|>",
+    "<|special_1307|>",
+    "<|special_1308|>",
+    "<|special_1309|>",
+    "<|special_1310|>",
+    "<|special_1311|>",
+    "<|special_1312|>",
+    "<|special_1313|>",
+    "<|special_1314|>",
+    "<|special_1315|>",
+    "<|special_1316|>",
+    "<|special_1317|>",
+    "<|special_1318|>",
+    "<|special_1319|>",
+    "<|special_1320|>",
+    "<|special_1321|>",
+    "<|special_1322|>",
+    "<|special_1323|>",
+    "<|special_1324|>",
+    "<|special_1325|>",
+    "<|special_1326|>",
+    "<|special_1327|>",
+    "<|special_1328|>",
+    "<|special_1329|>",
+    "<|special_1330|>",
+    "<|special_1331|>",
+    "<|special_1332|>",
+    "<|special_1333|>",
+    "<|special_1334|>",
+    "<|special_1335|>",
+    "<|special_1336|>",
+    "<|special_1337|>",
+    "<|special_1338|>",
+    "<|special_1339|>",
+    "<|special_1340|>",
+    "<|special_1341|>",
+    "<|special_1342|>",
+    "<|special_1343|>",
+    "<|special_1344|>",
+    "<|special_1345|>",
+    "<|special_1346|>",
+    "<|special_1347|>",
+    "<|special_1348|>",
+    "<|special_1349|>",
+    "<|special_1350|>",
+    "<|special_1351|>",
+    "<|special_1352|>",
+    "<|special_1353|>",
+    "<|special_1354|>",
+    "<|special_1355|>",
+    "<|special_1356|>",
+    "<|special_1357|>",
+    "<|special_1358|>",
+    "<|special_1359|>",
+    "<|special_1360|>",
+    "<|special_1361|>",
+    "<|special_1362|>",
+    "<|special_1363|>",
+    "<|special_1364|>",
+    "<|special_1365|>",
+    "<|special_1366|>",
+    "<|special_1367|>",
+    "<|special_1368|>",
+    "<|special_1369|>",
+    "<|special_1370|>",
+    "<|special_1371|>",
+    "<|special_1372|>",
+    "<|special_1373|>",
+    "<|special_1374|>",
+    "<|special_1375|>",
+    "<|special_1376|>",
+    "<|special_1377|>",
+    "<|special_1378|>",
+    "<|special_1379|>",
+    "<|special_1380|>",
+    "<|special_1381|>",
+    "<|special_1382|>",
+    "<|special_1383|>",
+    "<|special_1384|>",
+    "<|special_1385|>",
+    "<|special_1386|>",
+    "<|special_1387|>",
+    "<|special_1388|>",
+    "<|special_1389|>",
+    "<|special_1390|>",
+    "<|special_1391|>",
+    "<|special_1392|>",
+    "<|special_1393|>",
+    "<|special_1394|>",
+    "<|special_1395|>",
+    "<|special_1396|>",
+    "<|special_1397|>",
+    "<|special_1398|>",
+    "<|special_1399|>",
+    "<|special_1400|>",
+    "<|special_1401|>",
+    "<|special_1402|>",
+    "<|special_1403|>",
+    "<|special_1404|>",
+    "<|special_1405|>",
+    "<|special_1406|>",
+    "<|special_1407|>",
+    "<|special_1408|>",
+    "<|special_1409|>",
+    "<|special_1410|>",
+    "<|special_1411|>",
+    "<|special_1412|>",
+    "<|special_1413|>",
+    "<|special_1414|>",
+    "<|special_1415|>",
+    "<|special_1416|>",
+    "<|special_1417|>",
+    "<|special_1418|>",
+    "<|special_1419|>",
+    "<|special_1420|>",
+    "<|special_1421|>",
+    "<|special_1422|>",
+    "<|special_1423|>",
+    "<|special_1424|>",
+    "<|special_1425|>",
+    "<|special_1426|>",
+    "<|special_1427|>",
+    "<|special_1428|>",
+    "<|special_1429|>",
+    "<|special_1430|>",
+    "<|special_1431|>",
+    "<|special_1432|>",
+    "<|special_1433|>",
+    "<|special_1434|>",
+    "<|special_1435|>",
+    "<|special_1436|>",
+    "<|special_1437|>",
+    "<|special_1438|>",
+    "<|special_1439|>",
+    "<|special_1440|>",
+    "<|special_1441|>",
+    "<|special_1442|>",
+    "<|special_1443|>",
+    "<|special_1444|>",
+    "<|special_1445|>",
+    "<|special_1446|>",
+    "<|special_1447|>",
+    "<|special_1448|>",
+    "<|special_1449|>",
+    "<|special_1450|>",
+    "<|special_1451|>",
+    "<|special_1452|>",
+    "<|special_1453|>",
+    "<|special_1454|>",
+    "<|special_1455|>",
+    "<|special_1456|>",
+    "<|special_1457|>",
+    "<|special_1458|>",
+    "<|special_1459|>",
+    "<|special_1460|>",
+    "<|special_1461|>",
+    "<|special_1462|>",
+    "<|special_1463|>",
+    "<|special_1464|>",
+    "<|special_1465|>",
+    "<|special_1466|>",
+    "<|special_1467|>",
+    "<|special_1468|>",
+    "<|special_1469|>",
+    "<|special_1470|>",
+    "<|special_1471|>",
+    "<|special_1472|>",
+    "<|special_1473|>",
+    "<|special_1474|>",
+    "<|special_1475|>",
+    "<|special_1476|>",
+    "<|special_1477|>",
+    "<|special_1478|>",
+    "<|special_1479|>",
+    "<|special_1480|>",
+    "<|special_1481|>",
+    "<|special_1482|>",
+    "<|special_1483|>",
+    "<|special_1484|>",
+    "<|special_1485|>",
+    "<|special_1486|>",
+    "<|special_1487|>",
+    "<|special_1488|>",
+    "<|special_1489|>",
+    "<|special_1490|>",
+    "<|special_1491|>",
+    "<|special_1492|>",
+    "<|special_1493|>",
+    "<|special_1494|>",
+    "<|special_1495|>",
+    "<|special_1496|>",
+    "<|special_1497|>",
+    "<|special_1498|>",
+    "<|special_1499|>",
+    "<|special_1500|>",
+    "<|special_1501|>",
+    "<|special_1502|>",
+    "<|special_1503|>",
+    "<|special_1504|>",
+    "<|special_1505|>",
+    "<|special_1506|>",
+    "<|special_1507|>",
+    "<|special_1508|>",
+    "<|special_1509|>",
+    "<|special_1510|>",
+    "<|special_1511|>",
+    "<|special_1512|>",
+    "<|special_1513|>",
+    "<|special_1514|>",
+    "<|special_1515|>",
+    "<|special_1516|>",
+    "<|special_1517|>",
+    "<|special_1518|>",
+    "<|special_1519|>",
+    "<|special_1520|>",
+    "<|special_1521|>",
+    "<|special_1522|>",
+    "<|special_1523|>",
+    "<|special_1524|>",
+    "<|special_1525|>",
+    "<|special_1526|>",
+    "<|special_1527|>",
+    "<|special_1528|>",
+    "<|special_1529|>",
+    "<|special_1530|>",
+    "<|special_1531|>",
+    "<|special_1532|>",
+    "<|special_1533|>",
+    "<|special_1534|>",
+    "<|special_1535|>",
+    "<|special_1536|>",
+    "<|special_1537|>",
+    "<|special_1538|>",
+    "<|special_1539|>",
+    "<|special_1540|>",
+    "<|special_1541|>",
+    "<|special_1542|>",
+    "<|special_1543|>",
+    "<|special_1544|>",
+    "<|special_1545|>",
+    "<|special_1546|>",
+    "<|special_1547|>",
+    "<|special_1548|>",
+    "<|special_1549|>",
+    "<|special_1550|>",
+    "<|special_1551|>",
+    "<|special_1552|>",
+    "<|special_1553|>",
+    "<|special_1554|>",
+    "<|special_1555|>",
+    "<|special_1556|>",
+    "<|special_1557|>",
+    "<|special_1558|>",
+    "<|special_1559|>",
+    "<|special_1560|>",
+    "<|special_1561|>",
+    "<|special_1562|>",
+    "<|special_1563|>",
+    "<|special_1564|>",
+    "<|special_1565|>",
+    "<|special_1566|>",
+    "<|special_1567|>",
+    "<|special_1568|>",
+    "<|special_1569|>",
+    "<|special_1570|>",
+    "<|special_1571|>",
+    "<|special_1572|>",
+    "<|special_1573|>",
+    "<|special_1574|>",
+    "<|special_1575|>",
+    "<|special_1576|>",
+    "<|special_1577|>",
+    "<|special_1578|>",
+    "<|special_1579|>",
+    "<|special_1580|>",
+    "<|special_1581|>",
+    "<|special_1582|>",
+    "<|special_1583|>",
+    "<|special_1584|>",
+    "<|special_1585|>",
+    "<|special_1586|>",
+    "<|special_1587|>",
+    "<|special_1588|>",
+    "<|special_1589|>",
+    "<|special_1590|>",
+    "<|special_1591|>",
+    "<|special_1592|>",
+    "<|special_1593|>",
+    "<|special_1594|>",
+    "<|special_1595|>",
+    "<|special_1596|>",
+    "<|special_1597|>",
+    "<|special_1598|>",
+    "<|special_1599|>",
+    "<|special_1600|>",
+    "<|special_1601|>",
+    "<|special_1602|>",
+    "<|special_1603|>",
+    "<|special_1604|>",
+    "<|special_1605|>",
+    "<|special_1606|>",
+    "<|special_1607|>",
+    "<|special_1608|>",
+    "<|special_1609|>",
+    "<|special_1610|>",
+    "<|special_1611|>",
+    "<|special_1612|>",
+    "<|special_1613|>",
+    "<|special_1614|>",
+    "<|special_1615|>",
+    "<|special_1616|>",
+    "<|special_1617|>",
+    "<|special_1618|>",
+    "<|special_1619|>",
+    "<|special_1620|>",
+    "<|special_1621|>",
+    "<|special_1622|>",
+    "<|special_1623|>",
+    "<|special_1624|>",
+    "<|special_1625|>",
+    "<|special_1626|>",
+    "<|special_1627|>",
+    "<|special_1628|>",
+    "<|special_1629|>",
+    "<|special_1630|>",
+    "<|special_1631|>",
+    "<|special_1632|>",
+    "<|special_1633|>",
+    "<|special_1634|>",
+    "<|special_1635|>",
+    "<|special_1636|>",
+    "<|special_1637|>",
+    "<|special_1638|>",
+    "<|special_1639|>",
+    "<|special_1640|>",
+    "<|special_1641|>",
+    "<|special_1642|>",
+    "<|special_1643|>",
+    "<|special_1644|>",
+    "<|special_1645|>",
+    "<|special_1646|>",
+    "<|special_1647|>",
+    "<|special_1648|>",
+    "<|special_1649|>",
+    "<|special_1650|>",
+    "<|special_1651|>",
+    "<|special_1652|>",
+    "<|special_1653|>",
+    "<|special_1654|>",
+    "<|special_1655|>",
+    "<|special_1656|>",
+    "<|special_1657|>",
+    "<|special_1658|>",
+    "<|special_1659|>",
+    "<|special_1660|>",
+    "<|special_1661|>",
+    "<|special_1662|>",
+    "<|special_1663|>",
+    "<|special_1664|>",
+    "<|special_1665|>",
+    "<|special_1666|>",
+    "<|special_1667|>",
+    "<|special_1668|>",
+    "<|special_1669|>",
+    "<|special_1670|>",
+    "<|special_1671|>",
+    "<|special_1672|>",
+    "<|special_1673|>",
+    "<|special_1674|>",
+    "<|special_1675|>",
+    "<|special_1676|>",
+    "<|special_1677|>",
+    "<|special_1678|>",
+    "<|special_1679|>",
+    "<|special_1680|>",
+    "<|special_1681|>",
+    "<|special_1682|>",
+    "<|special_1683|>",
+    "<|special_1684|>",
+    "<|special_1685|>",
+    "<|special_1686|>",
+    "<|special_1687|>",
+    "<|special_1688|>",
+    "<|special_1689|>",
+    "<|special_1690|>",
+    "<|special_1691|>",
+    "<|special_1692|>",
+    "<|special_1693|>",
+    "<|special_1694|>",
+    "<|special_1695|>",
+    "<|special_1696|>",
+    "<|special_1697|>",
+    "<|special_1698|>",
+    "<|special_1699|>",
+    "<|special_1700|>",
+    "<|special_1701|>",
+    "<|special_1702|>",
+    "<|special_1703|>",
+    "<|special_1704|>",
+    "<|special_1705|>",
+    "<|special_1706|>",
+    "<|special_1707|>",
+    "<|special_1708|>",
+    "<|special_1709|>",
+    "<|special_1710|>",
+    "<|special_1711|>",
+    "<|special_1712|>",
+    "<|special_1713|>",
+    "<|special_1714|>",
+    "<|special_1715|>",
+    "<|special_1716|>",
+    "<|special_1717|>",
+    "<|special_1718|>",
+    "<|special_1719|>",
+    "<|special_1720|>",
+    "<|special_1721|>",
+    "<|special_1722|>",
+    "<|special_1723|>",
+    "<|special_1724|>",
+    "<|special_1725|>",
+    "<|special_1726|>",
+    "<|special_1727|>",
+    "<|special_1728|>",
+    "<|special_1729|>",
+    "<|special_1730|>",
+    "<|special_1731|>",
+    "<|special_1732|>",
+    "<|special_1733|>",
+    "<|special_1734|>",
+    "<|special_1735|>",
+    "<|special_1736|>",
+    "<|special_1737|>",
+    "<|special_1738|>",
+    "<|special_1739|>",
+    "<|special_1740|>",
+    "<|special_1741|>",
+    "<|special_1742|>",
+    "<|special_1743|>",
+    "<|special_1744|>",
+    "<|special_1745|>",
+    "<|special_1746|>",
+    "<|special_1747|>",
+    "<|special_1748|>",
+    "<|special_1749|>",
+    "<|special_1750|>",
+    "<|special_1751|>",
+    "<|special_1752|>",
+    "<|special_1753|>",
+    "<|special_1754|>",
+    "<|special_1755|>",
+    "<|special_1756|>",
+    "<|special_1757|>",
+    "<|special_1758|>",
+    "<|special_1759|>",
+    "<|special_1760|>",
+    "<|special_1761|>",
+    "<|special_1762|>",
+    "<|special_1763|>",
+    "<|special_1764|>",
+    "<|special_1765|>",
+    "<|special_1766|>",
+    "<|special_1767|>",
+    "<|special_1768|>",
+    "<|special_1769|>",
+    "<|special_1770|>",
+    "<|special_1771|>",
+    "<|special_1772|>",
+    "<|special_1773|>",
+    "<|special_1774|>",
+    "<|special_1775|>",
+    "<|special_1776|>",
+    "<|special_1777|>",
+    "<|special_1778|>",
+    "<|special_1779|>",
+    "<|special_1780|>",
+    "<|special_1781|>",
+    "<|special_1782|>",
+    "<|special_1783|>",
+    "<|special_1784|>",
+    "<|special_1785|>",
+    "<|special_1786|>",
+    "<|special_1787|>",
+    "<|special_1788|>",
+    "<|special_1789|>",
+    "<|special_1790|>",
+    "<|special_1791|>",
+    "<|special_1792|>",
+    "<|special_1793|>",
+    "<|special_1794|>",
+    "<|special_1795|>",
+    "<|special_1796|>",
+    "<|special_1797|>",
+    "<|special_1798|>",
+    "<|special_1799|>",
+    "<|special_1800|>",
+    "<|special_1801|>",
+    "<|special_1802|>",
+    "<|special_1803|>",
+    "<|special_1804|>",
+    "<|special_1805|>",
+    "<|special_1806|>",
+    "<|special_1807|>",
+    "<|special_1808|>",
+    "<|special_1809|>",
+    "<|special_1810|>",
+    "<|special_1811|>",
+    "<|special_1812|>",
+    "<|special_1813|>",
+    "<|special_1814|>",
+    "<|special_1815|>",
+    "<|special_1816|>",
+    "<|special_1817|>",
+    "<|special_1818|>",
+    "<|special_1819|>",
+    "<|special_1820|>",
+    "<|special_1821|>",
+    "<|special_1822|>",
+    "<|special_1823|>",
+    "<|special_1824|>",
+    "<|special_1825|>",
+    "<|special_1826|>",
+    "<|special_1827|>",
+    "<|special_1828|>",
+    "<|special_1829|>",
+    "<|special_1830|>",
+    "<|special_1831|>",
+    "<|special_1832|>",
+    "<|special_1833|>",
+    "<|special_1834|>",
+    "<|special_1835|>",
+    "<|special_1836|>",
+    "<|special_1837|>",
+    "<|special_1838|>",
+    "<|special_1839|>",
+    "<|special_1840|>",
+    "<|special_1841|>",
+    "<|special_1842|>",
+    "<|special_1843|>",
+    "<|special_1844|>",
+    "<|special_1845|>",
+    "<|special_1846|>",
+    "<|special_1847|>",
+    "<|special_1848|>",
+    "<|special_1849|>",
+    "<|special_1850|>",
+    "<|special_1851|>",
+    "<|special_1852|>",
+    "<|special_1853|>",
+    "<|special_1854|>",
+    "<|special_1855|>",
+    "<|special_1856|>",
+    "<|special_1857|>",
+    "<|special_1858|>",
+    "<|special_1859|>",
+    "<|special_1860|>",
+    "<|special_1861|>",
+    "<|special_1862|>",
+    "<|special_1863|>",
+    "<|special_1864|>",
+    "<|special_1865|>",
+    "<|special_1866|>",
+    "<|special_1867|>",
+    "<|special_1868|>",
+    "<|special_1869|>",
+    "<|special_1870|>",
+    "<|special_1871|>",
+    "<|special_1872|>",
+    "<|special_1873|>",
+    "<|special_1874|>",
+    "<|special_1875|>",
+    "<|special_1876|>",
+    "<|special_1877|>",
+    "<|special_1878|>",
+    "<|special_1879|>",
+    "<|special_1880|>",
+    "<|special_1881|>",
+    "<|special_1882|>",
+    "<|special_1883|>",
+    "<|special_1884|>",
+    "<|special_1885|>",
+    "<|special_1886|>",
+    "<|special_1887|>",
+    "<|special_1888|>",
+    "<|special_1889|>",
+    "<|special_1890|>",
+    "<|special_1891|>",
+    "<|special_1892|>",
+    "<|special_1893|>",
+    "<|special_1894|>",
+    "<|special_1895|>",
+    "<|special_1896|>",
+    "<|special_1897|>",
+    "<|special_1898|>",
+    "<|special_1899|>",
+    "<|special_1900|>",
+    "<|special_1901|>",
+    "<|special_1902|>",
+    "<|special_1903|>",
+    "<|special_1904|>",
+    "<|special_1905|>",
+    "<|special_1906|>",
+    "<|special_1907|>",
+    "<|special_1908|>",
+    "<|special_1909|>",
+    "<|special_1910|>",
+    "<|special_1911|>",
+    "<|special_1912|>",
+    "<|special_1913|>",
+    "<|special_1914|>",
+    "<|special_1915|>",
+    "<|special_1916|>",
+    "<|special_1917|>",
+    "<|special_1918|>",
+    "<|special_1919|>",
+    "<|special_1920|>",
+    "<|special_1921|>",
+    "<|special_1922|>",
+    "<|special_1923|>",
+    "<|special_1924|>",
+    "<|special_1925|>",
+    "<|special_1926|>",
+    "<|special_1927|>",
+    "<|special_1928|>",
+    "<|special_1929|>",
+    "<|special_1930|>",
+    "<|special_1931|>",
+    "<|special_1932|>",
+    "<|special_1933|>",
+    "<|special_1934|>",
+    "<|special_1935|>",
+    "<|special_1936|>",
+    "<|special_1937|>",
+    "<|special_1938|>",
+    "<|special_1939|>",
+    "<|special_1940|>",
+    "<|special_1941|>",
+    "<|special_1942|>",
+    "<|special_1943|>",
+    "<|special_1944|>",
+    "<|special_1945|>",
+    "<|special_1946|>",
+    "<|special_1947|>",
+    "<|special_1948|>",
+    "<|special_1949|>",
+    "<|special_1950|>",
+    "<|special_1951|>",
+    "<|special_1952|>",
+    "<|special_1953|>",
+    "<|special_1954|>",
+    "<|special_1955|>",
+    "<|special_1956|>",
+    "<|special_1957|>",
+    "<|special_1958|>",
+    "<|special_1959|>",
+    "<|special_1960|>",
+    "<|special_1961|>",
+    "<|special_1962|>",
+    "<|special_1963|>",
+    "<|special_1964|>",
+    "<|special_1965|>",
+    "<|special_1966|>",
+    "<|special_1967|>",
+    "<|special_1968|>",
+    "<|special_1969|>",
+    "<|special_1970|>",
+    "<|special_1971|>",
+    "<|special_1972|>",
+    "<|special_1973|>",
+    "<|special_1974|>",
+    "<|special_1975|>",
+    "<|special_1976|>",
+    "<|special_1977|>",
+    "<|special_1978|>",
+    "<|special_1979|>",
+    "<|special_1980|>",
+    "<|special_1981|>",
+    "<|special_1982|>",
+    "<|special_1983|>",
+    "<|special_1984|>",
+    "<|special_1985|>",
+    "<|special_1986|>",
+    "<|special_1987|>",
+    "<|special_1988|>",
+    "<|special_1989|>",
+    "<|special_1990|>",
+    "<|special_1991|>",
+    "<|special_1992|>",
+    "<|special_1993|>",
+    "<|special_1994|>",
+    "<|special_1995|>",
+    "<|special_1996|>",
+    "<|special_1997|>",
+    "<|special_1998|>",
+    "<|special_1999|>",
+    "<|special_2000|>",
+    "<|special_2001|>",
+    "<|special_2002|>",
+    "<|special_2003|>",
+    "<|special_2004|>",
+    "<|special_2005|>",
+    "<|special_2006|>",
+    "<|special_2007|>",
+    "<|special_2008|>",
+    "<|special_2009|>",
+    "<|special_2010|>",
+    "<|special_2011|>",
+    "<|special_2012|>",
+    "<|special_2013|>",
+    "<|special_2014|>",
+    "<|special_2015|>",
+    "<|special_2016|>",
+    "<|special_2017|>",
+    "<|special_2018|>",
+    "<|special_2019|>",
+    "<|special_2020|>",
+    "<|special_2021|>",
+    "<|special_2022|>",
+    "<|special_2023|>",
+    "<|special_2024|>",
+    "<|special_2025|>",
+    "<|special_2026|>",
+    "<|special_2027|>",
+    "<|special_2028|>",
+    "<|special_2029|>",
+    "<|special_2030|>",
+    "<|special_2031|>",
+    "<|special_2032|>",
+    "<|special_2033|>",
+    "<|special_2034|>",
+    "<|special_2035|>",
+    "<|special_2036|>",
+    "<|special_2037|>",
+    "<|special_2038|>",
+    "<|special_2039|>",
+    "<|special_2040|>",
+    "<|special_2041|>",
+    "<|special_2042|>",
+    "<|special_2043|>",
+    "<|special_2044|>",
+    "<|special_2045|>",
+    "<|special_2046|>",
+    "<|special_2047|>",
+    "<|special_2048|>",
+    "<|special_2049|>",
+    "<|special_2050|>",
+    "<|special_2051|>",
+    "<|special_2052|>",
+    "<|special_2053|>",
+    "<|special_2054|>",
+    "<|special_2055|>",
+    "<|special_2056|>",
+    "<|special_2057|>",
+    "<|special_2058|>",
+    "<|special_2059|>",
+    "<|special_2060|>",
+    "<|special_2061|>",
+    "<|special_2062|>",
+    "<|special_2063|>",
+    "<|special_2064|>",
+    "<|special_2065|>",
+    "<|special_2066|>",
+    "<|special_2067|>",
+    "<|special_2068|>",
+    "<|special_2069|>",
+    "<|special_2070|>",
+    "<|special_2071|>",
+    "<|special_2072|>",
+    "<|special_2073|>",
+    "<|special_2074|>",
+    "<|special_2075|>",
+    "<|special_2076|>",
+    "<|special_2077|>",
+    "<|special_2078|>",
+    "<|special_2079|>",
+    "<|special_2080|>",
+    "<|special_2081|>",
+    "<|special_2082|>",
+    "<|special_2083|>",
+    "<|special_2084|>",
+    "<|special_2085|>",
+    "<|special_2086|>",
+    "<|special_2087|>",
+    "<|special_2088|>",
+    "<|special_2089|>",
+    "<|special_2090|>",
+    "<|special_2091|>",
+    "<|special_2092|>",
+    "<|special_2093|>",
+    "<|special_2094|>",
+    "<|special_2095|>",
+    "<|special_2096|>",
+    "<|special_2097|>",
+    "<|special_2098|>",
+    "<|special_2099|>",
+    "<|special_2100|>",
+    "<|special_2101|>",
+    "<|special_2102|>",
+    "<|special_2103|>",
+    "<|special_2104|>",
+    "<|special_2105|>",
+    "<|special_2106|>",
+    "<|special_2107|>",
+    "<|special_2108|>",
+    "<|special_2109|>",
+    "<|special_2110|>",
+    "<|special_2111|>",
+    "<|special_2112|>",
+    "<|special_2113|>",
+    "<|special_2114|>",
+    "<|special_2115|>",
+    "<|special_2116|>",
+    "<|special_2117|>",
+    "<|special_2118|>",
+    "<|special_2119|>",
+    "<|special_2120|>",
+    "<|special_2121|>",
+    "<|special_2122|>",
+    "<|special_2123|>",
+    "<|special_2124|>",
+    "<|special_2125|>",
+    "<|special_2126|>",
+    "<|special_2127|>",
+    "<|special_2128|>",
+    "<|special_2129|>",
+    "<|special_2130|>",
+    "<|special_2131|>",
+    "<|special_2132|>",
+    "<|special_2133|>",
+    "<|special_2134|>",
+    "<|special_2135|>",
+    "<|special_2136|>",
+    "<|special_2137|>",
+    "<|special_2138|>",
+    "<|special_2139|>",
+    "<|special_2140|>",
+    "<|special_2141|>",
+    "<|special_2142|>",
+    "<|special_2143|>",
+    "<|special_2144|>",
+    "<|special_2145|>",
+    "<|special_2146|>",
+    "<|special_2147|>",
+    "<|special_2148|>",
+    "<|special_2149|>",
+    "<|special_2150|>",
+    "<|special_2151|>",
+    "<|special_2152|>",
+    "<|special_2153|>",
+    "<|special_2154|>",
+    "<|special_2155|>",
+    "<|special_2156|>",
+    "<|special_2157|>",
+    "<|special_2158|>",
+    "<|special_2159|>",
+    "<|special_2160|>",
+    "<|special_2161|>",
+    "<|special_2162|>",
+    "<|special_2163|>",
+    "<|special_2164|>",
+    "<|special_2165|>",
+    "<|special_2166|>",
+    "<|special_2167|>",
+    "<|special_2168|>",
+    "<|special_2169|>",
+    "<|special_2170|>",
+    "<|special_2171|>",
+    "<|special_2172|>",
+    "<|special_2173|>",
+    "<|special_2174|>",
+    "<|special_2175|>",
+    "<|special_2176|>",
+    "<|special_2177|>",
+    "<|special_2178|>",
+    "<|special_2179|>",
+    "<|special_2180|>",
+    "<|special_2181|>",
+    "<|special_2182|>",
+    "<|special_2183|>",
+    "<|special_2184|>",
+    "<|special_2185|>",
+    "<|special_2186|>",
+    "<|special_2187|>",
+    "<|special_2188|>",
+    "<|special_2189|>",
+    "<|special_2190|>",
+    "<|special_2191|>",
+    "<|special_2192|>",
+    "<|special_2193|>",
+    "<|special_2194|>",
+    "<|special_2195|>",
+    "<|special_2196|>",
+    "<|special_2197|>",
+    "<|special_2198|>",
+    "<|special_2199|>",
+    "<|special_2200|>",
+    "<|special_2201|>",
+    "<|special_2202|>",
+    "<|special_2203|>",
+    "<|special_2204|>",
+    "<|special_2205|>",
+    "<|special_2206|>",
+    "<|special_2207|>",
+    "<|special_2208|>",
+    "<|special_2209|>",
+    "<|special_2210|>",
+    "<|special_2211|>",
+    "<|special_2212|>",
+    "<|special_2213|>",
+    "<|special_2214|>",
+    "<|special_2215|>",
+    "<|special_2216|>",
+    "<|special_2217|>",
+    "<|special_2218|>",
+    "<|special_2219|>",
+    "<|special_2220|>",
+    "<|special_2221|>",
+    "<|special_2222|>",
+    "<|special_2223|>",
+    "<|special_2224|>",
+    "<|special_2225|>",
+    "<|special_2226|>",
+    "<|special_2227|>",
+    "<|special_2228|>",
+    "<|special_2229|>",
+    "<|special_2230|>",
+    "<|special_2231|>",
+    "<|special_2232|>",
+    "<|special_2233|>",
+    "<|special_2234|>",
+    "<|special_2235|>",
+    "<|special_2236|>",
+    "<|special_2237|>",
+    "<|special_2238|>",
+    "<|special_2239|>",
+    "<|special_2240|>",
+    "<|special_2241|>",
+    "<|special_2242|>",
+    "<|special_2243|>",
+    "<|special_2244|>",
+    "<|special_2245|>",
+    "<|special_2246|>",
+    "<|special_2247|>",
+    "<|special_2248|>",
+    "<|special_2249|>",
+    "<|special_2250|>",
+    "<|special_2251|>",
+    "<|special_2252|>",
+    "<|special_2253|>",
+    "<|special_2254|>",
+    "<|special_2255|>",
+    "<|special_2256|>",
+    "<|special_2257|>",
+    "<|special_2258|>",
+    "<|special_2259|>",
+    "<|special_2260|>",
+    "<|special_2261|>",
+    "<|special_2262|>",
+    "<|special_2263|>",
+    "<|special_2264|>",
+    "<|special_2265|>",
+    "<|special_2266|>",
+    "<|special_2267|>",
+    "<|special_2268|>",
+    "<|special_2269|>",
+    "<|special_2270|>",
+    "<|special_2271|>",
+    "<|special_2272|>",
+    "<|special_2273|>",
+    "<|special_2274|>",
+    "<|special_2275|>",
+    "<|special_2276|>",
+    "<|special_2277|>",
+    "<|special_2278|>",
+    "<|special_2279|>",
+    "<|special_2280|>",
+    "<|special_2281|>",
+    "<|special_2282|>",
+    "<|special_2283|>",
+    "<|special_2284|>",
+    "<|special_2285|>",
+    "<|special_2286|>",
+    "<|special_2287|>",
+    "<|special_2288|>",
+    "<|special_2289|>",
+    "<|special_2290|>",
+    "<|special_2291|>",
+    "<|special_2292|>",
+    "<|special_2293|>",
+    "<|special_2294|>",
+    "<|special_2295|>",
+    "<|special_2296|>",
+    "<|special_2297|>",
+    "<|special_2298|>",
+    "<|special_2299|>",
+    "<|special_2300|>",
+    "<|special_2301|>",
+    "<|special_2302|>",
+    "<|special_2303|>",
+    "<|special_2304|>",
+    "<|special_2305|>",
+    "<|special_2306|>",
+    "<|special_2307|>",
+    "<|special_2308|>",
+    "<|special_2309|>",
+    "<|special_2310|>",
+    "<|special_2311|>",
+    "<|special_2312|>",
+    "<|special_2313|>",
+    "<|special_2314|>",
+    "<|special_2315|>",
+    "<|special_2316|>",
+    "<|special_2317|>",
+    "<|special_2318|>",
+    "<|special_2319|>",
+    "<|special_2320|>",
+    "<|special_2321|>",
+    "<|special_2322|>",
+    "<|special_2323|>",
+    "<|special_2324|>",
+    "<|special_2325|>",
+    "<|special_2326|>",
+    "<|special_2327|>",
+    "<|special_2328|>",
+    "<|special_2329|>",
+    "<|special_2330|>",
+    "<|special_2331|>",
+    "<|special_2332|>",
+    "<|special_2333|>",
+    "<|special_2334|>",
+    "<|special_2335|>",
+    "<|special_2336|>",
+    "<|special_2337|>",
+    "<|special_2338|>",
+    "<|special_2339|>",
+    "<|special_2340|>",
+    "<|special_2341|>",
+    "<|special_2342|>",
+    "<|special_2343|>",
+    "<|special_2344|>",
+    "<|special_2345|>",
+    "<|special_2346|>",
+    "<|special_2347|>",
+    "<|special_2348|>",
+    "<|special_2349|>",
+    "<|special_2350|>",
+    "<|special_2351|>",
+    "<|special_2352|>",
+    "<|special_2353|>",
+    "<|special_2354|>",
+    "<|special_2355|>",
+    "<|special_2356|>",
+    "<|special_2357|>",
+    "<|special_2358|>",
+    "<|special_2359|>",
+    "<|special_2360|>",
+    "<|special_2361|>",
+    "<|special_2362|>",
+    "<|special_2363|>",
+    "<|special_2364|>",
+    "<|special_2365|>",
+    "<|special_2366|>",
+    "<|special_2367|>",
+    "<|special_2368|>",
+    "<|special_2369|>",
+    "<|special_2370|>",
+    "<|special_2371|>",
+    "<|special_2372|>",
+    "<|special_2373|>",
+    "<|special_2374|>",
+    "<|special_2375|>",
+    "<|special_2376|>",
+    "<|special_2377|>",
+    "<|special_2378|>",
+    "<|special_2379|>",
+    "<|special_2380|>",
+    "<|special_2381|>",
+    "<|special_2382|>",
+    "<|special_2383|>",
+    "<|special_2384|>",
+    "<|special_2385|>",
+    "<|special_2386|>",
+    "<|special_2387|>",
+    "<|special_2388|>",
+    "<|special_2389|>",
+    "<|special_2390|>",
+    "<|special_2391|>",
+    "<|special_2392|>",
+    "<|special_2393|>",
+    "<|special_2394|>",
+    "<|special_2395|>",
+    "<|special_2396|>",
+    "<|special_2397|>",
+    "<|special_2398|>",
+    "<|special_2399|>",
+    "<|special_2400|>",
+    "<|special_2401|>",
+    "<|special_2402|>",
+    "<|special_2403|>",
+    "<|special_2404|>",
+    "<|special_2405|>",
+    "<|special_2406|>",
+    "<|special_2407|>",
+    "<|special_2408|>",
+    "<|special_2409|>",
+    "<|special_2410|>",
+    "<|special_2411|>",
+    "<|special_2412|>",
+    "<|special_2413|>",
+    "<|special_2414|>",
+    "<|special_2415|>",
+    "<|special_2416|>",
+    "<|special_2417|>",
+    "<|special_2418|>",
+    "<|special_2419|>",
+    "<|special_2420|>",
+    "<|special_2421|>",
+    "<|special_2422|>",
+    "<|special_2423|>",
+    "<|special_2424|>",
+    "<|special_2425|>",
+    "<|special_2426|>",
+    "<|special_2427|>",
+    "<|special_2428|>",
+    "<|special_2429|>",
+    "<|special_2430|>",
+    "<|special_2431|>",
+    "<|special_2432|>",
+    "<|special_2433|>",
+    "<|special_2434|>",
+    "<|special_2435|>",
+    "<|special_2436|>",
+    "<|special_2437|>",
+    "<|special_2438|>",
+    "<|special_2439|>",
+    "<|special_2440|>",
+    "<|special_2441|>",
+    "<|special_2442|>",
+    "<|special_2443|>",
+    "<|special_2444|>",
+    "<|special_2445|>",
+    "<|special_2446|>",
+    "<|special_2447|>",
+    "<|special_2448|>",
+    "<|special_2449|>",
+    "<|special_2450|>",
+    "<|special_2451|>",
+    "<|special_2452|>",
+    "<|special_2453|>",
+    "<|special_2454|>",
+    "<|special_2455|>",
+    "<|special_2456|>",
+    "<|special_2457|>",
+    "<|special_2458|>",
+    "<|special_2459|>",
+    "<|special_2460|>",
+    "<|special_2461|>",
+    "<|special_2462|>",
+    "<|special_2463|>",
+    "<|special_2464|>",
+    "<|special_2465|>",
+    "<|special_2466|>",
+    "<|special_2467|>",
+    "<|special_2468|>",
+    "<|special_2469|>",
+    "<|special_2470|>",
+    "<|special_2471|>",
+    "<|special_2472|>",
+    "<|special_2473|>",
+    "<|special_2474|>",
+    "<|special_2475|>",
+    "<|special_2476|>",
+    "<|special_2477|>",
+    "<|special_2478|>",
+    "<|special_2479|>",
+    "<|special_2480|>",
+    "<|special_2481|>",
+    "<|special_2482|>",
+    "<|special_2483|>",
+    "<|special_2484|>",
+    "<|special_2485|>",
+    "<|special_2486|>",
+    "<|special_2487|>",
+    "<|special_2488|>",
+    "<|special_2489|>",
+    "<|special_2490|>",
+    "<|special_2491|>",
+    "<|special_2492|>",
+    "<|special_2493|>",
+    "<|special_2494|>",
+    "<|special_2495|>",
+    "<|special_2496|>",
+    "<|special_2497|>",
+    "<|special_2498|>",
+    "<|special_2499|>",
+    "<|special_2500|>",
+    "<|special_2501|>",
+    "<|special_2502|>",
+    "<|special_2503|>",
+    "<|special_2504|>",
+    "<|special_2505|>",
+    "<|special_2506|>",
+    "<|special_2507|>",
+    "<|special_2508|>",
+    "<|special_2509|>",
+    "<|special_2510|>",
+    "<|special_2511|>",
+    "<|special_2512|>",
+    "<|special_2513|>",
+    "<|special_2514|>",
+    "<|special_2515|>",
+    "<|special_2516|>",
+    "<|special_2517|>",
+    "<|special_2518|>",
+    "<|special_2519|>",
+    "<|special_2520|>",
+    "<|special_2521|>",
+    "<|special_2522|>",
+    "<|special_2523|>",
+    "<|special_2524|>",
+    "<|special_2525|>",
+    "<|special_2526|>",
+    "<|special_2527|>",
+    "<|special_2528|>",
+    "<|special_2529|>",
+    "<|special_2530|>",
+    "<|special_2531|>",
+    "<|special_2532|>",
+    "<|special_2533|>",
+    "<|special_2534|>",
+    "<|special_2535|>",
+    "<|special_2536|>",
+    "<|special_2537|>",
+    "<|special_2538|>",
+    "<|special_2539|>",
+    "<|special_2540|>",
+    "<|special_2541|>",
+    "<|special_2542|>",
+    "<|special_2543|>",
+    "<|special_2544|>",
+    "<|special_2545|>",
+    "<|special_2546|>",
+    "<|special_2547|>",
+    "<|special_2548|>",
+    "<|special_2549|>",
+    "<|special_2550|>",
+    "<|special_2551|>",
+    "<|special_2552|>",
+    "<|special_2553|>",
+    "<|special_2554|>",
+    "<|special_2555|>",
+    "<|special_2556|>",
+    "<|special_2557|>",
+    "<|special_2558|>",
+    "<|special_2559|>",
+    "<|special_2560|>",
+    "<|special_2561|>",
+    "<|special_2562|>",
+    "<|special_2563|>",
+    "<|special_2564|>",
+    "<|special_2565|>",
+    "<|special_2566|>",
+    "<|special_2567|>",
+    "<|special_2568|>",
+    "<|special_2569|>",
+    "<|special_2570|>",
+    "<|special_2571|>",
+    "<|special_2572|>",
+    "<|special_2573|>",
+    "<|special_2574|>",
+    "<|special_2575|>",
+    "<|special_2576|>",
+    "<|special_2577|>",
+    "<|special_2578|>",
+    "<|special_2579|>",
+    "<|special_2580|>",
+    "<|special_2581|>",
+    "<|special_2582|>",
+    "<|special_2583|>",
+    "<|special_2584|>",
+    "<|special_2585|>",
+    "<|special_2586|>",
+    "<|special_2587|>",
+    "<|special_2588|>",
+    "<|special_2589|>",
+    "<|special_2590|>",
+    "<|special_2591|>",
+    "<|special_2592|>",
+    "<|special_2593|>",
+    "<|special_2594|>",
+    "<|special_2595|>",
+    "<|special_2596|>",
+    "<|special_2597|>",
+    "<|special_2598|>",
+    "<|special_2599|>",
+    "<|special_2600|>",
+    "<|special_2601|>",
+    "<|special_2602|>",
+    "<|special_2603|>",
+    "<|special_2604|>",
+    "<|special_2605|>",
+    "<|special_2606|>",
+    "<|special_2607|>",
+    "<|special_2608|>",
+    "<|special_2609|>",
+    "<|special_2610|>",
+    "<|special_2611|>",
+    "<|special_2612|>",
+    "<|special_2613|>",
+    "<|special_2614|>",
+    "<|special_2615|>",
+    "<|special_2616|>",
+    "<|special_2617|>",
+    "<|special_2618|>",
+    "<|special_2619|>",
+    "<|special_2620|>",
+    "<|special_2621|>",
+    "<|special_2622|>",
+    "<|special_2623|>",
+    "<|special_2624|>",
+    "<|special_2625|>",
+    "<|special_2626|>",
+    "<|special_2627|>",
+    "<|special_2628|>",
+    "<|special_2629|>",
+    "<|special_2630|>",
+    "<|special_2631|>",
+    "<|special_2632|>",
+    "<|special_2633|>",
+    "<|special_2634|>",
+    "<|special_2635|>",
+    "<|special_2636|>",
+    "<|special_2637|>",
+    "<|special_2638|>",
+    "<|special_2639|>",
+    "<|special_2640|>",
+    "<|special_2641|>",
+    "<|special_2642|>",
+    "<|special_2643|>",
+    "<|special_2644|>",
+    "<|special_2645|>",
+    "<|special_2646|>",
+    "<|special_2647|>",
+    "<|special_2648|>",
+    "<|special_2649|>",
+    "<|special_2650|>",
+    "<|special_2651|>",
+    "<|special_2652|>",
+    "<|special_2653|>",
+    "<|special_2654|>",
+    "<|special_2655|>",
+    "<|special_2656|>",
+    "<|special_2657|>",
+    "<|special_2658|>",
+    "<|special_2659|>",
+    "<|special_2660|>",
+    "<|special_2661|>",
+    "<|special_2662|>",
+    "<|special_2663|>",
+    "<|special_2664|>",
+    "<|special_2665|>",
+    "<|special_2666|>",
+    "<|special_2667|>",
+    "<|special_2668|>",
+    "<|special_2669|>",
+    "<|special_2670|>",
+    "<|special_2671|>",
+    "<|special_2672|>",
+    "<|special_2673|>",
+    "<|special_2674|>",
+    "<|special_2675|>",
+    "<|special_2676|>",
+    "<|special_2677|>",
+    "<|special_2678|>",
+    "<|special_2679|>",
+    "<|special_2680|>",
+    "<|special_2681|>",
+    "<|special_2682|>",
+    "<|special_2683|>",
+    "<|special_2684|>",
+    "<|special_2685|>",
+    "<|special_2686|>",
+    "<|special_2687|>",
+    "<|special_2688|>",
+    "<|special_2689|>",
+    "<|special_2690|>",
+    "<|special_2691|>",
+    "<|special_2692|>",
+    "<|special_2693|>",
+    "<|special_2694|>",
+    "<|special_2695|>",
+    "<|special_2696|>",
+    "<|special_2697|>",
+    "<|special_2698|>",
+    "<|special_2699|>",
+    "<|special_2700|>",
+    "<|special_2701|>",
+    "<|special_2702|>",
+    "<|special_2703|>",
+    "<|special_2704|>",
+    "<|special_2705|>",
+    "<|special_2706|>",
+    "<|special_2707|>",
+    "<|special_2708|>",
+    "<|special_2709|>",
+    "<|special_2710|>",
+    "<|special_2711|>",
+    "<|special_2712|>",
+    "<|special_2713|>",
+    "<|special_2714|>",
+    "<|special_2715|>",
+    "<|special_2716|>",
+    "<|special_2717|>",
+    "<|special_2718|>",
+    "<|special_2719|>",
+    "<|special_2720|>",
+    "<|special_2721|>",
+    "<|special_2722|>",
+    "<|special_2723|>",
+    "<|special_2724|>",
+    "<|special_2725|>",
+    "<|special_2726|>",
+    "<|special_2727|>",
+    "<|special_2728|>",
+    "<|special_2729|>",
+    "<|special_2730|>",
+    "<|special_2731|>",
+    "<|special_2732|>",
+    "<|special_2733|>",
+    "<|special_2734|>",
+    "<|special_2735|>",
+    "<|special_2736|>",
+    "<|special_2737|>",
+    "<|special_2738|>",
+    "<|special_2739|>",
+    "<|special_2740|>",
+    "<|special_2741|>",
+    "<|special_2742|>",
+    "<|special_2743|>",
+    "<|special_2744|>",
+    "<|special_2745|>",
+    "<|special_2746|>",
+    "<|special_2747|>",
+    "<|special_2748|>",
+    "<|special_2749|>",
+    "<|special_2750|>",
+    "<|special_2751|>",
+    "<|special_2752|>",
+    "<|special_2753|>",
+    "<|special_2754|>",
+    "<|special_2755|>",
+    "<|special_2756|>",
+    "<|special_2757|>",
+    "<|special_2758|>",
+    "<|special_2759|>",
+    "<|special_2760|>",
+    "<|special_2761|>",
+    "<|special_2762|>",
+    "<|special_2763|>",
+    "<|special_2764|>",
+    "<|special_2765|>",
+    "<|special_2766|>",
+    "<|special_2767|>",
+    "<|special_2768|>",
+    "<|special_2769|>",
+    "<|special_2770|>",
+    "<|special_2771|>",
+    "<|special_2772|>",
+    "<|special_2773|>",
+    "<|special_2774|>",
+    "<|special_2775|>",
+    "<|special_2776|>",
+    "<|special_2777|>",
+    "<|special_2778|>",
+    "<|special_2779|>",
+    "<|special_2780|>",
+    "<|special_2781|>",
+    "<|special_2782|>",
+    "<|special_2783|>",
+    "<|special_2784|>",
+    "<|special_2785|>",
+    "<|special_2786|>",
+    "<|special_2787|>",
+    "<|special_2788|>",
+    "<|special_2789|>",
+    "<|special_2790|>",
+    "<|special_2791|>",
+    "<|special_2792|>",
+    "<|special_2793|>",
+    "<|special_2794|>",
+    "<|special_2795|>",
+    "<|special_2796|>",
+    "<|special_2797|>",
+    "<|special_2798|>",
+    "<|special_2799|>",
+    "<|special_2800|>",
+    "<|special_2801|>",
+    "<|special_2802|>",
+    "<|special_2803|>",
+    "<|special_2804|>",
+    "<|special_2805|>",
+    "<|special_2806|>",
+    "<|special_2807|>",
+    "<|special_2808|>",
+    "<|special_2809|>",
+    "<|special_2810|>",
+    "<|special_2811|>",
+    "<|special_2812|>",
+    "<|special_2813|>",
+    "<|special_2814|>",
+    "<|special_2815|>",
+    "<|special_2816|>",
+    "<|special_2817|>",
+    "<|special_2818|>",
+    "<|special_2819|>",
+    "<|special_2820|>",
+    "<|special_2821|>",
+    "<|special_2822|>",
+    "<|special_2823|>",
+    "<|special_2824|>",
+    "<|special_2825|>",
+    "<|special_2826|>",
+    "<|special_2827|>",
+    "<|special_2828|>",
+    "<|special_2829|>",
+    "<|special_2830|>",
+    "<|special_2831|>",
+    "<|special_2832|>",
+    "<|special_2833|>",
+    "<|special_2834|>",
+    "<|special_2835|>",
+    "<|special_2836|>",
+    "<|special_2837|>",
+    "<|special_2838|>",
+    "<|special_2839|>",
+    "<|special_2840|>",
+    "<|special_2841|>",
+    "<|special_2842|>",
+    "<|special_2843|>",
+    "<|special_2844|>",
+    "<|special_2845|>",
+    "<|special_2846|>",
+    "<|special_2847|>",
+    "<|special_2848|>",
+    "<|special_2849|>",
+    "<|special_2850|>",
+    "<|special_2851|>",
+    "<|special_2852|>",
+    "<|special_2853|>",
+    "<|special_2854|>",
+    "<|special_2855|>",
+    "<|special_2856|>",
+    "<|special_2857|>",
+    "<|special_2858|>",
+    "<|special_2859|>",
+    "<|special_2860|>",
+    "<|special_2861|>",
+    "<|special_2862|>",
+    "<|special_2863|>",
+    "<|special_2864|>",
+    "<|special_2865|>",
+    "<|special_2866|>",
+    "<|special_2867|>",
+    "<|special_2868|>",
+    "<|special_2869|>",
+    "<|special_2870|>",
+    "<|special_2871|>",
+    "<|special_2872|>",
+    "<|special_2873|>",
+    "<|special_2874|>",
+    "<|special_2875|>",
+    "<|special_2876|>",
+    "<|special_2877|>",
+    "<|special_2878|>",
+    "<|special_2879|>",
+    "<|special_2880|>",
+    "<|special_2881|>",
+    "<|special_2882|>",
+    "<|special_2883|>",
+    "<|special_2884|>",
+    "<|special_2885|>",
+    "<|special_2886|>",
+    "<|special_2887|>",
+    "<|special_2888|>",
+    "<|special_2889|>",
+    "<|special_2890|>",
+    "<|special_2891|>",
+    "<|special_2892|>",
+    "<|special_2893|>",
+    "<|special_2894|>",
+    "<|special_2895|>",
+    "<|special_2896|>",
+    "<|special_2897|>",
+    "<|special_2898|>",
+    "<|special_2899|>",
+    "<|special_2900|>",
+    "<|special_2901|>",
+    "<|special_2902|>",
+    "<|special_2903|>",
+    "<|special_2904|>",
+    "<|special_2905|>",
+    "<|special_2906|>",
+    "<|special_2907|>",
+    "<|special_2908|>",
+    "<|special_2909|>",
+    "<|special_2910|>",
+    "<|special_2911|>",
+    "<|special_2912|>",
+    "<|special_2913|>",
+    "<|special_2914|>",
+    "<|special_2915|>",
+    "<|special_2916|>",
+    "<|special_2917|>",
+    "<|special_2918|>",
+    "<|special_2919|>",
+    "<|special_2920|>",
+    "<|special_2921|>",
+    "<|special_2922|>",
+    "<|special_2923|>",
+    "<|special_2924|>",
+    "<|special_2925|>",
+    "<|special_2926|>",
+    "<|special_2927|>",
+    "<|special_2928|>",
+    "<|special_2929|>",
+    "<|special_2930|>",
+    "<|special_2931|>",
+    "<|special_2932|>",
+    "<|special_2933|>",
+    "<|special_2934|>",
+    "<|special_2935|>",
+    "<|special_2936|>",
+    "<|special_2937|>",
+    "<|special_2938|>",
+    "<|special_2939|>",
+    "<|special_2940|>",
+    "<|special_2941|>",
+    "<|special_2942|>",
+    "<|special_2943|>",
+    "<|special_2944|>",
+    "<|special_2945|>",
+    "<|special_2946|>",
+    "<|special_2947|>",
+    "<|special_2948|>",
+    "<|special_2949|>",
+    "<|special_2950|>",
+    "<|special_2951|>",
+    "<|special_2952|>",
+    "<|special_2953|>",
+    "<|special_2954|>",
+    "<|special_2955|>",
+    "<|special_2956|>",
+    "<|special_2957|>",
+    "<|special_2958|>",
+    "<|special_2959|>",
+    "<|special_2960|>",
+    "<|special_2961|>",
+    "<|special_2962|>",
+    "<|special_2963|>",
+    "<|special_2964|>",
+    "<|special_2965|>",
+    "<|special_2966|>",
+    "<|special_2967|>",
+    "<|special_2968|>",
+    "<|special_2969|>",
+    "<|special_2970|>",
+    "<|special_2971|>",
+    "<|special_2972|>",
+    "<|special_2973|>",
+    "<|special_2974|>",
+    "<|special_2975|>",
+    "<|special_2976|>",
+    "<|special_2977|>",
+    "<|special_2978|>",
+    "<|special_2979|>",
+    "<|special_2980|>",
+    "<|special_2981|>",
+    "<|special_2982|>",
+    "<|special_2983|>",
+    "<|special_2984|>",
+    "<|special_2985|>",
+    "<|special_2986|>",
+    "<|special_2987|>",
+    "<|special_2988|>",
+    "<|special_2989|>",
+    "<|special_2990|>",
+    "<|special_2991|>",
+    "<|special_2992|>",
+    "<|special_2993|>",
+    "<|special_2994|>",
+    "<|special_2995|>",
+    "<|special_2996|>",
+    "<|special_2997|>",
+    "<|special_2998|>",
+    "<|special_2999|>",
+    "<|special_3000|>",
+    "<|special_3001|>",
+    "<|special_3002|>",
+    "<|special_3003|>",
+    "<|special_3004|>",
+    "<|special_3005|>",
+    "<|special_3006|>",
+    "<|special_3007|>",
+    "<|special_3008|>",
+    "<|special_3009|>",
+    "<|special_3010|>",
+    "<|special_3011|>",
+    "<|special_3012|>",
+    "<|special_3013|>",
+    "<|special_3014|>",
+    "<|special_3015|>",
+    "<|special_3016|>",
+    "<|special_3017|>",
+    "<|special_3018|>",
+    "<|special_3019|>",
+    "<|special_3020|>",
+    "<|special_3021|>",
+    "<|special_3022|>",
+    "<|special_3023|>",
+    "<|special_3024|>",
+    "<|special_3025|>",
+    "<|special_3026|>",
+    "<|special_3027|>",
+    "<|special_3028|>",
+    "<|special_3029|>",
+    "<|special_3030|>",
+    "<|special_3031|>",
+    "<|special_3032|>",
+    "<|special_3033|>",
+    "<|special_3034|>",
+    "<|special_3035|>",
+    "<|special_3036|>",
+    "<|special_3037|>",
+    "<|special_3038|>",
+    "<|special_3039|>",
+    "<|special_3040|>",
+    "<|special_3041|>",
+    "<|special_3042|>",
+    "<|special_3043|>",
+    "<|special_3044|>",
+    "<|special_3045|>",
+    "<|special_3046|>",
+    "<|special_3047|>",
+    "<|special_3048|>",
+    "<|special_3049|>",
+    "<|special_3050|>",
+    "<|special_3051|>",
+    "<|special_3052|>",
+    "<|special_3053|>",
+    "<|special_3054|>",
+    "<|special_3055|>",
+    "<|special_3056|>",
+    "<|special_3057|>",
+    "<|special_3058|>",
+    "<|special_3059|>",
+    "<|special_3060|>",
+    "<|special_3061|>",
+    "<|special_3062|>",
+    "<|special_3063|>",
+    "<|special_3064|>",
+    "<|special_3065|>",
+    "<|special_3066|>",
+    "<|special_3067|>",
+    "<|special_3068|>",
+    "<|special_3069|>",
+    "<|special_3070|>",
+    "<|special_3071|>",
+    "<|special_3072|>",
+    "<|special_3073|>",
+    "<|special_3074|>",
+    "<|special_3075|>",
+    "<|special_3076|>",
+    "<|special_3077|>",
+    "<|special_3078|>",
+    "<|special_3079|>",
+    "<|special_3080|>",
+    "<|special_3081|>",
+    "<|special_3082|>",
+    "<|special_3083|>",
+    "<|special_3084|>",
+    "<|special_3085|>",
+    "<|special_3086|>",
+    "<|special_3087|>",
+    "<|special_3088|>",
+    "<|special_3089|>",
+    "<|special_3090|>",
+    "<|special_3091|>",
+    "<|special_3092|>",
+    "<|special_3093|>",
+    "<|special_3094|>",
+    "<|special_3095|>",
+    "<|special_3096|>",
+    "<|special_3097|>",
+    "<|special_3098|>",
+    "<|special_3099|>",
+    "<|special_3100|>",
+    "<|special_3101|>",
+    "<|special_3102|>",
+    "<|special_3103|>",
+    "<|special_3104|>",
+    "<|special_3105|>",
+    "<|special_3106|>",
+    "<|special_3107|>",
+    "<|special_3108|>",
+    "<|special_3109|>",
+    "<|special_3110|>",
+    "<|special_3111|>",
+    "<|special_3112|>",
+    "<|special_3113|>",
+    "<|special_3114|>",
+    "<|special_3115|>",
+    "<|special_3116|>",
+    "<|special_3117|>",
+    "<|special_3118|>",
+    "<|special_3119|>",
+    "<|special_3120|>",
+    "<|special_3121|>",
+    "<|special_3122|>",
+    "<|special_3123|>",
+    "<|special_3124|>",
+    "<|special_3125|>",
+    "<|special_3126|>",
+    "<|special_3127|>",
+    "<|special_3128|>",
+    "<|special_3129|>",
+    "<|special_3130|>",
+    "<|special_3131|>",
+    "<|special_3132|>",
+    "<|special_3133|>",
+    "<|special_3134|>",
+    "<|special_3135|>",
+    "<|special_3136|>",
+    "<|special_3137|>",
+    "<|special_3138|>",
+    "<|special_3139|>",
+    "<|special_3140|>",
+    "<|special_3141|>",
+    "<|special_3142|>",
+    "<|special_3143|>",
+    "<|special_3144|>",
+    "<|special_3145|>",
+    "<|special_3146|>",
+    "<|special_3147|>",
+    "<|special_3148|>",
+    "<|special_3149|>",
+    "<|special_3150|>",
+    "<|special_3151|>",
+    "<|special_3152|>",
+    "<|special_3153|>",
+    "<|special_3154|>",
+    "<|special_3155|>",
+    "<|special_3156|>",
+    "<|special_3157|>",
+    "<|special_3158|>",
+    "<|special_3159|>",
+    "<|special_3160|>",
+    "<|special_3161|>",
+    "<|special_3162|>",
+    "<|special_3163|>",
+    "<|special_3164|>",
+    "<|special_3165|>",
+    "<|special_3166|>",
+    "<|special_3167|>",
+    "<|special_3168|>",
+    "<|special_3169|>",
+    "<|special_3170|>",
+    "<|special_3171|>",
+    "<|special_3172|>",
+    "<|special_3173|>",
+    "<|special_3174|>",
+    "<|special_3175|>",
+    "<|special_3176|>",
+    "<|special_3177|>",
+    "<|special_3178|>",
+    "<|special_3179|>",
+    "<|special_3180|>",
+    "<|special_3181|>",
+    "<|special_3182|>",
+    "<|special_3183|>",
+    "<|special_3184|>",
+    "<|special_3185|>",
+    "<|special_3186|>",
+    "<|special_3187|>",
+    "<|special_3188|>",
+    "<|special_3189|>",
+    "<|special_3190|>",
+    "<|special_3191|>",
+    "<|special_3192|>",
+    "<|special_3193|>",
+    "<|special_3194|>",
+    "<|special_3195|>",
+    "<|special_3196|>",
+    "<|special_3197|>",
+    "<|special_3198|>",
+    "<|special_3199|>",
+    "<|special_3200|>",
+    "<|special_3201|>",
+    "<|special_3202|>",
+    "<|special_3203|>",
+    "<|special_3204|>",
+    "<|special_3205|>",
+    "<|special_3206|>",
+    "<|special_3207|>",
+    "<|special_3208|>",
+    "<|special_3209|>",
+    "<|special_3210|>",
+    "<|special_3211|>",
+    "<|special_3212|>",
+    "<|special_3213|>",
+    "<|special_3214|>",
+    "<|special_3215|>",
+    "<|special_3216|>",
+    "<|special_3217|>",
+    "<|special_3218|>",
+    "<|special_3219|>",
+    "<|special_3220|>",
+    "<|special_3221|>",
+    "<|special_3222|>",
+    "<|special_3223|>",
+    "<|special_3224|>",
+    "<|special_3225|>",
+    "<|special_3226|>",
+    "<|special_3227|>",
+    "<|special_3228|>",
+    "<|special_3229|>",
+    "<|special_3230|>",
+    "<|special_3231|>",
+    "<|special_3232|>",
+    "<|special_3233|>",
+    "<|special_3234|>",
+    "<|special_3235|>",
+    "<|special_3236|>",
+    "<|special_3237|>",
+    "<|special_3238|>",
+    "<|special_3239|>",
+    "<|special_3240|>",
+    "<|special_3241|>",
+    "<|special_3242|>",
+    "<|special_3243|>",
+    "<|special_3244|>",
+    "<|special_3245|>",
+    "<|special_3246|>",
+    "<|special_3247|>",
+    "<|special_3248|>",
+    "<|special_3249|>",
+    "<|special_3250|>",
+    "<|special_3251|>",
+    "<|special_3252|>",
+    "<|special_3253|>",
+    "<|special_3254|>",
+    "<|special_3255|>",
+    "<|special_3256|>",
+    "<|special_3257|>",
+    "<|special_3258|>",
+    "<|special_3259|>",
+    "<|special_3260|>",
+    "<|special_3261|>",
+    "<|special_3262|>",
+    "<|special_3263|>",
+    "<|special_3264|>",
+    "<|special_3265|>",
+    "<|special_3266|>",
+    "<|special_3267|>",
+    "<|special_3268|>",
+    "<|special_3269|>",
+    "<|special_3270|>",
+    "<|special_3271|>",
+    "<|special_3272|>",
+    "<|special_3273|>",
+    "<|special_3274|>",
+    "<|special_3275|>",
+    "<|special_3276|>",
+    "<|special_3277|>",
+    "<|special_3278|>",
+    "<|special_3279|>",
+    "<|special_3280|>",
+    "<|special_3281|>",
+    "<|special_3282|>",
+    "<|special_3283|>",
+    "<|special_3284|>",
+    "<|special_3285|>",
+    "<|special_3286|>",
+    "<|special_3287|>",
+    "<|special_3288|>",
+    "<|special_3289|>",
+    "<|special_3290|>",
+    "<|special_3291|>",
+    "<|special_3292|>",
+    "<|special_3293|>",
+    "<|special_3294|>",
+    "<|special_3295|>",
+    "<|special_3296|>",
+    "<|special_3297|>",
+    "<|special_3298|>",
+    "<|special_3299|>",
+    "<|special_3300|>",
+    "<|special_3301|>",
+    "<|special_3302|>",
+    "<|special_3303|>",
+    "<|special_3304|>",
+    "<|special_3305|>",
+    "<|special_3306|>",
+    "<|special_3307|>",
+    "<|special_3308|>",
+    "<|special_3309|>",
+    "<|special_3310|>",
+    "<|special_3311|>",
+    "<|special_3312|>",
+    "<|special_3313|>",
+    "<|special_3314|>",
+    "<|special_3315|>",
+    "<|special_3316|>",
+    "<|special_3317|>",
+    "<|special_3318|>",
+    "<|special_3319|>",
+    "<|special_3320|>",
+    "<|special_3321|>",
+    "<|special_3322|>",
+    "<|special_3323|>",
+    "<|special_3324|>",
+    "<|special_3325|>",
+    "<|special_3326|>",
+    "<|special_3327|>",
+    "<|special_3328|>",
+    "<|special_3329|>",
+    "<|special_3330|>",
+    "<|special_3331|>",
+    "<|special_3332|>",
+    "<|special_3333|>",
+    "<|special_3334|>",
+    "<|special_3335|>",
+    "<|special_3336|>",
+    "<|special_3337|>",
+    "<|special_3338|>",
+    "<|special_3339|>",
+    "<|special_3340|>",
+    "<|special_3341|>",
+    "<|special_3342|>",
+    "<|special_3343|>",
+    "<|special_3344|>",
+    "<|special_3345|>",
+    "<|special_3346|>",
+    "<|special_3347|>",
+    "<|special_3348|>",
+    "<|special_3349|>",
+    "<|special_3350|>",
+    "<|special_3351|>",
+    "<|special_3352|>",
+    "<|special_3353|>",
+    "<|special_3354|>",
+    "<|special_3355|>",
+    "<|special_3356|>",
+    "<|special_3357|>",
+    "<|special_3358|>",
+    "<|special_3359|>",
+    "<|special_3360|>",
+    "<|special_3361|>",
+    "<|special_3362|>",
+    "<|special_3363|>",
+    "<|special_3364|>",
+    "<|special_3365|>",
+    "<|special_3366|>",
+    "<|special_3367|>",
+    "<|special_3368|>",
+    "<|special_3369|>",
+    "<|special_3370|>",
+    "<|special_3371|>",
+    "<|special_3372|>",
+    "<|special_3373|>",
+    "<|special_3374|>",
+    "<|special_3375|>",
+    "<|special_3376|>",
+    "<|special_3377|>",
+    "<|special_3378|>",
+    "<|special_3379|>",
+    "<|special_3380|>",
+    "<|special_3381|>",
+    "<|special_3382|>",
+    "<|special_3383|>",
+    "<|special_3384|>",
+    "<|special_3385|>",
+    "<|special_3386|>",
+    "<|special_3387|>",
+    "<|special_3388|>",
+    "<|special_3389|>",
+    "<|special_3390|>",
+    "<|special_3391|>",
+    "<|special_3392|>",
+    "<|special_3393|>",
+    "<|special_3394|>",
+    "<|special_3395|>",
+    "<|special_3396|>",
+    "<|special_3397|>",
+    "<|special_3398|>",
+    "<|special_3399|>",
+    "<|special_3400|>",
+    "<|special_3401|>",
+    "<|special_3402|>",
+    "<|special_3403|>",
+    "<|special_3404|>",
+    "<|special_3405|>",
+    "<|special_3406|>",
+    "<|special_3407|>",
+    "<|special_3408|>",
+    "<|special_3409|>",
+    "<|special_3410|>",
+    "<|special_3411|>",
+    "<|special_3412|>",
+    "<|special_3413|>",
+    "<|special_3414|>",
+    "<|special_3415|>",
+    "<|special_3416|>",
+    "<|special_3417|>",
+    "<|special_3418|>",
+    "<|special_3419|>",
+    "<|special_3420|>",
+    "<|special_3421|>",
+    "<|special_3422|>",
+    "<|special_3423|>",
+    "<|special_3424|>",
+    "<|special_3425|>",
+    "<|special_3426|>",
+    "<|special_3427|>",
+    "<|special_3428|>",
+    "<|special_3429|>",
+    "<|special_3430|>",
+    "<|special_3431|>",
+    "<|special_3432|>",
+    "<|special_3433|>",
+    "<|special_3434|>",
+    "<|special_3435|>",
+    "<|special_3436|>",
+    "<|special_3437|>",
+    "<|special_3438|>",
+    "<|special_3439|>",
+    "<|special_3440|>",
+    "<|special_3441|>",
+    "<|special_3442|>",
+    "<|special_3443|>",
+    "<|special_3444|>",
+    "<|special_3445|>",
+    "<|special_3446|>",
+    "<|special_3447|>",
+    "<|special_3448|>",
+    "<|special_3449|>",
+    "<|special_3450|>",
+    "<|special_3451|>",
+    "<|special_3452|>",
+    "<|special_3453|>",
+    "<|special_3454|>",
+    "<|special_3455|>",
+    "<|special_3456|>",
+    "<|special_3457|>",
+    "<|special_3458|>",
+    "<|special_3459|>",
+    "<|special_3460|>",
+    "<|special_3461|>",
+    "<|special_3462|>",
+    "<|special_3463|>",
+    "<|special_3464|>",
+    "<|special_3465|>",
+    "<|special_3466|>",
+    "<|special_3467|>",
+    "<|special_3468|>",
+    "<|special_3469|>",
+    "<|special_3470|>",
+    "<|special_3471|>",
+    "<|special_3472|>",
+    "<|special_3473|>",
+    "<|special_3474|>",
+    "<|special_3475|>",
+    "<|special_3476|>",
+    "<|special_3477|>",
+    "<|special_3478|>",
+    "<|special_3479|>",
+    "<|special_3480|>",
+    "<|special_3481|>",
+    "<|special_3482|>",
+    "<|special_3483|>",
+    "<|special_3484|>",
+    "<|special_3485|>",
+    "<|special_3486|>",
+    "<|special_3487|>",
+    "<|special_3488|>",
+    "<|special_3489|>",
+    "<|special_3490|>",
+    "<|special_3491|>",
+    "<|special_3492|>",
+    "<|special_3493|>",
+    "<|special_3494|>",
+    "<|special_3495|>",
+    "<|special_3496|>",
+    "<|special_3497|>",
+    "<|special_3498|>",
+    "<|special_3499|>",
+    "<|special_3500|>",
+    "<|special_3501|>",
+    "<|special_3502|>",
+    "<|special_3503|>",
+    "<|special_3504|>",
+    "<|special_3505|>",
+    "<|special_3506|>",
+    "<|special_3507|>",
+    "<|special_3508|>",
+    "<|special_3509|>",
+    "<|special_3510|>",
+    "<|special_3511|>",
+    "<|special_3512|>",
+    "<|special_3513|>",
+    "<|special_3514|>",
+    "<|special_3515|>",
+    "<|special_3516|>",
+    "<|special_3517|>",
+    "<|special_3518|>",
+    "<|special_3519|>",
+    "<|special_3520|>",
+    "<|special_3521|>",
+    "<|special_3522|>",
+    "<|special_3523|>",
+    "<|special_3524|>",
+    "<|special_3525|>",
+    "<|special_3526|>",
+    "<|special_3527|>",
+    "<|special_3528|>",
+    "<|special_3529|>",
+    "<|special_3530|>",
+    "<|special_3531|>",
+    "<|special_3532|>",
+    "<|special_3533|>",
+    "<|special_3534|>",
+    "<|special_3535|>",
+    "<|special_3536|>",
+    "<|special_3537|>",
+    "<|special_3538|>",
+    "<|special_3539|>",
+    "<|special_3540|>",
+    "<|special_3541|>",
+    "<|special_3542|>",
+    "<|special_3543|>",
+    "<|special_3544|>",
+    "<|special_3545|>",
+    "<|special_3546|>",
+    "<|special_3547|>",
+    "<|special_3548|>",
+    "<|special_3549|>",
+    "<|special_3550|>",
+    "<|special_3551|>",
+    "<|special_3552|>",
+    "<|special_3553|>",
+    "<|special_3554|>",
+    "<|special_3555|>",
+    "<|special_3556|>",
+    "<|special_3557|>",
+    "<|special_3558|>",
+    "<|special_3559|>",
+    "<|special_3560|>",
+    "<|special_3561|>",
+    "<|special_3562|>",
+    "<|special_3563|>",
+    "<|special_3564|>",
+    "<|special_3565|>",
+    "<|special_3566|>",
+    "<|special_3567|>",
+    "<|special_3568|>",
+    "<|special_3569|>",
+    "<|special_3570|>",
+    "<|special_3571|>",
+    "<|special_3572|>",
+    "<|special_3573|>",
+    "<|special_3574|>",
+    "<|special_3575|>",
+    "<|special_3576|>",
+    "<|special_3577|>",
+    "<|special_3578|>",
+    "<|special_3579|>",
+    "<|special_3580|>",
+    "<|special_3581|>",
+    "<|special_3582|>",
+    "<|special_3583|>",
+    "<|special_3584|>",
+    "<|special_3585|>",
+    "<|special_3586|>",
+    "<|special_3587|>",
+    "<|special_3588|>",
+    "<|special_3589|>",
+    "<|special_3590|>",
+    "<|special_3591|>",
+    "<|special_3592|>",
+    "<|special_3593|>",
+    "<|special_3594|>",
+    "<|special_3595|>",
+    "<|special_3596|>",
+    "<|special_3597|>",
+    "<|special_3598|>",
+    "<|special_3599|>",
+    "<|special_3600|>",
+    "<|special_3601|>",
+    "<|special_3602|>",
+    "<|special_3603|>",
+    "<|special_3604|>",
+    "<|special_3605|>",
+    "<|special_3606|>",
+    "<|special_3607|>",
+    "<|special_3608|>",
+    "<|special_3609|>",
+    "<|special_3610|>",
+    "<|special_3611|>",
+    "<|special_3612|>",
+    "<|special_3613|>",
+    "<|special_3614|>",
+    "<|special_3615|>",
+    "<|special_3616|>",
+    "<|special_3617|>",
+    "<|special_3618|>",
+    "<|special_3619|>",
+    "<|special_3620|>",
+    "<|special_3621|>",
+    "<|special_3622|>",
+    "<|special_3623|>",
+    "<|special_3624|>",
+    "<|special_3625|>",
+    "<|special_3626|>",
+    "<|special_3627|>",
+    "<|special_3628|>",
+    "<|special_3629|>",
+    "<|special_3630|>",
+    "<|special_3631|>",
+    "<|special_3632|>",
+    "<|special_3633|>",
+    "<|special_3634|>",
+    "<|special_3635|>",
+    "<|special_3636|>",
+    "<|special_3637|>",
+    "<|special_3638|>",
+    "<|special_3639|>",
+    "<|special_3640|>",
+    "<|special_3641|>",
+    "<|special_3642|>",
+    "<|special_3643|>",
+    "<|special_3644|>",
+    "<|special_3645|>",
+    "<|special_3646|>",
+    "<|special_3647|>",
+    "<|special_3648|>",
+    "<|special_3649|>",
+    "<|special_3650|>",
+    "<|special_3651|>",
+    "<|special_3652|>",
+    "<|special_3653|>",
+    "<|special_3654|>",
+    "<|special_3655|>",
+    "<|special_3656|>",
+    "<|special_3657|>",
+    "<|special_3658|>",
+    "<|special_3659|>",
+    "<|special_3660|>",
+    "<|special_3661|>",
+    "<|special_3662|>",
+    "<|special_3663|>",
+    "<|special_3664|>",
+    "<|special_3665|>",
+    "<|special_3666|>",
+    "<|special_3667|>",
+    "<|special_3668|>",
+    "<|special_3669|>",
+    "<|special_3670|>",
+    "<|special_3671|>",
+    "<|special_3672|>",
+    "<|special_3673|>",
+    "<|special_3674|>",
+    "<|special_3675|>",
+    "<|special_3676|>",
+    "<|special_3677|>",
+    "<|special_3678|>",
+    "<|special_3679|>",
+    "<|special_3680|>",
+    "<|special_3681|>",
+    "<|special_3682|>",
+    "<|special_3683|>",
+    "<|special_3684|>",
+    "<|special_3685|>",
+    "<|special_3686|>",
+    "<|special_3687|>",
+    "<|special_3688|>",
+    "<|special_3689|>",
+    "<|special_3690|>",
+    "<|special_3691|>",
+    "<|special_3692|>",
+    "<|special_3693|>",
+    "<|special_3694|>",
+    "<|special_3695|>",
+    "<|special_3696|>",
+    "<|special_3697|>",
+    "<|special_3698|>",
+    "<|special_3699|>",
+    "<|special_3700|>",
+    "<|special_3701|>",
+    "<|special_3702|>",
+    "<|special_3703|>",
+    "<|special_3704|>",
+    "<|special_3705|>",
+    "<|special_3706|>",
+    "<|special_3707|>",
+    "<|special_3708|>",
+    "<|special_3709|>",
+    "<|special_3710|>",
+    "<|special_3711|>",
+    "<|special_3712|>",
+    "<|special_3713|>",
+    "<|special_3714|>",
+    "<|special_3715|>",
+    "<|special_3716|>",
+    "<|special_3717|>",
+    "<|special_3718|>",
+    "<|special_3719|>",
+    "<|special_3720|>",
+    "<|special_3721|>",
+    "<|special_3722|>",
+    "<|special_3723|>",
+    "<|special_3724|>",
+    "<|special_3725|>",
+    "<|special_3726|>",
+    "<|special_3727|>",
+    "<|special_3728|>",
+    "<|special_3729|>",
+    "<|special_3730|>",
+    "<|special_3731|>",
+    "<|special_3732|>",
+    "<|special_3733|>",
+    "<|special_3734|>",
+    "<|special_3735|>",
+    "<|special_3736|>",
+    "<|special_3737|>",
+    "<|special_3738|>",
+    "<|special_3739|>",
+    "<|special_3740|>",
+    "<|special_3741|>",
+    "<|special_3742|>",
+    "<|special_3743|>",
+    "<|special_3744|>",
+    "<|special_3745|>",
+    "<|special_3746|>",
+    "<|special_3747|>",
+    "<|special_3748|>",
+    "<|special_3749|>",
+    "<|special_3750|>",
+    "<|special_3751|>",
+    "<|special_3752|>",
+    "<|special_3753|>",
+    "<|special_3754|>",
+    "<|special_3755|>",
+    "<|special_3756|>",
+    "<|special_3757|>",
+    "<|special_3758|>",
+    "<|special_3759|>",
+    "<|special_3760|>",
+    "<|special_3761|>",
+    "<|special_3762|>",
+    "<|special_3763|>",
+    "<|special_3764|>",
+    "<|special_3765|>",
+    "<|special_3766|>",
+    "<|special_3767|>",
+    "<|special_3768|>",
+    "<|special_3769|>",
+    "<|special_3770|>",
+    "<|special_3771|>",
+    "<|special_3772|>",
+    "<|special_3773|>",
+    "<|special_3774|>",
+    "<|special_3775|>",
+    "<|special_3776|>",
+    "<|special_3777|>",
+    "<|special_3778|>",
+    "<|special_3779|>",
+    "<|special_3780|>",
+    "<|special_3781|>",
+    "<|special_3782|>",
+    "<|special_3783|>",
+    "<|special_3784|>",
+    "<|special_3785|>",
+    "<|special_3786|>",
+    "<|special_3787|>",
+    "<|special_3788|>",
+    "<|special_3789|>",
+    "<|special_3790|>",
+    "<|special_3791|>",
+    "<|special_3792|>",
+    "<|special_3793|>",
+    "<|special_3794|>",
+    "<|special_3795|>",
+    "<|special_3796|>",
+    "<|special_3797|>",
+    "<|special_3798|>",
+    "<|special_3799|>",
+    "<|special_3800|>",
+    "<|special_3801|>",
+    "<|special_3802|>",
+    "<|special_3803|>",
+    "<|special_3804|>",
+    "<|special_3805|>",
+    "<|special_3806|>",
+    "<|special_3807|>",
+    "<|special_3808|>",
+    "<|special_3809|>",
+    "<|special_3810|>",
+    "<|special_3811|>",
+    "<|special_3812|>",
+    "<|special_3813|>",
+    "<|special_3814|>",
+    "<|special_3815|>",
+    "<|special_3816|>",
+    "<|special_3817|>",
+    "<|special_3818|>",
+    "<|special_3819|>",
+    "<|special_3820|>",
+    "<|special_3821|>",
+    "<|special_3822|>",
+    "<|special_3823|>",
+    "<|special_3824|>",
+    "<|special_3825|>",
+    "<|special_3826|>",
+    "<|special_3827|>",
+    "<|special_3828|>",
+    "<|special_3829|>",
+    "<|special_3830|>",
+    "<|special_3831|>",
+    "<|special_3832|>",
+    "<|special_3833|>",
+    "<|special_3834|>",
+    "<|special_3835|>",
+    "<|special_3836|>",
+    "<|special_3837|>",
+    "<|special_3838|>",
+    "<|special_3839|>",
+    "<|special_3840|>",
+    "<|special_3841|>",
+    "<|special_3842|>",
+    "<|special_3843|>",
+    "<|special_3844|>",
+    "<|special_3845|>",
+    "<|special_3846|>",
+    "<|special_3847|>",
+    "<|special_3848|>",
+    "<|special_3849|>",
+    "<|special_3850|>",
+    "<|special_3851|>",
+    "<|special_3852|>",
+    "<|special_3853|>",
+    "<|special_3854|>",
+    "<|special_3855|>",
+    "<|special_3856|>",
+    "<|special_3857|>",
+    "<|special_3858|>",
+    "<|special_3859|>",
+    "<|special_3860|>",
+    "<|special_3861|>",
+    "<|special_3862|>",
+    "<|special_3863|>",
+    "<|special_3864|>",
+    "<|special_3865|>",
+    "<|special_3866|>",
+    "<|special_3867|>",
+    "<|special_3868|>",
+    "<|special_3869|>",
+    "<|special_3870|>",
+    "<|special_3871|>",
+    "<|special_3872|>",
+    "<|special_3873|>",
+    "<|special_3874|>",
+    "<|special_3875|>",
+    "<|special_3876|>",
+    "<|special_3877|>",
+    "<|special_3878|>",
+    "<|special_3879|>",
+    "<|special_3880|>",
+    "<|special_3881|>",
+    "<|special_3882|>",
+    "<|special_3883|>",
+    "<|special_3884|>",
+    "<|special_3885|>",
+    "<|special_3886|>",
+    "<|special_3887|>",
+    "<|special_3888|>",
+    "<|special_3889|>",
+    "<|special_3890|>",
+    "<|special_3891|>",
+    "<|special_3892|>",
+    "<|special_3893|>",
+    "<|special_3894|>",
+    "<|special_3895|>",
+    "<|special_3896|>",
+    "<|special_3897|>",
+    "<|special_3898|>",
+    "<|special_3899|>",
+    "<|special_3900|>",
+    "<|special_3901|>",
+    "<|special_3902|>",
+    "<|special_3903|>",
+    "<|special_3904|>",
+    "<|special_3905|>",
+    "<|special_3906|>",
+    "<|special_3907|>",
+    "<|special_3908|>",
+    "<|special_3909|>",
+    "<|special_3910|>",
+    "<|special_3911|>",
+    "<|special_3912|>",
+    "<|special_3913|>",
+    "<|special_3914|>",
+    "<|special_3915|>",
+    "<|special_3916|>",
+    "<|special_3917|>",
+    "<|special_3918|>",
+    "<|special_3919|>",
+    "<|special_3920|>",
+    "<|special_3921|>",
+    "<|special_3922|>",
+    "<|special_3923|>",
+    "<|special_3924|>",
+    "<|special_3925|>",
+    "<|special_3926|>",
+    "<|special_3927|>",
+    "<|special_3928|>",
+    "<|special_3929|>",
+    "<|special_3930|>",
+    "<|special_3931|>",
+    "<|special_3932|>",
+    "<|special_3933|>",
+    "<|special_3934|>",
+    "<|special_3935|>",
+    "<|special_3936|>",
+    "<|special_3937|>",
+    "<|special_3938|>",
+    "<|special_3939|>",
+    "<|special_3940|>",
+    "<|special_3941|>",
+    "<|special_3942|>",
+    "<|special_3943|>",
+    "<|special_3944|>",
+    "<|special_3945|>",
+    "<|special_3946|>",
+    "<|special_3947|>",
+    "<|special_3948|>",
+    "<|special_3949|>",
+    "<|special_3950|>",
+    "<|special_3951|>",
+    "<|special_3952|>",
+    "<|special_3953|>",
+    "<|special_3954|>",
+    "<|special_3955|>",
+    "<|special_3956|>",
+    "<|special_3957|>",
+    "<|special_3958|>",
+    "<|special_3959|>",
+    "<|special_3960|>",
+    "<|special_3961|>",
+    "<|special_3962|>",
+    "<|special_3963|>",
+    "<|special_3964|>",
+    "<|special_3965|>",
+    "<|special_3966|>",
+    "<|special_3967|>",
+    "<|special_3968|>",
+    "<|special_3969|>",
+    "<|special_3970|>",
+    "<|special_3971|>",
+    "<|special_3972|>",
+    "<|special_3973|>",
+    "<|special_3974|>",
+    "<|special_3975|>",
+    "<|special_3976|>",
+    "<|special_3977|>",
+    "<|special_3978|>",
+    "<|special_3979|>",
+    "<|special_3980|>",
+    "<|special_3981|>",
+    "<|special_3982|>",
+    "<|special_3983|>",
+    "<|special_3984|>",
+    "<|special_3985|>",
+    "<|special_3986|>",
+    "<|special_3987|>",
+    "<|special_3988|>",
+    "<|special_3989|>",
+    "<|special_3990|>",
+    "<|special_3991|>",
+    "<|special_3992|>",
+    "<|special_3993|>",
+    "<|special_3994|>",
+    "<|special_3995|>",
+    "<|special_3996|>",
+    "<|special_3997|>",
+    "<|special_3998|>",
+    "<|special_3999|>",
+    "<|special_4000|>",
+    "<|special_4001|>",
+    "<|special_4002|>",
+    "<|special_4003|>",
+    "<|special_4004|>",
+    "<|special_4005|>",
+    "<|special_4006|>",
+    "<|special_4007|>",
+    "<|special_4008|>",
+    "<|special_4009|>",
+    "<|special_4010|>",
+    "<|special_4011|>",
+    "<|special_4012|>",
+    "<|special_4013|>",
+    "<|special_4014|>",
+    "<|special_4015|>",
+    "<|special_4016|>",
+    "<|special_4017|>",
+    "<|special_4018|>",
+    "<|special_4019|>",
+    "<|special_4020|>",
+    "<|special_4021|>",
+    "<|special_4022|>",
+    "<|special_4023|>",
+    "<|special_4024|>",
+    "<|special_4025|>",
+    "<|special_4026|>",
+    "<|special_4027|>",
+    "<|special_4028|>",
+    "<|special_4029|>",
+    "<|special_4030|>",
+    "<|special_4031|>",
+    "<|special_4032|>",
+    "<|special_4033|>",
+    "<|special_4034|>",
+    "<|special_4035|>",
+    "<|special_4036|>",
+    "<|special_4037|>",
+    "<|special_4038|>",
+    "<|special_4039|>",
+    "<|special_4040|>",
+    "<|special_4041|>",
+    "<|special_4042|>",
+    "<|special_4043|>",
+    "<|special_4044|>",
+    "<|special_4045|>",
+    "<|special_4046|>",
+    "<|special_4047|>",
+    "<|special_4048|>",
+    "<|special_4049|>",
+    "<|special_4050|>",
+    "<|special_4051|>",
+    "<|special_4052|>",
+    "<|special_4053|>",
+    "<|special_4054|>",
+    "<|special_4055|>",
+    "<|special_4056|>",
+    "<|special_4057|>",
+    "<|special_4058|>",
+    "<|special_4059|>",
+    "<|special_4060|>",
+    "<|special_4061|>",
+    "<|special_4062|>",
+    "<|special_4063|>",
+    "<|special_4064|>",
+    "<|special_4065|>",
+    "<|special_4066|>",
+    "<|special_4067|>",
+    "<|special_4068|>",
+    "<|special_4069|>",
+    "<|special_4070|>",
+    "<|special_4071|>",
+    "<|special_4072|>",
+    "<|special_4073|>",
+    "<|special_4074|>",
+    "<|special_4075|>",
+    "<|special_4076|>",
+    "<|special_4077|>",
+    "<|special_4078|>",
+    "<|special_4079|>",
+    "<|special_4080|>",
+    "<|special_4081|>",
+    "<|special_4082|>",
+    "<|special_4083|>",
+    "<|special_4084|>",
+    "<|special_4085|>",
+    "<|special_4086|>",
+    "<|special_4087|>",
+    "<|special_4088|>",
+    "<|special_4089|>",
+    "<|special_4090|>",
+    "<|special_4091|>",
+    "<|special_4092|>",
+    "<|special_4093|>",
+    "<|special_4094|>",
+    "<|special_4095|>"
+  ],
+  "bos_token": {
+    "content": "<|startoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cad2dc0cc1dd988ef022be5d94131ec9eecfbf66a955ca820fde0c462fa69add
+size 16473584