Hugging Face's logo

zededa
/
Llama-3.2-3B-Instruct-QCS9075-HTP

llama
qualcomm
qcs9075
edge-ai
genie
qnn-htp
llm
Model card Files
xet
 Community
Hariharasubramanian commited on
Commit
e148d8a
·
verified ·
1 Parent(s): 5965cbb

Upload Llama-3.2-3B-Instruct model for QCS9075 (HTP backend)

Browse files
Files changed (15) hide show
  1. .gitattributes +2 -0
  2. README.md +140 -0
  3. config.json +40 -0
  4. genie_config.json +71 -0
  5. htp_backend_ext_config.json +1 -0
  6. libGenie.so +3 -0
  7. llama_v3_2_3b_instruct_part_1_of_3.bin +3 -0
  8. llama_v3_2_3b_instruct_part_2_of_3.bin +3 -0
  9. llama_v3_2_3b_instruct_part_3_of_3.bin +3 -0
  10. metadata.yaml +2481 -0
  11. sample_prompt.txt +9 -0
  12. server_persistent.py +664 -0
  13. tokenizer.json +3 -0
  14. tokenizer_config.json +2064 -0
  15. tool-versions.yaml +1 -0
.gitattributes CHANGED
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ libGenie.so filter=lfs diff=lfs merge=lfs -text
37
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,140 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ tags:
4
+ - qualcomm
5
+ - qcs9075
6
+ - edge-ai
7
+ - genie
8
+ - qnn-htp
9
+ - llm
10
+ base_model: meta-llama/Llama-3.2-3B-Instruct
11
+ ---
12
+
13
+ # Llama-3.2-3B-Instruct-QCS9075-HTP
14
+
15
+ This is a pre-compiled version of [meta-llama/Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct) optimized for the **Qualcomm QCS9075 SoC** using the **Qualcomm Genie SDK**.
16
+
17
+ ## Model Details
18
+
19
+ - **Base Model**: [meta-llama/Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct)
20
+ - **Target Hardware**: Qualcomm QCS9075 (IQ-9075 EVK)
21
+ - **Backend**: QnnHtp (NPU)
22
+ - **Quantization**: W4A16
23
+ - **Compilation**: Qualcomm AI Hub (QAIRT 2.42)
24
+
25
+ ## Performance
26
+
27
+ | Model | Backend | Performance | Size |
28
+ |-------|---------|-------------|------|
29
+ | Llama-3.2-3B-Instruct-QCS9075-HTP | QnnHtp (NPU) | ~18.7 TPS on QCS9075 | 2.5G |
30
+
31
+ *TPS = Tokens Per Second (generation speed)*
32
+
33
+ ## Hardware Requirements
34
+
35
+ - **Device**: Qualcomm IQ-9075 EVK or QCS9075-based device
36
+ - **OS**: Ubuntu 22.04 (recommended)
37
+ - **SDK**: Qualcomm Genie SDK
38
+ - **QAIRT**: Version 2.42 or later
39
+
40
+ ## Usage
41
+
42
+ ### Prerequisites
43
+
44
+ 1. Install the Qualcomm Genie SDK on your QCS9075 device
45
+ 2. Download all model files from this repository
46
+ 3. Ensure QAIRT 2.42 libraries are available
47
+
48
+ ### Environment Setup
49
+
50
+ For HTP models, the LD_LIBRARY_PATH ordering is critical:
51
+
52
+ ```bash
53
+ export LD_LIBRARY_PATH=/opt/qcom/aistack/qairt/2.42.0.250923/lib/aarch64-linux-gnu:/opt/qcom/aistack/genie/qnn/libs:$LD_LIBRARY_PATH
54
+ ```
55
+
56
+ ### Configuration
57
+
58
+ Create a `genie_config.json` file:
59
+
60
+ ```json
61
+ {
62
+ "model_path": "/path/to/model/files",
63
+ "backend": "QnnHtp",
64
+ "device": "0"
65
+ }
66
+ ```
67
+
68
+ ### Running the Model
69
+
70
+ ```bash
71
+ # Using the Genie server
72
+ python3 /opt/qcom/aistack/genie/examples/server_persistent.py \
73
+ --config genie_config.json \
74
+ --port 8000
75
+ ```
76
+
77
+ ### Kubernetes Deployment
78
+
79
+ For deploying on Kubernetes clusters with QCS9075 nodes, refer to the deployment pattern:
80
+
81
+ ```yaml
82
+ apiVersion: v1
83
+ kind: Pod
84
+ metadata:
85
+ name: genie-llm-server
86
+ spec:
87
+ containers:
88
+ - name: genie
89
+ image: your-registry/genie-runtime:latest
90
+ env:
91
+ - name: LD_LIBRARY_PATH
92
+ value: "'/opt/qcom/aistack/qairt/2.42.0.250923/lib/aarch64-linux-gnu:/opt/qcom/aistack/genie/qnn/libs'"
93
+ volumeMounts:
94
+ - name: model-storage
95
+ mountPath: /models
96
+ - name: qcom-libs
97
+ mountPath: /opt/qcom/aistack
98
+ volumes:
99
+ - name: model-storage
100
+ hostPath:
101
+ path: /mnt/models/llama-3.2-3b-instruct-qcs9075-htp
102
+ - name: qcom-libs
103
+ hostPath:
104
+ path: /opt/qcom/aistack
105
+ ```
106
+
107
+ ## File Structure
108
+
109
+ This repository contains:
110
+ - Compiled model artifacts (.bin files)
111
+ - Configuration files (genie_config.json)
112
+ - QNN HTP context binaries
113
+
114
+ ## Benchmarking Notes
115
+
116
+ - Performance metrics measured on Qualcomm IQ-9075 EVK
117
+ - TPS (Tokens Per Second) measured during generation phase
118
+ - Results may vary based on prompt length and complexity
119
+ - HTP backend utilizes the NPU for acceleration
120
+
121
+ ## License
122
+
123
+ This model follows the license of the base model [meta-llama/Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct). Please refer to the original model card for license details.
124
+
125
+ ## Acknowledgments
126
+
127
+ - Base model: [meta-llama/Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct)
128
+ - Compiled using Qualcomm AI Hub with QAIRT 2.42
129
+ - Target hardware: Qualcomm QCS9075 SoC
130
+
131
+ ## Support
132
+
133
+ For issues related to:
134
+ - **Model compilation**: Contact Qualcomm AI Hub support
135
+ - **Genie SDK**: Refer to Qualcomm Genie documentation
136
+ - **Deployment**: Open an issue in this repository
137
+
138
+ ---
139
+
140
+ *This model is optimized for edge deployment on Qualcomm QCS9075 devices and may not work on other hardware platforms.*
config.json ADDED
@@ -0,0 +1,40 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "meta-llama/Llama-3.2-3B-Instruct",
3
+ "architectures": [
4
+ "LlamaForCausalLM"
5
+ ],
6
+ "attention_bias": false,
7
+ "attention_dropout": 0.0,
8
+ "bos_token_id": 128000,
9
+ "eos_token_id": [
10
+ 128001,
11
+ 128008,
12
+ 128009
13
+ ],
14
+ "head_dim": 128,
15
+ "hidden_act": "silu",
16
+ "hidden_size": 3072,
17
+ "initializer_range": 0.02,
18
+ "intermediate_size": 8192,
19
+ "max_position_embeddings": 131072,
20
+ "mlp_bias": false,
21
+ "model_type": "llama",
22
+ "num_attention_heads": 24,
23
+ "num_hidden_layers": 28,
24
+ "num_key_value_heads": 8,
25
+ "pretraining_tp": 1,
26
+ "rms_norm_eps": 1e-05,
27
+ "rope_scaling": {
28
+ "factor": 32.0,
29
+ "high_freq_factor": 4.0,
30
+ "low_freq_factor": 1.0,
31
+ "original_max_position_embeddings": 8192,
32
+ "rope_type": "llama3"
33
+ },
34
+ "rope_theta": 500000.0,
35
+ "tie_word_embeddings": true,
36
+ "torch_dtype": "bfloat16",
37
+ "transformers_version": "4.45.0",
38
+ "use_cache": true,
39
+ "vocab_size": 128256
40
+ }
genie_config.json ADDED
@@ -0,0 +1,71 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "dialog": {
3
+ "version": 1,
4
+ "type": "basic",
5
+ "context": {
6
+ "version": 1,
7
+ "size": 4096,
8
+ "n-vocab": 128256,
9
+ "bos-token": 128000,
10
+ "eos-token": [
11
+ 128001,
12
+ 128008,
13
+ 128009
14
+ ]
15
+ },
16
+ "sampler": {
17
+ "version": 1,
18
+ "seed": 42,
19
+ "temp": 0.8,
20
+ "top-k": 40,
21
+ "top-p": 0.95
22
+ },
23
+ "tokenizer": {
24
+ "version": 1,
25
+ "path": "tokenizer.json"
26
+ },
27
+ "engine": {
28
+ "version": 1,
29
+ "n-threads": 3,
30
+ "backend": {
31
+ "version": 1,
32
+ "type": "QnnHtp",
33
+ "QnnHtp": {
34
+ "version": 1,
35
+ "use-mmap": true,
36
+ "spill-fill-bufsize": 0,
37
+ "mmap-budget": 0,
38
+ "poll": true,
39
+ "cpu-mask": "0xe0",
40
+ "kv-dim": 128,
41
+ "allow-async-init": false
42
+ },
43
+ "extensions": "htp_backend_ext_config.json"
44
+ },
45
+ "model": {
46
+ "version": 1,
47
+ "type": "binary",
48
+ "binary": {
49
+ "version": 1,
50
+ "ctx-bins": [
51
+ "llama_v3_2_3b_instruct_part_1_of_3.bin",
52
+ "llama_v3_2_3b_instruct_part_2_of_3.bin",
53
+ "llama_v3_2_3b_instruct_part_3_of_3.bin"
54
+ ]
55
+ },
56
+ "positional-encoding": {
57
+ "type": "rope",
58
+ "rope-dim": 64,
59
+ "rope-theta": 500000,
60
+ "rope-scaling": {
61
+ "rope-type": "llama3",
62
+ "factor": 8.0,
63
+ "low-freq-factor": 1.0,
64
+ "high-freq-factor": 4.0,
65
+ "original-max-position-embeddings": 8192
66
+ }
67
+ }
68
+ }
69
+ }
70
+ }
71
+ }
htp_backend_ext_config.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"devices": [{"soc_model": 77, "dsp_arch": "v73", "cores": [{"core_id": 0, "perf_profile": "burst", "rpc_control_latency": 100}]}], "memory": {"mem_type": "shared_buffer"}, "context": {"weight_sharing_enabled": true}}
libGenie.so ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c0d13009c32d2548b63de831a515ed893508ca7fa685cf5bd4c327d990c86478
3
+ size 7889648
llama_v3_2_3b_instruct_part_1_of_3.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c4bc997ad4a2ae9a5d0c6f8c74206642c8bc68968c539279d285508d015cb544
3
+ size 788066304
llama_v3_2_3b_instruct_part_2_of_3.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b1cee7231944a0e517583f535a6a71c60873a5a3b3e784cfae8ffbbc1a81a2a7
3
+ size 725401600
llama_v3_2_3b_instruct_part_3_of_3.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:90eeb1f9941185f605a6d3f9b889e77eb8225e4a016fdf229adfc631e5e0761b
3
+ size 1125314560
metadata.yaml ADDED
@@ -0,0 +1,2481 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ components:
2
+ ar128_cl4096_1_of_3:
3
+ inputs:
4
+ input_ids:
5
+ shape:
6
+ - 1
7
+ - 128
8
+ dtype: int32
9
+ outputs:
10
+ /model/model/embed_tokens/Gather_output_0:
11
+ shape:
12
+ - 1
13
+ - 128
14
+ - 3072
15
+ dtype: uint16
16
+ quantization_parameters:
17
+ scale: 8.687509536888685e-06
18
+ offset: -27540
19
+ ar128_cl4096_2_of_3:
20
+ inputs:
21
+ /model/model/embed_tokens/Gather_output_0:
22
+ shape:
23
+ - 1
24
+ - 128
25
+ - 3072
26
+ dtype: uint16
27
+ quantization_parameters:
28
+ scale: 8.687509536888685e-06
29
+ offset: -27540
30
+ attention_mask:
31
+ shape:
32
+ - 1
33
+ - 1
34
+ - 128
35
+ - 4096
36
+ dtype: uint16
37
+ quantization_parameters:
38
+ scale: 0.0007629510948348211
39
+ offset: -65535
40
+ position_ids_cos:
41
+ shape:
42
+ - 1
43
+ - 1
44
+ - 128
45
+ - 64
46
+ dtype: uint16
47
+ quantization_parameters:
48
+ scale: 3.0518043793392844e-05
49
+ offset: -32768
50
+ position_ids_sin:
51
+ shape:
52
+ - 1
53
+ - 1
54
+ - 128
55
+ - 64
56
+ dtype: uint16
57
+ quantization_parameters:
58
+ scale: 3.0518043793392844e-05
59
+ offset: -32768
60
+ past_key_0_in:
61
+ shape:
62
+ - 8
63
+ - 1
64
+ - 128
65
+ - 3968
66
+ dtype: uint8
67
+ quantization_parameters:
68
+ scale: 0.15415094792842865
69
+ offset: -128
70
+ past_value_0_in:
71
+ shape:
72
+ - 8
73
+ - 1
74
+ - 3968
75
+ - 128
76
+ dtype: uint8
77
+ quantization_parameters:
78
+ scale: 0.0076334718614816666
79
+ offset: -128
80
+ past_key_1_in:
81
+ shape:
82
+ - 8
83
+ - 1
84
+ - 128
85
+ - 3968
86
+ dtype: uint8
87
+ quantization_parameters:
88
+ scale: 0.09501257538795471
89
+ offset: -128
90
+ past_value_1_in:
91
+ shape:
92
+ - 8
93
+ - 1
94
+ - 3968
95
+ - 128
96
+ dtype: uint8
97
+ quantization_parameters:
98
+ scale: 0.018255468428604246
99
+ offset: -128
100
+ past_key_2_in:
101
+ shape:
102
+ - 8
103
+ - 1
104
+ - 128
105
+ - 3968
106
+ dtype: uint8
107
+ quantization_parameters:
108
+ scale: 0.12992745637893677
109
+ offset: -128
110
+ past_value_2_in:
111
+ shape:
112
+ - 8
113
+ - 1
114
+ - 3968
115
+ - 128
116
+ dtype: uint8
117
+ quantization_parameters:
118
+ scale: 0.025279987515426997
119
+ offset: -128
120
+ past_key_3_in:
121
+ shape:
122
+ - 8
123
+ - 1
124
+ - 128
125
+ - 3968
126
+ dtype: uint8
127
+ quantization_parameters:
128
+ scale: 0.1447511507770208
129
+ offset: -128
130
+ past_value_3_in:
131
+ shape:
132
+ - 8
133
+ - 1
134
+ - 3968
135
+ - 128
136
+ dtype: uint8
137
+ quantization_parameters:
138
+ scale: 0.028148770332336426
139
+ offset: -128
140
+ past_key_4_in:
141
+ shape:
142
+ - 8
143
+ - 1
144
+ - 128
145
+ - 3968
146
+ dtype: uint8
147
+ quantization_parameters:
148
+ scale: 0.12199506233996293
149
+ offset: -128
150
+ past_value_4_in:
151
+ shape:
152
+ - 8
153
+ - 1
154
+ - 3968
155
+ - 128
156
+ dtype: uint8
157
+ quantization_parameters:
158
+ scale: 0.02944980831596795
159
+ offset: -128
160
+ past_key_5_in:
161
+ shape:
162
+ - 8
163
+ - 1
164
+ - 128
165
+ - 3968
166
+ dtype: uint8
167
+ quantization_parameters:
168
+ scale: 0.12595602869987488
169
+ offset: -128
170
+ past_value_5_in:
171
+ shape:
172
+ - 8
173
+ - 1
174
+ - 3968
175
+ - 128
176
+ dtype: uint8
177
+ quantization_parameters:
178
+ scale: 0.028479063883423805
179
+ offset: -128
180
+ past_key_6_in:
181
+ shape:
182
+ - 8
183
+ - 1
184
+ - 128
185
+ - 3968
186
+ dtype: uint8
187
+ quantization_parameters:
188
+ scale: 0.12635421752929688
189
+ offset: -128
190
+ past_value_6_in:
191
+ shape:
192
+ - 8
193
+ - 1
194
+ - 3968
195
+ - 128
196
+ dtype: uint8
197
+ quantization_parameters:
198
+ scale: 0.025955546647310257
199
+ offset: -128
200
+ past_key_7_in:
201
+ shape:
202
+ - 8
203
+ - 1
204
+ - 128
205
+ - 3968
206
+ dtype: uint8
207
+ quantization_parameters:
208
+ scale: 0.13450774550437927
209
+ offset: -128
210
+ past_value_7_in:
211
+ shape:
212
+ - 8
213
+ - 1
214
+ - 3968
215
+ - 128
216
+ dtype: uint8
217
+ quantization_parameters:
218
+ scale: 0.024671796738632082
219
+ offset: -128
220
+ past_key_8_in:
221
+ shape:
222
+ - 8
223
+ - 1
224
+ - 128
225
+ - 3968
226
+ dtype: uint8
227
+ quantization_parameters:
228
+ scale: 0.1284546439103254
229
+ offset: -128
230
+ past_value_8_in:
231
+ shape:
232
+ - 8
233
+ - 1
234
+ - 3968
235
+ - 128
236
+ dtype: uint8
237
+ quantization_parameters:
238
+ scale: 0.030749136582016945
239
+ offset: -128
240
+ past_key_9_in:
241
+ shape:
242
+ - 8
243
+ - 1
244
+ - 128
245
+ - 3968
246
+ dtype: uint8
247
+ quantization_parameters:
248
+ scale: 0.16326546669006348
249
+ offset: -128
250
+ past_value_9_in:
251
+ shape:
252
+ - 8
253
+ - 1
254
+ - 3968
255
+ - 128
256
+ dtype: uint8
257
+ quantization_parameters:
258
+ scale: 0.03575011125699742
259
+ offset: -128
260
+ past_key_10_in:
261
+ shape:
262
+ - 8
263
+ - 1
264
+ - 128
265
+ - 3968
266
+ dtype: uint8
267
+ quantization_parameters:
268
+ scale: 0.1229240819811821
269
+ offset: -128
270
+ past_value_10_in:
271
+ shape:
272
+ - 8
273
+ - 1
274
+ - 3968
275
+ - 128
276
+ dtype: uint8
277
+ quantization_parameters:
278
+ scale: 0.032141510397195816
279
+ offset: -128
280
+ past_key_11_in:
281
+ shape:
282
+ - 8
283
+ - 1
284
+ - 128
285
+ - 3968
286
+ dtype: uint8
287
+ quantization_parameters:
288
+ scale: 0.13410881161689758
289
+ offset: -128
290
+ past_value_11_in:
291
+ shape:
292
+ - 8
293
+ - 1
294
+ - 3968
295
+ - 128
296
+ dtype: uint8
297
+ quantization_parameters:
298
+ scale: 0.02836606258482445
299
+ offset: -128
300
+ past_key_12_in:
301
+ shape:
302
+ - 8
303
+ - 1
304
+ - 128
305
+ - 3968
306
+ dtype: uint8
307
+ quantization_parameters:
308
+ scale: 0.11923930794000626
309
+ offset: -128
310
+ past_value_12_in:
311
+ shape:
312
+ - 8
313
+ - 1
314
+ - 3968
315
+ - 128
316
+ dtype: uint8
317
+ quantization_parameters:
318
+ scale: 0.03199498108991488
319
+ offset: -128
320
+ past_key_13_in:
321
+ shape:
322
+ - 8
323
+ - 1
324
+ - 128
325
+ - 3968
326
+ dtype: uint8
327
+ quantization_parameters:
328
+ scale: 0.14966769481268455
329
+ offset: -128
330
+ past_value_13_in:
331
+ shape:
332
+ - 8
333
+ - 1
334
+ - 3968
335
+ - 128
336
+ dtype: uint8
337
+ quantization_parameters:
338
+ scale: 0.028661564579160196
339
+ offset: -128
340
+ outputs:
341
+ /model/model/layers.13/Add_1_output_0:
342
+ shape:
343
+ - 1
344
+ - 128
345
+ - 3072
346
+ dtype: uint16
347
+ quantization_parameters:
348
+ scale: 0.011241219636185578
349
+ offset: -31773
350
+ past_key_0_out:
351
+ shape:
352
+ - 8
353
+ - 1
354
+ - 128
355
+ - 128
356
+ dtype: uint8
357
+ quantization_parameters:
358
+ scale: 0.15415094792842865
359
+ offset: -128
360
+ past_value_0_out:
361
+ shape:
362
+ - 8
363
+ - 1
364
+ - 128
365
+ - 128
366
+ dtype: uint8
367
+ quantization_parameters:
368
+ scale: 0.0076334718614816666
369
+ offset: -128
370
+ past_key_1_out:
371
+ shape:
372
+ - 8
373
+ - 1
374
+ - 128
375
+ - 128
376
+ dtype: uint8
377
+ quantization_parameters:
378
+ scale: 0.09501257538795471
379
+ offset: -128
380
+ past_value_1_out:
381
+ shape:
382
+ - 8
383
+ - 1
384
+ - 128
385
+ - 128
386
+ dtype: uint8
387
+ quantization_parameters:
388
+ scale: 0.018255468428604246
389
+ offset: -128
390
+ past_key_2_out:
391
+ shape:
392
+ - 8
393
+ - 1
394
+ - 128
395
+ - 128
396
+ dtype: uint8
397
+ quantization_parameters:
398
+ scale: 0.12992745637893677
399
+ offset: -128
400
+ past_value_2_out:
401
+ shape:
402
+ - 8
403
+ - 1
404
+ - 128
405
+ - 128
406
+ dtype: uint8
407
+ quantization_parameters:
408
+ scale: 0.025279987515426997
409
+ offset: -128
410
+ past_key_3_out:
411
+ shape:
412
+ - 8
413
+ - 1
414
+ - 128
415
+ - 128
416
+ dtype: uint8
417
+ quantization_parameters:
418
+ scale: 0.1447511507770208
419
+ offset: -128
420
+ past_value_3_out:
421
+ shape:
422
+ - 8
423
+ - 1
424
+ - 128
425
+ - 128
426
+ dtype: uint8
427
+ quantization_parameters:
428
+ scale: 0.028148770332336426
429
+ offset: -128
430
+ past_key_4_out:
431
+ shape:
432
+ - 8
433
+ - 1
434
+ - 128
435
+ - 128
436
+ dtype: uint8
437
+ quantization_parameters:
438
+ scale: 0.12199506233996293
439
+ offset: -128
440
+ past_value_4_out:
441
+ shape:
442
+ - 8
443
+ - 1
444
+ - 128
445
+ - 128
446
+ dtype: uint8
447
+ quantization_parameters:
448
+ scale: 0.02944980831596795
449
+ offset: -128
450
+ past_key_5_out:
451
+ shape:
452
+ - 8
453
+ - 1
454
+ - 128
455
+ - 128
456
+ dtype: uint8
457
+ quantization_parameters:
458
+ scale: 0.12595602869987488
459
+ offset: -128
460
+ past_value_5_out:
461
+ shape:
462
+ - 8
463
+ - 1
464
+ - 128
465
+ - 128
466
+ dtype: uint8
467
+ quantization_parameters:
468
+ scale: 0.028479063883423805
469
+ offset: -128
470
+ past_key_6_out:
471
+ shape:
472
+ - 8
473
+ - 1
474
+ - 128
475
+ - 128
476
+ dtype: uint8
477
+ quantization_parameters:
478
+ scale: 0.12635421752929688
479
+ offset: -128
480
+ past_value_6_out:
481
+ shape:
482
+ - 8
483
+ - 1
484
+ - 128
485
+ - 128
486
+ dtype: uint8
487
+ quantization_parameters:
488
+ scale: 0.025955546647310257
489
+ offset: -128
490
+ past_key_7_out:
491
+ shape:
492
+ - 8
493
+ - 1
494
+ - 128
495
+ - 128
496
+ dtype: uint8
497
+ quantization_parameters:
498
+ scale: 0.13450774550437927
499
+ offset: -128
500
+ past_value_7_out:
501
+ shape:
502
+ - 8
503
+ - 1
504
+ - 128
505
+ - 128
506
+ dtype: uint8
507
+ quantization_parameters:
508
+ scale: 0.024671796738632082
509
+ offset: -128
510
+ past_key_8_out:
511
+ shape:
512
+ - 8
513
+ - 1
514
+ - 128
515
+ - 128
516
+ dtype: uint8
517
+ quantization_parameters:
518
+ scale: 0.1284546439103254
519
+ offset: -128
520
+ past_value_8_out:
521
+ shape:
522
+ - 8
523
+ - 1
524
+ - 128
525
+ - 128
526
+ dtype: uint8
527
+ quantization_parameters:
528
+ scale: 0.030749136582016945
529
+ offset: -128
530
+ past_key_9_out:
531
+ shape:
532
+ - 8
533
+ - 1
534
+ - 128
535
+ - 128
536
+ dtype: uint8
537
+ quantization_parameters:
538
+ scale: 0.16326546669006348
539
+ offset: -128
540
+ past_value_9_out:
541
+ shape:
542
+ - 8
543
+ - 1
544
+ - 128
545
+ - 128
546
+ dtype: uint8
547
+ quantization_parameters:
548
+ scale: 0.03575011125699742
549
+ offset: -128
550
+ past_key_10_out:
551
+ shape:
552
+ - 8
553
+ - 1
554
+ - 128
555
+ - 128
556
+ dtype: uint8
557
+ quantization_parameters:
558
+ scale: 0.1229240819811821
559
+ offset: -128
560
+ past_value_10_out:
561
+ shape:
562
+ - 8
563
+ - 1
564
+ - 128
565
+ - 128
566
+ dtype: uint8
567
+ quantization_parameters:
568
+ scale: 0.032141510397195816
569
+ offset: -128
570
+ past_key_11_out:
571
+ shape:
572
+ - 8
573
+ - 1
574
+ - 128
575
+ - 128
576
+ dtype: uint8
577
+ quantization_parameters:
578
+ scale: 0.13410881161689758
579
+ offset: -128
580
+ past_value_11_out:
581
+ shape:
582
+ - 8
583
+ - 1
584
+ - 128
585
+ - 128
586
+ dtype: uint8
587
+ quantization_parameters:
588
+ scale: 0.02836606258482445
589
+ offset: -128
590
+ past_key_12_out:
591
+ shape:
592
+ - 8
593
+ - 1
594
+ - 128
595
+ - 128
596
+ dtype: uint8
597
+ quantization_parameters:
598
+ scale: 0.11923930794000626
599
+ offset: -128
600
+ past_value_12_out:
601
+ shape:
602
+ - 8
603
+ - 1
604
+ - 128
605
+ - 128
606
+ dtype: uint8
607
+ quantization_parameters:
608
+ scale: 0.03199498108991488
609
+ offset: -128
610
+ past_key_13_out:
611
+ shape:
612
+ - 8
613
+ - 1
614
+ - 128
615
+ - 128
616
+ dtype: uint8
617
+ quantization_parameters:
618
+ scale: 0.14966769481268455
619
+ offset: -128
620
+ past_value_13_out:
621
+ shape:
622
+ - 8
623
+ - 1
624
+ - 128
625
+ - 128
626
+ dtype: uint8
627
+ quantization_parameters:
628
+ scale: 0.028661564579160196
629
+ offset: -128
630
+ ar128_cl4096_3_of_3:
631
+ inputs:
632
+ /model/model/layers.13/Add_1_output_0:
633
+ shape:
634
+ - 1
635
+ - 128
636
+ - 3072
637
+ dtype: uint16
638
+ quantization_parameters:
639
+ scale: 0.011241219636185578
640
+ offset: -31773
641
+ attention_mask:
642
+ shape:
643
+ - 1
644
+ - 1
645
+ - 128
646
+ - 4096
647
+ dtype: uint16
648
+ quantization_parameters:
649
+ scale: 0.0007629510948348211
650
+ offset: -65535
651
+ position_ids_cos:
652
+ shape:
653
+ - 1
654
+ - 1
655
+ - 128
656
+ - 64
657
+ dtype: uint16
658
+ quantization_parameters:
659
+ scale: 3.0518043793392844e-05
660
+ offset: -32768
661
+ position_ids_sin:
662
+ shape:
663
+ - 1
664
+ - 1
665
+ - 128
666
+ - 64
667
+ dtype: uint16
668
+ quantization_parameters:
669
+ scale: 3.0518043793392844e-05
670
+ offset: -32768
671
+ past_key_14_in:
672
+ shape:
673
+ - 8
674
+ - 1
675
+ - 128
676
+ - 3968
677
+ dtype: uint8
678
+ quantization_parameters:
679
+ scale: 0.13996938269908035
680
+ offset: -128
681
+ past_value_14_in:
682
+ shape:
683
+ - 8
684
+ - 1
685
+ - 3968
686
+ - 128
687
+ dtype: uint8
688
+ quantization_parameters:
689
+ scale: 0.028563514469176765
690
+ offset: -128
691
+ past_key_15_in:
692
+ shape:
693
+ - 8
694
+ - 1
695
+ - 128
696
+ - 3968
697
+ dtype: uint8
698
+ quantization_parameters:
699
+ scale: 0.125163213474544
700
+ offset: -128
701
+ past_value_15_in:
702
+ shape:
703
+ - 8
704
+ - 1
705
+ - 3968
706
+ - 128
707
+ dtype: uint8
708
+ quantization_parameters:
709
+ scale: 0.027315943259892502
710
+ offset: -128
711
+ past_key_16_in:
712
+ shape:
713
+ - 8
714
+ - 1
715
+ - 128
716
+ - 3968
717
+ dtype: uint8
718
+ quantization_parameters:
719
+ scale: 0.12602750522883857
720
+ offset: -128
721
+ past_value_16_in:
722
+ shape:
723
+ - 8
724
+ - 1
725
+ - 3968
726
+ - 128
727
+ dtype: uint8
728
+ quantization_parameters:
729
+ scale: 0.033199068158864975
730
+ offset: -128
731
+ past_key_17_in:
732
+ shape:
733
+ - 8
734
+ - 1
735
+ - 128
736
+ - 3968
737
+ dtype: uint8
738
+ quantization_parameters:
739
+ scale: 0.13656628131866455
740
+ offset: -128
741
+ past_value_17_in:
742
+ shape:
743
+ - 8
744
+ - 1
745
+ - 3968
746
+ - 128
747
+ dtype: uint8
748
+ quantization_parameters:
749
+ scale: 0.030598552208247146
750
+ offset: -128
751
+ past_key_18_in:
752
+ shape:
753
+ - 8
754
+ - 1
755
+ - 128
756
+ - 3968
757
+ dtype: uint8
758
+ quantization_parameters:
759
+ scale: 0.11889628913458877
760
+ offset: -128
761
+ past_value_18_in:
762
+ shape:
763
+ - 8
764
+ - 1
765
+ - 3968
766
+ - 128
767
+ dtype: uint8
768
+ quantization_parameters:
769
+ scale: 0.032141249626874924
770
+ offset: -128
771
+ past_key_19_in:
772
+ shape:
773
+ - 8
774
+ - 1
775
+ - 128
776
+ - 3968
777
+ dtype: uint8
778
+ quantization_parameters:
779
+ scale: 0.13976185023784637
780
+ offset: -128
781
+ past_value_19_in:
782
+ shape:
783
+ - 8
784
+ - 1
785
+ - 3968
786
+ - 128
787
+ dtype: uint8
788
+ quantization_parameters:
789
+ scale: 0.032557712765190545
790
+ offset: -128
791
+ past_key_20_in:
792
+ shape:
793
+ - 8
794
+ - 1
795
+ - 128
796
+ - 3968
797
+ dtype: uint8
798
+ quantization_parameters:
799
+ scale: 0.11934278142733837
800
+ offset: -128
801
+ past_value_20_in:
802
+ shape:
803
+ - 8
804
+ - 1
805
+ - 3968
806
+ - 128
807
+ dtype: uint8
808
+ quantization_parameters:
809
+ scale: 0.027935825288295746
810
+ offset: -128
811
+ past_key_21_in:
812
+ shape:
813
+ - 8
814
+ - 1
815
+ - 128
816
+ - 3968
817
+ dtype: uint8
818
+ quantization_parameters:
819
+ scale: 0.13167049377922
820
+ offset: -128
821
+ past_value_21_in:
822
+ shape:
823
+ - 8
824
+ - 1
825
+ - 3968
826
+ - 128
827
+ dtype: uint8
828
+ quantization_parameters:
829
+ scale: 0.04469357803463936
830
+ offset: -128
831
+ past_key_22_in:
832
+ shape:
833
+ - 8
834
+ - 1
835
+ - 128
836
+ - 3968
837
+ dtype: uint8
838
+ quantization_parameters:
839
+ scale: 0.13328912689929873
840
+ offset: -128
841
+ past_value_22_in:
842
+ shape:
843
+ - 8
844
+ - 1
845
+ - 3968
846
+ - 128
847
+ dtype: uint8
848
+ quantization_parameters:
849
+ scale: 0.03745593875646591
850
+ offset: -128
851
+ past_key_23_in:
852
+ shape:
853
+ - 8
854
+ - 1
855
+ - 128
856
+ - 3968
857
+ dtype: uint8
858
+ quantization_parameters:
859
+ scale: 0.12827403151144193
860
+ offset: -128
861
+ past_value_23_in:
862
+ shape:
863
+ - 8
864
+ - 1
865
+ - 3968
866
+ - 128
867
+ dtype: uint8
868
+ quantization_parameters:
869
+ scale: 0.032266516238451004
870
+ offset: -128
871
+ past_key_24_in:
872
+ shape:
873
+ - 8
874
+ - 1
875
+ - 128
876
+ - 3968
877
+ dtype: uint8
878
+ quantization_parameters:
879
+ scale: 0.14478352665901184
880
+ offset: -128
881
+ past_value_24_in:
882
+ shape:
883
+ - 8
884
+ - 1
885
+ - 3968
886
+ - 128
887
+ dtype: uint8
888
+ quantization_parameters:
889
+ scale: 0.04626827540360098
890
+ offset: -128
891
+ past_key_25_in:
892
+ shape:
893
+ - 8
894
+ - 1
895
+ - 128
896
+ - 3968
897
+ dtype: uint8
898
+ quantization_parameters:
899
+ scale: 0.17984355716254768
900
+ offset: -128
901
+ past_value_25_in:
902
+ shape:
903
+ - 8
904
+ - 1
905
+ - 3968
906
+ - 128
907
+ dtype: uint8
908
+ quantization_parameters:
909
+ scale: 0.04539092882411686
910
+ offset: -128
911
+ past_key_26_in:
912
+ shape:
913
+ - 8
914
+ - 1
915
+ - 128
916
+ - 3968
917
+ dtype: uint8
918
+ quantization_parameters:
919
+ scale: 0.1248558149563046
920
+ offset: -128
921
+ past_value_26_in:
922
+ shape:
923
+ - 8
924
+ - 1
925
+ - 3968
926
+ - 128
927
+ dtype: uint8
928
+ quantization_parameters:
929
+ scale: 0.05314682051539421
930
+ offset: -128
931
+ past_key_27_in:
932
+ shape:
933
+ - 8
934
+ - 1
935
+ - 128
936
+ - 3968
937
+ dtype: uint8
938
+ quantization_parameters:
939
+ scale: 0.13326138257980347
940
+ offset: -128
941
+ past_value_27_in:
942
+ shape:
943
+ - 8
944
+ - 1
945
+ - 3968
946
+ - 128
947
+ dtype: uint8
948
+ quantization_parameters:
949
+ scale: 0.04699636995792389
950
+ offset: -128
951
+ outputs:
952
+ logits:
953
+ shape:
954
+ - 1
955
+ - 128
956
+ - 128256
957
+ dtype: uint16
958
+ quantization_parameters:
959
+ scale: 0.0006455201492198974
960
+ offset: -24293
961
+ past_key_14_out:
962
+ shape:
963
+ - 8
964
+ - 1
965
+ - 128
966
+ - 128
967
+ dtype: uint8
968
+ quantization_parameters:
969
+ scale: 0.13996938269908035
970
+ offset: -128
971
+ past_value_14_out:
972
+ shape:
973
+ - 8
974
+ - 1
975
+ - 128
976
+ - 128
977
+ dtype: uint8
978
+ quantization_parameters:
979
+ scale: 0.028563514469176765
980
+ offset: -128
981
+ past_key_15_out:
982
+ shape:
983
+ - 8
984
+ - 1
985
+ - 128
986
+ - 128
987
+ dtype: uint8
988
+ quantization_parameters:
989
+ scale: 0.125163213474544
990
+ offset: -128
991
+ past_value_15_out:
992
+ shape:
993
+ - 8
994
+ - 1
995
+ - 128
996
+ - 128
997
+ dtype: uint8
998
+ quantization_parameters:
999
+ scale: 0.027315943259892502
1000
+ offset: -128
1001
+ past_key_16_out:
1002
+ shape:
1003
+ - 8
1004
+ - 1
1005
+ - 128
1006
+ - 128
1007
+ dtype: uint8
1008
+ quantization_parameters:
1009
+ scale: 0.12602750522883857
1010
+ offset: -128
1011
+ past_value_16_out:
1012
+ shape:
1013
+ - 8
1014
+ - 1
1015
+ - 128
1016
+ - 128
1017
+ dtype: uint8
1018
+ quantization_parameters:
1019
+ scale: 0.033199068158864975
1020
+ offset: -128
1021
+ past_key_17_out:
1022
+ shape:
1023
+ - 8
1024
+ - 1
1025
+ - 128
1026
+ - 128
1027
+ dtype: uint8
1028
+ quantization_parameters:
1029
+ scale: 0.13656628131866455
1030
+ offset: -128
1031
+ past_value_17_out:
1032
+ shape:
1033
+ - 8
1034
+ - 1
1035
+ - 128
1036
+ - 128
1037
+ dtype: uint8
1038
+ quantization_parameters:
1039
+ scale: 0.030598552208247146
1040
+ offset: -128
1041
+ past_key_18_out:
1042
+ shape:
1043
+ - 8
1044
+ - 1
1045
+ - 128
1046
+ - 128
1047
+ dtype: uint8
1048
+ quantization_parameters:
1049
+ scale: 0.11889628913458877
1050
+ offset: -128
1051
+ past_value_18_out:
1052
+ shape:
1053
+ - 8
1054
+ - 1
1055
+ - 128
1056
+ - 128
1057
+ dtype: uint8
1058
+ quantization_parameters:
1059
+ scale: 0.032141249626874924
1060
+ offset: -128
1061
+ past_key_19_out:
1062
+ shape:
1063
+ - 8
1064
+ - 1
1065
+ - 128
1066
+ - 128
1067
+ dtype: uint8
1068
+ quantization_parameters:
1069
+ scale: 0.13976185023784637
1070
+ offset: -128
1071
+ past_value_19_out:
1072
+ shape:
1073
+ - 8
1074
+ - 1
1075
+ - 128
1076
+ - 128
1077
+ dtype: uint8
1078
+ quantization_parameters:
1079
+ scale: 0.032557712765190545
1080
+ offset: -128
1081
+ past_key_20_out:
1082
+ shape:
1083
+ - 8
1084
+ - 1
1085
+ - 128
1086
+ - 128
1087
+ dtype: uint8
1088
+ quantization_parameters:
1089
+ scale: 0.11934278142733837
1090
+ offset: -128
1091
+ past_value_20_out:
1092
+ shape:
1093
+ - 8
1094
+ - 1
1095
+ - 128
1096
+ - 128
1097
+ dtype: uint8
1098
+ quantization_parameters:
1099
+ scale: 0.027935825288295746
1100
+ offset: -128
1101
+ past_key_21_out:
1102
+ shape:
1103
+ - 8
1104
+ - 1
1105
+ - 128
1106
+ - 128
1107
+ dtype: uint8
1108
+ quantization_parameters:
1109
+ scale: 0.13167049377922
1110
+ offset: -128
1111
+ past_value_21_out:
1112
+ shape:
1113
+ - 8
1114
+ - 1
1115
+ - 128
1116
+ - 128
1117
+ dtype: uint8
1118
+ quantization_parameters:
1119
+ scale: 0.04469357803463936
1120
+ offset: -128
1121
+ past_key_22_out:
1122
+ shape:
1123
+ - 8
1124
+ - 1
1125
+ - 128
1126
+ - 128
1127
+ dtype: uint8
1128
+ quantization_parameters:
1129
+ scale: 0.13328912689929873
1130
+ offset: -128
1131
+ past_value_22_out:
1132
+ shape:
1133
+ - 8
1134
+ - 1
1135
+ - 128
1136
+ - 128
1137
+ dtype: uint8
1138
+ quantization_parameters:
1139
+ scale: 0.03745593875646591
1140
+ offset: -128
1141
+ past_key_23_out:
1142
+ shape:
1143
+ - 8
1144
+ - 1
1145
+ - 128
1146
+ - 128
1147
+ dtype: uint8
1148
+ quantization_parameters:
1149
+ scale: 0.12827403151144193
1150
+ offset: -128
1151
+ past_value_23_out:
1152
+ shape:
1153
+ - 8
1154
+ - 1
1155
+ - 128
1156
+ - 128
1157
+ dtype: uint8
1158
+ quantization_parameters:
1159
+ scale: 0.032266516238451004
1160
+ offset: -128
1161
+ past_key_24_out:
1162
+ shape:
1163
+ - 8
1164
+ - 1
1165
+ - 128
1166
+ - 128
1167
+ dtype: uint8
1168
+ quantization_parameters:
1169
+ scale: 0.14478352665901184
1170
+ offset: -128
1171
+ past_value_24_out:
1172
+ shape:
1173
+ - 8
1174
+ - 1
1175
+ - 128
1176
+ - 128
1177
+ dtype: uint8
1178
+ quantization_parameters:
1179
+ scale: 0.04626827540360098
1180
+ offset: -128
1181
+ past_key_25_out:
1182
+ shape:
1183
+ - 8
1184
+ - 1
1185
+ - 128
1186
+ - 128
1187
+ dtype: uint8
1188
+ quantization_parameters:
1189
+ scale: 0.17984355716254768
1190
+ offset: -128
1191
+ past_value_25_out:
1192
+ shape:
1193
+ - 8
1194
+ - 1
1195
+ - 128
1196
+ - 128
1197
+ dtype: uint8
1198
+ quantization_parameters:
1199
+ scale: 0.04539092882411686
1200
+ offset: -128
1201
+ past_key_26_out:
1202
+ shape:
1203
+ - 8
1204
+ - 1
1205
+ - 128
1206
+ - 128
1207
+ dtype: uint8
1208
+ quantization_parameters:
1209
+ scale: 0.1248558149563046
1210
+ offset: -128
1211
+ past_value_26_out:
1212
+ shape:
1213
+ - 8
1214
+ - 1
1215
+ - 128
1216
+ - 128
1217
+ dtype: uint8
1218
+ quantization_parameters:
1219
+ scale: 0.05314682051539421
1220
+ offset: -128
1221
+ past_key_27_out:
1222
+ shape:
1223
+ - 8
1224
+ - 1
1225
+ - 128
1226
+ - 128
1227
+ dtype: uint8
1228
+ quantization_parameters:
1229
+ scale: 0.13326138257980347
1230
+ offset: -128
1231
+ past_value_27_out:
1232
+ shape:
1233
+ - 8
1234
+ - 1
1235
+ - 128
1236
+ - 128
1237
+ dtype: uint8
1238
+ quantization_parameters:
1239
+ scale: 0.04699636995792389
1240
+ offset: -128
1241
+ ar1_cl4096_1_of_3:
1242
+ inputs:
1243
+ input_ids:
1244
+ shape:
1245
+ - 1
1246
+ - 1
1247
+ dtype: int32
1248
+ outputs:
1249
+ /model/model/embed_tokens/Gather_output_0:
1250
+ shape:
1251
+ - 1
1252
+ - 1
1253
+ - 3072
1254
+ dtype: uint16
1255
+ quantization_parameters:
1256
+ scale: 8.687509536888685e-06
1257
+ offset: -27540
1258
+ ar1_cl4096_2_of_3:
1259
+ inputs:
1260
+ /model/model/embed_tokens/Gather_output_0:
1261
+ shape:
1262
+ - 1
1263
+ - 1
1264
+ - 3072
1265
+ dtype: uint16
1266
+ quantization_parameters:
1267
+ scale: 8.687509536888685e-06
1268
+ offset: -27540
1269
+ attention_mask:
1270
+ shape:
1271
+ - 1
1272
+ - 1
1273
+ - 1
1274
+ - 4096
1275
+ dtype: uint16
1276
+ quantization_parameters:
1277
+ scale: 0.0007629510948348211
1278
+ offset: -65535
1279
+ position_ids_cos:
1280
+ shape:
1281
+ - 1
1282
+ - 1
1283
+ - 1
1284
+ - 64
1285
+ dtype: uint16
1286
+ quantization_parameters:
1287
+ scale: 3.0518043793392844e-05
1288
+ offset: -32768
1289
+ position_ids_sin:
1290
+ shape:
1291
+ - 1
1292
+ - 1
1293
+ - 1
1294
+ - 64
1295
+ dtype: uint16
1296
+ quantization_parameters:
1297
+ scale: 3.0518043793392844e-05
1298
+ offset: -32768
1299
+ past_key_0_in:
1300
+ shape:
1301
+ - 8
1302
+ - 1
1303
+ - 128
1304
+ - 4095
1305
+ dtype: uint8
1306
+ quantization_parameters:
1307
+ scale: 0.15415094792842865
1308
+ offset: -128
1309
+ past_value_0_in:
1310
+ shape:
1311
+ - 8
1312
+ - 1
1313
+ - 4095
1314
+ - 128
1315
+ dtype: uint8
1316
+ quantization_parameters:
1317
+ scale: 0.0076334718614816666
1318
+ offset: -128
1319
+ past_key_1_in:
1320
+ shape:
1321
+ - 8
1322
+ - 1
1323
+ - 128
1324
+ - 4095
1325
+ dtype: uint8
1326
+ quantization_parameters:
1327
+ scale: 0.09501257538795471
1328
+ offset: -128
1329
+ past_value_1_in:
1330
+ shape:
1331
+ - 8
1332
+ - 1
1333
+ - 4095
1334
+ - 128
1335
+ dtype: uint8
1336
+ quantization_parameters:
1337
+ scale: 0.018255468428604246
1338
+ offset: -128
1339
+ past_key_2_in:
1340
+ shape:
1341
+ - 8
1342
+ - 1
1343
+ - 128
1344
+ - 4095
1345
+ dtype: uint8
1346
+ quantization_parameters:
1347
+ scale: 0.12992745637893677
1348
+ offset: -128
1349
+ past_value_2_in:
1350
+ shape:
1351
+ - 8
1352
+ - 1
1353
+ - 4095
1354
+ - 128
1355
+ dtype: uint8
1356
+ quantization_parameters:
1357
+ scale: 0.025279987515426997
1358
+ offset: -128
1359
+ past_key_3_in:
1360
+ shape:
1361
+ - 8
1362
+ - 1
1363
+ - 128
1364
+ - 4095
1365
+ dtype: uint8
1366
+ quantization_parameters:
1367
+ scale: 0.1447511507770208
1368
+ offset: -128
1369
+ past_value_3_in:
1370
+ shape:
1371
+ - 8
1372
+ - 1
1373
+ - 4095
1374
+ - 128
1375
+ dtype: uint8
1376
+ quantization_parameters:
1377
+ scale: 0.028148770332336426
1378
+ offset: -128
1379
+ past_key_4_in:
1380
+ shape:
1381
+ - 8
1382
+ - 1
1383
+ - 128
1384
+ - 4095
1385
+ dtype: uint8
1386
+ quantization_parameters:
1387
+ scale: 0.12199506233996293
1388
+ offset: -128
1389
+ past_value_4_in:
1390
+ shape:
1391
+ - 8
1392
+ - 1
1393
+ - 4095
1394
+ - 128
1395
+ dtype: uint8
1396
+ quantization_parameters:
1397
+ scale: 0.02944980831596795
1398
+ offset: -128
1399
+ past_key_5_in:
1400
+ shape:
1401
+ - 8
1402
+ - 1
1403
+ - 128
1404
+ - 4095
1405
+ dtype: uint8
1406
+ quantization_parameters:
1407
+ scale: 0.12595602869987488
1408
+ offset: -128
1409
+ past_value_5_in:
1410
+ shape:
1411
+ - 8
1412
+ - 1
1413
+ - 4095
1414
+ - 128
1415
+ dtype: uint8
1416
+ quantization_parameters:
1417
+ scale: 0.028479063883423805
1418
+ offset: -128
1419
+ past_key_6_in:
1420
+ shape:
1421
+ - 8
1422
+ - 1
1423
+ - 128
1424
+ - 4095
1425
+ dtype: uint8
1426
+ quantization_parameters:
1427
+ scale: 0.12635421752929688
1428
+ offset: -128
1429
+ past_value_6_in:
1430
+ shape:
1431
+ - 8
1432
+ - 1
1433
+ - 4095
1434
+ - 128
1435
+ dtype: uint8
1436
+ quantization_parameters:
1437
+ scale: 0.025955546647310257
1438
+ offset: -128
1439
+ past_key_7_in:
1440
+ shape:
1441
+ - 8
1442
+ - 1
1443
+ - 128
1444
+ - 4095
1445
+ dtype: uint8
1446
+ quantization_parameters:
1447
+ scale: 0.13450774550437927
1448
+ offset: -128
1449
+ past_value_7_in:
1450
+ shape:
1451
+ - 8
1452
+ - 1
1453
+ - 4095
1454
+ - 128
1455
+ dtype: uint8
1456
+ quantization_parameters:
1457
+ scale: 0.024671796738632082
1458
+ offset: -128
1459
+ past_key_8_in:
1460
+ shape:
1461
+ - 8
1462
+ - 1
1463
+ - 128
1464
+ - 4095
1465
+ dtype: uint8
1466
+ quantization_parameters:
1467
+ scale: 0.1284546439103254
1468
+ offset: -128
1469
+ past_value_8_in:
1470
+ shape:
1471
+ - 8
1472
+ - 1
1473
+ - 4095
1474
+ - 128
1475
+ dtype: uint8
1476
+ quantization_parameters:
1477
+ scale: 0.030749136582016945
1478
+ offset: -128
1479
+ past_key_9_in:
1480
+ shape:
1481
+ - 8
1482
+ - 1
1483
+ - 128
1484
+ - 4095
1485
+ dtype: uint8
1486
+ quantization_parameters:
1487
+ scale: 0.16326546669006348
1488
+ offset: -128
1489
+ past_value_9_in:
1490
+ shape:
1491
+ - 8
1492
+ - 1
1493
+ - 4095
1494
+ - 128
1495
+ dtype: uint8
1496
+ quantization_parameters:
1497
+ scale: 0.03575011125699742
1498
+ offset: -128
1499
+ past_key_10_in:
1500
+ shape:
1501
+ - 8
1502
+ - 1
1503
+ - 128
1504
+ - 4095
1505
+ dtype: uint8
1506
+ quantization_parameters:
1507
+ scale: 0.1229240819811821
1508
+ offset: -128
1509
+ past_value_10_in:
1510
+ shape:
1511
+ - 8
1512
+ - 1
1513
+ - 4095
1514
+ - 128
1515
+ dtype: uint8
1516
+ quantization_parameters:
1517
+ scale: 0.032141510397195816
1518
+ offset: -128
1519
+ past_key_11_in:
1520
+ shape:
1521
+ - 8
1522
+ - 1
1523
+ - 128
1524
+ - 4095
1525
+ dtype: uint8
1526
+ quantization_parameters:
1527
+ scale: 0.13410881161689758
1528
+ offset: -128
1529
+ past_value_11_in:
1530
+ shape:
1531
+ - 8
1532
+ - 1
1533
+ - 4095
1534
+ - 128
1535
+ dtype: uint8
1536
+ quantization_parameters:
1537
+ scale: 0.02836606258482445
1538
+ offset: -128
1539
+ past_key_12_in:
1540
+ shape:
1541
+ - 8
1542
+ - 1
1543
+ - 128
1544
+ - 4095
1545
+ dtype: uint8
1546
+ quantization_parameters:
1547
+ scale: 0.11923930794000626
1548
+ offset: -128
1549
+ past_value_12_in:
1550
+ shape:
1551
+ - 8
1552
+ - 1
1553
+ - 4095
1554
+ - 128
1555
+ dtype: uint8
1556
+ quantization_parameters:
1557
+ scale: 0.03199498108991488
1558
+ offset: -128
1559
+ past_key_13_in:
1560
+ shape:
1561
+ - 8
1562
+ - 1
1563
+ - 128
1564
+ - 4095
1565
+ dtype: uint8
1566
+ quantization_parameters:
1567
+ scale: 0.14966769481268455
1568
+ offset: -128
1569
+ past_value_13_in:
1570
+ shape:
1571
+ - 8
1572
+ - 1
1573
+ - 4095
1574
+ - 128
1575
+ dtype: uint8
1576
+ quantization_parameters:
1577
+ scale: 0.028661564579160196
1578
+ offset: -128
1579
+ outputs:
1580
+ /model/model/layers.13/Add_1_output_0:
1581
+ shape:
1582
+ - 1
1583
+ - 1
1584
+ - 3072
1585
+ dtype: uint16
1586
+ quantization_parameters:
1587
+ scale: 0.011241219636185578
1588
+ offset: -31773
1589
+ past_key_0_out:
1590
+ shape:
1591
+ - 8
1592
+ - 1
1593
+ - 128
1594
+ - 1
1595
+ dtype: uint8
1596
+ quantization_parameters:
1597
+ scale: 0.15415094792842865
1598
+ offset: -128
1599
+ past_value_0_out:
1600
+ shape:
1601
+ - 8
1602
+ - 1
1603
+ - 1
1604
+ - 128
1605
+ dtype: uint8
1606
+ quantization_parameters:
1607
+ scale: 0.0076334718614816666
1608
+ offset: -128
1609
+ past_key_1_out:
1610
+ shape:
1611
+ - 8
1612
+ - 1
1613
+ - 128
1614
+ - 1
1615
+ dtype: uint8
1616
+ quantization_parameters:
1617
+ scale: 0.09501257538795471
1618
+ offset: -128
1619
+ past_value_1_out:
1620
+ shape:
1621
+ - 8
1622
+ - 1
1623
+ - 1
1624
+ - 128
1625
+ dtype: uint8
1626
+ quantization_parameters:
1627
+ scale: 0.018255468428604246
1628
+ offset: -128
1629
+ past_key_2_out:
1630
+ shape:
1631
+ - 8
1632
+ - 1
1633
+ - 128
1634
+ - 1
1635
+ dtype: uint8
1636
+ quantization_parameters:
1637
+ scale: 0.12992745637893677
1638
+ offset: -128
1639
+ past_value_2_out:
1640
+ shape:
1641
+ - 8
1642
+ - 1
1643
+ - 1
1644
+ - 128
1645
+ dtype: uint8
1646
+ quantization_parameters:
1647
+ scale: 0.025279987515426997
1648
+ offset: -128
1649
+ past_key_3_out:
1650
+ shape:
1651
+ - 8
1652
+ - 1
1653
+ - 128
1654
+ - 1
1655
+ dtype: uint8
1656
+ quantization_parameters:
1657
+ scale: 0.1447511507770208
1658
+ offset: -128
1659
+ past_value_3_out:
1660
+ shape:
1661
+ - 8
1662
+ - 1
1663
+ - 1
1664
+ - 128
1665
+ dtype: uint8
1666
+ quantization_parameters:
1667
+ scale: 0.028148770332336426
1668
+ offset: -128
1669
+ past_key_4_out:
1670
+ shape:
1671
+ - 8
1672
+ - 1
1673
+ - 128
1674
+ - 1
1675
+ dtype: uint8
1676
+ quantization_parameters:
1677
+ scale: 0.12199506233996293
1678
+ offset: -128
1679
+ past_value_4_out:
1680
+ shape:
1681
+ - 8
1682
+ - 1
1683
+ - 1
1684
+ - 128
1685
+ dtype: uint8
1686
+ quantization_parameters:
1687
+ scale: 0.02944980831596795
1688
+ offset: -128
1689
+ past_key_5_out:
1690
+ shape:
1691
+ - 8
1692
+ - 1
1693
+ - 128
1694
+ - 1
1695
+ dtype: uint8
1696
+ quantization_parameters:
1697
+ scale: 0.12595602869987488
1698
+ offset: -128
1699
+ past_value_5_out:
1700
+ shape:
1701
+ - 8
1702
+ - 1
1703
+ - 1
1704
+ - 128
1705
+ dtype: uint8
1706
+ quantization_parameters:
1707
+ scale: 0.028479063883423805
1708
+ offset: -128
1709
+ past_key_6_out:
1710
+ shape:
1711
+ - 8
1712
+ - 1
1713
+ - 128
1714
+ - 1
1715
+ dtype: uint8
1716
+ quantization_parameters:
1717
+ scale: 0.12635421752929688
1718
+ offset: -128
1719
+ past_value_6_out:
1720
+ shape:
1721
+ - 8
1722
+ - 1
1723
+ - 1
1724
+ - 128
1725
+ dtype: uint8
1726
+ quantization_parameters:
1727
+ scale: 0.025955546647310257
1728
+ offset: -128
1729
+ past_key_7_out:
1730
+ shape:
1731
+ - 8
1732
+ - 1
1733
+ - 128
1734
+ - 1
1735
+ dtype: uint8
1736
+ quantization_parameters:
1737
+ scale: 0.13450774550437927
1738
+ offset: -128
1739
+ past_value_7_out:
1740
+ shape:
1741
+ - 8
1742
+ - 1
1743
+ - 1
1744
+ - 128
1745
+ dtype: uint8
1746
+ quantization_parameters:
1747
+ scale: 0.024671796738632082
1748
+ offset: -128
1749
+ past_key_8_out:
1750
+ shape:
1751
+ - 8
1752
+ - 1
1753
+ - 128
1754
+ - 1
1755
+ dtype: uint8
1756
+ quantization_parameters:
1757
+ scale: 0.1284546439103254
1758
+ offset: -128
1759
+ past_value_8_out:
1760
+ shape:
1761
+ - 8
1762
+ - 1
1763
+ - 1
1764
+ - 128
1765
+ dtype: uint8
1766
+ quantization_parameters:
1767
+ scale: 0.030749136582016945
1768
+ offset: -128
1769
+ past_key_9_out:
1770
+ shape:
1771
+ - 8
1772
+ - 1
1773
+ - 128
1774
+ - 1
1775
+ dtype: uint8
1776
+ quantization_parameters:
1777
+ scale: 0.16326546669006348
1778
+ offset: -128
1779
+ past_value_9_out:
1780
+ shape:
1781
+ - 8
1782
+ - 1
1783
+ - 1
1784
+ - 128
1785
+ dtype: uint8
1786
+ quantization_parameters:
1787
+ scale: 0.03575011125699742
1788
+ offset: -128
1789
+ past_key_10_out:
1790
+ shape:
1791
+ - 8
1792
+ - 1
1793
+ - 128
1794
+ - 1
1795
+ dtype: uint8
1796
+ quantization_parameters:
1797
+ scale: 0.1229240819811821
1798
+ offset: -128
1799
+ past_value_10_out:
1800
+ shape:
1801
+ - 8
1802
+ - 1
1803
+ - 1
1804
+ - 128
1805
+ dtype: uint8
1806
+ quantization_parameters:
1807
+ scale: 0.032141510397195816
1808
+ offset: -128
1809
+ past_key_11_out:
1810
+ shape:
1811
+ - 8
1812
+ - 1
1813
+ - 128
1814
+ - 1
1815
+ dtype: uint8
1816
+ quantization_parameters:
1817
+ scale: 0.13410881161689758
1818
+ offset: -128
1819
+ past_value_11_out:
1820
+ shape:
1821
+ - 8
1822
+ - 1
1823
+ - 1
1824
+ - 128
1825
+ dtype: uint8
1826
+ quantization_parameters:
1827
+ scale: 0.02836606258482445
1828
+ offset: -128
1829
+ past_key_12_out:
1830
+ shape:
1831
+ - 8
1832
+ - 1
1833
+ - 128
1834
+ - 1
1835
+ dtype: uint8
1836
+ quantization_parameters:
1837
+ scale: 0.11923930794000626
1838
+ offset: -128
1839
+ past_value_12_out:
1840
+ shape:
1841
+ - 8
1842
+ - 1
1843
+ - 1
1844
+ - 128
1845
+ dtype: uint8
1846
+ quantization_parameters:
1847
+ scale: 0.03199498108991488
1848
+ offset: -128
1849
+ past_key_13_out:
1850
+ shape:
1851
+ - 8
1852
+ - 1
1853
+ - 128
1854
+ - 1
1855
+ dtype: uint8
1856
+ quantization_parameters:
1857
+ scale: 0.14966769481268455
1858
+ offset: -128
1859
+ past_value_13_out:
1860
+ shape:
1861
+ - 8
1862
+ - 1
1863
+ - 1
1864
+ - 128
1865
+ dtype: uint8
1866
+ quantization_parameters:
1867
+ scale: 0.028661564579160196
1868
+ offset: -128
1869
+ ar1_cl4096_3_of_3:
1870
+ inputs:
1871
+ /model/model/layers.13/Add_1_output_0:
1872
+ shape:
1873
+ - 1
1874
+ - 1
1875
+ - 3072
1876
+ dtype: uint16
1877
+ quantization_parameters:
1878
+ scale: 0.011241219636185578
1879
+ offset: -31773
1880
+ attention_mask:
1881
+ shape:
1882
+ - 1
1883
+ - 1
1884
+ - 1
1885
+ - 4096
1886
+ dtype: uint16
1887
+ quantization_parameters:
1888
+ scale: 0.0007629510948348211
1889
+ offset: -65535
1890
+ position_ids_cos:
1891
+ shape:
1892
+ - 1
1893
+ - 1
1894
+ - 1
1895
+ - 64
1896
+ dtype: uint16
1897
+ quantization_parameters:
1898
+ scale: 3.0518043793392844e-05
1899
+ offset: -32768
1900
+ position_ids_sin:
1901
+ shape:
1902
+ - 1
1903
+ - 1
1904
+ - 1
1905
+ - 64
1906
+ dtype: uint16
1907
+ quantization_parameters:
1908
+ scale: 3.0518043793392844e-05
1909
+ offset: -32768
1910
+ past_key_14_in:
1911
+ shape:
1912
+ - 8
1913
+ - 1
1914
+ - 128
1915
+ - 4095
1916
+ dtype: uint8
1917
+ quantization_parameters:
1918
+ scale: 0.13996938269908035
1919
+ offset: -128
1920
+ past_value_14_in:
1921
+ shape:
1922
+ - 8
1923
+ - 1
1924
+ - 4095
1925
+ - 128
1926
+ dtype: uint8
1927
+ quantization_parameters:
1928
+ scale: 0.028563514469176765
1929
+ offset: -128
1930
+ past_key_15_in:
1931
+ shape:
1932
+ - 8
1933
+ - 1
1934
+ - 128
1935
+ - 4095
1936
+ dtype: uint8
1937
+ quantization_parameters:
1938
+ scale: 0.125163213474544
1939
+ offset: -128
1940
+ past_value_15_in:
1941
+ shape:
1942
+ - 8
1943
+ - 1
1944
+ - 4095
1945
+ - 128
1946
+ dtype: uint8
1947
+ quantization_parameters:
1948
+ scale: 0.027315943259892502
1949
+ offset: -128
1950
+ past_key_16_in:
1951
+ shape:
1952
+ - 8
1953
+ - 1
1954
+ - 128
1955
+ - 4095
1956
+ dtype: uint8
1957
+ quantization_parameters:
1958
+ scale: 0.12602750522883857
1959
+ offset: -128
1960
+ past_value_16_in:
1961
+ shape:
1962
+ - 8
1963
+ - 1
1964
+ - 4095
1965
+ - 128
1966
+ dtype: uint8
1967
+ quantization_parameters:
1968
+ scale: 0.033199068158864975
1969
+ offset: -128
1970
+ past_key_17_in:
1971
+ shape:
1972
+ - 8
1973
+ - 1
1974
+ - 128
1975
+ - 4095
1976
+ dtype: uint8
1977
+ quantization_parameters:
1978
+ scale: 0.13656628131866455
1979
+ offset: -128
1980
+ past_value_17_in:
1981
+ shape:
1982
+ - 8
1983
+ - 1
1984
+ - 4095
1985
+ - 128
1986
+ dtype: uint8
1987
+ quantization_parameters:
1988
+ scale: 0.030598552208247146
1989
+ offset: -128
1990
+ past_key_18_in:
1991
+ shape:
1992
+ - 8
1993
+ - 1
1994
+ - 128
1995
+ - 4095
1996
+ dtype: uint8
1997
+ quantization_parameters:
1998
+ scale: 0.11889628913458877
1999
+ offset: -128
2000
+ past_value_18_in:
2001
+ shape:
2002
+ - 8
2003
+ - 1
2004
+ - 4095
2005
+ - 128
2006
+ dtype: uint8
2007
+ quantization_parameters:
2008
+ scale: 0.032141249626874924
2009
+ offset: -128
2010
+ past_key_19_in:
2011
+ shape:
2012
+ - 8
2013
+ - 1
2014
+ - 128
2015
+ - 4095
2016
+ dtype: uint8
2017
+ quantization_parameters:
2018
+ scale: 0.13976185023784637
2019
+ offset: -128
2020
+ past_value_19_in:
2021
+ shape:
2022
+ - 8
2023
+ - 1
2024
+ - 4095
2025
+ - 128
2026
+ dtype: uint8
2027
+ quantization_parameters:
2028
+ scale: 0.032557712765190545
2029
+ offset: -128
2030
+ past_key_20_in:
2031
+ shape:
2032
+ - 8
2033
+ - 1
2034
+ - 128
2035
+ - 4095
2036
+ dtype: uint8
2037
+ quantization_parameters:
2038
+ scale: 0.11934278142733837
2039
+ offset: -128
2040
+ past_value_20_in:
2041
+ shape:
2042
+ - 8
2043
+ - 1
2044
+ - 4095
2045
+ - 128
2046
+ dtype: uint8
2047
+ quantization_parameters:
2048
+ scale: 0.027935825288295746
2049
+ offset: -128
2050
+ past_key_21_in:
2051
+ shape:
2052
+ - 8
2053
+ - 1
2054
+ - 128
2055
+ - 4095
2056
+ dtype: uint8
2057
+ quantization_parameters:
2058
+ scale: 0.13167049377922
2059
+ offset: -128
2060
+ past_value_21_in:
2061
+ shape:
2062
+ - 8
2063
+ - 1
2064
+ - 4095
2065
+ - 128
2066
+ dtype: uint8
2067
+ quantization_parameters:
2068
+ scale: 0.04469357803463936
2069
+ offset: -128
2070
+ past_key_22_in:
2071
+ shape:
2072
+ - 8
2073
+ - 1
2074
+ - 128
2075
+ - 4095
2076
+ dtype: uint8
2077
+ quantization_parameters:
2078
+ scale: 0.13328912689929873
2079
+ offset: -128
2080
+ past_value_22_in:
2081
+ shape:
2082
+ - 8
2083
+ - 1
2084
+ - 4095
2085
+ - 128
2086
+ dtype: uint8
2087
+ quantization_parameters:
2088
+ scale: 0.03745593875646591
2089
+ offset: -128
2090
+ past_key_23_in:
2091
+ shape:
2092
+ - 8
2093
+ - 1
2094
+ - 128
2095
+ - 4095
2096
+ dtype: uint8
2097
+ quantization_parameters:
2098
+ scale: 0.12827403151144193
2099
+ offset: -128
2100
+ past_value_23_in:
2101
+ shape:
2102
+ - 8
2103
+ - 1
2104
+ - 4095
2105
+ - 128
2106
+ dtype: uint8
2107
+ quantization_parameters:
2108
+ scale: 0.032266516238451004
2109
+ offset: -128
2110
+ past_key_24_in:
2111
+ shape:
2112
+ - 8
2113
+ - 1
2114
+ - 128
2115
+ - 4095
2116
+ dtype: uint8
2117
+ quantization_parameters:
2118
+ scale: 0.14478352665901184
2119
+ offset: -128
2120
+ past_value_24_in:
2121
+ shape:
2122
+ - 8
2123
+ - 1
2124
+ - 4095
2125
+ - 128
2126
+ dtype: uint8
2127
+ quantization_parameters:
2128
+ scale: 0.04626827540360098
2129
+ offset: -128
2130
+ past_key_25_in:
2131
+ shape:
2132
+ - 8
2133
+ - 1
2134
+ - 128
2135
+ - 4095
2136
+ dtype: uint8
2137
+ quantization_parameters:
2138
+ scale: 0.17984355716254768
2139
+ offset: -128
2140
+ past_value_25_in:
2141
+ shape:
2142
+ - 8
2143
+ - 1
2144
+ - 4095
2145
+ - 128
2146
+ dtype: uint8
2147
+ quantization_parameters:
2148
+ scale: 0.04539092882411686
2149
+ offset: -128
2150
+ past_key_26_in:
2151
+ shape:
2152
+ - 8
2153
+ - 1
2154
+ - 128
2155
+ - 4095
2156
+ dtype: uint8
2157
+ quantization_parameters:
2158
+ scale: 0.1248558149563046
2159
+ offset: -128
2160
+ past_value_26_in:
2161
+ shape:
2162
+ - 8
2163
+ - 1
2164
+ - 4095
2165
+ - 128
2166
+ dtype: uint8
2167
+ quantization_parameters:
2168
+ scale: 0.05314682051539421
2169
+ offset: -128
2170
+ past_key_27_in:
2171
+ shape:
2172
+ - 8
2173
+ - 1
2174
+ - 128
2175
+ - 4095
2176
+ dtype: uint8
2177
+ quantization_parameters:
2178
+ scale: 0.13326138257980347
2179
+ offset: -128
2180
+ past_value_27_in:
2181
+ shape:
2182
+ - 8
2183
+ - 1
2184
+ - 4095
2185
+ - 128
2186
+ dtype: uint8
2187
+ quantization_parameters:
2188
+ scale: 0.04699636995792389
2189
+ offset: -128
2190
+ outputs:
2191
+ logits:
2192
+ shape:
2193
+ - 1
2194
+ - 1
2195
+ - 128256
2196
+ dtype: uint16
2197
+ quantization_parameters:
2198
+ scale: 0.0006455201492198974
2199
+ offset: -24293
2200
+ past_key_14_out:
2201
+ shape:
2202
+ - 8
2203
+ - 1
2204
+ - 128
2205
+ - 1
2206
+ dtype: uint8
2207
+ quantization_parameters:
2208
+ scale: 0.13996938269908035
2209
+ offset: -128
2210
+ past_value_14_out:
2211
+ shape:
2212
+ - 8
2213
+ - 1
2214
+ - 1
2215
+ - 128
2216
+ dtype: uint8
2217
+ quantization_parameters:
2218
+ scale: 0.028563514469176765
2219
+ offset: -128
2220
+ past_key_15_out:
2221
+ shape:
2222
+ - 8
2223
+ - 1
2224
+ - 128
2225
+ - 1
2226
+ dtype: uint8
2227
+ quantization_parameters:
2228
+ scale: 0.125163213474544
2229
+ offset: -128
2230
+ past_value_15_out:
2231
+ shape:
2232
+ - 8
2233
+ - 1
2234
+ - 1
2235
+ - 128
2236
+ dtype: uint8
2237
+ quantization_parameters:
2238
+ scale: 0.027315943259892502
2239
+ offset: -128
2240
+ past_key_16_out:
2241
+ shape:
2242
+ - 8
2243
+ - 1
2244
+ - 128
2245
+ - 1
2246
+ dtype: uint8
2247
+ quantization_parameters:
2248
+ scale: 0.12602750522883857
2249
+ offset: -128
2250
+ past_value_16_out:
2251
+ shape:
2252
+ - 8
2253
+ - 1
2254
+ - 1
2255
+ - 128
2256
+ dtype: uint8
2257
+ quantization_parameters:
2258
+ scale: 0.033199068158864975
2259
+ offset: -128
2260
+ past_key_17_out:
2261
+ shape:
2262
+ - 8
2263
+ - 1
2264
+ - 128
2265
+ - 1
2266
+ dtype: uint8
2267
+ quantization_parameters:
2268
+ scale: 0.13656628131866455
2269
+ offset: -128
2270
+ past_value_17_out:
2271
+ shape:
2272
+ - 8
2273
+ - 1
2274
+ - 1
2275
+ - 128
2276
+ dtype: uint8
2277
+ quantization_parameters:
2278
+ scale: 0.030598552208247146
2279
+ offset: -128
2280
+ past_key_18_out:
2281
+ shape:
2282
+ - 8
2283
+ - 1
2284
+ - 128
2285
+ - 1
2286
+ dtype: uint8
2287
+ quantization_parameters:
2288
+ scale: 0.11889628913458877
2289
+ offset: -128
2290
+ past_value_18_out:
2291
+ shape:
2292
+ - 8
2293
+ - 1
2294
+ - 1
2295
+ - 128
2296
+ dtype: uint8
2297
+ quantization_parameters:
2298
+ scale: 0.032141249626874924
2299
+ offset: -128
2300
+ past_key_19_out:
2301
+ shape:
2302
+ - 8
2303
+ - 1
2304
+ - 128
2305
+ - 1
2306
+ dtype: uint8
2307
+ quantization_parameters:
2308
+ scale: 0.13976185023784637
2309
+ offset: -128
2310
+ past_value_19_out:
2311
+ shape:
2312
+ - 8
2313
+ - 1
2314
+ - 1
2315
+ - 128
2316
+ dtype: uint8
2317
+ quantization_parameters:
2318
+ scale: 0.032557712765190545
2319
+ offset: -128
2320
+ past_key_20_out:
2321
+ shape:
2322
+ - 8
2323
+ - 1
2324
+ - 128
2325
+ - 1
2326
+ dtype: uint8
2327
+ quantization_parameters:
2328
+ scale: 0.11934278142733837
2329
+ offset: -128
2330
+ past_value_20_out:
2331
+ shape:
2332
+ - 8
2333
+ - 1
2334
+ - 1
2335
+ - 128
2336
+ dtype: uint8
2337
+ quantization_parameters:
2338
+ scale: 0.027935825288295746
2339
+ offset: -128
2340
+ past_key_21_out:
2341
+ shape:
2342
+ - 8
2343
+ - 1
2344
+ - 128
2345
+ - 1
2346
+ dtype: uint8
2347
+ quantization_parameters:
2348
+ scale: 0.13167049377922
2349
+ offset: -128
2350
+ past_value_21_out:
2351
+ shape:
2352
+ - 8
2353
+ - 1
2354
+ - 1
2355
+ - 128
2356
+ dtype: uint8
2357
+ quantization_parameters:
2358
+ scale: 0.04469357803463936
2359
+ offset: -128
2360
+ past_key_22_out:
2361
+ shape:
2362
+ - 8
2363
+ - 1
2364
+ - 128
2365
+ - 1
2366
+ dtype: uint8
2367
+ quantization_parameters:
2368
+ scale: 0.13328912689929873
2369
+ offset: -128
2370
+ past_value_22_out:
2371
+ shape:
2372
+ - 8
2373
+ - 1
2374
+ - 1
2375
+ - 128
2376
+ dtype: uint8
2377
+ quantization_parameters:
2378
+ scale: 0.03745593875646591
2379
+ offset: -128
2380
+ past_key_23_out:
2381
+ shape:
2382
+ - 8
2383
+ - 1
2384
+ - 128
2385
+ - 1
2386
+ dtype: uint8
2387
+ quantization_parameters:
2388
+ scale: 0.12827403151144193
2389
+ offset: -128
2390
+ past_value_23_out:
2391
+ shape:
2392
+ - 8
2393
+ - 1
2394
+ - 1
2395
+ - 128
2396
+ dtype: uint8
2397
+ quantization_parameters:
2398
+ scale: 0.032266516238451004
2399
+ offset: -128
2400
+ past_key_24_out:
2401
+ shape:
2402
+ - 8
2403
+ - 1
2404
+ - 128
2405
+ - 1
2406
+ dtype: uint8
2407
+ quantization_parameters:
2408
+ scale: 0.14478352665901184
2409
+ offset: -128
2410
+ past_value_24_out:
2411
+ shape:
2412
+ - 8
2413
+ - 1
2414
+ - 1
2415
+ - 128
2416
+ dtype: uint8
2417
+ quantization_parameters:
2418
+ scale: 0.04626827540360098
2419
+ offset: -128
2420
+ past_key_25_out:
2421
+ shape:
2422
+ - 8
2423
+ - 1
2424
+ - 128
2425
+ - 1
2426
+ dtype: uint8
2427
+ quantization_parameters:
2428
+ scale: 0.17984355716254768
2429
+ offset: -128
2430
+ past_value_25_out:
2431
+ shape:
2432
+ - 8
2433
+ - 1
2434
+ - 1
2435
+ - 128
2436
+ dtype: uint8
2437
+ quantization_parameters:
2438
+ scale: 0.04539092882411686
2439
+ offset: -128
2440
+ past_key_26_out:
2441
+ shape:
2442
+ - 8
2443
+ - 1
2444
+ - 128
2445
+ - 1
2446
+ dtype: uint8
2447
+ quantization_parameters:
2448
+ scale: 0.1248558149563046
2449
+ offset: -128
2450
+ past_value_26_out:
2451
+ shape:
2452
+ - 8
2453
+ - 1
2454
+ - 1
2455
+ - 128
2456
+ dtype: uint8
2457
+ quantization_parameters:
2458
+ scale: 0.05314682051539421
2459
+ offset: -128
2460
+ past_key_27_out:
2461
+ shape:
2462
+ - 8
2463
+ - 1
2464
+ - 128
2465
+ - 1
2466
+ dtype: uint8
2467
+ quantization_parameters:
2468
+ scale: 0.13326138257980347
2469
+ offset: -128
2470
+ past_value_27_out:
2471
+ shape:
2472
+ - 8
2473
+ - 1
2474
+ - 1
2475
+ - 128
2476
+ dtype: uint8
2477
+ quantization_parameters:
2478
+ scale: 0.04699636995792389
2479
+ offset: -128
2480
+ precision: w4a16
2481
+ runtime: genie
sample_prompt.txt ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ <|begin_of_text|><|start_header_id|>system<|end_header_id|>
2
+
3
+ Cutting Knowledge Date: December 2023
4
+ Today Date: 09 Apr 2026
5
+
6
+ You are a helpful AI assistant<|eot_id|><|start_header_id|>user<|end_header_id|>
7
+
8
+ What do llamas eat? Keep the answer under ten words.<|eot_id|><|start_header_id|>assistant<|end_header_id|>
9
+
server_persistent.py ADDED
@@ -0,0 +1,664 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """
3
+ Persistent OpenAI-compatible API server for Qualcomm Genie on NPU.
4
+
5
+ Uses Python ctypes to call libGenie.so C API directly, keeping the model
6
+ loaded in memory between requests. This eliminates the ~3s model reload
7
+ overhead that occurs when spawning genie-t2t-run per request.
8
+
9
+ Architecture:
10
+ - At startup: GenieDialogConfig_createFromJson + GenieDialog_create (loads model once)
11
+ - Per request: GenieDialog_reset + GenieDialog_query (streaming via callback)
12
+ - At shutdown: GenieDialog_free + GenieDialogConfig_free
13
+
14
+ No external dependencies — uses only Python stdlib + ctypes.
15
+ """
16
+
17
+ import ctypes
18
+ import json
19
+ import os
20
+ import queue
21
+ import signal
22
+ import sys
23
+ import time
24
+ import traceback
25
+ import uuid
26
+ from http.server import HTTPServer, BaseHTTPRequestHandler
27
+ from socketserver import ThreadingMixIn
28
+ from threading import Lock, Thread, Event
29
+
30
+
31
+ class ThreadingHTTPServer(ThreadingMixIn, HTTPServer):
32
+ """HTTPServer that handles each request in a new thread.
33
+
34
+ This prevents Open WebUI's concurrent requests (e.g. /v1/models polling
35
+ while a streaming response is in progress) from blocking the server.
36
+ The Genie inference itself is still serialized via _inference_lock.
37
+ """
38
+ daemon_threads = True
39
+
40
+ # ---------------------------------------------------------------------------
41
+ # Configuration
42
+ # ---------------------------------------------------------------------------
43
+ MODEL_DIR = os.environ.get("MODEL_DIR", "/tmp/genie_bundle")
44
+ MODEL_NAME = os.environ.get("MODEL_NAME", "llama-3.2-3b-instruct-npu")
45
+ CONFIG_FILE = os.environ.get("GENIE_CONFIG", os.path.join(MODEL_DIR, "genie_config.json"))
46
+ SERVER_PORT = int(os.environ.get("PORT", "8000"))
47
+ LIB_GENIE_PATH = os.environ.get("LIB_GENIE_PATH", os.path.join(MODEL_DIR, "libGenie.so"))
48
+
49
+ # Library search paths
50
+ GENIE_LIB_PATH = os.environ.get("LD_LIBRARY_PATH", f"{MODEL_DIR}:/usr/lib")
51
+ ADSP_LIB_PATH = os.environ.get("ADSP_LIBRARY_PATH", "/usr/lib/dsp/cdsp;/usr/lib/dsp/cdsp1")
52
+
53
+ # ---------------------------------------------------------------------------
54
+ # Genie C API Constants (from GenieCommon.h / GenieDialog.h)
55
+ # ---------------------------------------------------------------------------
56
+ GENIE_STATUS_SUCCESS = 0
57
+ GENIE_STATUS_WARNING_ABORTED = 1
58
+ GENIE_STATUS_WARNING_CONTEXT_EXCEEDED = 4
59
+
60
+ # Sentence codes
61
+ GENIE_DIALOG_SENTENCE_COMPLETE = 0
62
+ GENIE_DIALOG_SENTENCE_BEGIN = 1
63
+ GENIE_DIALOG_SENTENCE_CONTINUE = 2
64
+ GENIE_DIALOG_SENTENCE_END = 3
65
+ GENIE_DIALOG_SENTENCE_ABORT = 4
66
+
67
+ # Actions
68
+ GENIE_DIALOG_ACTION_ABORT = 0x01
69
+
70
+ # ---------------------------------------------------------------------------
71
+ # Genie C API Callback Type
72
+ # ---------------------------------------------------------------------------
73
+ # typedef void (*GenieDialog_QueryCallback_t)(
74
+ # const char* response,
75
+ # const GenieDialog_SentenceCode_t sentenceCode,
76
+ # const void* userData);
77
+ QUERY_CALLBACK_TYPE = ctypes.CFUNCTYPE(
78
+ None, # return void
79
+ ctypes.c_char_p, # const char* response
80
+ ctypes.c_int, # GenieDialog_SentenceCode_t sentenceCode
81
+ ctypes.c_void_p, # const void* userData
82
+ )
83
+
84
+ # ---------------------------------------------------------------------------
85
+ # Genie Library Wrapper
86
+ # ---------------------------------------------------------------------------
87
+ class GenieLibrary:
88
+ """Wrapper around libGenie.so with ctypes bindings."""
89
+
90
+ def __init__(self, lib_path):
91
+ # Set environment before loading
92
+ os.environ["LD_LIBRARY_PATH"] = GENIE_LIB_PATH
93
+ os.environ["ADSP_LIBRARY_PATH"] = ADSP_LIB_PATH
94
+
95
+ print(f"[GENIE] Loading libGenie.so from {lib_path}")
96
+ self.lib = ctypes.CDLL(lib_path)
97
+
98
+ # Set up function signatures
99
+ self._setup_signatures()
100
+
101
+ # Get API version
102
+ major = self.lib.Genie_getApiMajorVersion()
103
+ minor = self.lib.Genie_getApiMinorVersion()
104
+ patch = self.lib.Genie_getApiPatchVersion()
105
+ print(f"[GENIE] API version: {major}.{minor}.{patch}")
106
+
107
+ def _setup_signatures(self):
108
+ """Define argument and return types for all functions."""
109
+ lib = self.lib
110
+
111
+ # GenieDialogConfig_createFromJson(const char* str, GenieDialogConfig_Handle_t* configHandle)
112
+ lib.GenieDialogConfig_createFromJson.argtypes = [ctypes.c_char_p, ctypes.POINTER(ctypes.c_void_p)]
113
+ lib.GenieDialogConfig_createFromJson.restype = ctypes.c_int32
114
+
115
+ # GenieDialogConfig_free(GenieDialogConfig_Handle_t configHandle)
116
+ lib.GenieDialogConfig_free.argtypes = [ctypes.c_void_p]
117
+ lib.GenieDialogConfig_free.restype = ctypes.c_int32
118
+
119
+ # GenieDialog_create(GenieDialogConfig_Handle_t configHandle, GenieDialog_Handle_t* dialogHandle)
120
+ lib.GenieDialog_create.argtypes = [ctypes.c_void_p, ctypes.POINTER(ctypes.c_void_p)]
121
+ lib.GenieDialog_create.restype = ctypes.c_int32
122
+
123
+ # GenieDialog_query(GenieDialog_Handle_t dialogHandle, const char* queryStr,
124
+ # GenieDialog_SentenceCode_t sentenceCode,
125
+ # GenieDialog_QueryCallback_t callback, const void* userData)
126
+ lib.GenieDialog_query.argtypes = [
127
+ ctypes.c_void_p, # dialogHandle
128
+ ctypes.c_char_p, # queryStr
129
+ ctypes.c_int, # sentenceCode
130
+ QUERY_CALLBACK_TYPE, # callback
131
+ ctypes.c_void_p, # userData
132
+ ]
133
+ lib.GenieDialog_query.restype = ctypes.c_int32
134
+
135
+ # GenieDialog_reset(GenieDialog_Handle_t dialogHandle)
136
+ lib.GenieDialog_reset.argtypes = [ctypes.c_void_p]
137
+ lib.GenieDialog_reset.restype = ctypes.c_int32
138
+
139
+ # GenieDialog_free(GenieDialog_Handle_t dialogHandle)
140
+ lib.GenieDialog_free.argtypes = [ctypes.c_void_p]
141
+ lib.GenieDialog_free.restype = ctypes.c_int32
142
+
143
+ # GenieDialog_signal(GenieDialog_Handle_t dialogHandle, GenieDialog_Action_t action)
144
+ lib.GenieDialog_signal.argtypes = [ctypes.c_void_p, ctypes.c_int]
145
+ lib.GenieDialog_signal.restype = ctypes.c_int32
146
+
147
+ # GenieDialog_save(GenieDialog_Handle_t dialogHandle, const char* path)
148
+ lib.GenieDialog_save.argtypes = [ctypes.c_void_p, ctypes.c_char_p]
149
+ lib.GenieDialog_save.restype = ctypes.c_int32
150
+
151
+ # GenieDialog_restore(GenieDialog_Handle_t dialogHandle, const char* path)
152
+ lib.GenieDialog_restore.argtypes = [ctypes.c_void_p, ctypes.c_char_p]
153
+ lib.GenieDialog_restore.restype = ctypes.c_int32
154
+
155
+ # GenieDialog_setStopSequence(GenieDialog_Handle_t dialogHandle, const char* newStopSequences)
156
+ lib.GenieDialog_setStopSequence.argtypes = [ctypes.c_void_p, ctypes.c_char_p]
157
+ lib.GenieDialog_setStopSequence.restype = ctypes.c_int32
158
+
159
+ # GenieSamplerConfig_createFromJson(const char* str, GenieSamplerConfig_Handle_t* configHandle)
160
+ lib.GenieSamplerConfig_createFromJson.argtypes = [ctypes.c_char_p, ctypes.POINTER(ctypes.c_void_p)]
161
+ lib.GenieSamplerConfig_createFromJson.restype = ctypes.c_int32
162
+
163
+ # GenieSampler_applyConfig(GenieSampler_Handle_t samplerHandle, GenieSamplerConfig_Handle_t configHandle)
164
+ lib.GenieSampler_applyConfig.argtypes = [ctypes.c_void_p, ctypes.c_void_p]
165
+ lib.GenieSampler_applyConfig.restype = ctypes.c_int32
166
+
167
+ # GenieSamplerConfig_free(GenieSamplerConfig_Handle_t configHandle)
168
+ lib.GenieSamplerConfig_free.argtypes = [ctypes.c_void_p]
169
+ lib.GenieSamplerConfig_free.restype = ctypes.c_int32
170
+
171
+ # GenieDialog_getSampler(GenieDialog_Handle_t dialogHandle, GenieSampler_Handle_t* samplerHandle)
172
+ lib.GenieDialog_getSampler.argtypes = [ctypes.c_void_p, ctypes.POINTER(ctypes.c_void_p)]
173
+ lib.GenieDialog_getSampler.restype = ctypes.c_int32
174
+
175
+
176
+ def _check_status(status, func_name):
177
+ """Check Genie API return status and raise on error."""
178
+ if status < 0:
179
+ raise RuntimeError(f"[GENIE] {func_name} failed with status {status}")
180
+ if status > 0:
181
+ print(f"[GENIE] {func_name} returned warning status {status}", file=sys.stderr)
182
+ return status
183
+
184
+
185
+ # ---------------------------------------------------------------------------
186
+ # Persistent Genie Dialog Manager
187
+ # ---------------------------------------------------------------------------
188
+ class GenieDialogManager:
189
+ """Manages a persistent Genie dialog instance.
190
+
191
+ Loads the model once at startup and handles queries by:
192
+ 1. Resetting the dialog (clears KV cache, keeps model loaded)
193
+ 2. Executing query with streaming callback
194
+ """
195
+
196
+ def __init__(self, lib_path, config_path, model_dir):
197
+ self._lib = GenieLibrary(lib_path)
198
+ self._config_handle = ctypes.c_void_p()
199
+ self._dialog_handle = ctypes.c_void_p()
200
+ self._inference_lock = Lock()
201
+ self._model_loaded = False
202
+ self._model_dir = model_dir
203
+
204
+ # Load model
205
+ self._load_model(config_path)
206
+
207
+ def _load_model(self, config_path):
208
+ """Load model onto NPU (one-time operation)."""
209
+ print(f"[GENIE] Loading config from {config_path}")
210
+ t0 = time.time()
211
+
212
+ # Read config JSON
213
+ with open(config_path, "r") as f:
214
+ config_str = f.read()
215
+
216
+ # Create dialog config from JSON
217
+ status = self._lib.lib.GenieDialogConfig_createFromJson(
218
+ config_str.encode("utf-8"),
219
+ ctypes.byref(self._config_handle),
220
+ )
221
+ _check_status(status, "GenieDialogConfig_createFromJson")
222
+ print(f"[GENIE] Config created in {time.time() - t0:.2f}s")
223
+
224
+ # Create dialog (this loads model weights onto NPU — the expensive step)
225
+ print(f"[GENIE] Creating dialog (loading model onto NPU)...")
226
+ t1 = time.time()
227
+ status = self._lib.lib.GenieDialog_create(
228
+ self._config_handle,
229
+ ctypes.byref(self._dialog_handle),
230
+ )
231
+ _check_status(status, "GenieDialog_create")
232
+ load_time = time.time() - t1
233
+ print(f"[GENIE] Model loaded onto NPU in {load_time:.2f}s")
234
+ print(f"[GENIE] Total startup time: {time.time() - t0:.2f}s")
235
+
236
+ self._model_loaded = True
237
+
238
+ def query_blocking(self, prompt_text, max_tokens=2048):
239
+ """Execute a blocking query and return the complete response text.
240
+
241
+ Returns (response_text, token_count_estimate).
242
+ """
243
+ if not self._model_loaded:
244
+ raise RuntimeError("Model not loaded")
245
+
246
+ collected_tokens = []
247
+
248
+ # Define the callback that collects tokens
249
+ @QUERY_CALLBACK_TYPE
250
+ def callback(response, sentence_code, user_data):
251
+ if response:
252
+ text = response.decode("utf-8", errors="replace")
253
+ collected_tokens.append(text)
254
+
255
+ with self._inference_lock:
256
+ # Reset dialog KV cache for a fresh independent query
257
+ status = self._lib.lib.GenieDialog_reset(self._dialog_handle)
258
+ _check_status(status, "GenieDialog_reset")
259
+
260
+ # Execute query
261
+ prompt_bytes = prompt_text.encode("utf-8")
262
+ status = self._lib.lib.GenieDialog_query(
263
+ self._dialog_handle,
264
+ prompt_bytes,
265
+ GENIE_DIALOG_SENTENCE_COMPLETE,
266
+ callback,
267
+ None,
268
+ )
269
+ if status < 0:
270
+ print(f"[GENIE] GenieDialog_query returned error {status}", file=sys.stderr)
271
+
272
+ return "".join(collected_tokens), len(collected_tokens)
273
+
274
+ def query_streaming(self, prompt_text, max_tokens=2048):
275
+ """Execute a query and yield text chunks as they arrive via callback.
276
+
277
+ Returns a generator that yields (text, sentence_code) tuples.
278
+ """
279
+ if not self._model_loaded:
280
+ raise RuntimeError("Model not loaded")
281
+
282
+ token_queue = queue.Queue()
283
+ done_event = Event()
284
+
285
+ # Define the callback that pushes tokens to the queue
286
+ @QUERY_CALLBACK_TYPE
287
+ def callback(response, sentence_code, user_data):
288
+ if response:
289
+ text = response.decode("utf-8", errors="replace")
290
+ token_queue.put((text, sentence_code))
291
+ if sentence_code in (GENIE_DIALOG_SENTENCE_END,
292
+ GENIE_DIALOG_SENTENCE_COMPLETE,
293
+ GENIE_DIALOG_SENTENCE_ABORT):
294
+ done_event.set()
295
+
296
+ def run_query():
297
+ try:
298
+ with self._inference_lock:
299
+ # Reset dialog KV cache
300
+ status = self._lib.lib.GenieDialog_reset(self._dialog_handle)
301
+ _check_status(status, "GenieDialog_reset")
302
+
303
+ # Execute query — this blocks until generation completes
304
+ prompt_bytes = prompt_text.encode("utf-8")
305
+ status = self._lib.lib.GenieDialog_query(
306
+ self._dialog_handle,
307
+ prompt_bytes,
308
+ GENIE_DIALOG_SENTENCE_COMPLETE,
309
+ callback,
310
+ None,
311
+ )
312
+ if status < 0:
313
+ print(f"[GENIE] GenieDialog_query error {status}", file=sys.stderr)
314
+ except Exception as e:
315
+ print(f"[GENIE] Query thread error: {e}", file=sys.stderr)
316
+ finally:
317
+ done_event.set()
318
+ token_queue.put((None, GENIE_DIALOG_SENTENCE_END)) # Sentinel
319
+
320
+ # Run query in a separate thread so we can yield tokens as they arrive
321
+ query_thread = Thread(target=run_query, daemon=True)
322
+ query_thread.start()
323
+
324
+ # Yield tokens from the queue
325
+ while True:
326
+ try:
327
+ text, sentence_code = token_queue.get(timeout=120)
328
+ if text is None:
329
+ break
330
+ yield text, sentence_code
331
+ except queue.Empty:
332
+ print("[GENIE] Query timed out waiting for tokens", file=sys.stderr)
333
+ break
334
+
335
+ query_thread.join(timeout=5)
336
+
337
+ def shutdown(self):
338
+ """Free dialog and config handles."""
339
+ if self._model_loaded:
340
+ print("[GENIE] Freeing dialog...")
341
+ self._lib.lib.GenieDialog_free(self._dialog_handle)
342
+ print("[GENIE] Freeing config...")
343
+ self._lib.lib.GenieDialogConfig_free(self._config_handle)
344
+ self._model_loaded = False
345
+ print("[GENIE] Shutdown complete")
346
+
347
+
348
+ # ---------------------------------------------------------------------------
349
+ # Content extraction (handles OpenAI vision format from Open WebUI)
350
+ # ---------------------------------------------------------------------------
351
+ def _extract_text_from_content(content):
352
+ """Extract text from OpenAI message content, stripping image data.
353
+
354
+ Open WebUI sends images in OpenAI vision format:
355
+ [{"type": "text", "text": "..."}, {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}]
356
+
357
+ This model is text-only (Llama 3.1 8B Instruct) and cannot process images.
358
+ We extract only the text parts and add a note about stripped images.
359
+ """
360
+ if isinstance(content, str):
361
+ return content
362
+ if isinstance(content, list):
363
+ text_parts = []
364
+ has_images = False
365
+ for part in content:
366
+ if isinstance(part, dict):
367
+ if part.get("type") == "text":
368
+ text_parts.append(part.get("text", ""))
369
+ elif part.get("type") == "image_url":
370
+ has_images = True
371
+ result = " ".join(text_parts).strip()
372
+ if has_images:
373
+ if result:
374
+ result += "\n\n[Note: An image was attached but this model is text-only and cannot process images.]"
375
+ else:
376
+ result = "[An image was sent but this model is text-only and cannot process images. Please send a text message instead.]"
377
+ print(f"[GUARD] Stripped image data from message, text-only content: {result[:100]}...", flush=True)
378
+ return result
379
+ return str(content)
380
+
381
+
382
+ # ---------------------------------------------------------------------------
383
+ # Llama 3.2 Instruct prompt formatting
384
+ # ---------------------------------------------------------------------------
385
+ def format_llama32_prompt(messages):
386
+ parts = ["<|begin_of_text|>"]
387
+ for msg in messages:
388
+ role = msg.get("role", "user")
389
+ content = _extract_text_from_content(msg.get("content", ""))
390
+ parts.append(f"<|start_header_id|>{role}<|end_header_id|>\n\n{content}<|eot_id|>")
391
+ parts.append("<|start_header_id|>assistant<|end_header_id|>\n\n")
392
+ return "".join(parts)
393
+
394
+
395
+ # ---------------------------------------------------------------------------
396
+ # HTTP Handler
397
+ # ---------------------------------------------------------------------------
398
+ class PersistentGenieHandler(BaseHTTPRequestHandler):
399
+ """OpenAI-compatible HTTP handler using persistent Genie dialog."""
400
+
401
+ # Class-level reference to the dialog manager (set in main)
402
+ dialog_manager = None
403
+ _startup_time = None
404
+ _request_count = 0
405
+
406
+ def log_message(self, format, *args):
407
+ print(f"[HTTP] {self.client_address[0]} - {format % args}", file=sys.stderr)
408
+
409
+ def _send_json(self, data, status=200):
410
+ body = json.dumps(data).encode("utf-8")
411
+ self.send_response(status)
412
+ self.send_header("Content-Type", "application/json")
413
+ self.send_header("Content-Length", str(len(body)))
414
+ self.send_header("Access-Control-Allow-Origin", "*")
415
+ self.end_headers()
416
+ self.wfile.write(body)
417
+
418
+ def do_OPTIONS(self):
419
+ self.send_response(200)
420
+ self.send_header("Access-Control-Allow-Origin", "*")
421
+ self.send_header("Access-Control-Allow-Methods", "GET, POST, OPTIONS")
422
+ self.send_header("Access-Control-Allow-Headers", "Content-Type, Authorization")
423
+ self.send_header("Content-Length", "0")
424
+ self.end_headers()
425
+
426
+ def do_GET(self):
427
+ if self.path == "/health":
428
+ uptime = time.time() - (self._startup_time or time.time())
429
+ self._send_json({
430
+ "status": "ok",
431
+ "model_loaded": self.dialog_manager._model_loaded if self.dialog_manager else False,
432
+ "mode": "persistent",
433
+ "uptime_seconds": round(uptime, 1),
434
+ "requests_served": self._request_count,
435
+ })
436
+ elif self.path in ("/", ""):
437
+ self._send_json({
438
+ "message": "Genie LLM Server (Persistent Mode - OpenAI-compatible)",
439
+ "model": MODEL_NAME,
440
+ "mode": "persistent",
441
+ "description": "Model kept loaded on NPU between requests — no reload overhead",
442
+ })
443
+ elif self.path == "/v1/models":
444
+ self._send_json({
445
+ "object": "list",
446
+ "data": [{
447
+ "id": MODEL_NAME,
448
+ "object": "model",
449
+ "created": int(time.time()),
450
+ "owned_by": "qualcomm-genie",
451
+ "permission": [],
452
+ "root": MODEL_NAME,
453
+ "parent": None,
454
+ }]
455
+ })
456
+ elif self.path.startswith("/v1/models/"):
457
+ self._send_json({
458
+ "id": MODEL_NAME,
459
+ "object": "model",
460
+ "created": int(time.time()),
461
+ "owned_by": "qualcomm-genie",
462
+ })
463
+ else:
464
+ self._send_json({"error": "Not found"}, 404)
465
+
466
+ def do_POST(self):
467
+ if self.path != "/v1/chat/completions":
468
+ self._send_json({"error": "Not found"}, 404)
469
+ return
470
+
471
+ content_length = int(self.headers.get("Content-Length", 0))
472
+ body = self.rfile.read(content_length)
473
+
474
+ try:
475
+ request = json.loads(body)
476
+ except json.JSONDecodeError as e:
477
+ self._send_json({"error": f"Invalid JSON: {e}"}, 400)
478
+ return
479
+
480
+ messages = request.get("messages", [])
481
+ stream = request.get("stream", False)
482
+ max_tokens = request.get("max_tokens", 2048)
483
+
484
+ prompt = format_llama32_prompt(messages)
485
+ request_id = f"chatcmpl-{uuid.uuid4().hex[:12]}"
486
+ created = int(time.time())
487
+
488
+ PersistentGenieHandler._request_count += 1
489
+ t0 = time.time()
490
+
491
+ try:
492
+ if stream:
493
+ self._handle_streaming(prompt, request_id, created, max_tokens)
494
+ else:
495
+ self._handle_non_streaming(prompt, request_id, created, request, max_tokens)
496
+ except (BrokenPipeError, ConnectionResetError, ConnectionAbortedError):
497
+ print(f"[HTTP] Client disconnected during request {self._request_count}", file=sys.stderr)
498
+ except Exception as e:
499
+ print(f"[HTTP] Unexpected error in request {self._request_count}: {e}", file=sys.stderr)
500
+ traceback.print_exc()
501
+
502
+ elapsed = time.time() - t0
503
+ print(f"[PERF] Request {self._request_count}: {elapsed:.2f}s ({'stream' if stream else 'batch'})", file=sys.stderr)
504
+
505
+ def _handle_non_streaming(self, prompt, request_id, created, request, max_tokens):
506
+ try:
507
+ generated_text, token_count = self.dialog_manager.query_blocking(prompt, max_tokens)
508
+ except Exception as e:
509
+ self._send_json({"error": str(e)}, 500)
510
+ return
511
+
512
+ prompt_tokens = len(prompt) // 4
513
+ completion_tokens = max(token_count, len(generated_text) // 4)
514
+
515
+ response = {
516
+ "id": request_id,
517
+ "object": "chat.completion",
518
+ "created": created,
519
+ "model": request.get("model", MODEL_NAME),
520
+ "choices": [{
521
+ "index": 0,
522
+ "message": {"role": "assistant", "content": generated_text},
523
+ "finish_reason": "stop",
524
+ }],
525
+ "usage": {
526
+ "prompt_tokens": prompt_tokens,
527
+ "completion_tokens": completion_tokens,
528
+ "total_tokens": prompt_tokens + completion_tokens,
529
+ }
530
+ }
531
+ self._send_json(response)
532
+
533
+ def _handle_streaming(self, prompt, request_id, created, max_tokens):
534
+ self.send_response(200)
535
+ self.send_header("Content-Type", "text/event-stream")
536
+ self.send_header("Cache-Control", "no-cache")
537
+ self.send_header("Connection", "keep-alive")
538
+ self.send_header("Access-Control-Allow-Origin", "*")
539
+ self.send_header("X-Accel-Buffering", "no")
540
+ self.end_headers()
541
+
542
+ def send_chunk(data):
543
+ line = f"data: {json.dumps(data)}\n\n"
544
+ self.wfile.write(line.encode("utf-8"))
545
+ self.wfile.flush()
546
+
547
+ try:
548
+ # Initial role chunk
549
+ send_chunk({
550
+ "id": request_id,
551
+ "object": "chat.completion.chunk",
552
+ "created": created,
553
+ "model": MODEL_NAME,
554
+ "choices": [{"index": 0, "delta": {"role": "assistant", "content": ""}, "finish_reason": None}],
555
+ })
556
+
557
+ # Stream tokens from callback
558
+ for token_text, sentence_code in self.dialog_manager.query_streaming(prompt, max_tokens):
559
+ send_chunk({
560
+ "id": request_id,
561
+ "object": "chat.completion.chunk",
562
+ "created": created,
563
+ "model": MODEL_NAME,
564
+ "choices": [{"index": 0, "delta": {"content": token_text}, "finish_reason": None}],
565
+ })
566
+
567
+ # Final chunk
568
+ send_chunk({
569
+ "id": request_id,
570
+ "object": "chat.completion.chunk",
571
+ "created": created,
572
+ "model": MODEL_NAME,
573
+ "choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}],
574
+ })
575
+
576
+ self.wfile.write(b"data: [DONE]\n\n")
577
+ self.wfile.flush()
578
+
579
+ except (BrokenPipeError, ConnectionResetError, ConnectionAbortedError) as e:
580
+ # Client disconnected mid-stream (e.g. user opened a new chat in Open WebUI)
581
+ # This is normal and should NOT crash the server
582
+ print(f"[HTTP] Client disconnected during stream (normal): {type(e).__name__}", file=sys.stderr)
583
+ except Exception as e:
584
+ print(f"[HTTP] Streaming error: {e}", file=sys.stderr)
585
+
586
+
587
+ # ---------------------------------------------------------------------------
588
+ # Main
589
+ # ---------------------------------------------------------------------------
590
+ def main():
591
+ print("=" * 60)
592
+ print("Genie LLM Server — PERSISTENT MODE")
593
+ print("=" * 60)
594
+ print(f" Model: {MODEL_NAME}")
595
+ print(f" Config: {CONFIG_FILE}")
596
+ print(f" Model dir: {MODEL_DIR}")
597
+ print(f" libGenie.so: {LIB_GENIE_PATH}")
598
+ print(f" Library path: {GENIE_LIB_PATH}")
599
+ print(f" ADSP path: {ADSP_LIB_PATH}")
600
+ print(f" Port: {SERVER_PORT}")
601
+ print(f" Mode: PERSISTENT (model loaded once, kept resident)")
602
+ print()
603
+
604
+ # Verify FastRPC devices
605
+ fastrpc_devs = [d for d in os.listdir("/dev") if d.startswith("fastrpc")]
606
+ if fastrpc_devs:
607
+ print(f" FastRPC devices: {', '.join(sorted(fastrpc_devs))}")
608
+ else:
609
+ print(" WARNING: No /dev/fastrpc-* devices found!", file=sys.stderr)
610
+ print(" Running on host or ensure CDI/device-plugin injects devices", file=sys.stderr)
611
+
612
+ # Verify libGenie.so exists
613
+ if not os.path.isfile(LIB_GENIE_PATH):
614
+ print(f" ERROR: libGenie.so not found: {LIB_GENIE_PATH}", file=sys.stderr)
615
+ sys.exit(1)
616
+
617
+ # Verify config exists
618
+ if not os.path.isfile(CONFIG_FILE):
619
+ print(f" ERROR: Config not found: {CONFIG_FILE}", file=sys.stderr)
620
+ sys.exit(1)
621
+
622
+ print()
623
+ # Change to model directory so Genie resolves relative paths
624
+ # (tokenizer.json, .bin files referenced in genie_config.json)
625
+ print(f"[STARTUP] Changing working directory to {MODEL_DIR}")
626
+ os.chdir(MODEL_DIR)
627
+
628
+ print("[STARTUP] Loading model onto NPU (one-time cost)...")
629
+ print()
630
+
631
+ # Create the persistent dialog manager
632
+ try:
633
+ dialog_manager = GenieDialogManager(LIB_GENIE_PATH, CONFIG_FILE, MODEL_DIR)
634
+ except Exception as e:
635
+ print(f"[FATAL] Failed to initialize Genie: {e}", file=sys.stderr)
636
+ traceback.print_exc()
637
+ sys.exit(1)
638
+
639
+ # Set up HTTP server
640
+ PersistentGenieHandler.dialog_manager = dialog_manager
641
+ PersistentGenieHandler._startup_time = time.time()
642
+
643
+ ThreadingHTTPServer.allow_reuse_address = True
644
+ server = ThreadingHTTPServer(("0.0.0.0", SERVER_PORT), PersistentGenieHandler)
645
+
646
+ def shutdown_handler(sig, frame):
647
+ print("\n[SHUTDOWN] Received signal, shutting down...")
648
+ dialog_manager.shutdown()
649
+ server.shutdown()
650
+ sys.exit(0)
651
+
652
+ signal.signal(signal.SIGINT, shutdown_handler)
653
+ signal.signal(signal.SIGTERM, shutdown_handler)
654
+
655
+ print()
656
+ print(f"[READY] Listening on http://0.0.0.0:{SERVER_PORT}")
657
+ print(f"[READY] Model loaded and resident on NPU — no per-request reload!")
658
+ print()
659
+
660
+ server.serve_forever()
661
+
662
+
663
+ if __name__ == "__main__":
664
+ main()
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6b9e4e7fb171f92fd137b777cc2714bf87d11576700a1dcd7a399e7bbe39537b
3
+ size 17209920
tokenizer_config.json ADDED
@@ -0,0 +1,2064 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "added_tokens_decoder": {
3
+ "128000": {
4
+ "content": "<|begin_of_text|>",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false,
9
+ "special": true
10
+ },
11
+ "128001": {
12
+ "content": "<|end_of_text|>",
13
+ "lstrip": false,
14
+ "normalized": false,
15
+ "rstrip": false,
16
+ "single_word": false,
17
+ "special": true
18
+ },
19
+ "128002": {
20
+ "content": "<|reserved_special_token_0|>",
21
+ "lstrip": false,
22
+ "normalized": false,
23
+ "rstrip": false,
24
+ "single_word": false,
25
+ "special": true
26
+ },
27
+ "128003": {
28
+ "content": "<|reserved_special_token_1|>",
29
+ "lstrip": false,
30
+ "normalized": false,
31
+ "rstrip": false,
32
+ "single_word": false,
33
+ "special": true
34
+ },
35
+ "128004": {
36
+ "content": "<|finetune_right_pad_id|>",
37
+ "lstrip": false,
38
+ "normalized": false,
39
+ "rstrip": false,
40
+ "single_word": false,
41
+ "special": true
42
+ },
43
+ "128005": {
44
+ "content": "<|reserved_special_token_2|>",
45
+ "lstrip": false,
46
+ "normalized": false,
47
+ "rstrip": false,
48
+ "single_word": false,
49
+ "special": true
50
+ },
51
+ "128006": {
52
+ "content": "<|start_header_id|>",
53
+ "lstrip": false,
54
+ "normalized": false,
55
+ "rstrip": false,
56
+ "single_word": false,
57
+ "special": true
58
+ },
59
+ "128007": {
60
+ "content": "<|end_header_id|>",
61
+ "lstrip": false,
62
+ "normalized": false,
63
+ "rstrip": false,
64
+ "single_word": false,
65
+ "special": true
66
+ },
67
+ "128008": {
68
+ "content": "<|eom_id|>",
69
+ "lstrip": false,
70
+ "normalized": false,
71
+ "rstrip": false,
72
+ "single_word": false,
73
+ "special": true
74
+ },
75
+ "128009": {
76
+ "content": "<|eot_id|>",
77
+ "lstrip": false,
78
+ "normalized": false,
79
+ "rstrip": false,
80
+ "single_word": false,
81
+ "special": true
82
+ },
83
+ "128010": {
84
+ "content": "<|python_tag|>",
85
+ "lstrip": false,
86
+ "normalized": false,
87
+ "rstrip": false,
88
+ "single_word": false,
89
+ "special": true
90
+ },
91
+ "128011": {
92
+ "content": "<|reserved_special_token_3|>",
93
+ "lstrip": false,
94
+ "normalized": false,
95
+ "rstrip": false,
96
+ "single_word": false,
97
+ "special": true
98
+ },
99
+ "128012": {
100
+ "content": "<|reserved_special_token_4|>",
101
+ "lstrip": false,
102
+ "normalized": false,
103
+ "rstrip": false,
104
+ "single_word": false,
105
+ "special": true
106
+ },
107
+ "128013": {
108
+ "content": "<|reserved_special_token_5|>",
109
+ "lstrip": false,
110
+ "normalized": false,
111
+ "rstrip": false,
112
+ "single_word": false,
113
+ "special": true
114
+ },
115
+ "128014": {
116
+ "content": "<|reserved_special_token_6|>",
117
+ "lstrip": false,
118
+ "normalized": false,
119
+ "rstrip": false,
120
+ "single_word": false,
121
+ "special": true
122
+ },
123
+ "128015": {
124
+ "content": "<|reserved_special_token_7|>",
125
+ "lstrip": false,
126
+ "normalized": false,
127
+ "rstrip": false,
128
+ "single_word": false,
129
+ "special": true
130
+ },
131
+ "128016": {
132
+ "content": "<|reserved_special_token_8|>",
133
+ "lstrip": false,
134
+ "normalized": false,
135
+ "rstrip": false,
136
+ "single_word": false,
137
+ "special": true
138
+ },
139
+ "128017": {
140
+ "content": "<|reserved_special_token_9|>",
141
+ "lstrip": false,
142
+ "normalized": false,
143
+ "rstrip": false,
144
+ "single_word": false,
145
+ "special": true
146
+ },
147
+ "128018": {
148
+ "content": "<|reserved_special_token_10|>",
149
+ "lstrip": false,
150
+ "normalized": false,
151
+ "rstrip": false,
152
+ "single_word": false,
153
+ "special": true
154
+ },
155
+ "128019": {
156
+ "content": "<|reserved_special_token_11|>",
157
+ "lstrip": false,
158
+ "normalized": false,
159
+ "rstrip": false,
160
+ "single_word": false,
161
+ "special": true
162
+ },
163
+ "128020": {
164
+ "content": "<|reserved_special_token_12|>",
165
+ "lstrip": false,
166
+ "normalized": false,
167
+ "rstrip": false,
168
+ "single_word": false,
169
+ "special": true
170
+ },
171
+ "128021": {
172
+ "content": "<|reserved_special_token_13|>",
173
+ "lstrip": false,
174
+ "normalized": false,
175
+ "rstrip": false,
176
+ "single_word": false,
177
+ "special": true
178
+ },
179
+ "128022": {
180
+ "content": "<|reserved_special_token_14|>",
181
+ "lstrip": false,
182
+ "normalized": false,
183
+ "rstrip": false,
184
+ "single_word": false,
185
+ "special": true
186
+ },
187
+ "128023": {
188
+ "content": "<|reserved_special_token_15|>",
189
+ "lstrip": false,
190
+ "normalized": false,
191
+ "rstrip": false,
192
+ "single_word": false,
193
+ "special": true
194
+ },
195
+ "128024": {
196
+ "content": "<|reserved_special_token_16|>",
197
+ "lstrip": false,
198
+ "normalized": false,
199
+ "rstrip": false,
200
+ "single_word": false,
201
+ "special": true
202
+ },
203
+ "128025": {
204
+ "content": "<|reserved_special_token_17|>",
205
+ "lstrip": false,
206
+ "normalized": false,
207
+ "rstrip": false,
208
+ "single_word": false,
209
+ "special": true
210
+ },
211
+ "128026": {
212
+ "content": "<|reserved_special_token_18|>",
213
+ "lstrip": false,
214
+ "normalized": false,
215
+ "rstrip": false,
216
+ "single_word": false,
217
+ "special": true
218
+ },
219
+ "128027": {
220
+ "content": "<|reserved_special_token_19|>",
221
+ "lstrip": false,
222
+ "normalized": false,
223
+ "rstrip": false,
224
+ "single_word": false,
225
+ "special": true
226
+ },
227
+ "128028": {
228
+ "content": "<|reserved_special_token_20|>",
229
+ "lstrip": false,
230
+ "normalized": false,
231
+ "rstrip": false,
232
+ "single_word": false,
233
+ "special": true
234
+ },
235
+ "128029": {
236
+ "content": "<|reserved_special_token_21|>",
237
+ "lstrip": false,
238
+ "normalized": false,
239
+ "rstrip": false,
240
+ "single_word": false,
241
+ "special": true
242
+ },
243
+ "128030": {
244
+ "content": "<|reserved_special_token_22|>",
245
+ "lstrip": false,
246
+ "normalized": false,
247
+ "rstrip": false,
248
+ "single_word": false,
249
+ "special": true
250
+ },
251
+ "128031": {
252
+ "content": "<|reserved_special_token_23|>",
253
+ "lstrip": false,
254
+ "normalized": false,
255
+ "rstrip": false,
256
+ "single_word": false,
257
+ "special": true
258
+ },
259
+ "128032": {
260
+ "content": "<|reserved_special_token_24|>",
261
+ "lstrip": false,
262
+ "normalized": false,
263
+ "rstrip": false,
264
+ "single_word": false,
265
+ "special": true
266
+ },
267
+ "128033": {
268
+ "content": "<|reserved_special_token_25|>",
269
+ "lstrip": false,
270
+ "normalized": false,
271
+ "rstrip": false,
272
+ "single_word": false,
273
+ "special": true
274
+ },
275
+ "128034": {
276
+ "content": "<|reserved_special_token_26|>",
277
+ "lstrip": false,
278
+ "normalized": false,
279
+ "rstrip": false,
280
+ "single_word": false,
281
+ "special": true
282
+ },
283
+ "128035": {
284
+ "content": "<|reserved_special_token_27|>",
285
+ "lstrip": false,
286
+ "normalized": false,
287
+ "rstrip": false,
288
+ "single_word": false,
289
+ "special": true
290
+ },
291
+ "128036": {
292
+ "content": "<|reserved_special_token_28|>",
293
+ "lstrip": false,
294
+ "normalized": false,
295
+ "rstrip": false,
296
+ "single_word": false,
297
+ "special": true
298
+ },
299
+ "128037": {
300
+ "content": "<|reserved_special_token_29|>",
301
+ "lstrip": false,
302
+ "normalized": false,
303
+ "rstrip": false,
304
+ "single_word": false,
305
+ "special": true
306
+ },
307
+ "128038": {
308
+ "content": "<|reserved_special_token_30|>",
309
+ "lstrip": false,
310
+ "normalized": false,
311
+ "rstrip": false,
312
+ "single_word": false,
313
+ "special": true
314
+ },
315
+ "128039": {
316
+ "content": "<|reserved_special_token_31|>",
317
+ "lstrip": false,
318
+ "normalized": false,
319
+ "rstrip": false,
320
+ "single_word": false,
321
+ "special": true
322
+ },
323
+ "128040": {
324
+ "content": "<|reserved_special_token_32|>",
325
+ "lstrip": false,
326
+ "normalized": false,
327
+ "rstrip": false,
328
+ "single_word": false,
329
+ "special": true
330
+ },
331
+ "128041": {
332
+ "content": "<|reserved_special_token_33|>",
333
+ "lstrip": false,
334
+ "normalized": false,
335
+ "rstrip": false,
336
+ "single_word": false,
337
+ "special": true
338
+ },
339
+ "128042": {
340
+ "content": "<|reserved_special_token_34|>",
341
+ "lstrip": false,
342
+ "normalized": false,
343
+ "rstrip": false,
344
+ "single_word": false,
345
+ "special": true
346
+ },
347
+ "128043": {
348
+ "content": "<|reserved_special_token_35|>",
349
+ "lstrip": false,
350
+ "normalized": false,
351
+ "rstrip": false,
352
+ "single_word": false,
353
+ "special": true
354
+ },
355
+ "128044": {
356
+ "content": "<|reserved_special_token_36|>",
357
+ "lstrip": false,
358
+ "normalized": false,
359
+ "rstrip": false,
360
+ "single_word": false,
361
+ "special": true
362
+ },
363
+ "128045": {
364
+ "content": "<|reserved_special_token_37|>",
365
+ "lstrip": false,
366
+ "normalized": false,
367
+ "rstrip": false,
368
+ "single_word": false,
369
+ "special": true
370
+ },
371
+ "128046": {
372
+ "content": "<|reserved_special_token_38|>",
373
+ "lstrip": false,
374
+ "normalized": false,
375
+ "rstrip": false,
376
+ "single_word": false,
377
+ "special": true
378
+ },
379
+ "128047": {
380
+ "content": "<|reserved_special_token_39|>",
381
+ "lstrip": false,
382
+ "normalized": false,
383
+ "rstrip": false,
384
+ "single_word": false,
385
+ "special": true
386
+ },
387
+ "128048": {
388
+ "content": "<|reserved_special_token_40|>",
389
+ "lstrip": false,
390
+ "normalized": false,
391
+ "rstrip": false,
392
+ "single_word": false,
393
+ "special": true
394
+ },
395
+ "128049": {
396
+ "content": "<|reserved_special_token_41|>",
397
+ "lstrip": false,
398
+ "normalized": false,
399
+ "rstrip": false,
400
+ "single_word": false,
401
+ "special": true
402
+ },
403
+ "128050": {
404
+ "content": "<|reserved_special_token_42|>",
405
+ "lstrip": false,
406
+ "normalized": false,
407
+ "rstrip": false,
408
+ "single_word": false,
409
+ "special": true
410
+ },
411
+ "128051": {
412
+ "content": "<|reserved_special_token_43|>",
413
+ "lstrip": false,
414
+ "normalized": false,
415
+ "rstrip": false,
416
+ "single_word": false,
417
+ "special": true
418
+ },
419
+ "128052": {
420
+ "content": "<|reserved_special_token_44|>",
421
+ "lstrip": false,
422
+ "normalized": false,
423
+ "rstrip": false,
424
+ "single_word": false,
425
+ "special": true
426
+ },
427
+ "128053": {
428
+ "content": "<|reserved_special_token_45|>",
429
+ "lstrip": false,
430
+ "normalized": false,
431
+ "rstrip": false,
432
+ "single_word": false,
433
+ "special": true
434
+ },
435
+ "128054": {
436
+ "content": "<|reserved_special_token_46|>",
437
+ "lstrip": false,
438
+ "normalized": false,
439
+ "rstrip": false,
440
+ "single_word": false,
441
+ "special": true
442
+ },
443
+ "128055": {
444
+ "content": "<|reserved_special_token_47|>",
445
+ "lstrip": false,
446
+ "normalized": false,
447
+ "rstrip": false,
448
+ "single_word": false,
449
+ "special": true
450
+ },
451
+ "128056": {
452
+ "content": "<|reserved_special_token_48|>",
453
+ "lstrip": false,
454
+ "normalized": false,
455
+ "rstrip": false,
456
+ "single_word": false,
457
+ "special": true
458
+ },
459
+ "128057": {
460
+ "content": "<|reserved_special_token_49|>",
461
+ "lstrip": false,
462
+ "normalized": false,
463
+ "rstrip": false,
464
+ "single_word": false,
465
+ "special": true
466
+ },
467
+ "128058": {
468
+ "content": "<|reserved_special_token_50|>",
469
+ "lstrip": false,
470
+ "normalized": false,
471
+ "rstrip": false,
472
+ "single_word": false,
473
+ "special": true
474
+ },
475
+ "128059": {
476
+ "content": "<|reserved_special_token_51|>",
477
+ "lstrip": false,
478
+ "normalized": false,
479
+ "rstrip": false,
480
+ "single_word": false,
481
+ "special": true
482
+ },
483
+ "128060": {
484
+ "content": "<|reserved_special_token_52|>",
485
+ "lstrip": false,
486
+ "normalized": false,
487
+ "rstrip": false,
488
+ "single_word": false,
489
+ "special": true
490
+ },
491
+ "128061": {
492
+ "content": "<|reserved_special_token_53|>",
493
+ "lstrip": false,
494
+ "normalized": false,
495
+ "rstrip": false,
496
+ "single_word": false,
497
+ "special": true
498
+ },
499
+ "128062": {
500
+ "content": "<|reserved_special_token_54|>",
501
+ "lstrip": false,
502
+ "normalized": false,
503
+ "rstrip": false,
504
+ "single_word": false,
505
+ "special": true
506
+ },
507
+ "128063": {
508
+ "content": "<|reserved_special_token_55|>",
509
+ "lstrip": false,
510
+ "normalized": false,
511
+ "rstrip": false,
512
+ "single_word": false,
513
+ "special": true
514
+ },
515
+ "128064": {
516
+ "content": "<|reserved_special_token_56|>",
517
+ "lstrip": false,
518
+ "normalized": false,
519
+ "rstrip": false,
520
+ "single_word": false,
521
+ "special": true
522
+ },
523
+ "128065": {
524
+ "content": "<|reserved_special_token_57|>",
525
+ "lstrip": false,
526
+ "normalized": false,
527
+ "rstrip": false,
528
+ "single_word": false,
529
+ "special": true
530
+ },
531
+ "128066": {
532
+ "content": "<|reserved_special_token_58|>",
533
+ "lstrip": false,
534
+ "normalized": false,
535
+ "rstrip": false,
536
+ "single_word": false,
537
+ "special": true
538
+ },
539
+ "128067": {
540
+ "content": "<|reserved_special_token_59|>",
541
+ "lstrip": false,
542
+ "normalized": false,
543
+ "rstrip": false,
544
+ "single_word": false,
545
+ "special": true
546
+ },
547
+ "128068": {
548
+ "content": "<|reserved_special_token_60|>",
549
+ "lstrip": false,
550
+ "normalized": false,
551
+ "rstrip": false,
552
+ "single_word": false,
553
+ "special": true
554
+ },
555
+ "128069": {
556
+ "content": "<|reserved_special_token_61|>",
557
+ "lstrip": false,
558
+ "normalized": false,
559
+ "rstrip": false,
560
+ "single_word": false,
561
+ "special": true
562
+ },
563
+ "128070": {
564
+ "content": "<|reserved_special_token_62|>",
565
+ "lstrip": false,
566
+ "normalized": false,
567
+ "rstrip": false,
568
+ "single_word": false,
569
+ "special": true
570
+ },
571
+ "128071": {
572
+ "content": "<|reserved_special_token_63|>",
573
+ "lstrip": false,
574
+ "normalized": false,
575
+ "rstrip": false,
576
+ "single_word": false,
577
+ "special": true
578
+ },
579
+ "128072": {
580
+ "content": "<|reserved_special_token_64|>",
581
+ "lstrip": false,
582
+ "normalized": false,
583
+ "rstrip": false,
584
+ "single_word": false,
585
+ "special": true
586
+ },
587
+ "128073": {
588
+ "content": "<|reserved_special_token_65|>",
589
+ "lstrip": false,
590
+ "normalized": false,
591
+ "rstrip": false,
592
+ "single_word": false,
593
+ "special": true
594
+ },
595
+ "128074": {
596
+ "content": "<|reserved_special_token_66|>",
597
+ "lstrip": false,
598
+ "normalized": false,
599
+ "rstrip": false,
600
+ "single_word": false,
601
+ "special": true
602
+ },
603
+ "128075": {
604
+ "content": "<|reserved_special_token_67|>",
605
+ "lstrip": false,
606
+ "normalized": false,
607
+ "rstrip": false,
608
+ "single_word": false,
609
+ "special": true
610
+ },
611
+ "128076": {
612
+ "content": "<|reserved_special_token_68|>",
613
+ "lstrip": false,
614
+ "normalized": false,
615
+ "rstrip": false,
616
+ "single_word": false,
617
+ "special": true
618
+ },
619
+ "128077": {
620
+ "content": "<|reserved_special_token_69|>",
621
+ "lstrip": false,
622
+ "normalized": false,
623
+ "rstrip": false,
624
+ "single_word": false,
625
+ "special": true
626
+ },
627
+ "128078": {
628
+ "content": "<|reserved_special_token_70|>",
629
+ "lstrip": false,
630
+ "normalized": false,
631
+ "rstrip": false,
632
+ "single_word": false,
633
+ "special": true
634
+ },
635
+ "128079": {
636
+ "content": "<|reserved_special_token_71|>",
637
+ "lstrip": false,
638
+ "normalized": false,
639
+ "rstrip": false,
640
+ "single_word": false,
641
+ "special": true
642
+ },
643
+ "128080": {
644
+ "content": "<|reserved_special_token_72|>",
645
+ "lstrip": false,
646
+ "normalized": false,
647
+ "rstrip": false,
648
+ "single_word": false,
649
+ "special": true
650
+ },
651
+ "128081": {
652
+ "content": "<|reserved_special_token_73|>",
653
+ "lstrip": false,
654
+ "normalized": false,
655
+ "rstrip": false,
656
+ "single_word": false,
657
+ "special": true
658
+ },
659
+ "128082": {
660
+ "content": "<|reserved_special_token_74|>",
661
+ "lstrip": false,
662
+ "normalized": false,
663
+ "rstrip": false,
664
+ "single_word": false,
665
+ "special": true
666
+ },
667
+ "128083": {
668
+ "content": "<|reserved_special_token_75|>",
669
+ "lstrip": false,
670
+ "normalized": false,
671
+ "rstrip": false,
672
+ "single_word": false,
673
+ "special": true
674
+ },
675
+ "128084": {
676
+ "content": "<|reserved_special_token_76|>",
677
+ "lstrip": false,
678
+ "normalized": false,
679
+ "rstrip": false,
680
+ "single_word": false,
681
+ "special": true
682
+ },
683
+ "128085": {
684
+ "content": "<|reserved_special_token_77|>",
685
+ "lstrip": false,
686
+ "normalized": false,
687
+ "rstrip": false,
688
+ "single_word": false,
689
+ "special": true
690
+ },
691
+ "128086": {
692
+ "content": "<|reserved_special_token_78|>",
693
+ "lstrip": false,
694
+ "normalized": false,
695
+ "rstrip": false,
696
+ "single_word": false,
697
+ "special": true
698
+ },
699
+ "128087": {
700
+ "content": "<|reserved_special_token_79|>",
701
+ "lstrip": false,
702
+ "normalized": false,
703
+ "rstrip": false,
704
+ "single_word": false,
705
+ "special": true
706
+ },
707
+ "128088": {
708
+ "content": "<|reserved_special_token_80|>",
709
+ "lstrip": false,
710
+ "normalized": false,
711
+ "rstrip": false,
712
+ "single_word": false,
713
+ "special": true
714
+ },
715
+ "128089": {
716
+ "content": "<|reserved_special_token_81|>",
717
+ "lstrip": false,
718
+ "normalized": false,
719
+ "rstrip": false,
720
+ "single_word": false,
721
+ "special": true
722
+ },
723
+ "128090": {
724
+ "content": "<|reserved_special_token_82|>",
725
+ "lstrip": false,
726
+ "normalized": false,
727
+ "rstrip": false,
728
+ "single_word": false,
729
+ "special": true
730
+ },
731
+ "128091": {
732
+ "content": "<|reserved_special_token_83|>",
733
+ "lstrip": false,
734
+ "normalized": false,
735
+ "rstrip": false,
736
+ "single_word": false,
737
+ "special": true
738
+ },
739
+ "128092": {
740
+ "content": "<|reserved_special_token_84|>",
741
+ "lstrip": false,
742
+ "normalized": false,
743
+ "rstrip": false,
744
+ "single_word": false,
745
+ "special": true
746
+ },
747
+ "128093": {
748
+ "content": "<|reserved_special_token_85|>",
749
+ "lstrip": false,
750
+ "normalized": false,
751
+ "rstrip": false,
752
+ "single_word": false,
753
+ "special": true
754
+ },
755
+ "128094": {
756
+ "content": "<|reserved_special_token_86|>",
757
+ "lstrip": false,
758
+ "normalized": false,
759
+ "rstrip": false,
760
+ "single_word": false,
761
+ "special": true
762
+ },
763
+ "128095": {
764
+ "content": "<|reserved_special_token_87|>",
765
+ "lstrip": false,
766
+ "normalized": false,
767
+ "rstrip": false,
768
+ "single_word": false,
769
+ "special": true
770
+ },
771
+ "128096": {
772
+ "content": "<|reserved_special_token_88|>",
773
+ "lstrip": false,
774
+ "normalized": false,
775
+ "rstrip": false,
776
+ "single_word": false,
777
+ "special": true
778
+ },
779
+ "128097": {
780
+ "content": "<|reserved_special_token_89|>",
781
+ "lstrip": false,
782
+ "normalized": false,
783
+ "rstrip": false,
784
+ "single_word": false,
785
+ "special": true
786
+ },
787
+ "128098": {
788
+ "content": "<|reserved_special_token_90|>",
789
+ "lstrip": false,
790
+ "normalized": false,
791
+ "rstrip": false,
792
+ "single_word": false,
793
+ "special": true
794
+ },
795
+ "128099": {
796
+ "content": "<|reserved_special_token_91|>",
797
+ "lstrip": false,
798
+ "normalized": false,
799
+ "rstrip": false,
800
+ "single_word": false,
801
+ "special": true
802
+ },
803
+ "128100": {
804
+ "content": "<|reserved_special_token_92|>",
805
+ "lstrip": false,
806
+ "normalized": false,
807
+ "rstrip": false,
808
+ "single_word": false,
809
+ "special": true
810
+ },
811
+ "128101": {
812
+ "content": "<|reserved_special_token_93|>",
813
+ "lstrip": false,
814
+ "normalized": false,
815
+ "rstrip": false,
816
+ "single_word": false,
817
+ "special": true
818
+ },
819
+ "128102": {
820
+ "content": "<|reserved_special_token_94|>",
821
+ "lstrip": false,
822
+ "normalized": false,
823
+ "rstrip": false,
824
+ "single_word": false,
825
+ "special": true
826
+ },
827
+ "128103": {
828
+ "content": "<|reserved_special_token_95|>",
829
+ "lstrip": false,
830
+ "normalized": false,
831
+ "rstrip": false,
832
+ "single_word": false,
833
+ "special": true
834
+ },
835
+ "128104": {
836
+ "content": "<|reserved_special_token_96|>",
837
+ "lstrip": false,
838
+ "normalized": false,
839
+ "rstrip": false,
840
+ "single_word": false,
841
+ "special": true
842
+ },
843
+ "128105": {
844
+ "content": "<|reserved_special_token_97|>",
845
+ "lstrip": false,
846
+ "normalized": false,
847
+ "rstrip": false,
848
+ "single_word": false,
849
+ "special": true
850
+ },
851
+ "128106": {
852
+ "content": "<|reserved_special_token_98|>",
853
+ "lstrip": false,
854
+ "normalized": false,
855
+ "rstrip": false,
856
+ "single_word": false,
857
+ "special": true
858
+ },
859
+ "128107": {
860
+ "content": "<|reserved_special_token_99|>",
861
+ "lstrip": false,
862
+ "normalized": false,
863
+ "rstrip": false,
864
+ "single_word": false,
865
+ "special": true
866
+ },
867
+ "128108": {
868
+ "content": "<|reserved_special_token_100|>",
869
+ "lstrip": false,
870
+ "normalized": false,
871
+ "rstrip": false,
872
+ "single_word": false,
873
+ "special": true
874
+ },
875
+ "128109": {
876
+ "content": "<|reserved_special_token_101|>",
877
+ "lstrip": false,
878
+ "normalized": false,
879
+ "rstrip": false,
880
+ "single_word": false,
881
+ "special": true
882
+ },
883
+ "128110": {
884
+ "content": "<|reserved_special_token_102|>",
885
+ "lstrip": false,
886
+ "normalized": false,
887
+ "rstrip": false,
888
+ "single_word": false,
889
+ "special": true
890
+ },
891
+ "128111": {
892
+ "content": "<|reserved_special_token_103|>",
893
+ "lstrip": false,
894
+ "normalized": false,
895
+ "rstrip": false,
896
+ "single_word": false,
897
+ "special": true
898
+ },
899
+ "128112": {
900
+ "content": "<|reserved_special_token_104|>",
901
+ "lstrip": false,
902
+ "normalized": false,
903
+ "rstrip": false,
904
+ "single_word": false,
905
+ "special": true
906
+ },
907
+ "128113": {
908
+ "content": "<|reserved_special_token_105|>",
909
+ "lstrip": false,
910
+ "normalized": false,
911
+ "rstrip": false,
912
+ "single_word": false,
913
+ "special": true
914
+ },
915
+ "128114": {
916
+ "content": "<|reserved_special_token_106|>",
917
+ "lstrip": false,
918
+ "normalized": false,
919
+ "rstrip": false,
920
+ "single_word": false,
921
+ "special": true
922
+ },
923
+ "128115": {
924
+ "content": "<|reserved_special_token_107|>",
925
+ "lstrip": false,
926
+ "normalized": false,
927
+ "rstrip": false,
928
+ "single_word": false,
929
+ "special": true
930
+ },
931
+ "128116": {
932
+ "content": "<|reserved_special_token_108|>",
933
+ "lstrip": false,
934
+ "normalized": false,
935
+ "rstrip": false,
936
+ "single_word": false,
937
+ "special": true
938
+ },
939
+ "128117": {
940
+ "content": "<|reserved_special_token_109|>",
941
+ "lstrip": false,
942
+ "normalized": false,
943
+ "rstrip": false,
944
+ "single_word": false,
945
+ "special": true
946
+ },
947
+ "128118": {
948
+ "content": "<|reserved_special_token_110|>",
949
+ "lstrip": false,
950
+ "normalized": false,
951
+ "rstrip": false,
952
+ "single_word": false,
953
+ "special": true
954
+ },
955
+ "128119": {
956
+ "content": "<|reserved_special_token_111|>",
957
+ "lstrip": false,
958
+ "normalized": false,
959
+ "rstrip": false,
960
+ "single_word": false,
961
+ "special": true
962
+ },
963
+ "128120": {
964
+ "content": "<|reserved_special_token_112|>",
965
+ "lstrip": false,
966
+ "normalized": false,
967
+ "rstrip": false,
968
+ "single_word": false,
969
+ "special": true
970
+ },
971
+ "128121": {
972
+ "content": "<|reserved_special_token_113|>",
973
+ "lstrip": false,
974
+ "normalized": false,
975
+ "rstrip": false,
976
+ "single_word": false,
977
+ "special": true
978
+ },
979
+ "128122": {
980
+ "content": "<|reserved_special_token_114|>",
981
+ "lstrip": false,
982
+ "normalized": false,
983
+ "rstrip": false,
984
+ "single_word": false,
985
+ "special": true
986
+ },
987
+ "128123": {
988
+ "content": "<|reserved_special_token_115|>",
989
+ "lstrip": false,
990
+ "normalized": false,
991
+ "rstrip": false,
992
+ "single_word": false,
993
+ "special": true
994
+ },
995
+ "128124": {
996
+ "content": "<|reserved_special_token_116|>",
997
+ "lstrip": false,
998
+ "normalized": false,
999
+ "rstrip": false,
1000
+ "single_word": false,
1001
+ "special": true
1002
+ },
1003
+ "128125": {
1004
+ "content": "<|reserved_special_token_117|>",
1005
+ "lstrip": false,
1006
+ "normalized": false,
1007
+ "rstrip": false,
1008
+ "single_word": false,
1009
+ "special": true
1010
+ },
1011
+ "128126": {
1012
+ "content": "<|reserved_special_token_118|>",
1013
+ "lstrip": false,
1014
+ "normalized": false,
1015
+ "rstrip": false,
1016
+ "single_word": false,
1017
+ "special": true
1018
+ },
1019
+ "128127": {
1020
+ "content": "<|reserved_special_token_119|>",
1021
+ "lstrip": false,
1022
+ "normalized": false,
1023
+ "rstrip": false,
1024
+ "single_word": false,
1025
+ "special": true
1026
+ },
1027
+ "128128": {
1028
+ "content": "<|reserved_special_token_120|>",
1029
+ "lstrip": false,
1030
+ "normalized": false,
1031
+ "rstrip": false,
1032
+ "single_word": false,
1033
+ "special": true
1034
+ },
1035
+ "128129": {
1036
+ "content": "<|reserved_special_token_121|>",
1037
+ "lstrip": false,
1038
+ "normalized": false,
1039
+ "rstrip": false,
1040
+ "single_word": false,
1041
+ "special": true
1042
+ },
1043
+ "128130": {
1044
+ "content": "<|reserved_special_token_122|>",
1045
+ "lstrip": false,
1046
+ "normalized": false,
1047
+ "rstrip": false,
1048
+ "single_word": false,
1049
+ "special": true
1050
+ },
1051
+ "128131": {
1052
+ "content": "<|reserved_special_token_123|>",
1053
+ "lstrip": false,
1054
+ "normalized": false,
1055
+ "rstrip": false,
1056
+ "single_word": false,
1057
+ "special": true
1058
+ },
1059
+ "128132": {
1060
+ "content": "<|reserved_special_token_124|>",
1061
+ "lstrip": false,
1062
+ "normalized": false,
1063
+ "rstrip": false,
1064
+ "single_word": false,
1065
+ "special": true
1066
+ },
1067
+ "128133": {
1068
+ "content": "<|reserved_special_token_125|>",
1069
+ "lstrip": false,
1070
+ "normalized": false,
1071
+ "rstrip": false,
1072
+ "single_word": false,
1073
+ "special": true
1074
+ },
1075
+ "128134": {
1076
+ "content": "<|reserved_special_token_126|>",
1077
+ "lstrip": false,
1078
+ "normalized": false,
1079
+ "rstrip": false,
1080
+ "single_word": false,
1081
+ "special": true
1082
+ },
1083
+ "128135": {
1084
+ "content": "<|reserved_special_token_127|>",
1085
+ "lstrip": false,
1086
+ "normalized": false,
1087
+ "rstrip": false,
1088
+ "single_word": false,
1089
+ "special": true
1090
+ },
1091
+ "128136": {
1092
+ "content": "<|reserved_special_token_128|>",
1093
+ "lstrip": false,
1094
+ "normalized": false,
1095
+ "rstrip": false,
1096
+ "single_word": false,
1097
+ "special": true
1098
+ },
1099
+ "128137": {
1100
+ "content": "<|reserved_special_token_129|>",
1101
+ "lstrip": false,
1102
+ "normalized": false,
1103
+ "rstrip": false,
1104
+ "single_word": false,
1105
+ "special": true
1106
+ },
1107
+ "128138": {
1108
+ "content": "<|reserved_special_token_130|>",
1109
+ "lstrip": false,
1110
+ "normalized": false,
1111
+ "rstrip": false,
1112
+ "single_word": false,
1113
+ "special": true
1114
+ },
1115
+ "128139": {
1116
+ "content": "<|reserved_special_token_131|>",
1117
+ "lstrip": false,
1118
+ "normalized": false,
1119
+ "rstrip": false,
1120
+ "single_word": false,
1121
+ "special": true
1122
+ },
1123
+ "128140": {
1124
+ "content": "<|reserved_special_token_132|>",
1125
+ "lstrip": false,
1126
+ "normalized": false,
1127
+ "rstrip": false,
1128
+ "single_word": false,
1129
+ "special": true
1130
+ },
1131
+ "128141": {
1132
+ "content": "<|reserved_special_token_133|>",
1133
+ "lstrip": false,
1134
+ "normalized": false,
1135
+ "rstrip": false,
1136
+ "single_word": false,
1137
+ "special": true
1138
+ },
1139
+ "128142": {
1140
+ "content": "<|reserved_special_token_134|>",
1141
+ "lstrip": false,
1142
+ "normalized": false,
1143
+ "rstrip": false,
1144
+ "single_word": false,
1145
+ "special": true
1146
+ },
1147
+ "128143": {
1148
+ "content": "<|reserved_special_token_135|>",
1149
+ "lstrip": false,
1150
+ "normalized": false,
1151
+ "rstrip": false,
1152
+ "single_word": false,
1153
+ "special": true
1154
+ },
1155
+ "128144": {
1156
+ "content": "<|reserved_special_token_136|>",
1157
+ "lstrip": false,
1158
+ "normalized": false,
1159
+ "rstrip": false,
1160
+ "single_word": false,
1161
+ "special": true
1162
+ },
1163
+ "128145": {
1164
+ "content": "<|reserved_special_token_137|>",
1165
+ "lstrip": false,
1166
+ "normalized": false,
1167
+ "rstrip": false,
1168
+ "single_word": false,
1169
+ "special": true
1170
+ },
1171
+ "128146": {
1172
+ "content": "<|reserved_special_token_138|>",
1173
+ "lstrip": false,
1174
+ "normalized": false,
1175
+ "rstrip": false,
1176
+ "single_word": false,
1177
+ "special": true
1178
+ },
1179
+ "128147": {
1180
+ "content": "<|reserved_special_token_139|>",
1181
+ "lstrip": false,
1182
+ "normalized": false,
1183
+ "rstrip": false,
1184
+ "single_word": false,
1185
+ "special": true
1186
+ },
1187
+ "128148": {
1188
+ "content": "<|reserved_special_token_140|>",
1189
+ "lstrip": false,
1190
+ "normalized": false,
1191
+ "rstrip": false,
1192
+ "single_word": false,
1193
+ "special": true
1194
+ },
1195
+ "128149": {
1196
+ "content": "<|reserved_special_token_141|>",
1197
+ "lstrip": false,
1198
+ "normalized": false,
1199
+ "rstrip": false,
1200
+ "single_word": false,
1201
+ "special": true
1202
+ },
1203
+ "128150": {
1204
+ "content": "<|reserved_special_token_142|>",
1205
+ "lstrip": false,
1206
+ "normalized": false,
1207
+ "rstrip": false,
1208
+ "single_word": false,
1209
+ "special": true
1210
+ },
1211
+ "128151": {
1212
+ "content": "<|reserved_special_token_143|>",
1213
+ "lstrip": false,
1214
+ "normalized": false,
1215
+ "rstrip": false,
1216
+ "single_word": false,
1217
+ "special": true
1218
+ },
1219
+ "128152": {
1220
+ "content": "<|reserved_special_token_144|>",
1221
+ "lstrip": false,
1222
+ "normalized": false,
1223
+ "rstrip": false,
1224
+ "single_word": false,
1225
+ "special": true
1226
+ },
1227
+ "128153": {
1228
+ "content": "<|reserved_special_token_145|>",
1229
+ "lstrip": false,
1230
+ "normalized": false,
1231
+ "rstrip": false,
1232
+ "single_word": false,
1233
+ "special": true
1234
+ },
1235
+ "128154": {
1236
+ "content": "<|reserved_special_token_146|>",
1237
+ "lstrip": false,
1238
+ "normalized": false,
1239
+ "rstrip": false,
1240
+ "single_word": false,
1241
+ "special": true
1242
+ },
1243
+ "128155": {
1244
+ "content": "<|reserved_special_token_147|>",
1245
+ "lstrip": false,
1246
+ "normalized": false,
1247
+ "rstrip": false,
1248
+ "single_word": false,
1249
+ "special": true
1250
+ },
1251
+ "128156": {
1252
+ "content": "<|reserved_special_token_148|>",
1253
+ "lstrip": false,
1254
+ "normalized": false,
1255
+ "rstrip": false,
1256
+ "single_word": false,
1257
+ "special": true
1258
+ },
1259
+ "128157": {
1260
+ "content": "<|reserved_special_token_149|>",
1261
+ "lstrip": false,
1262
+ "normalized": false,
1263
+ "rstrip": false,
1264
+ "single_word": false,
1265
+ "special": true
1266
+ },
1267
+ "128158": {
1268
+ "content": "<|reserved_special_token_150|>",
1269
+ "lstrip": false,
1270
+ "normalized": false,
1271
+ "rstrip": false,
1272
+ "single_word": false,
1273
+ "special": true
1274
+ },
1275
+ "128159": {
1276
+ "content": "<|reserved_special_token_151|>",
1277
+ "lstrip": false,
1278
+ "normalized": false,
1279
+ "rstrip": false,
1280
+ "single_word": false,
1281
+ "special": true
1282
+ },
1283
+ "128160": {
1284
+ "content": "<|reserved_special_token_152|>",
1285
+ "lstrip": false,
1286
+ "normalized": false,
1287
+ "rstrip": false,
1288
+ "single_word": false,
1289
+ "special": true
1290
+ },
1291
+ "128161": {
1292
+ "content": "<|reserved_special_token_153|>",
1293
+ "lstrip": false,
1294
+ "normalized": false,
1295
+ "rstrip": false,
1296
+ "single_word": false,
1297
+ "special": true
1298
+ },
1299
+ "128162": {
1300
+ "content": "<|reserved_special_token_154|>",
1301
+ "lstrip": false,
1302
+ "normalized": false,
1303
+ "rstrip": false,
1304
+ "single_word": false,
1305
+ "special": true
1306
+ },
1307
+ "128163": {
1308
+ "content": "<|reserved_special_token_155|>",
1309
+ "lstrip": false,
1310
+ "normalized": false,
1311
+ "rstrip": false,
1312
+ "single_word": false,
1313
+ "special": true
1314
+ },
1315
+ "128164": {
1316
+ "content": "<|reserved_special_token_156|>",
1317
+ "lstrip": false,
1318
+ "normalized": false,
1319
+ "rstrip": false,
1320
+ "single_word": false,
1321
+ "special": true
1322
+ },
1323
+ "128165": {
1324
+ "content": "<|reserved_special_token_157|>",
1325
+ "lstrip": false,
1326
+ "normalized": false,
1327
+ "rstrip": false,
1328
+ "single_word": false,
1329
+ "special": true
1330
+ },
1331
+ "128166": {
1332
+ "content": "<|reserved_special_token_158|>",
1333
+ "lstrip": false,
1334
+ "normalized": false,
1335
+ "rstrip": false,
1336
+ "single_word": false,
1337
+ "special": true
1338
+ },
1339
+ "128167": {
1340
+ "content": "<|reserved_special_token_159|>",
1341
+ "lstrip": false,
1342
+ "normalized": false,
1343
+ "rstrip": false,
1344
+ "single_word": false,
1345
+ "special": true
1346
+ },
1347
+ "128168": {
1348
+ "content": "<|reserved_special_token_160|>",
1349
+ "lstrip": false,
1350
+ "normalized": false,
1351
+ "rstrip": false,
1352
+ "single_word": false,
1353
+ "special": true
1354
+ },
1355
+ "128169": {
1356
+ "content": "<|reserved_special_token_161|>",
1357
+ "lstrip": false,
1358
+ "normalized": false,
1359
+ "rstrip": false,
1360
+ "single_word": false,
1361
+ "special": true
1362
+ },
1363
+ "128170": {
1364
+ "content": "<|reserved_special_token_162|>",
1365
+ "lstrip": false,
1366
+ "normalized": false,
1367
+ "rstrip": false,
1368
+ "single_word": false,
1369
+ "special": true
1370
+ },
1371
+ "128171": {
1372
+ "content": "<|reserved_special_token_163|>",
1373
+ "lstrip": false,
1374
+ "normalized": false,
1375
+ "rstrip": false,
1376
+ "single_word": false,
1377
+ "special": true
1378
+ },
1379
+ "128172": {
1380
+ "content": "<|reserved_special_token_164|>",
1381
+ "lstrip": false,
1382
+ "normalized": false,
1383
+ "rstrip": false,
1384
+ "single_word": false,
1385
+ "special": true
1386
+ },
1387
+ "128173": {
1388
+ "content": "<|reserved_special_token_165|>",
1389
+ "lstrip": false,
1390
+ "normalized": false,
1391
+ "rstrip": false,
1392
+ "single_word": false,
1393
+ "special": true
1394
+ },
1395
+ "128174": {
1396
+ "content": "<|reserved_special_token_166|>",
1397
+ "lstrip": false,
1398
+ "normalized": false,
1399
+ "rstrip": false,
1400
+ "single_word": false,
1401
+ "special": true
1402
+ },
1403
+ "128175": {
1404
+ "content": "<|reserved_special_token_167|>",
1405
+ "lstrip": false,
1406
+ "normalized": false,
1407
+ "rstrip": false,
1408
+ "single_word": false,
1409
+ "special": true
1410
+ },
1411
+ "128176": {
1412
+ "content": "<|reserved_special_token_168|>",
1413
+ "lstrip": false,
1414
+ "normalized": false,
1415
+ "rstrip": false,
1416
+ "single_word": false,
1417
+ "special": true
1418
+ },
1419
+ "128177": {
1420
+ "content": "<|reserved_special_token_169|>",
1421
+ "lstrip": false,
1422
+ "normalized": false,
1423
+ "rstrip": false,
1424
+ "single_word": false,
1425
+ "special": true
1426
+ },
1427
+ "128178": {
1428
+ "content": "<|reserved_special_token_170|>",
1429
+ "lstrip": false,
1430
+ "normalized": false,
1431
+ "rstrip": false,
1432
+ "single_word": false,
1433
+ "special": true
1434
+ },
1435
+ "128179": {
1436
+ "content": "<|reserved_special_token_171|>",
1437
+ "lstrip": false,
1438
+ "normalized": false,
1439
+ "rstrip": false,
1440
+ "single_word": false,
1441
+ "special": true
1442
+ },
1443
+ "128180": {
1444
+ "content": "<|reserved_special_token_172|>",
1445
+ "lstrip": false,
1446
+ "normalized": false,
1447
+ "rstrip": false,
1448
+ "single_word": false,
1449
+ "special": true
1450
+ },
1451
+ "128181": {
1452
+ "content": "<|reserved_special_token_173|>",
1453
+ "lstrip": false,
1454
+ "normalized": false,
1455
+ "rstrip": false,
1456
+ "single_word": false,
1457
+ "special": true
1458
+ },
1459
+ "128182": {
1460
+ "content": "<|reserved_special_token_174|>",
1461
+ "lstrip": false,
1462
+ "normalized": false,
1463
+ "rstrip": false,
1464
+ "single_word": false,
1465
+ "special": true
1466
+ },
1467
+ "128183": {
1468
+ "content": "<|reserved_special_token_175|>",
1469
+ "lstrip": false,
1470
+ "normalized": false,
1471
+ "rstrip": false,
1472
+ "single_word": false,
1473
+ "special": true
1474
+ },
1475
+ "128184": {
1476
+ "content": "<|reserved_special_token_176|>",
1477
+ "lstrip": false,
1478
+ "normalized": false,
1479
+ "rstrip": false,
1480
+ "single_word": false,
1481
+ "special": true
1482
+ },
1483
+ "128185": {
1484
+ "content": "<|reserved_special_token_177|>",
1485
+ "lstrip": false,
1486
+ "normalized": false,
1487
+ "rstrip": false,
1488
+ "single_word": false,
1489
+ "special": true
1490
+ },
1491
+ "128186": {
1492
+ "content": "<|reserved_special_token_178|>",
1493
+ "lstrip": false,
1494
+ "normalized": false,
1495
+ "rstrip": false,
1496
+ "single_word": false,
1497
+ "special": true
1498
+ },
1499
+ "128187": {
1500
+ "content": "<|reserved_special_token_179|>",
1501
+ "lstrip": false,
1502
+ "normalized": false,
1503
+ "rstrip": false,
1504
+ "single_word": false,
1505
+ "special": true
1506
+ },
1507
+ "128188": {
1508
+ "content": "<|reserved_special_token_180|>",
1509
+ "lstrip": false,
1510
+ "normalized": false,
1511
+ "rstrip": false,
1512
+ "single_word": false,
1513
+ "special": true
1514
+ },
1515
+ "128189": {
1516
+ "content": "<|reserved_special_token_181|>",
1517
+ "lstrip": false,
1518
+ "normalized": false,
1519
+ "rstrip": false,
1520
+ "single_word": false,
1521
+ "special": true
1522
+ },
1523
+ "128190": {
1524
+ "content": "<|reserved_special_token_182|>",
1525
+ "lstrip": false,
1526
+ "normalized": false,
1527
+ "rstrip": false,
1528
+ "single_word": false,
1529
+ "special": true
1530
+ },
1531
+ "128191": {
1532
+ "content": "<|reserved_special_token_183|>",
1533
+ "lstrip": false,
1534
+ "normalized": false,
1535
+ "rstrip": false,
1536
+ "single_word": false,
1537
+ "special": true
1538
+ },
1539
+ "128192": {
1540
+ "content": "<|reserved_special_token_184|>",
1541
+ "lstrip": false,
1542
+ "normalized": false,
1543
+ "rstrip": false,
1544
+ "single_word": false,
1545
+ "special": true
1546
+ },
1547
+ "128193": {
1548
+ "content": "<|reserved_special_token_185|>",
1549
+ "lstrip": false,
1550
+ "normalized": false,
1551
+ "rstrip": false,
1552
+ "single_word": false,
1553
+ "special": true
1554
+ },
1555
+ "128194": {
1556
+ "content": "<|reserved_special_token_186|>",
1557
+ "lstrip": false,
1558
+ "normalized": false,
1559
+ "rstrip": false,
1560
+ "single_word": false,
1561
+ "special": true
1562
+ },
1563
+ "128195": {
1564
+ "content": "<|reserved_special_token_187|>",
1565
+ "lstrip": false,
1566
+ "normalized": false,
1567
+ "rstrip": false,
1568
+ "single_word": false,
1569
+ "special": true
1570
+ },
1571
+ "128196": {
1572
+ "content": "<|reserved_special_token_188|>",
1573
+ "lstrip": false,
1574
+ "normalized": false,
1575
+ "rstrip": false,
1576
+ "single_word": false,
1577
+ "special": true
1578
+ },
1579
+ "128197": {
1580
+ "content": "<|reserved_special_token_189|>",
1581
+ "lstrip": false,
1582
+ "normalized": false,
1583
+ "rstrip": false,
1584
+ "single_word": false,
1585
+ "special": true
1586
+ },
1587
+ "128198": {
1588
+ "content": "<|reserved_special_token_190|>",
1589
+ "lstrip": false,
1590
+ "normalized": false,
1591
+ "rstrip": false,
1592
+ "single_word": false,
1593
+ "special": true
1594
+ },
1595
+ "128199": {
1596
+ "content": "<|reserved_special_token_191|>",
1597
+ "lstrip": false,
1598
+ "normalized": false,
1599
+ "rstrip": false,
1600
+ "single_word": false,
1601
+ "special": true
1602
+ },
1603
+ "128200": {
1604
+ "content": "<|reserved_special_token_192|>",
1605
+ "lstrip": false,
1606
+ "normalized": false,
1607
+ "rstrip": false,
1608
+ "single_word": false,
1609
+ "special": true
1610
+ },
1611
+ "128201": {
1612
+ "content": "<|reserved_special_token_193|>",
1613
+ "lstrip": false,
1614
+ "normalized": false,
1615
+ "rstrip": false,
1616
+ "single_word": false,
1617
+ "special": true
1618
+ },
1619
+ "128202": {
1620
+ "content": "<|reserved_special_token_194|>",
1621
+ "lstrip": false,
1622
+ "normalized": false,
1623
+ "rstrip": false,
1624
+ "single_word": false,
1625
+ "special": true
1626
+ },
1627
+ "128203": {
1628
+ "content": "<|reserved_special_token_195|>",
1629
+ "lstrip": false,
1630
+ "normalized": false,
1631
+ "rstrip": false,
1632
+ "single_word": false,
1633
+ "special": true
1634
+ },
1635
+ "128204": {
1636
+ "content": "<|reserved_special_token_196|>",
1637
+ "lstrip": false,
1638
+ "normalized": false,
1639
+ "rstrip": false,
1640
+ "single_word": false,
1641
+ "special": true
1642
+ },
1643
+ "128205": {
1644
+ "content": "<|reserved_special_token_197|>",
1645
+ "lstrip": false,
1646
+ "normalized": false,
1647
+ "rstrip": false,
1648
+ "single_word": false,
1649
+ "special": true
1650
+ },
1651
+ "128206": {
1652
+ "content": "<|reserved_special_token_198|>",
1653
+ "lstrip": false,
1654
+ "normalized": false,
1655
+ "rstrip": false,
1656
+ "single_word": false,
1657
+ "special": true
1658
+ },
1659
+ "128207": {
1660
+ "content": "<|reserved_special_token_199|>",
1661
+ "lstrip": false,
1662
+ "normalized": false,
1663
+ "rstrip": false,
1664
+ "single_word": false,
1665
+ "special": true
1666
+ },
1667
+ "128208": {
1668
+ "content": "<|reserved_special_token_200|>",
1669
+ "lstrip": false,
1670
+ "normalized": false,
1671
+ "rstrip": false,
1672
+ "single_word": false,
1673
+ "special": true
1674
+ },
1675
+ "128209": {
1676
+ "content": "<|reserved_special_token_201|>",
1677
+ "lstrip": false,
1678
+ "normalized": false,
1679
+ "rstrip": false,
1680
+ "single_word": false,
1681
+ "special": true
1682
+ },
1683
+ "128210": {
1684
+ "content": "<|reserved_special_token_202|>",
1685
+ "lstrip": false,
1686
+ "normalized": false,
1687
+ "rstrip": false,
1688
+ "single_word": false,
1689
+ "special": true
1690
+ },
1691
+ "128211": {
1692
+ "content": "<|reserved_special_token_203|>",
1693
+ "lstrip": false,
1694
+ "normalized": false,
1695
+ "rstrip": false,
1696
+ "single_word": false,
1697
+ "special": true
1698
+ },
1699
+ "128212": {
1700
+ "content": "<|reserved_special_token_204|>",
1701
+ "lstrip": false,
1702
+ "normalized": false,
1703
+ "rstrip": false,
1704
+ "single_word": false,
1705
+ "special": true
1706
+ },
1707
+ "128213": {
1708
+ "content": "<|reserved_special_token_205|>",
1709
+ "lstrip": false,
1710
+ "normalized": false,
1711
+ "rstrip": false,
1712
+ "single_word": false,
1713
+ "special": true
1714
+ },
1715
+ "128214": {
1716
+ "content": "<|reserved_special_token_206|>",
1717
+ "lstrip": false,
1718
+ "normalized": false,
1719
+ "rstrip": false,
1720
+ "single_word": false,
1721
+ "special": true
1722
+ },
1723
+ "128215": {
1724
+ "content": "<|reserved_special_token_207|>",
1725
+ "lstrip": false,
1726
+ "normalized": false,
1727
+ "rstrip": false,
1728
+ "single_word": false,
1729
+ "special": true
1730
+ },
1731
+ "128216": {
1732
+ "content": "<|reserved_special_token_208|>",
1733
+ "lstrip": false,
1734
+ "normalized": false,
1735
+ "rstrip": false,
1736
+ "single_word": false,
1737
+ "special": true
1738
+ },
1739
+ "128217": {
1740
+ "content": "<|reserved_special_token_209|>",
1741
+ "lstrip": false,
1742
+ "normalized": false,
1743
+ "rstrip": false,
1744
+ "single_word": false,
1745
+ "special": true
1746
+ },
1747
+ "128218": {
1748
+ "content": "<|reserved_special_token_210|>",
1749
+ "lstrip": false,
1750
+ "normalized": false,
1751
+ "rstrip": false,
1752
+ "single_word": false,
1753
+ "special": true
1754
+ },
1755
+ "128219": {
1756
+ "content": "<|reserved_special_token_211|>",
1757
+ "lstrip": false,
1758
+ "normalized": false,
1759
+ "rstrip": false,
1760
+ "single_word": false,
1761
+ "special": true
1762
+ },
1763
+ "128220": {
1764
+ "content": "<|reserved_special_token_212|>",
1765
+ "lstrip": false,
1766
+ "normalized": false,
1767
+ "rstrip": false,
1768
+ "single_word": false,
1769
+ "special": true
1770
+ },
1771
+ "128221": {
1772
+ "content": "<|reserved_special_token_213|>",
1773
+ "lstrip": false,
1774
+ "normalized": false,
1775
+ "rstrip": false,
1776
+ "single_word": false,
1777
+ "special": true
1778
+ },
1779
+ "128222": {
1780
+ "content": "<|reserved_special_token_214|>",
1781
+ "lstrip": false,
1782
+ "normalized": false,
1783
+ "rstrip": false,
1784
+ "single_word": false,
1785
+ "special": true
1786
+ },
1787
+ "128223": {
1788
+ "content": "<|reserved_special_token_215|>",
1789
+ "lstrip": false,
1790
+ "normalized": false,
1791
+ "rstrip": false,
1792
+ "single_word": false,
1793
+ "special": true
1794
+ },
1795
+ "128224": {
1796
+ "content": "<|reserved_special_token_216|>",
1797
+ "lstrip": false,
1798
+ "normalized": false,
1799
+ "rstrip": false,
1800
+ "single_word": false,
1801
+ "special": true
1802
+ },
1803
+ "128225": {
1804
+ "content": "<|reserved_special_token_217|>",
1805
+ "lstrip": false,
1806
+ "normalized": false,
1807
+ "rstrip": false,
1808
+ "single_word": false,
1809
+ "special": true
1810
+ },
1811
+ "128226": {
1812
+ "content": "<|reserved_special_token_218|>",
1813
+ "lstrip": false,
1814
+ "normalized": false,
1815
+ "rstrip": false,
1816
+ "single_word": false,
1817
+ "special": true
1818
+ },
1819
+ "128227": {
1820
+ "content": "<|reserved_special_token_219|>",
1821
+ "lstrip": false,
1822
+ "normalized": false,
1823
+ "rstrip": false,
1824
+ "single_word": false,
1825
+ "special": true
1826
+ },
1827
+ "128228": {
1828
+ "content": "<|reserved_special_token_220|>",
1829
+ "lstrip": false,
1830
+ "normalized": false,
1831
+ "rstrip": false,
1832
+ "single_word": false,
1833
+ "special": true
1834
+ },
1835
+ "128229": {
1836
+ "content": "<|reserved_special_token_221|>",
1837
+ "lstrip": false,
1838
+ "normalized": false,
1839
+ "rstrip": false,
1840
+ "single_word": false,
1841
+ "special": true
1842
+ },
1843
+ "128230": {
1844
+ "content": "<|reserved_special_token_222|>",
1845
+ "lstrip": false,
1846
+ "normalized": false,
1847
+ "rstrip": false,
1848
+ "single_word": false,
1849
+ "special": true
1850
+ },
1851
+ "128231": {
1852
+ "content": "<|reserved_special_token_223|>",
1853
+ "lstrip": false,
1854
+ "normalized": false,
1855
+ "rstrip": false,
1856
+ "single_word": false,
1857
+ "special": true
1858
+ },
1859
+ "128232": {
1860
+ "content": "<|reserved_special_token_224|>",
1861
+ "lstrip": false,
1862
+ "normalized": false,
1863
+ "rstrip": false,
1864
+ "single_word": false,
1865
+ "special": true
1866
+ },
1867
+ "128233": {
1868
+ "content": "<|reserved_special_token_225|>",
1869
+ "lstrip": false,
1870
+ "normalized": false,
1871
+ "rstrip": false,
1872
+ "single_word": false,
1873
+ "special": true
1874
+ },
1875
+ "128234": {
1876
+ "content": "<|reserved_special_token_226|>",
1877
+ "lstrip": false,
1878
+ "normalized": false,
1879
+ "rstrip": false,
1880
+ "single_word": false,
1881
+ "special": true
1882
+ },
1883
+ "128235": {
1884
+ "content": "<|reserved_special_token_227|>",
1885
+ "lstrip": false,
1886
+ "normalized": false,
1887
+ "rstrip": false,
1888
+ "single_word": false,
1889
+ "special": true
1890
+ },
1891
+ "128236": {
1892
+ "content": "<|reserved_special_token_228|>",
1893
+ "lstrip": false,
1894
+ "normalized": false,
1895
+ "rstrip": false,
1896
+ "single_word": false,
1897
+ "special": true
1898
+ },
1899
+ "128237": {
1900
+ "content": "<|reserved_special_token_229|>",
1901
+ "lstrip": false,
1902
+ "normalized": false,
1903
+ "rstrip": false,
1904
+ "single_word": false,
1905
+ "special": true
1906
+ },
1907
+ "128238": {
1908
+ "content": "<|reserved_special_token_230|>",
1909
+ "lstrip": false,
1910
+ "normalized": false,
1911
+ "rstrip": false,
1912
+ "single_word": false,
1913
+ "special": true
1914
+ },
1915
+ "128239": {
1916
+ "content": "<|reserved_special_token_231|>",
1917
+ "lstrip": false,
1918
+ "normalized": false,
1919
+ "rstrip": false,
1920
+ "single_word": false,
1921
+ "special": true
1922
+ },
1923
+ "128240": {
1924
+ "content": "<|reserved_special_token_232|>",
1925
+ "lstrip": false,
1926
+ "normalized": false,
1927
+ "rstrip": false,
1928
+ "single_word": false,
1929
+ "special": true
1930
+ },
1931
+ "128241": {
1932
+ "content": "<|reserved_special_token_233|>",
1933
+ "lstrip": false,
1934
+ "normalized": false,
1935
+ "rstrip": false,
1936
+ "single_word": false,
1937
+ "special": true
1938
+ },
1939
+ "128242": {
1940
+ "content": "<|reserved_special_token_234|>",
1941
+ "lstrip": false,
1942
+ "normalized": false,
1943
+ "rstrip": false,
1944
+ "single_word": false,
1945
+ "special": true
1946
+ },
1947
+ "128243": {
1948
+ "content": "<|reserved_special_token_235|>",
1949
+ "lstrip": false,
1950
+ "normalized": false,
1951
+ "rstrip": false,
1952
+ "single_word": false,
1953
+ "special": true
1954
+ },
1955
+ "128244": {
1956
+ "content": "<|reserved_special_token_236|>",
1957
+ "lstrip": false,
1958
+ "normalized": false,
1959
+ "rstrip": false,
1960
+ "single_word": false,
1961
+ "special": true
1962
+ },
1963
+ "128245": {
1964
+ "content": "<|reserved_special_token_237|>",
1965
+ "lstrip": false,
1966
+ "normalized": false,
1967
+ "rstrip": false,
1968
+ "single_word": false,
1969
+ "special": true
1970
+ },
1971
+ "128246": {
1972
+ "content": "<|reserved_special_token_238|>",
1973
+ "lstrip": false,
1974
+ "normalized": false,
1975
+ "rstrip": false,
1976
+ "single_word": false,
1977
+ "special": true
1978
+ },
1979
+ "128247": {
1980
+ "content": "<|reserved_special_token_239|>",
1981
+ "lstrip": false,
1982
+ "normalized": false,
1983
+ "rstrip": false,
1984
+ "single_word": false,
1985
+ "special": true
1986
+ },
1987
+ "128248": {
1988
+ "content": "<|reserved_special_token_240|>",
1989
+ "lstrip": false,
1990
+ "normalized": false,
1991
+ "rstrip": false,
1992
+ "single_word": false,
1993
+ "special": true
1994
+ },
1995
+ "128249": {
1996
+ "content": "<|reserved_special_token_241|>",
1997
+ "lstrip": false,
1998
+ "normalized": false,
1999
+ "rstrip": false,
2000
+ "single_word": false,
2001
+ "special": true
2002
+ },
2003
+ "128250": {
2004
+ "content": "<|reserved_special_token_242|>",
2005
+ "lstrip": false,
2006
+ "normalized": false,
2007
+ "rstrip": false,
2008
+ "single_word": false,
2009
+ "special": true
2010
+ },
2011
+ "128251": {
2012
+ "content": "<|reserved_special_token_243|>",
2013
+ "lstrip": false,
2014
+ "normalized": false,
2015
+ "rstrip": false,
2016
+ "single_word": false,
2017
+ "special": true
2018
+ },
2019
+ "128252": {
2020
+ "content": "<|reserved_special_token_244|>",
2021
+ "lstrip": false,
2022
+ "normalized": false,
2023
+ "rstrip": false,
2024
+ "single_word": false,
2025
+ "special": true
2026
+ },
2027
+ "128253": {
2028
+ "content": "<|reserved_special_token_245|>",
2029
+ "lstrip": false,
2030
+ "normalized": false,
2031
+ "rstrip": false,
2032
+ "single_word": false,
2033
+ "special": true
2034
+ },
2035
+ "128254": {
2036
+ "content": "<|reserved_special_token_246|>",
2037
+ "lstrip": false,
2038
+ "normalized": false,
2039
+ "rstrip": false,
2040
+ "single_word": false,
2041
+ "special": true
2042
+ },
2043
+ "128255": {
2044
+ "content": "<|reserved_special_token_247|>",
2045
+ "lstrip": false,
2046
+ "normalized": false,
2047
+ "rstrip": false,
2048
+ "single_word": false,
2049
+ "special": true
2050
+ }
2051
+ },
2052
+ "bos_token": "<|begin_of_text|>",
2053
+ "chat_template": "{{- bos_token }}\n{%- if custom_tools is defined %}\n {%- set tools = custom_tools %}\n{%- endif %}\n{%- if not tools_in_user_message is defined %}\n {%- set tools_in_user_message = true %}\n{%- endif %}\n{%- if not date_string is defined %}\n {%- if strftime_now is defined %}\n {%- set date_string = strftime_now(\"%d %b %Y\") %}\n {%- else %}\n {%- set date_string = \"26 Jul 2024\" %}\n {%- endif %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n\n{#- This block extracts the system message, so we can slot it into the right place. #}\n{%- if messages[0]['role'] == 'system' %}\n {%- set system_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n{%- else %}\n {%- set system_message = \"\" %}\n{%- endif %}\n\n{#- System message #}\n{{- \"<|start_header_id|>system<|end_header_id|>\\n\\n\" }}\n{%- if tools is not none %}\n {{- \"Environment: ipython\\n\" }}\n{%- endif %}\n{{- \"Cutting Knowledge Date: December 2023\\n\" }}\n{{- \"Today Date: \" + date_string + \"\\n\\n\" }}\n{%- if tools is not none and not tools_in_user_message %}\n {{- \"You have access to the following functions. To call a function, please respond with JSON for a function call.\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n{%- endif %}\n{{- system_message }}\n{{- \"<|eot_id|>\" }}\n\n{#- Custom tools are passed in a user message with some extra guidance #}\n{%- if tools_in_user_message and not tools is none %}\n {#- Extract the first user message so we can plug it in here #}\n {%- if messages | length != 0 %}\n {%- set first_user_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n {%- else %}\n {{- raise_exception(\"Cannot put tools in the first user message when there's no first user message!\") }}\n{%- endif %}\n {{- '<|start_header_id|>user<|end_header_id|>\\n\\n' -}}\n {{- \"Given the following functions, please respond with a JSON for a function call \" }}\n {{- \"with its proper arguments that best answers the given prompt.\\n\\n\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n {{- first_user_message + \"<|eot_id|>\"}}\n{%- endif %}\n\n{%- for message in messages %}\n {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}\n {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\\n\\n'+ message['content'] | trim + '<|eot_id|>' }}\n {%- elif 'tool_calls' in message %}\n {%- if not message.tool_calls|length == 1 %}\n {{- raise_exception(\"This model only supports single tool-calls at once!\") }}\n {%- endif %}\n {%- set tool_call = message.tool_calls[0].function %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- '{\"name\": \"' + tool_call.name + '\", ' }}\n {{- '\"parameters\": ' }}\n {{- tool_call.arguments | tojson }}\n {{- \"}\" }}\n {{- \"<|eot_id|>\" }}\n {%- elif message.role == \"tool\" or message.role == \"ipython\" %}\n {{- \"<|start_header_id|>ipython<|end_header_id|>\\n\\n\" }}\n {%- if message.content is mapping or message.content is iterable %}\n {{- message.content | tojson }}\n {%- else %}\n {{- message.content }}\n {%- endif %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }}\n{%- endif %}\n",
2054
+ "clean_up_tokenization_spaces": true,
2055
+ "eos_token": "<|eot_id|>",
2056
+ "is_fast": true,
2057
+ "model_input_names": [
2058
+ "input_ids",
2059
+ "attention_mask"
2060
+ ],
2061
+ "model_max_length": 131072,
2062
+ "pad_token": "<|eot_id|>",
2063
+ "tokenizer_class": "PreTrainedTokenizerFast"
2064
+ }
tool-versions.yaml ADDED
@@ -0,0 +1 @@
 
 
1
+ qairt: 2.42.0.251225135753_193295