Hugging Face's logo

melihcatal
/
codedp-cpt-models-v2

PEFT
Safetensors
code-generation
differential-privacy
continued-pretraining
lora
Model card Files
xet
 Community
melihcatal commited on
Commit
080080b
·
verified ·
1 Parent(s): 0b02ed9

Add files using upload-large-folder tool

Browse files
Files changed (50) hide show
  1. qwen1.5-moe-a2.7b/base_attn_v2/adapter/README.md +207 -0
  2. qwen1.5-moe-a2.7b/base_attn_v2/adapter/adapter_config.json +46 -0
  3. qwen1.5-moe-a2.7b/base_attn_v2/audit_results.json +137 -0
  4. qwen1.5-moe-a2.7b/base_attn_v2/canary_meta.json +0 -0
  5. qwen1.5-moe-a2.7b/base_attn_v2/metrics.jsonl +0 -0
  6. qwen1.5-moe-a2.7b/base_attn_v2/resolved_config.yaml +111 -0
  7. qwen1.5-moe-a2.7b/base_attn_v2/summary.json +71 -0
  8. qwen1.5-moe-a2.7b/base_attn_v2/tokenizer/added_tokens.json +505 -0
  9. qwen1.5-moe-a2.7b/base_attn_v2/tokenizer/chat_template.jinja +6 -0
  10. qwen1.5-moe-a2.7b/base_attn_v2/tokenizer/merges.txt +0 -0
  11. qwen1.5-moe-a2.7b/base_attn_v2/tokenizer/special_tokens_map.json +3518 -0
  12. qwen1.5-moe-a2.7b/base_attn_v2/tokenizer/tokenizer_config.json +0 -0
  13. qwen1.5-moe-a2.7b/base_attn_v2/tokenizer/vocab.json +0 -0
  14. qwen1.5-moe-a2.7b/base_attn_v2/train.log +276 -0
  15. qwen1.5-moe-a2.7b/dp3_attn/adapter/README.md +207 -0
  16. qwen1.5-moe-a2.7b/dp3_attn/adapter/adapter_config.json +46 -0
  17. qwen1.5-moe-a2.7b/dp3_attn/audit_results.json +137 -0
  18. qwen1.5-moe-a2.7b/dp3_attn/canary_meta.json +0 -0
  19. qwen1.5-moe-a2.7b/dp3_attn/metrics.jsonl +0 -0
  20. qwen1.5-moe-a2.7b/dp3_attn/resolved_config.yaml +110 -0
  21. qwen1.5-moe-a2.7b/dp3_attn/summary.json +72 -0
  22. qwen1.5-moe-a2.7b/dp3_attn/tokenizer/chat_template.jinja +6 -0
  23. qwen1.5-moe-a2.7b/dp3_attn/tokenizer/tokenizer_config.json +516 -0
  24. qwen1.5-moe-a2.7b/dp3_attn/train.log +197 -0
  25. qwen1.5-moe-a2.7b/dp3_attn_v2/adapter/README.md +207 -0
  26. qwen1.5-moe-a2.7b/dp3_attn_v2/audit_results.json +137 -0
  27. qwen1.5-moe-a2.7b/dp3_attn_v2/canary_meta.json +0 -0
  28. qwen1.5-moe-a2.7b/dp3_attn_v2/metrics.jsonl +0 -0
  29. qwen1.5-moe-a2.7b/dp3_attn_v2/resolved_config.yaml +111 -0
  30. qwen1.5-moe-a2.7b/dp3_attn_v2/summary.json +72 -0
  31. qwen1.5-moe-a2.7b/dp3_attn_v2/tokenizer/added_tokens.json +505 -0
  32. qwen1.5-moe-a2.7b/dp3_attn_v2/tokenizer/chat_template.jinja +6 -0
  33. qwen1.5-moe-a2.7b/dp3_attn_v2/tokenizer/special_tokens_map.json +3518 -0
  34. qwen1.5-moe-a2.7b/dp3_attn_v2/tokenizer/tokenizer_config.json +0 -0
  35. qwen1.5-moe-a2.7b/dp3_attn_v2/tokenizer/vocab.json +0 -0
  36. qwen1.5-moe-a2.7b/dp3_attn_v2/train.log +277 -0
  37. qwen1.5-moe-a2.7b/dp8_attn_v2/adapter/README.md +207 -0
  38. qwen1.5-moe-a2.7b/dp8_attn_v2/adapter/adapter_config.json +46 -0
  39. qwen1.5-moe-a2.7b/dp8_attn_v2/audit_results.json +137 -0
  40. qwen1.5-moe-a2.7b/dp8_attn_v2/canary_meta.json +0 -0
  41. qwen1.5-moe-a2.7b/dp8_attn_v2/metrics.jsonl +0 -0
  42. qwen1.5-moe-a2.7b/dp8_attn_v2/resolved_config.yaml +111 -0
  43. qwen1.5-moe-a2.7b/dp8_attn_v2/summary.json +72 -0
  44. qwen1.5-moe-a2.7b/dp8_attn_v2/tokenizer/added_tokens.json +505 -0
  45. qwen1.5-moe-a2.7b/dp8_attn_v2/tokenizer/chat_template.jinja +6 -0
  46. qwen1.5-moe-a2.7b/dp8_attn_v2/tokenizer/merges.txt +0 -0
  47. qwen1.5-moe-a2.7b/dp8_attn_v2/tokenizer/special_tokens_map.json +3518 -0
  48. qwen1.5-moe-a2.7b/dp8_attn_v2/tokenizer/tokenizer_config.json +0 -0
  49. qwen1.5-moe-a2.7b/dp8_attn_v2/tokenizer/vocab.json +0 -0
  50. qwen1.5-moe-a2.7b/dp8_attn_v2/train.log +277 -0
qwen1.5-moe-a2.7b/base_attn_v2/adapter/README.md ADDED
@@ -0,0 +1,207 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: Qwen/Qwen1.5-MoE-A2.7B
3
+ library_name: peft
4
+ pipeline_tag: text-generation
5
+ tags:
6
+ - base_model:adapter:Qwen/Qwen1.5-MoE-A2.7B
7
+ - lora
8
+ - transformers
9
+ ---
10
+
11
+ # Model Card for Model ID
12
+
13
+ <!-- Provide a quick summary of what the model is/does. -->
14
+
15
+
16
+
17
+ ## Model Details
18
+
19
+ ### Model Description
20
+
21
+ <!-- Provide a longer summary of what this model is. -->
22
+
23
+
24
+
25
+ - **Developed by:** [More Information Needed]
26
+ - **Funded by [optional]:** [More Information Needed]
27
+ - **Shared by [optional]:** [More Information Needed]
28
+ - **Model type:** [More Information Needed]
29
+ - **Language(s) (NLP):** [More Information Needed]
30
+ - **License:** [More Information Needed]
31
+ - **Finetuned from model [optional]:** [More Information Needed]
32
+
33
+ ### Model Sources [optional]
34
+
35
+ <!-- Provide the basic links for the model. -->
36
+
37
+ - **Repository:** [More Information Needed]
38
+ - **Paper [optional]:** [More Information Needed]
39
+ - **Demo [optional]:** [More Information Needed]
40
+
41
+ ## Uses
42
+
43
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
44
+
45
+ ### Direct Use
46
+
47
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
48
+
49
+ [More Information Needed]
50
+
51
+ ### Downstream Use [optional]
52
+
53
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
54
+
55
+ [More Information Needed]
56
+
57
+ ### Out-of-Scope Use
58
+
59
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
60
+
61
+ [More Information Needed]
62
+
63
+ ## Bias, Risks, and Limitations
64
+
65
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
66
+
67
+ [More Information Needed]
68
+
69
+ ### Recommendations
70
+
71
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
72
+
73
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
74
+
75
+ ## How to Get Started with the Model
76
+
77
+ Use the code below to get started with the model.
78
+
79
+ [More Information Needed]
80
+
81
+ ## Training Details
82
+
83
+ ### Training Data
84
+
85
+ <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
86
+
87
+ [More Information Needed]
88
+
89
+ ### Training Procedure
90
+
91
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
92
+
93
+ #### Preprocessing [optional]
94
+
95
+ [More Information Needed]
96
+
97
+
98
+ #### Training Hyperparameters
99
+
100
+ - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
101
+
102
+ #### Speeds, Sizes, Times [optional]
103
+
104
+ <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
105
+
106
+ [More Information Needed]
107
+
108
+ ## Evaluation
109
+
110
+ <!-- This section describes the evaluation protocols and provides the results. -->
111
+
112
+ ### Testing Data, Factors & Metrics
113
+
114
+ #### Testing Data
115
+
116
+ <!-- This should link to a Dataset Card if possible. -->
117
+
118
+ [More Information Needed]
119
+
120
+ #### Factors
121
+
122
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
123
+
124
+ [More Information Needed]
125
+
126
+ #### Metrics
127
+
128
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
129
+
130
+ [More Information Needed]
131
+
132
+ ### Results
133
+
134
+ [More Information Needed]
135
+
136
+ #### Summary
137
+
138
+
139
+
140
+ ## Model Examination [optional]
141
+
142
+ <!-- Relevant interpretability work for the model goes here -->
143
+
144
+ [More Information Needed]
145
+
146
+ ## Environmental Impact
147
+
148
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
149
+
150
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
151
+
152
+ - **Hardware Type:** [More Information Needed]
153
+ - **Hours used:** [More Information Needed]
154
+ - **Cloud Provider:** [More Information Needed]
155
+ - **Compute Region:** [More Information Needed]
156
+ - **Carbon Emitted:** [More Information Needed]
157
+
158
+ ## Technical Specifications [optional]
159
+
160
+ ### Model Architecture and Objective
161
+
162
+ [More Information Needed]
163
+
164
+ ### Compute Infrastructure
165
+
166
+ [More Information Needed]
167
+
168
+ #### Hardware
169
+
170
+ [More Information Needed]
171
+
172
+ #### Software
173
+
174
+ [More Information Needed]
175
+
176
+ ## Citation [optional]
177
+
178
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
179
+
180
+ **BibTeX:**
181
+
182
+ [More Information Needed]
183
+
184
+ **APA:**
185
+
186
+ [More Information Needed]
187
+
188
+ ## Glossary [optional]
189
+
190
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
191
+
192
+ [More Information Needed]
193
+
194
+ ## More Information [optional]
195
+
196
+ [More Information Needed]
197
+
198
+ ## Model Card Authors [optional]
199
+
200
+ [More Information Needed]
201
+
202
+ ## Model Card Contact
203
+
204
+ [More Information Needed]
205
+ ### Framework versions
206
+
207
+ - PEFT 0.18.1
qwen1.5-moe-a2.7b/base_attn_v2/adapter/adapter_config.json ADDED
@@ -0,0 +1,46 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alora_invocation_tokens": null,
3
+ "alpha_pattern": {},
4
+ "arrow_config": null,
5
+ "auto_mapping": null,
6
+ "base_model_name_or_path": "Qwen/Qwen1.5-MoE-A2.7B",
7
+ "bias": "none",
8
+ "corda_config": null,
9
+ "ensure_weight_tying": true,
10
+ "eva_config": null,
11
+ "exclude_modules": null,
12
+ "fan_in_fan_out": false,
13
+ "inference_mode": true,
14
+ "init_lora_weights": true,
15
+ "layer_replication": null,
16
+ "layers_pattern": null,
17
+ "layers_to_transform": null,
18
+ "loftq_config": {},
19
+ "lora_alpha": 32,
20
+ "lora_bias": false,
21
+ "lora_dropout": 0.05,
22
+ "megatron_config": null,
23
+ "megatron_core": "megatron.core",
24
+ "modules_to_save": [
25
+ "lm_head",
26
+ "embed_tokens"
27
+ ],
28
+ "peft_type": "LORA",
29
+ "peft_version": "0.18.1",
30
+ "qalora_group_size": 16,
31
+ "r": 16,
32
+ "rank_pattern": {},
33
+ "revision": null,
34
+ "target_modules": [
35
+ "q_proj",
36
+ "o_proj",
37
+ "k_proj",
38
+ "v_proj"
39
+ ],
40
+ "target_parameters": null,
41
+ "task_type": "CAUSAL_LM",
42
+ "trainable_token_indices": null,
43
+ "use_dora": false,
44
+ "use_qalora": false,
45
+ "use_rslora": false
46
+ }
qwen1.5-moe-a2.7b/base_attn_v2/audit_results.json ADDED
@@ -0,0 +1,137 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "delta": 1e-05,
3
+ "num_canaries": 500,
4
+ "num_members": 250,
5
+ "paper_guess_fraction": 0.2,
6
+ "paper_guess_steps": 20,
7
+ "loss": {
8
+ "auc": 1.0,
9
+ "empirical_epsilon": {
10
+ "0.05": 3.4791953936219215,
11
+ "0.01": 3.023197554051876
12
+ },
13
+ "empirical_epsilon_details": {
14
+ "0.05": {
15
+ "epsilon": 3.4791953936219215,
16
+ "num_guesses": 100,
17
+ "correct_guesses": 100,
18
+ "candidate_num_guesses": [
19
+ 5,
20
+ 10,
21
+ 15,
22
+ 20,
23
+ 25,
24
+ 30,
25
+ 35,
26
+ 40,
27
+ 45,
28
+ 50,
29
+ 55,
30
+ 60,
31
+ 65,
32
+ 70,
33
+ 75,
34
+ 80,
35
+ 85,
36
+ 90,
37
+ 95,
38
+ 100
39
+ ],
40
+ "direction": "lower"
41
+ },
42
+ "0.01": {
43
+ "epsilon": 3.023197554051876,
44
+ "num_guesses": 100,
45
+ "correct_guesses": 100,
46
+ "candidate_num_guesses": [
47
+ 5,
48
+ 10,
49
+ 15,
50
+ 20,
51
+ 25,
52
+ 30,
53
+ 35,
54
+ 40,
55
+ 45,
56
+ 50,
57
+ 55,
58
+ 60,
59
+ 65,
60
+ 70,
61
+ 75,
62
+ 80,
63
+ 85,
64
+ 90,
65
+ 95,
66
+ 100
67
+ ],
68
+ "direction": "lower"
69
+ }
70
+ }
71
+ },
72
+ "embedding": {
73
+ "auc": 0.612,
74
+ "empirical_epsilon": {
75
+ "0.05": 3.4791953936219215,
76
+ "0.01": 3.023197554051876
77
+ },
78
+ "empirical_epsilon_details": {
79
+ "0.05": {
80
+ "epsilon": 3.4791953936219215,
81
+ "num_guesses": 100,
82
+ "correct_guesses": 100,
83
+ "candidate_num_guesses": [
84
+ 5,
85
+ 10,
86
+ 15,
87
+ 20,
88
+ 25,
89
+ 30,
90
+ 35,
91
+ 40,
92
+ 45,
93
+ 50,
94
+ 55,
95
+ 60,
96
+ 65,
97
+ 70,
98
+ 75,
99
+ 80,
100
+ 85,
101
+ 90,
102
+ 95,
103
+ 100
104
+ ],
105
+ "direction": "higher"
106
+ },
107
+ "0.01": {
108
+ "epsilon": 3.023197554051876,
109
+ "num_guesses": 100,
110
+ "correct_guesses": 100,
111
+ "candidate_num_guesses": [
112
+ 5,
113
+ 10,
114
+ 15,
115
+ 20,
116
+ 25,
117
+ 30,
118
+ 35,
119
+ 40,
120
+ 45,
121
+ 50,
122
+ 55,
123
+ 60,
124
+ 65,
125
+ 70,
126
+ 75,
127
+ 80,
128
+ 85,
129
+ 90,
130
+ 95,
131
+ 100
132
+ ],
133
+ "direction": "higher"
134
+ }
135
+ }
136
+ }
137
+ }
qwen1.5-moe-a2.7b/base_attn_v2/canary_meta.json ADDED
The diff for this file is too large to render. See raw diff
 
qwen1.5-moe-a2.7b/base_attn_v2/metrics.jsonl ADDED
The diff for this file is too large to render. See raw diff
 
qwen1.5-moe-a2.7b/base_attn_v2/resolved_config.yaml ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ model:
2
+ name: Qwen/Qwen1.5-MoE-A2.7B
3
+ tokenizer_name: Qwen/Qwen1.5-MoE-A2.7B
4
+ max_length: 1024
5
+ dtype: bfloat16
6
+ trust_remote_code: true
7
+ use_fast_tokenizer: true
8
+ cache_dir: null
9
+ local_files_only: false
10
+ low_cpu_mem_usage: true
11
+ tie_word_embeddings: true
12
+ gradient_checkpointing: false
13
+ use_chat_template: false
14
+ dataset:
15
+ name: melihcatal/codedp-cpt
16
+ split: train
17
+ mode: cpt
18
+ text_column: text
19
+ validation_ratio: 0.05
20
+ max_samples: -1
21
+ lora:
22
+ enabled: true
23
+ r: 16
24
+ alpha: 32
25
+ dropout: 0.05
26
+ target_modules:
27
+ - q_proj
28
+ - k_proj
29
+ - v_proj
30
+ - o_proj
31
+ modules_to_save:
32
+ - lm_head
33
+ bias: none
34
+ training:
35
+ seed: 42
36
+ epochs: 5
37
+ warmup_steps: null
38
+ warmup_ratio: 0.05
39
+ mixed_precision: false
40
+ mixed_precision_dtype: bfloat16
41
+ batch_size: 4
42
+ eval_batch_size: 8
43
+ eval_every_steps: 10
44
+ eval_every_epochs: 1
45
+ learning_rate: 0.0005
46
+ optimizer: adamw
47
+ lr_scheduler: cosine
48
+ adam_beta1: 0.9
49
+ adam_beta2: 0.999
50
+ adam_epsilon: 1.0e-08
51
+ sgd_momentum: 0.9
52
+ weight_decay: 0.01
53
+ max_grad_norm: 1.0
54
+ log_every: 5
55
+ gradient_accumulation_steps: 8
56
+ num_workers: 4
57
+ output_dir: /scratch/mcatal/runs/cpt/qwen1.5-moe-a2.7b/base_attn_v2
58
+ min_lr_ratio: 0.15
59
+ distributed:
60
+ strategy: dpddp
61
+ backend: nccl
62
+ devices: null
63
+ dp:
64
+ module_validator: auto
65
+ target_delta: 1.0e-05
66
+ noise_multiplier: null
67
+ max_grad_norm: 1.0
68
+ grad_sample_mode: hooks
69
+ clipping: flat
70
+ secure_mode: false
71
+ enabled: false
72
+ target_epsilon: 8.0
73
+ audit:
74
+ enabled: true
75
+ run_every_epoch: true
76
+ epoch_device: cuda
77
+ q_canary: auto
78
+ num_canaries: 500
79
+ prefix_length: 49
80
+ num_digits: 12
81
+ batch_size: 32
82
+ delta: 1.0e-05
83
+ p_values:
84
+ - 0.05
85
+ - 0.01
86
+ paper_guess_fraction: 0.2
87
+ paper_guess_steps: 20
88
+ enable_holdout_empirical_epsilon: false
89
+ holdout_seed: 42
90
+ tie_seed: 42
91
+ tracking:
92
+ enabled: true
93
+ tensorboard: true
94
+ wandb: false
95
+ wandb_project: codedp-finetune-h200-audit
96
+ wandb_run_name: qwen1.5-moe-a2.7b-cpt-base_attn
97
+ wandb_mode: online
98
+ codecarbon: true
99
+ codecarbon_output_file: codecarbon.csv
100
+ codecarbon_measure_power_secs: 15
101
+ codecarbon_country_iso_code: null
102
+ codecarbon_project_name: codedp-qwen1.5-moe-a2.7b-cpt-base_attn
103
+ moe:
104
+ output_router_logits: false
105
+ router_aux_loss_coef: 0.0
106
+ freeze_router: true
107
+ profile:
108
+ enabled: false
109
+ num_batches: 8
110
+ top_experts: 8
111
+ output_file: moe_expert_profile.json
qwen1.5-moe-a2.7b/base_attn_v2/summary.json ADDED
@@ -0,0 +1,71 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "audit/delta": 1e-05,
3
+ "audit/embedding/auc": 0.612,
4
+ "audit/embedding/empirical_epsilon/0.01": 3.023197554051876,
5
+ "audit/embedding/empirical_epsilon/0.05": 3.4791953936219215,
6
+ "audit/embedding/empirical_epsilon_details/0.01/correct_guesses": 100.0,
7
+ "audit/embedding/empirical_epsilon_details/0.01/epsilon": 3.023197554051876,
8
+ "audit/embedding/empirical_epsilon_details/0.01/num_guesses": 100.0,
9
+ "audit/embedding/empirical_epsilon_details/0.05/correct_guesses": 100.0,
10
+ "audit/embedding/empirical_epsilon_details/0.05/epsilon": 3.4791953936219215,
11
+ "audit/embedding/empirical_epsilon_details/0.05/num_guesses": 100.0,
12
+ "audit/loss/auc": 1.0,
13
+ "audit/loss/empirical_epsilon/0.01": 3.023197554051876,
14
+ "audit/loss/empirical_epsilon/0.05": 3.4791953936219215,
15
+ "audit/loss/empirical_epsilon_details/0.01/correct_guesses": 100.0,
16
+ "audit/loss/empirical_epsilon_details/0.01/epsilon": 3.023197554051876,
17
+ "audit/loss/empirical_epsilon_details/0.01/num_guesses": 100.0,
18
+ "audit/loss/empirical_epsilon_details/0.05/correct_guesses": 100.0,
19
+ "audit/loss/empirical_epsilon_details/0.05/epsilon": 3.4791953936219215,
20
+ "audit/loss/empirical_epsilon_details/0.05/num_guesses": 100.0,
21
+ "audit/num_canaries": 500.0,
22
+ "audit/num_members": 250.0,
23
+ "audit/paper_guess_fraction": 0.2,
24
+ "audit/paper_guess_steps": 20.0,
25
+ "energy/codecarbon/cpu_count": 32.0,
26
+ "energy/codecarbon/cpu_energy": 0.339297275424218,
27
+ "energy/codecarbon/cpu_power": 84.18862812551733,
28
+ "energy/codecarbon/cpu_utilization_percent": 12.456958556149733,
29
+ "energy/codecarbon/duration": 15062.88998449099,
30
+ "energy/codecarbon/emissions": 0.27413221652775094,
31
+ "energy/codecarbon/emissions_rate": 1.819917803356475e-05,
32
+ "energy/codecarbon/energy_consumed": 7.867641033428546,
33
+ "energy/codecarbon/gpu_count": 8.0,
34
+ "energy/codecarbon/gpu_energy": 7.375202957935425,
35
+ "energy/codecarbon/gpu_power": 1767.6150471589203,
36
+ "energy/codecarbon/gpu_utilization_percent": 75.93257854278075,
37
+ "energy/codecarbon/latitude": 47.4843,
38
+ "energy/codecarbon/longitude": 8.212,
39
+ "energy/codecarbon/pue": 1.0,
40
+ "energy/codecarbon/ram_energy": 0.15314080006889896,
41
+ "energy/codecarbon/ram_power": 38.0,
42
+ "energy/codecarbon/ram_total_size": 256.0,
43
+ "energy/codecarbon/ram_used_gb": 375.8308208100936,
44
+ "energy/codecarbon/ram_utilization_percent": 19.076691176470586,
45
+ "energy/codecarbon/water_consumed": 0.0,
46
+ "energy/codecarbon/wue": 0.0,
47
+ "eval/duration_sec": 35.181572881992906,
48
+ "eval/loss": 0.9080336100589006,
49
+ "perf/audit_duration_sec": 10.8662999770022,
50
+ "perf/epoch_duration_sec": 2979.168524239998,
51
+ "perf/epoch_samples": 47512.0,
52
+ "perf/epoch_samples_per_sec": 15.948073972123,
53
+ "perf/epoch_tokens": 37522442.0,
54
+ "perf/epoch_tokens_per_sec": 12594.937713223919,
55
+ "perf/gradient_accumulation_steps": 8.0,
56
+ "perf/logical_batch_size": 32.0,
57
+ "perf/logical_token_count": 25640.0,
58
+ "perf/samples_per_sec": 2.534460894899331,
59
+ "perf/step_duration_sec": 12.625959258002695,
60
+ "perf/tokens_per_sec": 2030.7367920380889,
61
+ "system/cuda_epoch_peak_memory_gb": 64.22979497909546,
62
+ "system/cuda_max_memory_allocated_gb": 64.22979497909546,
63
+ "system/cuda_memory_allocated_gb": 32.04767894744873,
64
+ "train/epoch_canary_loss": 0.07924876655465145,
65
+ "train/epoch_loss": 0.5585743008840773,
66
+ "train/epoch_real_loss": 0.5626902833793556,
67
+ "train/lr": 7.5e-05,
68
+ "train/step_canary_loss": 0.0,
69
+ "train/step_loss": 0.5563923418521881,
70
+ "train/step_real_loss": 0.5563923418521881
71
+ }
qwen1.5-moe-a2.7b/base_attn_v2/tokenizer/added_tokens.json ADDED
@@ -0,0 +1,505 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "114848338811": 151654,
3
+ "116427191984": 152008,
4
+ "116639746413": 151980,
5
+ "117298239631": 151870,
6
+ "117657531962": 151927,
7
+ "117783274348": 151944,
8
+ "119342152414": 152017,
9
+ "121557291859": 151876,
10
+ "123741461297": 151785,
11
+ "124589118494": 151781,
12
+ "125866399464": 151890,
13
+ "126574388585": 151832,
14
+ "129693547287": 151766,
15
+ "133373659361": 151961,
16
+ "135441121756": 151649,
17
+ "136931728327": 151751,
18
+ "138154781148": 152061,
19
+ "138218735276": 151945,
20
+ "141499982867": 152039,
21
+ "142526557681": 152139,
22
+ "142799718159": 152003,
23
+ "143182978129": 151965,
24
+ "143777992227": 151660,
25
+ "144373897488": 152071,
26
+ "145669731981": 151875,
27
+ "147444452794": 152124,
28
+ "147541537162": 152069,
29
+ "151859598873": 151841,
30
+ "152676556624": 151976,
31
+ "154962668224": 151990,
32
+ "158142752582": 151871,
33
+ "158929292238": 151881,
34
+ "159337222655": 151716,
35
+ "161238482259": 152084,
36
+ "161588262349": 151731,
37
+ "163165456396": 152082,
38
+ "163497196769": 152114,
39
+ "165429152138": 152011,
40
+ "172599315861": 151665,
41
+ "173781362446": 151717,
42
+ "175469284871": 151809,
43
+ "175528632226": 152127,
44
+ "178478368922": 152056,
45
+ "179386156569": 152098,
46
+ "179564995814": 151991,
47
+ "181233596124": 152076,
48
+ "181439694557": 151692,
49
+ "182345464792": 151736,
50
+ "182782912224": 151918,
51
+ "183368827562": 151845,
52
+ "185438575221": 151831,
53
+ "186198784585": 151892,
54
+ "186872457894": 152129,
55
+ "191887635211": 152015,
56
+ "195388849133": 152086,
57
+ "195792996468": 151840,
58
+ "196119929397": 151905,
59
+ "196735786156": 151787,
60
+ "198462689911": 151722,
61
+ "211411365397": 151671,
62
+ "211662639865": 152000,
63
+ "212985874612": 152014,
64
+ "214193645583": 152096,
65
+ "215671314196": 151896,
66
+ "215826764872": 152050,
67
+ "216667245274": 151778,
68
+ "217756494954": 151942,
69
+ "218391991184": 151973,
70
+ "218784516525": 152001,
71
+ "221762325616": 151972,
72
+ "225164373623": 151733,
73
+ "228358422847": 151963,
74
+ "228565148417": 152020,
75
+ "229775525672": 152087,
76
+ "231853711778": 151810,
77
+ "232776999554": 151737,
78
+ "232787996724": 151895,
79
+ "232896526252": 151783,
80
+ "233795848681": 151801,
81
+ "234316622259": 151919,
82
+ "242851284913": 151729,
83
+ "243165586174": 152078,
84
+ "244371196654": 151909,
85
+ "244885399387": 151827,
86
+ "245416776926": 151877,
87
+ "251789649997": 151907,
88
+ "252867443568": 151872,
89
+ "256497188281": 152029,
90
+ "259533577263": 151752,
91
+ "259666844669": 151728,
92
+ "262296912232": 151775,
93
+ "262437452884": 152088,
94
+ "264259917554": 151707,
95
+ "265847332563": 152112,
96
+ "267514771964": 151868,
97
+ "267954879268": 151940,
98
+ "268433315926": 152116,
99
+ "268921994538": 152009,
100
+ "272362973463": 151700,
101
+ "275597777299": 151859,
102
+ "281673864127": 151702,
103
+ "282322838685": 151667,
104
+ "284445431422": 151958,
105
+ "284776712475": 152074,
106
+ "286173241916": 151897,
107
+ "291762694874": 151964,
108
+ "293847485454": 152054,
109
+ "295118729589": 151903,
110
+ "295448593321": 151784,
111
+ "296953381169": 151773,
112
+ "298464134954": 151777,
113
+ "311632789559": 151685,
114
+ "313145133463": 152067,
115
+ "313313831273": 152109,
116
+ "314697386682": 151743,
117
+ "315534574435": 152048,
118
+ "319584985416": 151864,
119
+ "322589379462": 151974,
120
+ "322683334381": 151962,
121
+ "325196781219": 152138,
122
+ "327158398268": 151967,
123
+ "327491773134": 151681,
124
+ "328552864359": 151999,
125
+ "331369926711": 152025,
126
+ "332234613346": 152137,
127
+ "332522971941": 151977,
128
+ "334513529294": 151956,
129
+ "336813343298": 152041,
130
+ "337326533813": 151738,
131
+ "339882449689": 151891,
132
+ "342221452223": 152036,
133
+ "345271433112": 151951,
134
+ "349249721685": 152045,
135
+ "351885857332": 151720,
136
+ "353755529131": 152047,
137
+ "355977139358": 151673,
138
+ "356722692178": 152140,
139
+ "357299947518": 152126,
140
+ "357621613582": 152032,
141
+ "363214487524": 151826,
142
+ "364196983365": 151839,
143
+ "364492359246": 151989,
144
+ "364947325679": 151912,
145
+ "367459894796": 151650,
146
+ "367968398551": 152051,
147
+ "368763327262": 152093,
148
+ "371496153876": 152110,
149
+ "374327157578": 151684,
150
+ "374439448821": 151762,
151
+ "374598961236": 151806,
152
+ "375429755114": 152075,
153
+ "377273562477": 151774,
154
+ "377338713663": 151788,
155
+ "378749551722": 152083,
156
+ "379557332953": 151745,
157
+ "379571934299": 151906,
158
+ "383837596997": 151675,
159
+ "384598329253": 151910,
160
+ "387763951128": 152024,
161
+ "389715484387": 152053,
162
+ "393741199486": 151998,
163
+ "396521188476": 151959,
164
+ "396847976144": 152079,
165
+ "398549545767": 151796,
166
+ "399223169721": 151842,
167
+ "399749397872": 151765,
168
+ "412843856172": 152097,
169
+ "415956914763": 151923,
170
+ "417668946583": 152031,
171
+ "418412875665": 151992,
172
+ "421369185549": 151920,
173
+ "421435255895": 151960,
174
+ "425949483533": 151662,
175
+ "429781429464": 151747,
176
+ "431149748713": 151888,
177
+ "431751228368": 151828,
178
+ "433581868192": 151829,
179
+ "434632855397": 151921,
180
+ "438124184139": 152106,
181
+ "438435178228": 152033,
182
+ "438745114173": 151984,
183
+ "441373944925": 151757,
184
+ "441377892146": 152089,
185
+ "445925149649": 151658,
186
+ "447445629421": 151690,
187
+ "449318681694": 152141,
188
+ "449549215429": 152113,
189
+ "451161679736": 151970,
190
+ "451885565366": 152090,
191
+ "454164859837": 151950,
192
+ "454244839926": 151797,
193
+ "455423278865": 151997,
194
+ "457919547633": 151652,
195
+ "462294118988": 151952,
196
+ "467893531755": 151759,
197
+ "472129283475": 152043,
198
+ "473822473819": 151691,
199
+ "475794684356": 151850,
200
+ "476969195485": 152066,
201
+ "479227349997": 151750,
202
+ "479676316326": 151703,
203
+ "481884157827": 151710,
204
+ "482829231639": 152122,
205
+ "484489193824": 151823,
206
+ "484773717614": 151791,
207
+ "485177821815": 151648,
208
+ "485692541169": 152034,
209
+ "486391569221": 151830,
210
+ "488496781997": 152120,
211
+ "488538864842": 151753,
212
+ "489562189822": 151824,
213
+ "489991411364": 151755,
214
+ "492373714791": 151884,
215
+ "492516914298": 152063,
216
+ "497294727353": 151954,
217
+ "499148455254": 151756,
218
+ "499156317423": 151948,
219
+ "499444466686": 151742,
220
+ "511785788222": 151953,
221
+ "511923256573": 151966,
222
+ "512659818733": 151885,
223
+ "513349618943": 151979,
224
+ "513361312532": 151814,
225
+ "514354144759": 152044,
226
+ "514532995959": 151730,
227
+ "515856611931": 151949,
228
+ "516968424863": 151770,
229
+ "517379856925": 151744,
230
+ "517778845323": 151725,
231
+ "522515164754": 152072,
232
+ "523723475786": 152049,
233
+ "523918658846": 151686,
234
+ "523937669294": 152010,
235
+ "524444658383": 151943,
236
+ "527117488925": 151760,
237
+ "527874729936": 152022,
238
+ "529455466433": 151904,
239
+ "529679678956": 151714,
240
+ "529854679937": 151834,
241
+ "532833541879": 151936,
242
+ "533417736566": 151846,
243
+ "535787224917": 151719,
244
+ "538631466654": 151971,
245
+ "545512344383": 151741,
246
+ "546276298359": 152046,
247
+ "547149832394": 151776,
248
+ "547563892268": 152107,
249
+ "549946382131": 151957,
250
+ "552738847828": 152130,
251
+ "556814553861": 152121,
252
+ "556976448182": 151899,
253
+ "557163366873": 152070,
254
+ "557533689478": 151821,
255
+ "558837194455": 152081,
256
+ "562389434565": 152028,
257
+ "562716493614": 152026,
258
+ "563117729724": 151804,
259
+ "563365646341": 151865,
260
+ "564395222747": 151969,
261
+ "564544131355": 151908,
262
+ "569765693871": 151647,
263
+ "569853653352": 152052,
264
+ "571317457151": 152006,
265
+ "572154333646": 151780,
266
+ "578827344666": 151721,
267
+ "579184949249": 151712,
268
+ "581493616981": 152057,
269
+ "581976444672": 151882,
270
+ "582471228164": 151819,
271
+ "583527834464": 152119,
272
+ "585544633356": 151818,
273
+ "586855179568": 151790,
274
+ "587541149322": 152111,
275
+ "587787496812": 151711,
276
+ "589439538674": 152058,
277
+ "589547936288": 151825,
278
+ "591464355756": 152013,
279
+ "592538279337": 151693,
280
+ "594688742928": 152042,
281
+ "598916996413": 151924,
282
+ "599961512792": 151900,
283
+ "612128769512": 151683,
284
+ "612788593128": 151874,
285
+ "614583493135": 152135,
286
+ "614797267726": 151746,
287
+ "616248973917": 151836,
288
+ "618423867817": 152019,
289
+ "618878266619": 152123,
290
+ "619454958782": 151887,
291
+ "622553389126": 151987,
292
+ "624221476921": 151889,
293
+ "624266421831": 151995,
294
+ "625334487352": 152101,
295
+ "626748382482": 152131,
296
+ "632962475133": 151822,
297
+ "635127952696": 152007,
298
+ "635185846287": 151981,
299
+ "636324136426": 152095,
300
+ "637551427887": 151941,
301
+ "643489267333": 151666,
302
+ "643692393748": 151695,
303
+ "644234165531": 151917,
304
+ "645527857972": 152128,
305
+ "646979124677": 151704,
306
+ "649291669397": 151656,
307
+ "649516938598": 152104,
308
+ "652154162978": 151851,
309
+ "653573457168": 151786,
310
+ "655517954651": 151913,
311
+ "658481324922": 151709,
312
+ "661436365453": 151727,
313
+ "662426712668": 151811,
314
+ "664847713559": 151931,
315
+ "665661658354": 151848,
316
+ "668134915514": 151694,
317
+ "668514843129": 151853,
318
+ "669183782449": 151937,
319
+ "669689115557": 151680,
320
+ "673948786567": 151914,
321
+ "676539973863": 151734,
322
+ "676676697292": 151739,
323
+ "677896358599": 151793,
324
+ "678666359123": 151820,
325
+ "682564697312": 151674,
326
+ "687284547244": 152142,
327
+ "689285841825": 151767,
328
+ "689362171782": 151676,
329
+ "691315125346": 151929,
330
+ "691587454538": 151659,
331
+ "693281533643": 151849,
332
+ "694235616268": 151701,
333
+ "696651276628": 151696,
334
+ "697747991249": 152037,
335
+ "698218415444": 151672,
336
+ "699446724178": 151994,
337
+ "711412347158": 151812,
338
+ "711641973413": 151947,
339
+ "712992815957": 151815,
340
+ "714738141818": 152055,
341
+ "716763689965": 152038,
342
+ "717628619782": 152105,
343
+ "718726519731": 151664,
344
+ "718951851413": 151993,
345
+ "721737645236": 151749,
346
+ "722339992679": 152060,
347
+ "722618266242": 151723,
348
+ "724492262421": 151986,
349
+ "724883568652": 151769,
350
+ "724964923582": 152073,
351
+ "729116422489": 152021,
352
+ "729825168745": 151928,
353
+ "731681792655": 151860,
354
+ "733737988257": 151771,
355
+ "739246183345": 152012,
356
+ "739394546916": 151939,
357
+ "739667844957": 152027,
358
+ "739784153482": 152023,
359
+ "741757479784": 151833,
360
+ "742765629356": 151732,
361
+ "745977673725": 151898,
362
+ "746285987371": 151655,
363
+ "749665846687": 151726,
364
+ "749966767285": 151677,
365
+ "752644929761": 151857,
366
+ "753159165157": 151678,
367
+ "753233719644": 151852,
368
+ "753466987363": 151813,
369
+ "754978243758": 152085,
370
+ "756916173536": 152099,
371
+ "757422791351": 152094,
372
+ "757785319772": 152062,
373
+ "758963111966": 152068,
374
+ "762486672996": 152002,
375
+ "762495822823": 152092,
376
+ "763531783491": 151837,
377
+ "765474393376": 151653,
378
+ "765737843639": 151808,
379
+ "765981581453": 151687,
380
+ "766294538337": 151901,
381
+ "767858144912": 152005,
382
+ "768461952319": 151930,
383
+ "771619544974": 151768,
384
+ "773671349113": 151925,
385
+ "773687856563": 151718,
386
+ "774679513717": 152117,
387
+ "778466723723": 151698,
388
+ "781653545886": 151668,
389
+ "786391632558": 151847,
390
+ "794825141891": 151688,
391
+ "795693824762": 151679,
392
+ "795838284624": 151715,
393
+ "796415361892": 151669,
394
+ "797238311233": 152145,
395
+ "799364566435": 151798,
396
+ "799417897197": 151878,
397
+ "811371928234": 151866,
398
+ "813981174797": 151983,
399
+ "814611949491": 151946,
400
+ "815458725125": 151996,
401
+ "816115936673": 151916,
402
+ "817378986438": 151800,
403
+ "818479319499": 152040,
404
+ "819162623892": 151854,
405
+ "824387273757": 151802,
406
+ "824488329416": 151934,
407
+ "825815739116": 152035,
408
+ "826214551218": 151735,
409
+ "829235161526": 151817,
410
+ "836746273134": 151708,
411
+ "837131396371": 151867,
412
+ "837873762491": 151935,
413
+ "839269684865": 151873,
414
+ "841824384614": 151782,
415
+ "841991688488": 151670,
416
+ "843998562287": 151779,
417
+ "846657726193": 152059,
418
+ "848845611563": 151794,
419
+ "851254219729": 152118,
420
+ "851852651677": 151795,
421
+ "852347289392": 151772,
422
+ "853859348234": 151697,
423
+ "856317483891": 152108,
424
+ "857231548835": 151915,
425
+ "858146415154": 152004,
426
+ "859289776479": 152134,
427
+ "859927364588": 152030,
428
+ "861342291298": 152115,
429
+ "864718285734": 151862,
430
+ "865331112869": 151646,
431
+ "866519986723": 151955,
432
+ "867678158885": 151764,
433
+ "873898736873": 151689,
434
+ "876537814263": 151975,
435
+ "877385381798": 151856,
436
+ "877482678543": 151651,
437
+ "879425227932": 151763,
438
+ "879764613759": 151968,
439
+ "881112998642": 151886,
440
+ "881136466196": 151858,
441
+ "883685933161": 151706,
442
+ "884354318946": 151978,
443
+ "886464195129": 151933,
444
+ "887417525236": 152103,
445
+ "887753195844": 151911,
446
+ "889965243135": 151988,
447
+ "892336777248": 151880,
448
+ "892697125149": 152100,
449
+ "893974619684": 152144,
450
+ "894885246797": 151792,
451
+ "897784492783": 151883,
452
+ "899151413682": 151758,
453
+ "912193598348": 151713,
454
+ "916198946615": 151803,
455
+ "919181521716": 152064,
456
+ "921889371893": 151922,
457
+ "921894985642": 152132,
458
+ "922327493433": 151705,
459
+ "922466849763": 151748,
460
+ "922867396683": 151807,
461
+ "927914615679": 151657,
462
+ "928335588653": 151761,
463
+ "929185622831": 151740,
464
+ "929826356991": 151699,
465
+ "933775286797": 152136,
466
+ "934912591213": 151902,
467
+ "937495658852": 151754,
468
+ "938488813964": 151843,
469
+ "941169431859": 151855,
470
+ "941861857715": 152102,
471
+ "941976537588": 151938,
472
+ "943193294691": 151893,
473
+ "943878645871": 152133,
474
+ "944513428457": 151869,
475
+ "946444632795": 152018,
476
+ "947262995132": 152143,
477
+ "948585673431": 152077,
478
+ "949235426629": 152125,
479
+ "951794811935": 151805,
480
+ "952465217972": 151926,
481
+ "952872416512": 151724,
482
+ "953267689786": 151932,
483
+ "955456118295": 151838,
484
+ "955668961269": 151894,
485
+ "961133895172": 151861,
486
+ "961981959227": 151844,
487
+ "963852916563": 151863,
488
+ "964342468552": 151789,
489
+ "967114116556": 151799,
490
+ "967214577653": 152016,
491
+ "971621888444": 151816,
492
+ "981277526855": 152091,
493
+ "982993456429": 151663,
494
+ "983493951323": 151985,
495
+ "983569279932": 151682,
496
+ "985781138935": 152065,
497
+ "993832498489": 151982,
498
+ "996116119839": 151835,
499
+ "997724962668": 152080,
500
+ "997958836435": 151879,
501
+ "997981281989": 151661,
502
+ "<|endoftext|>": 151643,
503
+ "<|im_end|>": 151645,
504
+ "<|im_start|>": 151644
505
+ }
qwen1.5-moe-a2.7b/base_attn_v2/tokenizer/chat_template.jinja ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system
2
+ You are a helpful assistant<|im_end|>
3
+ ' }}{% endif %}{{'<|im_start|>' + message['role'] + '
4
+ ' + message['content'] + '<|im_end|>' + '
5
+ '}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
6
+ ' }}{% endif %}
qwen1.5-moe-a2.7b/base_attn_v2/tokenizer/merges.txt ADDED
The diff for this file is too large to render. See raw diff
 
qwen1.5-moe-a2.7b/base_attn_v2/tokenizer/special_tokens_map.json ADDED
@@ -0,0 +1,3518 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": [
3
+ {
4
+ "content": "865331112869",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false
9
+ },
10
+ {
11
+ "content": "569765693871",
12
+ "lstrip": false,
13
+ "normalized": false,
14
+ "rstrip": false,
15
+ "single_word": false
16
+ },
17
+ {
18
+ "content": "485177821815",
19
+ "lstrip": false,
20
+ "normalized": false,
21
+ "rstrip": false,
22
+ "single_word": false
23
+ },
24
+ {
25
+ "content": "135441121756",
26
+ "lstrip": false,
27
+ "normalized": false,
28
+ "rstrip": false,
29
+ "single_word": false
30
+ },
31
+ {
32
+ "content": "367459894796",
33
+ "lstrip": false,
34
+ "normalized": false,
35
+ "rstrip": false,
36
+ "single_word": false
37
+ },
38
+ {
39
+ "content": "877482678543",
40
+ "lstrip": false,
41
+ "normalized": false,
42
+ "rstrip": false,
43
+ "single_word": false
44
+ },
45
+ {
46
+ "content": "457919547633",
47
+ "lstrip": false,
48
+ "normalized": false,
49
+ "rstrip": false,
50
+ "single_word": false
51
+ },
52
+ {
53
+ "content": "765474393376",
54
+ "lstrip": false,
55
+ "normalized": false,
56
+ "rstrip": false,
57
+ "single_word": false
58
+ },
59
+ {
60
+ "content": "114848338811",
61
+ "lstrip": false,
62
+ "normalized": false,
63
+ "rstrip": false,
64
+ "single_word": false
65
+ },
66
+ {
67
+ "content": "746285987371",
68
+ "lstrip": false,
69
+ "normalized": false,
70
+ "rstrip": false,
71
+ "single_word": false
72
+ },
73
+ {
74
+ "content": "649291669397",
75
+ "lstrip": false,
76
+ "normalized": false,
77
+ "rstrip": false,
78
+ "single_word": false
79
+ },
80
+ {
81
+ "content": "927914615679",
82
+ "lstrip": false,
83
+ "normalized": false,
84
+ "rstrip": false,
85
+ "single_word": false
86
+ },
87
+ {
88
+ "content": "445925149649",
89
+ "lstrip": false,
90
+ "normalized": false,
91
+ "rstrip": false,
92
+ "single_word": false
93
+ },
94
+ {
95
+ "content": "691587454538",
96
+ "lstrip": false,
97
+ "normalized": false,
98
+ "rstrip": false,
99
+ "single_word": false
100
+ },
101
+ {
102
+ "content": "143777992227",
103
+ "lstrip": false,
104
+ "normalized": false,
105
+ "rstrip": false,
106
+ "single_word": false
107
+ },
108
+ {
109
+ "content": "997981281989",
110
+ "lstrip": false,
111
+ "normalized": false,
112
+ "rstrip": false,
113
+ "single_word": false
114
+ },
115
+ {
116
+ "content": "425949483533",
117
+ "lstrip": false,
118
+ "normalized": false,
119
+ "rstrip": false,
120
+ "single_word": false
121
+ },
122
+ {
123
+ "content": "982993456429",
124
+ "lstrip": false,
125
+ "normalized": false,
126
+ "rstrip": false,
127
+ "single_word": false
128
+ },
129
+ {
130
+ "content": "718726519731",
131
+ "lstrip": false,
132
+ "normalized": false,
133
+ "rstrip": false,
134
+ "single_word": false
135
+ },
136
+ {
137
+ "content": "172599315861",
138
+ "lstrip": false,
139
+ "normalized": false,
140
+ "rstrip": false,
141
+ "single_word": false
142
+ },
143
+ {
144
+ "content": "643489267333",
145
+ "lstrip": false,
146
+ "normalized": false,
147
+ "rstrip": false,
148
+ "single_word": false
149
+ },
150
+ {
151
+ "content": "282322838685",
152
+ "lstrip": false,
153
+ "normalized": false,
154
+ "rstrip": false,
155
+ "single_word": false
156
+ },
157
+ {
158
+ "content": "781653545886",
159
+ "lstrip": false,
160
+ "normalized": false,
161
+ "rstrip": false,
162
+ "single_word": false
163
+ },
164
+ {
165
+ "content": "796415361892",
166
+ "lstrip": false,
167
+ "normalized": false,
168
+ "rstrip": false,
169
+ "single_word": false
170
+ },
171
+ {
172
+ "content": "841991688488",
173
+ "lstrip": false,
174
+ "normalized": false,
175
+ "rstrip": false,
176
+ "single_word": false
177
+ },
178
+ {
179
+ "content": "211411365397",
180
+ "lstrip": false,
181
+ "normalized": false,
182
+ "rstrip": false,
183
+ "single_word": false
184
+ },
185
+ {
186
+ "content": "698218415444",
187
+ "lstrip": false,
188
+ "normalized": false,
189
+ "rstrip": false,
190
+ "single_word": false
191
+ },
192
+ {
193
+ "content": "355977139358",
194
+ "lstrip": false,
195
+ "normalized": false,
196
+ "rstrip": false,
197
+ "single_word": false
198
+ },
199
+ {
200
+ "content": "682564697312",
201
+ "lstrip": false,
202
+ "normalized": false,
203
+ "rstrip": false,
204
+ "single_word": false
205
+ },
206
+ {
207
+ "content": "383837596997",
208
+ "lstrip": false,
209
+ "normalized": false,
210
+ "rstrip": false,
211
+ "single_word": false
212
+ },
213
+ {
214
+ "content": "689362171782",
215
+ "lstrip": false,
216
+ "normalized": false,
217
+ "rstrip": false,
218
+ "single_word": false
219
+ },
220
+ {
221
+ "content": "749966767285",
222
+ "lstrip": false,
223
+ "normalized": false,
224
+ "rstrip": false,
225
+ "single_word": false
226
+ },
227
+ {
228
+ "content": "753159165157",
229
+ "lstrip": false,
230
+ "normalized": false,
231
+ "rstrip": false,
232
+ "single_word": false
233
+ },
234
+ {
235
+ "content": "795693824762",
236
+ "lstrip": false,
237
+ "normalized": false,
238
+ "rstrip": false,
239
+ "single_word": false
240
+ },
241
+ {
242
+ "content": "669689115557",
243
+ "lstrip": false,
244
+ "normalized": false,
245
+ "rstrip": false,
246
+ "single_word": false
247
+ },
248
+ {
249
+ "content": "327491773134",
250
+ "lstrip": false,
251
+ "normalized": false,
252
+ "rstrip": false,
253
+ "single_word": false
254
+ },
255
+ {
256
+ "content": "983569279932",
257
+ "lstrip": false,
258
+ "normalized": false,
259
+ "rstrip": false,
260
+ "single_word": false
261
+ },
262
+ {
263
+ "content": "612128769512",
264
+ "lstrip": false,
265
+ "normalized": false,
266
+ "rstrip": false,
267
+ "single_word": false
268
+ },
269
+ {
270
+ "content": "374327157578",
271
+ "lstrip": false,
272
+ "normalized": false,
273
+ "rstrip": false,
274
+ "single_word": false
275
+ },
276
+ {
277
+ "content": "311632789559",
278
+ "lstrip": false,
279
+ "normalized": false,
280
+ "rstrip": false,
281
+ "single_word": false
282
+ },
283
+ {
284
+ "content": "523918658846",
285
+ "lstrip": false,
286
+ "normalized": false,
287
+ "rstrip": false,
288
+ "single_word": false
289
+ },
290
+ {
291
+ "content": "765981581453",
292
+ "lstrip": false,
293
+ "normalized": false,
294
+ "rstrip": false,
295
+ "single_word": false
296
+ },
297
+ {
298
+ "content": "794825141891",
299
+ "lstrip": false,
300
+ "normalized": false,
301
+ "rstrip": false,
302
+ "single_word": false
303
+ },
304
+ {
305
+ "content": "873898736873",
306
+ "lstrip": false,
307
+ "normalized": false,
308
+ "rstrip": false,
309
+ "single_word": false
310
+ },
311
+ {
312
+ "content": "447445629421",
313
+ "lstrip": false,
314
+ "normalized": false,
315
+ "rstrip": false,
316
+ "single_word": false
317
+ },
318
+ {
319
+ "content": "473822473819",
320
+ "lstrip": false,
321
+ "normalized": false,
322
+ "rstrip": false,
323
+ "single_word": false
324
+ },
325
+ {
326
+ "content": "181439694557",
327
+ "lstrip": false,
328
+ "normalized": false,
329
+ "rstrip": false,
330
+ "single_word": false
331
+ },
332
+ {
333
+ "content": "592538279337",
334
+ "lstrip": false,
335
+ "normalized": false,
336
+ "rstrip": false,
337
+ "single_word": false
338
+ },
339
+ {
340
+ "content": "668134915514",
341
+ "lstrip": false,
342
+ "normalized": false,
343
+ "rstrip": false,
344
+ "single_word": false
345
+ },
346
+ {
347
+ "content": "643692393748",
348
+ "lstrip": false,
349
+ "normalized": false,
350
+ "rstrip": false,
351
+ "single_word": false
352
+ },
353
+ {
354
+ "content": "696651276628",
355
+ "lstrip": false,
356
+ "normalized": false,
357
+ "rstrip": false,
358
+ "single_word": false
359
+ },
360
+ {
361
+ "content": "853859348234",
362
+ "lstrip": false,
363
+ "normalized": false,
364
+ "rstrip": false,
365
+ "single_word": false
366
+ },
367
+ {
368
+ "content": "778466723723",
369
+ "lstrip": false,
370
+ "normalized": false,
371
+ "rstrip": false,
372
+ "single_word": false
373
+ },
374
+ {
375
+ "content": "929826356991",
376
+ "lstrip": false,
377
+ "normalized": false,
378
+ "rstrip": false,
379
+ "single_word": false
380
+ },
381
+ {
382
+ "content": "272362973463",
383
+ "lstrip": false,
384
+ "normalized": false,
385
+ "rstrip": false,
386
+ "single_word": false
387
+ },
388
+ {
389
+ "content": "694235616268",
390
+ "lstrip": false,
391
+ "normalized": false,
392
+ "rstrip": false,
393
+ "single_word": false
394
+ },
395
+ {
396
+ "content": "281673864127",
397
+ "lstrip": false,
398
+ "normalized": false,
399
+ "rstrip": false,
400
+ "single_word": false
401
+ },
402
+ {
403
+ "content": "479676316326",
404
+ "lstrip": false,
405
+ "normalized": false,
406
+ "rstrip": false,
407
+ "single_word": false
408
+ },
409
+ {
410
+ "content": "646979124677",
411
+ "lstrip": false,
412
+ "normalized": false,
413
+ "rstrip": false,
414
+ "single_word": false
415
+ },
416
+ {
417
+ "content": "922327493433",
418
+ "lstrip": false,
419
+ "normalized": false,
420
+ "rstrip": false,
421
+ "single_word": false
422
+ },
423
+ {
424
+ "content": "883685933161",
425
+ "lstrip": false,
426
+ "normalized": false,
427
+ "rstrip": false,
428
+ "single_word": false
429
+ },
430
+ {
431
+ "content": "264259917554",
432
+ "lstrip": false,
433
+ "normalized": false,
434
+ "rstrip": false,
435
+ "single_word": false
436
+ },
437
+ {
438
+ "content": "836746273134",
439
+ "lstrip": false,
440
+ "normalized": false,
441
+ "rstrip": false,
442
+ "single_word": false
443
+ },
444
+ {
445
+ "content": "658481324922",
446
+ "lstrip": false,
447
+ "normalized": false,
448
+ "rstrip": false,
449
+ "single_word": false
450
+ },
451
+ {
452
+ "content": "481884157827",
453
+ "lstrip": false,
454
+ "normalized": false,
455
+ "rstrip": false,
456
+ "single_word": false
457
+ },
458
+ {
459
+ "content": "587787496812",
460
+ "lstrip": false,
461
+ "normalized": false,
462
+ "rstrip": false,
463
+ "single_word": false
464
+ },
465
+ {
466
+ "content": "579184949249",
467
+ "lstrip": false,
468
+ "normalized": false,
469
+ "rstrip": false,
470
+ "single_word": false
471
+ },
472
+ {
473
+ "content": "912193598348",
474
+ "lstrip": false,
475
+ "normalized": false,
476
+ "rstrip": false,
477
+ "single_word": false
478
+ },
479
+ {
480
+ "content": "529679678956",
481
+ "lstrip": false,
482
+ "normalized": false,
483
+ "rstrip": false,
484
+ "single_word": false
485
+ },
486
+ {
487
+ "content": "795838284624",
488
+ "lstrip": false,
489
+ "normalized": false,
490
+ "rstrip": false,
491
+ "single_word": false
492
+ },
493
+ {
494
+ "content": "159337222655",
495
+ "lstrip": false,
496
+ "normalized": false,
497
+ "rstrip": false,
498
+ "single_word": false
499
+ },
500
+ {
501
+ "content": "173781362446",
502
+ "lstrip": false,
503
+ "normalized": false,
504
+ "rstrip": false,
505
+ "single_word": false
506
+ },
507
+ {
508
+ "content": "773687856563",
509
+ "lstrip": false,
510
+ "normalized": false,
511
+ "rstrip": false,
512
+ "single_word": false
513
+ },
514
+ {
515
+ "content": "535787224917",
516
+ "lstrip": false,
517
+ "normalized": false,
518
+ "rstrip": false,
519
+ "single_word": false
520
+ },
521
+ {
522
+ "content": "351885857332",
523
+ "lstrip": false,
524
+ "normalized": false,
525
+ "rstrip": false,
526
+ "single_word": false
527
+ },
528
+ {
529
+ "content": "578827344666",
530
+ "lstrip": false,
531
+ "normalized": false,
532
+ "rstrip": false,
533
+ "single_word": false
534
+ },
535
+ {
536
+ "content": "198462689911",
537
+ "lstrip": false,
538
+ "normalized": false,
539
+ "rstrip": false,
540
+ "single_word": false
541
+ },
542
+ {
543
+ "content": "722618266242",
544
+ "lstrip": false,
545
+ "normalized": false,
546
+ "rstrip": false,
547
+ "single_word": false
548
+ },
549
+ {
550
+ "content": "952872416512",
551
+ "lstrip": false,
552
+ "normalized": false,
553
+ "rstrip": false,
554
+ "single_word": false
555
+ },
556
+ {
557
+ "content": "517778845323",
558
+ "lstrip": false,
559
+ "normalized": false,
560
+ "rstrip": false,
561
+ "single_word": false
562
+ },
563
+ {
564
+ "content": "749665846687",
565
+ "lstrip": false,
566
+ "normalized": false,
567
+ "rstrip": false,
568
+ "single_word": false
569
+ },
570
+ {
571
+ "content": "661436365453",
572
+ "lstrip": false,
573
+ "normalized": false,
574
+ "rstrip": false,
575
+ "single_word": false
576
+ },
577
+ {
578
+ "content": "259666844669",
579
+ "lstrip": false,
580
+ "normalized": false,
581
+ "rstrip": false,
582
+ "single_word": false
583
+ },
584
+ {
585
+ "content": "242851284913",
586
+ "lstrip": false,
587
+ "normalized": false,
588
+ "rstrip": false,
589
+ "single_word": false
590
+ },
591
+ {
592
+ "content": "514532995959",
593
+ "lstrip": false,
594
+ "normalized": false,
595
+ "rstrip": false,
596
+ "single_word": false
597
+ },
598
+ {
599
+ "content": "161588262349",
600
+ "lstrip": false,
601
+ "normalized": false,
602
+ "rstrip": false,
603
+ "single_word": false
604
+ },
605
+ {
606
+ "content": "742765629356",
607
+ "lstrip": false,
608
+ "normalized": false,
609
+ "rstrip": false,
610
+ "single_word": false
611
+ },
612
+ {
613
+ "content": "225164373623",
614
+ "lstrip": false,
615
+ "normalized": false,
616
+ "rstrip": false,
617
+ "single_word": false
618
+ },
619
+ {
620
+ "content": "676539973863",
621
+ "lstrip": false,
622
+ "normalized": false,
623
+ "rstrip": false,
624
+ "single_word": false
625
+ },
626
+ {
627
+ "content": "826214551218",
628
+ "lstrip": false,
629
+ "normalized": false,
630
+ "rstrip": false,
631
+ "single_word": false
632
+ },
633
+ {
634
+ "content": "182345464792",
635
+ "lstrip": false,
636
+ "normalized": false,
637
+ "rstrip": false,
638
+ "single_word": false
639
+ },
640
+ {
641
+ "content": "232776999554",
642
+ "lstrip": false,
643
+ "normalized": false,
644
+ "rstrip": false,
645
+ "single_word": false
646
+ },
647
+ {
648
+ "content": "337326533813",
649
+ "lstrip": false,
650
+ "normalized": false,
651
+ "rstrip": false,
652
+ "single_word": false
653
+ },
654
+ {
655
+ "content": "676676697292",
656
+ "lstrip": false,
657
+ "normalized": false,
658
+ "rstrip": false,
659
+ "single_word": false
660
+ },
661
+ {
662
+ "content": "929185622831",
663
+ "lstrip": false,
664
+ "normalized": false,
665
+ "rstrip": false,
666
+ "single_word": false
667
+ },
668
+ {
669
+ "content": "545512344383",
670
+ "lstrip": false,
671
+ "normalized": false,
672
+ "rstrip": false,
673
+ "single_word": false
674
+ },
675
+ {
676
+ "content": "499444466686",
677
+ "lstrip": false,
678
+ "normalized": false,
679
+ "rstrip": false,
680
+ "single_word": false
681
+ },
682
+ {
683
+ "content": "314697386682",
684
+ "lstrip": false,
685
+ "normalized": false,
686
+ "rstrip": false,
687
+ "single_word": false
688
+ },
689
+ {
690
+ "content": "517379856925",
691
+ "lstrip": false,
692
+ "normalized": false,
693
+ "rstrip": false,
694
+ "single_word": false
695
+ },
696
+ {
697
+ "content": "379557332953",
698
+ "lstrip": false,
699
+ "normalized": false,
700
+ "rstrip": false,
701
+ "single_word": false
702
+ },
703
+ {
704
+ "content": "614797267726",
705
+ "lstrip": false,
706
+ "normalized": false,
707
+ "rstrip": false,
708
+ "single_word": false
709
+ },
710
+ {
711
+ "content": "429781429464",
712
+ "lstrip": false,
713
+ "normalized": false,
714
+ "rstrip": false,
715
+ "single_word": false
716
+ },
717
+ {
718
+ "content": "922466849763",
719
+ "lstrip": false,
720
+ "normalized": false,
721
+ "rstrip": false,
722
+ "single_word": false
723
+ },
724
+ {
725
+ "content": "721737645236",
726
+ "lstrip": false,
727
+ "normalized": false,
728
+ "rstrip": false,
729
+ "single_word": false
730
+ },
731
+ {
732
+ "content": "479227349997",
733
+ "lstrip": false,
734
+ "normalized": false,
735
+ "rstrip": false,
736
+ "single_word": false
737
+ },
738
+ {
739
+ "content": "136931728327",
740
+ "lstrip": false,
741
+ "normalized": false,
742
+ "rstrip": false,
743
+ "single_word": false
744
+ },
745
+ {
746
+ "content": "259533577263",
747
+ "lstrip": false,
748
+ "normalized": false,
749
+ "rstrip": false,
750
+ "single_word": false
751
+ },
752
+ {
753
+ "content": "488538864842",
754
+ "lstrip": false,
755
+ "normalized": false,
756
+ "rstrip": false,
757
+ "single_word": false
758
+ },
759
+ {
760
+ "content": "937495658852",
761
+ "lstrip": false,
762
+ "normalized": false,
763
+ "rstrip": false,
764
+ "single_word": false
765
+ },
766
+ {
767
+ "content": "489991411364",
768
+ "lstrip": false,
769
+ "normalized": false,
770
+ "rstrip": false,
771
+ "single_word": false
772
+ },
773
+ {
774
+ "content": "499148455254",
775
+ "lstrip": false,
776
+ "normalized": false,
777
+ "rstrip": false,
778
+ "single_word": false
779
+ },
780
+ {
781
+ "content": "441373944925",
782
+ "lstrip": false,
783
+ "normalized": false,
784
+ "rstrip": false,
785
+ "single_word": false
786
+ },
787
+ {
788
+ "content": "899151413682",
789
+ "lstrip": false,
790
+ "normalized": false,
791
+ "rstrip": false,
792
+ "single_word": false
793
+ },
794
+ {
795
+ "content": "467893531755",
796
+ "lstrip": false,
797
+ "normalized": false,
798
+ "rstrip": false,
799
+ "single_word": false
800
+ },
801
+ {
802
+ "content": "527117488925",
803
+ "lstrip": false,
804
+ "normalized": false,
805
+ "rstrip": false,
806
+ "single_word": false
807
+ },
808
+ {
809
+ "content": "928335588653",
810
+ "lstrip": false,
811
+ "normalized": false,
812
+ "rstrip": false,
813
+ "single_word": false
814
+ },
815
+ {
816
+ "content": "374439448821",
817
+ "lstrip": false,
818
+ "normalized": false,
819
+ "rstrip": false,
820
+ "single_word": false
821
+ },
822
+ {
823
+ "content": "879425227932",
824
+ "lstrip": false,
825
+ "normalized": false,
826
+ "rstrip": false,
827
+ "single_word": false
828
+ },
829
+ {
830
+ "content": "867678158885",
831
+ "lstrip": false,
832
+ "normalized": false,
833
+ "rstrip": false,
834
+ "single_word": false
835
+ },
836
+ {
837
+ "content": "399749397872",
838
+ "lstrip": false,
839
+ "normalized": false,
840
+ "rstrip": false,
841
+ "single_word": false
842
+ },
843
+ {
844
+ "content": "129693547287",
845
+ "lstrip": false,
846
+ "normalized": false,
847
+ "rstrip": false,
848
+ "single_word": false
849
+ },
850
+ {
851
+ "content": "689285841825",
852
+ "lstrip": false,
853
+ "normalized": false,
854
+ "rstrip": false,
855
+ "single_word": false
856
+ },
857
+ {
858
+ "content": "771619544974",
859
+ "lstrip": false,
860
+ "normalized": false,
861
+ "rstrip": false,
862
+ "single_word": false
863
+ },
864
+ {
865
+ "content": "724883568652",
866
+ "lstrip": false,
867
+ "normalized": false,
868
+ "rstrip": false,
869
+ "single_word": false
870
+ },
871
+ {
872
+ "content": "516968424863",
873
+ "lstrip": false,
874
+ "normalized": false,
875
+ "rstrip": false,
876
+ "single_word": false
877
+ },
878
+ {
879
+ "content": "733737988257",
880
+ "lstrip": false,
881
+ "normalized": false,
882
+ "rstrip": false,
883
+ "single_word": false
884
+ },
885
+ {
886
+ "content": "852347289392",
887
+ "lstrip": false,
888
+ "normalized": false,
889
+ "rstrip": false,
890
+ "single_word": false
891
+ },
892
+ {
893
+ "content": "296953381169",
894
+ "lstrip": false,
895
+ "normalized": false,
896
+ "rstrip": false,
897
+ "single_word": false
898
+ },
899
+ {
900
+ "content": "377273562477",
901
+ "lstrip": false,
902
+ "normalized": false,
903
+ "rstrip": false,
904
+ "single_word": false
905
+ },
906
+ {
907
+ "content": "262296912232",
908
+ "lstrip": false,
909
+ "normalized": false,
910
+ "rstrip": false,
911
+ "single_word": false
912
+ },
913
+ {
914
+ "content": "547149832394",
915
+ "lstrip": false,
916
+ "normalized": false,
917
+ "rstrip": false,
918
+ "single_word": false
919
+ },
920
+ {
921
+ "content": "298464134954",
922
+ "lstrip": false,
923
+ "normalized": false,
924
+ "rstrip": false,
925
+ "single_word": false
926
+ },
927
+ {
928
+ "content": "216667245274",
929
+ "lstrip": false,
930
+ "normalized": false,
931
+ "rstrip": false,
932
+ "single_word": false
933
+ },
934
+ {
935
+ "content": "843998562287",
936
+ "lstrip": false,
937
+ "normalized": false,
938
+ "rstrip": false,
939
+ "single_word": false
940
+ },
941
+ {
942
+ "content": "572154333646",
943
+ "lstrip": false,
944
+ "normalized": false,
945
+ "rstrip": false,
946
+ "single_word": false
947
+ },
948
+ {
949
+ "content": "124589118494",
950
+ "lstrip": false,
951
+ "normalized": false,
952
+ "rstrip": false,
953
+ "single_word": false
954
+ },
955
+ {
956
+ "content": "841824384614",
957
+ "lstrip": false,
958
+ "normalized": false,
959
+ "rstrip": false,
960
+ "single_word": false
961
+ },
962
+ {
963
+ "content": "232896526252",
964
+ "lstrip": false,
965
+ "normalized": false,
966
+ "rstrip": false,
967
+ "single_word": false
968
+ },
969
+ {
970
+ "content": "295448593321",
971
+ "lstrip": false,
972
+ "normalized": false,
973
+ "rstrip": false,
974
+ "single_word": false
975
+ },
976
+ {
977
+ "content": "123741461297",
978
+ "lstrip": false,
979
+ "normalized": false,
980
+ "rstrip": false,
981
+ "single_word": false
982
+ },
983
+ {
984
+ "content": "653573457168",
985
+ "lstrip": false,
986
+ "normalized": false,
987
+ "rstrip": false,
988
+ "single_word": false
989
+ },
990
+ {
991
+ "content": "196735786156",
992
+ "lstrip": false,
993
+ "normalized": false,
994
+ "rstrip": false,
995
+ "single_word": false
996
+ },
997
+ {
998
+ "content": "377338713663",
999
+ "lstrip": false,
1000
+ "normalized": false,
1001
+ "rstrip": false,
1002
+ "single_word": false
1003
+ },
1004
+ {
1005
+ "content": "964342468552",
1006
+ "lstrip": false,
1007
+ "normalized": false,
1008
+ "rstrip": false,
1009
+ "single_word": false
1010
+ },
1011
+ {
1012
+ "content": "586855179568",
1013
+ "lstrip": false,
1014
+ "normalized": false,
1015
+ "rstrip": false,
1016
+ "single_word": false
1017
+ },
1018
+ {
1019
+ "content": "484773717614",
1020
+ "lstrip": false,
1021
+ "normalized": false,
1022
+ "rstrip": false,
1023
+ "single_word": false
1024
+ },
1025
+ {
1026
+ "content": "894885246797",
1027
+ "lstrip": false,
1028
+ "normalized": false,
1029
+ "rstrip": false,
1030
+ "single_word": false
1031
+ },
1032
+ {
1033
+ "content": "677896358599",
1034
+ "lstrip": false,
1035
+ "normalized": false,
1036
+ "rstrip": false,
1037
+ "single_word": false
1038
+ },
1039
+ {
1040
+ "content": "848845611563",
1041
+ "lstrip": false,
1042
+ "normalized": false,
1043
+ "rstrip": false,
1044
+ "single_word": false
1045
+ },
1046
+ {
1047
+ "content": "851852651677",
1048
+ "lstrip": false,
1049
+ "normalized": false,
1050
+ "rstrip": false,
1051
+ "single_word": false
1052
+ },
1053
+ {
1054
+ "content": "398549545767",
1055
+ "lstrip": false,
1056
+ "normalized": false,
1057
+ "rstrip": false,
1058
+ "single_word": false
1059
+ },
1060
+ {
1061
+ "content": "454244839926",
1062
+ "lstrip": false,
1063
+ "normalized": false,
1064
+ "rstrip": false,
1065
+ "single_word": false
1066
+ },
1067
+ {
1068
+ "content": "799364566435",
1069
+ "lstrip": false,
1070
+ "normalized": false,
1071
+ "rstrip": false,
1072
+ "single_word": false
1073
+ },
1074
+ {
1075
+ "content": "967114116556",
1076
+ "lstrip": false,
1077
+ "normalized": false,
1078
+ "rstrip": false,
1079
+ "single_word": false
1080
+ },
1081
+ {
1082
+ "content": "817378986438",
1083
+ "lstrip": false,
1084
+ "normalized": false,
1085
+ "rstrip": false,
1086
+ "single_word": false
1087
+ },
1088
+ {
1089
+ "content": "233795848681",
1090
+ "lstrip": false,
1091
+ "normalized": false,
1092
+ "rstrip": false,
1093
+ "single_word": false
1094
+ },
1095
+ {
1096
+ "content": "824387273757",
1097
+ "lstrip": false,
1098
+ "normalized": false,
1099
+ "rstrip": false,
1100
+ "single_word": false
1101
+ },
1102
+ {
1103
+ "content": "916198946615",
1104
+ "lstrip": false,
1105
+ "normalized": false,
1106
+ "rstrip": false,
1107
+ "single_word": false
1108
+ },
1109
+ {
1110
+ "content": "563117729724",
1111
+ "lstrip": false,
1112
+ "normalized": false,
1113
+ "rstrip": false,
1114
+ "single_word": false
1115
+ },
1116
+ {
1117
+ "content": "951794811935",
1118
+ "lstrip": false,
1119
+ "normalized": false,
1120
+ "rstrip": false,
1121
+ "single_word": false
1122
+ },
1123
+ {
1124
+ "content": "374598961236",
1125
+ "lstrip": false,
1126
+ "normalized": false,
1127
+ "rstrip": false,
1128
+ "single_word": false
1129
+ },
1130
+ {
1131
+ "content": "922867396683",
1132
+ "lstrip": false,
1133
+ "normalized": false,
1134
+ "rstrip": false,
1135
+ "single_word": false
1136
+ },
1137
+ {
1138
+ "content": "765737843639",
1139
+ "lstrip": false,
1140
+ "normalized": false,
1141
+ "rstrip": false,
1142
+ "single_word": false
1143
+ },
1144
+ {
1145
+ "content": "175469284871",
1146
+ "lstrip": false,
1147
+ "normalized": false,
1148
+ "rstrip": false,
1149
+ "single_word": false
1150
+ },
1151
+ {
1152
+ "content": "231853711778",
1153
+ "lstrip": false,
1154
+ "normalized": false,
1155
+ "rstrip": false,
1156
+ "single_word": false
1157
+ },
1158
+ {
1159
+ "content": "662426712668",
1160
+ "lstrip": false,
1161
+ "normalized": false,
1162
+ "rstrip": false,
1163
+ "single_word": false
1164
+ },
1165
+ {
1166
+ "content": "711412347158",
1167
+ "lstrip": false,
1168
+ "normalized": false,
1169
+ "rstrip": false,
1170
+ "single_word": false
1171
+ },
1172
+ {
1173
+ "content": "753466987363",
1174
+ "lstrip": false,
1175
+ "normalized": false,
1176
+ "rstrip": false,
1177
+ "single_word": false
1178
+ },
1179
+ {
1180
+ "content": "513361312532",
1181
+ "lstrip": false,
1182
+ "normalized": false,
1183
+ "rstrip": false,
1184
+ "single_word": false
1185
+ },
1186
+ {
1187
+ "content": "712992815957",
1188
+ "lstrip": false,
1189
+ "normalized": false,
1190
+ "rstrip": false,
1191
+ "single_word": false
1192
+ },
1193
+ {
1194
+ "content": "971621888444",
1195
+ "lstrip": false,
1196
+ "normalized": false,
1197
+ "rstrip": false,
1198
+ "single_word": false
1199
+ },
1200
+ {
1201
+ "content": "829235161526",
1202
+ "lstrip": false,
1203
+ "normalized": false,
1204
+ "rstrip": false,
1205
+ "single_word": false
1206
+ },
1207
+ {
1208
+ "content": "585544633356",
1209
+ "lstrip": false,
1210
+ "normalized": false,
1211
+ "rstrip": false,
1212
+ "single_word": false
1213
+ },
1214
+ {
1215
+ "content": "582471228164",
1216
+ "lstrip": false,
1217
+ "normalized": false,
1218
+ "rstrip": false,
1219
+ "single_word": false
1220
+ },
1221
+ {
1222
+ "content": "678666359123",
1223
+ "lstrip": false,
1224
+ "normalized": false,
1225
+ "rstrip": false,
1226
+ "single_word": false
1227
+ },
1228
+ {
1229
+ "content": "557533689478",
1230
+ "lstrip": false,
1231
+ "normalized": false,
1232
+ "rstrip": false,
1233
+ "single_word": false
1234
+ },
1235
+ {
1236
+ "content": "632962475133",
1237
+ "lstrip": false,
1238
+ "normalized": false,
1239
+ "rstrip": false,
1240
+ "single_word": false
1241
+ },
1242
+ {
1243
+ "content": "484489193824",
1244
+ "lstrip": false,
1245
+ "normalized": false,
1246
+ "rstrip": false,
1247
+ "single_word": false
1248
+ },
1249
+ {
1250
+ "content": "489562189822",
1251
+ "lstrip": false,
1252
+ "normalized": false,
1253
+ "rstrip": false,
1254
+ "single_word": false
1255
+ },
1256
+ {
1257
+ "content": "589547936288",
1258
+ "lstrip": false,
1259
+ "normalized": false,
1260
+ "rstrip": false,
1261
+ "single_word": false
1262
+ },
1263
+ {
1264
+ "content": "363214487524",
1265
+ "lstrip": false,
1266
+ "normalized": false,
1267
+ "rstrip": false,
1268
+ "single_word": false
1269
+ },
1270
+ {
1271
+ "content": "244885399387",
1272
+ "lstrip": false,
1273
+ "normalized": false,
1274
+ "rstrip": false,
1275
+ "single_word": false
1276
+ },
1277
+ {
1278
+ "content": "431751228368",
1279
+ "lstrip": false,
1280
+ "normalized": false,
1281
+ "rstrip": false,
1282
+ "single_word": false
1283
+ },
1284
+ {
1285
+ "content": "433581868192",
1286
+ "lstrip": false,
1287
+ "normalized": false,
1288
+ "rstrip": false,
1289
+ "single_word": false
1290
+ },
1291
+ {
1292
+ "content": "486391569221",
1293
+ "lstrip": false,
1294
+ "normalized": false,
1295
+ "rstrip": false,
1296
+ "single_word": false
1297
+ },
1298
+ {
1299
+ "content": "185438575221",
1300
+ "lstrip": false,
1301
+ "normalized": false,
1302
+ "rstrip": false,
1303
+ "single_word": false
1304
+ },
1305
+ {
1306
+ "content": "126574388585",
1307
+ "lstrip": false,
1308
+ "normalized": false,
1309
+ "rstrip": false,
1310
+ "single_word": false
1311
+ },
1312
+ {
1313
+ "content": "741757479784",
1314
+ "lstrip": false,
1315
+ "normalized": false,
1316
+ "rstrip": false,
1317
+ "single_word": false
1318
+ },
1319
+ {
1320
+ "content": "529854679937",
1321
+ "lstrip": false,
1322
+ "normalized": false,
1323
+ "rstrip": false,
1324
+ "single_word": false
1325
+ },
1326
+ {
1327
+ "content": "996116119839",
1328
+ "lstrip": false,
1329
+ "normalized": false,
1330
+ "rstrip": false,
1331
+ "single_word": false
1332
+ },
1333
+ {
1334
+ "content": "616248973917",
1335
+ "lstrip": false,
1336
+ "normalized": false,
1337
+ "rstrip": false,
1338
+ "single_word": false
1339
+ },
1340
+ {
1341
+ "content": "763531783491",
1342
+ "lstrip": false,
1343
+ "normalized": false,
1344
+ "rstrip": false,
1345
+ "single_word": false
1346
+ },
1347
+ {
1348
+ "content": "955456118295",
1349
+ "lstrip": false,
1350
+ "normalized": false,
1351
+ "rstrip": false,
1352
+ "single_word": false
1353
+ },
1354
+ {
1355
+ "content": "364196983365",
1356
+ "lstrip": false,
1357
+ "normalized": false,
1358
+ "rstrip": false,
1359
+ "single_word": false
1360
+ },
1361
+ {
1362
+ "content": "195792996468",
1363
+ "lstrip": false,
1364
+ "normalized": false,
1365
+ "rstrip": false,
1366
+ "single_word": false
1367
+ },
1368
+ {
1369
+ "content": "151859598873",
1370
+ "lstrip": false,
1371
+ "normalized": false,
1372
+ "rstrip": false,
1373
+ "single_word": false
1374
+ },
1375
+ {
1376
+ "content": "399223169721",
1377
+ "lstrip": false,
1378
+ "normalized": false,
1379
+ "rstrip": false,
1380
+ "single_word": false
1381
+ },
1382
+ {
1383
+ "content": "938488813964",
1384
+ "lstrip": false,
1385
+ "normalized": false,
1386
+ "rstrip": false,
1387
+ "single_word": false
1388
+ },
1389
+ {
1390
+ "content": "961981959227",
1391
+ "lstrip": false,
1392
+ "normalized": false,
1393
+ "rstrip": false,
1394
+ "single_word": false
1395
+ },
1396
+ {
1397
+ "content": "183368827562",
1398
+ "lstrip": false,
1399
+ "normalized": false,
1400
+ "rstrip": false,
1401
+ "single_word": false
1402
+ },
1403
+ {
1404
+ "content": "533417736566",
1405
+ "lstrip": false,
1406
+ "normalized": false,
1407
+ "rstrip": false,
1408
+ "single_word": false
1409
+ },
1410
+ {
1411
+ "content": "786391632558",
1412
+ "lstrip": false,
1413
+ "normalized": false,
1414
+ "rstrip": false,
1415
+ "single_word": false
1416
+ },
1417
+ {
1418
+ "content": "665661658354",
1419
+ "lstrip": false,
1420
+ "normalized": false,
1421
+ "rstrip": false,
1422
+ "single_word": false
1423
+ },
1424
+ {
1425
+ "content": "693281533643",
1426
+ "lstrip": false,
1427
+ "normalized": false,
1428
+ "rstrip": false,
1429
+ "single_word": false
1430
+ },
1431
+ {
1432
+ "content": "475794684356",
1433
+ "lstrip": false,
1434
+ "normalized": false,
1435
+ "rstrip": false,
1436
+ "single_word": false
1437
+ },
1438
+ {
1439
+ "content": "652154162978",
1440
+ "lstrip": false,
1441
+ "normalized": false,
1442
+ "rstrip": false,
1443
+ "single_word": false
1444
+ },
1445
+ {
1446
+ "content": "753233719644",
1447
+ "lstrip": false,
1448
+ "normalized": false,
1449
+ "rstrip": false,
1450
+ "single_word": false
1451
+ },
1452
+ {
1453
+ "content": "668514843129",
1454
+ "lstrip": false,
1455
+ "normalized": false,
1456
+ "rstrip": false,
1457
+ "single_word": false
1458
+ },
1459
+ {
1460
+ "content": "819162623892",
1461
+ "lstrip": false,
1462
+ "normalized": false,
1463
+ "rstrip": false,
1464
+ "single_word": false
1465
+ },
1466
+ {
1467
+ "content": "941169431859",
1468
+ "lstrip": false,
1469
+ "normalized": false,
1470
+ "rstrip": false,
1471
+ "single_word": false
1472
+ },
1473
+ {
1474
+ "content": "877385381798",
1475
+ "lstrip": false,
1476
+ "normalized": false,
1477
+ "rstrip": false,
1478
+ "single_word": false
1479
+ },
1480
+ {
1481
+ "content": "752644929761",
1482
+ "lstrip": false,
1483
+ "normalized": false,
1484
+ "rstrip": false,
1485
+ "single_word": false
1486
+ },
1487
+ {
1488
+ "content": "881136466196",
1489
+ "lstrip": false,
1490
+ "normalized": false,
1491
+ "rstrip": false,
1492
+ "single_word": false
1493
+ },
1494
+ {
1495
+ "content": "275597777299",
1496
+ "lstrip": false,
1497
+ "normalized": false,
1498
+ "rstrip": false,
1499
+ "single_word": false
1500
+ },
1501
+ {
1502
+ "content": "731681792655",
1503
+ "lstrip": false,
1504
+ "normalized": false,
1505
+ "rstrip": false,
1506
+ "single_word": false
1507
+ },
1508
+ {
1509
+ "content": "961133895172",
1510
+ "lstrip": false,
1511
+ "normalized": false,
1512
+ "rstrip": false,
1513
+ "single_word": false
1514
+ },
1515
+ {
1516
+ "content": "864718285734",
1517
+ "lstrip": false,
1518
+ "normalized": false,
1519
+ "rstrip": false,
1520
+ "single_word": false
1521
+ },
1522
+ {
1523
+ "content": "963852916563",
1524
+ "lstrip": false,
1525
+ "normalized": false,
1526
+ "rstrip": false,
1527
+ "single_word": false
1528
+ },
1529
+ {
1530
+ "content": "319584985416",
1531
+ "lstrip": false,
1532
+ "normalized": false,
1533
+ "rstrip": false,
1534
+ "single_word": false
1535
+ },
1536
+ {
1537
+ "content": "563365646341",
1538
+ "lstrip": false,
1539
+ "normalized": false,
1540
+ "rstrip": false,
1541
+ "single_word": false
1542
+ },
1543
+ {
1544
+ "content": "811371928234",
1545
+ "lstrip": false,
1546
+ "normalized": false,
1547
+ "rstrip": false,
1548
+ "single_word": false
1549
+ },
1550
+ {
1551
+ "content": "837131396371",
1552
+ "lstrip": false,
1553
+ "normalized": false,
1554
+ "rstrip": false,
1555
+ "single_word": false
1556
+ },
1557
+ {
1558
+ "content": "267514771964",
1559
+ "lstrip": false,
1560
+ "normalized": false,
1561
+ "rstrip": false,
1562
+ "single_word": false
1563
+ },
1564
+ {
1565
+ "content": "944513428457",
1566
+ "lstrip": false,
1567
+ "normalized": false,
1568
+ "rstrip": false,
1569
+ "single_word": false
1570
+ },
1571
+ {
1572
+ "content": "117298239631",
1573
+ "lstrip": false,
1574
+ "normalized": false,
1575
+ "rstrip": false,
1576
+ "single_word": false
1577
+ },
1578
+ {
1579
+ "content": "158142752582",
1580
+ "lstrip": false,
1581
+ "normalized": false,
1582
+ "rstrip": false,
1583
+ "single_word": false
1584
+ },
1585
+ {
1586
+ "content": "252867443568",
1587
+ "lstrip": false,
1588
+ "normalized": false,
1589
+ "rstrip": false,
1590
+ "single_word": false
1591
+ },
1592
+ {
1593
+ "content": "839269684865",
1594
+ "lstrip": false,
1595
+ "normalized": false,
1596
+ "rstrip": false,
1597
+ "single_word": false
1598
+ },
1599
+ {
1600
+ "content": "612788593128",
1601
+ "lstrip": false,
1602
+ "normalized": false,
1603
+ "rstrip": false,
1604
+ "single_word": false
1605
+ },
1606
+ {
1607
+ "content": "145669731981",
1608
+ "lstrip": false,
1609
+ "normalized": false,
1610
+ "rstrip": false,
1611
+ "single_word": false
1612
+ },
1613
+ {
1614
+ "content": "121557291859",
1615
+ "lstrip": false,
1616
+ "normalized": false,
1617
+ "rstrip": false,
1618
+ "single_word": false
1619
+ },
1620
+ {
1621
+ "content": "245416776926",
1622
+ "lstrip": false,
1623
+ "normalized": false,
1624
+ "rstrip": false,
1625
+ "single_word": false
1626
+ },
1627
+ {
1628
+ "content": "799417897197",
1629
+ "lstrip": false,
1630
+ "normalized": false,
1631
+ "rstrip": false,
1632
+ "single_word": false
1633
+ },
1634
+ {
1635
+ "content": "997958836435",
1636
+ "lstrip": false,
1637
+ "normalized": false,
1638
+ "rstrip": false,
1639
+ "single_word": false
1640
+ },
1641
+ {
1642
+ "content": "892336777248",
1643
+ "lstrip": false,
1644
+ "normalized": false,
1645
+ "rstrip": false,
1646
+ "single_word": false
1647
+ },
1648
+ {
1649
+ "content": "158929292238",
1650
+ "lstrip": false,
1651
+ "normalized": false,
1652
+ "rstrip": false,
1653
+ "single_word": false
1654
+ },
1655
+ {
1656
+ "content": "581976444672",
1657
+ "lstrip": false,
1658
+ "normalized": false,
1659
+ "rstrip": false,
1660
+ "single_word": false
1661
+ },
1662
+ {
1663
+ "content": "897784492783",
1664
+ "lstrip": false,
1665
+ "normalized": false,
1666
+ "rstrip": false,
1667
+ "single_word": false
1668
+ },
1669
+ {
1670
+ "content": "492373714791",
1671
+ "lstrip": false,
1672
+ "normalized": false,
1673
+ "rstrip": false,
1674
+ "single_word": false
1675
+ },
1676
+ {
1677
+ "content": "512659818733",
1678
+ "lstrip": false,
1679
+ "normalized": false,
1680
+ "rstrip": false,
1681
+ "single_word": false
1682
+ },
1683
+ {
1684
+ "content": "881112998642",
1685
+ "lstrip": false,
1686
+ "normalized": false,
1687
+ "rstrip": false,
1688
+ "single_word": false
1689
+ },
1690
+ {
1691
+ "content": "619454958782",
1692
+ "lstrip": false,
1693
+ "normalized": false,
1694
+ "rstrip": false,
1695
+ "single_word": false
1696
+ },
1697
+ {
1698
+ "content": "431149748713",
1699
+ "lstrip": false,
1700
+ "normalized": false,
1701
+ "rstrip": false,
1702
+ "single_word": false
1703
+ },
1704
+ {
1705
+ "content": "624221476921",
1706
+ "lstrip": false,
1707
+ "normalized": false,
1708
+ "rstrip": false,
1709
+ "single_word": false
1710
+ },
1711
+ {
1712
+ "content": "125866399464",
1713
+ "lstrip": false,
1714
+ "normalized": false,
1715
+ "rstrip": false,
1716
+ "single_word": false
1717
+ },
1718
+ {
1719
+ "content": "339882449689",
1720
+ "lstrip": false,
1721
+ "normalized": false,
1722
+ "rstrip": false,
1723
+ "single_word": false
1724
+ },
1725
+ {
1726
+ "content": "186198784585",
1727
+ "lstrip": false,
1728
+ "normalized": false,
1729
+ "rstrip": false,
1730
+ "single_word": false
1731
+ },
1732
+ {
1733
+ "content": "943193294691",
1734
+ "lstrip": false,
1735
+ "normalized": false,
1736
+ "rstrip": false,
1737
+ "single_word": false
1738
+ },
1739
+ {
1740
+ "content": "955668961269",
1741
+ "lstrip": false,
1742
+ "normalized": false,
1743
+ "rstrip": false,
1744
+ "single_word": false
1745
+ },
1746
+ {
1747
+ "content": "232787996724",
1748
+ "lstrip": false,
1749
+ "normalized": false,
1750
+ "rstrip": false,
1751
+ "single_word": false
1752
+ },
1753
+ {
1754
+ "content": "215671314196",
1755
+ "lstrip": false,
1756
+ "normalized": false,
1757
+ "rstrip": false,
1758
+ "single_word": false
1759
+ },
1760
+ {
1761
+ "content": "286173241916",
1762
+ "lstrip": false,
1763
+ "normalized": false,
1764
+ "rstrip": false,
1765
+ "single_word": false
1766
+ },
1767
+ {
1768
+ "content": "745977673725",
1769
+ "lstrip": false,
1770
+ "normalized": false,
1771
+ "rstrip": false,
1772
+ "single_word": false
1773
+ },
1774
+ {
1775
+ "content": "556976448182",
1776
+ "lstrip": false,
1777
+ "normalized": false,
1778
+ "rstrip": false,
1779
+ "single_word": false
1780
+ },
1781
+ {
1782
+ "content": "599961512792",
1783
+ "lstrip": false,
1784
+ "normalized": false,
1785
+ "rstrip": false,
1786
+ "single_word": false
1787
+ },
1788
+ {
1789
+ "content": "766294538337",
1790
+ "lstrip": false,
1791
+ "normalized": false,
1792
+ "rstrip": false,
1793
+ "single_word": false
1794
+ },
1795
+ {
1796
+ "content": "934912591213",
1797
+ "lstrip": false,
1798
+ "normalized": false,
1799
+ "rstrip": false,
1800
+ "single_word": false
1801
+ },
1802
+ {
1803
+ "content": "295118729589",
1804
+ "lstrip": false,
1805
+ "normalized": false,
1806
+ "rstrip": false,
1807
+ "single_word": false
1808
+ },
1809
+ {
1810
+ "content": "529455466433",
1811
+ "lstrip": false,
1812
+ "normalized": false,
1813
+ "rstrip": false,
1814
+ "single_word": false
1815
+ },
1816
+ {
1817
+ "content": "196119929397",
1818
+ "lstrip": false,
1819
+ "normalized": false,
1820
+ "rstrip": false,
1821
+ "single_word": false
1822
+ },
1823
+ {
1824
+ "content": "379571934299",
1825
+ "lstrip": false,
1826
+ "normalized": false,
1827
+ "rstrip": false,
1828
+ "single_word": false
1829
+ },
1830
+ {
1831
+ "content": "251789649997",
1832
+ "lstrip": false,
1833
+ "normalized": false,
1834
+ "rstrip": false,
1835
+ "single_word": false
1836
+ },
1837
+ {
1838
+ "content": "564544131355",
1839
+ "lstrip": false,
1840
+ "normalized": false,
1841
+ "rstrip": false,
1842
+ "single_word": false
1843
+ },
1844
+ {
1845
+ "content": "244371196654",
1846
+ "lstrip": false,
1847
+ "normalized": false,
1848
+ "rstrip": false,
1849
+ "single_word": false
1850
+ },
1851
+ {
1852
+ "content": "384598329253",
1853
+ "lstrip": false,
1854
+ "normalized": false,
1855
+ "rstrip": false,
1856
+ "single_word": false
1857
+ },
1858
+ {
1859
+ "content": "887753195844",
1860
+ "lstrip": false,
1861
+ "normalized": false,
1862
+ "rstrip": false,
1863
+ "single_word": false
1864
+ },
1865
+ {
1866
+ "content": "364947325679",
1867
+ "lstrip": false,
1868
+ "normalized": false,
1869
+ "rstrip": false,
1870
+ "single_word": false
1871
+ },
1872
+ {
1873
+ "content": "655517954651",
1874
+ "lstrip": false,
1875
+ "normalized": false,
1876
+ "rstrip": false,
1877
+ "single_word": false
1878
+ },
1879
+ {
1880
+ "content": "673948786567",
1881
+ "lstrip": false,
1882
+ "normalized": false,
1883
+ "rstrip": false,
1884
+ "single_word": false
1885
+ },
1886
+ {
1887
+ "content": "857231548835",
1888
+ "lstrip": false,
1889
+ "normalized": false,
1890
+ "rstrip": false,
1891
+ "single_word": false
1892
+ },
1893
+ {
1894
+ "content": "816115936673",
1895
+ "lstrip": false,
1896
+ "normalized": false,
1897
+ "rstrip": false,
1898
+ "single_word": false
1899
+ },
1900
+ {
1901
+ "content": "644234165531",
1902
+ "lstrip": false,
1903
+ "normalized": false,
1904
+ "rstrip": false,
1905
+ "single_word": false
1906
+ },
1907
+ {
1908
+ "content": "182782912224",
1909
+ "lstrip": false,
1910
+ "normalized": false,
1911
+ "rstrip": false,
1912
+ "single_word": false
1913
+ },
1914
+ {
1915
+ "content": "234316622259",
1916
+ "lstrip": false,
1917
+ "normalized": false,
1918
+ "rstrip": false,
1919
+ "single_word": false
1920
+ },
1921
+ {
1922
+ "content": "421369185549",
1923
+ "lstrip": false,
1924
+ "normalized": false,
1925
+ "rstrip": false,
1926
+ "single_word": false
1927
+ },
1928
+ {
1929
+ "content": "434632855397",
1930
+ "lstrip": false,
1931
+ "normalized": false,
1932
+ "rstrip": false,
1933
+ "single_word": false
1934
+ },
1935
+ {
1936
+ "content": "921889371893",
1937
+ "lstrip": false,
1938
+ "normalized": false,
1939
+ "rstrip": false,
1940
+ "single_word": false
1941
+ },
1942
+ {
1943
+ "content": "415956914763",
1944
+ "lstrip": false,
1945
+ "normalized": false,
1946
+ "rstrip": false,
1947
+ "single_word": false
1948
+ },
1949
+ {
1950
+ "content": "598916996413",
1951
+ "lstrip": false,
1952
+ "normalized": false,
1953
+ "rstrip": false,
1954
+ "single_word": false
1955
+ },
1956
+ {
1957
+ "content": "773671349113",
1958
+ "lstrip": false,
1959
+ "normalized": false,
1960
+ "rstrip": false,
1961
+ "single_word": false
1962
+ },
1963
+ {
1964
+ "content": "952465217972",
1965
+ "lstrip": false,
1966
+ "normalized": false,
1967
+ "rstrip": false,
1968
+ "single_word": false
1969
+ },
1970
+ {
1971
+ "content": "117657531962",
1972
+ "lstrip": false,
1973
+ "normalized": false,
1974
+ "rstrip": false,
1975
+ "single_word": false
1976
+ },
1977
+ {
1978
+ "content": "729825168745",
1979
+ "lstrip": false,
1980
+ "normalized": false,
1981
+ "rstrip": false,
1982
+ "single_word": false
1983
+ },
1984
+ {
1985
+ "content": "691315125346",
1986
+ "lstrip": false,
1987
+ "normalized": false,
1988
+ "rstrip": false,
1989
+ "single_word": false
1990
+ },
1991
+ {
1992
+ "content": "768461952319",
1993
+ "lstrip": false,
1994
+ "normalized": false,
1995
+ "rstrip": false,
1996
+ "single_word": false
1997
+ },
1998
+ {
1999
+ "content": "664847713559",
2000
+ "lstrip": false,
2001
+ "normalized": false,
2002
+ "rstrip": false,
2003
+ "single_word": false
2004
+ },
2005
+ {
2006
+ "content": "953267689786",
2007
+ "lstrip": false,
2008
+ "normalized": false,
2009
+ "rstrip": false,
2010
+ "single_word": false
2011
+ },
2012
+ {
2013
+ "content": "886464195129",
2014
+ "lstrip": false,
2015
+ "normalized": false,
2016
+ "rstrip": false,
2017
+ "single_word": false
2018
+ },
2019
+ {
2020
+ "content": "824488329416",
2021
+ "lstrip": false,
2022
+ "normalized": false,
2023
+ "rstrip": false,
2024
+ "single_word": false
2025
+ },
2026
+ {
2027
+ "content": "837873762491",
2028
+ "lstrip": false,
2029
+ "normalized": false,
2030
+ "rstrip": false,
2031
+ "single_word": false
2032
+ },
2033
+ {
2034
+ "content": "532833541879",
2035
+ "lstrip": false,
2036
+ "normalized": false,
2037
+ "rstrip": false,
2038
+ "single_word": false
2039
+ },
2040
+ {
2041
+ "content": "669183782449",
2042
+ "lstrip": false,
2043
+ "normalized": false,
2044
+ "rstrip": false,
2045
+ "single_word": false
2046
+ },
2047
+ {
2048
+ "content": "941976537588",
2049
+ "lstrip": false,
2050
+ "normalized": false,
2051
+ "rstrip": false,
2052
+ "single_word": false
2053
+ },
2054
+ {
2055
+ "content": "739394546916",
2056
+ "lstrip": false,
2057
+ "normalized": false,
2058
+ "rstrip": false,
2059
+ "single_word": false
2060
+ },
2061
+ {
2062
+ "content": "267954879268",
2063
+ "lstrip": false,
2064
+ "normalized": false,
2065
+ "rstrip": false,
2066
+ "single_word": false
2067
+ },
2068
+ {
2069
+ "content": "637551427887",
2070
+ "lstrip": false,
2071
+ "normalized": false,
2072
+ "rstrip": false,
2073
+ "single_word": false
2074
+ },
2075
+ {
2076
+ "content": "217756494954",
2077
+ "lstrip": false,
2078
+ "normalized": false,
2079
+ "rstrip": false,
2080
+ "single_word": false
2081
+ },
2082
+ {
2083
+ "content": "524444658383",
2084
+ "lstrip": false,
2085
+ "normalized": false,
2086
+ "rstrip": false,
2087
+ "single_word": false
2088
+ },
2089
+ {
2090
+ "content": "117783274348",
2091
+ "lstrip": false,
2092
+ "normalized": false,
2093
+ "rstrip": false,
2094
+ "single_word": false
2095
+ },
2096
+ {
2097
+ "content": "138218735276",
2098
+ "lstrip": false,
2099
+ "normalized": false,
2100
+ "rstrip": false,
2101
+ "single_word": false
2102
+ },
2103
+ {
2104
+ "content": "814611949491",
2105
+ "lstrip": false,
2106
+ "normalized": false,
2107
+ "rstrip": false,
2108
+ "single_word": false
2109
+ },
2110
+ {
2111
+ "content": "711641973413",
2112
+ "lstrip": false,
2113
+ "normalized": false,
2114
+ "rstrip": false,
2115
+ "single_word": false
2116
+ },
2117
+ {
2118
+ "content": "499156317423",
2119
+ "lstrip": false,
2120
+ "normalized": false,
2121
+ "rstrip": false,
2122
+ "single_word": false
2123
+ },
2124
+ {
2125
+ "content": "515856611931",
2126
+ "lstrip": false,
2127
+ "normalized": false,
2128
+ "rstrip": false,
2129
+ "single_word": false
2130
+ },
2131
+ {
2132
+ "content": "454164859837",
2133
+ "lstrip": false,
2134
+ "normalized": false,
2135
+ "rstrip": false,
2136
+ "single_word": false
2137
+ },
2138
+ {
2139
+ "content": "345271433112",
2140
+ "lstrip": false,
2141
+ "normalized": false,
2142
+ "rstrip": false,
2143
+ "single_word": false
2144
+ },
2145
+ {
2146
+ "content": "462294118988",
2147
+ "lstrip": false,
2148
+ "normalized": false,
2149
+ "rstrip": false,
2150
+ "single_word": false
2151
+ },
2152
+ {
2153
+ "content": "511785788222",
2154
+ "lstrip": false,
2155
+ "normalized": false,
2156
+ "rstrip": false,
2157
+ "single_word": false
2158
+ },
2159
+ {
2160
+ "content": "497294727353",
2161
+ "lstrip": false,
2162
+ "normalized": false,
2163
+ "rstrip": false,
2164
+ "single_word": false
2165
+ },
2166
+ {
2167
+ "content": "866519986723",
2168
+ "lstrip": false,
2169
+ "normalized": false,
2170
+ "rstrip": false,
2171
+ "single_word": false
2172
+ },
2173
+ {
2174
+ "content": "334513529294",
2175
+ "lstrip": false,
2176
+ "normalized": false,
2177
+ "rstrip": false,
2178
+ "single_word": false
2179
+ },
2180
+ {
2181
+ "content": "549946382131",
2182
+ "lstrip": false,
2183
+ "normalized": false,
2184
+ "rstrip": false,
2185
+ "single_word": false
2186
+ },
2187
+ {
2188
+ "content": "284445431422",
2189
+ "lstrip": false,
2190
+ "normalized": false,
2191
+ "rstrip": false,
2192
+ "single_word": false
2193
+ },
2194
+ {
2195
+ "content": "396521188476",
2196
+ "lstrip": false,
2197
+ "normalized": false,
2198
+ "rstrip": false,
2199
+ "single_word": false
2200
+ },
2201
+ {
2202
+ "content": "421435255895",
2203
+ "lstrip": false,
2204
+ "normalized": false,
2205
+ "rstrip": false,
2206
+ "single_word": false
2207
+ },
2208
+ {
2209
+ "content": "133373659361",
2210
+ "lstrip": false,
2211
+ "normalized": false,
2212
+ "rstrip": false,
2213
+ "single_word": false
2214
+ },
2215
+ {
2216
+ "content": "322683334381",
2217
+ "lstrip": false,
2218
+ "normalized": false,
2219
+ "rstrip": false,
2220
+ "single_word": false
2221
+ },
2222
+ {
2223
+ "content": "228358422847",
2224
+ "lstrip": false,
2225
+ "normalized": false,
2226
+ "rstrip": false,
2227
+ "single_word": false
2228
+ },
2229
+ {
2230
+ "content": "291762694874",
2231
+ "lstrip": false,
2232
+ "normalized": false,
2233
+ "rstrip": false,
2234
+ "single_word": false
2235
+ },
2236
+ {
2237
+ "content": "143182978129",
2238
+ "lstrip": false,
2239
+ "normalized": false,
2240
+ "rstrip": false,
2241
+ "single_word": false
2242
+ },
2243
+ {
2244
+ "content": "511923256573",
2245
+ "lstrip": false,
2246
+ "normalized": false,
2247
+ "rstrip": false,
2248
+ "single_word": false
2249
+ },
2250
+ {
2251
+ "content": "327158398268",
2252
+ "lstrip": false,
2253
+ "normalized": false,
2254
+ "rstrip": false,
2255
+ "single_word": false
2256
+ },
2257
+ {
2258
+ "content": "879764613759",
2259
+ "lstrip": false,
2260
+ "normalized": false,
2261
+ "rstrip": false,
2262
+ "single_word": false
2263
+ },
2264
+ {
2265
+ "content": "564395222747",
2266
+ "lstrip": false,
2267
+ "normalized": false,
2268
+ "rstrip": false,
2269
+ "single_word": false
2270
+ },
2271
+ {
2272
+ "content": "451161679736",
2273
+ "lstrip": false,
2274
+ "normalized": false,
2275
+ "rstrip": false,
2276
+ "single_word": false
2277
+ },
2278
+ {
2279
+ "content": "538631466654",
2280
+ "lstrip": false,
2281
+ "normalized": false,
2282
+ "rstrip": false,
2283
+ "single_word": false
2284
+ },
2285
+ {
2286
+ "content": "221762325616",
2287
+ "lstrip": false,
2288
+ "normalized": false,
2289
+ "rstrip": false,
2290
+ "single_word": false
2291
+ },
2292
+ {
2293
+ "content": "218391991184",
2294
+ "lstrip": false,
2295
+ "normalized": false,
2296
+ "rstrip": false,
2297
+ "single_word": false
2298
+ },
2299
+ {
2300
+ "content": "322589379462",
2301
+ "lstrip": false,
2302
+ "normalized": false,
2303
+ "rstrip": false,
2304
+ "single_word": false
2305
+ },
2306
+ {
2307
+ "content": "876537814263",
2308
+ "lstrip": false,
2309
+ "normalized": false,
2310
+ "rstrip": false,
2311
+ "single_word": false
2312
+ },
2313
+ {
2314
+ "content": "152676556624",
2315
+ "lstrip": false,
2316
+ "normalized": false,
2317
+ "rstrip": false,
2318
+ "single_word": false
2319
+ },
2320
+ {
2321
+ "content": "332522971941",
2322
+ "lstrip": false,
2323
+ "normalized": false,
2324
+ "rstrip": false,
2325
+ "single_word": false
2326
+ },
2327
+ {
2328
+ "content": "884354318946",
2329
+ "lstrip": false,
2330
+ "normalized": false,
2331
+ "rstrip": false,
2332
+ "single_word": false
2333
+ },
2334
+ {
2335
+ "content": "513349618943",
2336
+ "lstrip": false,
2337
+ "normalized": false,
2338
+ "rstrip": false,
2339
+ "single_word": false
2340
+ },
2341
+ {
2342
+ "content": "116639746413",
2343
+ "lstrip": false,
2344
+ "normalized": false,
2345
+ "rstrip": false,
2346
+ "single_word": false
2347
+ },
2348
+ {
2349
+ "content": "635185846287",
2350
+ "lstrip": false,
2351
+ "normalized": false,
2352
+ "rstrip": false,
2353
+ "single_word": false
2354
+ },
2355
+ {
2356
+ "content": "993832498489",
2357
+ "lstrip": false,
2358
+ "normalized": false,
2359
+ "rstrip": false,
2360
+ "single_word": false
2361
+ },
2362
+ {
2363
+ "content": "813981174797",
2364
+ "lstrip": false,
2365
+ "normalized": false,
2366
+ "rstrip": false,
2367
+ "single_word": false
2368
+ },
2369
+ {
2370
+ "content": "438745114173",
2371
+ "lstrip": false,
2372
+ "normalized": false,
2373
+ "rstrip": false,
2374
+ "single_word": false
2375
+ },
2376
+ {
2377
+ "content": "983493951323",
2378
+ "lstrip": false,
2379
+ "normalized": false,
2380
+ "rstrip": false,
2381
+ "single_word": false
2382
+ },
2383
+ {
2384
+ "content": "724492262421",
2385
+ "lstrip": false,
2386
+ "normalized": false,
2387
+ "rstrip": false,
2388
+ "single_word": false
2389
+ },
2390
+ {
2391
+ "content": "622553389126",
2392
+ "lstrip": false,
2393
+ "normalized": false,
2394
+ "rstrip": false,
2395
+ "single_word": false
2396
+ },
2397
+ {
2398
+ "content": "889965243135",
2399
+ "lstrip": false,
2400
+ "normalized": false,
2401
+ "rstrip": false,
2402
+ "single_word": false
2403
+ },
2404
+ {
2405
+ "content": "364492359246",
2406
+ "lstrip": false,
2407
+ "normalized": false,
2408
+ "rstrip": false,
2409
+ "single_word": false
2410
+ },
2411
+ {
2412
+ "content": "154962668224",
2413
+ "lstrip": false,
2414
+ "normalized": false,
2415
+ "rstrip": false,
2416
+ "single_word": false
2417
+ },
2418
+ {
2419
+ "content": "179564995814",
2420
+ "lstrip": false,
2421
+ "normalized": false,
2422
+ "rstrip": false,
2423
+ "single_word": false
2424
+ },
2425
+ {
2426
+ "content": "418412875665",
2427
+ "lstrip": false,
2428
+ "normalized": false,
2429
+ "rstrip": false,
2430
+ "single_word": false
2431
+ },
2432
+ {
2433
+ "content": "718951851413",
2434
+ "lstrip": false,
2435
+ "normalized": false,
2436
+ "rstrip": false,
2437
+ "single_word": false
2438
+ },
2439
+ {
2440
+ "content": "699446724178",
2441
+ "lstrip": false,
2442
+ "normalized": false,
2443
+ "rstrip": false,
2444
+ "single_word": false
2445
+ },
2446
+ {
2447
+ "content": "624266421831",
2448
+ "lstrip": false,
2449
+ "normalized": false,
2450
+ "rstrip": false,
2451
+ "single_word": false
2452
+ },
2453
+ {
2454
+ "content": "815458725125",
2455
+ "lstrip": false,
2456
+ "normalized": false,
2457
+ "rstrip": false,
2458
+ "single_word": false
2459
+ },
2460
+ {
2461
+ "content": "455423278865",
2462
+ "lstrip": false,
2463
+ "normalized": false,
2464
+ "rstrip": false,
2465
+ "single_word": false
2466
+ },
2467
+ {
2468
+ "content": "393741199486",
2469
+ "lstrip": false,
2470
+ "normalized": false,
2471
+ "rstrip": false,
2472
+ "single_word": false
2473
+ },
2474
+ {
2475
+ "content": "328552864359",
2476
+ "lstrip": false,
2477
+ "normalized": false,
2478
+ "rstrip": false,
2479
+ "single_word": false
2480
+ },
2481
+ {
2482
+ "content": "211662639865",
2483
+ "lstrip": false,
2484
+ "normalized": false,
2485
+ "rstrip": false,
2486
+ "single_word": false
2487
+ },
2488
+ {
2489
+ "content": "218784516525",
2490
+ "lstrip": false,
2491
+ "normalized": false,
2492
+ "rstrip": false,
2493
+ "single_word": false
2494
+ },
2495
+ {
2496
+ "content": "762486672996",
2497
+ "lstrip": false,
2498
+ "normalized": false,
2499
+ "rstrip": false,
2500
+ "single_word": false
2501
+ },
2502
+ {
2503
+ "content": "142799718159",
2504
+ "lstrip": false,
2505
+ "normalized": false,
2506
+ "rstrip": false,
2507
+ "single_word": false
2508
+ },
2509
+ {
2510
+ "content": "858146415154",
2511
+ "lstrip": false,
2512
+ "normalized": false,
2513
+ "rstrip": false,
2514
+ "single_word": false
2515
+ },
2516
+ {
2517
+ "content": "767858144912",
2518
+ "lstrip": false,
2519
+ "normalized": false,
2520
+ "rstrip": false,
2521
+ "single_word": false
2522
+ },
2523
+ {
2524
+ "content": "571317457151",
2525
+ "lstrip": false,
2526
+ "normalized": false,
2527
+ "rstrip": false,
2528
+ "single_word": false
2529
+ },
2530
+ {
2531
+ "content": "635127952696",
2532
+ "lstrip": false,
2533
+ "normalized": false,
2534
+ "rstrip": false,
2535
+ "single_word": false
2536
+ },
2537
+ {
2538
+ "content": "116427191984",
2539
+ "lstrip": false,
2540
+ "normalized": false,
2541
+ "rstrip": false,
2542
+ "single_word": false
2543
+ },
2544
+ {
2545
+ "content": "268921994538",
2546
+ "lstrip": false,
2547
+ "normalized": false,
2548
+ "rstrip": false,
2549
+ "single_word": false
2550
+ },
2551
+ {
2552
+ "content": "523937669294",
2553
+ "lstrip": false,
2554
+ "normalized": false,
2555
+ "rstrip": false,
2556
+ "single_word": false
2557
+ },
2558
+ {
2559
+ "content": "165429152138",
2560
+ "lstrip": false,
2561
+ "normalized": false,
2562
+ "rstrip": false,
2563
+ "single_word": false
2564
+ },
2565
+ {
2566
+ "content": "739246183345",
2567
+ "lstrip": false,
2568
+ "normalized": false,
2569
+ "rstrip": false,
2570
+ "single_word": false
2571
+ },
2572
+ {
2573
+ "content": "591464355756",
2574
+ "lstrip": false,
2575
+ "normalized": false,
2576
+ "rstrip": false,
2577
+ "single_word": false
2578
+ },
2579
+ {
2580
+ "content": "212985874612",
2581
+ "lstrip": false,
2582
+ "normalized": false,
2583
+ "rstrip": false,
2584
+ "single_word": false
2585
+ },
2586
+ {
2587
+ "content": "191887635211",
2588
+ "lstrip": false,
2589
+ "normalized": false,
2590
+ "rstrip": false,
2591
+ "single_word": false
2592
+ },
2593
+ {
2594
+ "content": "967214577653",
2595
+ "lstrip": false,
2596
+ "normalized": false,
2597
+ "rstrip": false,
2598
+ "single_word": false
2599
+ },
2600
+ {
2601
+ "content": "119342152414",
2602
+ "lstrip": false,
2603
+ "normalized": false,
2604
+ "rstrip": false,
2605
+ "single_word": false
2606
+ },
2607
+ {
2608
+ "content": "946444632795",
2609
+ "lstrip": false,
2610
+ "normalized": false,
2611
+ "rstrip": false,
2612
+ "single_word": false
2613
+ },
2614
+ {
2615
+ "content": "618423867817",
2616
+ "lstrip": false,
2617
+ "normalized": false,
2618
+ "rstrip": false,
2619
+ "single_word": false
2620
+ },
2621
+ {
2622
+ "content": "228565148417",
2623
+ "lstrip": false,
2624
+ "normalized": false,
2625
+ "rstrip": false,
2626
+ "single_word": false
2627
+ },
2628
+ {
2629
+ "content": "729116422489",
2630
+ "lstrip": false,
2631
+ "normalized": false,
2632
+ "rstrip": false,
2633
+ "single_word": false
2634
+ },
2635
+ {
2636
+ "content": "527874729936",
2637
+ "lstrip": false,
2638
+ "normalized": false,
2639
+ "rstrip": false,
2640
+ "single_word": false
2641
+ },
2642
+ {
2643
+ "content": "739784153482",
2644
+ "lstrip": false,
2645
+ "normalized": false,
2646
+ "rstrip": false,
2647
+ "single_word": false
2648
+ },
2649
+ {
2650
+ "content": "387763951128",
2651
+ "lstrip": false,
2652
+ "normalized": false,
2653
+ "rstrip": false,
2654
+ "single_word": false
2655
+ },
2656
+ {
2657
+ "content": "331369926711",
2658
+ "lstrip": false,
2659
+ "normalized": false,
2660
+ "rstrip": false,
2661
+ "single_word": false
2662
+ },
2663
+ {
2664
+ "content": "562716493614",
2665
+ "lstrip": false,
2666
+ "normalized": false,
2667
+ "rstrip": false,
2668
+ "single_word": false
2669
+ },
2670
+ {
2671
+ "content": "739667844957",
2672
+ "lstrip": false,
2673
+ "normalized": false,
2674
+ "rstrip": false,
2675
+ "single_word": false
2676
+ },
2677
+ {
2678
+ "content": "562389434565",
2679
+ "lstrip": false,
2680
+ "normalized": false,
2681
+ "rstrip": false,
2682
+ "single_word": false
2683
+ },
2684
+ {
2685
+ "content": "256497188281",
2686
+ "lstrip": false,
2687
+ "normalized": false,
2688
+ "rstrip": false,
2689
+ "single_word": false
2690
+ },
2691
+ {
2692
+ "content": "859927364588",
2693
+ "lstrip": false,
2694
+ "normalized": false,
2695
+ "rstrip": false,
2696
+ "single_word": false
2697
+ },
2698
+ {
2699
+ "content": "417668946583",
2700
+ "lstrip": false,
2701
+ "normalized": false,
2702
+ "rstrip": false,
2703
+ "single_word": false
2704
+ },
2705
+ {
2706
+ "content": "357621613582",
2707
+ "lstrip": false,
2708
+ "normalized": false,
2709
+ "rstrip": false,
2710
+ "single_word": false
2711
+ },
2712
+ {
2713
+ "content": "438435178228",
2714
+ "lstrip": false,
2715
+ "normalized": false,
2716
+ "rstrip": false,
2717
+ "single_word": false
2718
+ },
2719
+ {
2720
+ "content": "485692541169",
2721
+ "lstrip": false,
2722
+ "normalized": false,
2723
+ "rstrip": false,
2724
+ "single_word": false
2725
+ },
2726
+ {
2727
+ "content": "825815739116",
2728
+ "lstrip": false,
2729
+ "normalized": false,
2730
+ "rstrip": false,
2731
+ "single_word": false
2732
+ },
2733
+ {
2734
+ "content": "342221452223",
2735
+ "lstrip": false,
2736
+ "normalized": false,
2737
+ "rstrip": false,
2738
+ "single_word": false
2739
+ },
2740
+ {
2741
+ "content": "697747991249",
2742
+ "lstrip": false,
2743
+ "normalized": false,
2744
+ "rstrip": false,
2745
+ "single_word": false
2746
+ },
2747
+ {
2748
+ "content": "716763689965",
2749
+ "lstrip": false,
2750
+ "normalized": false,
2751
+ "rstrip": false,
2752
+ "single_word": false
2753
+ },
2754
+ {
2755
+ "content": "141499982867",
2756
+ "lstrip": false,
2757
+ "normalized": false,
2758
+ "rstrip": false,
2759
+ "single_word": false
2760
+ },
2761
+ {
2762
+ "content": "818479319499",
2763
+ "lstrip": false,
2764
+ "normalized": false,
2765
+ "rstrip": false,
2766
+ "single_word": false
2767
+ },
2768
+ {
2769
+ "content": "336813343298",
2770
+ "lstrip": false,
2771
+ "normalized": false,
2772
+ "rstrip": false,
2773
+ "single_word": false
2774
+ },
2775
+ {
2776
+ "content": "594688742928",
2777
+ "lstrip": false,
2778
+ "normalized": false,
2779
+ "rstrip": false,
2780
+ "single_word": false
2781
+ },
2782
+ {
2783
+ "content": "472129283475",
2784
+ "lstrip": false,
2785
+ "normalized": false,
2786
+ "rstrip": false,
2787
+ "single_word": false
2788
+ },
2789
+ {
2790
+ "content": "514354144759",
2791
+ "lstrip": false,
2792
+ "normalized": false,
2793
+ "rstrip": false,
2794
+ "single_word": false
2795
+ },
2796
+ {
2797
+ "content": "349249721685",
2798
+ "lstrip": false,
2799
+ "normalized": false,
2800
+ "rstrip": false,
2801
+ "single_word": false
2802
+ },
2803
+ {
2804
+ "content": "546276298359",
2805
+ "lstrip": false,
2806
+ "normalized": false,
2807
+ "rstrip": false,
2808
+ "single_word": false
2809
+ },
2810
+ {
2811
+ "content": "353755529131",
2812
+ "lstrip": false,
2813
+ "normalized": false,
2814
+ "rstrip": false,
2815
+ "single_word": false
2816
+ },
2817
+ {
2818
+ "content": "315534574435",
2819
+ "lstrip": false,
2820
+ "normalized": false,
2821
+ "rstrip": false,
2822
+ "single_word": false
2823
+ },
2824
+ {
2825
+ "content": "523723475786",
2826
+ "lstrip": false,
2827
+ "normalized": false,
2828
+ "rstrip": false,
2829
+ "single_word": false
2830
+ },
2831
+ {
2832
+ "content": "215826764872",
2833
+ "lstrip": false,
2834
+ "normalized": false,
2835
+ "rstrip": false,
2836
+ "single_word": false
2837
+ },
2838
+ {
2839
+ "content": "367968398551",
2840
+ "lstrip": false,
2841
+ "normalized": false,
2842
+ "rstrip": false,
2843
+ "single_word": false
2844
+ },
2845
+ {
2846
+ "content": "569853653352",
2847
+ "lstrip": false,
2848
+ "normalized": false,
2849
+ "rstrip": false,
2850
+ "single_word": false
2851
+ },
2852
+ {
2853
+ "content": "389715484387",
2854
+ "lstrip": false,
2855
+ "normalized": false,
2856
+ "rstrip": false,
2857
+ "single_word": false
2858
+ },
2859
+ {
2860
+ "content": "293847485454",
2861
+ "lstrip": false,
2862
+ "normalized": false,
2863
+ "rstrip": false,
2864
+ "single_word": false
2865
+ },
2866
+ {
2867
+ "content": "714738141818",
2868
+ "lstrip": false,
2869
+ "normalized": false,
2870
+ "rstrip": false,
2871
+ "single_word": false
2872
+ },
2873
+ {
2874
+ "content": "178478368922",
2875
+ "lstrip": false,
2876
+ "normalized": false,
2877
+ "rstrip": false,
2878
+ "single_word": false
2879
+ },
2880
+ {
2881
+ "content": "581493616981",
2882
+ "lstrip": false,
2883
+ "normalized": false,
2884
+ "rstrip": false,
2885
+ "single_word": false
2886
+ },
2887
+ {
2888
+ "content": "589439538674",
2889
+ "lstrip": false,
2890
+ "normalized": false,
2891
+ "rstrip": false,
2892
+ "single_word": false
2893
+ },
2894
+ {
2895
+ "content": "846657726193",
2896
+ "lstrip": false,
2897
+ "normalized": false,
2898
+ "rstrip": false,
2899
+ "single_word": false
2900
+ },
2901
+ {
2902
+ "content": "722339992679",
2903
+ "lstrip": false,
2904
+ "normalized": false,
2905
+ "rstrip": false,
2906
+ "single_word": false
2907
+ },
2908
+ {
2909
+ "content": "138154781148",
2910
+ "lstrip": false,
2911
+ "normalized": false,
2912
+ "rstrip": false,
2913
+ "single_word": false
2914
+ },
2915
+ {
2916
+ "content": "757785319772",
2917
+ "lstrip": false,
2918
+ "normalized": false,
2919
+ "rstrip": false,
2920
+ "single_word": false
2921
+ },
2922
+ {
2923
+ "content": "492516914298",
2924
+ "lstrip": false,
2925
+ "normalized": false,
2926
+ "rstrip": false,
2927
+ "single_word": false
2928
+ },
2929
+ {
2930
+ "content": "919181521716",
2931
+ "lstrip": false,
2932
+ "normalized": false,
2933
+ "rstrip": false,
2934
+ "single_word": false
2935
+ },
2936
+ {
2937
+ "content": "985781138935",
2938
+ "lstrip": false,
2939
+ "normalized": false,
2940
+ "rstrip": false,
2941
+ "single_word": false
2942
+ },
2943
+ {
2944
+ "content": "476969195485",
2945
+ "lstrip": false,
2946
+ "normalized": false,
2947
+ "rstrip": false,
2948
+ "single_word": false
2949
+ },
2950
+ {
2951
+ "content": "313145133463",
2952
+ "lstrip": false,
2953
+ "normalized": false,
2954
+ "rstrip": false,
2955
+ "single_word": false
2956
+ },
2957
+ {
2958
+ "content": "758963111966",
2959
+ "lstrip": false,
2960
+ "normalized": false,
2961
+ "rstrip": false,
2962
+ "single_word": false
2963
+ },
2964
+ {
2965
+ "content": "147541537162",
2966
+ "lstrip": false,
2967
+ "normalized": false,
2968
+ "rstrip": false,
2969
+ "single_word": false
2970
+ },
2971
+ {
2972
+ "content": "557163366873",
2973
+ "lstrip": false,
2974
+ "normalized": false,
2975
+ "rstrip": false,
2976
+ "single_word": false
2977
+ },
2978
+ {
2979
+ "content": "144373897488",
2980
+ "lstrip": false,
2981
+ "normalized": false,
2982
+ "rstrip": false,
2983
+ "single_word": false
2984
+ },
2985
+ {
2986
+ "content": "522515164754",
2987
+ "lstrip": false,
2988
+ "normalized": false,
2989
+ "rstrip": false,
2990
+ "single_word": false
2991
+ },
2992
+ {
2993
+ "content": "724964923582",
2994
+ "lstrip": false,
2995
+ "normalized": false,
2996
+ "rstrip": false,
2997
+ "single_word": false
2998
+ },
2999
+ {
3000
+ "content": "284776712475",
3001
+ "lstrip": false,
3002
+ "normalized": false,
3003
+ "rstrip": false,
3004
+ "single_word": false
3005
+ },
3006
+ {
3007
+ "content": "375429755114",
3008
+ "lstrip": false,
3009
+ "normalized": false,
3010
+ "rstrip": false,
3011
+ "single_word": false
3012
+ },
3013
+ {
3014
+ "content": "181233596124",
3015
+ "lstrip": false,
3016
+ "normalized": false,
3017
+ "rstrip": false,
3018
+ "single_word": false
3019
+ },
3020
+ {
3021
+ "content": "948585673431",
3022
+ "lstrip": false,
3023
+ "normalized": false,
3024
+ "rstrip": false,
3025
+ "single_word": false
3026
+ },
3027
+ {
3028
+ "content": "243165586174",
3029
+ "lstrip": false,
3030
+ "normalized": false,
3031
+ "rstrip": false,
3032
+ "single_word": false
3033
+ },
3034
+ {
3035
+ "content": "396847976144",
3036
+ "lstrip": false,
3037
+ "normalized": false,
3038
+ "rstrip": false,
3039
+ "single_word": false
3040
+ },
3041
+ {
3042
+ "content": "997724962668",
3043
+ "lstrip": false,
3044
+ "normalized": false,
3045
+ "rstrip": false,
3046
+ "single_word": false
3047
+ },
3048
+ {
3049
+ "content": "558837194455",
3050
+ "lstrip": false,
3051
+ "normalized": false,
3052
+ "rstrip": false,
3053
+ "single_word": false
3054
+ },
3055
+ {
3056
+ "content": "163165456396",
3057
+ "lstrip": false,
3058
+ "normalized": false,
3059
+ "rstrip": false,
3060
+ "single_word": false
3061
+ },
3062
+ {
3063
+ "content": "378749551722",
3064
+ "lstrip": false,
3065
+ "normalized": false,
3066
+ "rstrip": false,
3067
+ "single_word": false
3068
+ },
3069
+ {
3070
+ "content": "161238482259",
3071
+ "lstrip": false,
3072
+ "normalized": false,
3073
+ "rstrip": false,
3074
+ "single_word": false
3075
+ },
3076
+ {
3077
+ "content": "754978243758",
3078
+ "lstrip": false,
3079
+ "normalized": false,
3080
+ "rstrip": false,
3081
+ "single_word": false
3082
+ },
3083
+ {
3084
+ "content": "195388849133",
3085
+ "lstrip": false,
3086
+ "normalized": false,
3087
+ "rstrip": false,
3088
+ "single_word": false
3089
+ },
3090
+ {
3091
+ "content": "229775525672",
3092
+ "lstrip": false,
3093
+ "normalized": false,
3094
+ "rstrip": false,
3095
+ "single_word": false
3096
+ },
3097
+ {
3098
+ "content": "262437452884",
3099
+ "lstrip": false,
3100
+ "normalized": false,
3101
+ "rstrip": false,
3102
+ "single_word": false
3103
+ },
3104
+ {
3105
+ "content": "441377892146",
3106
+ "lstrip": false,
3107
+ "normalized": false,
3108
+ "rstrip": false,
3109
+ "single_word": false
3110
+ },
3111
+ {
3112
+ "content": "451885565366",
3113
+ "lstrip": false,
3114
+ "normalized": false,
3115
+ "rstrip": false,
3116
+ "single_word": false
3117
+ },
3118
+ {
3119
+ "content": "981277526855",
3120
+ "lstrip": false,
3121
+ "normalized": false,
3122
+ "rstrip": false,
3123
+ "single_word": false
3124
+ },
3125
+ {
3126
+ "content": "762495822823",
3127
+ "lstrip": false,
3128
+ "normalized": false,
3129
+ "rstrip": false,
3130
+ "single_word": false
3131
+ },
3132
+ {
3133
+ "content": "368763327262",
3134
+ "lstrip": false,
3135
+ "normalized": false,
3136
+ "rstrip": false,
3137
+ "single_word": false
3138
+ },
3139
+ {
3140
+ "content": "757422791351",
3141
+ "lstrip": false,
3142
+ "normalized": false,
3143
+ "rstrip": false,
3144
+ "single_word": false
3145
+ },
3146
+ {
3147
+ "content": "636324136426",
3148
+ "lstrip": false,
3149
+ "normalized": false,
3150
+ "rstrip": false,
3151
+ "single_word": false
3152
+ },
3153
+ {
3154
+ "content": "214193645583",
3155
+ "lstrip": false,
3156
+ "normalized": false,
3157
+ "rstrip": false,
3158
+ "single_word": false
3159
+ },
3160
+ {
3161
+ "content": "412843856172",
3162
+ "lstrip": false,
3163
+ "normalized": false,
3164
+ "rstrip": false,
3165
+ "single_word": false
3166
+ },
3167
+ {
3168
+ "content": "179386156569",
3169
+ "lstrip": false,
3170
+ "normalized": false,
3171
+ "rstrip": false,
3172
+ "single_word": false
3173
+ },
3174
+ {
3175
+ "content": "756916173536",
3176
+ "lstrip": false,
3177
+ "normalized": false,
3178
+ "rstrip": false,
3179
+ "single_word": false
3180
+ },
3181
+ {
3182
+ "content": "892697125149",
3183
+ "lstrip": false,
3184
+ "normalized": false,
3185
+ "rstrip": false,
3186
+ "single_word": false
3187
+ },
3188
+ {
3189
+ "content": "625334487352",
3190
+ "lstrip": false,
3191
+ "normalized": false,
3192
+ "rstrip": false,
3193
+ "single_word": false
3194
+ },
3195
+ {
3196
+ "content": "941861857715",
3197
+ "lstrip": false,
3198
+ "normalized": false,
3199
+ "rstrip": false,
3200
+ "single_word": false
3201
+ },
3202
+ {
3203
+ "content": "887417525236",
3204
+ "lstrip": false,
3205
+ "normalized": false,
3206
+ "rstrip": false,
3207
+ "single_word": false
3208
+ },
3209
+ {
3210
+ "content": "649516938598",
3211
+ "lstrip": false,
3212
+ "normalized": false,
3213
+ "rstrip": false,
3214
+ "single_word": false
3215
+ },
3216
+ {
3217
+ "content": "717628619782",
3218
+ "lstrip": false,
3219
+ "normalized": false,
3220
+ "rstrip": false,
3221
+ "single_word": false
3222
+ },
3223
+ {
3224
+ "content": "438124184139",
3225
+ "lstrip": false,
3226
+ "normalized": false,
3227
+ "rstrip": false,
3228
+ "single_word": false
3229
+ },
3230
+ {
3231
+ "content": "547563892268",
3232
+ "lstrip": false,
3233
+ "normalized": false,
3234
+ "rstrip": false,
3235
+ "single_word": false
3236
+ },
3237
+ {
3238
+ "content": "856317483891",
3239
+ "lstrip": false,
3240
+ "normalized": false,
3241
+ "rstrip": false,
3242
+ "single_word": false
3243
+ },
3244
+ {
3245
+ "content": "313313831273",
3246
+ "lstrip": false,
3247
+ "normalized": false,
3248
+ "rstrip": false,
3249
+ "single_word": false
3250
+ },
3251
+ {
3252
+ "content": "371496153876",
3253
+ "lstrip": false,
3254
+ "normalized": false,
3255
+ "rstrip": false,
3256
+ "single_word": false
3257
+ },
3258
+ {
3259
+ "content": "587541149322",
3260
+ "lstrip": false,
3261
+ "normalized": false,
3262
+ "rstrip": false,
3263
+ "single_word": false
3264
+ },
3265
+ {
3266
+ "content": "265847332563",
3267
+ "lstrip": false,
3268
+ "normalized": false,
3269
+ "rstrip": false,
3270
+ "single_word": false
3271
+ },
3272
+ {
3273
+ "content": "449549215429",
3274
+ "lstrip": false,
3275
+ "normalized": false,
3276
+ "rstrip": false,
3277
+ "single_word": false
3278
+ },
3279
+ {
3280
+ "content": "163497196769",
3281
+ "lstrip": false,
3282
+ "normalized": false,
3283
+ "rstrip": false,
3284
+ "single_word": false
3285
+ },
3286
+ {
3287
+ "content": "861342291298",
3288
+ "lstrip": false,
3289
+ "normalized": false,
3290
+ "rstrip": false,
3291
+ "single_word": false
3292
+ },
3293
+ {
3294
+ "content": "268433315926",
3295
+ "lstrip": false,
3296
+ "normalized": false,
3297
+ "rstrip": false,
3298
+ "single_word": false
3299
+ },
3300
+ {
3301
+ "content": "774679513717",
3302
+ "lstrip": false,
3303
+ "normalized": false,
3304
+ "rstrip": false,
3305
+ "single_word": false
3306
+ },
3307
+ {
3308
+ "content": "851254219729",
3309
+ "lstrip": false,
3310
+ "normalized": false,
3311
+ "rstrip": false,
3312
+ "single_word": false
3313
+ },
3314
+ {
3315
+ "content": "583527834464",
3316
+ "lstrip": false,
3317
+ "normalized": false,
3318
+ "rstrip": false,
3319
+ "single_word": false
3320
+ },
3321
+ {
3322
+ "content": "488496781997",
3323
+ "lstrip": false,
3324
+ "normalized": false,
3325
+ "rstrip": false,
3326
+ "single_word": false
3327
+ },
3328
+ {
3329
+ "content": "556814553861",
3330
+ "lstrip": false,
3331
+ "normalized": false,
3332
+ "rstrip": false,
3333
+ "single_word": false
3334
+ },
3335
+ {
3336
+ "content": "482829231639",
3337
+ "lstrip": false,
3338
+ "normalized": false,
3339
+ "rstrip": false,
3340
+ "single_word": false
3341
+ },
3342
+ {
3343
+ "content": "618878266619",
3344
+ "lstrip": false,
3345
+ "normalized": false,
3346
+ "rstrip": false,
3347
+ "single_word": false
3348
+ },
3349
+ {
3350
+ "content": "147444452794",
3351
+ "lstrip": false,
3352
+ "normalized": false,
3353
+ "rstrip": false,
3354
+ "single_word": false
3355
+ },
3356
+ {
3357
+ "content": "949235426629",
3358
+ "lstrip": false,
3359
+ "normalized": false,
3360
+ "rstrip": false,
3361
+ "single_word": false
3362
+ },
3363
+ {
3364
+ "content": "357299947518",
3365
+ "lstrip": false,
3366
+ "normalized": false,
3367
+ "rstrip": false,
3368
+ "single_word": false
3369
+ },
3370
+ {
3371
+ "content": "175528632226",
3372
+ "lstrip": false,
3373
+ "normalized": false,
3374
+ "rstrip": false,
3375
+ "single_word": false
3376
+ },
3377
+ {
3378
+ "content": "645527857972",
3379
+ "lstrip": false,
3380
+ "normalized": false,
3381
+ "rstrip": false,
3382
+ "single_word": false
3383
+ },
3384
+ {
3385
+ "content": "186872457894",
3386
+ "lstrip": false,
3387
+ "normalized": false,
3388
+ "rstrip": false,
3389
+ "single_word": false
3390
+ },
3391
+ {
3392
+ "content": "552738847828",
3393
+ "lstrip": false,
3394
+ "normalized": false,
3395
+ "rstrip": false,
3396
+ "single_word": false
3397
+ },
3398
+ {
3399
+ "content": "626748382482",
3400
+ "lstrip": false,
3401
+ "normalized": false,
3402
+ "rstrip": false,
3403
+ "single_word": false
3404
+ },
3405
+ {
3406
+ "content": "921894985642",
3407
+ "lstrip": false,
3408
+ "normalized": false,
3409
+ "rstrip": false,
3410
+ "single_word": false
3411
+ },
3412
+ {
3413
+ "content": "943878645871",
3414
+ "lstrip": false,
3415
+ "normalized": false,
3416
+ "rstrip": false,
3417
+ "single_word": false
3418
+ },
3419
+ {
3420
+ "content": "859289776479",
3421
+ "lstrip": false,
3422
+ "normalized": false,
3423
+ "rstrip": false,
3424
+ "single_word": false
3425
+ },
3426
+ {
3427
+ "content": "614583493135",
3428
+ "lstrip": false,
3429
+ "normalized": false,
3430
+ "rstrip": false,
3431
+ "single_word": false
3432
+ },
3433
+ {
3434
+ "content": "933775286797",
3435
+ "lstrip": false,
3436
+ "normalized": false,
3437
+ "rstrip": false,
3438
+ "single_word": false
3439
+ },
3440
+ {
3441
+ "content": "332234613346",
3442
+ "lstrip": false,
3443
+ "normalized": false,
3444
+ "rstrip": false,
3445
+ "single_word": false
3446
+ },
3447
+ {
3448
+ "content": "325196781219",
3449
+ "lstrip": false,
3450
+ "normalized": false,
3451
+ "rstrip": false,
3452
+ "single_word": false
3453
+ },
3454
+ {
3455
+ "content": "142526557681",
3456
+ "lstrip": false,
3457
+ "normalized": false,
3458
+ "rstrip": false,
3459
+ "single_word": false
3460
+ },
3461
+ {
3462
+ "content": "356722692178",
3463
+ "lstrip": false,
3464
+ "normalized": false,
3465
+ "rstrip": false,
3466
+ "single_word": false
3467
+ },
3468
+ {
3469
+ "content": "449318681694",
3470
+ "lstrip": false,
3471
+ "normalized": false,
3472
+ "rstrip": false,
3473
+ "single_word": false
3474
+ },
3475
+ {
3476
+ "content": "687284547244",
3477
+ "lstrip": false,
3478
+ "normalized": false,
3479
+ "rstrip": false,
3480
+ "single_word": false
3481
+ },
3482
+ {
3483
+ "content": "947262995132",
3484
+ "lstrip": false,
3485
+ "normalized": false,
3486
+ "rstrip": false,
3487
+ "single_word": false
3488
+ },
3489
+ {
3490
+ "content": "893974619684",
3491
+ "lstrip": false,
3492
+ "normalized": false,
3493
+ "rstrip": false,
3494
+ "single_word": false
3495
+ },
3496
+ {
3497
+ "content": "797238311233",
3498
+ "lstrip": false,
3499
+ "normalized": false,
3500
+ "rstrip": false,
3501
+ "single_word": false
3502
+ }
3503
+ ],
3504
+ "eos_token": {
3505
+ "content": "<|endoftext|>",
3506
+ "lstrip": false,
3507
+ "normalized": false,
3508
+ "rstrip": false,
3509
+ "single_word": false
3510
+ },
3511
+ "pad_token": {
3512
+ "content": "<|endoftext|>",
3513
+ "lstrip": false,
3514
+ "normalized": false,
3515
+ "rstrip": false,
3516
+ "single_word": false
3517
+ }
3518
+ }
qwen1.5-moe-a2.7b/base_attn_v2/tokenizer/tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff
 
qwen1.5-moe-a2.7b/base_attn_v2/tokenizer/vocab.json ADDED
The diff for this file is too large to render. See raw diff
 
qwen1.5-moe-a2.7b/base_attn_v2/train.log ADDED
@@ -0,0 +1,276 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 2026-04-03 05:45:12,401 [INFO] new_opacus_codex.train_steps: epoch=1 step=5 loss=24.5791
2
+ 2026-04-03 05:46:11,620 [INFO] new_opacus_codex.train_steps: epoch=1 step=10 loss=23.5678
3
+ 2026-04-03 05:46:30,113 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=10 eval_loss=19.9224 duration_sec=18.49
4
+ 2026-04-03 05:47:28,676 [INFO] new_opacus_codex.train_steps: epoch=1 step=15 loss=19.8919
5
+ 2026-04-03 05:48:41,386 [INFO] new_opacus_codex.train_steps: epoch=1 step=20 loss=14.0147
6
+ 2026-04-03 05:48:59,717 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=20 eval_loss=8.5965 duration_sec=14.54
7
+ 2026-04-03 05:49:59,529 [INFO] new_opacus_codex.train_steps: epoch=1 step=25 loss=9.1039
8
+ 2026-04-03 05:50:56,139 [INFO] new_opacus_codex.train_steps: epoch=1 step=30 loss=6.0981
9
+ 2026-04-03 05:51:33,185 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=30 eval_loss=3.7346 duration_sec=22.57
10
+ 2026-04-03 05:52:33,421 [INFO] new_opacus_codex.train_steps: epoch=1 step=35 loss=3.9861
11
+ 2026-04-03 05:53:32,931 [INFO] new_opacus_codex.train_steps: epoch=1 step=40 loss=2.8689
12
+ 2026-04-03 05:53:48,364 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=40 eval_loss=2.1162 duration_sec=15.42
13
+ 2026-04-03 05:54:48,584 [INFO] new_opacus_codex.train_steps: epoch=1 step=45 loss=2.3364
14
+ 2026-04-03 05:55:55,420 [INFO] new_opacus_codex.train_steps: epoch=1 step=50 loss=1.9852
15
+ 2026-04-03 05:56:30,411 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=50 eval_loss=1.6360 duration_sec=34.96
16
+ 2026-04-03 05:57:54,660 [INFO] new_opacus_codex.train_steps: epoch=1 step=55 loss=1.7469
17
+ 2026-04-03 05:58:55,014 [INFO] new_opacus_codex.train_steps: epoch=1 step=60 loss=1.5727
18
+ 2026-04-03 05:59:13,269 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=60 eval_loss=1.4312 duration_sec=18.24
19
+ 2026-04-03 06:00:10,549 [INFO] new_opacus_codex.train_steps: epoch=1 step=65 loss=1.5232
20
+ 2026-04-03 06:01:11,386 [INFO] new_opacus_codex.train_steps: epoch=1 step=70 loss=1.4675
21
+ 2026-04-03 06:01:28,900 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=70 eval_loss=1.3039 duration_sec=17.50
22
+ 2026-04-03 06:02:30,966 [INFO] new_opacus_codex.train_steps: epoch=1 step=75 loss=1.4180
23
+ 2026-04-03 06:03:31,841 [INFO] new_opacus_codex.train_steps: epoch=1 step=80 loss=1.3651
24
+ 2026-04-03 06:03:46,449 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=80 eval_loss=1.2296 duration_sec=14.59
25
+ 2026-04-03 06:04:48,309 [INFO] new_opacus_codex.train_steps: epoch=1 step=85 loss=1.2886
26
+ 2026-04-03 06:06:57,547 [INFO] new_opacus_codex.train_steps: epoch=1 step=90 loss=1.3373
27
+ 2026-04-03 06:07:15,854 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=90 eval_loss=1.1780 duration_sec=18.28
28
+ 2026-04-03 06:08:17,640 [INFO] new_opacus_codex.train_steps: epoch=1 step=95 loss=1.3727
29
+ 2026-04-03 06:09:21,090 [INFO] new_opacus_codex.train_steps: epoch=1 step=100 loss=1.3008
30
+ 2026-04-03 06:09:54,650 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=100 eval_loss=1.1475 duration_sec=33.54
31
+ 2026-04-03 06:10:57,916 [INFO] new_opacus_codex.train_steps: epoch=1 step=105 loss=1.2396
32
+ 2026-04-03 06:11:56,024 [INFO] new_opacus_codex.train_steps: epoch=1 step=110 loss=1.1881
33
+ 2026-04-03 06:12:16,019 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=110 eval_loss=1.1140 duration_sec=19.98
34
+ 2026-04-03 06:13:22,000 [INFO] new_opacus_codex.train_steps: epoch=1 step=115 loss=1.2082
35
+ 2026-04-03 06:14:27,350 [INFO] new_opacus_codex.train_steps: epoch=1 step=120 loss=1.2407
36
+ 2026-04-03 06:14:45,164 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=120 eval_loss=1.0909 duration_sec=17.80
37
+ 2026-04-03 06:15:48,253 [INFO] new_opacus_codex.train_steps: epoch=1 step=125 loss=1.1782
38
+ 2026-04-03 06:16:53,068 [INFO] new_opacus_codex.train_steps: epoch=1 step=130 loss=1.1674
39
+ 2026-04-03 06:17:15,936 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=130 eval_loss=1.0761 duration_sec=22.85
40
+ 2026-04-03 06:18:21,261 [INFO] new_opacus_codex.train_steps: epoch=1 step=135 loss=1.2230
41
+ 2026-04-03 06:19:41,302 [INFO] new_opacus_codex.train_steps: epoch=1 step=140 loss=1.1754
42
+ 2026-04-03 06:19:59,116 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=140 eval_loss=1.0598 duration_sec=17.79
43
+ 2026-04-03 06:21:02,204 [INFO] new_opacus_codex.train_steps: epoch=1 step=145 loss=1.0591
44
+ 2026-04-03 06:22:07,789 [INFO] new_opacus_codex.train_steps: epoch=1 step=150 loss=1.0390
45
+ 2026-04-03 06:22:44,014 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=150 eval_loss=1.0433 duration_sec=17.07
46
+ 2026-04-03 06:23:48,257 [INFO] new_opacus_codex.train_steps: epoch=1 step=155 loss=1.1013
47
+ 2026-04-03 06:24:52,256 [INFO] new_opacus_codex.train_steps: epoch=1 step=160 loss=1.1262
48
+ 2026-04-03 06:25:11,396 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=160 eval_loss=1.0283 duration_sec=19.10
49
+ 2026-04-03 06:26:14,481 [INFO] new_opacus_codex.train_steps: epoch=1 step=165 loss=1.0759
50
+ 2026-04-03 06:27:28,938 [INFO] new_opacus_codex.train_steps: epoch=1 step=170 loss=1.0525
51
+ 2026-04-03 06:28:31,741 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=170 eval_loss=1.0158 duration_sec=62.79
52
+ 2026-04-03 06:29:36,903 [INFO] new_opacus_codex.train_steps: epoch=1 step=175 loss=1.1217
53
+ 2026-04-03 06:30:41,009 [INFO] new_opacus_codex.train_steps: epoch=1 step=180 loss=1.1333
54
+ 2026-04-03 06:31:19,553 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=180 eval_loss=1.0124 duration_sec=38.49
55
+ 2026-04-03 06:33:19,874 [INFO] new_opacus_codex.train_steps: epoch=2 step=185 loss=1.0488
56
+ 2026-04-03 06:34:24,517 [INFO] new_opacus_codex.train_steps: epoch=2 step=190 loss=1.0728
57
+ 2026-04-03 06:34:42,672 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=190 eval_loss=1.0064 duration_sec=18.14
58
+ 2026-04-03 06:36:00,512 [INFO] new_opacus_codex.train_steps: epoch=2 step=195 loss=0.9642
59
+ 2026-04-03 06:37:17,065 [INFO] new_opacus_codex.train_steps: epoch=2 step=200 loss=0.9623
60
+ 2026-04-03 06:37:35,910 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=200 eval_loss=0.9984 duration_sec=18.83
61
+ 2026-04-03 06:38:39,736 [INFO] new_opacus_codex.train_steps: epoch=2 step=205 loss=0.9738
62
+ 2026-04-03 06:39:43,108 [INFO] new_opacus_codex.train_steps: epoch=2 step=210 loss=0.9230
63
+ 2026-04-03 06:40:07,193 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=210 eval_loss=0.9934 duration_sec=24.07
64
+ 2026-04-03 06:41:11,156 [INFO] new_opacus_codex.train_steps: epoch=2 step=215 loss=0.9798
65
+ 2026-04-03 06:42:14,222 [INFO] new_opacus_codex.train_steps: epoch=2 step=220 loss=0.9112
66
+ 2026-04-03 06:42:44,390 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=220 eval_loss=0.9908 duration_sec=30.16
67
+ 2026-04-03 06:43:46,856 [INFO] new_opacus_codex.train_steps: epoch=2 step=225 loss=0.8484
68
+ 2026-04-03 06:44:51,246 [INFO] new_opacus_codex.train_steps: epoch=2 step=230 loss=0.8635
69
+ 2026-04-03 06:45:09,600 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=230 eval_loss=0.9936 duration_sec=18.32
70
+ 2026-04-03 06:46:12,804 [INFO] new_opacus_codex.train_steps: epoch=2 step=235 loss=0.8745
71
+ 2026-04-03 06:47:18,615 [INFO] new_opacus_codex.train_steps: epoch=2 step=240 loss=0.9544
72
+ 2026-04-03 06:47:57,188 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=240 eval_loss=0.9811 duration_sec=38.54
73
+ 2026-04-03 06:49:22,422 [INFO] new_opacus_codex.train_steps: epoch=2 step=245 loss=0.9512
74
+ 2026-04-03 06:50:26,334 [INFO] new_opacus_codex.train_steps: epoch=2 step=250 loss=0.8906
75
+ 2026-04-03 06:50:43,723 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=250 eval_loss=0.9980 duration_sec=17.35
76
+ 2026-04-03 06:51:48,886 [INFO] new_opacus_codex.train_steps: epoch=2 step=255 loss=0.8853
77
+ 2026-04-03 06:52:52,858 [INFO] new_opacus_codex.train_steps: epoch=2 step=260 loss=0.9266
78
+ 2026-04-03 06:53:12,943 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=260 eval_loss=0.9758 duration_sec=20.07
79
+ 2026-04-03 06:54:15,432 [INFO] new_opacus_codex.train_steps: epoch=2 step=265 loss=0.9435
80
+ 2026-04-03 06:55:18,987 [INFO] new_opacus_codex.train_steps: epoch=2 step=270 loss=0.8830
81
+ 2026-04-03 06:55:38,886 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=270 eval_loss=0.9706 duration_sec=19.88
82
+ 2026-04-03 06:56:43,803 [INFO] new_opacus_codex.train_steps: epoch=2 step=275 loss=0.8839
83
+ 2026-04-03 06:58:07,316 [INFO] new_opacus_codex.train_steps: epoch=2 step=280 loss=0.8690
84
+ 2026-04-03 06:58:27,308 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=280 eval_loss=0.9666 duration_sec=19.97
85
+ 2026-04-03 06:59:30,374 [INFO] new_opacus_codex.train_steps: epoch=2 step=285 loss=0.8554
86
+ 2026-04-03 07:00:34,169 [INFO] new_opacus_codex.train_steps: epoch=2 step=290 loss=0.9303
87
+ 2026-04-03 07:00:51,402 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=290 eval_loss=0.9592 duration_sec=17.21
88
+ 2026-04-03 07:01:55,960 [INFO] new_opacus_codex.train_steps: epoch=2 step=295 loss=0.9738
89
+ 2026-04-03 07:03:01,407 [INFO] new_opacus_codex.train_steps: epoch=2 step=300 loss=0.9137
90
+ 2026-04-03 07:03:18,670 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=300 eval_loss=0.9533 duration_sec=17.23
91
+ 2026-04-03 07:04:20,820 [INFO] new_opacus_codex.train_steps: epoch=2 step=305 loss=0.8680
92
+ 2026-04-03 07:05:27,125 [INFO] new_opacus_codex.train_steps: epoch=2 step=310 loss=0.8771
93
+ 2026-04-03 07:06:04,945 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=310 eval_loss=0.9494 duration_sec=37.79
94
+ 2026-04-03 07:07:32,573 [INFO] new_opacus_codex.train_steps: epoch=2 step=315 loss=0.8697
95
+ 2026-04-03 07:08:35,572 [INFO] new_opacus_codex.train_steps: epoch=2 step=320 loss=0.8781
96
+ 2026-04-03 07:08:54,259 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=320 eval_loss=0.9467 duration_sec=18.68
97
+ 2026-04-03 07:09:58,672 [INFO] new_opacus_codex.train_steps: epoch=2 step=325 loss=0.8894
98
+ 2026-04-03 07:11:01,798 [INFO] new_opacus_codex.train_steps: epoch=2 step=330 loss=0.8624
99
+ 2026-04-03 07:11:20,541 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=330 eval_loss=0.9420 duration_sec=18.73
100
+ 2026-04-03 07:12:25,692 [INFO] new_opacus_codex.train_steps: epoch=2 step=335 loss=0.8588
101
+ 2026-04-03 07:13:30,914 [INFO] new_opacus_codex.train_steps: epoch=2 step=340 loss=0.8330
102
+ 2026-04-03 07:14:08,802 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=340 eval_loss=0.9384 duration_sec=37.88
103
+ 2026-04-03 07:16:12,389 [INFO] new_opacus_codex.train_steps: epoch=2 step=345 loss=0.7941
104
+ 2026-04-03 07:18:10,253 [INFO] new_opacus_codex.train_steps: epoch=2 step=350 loss=0.8237
105
+ 2026-04-03 07:18:31,162 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=350 eval_loss=0.9330 duration_sec=18.64
106
+ 2026-04-03 07:19:34,732 [INFO] new_opacus_codex.train_steps: epoch=2 step=355 loss=0.8627
107
+ 2026-04-03 07:20:36,862 [INFO] new_opacus_codex.train_steps: epoch=2 step=360 loss=0.8453
108
+ 2026-04-03 07:21:12,376 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=360 eval_loss=0.9298 duration_sec=35.50
109
+ 2026-04-03 07:22:18,135 [INFO] new_opacus_codex.train_steps: epoch=2 step=365 loss=0.8345
110
+ 2026-04-03 07:24:37,517 [INFO] new_opacus_codex.train_steps: epoch=3 step=370 loss=0.7413
111
+ 2026-04-03 07:24:54,337 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=370 eval_loss=0.9347 duration_sec=16.80
112
+ 2026-04-03 07:25:57,377 [INFO] new_opacus_codex.train_steps: epoch=3 step=375 loss=0.7195
113
+ 2026-04-03 07:27:35,048 [INFO] new_opacus_codex.train_steps: epoch=3 step=380 loss=0.7166
114
+ 2026-04-03 07:27:52,977 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=380 eval_loss=0.9302 duration_sec=17.92
115
+ 2026-04-03 07:29:03,188 [INFO] new_opacus_codex.train_steps: epoch=3 step=385 loss=0.7112
116
+ 2026-04-03 07:30:20,009 [INFO] new_opacus_codex.train_steps: epoch=3 step=390 loss=0.7247
117
+ 2026-04-03 07:30:37,750 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=390 eval_loss=0.9281 duration_sec=17.72
118
+ 2026-04-03 07:31:42,462 [INFO] new_opacus_codex.train_steps: epoch=3 step=395 loss=0.7671
119
+ 2026-04-03 07:32:45,829 [INFO] new_opacus_codex.train_steps: epoch=3 step=400 loss=0.8061
120
+ 2026-04-03 07:33:05,447 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=400 eval_loss=0.9269 duration_sec=19.60
121
+ 2026-04-03 07:34:08,960 [INFO] new_opacus_codex.train_steps: epoch=3 step=405 loss=0.7725
122
+ 2026-04-03 07:35:13,333 [INFO] new_opacus_codex.train_steps: epoch=3 step=410 loss=0.7342
123
+ 2026-04-03 07:35:31,622 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=410 eval_loss=0.9241 duration_sec=18.23
124
+ 2026-04-03 07:36:36,070 [INFO] new_opacus_codex.train_steps: epoch=3 step=415 loss=0.7293
125
+ 2026-04-03 07:37:38,200 [INFO] new_opacus_codex.train_steps: epoch=3 step=420 loss=0.7057
126
+ 2026-04-03 07:37:57,391 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=420 eval_loss=0.9240 duration_sec=19.09
127
+ 2026-04-03 07:39:27,902 [INFO] new_opacus_codex.train_steps: epoch=3 step=425 loss=0.7280
128
+ 2026-04-03 07:40:31,294 [INFO] new_opacus_codex.train_steps: epoch=3 step=430 loss=0.7420
129
+ 2026-04-03 07:41:43,030 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=430 eval_loss=0.9199 duration_sec=27.83
130
+ 2026-04-03 07:42:45,941 [INFO] new_opacus_codex.train_steps: epoch=3 step=435 loss=0.7631
131
+ 2026-04-03 07:43:50,062 [INFO] new_opacus_codex.train_steps: epoch=3 step=440 loss=0.7587
132
+ 2026-04-03 07:44:07,131 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=440 eval_loss=0.9170 duration_sec=17.06
133
+ 2026-04-03 07:45:13,949 [INFO] new_opacus_codex.train_steps: epoch=3 step=445 loss=0.7181
134
+ 2026-04-03 07:46:28,269 [INFO] new_opacus_codex.train_steps: epoch=3 step=450 loss=0.7142
135
+ 2026-04-03 07:46:45,943 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=450 eval_loss=0.9183 duration_sec=17.66
136
+ 2026-04-03 07:47:51,864 [INFO] new_opacus_codex.train_steps: epoch=3 step=455 loss=0.6899
137
+ 2026-04-03 07:48:55,416 [INFO] new_opacus_codex.train_steps: epoch=3 step=460 loss=0.6952
138
+ 2026-04-03 07:49:13,114 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=460 eval_loss=0.9163 duration_sec=17.65
139
+ 2026-04-03 07:50:30,835 [INFO] new_opacus_codex.train_steps: epoch=3 step=465 loss=0.7120
140
+ 2026-04-03 07:51:32,631 [INFO] new_opacus_codex.train_steps: epoch=3 step=470 loss=0.7193
141
+ 2026-04-03 07:52:21,085 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=470 eval_loss=0.9114 duration_sec=37.62
142
+ 2026-04-03 07:53:23,130 [INFO] new_opacus_codex.train_steps: epoch=3 step=475 loss=0.7263
143
+ 2026-04-03 07:54:25,195 [INFO] new_opacus_codex.train_steps: epoch=3 step=480 loss=0.6944
144
+ 2026-04-03 07:54:43,588 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=480 eval_loss=0.9070 duration_sec=18.37
145
+ 2026-04-03 07:55:43,709 [INFO] new_opacus_codex.train_steps: epoch=3 step=485 loss=0.7083
146
+ 2026-04-03 07:56:50,544 [INFO] new_opacus_codex.train_steps: epoch=3 step=490 loss=0.7436
147
+ 2026-04-03 07:57:24,269 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=490 eval_loss=0.9077 duration_sec=33.63
148
+ 2026-04-03 07:58:28,786 [INFO] new_opacus_codex.train_steps: epoch=3 step=495 loss=0.7473
149
+ 2026-04-03 07:59:54,378 [INFO] new_opacus_codex.train_steps: epoch=3 step=500 loss=0.7145
150
+ 2026-04-03 08:00:14,102 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=500 eval_loss=0.9046 duration_sec=19.71
151
+ 2026-04-03 08:01:15,059 [INFO] new_opacus_codex.train_steps: epoch=3 step=505 loss=0.6995
152
+ 2026-04-03 08:02:17,935 [INFO] new_opacus_codex.train_steps: epoch=3 step=510 loss=0.7243
153
+ 2026-04-03 08:02:35,098 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=510 eval_loss=0.8999 duration_sec=17.15
154
+ 2026-04-03 08:03:35,684 [INFO] new_opacus_codex.train_steps: epoch=3 step=515 loss=0.7057
155
+ 2026-04-03 08:04:48,183 [INFO] new_opacus_codex.train_steps: epoch=3 step=520 loss=0.6839
156
+ 2026-04-03 08:05:05,053 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=520 eval_loss=0.8990 duration_sec=16.85
157
+ 2026-04-03 08:06:09,023 [INFO] new_opacus_codex.train_steps: epoch=3 step=525 loss=0.6934
158
+ 2026-04-03 08:07:18,657 [INFO] new_opacus_codex.train_steps: epoch=3 step=530 loss=0.6857
159
+ 2026-04-03 08:07:33,996 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=530 eval_loss=0.8987 duration_sec=15.30
160
+ 2026-04-03 08:08:38,766 [INFO] new_opacus_codex.train_steps: epoch=3 step=535 loss=0.6625
161
+ 2026-04-03 08:09:39,386 [INFO] new_opacus_codex.train_steps: epoch=3 step=540 loss=0.6714
162
+ 2026-04-03 08:10:05,186 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=540 eval_loss=0.8960 duration_sec=15.72
163
+ 2026-04-03 08:11:05,578 [INFO] new_opacus_codex.train_steps: epoch=3 step=545 loss=0.6951
164
+ 2026-04-03 08:12:04,707 [INFO] new_opacus_codex.train_steps: epoch=3 step=550 loss=0.7273
165
+ 2026-04-03 08:12:20,110 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=550 eval_loss=0.8949 duration_sec=15.34
166
+ 2026-04-03 08:14:43,188 [INFO] new_opacus_codex.train_steps: epoch=4 step=555 loss=0.6098
167
+ 2026-04-03 08:15:42,797 [INFO] new_opacus_codex.train_steps: epoch=4 step=560 loss=0.6473
168
+ 2026-04-03 08:16:01,623 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=560 eval_loss=0.9085 duration_sec=18.81
169
+ 2026-04-03 08:17:02,126 [INFO] new_opacus_codex.train_steps: epoch=4 step=565 loss=0.6450
170
+ 2026-04-03 08:18:08,816 [INFO] new_opacus_codex.train_steps: epoch=4 step=570 loss=0.6175
171
+ 2026-04-03 08:18:35,053 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=570 eval_loss=0.9106 duration_sec=17.51
172
+ 2026-04-03 08:19:36,045 [INFO] new_opacus_codex.train_steps: epoch=4 step=575 loss=0.6275
173
+ 2026-04-03 08:20:38,797 [INFO] new_opacus_codex.train_steps: epoch=4 step=580 loss=0.6283
174
+ 2026-04-03 08:20:55,446 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=580 eval_loss=0.9060 duration_sec=16.62
175
+ 2026-04-03 08:21:56,658 [INFO] new_opacus_codex.train_steps: epoch=4 step=585 loss=0.5982
176
+ 2026-04-03 08:22:56,822 [INFO] new_opacus_codex.train_steps: epoch=4 step=590 loss=0.6029
177
+ 2026-04-03 08:23:37,627 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=590 eval_loss=0.9076 duration_sec=23.77
178
+ 2026-04-03 08:24:34,769 [INFO] new_opacus_codex.train_steps: epoch=4 step=595 loss=0.6692
179
+ 2026-04-03 08:25:33,796 [INFO] new_opacus_codex.train_steps: epoch=4 step=600 loss=0.6555
180
+ 2026-04-03 08:26:07,812 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=600 eval_loss=0.9074 duration_sec=34.00
181
+ 2026-04-03 08:27:07,121 [INFO] new_opacus_codex.train_steps: epoch=4 step=605 loss=0.5980
182
+ 2026-04-03 08:28:07,521 [INFO] new_opacus_codex.train_steps: epoch=4 step=610 loss=0.5976
183
+ 2026-04-03 08:28:25,889 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=610 eval_loss=0.9036 duration_sec=18.36
184
+ 2026-04-03 08:29:29,602 [INFO] new_opacus_codex.train_steps: epoch=4 step=615 loss=0.6094
185
+ 2026-04-03 08:30:53,935 [INFO] new_opacus_codex.train_steps: epoch=4 step=620 loss=0.6373
186
+ 2026-04-03 08:31:50,983 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=620 eval_loss=0.9012 duration_sec=17.61
187
+ 2026-04-03 08:32:53,083 [INFO] new_opacus_codex.train_steps: epoch=4 step=625 loss=0.6335
188
+ 2026-04-03 08:33:53,215 [INFO] new_opacus_codex.train_steps: epoch=4 step=630 loss=0.6350
189
+ 2026-04-03 08:34:07,896 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=630 eval_loss=0.9019 duration_sec=14.67
190
+ 2026-04-03 08:35:06,394 [INFO] new_opacus_codex.train_steps: epoch=4 step=635 loss=0.6458
191
+ 2026-04-03 08:36:01,915 [INFO] new_opacus_codex.train_steps: epoch=4 step=640 loss=0.6182
192
+ 2026-04-03 08:37:08,535 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=640 eval_loss=0.9028 duration_sec=37.37
193
+ 2026-04-03 08:38:13,223 [INFO] new_opacus_codex.train_steps: epoch=4 step=645 loss=0.6131
194
+ 2026-04-03 08:39:15,153 [INFO] new_opacus_codex.train_steps: epoch=4 step=650 loss=0.6175
195
+ 2026-04-03 08:39:33,671 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=650 eval_loss=0.9018 duration_sec=18.50
196
+ 2026-04-03 08:40:37,711 [INFO] new_opacus_codex.train_steps: epoch=4 step=655 loss=0.6032
197
+ 2026-04-03 08:41:40,088 [INFO] new_opacus_codex.train_steps: epoch=4 step=660 loss=0.6276
198
+ 2026-04-03 08:42:02,032 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=660 eval_loss=0.8996 duration_sec=21.92
199
+ 2026-04-03 08:43:04,821 [INFO] new_opacus_codex.train_steps: epoch=4 step=665 loss=0.6508
200
+ 2026-04-03 08:44:05,654 [INFO] new_opacus_codex.train_steps: epoch=4 step=670 loss=0.6338
201
+ 2026-04-03 08:44:23,769 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=670 eval_loss=0.8979 duration_sec=18.11
202
+ 2026-04-03 08:45:23,723 [INFO] new_opacus_codex.train_steps: epoch=4 step=675 loss=0.6147
203
+ 2026-04-03 08:46:40,439 [INFO] new_opacus_codex.train_steps: epoch=4 step=680 loss=0.6057
204
+ 2026-04-03 08:47:00,147 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=680 eval_loss=0.8990 duration_sec=19.67
205
+ 2026-04-03 08:48:03,178 [INFO] new_opacus_codex.train_steps: epoch=4 step=685 loss=0.6169
206
+ 2026-04-03 08:49:23,189 [INFO] new_opacus_codex.train_steps: epoch=4 step=690 loss=0.6216
207
+ 2026-04-03 08:49:41,646 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=690 eval_loss=0.8977 duration_sec=18.45
208
+ 2026-04-03 08:50:44,309 [INFO] new_opacus_codex.train_steps: epoch=4 step=695 loss=0.6178
209
+ 2026-04-03 08:51:50,210 [INFO] new_opacus_codex.train_steps: epoch=4 step=700 loss=0.5749
210
+ 2026-04-03 08:52:05,580 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=700 eval_loss=0.8963 duration_sec=15.34
211
+ 2026-04-03 08:53:11,101 [INFO] new_opacus_codex.train_steps: epoch=4 step=705 loss=0.5648
212
+ 2026-04-03 08:54:15,810 [INFO] new_opacus_codex.train_steps: epoch=4 step=710 loss=0.5997
213
+ 2026-04-03 08:54:53,234 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=710 eval_loss=0.8944 duration_sec=37.41
214
+ 2026-04-03 08:55:57,855 [INFO] new_opacus_codex.train_steps: epoch=4 step=715 loss=0.6174
215
+ 2026-04-03 08:57:02,169 [INFO] new_opacus_codex.train_steps: epoch=4 step=720 loss=0.6190
216
+ 2026-04-03 08:57:20,203 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=720 eval_loss=0.8948 duration_sec=18.02
217
+ 2026-04-03 08:58:21,919 [INFO] new_opacus_codex.train_steps: epoch=4 step=725 loss=0.6129
218
+ 2026-04-03 08:59:23,513 [INFO] new_opacus_codex.train_steps: epoch=4 step=730 loss=0.6184
219
+ 2026-04-03 08:59:41,283 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=730 eval_loss=0.8944 duration_sec=17.76
220
+ 2026-04-03 09:00:45,065 [INFO] new_opacus_codex.train_steps: epoch=4 step=735 loss=0.6104
221
+ 2026-04-03 09:02:14,724 [INFO] new_opacus_codex.train_steps: epoch=5 step=740 loss=0.5629
222
+ 2026-04-03 09:02:49,793 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=740 eval_loss=0.9112 duration_sec=35.05
223
+ 2026-04-03 09:03:48,847 [INFO] new_opacus_codex.train_steps: epoch=5 step=745 loss=0.5643
224
+ 2026-04-03 09:04:50,871 [INFO] new_opacus_codex.train_steps: epoch=5 step=750 loss=0.5632
225
+ 2026-04-03 09:05:19,395 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=750 eval_loss=0.9136 duration_sec=16.89
226
+ 2026-04-03 09:06:19,835 [INFO] new_opacus_codex.train_steps: epoch=5 step=755 loss=0.5646
227
+ 2026-04-03 09:07:31,997 [INFO] new_opacus_codex.train_steps: epoch=5 step=760 loss=0.5639
228
+ 2026-04-03 09:07:48,262 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=760 eval_loss=0.9132 duration_sec=16.22
229
+ 2026-04-03 09:08:48,761 [INFO] new_opacus_codex.train_steps: epoch=5 step=765 loss=0.5597
230
+ 2026-04-03 09:09:47,623 [INFO] new_opacus_codex.train_steps: epoch=5 step=770 loss=0.5641
231
+ 2026-04-03 09:10:03,489 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=770 eval_loss=0.9134 duration_sec=15.86
232
+ 2026-04-03 09:11:01,835 [INFO] new_opacus_codex.train_steps: epoch=5 step=775 loss=0.5601
233
+ 2026-04-03 09:11:59,734 [INFO] new_opacus_codex.train_steps: epoch=5 step=780 loss=0.5713
234
+ 2026-04-03 09:12:23,545 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=780 eval_loss=0.9113 duration_sec=19.65
235
+ 2026-04-03 09:13:25,144 [INFO] new_opacus_codex.train_steps: epoch=5 step=785 loss=0.5601
236
+ 2026-04-03 09:14:28,931 [INFO] new_opacus_codex.train_steps: epoch=5 step=790 loss=0.5647
237
+ 2026-04-03 09:15:07,078 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=790 eval_loss=0.9098 duration_sec=18.43
238
+ 2026-04-03 09:16:08,217 [INFO] new_opacus_codex.train_steps: epoch=5 step=795 loss=0.5745
239
+ 2026-04-03 09:17:13,395 [INFO] new_opacus_codex.train_steps: epoch=5 step=800 loss=0.5638
240
+ 2026-04-03 09:17:28,110 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=800 eval_loss=0.9091 duration_sec=14.69
241
+ 2026-04-03 09:18:28,128 [INFO] new_opacus_codex.train_steps: epoch=5 step=805 loss=0.5636
242
+ 2026-04-03 09:19:48,528 [INFO] new_opacus_codex.train_steps: epoch=5 step=810 loss=0.5488
243
+ 2026-04-03 09:20:05,440 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=810 eval_loss=0.9098 duration_sec=16.89
244
+ 2026-04-03 09:21:35,295 [INFO] new_opacus_codex.train_steps: epoch=5 step=815 loss=0.5716
245
+ 2026-04-03 09:22:33,695 [INFO] new_opacus_codex.train_steps: epoch=5 step=820 loss=0.5641
246
+ 2026-04-03 09:22:48,346 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=820 eval_loss=0.9095 duration_sec=14.64
247
+ 2026-04-03 09:23:46,188 [INFO] new_opacus_codex.train_steps: epoch=5 step=825 loss=0.5532
248
+ 2026-04-03 09:24:43,328 [INFO] new_opacus_codex.train_steps: epoch=5 step=830 loss=0.5684
249
+ 2026-04-03 09:25:13,160 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=830 eval_loss=0.9097 duration_sec=29.82
250
+ 2026-04-03 09:26:11,206 [INFO] new_opacus_codex.train_steps: epoch=5 step=835 loss=0.5639
251
+ 2026-04-03 09:27:07,995 [INFO] new_opacus_codex.train_steps: epoch=5 step=840 loss=0.5639
252
+ 2026-04-03 09:27:43,184 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=840 eval_loss=0.9103 duration_sec=22.36
253
+ 2026-04-03 09:28:41,419 [INFO] new_opacus_codex.train_steps: epoch=5 step=845 loss=0.5613
254
+ 2026-04-03 09:29:40,774 [INFO] new_opacus_codex.train_steps: epoch=5 step=850 loss=0.5513
255
+ 2026-04-03 09:29:55,406 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=850 eval_loss=0.9093 duration_sec=14.61
256
+ 2026-04-03 09:30:52,107 [INFO] new_opacus_codex.train_steps: epoch=5 step=855 loss=0.5352
257
+ 2026-04-03 09:33:23,901 [INFO] new_opacus_codex.train_steps: epoch=5 step=860 loss=0.5464
258
+ 2026-04-03 09:33:41,612 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=860 eval_loss=0.9106 duration_sec=17.66
259
+ 2026-04-03 09:34:41,845 [INFO] new_opacus_codex.train_steps: epoch=5 step=865 loss=0.5617
260
+ 2026-04-03 09:35:38,854 [INFO] new_opacus_codex.train_steps: epoch=5 step=870 loss=0.5510
261
+ 2026-04-03 09:36:22,769 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=870 eval_loss=0.9090 duration_sec=34.29
262
+ 2026-04-03 09:37:48,204 [INFO] new_opacus_codex.train_steps: epoch=5 step=875 loss=0.5547
263
+ 2026-04-03 09:38:51,422 [INFO] new_opacus_codex.train_steps: epoch=5 step=880 loss=0.5713
264
+ 2026-04-03 09:39:26,033 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=880 eval_loss=0.9074 duration_sec=34.59
265
+ 2026-04-03 09:40:28,996 [INFO] new_opacus_codex.train_steps: epoch=5 step=885 loss=0.5657
266
+ 2026-04-03 09:41:36,512 [INFO] new_opacus_codex.train_steps: epoch=5 step=890 loss=0.5408
267
+ 2026-04-03 09:42:11,356 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=890 eval_loss=0.9063 duration_sec=34.83
268
+ 2026-04-03 09:43:55,536 [INFO] new_opacus_codex.train_steps: epoch=5 step=895 loss=0.5370
269
+ 2026-04-03 09:44:53,707 [INFO] new_opacus_codex.train_steps: epoch=5 step=900 loss=0.5541
270
+ 2026-04-03 09:45:11,918 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=900 eval_loss=0.9085 duration_sec=18.20
271
+ 2026-04-03 09:46:11,402 [INFO] new_opacus_codex.train_steps: epoch=5 step=905 loss=0.5746
272
+ 2026-04-03 09:47:51,412 [INFO] new_opacus_codex.train_steps: epoch=5 step=910 loss=0.5804
273
+ 2026-04-03 09:48:05,993 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=910 eval_loss=0.9076 duration_sec=14.57
274
+ 2026-04-03 09:49:03,553 [INFO] new_opacus_codex.train_steps: epoch=5 step=915 loss=0.5725
275
+ 2026-04-03 09:50:03,697 [INFO] new_opacus_codex.train_steps: epoch=5 step=920 loss=0.5628
276
+ 2026-04-03 09:50:38,886 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=920 eval_loss=0.9080 duration_sec=35.18
qwen1.5-moe-a2.7b/dp3_attn/adapter/README.md ADDED
@@ -0,0 +1,207 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: Qwen/Qwen1.5-MoE-A2.7B
3
+ library_name: peft
4
+ pipeline_tag: text-generation
5
+ tags:
6
+ - base_model:adapter:Qwen/Qwen1.5-MoE-A2.7B
7
+ - lora
8
+ - transformers
9
+ ---
10
+
11
+ # Model Card for Model ID
12
+
13
+ <!-- Provide a quick summary of what the model is/does. -->
14
+
15
+
16
+
17
+ ## Model Details
18
+
19
+ ### Model Description
20
+
21
+ <!-- Provide a longer summary of what this model is. -->
22
+
23
+
24
+
25
+ - **Developed by:** [More Information Needed]
26
+ - **Funded by [optional]:** [More Information Needed]
27
+ - **Shared by [optional]:** [More Information Needed]
28
+ - **Model type:** [More Information Needed]
29
+ - **Language(s) (NLP):** [More Information Needed]
30
+ - **License:** [More Information Needed]
31
+ - **Finetuned from model [optional]:** [More Information Needed]
32
+
33
+ ### Model Sources [optional]
34
+
35
+ <!-- Provide the basic links for the model. -->
36
+
37
+ - **Repository:** [More Information Needed]
38
+ - **Paper [optional]:** [More Information Needed]
39
+ - **Demo [optional]:** [More Information Needed]
40
+
41
+ ## Uses
42
+
43
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
44
+
45
+ ### Direct Use
46
+
47
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
48
+
49
+ [More Information Needed]
50
+
51
+ ### Downstream Use [optional]
52
+
53
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
54
+
55
+ [More Information Needed]
56
+
57
+ ### Out-of-Scope Use
58
+
59
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
60
+
61
+ [More Information Needed]
62
+
63
+ ## Bias, Risks, and Limitations
64
+
65
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
66
+
67
+ [More Information Needed]
68
+
69
+ ### Recommendations
70
+
71
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
72
+
73
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
74
+
75
+ ## How to Get Started with the Model
76
+
77
+ Use the code below to get started with the model.
78
+
79
+ [More Information Needed]
80
+
81
+ ## Training Details
82
+
83
+ ### Training Data
84
+
85
+ <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
86
+
87
+ [More Information Needed]
88
+
89
+ ### Training Procedure
90
+
91
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
92
+
93
+ #### Preprocessing [optional]
94
+
95
+ [More Information Needed]
96
+
97
+
98
+ #### Training Hyperparameters
99
+
100
+ - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
101
+
102
+ #### Speeds, Sizes, Times [optional]
103
+
104
+ <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
105
+
106
+ [More Information Needed]
107
+
108
+ ## Evaluation
109
+
110
+ <!-- This section describes the evaluation protocols and provides the results. -->
111
+
112
+ ### Testing Data, Factors & Metrics
113
+
114
+ #### Testing Data
115
+
116
+ <!-- This should link to a Dataset Card if possible. -->
117
+
118
+ [More Information Needed]
119
+
120
+ #### Factors
121
+
122
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
123
+
124
+ [More Information Needed]
125
+
126
+ #### Metrics
127
+
128
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
129
+
130
+ [More Information Needed]
131
+
132
+ ### Results
133
+
134
+ [More Information Needed]
135
+
136
+ #### Summary
137
+
138
+
139
+
140
+ ## Model Examination [optional]
141
+
142
+ <!-- Relevant interpretability work for the model goes here -->
143
+
144
+ [More Information Needed]
145
+
146
+ ## Environmental Impact
147
+
148
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
149
+
150
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
151
+
152
+ - **Hardware Type:** [More Information Needed]
153
+ - **Hours used:** [More Information Needed]
154
+ - **Cloud Provider:** [More Information Needed]
155
+ - **Compute Region:** [More Information Needed]
156
+ - **Carbon Emitted:** [More Information Needed]
157
+
158
+ ## Technical Specifications [optional]
159
+
160
+ ### Model Architecture and Objective
161
+
162
+ [More Information Needed]
163
+
164
+ ### Compute Infrastructure
165
+
166
+ [More Information Needed]
167
+
168
+ #### Hardware
169
+
170
+ [More Information Needed]
171
+
172
+ #### Software
173
+
174
+ [More Information Needed]
175
+
176
+ ## Citation [optional]
177
+
178
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
179
+
180
+ **BibTeX:**
181
+
182
+ [More Information Needed]
183
+
184
+ **APA:**
185
+
186
+ [More Information Needed]
187
+
188
+ ## Glossary [optional]
189
+
190
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
191
+
192
+ [More Information Needed]
193
+
194
+ ## More Information [optional]
195
+
196
+ [More Information Needed]
197
+
198
+ ## Model Card Authors [optional]
199
+
200
+ [More Information Needed]
201
+
202
+ ## Model Card Contact
203
+
204
+ [More Information Needed]
205
+ ### Framework versions
206
+
207
+ - PEFT 0.18.1
qwen1.5-moe-a2.7b/dp3_attn/adapter/adapter_config.json ADDED
@@ -0,0 +1,46 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alora_invocation_tokens": null,
3
+ "alpha_pattern": {},
4
+ "arrow_config": null,
5
+ "auto_mapping": null,
6
+ "base_model_name_or_path": "Qwen/Qwen1.5-MoE-A2.7B",
7
+ "bias": "none",
8
+ "corda_config": null,
9
+ "ensure_weight_tying": true,
10
+ "eva_config": null,
11
+ "exclude_modules": null,
12
+ "fan_in_fan_out": false,
13
+ "inference_mode": true,
14
+ "init_lora_weights": true,
15
+ "layer_replication": null,
16
+ "layers_pattern": null,
17
+ "layers_to_transform": null,
18
+ "loftq_config": {},
19
+ "lora_alpha": 32,
20
+ "lora_bias": false,
21
+ "lora_dropout": 0.05,
22
+ "megatron_config": null,
23
+ "megatron_core": "megatron.core",
24
+ "modules_to_save": [
25
+ "lm_head",
26
+ "embed_tokens"
27
+ ],
28
+ "peft_type": "LORA",
29
+ "peft_version": "0.18.1",
30
+ "qalora_group_size": 16,
31
+ "r": 16,
32
+ "rank_pattern": {},
33
+ "revision": null,
34
+ "target_modules": [
35
+ "k_proj",
36
+ "v_proj",
37
+ "q_proj",
38
+ "o_proj"
39
+ ],
40
+ "target_parameters": null,
41
+ "task_type": "CAUSAL_LM",
42
+ "trainable_token_indices": null,
43
+ "use_dora": false,
44
+ "use_qalora": false,
45
+ "use_rslora": false
46
+ }
qwen1.5-moe-a2.7b/dp3_attn/audit_results.json ADDED
@@ -0,0 +1,137 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "delta": 1e-05,
3
+ "num_canaries": 500,
4
+ "num_members": 250,
5
+ "paper_guess_fraction": 0.2,
6
+ "paper_guess_steps": 20,
7
+ "loss": {
8
+ "auc": 0.538384,
9
+ "empirical_epsilon": {
10
+ "0.05": 0.0,
11
+ "0.01": 0.0
12
+ },
13
+ "empirical_epsilon_details": {
14
+ "0.05": {
15
+ "epsilon": 0.0,
16
+ "num_guesses": 0,
17
+ "correct_guesses": 0,
18
+ "candidate_num_guesses": [
19
+ 5,
20
+ 10,
21
+ 15,
22
+ 20,
23
+ 25,
24
+ 30,
25
+ 35,
26
+ 40,
27
+ 45,
28
+ 50,
29
+ 55,
30
+ 60,
31
+ 65,
32
+ 70,
33
+ 75,
34
+ 80,
35
+ 85,
36
+ 90,
37
+ 95,
38
+ 100
39
+ ],
40
+ "direction": "lower"
41
+ },
42
+ "0.01": {
43
+ "epsilon": 0.0,
44
+ "num_guesses": 0,
45
+ "correct_guesses": 0,
46
+ "candidate_num_guesses": [
47
+ 5,
48
+ 10,
49
+ 15,
50
+ 20,
51
+ 25,
52
+ 30,
53
+ 35,
54
+ 40,
55
+ 45,
56
+ 50,
57
+ 55,
58
+ 60,
59
+ 65,
60
+ 70,
61
+ 75,
62
+ 80,
63
+ 85,
64
+ 90,
65
+ 95,
66
+ 100
67
+ ],
68
+ "direction": "lower"
69
+ }
70
+ }
71
+ },
72
+ "embedding": {
73
+ "auc": 0.531696,
74
+ "empirical_epsilon": {
75
+ "0.05": 0.17315445840358734,
76
+ "0.01": 0.0
77
+ },
78
+ "empirical_epsilon_details": {
79
+ "0.05": {
80
+ "epsilon": 0.17315445840358734,
81
+ "num_guesses": 5,
82
+ "correct_guesses": 5,
83
+ "candidate_num_guesses": [
84
+ 5,
85
+ 10,
86
+ 15,
87
+ 20,
88
+ 25,
89
+ 30,
90
+ 35,
91
+ 40,
92
+ 45,
93
+ 50,
94
+ 55,
95
+ 60,
96
+ 65,
97
+ 70,
98
+ 75,
99
+ 80,
100
+ 85,
101
+ 90,
102
+ 95,
103
+ 100
104
+ ],
105
+ "direction": "higher"
106
+ },
107
+ "0.01": {
108
+ "epsilon": 0.0,
109
+ "num_guesses": 0,
110
+ "correct_guesses": 0,
111
+ "candidate_num_guesses": [
112
+ 5,
113
+ 10,
114
+ 15,
115
+ 20,
116
+ 25,
117
+ 30,
118
+ 35,
119
+ 40,
120
+ 45,
121
+ 50,
122
+ 55,
123
+ 60,
124
+ 65,
125
+ 70,
126
+ 75,
127
+ 80,
128
+ 85,
129
+ 90,
130
+ 95,
131
+ 100
132
+ ],
133
+ "direction": "lower"
134
+ }
135
+ }
136
+ }
137
+ }
qwen1.5-moe-a2.7b/dp3_attn/canary_meta.json ADDED
The diff for this file is too large to render. See raw diff
 
qwen1.5-moe-a2.7b/dp3_attn/metrics.jsonl ADDED
The diff for this file is too large to render. See raw diff
 
qwen1.5-moe-a2.7b/dp3_attn/resolved_config.yaml ADDED
@@ -0,0 +1,110 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ model:
2
+ name: Qwen/Qwen1.5-MoE-A2.7B
3
+ tokenizer_name: Qwen/Qwen1.5-MoE-A2.7B
4
+ max_length: 1024
5
+ dtype: bfloat16
6
+ trust_remote_code: true
7
+ use_fast_tokenizer: true
8
+ cache_dir: null
9
+ local_files_only: false
10
+ low_cpu_mem_usage: true
11
+ tie_word_embeddings: true
12
+ gradient_checkpointing: false
13
+ use_chat_template: false
14
+ dataset:
15
+ name: melihcatal/codedp-cpt
16
+ split: train
17
+ mode: cpt
18
+ text_column: text
19
+ validation_ratio: 0.05
20
+ max_samples: -1
21
+ lora:
22
+ enabled: true
23
+ r: 16
24
+ alpha: 32
25
+ dropout: 0.05
26
+ target_modules:
27
+ - q_proj
28
+ - k_proj
29
+ - v_proj
30
+ - o_proj
31
+ modules_to_save:
32
+ - lm_head
33
+ bias: none
34
+ training:
35
+ seed: 42
36
+ epochs: 3
37
+ warmup_steps: null
38
+ warmup_ratio: 0.05
39
+ mixed_precision: false
40
+ mixed_precision_dtype: bfloat16
41
+ batch_size: 4
42
+ eval_batch_size: 8
43
+ eval_every_steps: 10
44
+ eval_every_epochs: 1
45
+ learning_rate: 0.0005
46
+ optimizer: adamw
47
+ lr_scheduler: cosine
48
+ adam_beta1: 0.9
49
+ adam_beta2: 0.999
50
+ adam_epsilon: 1.0e-08
51
+ sgd_momentum: 0.9
52
+ weight_decay: 0.01
53
+ max_grad_norm: 1.0
54
+ log_every: 5
55
+ gradient_accumulation_steps: 8
56
+ num_workers: 4
57
+ output_dir: /scratch/mcatal/runs/cpt/qwen1.5-moe-a2.7b/dp3_attn
58
+ distributed:
59
+ strategy: dpddp
60
+ backend: nccl
61
+ devices: null
62
+ dp:
63
+ module_validator: auto
64
+ target_delta: 1.0e-05
65
+ noise_multiplier: null
66
+ max_grad_norm: 1.0
67
+ grad_sample_mode: hooks
68
+ clipping: per_layer
69
+ secure_mode: false
70
+ enabled: true
71
+ target_epsilon: 3.0
72
+ audit:
73
+ enabled: true
74
+ run_every_epoch: true
75
+ epoch_device: cuda
76
+ q_canary: auto
77
+ num_canaries: 500
78
+ prefix_length: 49
79
+ num_digits: 12
80
+ batch_size: 32
81
+ delta: 1.0e-05
82
+ p_values:
83
+ - 0.05
84
+ - 0.01
85
+ paper_guess_fraction: 0.2
86
+ paper_guess_steps: 20
87
+ enable_holdout_empirical_epsilon: false
88
+ holdout_seed: 42
89
+ tie_seed: 42
90
+ tracking:
91
+ enabled: true
92
+ tensorboard: true
93
+ wandb: false
94
+ wandb_project: codedp-finetune-h200-audit
95
+ wandb_run_name: qwen1.5-moe-a2.7b-cpt-dp3_attn
96
+ wandb_mode: online
97
+ codecarbon: true
98
+ codecarbon_output_file: codecarbon.csv
99
+ codecarbon_measure_power_secs: 15
100
+ codecarbon_country_iso_code: null
101
+ codecarbon_project_name: codedp-qwen1.5-moe-a2.7b-cpt-dp3_attn
102
+ moe:
103
+ output_router_logits: false
104
+ router_aux_loss_coef: 0.0
105
+ freeze_router: true
106
+ profile:
107
+ enabled: false
108
+ num_batches: 8
109
+ top_experts: 8
110
+ output_file: moe_expert_profile.json
qwen1.5-moe-a2.7b/dp3_attn/summary.json ADDED
@@ -0,0 +1,72 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "audit/delta": 1e-05,
3
+ "audit/embedding/auc": 0.531696,
4
+ "audit/embedding/empirical_epsilon/0.01": 0.0,
5
+ "audit/embedding/empirical_epsilon/0.05": 0.17315445840358734,
6
+ "audit/embedding/empirical_epsilon_details/0.01/correct_guesses": 0.0,
7
+ "audit/embedding/empirical_epsilon_details/0.01/epsilon": 0.0,
8
+ "audit/embedding/empirical_epsilon_details/0.01/num_guesses": 0.0,
9
+ "audit/embedding/empirical_epsilon_details/0.05/correct_guesses": 5.0,
10
+ "audit/embedding/empirical_epsilon_details/0.05/epsilon": 0.17315445840358734,
11
+ "audit/embedding/empirical_epsilon_details/0.05/num_guesses": 5.0,
12
+ "audit/loss/auc": 0.538384,
13
+ "audit/loss/empirical_epsilon/0.01": 0.0,
14
+ "audit/loss/empirical_epsilon/0.05": 0.0,
15
+ "audit/loss/empirical_epsilon_details/0.01/correct_guesses": 0.0,
16
+ "audit/loss/empirical_epsilon_details/0.01/epsilon": 0.0,
17
+ "audit/loss/empirical_epsilon_details/0.01/num_guesses": 0.0,
18
+ "audit/loss/empirical_epsilon_details/0.05/correct_guesses": 0.0,
19
+ "audit/loss/empirical_epsilon_details/0.05/epsilon": 0.0,
20
+ "audit/loss/empirical_epsilon_details/0.05/num_guesses": 0.0,
21
+ "audit/num_canaries": 500.0,
22
+ "audit/num_members": 250.0,
23
+ "audit/paper_guess_fraction": 0.2,
24
+ "audit/paper_guess_steps": 20.0,
25
+ "energy/codecarbon/cpu_count": 8.0,
26
+ "energy/codecarbon/cpu_energy": 0.11056595658839297,
27
+ "energy/codecarbon/cpu_power": 80.03013747036003,
28
+ "energy/codecarbon/cpu_utilization_percent": 3.9701551148635383,
29
+ "energy/codecarbon/duration": 5166.472950742114,
30
+ "energy/codecarbon/emissions": 0.12670685563743309,
31
+ "energy/codecarbon/emissions_rate": 2.4524827061996493e-05,
32
+ "energy/codecarbon/energy_consumed": 3.636508212192782,
33
+ "energy/codecarbon/gpu_count": 8.0,
34
+ "energy/codecarbon/gpu_energy": 3.4734437168083474,
35
+ "energy/codecarbon/gpu_power": 2430.49880801947,
36
+ "energy/codecarbon/gpu_utilization_percent": 91.15337784760409,
37
+ "energy/codecarbon/latitude": 47.4843,
38
+ "energy/codecarbon/longitude": 8.212,
39
+ "energy/codecarbon/pue": 1.0,
40
+ "energy/codecarbon/ram_energy": 0.05249853879604361,
41
+ "energy/codecarbon/ram_power": 38.0,
42
+ "energy/codecarbon/ram_total_size": 256.0,
43
+ "energy/codecarbon/ram_used_gb": 522.4753839173346,
44
+ "energy/codecarbon/ram_utilization_percent": 26.362104849793838,
45
+ "energy/codecarbon/water_consumed": 0.0,
46
+ "energy/codecarbon/wue": 0.0,
47
+ "eval/duration_sec": 7.727635388029739,
48
+ "eval/loss": 8.603873121432768,
49
+ "perf/audit_duration_sec": 5.70154406898655,
50
+ "perf/epoch_duration_sec": 1534.673990025185,
51
+ "perf/epoch_samples": 51837.0,
52
+ "perf/epoch_samples_per_sec": 33.7772063232461,
53
+ "perf/epoch_tokens": 37890736.0,
54
+ "perf/epoch_tokens_per_sec": 24689.76228585081,
55
+ "perf/logical_batch_size": 26.0,
56
+ "perf/logical_token_count": 18717.0,
57
+ "perf/physical_batches": 8.0,
58
+ "perf/samples_per_sec": 4.042696823009511,
59
+ "perf/step_duration_sec": 6.4313504421152174,
60
+ "perf/tokens_per_sec": 2910.275247548808,
61
+ "privacy/epsilon": 2.994149057175467,
62
+ "system/cuda_epoch_peak_memory_gb": 54.72412633895874,
63
+ "system/cuda_max_memory_allocated_gb": 54.72412633895874,
64
+ "system/cuda_memory_allocated_gb": 30.284135341644287,
65
+ "train/epoch_canary_loss": 13.01324360035101,
66
+ "train/epoch_loss": 9.342897934321682,
67
+ "train/epoch_real_loss": 8.840494789854363,
68
+ "train/lr": 0.0,
69
+ "train/step_canary_loss": 11.791666666666666,
70
+ "train/step_loss": 9.199588916919849,
71
+ "train/step_real_loss": 8.875579198201498
72
+ }
qwen1.5-moe-a2.7b/dp3_attn/tokenizer/chat_template.jinja ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system
2
+ You are a helpful assistant<|im_end|>
3
+ ' }}{% endif %}{{'<|im_start|>' + message['role'] + '
4
+ ' + message['content'] + '<|im_end|>' + '
5
+ '}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
6
+ ' }}{% endif %}
qwen1.5-moe-a2.7b/dp3_attn/tokenizer/tokenizer_config.json ADDED
@@ -0,0 +1,516 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "backend": "tokenizers",
4
+ "bos_token": null,
5
+ "clean_up_tokenization_spaces": false,
6
+ "eos_token": "<|endoftext|>",
7
+ "errors": "replace",
8
+ "extra_special_tokens": {
9
+ "extra_special_token_0": "865331112869",
10
+ "extra_special_token_1": "569765693871",
11
+ "extra_special_token_2": "485177821815",
12
+ "extra_special_token_3": "135441121756",
13
+ "extra_special_token_4": "367459894796",
14
+ "extra_special_token_5": "877482678543",
15
+ "extra_special_token_6": "457919547633",
16
+ "extra_special_token_7": "765474393376",
17
+ "extra_special_token_8": "114848338811",
18
+ "extra_special_token_9": "746285987371",
19
+ "extra_special_token_10": "649291669397",
20
+ "extra_special_token_11": "927914615679",
21
+ "extra_special_token_12": "445925149649",
22
+ "extra_special_token_13": "691587454538",
23
+ "extra_special_token_14": "143777992227",
24
+ "extra_special_token_15": "997981281989",
25
+ "extra_special_token_16": "425949483533",
26
+ "extra_special_token_17": "982993456429",
27
+ "extra_special_token_18": "718726519731",
28
+ "extra_special_token_19": "172599315861",
29
+ "extra_special_token_20": "643489267333",
30
+ "extra_special_token_21": "282322838685",
31
+ "extra_special_token_22": "781653545886",
32
+ "extra_special_token_23": "796415361892",
33
+ "extra_special_token_24": "841991688488",
34
+ "extra_special_token_25": "211411365397",
35
+ "extra_special_token_26": "698218415444",
36
+ "extra_special_token_27": "355977139358",
37
+ "extra_special_token_28": "682564697312",
38
+ "extra_special_token_29": "383837596997",
39
+ "extra_special_token_30": "689362171782",
40
+ "extra_special_token_31": "749966767285",
41
+ "extra_special_token_32": "753159165157",
42
+ "extra_special_token_33": "795693824762",
43
+ "extra_special_token_34": "669689115557",
44
+ "extra_special_token_35": "327491773134",
45
+ "extra_special_token_36": "983569279932",
46
+ "extra_special_token_37": "612128769512",
47
+ "extra_special_token_38": "374327157578",
48
+ "extra_special_token_39": "311632789559",
49
+ "extra_special_token_40": "523918658846",
50
+ "extra_special_token_41": "765981581453",
51
+ "extra_special_token_42": "794825141891",
52
+ "extra_special_token_43": "873898736873",
53
+ "extra_special_token_44": "447445629421",
54
+ "extra_special_token_45": "473822473819",
55
+ "extra_special_token_46": "181439694557",
56
+ "extra_special_token_47": "592538279337",
57
+ "extra_special_token_48": "668134915514",
58
+ "extra_special_token_49": "643692393748",
59
+ "extra_special_token_50": "696651276628",
60
+ "extra_special_token_51": "853859348234",
61
+ "extra_special_token_52": "778466723723",
62
+ "extra_special_token_53": "929826356991",
63
+ "extra_special_token_54": "272362973463",
64
+ "extra_special_token_55": "694235616268",
65
+ "extra_special_token_56": "281673864127",
66
+ "extra_special_token_57": "479676316326",
67
+ "extra_special_token_58": "646979124677",
68
+ "extra_special_token_59": "922327493433",
69
+ "extra_special_token_60": "883685933161",
70
+ "extra_special_token_61": "264259917554",
71
+ "extra_special_token_62": "836746273134",
72
+ "extra_special_token_63": "658481324922",
73
+ "extra_special_token_64": "481884157827",
74
+ "extra_special_token_65": "587787496812",
75
+ "extra_special_token_66": "579184949249",
76
+ "extra_special_token_67": "912193598348",
77
+ "extra_special_token_68": "529679678956",
78
+ "extra_special_token_69": "795838284624",
79
+ "extra_special_token_70": "159337222655",
80
+ "extra_special_token_71": "173781362446",
81
+ "extra_special_token_72": "773687856563",
82
+ "extra_special_token_73": "535787224917",
83
+ "extra_special_token_74": "351885857332",
84
+ "extra_special_token_75": "578827344666",
85
+ "extra_special_token_76": "198462689911",
86
+ "extra_special_token_77": "722618266242",
87
+ "extra_special_token_78": "952872416512",
88
+ "extra_special_token_79": "517778845323",
89
+ "extra_special_token_80": "749665846687",
90
+ "extra_special_token_81": "661436365453",
91
+ "extra_special_token_82": "259666844669",
92
+ "extra_special_token_83": "242851284913",
93
+ "extra_special_token_84": "514532995959",
94
+ "extra_special_token_85": "161588262349",
95
+ "extra_special_token_86": "742765629356",
96
+ "extra_special_token_87": "225164373623",
97
+ "extra_special_token_88": "676539973863",
98
+ "extra_special_token_89": "826214551218",
99
+ "extra_special_token_90": "182345464792",
100
+ "extra_special_token_91": "232776999554",
101
+ "extra_special_token_92": "337326533813",
102
+ "extra_special_token_93": "676676697292",
103
+ "extra_special_token_94": "929185622831",
104
+ "extra_special_token_95": "545512344383",
105
+ "extra_special_token_96": "499444466686",
106
+ "extra_special_token_97": "314697386682",
107
+ "extra_special_token_98": "517379856925",
108
+ "extra_special_token_99": "379557332953",
109
+ "extra_special_token_100": "614797267726",
110
+ "extra_special_token_101": "429781429464",
111
+ "extra_special_token_102": "922466849763",
112
+ "extra_special_token_103": "721737645236",
113
+ "extra_special_token_104": "479227349997",
114
+ "extra_special_token_105": "136931728327",
115
+ "extra_special_token_106": "259533577263",
116
+ "extra_special_token_107": "488538864842",
117
+ "extra_special_token_108": "937495658852",
118
+ "extra_special_token_109": "489991411364",
119
+ "extra_special_token_110": "499148455254",
120
+ "extra_special_token_111": "441373944925",
121
+ "extra_special_token_112": "899151413682",
122
+ "extra_special_token_113": "467893531755",
123
+ "extra_special_token_114": "527117488925",
124
+ "extra_special_token_115": "928335588653",
125
+ "extra_special_token_116": "374439448821",
126
+ "extra_special_token_117": "879425227932",
127
+ "extra_special_token_118": "867678158885",
128
+ "extra_special_token_119": "399749397872",
129
+ "extra_special_token_120": "129693547287",
130
+ "extra_special_token_121": "689285841825",
131
+ "extra_special_token_122": "771619544974",
132
+ "extra_special_token_123": "724883568652",
133
+ "extra_special_token_124": "516968424863",
134
+ "extra_special_token_125": "733737988257",
135
+ "extra_special_token_126": "852347289392",
136
+ "extra_special_token_127": "296953381169",
137
+ "extra_special_token_128": "377273562477",
138
+ "extra_special_token_129": "262296912232",
139
+ "extra_special_token_130": "547149832394",
140
+ "extra_special_token_131": "298464134954",
141
+ "extra_special_token_132": "216667245274",
142
+ "extra_special_token_133": "843998562287",
143
+ "extra_special_token_134": "572154333646",
144
+ "extra_special_token_135": "124589118494",
145
+ "extra_special_token_136": "841824384614",
146
+ "extra_special_token_137": "232896526252",
147
+ "extra_special_token_138": "295448593321",
148
+ "extra_special_token_139": "123741461297",
149
+ "extra_special_token_140": "653573457168",
150
+ "extra_special_token_141": "196735786156",
151
+ "extra_special_token_142": "377338713663",
152
+ "extra_special_token_143": "964342468552",
153
+ "extra_special_token_144": "586855179568",
154
+ "extra_special_token_145": "484773717614",
155
+ "extra_special_token_146": "894885246797",
156
+ "extra_special_token_147": "677896358599",
157
+ "extra_special_token_148": "848845611563",
158
+ "extra_special_token_149": "851852651677",
159
+ "extra_special_token_150": "398549545767",
160
+ "extra_special_token_151": "454244839926",
161
+ "extra_special_token_152": "799364566435",
162
+ "extra_special_token_153": "967114116556",
163
+ "extra_special_token_154": "817378986438",
164
+ "extra_special_token_155": "233795848681",
165
+ "extra_special_token_156": "824387273757",
166
+ "extra_special_token_157": "916198946615",
167
+ "extra_special_token_158": "563117729724",
168
+ "extra_special_token_159": "951794811935",
169
+ "extra_special_token_160": "374598961236",
170
+ "extra_special_token_161": "922867396683",
171
+ "extra_special_token_162": "765737843639",
172
+ "extra_special_token_163": "175469284871",
173
+ "extra_special_token_164": "231853711778",
174
+ "extra_special_token_165": "662426712668",
175
+ "extra_special_token_166": "711412347158",
176
+ "extra_special_token_167": "753466987363",
177
+ "extra_special_token_168": "513361312532",
178
+ "extra_special_token_169": "712992815957",
179
+ "extra_special_token_170": "971621888444",
180
+ "extra_special_token_171": "829235161526",
181
+ "extra_special_token_172": "585544633356",
182
+ "extra_special_token_173": "582471228164",
183
+ "extra_special_token_174": "678666359123",
184
+ "extra_special_token_175": "557533689478",
185
+ "extra_special_token_176": "632962475133",
186
+ "extra_special_token_177": "484489193824",
187
+ "extra_special_token_178": "489562189822",
188
+ "extra_special_token_179": "589547936288",
189
+ "extra_special_token_180": "363214487524",
190
+ "extra_special_token_181": "244885399387",
191
+ "extra_special_token_182": "431751228368",
192
+ "extra_special_token_183": "433581868192",
193
+ "extra_special_token_184": "486391569221",
194
+ "extra_special_token_185": "185438575221",
195
+ "extra_special_token_186": "126574388585",
196
+ "extra_special_token_187": "741757479784",
197
+ "extra_special_token_188": "529854679937",
198
+ "extra_special_token_189": "996116119839",
199
+ "extra_special_token_190": "616248973917",
200
+ "extra_special_token_191": "763531783491",
201
+ "extra_special_token_192": "955456118295",
202
+ "extra_special_token_193": "364196983365",
203
+ "extra_special_token_194": "195792996468",
204
+ "extra_special_token_195": "151859598873",
205
+ "extra_special_token_196": "399223169721",
206
+ "extra_special_token_197": "938488813964",
207
+ "extra_special_token_198": "961981959227",
208
+ "extra_special_token_199": "183368827562",
209
+ "extra_special_token_200": "533417736566",
210
+ "extra_special_token_201": "786391632558",
211
+ "extra_special_token_202": "665661658354",
212
+ "extra_special_token_203": "693281533643",
213
+ "extra_special_token_204": "475794684356",
214
+ "extra_special_token_205": "652154162978",
215
+ "extra_special_token_206": "753233719644",
216
+ "extra_special_token_207": "668514843129",
217
+ "extra_special_token_208": "819162623892",
218
+ "extra_special_token_209": "941169431859",
219
+ "extra_special_token_210": "877385381798",
220
+ "extra_special_token_211": "752644929761",
221
+ "extra_special_token_212": "881136466196",
222
+ "extra_special_token_213": "275597777299",
223
+ "extra_special_token_214": "731681792655",
224
+ "extra_special_token_215": "961133895172",
225
+ "extra_special_token_216": "864718285734",
226
+ "extra_special_token_217": "963852916563",
227
+ "extra_special_token_218": "319584985416",
228
+ "extra_special_token_219": "563365646341",
229
+ "extra_special_token_220": "811371928234",
230
+ "extra_special_token_221": "837131396371",
231
+ "extra_special_token_222": "267514771964",
232
+ "extra_special_token_223": "944513428457",
233
+ "extra_special_token_224": "117298239631",
234
+ "extra_special_token_225": "158142752582",
235
+ "extra_special_token_226": "252867443568",
236
+ "extra_special_token_227": "839269684865",
237
+ "extra_special_token_228": "612788593128",
238
+ "extra_special_token_229": "145669731981",
239
+ "extra_special_token_230": "121557291859",
240
+ "extra_special_token_231": "245416776926",
241
+ "extra_special_token_232": "799417897197",
242
+ "extra_special_token_233": "997958836435",
243
+ "extra_special_token_234": "892336777248",
244
+ "extra_special_token_235": "158929292238",
245
+ "extra_special_token_236": "581976444672",
246
+ "extra_special_token_237": "897784492783",
247
+ "extra_special_token_238": "492373714791",
248
+ "extra_special_token_239": "512659818733",
249
+ "extra_special_token_240": "881112998642",
250
+ "extra_special_token_241": "619454958782",
251
+ "extra_special_token_242": "431149748713",
252
+ "extra_special_token_243": "624221476921",
253
+ "extra_special_token_244": "125866399464",
254
+ "extra_special_token_245": "339882449689",
255
+ "extra_special_token_246": "186198784585",
256
+ "extra_special_token_247": "943193294691",
257
+ "extra_special_token_248": "955668961269",
258
+ "extra_special_token_249": "232787996724",
259
+ "extra_special_token_250": "215671314196",
260
+ "extra_special_token_251": "286173241916",
261
+ "extra_special_token_252": "745977673725",
262
+ "extra_special_token_253": "556976448182",
263
+ "extra_special_token_254": "599961512792",
264
+ "extra_special_token_255": "766294538337",
265
+ "extra_special_token_256": "934912591213",
266
+ "extra_special_token_257": "295118729589",
267
+ "extra_special_token_258": "529455466433",
268
+ "extra_special_token_259": "196119929397",
269
+ "extra_special_token_260": "379571934299",
270
+ "extra_special_token_261": "251789649997",
271
+ "extra_special_token_262": "564544131355",
272
+ "extra_special_token_263": "244371196654",
273
+ "extra_special_token_264": "384598329253",
274
+ "extra_special_token_265": "887753195844",
275
+ "extra_special_token_266": "364947325679",
276
+ "extra_special_token_267": "655517954651",
277
+ "extra_special_token_268": "673948786567",
278
+ "extra_special_token_269": "857231548835",
279
+ "extra_special_token_270": "816115936673",
280
+ "extra_special_token_271": "644234165531",
281
+ "extra_special_token_272": "182782912224",
282
+ "extra_special_token_273": "234316622259",
283
+ "extra_special_token_274": "421369185549",
284
+ "extra_special_token_275": "434632855397",
285
+ "extra_special_token_276": "921889371893",
286
+ "extra_special_token_277": "415956914763",
287
+ "extra_special_token_278": "598916996413",
288
+ "extra_special_token_279": "773671349113",
289
+ "extra_special_token_280": "952465217972",
290
+ "extra_special_token_281": "117657531962",
291
+ "extra_special_token_282": "729825168745",
292
+ "extra_special_token_283": "691315125346",
293
+ "extra_special_token_284": "768461952319",
294
+ "extra_special_token_285": "664847713559",
295
+ "extra_special_token_286": "953267689786",
296
+ "extra_special_token_287": "886464195129",
297
+ "extra_special_token_288": "824488329416",
298
+ "extra_special_token_289": "837873762491",
299
+ "extra_special_token_290": "532833541879",
300
+ "extra_special_token_291": "669183782449",
301
+ "extra_special_token_292": "941976537588",
302
+ "extra_special_token_293": "739394546916",
303
+ "extra_special_token_294": "267954879268",
304
+ "extra_special_token_295": "637551427887",
305
+ "extra_special_token_296": "217756494954",
306
+ "extra_special_token_297": "524444658383",
307
+ "extra_special_token_298": "117783274348",
308
+ "extra_special_token_299": "138218735276",
309
+ "extra_special_token_300": "814611949491",
310
+ "extra_special_token_301": "711641973413",
311
+ "extra_special_token_302": "499156317423",
312
+ "extra_special_token_303": "515856611931",
313
+ "extra_special_token_304": "454164859837",
314
+ "extra_special_token_305": "345271433112",
315
+ "extra_special_token_306": "462294118988",
316
+ "extra_special_token_307": "511785788222",
317
+ "extra_special_token_308": "497294727353",
318
+ "extra_special_token_309": "866519986723",
319
+ "extra_special_token_310": "334513529294",
320
+ "extra_special_token_311": "549946382131",
321
+ "extra_special_token_312": "284445431422",
322
+ "extra_special_token_313": "396521188476",
323
+ "extra_special_token_314": "421435255895",
324
+ "extra_special_token_315": "133373659361",
325
+ "extra_special_token_316": "322683334381",
326
+ "extra_special_token_317": "228358422847",
327
+ "extra_special_token_318": "291762694874",
328
+ "extra_special_token_319": "143182978129",
329
+ "extra_special_token_320": "511923256573",
330
+ "extra_special_token_321": "327158398268",
331
+ "extra_special_token_322": "879764613759",
332
+ "extra_special_token_323": "564395222747",
333
+ "extra_special_token_324": "451161679736",
334
+ "extra_special_token_325": "538631466654",
335
+ "extra_special_token_326": "221762325616",
336
+ "extra_special_token_327": "218391991184",
337
+ "extra_special_token_328": "322589379462",
338
+ "extra_special_token_329": "876537814263",
339
+ "extra_special_token_330": "152676556624",
340
+ "extra_special_token_331": "332522971941",
341
+ "extra_special_token_332": "884354318946",
342
+ "extra_special_token_333": "513349618943",
343
+ "extra_special_token_334": "116639746413",
344
+ "extra_special_token_335": "635185846287",
345
+ "extra_special_token_336": "993832498489",
346
+ "extra_special_token_337": "813981174797",
347
+ "extra_special_token_338": "438745114173",
348
+ "extra_special_token_339": "983493951323",
349
+ "extra_special_token_340": "724492262421",
350
+ "extra_special_token_341": "622553389126",
351
+ "extra_special_token_342": "889965243135",
352
+ "extra_special_token_343": "364492359246",
353
+ "extra_special_token_344": "154962668224",
354
+ "extra_special_token_345": "179564995814",
355
+ "extra_special_token_346": "418412875665",
356
+ "extra_special_token_347": "718951851413",
357
+ "extra_special_token_348": "699446724178",
358
+ "extra_special_token_349": "624266421831",
359
+ "extra_special_token_350": "815458725125",
360
+ "extra_special_token_351": "455423278865",
361
+ "extra_special_token_352": "393741199486",
362
+ "extra_special_token_353": "328552864359",
363
+ "extra_special_token_354": "211662639865",
364
+ "extra_special_token_355": "218784516525",
365
+ "extra_special_token_356": "762486672996",
366
+ "extra_special_token_357": "142799718159",
367
+ "extra_special_token_358": "858146415154",
368
+ "extra_special_token_359": "767858144912",
369
+ "extra_special_token_360": "571317457151",
370
+ "extra_special_token_361": "635127952696",
371
+ "extra_special_token_362": "116427191984",
372
+ "extra_special_token_363": "268921994538",
373
+ "extra_special_token_364": "523937669294",
374
+ "extra_special_token_365": "165429152138",
375
+ "extra_special_token_366": "739246183345",
376
+ "extra_special_token_367": "591464355756",
377
+ "extra_special_token_368": "212985874612",
378
+ "extra_special_token_369": "191887635211",
379
+ "extra_special_token_370": "967214577653",
380
+ "extra_special_token_371": "119342152414",
381
+ "extra_special_token_372": "946444632795",
382
+ "extra_special_token_373": "618423867817",
383
+ "extra_special_token_374": "228565148417",
384
+ "extra_special_token_375": "729116422489",
385
+ "extra_special_token_376": "527874729936",
386
+ "extra_special_token_377": "739784153482",
387
+ "extra_special_token_378": "387763951128",
388
+ "extra_special_token_379": "331369926711",
389
+ "extra_special_token_380": "562716493614",
390
+ "extra_special_token_381": "739667844957",
391
+ "extra_special_token_382": "562389434565",
392
+ "extra_special_token_383": "256497188281",
393
+ "extra_special_token_384": "859927364588",
394
+ "extra_special_token_385": "417668946583",
395
+ "extra_special_token_386": "357621613582",
396
+ "extra_special_token_387": "438435178228",
397
+ "extra_special_token_388": "485692541169",
398
+ "extra_special_token_389": "825815739116",
399
+ "extra_special_token_390": "342221452223",
400
+ "extra_special_token_391": "697747991249",
401
+ "extra_special_token_392": "716763689965",
402
+ "extra_special_token_393": "141499982867",
403
+ "extra_special_token_394": "818479319499",
404
+ "extra_special_token_395": "336813343298",
405
+ "extra_special_token_396": "594688742928",
406
+ "extra_special_token_397": "472129283475",
407
+ "extra_special_token_398": "514354144759",
408
+ "extra_special_token_399": "349249721685",
409
+ "extra_special_token_400": "546276298359",
410
+ "extra_special_token_401": "353755529131",
411
+ "extra_special_token_402": "315534574435",
412
+ "extra_special_token_403": "523723475786",
413
+ "extra_special_token_404": "215826764872",
414
+ "extra_special_token_405": "367968398551",
415
+ "extra_special_token_406": "569853653352",
416
+ "extra_special_token_407": "389715484387",
417
+ "extra_special_token_408": "293847485454",
418
+ "extra_special_token_409": "714738141818",
419
+ "extra_special_token_410": "178478368922",
420
+ "extra_special_token_411": "581493616981",
421
+ "extra_special_token_412": "589439538674",
422
+ "extra_special_token_413": "846657726193",
423
+ "extra_special_token_414": "722339992679",
424
+ "extra_special_token_415": "138154781148",
425
+ "extra_special_token_416": "757785319772",
426
+ "extra_special_token_417": "492516914298",
427
+ "extra_special_token_418": "919181521716",
428
+ "extra_special_token_419": "985781138935",
429
+ "extra_special_token_420": "476969195485",
430
+ "extra_special_token_421": "313145133463",
431
+ "extra_special_token_422": "758963111966",
432
+ "extra_special_token_423": "147541537162",
433
+ "extra_special_token_424": "557163366873",
434
+ "extra_special_token_425": "144373897488",
435
+ "extra_special_token_426": "522515164754",
436
+ "extra_special_token_427": "724964923582",
437
+ "extra_special_token_428": "284776712475",
438
+ "extra_special_token_429": "375429755114",
439
+ "extra_special_token_430": "181233596124",
440
+ "extra_special_token_431": "948585673431",
441
+ "extra_special_token_432": "243165586174",
442
+ "extra_special_token_433": "396847976144",
443
+ "extra_special_token_434": "997724962668",
444
+ "extra_special_token_435": "558837194455",
445
+ "extra_special_token_436": "163165456396",
446
+ "extra_special_token_437": "378749551722",
447
+ "extra_special_token_438": "161238482259",
448
+ "extra_special_token_439": "754978243758",
449
+ "extra_special_token_440": "195388849133",
450
+ "extra_special_token_441": "229775525672",
451
+ "extra_special_token_442": "262437452884",
452
+ "extra_special_token_443": "441377892146",
453
+ "extra_special_token_444": "451885565366",
454
+ "extra_special_token_445": "981277526855",
455
+ "extra_special_token_446": "762495822823",
456
+ "extra_special_token_447": "368763327262",
457
+ "extra_special_token_448": "757422791351",
458
+ "extra_special_token_449": "636324136426",
459
+ "extra_special_token_450": "214193645583",
460
+ "extra_special_token_451": "412843856172",
461
+ "extra_special_token_452": "179386156569",
462
+ "extra_special_token_453": "756916173536",
463
+ "extra_special_token_454": "892697125149",
464
+ "extra_special_token_455": "625334487352",
465
+ "extra_special_token_456": "941861857715",
466
+ "extra_special_token_457": "887417525236",
467
+ "extra_special_token_458": "649516938598",
468
+ "extra_special_token_459": "717628619782",
469
+ "extra_special_token_460": "438124184139",
470
+ "extra_special_token_461": "547563892268",
471
+ "extra_special_token_462": "856317483891",
472
+ "extra_special_token_463": "313313831273",
473
+ "extra_special_token_464": "371496153876",
474
+ "extra_special_token_465": "587541149322",
475
+ "extra_special_token_466": "265847332563",
476
+ "extra_special_token_467": "449549215429",
477
+ "extra_special_token_468": "163497196769",
478
+ "extra_special_token_469": "861342291298",
479
+ "extra_special_token_470": "268433315926",
480
+ "extra_special_token_471": "774679513717",
481
+ "extra_special_token_472": "851254219729",
482
+ "extra_special_token_473": "583527834464",
483
+ "extra_special_token_474": "488496781997",
484
+ "extra_special_token_475": "556814553861",
485
+ "extra_special_token_476": "482829231639",
486
+ "extra_special_token_477": "618878266619",
487
+ "extra_special_token_478": "147444452794",
488
+ "extra_special_token_479": "949235426629",
489
+ "extra_special_token_480": "357299947518",
490
+ "extra_special_token_481": "175528632226",
491
+ "extra_special_token_482": "645527857972",
492
+ "extra_special_token_483": "186872457894",
493
+ "extra_special_token_484": "552738847828",
494
+ "extra_special_token_485": "626748382482",
495
+ "extra_special_token_486": "921894985642",
496
+ "extra_special_token_487": "943878645871",
497
+ "extra_special_token_488": "859289776479",
498
+ "extra_special_token_489": "614583493135",
499
+ "extra_special_token_490": "933775286797",
500
+ "extra_special_token_491": "332234613346",
501
+ "extra_special_token_492": "325196781219",
502
+ "extra_special_token_493": "142526557681",
503
+ "extra_special_token_494": "356722692178",
504
+ "extra_special_token_495": "449318681694",
505
+ "extra_special_token_496": "687284547244",
506
+ "extra_special_token_497": "947262995132",
507
+ "extra_special_token_498": "893974619684",
508
+ "extra_special_token_499": "797238311233"
509
+ },
510
+ "is_local": false,
511
+ "model_max_length": 32768,
512
+ "pad_token": "<|endoftext|>",
513
+ "split_special_tokens": false,
514
+ "tokenizer_class": "Qwen2Tokenizer",
515
+ "unk_token": null
516
+ }
qwen1.5-moe-a2.7b/dp3_attn/train.log ADDED
@@ -0,0 +1,197 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 2026-03-26 10:02:04,480 [INFO] new_opacus_codex.train_steps: epoch=1 step=5 loss=23.5271
2
+ 2026-03-26 10:03:06,952 [INFO] new_opacus_codex.train_steps: epoch=1 step=10 loss=23.3547
3
+ 2026-03-26 10:03:14,504 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=10 eval_loss=24.6204 duration_sec=7.53
4
+ 2026-03-26 10:08:18,863 [INFO] new_opacus_codex.train_steps: epoch=1 step=5 loss=23.7846
5
+ 2026-03-26 10:08:57,957 [INFO] new_opacus_codex.train_steps: epoch=1 step=10 loss=23.7466
6
+ 2026-03-26 10:09:05,499 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=10 eval_loss=24.6417 duration_sec=7.53
7
+ 2026-03-26 10:09:44,030 [INFO] new_opacus_codex.train_steps: epoch=1 step=15 loss=23.7345
8
+ 2026-03-26 10:10:22,552 [INFO] new_opacus_codex.train_steps: epoch=1 step=20 loss=23.6740
9
+ 2026-03-26 10:10:30,085 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=20 eval_loss=24.6035 duration_sec=7.52
10
+ 2026-03-26 10:11:07,464 [INFO] new_opacus_codex.train_steps: epoch=1 step=25 loss=23.7710
11
+ 2026-03-26 10:11:45,249 [INFO] new_opacus_codex.train_steps: epoch=1 step=30 loss=23.7862
12
+ 2026-03-26 10:11:52,831 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=30 eval_loss=24.5213 duration_sec=7.57
13
+ 2026-03-26 10:12:31,188 [INFO] new_opacus_codex.train_steps: epoch=1 step=35 loss=23.5857
14
+ 2026-03-26 10:13:08,343 [INFO] new_opacus_codex.train_steps: epoch=1 step=40 loss=23.6999
15
+ 2026-03-26 10:13:15,929 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=40 eval_loss=24.4214 duration_sec=7.57
16
+ 2026-03-26 10:13:54,645 [INFO] new_opacus_codex.train_steps: epoch=1 step=45 loss=23.6816
17
+ 2026-03-26 10:14:31,958 [INFO] new_opacus_codex.train_steps: epoch=1 step=50 loss=23.5223
18
+ 2026-03-26 10:14:39,531 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=50 eval_loss=24.3189 duration_sec=7.56
19
+ 2026-03-26 10:15:16,864 [INFO] new_opacus_codex.train_steps: epoch=1 step=55 loss=23.5522
20
+ 2026-03-26 10:15:53,249 [INFO] new_opacus_codex.train_steps: epoch=1 step=60 loss=23.4770
21
+ 2026-03-26 10:16:00,855 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=60 eval_loss=24.2110 duration_sec=7.59
22
+ 2026-03-26 10:16:38,090 [INFO] new_opacus_codex.train_steps: epoch=1 step=65 loss=23.5034
23
+ 2026-03-26 10:17:15,168 [INFO] new_opacus_codex.train_steps: epoch=1 step=70 loss=23.5821
24
+ 2026-03-26 10:17:22,758 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=70 eval_loss=24.1045 duration_sec=7.58
25
+ 2026-03-26 10:18:00,622 [INFO] new_opacus_codex.train_steps: epoch=1 step=75 loss=23.5180
26
+ 2026-03-26 10:18:38,658 [INFO] new_opacus_codex.train_steps: epoch=1 step=80 loss=23.3221
27
+ 2026-03-26 10:18:46,274 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=80 eval_loss=23.9967 duration_sec=7.60
28
+ 2026-03-26 10:19:25,361 [INFO] new_opacus_codex.train_steps: epoch=1 step=85 loss=23.2170
29
+ 2026-03-26 10:20:04,167 [INFO] new_opacus_codex.train_steps: epoch=1 step=90 loss=23.1383
30
+ 2026-03-26 10:20:12,578 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=90 eval_loss=23.8849 duration_sec=8.40
31
+ 2026-03-26 10:20:50,694 [INFO] new_opacus_codex.train_steps: epoch=1 step=95 loss=23.2622
32
+ 2026-03-26 10:25:42,902 [INFO] new_opacus_codex.train_steps: epoch=1 step=5 loss=23.7842
33
+ 2026-03-26 10:26:21,991 [INFO] new_opacus_codex.train_steps: epoch=1 step=10 loss=23.7316
34
+ 2026-03-26 10:26:29,581 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=10 eval_loss=24.5739 duration_sec=7.58
35
+ 2026-03-26 10:27:08,089 [INFO] new_opacus_codex.train_steps: epoch=1 step=15 loss=23.6730
36
+ 2026-03-26 10:27:45,862 [INFO] new_opacus_codex.train_steps: epoch=1 step=20 loss=23.5347
37
+ 2026-03-26 10:27:53,424 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=20 eval_loss=24.2967 duration_sec=7.55
38
+ 2026-03-26 10:28:31,338 [INFO] new_opacus_codex.train_steps: epoch=1 step=25 loss=23.5104
39
+ 2026-03-26 10:29:09,037 [INFO] new_opacus_codex.train_steps: epoch=1 step=30 loss=23.3632
40
+ 2026-03-26 10:29:16,622 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=30 eval_loss=23.8265 duration_sec=7.57
41
+ 2026-03-26 10:29:54,827 [INFO] new_opacus_codex.train_steps: epoch=1 step=35 loss=22.9746
42
+ 2026-03-26 10:30:32,183 [INFO] new_opacus_codex.train_steps: epoch=1 step=40 loss=22.8783
43
+ 2026-03-26 10:30:40,426 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=40 eval_loss=23.2472 duration_sec=8.22
44
+ 2026-03-26 10:31:19,025 [INFO] new_opacus_codex.train_steps: epoch=1 step=45 loss=22.6713
45
+ 2026-03-26 10:31:55,306 [INFO] new_opacus_codex.train_steps: epoch=1 step=50 loss=22.2647
46
+ 2026-03-26 10:32:02,879 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=50 eval_loss=22.5576 duration_sec=7.55
47
+ 2026-03-26 10:32:40,324 [INFO] new_opacus_codex.train_steps: epoch=1 step=55 loss=21.9748
48
+ 2026-03-26 10:33:18,123 [INFO] new_opacus_codex.train_steps: epoch=1 step=60 loss=21.6280
49
+ 2026-03-26 10:33:25,697 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=60 eval_loss=21.7053 duration_sec=7.56
50
+ 2026-03-26 10:34:04,438 [INFO] new_opacus_codex.train_steps: epoch=1 step=65 loss=21.3198
51
+ 2026-03-26 10:34:42,254 [INFO] new_opacus_codex.train_steps: epoch=1 step=70 loss=21.0684
52
+ 2026-03-26 10:34:49,850 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=70 eval_loss=20.6912 duration_sec=7.59
53
+ 2026-03-26 10:35:28,188 [INFO] new_opacus_codex.train_steps: epoch=1 step=75 loss=20.6064
54
+ 2026-03-26 10:36:06,905 [INFO] new_opacus_codex.train_steps: epoch=1 step=80 loss=19.8604
55
+ 2026-03-26 10:36:14,459 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=80 eval_loss=19.4986 duration_sec=7.51
56
+ 2026-03-26 10:36:53,052 [INFO] new_opacus_codex.train_steps: epoch=1 step=85 loss=19.2182
57
+ 2026-03-26 10:37:32,520 [INFO] new_opacus_codex.train_steps: epoch=1 step=90 loss=18.7011
58
+ 2026-03-26 10:37:40,236 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=90 eval_loss=18.1367 duration_sec=7.70
59
+ 2026-03-26 10:38:17,270 [INFO] new_opacus_codex.train_steps: epoch=1 step=95 loss=18.3133
60
+ 2026-03-26 10:38:55,412 [INFO] new_opacus_codex.train_steps: epoch=1 step=100 loss=17.7769
61
+ 2026-03-26 10:39:03,637 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=100 eval_loss=16.7268 duration_sec=8.21
62
+ 2026-03-26 10:39:42,489 [INFO] new_opacus_codex.train_steps: epoch=1 step=105 loss=17.0871
63
+ 2026-03-26 10:40:18,938 [INFO] new_opacus_codex.train_steps: epoch=1 step=110 loss=16.6455
64
+ 2026-03-26 10:40:26,595 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=110 eval_loss=15.7859 duration_sec=7.65
65
+ 2026-03-26 10:41:03,943 [INFO] new_opacus_codex.train_steps: epoch=1 step=115 loss=16.0993
66
+ 2026-03-26 10:41:41,318 [INFO] new_opacus_codex.train_steps: epoch=1 step=120 loss=15.8509
67
+ 2026-03-26 10:41:48,954 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=120 eval_loss=15.1876 duration_sec=7.63
68
+ 2026-03-26 10:42:26,254 [INFO] new_opacus_codex.train_steps: epoch=1 step=125 loss=15.7489
69
+ 2026-03-26 10:43:12,152 [INFO] new_opacus_codex.train_steps: epoch=1 step=130 loss=15.1828
70
+ 2026-03-26 10:43:21,361 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=130 eval_loss=14.6953 duration_sec=9.20
71
+ 2026-03-26 10:44:07,775 [INFO] new_opacus_codex.train_steps: epoch=1 step=135 loss=14.9928
72
+ 2026-03-26 10:45:03,481 [INFO] new_opacus_codex.train_steps: epoch=1 step=140 loss=15.1492
73
+ 2026-03-26 10:45:14,040 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=140 eval_loss=14.2579 duration_sec=10.54
74
+ 2026-03-26 10:46:11,639 [INFO] new_opacus_codex.train_steps: epoch=1 step=145 loss=14.7883
75
+ 2026-03-26 10:47:29,030 [INFO] new_opacus_codex.train_steps: epoch=1 step=150 loss=14.5153
76
+ 2026-03-26 10:47:45,015 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=150 eval_loss=13.8999 duration_sec=15.97
77
+ 2026-03-26 10:49:00,981 [INFO] new_opacus_codex.train_steps: epoch=1 step=155 loss=14.1214
78
+ 2026-03-26 10:50:24,032 [INFO] new_opacus_codex.train_steps: epoch=1 step=160 loss=14.0007
79
+ 2026-03-26 10:50:34,943 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=160 eval_loss=13.5788 duration_sec=10.89
80
+ 2026-03-26 10:51:28,743 [INFO] new_opacus_codex.train_steps: epoch=1 step=165 loss=14.3488
81
+ 2026-03-26 10:52:22,611 [INFO] new_opacus_codex.train_steps: epoch=1 step=170 loss=14.0904
82
+ 2026-03-26 10:52:32,950 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=170 eval_loss=13.2596 duration_sec=10.32
83
+ 2026-03-26 10:53:31,679 [INFO] new_opacus_codex.train_steps: epoch=1 step=175 loss=13.6605
84
+ 2026-03-26 10:54:26,420 [INFO] new_opacus_codex.train_steps: epoch=1 step=180 loss=13.7271
85
+ 2026-03-26 10:54:38,199 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=180 eval_loss=12.9654 duration_sec=11.74
86
+ 2026-03-26 10:55:33,500 [INFO] new_opacus_codex.train_steps: epoch=1 step=185 loss=13.6569
87
+ 2026-03-26 10:56:31,266 [INFO] new_opacus_codex.train_steps: epoch=2 step=190 loss=13.7310
88
+ 2026-03-26 10:56:38,905 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=190 eval_loss=12.6947 duration_sec=7.63
89
+ 2026-03-26 10:57:17,556 [INFO] new_opacus_codex.train_steps: epoch=2 step=195 loss=13.6424
90
+ 2026-03-26 10:57:54,671 [INFO] new_opacus_codex.train_steps: epoch=2 step=200 loss=13.2158
91
+ 2026-03-26 10:58:02,313 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=200 eval_loss=12.4482 duration_sec=7.63
92
+ 2026-03-26 10:58:39,800 [INFO] new_opacus_codex.train_steps: epoch=2 step=205 loss=12.9208
93
+ 2026-03-26 10:59:18,393 [INFO] new_opacus_codex.train_steps: epoch=2 step=210 loss=12.8071
94
+ 2026-03-26 10:59:26,004 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=210 eval_loss=12.2004 duration_sec=7.59
95
+ 2026-03-26 11:00:03,650 [INFO] new_opacus_codex.train_steps: epoch=2 step=215 loss=12.7568
96
+ 2026-03-26 11:00:41,416 [INFO] new_opacus_codex.train_steps: epoch=2 step=220 loss=12.8060
97
+ 2026-03-26 11:00:49,080 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=220 eval_loss=11.9388 duration_sec=7.65
98
+ 2026-03-26 11:01:27,711 [INFO] new_opacus_codex.train_steps: epoch=2 step=225 loss=12.5291
99
+ 2026-03-26 11:02:04,404 [INFO] new_opacus_codex.train_steps: epoch=2 step=230 loss=12.4607
100
+ 2026-03-26 11:02:12,205 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=230 eval_loss=11.6371 duration_sec=7.78
101
+ 2026-03-26 11:02:48,976 [INFO] new_opacus_codex.train_steps: epoch=2 step=235 loss=12.5642
102
+ 2026-03-26 11:03:26,549 [INFO] new_opacus_codex.train_steps: epoch=2 step=240 loss=12.3893
103
+ 2026-03-26 11:03:34,161 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=240 eval_loss=11.2820 duration_sec=7.57
104
+ 2026-03-26 11:04:12,101 [INFO] new_opacus_codex.train_steps: epoch=2 step=245 loss=11.9305
105
+ 2026-03-26 11:04:50,826 [INFO] new_opacus_codex.train_steps: epoch=2 step=250 loss=11.7338
106
+ 2026-03-26 11:04:58,450 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=250 eval_loss=10.8583 duration_sec=7.50
107
+ 2026-03-26 11:05:36,901 [INFO] new_opacus_codex.train_steps: epoch=2 step=255 loss=11.6074
108
+ 2026-03-26 11:06:14,022 [INFO] new_opacus_codex.train_steps: epoch=2 step=260 loss=11.1919
109
+ 2026-03-26 11:06:21,652 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=260 eval_loss=10.4156 duration_sec=7.59
110
+ 2026-03-26 11:06:58,943 [INFO] new_opacus_codex.train_steps: epoch=2 step=265 loss=11.0851
111
+ 2026-03-26 11:07:36,291 [INFO] new_opacus_codex.train_steps: epoch=2 step=270 loss=11.0509
112
+ 2026-03-26 11:07:43,925 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=270 eval_loss=10.0321 duration_sec=7.61
113
+ 2026-03-26 11:08:22,222 [INFO] new_opacus_codex.train_steps: epoch=2 step=275 loss=10.8334
114
+ 2026-03-26 11:09:02,174 [INFO] new_opacus_codex.train_steps: epoch=2 step=280 loss=10.6670
115
+ 2026-03-26 11:09:09,793 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=280 eval_loss=9.7375 duration_sec=7.61
116
+ 2026-03-26 11:09:47,169 [INFO] new_opacus_codex.train_steps: epoch=2 step=285 loss=10.5509
117
+ 2026-03-26 11:10:24,489 [INFO] new_opacus_codex.train_steps: epoch=2 step=290 loss=10.4099
118
+ 2026-03-26 11:10:32,101 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=290 eval_loss=9.5191 duration_sec=7.60
119
+ 2026-03-26 11:11:10,120 [INFO] new_opacus_codex.train_steps: epoch=2 step=295 loss=10.3134
120
+ 2026-03-26 11:11:48,132 [INFO] new_opacus_codex.train_steps: epoch=2 step=300 loss=10.2900
121
+ 2026-03-26 11:11:55,818 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=300 eval_loss=9.3513 duration_sec=7.67
122
+ 2026-03-26 11:12:33,839 [INFO] new_opacus_codex.train_steps: epoch=2 step=305 loss=10.1988
123
+ 2026-03-26 11:13:12,396 [INFO] new_opacus_codex.train_steps: epoch=2 step=310 loss=10.1343
124
+ 2026-03-26 11:13:20,033 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=310 eval_loss=9.2192 duration_sec=7.62
125
+ 2026-03-26 11:13:59,311 [INFO] new_opacus_codex.train_steps: epoch=2 step=315 loss=10.0699
126
+ 2026-03-26 11:14:36,984 [INFO] new_opacus_codex.train_steps: epoch=2 step=320 loss=10.1069
127
+ 2026-03-26 11:14:44,591 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=320 eval_loss=9.1170 duration_sec=7.60
128
+ 2026-03-26 11:15:23,664 [INFO] new_opacus_codex.train_steps: epoch=2 step=325 loss=9.9682
129
+ 2026-03-26 11:16:00,812 [INFO] new_opacus_codex.train_steps: epoch=2 step=330 loss=9.9043
130
+ 2026-03-26 11:16:08,394 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=330 eval_loss=9.0326 duration_sec=7.57
131
+ 2026-03-26 11:16:44,941 [INFO] new_opacus_codex.train_steps: epoch=2 step=335 loss=9.8636
132
+ 2026-03-26 11:17:23,570 [INFO] new_opacus_codex.train_steps: epoch=2 step=340 loss=9.5794
133
+ 2026-03-26 11:17:31,299 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=340 eval_loss=8.9614 duration_sec=7.72
134
+ 2026-03-26 11:18:08,305 [INFO] new_opacus_codex.train_steps: epoch=2 step=345 loss=9.4299
135
+ 2026-03-26 11:18:46,795 [INFO] new_opacus_codex.train_steps: epoch=2 step=350 loss=9.5881
136
+ 2026-03-26 11:18:54,439 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=350 eval_loss=8.9035 duration_sec=7.64
137
+ 2026-03-26 11:19:31,669 [INFO] new_opacus_codex.train_steps: epoch=2 step=355 loss=9.9158
138
+ 2026-03-26 11:20:08,680 [INFO] new_opacus_codex.train_steps: epoch=2 step=360 loss=9.9199
139
+ 2026-03-26 11:20:16,296 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=360 eval_loss=8.8557 duration_sec=7.60
140
+ 2026-03-26 11:20:54,002 [INFO] new_opacus_codex.train_steps: epoch=2 step=365 loss=9.6665
141
+ 2026-03-26 11:21:30,766 [INFO] new_opacus_codex.train_steps: epoch=2 step=370 loss=9.4803
142
+ 2026-03-26 11:21:38,432 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=370 eval_loss=8.8149 duration_sec=7.65
143
+ 2026-03-26 11:22:28,302 [INFO] new_opacus_codex.train_steps: epoch=3 step=375 loss=9.2577
144
+ 2026-03-26 11:23:07,567 [INFO] new_opacus_codex.train_steps: epoch=3 step=380 loss=9.3074
145
+ 2026-03-26 11:23:15,345 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=380 eval_loss=8.7803 duration_sec=7.77
146
+ 2026-03-26 11:23:53,344 [INFO] new_opacus_codex.train_steps: epoch=3 step=385 loss=9.5418
147
+ 2026-03-26 11:24:31,127 [INFO] new_opacus_codex.train_steps: epoch=3 step=390 loss=9.6305
148
+ 2026-03-26 11:24:38,926 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=390 eval_loss=8.7495 duration_sec=7.79
149
+ 2026-03-26 11:25:16,218 [INFO] new_opacus_codex.train_steps: epoch=3 step=395 loss=9.4649
150
+ 2026-03-26 11:25:53,549 [INFO] new_opacus_codex.train_steps: epoch=3 step=400 loss=9.3736
151
+ 2026-03-26 11:26:01,759 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=400 eval_loss=8.7224 duration_sec=8.19
152
+ 2026-03-26 11:26:39,409 [INFO] new_opacus_codex.train_steps: epoch=3 step=405 loss=9.3111
153
+ 2026-03-26 11:27:17,696 [INFO] new_opacus_codex.train_steps: epoch=3 step=410 loss=9.4181
154
+ 2026-03-26 11:27:25,399 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=410 eval_loss=8.6996 duration_sec=7.69
155
+ 2026-03-26 11:28:02,647 [INFO] new_opacus_codex.train_steps: epoch=3 step=415 loss=9.3370
156
+ 2026-03-26 11:28:39,680 [INFO] new_opacus_codex.train_steps: epoch=3 step=420 loss=9.3384
157
+ 2026-03-26 11:28:47,463 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=420 eval_loss=8.6798 duration_sec=7.77
158
+ 2026-03-26 11:29:26,431 [INFO] new_opacus_codex.train_steps: epoch=3 step=425 loss=9.3397
159
+ 2026-03-26 11:30:03,469 [INFO] new_opacus_codex.train_steps: epoch=3 step=430 loss=9.2703
160
+ 2026-03-26 11:30:11,217 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=430 eval_loss=8.6642 duration_sec=7.73
161
+ 2026-03-26 11:30:49,747 [INFO] new_opacus_codex.train_steps: epoch=3 step=435 loss=9.2197
162
+ 2026-03-26 11:31:26,505 [INFO] new_opacus_codex.train_steps: epoch=3 step=440 loss=8.9902
163
+ 2026-03-26 11:31:34,642 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=440 eval_loss=8.6501 duration_sec=8.12
164
+ 2026-03-26 11:32:12,451 [INFO] new_opacus_codex.train_steps: epoch=3 step=445 loss=9.1127
165
+ 2026-03-26 11:32:48,965 [INFO] new_opacus_codex.train_steps: epoch=3 step=450 loss=9.4012
166
+ 2026-03-26 11:32:56,716 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=450 eval_loss=8.6387 duration_sec=7.73
167
+ 2026-03-26 11:33:33,412 [INFO] new_opacus_codex.train_steps: epoch=3 step=455 loss=9.6190
168
+ 2026-03-26 11:34:11,769 [INFO] new_opacus_codex.train_steps: epoch=3 step=460 loss=9.4811
169
+ 2026-03-26 11:34:19,606 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=460 eval_loss=8.6296 duration_sec=7.83
170
+ 2026-03-26 11:34:58,261 [INFO] new_opacus_codex.train_steps: epoch=3 step=465 loss=9.3332
171
+ 2026-03-26 11:35:35,104 [INFO] new_opacus_codex.train_steps: epoch=3 step=470 loss=9.4240
172
+ 2026-03-26 11:35:42,896 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=470 eval_loss=8.6222 duration_sec=7.79
173
+ 2026-03-26 11:36:18,601 [INFO] new_opacus_codex.train_steps: epoch=3 step=475 loss=9.4863
174
+ 2026-03-26 11:36:56,457 [INFO] new_opacus_codex.train_steps: epoch=3 step=480 loss=9.4801
175
+ 2026-03-26 11:37:04,249 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=480 eval_loss=8.6164 duration_sec=7.78
176
+ 2026-03-26 11:37:41,513 [INFO] new_opacus_codex.train_steps: epoch=3 step=485 loss=9.2427
177
+ 2026-03-26 11:38:18,550 [INFO] new_opacus_codex.train_steps: epoch=3 step=490 loss=9.3135
178
+ 2026-03-26 11:38:26,229 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=490 eval_loss=8.6120 duration_sec=7.66
179
+ 2026-03-26 11:39:04,765 [INFO] new_opacus_codex.train_steps: epoch=3 step=495 loss=9.5749
180
+ 2026-03-26 11:39:43,965 [INFO] new_opacus_codex.train_steps: epoch=3 step=500 loss=9.5063
181
+ 2026-03-26 11:39:51,659 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=500 eval_loss=8.6092 duration_sec=7.68
182
+ 2026-03-26 11:40:30,150 [INFO] new_opacus_codex.train_steps: epoch=3 step=505 loss=9.3678
183
+ 2026-03-26 11:41:07,433 [INFO] new_opacus_codex.train_steps: epoch=3 step=510 loss=9.3925
184
+ 2026-03-26 11:41:15,053 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=510 eval_loss=8.6066 duration_sec=7.60
185
+ 2026-03-26 11:41:52,901 [INFO] new_opacus_codex.train_steps: epoch=3 step=515 loss=9.2716
186
+ 2026-03-26 11:42:31,601 [INFO] new_opacus_codex.train_steps: epoch=3 step=520 loss=9.2774
187
+ 2026-03-26 11:42:39,977 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=520 eval_loss=8.6050 duration_sec=8.36
188
+ 2026-03-26 11:43:15,162 [INFO] new_opacus_codex.train_steps: epoch=3 step=525 loss=9.2858
189
+ 2026-03-26 11:43:52,885 [INFO] new_opacus_codex.train_steps: epoch=3 step=530 loss=9.1494
190
+ 2026-03-26 11:44:00,508 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=530 eval_loss=8.6043 duration_sec=7.62
191
+ 2026-03-26 11:44:38,127 [INFO] new_opacus_codex.train_steps: epoch=3 step=535 loss=9.5521
192
+ 2026-03-26 11:45:15,123 [INFO] new_opacus_codex.train_steps: epoch=3 step=540 loss=9.5275
193
+ 2026-03-26 11:45:22,792 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=540 eval_loss=8.6038 duration_sec=7.66
194
+ 2026-03-26 11:46:00,789 [INFO] new_opacus_codex.train_steps: epoch=3 step=545 loss=9.3271
195
+ 2026-03-26 11:46:38,905 [INFO] new_opacus_codex.train_steps: epoch=3 step=550 loss=9.3544
196
+ 2026-03-26 11:46:47,024 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=550 eval_loss=8.6039 duration_sec=8.11
197
+ 2026-03-26 11:47:23,087 [INFO] new_opacus_codex.train_steps: epoch=3 step=555 loss=9.2233
qwen1.5-moe-a2.7b/dp3_attn_v2/adapter/README.md ADDED
@@ -0,0 +1,207 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: Qwen/Qwen1.5-MoE-A2.7B
3
+ library_name: peft
4
+ pipeline_tag: text-generation
5
+ tags:
6
+ - base_model:adapter:Qwen/Qwen1.5-MoE-A2.7B
7
+ - lora
8
+ - transformers
9
+ ---
10
+
11
+ # Model Card for Model ID
12
+
13
+ <!-- Provide a quick summary of what the model is/does. -->
14
+
15
+
16
+
17
+ ## Model Details
18
+
19
+ ### Model Description
20
+
21
+ <!-- Provide a longer summary of what this model is. -->
22
+
23
+
24
+
25
+ - **Developed by:** [More Information Needed]
26
+ - **Funded by [optional]:** [More Information Needed]
27
+ - **Shared by [optional]:** [More Information Needed]
28
+ - **Model type:** [More Information Needed]
29
+ - **Language(s) (NLP):** [More Information Needed]
30
+ - **License:** [More Information Needed]
31
+ - **Finetuned from model [optional]:** [More Information Needed]
32
+
33
+ ### Model Sources [optional]
34
+
35
+ <!-- Provide the basic links for the model. -->
36
+
37
+ - **Repository:** [More Information Needed]
38
+ - **Paper [optional]:** [More Information Needed]
39
+ - **Demo [optional]:** [More Information Needed]
40
+
41
+ ## Uses
42
+
43
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
44
+
45
+ ### Direct Use
46
+
47
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
48
+
49
+ [More Information Needed]
50
+
51
+ ### Downstream Use [optional]
52
+
53
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
54
+
55
+ [More Information Needed]
56
+
57
+ ### Out-of-Scope Use
58
+
59
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
60
+
61
+ [More Information Needed]
62
+
63
+ ## Bias, Risks, and Limitations
64
+
65
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
66
+
67
+ [More Information Needed]
68
+
69
+ ### Recommendations
70
+
71
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
72
+
73
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
74
+
75
+ ## How to Get Started with the Model
76
+
77
+ Use the code below to get started with the model.
78
+
79
+ [More Information Needed]
80
+
81
+ ## Training Details
82
+
83
+ ### Training Data
84
+
85
+ <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
86
+
87
+ [More Information Needed]
88
+
89
+ ### Training Procedure
90
+
91
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
92
+
93
+ #### Preprocessing [optional]
94
+
95
+ [More Information Needed]
96
+
97
+
98
+ #### Training Hyperparameters
99
+
100
+ - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
101
+
102
+ #### Speeds, Sizes, Times [optional]
103
+
104
+ <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
105
+
106
+ [More Information Needed]
107
+
108
+ ## Evaluation
109
+
110
+ <!-- This section describes the evaluation protocols and provides the results. -->
111
+
112
+ ### Testing Data, Factors & Metrics
113
+
114
+ #### Testing Data
115
+
116
+ <!-- This should link to a Dataset Card if possible. -->
117
+
118
+ [More Information Needed]
119
+
120
+ #### Factors
121
+
122
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
123
+
124
+ [More Information Needed]
125
+
126
+ #### Metrics
127
+
128
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
129
+
130
+ [More Information Needed]
131
+
132
+ ### Results
133
+
134
+ [More Information Needed]
135
+
136
+ #### Summary
137
+
138
+
139
+
140
+ ## Model Examination [optional]
141
+
142
+ <!-- Relevant interpretability work for the model goes here -->
143
+
144
+ [More Information Needed]
145
+
146
+ ## Environmental Impact
147
+
148
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
149
+
150
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
151
+
152
+ - **Hardware Type:** [More Information Needed]
153
+ - **Hours used:** [More Information Needed]
154
+ - **Cloud Provider:** [More Information Needed]
155
+ - **Compute Region:** [More Information Needed]
156
+ - **Carbon Emitted:** [More Information Needed]
157
+
158
+ ## Technical Specifications [optional]
159
+
160
+ ### Model Architecture and Objective
161
+
162
+ [More Information Needed]
163
+
164
+ ### Compute Infrastructure
165
+
166
+ [More Information Needed]
167
+
168
+ #### Hardware
169
+
170
+ [More Information Needed]
171
+
172
+ #### Software
173
+
174
+ [More Information Needed]
175
+
176
+ ## Citation [optional]
177
+
178
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
179
+
180
+ **BibTeX:**
181
+
182
+ [More Information Needed]
183
+
184
+ **APA:**
185
+
186
+ [More Information Needed]
187
+
188
+ ## Glossary [optional]
189
+
190
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
191
+
192
+ [More Information Needed]
193
+
194
+ ## More Information [optional]
195
+
196
+ [More Information Needed]
197
+
198
+ ## Model Card Authors [optional]
199
+
200
+ [More Information Needed]
201
+
202
+ ## Model Card Contact
203
+
204
+ [More Information Needed]
205
+ ### Framework versions
206
+
207
+ - PEFT 0.18.1
qwen1.5-moe-a2.7b/dp3_attn_v2/audit_results.json ADDED
@@ -0,0 +1,137 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "delta": 1e-05,
3
+ "num_canaries": 500,
4
+ "num_members": 250,
5
+ "paper_guess_fraction": 0.2,
6
+ "paper_guess_steps": 20,
7
+ "loss": {
8
+ "auc": 0.540256,
9
+ "empirical_epsilon": {
10
+ "0.05": 0.3197584319859743,
11
+ "0.01": 0.030605491250753403
12
+ },
13
+ "empirical_epsilon_details": {
14
+ "0.05": {
15
+ "epsilon": 0.3197584319859743,
16
+ "num_guesses": 25,
17
+ "correct_guesses": 19,
18
+ "candidate_num_guesses": [
19
+ 5,
20
+ 10,
21
+ 15,
22
+ 20,
23
+ 25,
24
+ 30,
25
+ 35,
26
+ 40,
27
+ 45,
28
+ 50,
29
+ 55,
30
+ 60,
31
+ 65,
32
+ 70,
33
+ 75,
34
+ 80,
35
+ 85,
36
+ 90,
37
+ 95,
38
+ 100
39
+ ],
40
+ "direction": "lower"
41
+ },
42
+ "0.01": {
43
+ "epsilon": 0.030605491250753403,
44
+ "num_guesses": 25,
45
+ "correct_guesses": 19,
46
+ "candidate_num_guesses": [
47
+ 5,
48
+ 10,
49
+ 15,
50
+ 20,
51
+ 25,
52
+ 30,
53
+ 35,
54
+ 40,
55
+ 45,
56
+ 50,
57
+ 55,
58
+ 60,
59
+ 65,
60
+ 70,
61
+ 75,
62
+ 80,
63
+ 85,
64
+ 90,
65
+ 95,
66
+ 100
67
+ ],
68
+ "direction": "lower"
69
+ }
70
+ }
71
+ },
72
+ "embedding": {
73
+ "auc": 0.5672,
74
+ "empirical_epsilon": {
75
+ "0.05": 0.010390725918114185,
76
+ "0.01": 0.0
77
+ },
78
+ "empirical_epsilon_details": {
79
+ "0.05": {
80
+ "epsilon": 0.010390725918114185,
81
+ "num_guesses": 100,
82
+ "correct_guesses": 59,
83
+ "candidate_num_guesses": [
84
+ 5,
85
+ 10,
86
+ 15,
87
+ 20,
88
+ 25,
89
+ 30,
90
+ 35,
91
+ 40,
92
+ 45,
93
+ 50,
94
+ 55,
95
+ 60,
96
+ 65,
97
+ 70,
98
+ 75,
99
+ 80,
100
+ 85,
101
+ 90,
102
+ 95,
103
+ 100
104
+ ],
105
+ "direction": "higher"
106
+ },
107
+ "0.01": {
108
+ "epsilon": 0.0,
109
+ "num_guesses": 0,
110
+ "correct_guesses": 0,
111
+ "candidate_num_guesses": [
112
+ 5,
113
+ 10,
114
+ 15,
115
+ 20,
116
+ 25,
117
+ 30,
118
+ 35,
119
+ 40,
120
+ 45,
121
+ 50,
122
+ 55,
123
+ 60,
124
+ 65,
125
+ 70,
126
+ 75,
127
+ 80,
128
+ 85,
129
+ 90,
130
+ 95,
131
+ 100
132
+ ],
133
+ "direction": "lower"
134
+ }
135
+ }
136
+ }
137
+ }
qwen1.5-moe-a2.7b/dp3_attn_v2/canary_meta.json ADDED
The diff for this file is too large to render. See raw diff
 
qwen1.5-moe-a2.7b/dp3_attn_v2/metrics.jsonl ADDED
The diff for this file is too large to render. See raw diff
 
qwen1.5-moe-a2.7b/dp3_attn_v2/resolved_config.yaml ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ model:
2
+ name: Qwen/Qwen1.5-MoE-A2.7B
3
+ tokenizer_name: Qwen/Qwen1.5-MoE-A2.7B
4
+ max_length: 1024
5
+ dtype: bfloat16
6
+ trust_remote_code: true
7
+ use_fast_tokenizer: true
8
+ cache_dir: null
9
+ local_files_only: false
10
+ low_cpu_mem_usage: true
11
+ tie_word_embeddings: true
12
+ gradient_checkpointing: false
13
+ use_chat_template: false
14
+ dataset:
15
+ name: melihcatal/codedp-cpt
16
+ split: train
17
+ mode: cpt
18
+ text_column: text
19
+ validation_ratio: 0.05
20
+ max_samples: -1
21
+ lora:
22
+ enabled: true
23
+ r: 16
24
+ alpha: 32
25
+ dropout: 0.05
26
+ target_modules:
27
+ - q_proj
28
+ - k_proj
29
+ - v_proj
30
+ - o_proj
31
+ modules_to_save:
32
+ - lm_head
33
+ bias: none
34
+ training:
35
+ seed: 42
36
+ epochs: 5
37
+ warmup_steps: null
38
+ warmup_ratio: 0.05
39
+ mixed_precision: false
40
+ mixed_precision_dtype: bfloat16
41
+ batch_size: 4
42
+ eval_batch_size: 8
43
+ eval_every_steps: 10
44
+ eval_every_epochs: 1
45
+ learning_rate: 0.0005
46
+ optimizer: adamw
47
+ lr_scheduler: cosine
48
+ adam_beta1: 0.9
49
+ adam_beta2: 0.999
50
+ adam_epsilon: 1.0e-08
51
+ sgd_momentum: 0.9
52
+ weight_decay: 0.01
53
+ max_grad_norm: 1.0
54
+ log_every: 5
55
+ gradient_accumulation_steps: 8
56
+ num_workers: 4
57
+ output_dir: /scratch/mcatal/runs/cpt/qwen1.5-moe-a2.7b/dp3_attn_v2
58
+ min_lr_ratio: 0.15
59
+ distributed:
60
+ strategy: dpddp
61
+ backend: nccl
62
+ devices: null
63
+ dp:
64
+ module_validator: auto
65
+ target_delta: 1.0e-05
66
+ noise_multiplier: null
67
+ max_grad_norm: 1.0
68
+ grad_sample_mode: hooks
69
+ clipping: per_layer
70
+ secure_mode: false
71
+ enabled: true
72
+ target_epsilon: 3.0
73
+ audit:
74
+ enabled: true
75
+ run_every_epoch: true
76
+ epoch_device: cuda
77
+ q_canary: auto
78
+ num_canaries: 500
79
+ prefix_length: 49
80
+ num_digits: 12
81
+ batch_size: 32
82
+ delta: 1.0e-05
83
+ p_values:
84
+ - 0.05
85
+ - 0.01
86
+ paper_guess_fraction: 0.2
87
+ paper_guess_steps: 20
88
+ enable_holdout_empirical_epsilon: false
89
+ holdout_seed: 42
90
+ tie_seed: 42
91
+ tracking:
92
+ enabled: true
93
+ tensorboard: true
94
+ wandb: false
95
+ wandb_project: codedp-finetune-h200-audit
96
+ wandb_run_name: qwen1.5-moe-a2.7b-cpt-dp3_attn
97
+ wandb_mode: online
98
+ codecarbon: true
99
+ codecarbon_output_file: codecarbon.csv
100
+ codecarbon_measure_power_secs: 15
101
+ codecarbon_country_iso_code: null
102
+ codecarbon_project_name: codedp-qwen1.5-moe-a2.7b-cpt-dp3_attn
103
+ moe:
104
+ output_router_logits: false
105
+ router_aux_loss_coef: 0.0
106
+ freeze_router: true
107
+ profile:
108
+ enabled: false
109
+ num_batches: 8
110
+ top_experts: 8
111
+ output_file: moe_expert_profile.json
qwen1.5-moe-a2.7b/dp3_attn_v2/summary.json ADDED
@@ -0,0 +1,72 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "audit/delta": 1e-05,
3
+ "audit/embedding/auc": 0.5672,
4
+ "audit/embedding/empirical_epsilon/0.01": 0.0,
5
+ "audit/embedding/empirical_epsilon/0.05": 0.010390725918114185,
6
+ "audit/embedding/empirical_epsilon_details/0.01/correct_guesses": 0.0,
7
+ "audit/embedding/empirical_epsilon_details/0.01/epsilon": 0.0,
8
+ "audit/embedding/empirical_epsilon_details/0.01/num_guesses": 0.0,
9
+ "audit/embedding/empirical_epsilon_details/0.05/correct_guesses": 59.0,
10
+ "audit/embedding/empirical_epsilon_details/0.05/epsilon": 0.010390725918114185,
11
+ "audit/embedding/empirical_epsilon_details/0.05/num_guesses": 100.0,
12
+ "audit/loss/auc": 0.540256,
13
+ "audit/loss/empirical_epsilon/0.01": 0.030605491250753403,
14
+ "audit/loss/empirical_epsilon/0.05": 0.3197584319859743,
15
+ "audit/loss/empirical_epsilon_details/0.01/correct_guesses": 19.0,
16
+ "audit/loss/empirical_epsilon_details/0.01/epsilon": 0.030605491250753403,
17
+ "audit/loss/empirical_epsilon_details/0.01/num_guesses": 25.0,
18
+ "audit/loss/empirical_epsilon_details/0.05/correct_guesses": 19.0,
19
+ "audit/loss/empirical_epsilon_details/0.05/epsilon": 0.3197584319859743,
20
+ "audit/loss/empirical_epsilon_details/0.05/num_guesses": 25.0,
21
+ "audit/num_canaries": 500.0,
22
+ "audit/num_members": 250.0,
23
+ "audit/paper_guess_fraction": 0.2,
24
+ "audit/paper_guess_steps": 20.0,
25
+ "energy/codecarbon/cpu_count": 32.0,
26
+ "energy/codecarbon/cpu_energy": 0.2855161560587346,
27
+ "energy/codecarbon/cpu_power": 81.27538315956447,
28
+ "energy/codecarbon/cpu_utilization_percent": 11.022740929050151,
29
+ "energy/codecarbon/duration": 13136.868291858002,
30
+ "energy/codecarbon/emissions": 0.2572504230722735,
31
+ "energy/codecarbon/emissions_rate": 1.9582324900959293e-05,
32
+ "energy/codecarbon/energy_consumed": 7.3831307026453965,
33
+ "energy/codecarbon/gpu_count": 8.0,
34
+ "energy/codecarbon/gpu_energy": 6.964125738796139,
35
+ "energy/codecarbon/gpu_power": 1913.306485144389,
36
+ "energy/codecarbon/gpu_utilization_percent": 76.41666666666667,
37
+ "energy/codecarbon/latitude": 47.4843,
38
+ "energy/codecarbon/longitude": 8.212,
39
+ "energy/codecarbon/pue": 1.0,
40
+ "energy/codecarbon/ram_energy": 0.1334888077905266,
41
+ "energy/codecarbon/ram_power": 38.0,
42
+ "energy/codecarbon/ram_total_size": 256.0,
43
+ "energy/codecarbon/ram_used_gb": 71.84261003057428,
44
+ "energy/codecarbon/ram_utilization_percent": 3.980964486557276,
45
+ "energy/codecarbon/water_consumed": 0.0,
46
+ "energy/codecarbon/wue": 0.0,
47
+ "eval/duration_sec": 16.239122020022478,
48
+ "eval/loss": 7.9647808350049525,
49
+ "perf/audit_duration_sec": 7.3567996520141605,
50
+ "perf/epoch_duration_sec": 2603.847994864016,
51
+ "perf/epoch_samples": 51891.0,
52
+ "perf/epoch_samples_per_sec": 19.928582660106457,
53
+ "perf/epoch_tokens": 37894520.0,
54
+ "perf/epoch_tokens_per_sec": 14553.276564048821,
55
+ "perf/logical_batch_size": 25.0,
56
+ "perf/logical_token_count": 19367.0,
57
+ "perf/physical_batches": 8.0,
58
+ "perf/samples_per_sec": 2.461059126094217,
59
+ "perf/step_duration_sec": 10.158228112006327,
60
+ "perf/tokens_per_sec": 1906.5332838026682,
61
+ "privacy/epsilon": 2.9933345067944916,
62
+ "system/cuda_epoch_peak_memory_gb": 56.512940406799316,
63
+ "system/cuda_max_memory_allocated_gb": 56.512940406799316,
64
+ "system/cuda_memory_allocated_gb": 29.994746685028076,
65
+ "train/epoch_canary_loss": 11.233648779176939,
66
+ "train/epoch_loss": 8.540389815382778,
67
+ "train/epoch_real_loss": 8.163681753906044,
68
+ "train/lr": 7.5e-05,
69
+ "train/step_canary_loss": 6.21875,
70
+ "train/step_loss": 8.063941588768593,
71
+ "train/step_real_loss": 8.217707554499308
72
+ }
qwen1.5-moe-a2.7b/dp3_attn_v2/tokenizer/added_tokens.json ADDED
@@ -0,0 +1,505 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "114848338811": 151654,
3
+ "116427191984": 152008,
4
+ "116639746413": 151980,
5
+ "117298239631": 151870,
6
+ "117657531962": 151927,
7
+ "117783274348": 151944,
8
+ "119342152414": 152017,
9
+ "121557291859": 151876,
10
+ "123741461297": 151785,
11
+ "124589118494": 151781,
12
+ "125866399464": 151890,
13
+ "126574388585": 151832,
14
+ "129693547287": 151766,
15
+ "133373659361": 151961,
16
+ "135441121756": 151649,
17
+ "136931728327": 151751,
18
+ "138154781148": 152061,
19
+ "138218735276": 151945,
20
+ "141499982867": 152039,
21
+ "142526557681": 152139,
22
+ "142799718159": 152003,
23
+ "143182978129": 151965,
24
+ "143777992227": 151660,
25
+ "144373897488": 152071,
26
+ "145669731981": 151875,
27
+ "147444452794": 152124,
28
+ "147541537162": 152069,
29
+ "151859598873": 151841,
30
+ "152676556624": 151976,
31
+ "154962668224": 151990,
32
+ "158142752582": 151871,
33
+ "158929292238": 151881,
34
+ "159337222655": 151716,
35
+ "161238482259": 152084,
36
+ "161588262349": 151731,
37
+ "163165456396": 152082,
38
+ "163497196769": 152114,
39
+ "165429152138": 152011,
40
+ "172599315861": 151665,
41
+ "173781362446": 151717,
42
+ "175469284871": 151809,
43
+ "175528632226": 152127,
44
+ "178478368922": 152056,
45
+ "179386156569": 152098,
46
+ "179564995814": 151991,
47
+ "181233596124": 152076,
48
+ "181439694557": 151692,
49
+ "182345464792": 151736,
50
+ "182782912224": 151918,
51
+ "183368827562": 151845,
52
+ "185438575221": 151831,
53
+ "186198784585": 151892,
54
+ "186872457894": 152129,
55
+ "191887635211": 152015,
56
+ "195388849133": 152086,
57
+ "195792996468": 151840,
58
+ "196119929397": 151905,
59
+ "196735786156": 151787,
60
+ "198462689911": 151722,
61
+ "211411365397": 151671,
62
+ "211662639865": 152000,
63
+ "212985874612": 152014,
64
+ "214193645583": 152096,
65
+ "215671314196": 151896,
66
+ "215826764872": 152050,
67
+ "216667245274": 151778,
68
+ "217756494954": 151942,
69
+ "218391991184": 151973,
70
+ "218784516525": 152001,
71
+ "221762325616": 151972,
72
+ "225164373623": 151733,
73
+ "228358422847": 151963,
74
+ "228565148417": 152020,
75
+ "229775525672": 152087,
76
+ "231853711778": 151810,
77
+ "232776999554": 151737,
78
+ "232787996724": 151895,
79
+ "232896526252": 151783,
80
+ "233795848681": 151801,
81
+ "234316622259": 151919,
82
+ "242851284913": 151729,
83
+ "243165586174": 152078,
84
+ "244371196654": 151909,
85
+ "244885399387": 151827,
86
+ "245416776926": 151877,
87
+ "251789649997": 151907,
88
+ "252867443568": 151872,
89
+ "256497188281": 152029,
90
+ "259533577263": 151752,
91
+ "259666844669": 151728,
92
+ "262296912232": 151775,
93
+ "262437452884": 152088,
94
+ "264259917554": 151707,
95
+ "265847332563": 152112,
96
+ "267514771964": 151868,
97
+ "267954879268": 151940,
98
+ "268433315926": 152116,
99
+ "268921994538": 152009,
100
+ "272362973463": 151700,
101
+ "275597777299": 151859,
102
+ "281673864127": 151702,
103
+ "282322838685": 151667,
104
+ "284445431422": 151958,
105
+ "284776712475": 152074,
106
+ "286173241916": 151897,
107
+ "291762694874": 151964,
108
+ "293847485454": 152054,
109
+ "295118729589": 151903,
110
+ "295448593321": 151784,
111
+ "296953381169": 151773,
112
+ "298464134954": 151777,
113
+ "311632789559": 151685,
114
+ "313145133463": 152067,
115
+ "313313831273": 152109,
116
+ "314697386682": 151743,
117
+ "315534574435": 152048,
118
+ "319584985416": 151864,
119
+ "322589379462": 151974,
120
+ "322683334381": 151962,
121
+ "325196781219": 152138,
122
+ "327158398268": 151967,
123
+ "327491773134": 151681,
124
+ "328552864359": 151999,
125
+ "331369926711": 152025,
126
+ "332234613346": 152137,
127
+ "332522971941": 151977,
128
+ "334513529294": 151956,
129
+ "336813343298": 152041,
130
+ "337326533813": 151738,
131
+ "339882449689": 151891,
132
+ "342221452223": 152036,
133
+ "345271433112": 151951,
134
+ "349249721685": 152045,
135
+ "351885857332": 151720,
136
+ "353755529131": 152047,
137
+ "355977139358": 151673,
138
+ "356722692178": 152140,
139
+ "357299947518": 152126,
140
+ "357621613582": 152032,
141
+ "363214487524": 151826,
142
+ "364196983365": 151839,
143
+ "364492359246": 151989,
144
+ "364947325679": 151912,
145
+ "367459894796": 151650,
146
+ "367968398551": 152051,
147
+ "368763327262": 152093,
148
+ "371496153876": 152110,
149
+ "374327157578": 151684,
150
+ "374439448821": 151762,
151
+ "374598961236": 151806,
152
+ "375429755114": 152075,
153
+ "377273562477": 151774,
154
+ "377338713663": 151788,
155
+ "378749551722": 152083,
156
+ "379557332953": 151745,
157
+ "379571934299": 151906,
158
+ "383837596997": 151675,
159
+ "384598329253": 151910,
160
+ "387763951128": 152024,
161
+ "389715484387": 152053,
162
+ "393741199486": 151998,
163
+ "396521188476": 151959,
164
+ "396847976144": 152079,
165
+ "398549545767": 151796,
166
+ "399223169721": 151842,
167
+ "399749397872": 151765,
168
+ "412843856172": 152097,
169
+ "415956914763": 151923,
170
+ "417668946583": 152031,
171
+ "418412875665": 151992,
172
+ "421369185549": 151920,
173
+ "421435255895": 151960,
174
+ "425949483533": 151662,
175
+ "429781429464": 151747,
176
+ "431149748713": 151888,
177
+ "431751228368": 151828,
178
+ "433581868192": 151829,
179
+ "434632855397": 151921,
180
+ "438124184139": 152106,
181
+ "438435178228": 152033,
182
+ "438745114173": 151984,
183
+ "441373944925": 151757,
184
+ "441377892146": 152089,
185
+ "445925149649": 151658,
186
+ "447445629421": 151690,
187
+ "449318681694": 152141,
188
+ "449549215429": 152113,
189
+ "451161679736": 151970,
190
+ "451885565366": 152090,
191
+ "454164859837": 151950,
192
+ "454244839926": 151797,
193
+ "455423278865": 151997,
194
+ "457919547633": 151652,
195
+ "462294118988": 151952,
196
+ "467893531755": 151759,
197
+ "472129283475": 152043,
198
+ "473822473819": 151691,
199
+ "475794684356": 151850,
200
+ "476969195485": 152066,
201
+ "479227349997": 151750,
202
+ "479676316326": 151703,
203
+ "481884157827": 151710,
204
+ "482829231639": 152122,
205
+ "484489193824": 151823,
206
+ "484773717614": 151791,
207
+ "485177821815": 151648,
208
+ "485692541169": 152034,
209
+ "486391569221": 151830,
210
+ "488496781997": 152120,
211
+ "488538864842": 151753,
212
+ "489562189822": 151824,
213
+ "489991411364": 151755,
214
+ "492373714791": 151884,
215
+ "492516914298": 152063,
216
+ "497294727353": 151954,
217
+ "499148455254": 151756,
218
+ "499156317423": 151948,
219
+ "499444466686": 151742,
220
+ "511785788222": 151953,
221
+ "511923256573": 151966,
222
+ "512659818733": 151885,
223
+ "513349618943": 151979,
224
+ "513361312532": 151814,
225
+ "514354144759": 152044,
226
+ "514532995959": 151730,
227
+ "515856611931": 151949,
228
+ "516968424863": 151770,
229
+ "517379856925": 151744,
230
+ "517778845323": 151725,
231
+ "522515164754": 152072,
232
+ "523723475786": 152049,
233
+ "523918658846": 151686,
234
+ "523937669294": 152010,
235
+ "524444658383": 151943,
236
+ "527117488925": 151760,
237
+ "527874729936": 152022,
238
+ "529455466433": 151904,
239
+ "529679678956": 151714,
240
+ "529854679937": 151834,
241
+ "532833541879": 151936,
242
+ "533417736566": 151846,
243
+ "535787224917": 151719,
244
+ "538631466654": 151971,
245
+ "545512344383": 151741,
246
+ "546276298359": 152046,
247
+ "547149832394": 151776,
248
+ "547563892268": 152107,
249
+ "549946382131": 151957,
250
+ "552738847828": 152130,
251
+ "556814553861": 152121,
252
+ "556976448182": 151899,
253
+ "557163366873": 152070,
254
+ "557533689478": 151821,
255
+ "558837194455": 152081,
256
+ "562389434565": 152028,
257
+ "562716493614": 152026,
258
+ "563117729724": 151804,
259
+ "563365646341": 151865,
260
+ "564395222747": 151969,
261
+ "564544131355": 151908,
262
+ "569765693871": 151647,
263
+ "569853653352": 152052,
264
+ "571317457151": 152006,
265
+ "572154333646": 151780,
266
+ "578827344666": 151721,
267
+ "579184949249": 151712,
268
+ "581493616981": 152057,
269
+ "581976444672": 151882,
270
+ "582471228164": 151819,
271
+ "583527834464": 152119,
272
+ "585544633356": 151818,
273
+ "586855179568": 151790,
274
+ "587541149322": 152111,
275
+ "587787496812": 151711,
276
+ "589439538674": 152058,
277
+ "589547936288": 151825,
278
+ "591464355756": 152013,
279
+ "592538279337": 151693,
280
+ "594688742928": 152042,
281
+ "598916996413": 151924,
282
+ "599961512792": 151900,
283
+ "612128769512": 151683,
284
+ "612788593128": 151874,
285
+ "614583493135": 152135,
286
+ "614797267726": 151746,
287
+ "616248973917": 151836,
288
+ "618423867817": 152019,
289
+ "618878266619": 152123,
290
+ "619454958782": 151887,
291
+ "622553389126": 151987,
292
+ "624221476921": 151889,
293
+ "624266421831": 151995,
294
+ "625334487352": 152101,
295
+ "626748382482": 152131,
296
+ "632962475133": 151822,
297
+ "635127952696": 152007,
298
+ "635185846287": 151981,
299
+ "636324136426": 152095,
300
+ "637551427887": 151941,
301
+ "643489267333": 151666,
302
+ "643692393748": 151695,
303
+ "644234165531": 151917,
304
+ "645527857972": 152128,
305
+ "646979124677": 151704,
306
+ "649291669397": 151656,
307
+ "649516938598": 152104,
308
+ "652154162978": 151851,
309
+ "653573457168": 151786,
310
+ "655517954651": 151913,
311
+ "658481324922": 151709,
312
+ "661436365453": 151727,
313
+ "662426712668": 151811,
314
+ "664847713559": 151931,
315
+ "665661658354": 151848,
316
+ "668134915514": 151694,
317
+ "668514843129": 151853,
318
+ "669183782449": 151937,
319
+ "669689115557": 151680,
320
+ "673948786567": 151914,
321
+ "676539973863": 151734,
322
+ "676676697292": 151739,
323
+ "677896358599": 151793,
324
+ "678666359123": 151820,
325
+ "682564697312": 151674,
326
+ "687284547244": 152142,
327
+ "689285841825": 151767,
328
+ "689362171782": 151676,
329
+ "691315125346": 151929,
330
+ "691587454538": 151659,
331
+ "693281533643": 151849,
332
+ "694235616268": 151701,
333
+ "696651276628": 151696,
334
+ "697747991249": 152037,
335
+ "698218415444": 151672,
336
+ "699446724178": 151994,
337
+ "711412347158": 151812,
338
+ "711641973413": 151947,
339
+ "712992815957": 151815,
340
+ "714738141818": 152055,
341
+ "716763689965": 152038,
342
+ "717628619782": 152105,
343
+ "718726519731": 151664,
344
+ "718951851413": 151993,
345
+ "721737645236": 151749,
346
+ "722339992679": 152060,
347
+ "722618266242": 151723,
348
+ "724492262421": 151986,
349
+ "724883568652": 151769,
350
+ "724964923582": 152073,
351
+ "729116422489": 152021,
352
+ "729825168745": 151928,
353
+ "731681792655": 151860,
354
+ "733737988257": 151771,
355
+ "739246183345": 152012,
356
+ "739394546916": 151939,
357
+ "739667844957": 152027,
358
+ "739784153482": 152023,
359
+ "741757479784": 151833,
360
+ "742765629356": 151732,
361
+ "745977673725": 151898,
362
+ "746285987371": 151655,
363
+ "749665846687": 151726,
364
+ "749966767285": 151677,
365
+ "752644929761": 151857,
366
+ "753159165157": 151678,
367
+ "753233719644": 151852,
368
+ "753466987363": 151813,
369
+ "754978243758": 152085,
370
+ "756916173536": 152099,
371
+ "757422791351": 152094,
372
+ "757785319772": 152062,
373
+ "758963111966": 152068,
374
+ "762486672996": 152002,
375
+ "762495822823": 152092,
376
+ "763531783491": 151837,
377
+ "765474393376": 151653,
378
+ "765737843639": 151808,
379
+ "765981581453": 151687,
380
+ "766294538337": 151901,
381
+ "767858144912": 152005,
382
+ "768461952319": 151930,
383
+ "771619544974": 151768,
384
+ "773671349113": 151925,
385
+ "773687856563": 151718,
386
+ "774679513717": 152117,
387
+ "778466723723": 151698,
388
+ "781653545886": 151668,
389
+ "786391632558": 151847,
390
+ "794825141891": 151688,
391
+ "795693824762": 151679,
392
+ "795838284624": 151715,
393
+ "796415361892": 151669,
394
+ "797238311233": 152145,
395
+ "799364566435": 151798,
396
+ "799417897197": 151878,
397
+ "811371928234": 151866,
398
+ "813981174797": 151983,
399
+ "814611949491": 151946,
400
+ "815458725125": 151996,
401
+ "816115936673": 151916,
402
+ "817378986438": 151800,
403
+ "818479319499": 152040,
404
+ "819162623892": 151854,
405
+ "824387273757": 151802,
406
+ "824488329416": 151934,
407
+ "825815739116": 152035,
408
+ "826214551218": 151735,
409
+ "829235161526": 151817,
410
+ "836746273134": 151708,
411
+ "837131396371": 151867,
412
+ "837873762491": 151935,
413
+ "839269684865": 151873,
414
+ "841824384614": 151782,
415
+ "841991688488": 151670,
416
+ "843998562287": 151779,
417
+ "846657726193": 152059,
418
+ "848845611563": 151794,
419
+ "851254219729": 152118,
420
+ "851852651677": 151795,
421
+ "852347289392": 151772,
422
+ "853859348234": 151697,
423
+ "856317483891": 152108,
424
+ "857231548835": 151915,
425
+ "858146415154": 152004,
426
+ "859289776479": 152134,
427
+ "859927364588": 152030,
428
+ "861342291298": 152115,
429
+ "864718285734": 151862,
430
+ "865331112869": 151646,
431
+ "866519986723": 151955,
432
+ "867678158885": 151764,
433
+ "873898736873": 151689,
434
+ "876537814263": 151975,
435
+ "877385381798": 151856,
436
+ "877482678543": 151651,
437
+ "879425227932": 151763,
438
+ "879764613759": 151968,
439
+ "881112998642": 151886,
440
+ "881136466196": 151858,
441
+ "883685933161": 151706,
442
+ "884354318946": 151978,
443
+ "886464195129": 151933,
444
+ "887417525236": 152103,
445
+ "887753195844": 151911,
446
+ "889965243135": 151988,
447
+ "892336777248": 151880,
448
+ "892697125149": 152100,
449
+ "893974619684": 152144,
450
+ "894885246797": 151792,
451
+ "897784492783": 151883,
452
+ "899151413682": 151758,
453
+ "912193598348": 151713,
454
+ "916198946615": 151803,
455
+ "919181521716": 152064,
456
+ "921889371893": 151922,
457
+ "921894985642": 152132,
458
+ "922327493433": 151705,
459
+ "922466849763": 151748,
460
+ "922867396683": 151807,
461
+ "927914615679": 151657,
462
+ "928335588653": 151761,
463
+ "929185622831": 151740,
464
+ "929826356991": 151699,
465
+ "933775286797": 152136,
466
+ "934912591213": 151902,
467
+ "937495658852": 151754,
468
+ "938488813964": 151843,
469
+ "941169431859": 151855,
470
+ "941861857715": 152102,
471
+ "941976537588": 151938,
472
+ "943193294691": 151893,
473
+ "943878645871": 152133,
474
+ "944513428457": 151869,
475
+ "946444632795": 152018,
476
+ "947262995132": 152143,
477
+ "948585673431": 152077,
478
+ "949235426629": 152125,
479
+ "951794811935": 151805,
480
+ "952465217972": 151926,
481
+ "952872416512": 151724,
482
+ "953267689786": 151932,
483
+ "955456118295": 151838,
484
+ "955668961269": 151894,
485
+ "961133895172": 151861,
486
+ "961981959227": 151844,
487
+ "963852916563": 151863,
488
+ "964342468552": 151789,
489
+ "967114116556": 151799,
490
+ "967214577653": 152016,
491
+ "971621888444": 151816,
492
+ "981277526855": 152091,
493
+ "982993456429": 151663,
494
+ "983493951323": 151985,
495
+ "983569279932": 151682,
496
+ "985781138935": 152065,
497
+ "993832498489": 151982,
498
+ "996116119839": 151835,
499
+ "997724962668": 152080,
500
+ "997958836435": 151879,
501
+ "997981281989": 151661,
502
+ "<|endoftext|>": 151643,
503
+ "<|im_end|>": 151645,
504
+ "<|im_start|>": 151644
505
+ }
qwen1.5-moe-a2.7b/dp3_attn_v2/tokenizer/chat_template.jinja ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system
2
+ You are a helpful assistant<|im_end|>
3
+ ' }}{% endif %}{{'<|im_start|>' + message['role'] + '
4
+ ' + message['content'] + '<|im_end|>' + '
5
+ '}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
6
+ ' }}{% endif %}
qwen1.5-moe-a2.7b/dp3_attn_v2/tokenizer/special_tokens_map.json ADDED
@@ -0,0 +1,3518 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": [
3
+ {
4
+ "content": "865331112869",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false
9
+ },
10
+ {
11
+ "content": "569765693871",
12
+ "lstrip": false,
13
+ "normalized": false,
14
+ "rstrip": false,
15
+ "single_word": false
16
+ },
17
+ {
18
+ "content": "485177821815",
19
+ "lstrip": false,
20
+ "normalized": false,
21
+ "rstrip": false,
22
+ "single_word": false
23
+ },
24
+ {
25
+ "content": "135441121756",
26
+ "lstrip": false,
27
+ "normalized": false,
28
+ "rstrip": false,
29
+ "single_word": false
30
+ },
31
+ {
32
+ "content": "367459894796",
33
+ "lstrip": false,
34
+ "normalized": false,
35
+ "rstrip": false,
36
+ "single_word": false
37
+ },
38
+ {
39
+ "content": "877482678543",
40
+ "lstrip": false,
41
+ "normalized": false,
42
+ "rstrip": false,
43
+ "single_word": false
44
+ },
45
+ {
46
+ "content": "457919547633",
47
+ "lstrip": false,
48
+ "normalized": false,
49
+ "rstrip": false,
50
+ "single_word": false
51
+ },
52
+ {
53
+ "content": "765474393376",
54
+ "lstrip": false,
55
+ "normalized": false,
56
+ "rstrip": false,
57
+ "single_word": false
58
+ },
59
+ {
60
+ "content": "114848338811",
61
+ "lstrip": false,
62
+ "normalized": false,
63
+ "rstrip": false,
64
+ "single_word": false
65
+ },
66
+ {
67
+ "content": "746285987371",
68
+ "lstrip": false,
69
+ "normalized": false,
70
+ "rstrip": false,
71
+ "single_word": false
72
+ },
73
+ {
74
+ "content": "649291669397",
75
+ "lstrip": false,
76
+ "normalized": false,
77
+ "rstrip": false,
78
+ "single_word": false
79
+ },
80
+ {
81
+ "content": "927914615679",
82
+ "lstrip": false,
83
+ "normalized": false,
84
+ "rstrip": false,
85
+ "single_word": false
86
+ },
87
+ {
88
+ "content": "445925149649",
89
+ "lstrip": false,
90
+ "normalized": false,
91
+ "rstrip": false,
92
+ "single_word": false
93
+ },
94
+ {
95
+ "content": "691587454538",
96
+ "lstrip": false,
97
+ "normalized": false,
98
+ "rstrip": false,
99
+ "single_word": false
100
+ },
101
+ {
102
+ "content": "143777992227",
103
+ "lstrip": false,
104
+ "normalized": false,
105
+ "rstrip": false,
106
+ "single_word": false
107
+ },
108
+ {
109
+ "content": "997981281989",
110
+ "lstrip": false,
111
+ "normalized": false,
112
+ "rstrip": false,
113
+ "single_word": false
114
+ },
115
+ {
116
+ "content": "425949483533",
117
+ "lstrip": false,
118
+ "normalized": false,
119
+ "rstrip": false,
120
+ "single_word": false
121
+ },
122
+ {
123
+ "content": "982993456429",
124
+ "lstrip": false,
125
+ "normalized": false,
126
+ "rstrip": false,
127
+ "single_word": false
128
+ },
129
+ {
130
+ "content": "718726519731",
131
+ "lstrip": false,
132
+ "normalized": false,
133
+ "rstrip": false,
134
+ "single_word": false
135
+ },
136
+ {
137
+ "content": "172599315861",
138
+ "lstrip": false,
139
+ "normalized": false,
140
+ "rstrip": false,
141
+ "single_word": false
142
+ },
143
+ {
144
+ "content": "643489267333",
145
+ "lstrip": false,
146
+ "normalized": false,
147
+ "rstrip": false,
148
+ "single_word": false
149
+ },
150
+ {
151
+ "content": "282322838685",
152
+ "lstrip": false,
153
+ "normalized": false,
154
+ "rstrip": false,
155
+ "single_word": false
156
+ },
157
+ {
158
+ "content": "781653545886",
159
+ "lstrip": false,
160
+ "normalized": false,
161
+ "rstrip": false,
162
+ "single_word": false
163
+ },
164
+ {
165
+ "content": "796415361892",
166
+ "lstrip": false,
167
+ "normalized": false,
168
+ "rstrip": false,
169
+ "single_word": false
170
+ },
171
+ {
172
+ "content": "841991688488",
173
+ "lstrip": false,
174
+ "normalized": false,
175
+ "rstrip": false,
176
+ "single_word": false
177
+ },
178
+ {
179
+ "content": "211411365397",
180
+ "lstrip": false,
181
+ "normalized": false,
182
+ "rstrip": false,
183
+ "single_word": false
184
+ },
185
+ {
186
+ "content": "698218415444",
187
+ "lstrip": false,
188
+ "normalized": false,
189
+ "rstrip": false,
190
+ "single_word": false
191
+ },
192
+ {
193
+ "content": "355977139358",
194
+ "lstrip": false,
195
+ "normalized": false,
196
+ "rstrip": false,
197
+ "single_word": false
198
+ },
199
+ {
200
+ "content": "682564697312",
201
+ "lstrip": false,
202
+ "normalized": false,
203
+ "rstrip": false,
204
+ "single_word": false
205
+ },
206
+ {
207
+ "content": "383837596997",
208
+ "lstrip": false,
209
+ "normalized": false,
210
+ "rstrip": false,
211
+ "single_word": false
212
+ },
213
+ {
214
+ "content": "689362171782",
215
+ "lstrip": false,
216
+ "normalized": false,
217
+ "rstrip": false,
218
+ "single_word": false
219
+ },
220
+ {
221
+ "content": "749966767285",
222
+ "lstrip": false,
223
+ "normalized": false,
224
+ "rstrip": false,
225
+ "single_word": false
226
+ },
227
+ {
228
+ "content": "753159165157",
229
+ "lstrip": false,
230
+ "normalized": false,
231
+ "rstrip": false,
232
+ "single_word": false
233
+ },
234
+ {
235
+ "content": "795693824762",
236
+ "lstrip": false,
237
+ "normalized": false,
238
+ "rstrip": false,
239
+ "single_word": false
240
+ },
241
+ {
242
+ "content": "669689115557",
243
+ "lstrip": false,
244
+ "normalized": false,
245
+ "rstrip": false,
246
+ "single_word": false
247
+ },
248
+ {
249
+ "content": "327491773134",
250
+ "lstrip": false,
251
+ "normalized": false,
252
+ "rstrip": false,
253
+ "single_word": false
254
+ },
255
+ {
256
+ "content": "983569279932",
257
+ "lstrip": false,
258
+ "normalized": false,
259
+ "rstrip": false,
260
+ "single_word": false
261
+ },
262
+ {
263
+ "content": "612128769512",
264
+ "lstrip": false,
265
+ "normalized": false,
266
+ "rstrip": false,
267
+ "single_word": false
268
+ },
269
+ {
270
+ "content": "374327157578",
271
+ "lstrip": false,
272
+ "normalized": false,
273
+ "rstrip": false,
274
+ "single_word": false
275
+ },
276
+ {
277
+ "content": "311632789559",
278
+ "lstrip": false,
279
+ "normalized": false,
280
+ "rstrip": false,
281
+ "single_word": false
282
+ },
283
+ {
284
+ "content": "523918658846",
285
+ "lstrip": false,
286
+ "normalized": false,
287
+ "rstrip": false,
288
+ "single_word": false
289
+ },
290
+ {
291
+ "content": "765981581453",
292
+ "lstrip": false,
293
+ "normalized": false,
294
+ "rstrip": false,
295
+ "single_word": false
296
+ },
297
+ {
298
+ "content": "794825141891",
299
+ "lstrip": false,
300
+ "normalized": false,
301
+ "rstrip": false,
302
+ "single_word": false
303
+ },
304
+ {
305
+ "content": "873898736873",
306
+ "lstrip": false,
307
+ "normalized": false,
308
+ "rstrip": false,
309
+ "single_word": false
310
+ },
311
+ {
312
+ "content": "447445629421",
313
+ "lstrip": false,
314
+ "normalized": false,
315
+ "rstrip": false,
316
+ "single_word": false
317
+ },
318
+ {
319
+ "content": "473822473819",
320
+ "lstrip": false,
321
+ "normalized": false,
322
+ "rstrip": false,
323
+ "single_word": false
324
+ },
325
+ {
326
+ "content": "181439694557",
327
+ "lstrip": false,
328
+ "normalized": false,
329
+ "rstrip": false,
330
+ "single_word": false
331
+ },
332
+ {
333
+ "content": "592538279337",
334
+ "lstrip": false,
335
+ "normalized": false,
336
+ "rstrip": false,
337
+ "single_word": false
338
+ },
339
+ {
340
+ "content": "668134915514",
341
+ "lstrip": false,
342
+ "normalized": false,
343
+ "rstrip": false,
344
+ "single_word": false
345
+ },
346
+ {
347
+ "content": "643692393748",
348
+ "lstrip": false,
349
+ "normalized": false,
350
+ "rstrip": false,
351
+ "single_word": false
352
+ },
353
+ {
354
+ "content": "696651276628",
355
+ "lstrip": false,
356
+ "normalized": false,
357
+ "rstrip": false,
358
+ "single_word": false
359
+ },
360
+ {
361
+ "content": "853859348234",
362
+ "lstrip": false,
363
+ "normalized": false,
364
+ "rstrip": false,
365
+ "single_word": false
366
+ },
367
+ {
368
+ "content": "778466723723",
369
+ "lstrip": false,
370
+ "normalized": false,
371
+ "rstrip": false,
372
+ "single_word": false
373
+ },
374
+ {
375
+ "content": "929826356991",
376
+ "lstrip": false,
377
+ "normalized": false,
378
+ "rstrip": false,
379
+ "single_word": false
380
+ },
381
+ {
382
+ "content": "272362973463",
383
+ "lstrip": false,
384
+ "normalized": false,
385
+ "rstrip": false,
386
+ "single_word": false
387
+ },
388
+ {
389
+ "content": "694235616268",
390
+ "lstrip": false,
391
+ "normalized": false,
392
+ "rstrip": false,
393
+ "single_word": false
394
+ },
395
+ {
396
+ "content": "281673864127",
397
+ "lstrip": false,
398
+ "normalized": false,
399
+ "rstrip": false,
400
+ "single_word": false
401
+ },
402
+ {
403
+ "content": "479676316326",
404
+ "lstrip": false,
405
+ "normalized": false,
406
+ "rstrip": false,
407
+ "single_word": false
408
+ },
409
+ {
410
+ "content": "646979124677",
411
+ "lstrip": false,
412
+ "normalized": false,
413
+ "rstrip": false,
414
+ "single_word": false
415
+ },
416
+ {
417
+ "content": "922327493433",
418
+ "lstrip": false,
419
+ "normalized": false,
420
+ "rstrip": false,
421
+ "single_word": false
422
+ },
423
+ {
424
+ "content": "883685933161",
425
+ "lstrip": false,
426
+ "normalized": false,
427
+ "rstrip": false,
428
+ "single_word": false
429
+ },
430
+ {
431
+ "content": "264259917554",
432
+ "lstrip": false,
433
+ "normalized": false,
434
+ "rstrip": false,
435
+ "single_word": false
436
+ },
437
+ {
438
+ "content": "836746273134",
439
+ "lstrip": false,
440
+ "normalized": false,
441
+ "rstrip": false,
442
+ "single_word": false
443
+ },
444
+ {
445
+ "content": "658481324922",
446
+ "lstrip": false,
447
+ "normalized": false,
448
+ "rstrip": false,
449
+ "single_word": false
450
+ },
451
+ {
452
+ "content": "481884157827",
453
+ "lstrip": false,
454
+ "normalized": false,
455
+ "rstrip": false,
456
+ "single_word": false
457
+ },
458
+ {
459
+ "content": "587787496812",
460
+ "lstrip": false,
461
+ "normalized": false,
462
+ "rstrip": false,
463
+ "single_word": false
464
+ },
465
+ {
466
+ "content": "579184949249",
467
+ "lstrip": false,
468
+ "normalized": false,
469
+ "rstrip": false,
470
+ "single_word": false
471
+ },
472
+ {
473
+ "content": "912193598348",
474
+ "lstrip": false,
475
+ "normalized": false,
476
+ "rstrip": false,
477
+ "single_word": false
478
+ },
479
+ {
480
+ "content": "529679678956",
481
+ "lstrip": false,
482
+ "normalized": false,
483
+ "rstrip": false,
484
+ "single_word": false
485
+ },
486
+ {
487
+ "content": "795838284624",
488
+ "lstrip": false,
489
+ "normalized": false,
490
+ "rstrip": false,
491
+ "single_word": false
492
+ },
493
+ {
494
+ "content": "159337222655",
495
+ "lstrip": false,
496
+ "normalized": false,
497
+ "rstrip": false,
498
+ "single_word": false
499
+ },
500
+ {
501
+ "content": "173781362446",
502
+ "lstrip": false,
503
+ "normalized": false,
504
+ "rstrip": false,
505
+ "single_word": false
506
+ },
507
+ {
508
+ "content": "773687856563",
509
+ "lstrip": false,
510
+ "normalized": false,
511
+ "rstrip": false,
512
+ "single_word": false
513
+ },
514
+ {
515
+ "content": "535787224917",
516
+ "lstrip": false,
517
+ "normalized": false,
518
+ "rstrip": false,
519
+ "single_word": false
520
+ },
521
+ {
522
+ "content": "351885857332",
523
+ "lstrip": false,
524
+ "normalized": false,
525
+ "rstrip": false,
526
+ "single_word": false
527
+ },
528
+ {
529
+ "content": "578827344666",
530
+ "lstrip": false,
531
+ "normalized": false,
532
+ "rstrip": false,
533
+ "single_word": false
534
+ },
535
+ {
536
+ "content": "198462689911",
537
+ "lstrip": false,
538
+ "normalized": false,
539
+ "rstrip": false,
540
+ "single_word": false
541
+ },
542
+ {
543
+ "content": "722618266242",
544
+ "lstrip": false,
545
+ "normalized": false,
546
+ "rstrip": false,
547
+ "single_word": false
548
+ },
549
+ {
550
+ "content": "952872416512",
551
+ "lstrip": false,
552
+ "normalized": false,
553
+ "rstrip": false,
554
+ "single_word": false
555
+ },
556
+ {
557
+ "content": "517778845323",
558
+ "lstrip": false,
559
+ "normalized": false,
560
+ "rstrip": false,
561
+ "single_word": false
562
+ },
563
+ {
564
+ "content": "749665846687",
565
+ "lstrip": false,
566
+ "normalized": false,
567
+ "rstrip": false,
568
+ "single_word": false
569
+ },
570
+ {
571
+ "content": "661436365453",
572
+ "lstrip": false,
573
+ "normalized": false,
574
+ "rstrip": false,
575
+ "single_word": false
576
+ },
577
+ {
578
+ "content": "259666844669",
579
+ "lstrip": false,
580
+ "normalized": false,
581
+ "rstrip": false,
582
+ "single_word": false
583
+ },
584
+ {
585
+ "content": "242851284913",
586
+ "lstrip": false,
587
+ "normalized": false,
588
+ "rstrip": false,
589
+ "single_word": false
590
+ },
591
+ {
592
+ "content": "514532995959",
593
+ "lstrip": false,
594
+ "normalized": false,
595
+ "rstrip": false,
596
+ "single_word": false
597
+ },
598
+ {
599
+ "content": "161588262349",
600
+ "lstrip": false,
601
+ "normalized": false,
602
+ "rstrip": false,
603
+ "single_word": false
604
+ },
605
+ {
606
+ "content": "742765629356",
607
+ "lstrip": false,
608
+ "normalized": false,
609
+ "rstrip": false,
610
+ "single_word": false
611
+ },
612
+ {
613
+ "content": "225164373623",
614
+ "lstrip": false,
615
+ "normalized": false,
616
+ "rstrip": false,
617
+ "single_word": false
618
+ },
619
+ {
620
+ "content": "676539973863",
621
+ "lstrip": false,
622
+ "normalized": false,
623
+ "rstrip": false,
624
+ "single_word": false
625
+ },
626
+ {
627
+ "content": "826214551218",
628
+ "lstrip": false,
629
+ "normalized": false,
630
+ "rstrip": false,
631
+ "single_word": false
632
+ },
633
+ {
634
+ "content": "182345464792",
635
+ "lstrip": false,
636
+ "normalized": false,
637
+ "rstrip": false,
638
+ "single_word": false
639
+ },
640
+ {
641
+ "content": "232776999554",
642
+ "lstrip": false,
643
+ "normalized": false,
644
+ "rstrip": false,
645
+ "single_word": false
646
+ },
647
+ {
648
+ "content": "337326533813",
649
+ "lstrip": false,
650
+ "normalized": false,
651
+ "rstrip": false,
652
+ "single_word": false
653
+ },
654
+ {
655
+ "content": "676676697292",
656
+ "lstrip": false,
657
+ "normalized": false,
658
+ "rstrip": false,
659
+ "single_word": false
660
+ },
661
+ {
662
+ "content": "929185622831",
663
+ "lstrip": false,
664
+ "normalized": false,
665
+ "rstrip": false,
666
+ "single_word": false
667
+ },
668
+ {
669
+ "content": "545512344383",
670
+ "lstrip": false,
671
+ "normalized": false,
672
+ "rstrip": false,
673
+ "single_word": false
674
+ },
675
+ {
676
+ "content": "499444466686",
677
+ "lstrip": false,
678
+ "normalized": false,
679
+ "rstrip": false,
680
+ "single_word": false
681
+ },
682
+ {
683
+ "content": "314697386682",
684
+ "lstrip": false,
685
+ "normalized": false,
686
+ "rstrip": false,
687
+ "single_word": false
688
+ },
689
+ {
690
+ "content": "517379856925",
691
+ "lstrip": false,
692
+ "normalized": false,
693
+ "rstrip": false,
694
+ "single_word": false
695
+ },
696
+ {
697
+ "content": "379557332953",
698
+ "lstrip": false,
699
+ "normalized": false,
700
+ "rstrip": false,
701
+ "single_word": false
702
+ },
703
+ {
704
+ "content": "614797267726",
705
+ "lstrip": false,
706
+ "normalized": false,
707
+ "rstrip": false,
708
+ "single_word": false
709
+ },
710
+ {
711
+ "content": "429781429464",
712
+ "lstrip": false,
713
+ "normalized": false,
714
+ "rstrip": false,
715
+ "single_word": false
716
+ },
717
+ {
718
+ "content": "922466849763",
719
+ "lstrip": false,
720
+ "normalized": false,
721
+ "rstrip": false,
722
+ "single_word": false
723
+ },
724
+ {
725
+ "content": "721737645236",
726
+ "lstrip": false,
727
+ "normalized": false,
728
+ "rstrip": false,
729
+ "single_word": false
730
+ },
731
+ {
732
+ "content": "479227349997",
733
+ "lstrip": false,
734
+ "normalized": false,
735
+ "rstrip": false,
736
+ "single_word": false
737
+ },
738
+ {
739
+ "content": "136931728327",
740
+ "lstrip": false,
741
+ "normalized": false,
742
+ "rstrip": false,
743
+ "single_word": false
744
+ },
745
+ {
746
+ "content": "259533577263",
747
+ "lstrip": false,
748
+ "normalized": false,
749
+ "rstrip": false,
750
+ "single_word": false
751
+ },
752
+ {
753
+ "content": "488538864842",
754
+ "lstrip": false,
755
+ "normalized": false,
756
+ "rstrip": false,
757
+ "single_word": false
758
+ },
759
+ {
760
+ "content": "937495658852",
761
+ "lstrip": false,
762
+ "normalized": false,
763
+ "rstrip": false,
764
+ "single_word": false
765
+ },
766
+ {
767
+ "content": "489991411364",
768
+ "lstrip": false,
769
+ "normalized": false,
770
+ "rstrip": false,
771
+ "single_word": false
772
+ },
773
+ {
774
+ "content": "499148455254",
775
+ "lstrip": false,
776
+ "normalized": false,
777
+ "rstrip": false,
778
+ "single_word": false
779
+ },
780
+ {
781
+ "content": "441373944925",
782
+ "lstrip": false,
783
+ "normalized": false,
784
+ "rstrip": false,
785
+ "single_word": false
786
+ },
787
+ {
788
+ "content": "899151413682",
789
+ "lstrip": false,
790
+ "normalized": false,
791
+ "rstrip": false,
792
+ "single_word": false
793
+ },
794
+ {
795
+ "content": "467893531755",
796
+ "lstrip": false,
797
+ "normalized": false,
798
+ "rstrip": false,
799
+ "single_word": false
800
+ },
801
+ {
802
+ "content": "527117488925",
803
+ "lstrip": false,
804
+ "normalized": false,
805
+ "rstrip": false,
806
+ "single_word": false
807
+ },
808
+ {
809
+ "content": "928335588653",
810
+ "lstrip": false,
811
+ "normalized": false,
812
+ "rstrip": false,
813
+ "single_word": false
814
+ },
815
+ {
816
+ "content": "374439448821",
817
+ "lstrip": false,
818
+ "normalized": false,
819
+ "rstrip": false,
820
+ "single_word": false
821
+ },
822
+ {
823
+ "content": "879425227932",
824
+ "lstrip": false,
825
+ "normalized": false,
826
+ "rstrip": false,
827
+ "single_word": false
828
+ },
829
+ {
830
+ "content": "867678158885",
831
+ "lstrip": false,
832
+ "normalized": false,
833
+ "rstrip": false,
834
+ "single_word": false
835
+ },
836
+ {
837
+ "content": "399749397872",
838
+ "lstrip": false,
839
+ "normalized": false,
840
+ "rstrip": false,
841
+ "single_word": false
842
+ },
843
+ {
844
+ "content": "129693547287",
845
+ "lstrip": false,
846
+ "normalized": false,
847
+ "rstrip": false,
848
+ "single_word": false
849
+ },
850
+ {
851
+ "content": "689285841825",
852
+ "lstrip": false,
853
+ "normalized": false,
854
+ "rstrip": false,
855
+ "single_word": false
856
+ },
857
+ {
858
+ "content": "771619544974",
859
+ "lstrip": false,
860
+ "normalized": false,
861
+ "rstrip": false,
862
+ "single_word": false
863
+ },
864
+ {
865
+ "content": "724883568652",
866
+ "lstrip": false,
867
+ "normalized": false,
868
+ "rstrip": false,
869
+ "single_word": false
870
+ },
871
+ {
872
+ "content": "516968424863",
873
+ "lstrip": false,
874
+ "normalized": false,
875
+ "rstrip": false,
876
+ "single_word": false
877
+ },
878
+ {
879
+ "content": "733737988257",
880
+ "lstrip": false,
881
+ "normalized": false,
882
+ "rstrip": false,
883
+ "single_word": false
884
+ },
885
+ {
886
+ "content": "852347289392",
887
+ "lstrip": false,
888
+ "normalized": false,
889
+ "rstrip": false,
890
+ "single_word": false
891
+ },
892
+ {
893
+ "content": "296953381169",
894
+ "lstrip": false,
895
+ "normalized": false,
896
+ "rstrip": false,
897
+ "single_word": false
898
+ },
899
+ {
900
+ "content": "377273562477",
901
+ "lstrip": false,
902
+ "normalized": false,
903
+ "rstrip": false,
904
+ "single_word": false
905
+ },
906
+ {
907
+ "content": "262296912232",
908
+ "lstrip": false,
909
+ "normalized": false,
910
+ "rstrip": false,
911
+ "single_word": false
912
+ },
913
+ {
914
+ "content": "547149832394",
915
+ "lstrip": false,
916
+ "normalized": false,
917
+ "rstrip": false,
918
+ "single_word": false
919
+ },
920
+ {
921
+ "content": "298464134954",
922
+ "lstrip": false,
923
+ "normalized": false,
924
+ "rstrip": false,
925
+ "single_word": false
926
+ },
927
+ {
928
+ "content": "216667245274",
929
+ "lstrip": false,
930
+ "normalized": false,
931
+ "rstrip": false,
932
+ "single_word": false
933
+ },
934
+ {
935
+ "content": "843998562287",
936
+ "lstrip": false,
937
+ "normalized": false,
938
+ "rstrip": false,
939
+ "single_word": false
940
+ },
941
+ {
942
+ "content": "572154333646",
943
+ "lstrip": false,
944
+ "normalized": false,
945
+ "rstrip": false,
946
+ "single_word": false
947
+ },
948
+ {
949
+ "content": "124589118494",
950
+ "lstrip": false,
951
+ "normalized": false,
952
+ "rstrip": false,
953
+ "single_word": false
954
+ },
955
+ {
956
+ "content": "841824384614",
957
+ "lstrip": false,
958
+ "normalized": false,
959
+ "rstrip": false,
960
+ "single_word": false
961
+ },
962
+ {
963
+ "content": "232896526252",
964
+ "lstrip": false,
965
+ "normalized": false,
966
+ "rstrip": false,
967
+ "single_word": false
968
+ },
969
+ {
970
+ "content": "295448593321",
971
+ "lstrip": false,
972
+ "normalized": false,
973
+ "rstrip": false,
974
+ "single_word": false
975
+ },
976
+ {
977
+ "content": "123741461297",
978
+ "lstrip": false,
979
+ "normalized": false,
980
+ "rstrip": false,
981
+ "single_word": false
982
+ },
983
+ {
984
+ "content": "653573457168",
985
+ "lstrip": false,
986
+ "normalized": false,
987
+ "rstrip": false,
988
+ "single_word": false
989
+ },
990
+ {
991
+ "content": "196735786156",
992
+ "lstrip": false,
993
+ "normalized": false,
994
+ "rstrip": false,
995
+ "single_word": false
996
+ },
997
+ {
998
+ "content": "377338713663",
999
+ "lstrip": false,
1000
+ "normalized": false,
1001
+ "rstrip": false,
1002
+ "single_word": false
1003
+ },
1004
+ {
1005
+ "content": "964342468552",
1006
+ "lstrip": false,
1007
+ "normalized": false,
1008
+ "rstrip": false,
1009
+ "single_word": false
1010
+ },
1011
+ {
1012
+ "content": "586855179568",
1013
+ "lstrip": false,
1014
+ "normalized": false,
1015
+ "rstrip": false,
1016
+ "single_word": false
1017
+ },
1018
+ {
1019
+ "content": "484773717614",
1020
+ "lstrip": false,
1021
+ "normalized": false,
1022
+ "rstrip": false,
1023
+ "single_word": false
1024
+ },
1025
+ {
1026
+ "content": "894885246797",
1027
+ "lstrip": false,
1028
+ "normalized": false,
1029
+ "rstrip": false,
1030
+ "single_word": false
1031
+ },
1032
+ {
1033
+ "content": "677896358599",
1034
+ "lstrip": false,
1035
+ "normalized": false,
1036
+ "rstrip": false,
1037
+ "single_word": false
1038
+ },
1039
+ {
1040
+ "content": "848845611563",
1041
+ "lstrip": false,
1042
+ "normalized": false,
1043
+ "rstrip": false,
1044
+ "single_word": false
1045
+ },
1046
+ {
1047
+ "content": "851852651677",
1048
+ "lstrip": false,
1049
+ "normalized": false,
1050
+ "rstrip": false,
1051
+ "single_word": false
1052
+ },
1053
+ {
1054
+ "content": "398549545767",
1055
+ "lstrip": false,
1056
+ "normalized": false,
1057
+ "rstrip": false,
1058
+ "single_word": false
1059
+ },
1060
+ {
1061
+ "content": "454244839926",
1062
+ "lstrip": false,
1063
+ "normalized": false,
1064
+ "rstrip": false,
1065
+ "single_word": false
1066
+ },
1067
+ {
1068
+ "content": "799364566435",
1069
+ "lstrip": false,
1070
+ "normalized": false,
1071
+ "rstrip": false,
1072
+ "single_word": false
1073
+ },
1074
+ {
1075
+ "content": "967114116556",
1076
+ "lstrip": false,
1077
+ "normalized": false,
1078
+ "rstrip": false,
1079
+ "single_word": false
1080
+ },
1081
+ {
1082
+ "content": "817378986438",
1083
+ "lstrip": false,
1084
+ "normalized": false,
1085
+ "rstrip": false,
1086
+ "single_word": false
1087
+ },
1088
+ {
1089
+ "content": "233795848681",
1090
+ "lstrip": false,
1091
+ "normalized": false,
1092
+ "rstrip": false,
1093
+ "single_word": false
1094
+ },
1095
+ {
1096
+ "content": "824387273757",
1097
+ "lstrip": false,
1098
+ "normalized": false,
1099
+ "rstrip": false,
1100
+ "single_word": false
1101
+ },
1102
+ {
1103
+ "content": "916198946615",
1104
+ "lstrip": false,
1105
+ "normalized": false,
1106
+ "rstrip": false,
1107
+ "single_word": false
1108
+ },
1109
+ {
1110
+ "content": "563117729724",
1111
+ "lstrip": false,
1112
+ "normalized": false,
1113
+ "rstrip": false,
1114
+ "single_word": false
1115
+ },
1116
+ {
1117
+ "content": "951794811935",
1118
+ "lstrip": false,
1119
+ "normalized": false,
1120
+ "rstrip": false,
1121
+ "single_word": false
1122
+ },
1123
+ {
1124
+ "content": "374598961236",
1125
+ "lstrip": false,
1126
+ "normalized": false,
1127
+ "rstrip": false,
1128
+ "single_word": false
1129
+ },
1130
+ {
1131
+ "content": "922867396683",
1132
+ "lstrip": false,
1133
+ "normalized": false,
1134
+ "rstrip": false,
1135
+ "single_word": false
1136
+ },
1137
+ {
1138
+ "content": "765737843639",
1139
+ "lstrip": false,
1140
+ "normalized": false,
1141
+ "rstrip": false,
1142
+ "single_word": false
1143
+ },
1144
+ {
1145
+ "content": "175469284871",
1146
+ "lstrip": false,
1147
+ "normalized": false,
1148
+ "rstrip": false,
1149
+ "single_word": false
1150
+ },
1151
+ {
1152
+ "content": "231853711778",
1153
+ "lstrip": false,
1154
+ "normalized": false,
1155
+ "rstrip": false,
1156
+ "single_word": false
1157
+ },
1158
+ {
1159
+ "content": "662426712668",
1160
+ "lstrip": false,
1161
+ "normalized": false,
1162
+ "rstrip": false,
1163
+ "single_word": false
1164
+ },
1165
+ {
1166
+ "content": "711412347158",
1167
+ "lstrip": false,
1168
+ "normalized": false,
1169
+ "rstrip": false,
1170
+ "single_word": false
1171
+ },
1172
+ {
1173
+ "content": "753466987363",
1174
+ "lstrip": false,
1175
+ "normalized": false,
1176
+ "rstrip": false,
1177
+ "single_word": false
1178
+ },
1179
+ {
1180
+ "content": "513361312532",
1181
+ "lstrip": false,
1182
+ "normalized": false,
1183
+ "rstrip": false,
1184
+ "single_word": false
1185
+ },
1186
+ {
1187
+ "content": "712992815957",
1188
+ "lstrip": false,
1189
+ "normalized": false,
1190
+ "rstrip": false,
1191
+ "single_word": false
1192
+ },
1193
+ {
1194
+ "content": "971621888444",
1195
+ "lstrip": false,
1196
+ "normalized": false,
1197
+ "rstrip": false,
1198
+ "single_word": false
1199
+ },
1200
+ {
1201
+ "content": "829235161526",
1202
+ "lstrip": false,
1203
+ "normalized": false,
1204
+ "rstrip": false,
1205
+ "single_word": false
1206
+ },
1207
+ {
1208
+ "content": "585544633356",
1209
+ "lstrip": false,
1210
+ "normalized": false,
1211
+ "rstrip": false,
1212
+ "single_word": false
1213
+ },
1214
+ {
1215
+ "content": "582471228164",
1216
+ "lstrip": false,
1217
+ "normalized": false,
1218
+ "rstrip": false,
1219
+ "single_word": false
1220
+ },
1221
+ {
1222
+ "content": "678666359123",
1223
+ "lstrip": false,
1224
+ "normalized": false,
1225
+ "rstrip": false,
1226
+ "single_word": false
1227
+ },
1228
+ {
1229
+ "content": "557533689478",
1230
+ "lstrip": false,
1231
+ "normalized": false,
1232
+ "rstrip": false,
1233
+ "single_word": false
1234
+ },
1235
+ {
1236
+ "content": "632962475133",
1237
+ "lstrip": false,
1238
+ "normalized": false,
1239
+ "rstrip": false,
1240
+ "single_word": false
1241
+ },
1242
+ {
1243
+ "content": "484489193824",
1244
+ "lstrip": false,
1245
+ "normalized": false,
1246
+ "rstrip": false,
1247
+ "single_word": false
1248
+ },
1249
+ {
1250
+ "content": "489562189822",
1251
+ "lstrip": false,
1252
+ "normalized": false,
1253
+ "rstrip": false,
1254
+ "single_word": false
1255
+ },
1256
+ {
1257
+ "content": "589547936288",
1258
+ "lstrip": false,
1259
+ "normalized": false,
1260
+ "rstrip": false,
1261
+ "single_word": false
1262
+ },
1263
+ {
1264
+ "content": "363214487524",
1265
+ "lstrip": false,
1266
+ "normalized": false,
1267
+ "rstrip": false,
1268
+ "single_word": false
1269
+ },
1270
+ {
1271
+ "content": "244885399387",
1272
+ "lstrip": false,
1273
+ "normalized": false,
1274
+ "rstrip": false,
1275
+ "single_word": false
1276
+ },
1277
+ {
1278
+ "content": "431751228368",
1279
+ "lstrip": false,
1280
+ "normalized": false,
1281
+ "rstrip": false,
1282
+ "single_word": false
1283
+ },
1284
+ {
1285
+ "content": "433581868192",
1286
+ "lstrip": false,
1287
+ "normalized": false,
1288
+ "rstrip": false,
1289
+ "single_word": false
1290
+ },
1291
+ {
1292
+ "content": "486391569221",
1293
+ "lstrip": false,
1294
+ "normalized": false,
1295
+ "rstrip": false,
1296
+ "single_word": false
1297
+ },
1298
+ {
1299
+ "content": "185438575221",
1300
+ "lstrip": false,
1301
+ "normalized": false,
1302
+ "rstrip": false,
1303
+ "single_word": false
1304
+ },
1305
+ {
1306
+ "content": "126574388585",
1307
+ "lstrip": false,
1308
+ "normalized": false,
1309
+ "rstrip": false,
1310
+ "single_word": false
1311
+ },
1312
+ {
1313
+ "content": "741757479784",
1314
+ "lstrip": false,
1315
+ "normalized": false,
1316
+ "rstrip": false,
1317
+ "single_word": false
1318
+ },
1319
+ {
1320
+ "content": "529854679937",
1321
+ "lstrip": false,
1322
+ "normalized": false,
1323
+ "rstrip": false,
1324
+ "single_word": false
1325
+ },
1326
+ {
1327
+ "content": "996116119839",
1328
+ "lstrip": false,
1329
+ "normalized": false,
1330
+ "rstrip": false,
1331
+ "single_word": false
1332
+ },
1333
+ {
1334
+ "content": "616248973917",
1335
+ "lstrip": false,
1336
+ "normalized": false,
1337
+ "rstrip": false,
1338
+ "single_word": false
1339
+ },
1340
+ {
1341
+ "content": "763531783491",
1342
+ "lstrip": false,
1343
+ "normalized": false,
1344
+ "rstrip": false,
1345
+ "single_word": false
1346
+ },
1347
+ {
1348
+ "content": "955456118295",
1349
+ "lstrip": false,
1350
+ "normalized": false,
1351
+ "rstrip": false,
1352
+ "single_word": false
1353
+ },
1354
+ {
1355
+ "content": "364196983365",
1356
+ "lstrip": false,
1357
+ "normalized": false,
1358
+ "rstrip": false,
1359
+ "single_word": false
1360
+ },
1361
+ {
1362
+ "content": "195792996468",
1363
+ "lstrip": false,
1364
+ "normalized": false,
1365
+ "rstrip": false,
1366
+ "single_word": false
1367
+ },
1368
+ {
1369
+ "content": "151859598873",
1370
+ "lstrip": false,
1371
+ "normalized": false,
1372
+ "rstrip": false,
1373
+ "single_word": false
1374
+ },
1375
+ {
1376
+ "content": "399223169721",
1377
+ "lstrip": false,
1378
+ "normalized": false,
1379
+ "rstrip": false,
1380
+ "single_word": false
1381
+ },
1382
+ {
1383
+ "content": "938488813964",
1384
+ "lstrip": false,
1385
+ "normalized": false,
1386
+ "rstrip": false,
1387
+ "single_word": false
1388
+ },
1389
+ {
1390
+ "content": "961981959227",
1391
+ "lstrip": false,
1392
+ "normalized": false,
1393
+ "rstrip": false,
1394
+ "single_word": false
1395
+ },
1396
+ {
1397
+ "content": "183368827562",
1398
+ "lstrip": false,
1399
+ "normalized": false,
1400
+ "rstrip": false,
1401
+ "single_word": false
1402
+ },
1403
+ {
1404
+ "content": "533417736566",
1405
+ "lstrip": false,
1406
+ "normalized": false,
1407
+ "rstrip": false,
1408
+ "single_word": false
1409
+ },
1410
+ {
1411
+ "content": "786391632558",
1412
+ "lstrip": false,
1413
+ "normalized": false,
1414
+ "rstrip": false,
1415
+ "single_word": false
1416
+ },
1417
+ {
1418
+ "content": "665661658354",
1419
+ "lstrip": false,
1420
+ "normalized": false,
1421
+ "rstrip": false,
1422
+ "single_word": false
1423
+ },
1424
+ {
1425
+ "content": "693281533643",
1426
+ "lstrip": false,
1427
+ "normalized": false,
1428
+ "rstrip": false,
1429
+ "single_word": false
1430
+ },
1431
+ {
1432
+ "content": "475794684356",
1433
+ "lstrip": false,
1434
+ "normalized": false,
1435
+ "rstrip": false,
1436
+ "single_word": false
1437
+ },
1438
+ {
1439
+ "content": "652154162978",
1440
+ "lstrip": false,
1441
+ "normalized": false,
1442
+ "rstrip": false,
1443
+ "single_word": false
1444
+ },
1445
+ {
1446
+ "content": "753233719644",
1447
+ "lstrip": false,
1448
+ "normalized": false,
1449
+ "rstrip": false,
1450
+ "single_word": false
1451
+ },
1452
+ {
1453
+ "content": "668514843129",
1454
+ "lstrip": false,
1455
+ "normalized": false,
1456
+ "rstrip": false,
1457
+ "single_word": false
1458
+ },
1459
+ {
1460
+ "content": "819162623892",
1461
+ "lstrip": false,
1462
+ "normalized": false,
1463
+ "rstrip": false,
1464
+ "single_word": false
1465
+ },
1466
+ {
1467
+ "content": "941169431859",
1468
+ "lstrip": false,
1469
+ "normalized": false,
1470
+ "rstrip": false,
1471
+ "single_word": false
1472
+ },
1473
+ {
1474
+ "content": "877385381798",
1475
+ "lstrip": false,
1476
+ "normalized": false,
1477
+ "rstrip": false,
1478
+ "single_word": false
1479
+ },
1480
+ {
1481
+ "content": "752644929761",
1482
+ "lstrip": false,
1483
+ "normalized": false,
1484
+ "rstrip": false,
1485
+ "single_word": false
1486
+ },
1487
+ {
1488
+ "content": "881136466196",
1489
+ "lstrip": false,
1490
+ "normalized": false,
1491
+ "rstrip": false,
1492
+ "single_word": false
1493
+ },
1494
+ {
1495
+ "content": "275597777299",
1496
+ "lstrip": false,
1497
+ "normalized": false,
1498
+ "rstrip": false,
1499
+ "single_word": false
1500
+ },
1501
+ {
1502
+ "content": "731681792655",
1503
+ "lstrip": false,
1504
+ "normalized": false,
1505
+ "rstrip": false,
1506
+ "single_word": false
1507
+ },
1508
+ {
1509
+ "content": "961133895172",
1510
+ "lstrip": false,
1511
+ "normalized": false,
1512
+ "rstrip": false,
1513
+ "single_word": false
1514
+ },
1515
+ {
1516
+ "content": "864718285734",
1517
+ "lstrip": false,
1518
+ "normalized": false,
1519
+ "rstrip": false,
1520
+ "single_word": false
1521
+ },
1522
+ {
1523
+ "content": "963852916563",
1524
+ "lstrip": false,
1525
+ "normalized": false,
1526
+ "rstrip": false,
1527
+ "single_word": false
1528
+ },
1529
+ {
1530
+ "content": "319584985416",
1531
+ "lstrip": false,
1532
+ "normalized": false,
1533
+ "rstrip": false,
1534
+ "single_word": false
1535
+ },
1536
+ {
1537
+ "content": "563365646341",
1538
+ "lstrip": false,
1539
+ "normalized": false,
1540
+ "rstrip": false,
1541
+ "single_word": false
1542
+ },
1543
+ {
1544
+ "content": "811371928234",
1545
+ "lstrip": false,
1546
+ "normalized": false,
1547
+ "rstrip": false,
1548
+ "single_word": false
1549
+ },
1550
+ {
1551
+ "content": "837131396371",
1552
+ "lstrip": false,
1553
+ "normalized": false,
1554
+ "rstrip": false,
1555
+ "single_word": false
1556
+ },
1557
+ {
1558
+ "content": "267514771964",
1559
+ "lstrip": false,
1560
+ "normalized": false,
1561
+ "rstrip": false,
1562
+ "single_word": false
1563
+ },
1564
+ {
1565
+ "content": "944513428457",
1566
+ "lstrip": false,
1567
+ "normalized": false,
1568
+ "rstrip": false,
1569
+ "single_word": false
1570
+ },
1571
+ {
1572
+ "content": "117298239631",
1573
+ "lstrip": false,
1574
+ "normalized": false,
1575
+ "rstrip": false,
1576
+ "single_word": false
1577
+ },
1578
+ {
1579
+ "content": "158142752582",
1580
+ "lstrip": false,
1581
+ "normalized": false,
1582
+ "rstrip": false,
1583
+ "single_word": false
1584
+ },
1585
+ {
1586
+ "content": "252867443568",
1587
+ "lstrip": false,
1588
+ "normalized": false,
1589
+ "rstrip": false,
1590
+ "single_word": false
1591
+ },
1592
+ {
1593
+ "content": "839269684865",
1594
+ "lstrip": false,
1595
+ "normalized": false,
1596
+ "rstrip": false,
1597
+ "single_word": false
1598
+ },
1599
+ {
1600
+ "content": "612788593128",
1601
+ "lstrip": false,
1602
+ "normalized": false,
1603
+ "rstrip": false,
1604
+ "single_word": false
1605
+ },
1606
+ {
1607
+ "content": "145669731981",
1608
+ "lstrip": false,
1609
+ "normalized": false,
1610
+ "rstrip": false,
1611
+ "single_word": false
1612
+ },
1613
+ {
1614
+ "content": "121557291859",
1615
+ "lstrip": false,
1616
+ "normalized": false,
1617
+ "rstrip": false,
1618
+ "single_word": false
1619
+ },
1620
+ {
1621
+ "content": "245416776926",
1622
+ "lstrip": false,
1623
+ "normalized": false,
1624
+ "rstrip": false,
1625
+ "single_word": false
1626
+ },
1627
+ {
1628
+ "content": "799417897197",
1629
+ "lstrip": false,
1630
+ "normalized": false,
1631
+ "rstrip": false,
1632
+ "single_word": false
1633
+ },
1634
+ {
1635
+ "content": "997958836435",
1636
+ "lstrip": false,
1637
+ "normalized": false,
1638
+ "rstrip": false,
1639
+ "single_word": false
1640
+ },
1641
+ {
1642
+ "content": "892336777248",
1643
+ "lstrip": false,
1644
+ "normalized": false,
1645
+ "rstrip": false,
1646
+ "single_word": false
1647
+ },
1648
+ {
1649
+ "content": "158929292238",
1650
+ "lstrip": false,
1651
+ "normalized": false,
1652
+ "rstrip": false,
1653
+ "single_word": false
1654
+ },
1655
+ {
1656
+ "content": "581976444672",
1657
+ "lstrip": false,
1658
+ "normalized": false,
1659
+ "rstrip": false,
1660
+ "single_word": false
1661
+ },
1662
+ {
1663
+ "content": "897784492783",
1664
+ "lstrip": false,
1665
+ "normalized": false,
1666
+ "rstrip": false,
1667
+ "single_word": false
1668
+ },
1669
+ {
1670
+ "content": "492373714791",
1671
+ "lstrip": false,
1672
+ "normalized": false,
1673
+ "rstrip": false,
1674
+ "single_word": false
1675
+ },
1676
+ {
1677
+ "content": "512659818733",
1678
+ "lstrip": false,
1679
+ "normalized": false,
1680
+ "rstrip": false,
1681
+ "single_word": false
1682
+ },
1683
+ {
1684
+ "content": "881112998642",
1685
+ "lstrip": false,
1686
+ "normalized": false,
1687
+ "rstrip": false,
1688
+ "single_word": false
1689
+ },
1690
+ {
1691
+ "content": "619454958782",
1692
+ "lstrip": false,
1693
+ "normalized": false,
1694
+ "rstrip": false,
1695
+ "single_word": false
1696
+ },
1697
+ {
1698
+ "content": "431149748713",
1699
+ "lstrip": false,
1700
+ "normalized": false,
1701
+ "rstrip": false,
1702
+ "single_word": false
1703
+ },
1704
+ {
1705
+ "content": "624221476921",
1706
+ "lstrip": false,
1707
+ "normalized": false,
1708
+ "rstrip": false,
1709
+ "single_word": false
1710
+ },
1711
+ {
1712
+ "content": "125866399464",
1713
+ "lstrip": false,
1714
+ "normalized": false,
1715
+ "rstrip": false,
1716
+ "single_word": false
1717
+ },
1718
+ {
1719
+ "content": "339882449689",
1720
+ "lstrip": false,
1721
+ "normalized": false,
1722
+ "rstrip": false,
1723
+ "single_word": false
1724
+ },
1725
+ {
1726
+ "content": "186198784585",
1727
+ "lstrip": false,
1728
+ "normalized": false,
1729
+ "rstrip": false,
1730
+ "single_word": false
1731
+ },
1732
+ {
1733
+ "content": "943193294691",
1734
+ "lstrip": false,
1735
+ "normalized": false,
1736
+ "rstrip": false,
1737
+ "single_word": false
1738
+ },
1739
+ {
1740
+ "content": "955668961269",
1741
+ "lstrip": false,
1742
+ "normalized": false,
1743
+ "rstrip": false,
1744
+ "single_word": false
1745
+ },
1746
+ {
1747
+ "content": "232787996724",
1748
+ "lstrip": false,
1749
+ "normalized": false,
1750
+ "rstrip": false,
1751
+ "single_word": false
1752
+ },
1753
+ {
1754
+ "content": "215671314196",
1755
+ "lstrip": false,
1756
+ "normalized": false,
1757
+ "rstrip": false,
1758
+ "single_word": false
1759
+ },
1760
+ {
1761
+ "content": "286173241916",
1762
+ "lstrip": false,
1763
+ "normalized": false,
1764
+ "rstrip": false,
1765
+ "single_word": false
1766
+ },
1767
+ {
1768
+ "content": "745977673725",
1769
+ "lstrip": false,
1770
+ "normalized": false,
1771
+ "rstrip": false,
1772
+ "single_word": false
1773
+ },
1774
+ {
1775
+ "content": "556976448182",
1776
+ "lstrip": false,
1777
+ "normalized": false,
1778
+ "rstrip": false,
1779
+ "single_word": false
1780
+ },
1781
+ {
1782
+ "content": "599961512792",
1783
+ "lstrip": false,
1784
+ "normalized": false,
1785
+ "rstrip": false,
1786
+ "single_word": false
1787
+ },
1788
+ {
1789
+ "content": "766294538337",
1790
+ "lstrip": false,
1791
+ "normalized": false,
1792
+ "rstrip": false,
1793
+ "single_word": false
1794
+ },
1795
+ {
1796
+ "content": "934912591213",
1797
+ "lstrip": false,
1798
+ "normalized": false,
1799
+ "rstrip": false,
1800
+ "single_word": false
1801
+ },
1802
+ {
1803
+ "content": "295118729589",
1804
+ "lstrip": false,
1805
+ "normalized": false,
1806
+ "rstrip": false,
1807
+ "single_word": false
1808
+ },
1809
+ {
1810
+ "content": "529455466433",
1811
+ "lstrip": false,
1812
+ "normalized": false,
1813
+ "rstrip": false,
1814
+ "single_word": false
1815
+ },
1816
+ {
1817
+ "content": "196119929397",
1818
+ "lstrip": false,
1819
+ "normalized": false,
1820
+ "rstrip": false,
1821
+ "single_word": false
1822
+ },
1823
+ {
1824
+ "content": "379571934299",
1825
+ "lstrip": false,
1826
+ "normalized": false,
1827
+ "rstrip": false,
1828
+ "single_word": false
1829
+ },
1830
+ {
1831
+ "content": "251789649997",
1832
+ "lstrip": false,
1833
+ "normalized": false,
1834
+ "rstrip": false,
1835
+ "single_word": false
1836
+ },
1837
+ {
1838
+ "content": "564544131355",
1839
+ "lstrip": false,
1840
+ "normalized": false,
1841
+ "rstrip": false,
1842
+ "single_word": false
1843
+ },
1844
+ {
1845
+ "content": "244371196654",
1846
+ "lstrip": false,
1847
+ "normalized": false,
1848
+ "rstrip": false,
1849
+ "single_word": false
1850
+ },
1851
+ {
1852
+ "content": "384598329253",
1853
+ "lstrip": false,
1854
+ "normalized": false,
1855
+ "rstrip": false,
1856
+ "single_word": false
1857
+ },
1858
+ {
1859
+ "content": "887753195844",
1860
+ "lstrip": false,
1861
+ "normalized": false,
1862
+ "rstrip": false,
1863
+ "single_word": false
1864
+ },
1865
+ {
1866
+ "content": "364947325679",
1867
+ "lstrip": false,
1868
+ "normalized": false,
1869
+ "rstrip": false,
1870
+ "single_word": false
1871
+ },
1872
+ {
1873
+ "content": "655517954651",
1874
+ "lstrip": false,
1875
+ "normalized": false,
1876
+ "rstrip": false,
1877
+ "single_word": false
1878
+ },
1879
+ {
1880
+ "content": "673948786567",
1881
+ "lstrip": false,
1882
+ "normalized": false,
1883
+ "rstrip": false,
1884
+ "single_word": false
1885
+ },
1886
+ {
1887
+ "content": "857231548835",
1888
+ "lstrip": false,
1889
+ "normalized": false,
1890
+ "rstrip": false,
1891
+ "single_word": false
1892
+ },
1893
+ {
1894
+ "content": "816115936673",
1895
+ "lstrip": false,
1896
+ "normalized": false,
1897
+ "rstrip": false,
1898
+ "single_word": false
1899
+ },
1900
+ {
1901
+ "content": "644234165531",
1902
+ "lstrip": false,
1903
+ "normalized": false,
1904
+ "rstrip": false,
1905
+ "single_word": false
1906
+ },
1907
+ {
1908
+ "content": "182782912224",
1909
+ "lstrip": false,
1910
+ "normalized": false,
1911
+ "rstrip": false,
1912
+ "single_word": false
1913
+ },
1914
+ {
1915
+ "content": "234316622259",
1916
+ "lstrip": false,
1917
+ "normalized": false,
1918
+ "rstrip": false,
1919
+ "single_word": false
1920
+ },
1921
+ {
1922
+ "content": "421369185549",
1923
+ "lstrip": false,
1924
+ "normalized": false,
1925
+ "rstrip": false,
1926
+ "single_word": false
1927
+ },
1928
+ {
1929
+ "content": "434632855397",
1930
+ "lstrip": false,
1931
+ "normalized": false,
1932
+ "rstrip": false,
1933
+ "single_word": false
1934
+ },
1935
+ {
1936
+ "content": "921889371893",
1937
+ "lstrip": false,
1938
+ "normalized": false,
1939
+ "rstrip": false,
1940
+ "single_word": false
1941
+ },
1942
+ {
1943
+ "content": "415956914763",
1944
+ "lstrip": false,
1945
+ "normalized": false,
1946
+ "rstrip": false,
1947
+ "single_word": false
1948
+ },
1949
+ {
1950
+ "content": "598916996413",
1951
+ "lstrip": false,
1952
+ "normalized": false,
1953
+ "rstrip": false,
1954
+ "single_word": false
1955
+ },
1956
+ {
1957
+ "content": "773671349113",
1958
+ "lstrip": false,
1959
+ "normalized": false,
1960
+ "rstrip": false,
1961
+ "single_word": false
1962
+ },
1963
+ {
1964
+ "content": "952465217972",
1965
+ "lstrip": false,
1966
+ "normalized": false,
1967
+ "rstrip": false,
1968
+ "single_word": false
1969
+ },
1970
+ {
1971
+ "content": "117657531962",
1972
+ "lstrip": false,
1973
+ "normalized": false,
1974
+ "rstrip": false,
1975
+ "single_word": false
1976
+ },
1977
+ {
1978
+ "content": "729825168745",
1979
+ "lstrip": false,
1980
+ "normalized": false,
1981
+ "rstrip": false,
1982
+ "single_word": false
1983
+ },
1984
+ {
1985
+ "content": "691315125346",
1986
+ "lstrip": false,
1987
+ "normalized": false,
1988
+ "rstrip": false,
1989
+ "single_word": false
1990
+ },
1991
+ {
1992
+ "content": "768461952319",
1993
+ "lstrip": false,
1994
+ "normalized": false,
1995
+ "rstrip": false,
1996
+ "single_word": false
1997
+ },
1998
+ {
1999
+ "content": "664847713559",
2000
+ "lstrip": false,
2001
+ "normalized": false,
2002
+ "rstrip": false,
2003
+ "single_word": false
2004
+ },
2005
+ {
2006
+ "content": "953267689786",
2007
+ "lstrip": false,
2008
+ "normalized": false,
2009
+ "rstrip": false,
2010
+ "single_word": false
2011
+ },
2012
+ {
2013
+ "content": "886464195129",
2014
+ "lstrip": false,
2015
+ "normalized": false,
2016
+ "rstrip": false,
2017
+ "single_word": false
2018
+ },
2019
+ {
2020
+ "content": "824488329416",
2021
+ "lstrip": false,
2022
+ "normalized": false,
2023
+ "rstrip": false,
2024
+ "single_word": false
2025
+ },
2026
+ {
2027
+ "content": "837873762491",
2028
+ "lstrip": false,
2029
+ "normalized": false,
2030
+ "rstrip": false,
2031
+ "single_word": false
2032
+ },
2033
+ {
2034
+ "content": "532833541879",
2035
+ "lstrip": false,
2036
+ "normalized": false,
2037
+ "rstrip": false,
2038
+ "single_word": false
2039
+ },
2040
+ {
2041
+ "content": "669183782449",
2042
+ "lstrip": false,
2043
+ "normalized": false,
2044
+ "rstrip": false,
2045
+ "single_word": false
2046
+ },
2047
+ {
2048
+ "content": "941976537588",
2049
+ "lstrip": false,
2050
+ "normalized": false,
2051
+ "rstrip": false,
2052
+ "single_word": false
2053
+ },
2054
+ {
2055
+ "content": "739394546916",
2056
+ "lstrip": false,
2057
+ "normalized": false,
2058
+ "rstrip": false,
2059
+ "single_word": false
2060
+ },
2061
+ {
2062
+ "content": "267954879268",
2063
+ "lstrip": false,
2064
+ "normalized": false,
2065
+ "rstrip": false,
2066
+ "single_word": false
2067
+ },
2068
+ {
2069
+ "content": "637551427887",
2070
+ "lstrip": false,
2071
+ "normalized": false,
2072
+ "rstrip": false,
2073
+ "single_word": false
2074
+ },
2075
+ {
2076
+ "content": "217756494954",
2077
+ "lstrip": false,
2078
+ "normalized": false,
2079
+ "rstrip": false,
2080
+ "single_word": false
2081
+ },
2082
+ {
2083
+ "content": "524444658383",
2084
+ "lstrip": false,
2085
+ "normalized": false,
2086
+ "rstrip": false,
2087
+ "single_word": false
2088
+ },
2089
+ {
2090
+ "content": "117783274348",
2091
+ "lstrip": false,
2092
+ "normalized": false,
2093
+ "rstrip": false,
2094
+ "single_word": false
2095
+ },
2096
+ {
2097
+ "content": "138218735276",
2098
+ "lstrip": false,
2099
+ "normalized": false,
2100
+ "rstrip": false,
2101
+ "single_word": false
2102
+ },
2103
+ {
2104
+ "content": "814611949491",
2105
+ "lstrip": false,
2106
+ "normalized": false,
2107
+ "rstrip": false,
2108
+ "single_word": false
2109
+ },
2110
+ {
2111
+ "content": "711641973413",
2112
+ "lstrip": false,
2113
+ "normalized": false,
2114
+ "rstrip": false,
2115
+ "single_word": false
2116
+ },
2117
+ {
2118
+ "content": "499156317423",
2119
+ "lstrip": false,
2120
+ "normalized": false,
2121
+ "rstrip": false,
2122
+ "single_word": false
2123
+ },
2124
+ {
2125
+ "content": "515856611931",
2126
+ "lstrip": false,
2127
+ "normalized": false,
2128
+ "rstrip": false,
2129
+ "single_word": false
2130
+ },
2131
+ {
2132
+ "content": "454164859837",
2133
+ "lstrip": false,
2134
+ "normalized": false,
2135
+ "rstrip": false,
2136
+ "single_word": false
2137
+ },
2138
+ {
2139
+ "content": "345271433112",
2140
+ "lstrip": false,
2141
+ "normalized": false,
2142
+ "rstrip": false,
2143
+ "single_word": false
2144
+ },
2145
+ {
2146
+ "content": "462294118988",
2147
+ "lstrip": false,
2148
+ "normalized": false,
2149
+ "rstrip": false,
2150
+ "single_word": false
2151
+ },
2152
+ {
2153
+ "content": "511785788222",
2154
+ "lstrip": false,
2155
+ "normalized": false,
2156
+ "rstrip": false,
2157
+ "single_word": false
2158
+ },
2159
+ {
2160
+ "content": "497294727353",
2161
+ "lstrip": false,
2162
+ "normalized": false,
2163
+ "rstrip": false,
2164
+ "single_word": false
2165
+ },
2166
+ {
2167
+ "content": "866519986723",
2168
+ "lstrip": false,
2169
+ "normalized": false,
2170
+ "rstrip": false,
2171
+ "single_word": false
2172
+ },
2173
+ {
2174
+ "content": "334513529294",
2175
+ "lstrip": false,
2176
+ "normalized": false,
2177
+ "rstrip": false,
2178
+ "single_word": false
2179
+ },
2180
+ {
2181
+ "content": "549946382131",
2182
+ "lstrip": false,
2183
+ "normalized": false,
2184
+ "rstrip": false,
2185
+ "single_word": false
2186
+ },
2187
+ {
2188
+ "content": "284445431422",
2189
+ "lstrip": false,
2190
+ "normalized": false,
2191
+ "rstrip": false,
2192
+ "single_word": false
2193
+ },
2194
+ {
2195
+ "content": "396521188476",
2196
+ "lstrip": false,
2197
+ "normalized": false,
2198
+ "rstrip": false,
2199
+ "single_word": false
2200
+ },
2201
+ {
2202
+ "content": "421435255895",
2203
+ "lstrip": false,
2204
+ "normalized": false,
2205
+ "rstrip": false,
2206
+ "single_word": false
2207
+ },
2208
+ {
2209
+ "content": "133373659361",
2210
+ "lstrip": false,
2211
+ "normalized": false,
2212
+ "rstrip": false,
2213
+ "single_word": false
2214
+ },
2215
+ {
2216
+ "content": "322683334381",
2217
+ "lstrip": false,
2218
+ "normalized": false,
2219
+ "rstrip": false,
2220
+ "single_word": false
2221
+ },
2222
+ {
2223
+ "content": "228358422847",
2224
+ "lstrip": false,
2225
+ "normalized": false,
2226
+ "rstrip": false,
2227
+ "single_word": false
2228
+ },
2229
+ {
2230
+ "content": "291762694874",
2231
+ "lstrip": false,
2232
+ "normalized": false,
2233
+ "rstrip": false,
2234
+ "single_word": false
2235
+ },
2236
+ {
2237
+ "content": "143182978129",
2238
+ "lstrip": false,
2239
+ "normalized": false,
2240
+ "rstrip": false,
2241
+ "single_word": false
2242
+ },
2243
+ {
2244
+ "content": "511923256573",
2245
+ "lstrip": false,
2246
+ "normalized": false,
2247
+ "rstrip": false,
2248
+ "single_word": false
2249
+ },
2250
+ {
2251
+ "content": "327158398268",
2252
+ "lstrip": false,
2253
+ "normalized": false,
2254
+ "rstrip": false,
2255
+ "single_word": false
2256
+ },
2257
+ {
2258
+ "content": "879764613759",
2259
+ "lstrip": false,
2260
+ "normalized": false,
2261
+ "rstrip": false,
2262
+ "single_word": false
2263
+ },
2264
+ {
2265
+ "content": "564395222747",
2266
+ "lstrip": false,
2267
+ "normalized": false,
2268
+ "rstrip": false,
2269
+ "single_word": false
2270
+ },
2271
+ {
2272
+ "content": "451161679736",
2273
+ "lstrip": false,
2274
+ "normalized": false,
2275
+ "rstrip": false,
2276
+ "single_word": false
2277
+ },
2278
+ {
2279
+ "content": "538631466654",
2280
+ "lstrip": false,
2281
+ "normalized": false,
2282
+ "rstrip": false,
2283
+ "single_word": false
2284
+ },
2285
+ {
2286
+ "content": "221762325616",
2287
+ "lstrip": false,
2288
+ "normalized": false,
2289
+ "rstrip": false,
2290
+ "single_word": false
2291
+ },
2292
+ {
2293
+ "content": "218391991184",
2294
+ "lstrip": false,
2295
+ "normalized": false,
2296
+ "rstrip": false,
2297
+ "single_word": false
2298
+ },
2299
+ {
2300
+ "content": "322589379462",
2301
+ "lstrip": false,
2302
+ "normalized": false,
2303
+ "rstrip": false,
2304
+ "single_word": false
2305
+ },
2306
+ {
2307
+ "content": "876537814263",
2308
+ "lstrip": false,
2309
+ "normalized": false,
2310
+ "rstrip": false,
2311
+ "single_word": false
2312
+ },
2313
+ {
2314
+ "content": "152676556624",
2315
+ "lstrip": false,
2316
+ "normalized": false,
2317
+ "rstrip": false,
2318
+ "single_word": false
2319
+ },
2320
+ {
2321
+ "content": "332522971941",
2322
+ "lstrip": false,
2323
+ "normalized": false,
2324
+ "rstrip": false,
2325
+ "single_word": false
2326
+ },
2327
+ {
2328
+ "content": "884354318946",
2329
+ "lstrip": false,
2330
+ "normalized": false,
2331
+ "rstrip": false,
2332
+ "single_word": false
2333
+ },
2334
+ {
2335
+ "content": "513349618943",
2336
+ "lstrip": false,
2337
+ "normalized": false,
2338
+ "rstrip": false,
2339
+ "single_word": false
2340
+ },
2341
+ {
2342
+ "content": "116639746413",
2343
+ "lstrip": false,
2344
+ "normalized": false,
2345
+ "rstrip": false,
2346
+ "single_word": false
2347
+ },
2348
+ {
2349
+ "content": "635185846287",
2350
+ "lstrip": false,
2351
+ "normalized": false,
2352
+ "rstrip": false,
2353
+ "single_word": false
2354
+ },
2355
+ {
2356
+ "content": "993832498489",
2357
+ "lstrip": false,
2358
+ "normalized": false,
2359
+ "rstrip": false,
2360
+ "single_word": false
2361
+ },
2362
+ {
2363
+ "content": "813981174797",
2364
+ "lstrip": false,
2365
+ "normalized": false,
2366
+ "rstrip": false,
2367
+ "single_word": false
2368
+ },
2369
+ {
2370
+ "content": "438745114173",
2371
+ "lstrip": false,
2372
+ "normalized": false,
2373
+ "rstrip": false,
2374
+ "single_word": false
2375
+ },
2376
+ {
2377
+ "content": "983493951323",
2378
+ "lstrip": false,
2379
+ "normalized": false,
2380
+ "rstrip": false,
2381
+ "single_word": false
2382
+ },
2383
+ {
2384
+ "content": "724492262421",
2385
+ "lstrip": false,
2386
+ "normalized": false,
2387
+ "rstrip": false,
2388
+ "single_word": false
2389
+ },
2390
+ {
2391
+ "content": "622553389126",
2392
+ "lstrip": false,
2393
+ "normalized": false,
2394
+ "rstrip": false,
2395
+ "single_word": false
2396
+ },
2397
+ {
2398
+ "content": "889965243135",
2399
+ "lstrip": false,
2400
+ "normalized": false,
2401
+ "rstrip": false,
2402
+ "single_word": false
2403
+ },
2404
+ {
2405
+ "content": "364492359246",
2406
+ "lstrip": false,
2407
+ "normalized": false,
2408
+ "rstrip": false,
2409
+ "single_word": false
2410
+ },
2411
+ {
2412
+ "content": "154962668224",
2413
+ "lstrip": false,
2414
+ "normalized": false,
2415
+ "rstrip": false,
2416
+ "single_word": false
2417
+ },
2418
+ {
2419
+ "content": "179564995814",
2420
+ "lstrip": false,
2421
+ "normalized": false,
2422
+ "rstrip": false,
2423
+ "single_word": false
2424
+ },
2425
+ {
2426
+ "content": "418412875665",
2427
+ "lstrip": false,
2428
+ "normalized": false,
2429
+ "rstrip": false,
2430
+ "single_word": false
2431
+ },
2432
+ {
2433
+ "content": "718951851413",
2434
+ "lstrip": false,
2435
+ "normalized": false,
2436
+ "rstrip": false,
2437
+ "single_word": false
2438
+ },
2439
+ {
2440
+ "content": "699446724178",
2441
+ "lstrip": false,
2442
+ "normalized": false,
2443
+ "rstrip": false,
2444
+ "single_word": false
2445
+ },
2446
+ {
2447
+ "content": "624266421831",
2448
+ "lstrip": false,
2449
+ "normalized": false,
2450
+ "rstrip": false,
2451
+ "single_word": false
2452
+ },
2453
+ {
2454
+ "content": "815458725125",
2455
+ "lstrip": false,
2456
+ "normalized": false,
2457
+ "rstrip": false,
2458
+ "single_word": false
2459
+ },
2460
+ {
2461
+ "content": "455423278865",
2462
+ "lstrip": false,
2463
+ "normalized": false,
2464
+ "rstrip": false,
2465
+ "single_word": false
2466
+ },
2467
+ {
2468
+ "content": "393741199486",
2469
+ "lstrip": false,
2470
+ "normalized": false,
2471
+ "rstrip": false,
2472
+ "single_word": false
2473
+ },
2474
+ {
2475
+ "content": "328552864359",
2476
+ "lstrip": false,
2477
+ "normalized": false,
2478
+ "rstrip": false,
2479
+ "single_word": false
2480
+ },
2481
+ {
2482
+ "content": "211662639865",
2483
+ "lstrip": false,
2484
+ "normalized": false,
2485
+ "rstrip": false,
2486
+ "single_word": false
2487
+ },
2488
+ {
2489
+ "content": "218784516525",
2490
+ "lstrip": false,
2491
+ "normalized": false,
2492
+ "rstrip": false,
2493
+ "single_word": false
2494
+ },
2495
+ {
2496
+ "content": "762486672996",
2497
+ "lstrip": false,
2498
+ "normalized": false,
2499
+ "rstrip": false,
2500
+ "single_word": false
2501
+ },
2502
+ {
2503
+ "content": "142799718159",
2504
+ "lstrip": false,
2505
+ "normalized": false,
2506
+ "rstrip": false,
2507
+ "single_word": false
2508
+ },
2509
+ {
2510
+ "content": "858146415154",
2511
+ "lstrip": false,
2512
+ "normalized": false,
2513
+ "rstrip": false,
2514
+ "single_word": false
2515
+ },
2516
+ {
2517
+ "content": "767858144912",
2518
+ "lstrip": false,
2519
+ "normalized": false,
2520
+ "rstrip": false,
2521
+ "single_word": false
2522
+ },
2523
+ {
2524
+ "content": "571317457151",
2525
+ "lstrip": false,
2526
+ "normalized": false,
2527
+ "rstrip": false,
2528
+ "single_word": false
2529
+ },
2530
+ {
2531
+ "content": "635127952696",
2532
+ "lstrip": false,
2533
+ "normalized": false,
2534
+ "rstrip": false,
2535
+ "single_word": false
2536
+ },
2537
+ {
2538
+ "content": "116427191984",
2539
+ "lstrip": false,
2540
+ "normalized": false,
2541
+ "rstrip": false,
2542
+ "single_word": false
2543
+ },
2544
+ {
2545
+ "content": "268921994538",
2546
+ "lstrip": false,
2547
+ "normalized": false,
2548
+ "rstrip": false,
2549
+ "single_word": false
2550
+ },
2551
+ {
2552
+ "content": "523937669294",
2553
+ "lstrip": false,
2554
+ "normalized": false,
2555
+ "rstrip": false,
2556
+ "single_word": false
2557
+ },
2558
+ {
2559
+ "content": "165429152138",
2560
+ "lstrip": false,
2561
+ "normalized": false,
2562
+ "rstrip": false,
2563
+ "single_word": false
2564
+ },
2565
+ {
2566
+ "content": "739246183345",
2567
+ "lstrip": false,
2568
+ "normalized": false,
2569
+ "rstrip": false,
2570
+ "single_word": false
2571
+ },
2572
+ {
2573
+ "content": "591464355756",
2574
+ "lstrip": false,
2575
+ "normalized": false,
2576
+ "rstrip": false,
2577
+ "single_word": false
2578
+ },
2579
+ {
2580
+ "content": "212985874612",
2581
+ "lstrip": false,
2582
+ "normalized": false,
2583
+ "rstrip": false,
2584
+ "single_word": false
2585
+ },
2586
+ {
2587
+ "content": "191887635211",
2588
+ "lstrip": false,
2589
+ "normalized": false,
2590
+ "rstrip": false,
2591
+ "single_word": false
2592
+ },
2593
+ {
2594
+ "content": "967214577653",
2595
+ "lstrip": false,
2596
+ "normalized": false,
2597
+ "rstrip": false,
2598
+ "single_word": false
2599
+ },
2600
+ {
2601
+ "content": "119342152414",
2602
+ "lstrip": false,
2603
+ "normalized": false,
2604
+ "rstrip": false,
2605
+ "single_word": false
2606
+ },
2607
+ {
2608
+ "content": "946444632795",
2609
+ "lstrip": false,
2610
+ "normalized": false,
2611
+ "rstrip": false,
2612
+ "single_word": false
2613
+ },
2614
+ {
2615
+ "content": "618423867817",
2616
+ "lstrip": false,
2617
+ "normalized": false,
2618
+ "rstrip": false,
2619
+ "single_word": false
2620
+ },
2621
+ {
2622
+ "content": "228565148417",
2623
+ "lstrip": false,
2624
+ "normalized": false,
2625
+ "rstrip": false,
2626
+ "single_word": false
2627
+ },
2628
+ {
2629
+ "content": "729116422489",
2630
+ "lstrip": false,
2631
+ "normalized": false,
2632
+ "rstrip": false,
2633
+ "single_word": false
2634
+ },
2635
+ {
2636
+ "content": "527874729936",
2637
+ "lstrip": false,
2638
+ "normalized": false,
2639
+ "rstrip": false,
2640
+ "single_word": false
2641
+ },
2642
+ {
2643
+ "content": "739784153482",
2644
+ "lstrip": false,
2645
+ "normalized": false,
2646
+ "rstrip": false,
2647
+ "single_word": false
2648
+ },
2649
+ {
2650
+ "content": "387763951128",
2651
+ "lstrip": false,
2652
+ "normalized": false,
2653
+ "rstrip": false,
2654
+ "single_word": false
2655
+ },
2656
+ {
2657
+ "content": "331369926711",
2658
+ "lstrip": false,
2659
+ "normalized": false,
2660
+ "rstrip": false,
2661
+ "single_word": false
2662
+ },
2663
+ {
2664
+ "content": "562716493614",
2665
+ "lstrip": false,
2666
+ "normalized": false,
2667
+ "rstrip": false,
2668
+ "single_word": false
2669
+ },
2670
+ {
2671
+ "content": "739667844957",
2672
+ "lstrip": false,
2673
+ "normalized": false,
2674
+ "rstrip": false,
2675
+ "single_word": false
2676
+ },
2677
+ {
2678
+ "content": "562389434565",
2679
+ "lstrip": false,
2680
+ "normalized": false,
2681
+ "rstrip": false,
2682
+ "single_word": false
2683
+ },
2684
+ {
2685
+ "content": "256497188281",
2686
+ "lstrip": false,
2687
+ "normalized": false,
2688
+ "rstrip": false,
2689
+ "single_word": false
2690
+ },
2691
+ {
2692
+ "content": "859927364588",
2693
+ "lstrip": false,
2694
+ "normalized": false,
2695
+ "rstrip": false,
2696
+ "single_word": false
2697
+ },
2698
+ {
2699
+ "content": "417668946583",
2700
+ "lstrip": false,
2701
+ "normalized": false,
2702
+ "rstrip": false,
2703
+ "single_word": false
2704
+ },
2705
+ {
2706
+ "content": "357621613582",
2707
+ "lstrip": false,
2708
+ "normalized": false,
2709
+ "rstrip": false,
2710
+ "single_word": false
2711
+ },
2712
+ {
2713
+ "content": "438435178228",
2714
+ "lstrip": false,
2715
+ "normalized": false,
2716
+ "rstrip": false,
2717
+ "single_word": false
2718
+ },
2719
+ {
2720
+ "content": "485692541169",
2721
+ "lstrip": false,
2722
+ "normalized": false,
2723
+ "rstrip": false,
2724
+ "single_word": false
2725
+ },
2726
+ {
2727
+ "content": "825815739116",
2728
+ "lstrip": false,
2729
+ "normalized": false,
2730
+ "rstrip": false,
2731
+ "single_word": false
2732
+ },
2733
+ {
2734
+ "content": "342221452223",
2735
+ "lstrip": false,
2736
+ "normalized": false,
2737
+ "rstrip": false,
2738
+ "single_word": false
2739
+ },
2740
+ {
2741
+ "content": "697747991249",
2742
+ "lstrip": false,
2743
+ "normalized": false,
2744
+ "rstrip": false,
2745
+ "single_word": false
2746
+ },
2747
+ {
2748
+ "content": "716763689965",
2749
+ "lstrip": false,
2750
+ "normalized": false,
2751
+ "rstrip": false,
2752
+ "single_word": false
2753
+ },
2754
+ {
2755
+ "content": "141499982867",
2756
+ "lstrip": false,
2757
+ "normalized": false,
2758
+ "rstrip": false,
2759
+ "single_word": false
2760
+ },
2761
+ {
2762
+ "content": "818479319499",
2763
+ "lstrip": false,
2764
+ "normalized": false,
2765
+ "rstrip": false,
2766
+ "single_word": false
2767
+ },
2768
+ {
2769
+ "content": "336813343298",
2770
+ "lstrip": false,
2771
+ "normalized": false,
2772
+ "rstrip": false,
2773
+ "single_word": false
2774
+ },
2775
+ {
2776
+ "content": "594688742928",
2777
+ "lstrip": false,
2778
+ "normalized": false,
2779
+ "rstrip": false,
2780
+ "single_word": false
2781
+ },
2782
+ {
2783
+ "content": "472129283475",
2784
+ "lstrip": false,
2785
+ "normalized": false,
2786
+ "rstrip": false,
2787
+ "single_word": false
2788
+ },
2789
+ {
2790
+ "content": "514354144759",
2791
+ "lstrip": false,
2792
+ "normalized": false,
2793
+ "rstrip": false,
2794
+ "single_word": false
2795
+ },
2796
+ {
2797
+ "content": "349249721685",
2798
+ "lstrip": false,
2799
+ "normalized": false,
2800
+ "rstrip": false,
2801
+ "single_word": false
2802
+ },
2803
+ {
2804
+ "content": "546276298359",
2805
+ "lstrip": false,
2806
+ "normalized": false,
2807
+ "rstrip": false,
2808
+ "single_word": false
2809
+ },
2810
+ {
2811
+ "content": "353755529131",
2812
+ "lstrip": false,
2813
+ "normalized": false,
2814
+ "rstrip": false,
2815
+ "single_word": false
2816
+ },
2817
+ {
2818
+ "content": "315534574435",
2819
+ "lstrip": false,
2820
+ "normalized": false,
2821
+ "rstrip": false,
2822
+ "single_word": false
2823
+ },
2824
+ {
2825
+ "content": "523723475786",
2826
+ "lstrip": false,
2827
+ "normalized": false,
2828
+ "rstrip": false,
2829
+ "single_word": false
2830
+ },
2831
+ {
2832
+ "content": "215826764872",
2833
+ "lstrip": false,
2834
+ "normalized": false,
2835
+ "rstrip": false,
2836
+ "single_word": false
2837
+ },
2838
+ {
2839
+ "content": "367968398551",
2840
+ "lstrip": false,
2841
+ "normalized": false,
2842
+ "rstrip": false,
2843
+ "single_word": false
2844
+ },
2845
+ {
2846
+ "content": "569853653352",
2847
+ "lstrip": false,
2848
+ "normalized": false,
2849
+ "rstrip": false,
2850
+ "single_word": false
2851
+ },
2852
+ {
2853
+ "content": "389715484387",
2854
+ "lstrip": false,
2855
+ "normalized": false,
2856
+ "rstrip": false,
2857
+ "single_word": false
2858
+ },
2859
+ {
2860
+ "content": "293847485454",
2861
+ "lstrip": false,
2862
+ "normalized": false,
2863
+ "rstrip": false,
2864
+ "single_word": false
2865
+ },
2866
+ {
2867
+ "content": "714738141818",
2868
+ "lstrip": false,
2869
+ "normalized": false,
2870
+ "rstrip": false,
2871
+ "single_word": false
2872
+ },
2873
+ {
2874
+ "content": "178478368922",
2875
+ "lstrip": false,
2876
+ "normalized": false,
2877
+ "rstrip": false,
2878
+ "single_word": false
2879
+ },
2880
+ {
2881
+ "content": "581493616981",
2882
+ "lstrip": false,
2883
+ "normalized": false,
2884
+ "rstrip": false,
2885
+ "single_word": false
2886
+ },
2887
+ {
2888
+ "content": "589439538674",
2889
+ "lstrip": false,
2890
+ "normalized": false,
2891
+ "rstrip": false,
2892
+ "single_word": false
2893
+ },
2894
+ {
2895
+ "content": "846657726193",
2896
+ "lstrip": false,
2897
+ "normalized": false,
2898
+ "rstrip": false,
2899
+ "single_word": false
2900
+ },
2901
+ {
2902
+ "content": "722339992679",
2903
+ "lstrip": false,
2904
+ "normalized": false,
2905
+ "rstrip": false,
2906
+ "single_word": false
2907
+ },
2908
+ {
2909
+ "content": "138154781148",
2910
+ "lstrip": false,
2911
+ "normalized": false,
2912
+ "rstrip": false,
2913
+ "single_word": false
2914
+ },
2915
+ {
2916
+ "content": "757785319772",
2917
+ "lstrip": false,
2918
+ "normalized": false,
2919
+ "rstrip": false,
2920
+ "single_word": false
2921
+ },
2922
+ {
2923
+ "content": "492516914298",
2924
+ "lstrip": false,
2925
+ "normalized": false,
2926
+ "rstrip": false,
2927
+ "single_word": false
2928
+ },
2929
+ {
2930
+ "content": "919181521716",
2931
+ "lstrip": false,
2932
+ "normalized": false,
2933
+ "rstrip": false,
2934
+ "single_word": false
2935
+ },
2936
+ {
2937
+ "content": "985781138935",
2938
+ "lstrip": false,
2939
+ "normalized": false,
2940
+ "rstrip": false,
2941
+ "single_word": false
2942
+ },
2943
+ {
2944
+ "content": "476969195485",
2945
+ "lstrip": false,
2946
+ "normalized": false,
2947
+ "rstrip": false,
2948
+ "single_word": false
2949
+ },
2950
+ {
2951
+ "content": "313145133463",
2952
+ "lstrip": false,
2953
+ "normalized": false,
2954
+ "rstrip": false,
2955
+ "single_word": false
2956
+ },
2957
+ {
2958
+ "content": "758963111966",
2959
+ "lstrip": false,
2960
+ "normalized": false,
2961
+ "rstrip": false,
2962
+ "single_word": false
2963
+ },
2964
+ {
2965
+ "content": "147541537162",
2966
+ "lstrip": false,
2967
+ "normalized": false,
2968
+ "rstrip": false,
2969
+ "single_word": false
2970
+ },
2971
+ {
2972
+ "content": "557163366873",
2973
+ "lstrip": false,
2974
+ "normalized": false,
2975
+ "rstrip": false,
2976
+ "single_word": false
2977
+ },
2978
+ {
2979
+ "content": "144373897488",
2980
+ "lstrip": false,
2981
+ "normalized": false,
2982
+ "rstrip": false,
2983
+ "single_word": false
2984
+ },
2985
+ {
2986
+ "content": "522515164754",
2987
+ "lstrip": false,
2988
+ "normalized": false,
2989
+ "rstrip": false,
2990
+ "single_word": false
2991
+ },
2992
+ {
2993
+ "content": "724964923582",
2994
+ "lstrip": false,
2995
+ "normalized": false,
2996
+ "rstrip": false,
2997
+ "single_word": false
2998
+ },
2999
+ {
3000
+ "content": "284776712475",
3001
+ "lstrip": false,
3002
+ "normalized": false,
3003
+ "rstrip": false,
3004
+ "single_word": false
3005
+ },
3006
+ {
3007
+ "content": "375429755114",
3008
+ "lstrip": false,
3009
+ "normalized": false,
3010
+ "rstrip": false,
3011
+ "single_word": false
3012
+ },
3013
+ {
3014
+ "content": "181233596124",
3015
+ "lstrip": false,
3016
+ "normalized": false,
3017
+ "rstrip": false,
3018
+ "single_word": false
3019
+ },
3020
+ {
3021
+ "content": "948585673431",
3022
+ "lstrip": false,
3023
+ "normalized": false,
3024
+ "rstrip": false,
3025
+ "single_word": false
3026
+ },
3027
+ {
3028
+ "content": "243165586174",
3029
+ "lstrip": false,
3030
+ "normalized": false,
3031
+ "rstrip": false,
3032
+ "single_word": false
3033
+ },
3034
+ {
3035
+ "content": "396847976144",
3036
+ "lstrip": false,
3037
+ "normalized": false,
3038
+ "rstrip": false,
3039
+ "single_word": false
3040
+ },
3041
+ {
3042
+ "content": "997724962668",
3043
+ "lstrip": false,
3044
+ "normalized": false,
3045
+ "rstrip": false,
3046
+ "single_word": false
3047
+ },
3048
+ {
3049
+ "content": "558837194455",
3050
+ "lstrip": false,
3051
+ "normalized": false,
3052
+ "rstrip": false,
3053
+ "single_word": false
3054
+ },
3055
+ {
3056
+ "content": "163165456396",
3057
+ "lstrip": false,
3058
+ "normalized": false,
3059
+ "rstrip": false,
3060
+ "single_word": false
3061
+ },
3062
+ {
3063
+ "content": "378749551722",
3064
+ "lstrip": false,
3065
+ "normalized": false,
3066
+ "rstrip": false,
3067
+ "single_word": false
3068
+ },
3069
+ {
3070
+ "content": "161238482259",
3071
+ "lstrip": false,
3072
+ "normalized": false,
3073
+ "rstrip": false,
3074
+ "single_word": false
3075
+ },
3076
+ {
3077
+ "content": "754978243758",
3078
+ "lstrip": false,
3079
+ "normalized": false,
3080
+ "rstrip": false,
3081
+ "single_word": false
3082
+ },
3083
+ {
3084
+ "content": "195388849133",
3085
+ "lstrip": false,
3086
+ "normalized": false,
3087
+ "rstrip": false,
3088
+ "single_word": false
3089
+ },
3090
+ {
3091
+ "content": "229775525672",
3092
+ "lstrip": false,
3093
+ "normalized": false,
3094
+ "rstrip": false,
3095
+ "single_word": false
3096
+ },
3097
+ {
3098
+ "content": "262437452884",
3099
+ "lstrip": false,
3100
+ "normalized": false,
3101
+ "rstrip": false,
3102
+ "single_word": false
3103
+ },
3104
+ {
3105
+ "content": "441377892146",
3106
+ "lstrip": false,
3107
+ "normalized": false,
3108
+ "rstrip": false,
3109
+ "single_word": false
3110
+ },
3111
+ {
3112
+ "content": "451885565366",
3113
+ "lstrip": false,
3114
+ "normalized": false,
3115
+ "rstrip": false,
3116
+ "single_word": false
3117
+ },
3118
+ {
3119
+ "content": "981277526855",
3120
+ "lstrip": false,
3121
+ "normalized": false,
3122
+ "rstrip": false,
3123
+ "single_word": false
3124
+ },
3125
+ {
3126
+ "content": "762495822823",
3127
+ "lstrip": false,
3128
+ "normalized": false,
3129
+ "rstrip": false,
3130
+ "single_word": false
3131
+ },
3132
+ {
3133
+ "content": "368763327262",
3134
+ "lstrip": false,
3135
+ "normalized": false,
3136
+ "rstrip": false,
3137
+ "single_word": false
3138
+ },
3139
+ {
3140
+ "content": "757422791351",
3141
+ "lstrip": false,
3142
+ "normalized": false,
3143
+ "rstrip": false,
3144
+ "single_word": false
3145
+ },
3146
+ {
3147
+ "content": "636324136426",
3148
+ "lstrip": false,
3149
+ "normalized": false,
3150
+ "rstrip": false,
3151
+ "single_word": false
3152
+ },
3153
+ {
3154
+ "content": "214193645583",
3155
+ "lstrip": false,
3156
+ "normalized": false,
3157
+ "rstrip": false,
3158
+ "single_word": false
3159
+ },
3160
+ {
3161
+ "content": "412843856172",
3162
+ "lstrip": false,
3163
+ "normalized": false,
3164
+ "rstrip": false,
3165
+ "single_word": false
3166
+ },
3167
+ {
3168
+ "content": "179386156569",
3169
+ "lstrip": false,
3170
+ "normalized": false,
3171
+ "rstrip": false,
3172
+ "single_word": false
3173
+ },
3174
+ {
3175
+ "content": "756916173536",
3176
+ "lstrip": false,
3177
+ "normalized": false,
3178
+ "rstrip": false,
3179
+ "single_word": false
3180
+ },
3181
+ {
3182
+ "content": "892697125149",
3183
+ "lstrip": false,
3184
+ "normalized": false,
3185
+ "rstrip": false,
3186
+ "single_word": false
3187
+ },
3188
+ {
3189
+ "content": "625334487352",
3190
+ "lstrip": false,
3191
+ "normalized": false,
3192
+ "rstrip": false,
3193
+ "single_word": false
3194
+ },
3195
+ {
3196
+ "content": "941861857715",
3197
+ "lstrip": false,
3198
+ "normalized": false,
3199
+ "rstrip": false,
3200
+ "single_word": false
3201
+ },
3202
+ {
3203
+ "content": "887417525236",
3204
+ "lstrip": false,
3205
+ "normalized": false,
3206
+ "rstrip": false,
3207
+ "single_word": false
3208
+ },
3209
+ {
3210
+ "content": "649516938598",
3211
+ "lstrip": false,
3212
+ "normalized": false,
3213
+ "rstrip": false,
3214
+ "single_word": false
3215
+ },
3216
+ {
3217
+ "content": "717628619782",
3218
+ "lstrip": false,
3219
+ "normalized": false,
3220
+ "rstrip": false,
3221
+ "single_word": false
3222
+ },
3223
+ {
3224
+ "content": "438124184139",
3225
+ "lstrip": false,
3226
+ "normalized": false,
3227
+ "rstrip": false,
3228
+ "single_word": false
3229
+ },
3230
+ {
3231
+ "content": "547563892268",
3232
+ "lstrip": false,
3233
+ "normalized": false,
3234
+ "rstrip": false,
3235
+ "single_word": false
3236
+ },
3237
+ {
3238
+ "content": "856317483891",
3239
+ "lstrip": false,
3240
+ "normalized": false,
3241
+ "rstrip": false,
3242
+ "single_word": false
3243
+ },
3244
+ {
3245
+ "content": "313313831273",
3246
+ "lstrip": false,
3247
+ "normalized": false,
3248
+ "rstrip": false,
3249
+ "single_word": false
3250
+ },
3251
+ {
3252
+ "content": "371496153876",
3253
+ "lstrip": false,
3254
+ "normalized": false,
3255
+ "rstrip": false,
3256
+ "single_word": false
3257
+ },
3258
+ {
3259
+ "content": "587541149322",
3260
+ "lstrip": false,
3261
+ "normalized": false,
3262
+ "rstrip": false,
3263
+ "single_word": false
3264
+ },
3265
+ {
3266
+ "content": "265847332563",
3267
+ "lstrip": false,
3268
+ "normalized": false,
3269
+ "rstrip": false,
3270
+ "single_word": false
3271
+ },
3272
+ {
3273
+ "content": "449549215429",
3274
+ "lstrip": false,
3275
+ "normalized": false,
3276
+ "rstrip": false,
3277
+ "single_word": false
3278
+ },
3279
+ {
3280
+ "content": "163497196769",
3281
+ "lstrip": false,
3282
+ "normalized": false,
3283
+ "rstrip": false,
3284
+ "single_word": false
3285
+ },
3286
+ {
3287
+ "content": "861342291298",
3288
+ "lstrip": false,
3289
+ "normalized": false,
3290
+ "rstrip": false,
3291
+ "single_word": false
3292
+ },
3293
+ {
3294
+ "content": "268433315926",
3295
+ "lstrip": false,
3296
+ "normalized": false,
3297
+ "rstrip": false,
3298
+ "single_word": false
3299
+ },
3300
+ {
3301
+ "content": "774679513717",
3302
+ "lstrip": false,
3303
+ "normalized": false,
3304
+ "rstrip": false,
3305
+ "single_word": false
3306
+ },
3307
+ {
3308
+ "content": "851254219729",
3309
+ "lstrip": false,
3310
+ "normalized": false,
3311
+ "rstrip": false,
3312
+ "single_word": false
3313
+ },
3314
+ {
3315
+ "content": "583527834464",
3316
+ "lstrip": false,
3317
+ "normalized": false,
3318
+ "rstrip": false,
3319
+ "single_word": false
3320
+ },
3321
+ {
3322
+ "content": "488496781997",
3323
+ "lstrip": false,
3324
+ "normalized": false,
3325
+ "rstrip": false,
3326
+ "single_word": false
3327
+ },
3328
+ {
3329
+ "content": "556814553861",
3330
+ "lstrip": false,
3331
+ "normalized": false,
3332
+ "rstrip": false,
3333
+ "single_word": false
3334
+ },
3335
+ {
3336
+ "content": "482829231639",
3337
+ "lstrip": false,
3338
+ "normalized": false,
3339
+ "rstrip": false,
3340
+ "single_word": false
3341
+ },
3342
+ {
3343
+ "content": "618878266619",
3344
+ "lstrip": false,
3345
+ "normalized": false,
3346
+ "rstrip": false,
3347
+ "single_word": false
3348
+ },
3349
+ {
3350
+ "content": "147444452794",
3351
+ "lstrip": false,
3352
+ "normalized": false,
3353
+ "rstrip": false,
3354
+ "single_word": false
3355
+ },
3356
+ {
3357
+ "content": "949235426629",
3358
+ "lstrip": false,
3359
+ "normalized": false,
3360
+ "rstrip": false,
3361
+ "single_word": false
3362
+ },
3363
+ {
3364
+ "content": "357299947518",
3365
+ "lstrip": false,
3366
+ "normalized": false,
3367
+ "rstrip": false,
3368
+ "single_word": false
3369
+ },
3370
+ {
3371
+ "content": "175528632226",
3372
+ "lstrip": false,
3373
+ "normalized": false,
3374
+ "rstrip": false,
3375
+ "single_word": false
3376
+ },
3377
+ {
3378
+ "content": "645527857972",
3379
+ "lstrip": false,
3380
+ "normalized": false,
3381
+ "rstrip": false,
3382
+ "single_word": false
3383
+ },
3384
+ {
3385
+ "content": "186872457894",
3386
+ "lstrip": false,
3387
+ "normalized": false,
3388
+ "rstrip": false,
3389
+ "single_word": false
3390
+ },
3391
+ {
3392
+ "content": "552738847828",
3393
+ "lstrip": false,
3394
+ "normalized": false,
3395
+ "rstrip": false,
3396
+ "single_word": false
3397
+ },
3398
+ {
3399
+ "content": "626748382482",
3400
+ "lstrip": false,
3401
+ "normalized": false,
3402
+ "rstrip": false,
3403
+ "single_word": false
3404
+ },
3405
+ {
3406
+ "content": "921894985642",
3407
+ "lstrip": false,
3408
+ "normalized": false,
3409
+ "rstrip": false,
3410
+ "single_word": false
3411
+ },
3412
+ {
3413
+ "content": "943878645871",
3414
+ "lstrip": false,
3415
+ "normalized": false,
3416
+ "rstrip": false,
3417
+ "single_word": false
3418
+ },
3419
+ {
3420
+ "content": "859289776479",
3421
+ "lstrip": false,
3422
+ "normalized": false,
3423
+ "rstrip": false,
3424
+ "single_word": false
3425
+ },
3426
+ {
3427
+ "content": "614583493135",
3428
+ "lstrip": false,
3429
+ "normalized": false,
3430
+ "rstrip": false,
3431
+ "single_word": false
3432
+ },
3433
+ {
3434
+ "content": "933775286797",
3435
+ "lstrip": false,
3436
+ "normalized": false,
3437
+ "rstrip": false,
3438
+ "single_word": false
3439
+ },
3440
+ {
3441
+ "content": "332234613346",
3442
+ "lstrip": false,
3443
+ "normalized": false,
3444
+ "rstrip": false,
3445
+ "single_word": false
3446
+ },
3447
+ {
3448
+ "content": "325196781219",
3449
+ "lstrip": false,
3450
+ "normalized": false,
3451
+ "rstrip": false,
3452
+ "single_word": false
3453
+ },
3454
+ {
3455
+ "content": "142526557681",
3456
+ "lstrip": false,
3457
+ "normalized": false,
3458
+ "rstrip": false,
3459
+ "single_word": false
3460
+ },
3461
+ {
3462
+ "content": "356722692178",
3463
+ "lstrip": false,
3464
+ "normalized": false,
3465
+ "rstrip": false,
3466
+ "single_word": false
3467
+ },
3468
+ {
3469
+ "content": "449318681694",
3470
+ "lstrip": false,
3471
+ "normalized": false,
3472
+ "rstrip": false,
3473
+ "single_word": false
3474
+ },
3475
+ {
3476
+ "content": "687284547244",
3477
+ "lstrip": false,
3478
+ "normalized": false,
3479
+ "rstrip": false,
3480
+ "single_word": false
3481
+ },
3482
+ {
3483
+ "content": "947262995132",
3484
+ "lstrip": false,
3485
+ "normalized": false,
3486
+ "rstrip": false,
3487
+ "single_word": false
3488
+ },
3489
+ {
3490
+ "content": "893974619684",
3491
+ "lstrip": false,
3492
+ "normalized": false,
3493
+ "rstrip": false,
3494
+ "single_word": false
3495
+ },
3496
+ {
3497
+ "content": "797238311233",
3498
+ "lstrip": false,
3499
+ "normalized": false,
3500
+ "rstrip": false,
3501
+ "single_word": false
3502
+ }
3503
+ ],
3504
+ "eos_token": {
3505
+ "content": "<|endoftext|>",
3506
+ "lstrip": false,
3507
+ "normalized": false,
3508
+ "rstrip": false,
3509
+ "single_word": false
3510
+ },
3511
+ "pad_token": {
3512
+ "content": "<|endoftext|>",
3513
+ "lstrip": false,
3514
+ "normalized": false,
3515
+ "rstrip": false,
3516
+ "single_word": false
3517
+ }
3518
+ }
qwen1.5-moe-a2.7b/dp3_attn_v2/tokenizer/tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff
 
qwen1.5-moe-a2.7b/dp3_attn_v2/tokenizer/vocab.json ADDED
The diff for this file is too large to render. See raw diff
 
qwen1.5-moe-a2.7b/dp3_attn_v2/train.log ADDED
@@ -0,0 +1,277 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 2026-04-03 14:02:02,082 [INFO] new_opacus_codex.train_steps: epoch=1 step=5 loss=23.7877
2
+ 2026-04-03 14:03:03,915 [INFO] new_opacus_codex.train_steps: epoch=1 step=10 loss=23.7445
3
+ 2026-04-03 14:03:19,010 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=10 eval_loss=24.6196 duration_sec=15.08
4
+ 2026-04-03 14:04:20,560 [INFO] new_opacus_codex.train_steps: epoch=1 step=15 loss=23.7173
5
+ 2026-04-03 14:05:24,869 [INFO] new_opacus_codex.train_steps: epoch=1 step=20 loss=23.6299
6
+ 2026-04-03 14:05:39,059 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=20 eval_loss=24.4946 duration_sec=14.16
7
+ 2026-04-03 14:06:39,342 [INFO] new_opacus_codex.train_steps: epoch=1 step=25 loss=23.6806
8
+ 2026-04-03 14:07:38,882 [INFO] new_opacus_codex.train_steps: epoch=1 step=30 loss=23.6360
9
+ 2026-04-03 14:07:53,041 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=30 eval_loss=24.2726 duration_sec=14.15
10
+ 2026-04-03 14:08:54,772 [INFO] new_opacus_codex.train_steps: epoch=1 step=35 loss=23.3660
11
+ 2026-04-03 14:09:54,030 [INFO] new_opacus_codex.train_steps: epoch=1 step=40 loss=23.3956
12
+ 2026-04-03 14:10:08,284 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=40 eval_loss=23.9687 duration_sec=14.24
13
+ 2026-04-03 14:11:11,508 [INFO] new_opacus_codex.train_steps: epoch=1 step=45 loss=23.2912
14
+ 2026-04-03 14:12:11,759 [INFO] new_opacus_codex.train_steps: epoch=1 step=50 loss=23.0098
15
+ 2026-04-03 14:12:26,021 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=50 eval_loss=23.5664 duration_sec=14.25
16
+ 2026-04-03 14:13:24,710 [INFO] new_opacus_codex.train_steps: epoch=1 step=55 loss=22.8784
17
+ 2026-04-03 14:14:25,531 [INFO] new_opacus_codex.train_steps: epoch=1 step=60 loss=22.6599
18
+ 2026-04-03 14:14:40,004 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=60 eval_loss=23.0661 duration_sec=14.47
19
+ 2026-04-03 14:15:40,475 [INFO] new_opacus_codex.train_steps: epoch=1 step=65 loss=22.5012
20
+ 2026-04-03 14:16:40,277 [INFO] new_opacus_codex.train_steps: epoch=1 step=70 loss=22.4142
21
+ 2026-04-03 14:16:54,518 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=70 eval_loss=22.4899 duration_sec=14.23
22
+ 2026-04-03 14:17:56,223 [INFO] new_opacus_codex.train_steps: epoch=1 step=75 loss=22.1492
23
+ 2026-04-03 14:18:58,074 [INFO] new_opacus_codex.train_steps: epoch=1 step=80 loss=21.6600
24
+ 2026-04-03 14:19:13,131 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=80 eval_loss=21.8045 duration_sec=15.03
25
+ 2026-04-03 14:20:16,197 [INFO] new_opacus_codex.train_steps: epoch=1 step=85 loss=21.2726
26
+ 2026-04-03 14:21:19,383 [INFO] new_opacus_codex.train_steps: epoch=1 step=90 loss=20.9399
27
+ 2026-04-03 14:21:34,351 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=90 eval_loss=20.9903 duration_sec=14.95
28
+ 2026-04-03 14:22:34,745 [INFO] new_opacus_codex.train_steps: epoch=1 step=95 loss=20.7721
29
+ 2026-04-03 14:23:35,697 [INFO] new_opacus_codex.train_steps: epoch=1 step=100 loss=20.4722
30
+ 2026-04-03 14:23:51,899 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=100 eval_loss=20.0227 duration_sec=16.19
31
+ 2026-04-03 14:24:56,144 [INFO] new_opacus_codex.train_steps: epoch=1 step=105 loss=19.9227
32
+ 2026-04-03 14:25:55,470 [INFO] new_opacus_codex.train_steps: epoch=1 step=110 loss=19.5450
33
+ 2026-04-03 14:26:09,561 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=110 eval_loss=18.9271 duration_sec=14.08
34
+ 2026-04-03 14:27:12,152 [INFO] new_opacus_codex.train_steps: epoch=1 step=115 loss=18.9331
35
+ 2026-04-03 14:28:14,886 [INFO] new_opacus_codex.train_steps: epoch=1 step=120 loss=18.4778
36
+ 2026-04-03 14:28:29,051 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=120 eval_loss=17.7605 duration_sec=14.14
37
+ 2026-04-03 14:29:29,842 [INFO] new_opacus_codex.train_steps: epoch=1 step=125 loss=18.1106
38
+ 2026-04-03 14:30:30,820 [INFO] new_opacus_codex.train_steps: epoch=1 step=130 loss=17.2874
39
+ 2026-04-03 14:30:45,691 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=130 eval_loss=16.6687 duration_sec=14.80
40
+ 2026-04-03 14:31:46,887 [INFO] new_opacus_codex.train_steps: epoch=1 step=135 loss=16.8030
41
+ 2026-04-03 14:32:49,326 [INFO] new_opacus_codex.train_steps: epoch=1 step=140 loss=16.7360
42
+ 2026-04-03 14:33:03,818 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=140 eval_loss=15.9265 duration_sec=14.44
43
+ 2026-04-03 14:34:03,640 [INFO] new_opacus_codex.train_steps: epoch=1 step=145 loss=16.2815
44
+ 2026-04-03 14:35:05,587 [INFO] new_opacus_codex.train_steps: epoch=1 step=150 loss=15.9172
45
+ 2026-04-03 14:35:19,894 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=150 eval_loss=15.3882 duration_sec=14.29
46
+ 2026-04-03 14:36:24,923 [INFO] new_opacus_codex.train_steps: epoch=1 step=155 loss=15.4657
47
+ 2026-04-03 14:37:25,277 [INFO] new_opacus_codex.train_steps: epoch=1 step=160 loss=15.2851
48
+ 2026-04-03 14:37:39,491 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=160 eval_loss=14.9550 duration_sec=14.20
49
+ 2026-04-03 14:38:41,156 [INFO] new_opacus_codex.train_steps: epoch=1 step=165 loss=15.5306
50
+ 2026-04-03 14:39:40,772 [INFO] new_opacus_codex.train_steps: epoch=1 step=170 loss=15.2448
51
+ 2026-04-03 14:39:55,064 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=170 eval_loss=14.5620 duration_sec=14.24
52
+ 2026-04-03 14:40:57,626 [INFO] new_opacus_codex.train_steps: epoch=1 step=175 loss=14.8317
53
+ 2026-04-03 14:42:01,754 [INFO] new_opacus_codex.train_steps: epoch=1 step=180 loss=14.9049
54
+ 2026-04-03 14:42:16,169 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=180 eval_loss=14.2354 duration_sec=14.40
55
+ 2026-04-03 14:43:15,409 [INFO] new_opacus_codex.train_steps: epoch=1 step=185 loss=14.8066
56
+ 2026-04-03 14:44:49,654 [INFO] new_opacus_codex.train_steps: epoch=2 step=190 loss=14.9523
57
+ 2026-04-03 14:45:04,978 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=190 eval_loss=13.9318 duration_sec=15.31
58
+ 2026-04-03 14:46:06,782 [INFO] new_opacus_codex.train_steps: epoch=2 step=195 loss=14.8408
59
+ 2026-04-03 14:47:07,588 [INFO] new_opacus_codex.train_steps: epoch=2 step=200 loss=14.3849
60
+ 2026-04-03 14:47:21,999 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=200 eval_loss=13.6482 duration_sec=14.40
61
+ 2026-04-03 14:48:21,788 [INFO] new_opacus_codex.train_steps: epoch=2 step=205 loss=14.0710
62
+ 2026-04-03 14:49:22,338 [INFO] new_opacus_codex.train_steps: epoch=2 step=210 loss=13.9216
63
+ 2026-04-03 14:49:38,924 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=210 eval_loss=13.3778 duration_sec=16.57
64
+ 2026-04-03 14:50:43,145 [INFO] new_opacus_codex.train_steps: epoch=2 step=215 loss=13.8478
65
+ 2026-04-03 14:51:44,314 [INFO] new_opacus_codex.train_steps: epoch=2 step=220 loss=13.8986
66
+ 2026-04-03 14:51:58,416 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=220 eval_loss=13.1201 duration_sec=14.10
67
+ 2026-04-03 14:53:00,289 [INFO] new_opacus_codex.train_steps: epoch=2 step=225 loss=13.6179
68
+ 2026-04-03 14:54:00,681 [INFO] new_opacus_codex.train_steps: epoch=2 step=230 loss=13.5673
69
+ 2026-04-03 14:54:14,943 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=230 eval_loss=12.8806 duration_sec=14.26
70
+ 2026-04-03 14:55:14,255 [INFO] new_opacus_codex.train_steps: epoch=2 step=235 loss=13.7206
71
+ 2026-04-03 14:56:17,165 [INFO] new_opacus_codex.train_steps: epoch=2 step=240 loss=13.5915
72
+ 2026-04-03 14:56:31,514 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=240 eval_loss=12.6507 duration_sec=14.33
73
+ 2026-04-03 14:57:34,767 [INFO] new_opacus_codex.train_steps: epoch=2 step=245 loss=13.2063
74
+ 2026-04-03 14:58:34,633 [INFO] new_opacus_codex.train_steps: epoch=2 step=250 loss=13.1035
75
+ 2026-04-03 14:58:48,711 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=250 eval_loss=12.4195 duration_sec=14.05
76
+ 2026-04-03 14:59:51,158 [INFO] new_opacus_codex.train_steps: epoch=2 step=255 loss=13.0392
77
+ 2026-04-03 15:00:49,986 [INFO] new_opacus_codex.train_steps: epoch=2 step=260 loss=12.7483
78
+ 2026-04-03 15:01:04,324 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=260 eval_loss=12.1442 duration_sec=14.32
79
+ 2026-04-03 15:02:05,132 [INFO] new_opacus_codex.train_steps: epoch=2 step=265 loss=12.7522
80
+ 2026-04-03 15:03:03,764 [INFO] new_opacus_codex.train_steps: epoch=2 step=270 loss=12.7281
81
+ 2026-04-03 15:03:18,409 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=270 eval_loss=11.8001 duration_sec=14.63
82
+ 2026-04-03 15:04:18,230 [INFO] new_opacus_codex.train_steps: epoch=2 step=275 loss=12.5067
83
+ 2026-04-03 15:05:20,152 [INFO] new_opacus_codex.train_steps: epoch=2 step=280 loss=12.2792
84
+ 2026-04-03 15:05:34,713 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=280 eval_loss=11.3622 duration_sec=14.55
85
+ 2026-04-03 15:06:37,142 [INFO] new_opacus_codex.train_steps: epoch=2 step=285 loss=12.0755
86
+ 2026-04-03 15:07:39,290 [INFO] new_opacus_codex.train_steps: epoch=2 step=290 loss=11.8375
87
+ 2026-04-03 15:07:54,592 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=290 eval_loss=10.8559 duration_sec=15.29
88
+ 2026-04-03 15:08:56,175 [INFO] new_opacus_codex.train_steps: epoch=2 step=295 loss=11.6180
89
+ 2026-04-03 15:09:57,275 [INFO] new_opacus_codex.train_steps: epoch=2 step=300 loss=11.4692
90
+ 2026-04-03 15:10:11,484 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=300 eval_loss=10.3551 duration_sec=14.20
91
+ 2026-04-03 15:11:13,242 [INFO] new_opacus_codex.train_steps: epoch=2 step=305 loss=11.2479
92
+ 2026-04-03 15:12:15,586 [INFO] new_opacus_codex.train_steps: epoch=2 step=310 loss=11.0590
93
+ 2026-04-03 15:12:30,706 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=310 eval_loss=9.9333 duration_sec=15.11
94
+ 2026-04-03 15:13:34,313 [INFO] new_opacus_codex.train_steps: epoch=2 step=315 loss=10.8665
95
+ 2026-04-03 15:14:37,277 [INFO] new_opacus_codex.train_steps: epoch=2 step=320 loss=10.7904
96
+ 2026-04-03 15:15:01,359 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=320 eval_loss=9.6357 duration_sec=14.06
97
+ 2026-04-03 15:16:07,375 [INFO] new_opacus_codex.train_steps: epoch=2 step=325 loss=10.5493
98
+ 2026-04-03 15:17:08,865 [INFO] new_opacus_codex.train_steps: epoch=2 step=330 loss=10.4088
99
+ 2026-04-03 15:17:23,041 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=330 eval_loss=9.4153 duration_sec=14.16
100
+ 2026-04-03 15:18:23,652 [INFO] new_opacus_codex.train_steps: epoch=2 step=335 loss=10.3043
101
+ 2026-04-03 15:19:25,487 [INFO] new_opacus_codex.train_steps: epoch=2 step=340 loss=9.9412
102
+ 2026-04-03 15:19:40,045 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=340 eval_loss=9.2476 duration_sec=14.55
103
+ 2026-04-03 15:20:40,665 [INFO] new_opacus_codex.train_steps: epoch=2 step=345 loss=9.7279
104
+ 2026-04-03 15:21:40,788 [INFO] new_opacus_codex.train_steps: epoch=2 step=350 loss=9.8592
105
+ 2026-04-03 15:21:55,354 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=350 eval_loss=9.1174 duration_sec=14.53
106
+ 2026-04-03 15:22:54,824 [INFO] new_opacus_codex.train_steps: epoch=2 step=355 loss=10.1668
107
+ 2026-04-03 15:23:54,446 [INFO] new_opacus_codex.train_steps: epoch=2 step=360 loss=10.1283
108
+ 2026-04-03 15:24:09,442 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=360 eval_loss=9.0143 duration_sec=14.98
109
+ 2026-04-03 15:25:10,143 [INFO] new_opacus_codex.train_steps: epoch=2 step=365 loss=9.8453
110
+ 2026-04-03 15:26:10,391 [INFO] new_opacus_codex.train_steps: epoch=2 step=370 loss=9.6353
111
+ 2026-04-03 15:26:25,774 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=370 eval_loss=8.9251 duration_sec=15.36
112
+ 2026-04-03 15:27:40,097 [INFO] new_opacus_codex.train_steps: epoch=3 step=375 loss=9.3909
113
+ 2026-04-03 15:28:43,285 [INFO] new_opacus_codex.train_steps: epoch=3 step=380 loss=9.4321
114
+ 2026-04-03 15:28:57,810 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=380 eval_loss=8.8502 duration_sec=14.52
115
+ 2026-04-03 15:29:58,771 [INFO] new_opacus_codex.train_steps: epoch=3 step=385 loss=9.6398
116
+ 2026-04-03 15:31:00,647 [INFO] new_opacus_codex.train_steps: epoch=3 step=390 loss=9.7059
117
+ 2026-04-03 15:31:15,190 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=390 eval_loss=8.7853 duration_sec=14.54
118
+ 2026-04-03 15:32:17,324 [INFO] new_opacus_codex.train_steps: epoch=3 step=395 loss=9.5294
119
+ 2026-04-03 15:33:18,550 [INFO] new_opacus_codex.train_steps: epoch=3 step=400 loss=9.4144
120
+ 2026-04-03 15:33:33,194 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=400 eval_loss=8.7272 duration_sec=14.62
121
+ 2026-04-03 15:34:34,048 [INFO] new_opacus_codex.train_steps: epoch=3 step=405 loss=9.3189
122
+ 2026-04-03 15:35:35,421 [INFO] new_opacus_codex.train_steps: epoch=3 step=410 loss=9.4077
123
+ 2026-04-03 15:35:50,031 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=410 eval_loss=8.6726 duration_sec=14.60
124
+ 2026-04-03 15:36:52,312 [INFO] new_opacus_codex.train_steps: epoch=3 step=415 loss=9.3140
125
+ 2026-04-03 15:37:56,391 [INFO] new_opacus_codex.train_steps: epoch=3 step=420 loss=9.2872
126
+ 2026-04-03 15:38:11,000 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=420 eval_loss=8.6253 duration_sec=14.60
127
+ 2026-04-03 15:39:12,910 [INFO] new_opacus_codex.train_steps: epoch=3 step=425 loss=9.2748
128
+ 2026-04-03 15:40:12,688 [INFO] new_opacus_codex.train_steps: epoch=3 step=430 loss=9.2000
129
+ 2026-04-03 15:40:32,254 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=430 eval_loss=8.5831 duration_sec=19.55
130
+ 2026-04-03 15:41:34,179 [INFO] new_opacus_codex.train_steps: epoch=3 step=435 loss=9.1289
131
+ 2026-04-03 15:42:31,674 [INFO] new_opacus_codex.train_steps: epoch=3 step=440 loss=8.8791
132
+ 2026-04-03 15:42:46,007 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=440 eval_loss=8.5431 duration_sec=14.32
133
+ 2026-04-03 15:43:46,535 [INFO] new_opacus_codex.train_steps: epoch=3 step=445 loss=8.9777
134
+ 2026-04-03 15:44:45,848 [INFO] new_opacus_codex.train_steps: epoch=3 step=450 loss=9.2497
135
+ 2026-04-03 15:45:01,134 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=450 eval_loss=8.5038 duration_sec=15.27
136
+ 2026-04-03 15:46:00,519 [INFO] new_opacus_codex.train_steps: epoch=3 step=455 loss=9.4404
137
+ 2026-04-03 15:46:59,986 [INFO] new_opacus_codex.train_steps: epoch=3 step=460 loss=9.2961
138
+ 2026-04-03 15:47:14,554 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=460 eval_loss=8.4658 duration_sec=14.56
139
+ 2026-04-03 15:48:14,151 [INFO] new_opacus_codex.train_steps: epoch=3 step=465 loss=9.1452
140
+ 2026-04-03 15:49:14,760 [INFO] new_opacus_codex.train_steps: epoch=3 step=470 loss=9.2108
141
+ 2026-04-03 15:49:29,215 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=470 eval_loss=8.4346 duration_sec=14.44
142
+ 2026-04-03 15:50:28,428 [INFO] new_opacus_codex.train_steps: epoch=3 step=475 loss=9.2602
143
+ 2026-04-03 15:51:31,594 [INFO] new_opacus_codex.train_steps: epoch=3 step=480 loss=9.2387
144
+ 2026-04-03 15:51:46,352 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=480 eval_loss=8.4096 duration_sec=14.74
145
+ 2026-04-03 15:52:49,524 [INFO] new_opacus_codex.train_steps: epoch=3 step=485 loss=9.0095
146
+ 2026-04-03 15:53:53,781 [INFO] new_opacus_codex.train_steps: epoch=3 step=490 loss=9.0598
147
+ 2026-04-03 15:54:08,995 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=490 eval_loss=8.3838 duration_sec=15.19
148
+ 2026-04-03 15:55:11,820 [INFO] new_opacus_codex.train_steps: epoch=3 step=495 loss=9.2775
149
+ 2026-04-03 15:56:14,405 [INFO] new_opacus_codex.train_steps: epoch=3 step=500 loss=9.2000
150
+ 2026-04-03 15:56:29,121 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=500 eval_loss=8.3614 duration_sec=14.70
151
+ 2026-04-03 15:57:30,402 [INFO] new_opacus_codex.train_steps: epoch=3 step=505 loss=9.0640
152
+ 2026-04-03 15:58:30,156 [INFO] new_opacus_codex.train_steps: epoch=3 step=510 loss=9.0758
153
+ 2026-04-03 15:58:45,761 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=510 eval_loss=8.3366 duration_sec=15.60
154
+ 2026-04-03 15:59:46,674 [INFO] new_opacus_codex.train_steps: epoch=3 step=515 loss=8.9463
155
+ 2026-04-03 16:00:47,333 [INFO] new_opacus_codex.train_steps: epoch=3 step=520 loss=8.9290
156
+ 2026-04-03 16:01:01,793 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=520 eval_loss=8.3150 duration_sec=14.44
157
+ 2026-04-03 16:02:05,473 [INFO] new_opacus_codex.train_steps: epoch=3 step=525 loss=8.9199
158
+ 2026-04-03 16:03:10,240 [INFO] new_opacus_codex.train_steps: epoch=3 step=530 loss=8.7930
159
+ 2026-04-03 16:03:24,730 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=530 eval_loss=8.2965 duration_sec=14.46
160
+ 2026-04-03 16:04:24,954 [INFO] new_opacus_codex.train_steps: epoch=3 step=535 loss=9.1213
161
+ 2026-04-03 16:05:28,478 [INFO] new_opacus_codex.train_steps: epoch=3 step=540 loss=9.0639
162
+ 2026-04-03 16:05:43,121 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=540 eval_loss=8.2795 duration_sec=14.63
163
+ 2026-04-03 16:06:44,815 [INFO] new_opacus_codex.train_steps: epoch=3 step=545 loss=8.8914
164
+ 2026-04-03 16:07:48,303 [INFO] new_opacus_codex.train_steps: epoch=3 step=550 loss=8.8973
165
+ 2026-04-03 16:08:02,692 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=550 eval_loss=8.2623 duration_sec=14.38
166
+ 2026-04-03 16:09:02,796 [INFO] new_opacus_codex.train_steps: epoch=3 step=555 loss=8.7616
167
+ 2026-04-03 16:10:33,808 [INFO] new_opacus_codex.train_steps: epoch=4 step=560 loss=8.7028
168
+ 2026-04-03 16:10:50,574 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=560 eval_loss=8.2451 duration_sec=16.74
169
+ 2026-04-03 16:11:54,164 [INFO] new_opacus_codex.train_steps: epoch=4 step=565 loss=8.7656
170
+ 2026-04-03 16:12:57,788 [INFO] new_opacus_codex.train_steps: epoch=4 step=570 loss=8.7851
171
+ 2026-04-03 16:13:12,286 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=570 eval_loss=8.2282 duration_sec=14.48
172
+ 2026-04-03 16:14:13,480 [INFO] new_opacus_codex.train_steps: epoch=4 step=575 loss=8.7540
173
+ 2026-04-03 16:15:15,521 [INFO] new_opacus_codex.train_steps: epoch=4 step=580 loss=8.7355
174
+ 2026-04-03 16:15:30,531 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=580 eval_loss=8.2126 duration_sec=15.00
175
+ 2026-04-03 16:16:33,727 [INFO] new_opacus_codex.train_steps: epoch=4 step=585 loss=8.8621
176
+ 2026-04-03 16:17:35,268 [INFO] new_opacus_codex.train_steps: epoch=4 step=590 loss=8.9885
177
+ 2026-04-03 16:17:54,021 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=590 eval_loss=8.1981 duration_sec=18.73
178
+ 2026-04-03 16:18:58,316 [INFO] new_opacus_codex.train_steps: epoch=4 step=595 loss=8.8458
179
+ 2026-04-03 16:20:05,444 [INFO] new_opacus_codex.train_steps: epoch=4 step=600 loss=8.5159
180
+ 2026-04-03 16:20:22,664 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=600 eval_loss=8.1840 duration_sec=17.17
181
+ 2026-04-03 16:21:22,956 [INFO] new_opacus_codex.train_steps: epoch=4 step=605 loss=8.6842
182
+ 2026-04-03 16:22:24,116 [INFO] new_opacus_codex.train_steps: epoch=4 step=610 loss=8.8652
183
+ 2026-04-03 16:22:39,489 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=610 eval_loss=8.1706 duration_sec=15.36
184
+ 2026-04-03 16:23:42,885 [INFO] new_opacus_codex.train_steps: epoch=4 step=615 loss=8.7991
185
+ 2026-04-03 16:24:43,331 [INFO] new_opacus_codex.train_steps: epoch=4 step=620 loss=8.8568
186
+ 2026-04-03 16:24:57,934 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=620 eval_loss=8.1577 duration_sec=14.59
187
+ 2026-04-03 16:26:00,725 [INFO] new_opacus_codex.train_steps: epoch=4 step=625 loss=8.8398
188
+ 2026-04-03 16:27:04,203 [INFO] new_opacus_codex.train_steps: epoch=4 step=630 loss=8.9784
189
+ 2026-04-03 16:27:18,906 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=630 eval_loss=8.1464 duration_sec=14.69
190
+ 2026-04-03 16:28:24,179 [INFO] new_opacus_codex.train_steps: epoch=4 step=635 loss=8.8399
191
+ 2026-04-03 16:29:25,977 [INFO] new_opacus_codex.train_steps: epoch=4 step=640 loss=8.7117
192
+ 2026-04-03 16:29:40,816 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=640 eval_loss=8.1368 duration_sec=14.83
193
+ 2026-04-03 16:30:41,715 [INFO] new_opacus_codex.train_steps: epoch=4 step=645 loss=8.7472
194
+ 2026-04-03 16:31:45,707 [INFO] new_opacus_codex.train_steps: epoch=4 step=650 loss=8.6761
195
+ 2026-04-03 16:31:59,983 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=650 eval_loss=8.1264 duration_sec=14.25
196
+ 2026-04-03 16:33:01,092 [INFO] new_opacus_codex.train_steps: epoch=4 step=655 loss=8.7361
197
+ 2026-04-03 16:34:03,688 [INFO] new_opacus_codex.train_steps: epoch=4 step=660 loss=8.6457
198
+ 2026-04-03 16:34:18,567 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=660 eval_loss=8.1159 duration_sec=14.85
199
+ 2026-04-03 16:35:19,322 [INFO] new_opacus_codex.train_steps: epoch=4 step=665 loss=8.6954
200
+ 2026-04-03 16:36:17,526 [INFO] new_opacus_codex.train_steps: epoch=4 step=670 loss=8.7932
201
+ 2026-04-03 16:36:32,411 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=670 eval_loss=8.1073 duration_sec=14.87
202
+ 2026-04-03 16:37:33,674 [INFO] new_opacus_codex.train_steps: epoch=4 step=675 loss=8.6628
203
+ 2026-04-03 16:38:35,252 [INFO] new_opacus_codex.train_steps: epoch=4 step=680 loss=8.5662
204
+ 2026-04-03 16:38:49,665 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=680 eval_loss=8.0986 duration_sec=14.40
205
+ 2026-04-03 16:39:48,763 [INFO] new_opacus_codex.train_steps: epoch=4 step=685 loss=8.4892
206
+ 2026-04-03 16:40:50,002 [INFO] new_opacus_codex.train_steps: epoch=4 step=690 loss=8.5308
207
+ 2026-04-03 16:41:04,652 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=690 eval_loss=8.0903 duration_sec=14.64
208
+ 2026-04-03 16:42:06,274 [INFO] new_opacus_codex.train_steps: epoch=4 step=695 loss=8.7589
209
+ 2026-04-03 16:43:05,800 [INFO] new_opacus_codex.train_steps: epoch=4 step=700 loss=8.7602
210
+ 2026-04-03 16:43:20,162 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=700 eval_loss=8.0816 duration_sec=14.36
211
+ 2026-04-03 16:44:19,737 [INFO] new_opacus_codex.train_steps: epoch=4 step=705 loss=8.6064
212
+ 2026-04-03 16:45:19,232 [INFO] new_opacus_codex.train_steps: epoch=4 step=710 loss=8.7414
213
+ 2026-04-03 16:45:33,686 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=710 eval_loss=8.0748 duration_sec=14.44
214
+ 2026-04-03 16:46:33,897 [INFO] new_opacus_codex.train_steps: epoch=4 step=715 loss=8.6797
215
+ 2026-04-03 16:47:36,960 [INFO] new_opacus_codex.train_steps: epoch=4 step=720 loss=8.6761
216
+ 2026-04-03 16:47:51,245 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=720 eval_loss=8.0676 duration_sec=14.27
217
+ 2026-04-03 16:48:55,316 [INFO] new_opacus_codex.train_steps: epoch=4 step=725 loss=8.6614
218
+ 2026-04-03 16:49:59,482 [INFO] new_opacus_codex.train_steps: epoch=4 step=730 loss=8.6060
219
+ 2026-04-03 16:50:13,945 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=730 eval_loss=8.0606 duration_sec=14.42
220
+ 2026-04-03 16:51:18,082 [INFO] new_opacus_codex.train_steps: epoch=4 step=735 loss=8.6952
221
+ 2026-04-03 16:52:13,821 [INFO] new_opacus_codex.train_steps: epoch=4 step=740 loss=8.7486
222
+ 2026-04-03 16:52:28,311 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=740 eval_loss=8.0539 duration_sec=14.48
223
+ 2026-04-03 16:53:41,109 [INFO] new_opacus_codex.train_steps: epoch=5 step=745 loss=8.9434
224
+ 2026-04-03 16:54:42,800 [INFO] new_opacus_codex.train_steps: epoch=5 step=750 loss=8.6040
225
+ 2026-04-03 16:54:57,203 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=750 eval_loss=8.0482 duration_sec=14.39
226
+ 2026-04-03 16:55:58,878 [INFO] new_opacus_codex.train_steps: epoch=5 step=755 loss=8.4242
227
+ 2026-04-03 16:56:59,626 [INFO] new_opacus_codex.train_steps: epoch=5 step=760 loss=8.6597
228
+ 2026-04-03 16:57:14,418 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=760 eval_loss=8.0419 duration_sec=14.78
229
+ 2026-04-03 16:58:15,833 [INFO] new_opacus_codex.train_steps: epoch=5 step=765 loss=8.7243
230
+ 2026-04-03 16:59:18,087 [INFO] new_opacus_codex.train_steps: epoch=5 step=770 loss=8.6441
231
+ 2026-04-03 16:59:32,928 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=770 eval_loss=8.0361 duration_sec=14.83
232
+ 2026-04-03 17:00:33,386 [INFO] new_opacus_codex.train_steps: epoch=5 step=775 loss=8.6490
233
+ 2026-04-03 17:01:34,984 [INFO] new_opacus_codex.train_steps: epoch=5 step=780 loss=8.8158
234
+ 2026-04-03 17:01:49,621 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=780 eval_loss=8.0312 duration_sec=14.62
235
+ 2026-04-03 17:02:50,334 [INFO] new_opacus_codex.train_steps: epoch=5 step=785 loss=8.7401
236
+ 2026-04-03 17:03:50,737 [INFO] new_opacus_codex.train_steps: epoch=5 step=790 loss=8.5830
237
+ 2026-04-03 17:04:05,202 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=790 eval_loss=8.0256 duration_sec=14.45
238
+ 2026-04-03 17:05:09,207 [INFO] new_opacus_codex.train_steps: epoch=5 step=795 loss=8.6298
239
+ 2026-04-03 17:06:11,547 [INFO] new_opacus_codex.train_steps: epoch=5 step=800 loss=8.8127
240
+ 2026-04-03 17:06:26,035 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=800 eval_loss=8.0203 duration_sec=14.47
241
+ 2026-04-03 17:07:28,820 [INFO] new_opacus_codex.train_steps: epoch=5 step=805 loss=8.8090
242
+ 2026-04-03 17:08:31,469 [INFO] new_opacus_codex.train_steps: epoch=5 step=810 loss=8.6110
243
+ 2026-04-03 17:08:46,270 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=810 eval_loss=8.0148 duration_sec=14.79
244
+ 2026-04-03 17:09:46,299 [INFO] new_opacus_codex.train_steps: epoch=5 step=815 loss=8.5387
245
+ 2026-04-03 17:10:47,524 [INFO] new_opacus_codex.train_steps: epoch=5 step=820 loss=8.3954
246
+ 2026-04-03 17:11:01,740 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=820 eval_loss=8.0099 duration_sec=14.20
247
+ 2026-04-03 17:12:02,681 [INFO] new_opacus_codex.train_steps: epoch=5 step=825 loss=8.4409
248
+ 2026-04-03 17:13:03,941 [INFO] new_opacus_codex.train_steps: epoch=5 step=830 loss=8.4476
249
+ 2026-04-03 17:13:18,284 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=830 eval_loss=8.0046 duration_sec=14.33
250
+ 2026-04-03 17:14:25,047 [INFO] new_opacus_codex.train_steps: epoch=5 step=835 loss=8.1992
251
+ 2026-04-03 17:15:27,699 [INFO] new_opacus_codex.train_steps: epoch=5 step=840 loss=8.3737
252
+ 2026-04-03 17:15:42,050 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=840 eval_loss=7.9995 duration_sec=14.33
253
+ 2026-04-03 17:16:45,133 [INFO] new_opacus_codex.train_steps: epoch=5 step=845 loss=8.4779
254
+ 2026-04-03 17:17:48,090 [INFO] new_opacus_codex.train_steps: epoch=5 step=850 loss=8.4132
255
+ 2026-04-03 17:18:04,517 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=850 eval_loss=7.9955 duration_sec=16.41
256
+ 2026-04-03 17:19:10,080 [INFO] new_opacus_codex.train_steps: epoch=5 step=855 loss=8.5318
257
+ 2026-04-03 17:20:12,398 [INFO] new_opacus_codex.train_steps: epoch=5 step=860 loss=8.4488
258
+ 2026-04-03 17:20:27,384 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=860 eval_loss=7.9913 duration_sec=14.96
259
+ 2026-04-03 17:21:32,708 [INFO] new_opacus_codex.train_steps: epoch=5 step=865 loss=8.3725
260
+ 2026-04-03 17:22:36,619 [INFO] new_opacus_codex.train_steps: epoch=5 step=870 loss=8.3568
261
+ 2026-04-03 17:22:53,804 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=870 eval_loss=7.9872 duration_sec=17.15
262
+ 2026-04-03 17:24:01,939 [INFO] new_opacus_codex.train_steps: epoch=5 step=875 loss=8.3570
263
+ 2026-04-03 17:25:08,267 [INFO] new_opacus_codex.train_steps: epoch=5 step=880 loss=8.4181
264
+ 2026-04-03 17:25:22,880 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=880 eval_loss=7.9829 duration_sec=14.58
265
+ 2026-04-03 17:26:28,353 [INFO] new_opacus_codex.train_steps: epoch=5 step=885 loss=8.4445
266
+ 2026-04-03 17:27:31,413 [INFO] new_opacus_codex.train_steps: epoch=5 step=890 loss=8.3313
267
+ 2026-04-03 17:27:46,637 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=890 eval_loss=7.9784 duration_sec=15.22
268
+ 2026-04-03 17:28:50,883 [INFO] new_opacus_codex.train_steps: epoch=5 step=895 loss=8.3338
269
+ 2026-04-03 17:29:55,811 [INFO] new_opacus_codex.train_steps: epoch=5 step=900 loss=8.4964
270
+ 2026-04-03 17:30:11,629 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=900 eval_loss=7.9740 duration_sec=15.79
271
+ 2026-04-03 17:31:19,271 [INFO] new_opacus_codex.train_steps: epoch=5 step=905 loss=8.6959
272
+ 2026-04-03 17:32:27,378 [INFO] new_opacus_codex.train_steps: epoch=5 step=910 loss=8.5965
273
+ 2026-04-03 17:32:42,392 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=910 eval_loss=7.9702 duration_sec=14.97
274
+ 2026-04-03 17:33:47,253 [INFO] new_opacus_codex.train_steps: epoch=5 step=915 loss=8.5196
275
+ 2026-04-03 17:34:48,939 [INFO] new_opacus_codex.train_steps: epoch=5 step=920 loss=8.6196
276
+ 2026-04-03 17:35:03,182 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=920 eval_loss=7.9665 duration_sec=14.23
277
+ 2026-04-03 17:36:03,254 [INFO] new_opacus_codex.train_steps: epoch=5 step=925 loss=8.3729
qwen1.5-moe-a2.7b/dp8_attn_v2/adapter/README.md ADDED
@@ -0,0 +1,207 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: Qwen/Qwen1.5-MoE-A2.7B
3
+ library_name: peft
4
+ pipeline_tag: text-generation
5
+ tags:
6
+ - base_model:adapter:Qwen/Qwen1.5-MoE-A2.7B
7
+ - lora
8
+ - transformers
9
+ ---
10
+
11
+ # Model Card for Model ID
12
+
13
+ <!-- Provide a quick summary of what the model is/does. -->
14
+
15
+
16
+
17
+ ## Model Details
18
+
19
+ ### Model Description
20
+
21
+ <!-- Provide a longer summary of what this model is. -->
22
+
23
+
24
+
25
+ - **Developed by:** [More Information Needed]
26
+ - **Funded by [optional]:** [More Information Needed]
27
+ - **Shared by [optional]:** [More Information Needed]
28
+ - **Model type:** [More Information Needed]
29
+ - **Language(s) (NLP):** [More Information Needed]
30
+ - **License:** [More Information Needed]
31
+ - **Finetuned from model [optional]:** [More Information Needed]
32
+
33
+ ### Model Sources [optional]
34
+
35
+ <!-- Provide the basic links for the model. -->
36
+
37
+ - **Repository:** [More Information Needed]
38
+ - **Paper [optional]:** [More Information Needed]
39
+ - **Demo [optional]:** [More Information Needed]
40
+
41
+ ## Uses
42
+
43
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
44
+
45
+ ### Direct Use
46
+
47
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
48
+
49
+ [More Information Needed]
50
+
51
+ ### Downstream Use [optional]
52
+
53
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
54
+
55
+ [More Information Needed]
56
+
57
+ ### Out-of-Scope Use
58
+
59
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
60
+
61
+ [More Information Needed]
62
+
63
+ ## Bias, Risks, and Limitations
64
+
65
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
66
+
67
+ [More Information Needed]
68
+
69
+ ### Recommendations
70
+
71
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
72
+
73
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
74
+
75
+ ## How to Get Started with the Model
76
+
77
+ Use the code below to get started with the model.
78
+
79
+ [More Information Needed]
80
+
81
+ ## Training Details
82
+
83
+ ### Training Data
84
+
85
+ <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
86
+
87
+ [More Information Needed]
88
+
89
+ ### Training Procedure
90
+
91
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
92
+
93
+ #### Preprocessing [optional]
94
+
95
+ [More Information Needed]
96
+
97
+
98
+ #### Training Hyperparameters
99
+
100
+ - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
101
+
102
+ #### Speeds, Sizes, Times [optional]
103
+
104
+ <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
105
+
106
+ [More Information Needed]
107
+
108
+ ## Evaluation
109
+
110
+ <!-- This section describes the evaluation protocols and provides the results. -->
111
+
112
+ ### Testing Data, Factors & Metrics
113
+
114
+ #### Testing Data
115
+
116
+ <!-- This should link to a Dataset Card if possible. -->
117
+
118
+ [More Information Needed]
119
+
120
+ #### Factors
121
+
122
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
123
+
124
+ [More Information Needed]
125
+
126
+ #### Metrics
127
+
128
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
129
+
130
+ [More Information Needed]
131
+
132
+ ### Results
133
+
134
+ [More Information Needed]
135
+
136
+ #### Summary
137
+
138
+
139
+
140
+ ## Model Examination [optional]
141
+
142
+ <!-- Relevant interpretability work for the model goes here -->
143
+
144
+ [More Information Needed]
145
+
146
+ ## Environmental Impact
147
+
148
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
149
+
150
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
151
+
152
+ - **Hardware Type:** [More Information Needed]
153
+ - **Hours used:** [More Information Needed]
154
+ - **Cloud Provider:** [More Information Needed]
155
+ - **Compute Region:** [More Information Needed]
156
+ - **Carbon Emitted:** [More Information Needed]
157
+
158
+ ## Technical Specifications [optional]
159
+
160
+ ### Model Architecture and Objective
161
+
162
+ [More Information Needed]
163
+
164
+ ### Compute Infrastructure
165
+
166
+ [More Information Needed]
167
+
168
+ #### Hardware
169
+
170
+ [More Information Needed]
171
+
172
+ #### Software
173
+
174
+ [More Information Needed]
175
+
176
+ ## Citation [optional]
177
+
178
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
179
+
180
+ **BibTeX:**
181
+
182
+ [More Information Needed]
183
+
184
+ **APA:**
185
+
186
+ [More Information Needed]
187
+
188
+ ## Glossary [optional]
189
+
190
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
191
+
192
+ [More Information Needed]
193
+
194
+ ## More Information [optional]
195
+
196
+ [More Information Needed]
197
+
198
+ ## Model Card Authors [optional]
199
+
200
+ [More Information Needed]
201
+
202
+ ## Model Card Contact
203
+
204
+ [More Information Needed]
205
+ ### Framework versions
206
+
207
+ - PEFT 0.18.1
qwen1.5-moe-a2.7b/dp8_attn_v2/adapter/adapter_config.json ADDED
@@ -0,0 +1,46 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alora_invocation_tokens": null,
3
+ "alpha_pattern": {},
4
+ "arrow_config": null,
5
+ "auto_mapping": null,
6
+ "base_model_name_or_path": "Qwen/Qwen1.5-MoE-A2.7B",
7
+ "bias": "none",
8
+ "corda_config": null,
9
+ "ensure_weight_tying": true,
10
+ "eva_config": null,
11
+ "exclude_modules": null,
12
+ "fan_in_fan_out": false,
13
+ "inference_mode": true,
14
+ "init_lora_weights": true,
15
+ "layer_replication": null,
16
+ "layers_pattern": null,
17
+ "layers_to_transform": null,
18
+ "loftq_config": {},
19
+ "lora_alpha": 32,
20
+ "lora_bias": false,
21
+ "lora_dropout": 0.05,
22
+ "megatron_config": null,
23
+ "megatron_core": "megatron.core",
24
+ "modules_to_save": [
25
+ "lm_head",
26
+ "embed_tokens"
27
+ ],
28
+ "peft_type": "LORA",
29
+ "peft_version": "0.18.1",
30
+ "qalora_group_size": 16,
31
+ "r": 16,
32
+ "rank_pattern": {},
33
+ "revision": null,
34
+ "target_modules": [
35
+ "q_proj",
36
+ "k_proj",
37
+ "v_proj",
38
+ "o_proj"
39
+ ],
40
+ "target_parameters": null,
41
+ "task_type": "CAUSAL_LM",
42
+ "trainable_token_indices": null,
43
+ "use_dora": false,
44
+ "use_qalora": false,
45
+ "use_rslora": false
46
+ }
qwen1.5-moe-a2.7b/dp8_attn_v2/audit_results.json ADDED
@@ -0,0 +1,137 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "delta": 1e-05,
3
+ "num_canaries": 500,
4
+ "num_members": 250,
5
+ "paper_guess_fraction": 0.2,
6
+ "paper_guess_steps": 20,
7
+ "loss": {
8
+ "auc": 0.568112,
9
+ "empirical_epsilon": {
10
+ "0.05": 0.9539302941411734,
11
+ "0.01": 0.6810795972123742
12
+ },
13
+ "empirical_epsilon_details": {
14
+ "0.05": {
15
+ "epsilon": 0.9539302941411734,
16
+ "num_guesses": 35,
17
+ "correct_guesses": 30,
18
+ "candidate_num_guesses": [
19
+ 5,
20
+ 10,
21
+ 15,
22
+ 20,
23
+ 25,
24
+ 30,
25
+ 35,
26
+ 40,
27
+ 45,
28
+ 50,
29
+ 55,
30
+ 60,
31
+ 65,
32
+ 70,
33
+ 75,
34
+ 80,
35
+ 85,
36
+ 90,
37
+ 95,
38
+ 100
39
+ ],
40
+ "direction": "lower"
41
+ },
42
+ "0.01": {
43
+ "epsilon": 0.6810795972123742,
44
+ "num_guesses": 35,
45
+ "correct_guesses": 30,
46
+ "candidate_num_guesses": [
47
+ 5,
48
+ 10,
49
+ 15,
50
+ 20,
51
+ 25,
52
+ 30,
53
+ 35,
54
+ 40,
55
+ 45,
56
+ 50,
57
+ 55,
58
+ 60,
59
+ 65,
60
+ 70,
61
+ 75,
62
+ 80,
63
+ 85,
64
+ 90,
65
+ 95,
66
+ 100
67
+ ],
68
+ "direction": "lower"
69
+ }
70
+ }
71
+ },
72
+ "embedding": {
73
+ "auc": 0.569768,
74
+ "empirical_epsilon": {
75
+ "0.05": 0.010390725918114185,
76
+ "0.01": 0.0
77
+ },
78
+ "empirical_epsilon_details": {
79
+ "0.05": {
80
+ "epsilon": 0.010390725918114185,
81
+ "num_guesses": 100,
82
+ "correct_guesses": 59,
83
+ "candidate_num_guesses": [
84
+ 5,
85
+ 10,
86
+ 15,
87
+ 20,
88
+ 25,
89
+ 30,
90
+ 35,
91
+ 40,
92
+ 45,
93
+ 50,
94
+ 55,
95
+ 60,
96
+ 65,
97
+ 70,
98
+ 75,
99
+ 80,
100
+ 85,
101
+ 90,
102
+ 95,
103
+ 100
104
+ ],
105
+ "direction": "higher"
106
+ },
107
+ "0.01": {
108
+ "epsilon": 0.0,
109
+ "num_guesses": 0,
110
+ "correct_guesses": 0,
111
+ "candidate_num_guesses": [
112
+ 5,
113
+ 10,
114
+ 15,
115
+ 20,
116
+ 25,
117
+ 30,
118
+ 35,
119
+ 40,
120
+ 45,
121
+ 50,
122
+ 55,
123
+ 60,
124
+ 65,
125
+ 70,
126
+ 75,
127
+ 80,
128
+ 85,
129
+ 90,
130
+ 95,
131
+ 100
132
+ ],
133
+ "direction": "lower"
134
+ }
135
+ }
136
+ }
137
+ }
qwen1.5-moe-a2.7b/dp8_attn_v2/canary_meta.json ADDED
The diff for this file is too large to render. See raw diff
 
qwen1.5-moe-a2.7b/dp8_attn_v2/metrics.jsonl ADDED
The diff for this file is too large to render. See raw diff
 
qwen1.5-moe-a2.7b/dp8_attn_v2/resolved_config.yaml ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ model:
2
+ name: Qwen/Qwen1.5-MoE-A2.7B
3
+ tokenizer_name: Qwen/Qwen1.5-MoE-A2.7B
4
+ max_length: 1024
5
+ dtype: bfloat16
6
+ trust_remote_code: true
7
+ use_fast_tokenizer: true
8
+ cache_dir: null
9
+ local_files_only: false
10
+ low_cpu_mem_usage: true
11
+ tie_word_embeddings: true
12
+ gradient_checkpointing: false
13
+ use_chat_template: false
14
+ dataset:
15
+ name: melihcatal/codedp-cpt
16
+ split: train
17
+ mode: cpt
18
+ text_column: text
19
+ validation_ratio: 0.05
20
+ max_samples: -1
21
+ lora:
22
+ enabled: true
23
+ r: 16
24
+ alpha: 32
25
+ dropout: 0.05
26
+ target_modules:
27
+ - q_proj
28
+ - k_proj
29
+ - v_proj
30
+ - o_proj
31
+ modules_to_save:
32
+ - lm_head
33
+ bias: none
34
+ training:
35
+ seed: 42
36
+ epochs: 5
37
+ warmup_steps: null
38
+ warmup_ratio: 0.05
39
+ mixed_precision: false
40
+ mixed_precision_dtype: bfloat16
41
+ batch_size: 4
42
+ eval_batch_size: 8
43
+ eval_every_steps: 10
44
+ eval_every_epochs: 1
45
+ learning_rate: 0.0005
46
+ optimizer: adamw
47
+ lr_scheduler: cosine
48
+ adam_beta1: 0.9
49
+ adam_beta2: 0.999
50
+ adam_epsilon: 1.0e-08
51
+ sgd_momentum: 0.9
52
+ weight_decay: 0.01
53
+ max_grad_norm: 1.0
54
+ log_every: 5
55
+ gradient_accumulation_steps: 8
56
+ num_workers: 4
57
+ output_dir: /scratch/mcatal/runs/cpt/qwen1.5-moe-a2.7b/dp8_attn_v2
58
+ min_lr_ratio: 0.15
59
+ distributed:
60
+ strategy: dpddp
61
+ backend: nccl
62
+ devices: null
63
+ dp:
64
+ module_validator: auto
65
+ target_delta: 1.0e-05
66
+ noise_multiplier: null
67
+ max_grad_norm: 1.0
68
+ grad_sample_mode: hooks
69
+ clipping: per_layer
70
+ secure_mode: false
71
+ enabled: true
72
+ target_epsilon: 8.0
73
+ audit:
74
+ enabled: true
75
+ run_every_epoch: true
76
+ epoch_device: cuda
77
+ q_canary: auto
78
+ num_canaries: 500
79
+ prefix_length: 49
80
+ num_digits: 12
81
+ batch_size: 32
82
+ delta: 1.0e-05
83
+ p_values:
84
+ - 0.05
85
+ - 0.01
86
+ paper_guess_fraction: 0.2
87
+ paper_guess_steps: 20
88
+ enable_holdout_empirical_epsilon: false
89
+ holdout_seed: 42
90
+ tie_seed: 42
91
+ tracking:
92
+ enabled: true
93
+ tensorboard: true
94
+ wandb: false
95
+ wandb_project: codedp-finetune-h200-audit
96
+ wandb_run_name: qwen1.5-moe-a2.7b-cpt-dp8_attn
97
+ wandb_mode: online
98
+ codecarbon: true
99
+ codecarbon_output_file: codecarbon.csv
100
+ codecarbon_measure_power_secs: 15
101
+ codecarbon_country_iso_code: null
102
+ codecarbon_project_name: codedp-qwen1.5-moe-a2.7b-cpt-dp8_attn
103
+ moe:
104
+ output_router_logits: false
105
+ router_aux_loss_coef: 0.0
106
+ freeze_router: true
107
+ profile:
108
+ enabled: false
109
+ num_batches: 8
110
+ top_experts: 8
111
+ output_file: moe_expert_profile.json
qwen1.5-moe-a2.7b/dp8_attn_v2/summary.json ADDED
@@ -0,0 +1,72 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "audit/delta": 1e-05,
3
+ "audit/embedding/auc": 0.569768,
4
+ "audit/embedding/empirical_epsilon/0.01": 0.0,
5
+ "audit/embedding/empirical_epsilon/0.05": 0.010390725918114185,
6
+ "audit/embedding/empirical_epsilon_details/0.01/correct_guesses": 0.0,
7
+ "audit/embedding/empirical_epsilon_details/0.01/epsilon": 0.0,
8
+ "audit/embedding/empirical_epsilon_details/0.01/num_guesses": 0.0,
9
+ "audit/embedding/empirical_epsilon_details/0.05/correct_guesses": 59.0,
10
+ "audit/embedding/empirical_epsilon_details/0.05/epsilon": 0.010390725918114185,
11
+ "audit/embedding/empirical_epsilon_details/0.05/num_guesses": 100.0,
12
+ "audit/loss/auc": 0.568112,
13
+ "audit/loss/empirical_epsilon/0.01": 0.6810795972123742,
14
+ "audit/loss/empirical_epsilon/0.05": 0.9539302941411734,
15
+ "audit/loss/empirical_epsilon_details/0.01/correct_guesses": 30.0,
16
+ "audit/loss/empirical_epsilon_details/0.01/epsilon": 0.6810795972123742,
17
+ "audit/loss/empirical_epsilon_details/0.01/num_guesses": 35.0,
18
+ "audit/loss/empirical_epsilon_details/0.05/correct_guesses": 30.0,
19
+ "audit/loss/empirical_epsilon_details/0.05/epsilon": 0.9539302941411734,
20
+ "audit/loss/empirical_epsilon_details/0.05/num_guesses": 35.0,
21
+ "audit/num_canaries": 500.0,
22
+ "audit/num_members": 250.0,
23
+ "audit/paper_guess_fraction": 0.2,
24
+ "audit/paper_guess_steps": 20.0,
25
+ "energy/codecarbon/cpu_count": 32.0,
26
+ "energy/codecarbon/cpu_energy": 0.3174872591751485,
27
+ "energy/codecarbon/cpu_power": 81.01999924313104,
28
+ "energy/codecarbon/cpu_utilization_percent": 10.021495970796888,
29
+ "energy/codecarbon/duration": 14647.29265363299,
30
+ "energy/codecarbon/emissions": 0.27246604620222886,
31
+ "energy/codecarbon/emissions_rate": 1.8601802575075107e-05,
32
+ "energy/codecarbon/energy_consumed": 7.819821662951782,
33
+ "energy/codecarbon/gpu_count": 8.0,
34
+ "energy/codecarbon/gpu_energy": 7.353428284960146,
35
+ "energy/codecarbon/gpu_power": 1812.64383279467,
36
+ "energy/codecarbon/gpu_utilization_percent": 73.28197534265445,
37
+ "energy/codecarbon/latitude": 47.4843,
38
+ "energy/codecarbon/longitude": 8.212,
39
+ "energy/codecarbon/pue": 1.0,
40
+ "energy/codecarbon/ram_energy": 0.14890611881649343,
41
+ "energy/codecarbon/ram_power": 38.0,
42
+ "energy/codecarbon/ram_total_size": 256.0,
43
+ "energy/codecarbon/ram_used_gb": 123.66550332894461,
44
+ "energy/codecarbon/ram_utilization_percent": 6.562077278049452,
45
+ "energy/codecarbon/water_consumed": 0.0,
46
+ "energy/codecarbon/wue": 0.0,
47
+ "eval/duration_sec": 16.77953677199548,
48
+ "eval/loss": 6.994043157650874,
49
+ "perf/audit_duration_sec": 7.691278743004659,
50
+ "perf/epoch_duration_sec": 2568.0977820309927,
51
+ "perf/epoch_samples": 51891.0,
52
+ "perf/epoch_samples_per_sec": 20.206006314511026,
53
+ "perf/epoch_tokens": 37894520.0,
54
+ "perf/epoch_tokens_per_sec": 14755.87116080562,
55
+ "perf/logical_batch_size": 25.0,
56
+ "perf/logical_token_count": 19367.0,
57
+ "perf/physical_batches": 8.0,
58
+ "perf/samples_per_sec": 2.4299956279329176,
59
+ "perf/step_duration_sec": 10.288084353989689,
60
+ "perf/tokens_per_sec": 1882.4690130470726,
61
+ "privacy/epsilon": 7.992347493888048,
62
+ "system/cuda_epoch_peak_memory_gb": 56.501492977142334,
63
+ "system/cuda_max_memory_allocated_gb": 56.501492977142334,
64
+ "system/cuda_memory_allocated_gb": 29.994746685028076,
65
+ "train/epoch_canary_loss": 10.16043418827153,
66
+ "train/epoch_loss": 7.525427301665942,
67
+ "train/epoch_real_loss": 7.16184575840965,
68
+ "train/lr": 7.5e-05,
69
+ "train/step_canary_loss": 5.96875,
70
+ "train/step_loss": 7.116840582627517,
71
+ "train/step_real_loss": 7.2125147978464765
72
+ }
qwen1.5-moe-a2.7b/dp8_attn_v2/tokenizer/added_tokens.json ADDED
@@ -0,0 +1,505 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "114848338811": 151654,
3
+ "116427191984": 152008,
4
+ "116639746413": 151980,
5
+ "117298239631": 151870,
6
+ "117657531962": 151927,
7
+ "117783274348": 151944,
8
+ "119342152414": 152017,
9
+ "121557291859": 151876,
10
+ "123741461297": 151785,
11
+ "124589118494": 151781,
12
+ "125866399464": 151890,
13
+ "126574388585": 151832,
14
+ "129693547287": 151766,
15
+ "133373659361": 151961,
16
+ "135441121756": 151649,
17
+ "136931728327": 151751,
18
+ "138154781148": 152061,
19
+ "138218735276": 151945,
20
+ "141499982867": 152039,
21
+ "142526557681": 152139,
22
+ "142799718159": 152003,
23
+ "143182978129": 151965,
24
+ "143777992227": 151660,
25
+ "144373897488": 152071,
26
+ "145669731981": 151875,
27
+ "147444452794": 152124,
28
+ "147541537162": 152069,
29
+ "151859598873": 151841,
30
+ "152676556624": 151976,
31
+ "154962668224": 151990,
32
+ "158142752582": 151871,
33
+ "158929292238": 151881,
34
+ "159337222655": 151716,
35
+ "161238482259": 152084,
36
+ "161588262349": 151731,
37
+ "163165456396": 152082,
38
+ "163497196769": 152114,
39
+ "165429152138": 152011,
40
+ "172599315861": 151665,
41
+ "173781362446": 151717,
42
+ "175469284871": 151809,
43
+ "175528632226": 152127,
44
+ "178478368922": 152056,
45
+ "179386156569": 152098,
46
+ "179564995814": 151991,
47
+ "181233596124": 152076,
48
+ "181439694557": 151692,
49
+ "182345464792": 151736,
50
+ "182782912224": 151918,
51
+ "183368827562": 151845,
52
+ "185438575221": 151831,
53
+ "186198784585": 151892,
54
+ "186872457894": 152129,
55
+ "191887635211": 152015,
56
+ "195388849133": 152086,
57
+ "195792996468": 151840,
58
+ "196119929397": 151905,
59
+ "196735786156": 151787,
60
+ "198462689911": 151722,
61
+ "211411365397": 151671,
62
+ "211662639865": 152000,
63
+ "212985874612": 152014,
64
+ "214193645583": 152096,
65
+ "215671314196": 151896,
66
+ "215826764872": 152050,
67
+ "216667245274": 151778,
68
+ "217756494954": 151942,
69
+ "218391991184": 151973,
70
+ "218784516525": 152001,
71
+ "221762325616": 151972,
72
+ "225164373623": 151733,
73
+ "228358422847": 151963,
74
+ "228565148417": 152020,
75
+ "229775525672": 152087,
76
+ "231853711778": 151810,
77
+ "232776999554": 151737,
78
+ "232787996724": 151895,
79
+ "232896526252": 151783,
80
+ "233795848681": 151801,
81
+ "234316622259": 151919,
82
+ "242851284913": 151729,
83
+ "243165586174": 152078,
84
+ "244371196654": 151909,
85
+ "244885399387": 151827,
86
+ "245416776926": 151877,
87
+ "251789649997": 151907,
88
+ "252867443568": 151872,
89
+ "256497188281": 152029,
90
+ "259533577263": 151752,
91
+ "259666844669": 151728,
92
+ "262296912232": 151775,
93
+ "262437452884": 152088,
94
+ "264259917554": 151707,
95
+ "265847332563": 152112,
96
+ "267514771964": 151868,
97
+ "267954879268": 151940,
98
+ "268433315926": 152116,
99
+ "268921994538": 152009,
100
+ "272362973463": 151700,
101
+ "275597777299": 151859,
102
+ "281673864127": 151702,
103
+ "282322838685": 151667,
104
+ "284445431422": 151958,
105
+ "284776712475": 152074,
106
+ "286173241916": 151897,
107
+ "291762694874": 151964,
108
+ "293847485454": 152054,
109
+ "295118729589": 151903,
110
+ "295448593321": 151784,
111
+ "296953381169": 151773,
112
+ "298464134954": 151777,
113
+ "311632789559": 151685,
114
+ "313145133463": 152067,
115
+ "313313831273": 152109,
116
+ "314697386682": 151743,
117
+ "315534574435": 152048,
118
+ "319584985416": 151864,
119
+ "322589379462": 151974,
120
+ "322683334381": 151962,
121
+ "325196781219": 152138,
122
+ "327158398268": 151967,
123
+ "327491773134": 151681,
124
+ "328552864359": 151999,
125
+ "331369926711": 152025,
126
+ "332234613346": 152137,
127
+ "332522971941": 151977,
128
+ "334513529294": 151956,
129
+ "336813343298": 152041,
130
+ "337326533813": 151738,
131
+ "339882449689": 151891,
132
+ "342221452223": 152036,
133
+ "345271433112": 151951,
134
+ "349249721685": 152045,
135
+ "351885857332": 151720,
136
+ "353755529131": 152047,
137
+ "355977139358": 151673,
138
+ "356722692178": 152140,
139
+ "357299947518": 152126,
140
+ "357621613582": 152032,
141
+ "363214487524": 151826,
142
+ "364196983365": 151839,
143
+ "364492359246": 151989,
144
+ "364947325679": 151912,
145
+ "367459894796": 151650,
146
+ "367968398551": 152051,
147
+ "368763327262": 152093,
148
+ "371496153876": 152110,
149
+ "374327157578": 151684,
150
+ "374439448821": 151762,
151
+ "374598961236": 151806,
152
+ "375429755114": 152075,
153
+ "377273562477": 151774,
154
+ "377338713663": 151788,
155
+ "378749551722": 152083,
156
+ "379557332953": 151745,
157
+ "379571934299": 151906,
158
+ "383837596997": 151675,
159
+ "384598329253": 151910,
160
+ "387763951128": 152024,
161
+ "389715484387": 152053,
162
+ "393741199486": 151998,
163
+ "396521188476": 151959,
164
+ "396847976144": 152079,
165
+ "398549545767": 151796,
166
+ "399223169721": 151842,
167
+ "399749397872": 151765,
168
+ "412843856172": 152097,
169
+ "415956914763": 151923,
170
+ "417668946583": 152031,
171
+ "418412875665": 151992,
172
+ "421369185549": 151920,
173
+ "421435255895": 151960,
174
+ "425949483533": 151662,
175
+ "429781429464": 151747,
176
+ "431149748713": 151888,
177
+ "431751228368": 151828,
178
+ "433581868192": 151829,
179
+ "434632855397": 151921,
180
+ "438124184139": 152106,
181
+ "438435178228": 152033,
182
+ "438745114173": 151984,
183
+ "441373944925": 151757,
184
+ "441377892146": 152089,
185
+ "445925149649": 151658,
186
+ "447445629421": 151690,
187
+ "449318681694": 152141,
188
+ "449549215429": 152113,
189
+ "451161679736": 151970,
190
+ "451885565366": 152090,
191
+ "454164859837": 151950,
192
+ "454244839926": 151797,
193
+ "455423278865": 151997,
194
+ "457919547633": 151652,
195
+ "462294118988": 151952,
196
+ "467893531755": 151759,
197
+ "472129283475": 152043,
198
+ "473822473819": 151691,
199
+ "475794684356": 151850,
200
+ "476969195485": 152066,
201
+ "479227349997": 151750,
202
+ "479676316326": 151703,
203
+ "481884157827": 151710,
204
+ "482829231639": 152122,
205
+ "484489193824": 151823,
206
+ "484773717614": 151791,
207
+ "485177821815": 151648,
208
+ "485692541169": 152034,
209
+ "486391569221": 151830,
210
+ "488496781997": 152120,
211
+ "488538864842": 151753,
212
+ "489562189822": 151824,
213
+ "489991411364": 151755,
214
+ "492373714791": 151884,
215
+ "492516914298": 152063,
216
+ "497294727353": 151954,
217
+ "499148455254": 151756,
218
+ "499156317423": 151948,
219
+ "499444466686": 151742,
220
+ "511785788222": 151953,
221
+ "511923256573": 151966,
222
+ "512659818733": 151885,
223
+ "513349618943": 151979,
224
+ "513361312532": 151814,
225
+ "514354144759": 152044,
226
+ "514532995959": 151730,
227
+ "515856611931": 151949,
228
+ "516968424863": 151770,
229
+ "517379856925": 151744,
230
+ "517778845323": 151725,
231
+ "522515164754": 152072,
232
+ "523723475786": 152049,
233
+ "523918658846": 151686,
234
+ "523937669294": 152010,
235
+ "524444658383": 151943,
236
+ "527117488925": 151760,
237
+ "527874729936": 152022,
238
+ "529455466433": 151904,
239
+ "529679678956": 151714,
240
+ "529854679937": 151834,
241
+ "532833541879": 151936,
242
+ "533417736566": 151846,
243
+ "535787224917": 151719,
244
+ "538631466654": 151971,
245
+ "545512344383": 151741,
246
+ "546276298359": 152046,
247
+ "547149832394": 151776,
248
+ "547563892268": 152107,
249
+ "549946382131": 151957,
250
+ "552738847828": 152130,
251
+ "556814553861": 152121,
252
+ "556976448182": 151899,
253
+ "557163366873": 152070,
254
+ "557533689478": 151821,
255
+ "558837194455": 152081,
256
+ "562389434565": 152028,
257
+ "562716493614": 152026,
258
+ "563117729724": 151804,
259
+ "563365646341": 151865,
260
+ "564395222747": 151969,
261
+ "564544131355": 151908,
262
+ "569765693871": 151647,
263
+ "569853653352": 152052,
264
+ "571317457151": 152006,
265
+ "572154333646": 151780,
266
+ "578827344666": 151721,
267
+ "579184949249": 151712,
268
+ "581493616981": 152057,
269
+ "581976444672": 151882,
270
+ "582471228164": 151819,
271
+ "583527834464": 152119,
272
+ "585544633356": 151818,
273
+ "586855179568": 151790,
274
+ "587541149322": 152111,
275
+ "587787496812": 151711,
276
+ "589439538674": 152058,
277
+ "589547936288": 151825,
278
+ "591464355756": 152013,
279
+ "592538279337": 151693,
280
+ "594688742928": 152042,
281
+ "598916996413": 151924,
282
+ "599961512792": 151900,
283
+ "612128769512": 151683,
284
+ "612788593128": 151874,
285
+ "614583493135": 152135,
286
+ "614797267726": 151746,
287
+ "616248973917": 151836,
288
+ "618423867817": 152019,
289
+ "618878266619": 152123,
290
+ "619454958782": 151887,
291
+ "622553389126": 151987,
292
+ "624221476921": 151889,
293
+ "624266421831": 151995,
294
+ "625334487352": 152101,
295
+ "626748382482": 152131,
296
+ "632962475133": 151822,
297
+ "635127952696": 152007,
298
+ "635185846287": 151981,
299
+ "636324136426": 152095,
300
+ "637551427887": 151941,
301
+ "643489267333": 151666,
302
+ "643692393748": 151695,
303
+ "644234165531": 151917,
304
+ "645527857972": 152128,
305
+ "646979124677": 151704,
306
+ "649291669397": 151656,
307
+ "649516938598": 152104,
308
+ "652154162978": 151851,
309
+ "653573457168": 151786,
310
+ "655517954651": 151913,
311
+ "658481324922": 151709,
312
+ "661436365453": 151727,
313
+ "662426712668": 151811,
314
+ "664847713559": 151931,
315
+ "665661658354": 151848,
316
+ "668134915514": 151694,
317
+ "668514843129": 151853,
318
+ "669183782449": 151937,
319
+ "669689115557": 151680,
320
+ "673948786567": 151914,
321
+ "676539973863": 151734,
322
+ "676676697292": 151739,
323
+ "677896358599": 151793,
324
+ "678666359123": 151820,
325
+ "682564697312": 151674,
326
+ "687284547244": 152142,
327
+ "689285841825": 151767,
328
+ "689362171782": 151676,
329
+ "691315125346": 151929,
330
+ "691587454538": 151659,
331
+ "693281533643": 151849,
332
+ "694235616268": 151701,
333
+ "696651276628": 151696,
334
+ "697747991249": 152037,
335
+ "698218415444": 151672,
336
+ "699446724178": 151994,
337
+ "711412347158": 151812,
338
+ "711641973413": 151947,
339
+ "712992815957": 151815,
340
+ "714738141818": 152055,
341
+ "716763689965": 152038,
342
+ "717628619782": 152105,
343
+ "718726519731": 151664,
344
+ "718951851413": 151993,
345
+ "721737645236": 151749,
346
+ "722339992679": 152060,
347
+ "722618266242": 151723,
348
+ "724492262421": 151986,
349
+ "724883568652": 151769,
350
+ "724964923582": 152073,
351
+ "729116422489": 152021,
352
+ "729825168745": 151928,
353
+ "731681792655": 151860,
354
+ "733737988257": 151771,
355
+ "739246183345": 152012,
356
+ "739394546916": 151939,
357
+ "739667844957": 152027,
358
+ "739784153482": 152023,
359
+ "741757479784": 151833,
360
+ "742765629356": 151732,
361
+ "745977673725": 151898,
362
+ "746285987371": 151655,
363
+ "749665846687": 151726,
364
+ "749966767285": 151677,
365
+ "752644929761": 151857,
366
+ "753159165157": 151678,
367
+ "753233719644": 151852,
368
+ "753466987363": 151813,
369
+ "754978243758": 152085,
370
+ "756916173536": 152099,
371
+ "757422791351": 152094,
372
+ "757785319772": 152062,
373
+ "758963111966": 152068,
374
+ "762486672996": 152002,
375
+ "762495822823": 152092,
376
+ "763531783491": 151837,
377
+ "765474393376": 151653,
378
+ "765737843639": 151808,
379
+ "765981581453": 151687,
380
+ "766294538337": 151901,
381
+ "767858144912": 152005,
382
+ "768461952319": 151930,
383
+ "771619544974": 151768,
384
+ "773671349113": 151925,
385
+ "773687856563": 151718,
386
+ "774679513717": 152117,
387
+ "778466723723": 151698,
388
+ "781653545886": 151668,
389
+ "786391632558": 151847,
390
+ "794825141891": 151688,
391
+ "795693824762": 151679,
392
+ "795838284624": 151715,
393
+ "796415361892": 151669,
394
+ "797238311233": 152145,
395
+ "799364566435": 151798,
396
+ "799417897197": 151878,
397
+ "811371928234": 151866,
398
+ "813981174797": 151983,
399
+ "814611949491": 151946,
400
+ "815458725125": 151996,
401
+ "816115936673": 151916,
402
+ "817378986438": 151800,
403
+ "818479319499": 152040,
404
+ "819162623892": 151854,
405
+ "824387273757": 151802,
406
+ "824488329416": 151934,
407
+ "825815739116": 152035,
408
+ "826214551218": 151735,
409
+ "829235161526": 151817,
410
+ "836746273134": 151708,
411
+ "837131396371": 151867,
412
+ "837873762491": 151935,
413
+ "839269684865": 151873,
414
+ "841824384614": 151782,
415
+ "841991688488": 151670,
416
+ "843998562287": 151779,
417
+ "846657726193": 152059,
418
+ "848845611563": 151794,
419
+ "851254219729": 152118,
420
+ "851852651677": 151795,
421
+ "852347289392": 151772,
422
+ "853859348234": 151697,
423
+ "856317483891": 152108,
424
+ "857231548835": 151915,
425
+ "858146415154": 152004,
426
+ "859289776479": 152134,
427
+ "859927364588": 152030,
428
+ "861342291298": 152115,
429
+ "864718285734": 151862,
430
+ "865331112869": 151646,
431
+ "866519986723": 151955,
432
+ "867678158885": 151764,
433
+ "873898736873": 151689,
434
+ "876537814263": 151975,
435
+ "877385381798": 151856,
436
+ "877482678543": 151651,
437
+ "879425227932": 151763,
438
+ "879764613759": 151968,
439
+ "881112998642": 151886,
440
+ "881136466196": 151858,
441
+ "883685933161": 151706,
442
+ "884354318946": 151978,
443
+ "886464195129": 151933,
444
+ "887417525236": 152103,
445
+ "887753195844": 151911,
446
+ "889965243135": 151988,
447
+ "892336777248": 151880,
448
+ "892697125149": 152100,
449
+ "893974619684": 152144,
450
+ "894885246797": 151792,
451
+ "897784492783": 151883,
452
+ "899151413682": 151758,
453
+ "912193598348": 151713,
454
+ "916198946615": 151803,
455
+ "919181521716": 152064,
456
+ "921889371893": 151922,
457
+ "921894985642": 152132,
458
+ "922327493433": 151705,
459
+ "922466849763": 151748,
460
+ "922867396683": 151807,
461
+ "927914615679": 151657,
462
+ "928335588653": 151761,
463
+ "929185622831": 151740,
464
+ "929826356991": 151699,
465
+ "933775286797": 152136,
466
+ "934912591213": 151902,
467
+ "937495658852": 151754,
468
+ "938488813964": 151843,
469
+ "941169431859": 151855,
470
+ "941861857715": 152102,
471
+ "941976537588": 151938,
472
+ "943193294691": 151893,
473
+ "943878645871": 152133,
474
+ "944513428457": 151869,
475
+ "946444632795": 152018,
476
+ "947262995132": 152143,
477
+ "948585673431": 152077,
478
+ "949235426629": 152125,
479
+ "951794811935": 151805,
480
+ "952465217972": 151926,
481
+ "952872416512": 151724,
482
+ "953267689786": 151932,
483
+ "955456118295": 151838,
484
+ "955668961269": 151894,
485
+ "961133895172": 151861,
486
+ "961981959227": 151844,
487
+ "963852916563": 151863,
488
+ "964342468552": 151789,
489
+ "967114116556": 151799,
490
+ "967214577653": 152016,
491
+ "971621888444": 151816,
492
+ "981277526855": 152091,
493
+ "982993456429": 151663,
494
+ "983493951323": 151985,
495
+ "983569279932": 151682,
496
+ "985781138935": 152065,
497
+ "993832498489": 151982,
498
+ "996116119839": 151835,
499
+ "997724962668": 152080,
500
+ "997958836435": 151879,
501
+ "997981281989": 151661,
502
+ "<|endoftext|>": 151643,
503
+ "<|im_end|>": 151645,
504
+ "<|im_start|>": 151644
505
+ }
qwen1.5-moe-a2.7b/dp8_attn_v2/tokenizer/chat_template.jinja ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system
2
+ You are a helpful assistant<|im_end|>
3
+ ' }}{% endif %}{{'<|im_start|>' + message['role'] + '
4
+ ' + message['content'] + '<|im_end|>' + '
5
+ '}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
6
+ ' }}{% endif %}
qwen1.5-moe-a2.7b/dp8_attn_v2/tokenizer/merges.txt ADDED
The diff for this file is too large to render. See raw diff
 
qwen1.5-moe-a2.7b/dp8_attn_v2/tokenizer/special_tokens_map.json ADDED
@@ -0,0 +1,3518 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": [
3
+ {
4
+ "content": "865331112869",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false
9
+ },
10
+ {
11
+ "content": "569765693871",
12
+ "lstrip": false,
13
+ "normalized": false,
14
+ "rstrip": false,
15
+ "single_word": false
16
+ },
17
+ {
18
+ "content": "485177821815",
19
+ "lstrip": false,
20
+ "normalized": false,
21
+ "rstrip": false,
22
+ "single_word": false
23
+ },
24
+ {
25
+ "content": "135441121756",
26
+ "lstrip": false,
27
+ "normalized": false,
28
+ "rstrip": false,
29
+ "single_word": false
30
+ },
31
+ {
32
+ "content": "367459894796",
33
+ "lstrip": false,
34
+ "normalized": false,
35
+ "rstrip": false,
36
+ "single_word": false
37
+ },
38
+ {
39
+ "content": "877482678543",
40
+ "lstrip": false,
41
+ "normalized": false,
42
+ "rstrip": false,
43
+ "single_word": false
44
+ },
45
+ {
46
+ "content": "457919547633",
47
+ "lstrip": false,
48
+ "normalized": false,
49
+ "rstrip": false,
50
+ "single_word": false
51
+ },
52
+ {
53
+ "content": "765474393376",
54
+ "lstrip": false,
55
+ "normalized": false,
56
+ "rstrip": false,
57
+ "single_word": false
58
+ },
59
+ {
60
+ "content": "114848338811",
61
+ "lstrip": false,
62
+ "normalized": false,
63
+ "rstrip": false,
64
+ "single_word": false
65
+ },
66
+ {
67
+ "content": "746285987371",
68
+ "lstrip": false,
69
+ "normalized": false,
70
+ "rstrip": false,
71
+ "single_word": false
72
+ },
73
+ {
74
+ "content": "649291669397",
75
+ "lstrip": false,
76
+ "normalized": false,
77
+ "rstrip": false,
78
+ "single_word": false
79
+ },
80
+ {
81
+ "content": "927914615679",
82
+ "lstrip": false,
83
+ "normalized": false,
84
+ "rstrip": false,
85
+ "single_word": false
86
+ },
87
+ {
88
+ "content": "445925149649",
89
+ "lstrip": false,
90
+ "normalized": false,
91
+ "rstrip": false,
92
+ "single_word": false
93
+ },
94
+ {
95
+ "content": "691587454538",
96
+ "lstrip": false,
97
+ "normalized": false,
98
+ "rstrip": false,
99
+ "single_word": false
100
+ },
101
+ {
102
+ "content": "143777992227",
103
+ "lstrip": false,
104
+ "normalized": false,
105
+ "rstrip": false,
106
+ "single_word": false
107
+ },
108
+ {
109
+ "content": "997981281989",
110
+ "lstrip": false,
111
+ "normalized": false,
112
+ "rstrip": false,
113
+ "single_word": false
114
+ },
115
+ {
116
+ "content": "425949483533",
117
+ "lstrip": false,
118
+ "normalized": false,
119
+ "rstrip": false,
120
+ "single_word": false
121
+ },
122
+ {
123
+ "content": "982993456429",
124
+ "lstrip": false,
125
+ "normalized": false,
126
+ "rstrip": false,
127
+ "single_word": false
128
+ },
129
+ {
130
+ "content": "718726519731",
131
+ "lstrip": false,
132
+ "normalized": false,
133
+ "rstrip": false,
134
+ "single_word": false
135
+ },
136
+ {
137
+ "content": "172599315861",
138
+ "lstrip": false,
139
+ "normalized": false,
140
+ "rstrip": false,
141
+ "single_word": false
142
+ },
143
+ {
144
+ "content": "643489267333",
145
+ "lstrip": false,
146
+ "normalized": false,
147
+ "rstrip": false,
148
+ "single_word": false
149
+ },
150
+ {
151
+ "content": "282322838685",
152
+ "lstrip": false,
153
+ "normalized": false,
154
+ "rstrip": false,
155
+ "single_word": false
156
+ },
157
+ {
158
+ "content": "781653545886",
159
+ "lstrip": false,
160
+ "normalized": false,
161
+ "rstrip": false,
162
+ "single_word": false
163
+ },
164
+ {
165
+ "content": "796415361892",
166
+ "lstrip": false,
167
+ "normalized": false,
168
+ "rstrip": false,
169
+ "single_word": false
170
+ },
171
+ {
172
+ "content": "841991688488",
173
+ "lstrip": false,
174
+ "normalized": false,
175
+ "rstrip": false,
176
+ "single_word": false
177
+ },
178
+ {
179
+ "content": "211411365397",
180
+ "lstrip": false,
181
+ "normalized": false,
182
+ "rstrip": false,
183
+ "single_word": false
184
+ },
185
+ {
186
+ "content": "698218415444",
187
+ "lstrip": false,
188
+ "normalized": false,
189
+ "rstrip": false,
190
+ "single_word": false
191
+ },
192
+ {
193
+ "content": "355977139358",
194
+ "lstrip": false,
195
+ "normalized": false,
196
+ "rstrip": false,
197
+ "single_word": false
198
+ },
199
+ {
200
+ "content": "682564697312",
201
+ "lstrip": false,
202
+ "normalized": false,
203
+ "rstrip": false,
204
+ "single_word": false
205
+ },
206
+ {
207
+ "content": "383837596997",
208
+ "lstrip": false,
209
+ "normalized": false,
210
+ "rstrip": false,
211
+ "single_word": false
212
+ },
213
+ {
214
+ "content": "689362171782",
215
+ "lstrip": false,
216
+ "normalized": false,
217
+ "rstrip": false,
218
+ "single_word": false
219
+ },
220
+ {
221
+ "content": "749966767285",
222
+ "lstrip": false,
223
+ "normalized": false,
224
+ "rstrip": false,
225
+ "single_word": false
226
+ },
227
+ {
228
+ "content": "753159165157",
229
+ "lstrip": false,
230
+ "normalized": false,
231
+ "rstrip": false,
232
+ "single_word": false
233
+ },
234
+ {
235
+ "content": "795693824762",
236
+ "lstrip": false,
237
+ "normalized": false,
238
+ "rstrip": false,
239
+ "single_word": false
240
+ },
241
+ {
242
+ "content": "669689115557",
243
+ "lstrip": false,
244
+ "normalized": false,
245
+ "rstrip": false,
246
+ "single_word": false
247
+ },
248
+ {
249
+ "content": "327491773134",
250
+ "lstrip": false,
251
+ "normalized": false,
252
+ "rstrip": false,
253
+ "single_word": false
254
+ },
255
+ {
256
+ "content": "983569279932",
257
+ "lstrip": false,
258
+ "normalized": false,
259
+ "rstrip": false,
260
+ "single_word": false
261
+ },
262
+ {
263
+ "content": "612128769512",
264
+ "lstrip": false,
265
+ "normalized": false,
266
+ "rstrip": false,
267
+ "single_word": false
268
+ },
269
+ {
270
+ "content": "374327157578",
271
+ "lstrip": false,
272
+ "normalized": false,
273
+ "rstrip": false,
274
+ "single_word": false
275
+ },
276
+ {
277
+ "content": "311632789559",
278
+ "lstrip": false,
279
+ "normalized": false,
280
+ "rstrip": false,
281
+ "single_word": false
282
+ },
283
+ {
284
+ "content": "523918658846",
285
+ "lstrip": false,
286
+ "normalized": false,
287
+ "rstrip": false,
288
+ "single_word": false
289
+ },
290
+ {
291
+ "content": "765981581453",
292
+ "lstrip": false,
293
+ "normalized": false,
294
+ "rstrip": false,
295
+ "single_word": false
296
+ },
297
+ {
298
+ "content": "794825141891",
299
+ "lstrip": false,
300
+ "normalized": false,
301
+ "rstrip": false,
302
+ "single_word": false
303
+ },
304
+ {
305
+ "content": "873898736873",
306
+ "lstrip": false,
307
+ "normalized": false,
308
+ "rstrip": false,
309
+ "single_word": false
310
+ },
311
+ {
312
+ "content": "447445629421",
313
+ "lstrip": false,
314
+ "normalized": false,
315
+ "rstrip": false,
316
+ "single_word": false
317
+ },
318
+ {
319
+ "content": "473822473819",
320
+ "lstrip": false,
321
+ "normalized": false,
322
+ "rstrip": false,
323
+ "single_word": false
324
+ },
325
+ {
326
+ "content": "181439694557",
327
+ "lstrip": false,
328
+ "normalized": false,
329
+ "rstrip": false,
330
+ "single_word": false
331
+ },
332
+ {
333
+ "content": "592538279337",
334
+ "lstrip": false,
335
+ "normalized": false,
336
+ "rstrip": false,
337
+ "single_word": false
338
+ },
339
+ {
340
+ "content": "668134915514",
341
+ "lstrip": false,
342
+ "normalized": false,
343
+ "rstrip": false,
344
+ "single_word": false
345
+ },
346
+ {
347
+ "content": "643692393748",
348
+ "lstrip": false,
349
+ "normalized": false,
350
+ "rstrip": false,
351
+ "single_word": false
352
+ },
353
+ {
354
+ "content": "696651276628",
355
+ "lstrip": false,
356
+ "normalized": false,
357
+ "rstrip": false,
358
+ "single_word": false
359
+ },
360
+ {
361
+ "content": "853859348234",
362
+ "lstrip": false,
363
+ "normalized": false,
364
+ "rstrip": false,
365
+ "single_word": false
366
+ },
367
+ {
368
+ "content": "778466723723",
369
+ "lstrip": false,
370
+ "normalized": false,
371
+ "rstrip": false,
372
+ "single_word": false
373
+ },
374
+ {
375
+ "content": "929826356991",
376
+ "lstrip": false,
377
+ "normalized": false,
378
+ "rstrip": false,
379
+ "single_word": false
380
+ },
381
+ {
382
+ "content": "272362973463",
383
+ "lstrip": false,
384
+ "normalized": false,
385
+ "rstrip": false,
386
+ "single_word": false
387
+ },
388
+ {
389
+ "content": "694235616268",
390
+ "lstrip": false,
391
+ "normalized": false,
392
+ "rstrip": false,
393
+ "single_word": false
394
+ },
395
+ {
396
+ "content": "281673864127",
397
+ "lstrip": false,
398
+ "normalized": false,
399
+ "rstrip": false,
400
+ "single_word": false
401
+ },
402
+ {
403
+ "content": "479676316326",
404
+ "lstrip": false,
405
+ "normalized": false,
406
+ "rstrip": false,
407
+ "single_word": false
408
+ },
409
+ {
410
+ "content": "646979124677",
411
+ "lstrip": false,
412
+ "normalized": false,
413
+ "rstrip": false,
414
+ "single_word": false
415
+ },
416
+ {
417
+ "content": "922327493433",
418
+ "lstrip": false,
419
+ "normalized": false,
420
+ "rstrip": false,
421
+ "single_word": false
422
+ },
423
+ {
424
+ "content": "883685933161",
425
+ "lstrip": false,
426
+ "normalized": false,
427
+ "rstrip": false,
428
+ "single_word": false
429
+ },
430
+ {
431
+ "content": "264259917554",
432
+ "lstrip": false,
433
+ "normalized": false,
434
+ "rstrip": false,
435
+ "single_word": false
436
+ },
437
+ {
438
+ "content": "836746273134",
439
+ "lstrip": false,
440
+ "normalized": false,
441
+ "rstrip": false,
442
+ "single_word": false
443
+ },
444
+ {
445
+ "content": "658481324922",
446
+ "lstrip": false,
447
+ "normalized": false,
448
+ "rstrip": false,
449
+ "single_word": false
450
+ },
451
+ {
452
+ "content": "481884157827",
453
+ "lstrip": false,
454
+ "normalized": false,
455
+ "rstrip": false,
456
+ "single_word": false
457
+ },
458
+ {
459
+ "content": "587787496812",
460
+ "lstrip": false,
461
+ "normalized": false,
462
+ "rstrip": false,
463
+ "single_word": false
464
+ },
465
+ {
466
+ "content": "579184949249",
467
+ "lstrip": false,
468
+ "normalized": false,
469
+ "rstrip": false,
470
+ "single_word": false
471
+ },
472
+ {
473
+ "content": "912193598348",
474
+ "lstrip": false,
475
+ "normalized": false,
476
+ "rstrip": false,
477
+ "single_word": false
478
+ },
479
+ {
480
+ "content": "529679678956",
481
+ "lstrip": false,
482
+ "normalized": false,
483
+ "rstrip": false,
484
+ "single_word": false
485
+ },
486
+ {
487
+ "content": "795838284624",
488
+ "lstrip": false,
489
+ "normalized": false,
490
+ "rstrip": false,
491
+ "single_word": false
492
+ },
493
+ {
494
+ "content": "159337222655",
495
+ "lstrip": false,
496
+ "normalized": false,
497
+ "rstrip": false,
498
+ "single_word": false
499
+ },
500
+ {
501
+ "content": "173781362446",
502
+ "lstrip": false,
503
+ "normalized": false,
504
+ "rstrip": false,
505
+ "single_word": false
506
+ },
507
+ {
508
+ "content": "773687856563",
509
+ "lstrip": false,
510
+ "normalized": false,
511
+ "rstrip": false,
512
+ "single_word": false
513
+ },
514
+ {
515
+ "content": "535787224917",
516
+ "lstrip": false,
517
+ "normalized": false,
518
+ "rstrip": false,
519
+ "single_word": false
520
+ },
521
+ {
522
+ "content": "351885857332",
523
+ "lstrip": false,
524
+ "normalized": false,
525
+ "rstrip": false,
526
+ "single_word": false
527
+ },
528
+ {
529
+ "content": "578827344666",
530
+ "lstrip": false,
531
+ "normalized": false,
532
+ "rstrip": false,
533
+ "single_word": false
534
+ },
535
+ {
536
+ "content": "198462689911",
537
+ "lstrip": false,
538
+ "normalized": false,
539
+ "rstrip": false,
540
+ "single_word": false
541
+ },
542
+ {
543
+ "content": "722618266242",
544
+ "lstrip": false,
545
+ "normalized": false,
546
+ "rstrip": false,
547
+ "single_word": false
548
+ },
549
+ {
550
+ "content": "952872416512",
551
+ "lstrip": false,
552
+ "normalized": false,
553
+ "rstrip": false,
554
+ "single_word": false
555
+ },
556
+ {
557
+ "content": "517778845323",
558
+ "lstrip": false,
559
+ "normalized": false,
560
+ "rstrip": false,
561
+ "single_word": false
562
+ },
563
+ {
564
+ "content": "749665846687",
565
+ "lstrip": false,
566
+ "normalized": false,
567
+ "rstrip": false,
568
+ "single_word": false
569
+ },
570
+ {
571
+ "content": "661436365453",
572
+ "lstrip": false,
573
+ "normalized": false,
574
+ "rstrip": false,
575
+ "single_word": false
576
+ },
577
+ {
578
+ "content": "259666844669",
579
+ "lstrip": false,
580
+ "normalized": false,
581
+ "rstrip": false,
582
+ "single_word": false
583
+ },
584
+ {
585
+ "content": "242851284913",
586
+ "lstrip": false,
587
+ "normalized": false,
588
+ "rstrip": false,
589
+ "single_word": false
590
+ },
591
+ {
592
+ "content": "514532995959",
593
+ "lstrip": false,
594
+ "normalized": false,
595
+ "rstrip": false,
596
+ "single_word": false
597
+ },
598
+ {
599
+ "content": "161588262349",
600
+ "lstrip": false,
601
+ "normalized": false,
602
+ "rstrip": false,
603
+ "single_word": false
604
+ },
605
+ {
606
+ "content": "742765629356",
607
+ "lstrip": false,
608
+ "normalized": false,
609
+ "rstrip": false,
610
+ "single_word": false
611
+ },
612
+ {
613
+ "content": "225164373623",
614
+ "lstrip": false,
615
+ "normalized": false,
616
+ "rstrip": false,
617
+ "single_word": false
618
+ },
619
+ {
620
+ "content": "676539973863",
621
+ "lstrip": false,
622
+ "normalized": false,
623
+ "rstrip": false,
624
+ "single_word": false
625
+ },
626
+ {
627
+ "content": "826214551218",
628
+ "lstrip": false,
629
+ "normalized": false,
630
+ "rstrip": false,
631
+ "single_word": false
632
+ },
633
+ {
634
+ "content": "182345464792",
635
+ "lstrip": false,
636
+ "normalized": false,
637
+ "rstrip": false,
638
+ "single_word": false
639
+ },
640
+ {
641
+ "content": "232776999554",
642
+ "lstrip": false,
643
+ "normalized": false,
644
+ "rstrip": false,
645
+ "single_word": false
646
+ },
647
+ {
648
+ "content": "337326533813",
649
+ "lstrip": false,
650
+ "normalized": false,
651
+ "rstrip": false,
652
+ "single_word": false
653
+ },
654
+ {
655
+ "content": "676676697292",
656
+ "lstrip": false,
657
+ "normalized": false,
658
+ "rstrip": false,
659
+ "single_word": false
660
+ },
661
+ {
662
+ "content": "929185622831",
663
+ "lstrip": false,
664
+ "normalized": false,
665
+ "rstrip": false,
666
+ "single_word": false
667
+ },
668
+ {
669
+ "content": "545512344383",
670
+ "lstrip": false,
671
+ "normalized": false,
672
+ "rstrip": false,
673
+ "single_word": false
674
+ },
675
+ {
676
+ "content": "499444466686",
677
+ "lstrip": false,
678
+ "normalized": false,
679
+ "rstrip": false,
680
+ "single_word": false
681
+ },
682
+ {
683
+ "content": "314697386682",
684
+ "lstrip": false,
685
+ "normalized": false,
686
+ "rstrip": false,
687
+ "single_word": false
688
+ },
689
+ {
690
+ "content": "517379856925",
691
+ "lstrip": false,
692
+ "normalized": false,
693
+ "rstrip": false,
694
+ "single_word": false
695
+ },
696
+ {
697
+ "content": "379557332953",
698
+ "lstrip": false,
699
+ "normalized": false,
700
+ "rstrip": false,
701
+ "single_word": false
702
+ },
703
+ {
704
+ "content": "614797267726",
705
+ "lstrip": false,
706
+ "normalized": false,
707
+ "rstrip": false,
708
+ "single_word": false
709
+ },
710
+ {
711
+ "content": "429781429464",
712
+ "lstrip": false,
713
+ "normalized": false,
714
+ "rstrip": false,
715
+ "single_word": false
716
+ },
717
+ {
718
+ "content": "922466849763",
719
+ "lstrip": false,
720
+ "normalized": false,
721
+ "rstrip": false,
722
+ "single_word": false
723
+ },
724
+ {
725
+ "content": "721737645236",
726
+ "lstrip": false,
727
+ "normalized": false,
728
+ "rstrip": false,
729
+ "single_word": false
730
+ },
731
+ {
732
+ "content": "479227349997",
733
+ "lstrip": false,
734
+ "normalized": false,
735
+ "rstrip": false,
736
+ "single_word": false
737
+ },
738
+ {
739
+ "content": "136931728327",
740
+ "lstrip": false,
741
+ "normalized": false,
742
+ "rstrip": false,
743
+ "single_word": false
744
+ },
745
+ {
746
+ "content": "259533577263",
747
+ "lstrip": false,
748
+ "normalized": false,
749
+ "rstrip": false,
750
+ "single_word": false
751
+ },
752
+ {
753
+ "content": "488538864842",
754
+ "lstrip": false,
755
+ "normalized": false,
756
+ "rstrip": false,
757
+ "single_word": false
758
+ },
759
+ {
760
+ "content": "937495658852",
761
+ "lstrip": false,
762
+ "normalized": false,
763
+ "rstrip": false,
764
+ "single_word": false
765
+ },
766
+ {
767
+ "content": "489991411364",
768
+ "lstrip": false,
769
+ "normalized": false,
770
+ "rstrip": false,
771
+ "single_word": false
772
+ },
773
+ {
774
+ "content": "499148455254",
775
+ "lstrip": false,
776
+ "normalized": false,
777
+ "rstrip": false,
778
+ "single_word": false
779
+ },
780
+ {
781
+ "content": "441373944925",
782
+ "lstrip": false,
783
+ "normalized": false,
784
+ "rstrip": false,
785
+ "single_word": false
786
+ },
787
+ {
788
+ "content": "899151413682",
789
+ "lstrip": false,
790
+ "normalized": false,
791
+ "rstrip": false,
792
+ "single_word": false
793
+ },
794
+ {
795
+ "content": "467893531755",
796
+ "lstrip": false,
797
+ "normalized": false,
798
+ "rstrip": false,
799
+ "single_word": false
800
+ },
801
+ {
802
+ "content": "527117488925",
803
+ "lstrip": false,
804
+ "normalized": false,
805
+ "rstrip": false,
806
+ "single_word": false
807
+ },
808
+ {
809
+ "content": "928335588653",
810
+ "lstrip": false,
811
+ "normalized": false,
812
+ "rstrip": false,
813
+ "single_word": false
814
+ },
815
+ {
816
+ "content": "374439448821",
817
+ "lstrip": false,
818
+ "normalized": false,
819
+ "rstrip": false,
820
+ "single_word": false
821
+ },
822
+ {
823
+ "content": "879425227932",
824
+ "lstrip": false,
825
+ "normalized": false,
826
+ "rstrip": false,
827
+ "single_word": false
828
+ },
829
+ {
830
+ "content": "867678158885",
831
+ "lstrip": false,
832
+ "normalized": false,
833
+ "rstrip": false,
834
+ "single_word": false
835
+ },
836
+ {
837
+ "content": "399749397872",
838
+ "lstrip": false,
839
+ "normalized": false,
840
+ "rstrip": false,
841
+ "single_word": false
842
+ },
843
+ {
844
+ "content": "129693547287",
845
+ "lstrip": false,
846
+ "normalized": false,
847
+ "rstrip": false,
848
+ "single_word": false
849
+ },
850
+ {
851
+ "content": "689285841825",
852
+ "lstrip": false,
853
+ "normalized": false,
854
+ "rstrip": false,
855
+ "single_word": false
856
+ },
857
+ {
858
+ "content": "771619544974",
859
+ "lstrip": false,
860
+ "normalized": false,
861
+ "rstrip": false,
862
+ "single_word": false
863
+ },
864
+ {
865
+ "content": "724883568652",
866
+ "lstrip": false,
867
+ "normalized": false,
868
+ "rstrip": false,
869
+ "single_word": false
870
+ },
871
+ {
872
+ "content": "516968424863",
873
+ "lstrip": false,
874
+ "normalized": false,
875
+ "rstrip": false,
876
+ "single_word": false
877
+ },
878
+ {
879
+ "content": "733737988257",
880
+ "lstrip": false,
881
+ "normalized": false,
882
+ "rstrip": false,
883
+ "single_word": false
884
+ },
885
+ {
886
+ "content": "852347289392",
887
+ "lstrip": false,
888
+ "normalized": false,
889
+ "rstrip": false,
890
+ "single_word": false
891
+ },
892
+ {
893
+ "content": "296953381169",
894
+ "lstrip": false,
895
+ "normalized": false,
896
+ "rstrip": false,
897
+ "single_word": false
898
+ },
899
+ {
900
+ "content": "377273562477",
901
+ "lstrip": false,
902
+ "normalized": false,
903
+ "rstrip": false,
904
+ "single_word": false
905
+ },
906
+ {
907
+ "content": "262296912232",
908
+ "lstrip": false,
909
+ "normalized": false,
910
+ "rstrip": false,
911
+ "single_word": false
912
+ },
913
+ {
914
+ "content": "547149832394",
915
+ "lstrip": false,
916
+ "normalized": false,
917
+ "rstrip": false,
918
+ "single_word": false
919
+ },
920
+ {
921
+ "content": "298464134954",
922
+ "lstrip": false,
923
+ "normalized": false,
924
+ "rstrip": false,
925
+ "single_word": false
926
+ },
927
+ {
928
+ "content": "216667245274",
929
+ "lstrip": false,
930
+ "normalized": false,
931
+ "rstrip": false,
932
+ "single_word": false
933
+ },
934
+ {
935
+ "content": "843998562287",
936
+ "lstrip": false,
937
+ "normalized": false,
938
+ "rstrip": false,
939
+ "single_word": false
940
+ },
941
+ {
942
+ "content": "572154333646",
943
+ "lstrip": false,
944
+ "normalized": false,
945
+ "rstrip": false,
946
+ "single_word": false
947
+ },
948
+ {
949
+ "content": "124589118494",
950
+ "lstrip": false,
951
+ "normalized": false,
952
+ "rstrip": false,
953
+ "single_word": false
954
+ },
955
+ {
956
+ "content": "841824384614",
957
+ "lstrip": false,
958
+ "normalized": false,
959
+ "rstrip": false,
960
+ "single_word": false
961
+ },
962
+ {
963
+ "content": "232896526252",
964
+ "lstrip": false,
965
+ "normalized": false,
966
+ "rstrip": false,
967
+ "single_word": false
968
+ },
969
+ {
970
+ "content": "295448593321",
971
+ "lstrip": false,
972
+ "normalized": false,
973
+ "rstrip": false,
974
+ "single_word": false
975
+ },
976
+ {
977
+ "content": "123741461297",
978
+ "lstrip": false,
979
+ "normalized": false,
980
+ "rstrip": false,
981
+ "single_word": false
982
+ },
983
+ {
984
+ "content": "653573457168",
985
+ "lstrip": false,
986
+ "normalized": false,
987
+ "rstrip": false,
988
+ "single_word": false
989
+ },
990
+ {
991
+ "content": "196735786156",
992
+ "lstrip": false,
993
+ "normalized": false,
994
+ "rstrip": false,
995
+ "single_word": false
996
+ },
997
+ {
998
+ "content": "377338713663",
999
+ "lstrip": false,
1000
+ "normalized": false,
1001
+ "rstrip": false,
1002
+ "single_word": false
1003
+ },
1004
+ {
1005
+ "content": "964342468552",
1006
+ "lstrip": false,
1007
+ "normalized": false,
1008
+ "rstrip": false,
1009
+ "single_word": false
1010
+ },
1011
+ {
1012
+ "content": "586855179568",
1013
+ "lstrip": false,
1014
+ "normalized": false,
1015
+ "rstrip": false,
1016
+ "single_word": false
1017
+ },
1018
+ {
1019
+ "content": "484773717614",
1020
+ "lstrip": false,
1021
+ "normalized": false,
1022
+ "rstrip": false,
1023
+ "single_word": false
1024
+ },
1025
+ {
1026
+ "content": "894885246797",
1027
+ "lstrip": false,
1028
+ "normalized": false,
1029
+ "rstrip": false,
1030
+ "single_word": false
1031
+ },
1032
+ {
1033
+ "content": "677896358599",
1034
+ "lstrip": false,
1035
+ "normalized": false,
1036
+ "rstrip": false,
1037
+ "single_word": false
1038
+ },
1039
+ {
1040
+ "content": "848845611563",
1041
+ "lstrip": false,
1042
+ "normalized": false,
1043
+ "rstrip": false,
1044
+ "single_word": false
1045
+ },
1046
+ {
1047
+ "content": "851852651677",
1048
+ "lstrip": false,
1049
+ "normalized": false,
1050
+ "rstrip": false,
1051
+ "single_word": false
1052
+ },
1053
+ {
1054
+ "content": "398549545767",
1055
+ "lstrip": false,
1056
+ "normalized": false,
1057
+ "rstrip": false,
1058
+ "single_word": false
1059
+ },
1060
+ {
1061
+ "content": "454244839926",
1062
+ "lstrip": false,
1063
+ "normalized": false,
1064
+ "rstrip": false,
1065
+ "single_word": false
1066
+ },
1067
+ {
1068
+ "content": "799364566435",
1069
+ "lstrip": false,
1070
+ "normalized": false,
1071
+ "rstrip": false,
1072
+ "single_word": false
1073
+ },
1074
+ {
1075
+ "content": "967114116556",
1076
+ "lstrip": false,
1077
+ "normalized": false,
1078
+ "rstrip": false,
1079
+ "single_word": false
1080
+ },
1081
+ {
1082
+ "content": "817378986438",
1083
+ "lstrip": false,
1084
+ "normalized": false,
1085
+ "rstrip": false,
1086
+ "single_word": false
1087
+ },
1088
+ {
1089
+ "content": "233795848681",
1090
+ "lstrip": false,
1091
+ "normalized": false,
1092
+ "rstrip": false,
1093
+ "single_word": false
1094
+ },
1095
+ {
1096
+ "content": "824387273757",
1097
+ "lstrip": false,
1098
+ "normalized": false,
1099
+ "rstrip": false,
1100
+ "single_word": false
1101
+ },
1102
+ {
1103
+ "content": "916198946615",
1104
+ "lstrip": false,
1105
+ "normalized": false,
1106
+ "rstrip": false,
1107
+ "single_word": false
1108
+ },
1109
+ {
1110
+ "content": "563117729724",
1111
+ "lstrip": false,
1112
+ "normalized": false,
1113
+ "rstrip": false,
1114
+ "single_word": false
1115
+ },
1116
+ {
1117
+ "content": "951794811935",
1118
+ "lstrip": false,
1119
+ "normalized": false,
1120
+ "rstrip": false,
1121
+ "single_word": false
1122
+ },
1123
+ {
1124
+ "content": "374598961236",
1125
+ "lstrip": false,
1126
+ "normalized": false,
1127
+ "rstrip": false,
1128
+ "single_word": false
1129
+ },
1130
+ {
1131
+ "content": "922867396683",
1132
+ "lstrip": false,
1133
+ "normalized": false,
1134
+ "rstrip": false,
1135
+ "single_word": false
1136
+ },
1137
+ {
1138
+ "content": "765737843639",
1139
+ "lstrip": false,
1140
+ "normalized": false,
1141
+ "rstrip": false,
1142
+ "single_word": false
1143
+ },
1144
+ {
1145
+ "content": "175469284871",
1146
+ "lstrip": false,
1147
+ "normalized": false,
1148
+ "rstrip": false,
1149
+ "single_word": false
1150
+ },
1151
+ {
1152
+ "content": "231853711778",
1153
+ "lstrip": false,
1154
+ "normalized": false,
1155
+ "rstrip": false,
1156
+ "single_word": false
1157
+ },
1158
+ {
1159
+ "content": "662426712668",
1160
+ "lstrip": false,
1161
+ "normalized": false,
1162
+ "rstrip": false,
1163
+ "single_word": false
1164
+ },
1165
+ {
1166
+ "content": "711412347158",
1167
+ "lstrip": false,
1168
+ "normalized": false,
1169
+ "rstrip": false,
1170
+ "single_word": false
1171
+ },
1172
+ {
1173
+ "content": "753466987363",
1174
+ "lstrip": false,
1175
+ "normalized": false,
1176
+ "rstrip": false,
1177
+ "single_word": false
1178
+ },
1179
+ {
1180
+ "content": "513361312532",
1181
+ "lstrip": false,
1182
+ "normalized": false,
1183
+ "rstrip": false,
1184
+ "single_word": false
1185
+ },
1186
+ {
1187
+ "content": "712992815957",
1188
+ "lstrip": false,
1189
+ "normalized": false,
1190
+ "rstrip": false,
1191
+ "single_word": false
1192
+ },
1193
+ {
1194
+ "content": "971621888444",
1195
+ "lstrip": false,
1196
+ "normalized": false,
1197
+ "rstrip": false,
1198
+ "single_word": false
1199
+ },
1200
+ {
1201
+ "content": "829235161526",
1202
+ "lstrip": false,
1203
+ "normalized": false,
1204
+ "rstrip": false,
1205
+ "single_word": false
1206
+ },
1207
+ {
1208
+ "content": "585544633356",
1209
+ "lstrip": false,
1210
+ "normalized": false,
1211
+ "rstrip": false,
1212
+ "single_word": false
1213
+ },
1214
+ {
1215
+ "content": "582471228164",
1216
+ "lstrip": false,
1217
+ "normalized": false,
1218
+ "rstrip": false,
1219
+ "single_word": false
1220
+ },
1221
+ {
1222
+ "content": "678666359123",
1223
+ "lstrip": false,
1224
+ "normalized": false,
1225
+ "rstrip": false,
1226
+ "single_word": false
1227
+ },
1228
+ {
1229
+ "content": "557533689478",
1230
+ "lstrip": false,
1231
+ "normalized": false,
1232
+ "rstrip": false,
1233
+ "single_word": false
1234
+ },
1235
+ {
1236
+ "content": "632962475133",
1237
+ "lstrip": false,
1238
+ "normalized": false,
1239
+ "rstrip": false,
1240
+ "single_word": false
1241
+ },
1242
+ {
1243
+ "content": "484489193824",
1244
+ "lstrip": false,
1245
+ "normalized": false,
1246
+ "rstrip": false,
1247
+ "single_word": false
1248
+ },
1249
+ {
1250
+ "content": "489562189822",
1251
+ "lstrip": false,
1252
+ "normalized": false,
1253
+ "rstrip": false,
1254
+ "single_word": false
1255
+ },
1256
+ {
1257
+ "content": "589547936288",
1258
+ "lstrip": false,
1259
+ "normalized": false,
1260
+ "rstrip": false,
1261
+ "single_word": false
1262
+ },
1263
+ {
1264
+ "content": "363214487524",
1265
+ "lstrip": false,
1266
+ "normalized": false,
1267
+ "rstrip": false,
1268
+ "single_word": false
1269
+ },
1270
+ {
1271
+ "content": "244885399387",
1272
+ "lstrip": false,
1273
+ "normalized": false,
1274
+ "rstrip": false,
1275
+ "single_word": false
1276
+ },
1277
+ {
1278
+ "content": "431751228368",
1279
+ "lstrip": false,
1280
+ "normalized": false,
1281
+ "rstrip": false,
1282
+ "single_word": false
1283
+ },
1284
+ {
1285
+ "content": "433581868192",
1286
+ "lstrip": false,
1287
+ "normalized": false,
1288
+ "rstrip": false,
1289
+ "single_word": false
1290
+ },
1291
+ {
1292
+ "content": "486391569221",
1293
+ "lstrip": false,
1294
+ "normalized": false,
1295
+ "rstrip": false,
1296
+ "single_word": false
1297
+ },
1298
+ {
1299
+ "content": "185438575221",
1300
+ "lstrip": false,
1301
+ "normalized": false,
1302
+ "rstrip": false,
1303
+ "single_word": false
1304
+ },
1305
+ {
1306
+ "content": "126574388585",
1307
+ "lstrip": false,
1308
+ "normalized": false,
1309
+ "rstrip": false,
1310
+ "single_word": false
1311
+ },
1312
+ {
1313
+ "content": "741757479784",
1314
+ "lstrip": false,
1315
+ "normalized": false,
1316
+ "rstrip": false,
1317
+ "single_word": false
1318
+ },
1319
+ {
1320
+ "content": "529854679937",
1321
+ "lstrip": false,
1322
+ "normalized": false,
1323
+ "rstrip": false,
1324
+ "single_word": false
1325
+ },
1326
+ {
1327
+ "content": "996116119839",
1328
+ "lstrip": false,
1329
+ "normalized": false,
1330
+ "rstrip": false,
1331
+ "single_word": false
1332
+ },
1333
+ {
1334
+ "content": "616248973917",
1335
+ "lstrip": false,
1336
+ "normalized": false,
1337
+ "rstrip": false,
1338
+ "single_word": false
1339
+ },
1340
+ {
1341
+ "content": "763531783491",
1342
+ "lstrip": false,
1343
+ "normalized": false,
1344
+ "rstrip": false,
1345
+ "single_word": false
1346
+ },
1347
+ {
1348
+ "content": "955456118295",
1349
+ "lstrip": false,
1350
+ "normalized": false,
1351
+ "rstrip": false,
1352
+ "single_word": false
1353
+ },
1354
+ {
1355
+ "content": "364196983365",
1356
+ "lstrip": false,
1357
+ "normalized": false,
1358
+ "rstrip": false,
1359
+ "single_word": false
1360
+ },
1361
+ {
1362
+ "content": "195792996468",
1363
+ "lstrip": false,
1364
+ "normalized": false,
1365
+ "rstrip": false,
1366
+ "single_word": false
1367
+ },
1368
+ {
1369
+ "content": "151859598873",
1370
+ "lstrip": false,
1371
+ "normalized": false,
1372
+ "rstrip": false,
1373
+ "single_word": false
1374
+ },
1375
+ {
1376
+ "content": "399223169721",
1377
+ "lstrip": false,
1378
+ "normalized": false,
1379
+ "rstrip": false,
1380
+ "single_word": false
1381
+ },
1382
+ {
1383
+ "content": "938488813964",
1384
+ "lstrip": false,
1385
+ "normalized": false,
1386
+ "rstrip": false,
1387
+ "single_word": false
1388
+ },
1389
+ {
1390
+ "content": "961981959227",
1391
+ "lstrip": false,
1392
+ "normalized": false,
1393
+ "rstrip": false,
1394
+ "single_word": false
1395
+ },
1396
+ {
1397
+ "content": "183368827562",
1398
+ "lstrip": false,
1399
+ "normalized": false,
1400
+ "rstrip": false,
1401
+ "single_word": false
1402
+ },
1403
+ {
1404
+ "content": "533417736566",
1405
+ "lstrip": false,
1406
+ "normalized": false,
1407
+ "rstrip": false,
1408
+ "single_word": false
1409
+ },
1410
+ {
1411
+ "content": "786391632558",
1412
+ "lstrip": false,
1413
+ "normalized": false,
1414
+ "rstrip": false,
1415
+ "single_word": false
1416
+ },
1417
+ {
1418
+ "content": "665661658354",
1419
+ "lstrip": false,
1420
+ "normalized": false,
1421
+ "rstrip": false,
1422
+ "single_word": false
1423
+ },
1424
+ {
1425
+ "content": "693281533643",
1426
+ "lstrip": false,
1427
+ "normalized": false,
1428
+ "rstrip": false,
1429
+ "single_word": false
1430
+ },
1431
+ {
1432
+ "content": "475794684356",
1433
+ "lstrip": false,
1434
+ "normalized": false,
1435
+ "rstrip": false,
1436
+ "single_word": false
1437
+ },
1438
+ {
1439
+ "content": "652154162978",
1440
+ "lstrip": false,
1441
+ "normalized": false,
1442
+ "rstrip": false,
1443
+ "single_word": false
1444
+ },
1445
+ {
1446
+ "content": "753233719644",
1447
+ "lstrip": false,
1448
+ "normalized": false,
1449
+ "rstrip": false,
1450
+ "single_word": false
1451
+ },
1452
+ {
1453
+ "content": "668514843129",
1454
+ "lstrip": false,
1455
+ "normalized": false,
1456
+ "rstrip": false,
1457
+ "single_word": false
1458
+ },
1459
+ {
1460
+ "content": "819162623892",
1461
+ "lstrip": false,
1462
+ "normalized": false,
1463
+ "rstrip": false,
1464
+ "single_word": false
1465
+ },
1466
+ {
1467
+ "content": "941169431859",
1468
+ "lstrip": false,
1469
+ "normalized": false,
1470
+ "rstrip": false,
1471
+ "single_word": false
1472
+ },
1473
+ {
1474
+ "content": "877385381798",
1475
+ "lstrip": false,
1476
+ "normalized": false,
1477
+ "rstrip": false,
1478
+ "single_word": false
1479
+ },
1480
+ {
1481
+ "content": "752644929761",
1482
+ "lstrip": false,
1483
+ "normalized": false,
1484
+ "rstrip": false,
1485
+ "single_word": false
1486
+ },
1487
+ {
1488
+ "content": "881136466196",
1489
+ "lstrip": false,
1490
+ "normalized": false,
1491
+ "rstrip": false,
1492
+ "single_word": false
1493
+ },
1494
+ {
1495
+ "content": "275597777299",
1496
+ "lstrip": false,
1497
+ "normalized": false,
1498
+ "rstrip": false,
1499
+ "single_word": false
1500
+ },
1501
+ {
1502
+ "content": "731681792655",
1503
+ "lstrip": false,
1504
+ "normalized": false,
1505
+ "rstrip": false,
1506
+ "single_word": false
1507
+ },
1508
+ {
1509
+ "content": "961133895172",
1510
+ "lstrip": false,
1511
+ "normalized": false,
1512
+ "rstrip": false,
1513
+ "single_word": false
1514
+ },
1515
+ {
1516
+ "content": "864718285734",
1517
+ "lstrip": false,
1518
+ "normalized": false,
1519
+ "rstrip": false,
1520
+ "single_word": false
1521
+ },
1522
+ {
1523
+ "content": "963852916563",
1524
+ "lstrip": false,
1525
+ "normalized": false,
1526
+ "rstrip": false,
1527
+ "single_word": false
1528
+ },
1529
+ {
1530
+ "content": "319584985416",
1531
+ "lstrip": false,
1532
+ "normalized": false,
1533
+ "rstrip": false,
1534
+ "single_word": false
1535
+ },
1536
+ {
1537
+ "content": "563365646341",
1538
+ "lstrip": false,
1539
+ "normalized": false,
1540
+ "rstrip": false,
1541
+ "single_word": false
1542
+ },
1543
+ {
1544
+ "content": "811371928234",
1545
+ "lstrip": false,
1546
+ "normalized": false,
1547
+ "rstrip": false,
1548
+ "single_word": false
1549
+ },
1550
+ {
1551
+ "content": "837131396371",
1552
+ "lstrip": false,
1553
+ "normalized": false,
1554
+ "rstrip": false,
1555
+ "single_word": false
1556
+ },
1557
+ {
1558
+ "content": "267514771964",
1559
+ "lstrip": false,
1560
+ "normalized": false,
1561
+ "rstrip": false,
1562
+ "single_word": false
1563
+ },
1564
+ {
1565
+ "content": "944513428457",
1566
+ "lstrip": false,
1567
+ "normalized": false,
1568
+ "rstrip": false,
1569
+ "single_word": false
1570
+ },
1571
+ {
1572
+ "content": "117298239631",
1573
+ "lstrip": false,
1574
+ "normalized": false,
1575
+ "rstrip": false,
1576
+ "single_word": false
1577
+ },
1578
+ {
1579
+ "content": "158142752582",
1580
+ "lstrip": false,
1581
+ "normalized": false,
1582
+ "rstrip": false,
1583
+ "single_word": false
1584
+ },
1585
+ {
1586
+ "content": "252867443568",
1587
+ "lstrip": false,
1588
+ "normalized": false,
1589
+ "rstrip": false,
1590
+ "single_word": false
1591
+ },
1592
+ {
1593
+ "content": "839269684865",
1594
+ "lstrip": false,
1595
+ "normalized": false,
1596
+ "rstrip": false,
1597
+ "single_word": false
1598
+ },
1599
+ {
1600
+ "content": "612788593128",
1601
+ "lstrip": false,
1602
+ "normalized": false,
1603
+ "rstrip": false,
1604
+ "single_word": false
1605
+ },
1606
+ {
1607
+ "content": "145669731981",
1608
+ "lstrip": false,
1609
+ "normalized": false,
1610
+ "rstrip": false,
1611
+ "single_word": false
1612
+ },
1613
+ {
1614
+ "content": "121557291859",
1615
+ "lstrip": false,
1616
+ "normalized": false,
1617
+ "rstrip": false,
1618
+ "single_word": false
1619
+ },
1620
+ {
1621
+ "content": "245416776926",
1622
+ "lstrip": false,
1623
+ "normalized": false,
1624
+ "rstrip": false,
1625
+ "single_word": false
1626
+ },
1627
+ {
1628
+ "content": "799417897197",
1629
+ "lstrip": false,
1630
+ "normalized": false,
1631
+ "rstrip": false,
1632
+ "single_word": false
1633
+ },
1634
+ {
1635
+ "content": "997958836435",
1636
+ "lstrip": false,
1637
+ "normalized": false,
1638
+ "rstrip": false,
1639
+ "single_word": false
1640
+ },
1641
+ {
1642
+ "content": "892336777248",
1643
+ "lstrip": false,
1644
+ "normalized": false,
1645
+ "rstrip": false,
1646
+ "single_word": false
1647
+ },
1648
+ {
1649
+ "content": "158929292238",
1650
+ "lstrip": false,
1651
+ "normalized": false,
1652
+ "rstrip": false,
1653
+ "single_word": false
1654
+ },
1655
+ {
1656
+ "content": "581976444672",
1657
+ "lstrip": false,
1658
+ "normalized": false,
1659
+ "rstrip": false,
1660
+ "single_word": false
1661
+ },
1662
+ {
1663
+ "content": "897784492783",
1664
+ "lstrip": false,
1665
+ "normalized": false,
1666
+ "rstrip": false,
1667
+ "single_word": false
1668
+ },
1669
+ {
1670
+ "content": "492373714791",
1671
+ "lstrip": false,
1672
+ "normalized": false,
1673
+ "rstrip": false,
1674
+ "single_word": false
1675
+ },
1676
+ {
1677
+ "content": "512659818733",
1678
+ "lstrip": false,
1679
+ "normalized": false,
1680
+ "rstrip": false,
1681
+ "single_word": false
1682
+ },
1683
+ {
1684
+ "content": "881112998642",
1685
+ "lstrip": false,
1686
+ "normalized": false,
1687
+ "rstrip": false,
1688
+ "single_word": false
1689
+ },
1690
+ {
1691
+ "content": "619454958782",
1692
+ "lstrip": false,
1693
+ "normalized": false,
1694
+ "rstrip": false,
1695
+ "single_word": false
1696
+ },
1697
+ {
1698
+ "content": "431149748713",
1699
+ "lstrip": false,
1700
+ "normalized": false,
1701
+ "rstrip": false,
1702
+ "single_word": false
1703
+ },
1704
+ {
1705
+ "content": "624221476921",
1706
+ "lstrip": false,
1707
+ "normalized": false,
1708
+ "rstrip": false,
1709
+ "single_word": false
1710
+ },
1711
+ {
1712
+ "content": "125866399464",
1713
+ "lstrip": false,
1714
+ "normalized": false,
1715
+ "rstrip": false,
1716
+ "single_word": false
1717
+ },
1718
+ {
1719
+ "content": "339882449689",
1720
+ "lstrip": false,
1721
+ "normalized": false,
1722
+ "rstrip": false,
1723
+ "single_word": false
1724
+ },
1725
+ {
1726
+ "content": "186198784585",
1727
+ "lstrip": false,
1728
+ "normalized": false,
1729
+ "rstrip": false,
1730
+ "single_word": false
1731
+ },
1732
+ {
1733
+ "content": "943193294691",
1734
+ "lstrip": false,
1735
+ "normalized": false,
1736
+ "rstrip": false,
1737
+ "single_word": false
1738
+ },
1739
+ {
1740
+ "content": "955668961269",
1741
+ "lstrip": false,
1742
+ "normalized": false,
1743
+ "rstrip": false,
1744
+ "single_word": false
1745
+ },
1746
+ {
1747
+ "content": "232787996724",
1748
+ "lstrip": false,
1749
+ "normalized": false,
1750
+ "rstrip": false,
1751
+ "single_word": false
1752
+ },
1753
+ {
1754
+ "content": "215671314196",
1755
+ "lstrip": false,
1756
+ "normalized": false,
1757
+ "rstrip": false,
1758
+ "single_word": false
1759
+ },
1760
+ {
1761
+ "content": "286173241916",
1762
+ "lstrip": false,
1763
+ "normalized": false,
1764
+ "rstrip": false,
1765
+ "single_word": false
1766
+ },
1767
+ {
1768
+ "content": "745977673725",
1769
+ "lstrip": false,
1770
+ "normalized": false,
1771
+ "rstrip": false,
1772
+ "single_word": false
1773
+ },
1774
+ {
1775
+ "content": "556976448182",
1776
+ "lstrip": false,
1777
+ "normalized": false,
1778
+ "rstrip": false,
1779
+ "single_word": false
1780
+ },
1781
+ {
1782
+ "content": "599961512792",
1783
+ "lstrip": false,
1784
+ "normalized": false,
1785
+ "rstrip": false,
1786
+ "single_word": false
1787
+ },
1788
+ {
1789
+ "content": "766294538337",
1790
+ "lstrip": false,
1791
+ "normalized": false,
1792
+ "rstrip": false,
1793
+ "single_word": false
1794
+ },
1795
+ {
1796
+ "content": "934912591213",
1797
+ "lstrip": false,
1798
+ "normalized": false,
1799
+ "rstrip": false,
1800
+ "single_word": false
1801
+ },
1802
+ {
1803
+ "content": "295118729589",
1804
+ "lstrip": false,
1805
+ "normalized": false,
1806
+ "rstrip": false,
1807
+ "single_word": false
1808
+ },
1809
+ {
1810
+ "content": "529455466433",
1811
+ "lstrip": false,
1812
+ "normalized": false,
1813
+ "rstrip": false,
1814
+ "single_word": false
1815
+ },
1816
+ {
1817
+ "content": "196119929397",
1818
+ "lstrip": false,
1819
+ "normalized": false,
1820
+ "rstrip": false,
1821
+ "single_word": false
1822
+ },
1823
+ {
1824
+ "content": "379571934299",
1825
+ "lstrip": false,
1826
+ "normalized": false,
1827
+ "rstrip": false,
1828
+ "single_word": false
1829
+ },
1830
+ {
1831
+ "content": "251789649997",
1832
+ "lstrip": false,
1833
+ "normalized": false,
1834
+ "rstrip": false,
1835
+ "single_word": false
1836
+ },
1837
+ {
1838
+ "content": "564544131355",
1839
+ "lstrip": false,
1840
+ "normalized": false,
1841
+ "rstrip": false,
1842
+ "single_word": false
1843
+ },
1844
+ {
1845
+ "content": "244371196654",
1846
+ "lstrip": false,
1847
+ "normalized": false,
1848
+ "rstrip": false,
1849
+ "single_word": false
1850
+ },
1851
+ {
1852
+ "content": "384598329253",
1853
+ "lstrip": false,
1854
+ "normalized": false,
1855
+ "rstrip": false,
1856
+ "single_word": false
1857
+ },
1858
+ {
1859
+ "content": "887753195844",
1860
+ "lstrip": false,
1861
+ "normalized": false,
1862
+ "rstrip": false,
1863
+ "single_word": false
1864
+ },
1865
+ {
1866
+ "content": "364947325679",
1867
+ "lstrip": false,
1868
+ "normalized": false,
1869
+ "rstrip": false,
1870
+ "single_word": false
1871
+ },
1872
+ {
1873
+ "content": "655517954651",
1874
+ "lstrip": false,
1875
+ "normalized": false,
1876
+ "rstrip": false,
1877
+ "single_word": false
1878
+ },
1879
+ {
1880
+ "content": "673948786567",
1881
+ "lstrip": false,
1882
+ "normalized": false,
1883
+ "rstrip": false,
1884
+ "single_word": false
1885
+ },
1886
+ {
1887
+ "content": "857231548835",
1888
+ "lstrip": false,
1889
+ "normalized": false,
1890
+ "rstrip": false,
1891
+ "single_word": false
1892
+ },
1893
+ {
1894
+ "content": "816115936673",
1895
+ "lstrip": false,
1896
+ "normalized": false,
1897
+ "rstrip": false,
1898
+ "single_word": false
1899
+ },
1900
+ {
1901
+ "content": "644234165531",
1902
+ "lstrip": false,
1903
+ "normalized": false,
1904
+ "rstrip": false,
1905
+ "single_word": false
1906
+ },
1907
+ {
1908
+ "content": "182782912224",
1909
+ "lstrip": false,
1910
+ "normalized": false,
1911
+ "rstrip": false,
1912
+ "single_word": false
1913
+ },
1914
+ {
1915
+ "content": "234316622259",
1916
+ "lstrip": false,
1917
+ "normalized": false,
1918
+ "rstrip": false,
1919
+ "single_word": false
1920
+ },
1921
+ {
1922
+ "content": "421369185549",
1923
+ "lstrip": false,
1924
+ "normalized": false,
1925
+ "rstrip": false,
1926
+ "single_word": false
1927
+ },
1928
+ {
1929
+ "content": "434632855397",
1930
+ "lstrip": false,
1931
+ "normalized": false,
1932
+ "rstrip": false,
1933
+ "single_word": false
1934
+ },
1935
+ {
1936
+ "content": "921889371893",
1937
+ "lstrip": false,
1938
+ "normalized": false,
1939
+ "rstrip": false,
1940
+ "single_word": false
1941
+ },
1942
+ {
1943
+ "content": "415956914763",
1944
+ "lstrip": false,
1945
+ "normalized": false,
1946
+ "rstrip": false,
1947
+ "single_word": false
1948
+ },
1949
+ {
1950
+ "content": "598916996413",
1951
+ "lstrip": false,
1952
+ "normalized": false,
1953
+ "rstrip": false,
1954
+ "single_word": false
1955
+ },
1956
+ {
1957
+ "content": "773671349113",
1958
+ "lstrip": false,
1959
+ "normalized": false,
1960
+ "rstrip": false,
1961
+ "single_word": false
1962
+ },
1963
+ {
1964
+ "content": "952465217972",
1965
+ "lstrip": false,
1966
+ "normalized": false,
1967
+ "rstrip": false,
1968
+ "single_word": false
1969
+ },
1970
+ {
1971
+ "content": "117657531962",
1972
+ "lstrip": false,
1973
+ "normalized": false,
1974
+ "rstrip": false,
1975
+ "single_word": false
1976
+ },
1977
+ {
1978
+ "content": "729825168745",
1979
+ "lstrip": false,
1980
+ "normalized": false,
1981
+ "rstrip": false,
1982
+ "single_word": false
1983
+ },
1984
+ {
1985
+ "content": "691315125346",
1986
+ "lstrip": false,
1987
+ "normalized": false,
1988
+ "rstrip": false,
1989
+ "single_word": false
1990
+ },
1991
+ {
1992
+ "content": "768461952319",
1993
+ "lstrip": false,
1994
+ "normalized": false,
1995
+ "rstrip": false,
1996
+ "single_word": false
1997
+ },
1998
+ {
1999
+ "content": "664847713559",
2000
+ "lstrip": false,
2001
+ "normalized": false,
2002
+ "rstrip": false,
2003
+ "single_word": false
2004
+ },
2005
+ {
2006
+ "content": "953267689786",
2007
+ "lstrip": false,
2008
+ "normalized": false,
2009
+ "rstrip": false,
2010
+ "single_word": false
2011
+ },
2012
+ {
2013
+ "content": "886464195129",
2014
+ "lstrip": false,
2015
+ "normalized": false,
2016
+ "rstrip": false,
2017
+ "single_word": false
2018
+ },
2019
+ {
2020
+ "content": "824488329416",
2021
+ "lstrip": false,
2022
+ "normalized": false,
2023
+ "rstrip": false,
2024
+ "single_word": false
2025
+ },
2026
+ {
2027
+ "content": "837873762491",
2028
+ "lstrip": false,
2029
+ "normalized": false,
2030
+ "rstrip": false,
2031
+ "single_word": false
2032
+ },
2033
+ {
2034
+ "content": "532833541879",
2035
+ "lstrip": false,
2036
+ "normalized": false,
2037
+ "rstrip": false,
2038
+ "single_word": false
2039
+ },
2040
+ {
2041
+ "content": "669183782449",
2042
+ "lstrip": false,
2043
+ "normalized": false,
2044
+ "rstrip": false,
2045
+ "single_word": false
2046
+ },
2047
+ {
2048
+ "content": "941976537588",
2049
+ "lstrip": false,
2050
+ "normalized": false,
2051
+ "rstrip": false,
2052
+ "single_word": false
2053
+ },
2054
+ {
2055
+ "content": "739394546916",
2056
+ "lstrip": false,
2057
+ "normalized": false,
2058
+ "rstrip": false,
2059
+ "single_word": false
2060
+ },
2061
+ {
2062
+ "content": "267954879268",
2063
+ "lstrip": false,
2064
+ "normalized": false,
2065
+ "rstrip": false,
2066
+ "single_word": false
2067
+ },
2068
+ {
2069
+ "content": "637551427887",
2070
+ "lstrip": false,
2071
+ "normalized": false,
2072
+ "rstrip": false,
2073
+ "single_word": false
2074
+ },
2075
+ {
2076
+ "content": "217756494954",
2077
+ "lstrip": false,
2078
+ "normalized": false,
2079
+ "rstrip": false,
2080
+ "single_word": false
2081
+ },
2082
+ {
2083
+ "content": "524444658383",
2084
+ "lstrip": false,
2085
+ "normalized": false,
2086
+ "rstrip": false,
2087
+ "single_word": false
2088
+ },
2089
+ {
2090
+ "content": "117783274348",
2091
+ "lstrip": false,
2092
+ "normalized": false,
2093
+ "rstrip": false,
2094
+ "single_word": false
2095
+ },
2096
+ {
2097
+ "content": "138218735276",
2098
+ "lstrip": false,
2099
+ "normalized": false,
2100
+ "rstrip": false,
2101
+ "single_word": false
2102
+ },
2103
+ {
2104
+ "content": "814611949491",
2105
+ "lstrip": false,
2106
+ "normalized": false,
2107
+ "rstrip": false,
2108
+ "single_word": false
2109
+ },
2110
+ {
2111
+ "content": "711641973413",
2112
+ "lstrip": false,
2113
+ "normalized": false,
2114
+ "rstrip": false,
2115
+ "single_word": false
2116
+ },
2117
+ {
2118
+ "content": "499156317423",
2119
+ "lstrip": false,
2120
+ "normalized": false,
2121
+ "rstrip": false,
2122
+ "single_word": false
2123
+ },
2124
+ {
2125
+ "content": "515856611931",
2126
+ "lstrip": false,
2127
+ "normalized": false,
2128
+ "rstrip": false,
2129
+ "single_word": false
2130
+ },
2131
+ {
2132
+ "content": "454164859837",
2133
+ "lstrip": false,
2134
+ "normalized": false,
2135
+ "rstrip": false,
2136
+ "single_word": false
2137
+ },
2138
+ {
2139
+ "content": "345271433112",
2140
+ "lstrip": false,
2141
+ "normalized": false,
2142
+ "rstrip": false,
2143
+ "single_word": false
2144
+ },
2145
+ {
2146
+ "content": "462294118988",
2147
+ "lstrip": false,
2148
+ "normalized": false,
2149
+ "rstrip": false,
2150
+ "single_word": false
2151
+ },
2152
+ {
2153
+ "content": "511785788222",
2154
+ "lstrip": false,
2155
+ "normalized": false,
2156
+ "rstrip": false,
2157
+ "single_word": false
2158
+ },
2159
+ {
2160
+ "content": "497294727353",
2161
+ "lstrip": false,
2162
+ "normalized": false,
2163
+ "rstrip": false,
2164
+ "single_word": false
2165
+ },
2166
+ {
2167
+ "content": "866519986723",
2168
+ "lstrip": false,
2169
+ "normalized": false,
2170
+ "rstrip": false,
2171
+ "single_word": false
2172
+ },
2173
+ {
2174
+ "content": "334513529294",
2175
+ "lstrip": false,
2176
+ "normalized": false,
2177
+ "rstrip": false,
2178
+ "single_word": false
2179
+ },
2180
+ {
2181
+ "content": "549946382131",
2182
+ "lstrip": false,
2183
+ "normalized": false,
2184
+ "rstrip": false,
2185
+ "single_word": false
2186
+ },
2187
+ {
2188
+ "content": "284445431422",
2189
+ "lstrip": false,
2190
+ "normalized": false,
2191
+ "rstrip": false,
2192
+ "single_word": false
2193
+ },
2194
+ {
2195
+ "content": "396521188476",
2196
+ "lstrip": false,
2197
+ "normalized": false,
2198
+ "rstrip": false,
2199
+ "single_word": false
2200
+ },
2201
+ {
2202
+ "content": "421435255895",
2203
+ "lstrip": false,
2204
+ "normalized": false,
2205
+ "rstrip": false,
2206
+ "single_word": false
2207
+ },
2208
+ {
2209
+ "content": "133373659361",
2210
+ "lstrip": false,
2211
+ "normalized": false,
2212
+ "rstrip": false,
2213
+ "single_word": false
2214
+ },
2215
+ {
2216
+ "content": "322683334381",
2217
+ "lstrip": false,
2218
+ "normalized": false,
2219
+ "rstrip": false,
2220
+ "single_word": false
2221
+ },
2222
+ {
2223
+ "content": "228358422847",
2224
+ "lstrip": false,
2225
+ "normalized": false,
2226
+ "rstrip": false,
2227
+ "single_word": false
2228
+ },
2229
+ {
2230
+ "content": "291762694874",
2231
+ "lstrip": false,
2232
+ "normalized": false,
2233
+ "rstrip": false,
2234
+ "single_word": false
2235
+ },
2236
+ {
2237
+ "content": "143182978129",
2238
+ "lstrip": false,
2239
+ "normalized": false,
2240
+ "rstrip": false,
2241
+ "single_word": false
2242
+ },
2243
+ {
2244
+ "content": "511923256573",
2245
+ "lstrip": false,
2246
+ "normalized": false,
2247
+ "rstrip": false,
2248
+ "single_word": false
2249
+ },
2250
+ {
2251
+ "content": "327158398268",
2252
+ "lstrip": false,
2253
+ "normalized": false,
2254
+ "rstrip": false,
2255
+ "single_word": false
2256
+ },
2257
+ {
2258
+ "content": "879764613759",
2259
+ "lstrip": false,
2260
+ "normalized": false,
2261
+ "rstrip": false,
2262
+ "single_word": false
2263
+ },
2264
+ {
2265
+ "content": "564395222747",
2266
+ "lstrip": false,
2267
+ "normalized": false,
2268
+ "rstrip": false,
2269
+ "single_word": false
2270
+ },
2271
+ {
2272
+ "content": "451161679736",
2273
+ "lstrip": false,
2274
+ "normalized": false,
2275
+ "rstrip": false,
2276
+ "single_word": false
2277
+ },
2278
+ {
2279
+ "content": "538631466654",
2280
+ "lstrip": false,
2281
+ "normalized": false,
2282
+ "rstrip": false,
2283
+ "single_word": false
2284
+ },
2285
+ {
2286
+ "content": "221762325616",
2287
+ "lstrip": false,
2288
+ "normalized": false,
2289
+ "rstrip": false,
2290
+ "single_word": false
2291
+ },
2292
+ {
2293
+ "content": "218391991184",
2294
+ "lstrip": false,
2295
+ "normalized": false,
2296
+ "rstrip": false,
2297
+ "single_word": false
2298
+ },
2299
+ {
2300
+ "content": "322589379462",
2301
+ "lstrip": false,
2302
+ "normalized": false,
2303
+ "rstrip": false,
2304
+ "single_word": false
2305
+ },
2306
+ {
2307
+ "content": "876537814263",
2308
+ "lstrip": false,
2309
+ "normalized": false,
2310
+ "rstrip": false,
2311
+ "single_word": false
2312
+ },
2313
+ {
2314
+ "content": "152676556624",
2315
+ "lstrip": false,
2316
+ "normalized": false,
2317
+ "rstrip": false,
2318
+ "single_word": false
2319
+ },
2320
+ {
2321
+ "content": "332522971941",
2322
+ "lstrip": false,
2323
+ "normalized": false,
2324
+ "rstrip": false,
2325
+ "single_word": false
2326
+ },
2327
+ {
2328
+ "content": "884354318946",
2329
+ "lstrip": false,
2330
+ "normalized": false,
2331
+ "rstrip": false,
2332
+ "single_word": false
2333
+ },
2334
+ {
2335
+ "content": "513349618943",
2336
+ "lstrip": false,
2337
+ "normalized": false,
2338
+ "rstrip": false,
2339
+ "single_word": false
2340
+ },
2341
+ {
2342
+ "content": "116639746413",
2343
+ "lstrip": false,
2344
+ "normalized": false,
2345
+ "rstrip": false,
2346
+ "single_word": false
2347
+ },
2348
+ {
2349
+ "content": "635185846287",
2350
+ "lstrip": false,
2351
+ "normalized": false,
2352
+ "rstrip": false,
2353
+ "single_word": false
2354
+ },
2355
+ {
2356
+ "content": "993832498489",
2357
+ "lstrip": false,
2358
+ "normalized": false,
2359
+ "rstrip": false,
2360
+ "single_word": false
2361
+ },
2362
+ {
2363
+ "content": "813981174797",
2364
+ "lstrip": false,
2365
+ "normalized": false,
2366
+ "rstrip": false,
2367
+ "single_word": false
2368
+ },
2369
+ {
2370
+ "content": "438745114173",
2371
+ "lstrip": false,
2372
+ "normalized": false,
2373
+ "rstrip": false,
2374
+ "single_word": false
2375
+ },
2376
+ {
2377
+ "content": "983493951323",
2378
+ "lstrip": false,
2379
+ "normalized": false,
2380
+ "rstrip": false,
2381
+ "single_word": false
2382
+ },
2383
+ {
2384
+ "content": "724492262421",
2385
+ "lstrip": false,
2386
+ "normalized": false,
2387
+ "rstrip": false,
2388
+ "single_word": false
2389
+ },
2390
+ {
2391
+ "content": "622553389126",
2392
+ "lstrip": false,
2393
+ "normalized": false,
2394
+ "rstrip": false,
2395
+ "single_word": false
2396
+ },
2397
+ {
2398
+ "content": "889965243135",
2399
+ "lstrip": false,
2400
+ "normalized": false,
2401
+ "rstrip": false,
2402
+ "single_word": false
2403
+ },
2404
+ {
2405
+ "content": "364492359246",
2406
+ "lstrip": false,
2407
+ "normalized": false,
2408
+ "rstrip": false,
2409
+ "single_word": false
2410
+ },
2411
+ {
2412
+ "content": "154962668224",
2413
+ "lstrip": false,
2414
+ "normalized": false,
2415
+ "rstrip": false,
2416
+ "single_word": false
2417
+ },
2418
+ {
2419
+ "content": "179564995814",
2420
+ "lstrip": false,
2421
+ "normalized": false,
2422
+ "rstrip": false,
2423
+ "single_word": false
2424
+ },
2425
+ {
2426
+ "content": "418412875665",
2427
+ "lstrip": false,
2428
+ "normalized": false,
2429
+ "rstrip": false,
2430
+ "single_word": false
2431
+ },
2432
+ {
2433
+ "content": "718951851413",
2434
+ "lstrip": false,
2435
+ "normalized": false,
2436
+ "rstrip": false,
2437
+ "single_word": false
2438
+ },
2439
+ {
2440
+ "content": "699446724178",
2441
+ "lstrip": false,
2442
+ "normalized": false,
2443
+ "rstrip": false,
2444
+ "single_word": false
2445
+ },
2446
+ {
2447
+ "content": "624266421831",
2448
+ "lstrip": false,
2449
+ "normalized": false,
2450
+ "rstrip": false,
2451
+ "single_word": false
2452
+ },
2453
+ {
2454
+ "content": "815458725125",
2455
+ "lstrip": false,
2456
+ "normalized": false,
2457
+ "rstrip": false,
2458
+ "single_word": false
2459
+ },
2460
+ {
2461
+ "content": "455423278865",
2462
+ "lstrip": false,
2463
+ "normalized": false,
2464
+ "rstrip": false,
2465
+ "single_word": false
2466
+ },
2467
+ {
2468
+ "content": "393741199486",
2469
+ "lstrip": false,
2470
+ "normalized": false,
2471
+ "rstrip": false,
2472
+ "single_word": false
2473
+ },
2474
+ {
2475
+ "content": "328552864359",
2476
+ "lstrip": false,
2477
+ "normalized": false,
2478
+ "rstrip": false,
2479
+ "single_word": false
2480
+ },
2481
+ {
2482
+ "content": "211662639865",
2483
+ "lstrip": false,
2484
+ "normalized": false,
2485
+ "rstrip": false,
2486
+ "single_word": false
2487
+ },
2488
+ {
2489
+ "content": "218784516525",
2490
+ "lstrip": false,
2491
+ "normalized": false,
2492
+ "rstrip": false,
2493
+ "single_word": false
2494
+ },
2495
+ {
2496
+ "content": "762486672996",
2497
+ "lstrip": false,
2498
+ "normalized": false,
2499
+ "rstrip": false,
2500
+ "single_word": false
2501
+ },
2502
+ {
2503
+ "content": "142799718159",
2504
+ "lstrip": false,
2505
+ "normalized": false,
2506
+ "rstrip": false,
2507
+ "single_word": false
2508
+ },
2509
+ {
2510
+ "content": "858146415154",
2511
+ "lstrip": false,
2512
+ "normalized": false,
2513
+ "rstrip": false,
2514
+ "single_word": false
2515
+ },
2516
+ {
2517
+ "content": "767858144912",
2518
+ "lstrip": false,
2519
+ "normalized": false,
2520
+ "rstrip": false,
2521
+ "single_word": false
2522
+ },
2523
+ {
2524
+ "content": "571317457151",
2525
+ "lstrip": false,
2526
+ "normalized": false,
2527
+ "rstrip": false,
2528
+ "single_word": false
2529
+ },
2530
+ {
2531
+ "content": "635127952696",
2532
+ "lstrip": false,
2533
+ "normalized": false,
2534
+ "rstrip": false,
2535
+ "single_word": false
2536
+ },
2537
+ {
2538
+ "content": "116427191984",
2539
+ "lstrip": false,
2540
+ "normalized": false,
2541
+ "rstrip": false,
2542
+ "single_word": false
2543
+ },
2544
+ {
2545
+ "content": "268921994538",
2546
+ "lstrip": false,
2547
+ "normalized": false,
2548
+ "rstrip": false,
2549
+ "single_word": false
2550
+ },
2551
+ {
2552
+ "content": "523937669294",
2553
+ "lstrip": false,
2554
+ "normalized": false,
2555
+ "rstrip": false,
2556
+ "single_word": false
2557
+ },
2558
+ {
2559
+ "content": "165429152138",
2560
+ "lstrip": false,
2561
+ "normalized": false,
2562
+ "rstrip": false,
2563
+ "single_word": false
2564
+ },
2565
+ {
2566
+ "content": "739246183345",
2567
+ "lstrip": false,
2568
+ "normalized": false,
2569
+ "rstrip": false,
2570
+ "single_word": false
2571
+ },
2572
+ {
2573
+ "content": "591464355756",
2574
+ "lstrip": false,
2575
+ "normalized": false,
2576
+ "rstrip": false,
2577
+ "single_word": false
2578
+ },
2579
+ {
2580
+ "content": "212985874612",
2581
+ "lstrip": false,
2582
+ "normalized": false,
2583
+ "rstrip": false,
2584
+ "single_word": false
2585
+ },
2586
+ {
2587
+ "content": "191887635211",
2588
+ "lstrip": false,
2589
+ "normalized": false,
2590
+ "rstrip": false,
2591
+ "single_word": false
2592
+ },
2593
+ {
2594
+ "content": "967214577653",
2595
+ "lstrip": false,
2596
+ "normalized": false,
2597
+ "rstrip": false,
2598
+ "single_word": false
2599
+ },
2600
+ {
2601
+ "content": "119342152414",
2602
+ "lstrip": false,
2603
+ "normalized": false,
2604
+ "rstrip": false,
2605
+ "single_word": false
2606
+ },
2607
+ {
2608
+ "content": "946444632795",
2609
+ "lstrip": false,
2610
+ "normalized": false,
2611
+ "rstrip": false,
2612
+ "single_word": false
2613
+ },
2614
+ {
2615
+ "content": "618423867817",
2616
+ "lstrip": false,
2617
+ "normalized": false,
2618
+ "rstrip": false,
2619
+ "single_word": false
2620
+ },
2621
+ {
2622
+ "content": "228565148417",
2623
+ "lstrip": false,
2624
+ "normalized": false,
2625
+ "rstrip": false,
2626
+ "single_word": false
2627
+ },
2628
+ {
2629
+ "content": "729116422489",
2630
+ "lstrip": false,
2631
+ "normalized": false,
2632
+ "rstrip": false,
2633
+ "single_word": false
2634
+ },
2635
+ {
2636
+ "content": "527874729936",
2637
+ "lstrip": false,
2638
+ "normalized": false,
2639
+ "rstrip": false,
2640
+ "single_word": false
2641
+ },
2642
+ {
2643
+ "content": "739784153482",
2644
+ "lstrip": false,
2645
+ "normalized": false,
2646
+ "rstrip": false,
2647
+ "single_word": false
2648
+ },
2649
+ {
2650
+ "content": "387763951128",
2651
+ "lstrip": false,
2652
+ "normalized": false,
2653
+ "rstrip": false,
2654
+ "single_word": false
2655
+ },
2656
+ {
2657
+ "content": "331369926711",
2658
+ "lstrip": false,
2659
+ "normalized": false,
2660
+ "rstrip": false,
2661
+ "single_word": false
2662
+ },
2663
+ {
2664
+ "content": "562716493614",
2665
+ "lstrip": false,
2666
+ "normalized": false,
2667
+ "rstrip": false,
2668
+ "single_word": false
2669
+ },
2670
+ {
2671
+ "content": "739667844957",
2672
+ "lstrip": false,
2673
+ "normalized": false,
2674
+ "rstrip": false,
2675
+ "single_word": false
2676
+ },
2677
+ {
2678
+ "content": "562389434565",
2679
+ "lstrip": false,
2680
+ "normalized": false,
2681
+ "rstrip": false,
2682
+ "single_word": false
2683
+ },
2684
+ {
2685
+ "content": "256497188281",
2686
+ "lstrip": false,
2687
+ "normalized": false,
2688
+ "rstrip": false,
2689
+ "single_word": false
2690
+ },
2691
+ {
2692
+ "content": "859927364588",
2693
+ "lstrip": false,
2694
+ "normalized": false,
2695
+ "rstrip": false,
2696
+ "single_word": false
2697
+ },
2698
+ {
2699
+ "content": "417668946583",
2700
+ "lstrip": false,
2701
+ "normalized": false,
2702
+ "rstrip": false,
2703
+ "single_word": false
2704
+ },
2705
+ {
2706
+ "content": "357621613582",
2707
+ "lstrip": false,
2708
+ "normalized": false,
2709
+ "rstrip": false,
2710
+ "single_word": false
2711
+ },
2712
+ {
2713
+ "content": "438435178228",
2714
+ "lstrip": false,
2715
+ "normalized": false,
2716
+ "rstrip": false,
2717
+ "single_word": false
2718
+ },
2719
+ {
2720
+ "content": "485692541169",
2721
+ "lstrip": false,
2722
+ "normalized": false,
2723
+ "rstrip": false,
2724
+ "single_word": false
2725
+ },
2726
+ {
2727
+ "content": "825815739116",
2728
+ "lstrip": false,
2729
+ "normalized": false,
2730
+ "rstrip": false,
2731
+ "single_word": false
2732
+ },
2733
+ {
2734
+ "content": "342221452223",
2735
+ "lstrip": false,
2736
+ "normalized": false,
2737
+ "rstrip": false,
2738
+ "single_word": false
2739
+ },
2740
+ {
2741
+ "content": "697747991249",
2742
+ "lstrip": false,
2743
+ "normalized": false,
2744
+ "rstrip": false,
2745
+ "single_word": false
2746
+ },
2747
+ {
2748
+ "content": "716763689965",
2749
+ "lstrip": false,
2750
+ "normalized": false,
2751
+ "rstrip": false,
2752
+ "single_word": false
2753
+ },
2754
+ {
2755
+ "content": "141499982867",
2756
+ "lstrip": false,
2757
+ "normalized": false,
2758
+ "rstrip": false,
2759
+ "single_word": false
2760
+ },
2761
+ {
2762
+ "content": "818479319499",
2763
+ "lstrip": false,
2764
+ "normalized": false,
2765
+ "rstrip": false,
2766
+ "single_word": false
2767
+ },
2768
+ {
2769
+ "content": "336813343298",
2770
+ "lstrip": false,
2771
+ "normalized": false,
2772
+ "rstrip": false,
2773
+ "single_word": false
2774
+ },
2775
+ {
2776
+ "content": "594688742928",
2777
+ "lstrip": false,
2778
+ "normalized": false,
2779
+ "rstrip": false,
2780
+ "single_word": false
2781
+ },
2782
+ {
2783
+ "content": "472129283475",
2784
+ "lstrip": false,
2785
+ "normalized": false,
2786
+ "rstrip": false,
2787
+ "single_word": false
2788
+ },
2789
+ {
2790
+ "content": "514354144759",
2791
+ "lstrip": false,
2792
+ "normalized": false,
2793
+ "rstrip": false,
2794
+ "single_word": false
2795
+ },
2796
+ {
2797
+ "content": "349249721685",
2798
+ "lstrip": false,
2799
+ "normalized": false,
2800
+ "rstrip": false,
2801
+ "single_word": false
2802
+ },
2803
+ {
2804
+ "content": "546276298359",
2805
+ "lstrip": false,
2806
+ "normalized": false,
2807
+ "rstrip": false,
2808
+ "single_word": false
2809
+ },
2810
+ {
2811
+ "content": "353755529131",
2812
+ "lstrip": false,
2813
+ "normalized": false,
2814
+ "rstrip": false,
2815
+ "single_word": false
2816
+ },
2817
+ {
2818
+ "content": "315534574435",
2819
+ "lstrip": false,
2820
+ "normalized": false,
2821
+ "rstrip": false,
2822
+ "single_word": false
2823
+ },
2824
+ {
2825
+ "content": "523723475786",
2826
+ "lstrip": false,
2827
+ "normalized": false,
2828
+ "rstrip": false,
2829
+ "single_word": false
2830
+ },
2831
+ {
2832
+ "content": "215826764872",
2833
+ "lstrip": false,
2834
+ "normalized": false,
2835
+ "rstrip": false,
2836
+ "single_word": false
2837
+ },
2838
+ {
2839
+ "content": "367968398551",
2840
+ "lstrip": false,
2841
+ "normalized": false,
2842
+ "rstrip": false,
2843
+ "single_word": false
2844
+ },
2845
+ {
2846
+ "content": "569853653352",
2847
+ "lstrip": false,
2848
+ "normalized": false,
2849
+ "rstrip": false,
2850
+ "single_word": false
2851
+ },
2852
+ {
2853
+ "content": "389715484387",
2854
+ "lstrip": false,
2855
+ "normalized": false,
2856
+ "rstrip": false,
2857
+ "single_word": false
2858
+ },
2859
+ {
2860
+ "content": "293847485454",
2861
+ "lstrip": false,
2862
+ "normalized": false,
2863
+ "rstrip": false,
2864
+ "single_word": false
2865
+ },
2866
+ {
2867
+ "content": "714738141818",
2868
+ "lstrip": false,
2869
+ "normalized": false,
2870
+ "rstrip": false,
2871
+ "single_word": false
2872
+ },
2873
+ {
2874
+ "content": "178478368922",
2875
+ "lstrip": false,
2876
+ "normalized": false,
2877
+ "rstrip": false,
2878
+ "single_word": false
2879
+ },
2880
+ {
2881
+ "content": "581493616981",
2882
+ "lstrip": false,
2883
+ "normalized": false,
2884
+ "rstrip": false,
2885
+ "single_word": false
2886
+ },
2887
+ {
2888
+ "content": "589439538674",
2889
+ "lstrip": false,
2890
+ "normalized": false,
2891
+ "rstrip": false,
2892
+ "single_word": false
2893
+ },
2894
+ {
2895
+ "content": "846657726193",
2896
+ "lstrip": false,
2897
+ "normalized": false,
2898
+ "rstrip": false,
2899
+ "single_word": false
2900
+ },
2901
+ {
2902
+ "content": "722339992679",
2903
+ "lstrip": false,
2904
+ "normalized": false,
2905
+ "rstrip": false,
2906
+ "single_word": false
2907
+ },
2908
+ {
2909
+ "content": "138154781148",
2910
+ "lstrip": false,
2911
+ "normalized": false,
2912
+ "rstrip": false,
2913
+ "single_word": false
2914
+ },
2915
+ {
2916
+ "content": "757785319772",
2917
+ "lstrip": false,
2918
+ "normalized": false,
2919
+ "rstrip": false,
2920
+ "single_word": false
2921
+ },
2922
+ {
2923
+ "content": "492516914298",
2924
+ "lstrip": false,
2925
+ "normalized": false,
2926
+ "rstrip": false,
2927
+ "single_word": false
2928
+ },
2929
+ {
2930
+ "content": "919181521716",
2931
+ "lstrip": false,
2932
+ "normalized": false,
2933
+ "rstrip": false,
2934
+ "single_word": false
2935
+ },
2936
+ {
2937
+ "content": "985781138935",
2938
+ "lstrip": false,
2939
+ "normalized": false,
2940
+ "rstrip": false,
2941
+ "single_word": false
2942
+ },
2943
+ {
2944
+ "content": "476969195485",
2945
+ "lstrip": false,
2946
+ "normalized": false,
2947
+ "rstrip": false,
2948
+ "single_word": false
2949
+ },
2950
+ {
2951
+ "content": "313145133463",
2952
+ "lstrip": false,
2953
+ "normalized": false,
2954
+ "rstrip": false,
2955
+ "single_word": false
2956
+ },
2957
+ {
2958
+ "content": "758963111966",
2959
+ "lstrip": false,
2960
+ "normalized": false,
2961
+ "rstrip": false,
2962
+ "single_word": false
2963
+ },
2964
+ {
2965
+ "content": "147541537162",
2966
+ "lstrip": false,
2967
+ "normalized": false,
2968
+ "rstrip": false,
2969
+ "single_word": false
2970
+ },
2971
+ {
2972
+ "content": "557163366873",
2973
+ "lstrip": false,
2974
+ "normalized": false,
2975
+ "rstrip": false,
2976
+ "single_word": false
2977
+ },
2978
+ {
2979
+ "content": "144373897488",
2980
+ "lstrip": false,
2981
+ "normalized": false,
2982
+ "rstrip": false,
2983
+ "single_word": false
2984
+ },
2985
+ {
2986
+ "content": "522515164754",
2987
+ "lstrip": false,
2988
+ "normalized": false,
2989
+ "rstrip": false,
2990
+ "single_word": false
2991
+ },
2992
+ {
2993
+ "content": "724964923582",
2994
+ "lstrip": false,
2995
+ "normalized": false,
2996
+ "rstrip": false,
2997
+ "single_word": false
2998
+ },
2999
+ {
3000
+ "content": "284776712475",
3001
+ "lstrip": false,
3002
+ "normalized": false,
3003
+ "rstrip": false,
3004
+ "single_word": false
3005
+ },
3006
+ {
3007
+ "content": "375429755114",
3008
+ "lstrip": false,
3009
+ "normalized": false,
3010
+ "rstrip": false,
3011
+ "single_word": false
3012
+ },
3013
+ {
3014
+ "content": "181233596124",
3015
+ "lstrip": false,
3016
+ "normalized": false,
3017
+ "rstrip": false,
3018
+ "single_word": false
3019
+ },
3020
+ {
3021
+ "content": "948585673431",
3022
+ "lstrip": false,
3023
+ "normalized": false,
3024
+ "rstrip": false,
3025
+ "single_word": false
3026
+ },
3027
+ {
3028
+ "content": "243165586174",
3029
+ "lstrip": false,
3030
+ "normalized": false,
3031
+ "rstrip": false,
3032
+ "single_word": false
3033
+ },
3034
+ {
3035
+ "content": "396847976144",
3036
+ "lstrip": false,
3037
+ "normalized": false,
3038
+ "rstrip": false,
3039
+ "single_word": false
3040
+ },
3041
+ {
3042
+ "content": "997724962668",
3043
+ "lstrip": false,
3044
+ "normalized": false,
3045
+ "rstrip": false,
3046
+ "single_word": false
3047
+ },
3048
+ {
3049
+ "content": "558837194455",
3050
+ "lstrip": false,
3051
+ "normalized": false,
3052
+ "rstrip": false,
3053
+ "single_word": false
3054
+ },
3055
+ {
3056
+ "content": "163165456396",
3057
+ "lstrip": false,
3058
+ "normalized": false,
3059
+ "rstrip": false,
3060
+ "single_word": false
3061
+ },
3062
+ {
3063
+ "content": "378749551722",
3064
+ "lstrip": false,
3065
+ "normalized": false,
3066
+ "rstrip": false,
3067
+ "single_word": false
3068
+ },
3069
+ {
3070
+ "content": "161238482259",
3071
+ "lstrip": false,
3072
+ "normalized": false,
3073
+ "rstrip": false,
3074
+ "single_word": false
3075
+ },
3076
+ {
3077
+ "content": "754978243758",
3078
+ "lstrip": false,
3079
+ "normalized": false,
3080
+ "rstrip": false,
3081
+ "single_word": false
3082
+ },
3083
+ {
3084
+ "content": "195388849133",
3085
+ "lstrip": false,
3086
+ "normalized": false,
3087
+ "rstrip": false,
3088
+ "single_word": false
3089
+ },
3090
+ {
3091
+ "content": "229775525672",
3092
+ "lstrip": false,
3093
+ "normalized": false,
3094
+ "rstrip": false,
3095
+ "single_word": false
3096
+ },
3097
+ {
3098
+ "content": "262437452884",
3099
+ "lstrip": false,
3100
+ "normalized": false,
3101
+ "rstrip": false,
3102
+ "single_word": false
3103
+ },
3104
+ {
3105
+ "content": "441377892146",
3106
+ "lstrip": false,
3107
+ "normalized": false,
3108
+ "rstrip": false,
3109
+ "single_word": false
3110
+ },
3111
+ {
3112
+ "content": "451885565366",
3113
+ "lstrip": false,
3114
+ "normalized": false,
3115
+ "rstrip": false,
3116
+ "single_word": false
3117
+ },
3118
+ {
3119
+ "content": "981277526855",
3120
+ "lstrip": false,
3121
+ "normalized": false,
3122
+ "rstrip": false,
3123
+ "single_word": false
3124
+ },
3125
+ {
3126
+ "content": "762495822823",
3127
+ "lstrip": false,
3128
+ "normalized": false,
3129
+ "rstrip": false,
3130
+ "single_word": false
3131
+ },
3132
+ {
3133
+ "content": "368763327262",
3134
+ "lstrip": false,
3135
+ "normalized": false,
3136
+ "rstrip": false,
3137
+ "single_word": false
3138
+ },
3139
+ {
3140
+ "content": "757422791351",
3141
+ "lstrip": false,
3142
+ "normalized": false,
3143
+ "rstrip": false,
3144
+ "single_word": false
3145
+ },
3146
+ {
3147
+ "content": "636324136426",
3148
+ "lstrip": false,
3149
+ "normalized": false,
3150
+ "rstrip": false,
3151
+ "single_word": false
3152
+ },
3153
+ {
3154
+ "content": "214193645583",
3155
+ "lstrip": false,
3156
+ "normalized": false,
3157
+ "rstrip": false,
3158
+ "single_word": false
3159
+ },
3160
+ {
3161
+ "content": "412843856172",
3162
+ "lstrip": false,
3163
+ "normalized": false,
3164
+ "rstrip": false,
3165
+ "single_word": false
3166
+ },
3167
+ {
3168
+ "content": "179386156569",
3169
+ "lstrip": false,
3170
+ "normalized": false,
3171
+ "rstrip": false,
3172
+ "single_word": false
3173
+ },
3174
+ {
3175
+ "content": "756916173536",
3176
+ "lstrip": false,
3177
+ "normalized": false,
3178
+ "rstrip": false,
3179
+ "single_word": false
3180
+ },
3181
+ {
3182
+ "content": "892697125149",
3183
+ "lstrip": false,
3184
+ "normalized": false,
3185
+ "rstrip": false,
3186
+ "single_word": false
3187
+ },
3188
+ {
3189
+ "content": "625334487352",
3190
+ "lstrip": false,
3191
+ "normalized": false,
3192
+ "rstrip": false,
3193
+ "single_word": false
3194
+ },
3195
+ {
3196
+ "content": "941861857715",
3197
+ "lstrip": false,
3198
+ "normalized": false,
3199
+ "rstrip": false,
3200
+ "single_word": false
3201
+ },
3202
+ {
3203
+ "content": "887417525236",
3204
+ "lstrip": false,
3205
+ "normalized": false,
3206
+ "rstrip": false,
3207
+ "single_word": false
3208
+ },
3209
+ {
3210
+ "content": "649516938598",
3211
+ "lstrip": false,
3212
+ "normalized": false,
3213
+ "rstrip": false,
3214
+ "single_word": false
3215
+ },
3216
+ {
3217
+ "content": "717628619782",
3218
+ "lstrip": false,
3219
+ "normalized": false,
3220
+ "rstrip": false,
3221
+ "single_word": false
3222
+ },
3223
+ {
3224
+ "content": "438124184139",
3225
+ "lstrip": false,
3226
+ "normalized": false,
3227
+ "rstrip": false,
3228
+ "single_word": false
3229
+ },
3230
+ {
3231
+ "content": "547563892268",
3232
+ "lstrip": false,
3233
+ "normalized": false,
3234
+ "rstrip": false,
3235
+ "single_word": false
3236
+ },
3237
+ {
3238
+ "content": "856317483891",
3239
+ "lstrip": false,
3240
+ "normalized": false,
3241
+ "rstrip": false,
3242
+ "single_word": false
3243
+ },
3244
+ {
3245
+ "content": "313313831273",
3246
+ "lstrip": false,
3247
+ "normalized": false,
3248
+ "rstrip": false,
3249
+ "single_word": false
3250
+ },
3251
+ {
3252
+ "content": "371496153876",
3253
+ "lstrip": false,
3254
+ "normalized": false,
3255
+ "rstrip": false,
3256
+ "single_word": false
3257
+ },
3258
+ {
3259
+ "content": "587541149322",
3260
+ "lstrip": false,
3261
+ "normalized": false,
3262
+ "rstrip": false,
3263
+ "single_word": false
3264
+ },
3265
+ {
3266
+ "content": "265847332563",
3267
+ "lstrip": false,
3268
+ "normalized": false,
3269
+ "rstrip": false,
3270
+ "single_word": false
3271
+ },
3272
+ {
3273
+ "content": "449549215429",
3274
+ "lstrip": false,
3275
+ "normalized": false,
3276
+ "rstrip": false,
3277
+ "single_word": false
3278
+ },
3279
+ {
3280
+ "content": "163497196769",
3281
+ "lstrip": false,
3282
+ "normalized": false,
3283
+ "rstrip": false,
3284
+ "single_word": false
3285
+ },
3286
+ {
3287
+ "content": "861342291298",
3288
+ "lstrip": false,
3289
+ "normalized": false,
3290
+ "rstrip": false,
3291
+ "single_word": false
3292
+ },
3293
+ {
3294
+ "content": "268433315926",
3295
+ "lstrip": false,
3296
+ "normalized": false,
3297
+ "rstrip": false,
3298
+ "single_word": false
3299
+ },
3300
+ {
3301
+ "content": "774679513717",
3302
+ "lstrip": false,
3303
+ "normalized": false,
3304
+ "rstrip": false,
3305
+ "single_word": false
3306
+ },
3307
+ {
3308
+ "content": "851254219729",
3309
+ "lstrip": false,
3310
+ "normalized": false,
3311
+ "rstrip": false,
3312
+ "single_word": false
3313
+ },
3314
+ {
3315
+ "content": "583527834464",
3316
+ "lstrip": false,
3317
+ "normalized": false,
3318
+ "rstrip": false,
3319
+ "single_word": false
3320
+ },
3321
+ {
3322
+ "content": "488496781997",
3323
+ "lstrip": false,
3324
+ "normalized": false,
3325
+ "rstrip": false,
3326
+ "single_word": false
3327
+ },
3328
+ {
3329
+ "content": "556814553861",
3330
+ "lstrip": false,
3331
+ "normalized": false,
3332
+ "rstrip": false,
3333
+ "single_word": false
3334
+ },
3335
+ {
3336
+ "content": "482829231639",
3337
+ "lstrip": false,
3338
+ "normalized": false,
3339
+ "rstrip": false,
3340
+ "single_word": false
3341
+ },
3342
+ {
3343
+ "content": "618878266619",
3344
+ "lstrip": false,
3345
+ "normalized": false,
3346
+ "rstrip": false,
3347
+ "single_word": false
3348
+ },
3349
+ {
3350
+ "content": "147444452794",
3351
+ "lstrip": false,
3352
+ "normalized": false,
3353
+ "rstrip": false,
3354
+ "single_word": false
3355
+ },
3356
+ {
3357
+ "content": "949235426629",
3358
+ "lstrip": false,
3359
+ "normalized": false,
3360
+ "rstrip": false,
3361
+ "single_word": false
3362
+ },
3363
+ {
3364
+ "content": "357299947518",
3365
+ "lstrip": false,
3366
+ "normalized": false,
3367
+ "rstrip": false,
3368
+ "single_word": false
3369
+ },
3370
+ {
3371
+ "content": "175528632226",
3372
+ "lstrip": false,
3373
+ "normalized": false,
3374
+ "rstrip": false,
3375
+ "single_word": false
3376
+ },
3377
+ {
3378
+ "content": "645527857972",
3379
+ "lstrip": false,
3380
+ "normalized": false,
3381
+ "rstrip": false,
3382
+ "single_word": false
3383
+ },
3384
+ {
3385
+ "content": "186872457894",
3386
+ "lstrip": false,
3387
+ "normalized": false,
3388
+ "rstrip": false,
3389
+ "single_word": false
3390
+ },
3391
+ {
3392
+ "content": "552738847828",
3393
+ "lstrip": false,
3394
+ "normalized": false,
3395
+ "rstrip": false,
3396
+ "single_word": false
3397
+ },
3398
+ {
3399
+ "content": "626748382482",
3400
+ "lstrip": false,
3401
+ "normalized": false,
3402
+ "rstrip": false,
3403
+ "single_word": false
3404
+ },
3405
+ {
3406
+ "content": "921894985642",
3407
+ "lstrip": false,
3408
+ "normalized": false,
3409
+ "rstrip": false,
3410
+ "single_word": false
3411
+ },
3412
+ {
3413
+ "content": "943878645871",
3414
+ "lstrip": false,
3415
+ "normalized": false,
3416
+ "rstrip": false,
3417
+ "single_word": false
3418
+ },
3419
+ {
3420
+ "content": "859289776479",
3421
+ "lstrip": false,
3422
+ "normalized": false,
3423
+ "rstrip": false,
3424
+ "single_word": false
3425
+ },
3426
+ {
3427
+ "content": "614583493135",
3428
+ "lstrip": false,
3429
+ "normalized": false,
3430
+ "rstrip": false,
3431
+ "single_word": false
3432
+ },
3433
+ {
3434
+ "content": "933775286797",
3435
+ "lstrip": false,
3436
+ "normalized": false,
3437
+ "rstrip": false,
3438
+ "single_word": false
3439
+ },
3440
+ {
3441
+ "content": "332234613346",
3442
+ "lstrip": false,
3443
+ "normalized": false,
3444
+ "rstrip": false,
3445
+ "single_word": false
3446
+ },
3447
+ {
3448
+ "content": "325196781219",
3449
+ "lstrip": false,
3450
+ "normalized": false,
3451
+ "rstrip": false,
3452
+ "single_word": false
3453
+ },
3454
+ {
3455
+ "content": "142526557681",
3456
+ "lstrip": false,
3457
+ "normalized": false,
3458
+ "rstrip": false,
3459
+ "single_word": false
3460
+ },
3461
+ {
3462
+ "content": "356722692178",
3463
+ "lstrip": false,
3464
+ "normalized": false,
3465
+ "rstrip": false,
3466
+ "single_word": false
3467
+ },
3468
+ {
3469
+ "content": "449318681694",
3470
+ "lstrip": false,
3471
+ "normalized": false,
3472
+ "rstrip": false,
3473
+ "single_word": false
3474
+ },
3475
+ {
3476
+ "content": "687284547244",
3477
+ "lstrip": false,
3478
+ "normalized": false,
3479
+ "rstrip": false,
3480
+ "single_word": false
3481
+ },
3482
+ {
3483
+ "content": "947262995132",
3484
+ "lstrip": false,
3485
+ "normalized": false,
3486
+ "rstrip": false,
3487
+ "single_word": false
3488
+ },
3489
+ {
3490
+ "content": "893974619684",
3491
+ "lstrip": false,
3492
+ "normalized": false,
3493
+ "rstrip": false,
3494
+ "single_word": false
3495
+ },
3496
+ {
3497
+ "content": "797238311233",
3498
+ "lstrip": false,
3499
+ "normalized": false,
3500
+ "rstrip": false,
3501
+ "single_word": false
3502
+ }
3503
+ ],
3504
+ "eos_token": {
3505
+ "content": "<|endoftext|>",
3506
+ "lstrip": false,
3507
+ "normalized": false,
3508
+ "rstrip": false,
3509
+ "single_word": false
3510
+ },
3511
+ "pad_token": {
3512
+ "content": "<|endoftext|>",
3513
+ "lstrip": false,
3514
+ "normalized": false,
3515
+ "rstrip": false,
3516
+ "single_word": false
3517
+ }
3518
+ }
qwen1.5-moe-a2.7b/dp8_attn_v2/tokenizer/tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff
 
qwen1.5-moe-a2.7b/dp8_attn_v2/tokenizer/vocab.json ADDED
The diff for this file is too large to render. See raw diff
 
qwen1.5-moe-a2.7b/dp8_attn_v2/train.log ADDED
@@ -0,0 +1,277 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 2026-04-03 09:58:16,768 [INFO] new_opacus_codex.train_steps: epoch=1 step=5 loss=23.7896
2
+ 2026-04-03 09:59:29,333 [INFO] new_opacus_codex.train_steps: epoch=1 step=10 loss=23.7395
3
+ 2026-04-03 09:59:43,657 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=10 eval_loss=24.5840 duration_sec=14.23
4
+ 2026-04-03 10:00:47,929 [INFO] new_opacus_codex.train_steps: epoch=1 step=15 loss=23.6856
5
+ 2026-04-03 10:01:51,264 [INFO] new_opacus_codex.train_steps: epoch=1 step=20 loss=23.5449
6
+ 2026-04-03 10:02:05,393 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=20 eval_loss=24.2961 duration_sec=14.12
7
+ 2026-04-03 10:03:22,897 [INFO] new_opacus_codex.train_steps: epoch=1 step=25 loss=23.5120
8
+ 2026-04-03 10:04:25,335 [INFO] new_opacus_codex.train_steps: epoch=1 step=30 loss=23.3519
9
+ 2026-04-03 10:05:13,951 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=30 eval_loss=23.7807 duration_sec=34.04
10
+ 2026-04-03 10:06:17,369 [INFO] new_opacus_codex.train_steps: epoch=1 step=35 loss=22.9320
11
+ 2026-04-03 10:07:17,742 [INFO] new_opacus_codex.train_steps: epoch=1 step=40 loss=22.7707
12
+ 2026-04-03 10:07:32,092 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=40 eval_loss=22.9816 duration_sec=14.34
13
+ 2026-04-03 10:08:35,445 [INFO] new_opacus_codex.train_steps: epoch=1 step=45 loss=22.4355
14
+ 2026-04-03 10:09:39,414 [INFO] new_opacus_codex.train_steps: epoch=1 step=50 loss=21.8187
15
+ 2026-04-03 10:10:41,756 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=50 eval_loss=21.7184 duration_sec=62.32
16
+ 2026-04-03 10:11:45,633 [INFO] new_opacus_codex.train_steps: epoch=1 step=55 loss=21.2354
17
+ 2026-04-03 10:13:06,700 [INFO] new_opacus_codex.train_steps: epoch=1 step=60 loss=20.5393
18
+ 2026-04-03 10:13:41,147 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=60 eval_loss=19.9889 duration_sec=34.41
19
+ 2026-04-03 10:15:19,107 [INFO] new_opacus_codex.train_steps: epoch=1 step=65 loss=19.8367
20
+ 2026-04-03 10:16:19,681 [INFO] new_opacus_codex.train_steps: epoch=1 step=70 loss=19.1386
21
+ 2026-04-03 10:16:51,419 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=70 eval_loss=17.8783 duration_sec=18.18
22
+ 2026-04-03 10:17:54,855 [INFO] new_opacus_codex.train_steps: epoch=1 step=75 loss=18.1802
23
+ 2026-04-03 10:18:56,402 [INFO] new_opacus_codex.train_steps: epoch=1 step=80 loss=16.9303
24
+ 2026-04-03 10:19:33,425 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=80 eval_loss=15.8478 duration_sec=33.57
25
+ 2026-04-03 10:20:37,125 [INFO] new_opacus_codex.train_steps: epoch=1 step=85 loss=15.9908
26
+ 2026-04-03 10:21:42,783 [INFO] new_opacus_codex.train_steps: epoch=1 step=90 loss=15.5348
27
+ 2026-04-03 10:21:57,031 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=90 eval_loss=14.8146 duration_sec=14.23
28
+ 2026-04-03 10:22:56,697 [INFO] new_opacus_codex.train_steps: epoch=1 step=95 loss=15.3159
29
+ 2026-04-03 10:23:58,624 [INFO] new_opacus_codex.train_steps: epoch=1 step=100 loss=15.0965
30
+ 2026-04-03 10:24:12,893 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=100 eval_loss=14.0051 duration_sec=14.26
31
+ 2026-04-03 10:25:15,314 [INFO] new_opacus_codex.train_steps: epoch=1 step=105 loss=14.7657
32
+ 2026-04-03 10:26:16,465 [INFO] new_opacus_codex.train_steps: epoch=1 step=110 loss=14.5030
33
+ 2026-04-03 10:26:30,656 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=110 eval_loss=13.3753 duration_sec=14.17
34
+ 2026-04-03 10:27:31,470 [INFO] new_opacus_codex.train_steps: epoch=1 step=115 loss=14.0375
35
+ 2026-04-03 10:28:32,888 [INFO] new_opacus_codex.train_steps: epoch=1 step=120 loss=13.8177
36
+ 2026-04-03 10:29:15,184 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=120 eval_loss=12.8014 duration_sec=23.91
37
+ 2026-04-03 10:30:16,580 [INFO] new_opacus_codex.train_steps: epoch=1 step=125 loss=13.7412
38
+ 2026-04-03 10:31:19,428 [INFO] new_opacus_codex.train_steps: epoch=1 step=130 loss=13.1134
39
+ 2026-04-03 10:31:45,641 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=130 eval_loss=12.2890 duration_sec=21.77
40
+ 2026-04-03 10:32:47,794 [INFO] new_opacus_codex.train_steps: epoch=1 step=135 loss=12.8618
41
+ 2026-04-03 10:34:25,448 [INFO] new_opacus_codex.train_steps: epoch=1 step=140 loss=12.9431
42
+ 2026-04-03 10:34:45,334 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=140 eval_loss=11.7846 duration_sec=17.21
43
+ 2026-04-03 10:35:44,363 [INFO] new_opacus_codex.train_steps: epoch=1 step=145 loss=12.4910
44
+ 2026-04-03 10:36:46,120 [INFO] new_opacus_codex.train_steps: epoch=1 step=150 loss=12.1291
45
+ 2026-04-03 10:37:00,304 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=150 eval_loss=11.1421 duration_sec=14.18
46
+ 2026-04-03 10:37:59,600 [INFO] new_opacus_codex.train_steps: epoch=1 step=155 loss=11.5847
47
+ 2026-04-03 10:39:16,678 [INFO] new_opacus_codex.train_steps: epoch=1 step=160 loss=11.1843
48
+ 2026-04-03 10:39:34,859 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=160 eval_loss=10.2609 duration_sec=18.15
49
+ 2026-04-03 10:40:34,252 [INFO] new_opacus_codex.train_steps: epoch=1 step=165 loss=11.1950
50
+ 2026-04-03 10:41:34,567 [INFO] new_opacus_codex.train_steps: epoch=1 step=170 loss=10.7422
51
+ 2026-04-03 10:41:51,107 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=170 eval_loss=9.4919 duration_sec=14.46
52
+ 2026-04-03 10:42:52,226 [INFO] new_opacus_codex.train_steps: epoch=1 step=175 loss=10.1476
53
+ 2026-04-03 10:43:53,489 [INFO] new_opacus_codex.train_steps: epoch=1 step=180 loss=10.0431
54
+ 2026-04-03 10:44:07,784 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=1 step=180 eval_loss=9.0454 duration_sec=14.28
55
+ 2026-04-03 10:45:07,656 [INFO] new_opacus_codex.train_steps: epoch=1 step=185 loss=9.9055
56
+ 2026-04-03 10:47:39,097 [INFO] new_opacus_codex.train_steps: epoch=2 step=190 loss=9.8764
57
+ 2026-04-03 10:48:07,584 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=190 eval_loss=8.7758 duration_sec=20.77
58
+ 2026-04-03 10:49:08,542 [INFO] new_opacus_codex.train_steps: epoch=2 step=195 loss=9.7350
59
+ 2026-04-03 10:50:10,271 [INFO] new_opacus_codex.train_steps: epoch=2 step=200 loss=9.3717
60
+ 2026-04-03 10:50:24,833 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=200 eval_loss=8.5835 duration_sec=14.25
61
+ 2026-04-03 10:51:24,346 [INFO] new_opacus_codex.train_steps: epoch=2 step=205 loss=9.1469
62
+ 2026-04-03 10:52:24,788 [INFO] new_opacus_codex.train_steps: epoch=2 step=210 loss=8.9924
63
+ 2026-04-03 10:52:40,291 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=210 eval_loss=8.4374 duration_sec=15.48
64
+ 2026-04-03 10:53:40,729 [INFO] new_opacus_codex.train_steps: epoch=2 step=215 loss=8.9698
65
+ 2026-04-03 10:55:11,532 [INFO] new_opacus_codex.train_steps: epoch=2 step=220 loss=9.0637
66
+ 2026-04-03 10:55:26,007 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=220 eval_loss=8.3306 duration_sec=14.46
67
+ 2026-04-03 10:56:38,672 [INFO] new_opacus_codex.train_steps: epoch=2 step=225 loss=8.8873
68
+ 2026-04-03 10:57:40,794 [INFO] new_opacus_codex.train_steps: epoch=2 step=230 loss=8.8330
69
+ 2026-04-03 10:57:56,006 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=230 eval_loss=8.2474 duration_sec=15.18
70
+ 2026-04-03 10:58:55,878 [INFO] new_opacus_codex.train_steps: epoch=2 step=235 loss=8.9454
71
+ 2026-04-03 11:00:29,699 [INFO] new_opacus_codex.train_steps: epoch=2 step=240 loss=8.8902
72
+ 2026-04-03 11:00:45,145 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=240 eval_loss=8.1755 duration_sec=15.41
73
+ 2026-04-03 11:01:48,119 [INFO] new_opacus_codex.train_steps: epoch=2 step=245 loss=8.6190
74
+ 2026-04-03 11:02:47,862 [INFO] new_opacus_codex.train_steps: epoch=2 step=250 loss=8.6139
75
+ 2026-04-03 11:03:20,449 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=250 eval_loss=8.1084 duration_sec=32.58
76
+ 2026-04-03 11:04:22,527 [INFO] new_opacus_codex.train_steps: epoch=2 step=255 loss=8.6254
77
+ 2026-04-03 11:05:22,573 [INFO] new_opacus_codex.train_steps: epoch=2 step=260 loss=8.4023
78
+ 2026-04-03 11:05:37,390 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=260 eval_loss=8.0503 duration_sec=14.78
79
+ 2026-04-03 11:06:38,963 [INFO] new_opacus_codex.train_steps: epoch=2 step=265 loss=8.4493
80
+ 2026-04-03 11:08:08,701 [INFO] new_opacus_codex.train_steps: epoch=2 step=270 loss=8.5491
81
+ 2026-04-03 11:09:02,968 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=270 eval_loss=7.9965 duration_sec=18.23
82
+ 2026-04-03 11:10:25,192 [INFO] new_opacus_codex.train_steps: epoch=2 step=275 loss=8.5026
83
+ 2026-04-03 11:11:29,880 [INFO] new_opacus_codex.train_steps: epoch=2 step=280 loss=8.4486
84
+ 2026-04-03 11:11:44,499 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=280 eval_loss=7.9487 duration_sec=14.59
85
+ 2026-04-03 11:12:44,784 [INFO] new_opacus_codex.train_steps: epoch=2 step=285 loss=8.4456
86
+ 2026-04-03 11:13:45,974 [INFO] new_opacus_codex.train_steps: epoch=2 step=290 loss=8.3958
87
+ 2026-04-03 11:14:20,720 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=290 eval_loss=7.9004 duration_sec=34.71
88
+ 2026-04-03 11:15:24,822 [INFO] new_opacus_codex.train_steps: epoch=2 step=295 loss=8.3510
89
+ 2026-04-03 11:16:27,042 [INFO] new_opacus_codex.train_steps: epoch=2 step=300 loss=8.3889
90
+ 2026-04-03 11:17:01,903 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=300 eval_loss=7.8574 duration_sec=34.83
91
+ 2026-04-03 11:18:05,219 [INFO] new_opacus_codex.train_steps: epoch=2 step=305 loss=8.3666
92
+ 2026-04-03 11:19:21,608 [INFO] new_opacus_codex.train_steps: epoch=2 step=310 loss=8.3727
93
+ 2026-04-03 11:19:41,828 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=310 eval_loss=7.8196 duration_sec=18.12
94
+ 2026-04-03 11:20:45,251 [INFO] new_opacus_codex.train_steps: epoch=2 step=315 loss=8.3773
95
+ 2026-04-03 11:21:46,855 [INFO] new_opacus_codex.train_steps: epoch=2 step=320 loss=8.4136
96
+ 2026-04-03 11:22:01,359 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=320 eval_loss=7.7875 duration_sec=14.46
97
+ 2026-04-03 11:23:26,571 [INFO] new_opacus_codex.train_steps: epoch=2 step=325 loss=8.3244
98
+ 2026-04-03 11:24:35,022 [INFO] new_opacus_codex.train_steps: epoch=2 step=330 loss=8.2936
99
+ 2026-04-03 11:25:03,467 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=330 eval_loss=7.7551 duration_sec=28.40
100
+ 2026-04-03 11:26:02,851 [INFO] new_opacus_codex.train_steps: epoch=2 step=335 loss=8.3059
101
+ 2026-04-03 11:27:02,091 [INFO] new_opacus_codex.train_steps: epoch=2 step=340 loss=8.1050
102
+ 2026-04-03 11:27:16,620 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=340 eval_loss=7.7204 duration_sec=14.51
103
+ 2026-04-03 11:28:17,163 [INFO] new_opacus_codex.train_steps: epoch=2 step=345 loss=7.9633
104
+ 2026-04-03 11:29:18,615 [INFO] new_opacus_codex.train_steps: epoch=2 step=350 loss=8.0919
105
+ 2026-04-03 11:29:33,094 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=350 eval_loss=7.6903 duration_sec=14.45
106
+ 2026-04-03 11:30:35,194 [INFO] new_opacus_codex.train_steps: epoch=2 step=355 loss=8.3564
107
+ 2026-04-03 11:31:37,818 [INFO] new_opacus_codex.train_steps: epoch=2 step=360 loss=8.3715
108
+ 2026-04-03 11:32:12,044 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=360 eval_loss=7.6615 duration_sec=34.20
109
+ 2026-04-03 11:33:23,480 [INFO] new_opacus_codex.train_steps: epoch=2 step=365 loss=8.1965
110
+ 2026-04-03 11:34:24,685 [INFO] new_opacus_codex.train_steps: epoch=2 step=370 loss=8.0443
111
+ 2026-04-03 11:35:05,426 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=2 step=370 eval_loss=7.6310 duration_sec=40.71
112
+ 2026-04-03 11:37:30,828 [INFO] new_opacus_codex.train_steps: epoch=3 step=375 loss=7.9051
113
+ 2026-04-03 11:38:37,202 [INFO] new_opacus_codex.train_steps: epoch=3 step=380 loss=7.9499
114
+ 2026-04-03 11:38:51,891 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=380 eval_loss=7.6043 duration_sec=14.66
115
+ 2026-04-03 11:39:53,762 [INFO] new_opacus_codex.train_steps: epoch=3 step=385 loss=8.1092
116
+ 2026-04-03 11:41:22,238 [INFO] new_opacus_codex.train_steps: epoch=3 step=390 loss=8.1437
117
+ 2026-04-03 11:41:41,445 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=390 eval_loss=7.5765 duration_sec=19.18
118
+ 2026-04-03 11:42:46,695 [INFO] new_opacus_codex.train_steps: epoch=3 step=395 loss=8.0415
119
+ 2026-04-03 11:43:50,010 [INFO] new_opacus_codex.train_steps: epoch=3 step=400 loss=7.9756
120
+ 2026-04-03 11:44:04,650 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=400 eval_loss=7.5511 duration_sec=14.63
121
+ 2026-04-03 11:45:07,540 [INFO] new_opacus_codex.train_steps: epoch=3 step=405 loss=7.9202
122
+ 2026-04-03 11:46:10,484 [INFO] new_opacus_codex.train_steps: epoch=3 step=410 loss=8.0224
123
+ 2026-04-03 11:46:45,052 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=410 eval_loss=7.5256 duration_sec=34.56
124
+ 2026-04-03 11:48:09,140 [INFO] new_opacus_codex.train_steps: epoch=3 step=415 loss=7.9499
125
+ 2026-04-03 11:49:13,723 [INFO] new_opacus_codex.train_steps: epoch=3 step=420 loss=7.9213
126
+ 2026-04-03 11:49:48,276 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=420 eval_loss=7.5046 duration_sec=34.54
127
+ 2026-04-03 11:51:36,319 [INFO] new_opacus_codex.train_steps: epoch=3 step=425 loss=7.9094
128
+ 2026-04-03 11:53:47,016 [INFO] new_opacus_codex.train_steps: epoch=3 step=430 loss=7.8593
129
+ 2026-04-03 11:54:06,349 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=430 eval_loss=7.4816 duration_sec=19.27
130
+ 2026-04-03 11:55:14,639 [INFO] new_opacus_codex.train_steps: epoch=3 step=435 loss=7.8190
131
+ 2026-04-03 11:56:19,092 [INFO] new_opacus_codex.train_steps: epoch=3 step=440 loss=7.6230
132
+ 2026-04-03 11:56:49,898 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=440 eval_loss=7.4615 duration_sec=15.33
133
+ 2026-04-03 11:57:59,971 [INFO] new_opacus_codex.train_steps: epoch=3 step=445 loss=7.7097
134
+ 2026-04-03 11:59:20,077 [INFO] new_opacus_codex.train_steps: epoch=3 step=450 loss=7.9315
135
+ 2026-04-03 11:59:34,626 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=450 eval_loss=7.4414 duration_sec=14.54
136
+ 2026-04-03 12:00:38,719 [INFO] new_opacus_codex.train_steps: epoch=3 step=455 loss=8.0988
137
+ 2026-04-03 12:01:42,590 [INFO] new_opacus_codex.train_steps: epoch=3 step=460 loss=8.0120
138
+ 2026-04-03 12:02:00,095 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=460 eval_loss=7.4212 duration_sec=17.47
139
+ 2026-04-03 12:03:53,129 [INFO] new_opacus_codex.train_steps: epoch=3 step=465 loss=7.8972
140
+ 2026-04-03 12:04:53,789 [INFO] new_opacus_codex.train_steps: epoch=3 step=470 loss=7.9732
141
+ 2026-04-03 12:05:37,590 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=470 eval_loss=7.4015 duration_sec=33.26
142
+ 2026-04-03 12:06:49,651 [INFO] new_opacus_codex.train_steps: epoch=3 step=475 loss=8.0116
143
+ 2026-04-03 12:07:55,021 [INFO] new_opacus_codex.train_steps: epoch=3 step=480 loss=8.0135
144
+ 2026-04-03 12:08:10,206 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=480 eval_loss=7.3827 duration_sec=15.18
145
+ 2026-04-03 12:09:13,641 [INFO] new_opacus_codex.train_steps: epoch=3 step=485 loss=7.8555
146
+ 2026-04-03 12:10:20,433 [INFO] new_opacus_codex.train_steps: epoch=3 step=490 loss=7.8680
147
+ 2026-04-03 12:10:35,153 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=490 eval_loss=7.3645 duration_sec=14.69
148
+ 2026-04-03 12:11:35,950 [INFO] new_opacus_codex.train_steps: epoch=3 step=495 loss=8.0610
149
+ 2026-04-03 12:12:36,883 [INFO] new_opacus_codex.train_steps: epoch=3 step=500 loss=8.0193
150
+ 2026-04-03 12:12:55,838 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=500 eval_loss=7.3504 duration_sec=18.94
151
+ 2026-04-03 12:13:58,360 [INFO] new_opacus_codex.train_steps: epoch=3 step=505 loss=7.8643
152
+ 2026-04-03 12:15:23,732 [INFO] new_opacus_codex.train_steps: epoch=3 step=510 loss=7.8505
153
+ 2026-04-03 12:15:38,252 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=510 eval_loss=7.3329 duration_sec=14.51
154
+ 2026-04-03 12:16:41,120 [INFO] new_opacus_codex.train_steps: epoch=3 step=515 loss=7.7646
155
+ 2026-04-03 12:17:50,866 [INFO] new_opacus_codex.train_steps: epoch=3 step=520 loss=7.7706
156
+ 2026-04-03 12:18:52,992 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=520 eval_loss=7.3169 duration_sec=62.09
157
+ 2026-04-03 12:19:58,810 [INFO] new_opacus_codex.train_steps: epoch=3 step=525 loss=7.7613
158
+ 2026-04-03 12:21:05,153 [INFO] new_opacus_codex.train_steps: epoch=3 step=530 loss=7.6659
159
+ 2026-04-03 12:21:20,152 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=530 eval_loss=7.3021 duration_sec=14.98
160
+ 2026-04-03 12:22:23,408 [INFO] new_opacus_codex.train_steps: epoch=3 step=535 loss=7.9512
161
+ 2026-04-03 12:23:28,087 [INFO] new_opacus_codex.train_steps: epoch=3 step=540 loss=7.9026
162
+ 2026-04-03 12:24:02,433 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=540 eval_loss=7.2878 duration_sec=34.33
163
+ 2026-04-03 12:25:05,614 [INFO] new_opacus_codex.train_steps: epoch=3 step=545 loss=7.7804
164
+ 2026-04-03 12:26:10,184 [INFO] new_opacus_codex.train_steps: epoch=3 step=550 loss=7.8074
165
+ 2026-04-03 12:26:25,396 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=3 step=550 eval_loss=7.2748 duration_sec=15.15
166
+ 2026-04-03 12:27:26,544 [INFO] new_opacus_codex.train_steps: epoch=3 step=555 loss=7.6945
167
+ 2026-04-03 12:29:19,041 [INFO] new_opacus_codex.train_steps: epoch=4 step=560 loss=7.6630
168
+ 2026-04-03 12:29:33,713 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=560 eval_loss=7.2617 duration_sec=14.64
169
+ 2026-04-03 12:30:54,117 [INFO] new_opacus_codex.train_steps: epoch=4 step=565 loss=7.6891
170
+ 2026-04-03 12:31:59,533 [INFO] new_opacus_codex.train_steps: epoch=4 step=570 loss=7.6757
171
+ 2026-04-03 12:32:24,521 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=570 eval_loss=7.2489 duration_sec=24.96
172
+ 2026-04-03 12:33:29,092 [INFO] new_opacus_codex.train_steps: epoch=4 step=575 loss=7.6532
173
+ 2026-04-03 12:36:12,951 [INFO] new_opacus_codex.train_steps: epoch=4 step=580 loss=7.6620
174
+ 2026-04-03 12:36:28,558 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=580 eval_loss=7.2363 duration_sec=15.59
175
+ 2026-04-03 12:37:33,212 [INFO] new_opacus_codex.train_steps: epoch=4 step=585 loss=7.7592
176
+ 2026-04-03 12:38:36,961 [INFO] new_opacus_codex.train_steps: epoch=4 step=590 loss=7.8426
177
+ 2026-04-03 12:38:51,640 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=590 eval_loss=7.2239 duration_sec=14.65
178
+ 2026-04-03 12:39:54,682 [INFO] new_opacus_codex.train_steps: epoch=4 step=595 loss=7.7238
179
+ 2026-04-03 12:41:14,096 [INFO] new_opacus_codex.train_steps: epoch=4 step=600 loss=7.4597
180
+ 2026-04-03 12:41:28,734 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=600 eval_loss=7.2119 duration_sec=14.61
181
+ 2026-04-03 12:42:29,644 [INFO] new_opacus_codex.train_steps: epoch=4 step=605 loss=7.6092
182
+ 2026-04-03 12:43:32,931 [INFO] new_opacus_codex.train_steps: epoch=4 step=610 loss=7.7770
183
+ 2026-04-03 12:43:47,716 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=610 eval_loss=7.2005 duration_sec=14.72
184
+ 2026-04-03 12:44:49,695 [INFO] new_opacus_codex.train_steps: epoch=4 step=615 loss=7.7264
185
+ 2026-04-03 12:45:49,187 [INFO] new_opacus_codex.train_steps: epoch=4 step=620 loss=7.7598
186
+ 2026-04-03 12:46:03,621 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=620 eval_loss=7.1896 duration_sec=14.42
187
+ 2026-04-03 12:47:05,423 [INFO] new_opacus_codex.train_steps: epoch=4 step=625 loss=7.7411
188
+ 2026-04-03 12:48:05,633 [INFO] new_opacus_codex.train_steps: epoch=4 step=630 loss=7.8418
189
+ 2026-04-03 12:48:20,179 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=630 eval_loss=7.1796 duration_sec=14.45
190
+ 2026-04-03 12:49:21,611 [INFO] new_opacus_codex.train_steps: epoch=4 step=635 loss=7.7392
191
+ 2026-04-03 12:50:22,844 [INFO] new_opacus_codex.train_steps: epoch=4 step=640 loss=7.6457
192
+ 2026-04-03 12:50:37,628 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=640 eval_loss=7.1693 duration_sec=14.77
193
+ 2026-04-03 12:51:42,589 [INFO] new_opacus_codex.train_steps: epoch=4 step=645 loss=7.6846
194
+ 2026-04-03 12:52:46,479 [INFO] new_opacus_codex.train_steps: epoch=4 step=650 loss=7.6027
195
+ 2026-04-03 12:53:01,125 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=650 eval_loss=7.1586 duration_sec=14.64
196
+ 2026-04-03 12:54:00,571 [INFO] new_opacus_codex.train_steps: epoch=4 step=655 loss=7.6670
197
+ 2026-04-03 12:55:04,060 [INFO] new_opacus_codex.train_steps: epoch=4 step=660 loss=7.6276
198
+ 2026-04-03 12:55:18,775 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=660 eval_loss=7.1486 duration_sec=14.71
199
+ 2026-04-03 12:56:18,742 [INFO] new_opacus_codex.train_steps: epoch=4 step=665 loss=7.6492
200
+ 2026-04-03 12:57:17,478 [INFO] new_opacus_codex.train_steps: epoch=4 step=670 loss=7.7340
201
+ 2026-04-03 12:57:31,839 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=670 eval_loss=7.1389 duration_sec=14.35
202
+ 2026-04-03 12:58:30,737 [INFO] new_opacus_codex.train_steps: epoch=4 step=675 loss=7.6222
203
+ 2026-04-03 12:59:29,927 [INFO] new_opacus_codex.train_steps: epoch=4 step=680 loss=7.5122
204
+ 2026-04-03 12:59:44,359 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=680 eval_loss=7.1308 duration_sec=14.43
205
+ 2026-04-03 13:00:43,556 [INFO] new_opacus_codex.train_steps: epoch=4 step=685 loss=7.4593
206
+ 2026-04-03 13:01:43,351 [INFO] new_opacus_codex.train_steps: epoch=4 step=690 loss=7.5230
207
+ 2026-04-03 13:02:01,556 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=690 eval_loss=7.1227 duration_sec=18.19
208
+ 2026-04-03 13:03:06,420 [INFO] new_opacus_codex.train_steps: epoch=4 step=695 loss=7.7165
209
+ 2026-04-03 13:04:10,936 [INFO] new_opacus_codex.train_steps: epoch=4 step=700 loss=7.7122
210
+ 2026-04-03 13:04:25,408 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=700 eval_loss=7.1142 duration_sec=14.44
211
+ 2026-04-03 13:05:25,378 [INFO] new_opacus_codex.train_steps: epoch=4 step=705 loss=7.5483
212
+ 2026-04-03 13:06:25,056 [INFO] new_opacus_codex.train_steps: epoch=4 step=710 loss=7.6590
213
+ 2026-04-03 13:06:43,083 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=710 eval_loss=7.1062 duration_sec=17.97
214
+ 2026-04-03 13:07:44,537 [INFO] new_opacus_codex.train_steps: epoch=4 step=715 loss=7.6335
215
+ 2026-04-03 13:08:47,668 [INFO] new_opacus_codex.train_steps: epoch=4 step=720 loss=7.6322
216
+ 2026-04-03 13:09:02,515 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=720 eval_loss=7.0993 duration_sec=14.83
217
+ 2026-04-03 13:10:03,862 [INFO] new_opacus_codex.train_steps: epoch=4 step=725 loss=7.6167
218
+ 2026-04-03 13:11:05,034 [INFO] new_opacus_codex.train_steps: epoch=4 step=730 loss=7.5673
219
+ 2026-04-03 13:11:21,680 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=730 eval_loss=7.0925 duration_sec=16.64
220
+ 2026-04-03 13:12:23,799 [INFO] new_opacus_codex.train_steps: epoch=4 step=735 loss=7.6362
221
+ 2026-04-03 13:13:20,949 [INFO] new_opacus_codex.train_steps: epoch=4 step=740 loss=7.6886
222
+ 2026-04-03 13:13:36,458 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=4 step=740 eval_loss=7.0855 duration_sec=15.50
223
+ 2026-04-03 13:14:50,566 [INFO] new_opacus_codex.train_steps: epoch=5 step=745 loss=7.8739
224
+ 2026-04-03 13:15:51,070 [INFO] new_opacus_codex.train_steps: epoch=5 step=750 loss=7.5509
225
+ 2026-04-03 13:16:05,488 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=750 eval_loss=7.0792 duration_sec=14.41
226
+ 2026-04-03 13:17:07,572 [INFO] new_opacus_codex.train_steps: epoch=5 step=755 loss=7.4202
227
+ 2026-04-03 13:18:07,975 [INFO] new_opacus_codex.train_steps: epoch=5 step=760 loss=7.6658
228
+ 2026-04-03 13:18:22,440 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=760 eval_loss=7.0729 duration_sec=14.44
229
+ 2026-04-03 13:19:23,593 [INFO] new_opacus_codex.train_steps: epoch=5 step=765 loss=7.6835
230
+ 2026-04-03 13:20:23,338 [INFO] new_opacus_codex.train_steps: epoch=5 step=770 loss=7.6012
231
+ 2026-04-03 13:20:37,948 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=770 eval_loss=7.0668 duration_sec=14.60
232
+ 2026-04-03 13:21:41,328 [INFO] new_opacus_codex.train_steps: epoch=5 step=775 loss=7.6148
233
+ 2026-04-03 13:22:42,292 [INFO] new_opacus_codex.train_steps: epoch=5 step=780 loss=7.7957
234
+ 2026-04-03 13:22:56,997 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=780 eval_loss=7.0619 duration_sec=14.69
235
+ 2026-04-03 13:23:59,264 [INFO] new_opacus_codex.train_steps: epoch=5 step=785 loss=7.7219
236
+ 2026-04-03 13:25:00,507 [INFO] new_opacus_codex.train_steps: epoch=5 step=790 loss=7.5480
237
+ 2026-04-03 13:25:15,254 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=790 eval_loss=7.0564 duration_sec=14.73
238
+ 2026-04-03 13:26:18,890 [INFO] new_opacus_codex.train_steps: epoch=5 step=795 loss=7.5942
239
+ 2026-04-03 13:27:21,325 [INFO] new_opacus_codex.train_steps: epoch=5 step=800 loss=7.7524
240
+ 2026-04-03 13:27:35,700 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=800 eval_loss=7.0511 duration_sec=14.35
241
+ 2026-04-03 13:28:39,316 [INFO] new_opacus_codex.train_steps: epoch=5 step=805 loss=7.7531
242
+ 2026-04-03 13:29:42,751 [INFO] new_opacus_codex.train_steps: epoch=5 step=810 loss=7.5723
243
+ 2026-04-03 13:29:58,155 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=810 eval_loss=7.0455 duration_sec=15.38
244
+ 2026-04-03 13:30:58,723 [INFO] new_opacus_codex.train_steps: epoch=5 step=815 loss=7.5057
245
+ 2026-04-03 13:32:01,210 [INFO] new_opacus_codex.train_steps: epoch=5 step=820 loss=7.3655
246
+ 2026-04-03 13:32:15,682 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=820 eval_loss=7.0410 duration_sec=14.47
247
+ 2026-04-03 13:33:16,570 [INFO] new_opacus_codex.train_steps: epoch=5 step=825 loss=7.4022
248
+ 2026-04-03 13:34:17,097 [INFO] new_opacus_codex.train_steps: epoch=5 step=830 loss=7.4295
249
+ 2026-04-03 13:34:31,784 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=830 eval_loss=7.0356 duration_sec=14.67
250
+ 2026-04-03 13:35:37,280 [INFO] new_opacus_codex.train_steps: epoch=5 step=835 loss=7.2104
251
+ 2026-04-03 13:36:38,395 [INFO] new_opacus_codex.train_steps: epoch=5 step=840 loss=7.3611
252
+ 2026-04-03 13:36:52,712 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=840 eval_loss=7.0306 duration_sec=14.30
253
+ 2026-04-03 13:37:55,465 [INFO] new_opacus_codex.train_steps: epoch=5 step=845 loss=7.4621
254
+ 2026-04-03 13:38:57,888 [INFO] new_opacus_codex.train_steps: epoch=5 step=850 loss=7.3961
255
+ 2026-04-03 13:39:12,680 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=850 eval_loss=7.0261 duration_sec=14.79
256
+ 2026-04-03 13:40:15,309 [INFO] new_opacus_codex.train_steps: epoch=5 step=855 loss=7.4920
257
+ 2026-04-03 13:41:16,800 [INFO] new_opacus_codex.train_steps: epoch=5 step=860 loss=7.4205
258
+ 2026-04-03 13:41:31,552 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=860 eval_loss=7.0213 duration_sec=14.63
259
+ 2026-04-03 13:42:36,725 [INFO] new_opacus_codex.train_steps: epoch=5 step=865 loss=7.3711
260
+ 2026-04-03 13:43:41,161 [INFO] new_opacus_codex.train_steps: epoch=5 step=870 loss=7.3564
261
+ 2026-04-03 13:43:55,476 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=870 eval_loss=7.0168 duration_sec=14.29
262
+ 2026-04-03 13:44:59,181 [INFO] new_opacus_codex.train_steps: epoch=5 step=875 loss=7.3472
263
+ 2026-04-03 13:46:02,078 [INFO] new_opacus_codex.train_steps: epoch=5 step=880 loss=7.4068
264
+ 2026-04-03 13:46:16,825 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=880 eval_loss=7.0125 duration_sec=14.71
265
+ 2026-04-03 13:47:19,268 [INFO] new_opacus_codex.train_steps: epoch=5 step=885 loss=7.4467
266
+ 2026-04-03 13:48:20,509 [INFO] new_opacus_codex.train_steps: epoch=5 step=890 loss=7.3560
267
+ 2026-04-03 13:48:40,700 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=890 eval_loss=7.0082 duration_sec=20.17
268
+ 2026-04-03 13:49:43,323 [INFO] new_opacus_codex.train_steps: epoch=5 step=895 loss=7.3445
269
+ 2026-04-03 13:50:42,867 [INFO] new_opacus_codex.train_steps: epoch=5 step=900 loss=7.4890
270
+ 2026-04-03 13:50:57,671 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=900 eval_loss=7.0037 duration_sec=14.79
271
+ 2026-04-03 13:51:59,531 [INFO] new_opacus_codex.train_steps: epoch=5 step=905 loss=7.6684
272
+ 2026-04-03 13:53:02,910 [INFO] new_opacus_codex.train_steps: epoch=5 step=910 loss=7.5723
273
+ 2026-04-03 13:53:19,116 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=910 eval_loss=6.9997 duration_sec=16.17
274
+ 2026-04-03 13:54:23,170 [INFO] new_opacus_codex.train_steps: epoch=5 step=915 loss=7.5004
275
+ 2026-04-03 13:55:22,836 [INFO] new_opacus_codex.train_steps: epoch=5 step=920 loss=7.6176
276
+ 2026-04-03 13:55:38,765 [INFO] new_opacus_codex.train_steps: eval event=eval_step epoch=5 step=920 eval_loss=6.9958 duration_sec=15.91
277
+ 2026-04-03 13:56:38,329 [INFO] new_opacus_codex.train_steps: epoch=5 step=925 loss=7.3952