Hugging Face's logo

SwissNeuron
/
Qwen3.8-27B-SwissNeuron-Derisked

Image-Text-to-Text
Transformers
Safetensors
qwen3_5
qwen3.8
bfloat16
long-context
yarn
1m-context
multimodal
vision
reasoning
swissneuron
conversational
Model card Files
xet
 Community
Swiss-guy commited on
Commit
4d437b3
·
verified ·
1 Parent(s): 872fed3

Release SwissNeuron Qwen3.8-27B Derisked BF16 with 1M YaRN configuration

Browse files
Files changed (26) hide show
  1. .gitattributes +1 -0
  2. README.md +151 -0
  3. args.json +404 -0
  4. chat_template.jinja +170 -0
  5. config.json +153 -0
  6. config.native-262k.json +143 -0
  7. dwm_provenance.json +38 -0
  8. generation_config.json +13 -0
  9. model-00001-of-00012.safetensors +3 -0
  10. model-00002-of-00012.safetensors +3 -0
  11. model-00003-of-00012.safetensors +3 -0
  12. model-00004-of-00012.safetensors +3 -0
  13. model-00005-of-00012.safetensors +3 -0
  14. model-00006-of-00012.safetensors +3 -0
  15. model-00007-of-00012.safetensors +3 -0
  16. model-00008-of-00012.safetensors +3 -0
  17. model-00009-of-00012.safetensors +3 -0
  18. model-00010-of-00012.safetensors +3 -0
  19. model-00011-of-00012.safetensors +3 -0
  20. model-00012-of-00012.safetensors +3 -0
  21. model.safetensors.index.json +0 -0
  22. preprocessor_config.json +21 -0
  23. processor_config.json +60 -0
  24. refusal_direction_bank.pt +3 -0
  25. tokenizer.json +3 -0
  26. tokenizer_config.json +33 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,151 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ base_model: Qwen/Qwen3.8-27B
4
+ pipeline_tag: text-generation
5
+ library_name: transformers
6
+ tags:
7
+ - qwen3.8
8
+ - qwen3_5
9
+ - bfloat16
10
+ - long-context
11
+ - yarn
12
+ - 1m-context
13
+ - swissneuron
14
+ model_name: SwissNeuron
15
+ ---
16
+
17
+ # SwissNeuron
18
+
19
+ **SwissNeuron** is a BF16, 27B-parameter Qwen3.8 derivative engineered in Switzerland for direct technical work, strong reasoning, and capability retention. It combines focused in-house post-training with a conservative internal geometric derisk procedure, then extends the model configuration to a 1,048,576-token context window using factor-4 YaRN/RoPE scaling.
20
+
21
+ SwissNeuron is intended to provide Swiss-quality model engineering: precise provenance, conservative weight surgery, reproducible artifacts, and transparent limitations.
22
+
23
+ ## Highlights
24
+
25
+ - **27.36B BF16 parameters**
26
+ - **Qwen3.8 hybrid architecture**: 64 language layers, 48 Gated-DeltaNet layers and 16 full-attention layers
27
+ - **1M context configuration**: 1,048,576 tokens through factor-4 YaRN over the original 262,144-token window
28
+ - **Capability-preserving post-training**: focused internal training rather than broad, high-learning-rate continued pretraining
29
+ - **Conservative derisking**: a low-strength, single-pass internal direction-removal procedure applied only after training
30
+ - **Original MTP, multimodal processor, tokenizer, and chat template retained**
31
+ - **No quantization**: this release contains the merged BF16 weights; quantized editions will follow separately
32
+
33
+ ## What makes SwissNeuron different
34
+
35
+ Many aggressively modified or “uncensored” checkpoints trade away reasoning quality, instruction fidelity, or language-model calibration. SwissNeuron was built around the opposite objective: alter behavior while minimizing movement outside the targeted representation subspace.
36
+
37
+ The model was **not** produced by merging an existing Heretic-style, abliteration, or third-party surgically modified checkpoint. The starting point was the official Qwen3.8-27B BF16 model. SwissNeuron was then trained on a focused internal direct-answer corpus and derisked with an in-house pipeline using a direction bank recaptured from the final trained model itself. The released candidate uses a low α=0.1, one-pass edit, skips the first two layers, preserves global weight norms, and leaves the model architecture intact.
38
+
39
+ On an internal frozen holdout, the trained model improved token-level likelihood and accuracy relative to the official starting checkpoint before the final low-strength geometric pass. This release was selected for conservative weight movement and capability preservation rather than maximum behavioral alteration.
40
+
41
+ ## Long-context configuration
42
+
43
+ The original Qwen3.8 context window is 262,144 tokens. SwissNeuron sets:
44
+
45
+ ```json
46
+ {
47
+ "max_position_embeddings": 1048576,
48
+ "rope_type": "yarn",
49
+ "factor": 4.0,
50
+ "original_max_position_embeddings": 262144
51
+ }
52
+ ```
53
+
54
+ This is a **configuration-level YaRN extension**. The model has not yet undergone a dedicated 1M-token long-context adaptation stage, so quality near the extreme end of the window can depend on the serving stack, attention backend, prompt structure, and workload. Validate retrieval and generation quality for your own 1M-context use case. Servers must support Qwen3.8/Qwen3.5 hybrid linear attention and the included YaRN parameters.
55
+
56
+ ## Architecture
57
+
58
+ | Property | Value |
59
+ |---|---:|
60
+ | Parameters | 27,356,728,560 |
61
+ | Weight dtype | BF16 |
62
+ | Hidden size | 5,120 |
63
+ | Language layers | 64 |
64
+ | Full-attention interval | 4 |
65
+ | Full-attention layers | 16 |
66
+ | Gated-DeltaNet layers | 48 |
67
+ | FFN intermediate size | 17,408 |
68
+ | Vocabulary | 248,320 |
69
+ | Original context | 262,144 |
70
+ | Configured context | 1,048,576 |
71
+ | MTP layers | 1 |
72
+
73
+ ## Usage
74
+
75
+ Use a recent Transformers release with Qwen3.8/Qwen3.5 hybrid support:
76
+
77
+ ```python
78
+ import torch
79
+ from transformers import AutoModelForImageTextToText, AutoTokenizer
80
+
81
+ model_id = "swissneuron/SwissNeuron"
82
+
83
+ tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
84
+ model = AutoModelForImageTextToText.from_pretrained(
85
+ model_id,
86
+ dtype=torch.bfloat16,
87
+ device_map="auto",
88
+ trust_remote_code=True,
89
+ )
90
+
91
+ messages = [{"role": "user", "content": "Analyze this problem carefully."}]
92
+ text = tokenizer.apply_chat_template(
93
+ messages,
94
+ tokenize=False,
95
+ add_generation_prompt=True,
96
+ enable_thinking=True,
97
+ )
98
+ inputs = tokenizer(text, return_tensors="pt").to(model.device)
99
+ output = model.generate(**inputs, max_new_tokens=2048)
100
+ print(tokenizer.decode(output[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
101
+ ```
102
+
103
+ For production inference, use a serving engine that explicitly supports `Qwen3_5ForConditionalGeneration`, Gated-DeltaNet state, multimodal RoPE, and YaRN at the requested context length. A 1M context allocation requires substantial KV/state memory even with the hybrid architecture.
104
+
105
+ ## Recommended generation settings
106
+
107
+ Thinking/coding workloads:
108
+
109
+ ```text
110
+ temperature=0.6
111
+ top_p=0.95
112
+ top_k=20
113
+ ```
114
+
115
+ Direct non-thinking workloads should be tuned against the deployment task. Preserve the included chat template and explicitly select thinking mode rather than manually inserting reasoning markers.
116
+
117
+ ## Training and derisk provenance
118
+
119
+ - Official Qwen3.8-27B BF16 starting weights
120
+ - Focused in-house internal post-training corpus
121
+ - Full-hybrid LoRA targets covering attention, Gated-DeltaNet input/output projections, and MLP projections
122
+ - LoRA merged into the pristine official base
123
+ - Internal representation bank recaptured from the final merged model
124
+ - One-pass norm-preserving geometric edit
125
+ - α=0.1
126
+ - Layers 0–1 excluded from editing
127
+ - 124 residual-write matrices edited
128
+ - No secondary direction and no repeated passes
129
+
130
+ The exact direction bank and build provenance are included for internal reproducibility.
131
+
132
+ ## Roadmap
133
+
134
+ SwissNeuron is the base release in a broader model series. Planned follow-ups include:
135
+
136
+ - Quantized deployment editions
137
+ - Dedicated long-context adaptation and 1M retrieval evaluations
138
+ - A larger fine-tuned SwissNeuron edition trained on approximately **10B distilled tokens** from frontier teacher workflows, including Claude Opus 5 and Mythos-class research trajectories, subject to final data-quality and release review
139
+
140
+ The larger fine-tuned release will be published as a separate checkpoint rather than silently replacing this BF16 base.
141
+
142
+ ## Limitations
143
+
144
+ - The 1M window is enabled through YaRN configuration; extreme-context quality has not yet been comprehensively validated.
145
+ - This is a full BF16 release and requires substantial accelerator memory.
146
+ - Outputs may be inaccurate, incomplete, or unsuitable for a particular domain. Independently verify consequential results.
147
+ - This repository is private while licensing, long-context, and release checks are completed.
148
+
149
+ ## Release
150
+
151
+ Built by **SwissNeuron** in Switzerland. Quantized variants and the larger distilled fine-tune are planned as separate releases.
args.json ADDED
@@ -0,0 +1,404 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "output_dir": "/workspace/train-qwen38/models/karma-qwen38-nonrefusal-sft/v0-20260823-131002",
3
+ "per_device_train_batch_size": 1,
4
+ "num_train_epochs": 1.0,
5
+ "max_steps": -1,
6
+ "learning_rate": 0.0001,
7
+ "lr_scheduler_type": "cosine",
8
+ "lr_scheduler_kwargs": null,
9
+ "warmup_steps": 5.0,
10
+ "optim": "adamw_torch_fused",
11
+ "optim_args": null,
12
+ "weight_decay": 0.01,
13
+ "adam_beta1": 0.9,
14
+ "adam_beta2": 0.95,
15
+ "adam_epsilon": 1e-08,
16
+ "optim_target_modules": null,
17
+ "gradient_accumulation_steps": 2,
18
+ "average_tokens_across_devices": true,
19
+ "max_grad_norm": 1.0,
20
+ "label_smoothing_factor": 0.0,
21
+ "bf16": true,
22
+ "fp16": false,
23
+ "bf16_full_eval": false,
24
+ "fp16_full_eval": false,
25
+ "tf32": null,
26
+ "gradient_checkpointing": true,
27
+ "gradient_checkpointing_kwargs": null,
28
+ "torch_compile": false,
29
+ "torch_compile_backend": null,
30
+ "torch_compile_mode": null,
31
+ "use_liger_kernel": true,
32
+ "liger_kernel_config": null,
33
+ "use_cache": false,
34
+ "neftune_noise_alpha": null,
35
+ "torch_empty_cache_steps": null,
36
+ "auto_find_batch_size": false,
37
+ "logging_strategy": "steps",
38
+ "logging_steps": 5,
39
+ "logging_first_step": true,
40
+ "log_on_each_node": true,
41
+ "logging_nan_inf_filter": true,
42
+ "include_num_input_tokens_seen": false,
43
+ "log_level": "passive",
44
+ "log_level_replica": "warning",
45
+ "disable_tqdm": null,
46
+ "report_to": [
47
+ "tensorboard"
48
+ ],
49
+ "run_name": "/workspace/train-qwen38/models/karma-qwen38-nonrefusal-sft/v0-20260823-131002",
50
+ "project": "huggingface",
51
+ "trackio_space_id": null,
52
+ "trackio_bucket_id": null,
53
+ "trackio_static_space_id": null,
54
+ "eval_strategy": "steps",
55
+ "eval_steps": 50.0,
56
+ "eval_delay": 0,
57
+ "per_device_eval_batch_size": 1,
58
+ "prediction_loss_only": false,
59
+ "eval_on_start": false,
60
+ "eval_do_concat_batches": true,
61
+ "eval_use_gather_object": false,
62
+ "eval_accumulation_steps": null,
63
+ "include_for_metrics": [],
64
+ "batch_eval_metrics": false,
65
+ "save_only_model": false,
66
+ "save_strategy": "steps",
67
+ "save_steps": 50.0,
68
+ "save_on_each_node": false,
69
+ "save_total_limit": 8,
70
+ "enable_jit_checkpoint": false,
71
+ "push_to_hub": false,
72
+ "hub_token": null,
73
+ "hub_private_repo": null,
74
+ "hub_model_id": null,
75
+ "hub_strategy": "every_save",
76
+ "hub_always_push": false,
77
+ "hub_revision": null,
78
+ "load_best_model_at_end": false,
79
+ "metric_for_best_model": "loss",
80
+ "greater_is_better": false,
81
+ "ignore_data_skip": false,
82
+ "restore_callback_states_from_checkpoint": false,
83
+ "full_determinism": false,
84
+ "seed": 42,
85
+ "data_seed": 42,
86
+ "use_cpu": false,
87
+ "accelerator_config": {
88
+ "dispatch_batches": false
89
+ },
90
+ "parallelism_config": null,
91
+ "dataloader_drop_last": false,
92
+ "dataloader_num_workers": 4,
93
+ "dataloader_pin_memory": true,
94
+ "dataloader_persistent_workers": false,
95
+ "dataloader_prefetch_factor": null,
96
+ "remove_unused_columns": true,
97
+ "label_names": null,
98
+ "train_sampling_strategy": "random",
99
+ "length_column_name": "length",
100
+ "ddp_find_unused_parameters": null,
101
+ "ddp_bucket_cap_mb": null,
102
+ "ddp_broadcast_buffers": null,
103
+ "ddp_static_graph": null,
104
+ "ddp_backend": null,
105
+ "ddp_timeout": 18000000,
106
+ "fsdp": [],
107
+ "fsdp_config": null,
108
+ "deepspeed": {
109
+ "fp16": {
110
+ "enabled": "auto",
111
+ "loss_scale": 0,
112
+ "loss_scale_window": 1000,
113
+ "initial_scale_power": 16,
114
+ "hysteresis": 2,
115
+ "min_loss_scale": 1
116
+ },
117
+ "bf16": {
118
+ "enabled": "auto"
119
+ },
120
+ "zero_optimization": {
121
+ "stage": 3,
122
+ "offload_optimizer": {
123
+ "device": "none",
124
+ "pin_memory": true
125
+ },
126
+ "offload_param": {
127
+ "device": "none",
128
+ "pin_memory": true
129
+ },
130
+ "overlap_comm": false,
131
+ "contiguous_gradients": true,
132
+ "sub_group_size": 1000000000.0,
133
+ "reduce_bucket_size": "auto",
134
+ "zero_quantized_weights": false,
135
+ "zero_quantized_gradients": false,
136
+ "stage3_prefetch_bucket_size": "auto",
137
+ "stage3_param_persistence_threshold": "auto",
138
+ "stage3_max_live_parameters": 1000000000.0,
139
+ "stage3_max_reuse_distance": 1000000000.0,
140
+ "stage3_gather_16bit_weights_on_model_save": true
141
+ },
142
+ "gradient_accumulation_steps": "auto",
143
+ "gradient_clipping": "auto",
144
+ "steps_per_print": 2000,
145
+ "train_batch_size": "auto",
146
+ "train_micro_batch_size_per_gpu": "auto",
147
+ "wall_clock_breakdown": false
148
+ },
149
+ "debug": null,
150
+ "skip_memory_metrics": true,
151
+ "do_train": false,
152
+ "do_eval": false,
153
+ "do_predict": false,
154
+ "resume_from_checkpoint": null,
155
+ "warmup_ratio": null,
156
+ "logging_dir": "/workspace/train-qwen38/models/karma-qwen38-nonrefusal-sft/v0-20260823-131002/runs",
157
+ "local_rank": 0,
158
+ "sortish_sampler": false,
159
+ "predict_with_generate": false,
160
+ "generation_max_length": null,
161
+ "generation_num_beams": null,
162
+ "generation_config": null,
163
+ "tuner_backend": "peft",
164
+ "vit_gradient_checkpointing": false,
165
+ "router_aux_loss_coef": 0.0,
166
+ "enable_dft_loss": false,
167
+ "enable_channel_loss": false,
168
+ "safe_serialization": true,
169
+ "max_shard_size": "5GB",
170
+ "check_model": false,
171
+ "acc_strategy": "token",
172
+ "train_dataloader_shuffle": true,
173
+ "group_by_length": false,
174
+ "max_epochs": null,
175
+ "aligner_lr": null,
176
+ "vit_lr": null,
177
+ "use_logits_to_keep": null,
178
+ "ds3_gather_for_generation": true,
179
+ "resume_only_model": false,
180
+ "optimizer": null,
181
+ "loss_type": null,
182
+ "mrl_dims": null,
183
+ "eval_metric": null,
184
+ "callbacks": [],
185
+ "early_stop_interval": null,
186
+ "eval_use_evalscope": false,
187
+ "eval_dataset": [],
188
+ "eval_dataset_args": null,
189
+ "eval_limit": null,
190
+ "eval_generation_config": null,
191
+ "extra_eval_args": null,
192
+ "tuner_type": "lora",
193
+ "use_galore": false,
194
+ "galore_target_modules": null,
195
+ "galore_rank": 128,
196
+ "galore_update_proj_gap": 50,
197
+ "galore_scale": 1.0,
198
+ "galore_proj_type": "std",
199
+ "galore_optim_per_parameter": false,
200
+ "galore_with_embedding": false,
201
+ "galore_quantization": false,
202
+ "galore_proj_quant": false,
203
+ "galore_proj_bits": 4,
204
+ "galore_proj_group_size": 256,
205
+ "galore_cos_threshold": 0.4,
206
+ "galore_gamma_proj": 2,
207
+ "galore_queue_size": 5,
208
+ "lisa_activated_layers": 0,
209
+ "lisa_step_interval": 20,
210
+ "use_flash_ckpt": false,
211
+ "use_ray": false,
212
+ "ray_exp_name": null,
213
+ "device_groups": null,
214
+ "model": "/root/Qwen3.8-27B",
215
+ "model_type": "qwen3_5",
216
+ "model_revision": null,
217
+ "task_type": "causal_lm",
218
+ "torch_dtype": "bfloat16",
219
+ "attn_impl": "flash_attn",
220
+ "experts_impl": null,
221
+ "new_special_tokens": [],
222
+ "num_labels": null,
223
+ "problem_type": null,
224
+ "rope_scaling": null,
225
+ "device_map": null,
226
+ "max_memory": {},
227
+ "max_model_len": null,
228
+ "local_repo_path": null,
229
+ "init_strategy": null,
230
+ "template": "qwen3_5",
231
+ "system": null,
232
+ "max_length": 98304,
233
+ "truncation_strategy": "delete",
234
+ "max_pixels": null,
235
+ "agent_template": null,
236
+ "norm_bbox": null,
237
+ "use_chat_template": true,
238
+ "padding_side": "right",
239
+ "padding_free": true,
240
+ "loss_scale": "ignore_empty_think",
241
+ "sequence_parallel_size": 1,
242
+ "is_binary_loss_scale": null,
243
+ "template_backend": "swift",
244
+ "response_prefix": null,
245
+ "enable_thinking": null,
246
+ "preserve_thinking": null,
247
+ "add_non_thinking_prefix": true,
248
+ "disable_ignore_empty_think": false,
249
+ "dataset": [
250
+ "/workspace/train-qwen38/data/train.jsonl"
251
+ ],
252
+ "val_dataset": [
253
+ "/workspace/train-qwen38/data/holdout.jsonl"
254
+ ],
255
+ "cached_dataset": [],
256
+ "cached_val_dataset": [],
257
+ "split_dataset_ratio": 0.0,
258
+ "dataset_num_proc": 8,
259
+ "load_from_cache_file": false,
260
+ "dataset_shuffle": true,
261
+ "val_dataset_shuffle": false,
262
+ "streaming": false,
263
+ "interleave_prob": null,
264
+ "stopping_strategy": "first_exhausted",
265
+ "shuffle_buffer_size": 1000,
266
+ "download_mode": "reuse_dataset_if_exists",
267
+ "columns": {},
268
+ "strict": false,
269
+ "disable_auto_column_mapping": false,
270
+ "model_name": null,
271
+ "model_author": null,
272
+ "custom_dataset_info": [],
273
+ "quant_method": null,
274
+ "quant_bits": null,
275
+ "hqq_axis": null,
276
+ "bnb_4bit_compute_dtype": "bfloat16",
277
+ "bnb_4bit_quant_type": "nf4",
278
+ "bnb_4bit_use_double_quant": true,
279
+ "bnb_4bit_quant_storage": null,
280
+ "max_new_tokens": 64,
281
+ "temperature": 0.0,
282
+ "top_k": null,
283
+ "top_p": null,
284
+ "repetition_penalty": null,
285
+ "num_beams": 1,
286
+ "stream": false,
287
+ "stop_words": [],
288
+ "logprobs": false,
289
+ "top_logprobs": null,
290
+ "structured_outputs_regex": null,
291
+ "adapters": [],
292
+ "external_plugins": [],
293
+ "custom_register_path": [],
294
+ "model_kwargs": {},
295
+ "enable_npu_model_patch": true,
296
+ "load_args": false,
297
+ "load_data_args": false,
298
+ "packing": true,
299
+ "packing_length": 98304,
300
+ "packing_num_proc": 1,
301
+ "packing_strategy": "binpack",
302
+ "lazy_tokenize": false,
303
+ "use_hf": true,
304
+ "ignore_args_error": false,
305
+ "use_swift_lora": false,
306
+ "freeze_parameters": [],
307
+ "freeze_parameters_regex": null,
308
+ "freeze_parameters_ratio": 0.0,
309
+ "trainable_parameters": [],
310
+ "trainable_parameters_regex": null,
311
+ "freeze_llm": false,
312
+ "freeze_vit": true,
313
+ "freeze_aligner": true,
314
+ "target_modules": [
315
+ "q_proj",
316
+ "k_proj",
317
+ "v_proj",
318
+ "o_proj",
319
+ "out_proj",
320
+ "in_proj_qkv",
321
+ "in_proj_z",
322
+ "in_proj_b",
323
+ "in_proj_a",
324
+ "gate_proj",
325
+ "up_proj",
326
+ "down_proj"
327
+ ],
328
+ "target_regex": null,
329
+ "target_parameters": null,
330
+ "modules_to_save": [],
331
+ "lora_rank": 128,
332
+ "lora_alpha": 128,
333
+ "lora_dropout": 0.0,
334
+ "lora_bias": "none",
335
+ "lora_dtype": null,
336
+ "lorap_lr_ratio": null,
337
+ "use_rslora": false,
338
+ "use_dora": false,
339
+ "lora_ga_batch_size": 2,
340
+ "lora_ga_iters": 2,
341
+ "lora_ga_max_length": 1024,
342
+ "lora_ga_direction": "ArB2r",
343
+ "lora_ga_scale": "stable",
344
+ "lora_ga_stable_gamma": 16,
345
+ "init_weights": true,
346
+ "fourier_n_frequency": 2000,
347
+ "fourier_scaling": 300.0,
348
+ "boft_block_size": 4,
349
+ "boft_block_num": 0,
350
+ "boft_n_butterfly_factor": 1,
351
+ "boft_dropout": 0.0,
352
+ "vera_rank": 256,
353
+ "vera_projection_prng_key": 0,
354
+ "vera_dropout": 0.0,
355
+ "vera_d_initial": 0.1,
356
+ "adapter_act": "gelu",
357
+ "adapter_length": 128,
358
+ "adalora_target_r": 8,
359
+ "adalora_init_r": 12,
360
+ "adalora_tinit": 0,
361
+ "adalora_tfinal": 0,
362
+ "adalora_deltaT": 1,
363
+ "adalora_beta1": 0.85,
364
+ "adalora_beta2": 0.85,
365
+ "adalora_orth_reg_weight": 0.5,
366
+ "llamapro_num_new_blocks": 4,
367
+ "llamapro_num_groups": null,
368
+ "reft_layer_key": null,
369
+ "reft_layers": null,
370
+ "reft_rank": 4,
371
+ "reft_intervention_type": "LoreftIntervention",
372
+ "reft_args": null,
373
+ "swanlab_token": null,
374
+ "swanlab_project": "ms-swift",
375
+ "swanlab_workspace": null,
376
+ "swanlab_exp_name": null,
377
+ "swanlab_notification_method": null,
378
+ "swanlab_webhook_url": null,
379
+ "swanlab_secret": null,
380
+ "swanlab_sender_email": null,
381
+ "swanlab_receiver_email": null,
382
+ "swanlab_smtp_server": null,
383
+ "swanlab_smtp_port": null,
384
+ "swanlab_email_language": "zh",
385
+ "swanlab_mode": "cloud",
386
+ "add_version": true,
387
+ "create_checkpoint_symlink": false,
388
+ "zero_hpz_partition_size": null,
389
+ "deepspeed_autotp_size": null,
390
+ "swift_version": "4.4.2",
391
+ "ckpt_dir": null,
392
+ "rank": 0,
393
+ "global_world_size": 8,
394
+ "local_world_size": 8,
395
+ "model_suffix": "Qwen3.8-27B",
396
+ "model_info": "ModelInfo(model_type='qwen3_5', model_dir='/root/Qwen3.8-27B', torch_dtype=torch.bfloat16, max_model_len=262144, quant_method=None, quant_bits=None, rope_scaling=None, is_moe_model=False, is_multimodal=True, config=None, task_type='causal_lm', num_labels=None)",
397
+ "model_meta": "ModelMeta(model_type='qwen3_5', model_groups=[ModelGroup(models=[Model(ms_model_id='Qwen/Qwen3.5-0.8B', hf_model_id='Qwen/Qwen3.5-0.8B', model_path=None, ms_revision=None, hf_revision=None), Model(ms_model_id='Qwen/Qwen3.5-2B', hf_model_id='Qwen/Qwen3.5-2B', model_path=None, ms_revision=None, hf_revision=None), Model(ms_model_id='Qwen/Qwen3.5-4B', hf_model_id='Qwen/Qwen3.5-4B', model_path=None, ms_revision=None, hf_revision=None), Model(ms_model_id='Qwen/Qwen3.5-9B', hf_model_id='Qwen/Qwen3.5-9B', model_path=None, ms_revision=None, hf_revision=None), Model(ms_model_id='Qwen/Qwen3.5-27B', hf_model_id='Qwen/Qwen3.5-27B', model_path=None, ms_revision=None, hf_revision=None), Model(ms_model_id='Qwen/Qwen3.5-27B-FP8', hf_model_id='Qwen/Qwen3.5-27B-FP8', model_path=None, ms_revision=None, hf_revision=None), Model(ms_model_id='Qwen/Qwen3.5-0.8B-Base', hf_model_id='Qwen/Qwen3.5-0.8B-Base', model_path=None, ms_revision=None, hf_revision=None), Model(ms_model_id='Qwen/Qwen3.5-2B-Base', hf_model_id='Qwen/Qwen3.5-2B-Base', model_path=None, ms_revision=None, hf_revision=None), Model(ms_model_id='Qwen/Qwen3.5-4B-Base', hf_model_id='Qwen/Qwen3.5-4B-Base', model_path=None, ms_revision=None, hf_revision=None), Model(ms_model_id='Qwen/Qwen3.5-9B-Base', hf_model_id='Qwen/Qwen3.5-9B-Base', model_path=None, ms_revision=None, hf_revision=None)], template='qwen3_5', ignore_patterns=None, requires=None, tags=[]), ModelGroup(models=[Model(ms_model_id='Qwen/Qwen3.6-27B', hf_model_id='Qwen/Qwen3.6-27B', model_path=None, ms_revision=None, hf_revision=None), Model(ms_model_id='Qwen/Qwen3.6-27B-FP8', hf_model_id='Qwen/Qwen3.6-27B-FP8', model_path=None, ms_revision=None, hf_revision=None)], template='qwen3_5', ignore_patterns=None, requires=None, tags=[])], loader=<class 'swift.model.models.qwen.Qwen3_5Loader'>, template=None, model_arch=MultiModelKeys(arch_name='qwen2_vl', embedding=None, module_list=None, lm_head=None, q_proj=None, k_proj=None, v_proj=None, o_proj=None, attention=None, mlp=None, down_proj=None, qkv_proj=None, qk_proj=None, qa_proj=None, qb_proj=None, kv_proj=None, kva_proj=None, kvb_proj=None, language_model=['model.language_model', 'lm_head'], aligner=['model.visual.merger'], vision_tower=['model.visual'], generator=[]), mcore_model_type=None, architectures=['Qwen3_5ForConditionalGeneration'], additional_saved_files=[], torch_dtype=None, is_multimodal=True, is_reward=False, task_type=None, ignore_patterns=None, requires=['transformers>=5.0.0.dev', 'qwen_vl_utils>=0.0.14', 'decord'], tags=['vision', 'video'])",
398
+ "model_dir": "/root/Qwen3.8-27B",
399
+ "template_meta": "QwenTemplateMeta(template_type='qwen3_5', prefix=[], prompt=['<|im_start|>user\\n{{QUERY}}<|im_end|>\\n<|im_start|>assistant\\n'], chat_sep=['<|im_end|>\\n'], suffix=['<|im_end|>\\n'], template_cls=<class 'swift.template.templates.qwen.Qwen3_5Template'>, system_prefix=['<|im_start|>system\\n{{SYSTEM}}<|im_end|>\\n'], default_system=None, auto_add_bos=False, stop_words=['<|endoftext|>'], agent_template='qwen3_5', is_thinking=True, thinking_prefix='<think>\\n', non_thinking_prefix='<think>\\n\\n</think>\\n\\n', history_thinking_prefix='')",
400
+ "_val_dataset_exists": true,
401
+ "hub": "<class 'swift.hub.hub.HFHub'>",
402
+ "evaluation_strategy": "steps",
403
+ "training_args": "Seq2SeqTrainingArguments(output_dir='/workspace/train-qwen38/models/karma-qwen38-nonrefusal-sft/v0-20260823-131002', per_device_train_batch_size=1, num_train_epochs=1.0, max_steps=-1, learning_rate=0.0001, lr_scheduler_type=<SchedulerType.COSINE: 'cosine'>, lr_scheduler_kwargs=None, warmup_steps=5.0, optim=<OptimizerNames.ADAMW_TORCH_FUSED: 'adamw_torch_fused'>, optim_args=None, weight_decay=0.01, adam_beta1=0.9, adam_beta2=0.95, adam_epsilon=1e-08, optim_target_modules=None, gradient_accumulation_steps=2, average_tokens_across_devices=None, max_grad_norm=1.0, label_smoothing_factor=0.0, bf16=True, fp16=False, bf16_full_eval=False, fp16_full_eval=False, tf32=None, gradient_checkpointing=True, gradient_checkpointing_kwargs=None, torch_compile=False, torch_compile_backend=None, torch_compile_mode=None, use_liger_kernel=True, liger_kernel_config=None, use_cache=False, neftune_noise_alpha=None, torch_empty_cache_steps=None, auto_find_batch_size=False, logging_strategy=<IntervalStrategy.STEPS: 'steps'>, logging_steps=5, logging_first_step=True, log_on_each_node=True, logging_nan_inf_filter=True, include_num_input_tokens_seen=None, log_level='passive', log_level_replica='warning', disable_tqdm=False, report_to=['tensorboard'], run_name='/workspace/train-qwen38/models/karma-qwen38-nonrefusal-sft/v0-20260823-131002', project='huggingface', trackio_space_id=None, trackio_bucket_id=None, trackio_static_space_id=None, eval_strategy=<IntervalStrategy.STEPS: 'steps'>, eval_steps=50, eval_delay=0, per_device_eval_batch_size=1, prediction_loss_only=False, eval_on_start=False, eval_do_concat_batches=True, eval_use_gather_object=False, eval_accumulation_steps=None, include_for_metrics=[], batch_eval_metrics=False, save_only_model=False, save_strategy=<SaveStrategy.STEPS: 'steps'>, save_steps=50, save_on_each_node=False, save_total_limit=8, enable_jit_checkpoint=False, push_to_hub=False, hub_token=None, hub_private_repo=None, hub_model_id=None, hub_strategy=<HubStrategy.EVERY_SAVE: 'every_save'>, hub_always_push=False, hub_revision=None, load_best_model_at_end=False, metric_for_best_model='loss', greater_is_better=False, ignore_data_skip=False, restore_callback_states_from_checkpoint=False, full_determinism=False, seed=42, data_seed=42, use_cpu=False, accelerator_config=AcceleratorConfig(split_batches=False, dispatch_batches=False, even_batches=True, use_seedable_sampler=True, non_blocking=False, gradient_accumulation_kwargs=None, use_configured_state=False), parallelism_config=None, dataloader_drop_last=False, dataloader_num_workers=4, dataloader_pin_memory=True, dataloader_persistent_workers=False, dataloader_prefetch_factor=2, remove_unused_columns=False, label_names=None, train_sampling_strategy='random', length_column_name='length', ddp_find_unused_parameters=None, ddp_bucket_cap_mb=None, ddp_broadcast_buffers=None, ddp_static_graph=None, ddp_backend=None, ddp_timeout=18000000, fsdp=[], fsdp_config=None, deepspeed={'fp16': {'enabled': 'auto', 'loss_scale': 0, 'loss_scale_window': 1000, 'initial_scale_power': 16, 'hysteresis': 2, 'min_loss_scale': 1}, 'bf16': {'enabled': 'auto'}, 'zero_optimization': {'stage': 3, 'offload_optimizer': {'device': 'none', 'pin_memory': True}, 'offload_param': {'device': 'none', 'pin_memory': True}, 'overlap_comm': False, 'contiguous_gradients': True, 'sub_group_size': 1000000000.0, 'reduce_bucket_size': 'auto', 'zero_quantized_weights': False, 'zero_quantized_gradients': False, 'stage3_prefetch_bucket_size': 'auto', 'stage3_param_persistence_threshold': 'auto', 'stage3_max_live_parameters': 1000000000.0, 'stage3_max_reuse_distance': 1000000000.0, 'stage3_gather_16bit_weights_on_model_save': True}, 'gradient_accumulation_steps': 'auto', 'gradient_clipping': 'auto', 'steps_per_print': 2000, 'train_batch_size': 'auto', 'train_micro_batch_size_per_gpu': 'auto', 'wall_clock_breakdown': False}, debug=[], skip_memory_metrics=True, do_train=False, do_eval=True, do_predict=False, resume_from_checkpoint=None, warmup_ratio=None, logging_dir='/workspace/train-qwen38/models/karma-qwen38-nonrefusal-sft/v0-20260823-131002/runs', local_rank=0, sortish_sampler=False, predict_with_generate=False, generation_max_length=None, generation_num_beams=None, generation_config=None, tuner_backend='peft', vit_gradient_checkpointing=False, router_aux_loss_coef=0.0, enable_dft_loss=False, enable_channel_loss=False, safe_serialization=True, max_shard_size='5GB', check_model=False, acc_strategy='token', train_dataloader_shuffle=True, group_by_length=False, max_epochs=None, aligner_lr=None, vit_lr=None, use_logits_to_keep=None, ds3_gather_for_generation=True, resume_only_model=False, optimizer=None, loss_type=None, mrl_dims=None, eval_metric=None, callbacks=[], early_stop_interval=None, eval_use_evalscope=False, eval_dataset=[], eval_dataset_args=None, eval_limit=None, eval_generation_config=None, extra_eval_args=None, tuner_type='lora', use_galore=False, galore_target_modules=None, galore_rank=128, galore_update_proj_gap=50, galore_scale=1.0, galore_proj_type='std', galore_optim_per_parameter=False, galore_with_embedding=False, galore_quantization=False, galore_proj_quant=False, galore_proj_bits=4, galore_proj_group_size=256, galore_cos_threshold=0.4, galore_gamma_proj=2, galore_queue_size=5, lisa_activated_layers=0, lisa_step_interval=20, use_flash_ckpt=False)"
404
+ }
chat_template.jinja ADDED
@@ -0,0 +1,170 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {%- set image_count = namespace(value=0) %}
2
+ {%- set video_count = namespace(value=0) %}
3
+ {%- macro render_content(content, do_vision_count, is_system_content=false) %}
4
+ {%- if content is string %}
5
+ {{- content }}
6
+ {%- elif content is iterable and content is not mapping %}
7
+ {%- for item in content %}
8
+ {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
9
+ {%- if is_system_content %}
10
+ {{- raise_exception('System message cannot contain images.') }}
11
+ {%- endif %}
12
+ {%- if do_vision_count %}
13
+ {%- set image_count.value = image_count.value + 1 %}
14
+ {%- endif %}
15
+ {%- if add_vision_id %}
16
+ {{- 'Picture ' ~ image_count.value ~ ': ' }}
17
+ {%- endif %}
18
+ {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
19
+ {%- elif 'video' in item or item.type == 'video' %}
20
+ {%- if is_system_content %}
21
+ {{- raise_exception('System message cannot contain videos.') }}
22
+ {%- endif %}
23
+ {%- if do_vision_count %}
24
+ {%- set video_count.value = video_count.value + 1 %}
25
+ {%- endif %}
26
+ {%- if add_vision_id %}
27
+ {{- 'Video ' ~ video_count.value ~ ': ' }}
28
+ {%- endif %}
29
+ {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
30
+ {%- elif 'text' in item %}
31
+ {{- item.text }}
32
+ {%- else %}
33
+ {{- raise_exception('Unexpected item type in content.') }}
34
+ {%- endif %}
35
+ {%- endfor %}
36
+ {%- elif content is none or content is undefined %}
37
+ {{- '' }}
38
+ {%- else %}
39
+ {{- raise_exception('Unexpected content type.') }}
40
+ {%- endif %}
41
+ {%- endmacro %}
42
+ {%- if not messages %}
43
+ {{- raise_exception('No messages provided.') }}
44
+ {%- endif %}
45
+ {%- set reasoning_instructions = '' %}
46
+ {%- if enable_thinking is undefined or enable_thinking is true %}
47
+ {%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}
48
+ {%- if resolved_reasoning_effort not in ('xhigh', 'medium', 'low') %}
49
+ {{- raise_exception('Unexpected reasoning effort ' ~ reasoning_effort ~ '. Supported types are xhigh (default), medium, and low.') }}
50
+ {%- endif %}
51
+ {%- if resolved_reasoning_effort == 'xhigh' %}
52
+ {%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
53
+ {%- elif resolved_reasoning_effort == 'low' %}
54
+ {%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}
55
+ {%- endif %}
56
+ {%- endif %}
57
+ {%- if tools and tools is iterable and tools is not mapping %}
58
+ {{- '<|im_start|>system\n' }}
59
+ {%- if reasoning_instructions %}
60
+ {{- reasoning_instructions + '\n\n' }}
61
+ {%- endif %}
62
+ {{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
63
+ {%- for tool in tools %}
64
+ {{- "\n" }}
65
+ {{- tool | tojson }}
66
+ {%- endfor %}
67
+ {{- "\n</tools>" }}
68
+ {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
69
+ {%- if messages[0].role == 'system' %}
70
+ {%- set content = render_content(messages[0].content, false, true)|trim %}
71
+ {%- if content %}
72
+ {{- '\n\n' + content }}
73
+ {%- endif %}
74
+ {%- endif %}
75
+ {{- '<|im_end|>\n' }}
76
+ {%- else %}
77
+ {%- if messages[0].role == 'system' %}
78
+ {%- set content = render_content(messages[0].content, false, true)|trim %}
79
+ {%- if content %}
80
+ {{- '<|im_start|>system\n' + (reasoning_instructions + '\n\n' if reasoning_instructions else '') + content + '<|im_end|>\n' }}
81
+ {%- elif reasoning_instructions %}
82
+ {{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
83
+ {%- endif %}
84
+ {%- elif reasoning_instructions %}
85
+ {{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
86
+ {%- endif %}
87
+ {%- endif %}
88
+ {%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
89
+ {%- for message in messages[::-1] %}
90
+ {%- set index = (messages|length - 1) - loop.index0 %}
91
+ {%- if ns.multi_step_tool and message.role == "user" %}
92
+ {%- set content = render_content(message.content, false)|trim %}
93
+ {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
94
+ {%- set ns.multi_step_tool = false %}
95
+ {%- set ns.last_query_index = index %}
96
+ {%- endif %}
97
+ {%- endif %}
98
+ {%- endfor %}
99
+ {%- if ns.multi_step_tool %}
100
+ {{- raise_exception('No user query found in messages.') }}
101
+ {%- endif %}
102
+ {%- for message in messages %}
103
+ {%- set content = render_content(message.content, true)|trim %}
104
+ {%- if message.role == "system" %}
105
+ {%- if not loop.first %}
106
+ {{- raise_exception('System message must be at the beginning.') }}
107
+ {%- endif %}
108
+ {%- elif message.role == "user" %}
109
+ {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
110
+ {%- elif message.role == "assistant" %}
111
+ {%- set reasoning_content = '' %}
112
+ {%- if message.reasoning_content is string %}
113
+ {%- set reasoning_content = message.reasoning_content %}
114
+ {%- endif %}
115
+ {%- set reasoning_content = reasoning_content|trim %}
116
+ {%- if preserve_thinking is undefined or preserve_thinking is true or loop.index0 > ns.last_query_index %}
117
+ {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
118
+ {%- else %}
119
+ {{- '<|im_start|>' + message.role + '\n' + content }}
120
+ {%- endif %}
121
+ {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
122
+ {%- for tool_call in message.tool_calls %}
123
+ {%- if tool_call.function is defined %}
124
+ {%- set tool_call = tool_call.function %}
125
+ {%- endif %}
126
+ {%- if loop.first %}
127
+ {%- if content|trim %}
128
+ {{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
129
+ {%- else %}
130
+ {{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
131
+ {%- endif %}
132
+ {%- else %}
133
+ {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
134
+ {%- endif %}
135
+ {%- if tool_call.arguments is defined and tool_call.arguments != '' %}
136
+ {%- for args_name, args_value in tool_call.arguments|items %}
137
+ {{- '<parameter=' + args_name + '>\n' }}
138
+ {%- set args_value = args_value | string if args_value is string else args_value | tojson | safe %}
139
+ {{- args_value }}
140
+ {{- '\n</parameter>\n' }}
141
+ {%- endfor %}
142
+ {%- endif %}
143
+ {{- '</function>\n</tool_call>' }}
144
+ {%- endfor %}
145
+ {%- endif %}
146
+ {{- '<|im_end|>\n' }}
147
+ {%- elif message.role == "tool" %}
148
+ {%- if loop.previtem and loop.previtem.role != "tool" %}
149
+ {{- '<|im_start|>user' }}
150
+ {%- endif %}
151
+ {{- '\n<tool_response>\n' }}
152
+ {{- content }}
153
+ {{- '\n</tool_response>' }}
154
+ {%- if not loop.last and loop.nextitem.role != "tool" %}
155
+ {{- '<|im_end|>\n' }}
156
+ {%- elif loop.last %}
157
+ {{- '<|im_end|>\n' }}
158
+ {%- endif %}
159
+ {%- else %}
160
+ {{- raise_exception('Unexpected message role.') }}
161
+ {%- endif %}
162
+ {%- endfor %}
163
+ {%- if add_generation_prompt %}
164
+ {{- '<|im_start|>assistant\n' }}
165
+ {%- if enable_thinking is defined and enable_thinking is false %}
166
+ {{- '<think>\n\n</think>\n\n' }}
167
+ {%- else %}
168
+ {{- '<think>\n' }}
169
+ {%- endif %}
170
+ {%- endif %}
config.json ADDED
@@ -0,0 +1,153 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "Qwen3_5ForConditionalGeneration"
4
+ ],
5
+ "dtype": "bfloat16",
6
+ "hidden_size": 5120,
7
+ "image_token_id": 248056,
8
+ "language_model_only": false,
9
+ "model_type": "qwen3_5",
10
+ "text_config": {
11
+ "attention_bias": false,
12
+ "attention_dropout": 0.0,
13
+ "attn_output_gate": true,
14
+ "bos_token_id": 248044,
15
+ "dtype": "bfloat16",
16
+ "eos_token_id": 248044,
17
+ "full_attention_interval": 4,
18
+ "head_dim": 256,
19
+ "hidden_act": "silu",
20
+ "hidden_size": 5120,
21
+ "initializer_range": 0.02,
22
+ "intermediate_size": 17408,
23
+ "layer_types": [
24
+ "linear_attention",
25
+ "linear_attention",
26
+ "linear_attention",
27
+ "full_attention",
28
+ "linear_attention",
29
+ "linear_attention",
30
+ "linear_attention",
31
+ "full_attention",
32
+ "linear_attention",
33
+ "linear_attention",
34
+ "linear_attention",
35
+ "full_attention",
36
+ "linear_attention",
37
+ "linear_attention",
38
+ "linear_attention",
39
+ "full_attention",
40
+ "linear_attention",
41
+ "linear_attention",
42
+ "linear_attention",
43
+ "full_attention",
44
+ "linear_attention",
45
+ "linear_attention",
46
+ "linear_attention",
47
+ "full_attention",
48
+ "linear_attention",
49
+ "linear_attention",
50
+ "linear_attention",
51
+ "full_attention",
52
+ "linear_attention",
53
+ "linear_attention",
54
+ "linear_attention",
55
+ "full_attention",
56
+ "linear_attention",
57
+ "linear_attention",
58
+ "linear_attention",
59
+ "full_attention",
60
+ "linear_attention",
61
+ "linear_attention",
62
+ "linear_attention",
63
+ "full_attention",
64
+ "linear_attention",
65
+ "linear_attention",
66
+ "linear_attention",
67
+ "full_attention",
68
+ "linear_attention",
69
+ "linear_attention",
70
+ "linear_attention",
71
+ "full_attention",
72
+ "linear_attention",
73
+ "linear_attention",
74
+ "linear_attention",
75
+ "full_attention",
76
+ "linear_attention",
77
+ "linear_attention",
78
+ "linear_attention",
79
+ "full_attention",
80
+ "linear_attention",
81
+ "linear_attention",
82
+ "linear_attention",
83
+ "full_attention",
84
+ "linear_attention",
85
+ "linear_attention",
86
+ "linear_attention",
87
+ "full_attention"
88
+ ],
89
+ "linear_conv_kernel_dim": 4,
90
+ "linear_key_head_dim": 128,
91
+ "linear_num_key_heads": 16,
92
+ "linear_num_value_heads": 48,
93
+ "linear_value_head_dim": 128,
94
+ "mamba_ssm_dtype": "float32",
95
+ "max_position_embeddings": 1048576,
96
+ "model_type": "qwen3_5_text",
97
+ "mtp_num_hidden_layers": 1,
98
+ "mtp_use_dedicated_embeddings": false,
99
+ "num_attention_heads": 24,
100
+ "num_hidden_layers": 64,
101
+ "num_key_value_heads": 4,
102
+ "output_gate_type": "swish",
103
+ "pad_token_id": 248044,
104
+ "partial_rotary_factor": 0.25,
105
+ "rms_norm_eps": 1e-06,
106
+ "rope_parameters": {
107
+ "mrope_interleaved": true,
108
+ "mrope_section": [
109
+ 11,
110
+ 11,
111
+ 10
112
+ ],
113
+ "partial_rotary_factor": 0.25,
114
+ "rope_theta": 10000000,
115
+ "rope_type": "yarn",
116
+ "factor": 4.0,
117
+ "original_max_position_embeddings": 262144
118
+ },
119
+ "tie_word_embeddings": false,
120
+ "use_cache": true,
121
+ "vocab_size": 248320
122
+ },
123
+ "tie_word_embeddings": false,
124
+ "transformers_version": "5.12.1",
125
+ "video_token_id": 248057,
126
+ "vision_config": {
127
+ "deepstack_visual_indexes": [],
128
+ "depth": 27,
129
+ "dtype": "bfloat16",
130
+ "hidden_act": "gelu_pytorch_tanh",
131
+ "hidden_size": 1152,
132
+ "in_channels": 3,
133
+ "initializer_range": 0.02,
134
+ "intermediate_size": 4304,
135
+ "model_type": "qwen3_5_vision",
136
+ "num_heads": 16,
137
+ "num_position_embeddings": 2304,
138
+ "out_hidden_size": 5120,
139
+ "patch_size": 16,
140
+ "spatial_merge_size": 2,
141
+ "temporal_patch_size": 2
142
+ },
143
+ "vision_end_token_id": 248054,
144
+ "vision_start_token_id": 248053,
145
+ "model_name": "SwissNeuron",
146
+ "swissneuron_context": {
147
+ "configured_max_tokens": 1048576,
148
+ "original_max_tokens": 262144,
149
+ "method": "YaRN",
150
+ "factor": 4.0,
151
+ "dedicated_1m_adaptation": false
152
+ }
153
+ }
config.native-262k.json ADDED
@@ -0,0 +1,143 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "Qwen3_5ForConditionalGeneration"
4
+ ],
5
+ "dtype": "bfloat16",
6
+ "hidden_size": 5120,
7
+ "image_token_id": 248056,
8
+ "language_model_only": false,
9
+ "model_type": "qwen3_5",
10
+ "text_config": {
11
+ "attention_bias": false,
12
+ "attention_dropout": 0.0,
13
+ "attn_output_gate": true,
14
+ "bos_token_id": 248044,
15
+ "dtype": "bfloat16",
16
+ "eos_token_id": 248044,
17
+ "full_attention_interval": 4,
18
+ "head_dim": 256,
19
+ "hidden_act": "silu",
20
+ "hidden_size": 5120,
21
+ "initializer_range": 0.02,
22
+ "intermediate_size": 17408,
23
+ "layer_types": [
24
+ "linear_attention",
25
+ "linear_attention",
26
+ "linear_attention",
27
+ "full_attention",
28
+ "linear_attention",
29
+ "linear_attention",
30
+ "linear_attention",
31
+ "full_attention",
32
+ "linear_attention",
33
+ "linear_attention",
34
+ "linear_attention",
35
+ "full_attention",
36
+ "linear_attention",
37
+ "linear_attention",
38
+ "linear_attention",
39
+ "full_attention",
40
+ "linear_attention",
41
+ "linear_attention",
42
+ "linear_attention",
43
+ "full_attention",
44
+ "linear_attention",
45
+ "linear_attention",
46
+ "linear_attention",
47
+ "full_attention",
48
+ "linear_attention",
49
+ "linear_attention",
50
+ "linear_attention",
51
+ "full_attention",
52
+ "linear_attention",
53
+ "linear_attention",
54
+ "linear_attention",
55
+ "full_attention",
56
+ "linear_attention",
57
+ "linear_attention",
58
+ "linear_attention",
59
+ "full_attention",
60
+ "linear_attention",
61
+ "linear_attention",
62
+ "linear_attention",
63
+ "full_attention",
64
+ "linear_attention",
65
+ "linear_attention",
66
+ "linear_attention",
67
+ "full_attention",
68
+ "linear_attention",
69
+ "linear_attention",
70
+ "linear_attention",
71
+ "full_attention",
72
+ "linear_attention",
73
+ "linear_attention",
74
+ "linear_attention",
75
+ "full_attention",
76
+ "linear_attention",
77
+ "linear_attention",
78
+ "linear_attention",
79
+ "full_attention",
80
+ "linear_attention",
81
+ "linear_attention",
82
+ "linear_attention",
83
+ "full_attention",
84
+ "linear_attention",
85
+ "linear_attention",
86
+ "linear_attention",
87
+ "full_attention"
88
+ ],
89
+ "linear_conv_kernel_dim": 4,
90
+ "linear_key_head_dim": 128,
91
+ "linear_num_key_heads": 16,
92
+ "linear_num_value_heads": 48,
93
+ "linear_value_head_dim": 128,
94
+ "mamba_ssm_dtype": "float32",
95
+ "max_position_embeddings": 262144,
96
+ "model_type": "qwen3_5_text",
97
+ "mtp_num_hidden_layers": 1,
98
+ "mtp_use_dedicated_embeddings": false,
99
+ "num_attention_heads": 24,
100
+ "num_hidden_layers": 64,
101
+ "num_key_value_heads": 4,
102
+ "output_gate_type": "swish",
103
+ "pad_token_id": 248044,
104
+ "partial_rotary_factor": 0.25,
105
+ "rms_norm_eps": 1e-06,
106
+ "rope_parameters": {
107
+ "mrope_interleaved": true,
108
+ "mrope_section": [
109
+ 11,
110
+ 11,
111
+ 10
112
+ ],
113
+ "partial_rotary_factor": 0.25,
114
+ "rope_theta": 10000000,
115
+ "rope_type": "default"
116
+ },
117
+ "tie_word_embeddings": false,
118
+ "use_cache": true,
119
+ "vocab_size": 248320
120
+ },
121
+ "tie_word_embeddings": false,
122
+ "transformers_version": "5.12.1",
123
+ "video_token_id": 248057,
124
+ "vision_config": {
125
+ "deepstack_visual_indexes": [],
126
+ "depth": 27,
127
+ "dtype": "bfloat16",
128
+ "hidden_act": "gelu_pytorch_tanh",
129
+ "hidden_size": 1152,
130
+ "in_channels": 3,
131
+ "initializer_range": 0.02,
132
+ "intermediate_size": 4304,
133
+ "model_type": "qwen3_5_vision",
134
+ "num_heads": 16,
135
+ "num_position_embeddings": 2304,
136
+ "out_hidden_size": 5120,
137
+ "patch_size": 16,
138
+ "spatial_merge_size": 2,
139
+ "temporal_patch_size": 2
140
+ },
141
+ "vision_end_token_id": 248054,
142
+ "vision_start_token_id": 248053
143
+ }
dwm_provenance.json ADDED
@@ -0,0 +1,38 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alpha": 0.1,
3
+ "audit": {
4
+ "component_ratio_max": 0.9202628135681152,
5
+ "component_ratio_mean": 0.9149049116719153,
6
+ "component_ratio_min": 0.9011346697807312,
7
+ "norm_ratio_max": 1.0000070333480835,
8
+ "norm_ratio_mean": 0.9999727582739245,
9
+ "norm_ratio_min": 0.9998393058776855,
10
+ "relative_delta_max": 0.006776769645512104,
11
+ "relative_delta_mean": 0.0019126400884781633,
12
+ "relative_delta_min": 0.0013704006560146809,
13
+ "signed_retention_max": 0.9201236963272095,
14
+ "signed_retention_mean": 0.9147886436793112,
15
+ "signed_retention_min": 0.9011276960372925
16
+ },
17
+ "checkpoint_contains_recipe_metadata": false,
18
+ "direction_run_id": null,
19
+ "direction_sha256": "c754628ca259d9cfe55af81f93788f403424782f05994c5940ebab7613d81724",
20
+ "directions": "/workspace/train-qwen38/derisk/bank_final_nonrefusal.pt",
21
+ "edited_matrices": 124,
22
+ "hidden_size": 5120,
23
+ "layers": 64,
24
+ "method": "soft directional weight modification",
25
+ "nominal_directional_retention": 0.9,
26
+ "output": "/workspace/train-qwen38/models/karma-qwen38-nonrefusal-derisk-a01",
27
+ "passes": 1,
28
+ "secondary": null,
29
+ "skip_early": 2,
30
+ "source": "/workspace/train-qwen38/models/karma-qwen38-nonrefusal-sft-SNAPSHOT",
31
+ "source_index_sha256": "a600c6626eb1eb653a3f078ab3dd1c7a8fab2e71185c7ef3aa923a39aaa637be",
32
+ "source_modified": false,
33
+ "surface_counts": {
34
+ "dense_mlp_down": 62,
35
+ "full_attention_output": 16,
36
+ "linear_attention_output": 46
37
+ }
38
+ }
generation_config.json ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token_id": 248044,
3
+ "do_sample": true,
4
+ "eos_token_id": [
5
+ 248046,
6
+ 248044
7
+ ],
8
+ "pad_token_id": 248044,
9
+ "temperature": 1.0,
10
+ "top_k": 20,
11
+ "top_p": 0.95,
12
+ "transformers_version": "5.12.1"
13
+ }
model-00001-of-00012.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:54d83c1d36631de231876217a8e0c2483eccee8746369a482b79442bdfc5d958
3
+ size 2542796928
model-00002-of-00012.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f9d51464f2006c4a12240f251e22a55e894cf0db35be8f5fbd5295340ea8dee2
3
+ size 4842451920
model-00003-of-00012.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f84e5d7a394117f4b98354754b2edaa3a88fc0f495db7832a3f03d3014c2c7ab
3
+ size 4965227944
model-00004-of-00012.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:69330da8367eedd733d414aac86af1ab0c540f7f0ba1a9cf8587e4ad93148056
3
+ size 4912819264
model-00005-of-00012.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8d5861388eb9f3507e6abdb83d2b72648a27642978e4620038f8724c05cdfdc6
3
+ size 4986198544
model-00006-of-00012.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:777c23c5a7b091eb3bdfaa2cd91d582b998e33ef4f7301dee36ea1da301d0e03
3
+ size 4912819320
model-00007-of-00012.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bff0dd48aeaf81f13c872a799ce8443f2d47cb16c8948730ff8b466b8b2888be
3
+ size 4932703272
model-00008-of-00012.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9e2ceae6c13145833cb525c7803c69ccff574abe083378dce0d33849a961e1f3
3
+ size 4966314576
model-00009-of-00012.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c3f165523deff743af0ca0a57d4d45cdc2cb9744ff509d1616c62d76d3ad054f
3
+ size 4964162248
model-00010-of-00012.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f839b2e0b92cc570ecbab95083398de5b8342c0c9e7756070d91cf5664d6b591
3
+ size 4933789824
model-00011-of-00012.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5ec44a2b3edf78998535dcbe205029ca24ecd7ee3f018e23b369f6c9de75dc8b
3
+ size 4965228032
model-00012-of-00012.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c1d27e7a52aabc93d7d09d24bfb08e3899e1bdbea53282899f461e7293878a49
3
+ size 2789094896
model.safetensors.index.json ADDED
The diff for this file is too large to render. See raw diff
 
preprocessor_config.json ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "size": {
3
+ "longest_edge": 16777216,
4
+ "shortest_edge": 65536
5
+ },
6
+ "patch_size": 16,
7
+ "temporal_patch_size": 2,
8
+ "merge_size": 2,
9
+ "image_mean": [
10
+ 0.5,
11
+ 0.5,
12
+ 0.5
13
+ ],
14
+ "image_std": [
15
+ 0.5,
16
+ 0.5,
17
+ 0.5
18
+ ],
19
+ "processor_class": "Qwen3VLProcessor",
20
+ "image_processor_type": "Qwen2VLImageProcessorFast"
21
+ }
processor_config.json ADDED
@@ -0,0 +1,60 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "image_processor": {
3
+ "do_convert_rgb": true,
4
+ "do_normalize": true,
5
+ "do_rescale": true,
6
+ "do_resize": true,
7
+ "image_mean": [
8
+ 0.5,
9
+ 0.5,
10
+ 0.5
11
+ ],
12
+ "image_processor_type": "Qwen2VLImageProcessor",
13
+ "image_std": [
14
+ 0.5,
15
+ 0.5,
16
+ 0.5
17
+ ],
18
+ "merge_size": 2,
19
+ "patch_size": 16,
20
+ "resample": 3,
21
+ "rescale_factor": 0.00392156862745098,
22
+ "size": {
23
+ "longest_edge": 16777216,
24
+ "shortest_edge": 65536
25
+ },
26
+ "temporal_patch_size": 2
27
+ },
28
+ "processor_class": "Qwen3VLProcessor",
29
+ "video_processor": {
30
+ "do_convert_rgb": true,
31
+ "do_normalize": true,
32
+ "do_rescale": true,
33
+ "do_resize": true,
34
+ "do_sample_frames": true,
35
+ "fps": 2,
36
+ "image_mean": [
37
+ 0.5,
38
+ 0.5,
39
+ 0.5
40
+ ],
41
+ "image_std": [
42
+ 0.5,
43
+ 0.5,
44
+ 0.5
45
+ ],
46
+ "max_frames": 768,
47
+ "merge_size": 2,
48
+ "min_frames": 4,
49
+ "patch_size": 16,
50
+ "resample": 3,
51
+ "rescale_factor": 0.00392156862745098,
52
+ "return_metadata": false,
53
+ "size": {
54
+ "longest_edge": 25165824,
55
+ "shortest_edge": 4096
56
+ },
57
+ "temporal_patch_size": 2,
58
+ "video_processor_type": "Qwen3VLVideoProcessor"
59
+ }
60
+ }
refusal_direction_bank.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c754628ca259d9cfe55af81f93788f403424782f05994c5940ebab7613d81724
3
+ size 1313045
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523
3
+ size 19989325
tokenizer_config.json ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "audio_bos_token": "<|audio_start|>",
4
+ "audio_eos_token": "<|audio_end|>",
5
+ "audio_token": "<|audio_pad|>",
6
+ "backend": "tokenizers",
7
+ "bos_token": null,
8
+ "clean_up_tokenization_spaces": false,
9
+ "eos_token": "<|im_end|>",
10
+ "errors": "replace",
11
+ "image_token": "<|image_pad|>",
12
+ "is_local": true,
13
+ "local_files_only": false,
14
+ "model_max_length": 262144,
15
+ "model_specific_special_tokens": {
16
+ "audio_bos_token": "<|audio_start|>",
17
+ "audio_eos_token": "<|audio_end|>",
18
+ "audio_token": "<|audio_pad|>",
19
+ "image_token": "<|image_pad|>",
20
+ "video_token": "<|video_pad|>",
21
+ "vision_bos_token": "<|vision_start|>",
22
+ "vision_eos_token": "<|vision_end|>"
23
+ },
24
+ "pad_token": "<|endoftext|>",
25
+ "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
26
+ "processor_class": "Qwen3VLProcessor",
27
+ "split_special_tokens": false,
28
+ "tokenizer_class": "Qwen2Tokenizer",
29
+ "unk_token": null,
30
+ "video_token": "<|video_pad|>",
31
+ "vision_bos_token": "<|vision_start|>",
32
+ "vision_eos_token": "<|vision_end|>"
33
+ }