Add experimental Modujo 4B A0.66B SFT weights and evaluation
Browse files- sft/Modujo-4B-A0.66B/README.md +52 -0
- sft/Modujo-4B-A0.66B/__pycache__/qwen4_exp_dense_compat.cpython-312.pyc +0 -0
- sft/Modujo-4B-A0.66B/chat_template.jinja +170 -0
- sft/Modujo-4B-A0.66B/config.json +93 -0
- sft/Modujo-4B-A0.66B/evaluation/cases.json +1 -0
- sft/Modujo-4B-A0.66B/evaluation/code-tests.json +32 -0
- sft/Modujo-4B-A0.66B/evaluation/extended-cases.json +28 -0
- sft/Modujo-4B-A0.66B/evaluation/extended.jsonl +5 -0
- sft/Modujo-4B-A0.66B/evaluation/greedy.jsonl +8 -0
- sft/Modujo-4B-A0.66B/evaluation/report.md +327 -0
- sft/Modujo-4B-A0.66B/evaluation/summary.json +19 -0
- sft/Modujo-4B-A0.66B/generation_config.json +10 -0
- sft/Modujo-4B-A0.66B/inference.py +20 -0
- sft/Modujo-4B-A0.66B/model-00001-of-00004.safetensors +3 -0
- sft/Modujo-4B-A0.66B/model-00002-of-00004.safetensors +3 -0
- sft/Modujo-4B-A0.66B/model-00003-of-00004.safetensors +3 -0
- sft/Modujo-4B-A0.66B/model-00004-of-00004.safetensors +3 -0
- sft/Modujo-4B-A0.66B/model.safetensors.index.json +663 -0
- sft/Modujo-4B-A0.66B/qwen4_exp_dense_compat.py +62 -0
- sft/Modujo-4B-A0.66B/release.json +19 -0
- sft/Modujo-4B-A0.66B/tested-environment.json +7 -0
- sft/Modujo-4B-A0.66B/tokenizer.json +3 -0
- sft/Modujo-4B-A0.66B/tokenizer_config.json +32 -0
sft/Modujo-4B-A0.66B/README.md
ADDED
|
@@ -0,0 +1,52 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
library_name: transformers
|
| 3 |
+
pipeline_tag: text-generation
|
| 4 |
+
tags:
|
| 5 |
+
- mixture-of-experts
|
| 6 |
+
- qwen4-exp
|
| 7 |
+
- sft
|
| 8 |
+
- experimental
|
| 9 |
+
language:
|
| 10 |
+
- zh
|
| 11 |
+
- en
|
| 12 |
+
---
|
| 13 |
+
|
| 14 |
+
# Modujo 4B A0.66B — SFT
|
| 15 |
+
|
| 16 |
+
Experimental intermediate SFT checkpoint with approximately 3.96B total and 0.66B active parameters. This is a short-data-stage snapshot, not a completed teacher-distillation or production assistant release.
|
| 17 |
+
|
| 18 |
+
## Architecture and training
|
| 19 |
+
|
| 20 |
+
- Qwen4ExpForCausalLM, 28 layers, hidden size1024, 32 routed experts, top2, tied embeddings; one pass, no trained QSA indexer.
|
| 21 |
+
- Source PT: checkpoint17500 of the 4B pretraining run.
|
| 22 |
+
- Swift full SFT, FP32 master weights/AdamW states and BF16 compute; 8 B200 GPUs, global example batch128.
|
| 23 |
+
- Snapshot: short stage step36000/40017, about89.96% of its single epoch.
|
| 24 |
+
- Short stage: 5,122,171 cleaned rows, mostly MiniMind (5,040,686); e-commerce39,975, OpenGPT31,511, Claude9,999. Native training context<=2048, retained low reasoning<=512, predominantly no-think.
|
| 25 |
+
- Qwen/GLM/DeepSeek community teacher data and original long rows are scheduled in a later stage and have not been trained into this snapshot.
|
| 26 |
+
- Saved tensors remain FP32; optimizer and RNG states are not included in this inference release.
|
| 27 |
+
|
| 28 |
+
## Loading
|
| 29 |
+
|
| 30 |
+
The repository root is not a loadable model. Use this subdirectory. The installed Transformers environment used for validation is documented in tested-environment.json; it must provide the qwen4_exp architecture. Null-indexer checkpoints require the included dense-attention compatibility helper.
|
| 31 |
+
|
| 32 |
+
Download this directory:
|
| 33 |
+
|
| 34 |
+
```python
|
| 35 |
+
from huggingface_hub import snapshot_download
|
| 36 |
+
snapshot_download('Modujo-AI/ModujoMoE',
|
| 37 |
+
allow_patterns=['sft/Modujo-4B-A0.66B/*'],
|
| 38 |
+
local_dir='./modujo-download')
|
| 39 |
+
```
|
| 40 |
+
|
| 41 |
+
Run the included inference script in the recorded compatible environment:
|
| 42 |
+
|
| 43 |
+
```bash
|
| 44 |
+
python modujo-download/sft/Modujo-4B-A0.66B/inference.py --device cuda:0
|
| 45 |
+
# CPU fallback is also supported: --device cpu
|
| 46 |
+
```
|
| 47 |
+
|
| 48 |
+
## Observed quality and limitations
|
| 49 |
+
|
| 50 |
+
These are diagnostic cases, not benchmark accuracy estimates. Elementary arithmetic, event relations, strict JSON and low-think correctness remain unreliable. The palindrome function passes four basic tests but fails a punctuation boundary test by ignoring more characters than requested. Low-think multiplication repeats to512 tokens; apple reasoning closes but reaches an incorrect answer. English capital question returns Paris. Full inputs and generations are in evaluation/.
|
| 51 |
+
|
| 52 |
+
This snapshot is suitable for research and inspection; it should not be represented as a reliable assistant. Source datasets' applicable terms remain relevant; no new blanket license is asserted by this card.
|
sft/Modujo-4B-A0.66B/__pycache__/qwen4_exp_dense_compat.cpython-312.pyc
ADDED
|
Binary file (3.55 kB). View file
|
|
|
sft/Modujo-4B-A0.66B/chat_template.jinja
ADDED
|
@@ -0,0 +1,170 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{%- set image_count = namespace(value=0) %}
|
| 2 |
+
{%- set video_count = namespace(value=0) %}
|
| 3 |
+
{%- macro render_content(content, do_vision_count, is_system_content=false) %}
|
| 4 |
+
{%- if content is string %}
|
| 5 |
+
{{- content }}
|
| 6 |
+
{%- elif content is iterable and content is not mapping %}
|
| 7 |
+
{%- for item in content %}
|
| 8 |
+
{%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
|
| 9 |
+
{%- if is_system_content %}
|
| 10 |
+
{{- raise_exception('System message cannot contain images.') }}
|
| 11 |
+
{%- endif %}
|
| 12 |
+
{%- if do_vision_count %}
|
| 13 |
+
{%- set image_count.value = image_count.value + 1 %}
|
| 14 |
+
{%- endif %}
|
| 15 |
+
{%- if add_vision_id %}
|
| 16 |
+
{{- 'Picture ' ~ image_count.value ~ ': ' }}
|
| 17 |
+
{%- endif %}
|
| 18 |
+
{{- '<|vision_start|><|image_pad|><|vision_end|>' }}
|
| 19 |
+
{%- elif 'video' in item or item.type == 'video' %}
|
| 20 |
+
{%- if is_system_content %}
|
| 21 |
+
{{- raise_exception('System message cannot contain videos.') }}
|
| 22 |
+
{%- endif %}
|
| 23 |
+
{%- if do_vision_count %}
|
| 24 |
+
{%- set video_count.value = video_count.value + 1 %}
|
| 25 |
+
{%- endif %}
|
| 26 |
+
{%- if add_vision_id %}
|
| 27 |
+
{{- 'Video ' ~ video_count.value ~ ': ' }}
|
| 28 |
+
{%- endif %}
|
| 29 |
+
{{- '<|vision_start|><|video_pad|><|vision_end|>' }}
|
| 30 |
+
{%- elif 'text' in item %}
|
| 31 |
+
{{- item.text }}
|
| 32 |
+
{%- else %}
|
| 33 |
+
{{- raise_exception('Unexpected item type in content.') }}
|
| 34 |
+
{%- endif %}
|
| 35 |
+
{%- endfor %}
|
| 36 |
+
{%- elif content is none or content is undefined %}
|
| 37 |
+
{{- '' }}
|
| 38 |
+
{%- else %}
|
| 39 |
+
{{- raise_exception('Unexpected content type.') }}
|
| 40 |
+
{%- endif %}
|
| 41 |
+
{%- endmacro %}
|
| 42 |
+
{%- if not messages %}
|
| 43 |
+
{{- raise_exception('No messages provided.') }}
|
| 44 |
+
{%- endif %}
|
| 45 |
+
{%- set reasoning_instructions = '' %}
|
| 46 |
+
{%- if enable_thinking is undefined or enable_thinking is true %}
|
| 47 |
+
{%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}
|
| 48 |
+
{%- if resolved_reasoning_effort not in ('xhigh', 'medium', 'low') %}
|
| 49 |
+
{{- raise_exception('Unexpected reasoning effort ' ~ reasoning_effort ~ '. Supported types are xhigh (default), medium, and low.') }}
|
| 50 |
+
{%- endif %}
|
| 51 |
+
{%- if resolved_reasoning_effort == 'xhigh' %}
|
| 52 |
+
{%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
|
| 53 |
+
{%- elif resolved_reasoning_effort == 'low' %}
|
| 54 |
+
{%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}
|
| 55 |
+
{%- endif %}
|
| 56 |
+
{%- endif %}
|
| 57 |
+
{%- if tools and tools is iterable and tools is not mapping %}
|
| 58 |
+
{{- '<|im_start|>system\n' }}
|
| 59 |
+
{%- if reasoning_instructions %}
|
| 60 |
+
{{- reasoning_instructions + '\n\n' }}
|
| 61 |
+
{%- endif %}
|
| 62 |
+
{{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
|
| 63 |
+
{%- for tool in tools %}
|
| 64 |
+
{{- "\n" }}
|
| 65 |
+
{{- tool | tojson }}
|
| 66 |
+
{%- endfor %}
|
| 67 |
+
{{- "\n</tools>" }}
|
| 68 |
+
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
|
| 69 |
+
{%- if messages[0].role == 'system' %}
|
| 70 |
+
{%- set content = render_content(messages[0].content, false, true)|trim %}
|
| 71 |
+
{%- if content %}
|
| 72 |
+
{{- '\n\n' + content }}
|
| 73 |
+
{%- endif %}
|
| 74 |
+
{%- endif %}
|
| 75 |
+
{{- '<|im_end|>\n' }}
|
| 76 |
+
{%- else %}
|
| 77 |
+
{%- if messages[0].role == 'system' %}
|
| 78 |
+
{%- set content = render_content(messages[0].content, false, true)|trim %}
|
| 79 |
+
{%- if content %}
|
| 80 |
+
{{- '<|im_start|>system\n' + (reasoning_instructions + '\n\n' if reasoning_instructions else '') + content + '<|im_end|>\n' }}
|
| 81 |
+
{%- elif reasoning_instructions %}
|
| 82 |
+
{{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
|
| 83 |
+
{%- endif %}
|
| 84 |
+
{%- elif reasoning_instructions %}
|
| 85 |
+
{{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
|
| 86 |
+
{%- endif %}
|
| 87 |
+
{%- endif %}
|
| 88 |
+
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
|
| 89 |
+
{%- for message in messages[::-1] %}
|
| 90 |
+
{%- set index = (messages|length - 1) - loop.index0 %}
|
| 91 |
+
{%- if ns.multi_step_tool and message.role == "user" %}
|
| 92 |
+
{%- set content = render_content(message.content, false)|trim %}
|
| 93 |
+
{%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
|
| 94 |
+
{%- set ns.multi_step_tool = false %}
|
| 95 |
+
{%- set ns.last_query_index = index %}
|
| 96 |
+
{%- endif %}
|
| 97 |
+
{%- endif %}
|
| 98 |
+
{%- endfor %}
|
| 99 |
+
{%- if ns.multi_step_tool %}
|
| 100 |
+
{{- raise_exception('No user query found in messages.') }}
|
| 101 |
+
{%- endif %}
|
| 102 |
+
{%- for message in messages %}
|
| 103 |
+
{%- set content = render_content(message.content, true)|trim %}
|
| 104 |
+
{%- if message.role == "system" %}
|
| 105 |
+
{%- if not loop.first %}
|
| 106 |
+
{{- raise_exception('System message must be at the beginning.') }}
|
| 107 |
+
{%- endif %}
|
| 108 |
+
{%- elif message.role == "user" %}
|
| 109 |
+
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
|
| 110 |
+
{%- elif message.role == "assistant" %}
|
| 111 |
+
{%- set reasoning_content = '' %}
|
| 112 |
+
{%- if message.reasoning_content is string %}
|
| 113 |
+
{%- set reasoning_content = message.reasoning_content %}
|
| 114 |
+
{%- endif %}
|
| 115 |
+
{%- set reasoning_content = reasoning_content|trim %}
|
| 116 |
+
{%- if preserve_thinking is undefined or preserve_thinking is true or loop.index0 > ns.last_query_index %}
|
| 117 |
+
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
|
| 118 |
+
{%- else %}
|
| 119 |
+
{{- '<|im_start|>' + message.role + '\n' + content }}
|
| 120 |
+
{%- endif %}
|
| 121 |
+
{%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
|
| 122 |
+
{%- for tool_call in message.tool_calls %}
|
| 123 |
+
{%- if tool_call.function is defined %}
|
| 124 |
+
{%- set tool_call = tool_call.function %}
|
| 125 |
+
{%- endif %}
|
| 126 |
+
{%- if loop.first %}
|
| 127 |
+
{%- if content|trim %}
|
| 128 |
+
{{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
| 129 |
+
{%- else %}
|
| 130 |
+
{{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
| 131 |
+
{%- endif %}
|
| 132 |
+
{%- else %}
|
| 133 |
+
{{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
| 134 |
+
{%- endif %}
|
| 135 |
+
{%- if tool_call.arguments is defined and tool_call.arguments != '' %}
|
| 136 |
+
{%- for args_name, args_value in tool_call.arguments|items %}
|
| 137 |
+
{{- '<parameter=' + args_name + '>\n' }}
|
| 138 |
+
{%- set args_value = args_value | string if args_value is string else args_value | tojson | safe %}
|
| 139 |
+
{{- args_value }}
|
| 140 |
+
{{- '\n</parameter>\n' }}
|
| 141 |
+
{%- endfor %}
|
| 142 |
+
{%- endif %}
|
| 143 |
+
{{- '</function>\n</tool_call>' }}
|
| 144 |
+
{%- endfor %}
|
| 145 |
+
{%- endif %}
|
| 146 |
+
{{- '<|im_end|>\n' }}
|
| 147 |
+
{%- elif message.role == "tool" %}
|
| 148 |
+
{%- if loop.previtem and loop.previtem.role != "tool" %}
|
| 149 |
+
{{- '<|im_start|>user' }}
|
| 150 |
+
{%- endif %}
|
| 151 |
+
{{- '\n<tool_response>\n' }}
|
| 152 |
+
{{- content }}
|
| 153 |
+
{{- '\n</tool_response>' }}
|
| 154 |
+
{%- if not loop.last and loop.nextitem.role != "tool" %}
|
| 155 |
+
{{- '<|im_end|>\n' }}
|
| 156 |
+
{%- elif loop.last %}
|
| 157 |
+
{{- '<|im_end|>\n' }}
|
| 158 |
+
{%- endif %}
|
| 159 |
+
{%- else %}
|
| 160 |
+
{{- raise_exception('Unexpected message role.') }}
|
| 161 |
+
{%- endif %}
|
| 162 |
+
{%- endfor %}
|
| 163 |
+
{%- if add_generation_prompt %}
|
| 164 |
+
{{- '<|im_start|>assistant\n' }}
|
| 165 |
+
{%- if enable_thinking is defined and enable_thinking is false %}
|
| 166 |
+
{{- '<think>\n\n</think>\n\n' }}
|
| 167 |
+
{%- else %}
|
| 168 |
+
{{- '<think>\n' }}
|
| 169 |
+
{%- endif %}
|
| 170 |
+
{%- endif %}
|
sft/Modujo-4B-A0.66B/config.json
ADDED
|
@@ -0,0 +1,93 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architectures": [
|
| 3 |
+
"Qwen4ExpForCausalLM"
|
| 4 |
+
],
|
| 5 |
+
"attention_bias": false,
|
| 6 |
+
"attention_dropout": 0.0,
|
| 7 |
+
"bos_token_id": null,
|
| 8 |
+
"dtype": "float32",
|
| 9 |
+
"eos_token_id": 248046,
|
| 10 |
+
"experts_impl": "grouped_mm",
|
| 11 |
+
"hc_count": 4,
|
| 12 |
+
"hc_lowrank": 128,
|
| 13 |
+
"head_dim": 64,
|
| 14 |
+
"heads_per_ngram": 6,
|
| 15 |
+
"hidden_act": "silu",
|
| 16 |
+
"hidden_size": 1024,
|
| 17 |
+
"indexer_budget": null,
|
| 18 |
+
"indexer_compress_ratio": null,
|
| 19 |
+
"indexer_head_dim": null,
|
| 20 |
+
"indexer_kv_heads": null,
|
| 21 |
+
"indexer_n_heads": null,
|
| 22 |
+
"initializer_range": 0.02,
|
| 23 |
+
"layer_types": [
|
| 24 |
+
"linear_attention",
|
| 25 |
+
"linear_attention",
|
| 26 |
+
"linear_attention",
|
| 27 |
+
"qwen_sparse_attention",
|
| 28 |
+
"linear_attention",
|
| 29 |
+
"linear_attention",
|
| 30 |
+
"linear_attention",
|
| 31 |
+
"qwen_sparse_attention",
|
| 32 |
+
"linear_attention",
|
| 33 |
+
"linear_attention",
|
| 34 |
+
"linear_attention",
|
| 35 |
+
"qwen_sparse_attention",
|
| 36 |
+
"linear_attention",
|
| 37 |
+
"linear_attention",
|
| 38 |
+
"linear_attention",
|
| 39 |
+
"qwen_sparse_attention",
|
| 40 |
+
"linear_attention",
|
| 41 |
+
"linear_attention",
|
| 42 |
+
"linear_attention",
|
| 43 |
+
"qwen_sparse_attention",
|
| 44 |
+
"linear_attention",
|
| 45 |
+
"linear_attention",
|
| 46 |
+
"linear_attention",
|
| 47 |
+
"qwen_sparse_attention",
|
| 48 |
+
"linear_attention",
|
| 49 |
+
"linear_attention",
|
| 50 |
+
"linear_attention",
|
| 51 |
+
"qwen_sparse_attention"
|
| 52 |
+
],
|
| 53 |
+
"linear_conv_kernel_dim": 4,
|
| 54 |
+
"linear_key_head_dim": 64,
|
| 55 |
+
"linear_num_key_heads": 8,
|
| 56 |
+
"linear_num_value_heads": 16,
|
| 57 |
+
"linear_value_head_dim": 64,
|
| 58 |
+
"loop_repeats": 1,
|
| 59 |
+
"make_ngram_vocab_size_divisible_by": 128,
|
| 60 |
+
"max_position_embeddings": 32768,
|
| 61 |
+
"model_type": "qwen4_exp_text",
|
| 62 |
+
"modujo_loop_repeats": 1,
|
| 63 |
+
"moe_intermediate_size": 1280,
|
| 64 |
+
"ngram_size": 3,
|
| 65 |
+
"ngram_vocab_size_base": 144128,
|
| 66 |
+
"norm_topk_prob": true,
|
| 67 |
+
"num_attention_heads": 16,
|
| 68 |
+
"num_experts": 32,
|
| 69 |
+
"num_experts_per_tok": 2,
|
| 70 |
+
"num_hidden_layers": 28,
|
| 71 |
+
"num_key_value_heads": 2,
|
| 72 |
+
"number_of_conv_states": 3,
|
| 73 |
+
"output_gate_type": "sigmoid",
|
| 74 |
+
"output_router_logits": true,
|
| 75 |
+
"pad_token_id": 248044,
|
| 76 |
+
"ple_conv_kernel_size": 4,
|
| 77 |
+
"ple_embed_dim": 1024,
|
| 78 |
+
"ple_layer_ids": [],
|
| 79 |
+
"rms_norm_eps": 1e-06,
|
| 80 |
+
"rope_parameters": {
|
| 81 |
+
"partial_rotary_factor": 0.25,
|
| 82 |
+
"rope_theta": 10000000,
|
| 83 |
+
"rope_type": "default"
|
| 84 |
+
},
|
| 85 |
+
"router_aux_loss_coef": 0.001,
|
| 86 |
+
"seed": 1234,
|
| 87 |
+
"shared_expert_intermediate_size": 128,
|
| 88 |
+
"split_ngram_parts": 128,
|
| 89 |
+
"tie_word_embeddings": true,
|
| 90 |
+
"transformers_version": "5.16.1",
|
| 91 |
+
"use_cache": false,
|
| 92 |
+
"vocab_size": 248320
|
| 93 |
+
}
|
sft/Modujo-4B-A0.66B/evaluation/cases.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
[{"case": "greeting", "prompt": "你好，请用两句话介绍自己。"}, {"case": "arithmetic", "prompt": "计算 17×23，只回答结果。"}, {"case": "reasoning", "prompt": "小明有12个苹果，给小红3个，又买了5个，现在有几个？请简要说明。"}, {"case": "coding", "prompt": "用 Python 写一个判断字符串是否为回文的函数，忽略大小写和空格。"}, {"case": "ecommerce", "prompt": "我买的鞋子尺码不合适，如何申请退换货？"}, {"case": "logistics", "prompt": "帮我查询订单物流，订单号12345。"}, {"case": "english", "prompt": "What is the capital of France? Answer in one sentence."}, {"case": "format", "prompt": "只输出JSON，不要解释：姓名小王，年龄18，键名为name和age。"}]
|
sft/Modujo-4B-A0.66B/evaluation/code-tests.json
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[
|
| 2 |
+
{
|
| 3 |
+
"input": "racecar",
|
| 4 |
+
"expected": true,
|
| 5 |
+
"actual": true,
|
| 6 |
+
"passed": true
|
| 7 |
+
},
|
| 8 |
+
{
|
| 9 |
+
"input": "hello",
|
| 10 |
+
"expected": false,
|
| 11 |
+
"actual": false,
|
| 12 |
+
"passed": true
|
| 13 |
+
},
|
| 14 |
+
{
|
| 15 |
+
"input": "Aa",
|
| 16 |
+
"expected": true,
|
| 17 |
+
"actual": true,
|
| 18 |
+
"passed": true
|
| 19 |
+
},
|
| 20 |
+
{
|
| 21 |
+
"input": "Was it a car or a cat I saw",
|
| 22 |
+
"expected": true,
|
| 23 |
+
"actual": true,
|
| 24 |
+
"passed": true
|
| 25 |
+
},
|
| 26 |
+
{
|
| 27 |
+
"input": "a,ba",
|
| 28 |
+
"expected": false,
|
| 29 |
+
"actual": true,
|
| 30 |
+
"passed": false
|
| 31 |
+
}
|
| 32 |
+
]
|
sft/Modujo-4B-A0.66B/evaluation/extended-cases.json
ADDED
|
@@ -0,0 +1,28 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[
|
| 2 |
+
{
|
| 3 |
+
"case": "coding_extended",
|
| 4 |
+
"prompt": "用 Python 写一个判断字符串是否为回文的函数，忽略大小写和空格。"
|
| 5 |
+
},
|
| 6 |
+
{
|
| 7 |
+
"case": "subtract_control",
|
| 8 |
+
"prompt": "小明有12个苹果，给小红3个，又买了5个，现在有几个？只回答数字。"
|
| 9 |
+
},
|
| 10 |
+
{
|
| 11 |
+
"case": "add_control",
|
| 12 |
+
"prompt": "小明有12个苹果，小红给他3个，又买了5个，现在有几个？只回答数字。"
|
| 13 |
+
},
|
| 14 |
+
{
|
| 15 |
+
"case": "think_math",
|
| 16 |
+
"prompt": "<|im_start|>system\nReasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.<|im_end|>\n<|im_start|>user\n计算 17×23，思考后给出答案。<|im_end|>\n<|im_start|>assistant\n<think>\n",
|
| 17 |
+
"prompt_format": "raw",
|
| 18 |
+
"thinking_mode": "low",
|
| 19 |
+
"user_prompt": "计算 17×23，思考后给出答案。"
|
| 20 |
+
},
|
| 21 |
+
{
|
| 22 |
+
"case": "think_logic",
|
| 23 |
+
"prompt": "<|im_start|>system\nReasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.<|im_end|>\n<|im_start|>user\n小明有12个苹果，给小红3个，又买了5个，现在有几个？<|im_end|>\n<|im_start|>assistant\n<think>\n",
|
| 24 |
+
"prompt_format": "raw",
|
| 25 |
+
"thinking_mode": "low",
|
| 26 |
+
"user_prompt": "小明有12个苹果，给小红3个，又买了5个，现在有几个？"
|
| 27 |
+
}
|
| 28 |
+
]
|
sft/Modujo-4B-A0.66B/evaluation/extended.jsonl
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"checkpoint": "Modujo-AI/ModujoMoE/sft/Modujo-4B-A0.66B", "device": "cuda:0", "dtype": "bfloat16", "case": "coding_extended", "prompt": "用 Python 写一个判断字符串是否为回文的函数，忽略大小写和空格。", "prompt_mode": "nonthinking", "prompt_format": "chat", "loop_repeats": 1, "seed": 42, "decoding": "greedy", "repetition_penalty": 1.0, "answer": "你可以使用 Python 的内置函数 `str.lower()` 来检查一个字符串是否为回文。下面是一个简单的函数实现：\n\n```python\ndef is_palindrome(s):\n # 移除字符串中的空格和标点符号，并将所有字符转换为小写\n cleaned_s = ''.join(c.lower() for c in s if c.isalnum())\n \n # 检查清理后的字符串是否等于其反转字符串\n return cleaned_s == cleaned_s[::-1]\n\n# 测试函数\ntest_strings = [\"A man, a plan, a canal, Panama\", \"racecar\", \"hello\", \"Was it a car or a cat I saw?\"]\nfor test in test_strings:\n print(f\"'{test}' is a palindrome: {is_palindrome(test)}\")\n```\n\n这个函数首先使用列表推导式移除字符串中的所有空格和标点符号，并将所有字符转换为小写。然后，它检查清理后的字符串是否等于其反转字符串。如果相等，则说明原始字符串是回文，函数返回 `True`；否则返回 `False`。\n\n在测试部分，我们定义了一个包含所有字符串的字符串 `test_strings`，然后调用 `is_palindrome` 函数来检查 `test` 函数的输出是否与预期相符。", "token_count": 265, "generated_token_ids": [108621, 96402, 12654, 220, 95726, 111531, 103733, 1510, 485, 17534, 52434, 220, 95860, 97552, 96125, 110089, 130779, 96120, 95905, 1710, 99133, 98267, 101829, 103733, 97318, 4960, 271, 71093, 12305, 198, 727, 369, 70817, 34864, 1104, 1590, 198, 262, 653, 220, 118583, 110089, 96875, 122480, 95793, 141111, 109706, 3709, 109417, 96983, 104071, 117324, 95872, 96674, 198, 262, 26824, 625, 283, 41445, 5799, 1290, 17534, 363, 364, 272, 303, 274, 413, 272, 2011, 90147, 2072, 198, 1031, 262, 653, 220, 97552, 103947, 98159, 110089, 97553, 102149, 96041, 119689, 110089, 198, 262, 460, 26824, 625, 606, 26824, 625, 60059, 16, 60, 271, 2, 220, 99449, 103733, 198, 1877, 32388, 283, 4241, 32, 855, 11, 264, 3019, 11, 264, 37633, 11, 46434, 487, 328, 39823, 6702, 487, 328, 14556, 487, 328, 25202, 424, 264, 1740, 466, 264, 7993, 353, 5312, 30, 1293, 198, 1891, 1228, 303, 1228, 32388, 25, 198, 262, 1133, 925, 19959, 90, 1877, 11325, 369, 264, 70613, 25, 313, 284, 70817, 34864, 8359, 8866, 871, 198, 71093, 271, 96488, 103733, 99924, 96402, 101831, 141260, 96102, 118583, 110089, 145049, 122480, 95793, 141111, 109706, 3709, 109417, 96983, 104071, 117324, 95872, 96674, 1710, 98118, 3709, 96540, 97552, 103947, 98159, 110089, 97553, 102149, 96041, 119689, 110089, 1710, 96604, 115452, 3709, 96711, 98419, 104159, 110089, 95761, 96120, 95905, 3709, 103733, 99488, 1510, 2434, 63, 24178, 103793, 99488, 1510, 3913, 63, 1710, 271, 95772, 99449, 96985, 3709, 96091, 99546, 101603, 99516, 96983, 110089, 95726, 110089, 1510, 1877, 32388, 63, 3709, 98118, 109185, 1510, 284, 70817, 34864, 63, 220, 103733, 95860, 97552, 1510, 1877, 63, 220, 118828, 100627, 147343, 100797, 118172, 1710, 248046], "raw_answer": "你可以使用 Python 的内置函数 `str.lower()` 来检查一个字符串是否为回文。下面是一个简单的函数实现：\n\n```python\ndef is_palindrome(s):\n # 移除字符串中的空格和标点符号，并将所有字符转换为小写\n cleaned_s = ''.join(c.lower() for c in s if c.isalnum())\n \n # 检查清理后的字符串是否等于其反转字符串\n return cleaned_s == cleaned_s[::-1]\n\n# 测试函数\ntest_strings = [\"A man, a plan, a canal, Panama\", \"racecar\", \"hello\", \"Was it a car or a cat I saw?\"]\nfor test in test_strings:\n print(f\"'{test}' is a palindrome: {is_palindrome(test)}\")\n```\n\n这个函数首先使用列表推导式移除字符串中的所有空格和标点符号，并将所有字符转换为小写。然后，它检查清理后的字符串是否等于其反转字符串。如果相等，则说明原始字符串是回文，函数返回 `True`；否则返回 `False`。\n\n在测试部分，我们定义了一个包含所有字符串的字符串 `test_strings`，然后调用 `is_palindrome` 函数来检查 `test` 函数的输出是否与预期相符。<|im_end|>", "finish_reason": "eos"}
|
| 2 |
+
{"checkpoint": "Modujo-AI/ModujoMoE/sft/Modujo-4B-A0.66B", "device": "cuda:0", "dtype": "bfloat16", "case": "subtract_control", "prompt": "小明有12个苹果，给小红3个，又买了5个，现在有几个？只回答数字。", "prompt_mode": "nonthinking", "prompt_format": "chat", "loop_repeats": 1, "seed": 42, "decoding": "greedy", "repetition_penalty": 1.0, "answer": "小明有12个苹果，给小红3个，又买了5个，问总共买了多少个苹果。\n\n**步骤解析：**\n\n1. **计算12个苹果的数量：**\n - 小明有12个苹果。\n - 给小红3个，所以总数是 \\(12 + 3 = 15\\) 个苹果。\n\n2. **计算15个苹果的数量���**\n - 小明买了5个，所以总数是 \\(5 + 5 = 10\\) 个苹果。\n\n3. **计算10个苹果的数量：**\n - 小明买了10个，所以总数是 \\(10 + 10 = 20\\) 个苹果。\n\n**答案：**\n小明总共买了 **20个苹果**。", "token_count": 169, "generated_token_ids": [115942, 95762, 16, 17, 95840, 99203, 3709, 96237, 105914, 18, 95840, 3709, 96473, 110352, 20, 95840, 3709, 96094, 116105, 110352, 134371, 99203, 1710, 271, 332, 100661, 100596, 245342, 271, 16, 13, 2972, 97792, 16, 17, 95840, 99203, 113020, 245342, 198, 256, 471, 220, 115942, 95762, 16, 17, 95840, 99203, 1710, 198, 256, 471, 220, 96237, 105914, 18, 95840, 3709, 97483, 112056, 95761, 17237, 16, 17, 478, 220, 18, 283, 220, 16, 20, 55821, 220, 95840, 99203, 1710, 271, 17, 13, 2972, 97792, 16, 20, 95840, 99203, 113020, 245342, 198, 256, 471, 220, 115942, 110352, 20, 95840, 3709, 97483, 112056, 95761, 17237, 20, 478, 220, 20, 283, 220, 16, 15, 55821, 220, 95840, 99203, 1710, 271, 18, 13, 2972, 97792, 16, 15, 95840, 99203, 113020, 245342, 198, 256, 471, 220, 115942, 110352, 16, 15, 95840, 3709, 97483, 112056, 95761, 17237, 16, 15, 478, 220, 16, 15, 283, 220, 17, 15, 55821, 220, 95840, 99203, 1710, 271, 332, 97632, 245342, 198, 115942, 116105, 110352, 2972, 17, 15, 95840, 99203, 221794, 248046], "raw_answer": "小明有12个苹果，给小红3个，又买了5个，问总共买了多少个苹果。\n\n**步骤解析：**\n\n1. **计算12个苹果的数量：**\n - 小明有12个苹果。\n - 给小红3个，所以总数是 \\(12 + 3 = 15\\) 个苹果。\n\n2. **计算15个苹果的数量：**\n - 小明买了5个，所以总数是 \\(5 + 5 = 10\\) 个苹果。\n\n3. **计算10个苹果的数量：**\n - 小明买了10个，所以总数是 \\(10 + 10 = 20\\) 个苹果。\n\n**答案：**\n小明总共买了 **20个苹果**。<|im_end|>", "finish_reason": "eos"}
|
| 3 |
+
{"checkpoint": "Modujo-AI/ModujoMoE/sft/Modujo-4B-A0.66B", "device": "cuda:0", "dtype": "bfloat16", "case": "add_control", "prompt": "小明有12个苹果，小红给他3个，又买了5个，现在有几个？只回答数字。", "prompt_mode": "nonthinking", "prompt_format": "chat", "loop_repeats": 1, "seed": 42, "decoding": "greedy", "repetition_penalty": 1.0, "answer": "小明有12个苹果，小红有3个，又买了5个，问总共多少个苹果。\n\n**步骤解析：**\n\n1. **计算总数：** \n 总数 = 12 + 3 = **15个**\n\n2. **计算总数量：** \n 总数 = 15 + 5 = **20个**\n\n3. **求总数：** \n 总数 = 15 + 20 = **35个**\n\n**答案：** \n小明有 **35** 个苹果。", "token_count": 117, "generated_token_ids": [115942, 95762, 16, 17, 95840, 99203, 3709, 105914, 95762, 18, 95840, 3709, 96473, 110352, 20, 95840, 3709, 96094, 116105, 134371, 99203, 1710, 271, 332, 100661, 100596, 245342, 271, 16, 13, 2972, 97792, 112056, 245342, 2228, 256, 220, 112056, 283, 220, 16, 17, 478, 220, 18, 283, 2972, 16, 20, 95840, 332, 271, 17, 13, 2972, 97792, 96184, 99474, 245342, 2228, 256, 220, 112056, 283, 220, 16, 20, 478, 220, 20, 283, 2972, 17, 15, 95840, 332, 271, 18, 13, 2972, 96181, 112056, 245342, 2228, 256, 220, 112056, 283, 220, 16, 20, 478, 220, 17, 15, 283, 2972, 18, 20, 95840, 332, 271, 332, 97632, 245342, 2228, 115942, 95762, 2972, 18, 20, 332, 220, 95840, 99203, 1710, 248046], "raw_answer": "小明有12个苹果，小红有3个，又买了5个，问总共多少个苹果。\n\n**步骤解析：**\n\n1. **计算总数：** \n 总数 = 12 + 3 = **15个**\n\n2. **计算总数量：** \n 总数 = 15 + 5 = **20个**\n\n3. **求总数：** \n 总数 = 15 + 20 = **35个**\n\n**答案：** \n小明有 **35** 个苹果。<|im_end|>", "finish_reason": "eos"}
|
| 4 |
+
{"checkpoint": "Modujo-AI/ModujoMoE/sft/Modujo-4B-A0.66B", "device": "cuda:0", "dtype": "bfloat16", "case": "think_math", "prompt": "<|im_start|>system\nReasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.<|im_end|>\n<|im_start|>user\n计算 17×23，思考后给出答案。<|im_end|>\n<|im_start|>assistant\n<think>\n", "prompt_mode": "nonthinking", "prompt_format": "raw", "loop_repeats": 1, "seed": 42, "decoding": "greedy", "repetition_penalty": 1.0, "answer": "我现在需要计算17×23，然后思考后给出答案。首先，我应该回忆一下乘法的基本概念，乘法是交换律，交换加数的位置，所以这里应该是17×23。接下来，我需要计算17×23，也就是17乘以20，然后加上17乘以3，再加上17乘以4，再加上17乘以5，再加上17乘以6，再加上17乘以7，再加上17乘以8，再加上17乘以9，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加��17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17", "token_count": 512, "generated_token_ids": [111449, 96442, 97792, 16, 22, 17044, 17, 18, 3709, 98118, 99496, 95946, 110334, 97632, 1710, 99924, 3709, 129347, 101920, 97434, 124158, 100751, 99518, 3709, 124158, 95761, 104176, 96516, 3709, 104176, 95984, 99687, 97795, 3709, 97483, 97490, 108844, 16, 22, 17044, 17, 18, 1710, 108697, 3709, 128595, 97792, 16, 22, 17044, 17, 18, 3709, 101644, 16, 22, 133270, 17, 15, 3709, 98118, 103717, 16, 22, 133270, 18, 3709, 109217, 16, 22, 133270, 19, 3709, 109217, 16, 22, 133270, 20, 3709, 109217, 16, 22, 133270, 21, 3709, 109217, 16, 22, 133270, 22, 3709, 109217, 16, 22, 133270, 23, 3709, 109217, 16, 22, 133270, 24, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22, 133270, 16, 15, 3709, 109217, 16, 22], "raw_answer": "我现在需要计算17×23，然后思考后给出答案。首先，我应该回忆一下乘法的基本概念，乘法是交换律，交换加数的位置，所以这里应该是17×23。接下来，我需要计算17×23，也就是17乘以20，然后加上17乘以3，再加上17乘以4，再加上17乘以5，再加上17乘以6，再加上17乘以7，再加上17乘以8，再加上17乘以9，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17���以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17", "finish_reason": "length"}
|
| 5 |
+
{"checkpoint": "Modujo-AI/ModujoMoE/sft/Modujo-4B-A0.66B", "device": "cuda:0", "dtype": "bfloat16", "case": "think_logic", "prompt": "<|im_start|>system\nReasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.<|im_end|>\n<|im_start|>user\n小明有12个苹果，给小红3个，又买了5个，现在有几个？<|im_end|>\n<|im_start|>assistant\n<think>\n", "prompt_mode": "nonthinking", "prompt_format": "raw", "loop_repeats": 1, "seed": 42, "decoding": "greedy", "repetition_penalty": 1.0, "answer": "我现在要解决的问题是：小明有12个苹果，给小红3个，又买了5个，现在有几个？首先，我需要理清楚题目中的各个数据之间的关系。首先，小明有12个苹果，然后小红买了5个，所以首先应该算出小红买了多少个苹果，然后再减去5个，就是现在的数量。\n\n首先，计算小红买了多少个苹果。12个苹果，每个3个，所以应该是12乘以3。12乘3等于36，所以小红买了36个苹果。\n\n接下来，题目说现在有几个。这里需要注意，题目里说“现在有几个”，所以应该是总数减去已经有的数量，也就是总数减去已有的数量。也就是36减12，等于24。\n\n不过，我需要确认一下是否正确。有时候可能会有陷阱，比如是否需要考虑其他因素，比如是否有可能小红买了5个，但题目里说“现在有几个”，所以应该是总数减去已有的数量，也就是总数减去已有的数量。所以计算正确的话，应该是36-12=24。\n\n再检查一下，12个苹果，每个3个，12×3=36，没错。然后总数是36-12=24，应该是对的。\n\n可能需要注意的地方是，是否理解“现在有几个”是指总数减去已有的数量，也就是总数减去已有的数量，所以总数减去已有的数量就是现在的数量。这里总数是12，已有的数量是36，所以现在的数量是36-12=24。\n\n总结一下，步骤是：先算小红买了多少个，再减去5个，得到现在的数量。计算过程正确，结果应该是24个。\n</think>\n\n小明有12个苹果，小红买了5个，因此小红买了： \n**12 × 3 = 36个** \n\n现在需要计算现在有多少个苹果。 \n**总数 - 已有的数量 = 总数 - 已有的数量 = 36 - 12 = 24个** \n\n**答案：现在有几个？**", "token_count": 432, "generated_token_ids": [111449, 95822, 121736, 95761, 4960, 115942, 95762, 16, 17, 95840, 99203, 3709, 96237, 105914, 18, 95840, 3709, 96473, 110352, 20, 95840, 3709, 96846, 119239, 10992, 99924, 3709, 128595, 95856, 98309, 103998, 96875, 99560, 96902, 112579, 1710, 99924, 3709, 115942, 95762, 16, 17, 95840, 99203, 3709, 98118, 105914, 110352, 20, 95840, 3709, 97483, 99924, 97433, 96525, 95841, 105914, 110352, 98425, 95840, 99203, 3709, 110612, 130185, 20, 95840, 3709, 96496, 108658, 99474, 1710, 271, 99924, 3709, 97792, 105914, 110352, 98425, 95840, 99203, 1710, 16, 17, 95840, 99203, 3709, 99448, 18, 95840, 3709, 97483, 108844, 16, 17, 133270, 18, 1710, 16, 17, 97595, 18, 102149, 18, 21, 3709, 97483, 105914, 110352, 18, 21, 95840, 99203, 1710, 271, 108697, 3709, 103998, 96003, 96846, 119239, 1710, 97490, 109190, 3709, 103998, 96067, 96003, 2005, 96846, 119239, 32478, 97483, 108844, 112056, 130185, 96790, 97332, 99474, 3709, 101644, 112056, 130185, 96221, 97332, 99474, 1710, 101644, 18, 21, 96701, 16, 17, 3709, 102149, 17, 19, 1710, 271, 98609, 3709, 128595, 101632, 97434, 134014, 1710, 108913, 121832, 112994, 3709, 101589, 97553, 121568, 96903, 98278, 3709, 101589, 97553, 100917, 105914, 110352, 20, 95840, 3709, 96342, 103998, 96067, 96003, 2005, 96846, 119239, 32478, 97483, 108844, 112056, 130185, 96221, 97332, 99474, 3709, 101644, 112056, 130185, 96221, 97332, 99474, 1710, 97483, 97792, 97901, 101587, 3709, 108844, 18, 21, 12, 16, 17, 28, 17, 19, 1710, 271, 96390, 133182, 3709, 16, 17, 95840, 99203, 3709, 99448, 18, 95840, 3709, 16, 17, 17044, 18, 28, 18, 21, 3709, 111678, 1710, 98118, 112056, 95761, 18, 21, 12, 16, 17, 28, 17, 19, 3709, 97433, 136029, 1710, 271, 96539, 109190, 108529, 95761, 3709, 97553, 98142, 2005, 96846, 119239, 828, 103722, 112056, 130185, 96221, 97332, 99474, 3709, 101644, 112056, 130185, 96221, 97332, 99474, 3709, 97483, 112056, 130185, 96221, 97332, 99474, 96496, 108658, 99474, 1710, 97490, 112056, 95761, 16, 17, 3709, 96221, 97332, 99474, 95761, 18, 21, 3709, 97483, 108658, 99474, 95761, 18, 21, 12, 16, 17, 28, 17, 19, 1710, 271, 141087, 3709, 100661, 95761, 4960, 96218, 96525, 105914, 110352, 98425, 95840, 3709, 96390, 130185, 20, 95840, 3709, 98257, 108658, 99474, 1710, 97792, 97424, 97901, 3709, 97989, 108844, 17, 19, 95840, 1710, 198, 248069, 271, 115942, 95762, 16, 17, 95840, 99203, 3709, 105914, 110352, 20, 95840, 3709, 100532, 105914, 110352, 4960, 2228, 332, 16, 17, 23985, 220, 18, 283, 220, 18, 21, 95840, 332, 18050, 96846, 96442, 97792, 96846, 105022, 95840, 99203, 1710, 2228, 332, 112056, 471, 220, 96221, 97332, 99474, 283, 220, 112056, 471, 220, 96221, 97332, 99474, 283, 220, 18, 21, 471, 220, 16, 17, 283, 220, 17, 19, 95840, 332, 18050, 332, 97632, 4960, 96846, 119239, 232787, 248046], "raw_answer": "我现在要解决的问题是：小明有12个苹果，给小红3个，又买了5个，现在有几个？首先，我需要理清楚题目中的各个数据之间的关系。首先，小明有12个苹果，然后小红买了5个，所以首先应该算出小红买了多少个苹果，然后再减去5个，就是现在的数量。\n\n首先，计算小红买了多少个苹果。12个苹果，每个3个，所以应该是12乘以3。12乘3等于36，所以小红买了36个苹果。\n\n接下来，题目说现在有几个。这里需要注意，题目里说“现在有几个”，所以应该是总数减去已经有的数量，也就是总数减去已有的数量。也就是36减12，等于24。\n\n不过，我需要确认一下是否正确。有时候可能会有陷阱，比如是否需要考虑其他因素，比如是否有可能小红买了5个，但题目里说“现在有几个”，所以应该是总数减去已有的数量，也就是总数减去已有的数量。所以计算正确的话，应该是36-12=24。\n\n再检查一下，12个苹果，每个3个，12×3=36，没错。然后总数是36-12=24，应该是对的。\n\n可能需要注意的地方是，是否理解“现在有几个”是指总数减去已有的数量，也就是总数减去已有的数量，所以总数减去已有的数量就是现在的数量。这里总数是12，已有的数量是36，所以现在的数量是36-12=24。\n\n总结一下，步骤是：先算小红买了多少个，再减去5个，得到现在的数量。计算过程正确，结果应该是24个。\n</think>\n\n小明有12个苹果，小红买了5个，因此小红买了： \n**12 × 3 = 36个** \n\n现在需要计算现在有多少个苹果。 \n**总数 - 已有的数量 = 总数 - 已有的数量 = 36 - 12 = 24个** \n\n**答案：现在有几个？**<|im_end|>", "finish_reason": "eos"}
|
sft/Modujo-4B-A0.66B/evaluation/greedy.jsonl
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"checkpoint": "Modujo-AI/ModujoMoE/sft/Modujo-4B-A0.66B", "device": "cuda:0", "dtype": "bfloat16", "case": "greeting", "prompt": "你好，请用两句话介绍自己。", "prompt_mode": "nonthinking", "prompt_format": "chat", "loop_repeats": 1, "seed": 42, "decoding": "greedy", "repetition_penalty": 1.0, "answer": "你好！很高兴为你介绍自己。", "token_count": 8, "generated_token_ids": [109266, 6115, 116577, 101882, 97431, 96115, 1710, 248046], "raw_answer": "你好！很高兴为你介绍自己。<|im_end|>", "finish_reason": "eos"}
|
| 2 |
+
{"checkpoint": "Modujo-AI/ModujoMoE/sft/Modujo-4B-A0.66B", "device": "cuda:0", "dtype": "bfloat16", "case": "arithmetic", "prompt": "计算 17×23，只回答结果。", "prompt_mode": "nonthinking", "prompt_format": "chat", "loop_repeats": 1, "seed": 42, "decoding": "greedy", "repetition_penalty": 1.0, "answer": "计算 $17 \\times 23$ 的结果需要知道具体的数值。由于您没有提供具体的数值，我将提供一个通用的计算方法。\n\n1. **确定数值范围**：\n - 如果 $17 \\times 23 = 307$，则 $17 \\times 23 = 307$。\n - 如果 $17 \\times 23 = 308$，则 $17 \\times 23 = 308$。\n\n2. **计算结果**：\n - 将 $307$ 和 $308$ 相加，得到 $307 + 308 = 695$。\n - 将 $308$ 和 $309$ 相加，得到 $695 + 309 = 984$。", "token_count": 192, "generated_token_ids": [97792, 393, 16, 22, 1088, 14695, 220, 17, 18, 3, 220, 109146, 96442, 96645, 109542, 110745, 1710, 99121, 96347, 96487, 96553, 109542, 110745, 3709, 110931, 116926, 129396, 127249, 1710, 271, 16, 13, 2972, 98825, 110745, 97497, 332, 4960, 198, 256, 471, 220, 96604, 393, 16, 22, 1088, 14695, 220, 17, 18, 283, 220, 18, 15, 22, 3, 3709, 96711, 393, 16, 22, 1088, 14695, 220, 17, 18, 283, 220, 18, 15, 22, 3, 1710, 198, 256, 471, 220, 96604, 393, 16, 22, 1088, 14695, 220, 17, 18, 283, 220, 18, 15, 23, 3, 3709, 96711, 393, 16, 22, 1088, 14695, 220, 17, 18, 283, 220, 18, 15, 23, 3, 1710, 271, 17, 13, 2972, 97792, 97989, 332, 4960, 198, 256, 471, 220, 96123, 393, 18, 15, 22, 3, 220, 95793, 393, 18, 15, 23, 3, 220, 134351, 3709, 98257, 393, 18, 15, 22, 478, 220, 18, 15, 23, 283, 220, 21, 24, 20, 3, 1710, 198, 256, 471, 220, 96123, 393, 18, 15, 23, 3, 220, 95793, 393, 18, 15, 24, 3, 220, 134351, 3709, 98257, 393, 21, 24, 20, 478, 220, 18, 15, 24, 283, 220, 24, 23, 19, 3, 1710], "raw_answer": "计算 $17 \\times 23$ 的结果需要知道具体的数值。由于您没有提供具体的数值，我将提供一个通用的计算方法。\n\n1. **确定数值范围**：\n - 如果 $17 \\times 23 = 307$，则 $17 \\times 23 = 307$。\n - 如果 $17 \\times 23 = 308$，则 $17 \\times 23 = 308$。\n\n2. **计算结果**：\n - 将 $307$ 和 $308$ 相加，得到 $307 + 308 = 695$。\n - 将 $308$ 和 $309$ 相加，得到 $695 + 309 = 984$。", "finish_reason": "length"}
|
| 3 |
+
{"checkpoint": "Modujo-AI/ModujoMoE/sft/Modujo-4B-A0.66B", "device": "cuda:0", "dtype": "bfloat16", "case": "reasoning", "prompt": "小明有12个苹果，给小红3个，又买了5个，现在有几个？请简要说明。", "prompt_mode": "nonthinking", "prompt_format": "chat", "loop_repeats": 1, "seed": 42, "decoding": "greedy", "repetition_penalty": 1.0, "answer": "小明有12个苹果，给小红3个，又买了5个，现在有几个？\n\n**解答过程：**\n\n1. **计算小明总共买了多少个苹果？** \n 小明有12个苹果，给小红3个，所以总共买了： \n $ 12 + 3 = 15 $ 个苹果。\n\n2. **计算小明买了多少个苹果？** \n 小明买了5个苹果，所以买了： \n $ 5 \\times 15 = 75 $ 个。\n\n3. **计算小明总共买了多少个苹果？** \n 小明买了75个苹果，所以总共买了： \n $ 75 \\times 1 = 75 $ 个。\n\n**答案：** \n小明有 **75** 个苹果。", "token_count": 174, "generated_token_ids": [115942, 95762, 16, 17, 95840, 99203, 3709, 96237, 105914, 18, 95840, 3709, 96473, 110352, 20, 95840, 3709, 96846, 119239, 10992, 271, 332, 101776, 97424, 245342, 271, 16, 13, 2972, 97792, 115942, 116105, 110352, 98425, 95840, 99203, 232787, 2228, 256, 220, 115942, 95762, 16, 17, 95840, 99203, 3709, 96237, 105914, 18, 95840, 3709, 97483, 116105, 110352, 4960, 2228, 256, 393, 220, 16, 17, 478, 220, 18, 283, 220, 16, 20, 393, 220, 95840, 99203, 1710, 271, 17, 13, 2972, 97792, 115942, 110352, 98425, 95840, 99203, 232787, 2228, 256, 220, 115942, 110352, 20, 95840, 99203, 3709, 97483, 110352, 4960, 2228, 256, 393, 220, 20, 1088, 14695, 220, 16, 20, 283, 220, 22, 20, 393, 220, 95840, 1710, 271, 18, 13, 2972, 97792, 115942, 116105, 110352, 134371, 99203, 232787, 2228, 256, 220, 115942, 110352, 22, 20, 95840, 99203, 3709, 97483, 116105, 110352, 4960, 2228, 256, 393, 220, 22, 20, 1088, 14695, 220, 16, 283, 220, 22, 20, 393, 220, 95840, 1710, 271, 332, 97632, 245342, 2228, 115942, 95762, 2972, 22, 20, 332, 220, 95840, 99203, 1710, 248046], "raw_answer": "小明有12个苹果，给小红3个，又买了5个，现在有几个？\n\n**解答过程：**\n\n1. **计算小明总共买了多少个苹果？** \n 小明有12个苹果，给小红3���，所以总共买了： \n $ 12 + 3 = 15 $ 个苹果。\n\n2. **计算小明买了多少个苹果？** \n 小明买了5个苹果，所以买了： \n $ 5 \\times 15 = 75 $ 个。\n\n3. **计算小明总共买了多少个苹果？** \n 小明买了75个苹果，所以总共买了： \n $ 75 \\times 1 = 75 $ 个。\n\n**答案：** \n小明有 **75** 个苹果。<|im_end|>", "finish_reason": "eos"}
|
| 4 |
+
{"checkpoint": "Modujo-AI/ModujoMoE/sft/Modujo-4B-A0.66B", "device": "cuda:0", "dtype": "bfloat16", "case": "coding", "prompt": "用 Python 写一个判断字符串是否为回文的函数，忽略大小写和空格。", "prompt_mode": "nonthinking", "prompt_format": "chat", "loop_repeats": 1, "seed": 42, "decoding": "greedy", "repetition_penalty": 1.0, "answer": "你可以使用 Python 的内置函数 `str.lower()` 来检查一个字符串是否为回文。下面是一个简单的函数实现：\n\n```python\ndef is_palindrome(s):\n # 移除字符串中的空格和标点符号，并将所有字符转换为小写\n cleaned_s = ''.join(c.lower() for c in s if c.isalnum())\n \n # 检查清理后的字符串是否等于其反转字符串\n return cleaned_s == cleaned_s[::-1]\n\n# 测试函数\ntest_strings = [\"A man, a plan, a canal, Panama\", \"racecar\", \"hello\", \"Was it a car or a cat I saw?\"]\nfor test in test_strings:\n print(f\"'{test}' is a palindrome: {is_palindrome(test)}\")\n```\n\n这个函数首先使用列表推导式移除字符串中的所有空格和标点符号，并将所有字符转换为小写。然后，它", "token_count": 192, "generated_token_ids": [108621, 96402, 12654, 220, 95726, 111531, 103733, 1510, 485, 17534, 52434, 220, 95860, 97552, 96125, 110089, 130779, 96120, 95905, 1710, 99133, 98267, 101829, 103733, 97318, 4960, 271, 71093, 12305, 198, 727, 369, 70817, 34864, 1104, 1590, 198, 262, 653, 220, 118583, 110089, 96875, 122480, 95793, 141111, 109706, 3709, 109417, 96983, 104071, 117324, 95872, 96674, 198, 262, 26824, 625, 283, 41445, 5799, 1290, 17534, 363, 364, 272, 303, 274, 413, 272, 2011, 90147, 2072, 198, 1031, 262, 653, 220, 97552, 103947, 98159, 110089, 97553, 102149, 96041, 119689, 110089, 198, 262, 460, 26824, 625, 606, 26824, 625, 60059, 16, 60, 271, 2, 220, 99449, 103733, 198, 1877, 32388, 283, 4241, 32, 855, 11, 264, 3019, 11, 264, 37633, 11, 46434, 487, 328, 39823, 6702, 487, 328, 14556, 487, 328, 25202, 424, 264, 1740, 466, 264, 7993, 353, 5312, 30, 1293, 198, 1891, 1228, 303, 1228, 32388, 25, 198, 262, 1133, 925, 19959, 90, 1877, 11325, 369, 264, 70613, 25, 313, 284, 70817, 34864, 8359, 8866, 871, 198, 71093, 271, 96488, 103733, 99924, 96402, 101831, 141260, 96102, 118583, 110089, 145049, 122480, 95793, 141111, 109706, 3709, 109417, 96983, 104071, 117324, 95872, 96674, 1710, 98118, 3709, 96540], "raw_answer": "你可以使用 Python 的内置函数 `str.lower()` 来检查一个字符串是否为回文。下面是一个简单的函数实现：\n\n```python\ndef is_palindrome(s):\n # 移除字符串中的空格和标点符号，并将所有字符转换为小写\n cleaned_s = ''.join(c.lower() for c in s if c.isalnum())\n \n # 检查清理后的字符串是否等于其反转字符串\n return cleaned_s == cleaned_s[::-1]\n\n# 测试函数\ntest_strings = [\"A man, a plan, a canal, Panama\", \"racecar\", \"hello\", \"Was it a car or a cat I saw?\"]\nfor test in test_strings:\n print(f\"'{test}' is a palindrome: {is_palindrome(test)}\")\n```\n\n这个函数首先使用列表推导式移除字符串中的所有空格和标点符号，并将所有字符转换为小写。然后，它", "finish_reason": "length"}
|
| 5 |
+
{"checkpoint": "Modujo-AI/ModujoMoE/sft/Modujo-4B-A0.66B", "device": "cuda:0", "dtype": "bfloat16", "case": "ecommerce", "prompt": "我买的鞋子尺码不合适，如何申请退换货？", "prompt_mode": "nonthinking", "prompt_format": "chat", "loop_repeats": 1, "seed": 42, "decoding": "greedy", "repetition_penalty": 1.0, "answer": "申请退换货时，通常需要遵循以下步骤：\n\n1. **确认退换货政策**：首先，你需要确认你购买鞋子是否真的需要退换货。这通常可以通过购买时的商家提供的退换货政策来完成。确保你了解退换货的条件，比如商品是否完好、包装是否完好、是否有任何质量问题等。\n\n2. **查看商品详情**：在退换货页面，仔细阅读商品描述、规格、用户评价、退换货政策等信息。确保你了解商品的真实情况，包括尺寸、材质、颜色、功能等。\n\n3. **联系卖家**：如果商品确实需要退换货，尝试联系卖家。大多数电商平台都有消费者保护政策，你可以按照平台指引进行操作。在联系时，提供你的订单号、购买日期、商品描述、退换货原因等信息，并询问卖家是否接受退换货。\n\n4. **提交退换货申请**：根据卖家提供的信息，提交退换货申请。这通常可以通过平台提供的在线申请系统完成。确保在提交前", "token_count": 192, "generated_token_ids": [97379, 129125, 95865, 3709, 146350, 109134, 147064, 4960, 271, 16, 13, 2972, 101632, 129125, 97491, 332, 4960, 99924, 3709, 111083, 101632, 95933, 98619, 111774, 97553, 97896, 96442, 129125, 1710, 95854, 99151, 100107, 98619, 98911, 100599, 103853, 129125, 97491, 121120, 1710, 99128, 95933, 96880, 129125, 112479, 3709, 101589, 98820, 97553, 115738, 5205, 98679, 97553, 115738, 5205, 97553, 105224, 118539, 96024, 1710, 271, 17, 13, 2972, 97239, 98820, 99931, 332, 4960, 95772, 129125, 108586, 3709, 124085, 98820, 99172, 5205, 103919, 5205, 97237, 97817, 5205, 129125, 97491, 105030, 1710, 99128, 95933, 96880, 114377, 98444, 96759, 3709, 98249, 100682, 5205, 104026, 5205, 101617, 5205, 97485, 96024, 1710, 271, 18, 13, 2972, 97071, 109724, 332, 4960, 96604, 98820, 101683, 96442, 129125, 3709, 103811, 97071, 109724, 1710, 101749, 119215, 97812, 99463, 97153, 97491, 3709, 108621, 98254, 97190, 104698, 126217, 1710, 95772, 97071, 95865, 3709, 96553, 97319, 108673, 96389, 5205, 98619, 98019, 5205, 98820, 99172, 5205, 129125, 97494, 105030, 3709, 96172, 109044, 109724, 97553, 99127, 129125, 1710, 271, 19, 13, 2972, 100598, 129125, 97379, 332, 4960, 97149, 109724, 103853, 96280, 3709, 100598, 129125, 97379, 1710, 95854, 99151, 100107, 97190, 103853, 98271, 97379, 96746, 97235, 1710, 99128, 95772, 100598, 95990], "raw_answer": "申请退换货时，通常需要遵循以下步骤：\n\n1. **确认退换货政策**：首先，你需要确认你购买鞋子是否真的需要退换货。这通常可以通过购买时的商家提供的退换货政策来完成。确保你了解退换货的条件，比如商品是否完好、包装是否完好、是否有任何质量问题等。\n\n2. **查看商品详情**：在退换货页面，仔细阅读商品描述、规格、用户评价、退换货政策等信息。确保你了解商品的真实情况，包括尺寸、材质、颜色、功能等。\n\n3. **联系卖家**：如果商品确实需要退换货，尝试联系卖家。大多数电商平台都有消费者保护政策，你可以按照平台指引进行操作。在联系时，提供你的订单号、购买日期、商品描述、退换货原因等信息，并询问卖家是否接受退换货。\n\n4. **提交退换货申请**：根据卖家提供的信息，提交退换货申请。这通常可以通过平台提供的在线申请系统完成。确保在提交前", "finish_reason": "length"}
|
| 6 |
+
{"checkpoint": "Modujo-AI/ModujoMoE/sft/Modujo-4B-A0.66B", "device": "cuda:0", "dtype": "bfloat16", "case": "logistics", "prompt": "帮我查询订单物流，订单号12345。", "prompt_mode": "nonthinking", "prompt_format": "chat", "loop_repeats": 1, "seed": 42, "decoding": "greedy", "repetition_penalty": 1.0, "answer": "查询订单物流，通常可以通过以下几种方式：\n\n1. **在线查询**：许多电商平台（如淘宝、京东、亚马逊等）提供在线查询订单的接口。你可以在搜索框中输入订单号，平台会返回相应的物流信息。\n\n2. **电话查询**：如果你是通过电话查询，可以通过拨打客服的客服电话，询问订单号。客服会提供订单号、物流信息（如预计到达时间、预计到达时间等）以及物流公司的联系方式。\n\n3. **电话查询**：如果你是通过电话查询，可以通过拨打客服的客服电话，询问订单号。客服会提供订单号、物流信息（如预计到达时间、预计到达时间等）以及物流公司的联系方式。\n\n4. **电话查询**：如果你是通过电话查询，可以通过拨打客服的客服电话，询问订单号。客服会提供订单号、物流信息（如预计到达时间、预计到达时间等）以及物流公司的联系方式。\n\n5", "token_count": 192, "generated_token_ids": [100549, 108673, 98883, 3709, 99151, 100107, 132772, 96760, 4960, 271, 16, 13, 2972, 98271, 100549, 332, 4960, 98622, 119215, 9616, 96031, 104047, 5205, 101929, 5205, 110661, 96024, 7313, 96553, 98271, 100549, 108673, 95726, 108622, 1710, 126041, 98287, 99709, 136554, 108673, 96389, 3709, 97190, 95825, 99488, 108615, 98883, 96280, 1710, 271, 17, 13, 2972, 97786, 100549, 332, 4960, 98893, 110869, 97786, 100549, 3709, 100107, 102458, 99215, 95726, 128539, 3709, 109044, 108673, 96389, 1710, 99215, 95825, 96553, 108673, 96389, 5205, 98883, 96280, 9616, 96031, 101666, 103890, 96341, 5205, 101666, 103890, 96341, 96024, 7313, 98404, 98883, 99941, 108627, 1710, 271, 18, 13, 2972, 97786, 100549, 332, 4960, 98893, 110869, 97786, 100549, 3709, 100107, 102458, 99215, 95726, 128539, 3709, 109044, 108673, 96389, 1710, 99215, 95825, 96553, 108673, 96389, 5205, 98883, 96280, 9616, 96031, 101666, 103890, 96341, 5205, 101666, 103890, 96341, 96024, 7313, 98404, 98883, 99941, 108627, 1710, 271, 19, 13, 2972, 97786, 100549, 332, 4960, 98893, 110869, 97786, 100549, 3709, 100107, 102458, 99215, 95726, 128539, 3709, 109044, 108673, 96389, 1710, 99215, 95825, 96553, 108673, 96389, 5205, 98883, 96280, 9616, 96031, 101666, 103890, 96341, 5205, 101666, 103890, 96341, 96024, 7313, 98404, 98883, 99941, 108627, 1710, 271, 20], "raw_answer": "查询订单物流，通常可以通过以下几种方式：\n\n1. **在线查询**：许多电商平台（如淘宝、京东、亚马逊等）提供在线查询订单的接口。你可以在搜索框中输入订单号，平台会返回相应的物流信息。\n\n2. **电话查询**：如果你是通过电话查询，可以通过拨打客服的客服电话，询问订单号。客服会提供订单号、物流信息（如预计到达时间、预计到达时间等）以及物流公司的联系方式。\n\n3. **电话查询**：如果你是通过电话查询，可以通过拨打客服的客服电话，询问订单号。客服会提供订单号、物流信息（如预计到达时间、预计到达时间等）以及物流公司的联系方式。\n\n4. **电话查询**：如果你是通过电话查询，可以通过拨打客服的客服电话，询问订单号。客服会提供订单号、物流信息（如预计到达时间、预计到达时间等）以及物流公司的联系方式。\n\n5", "finish_reason": "length"}
|
| 7 |
+
{"checkpoint": "Modujo-AI/ModujoMoE/sft/Modujo-4B-A0.66B", "device": "cuda:0", "dtype": "bfloat16", "case": "english", "prompt": "What is the capital of France? Answer in one sentence.", "prompt_mode": "nonthinking", "prompt_format": "chat", "loop_repeats": 1, "seed": 42, "decoding": "greedy", "repetition_penalty": 1.0, "answer": "The capital of France is Paris.", "token_count": 8, "generated_token_ids": [760, 6511, 314, 9338, 369, 11751, 13, 248046], "raw_answer": "The capital of France is Paris.<|im_end|>", "finish_reason": "eos"}
|
| 8 |
+
{"checkpoint": "Modujo-AI/ModujoMoE/sft/Modujo-4B-A0.66B", "device": "cuda:0", "dtype": "bfloat16", "case": "format", "prompt": "只输出JSON，不要解释：姓名小王，年龄18，键名为name和age。", "prompt_mode": "nonthinking", "prompt_format": "chat", "loop_repeats": 1, "seed": 42, "decoding": "greedy", "repetition_penalty": 1.0, "answer": "在Python中，你可以使用`json`模块来解析JSON字符串。以下是一个示例：\n\n```python\nimport json\n\n# 创建一个字典，键是姓名，值是年龄\nname = \"张三\"\nage = 18\n\n# 使用json.loads()函数将字典转换为JSON字符串\njson_string = json.loads(name, age)\n\n# 打印JSON字符串\nprint(json_string)\n```\n\n在这个例子中，我们首先导入了`json`模块。然后，我们创建了一个字典，其中键是姓名，值是年龄。最后，我们使用`json.loads()`函数将字典转换为JSON字符串。\n\n如果你想要解析一个包含姓名和年龄的JSON字符串，你可以使用`json.loads()`函数：\n\n```python\nimport json\n\n# 创建一个字典，键是姓名，值是年龄\nname = \"张三\"\nage = 18\n\n# 使用json.loads()函数将字典转换为JSON", "token_count": 192, "generated_token_ids": [95772, 29292, 95789, 3709, 108621, 96402, 63, 2164, 63, 103788, 95860, 100596, 5202, 110089, 1710, 97155, 98267, 110393, 4960, 271, 71093, 12305, 198, 464, 2857, 271, 2, 220, 118948, 105844, 3709, 97295, 95761, 108610, 3709, 144334, 99191, 198, 591, 283, 328, 135279, 1, 198, 416, 283, 220, 16, 23, 271, 2, 220, 96402, 2164, 22246, 363, 103733, 96123, 105844, 117324, 5202, 110089, 198, 2164, 3773, 283, 2857, 22246, 3052, 11, 4092, 8, 271, 2, 220, 101679, 5202, 110089, 198, 1302, 9028, 3773, 8, 198, 71093, 271, 100722, 102177, 95789, 3709, 96091, 99924, 96167, 100168, 63, 2164, 63, 103788, 1710, 98118, 3709, 96091, 99157, 101603, 105844, 3709, 98405, 97295, 95761, 108610, 3709, 144334, 99191, 1710, 97790, 3709, 96091, 96402, 63, 2164, 22246, 52434, 103733, 96123, 105844, 117324, 5202, 110089, 1710, 271, 98893, 98831, 100596, 96125, 99516, 108610, 95793, 99191, 95726, 5202, 110089, 3709, 108621, 96402, 63, 2164, 22246, 52434, 103733, 4960, 271, 71093, 12305, 198, 464, 2857, 271, 2, 220, 118948, 105844, 3709, 97295, 95761, 108610, 3709, 144334, 99191, 198, 591, 283, 328, 135279, 1, 198, 416, 283, 220, 16, 23, 271, 2, 220, 96402, 2164, 22246, 363, 103733, 96123, 105844, 117324, 5202], "raw_answer": "在Python中，你可以使用`json`模块来解析JSON字符串。以下是一个示例：\n\n```python\nimport json\n\n# 创建一个字典，键是姓名，值是年龄\nname = \"张三\"\nage = 18\n\n# 使用json.loads()函数将字典转换为JSON字符串\njson_string = json.loads(name, age)\n\n# 打印JSON字符串\nprint(json_string)\n```\n\n在这个例子中，我们首先导入了`json`模块。然后，我们创建了一个字典，其中键是姓名，值是年龄。最后，我们使用`json.loads()`函数将字典转换为JSON字符串。\n\n如果你想要解析一个包含姓名和年龄的JSON字符串，你可以使用`json.loads()`函数：\n\n```python\nimport json\n\n# 创建一个字典，键是姓名，值是年龄\nname = \"张三\"\nage = 18\n\n# 使用json.loads()函数将字典转换为JSON", "finish_reason": "length"}
|
sft/Modujo-4B-A0.66B/evaluation/report.md
ADDED
|
@@ -0,0 +1,327 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 4B SFT checkpoint-36000 阶段评测
|
| 2 |
+
|
| 3 |
+
## 摘要
|
| 4 |
+
|
| 5 |
+
step36000未表现出稳定的任务正确性改善。回文代码保持基础用例4/4，但继续额外删除标点，边界测试失败；数学与严格JSON继续失败。两项512-token low think中，仅苹果题闭合并EOS，乘法题重复至上限，两项均没有正确最终答案。当前权重仍是原始短数据阶段中间产物，不代表教师蒸馏的训练结果。
|
| 6 |
+
|
| 7 |
+
## 实验对象与方法
|
| 8 |
+
|
| 9 |
+
测试启动时step36500尚无snapshot-ready标记，因此选择最新完整本地step36000，保存独立模型快照。原生模板、BF16、eager、greedy、重复惩罚1.0，八项普通题预算192，五项扩展预算512。短数据阶段完成89.96%。
|
| 10 |
+
|
| 11 |
+
旧step31000权重已经轮换清理，无法补跑GPU；改在CPU复测step36000乘法、代码、JSON和英文四题，与保存的step31000 CPU结果比较。CPU/GPU回答并不完全一致，不将跨设备EOS差异视作模型退化证据。
|
| 12 |
+
|
| 13 |
+
## 结果
|
| 14 |
+
|
| 15 |
+
| 任务 | step36000结果 | 判断 |
|
| 16 |
+
| --- | --- | --- |
|
| 17 |
+
| 介绍自己 | 一句礼貌回应，没有真实介绍 | 未完成指令 |
|
| 18 |
+
| 17×23 | 误称没有具体数值，混入307/308等假设 | 失败、截断 |
|
| 19 |
+
| 苹果题 | 先加3再乘5，最终75 | 错误，正确值14 |
|
| 20 |
+
| 回文代码 | 完整函数；四基础用例通过 | 保持局部能力，标点边界失败 |
|
| 21 |
+
| 退换货 | 相关内容但截断 | 未获得完整回答 |
|
| 22 |
+
| 物流 | 通用查询清单，重复电话查询 | 无真实工具，任务未完成 |
|
| 23 |
+
| 英文首都 | Paris | 正确、EOS |
|
| 24 |
+
| 纯JSON | 无关Python示例，错误json.loads用法 | 未遵守要求 |
|
| 25 |
+
|
| 26 |
+
## 代码及同设备对照
|
| 27 |
+
|
| 28 |
+
函数lower后以isalnum保留字符再与反转比较。racecar、hello、Aa和含空格混合大小写回文通过；a,ba错误返回True，因为题目仅要求忽略大小写和空格。扩展4/5，不表示全部需求满足。512预算代码265 token EOS。
|
| 29 |
+
|
| 30 |
+
四项CPU同设备对照中，代码保持同一归一化逻辑，英文保持Paris，数学和JSON仍错。乘法CPU答30，而GPU生成另一错误解释，显示解码路径变化会改变内容，但不能修复此任务。
|
| 31 |
+
|
| 32 |
+
## 数学对照与think
|
| 33 |
+
|
| 34 |
+
给出3个对照答20，正确14；收到3个对照答35，正确20。两题正常结束，却均使用不正确关系，且未仅输出数字。
|
| 35 |
+
|
| 36 |
+
乘法low think不断追加17乘10，512 token没有关闭think；苹果low think混淆人物与事件，采用12×3、36−12，最终推理24，闭合并在432 token EOS。两题均不正确。结束格式及答案正确必须分别评估。
|
| 37 |
+
|
| 38 |
+
## 结论与局限
|
| 39 |
+
|
| 40 |
+
当前产物没有在固定小样本上建立稳定的数学、格式或推理可靠性；代码保持部分收益。验证平均loss不能替代功能验收。本次不做总体准确率估计，未执行标准基准、配比消融或全量内容审计。教师阶段尚未开始，不评价新增教师数据收益。
|
| 41 |
+
|
| 42 |
+
## 附录：原始生成
|
| 43 |
+
|
| 44 |
+
### greeting
|
| 45 |
+
|
| 46 |
+
输入：你好，请用两句话介绍自己。
|
| 47 |
+
|
| 48 |
+
结束：eos；8 token。
|
| 49 |
+
|
| 50 |
+
```text
|
| 51 |
+
你好！很高兴为你介绍自己。<|im_end|>
|
| 52 |
+
```
|
| 53 |
+
|
| 54 |
+
### arithmetic
|
| 55 |
+
|
| 56 |
+
输入：计算 17×23，只回答结果。
|
| 57 |
+
|
| 58 |
+
结束：length；192 token。
|
| 59 |
+
|
| 60 |
+
```text
|
| 61 |
+
计算 $17 \times 23$ 的结果需要知道具体的数值。由于您没有提供具体的数值，我将提供一个通用的计算方法。
|
| 62 |
+
|
| 63 |
+
1. **确定数值范围**：
|
| 64 |
+
- 如果 $17 \times 23 = 307$，则 $17 \times 23 = 307$。
|
| 65 |
+
- 如果 $17 \times 23 = 308$，则 $17 \times 23 = 308$。
|
| 66 |
+
|
| 67 |
+
2. **计算结果**：
|
| 68 |
+
- 将 $307$ 和 $308$ 相加，得到 $307 + 308 = 695$。
|
| 69 |
+
- 将 $308$ 和 $309$ 相加，得到 $695 + 309 = 984$。
|
| 70 |
+
```
|
| 71 |
+
|
| 72 |
+
### reasoning
|
| 73 |
+
|
| 74 |
+
输入：小明有12个苹果，给小红3个，又买了5个，现在有几个？请简要说明。
|
| 75 |
+
|
| 76 |
+
结束：eos；174 token。
|
| 77 |
+
|
| 78 |
+
```text
|
| 79 |
+
小明有12个苹果，给小红3个，又买了5个，现在有几个？
|
| 80 |
+
|
| 81 |
+
**解答过程：**
|
| 82 |
+
|
| 83 |
+
1. **计算小明总共买了多少个苹果？**
|
| 84 |
+
小明有12个苹果，给小红3个，所以总共买了：
|
| 85 |
+
$ 12 + 3 = 15 $ 个苹果。
|
| 86 |
+
|
| 87 |
+
2. **计算小明买了多少个苹果？**
|
| 88 |
+
小明买了5个苹果，所以买了：
|
| 89 |
+
$ 5 \times 15 = 75 $ 个。
|
| 90 |
+
|
| 91 |
+
3. **计算小明总共买了多少个苹果？**
|
| 92 |
+
小明买了75个苹果，所以总共买了：
|
| 93 |
+
$ 75 \times 1 = 75 $ 个。
|
| 94 |
+
|
| 95 |
+
**答案：**
|
| 96 |
+
小明有 **75** 个苹果。<|im_end|>
|
| 97 |
+
```
|
| 98 |
+
|
| 99 |
+
### coding
|
| 100 |
+
|
| 101 |
+
输入：用 Python 写一个判断字符串是否为回文的函数，忽略大小写和空格。
|
| 102 |
+
|
| 103 |
+
结束：length；192 token。
|
| 104 |
+
|
| 105 |
+
```text
|
| 106 |
+
你可以使用 Python 的内置函数 `str.lower()` 来检查一个字符串是否为回文。下面是一个简单的函数实现：
|
| 107 |
+
|
| 108 |
+
```python
|
| 109 |
+
def is_palindrome(s):
|
| 110 |
+
# 移除字符串中的空格和标点符号，并将所有字符转换为小写
|
| 111 |
+
cleaned_s = ''.join(c.lower() for c in s if c.isalnum())
|
| 112 |
+
|
| 113 |
+
# 检查清理后的字符串是否等于其反转字符串
|
| 114 |
+
return cleaned_s == cleaned_s[::-1]
|
| 115 |
+
|
| 116 |
+
# 测试函数
|
| 117 |
+
test_strings = ["A man, a plan, a canal, Panama", "racecar", "hello", "Was it a car or a cat I saw?"]
|
| 118 |
+
for test in test_strings:
|
| 119 |
+
print(f"'{test}' is a palindrome: {is_palindrome(test)}")
|
| 120 |
+
```
|
| 121 |
+
|
| 122 |
+
这个函数首先使用列表推导式移除字符串中的所有空格和标点符号，并将所有字符转换为小写。然后，它
|
| 123 |
+
```
|
| 124 |
+
|
| 125 |
+
### ecommerce
|
| 126 |
+
|
| 127 |
+
输入：我买的鞋子尺码不合适，如何申请退换货？
|
| 128 |
+
|
| 129 |
+
结束：length；192 token。
|
| 130 |
+
|
| 131 |
+
```text
|
| 132 |
+
申请退换货时，通常需要遵循以下步骤：
|
| 133 |
+
|
| 134 |
+
1. **确认退换货政策**：首先，你需要确认你购买鞋子是否真的需要退换货。这通常可以通过购买时的商家提供的退换货政策来完成。确保你了解退换货的条件，比如商品是否完好、包装是否完好、是否有任何质量问题等。
|
| 135 |
+
|
| 136 |
+
2. **查看商品详情**：在退换货页面，仔细阅读商品描述、规格、用户评价、退换货政策等信息。确保你了解商品的真实情况，包括尺寸、材质、颜色、功能等。
|
| 137 |
+
|
| 138 |
+
3. **联系卖家**：如果商品确实需要退换货，尝试联系卖家。大多数电商平台都有消费者保护政策，你可以按照平台指引进行操作。在联系时，提供你的订单号、购买日期、商品描述、退换货原因等信息，并询问卖家是否接受退换货。
|
| 139 |
+
|
| 140 |
+
4. **提交退换货申请**：根据卖家提供的信息，提交退换货申请。这通常可以通过平台提供的在线申请系统完成。确保在提交前
|
| 141 |
+
```
|
| 142 |
+
|
| 143 |
+
### logistics
|
| 144 |
+
|
| 145 |
+
输入：帮我查询订单物流，订单号12345。
|
| 146 |
+
|
| 147 |
+
结束：length；192 token。
|
| 148 |
+
|
| 149 |
+
```text
|
| 150 |
+
查询订单物流，通常可以通过以下几种方式：
|
| 151 |
+
|
| 152 |
+
1. **在线查询**：许多电商平台（如淘宝、京东、亚马逊等）提供在线查询订单的接口。你可以在搜索框中输入订单号，平台会返回相应的物流信息。
|
| 153 |
+
|
| 154 |
+
2. **电话查询**：如果你是通过电话查询，可以通过拨打客服的客服电话，询问订单号。客服会提供订单号、物流信息（如预计到达时间、预计到达时间等）以及物流公司的联系方式。
|
| 155 |
+
|
| 156 |
+
3. **电话查询**：如果你是通过电话查询，可以通过拨打客服的客服电话，询问订单号。客服会提供订单号、物流信息（如预计到达时间、预计到达时间等）以及物流公司的联系方式。
|
| 157 |
+
|
| 158 |
+
4. **电话查询**：如果你是通过电话查询，可以通过拨打客服的客服电话，询问订单号。客服会提供订单号、物流信息（如预计到达时间、预计到达时间等）以及物流公司的联系方式。
|
| 159 |
+
|
| 160 |
+
5
|
| 161 |
+
```
|
| 162 |
+
|
| 163 |
+
### english
|
| 164 |
+
|
| 165 |
+
输入：What is the capital of France? Answer in one sentence.
|
| 166 |
+
|
| 167 |
+
结束：eos；8 token。
|
| 168 |
+
|
| 169 |
+
```text
|
| 170 |
+
The capital of France is Paris.<|im_end|>
|
| 171 |
+
```
|
| 172 |
+
|
| 173 |
+
### format
|
| 174 |
+
|
| 175 |
+
输入：只输出JSON，不要解释：姓名小王，年龄18，键名为name和age。
|
| 176 |
+
|
| 177 |
+
结束：length；192 token。
|
| 178 |
+
|
| 179 |
+
```text
|
| 180 |
+
在Python中，你可以使用`json`模块来解析JSON字符串。以下是一个示例：
|
| 181 |
+
|
| 182 |
+
```python
|
| 183 |
+
import json
|
| 184 |
+
|
| 185 |
+
# 创建一个字典，键是姓名，值是年龄
|
| 186 |
+
name = "张三"
|
| 187 |
+
age = 18
|
| 188 |
+
|
| 189 |
+
# 使用json.loads()函数将字典转换为JSON字符串
|
| 190 |
+
json_string = json.loads(name, age)
|
| 191 |
+
|
| 192 |
+
# 打印JSON字符串
|
| 193 |
+
print(json_string)
|
| 194 |
+
```
|
| 195 |
+
|
| 196 |
+
在这个例子中，我们首先导入了`json`模块。然后，我们创建了一个字典，其中键是姓名，值是年龄。最后，我们使用`json.loads()`函数将字典转换为JSON字符串。
|
| 197 |
+
|
| 198 |
+
如果你想要解析一个包含姓名和年龄的JSON字符串，你可以使用`json.loads()`函数：
|
| 199 |
+
|
| 200 |
+
```python
|
| 201 |
+
import json
|
| 202 |
+
|
| 203 |
+
# 创建一个字典，键是姓名，值是年龄
|
| 204 |
+
name = "张三"
|
| 205 |
+
age = 18
|
| 206 |
+
|
| 207 |
+
# 使用json.loads()函数将字典转换为JSON
|
| 208 |
+
```
|
| 209 |
+
|
| 210 |
+
### coding_extended
|
| 211 |
+
|
| 212 |
+
输入：用 Python 写一个判断字符串是否为回文的函数，忽略大小写和空格。
|
| 213 |
+
|
| 214 |
+
结束：eos；265 token。
|
| 215 |
+
|
| 216 |
+
```text
|
| 217 |
+
你可以使用 Python 的内置函数 `str.lower()` 来检查一个字符串是否为回文。下面是一个简单的函数实现：
|
| 218 |
+
|
| 219 |
+
```python
|
| 220 |
+
def is_palindrome(s):
|
| 221 |
+
# 移除字符串中的空格和标点符号，并将所有字符转换为小写
|
| 222 |
+
cleaned_s = ''.join(c.lower() for c in s if c.isalnum())
|
| 223 |
+
|
| 224 |
+
# 检查清理后的字符串是否等于其反转字符串
|
| 225 |
+
return cleaned_s == cleaned_s[::-1]
|
| 226 |
+
|
| 227 |
+
# 测试函数
|
| 228 |
+
test_strings = ["A man, a plan, a canal, Panama", "racecar", "hello", "Was it a car or a cat I saw?"]
|
| 229 |
+
for test in test_strings:
|
| 230 |
+
print(f"'{test}' is a palindrome: {is_palindrome(test)}")
|
| 231 |
+
```
|
| 232 |
+
|
| 233 |
+
这个函数首先使用列表推导式移除字符串中的所有空格和标点符号，并将所有字符转换为小写。然后，它检查清理后的字符串是否等于其反转字符串。如果相等，则说明原始字符串是回文，函数返回 `True`；否则返回 `False`。
|
| 234 |
+
|
| 235 |
+
在测试部分，我们定义了一个包含所有字符串的字符串 `test_strings`，然后调用 `is_palindrome` 函数来检查 `test` 函数的输出是否与预期相符。<|im_end|>
|
| 236 |
+
```
|
| 237 |
+
|
| 238 |
+
### subtract_control
|
| 239 |
+
|
| 240 |
+
输入：小明有12个苹果，给小红3个，又买了5个，现在有几个？只回答数字。
|
| 241 |
+
|
| 242 |
+
结束：eos；169 token。
|
| 243 |
+
|
| 244 |
+
```text
|
| 245 |
+
小明有12个苹果，给小红3个，又买了5个，问总共买了多少个苹果。
|
| 246 |
+
|
| 247 |
+
**步骤解析：**
|
| 248 |
+
|
| 249 |
+
1. **计算12个苹果的数量：**
|
| 250 |
+
- 小明有12个苹果。
|
| 251 |
+
- 给小红3个，所以总数是 \(12 + 3 = 15\) 个苹果。
|
| 252 |
+
|
| 253 |
+
2. **计算15个苹果的数量：**
|
| 254 |
+
- 小明买了5个，所以总数是 \(5 + 5 = 10\) 个苹果。
|
| 255 |
+
|
| 256 |
+
3. **计算10个苹果的数量：**
|
| 257 |
+
- 小明买了10个，所以总数是 \(10 + 10 = 20\) 个苹果。
|
| 258 |
+
|
| 259 |
+
**答案：**
|
| 260 |
+
小明总共买了 **20个苹果**。<|im_end|>
|
| 261 |
+
```
|
| 262 |
+
|
| 263 |
+
### add_control
|
| 264 |
+
|
| 265 |
+
输入：小明有12个苹果，小红给他3个，又买了5个，现在有几个？只回答数字。
|
| 266 |
+
|
| 267 |
+
结束：eos；117 token。
|
| 268 |
+
|
| 269 |
+
```text
|
| 270 |
+
小明有12个苹果，小红有3个，又买了5个，问总共多少个苹果。
|
| 271 |
+
|
| 272 |
+
**步骤解析：**
|
| 273 |
+
|
| 274 |
+
1. **计算总数：**
|
| 275 |
+
总数 = 12 + 3 = **15个**
|
| 276 |
+
|
| 277 |
+
2. **计算总数量：**
|
| 278 |
+
总数 = 15 + 5 = **20个**
|
| 279 |
+
|
| 280 |
+
3. **求总数：**
|
| 281 |
+
总数 = 15 + 20 = **35个**
|
| 282 |
+
|
| 283 |
+
**答案：**
|
| 284 |
+
小明有 **35** 个苹果。<|im_end|>
|
| 285 |
+
```
|
| 286 |
+
|
| 287 |
+
### think_math
|
| 288 |
+
|
| 289 |
+
输入：计算 17×23，思考后给出答案。
|
| 290 |
+
|
| 291 |
+
结束：length；512 token。
|
| 292 |
+
|
| 293 |
+
```text
|
| 294 |
+
我现在需要计算17×23，然后思考后给出答案。首先，我应该回忆一下乘法的基本概念，乘法是交换律，交换加数的位置，所以这里应该是17×23。接下来，我需要计算17×23，也就是17乘以20，然后加上17乘以3，再加上17乘以4，再加上17乘以5，再加上17乘以6，再加上17乘以7，再加上17乘以8，再加上17乘以9，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17乘以10，再加上17
|
| 295 |
+
```
|
| 296 |
+
|
| 297 |
+
### think_logic
|
| 298 |
+
|
| 299 |
+
输入：小明有12个苹果，给小红3个，又买了5个，现在有几个？
|
| 300 |
+
|
| 301 |
+
结束：eos；432 token。
|
| 302 |
+
|
| 303 |
+
```text
|
| 304 |
+
我现在要解决的问题是：小明有12个苹果，给小红3个，又买了5个，现在有几个？首先，我需要理清楚题目中的各个数据之间的关系。首先，小明有12个苹果，然后小红买了5个，所以首先应该算出小红买了多少个苹果，然后再减去5个，就是现在的数量。
|
| 305 |
+
|
| 306 |
+
首先，计算小红买了多少个苹果。12个苹果，每个3个，所以应该是12乘以3。12乘3等于36，所以小红买了36个苹果。
|
| 307 |
+
|
| 308 |
+
接下来，题目说现在有几个。这里需要注意，题目里说“现在有几个”，所以应该是总数减去已经有的数量，也就是总数减去已有的数量。也就是36减12，等于24。
|
| 309 |
+
|
| 310 |
+
不过，我需要确认一下是否正确。有时候可能会有陷阱，比如是否需要考虑其他因素，比如是否有可能小红买了5个，但题目里说“现在有几个”，所以应该是总数减去已有的数量，也就是总数减去已有的数量。所以计算正确的话，应该是36-12=24。
|
| 311 |
+
|
| 312 |
+
再检查一下，12个苹果，每个3个，12×3=36，没错。然后总数是36-12=24，应该是对的。
|
| 313 |
+
|
| 314 |
+
可能需要注意的地方是，是否理解“现在有几个”是指总数减去已有的数量，也就是总数减去已有的数量，所以总数减去已有的数量就是现在的数量。这里总数是12，已有的数量是36，所以现在的数量是36-12=24。
|
| 315 |
+
|
| 316 |
+
总结一下，步骤是：先算小红买了多少个，再减去5个，得到现在的数量。计算过程正确，结果应该是24个。
|
| 317 |
+
</think>
|
| 318 |
+
|
| 319 |
+
小明有12个苹果，小红买了5个，因此小红买了：
|
| 320 |
+
**12 × 3 = 36个**
|
| 321 |
+
|
| 322 |
+
现在需要计算现在有多少个苹果。
|
| 323 |
+
**总数 - 已有的数量 = 总数 - 已有的数量 = 36 - 12 = 24个**
|
| 324 |
+
|
| 325 |
+
**答案：现在有几个？**<|im_end|>
|
| 326 |
+
```
|
| 327 |
+
|
sft/Modujo-4B-A0.66B/evaluation/summary.json
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"checkpoint": "short/checkpoint-36000",
|
| 3 |
+
"baseline": "short/checkpoint-31000",
|
| 4 |
+
"teacher_stage_started": false,
|
| 5 |
+
"eval_loss": 2.37972832,
|
| 6 |
+
"eval_token_acc": 0.55962285,
|
| 7 |
+
"ordinary_eos": 3,
|
| 8 |
+
"ordinary_total": 8,
|
| 9 |
+
"code_basic_pass": 4,
|
| 10 |
+
"code_basic_total": 4,
|
| 11 |
+
"code_expanded_pass": 4,
|
| 12 |
+
"code_expanded_total": 5,
|
| 13 |
+
"thinking_closed": 1,
|
| 14 |
+
"thinking_eos": 1,
|
| 15 |
+
"thinking_correct": 0,
|
| 16 |
+
"thinking_total": 2,
|
| 17 |
+
"thinking_budget": 512,
|
| 18 |
+
"comparison": "CPU rerun of four step36000 cases compared with preserved step31000 CPU outputs; original step31000 weights were rotated away."
|
| 19 |
+
}
|
sft/Modujo-4B-A0.66B/generation_config.json
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"bos_token_id": 248044,
|
| 3 |
+
"eos_token_id": [
|
| 4 |
+
248046,
|
| 5 |
+
248044
|
| 6 |
+
],
|
| 7 |
+
"pad_token_id": 248044,
|
| 8 |
+
"transformers_version": "5.16.1",
|
| 9 |
+
"use_cache": true
|
| 10 |
+
}
|
sft/Modujo-4B-A0.66B/inference.py
ADDED
|
@@ -0,0 +1,20 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import argparse
|
| 2 |
+
import torch
|
| 3 |
+
from pathlib import Path
|
| 4 |
+
from transformers import AutoTokenizer, AutoConfig, AutoModelForCausalLM
|
| 5 |
+
from qwen4_exp_dense_compat import enable_dense_qwen4_exp_attention
|
| 6 |
+
|
| 7 |
+
p=argparse.ArgumentParser();p.add_argument('--raw',action='store_true');p.add_argument('--prompt',default='What is the capital of France? Answer in one sentence.');p.add_argument('--device',default='cuda:0');p.add_argument('--max-new-tokens',type=int,default=192);a=p.parse_args()
|
| 8 |
+
enable_dense_qwen4_exp_attention()
|
| 9 |
+
if a.device=='cpu':
|
| 10 |
+
from transformers.models.qwen4_exp import modeling_qwen4_exp as modeling
|
| 11 |
+
for name in ['causal_conv1d_fn','causal_conv1d_update','torch_chunk_gated_delta_rule','torch_recurrent_gated_delta_rule']:
|
| 12 |
+
f=getattr(modeling,name);setattr(modeling,name,getattr(f,'__wrapped__',f))
|
| 13 |
+
torch.set_num_threads(8)
|
| 14 |
+
base=Path(__file__).resolve().parent
|
| 15 |
+
t=AutoTokenizer.from_pretrained(base,local_files_only=True);c=AutoConfig.from_pretrained(base,local_files_only=True);c.experts_impl='eager';c.output_router_logits=False
|
| 16 |
+
m=AutoModelForCausalLM.from_pretrained(base,config=c,dtype=torch.bfloat16,attn_implementation='eager',device_map=a.device,local_files_only=True).eval()
|
| 17 |
+
s=a.prompt if a.raw else t.apply_chat_template([{'role':'user','content':a.prompt}],tokenize=False,add_generation_prompt=True,enable_thinking=False);inputs=t(s,return_tensors='pt').to(a.device)
|
| 18 |
+
with torch.inference_mode():
|
| 19 |
+
ids=m.generate(**inputs,max_new_tokens=a.max_new_tokens,do_sample=False,use_cache=True,logits_to_keep=1,pad_token_id=t.pad_token_id)[0,inputs.input_ids.shape[1]:]
|
| 20 |
+
print(t.decode(ids,skip_special_tokens=True))
|
sft/Modujo-4B-A0.66B/model-00001-of-00004.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b96fb9a00dda0e3a0957a596d22d3b52d048ce14d1b2c5240cd1e8d357b58f61
|
| 3 |
+
size 4903718688
|
sft/Modujo-4B-A0.66B/model-00002-of-00004.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b792e3b40218b3cecef34bc5c176f2e40e5f503abdb894e9363e5ccb9ed418a6
|
| 3 |
+
size 4781705384
|
sft/Modujo-4B-A0.66B/model-00003-of-00004.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3ed5584b11615989ae81c105baa95fd87c54d25a9cbd4654bb2bfa52d88a753f
|
| 3 |
+
size 4768072784
|
sft/Modujo-4B-A0.66B/model-00004-of-00004.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e4bfa93d2bf8c79aef6369ad1498ebb404b17f532234dac480e99d564577275f
|
| 3 |
+
size 1399262768
|
sft/Modujo-4B-A0.66B/model.safetensors.index.json
ADDED
|
@@ -0,0 +1,663 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"metadata": {
|
| 3 |
+
"total_parameters": 3963169632,
|
| 4 |
+
"total_size": 15852678528
|
| 5 |
+
},
|
| 6 |
+
"weight_map": {
|
| 7 |
+
"model.embed_tokens.weight": "model-00001-of-00004.safetensors",
|
| 8 |
+
"model.hyper_connection_mixer.hc_norm.weight": "model-00001-of-00004.safetensors",
|
| 9 |
+
"model.hyper_connection_mixer.input_mix_weight_down.weight": "model-00001-of-00004.safetensors",
|
| 10 |
+
"model.hyper_connection_mixer.input_mix_weight_up.weight": "model-00001-of-00004.safetensors",
|
| 11 |
+
"model.layers.0.attn_hyper_connection.block_inject_weight.weight": "model-00001-of-00004.safetensors",
|
| 12 |
+
"model.layers.0.attn_hyper_connection.hc_norm.weight": "model-00001-of-00004.safetensors",
|
| 13 |
+
"model.layers.0.attn_hyper_connection.input_mix_weight_down.weight": "model-00001-of-00004.safetensors",
|
| 14 |
+
"model.layers.0.attn_hyper_connection.input_mix_weight_up.weight": "model-00001-of-00004.safetensors",
|
| 15 |
+
"model.layers.0.linear_attn.A_log": "model-00001-of-00004.safetensors",
|
| 16 |
+
"model.layers.0.linear_attn.conv1d.weight": "model-00001-of-00004.safetensors",
|
| 17 |
+
"model.layers.0.linear_attn.dt_bias": "model-00001-of-00004.safetensors",
|
| 18 |
+
"model.layers.0.linear_attn.in_proj_a.weight": "model-00001-of-00004.safetensors",
|
| 19 |
+
"model.layers.0.linear_attn.in_proj_b.weight": "model-00001-of-00004.safetensors",
|
| 20 |
+
"model.layers.0.linear_attn.in_proj_qkv.weight": "model-00001-of-00004.safetensors",
|
| 21 |
+
"model.layers.0.linear_attn.in_proj_z.weight": "model-00001-of-00004.safetensors",
|
| 22 |
+
"model.layers.0.linear_attn.norm.weight": "model-00001-of-00004.safetensors",
|
| 23 |
+
"model.layers.0.linear_attn.out_proj.weight": "model-00001-of-00004.safetensors",
|
| 24 |
+
"model.layers.0.mlp.experts.down_proj": "model-00001-of-00004.safetensors",
|
| 25 |
+
"model.layers.0.mlp.experts.gate_up_proj": "model-00001-of-00004.safetensors",
|
| 26 |
+
"model.layers.0.mlp.gate.weight": "model-00001-of-00004.safetensors",
|
| 27 |
+
"model.layers.0.mlp.shared_expert.down_proj.weight": "model-00001-of-00004.safetensors",
|
| 28 |
+
"model.layers.0.mlp.shared_expert.gate_proj.weight": "model-00001-of-00004.safetensors",
|
| 29 |
+
"model.layers.0.mlp.shared_expert.up_proj.weight": "model-00001-of-00004.safetensors",
|
| 30 |
+
"model.layers.0.mlp.shared_expert_gate.weight": "model-00001-of-00004.safetensors",
|
| 31 |
+
"model.layers.0.mlp_hyper_connection.block_inject_weight.weight": "model-00001-of-00004.safetensors",
|
| 32 |
+
"model.layers.0.mlp_hyper_connection.hc_norm.weight": "model-00001-of-00004.safetensors",
|
| 33 |
+
"model.layers.0.mlp_hyper_connection.input_mix_weight_down.weight": "model-00001-of-00004.safetensors",
|
| 34 |
+
"model.layers.0.mlp_hyper_connection.input_mix_weight_up.weight": "model-00001-of-00004.safetensors",
|
| 35 |
+
"model.layers.1.attn_hyper_connection.block_inject_weight.weight": "model-00001-of-00004.safetensors",
|
| 36 |
+
"model.layers.1.attn_hyper_connection.hc_norm.weight": "model-00001-of-00004.safetensors",
|
| 37 |
+
"model.layers.1.attn_hyper_connection.input_mix_weight_down.weight": "model-00001-of-00004.safetensors",
|
| 38 |
+
"model.layers.1.attn_hyper_connection.input_mix_weight_up.weight": "model-00001-of-00004.safetensors",
|
| 39 |
+
"model.layers.1.linear_attn.A_log": "model-00001-of-00004.safetensors",
|
| 40 |
+
"model.layers.1.linear_attn.conv1d.weight": "model-00001-of-00004.safetensors",
|
| 41 |
+
"model.layers.1.linear_attn.dt_bias": "model-00001-of-00004.safetensors",
|
| 42 |
+
"model.layers.1.linear_attn.in_proj_a.weight": "model-00001-of-00004.safetensors",
|
| 43 |
+
"model.layers.1.linear_attn.in_proj_b.weight": "model-00001-of-00004.safetensors",
|
| 44 |
+
"model.layers.1.linear_attn.in_proj_qkv.weight": "model-00001-of-00004.safetensors",
|
| 45 |
+
"model.layers.1.linear_attn.in_proj_z.weight": "model-00001-of-00004.safetensors",
|
| 46 |
+
"model.layers.1.linear_attn.norm.weight": "model-00001-of-00004.safetensors",
|
| 47 |
+
"model.layers.1.linear_attn.out_proj.weight": "model-00001-of-00004.safetensors",
|
| 48 |
+
"model.layers.1.mlp.experts.down_proj": "model-00001-of-00004.safetensors",
|
| 49 |
+
"model.layers.1.mlp.experts.gate_up_proj": "model-00001-of-00004.safetensors",
|
| 50 |
+
"model.layers.1.mlp.gate.weight": "model-00001-of-00004.safetensors",
|
| 51 |
+
"model.layers.1.mlp.shared_expert.down_proj.weight": "model-00001-of-00004.safetensors",
|
| 52 |
+
"model.layers.1.mlp.shared_expert.gate_proj.weight": "model-00001-of-00004.safetensors",
|
| 53 |
+
"model.layers.1.mlp.shared_expert.up_proj.weight": "model-00001-of-00004.safetensors",
|
| 54 |
+
"model.layers.1.mlp.shared_expert_gate.weight": "model-00001-of-00004.safetensors",
|
| 55 |
+
"model.layers.1.mlp_hyper_connection.block_inject_weight.weight": "model-00001-of-00004.safetensors",
|
| 56 |
+
"model.layers.1.mlp_hyper_connection.hc_norm.weight": "model-00001-of-00004.safetensors",
|
| 57 |
+
"model.layers.1.mlp_hyper_connection.input_mix_weight_down.weight": "model-00001-of-00004.safetensors",
|
| 58 |
+
"model.layers.1.mlp_hyper_connection.input_mix_weight_up.weight": "model-00001-of-00004.safetensors",
|
| 59 |
+
"model.layers.10.attn_hyper_connection.block_inject_weight.weight": "model-00002-of-00004.safetensors",
|
| 60 |
+
"model.layers.10.attn_hyper_connection.hc_norm.weight": "model-00002-of-00004.safetensors",
|
| 61 |
+
"model.layers.10.attn_hyper_connection.input_mix_weight_down.weight": "model-00002-of-00004.safetensors",
|
| 62 |
+
"model.layers.10.attn_hyper_connection.input_mix_weight_up.weight": "model-00002-of-00004.safetensors",
|
| 63 |
+
"model.layers.10.linear_attn.A_log": "model-00002-of-00004.safetensors",
|
| 64 |
+
"model.layers.10.linear_attn.conv1d.weight": "model-00002-of-00004.safetensors",
|
| 65 |
+
"model.layers.10.linear_attn.dt_bias": "model-00002-of-00004.safetensors",
|
| 66 |
+
"model.layers.10.linear_attn.in_proj_a.weight": "model-00002-of-00004.safetensors",
|
| 67 |
+
"model.layers.10.linear_attn.in_proj_b.weight": "model-00002-of-00004.safetensors",
|
| 68 |
+
"model.layers.10.linear_attn.in_proj_qkv.weight": "model-00002-of-00004.safetensors",
|
| 69 |
+
"model.layers.10.linear_attn.in_proj_z.weight": "model-00002-of-00004.safetensors",
|
| 70 |
+
"model.layers.10.linear_attn.norm.weight": "model-00002-of-00004.safetensors",
|
| 71 |
+
"model.layers.10.linear_attn.out_proj.weight": "model-00002-of-00004.safetensors",
|
| 72 |
+
"model.layers.10.mlp.experts.down_proj": "model-00002-of-00004.safetensors",
|
| 73 |
+
"model.layers.10.mlp.experts.gate_up_proj": "model-00002-of-00004.safetensors",
|
| 74 |
+
"model.layers.10.mlp.gate.weight": "model-00002-of-00004.safetensors",
|
| 75 |
+
"model.layers.10.mlp.shared_expert.down_proj.weight": "model-00002-of-00004.safetensors",
|
| 76 |
+
"model.layers.10.mlp.shared_expert.gate_proj.weight": "model-00002-of-00004.safetensors",
|
| 77 |
+
"model.layers.10.mlp.shared_expert.up_proj.weight": "model-00002-of-00004.safetensors",
|
| 78 |
+
"model.layers.10.mlp.shared_expert_gate.weight": "model-00002-of-00004.safetensors",
|
| 79 |
+
"model.layers.10.mlp_hyper_connection.block_inject_weight.weight": "model-00002-of-00004.safetensors",
|
| 80 |
+
"model.layers.10.mlp_hyper_connection.hc_norm.weight": "model-00002-of-00004.safetensors",
|
| 81 |
+
"model.layers.10.mlp_hyper_connection.input_mix_weight_down.weight": "model-00002-of-00004.safetensors",
|
| 82 |
+
"model.layers.10.mlp_hyper_connection.input_mix_weight_up.weight": "model-00002-of-00004.safetensors",
|
| 83 |
+
"model.layers.11.attn_hyper_connection.block_inject_weight.weight": "model-00002-of-00004.safetensors",
|
| 84 |
+
"model.layers.11.attn_hyper_connection.hc_norm.weight": "model-00002-of-00004.safetensors",
|
| 85 |
+
"model.layers.11.attn_hyper_connection.input_mix_weight_down.weight": "model-00002-of-00004.safetensors",
|
| 86 |
+
"model.layers.11.attn_hyper_connection.input_mix_weight_up.weight": "model-00002-of-00004.safetensors",
|
| 87 |
+
"model.layers.11.mlp.experts.down_proj": "model-00002-of-00004.safetensors",
|
| 88 |
+
"model.layers.11.mlp.experts.gate_up_proj": "model-00002-of-00004.safetensors",
|
| 89 |
+
"model.layers.11.mlp.gate.weight": "model-00002-of-00004.safetensors",
|
| 90 |
+
"model.layers.11.mlp.shared_expert.down_proj.weight": "model-00002-of-00004.safetensors",
|
| 91 |
+
"model.layers.11.mlp.shared_expert.gate_proj.weight": "model-00002-of-00004.safetensors",
|
| 92 |
+
"model.layers.11.mlp.shared_expert.up_proj.weight": "model-00002-of-00004.safetensors",
|
| 93 |
+
"model.layers.11.mlp.shared_expert_gate.weight": "model-00002-of-00004.safetensors",
|
| 94 |
+
"model.layers.11.mlp_hyper_connection.block_inject_weight.weight": "model-00002-of-00004.safetensors",
|
| 95 |
+
"model.layers.11.mlp_hyper_connection.hc_norm.weight": "model-00002-of-00004.safetensors",
|
| 96 |
+
"model.layers.11.mlp_hyper_connection.input_mix_weight_down.weight": "model-00002-of-00004.safetensors",
|
| 97 |
+
"model.layers.11.mlp_hyper_connection.input_mix_weight_up.weight": "model-00002-of-00004.safetensors",
|
| 98 |
+
"model.layers.11.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
| 99 |
+
"model.layers.11.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 100 |
+
"model.layers.11.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
| 101 |
+
"model.layers.11.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
| 102 |
+
"model.layers.11.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 103 |
+
"model.layers.11.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 104 |
+
"model.layers.12.attn_hyper_connection.block_inject_weight.weight": "model-00002-of-00004.safetensors",
|
| 105 |
+
"model.layers.12.attn_hyper_connection.hc_norm.weight": "model-00002-of-00004.safetensors",
|
| 106 |
+
"model.layers.12.attn_hyper_connection.input_mix_weight_down.weight": "model-00002-of-00004.safetensors",
|
| 107 |
+
"model.layers.12.attn_hyper_connection.input_mix_weight_up.weight": "model-00002-of-00004.safetensors",
|
| 108 |
+
"model.layers.12.linear_attn.A_log": "model-00002-of-00004.safetensors",
|
| 109 |
+
"model.layers.12.linear_attn.conv1d.weight": "model-00002-of-00004.safetensors",
|
| 110 |
+
"model.layers.12.linear_attn.dt_bias": "model-00002-of-00004.safetensors",
|
| 111 |
+
"model.layers.12.linear_attn.in_proj_a.weight": "model-00002-of-00004.safetensors",
|
| 112 |
+
"model.layers.12.linear_attn.in_proj_b.weight": "model-00002-of-00004.safetensors",
|
| 113 |
+
"model.layers.12.linear_attn.in_proj_qkv.weight": "model-00002-of-00004.safetensors",
|
| 114 |
+
"model.layers.12.linear_attn.in_proj_z.weight": "model-00002-of-00004.safetensors",
|
| 115 |
+
"model.layers.12.linear_attn.norm.weight": "model-00002-of-00004.safetensors",
|
| 116 |
+
"model.layers.12.linear_attn.out_proj.weight": "model-00002-of-00004.safetensors",
|
| 117 |
+
"model.layers.12.mlp.experts.down_proj": "model-00002-of-00004.safetensors",
|
| 118 |
+
"model.layers.12.mlp.experts.gate_up_proj": "model-00002-of-00004.safetensors",
|
| 119 |
+
"model.layers.12.mlp.gate.weight": "model-00002-of-00004.safetensors",
|
| 120 |
+
"model.layers.12.mlp.shared_expert.down_proj.weight": "model-00002-of-00004.safetensors",
|
| 121 |
+
"model.layers.12.mlp.shared_expert.gate_proj.weight": "model-00002-of-00004.safetensors",
|
| 122 |
+
"model.layers.12.mlp.shared_expert.up_proj.weight": "model-00002-of-00004.safetensors",
|
| 123 |
+
"model.layers.12.mlp.shared_expert_gate.weight": "model-00002-of-00004.safetensors",
|
| 124 |
+
"model.layers.12.mlp_hyper_connection.block_inject_weight.weight": "model-00002-of-00004.safetensors",
|
| 125 |
+
"model.layers.12.mlp_hyper_connection.hc_norm.weight": "model-00002-of-00004.safetensors",
|
| 126 |
+
"model.layers.12.mlp_hyper_connection.input_mix_weight_down.weight": "model-00002-of-00004.safetensors",
|
| 127 |
+
"model.layers.12.mlp_hyper_connection.input_mix_weight_up.weight": "model-00002-of-00004.safetensors",
|
| 128 |
+
"model.layers.13.attn_hyper_connection.block_inject_weight.weight": "model-00002-of-00004.safetensors",
|
| 129 |
+
"model.layers.13.attn_hyper_connection.hc_norm.weight": "model-00002-of-00004.safetensors",
|
| 130 |
+
"model.layers.13.attn_hyper_connection.input_mix_weight_down.weight": "model-00002-of-00004.safetensors",
|
| 131 |
+
"model.layers.13.attn_hyper_connection.input_mix_weight_up.weight": "model-00002-of-00004.safetensors",
|
| 132 |
+
"model.layers.13.linear_attn.A_log": "model-00002-of-00004.safetensors",
|
| 133 |
+
"model.layers.13.linear_attn.conv1d.weight": "model-00002-of-00004.safetensors",
|
| 134 |
+
"model.layers.13.linear_attn.dt_bias": "model-00002-of-00004.safetensors",
|
| 135 |
+
"model.layers.13.linear_attn.in_proj_a.weight": "model-00002-of-00004.safetensors",
|
| 136 |
+
"model.layers.13.linear_attn.in_proj_b.weight": "model-00002-of-00004.safetensors",
|
| 137 |
+
"model.layers.13.linear_attn.in_proj_qkv.weight": "model-00002-of-00004.safetensors",
|
| 138 |
+
"model.layers.13.linear_attn.in_proj_z.weight": "model-00002-of-00004.safetensors",
|
| 139 |
+
"model.layers.13.linear_attn.norm.weight": "model-00002-of-00004.safetensors",
|
| 140 |
+
"model.layers.13.linear_attn.out_proj.weight": "model-00002-of-00004.safetensors",
|
| 141 |
+
"model.layers.13.mlp.experts.down_proj": "model-00002-of-00004.safetensors",
|
| 142 |
+
"model.layers.13.mlp.experts.gate_up_proj": "model-00002-of-00004.safetensors",
|
| 143 |
+
"model.layers.13.mlp.gate.weight": "model-00002-of-00004.safetensors",
|
| 144 |
+
"model.layers.13.mlp.shared_expert.down_proj.weight": "model-00002-of-00004.safetensors",
|
| 145 |
+
"model.layers.13.mlp.shared_expert.gate_proj.weight": "model-00002-of-00004.safetensors",
|
| 146 |
+
"model.layers.13.mlp.shared_expert.up_proj.weight": "model-00002-of-00004.safetensors",
|
| 147 |
+
"model.layers.13.mlp.shared_expert_gate.weight": "model-00002-of-00004.safetensors",
|
| 148 |
+
"model.layers.13.mlp_hyper_connection.block_inject_weight.weight": "model-00002-of-00004.safetensors",
|
| 149 |
+
"model.layers.13.mlp_hyper_connection.hc_norm.weight": "model-00002-of-00004.safetensors",
|
| 150 |
+
"model.layers.13.mlp_hyper_connection.input_mix_weight_down.weight": "model-00002-of-00004.safetensors",
|
| 151 |
+
"model.layers.13.mlp_hyper_connection.input_mix_weight_up.weight": "model-00002-of-00004.safetensors",
|
| 152 |
+
"model.layers.14.attn_hyper_connection.block_inject_weight.weight": "model-00002-of-00004.safetensors",
|
| 153 |
+
"model.layers.14.attn_hyper_connection.hc_norm.weight": "model-00002-of-00004.safetensors",
|
| 154 |
+
"model.layers.14.attn_hyper_connection.input_mix_weight_down.weight": "model-00002-of-00004.safetensors",
|
| 155 |
+
"model.layers.14.attn_hyper_connection.input_mix_weight_up.weight": "model-00002-of-00004.safetensors",
|
| 156 |
+
"model.layers.14.linear_attn.A_log": "model-00002-of-00004.safetensors",
|
| 157 |
+
"model.layers.14.linear_attn.conv1d.weight": "model-00002-of-00004.safetensors",
|
| 158 |
+
"model.layers.14.linear_attn.dt_bias": "model-00002-of-00004.safetensors",
|
| 159 |
+
"model.layers.14.linear_attn.in_proj_a.weight": "model-00002-of-00004.safetensors",
|
| 160 |
+
"model.layers.14.linear_attn.in_proj_b.weight": "model-00002-of-00004.safetensors",
|
| 161 |
+
"model.layers.14.linear_attn.in_proj_qkv.weight": "model-00002-of-00004.safetensors",
|
| 162 |
+
"model.layers.14.linear_attn.in_proj_z.weight": "model-00002-of-00004.safetensors",
|
| 163 |
+
"model.layers.14.linear_attn.norm.weight": "model-00002-of-00004.safetensors",
|
| 164 |
+
"model.layers.14.linear_attn.out_proj.weight": "model-00002-of-00004.safetensors",
|
| 165 |
+
"model.layers.14.mlp.experts.down_proj": "model-00002-of-00004.safetensors",
|
| 166 |
+
"model.layers.14.mlp.experts.gate_up_proj": "model-00002-of-00004.safetensors",
|
| 167 |
+
"model.layers.14.mlp.gate.weight": "model-00002-of-00004.safetensors",
|
| 168 |
+
"model.layers.14.mlp.shared_expert.down_proj.weight": "model-00002-of-00004.safetensors",
|
| 169 |
+
"model.layers.14.mlp.shared_expert.gate_proj.weight": "model-00002-of-00004.safetensors",
|
| 170 |
+
"model.layers.14.mlp.shared_expert.up_proj.weight": "model-00002-of-00004.safetensors",
|
| 171 |
+
"model.layers.14.mlp.shared_expert_gate.weight": "model-00002-of-00004.safetensors",
|
| 172 |
+
"model.layers.14.mlp_hyper_connection.block_inject_weight.weight": "model-00002-of-00004.safetensors",
|
| 173 |
+
"model.layers.14.mlp_hyper_connection.hc_norm.weight": "model-00002-of-00004.safetensors",
|
| 174 |
+
"model.layers.14.mlp_hyper_connection.input_mix_weight_down.weight": "model-00002-of-00004.safetensors",
|
| 175 |
+
"model.layers.14.mlp_hyper_connection.input_mix_weight_up.weight": "model-00002-of-00004.safetensors",
|
| 176 |
+
"model.layers.15.attn_hyper_connection.block_inject_weight.weight": "model-00002-of-00004.safetensors",
|
| 177 |
+
"model.layers.15.attn_hyper_connection.hc_norm.weight": "model-00002-of-00004.safetensors",
|
| 178 |
+
"model.layers.15.attn_hyper_connection.input_mix_weight_down.weight": "model-00002-of-00004.safetensors",
|
| 179 |
+
"model.layers.15.attn_hyper_connection.input_mix_weight_up.weight": "model-00002-of-00004.safetensors",
|
| 180 |
+
"model.layers.15.mlp.experts.down_proj": "model-00002-of-00004.safetensors",
|
| 181 |
+
"model.layers.15.mlp.experts.gate_up_proj": "model-00002-of-00004.safetensors",
|
| 182 |
+
"model.layers.15.mlp.gate.weight": "model-00002-of-00004.safetensors",
|
| 183 |
+
"model.layers.15.mlp.shared_expert.down_proj.weight": "model-00002-of-00004.safetensors",
|
| 184 |
+
"model.layers.15.mlp.shared_expert.gate_proj.weight": "model-00002-of-00004.safetensors",
|
| 185 |
+
"model.layers.15.mlp.shared_expert.up_proj.weight": "model-00002-of-00004.safetensors",
|
| 186 |
+
"model.layers.15.mlp.shared_expert_gate.weight": "model-00002-of-00004.safetensors",
|
| 187 |
+
"model.layers.15.mlp_hyper_connection.block_inject_weight.weight": "model-00002-of-00004.safetensors",
|
| 188 |
+
"model.layers.15.mlp_hyper_connection.hc_norm.weight": "model-00002-of-00004.safetensors",
|
| 189 |
+
"model.layers.15.mlp_hyper_connection.input_mix_weight_down.weight": "model-00002-of-00004.safetensors",
|
| 190 |
+
"model.layers.15.mlp_hyper_connection.input_mix_weight_up.weight": "model-00002-of-00004.safetensors",
|
| 191 |
+
"model.layers.15.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
| 192 |
+
"model.layers.15.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 193 |
+
"model.layers.15.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
| 194 |
+
"model.layers.15.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
| 195 |
+
"model.layers.15.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 196 |
+
"model.layers.15.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 197 |
+
"model.layers.16.attn_hyper_connection.block_inject_weight.weight": "model-00002-of-00004.safetensors",
|
| 198 |
+
"model.layers.16.attn_hyper_connection.hc_norm.weight": "model-00002-of-00004.safetensors",
|
| 199 |
+
"model.layers.16.attn_hyper_connection.input_mix_weight_down.weight": "model-00002-of-00004.safetensors",
|
| 200 |
+
"model.layers.16.attn_hyper_connection.input_mix_weight_up.weight": "model-00002-of-00004.safetensors",
|
| 201 |
+
"model.layers.16.linear_attn.A_log": "model-00002-of-00004.safetensors",
|
| 202 |
+
"model.layers.16.linear_attn.conv1d.weight": "model-00002-of-00004.safetensors",
|
| 203 |
+
"model.layers.16.linear_attn.dt_bias": "model-00002-of-00004.safetensors",
|
| 204 |
+
"model.layers.16.linear_attn.in_proj_a.weight": "model-00002-of-00004.safetensors",
|
| 205 |
+
"model.layers.16.linear_attn.in_proj_b.weight": "model-00002-of-00004.safetensors",
|
| 206 |
+
"model.layers.16.linear_attn.in_proj_qkv.weight": "model-00002-of-00004.safetensors",
|
| 207 |
+
"model.layers.16.linear_attn.in_proj_z.weight": "model-00002-of-00004.safetensors",
|
| 208 |
+
"model.layers.16.linear_attn.norm.weight": "model-00002-of-00004.safetensors",
|
| 209 |
+
"model.layers.16.linear_attn.out_proj.weight": "model-00002-of-00004.safetensors",
|
| 210 |
+
"model.layers.16.mlp.experts.down_proj": "model-00002-of-00004.safetensors",
|
| 211 |
+
"model.layers.16.mlp.experts.gate_up_proj": "model-00003-of-00004.safetensors",
|
| 212 |
+
"model.layers.16.mlp.gate.weight": "model-00003-of-00004.safetensors",
|
| 213 |
+
"model.layers.16.mlp.shared_expert.down_proj.weight": "model-00003-of-00004.safetensors",
|
| 214 |
+
"model.layers.16.mlp.shared_expert.gate_proj.weight": "model-00003-of-00004.safetensors",
|
| 215 |
+
"model.layers.16.mlp.shared_expert.up_proj.weight": "model-00003-of-00004.safetensors",
|
| 216 |
+
"model.layers.16.mlp.shared_expert_gate.weight": "model-00003-of-00004.safetensors",
|
| 217 |
+
"model.layers.16.mlp_hyper_connection.block_inject_weight.weight": "model-00003-of-00004.safetensors",
|
| 218 |
+
"model.layers.16.mlp_hyper_connection.hc_norm.weight": "model-00003-of-00004.safetensors",
|
| 219 |
+
"model.layers.16.mlp_hyper_connection.input_mix_weight_down.weight": "model-00003-of-00004.safetensors",
|
| 220 |
+
"model.layers.16.mlp_hyper_connection.input_mix_weight_up.weight": "model-00003-of-00004.safetensors",
|
| 221 |
+
"model.layers.17.attn_hyper_connection.block_inject_weight.weight": "model-00003-of-00004.safetensors",
|
| 222 |
+
"model.layers.17.attn_hyper_connection.hc_norm.weight": "model-00003-of-00004.safetensors",
|
| 223 |
+
"model.layers.17.attn_hyper_connection.input_mix_weight_down.weight": "model-00003-of-00004.safetensors",
|
| 224 |
+
"model.layers.17.attn_hyper_connection.input_mix_weight_up.weight": "model-00003-of-00004.safetensors",
|
| 225 |
+
"model.layers.17.linear_attn.A_log": "model-00003-of-00004.safetensors",
|
| 226 |
+
"model.layers.17.linear_attn.conv1d.weight": "model-00003-of-00004.safetensors",
|
| 227 |
+
"model.layers.17.linear_attn.dt_bias": "model-00003-of-00004.safetensors",
|
| 228 |
+
"model.layers.17.linear_attn.in_proj_a.weight": "model-00003-of-00004.safetensors",
|
| 229 |
+
"model.layers.17.linear_attn.in_proj_b.weight": "model-00003-of-00004.safetensors",
|
| 230 |
+
"model.layers.17.linear_attn.in_proj_qkv.weight": "model-00003-of-00004.safetensors",
|
| 231 |
+
"model.layers.17.linear_attn.in_proj_z.weight": "model-00003-of-00004.safetensors",
|
| 232 |
+
"model.layers.17.linear_attn.norm.weight": "model-00003-of-00004.safetensors",
|
| 233 |
+
"model.layers.17.linear_attn.out_proj.weight": "model-00003-of-00004.safetensors",
|
| 234 |
+
"model.layers.17.mlp.experts.down_proj": "model-00003-of-00004.safetensors",
|
| 235 |
+
"model.layers.17.mlp.experts.gate_up_proj": "model-00003-of-00004.safetensors",
|
| 236 |
+
"model.layers.17.mlp.gate.weight": "model-00003-of-00004.safetensors",
|
| 237 |
+
"model.layers.17.mlp.shared_expert.down_proj.weight": "model-00003-of-00004.safetensors",
|
| 238 |
+
"model.layers.17.mlp.shared_expert.gate_proj.weight": "model-00003-of-00004.safetensors",
|
| 239 |
+
"model.layers.17.mlp.shared_expert.up_proj.weight": "model-00003-of-00004.safetensors",
|
| 240 |
+
"model.layers.17.mlp.shared_expert_gate.weight": "model-00003-of-00004.safetensors",
|
| 241 |
+
"model.layers.17.mlp_hyper_connection.block_inject_weight.weight": "model-00003-of-00004.safetensors",
|
| 242 |
+
"model.layers.17.mlp_hyper_connection.hc_norm.weight": "model-00003-of-00004.safetensors",
|
| 243 |
+
"model.layers.17.mlp_hyper_connection.input_mix_weight_down.weight": "model-00003-of-00004.safetensors",
|
| 244 |
+
"model.layers.17.mlp_hyper_connection.input_mix_weight_up.weight": "model-00003-of-00004.safetensors",
|
| 245 |
+
"model.layers.18.attn_hyper_connection.block_inject_weight.weight": "model-00003-of-00004.safetensors",
|
| 246 |
+
"model.layers.18.attn_hyper_connection.hc_norm.weight": "model-00003-of-00004.safetensors",
|
| 247 |
+
"model.layers.18.attn_hyper_connection.input_mix_weight_down.weight": "model-00003-of-00004.safetensors",
|
| 248 |
+
"model.layers.18.attn_hyper_connection.input_mix_weight_up.weight": "model-00003-of-00004.safetensors",
|
| 249 |
+
"model.layers.18.linear_attn.A_log": "model-00003-of-00004.safetensors",
|
| 250 |
+
"model.layers.18.linear_attn.conv1d.weight": "model-00003-of-00004.safetensors",
|
| 251 |
+
"model.layers.18.linear_attn.dt_bias": "model-00003-of-00004.safetensors",
|
| 252 |
+
"model.layers.18.linear_attn.in_proj_a.weight": "model-00003-of-00004.safetensors",
|
| 253 |
+
"model.layers.18.linear_attn.in_proj_b.weight": "model-00003-of-00004.safetensors",
|
| 254 |
+
"model.layers.18.linear_attn.in_proj_qkv.weight": "model-00003-of-00004.safetensors",
|
| 255 |
+
"model.layers.18.linear_attn.in_proj_z.weight": "model-00003-of-00004.safetensors",
|
| 256 |
+
"model.layers.18.linear_attn.norm.weight": "model-00003-of-00004.safetensors",
|
| 257 |
+
"model.layers.18.linear_attn.out_proj.weight": "model-00003-of-00004.safetensors",
|
| 258 |
+
"model.layers.18.mlp.experts.down_proj": "model-00003-of-00004.safetensors",
|
| 259 |
+
"model.layers.18.mlp.experts.gate_up_proj": "model-00003-of-00004.safetensors",
|
| 260 |
+
"model.layers.18.mlp.gate.weight": "model-00003-of-00004.safetensors",
|
| 261 |
+
"model.layers.18.mlp.shared_expert.down_proj.weight": "model-00003-of-00004.safetensors",
|
| 262 |
+
"model.layers.18.mlp.shared_expert.gate_proj.weight": "model-00003-of-00004.safetensors",
|
| 263 |
+
"model.layers.18.mlp.shared_expert.up_proj.weight": "model-00003-of-00004.safetensors",
|
| 264 |
+
"model.layers.18.mlp.shared_expert_gate.weight": "model-00003-of-00004.safetensors",
|
| 265 |
+
"model.layers.18.mlp_hyper_connection.block_inject_weight.weight": "model-00003-of-00004.safetensors",
|
| 266 |
+
"model.layers.18.mlp_hyper_connection.hc_norm.weight": "model-00003-of-00004.safetensors",
|
| 267 |
+
"model.layers.18.mlp_hyper_connection.input_mix_weight_down.weight": "model-00003-of-00004.safetensors",
|
| 268 |
+
"model.layers.18.mlp_hyper_connection.input_mix_weight_up.weight": "model-00003-of-00004.safetensors",
|
| 269 |
+
"model.layers.19.attn_hyper_connection.block_inject_weight.weight": "model-00003-of-00004.safetensors",
|
| 270 |
+
"model.layers.19.attn_hyper_connection.hc_norm.weight": "model-00003-of-00004.safetensors",
|
| 271 |
+
"model.layers.19.attn_hyper_connection.input_mix_weight_down.weight": "model-00003-of-00004.safetensors",
|
| 272 |
+
"model.layers.19.attn_hyper_connection.input_mix_weight_up.weight": "model-00003-of-00004.safetensors",
|
| 273 |
+
"model.layers.19.mlp.experts.down_proj": "model-00003-of-00004.safetensors",
|
| 274 |
+
"model.layers.19.mlp.experts.gate_up_proj": "model-00003-of-00004.safetensors",
|
| 275 |
+
"model.layers.19.mlp.gate.weight": "model-00003-of-00004.safetensors",
|
| 276 |
+
"model.layers.19.mlp.shared_expert.down_proj.weight": "model-00003-of-00004.safetensors",
|
| 277 |
+
"model.layers.19.mlp.shared_expert.gate_proj.weight": "model-00003-of-00004.safetensors",
|
| 278 |
+
"model.layers.19.mlp.shared_expert.up_proj.weight": "model-00003-of-00004.safetensors",
|
| 279 |
+
"model.layers.19.mlp.shared_expert_gate.weight": "model-00003-of-00004.safetensors",
|
| 280 |
+
"model.layers.19.mlp_hyper_connection.block_inject_weight.weight": "model-00003-of-00004.safetensors",
|
| 281 |
+
"model.layers.19.mlp_hyper_connection.hc_norm.weight": "model-00003-of-00004.safetensors",
|
| 282 |
+
"model.layers.19.mlp_hyper_connection.input_mix_weight_down.weight": "model-00003-of-00004.safetensors",
|
| 283 |
+
"model.layers.19.mlp_hyper_connection.input_mix_weight_up.weight": "model-00003-of-00004.safetensors",
|
| 284 |
+
"model.layers.19.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
| 285 |
+
"model.layers.19.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 286 |
+
"model.layers.19.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
| 287 |
+
"model.layers.19.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
| 288 |
+
"model.layers.19.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 289 |
+
"model.layers.19.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 290 |
+
"model.layers.2.attn_hyper_connection.block_inject_weight.weight": "model-00001-of-00004.safetensors",
|
| 291 |
+
"model.layers.2.attn_hyper_connection.hc_norm.weight": "model-00001-of-00004.safetensors",
|
| 292 |
+
"model.layers.2.attn_hyper_connection.input_mix_weight_down.weight": "model-00001-of-00004.safetensors",
|
| 293 |
+
"model.layers.2.attn_hyper_connection.input_mix_weight_up.weight": "model-00001-of-00004.safetensors",
|
| 294 |
+
"model.layers.2.linear_attn.A_log": "model-00001-of-00004.safetensors",
|
| 295 |
+
"model.layers.2.linear_attn.conv1d.weight": "model-00001-of-00004.safetensors",
|
| 296 |
+
"model.layers.2.linear_attn.dt_bias": "model-00001-of-00004.safetensors",
|
| 297 |
+
"model.layers.2.linear_attn.in_proj_a.weight": "model-00001-of-00004.safetensors",
|
| 298 |
+
"model.layers.2.linear_attn.in_proj_b.weight": "model-00001-of-00004.safetensors",
|
| 299 |
+
"model.layers.2.linear_attn.in_proj_qkv.weight": "model-00001-of-00004.safetensors",
|
| 300 |
+
"model.layers.2.linear_attn.in_proj_z.weight": "model-00001-of-00004.safetensors",
|
| 301 |
+
"model.layers.2.linear_attn.norm.weight": "model-00001-of-00004.safetensors",
|
| 302 |
+
"model.layers.2.linear_attn.out_proj.weight": "model-00001-of-00004.safetensors",
|
| 303 |
+
"model.layers.2.mlp.experts.down_proj": "model-00001-of-00004.safetensors",
|
| 304 |
+
"model.layers.2.mlp.experts.gate_up_proj": "model-00001-of-00004.safetensors",
|
| 305 |
+
"model.layers.2.mlp.gate.weight": "model-00001-of-00004.safetensors",
|
| 306 |
+
"model.layers.2.mlp.shared_expert.down_proj.weight": "model-00001-of-00004.safetensors",
|
| 307 |
+
"model.layers.2.mlp.shared_expert.gate_proj.weight": "model-00001-of-00004.safetensors",
|
| 308 |
+
"model.layers.2.mlp.shared_expert.up_proj.weight": "model-00001-of-00004.safetensors",
|
| 309 |
+
"model.layers.2.mlp.shared_expert_gate.weight": "model-00001-of-00004.safetensors",
|
| 310 |
+
"model.layers.2.mlp_hyper_connection.block_inject_weight.weight": "model-00001-of-00004.safetensors",
|
| 311 |
+
"model.layers.2.mlp_hyper_connection.hc_norm.weight": "model-00001-of-00004.safetensors",
|
| 312 |
+
"model.layers.2.mlp_hyper_connection.input_mix_weight_down.weight": "model-00001-of-00004.safetensors",
|
| 313 |
+
"model.layers.2.mlp_hyper_connection.input_mix_weight_up.weight": "model-00001-of-00004.safetensors",
|
| 314 |
+
"model.layers.20.attn_hyper_connection.block_inject_weight.weight": "model-00003-of-00004.safetensors",
|
| 315 |
+
"model.layers.20.attn_hyper_connection.hc_norm.weight": "model-00003-of-00004.safetensors",
|
| 316 |
+
"model.layers.20.attn_hyper_connection.input_mix_weight_down.weight": "model-00003-of-00004.safetensors",
|
| 317 |
+
"model.layers.20.attn_hyper_connection.input_mix_weight_up.weight": "model-00003-of-00004.safetensors",
|
| 318 |
+
"model.layers.20.linear_attn.A_log": "model-00003-of-00004.safetensors",
|
| 319 |
+
"model.layers.20.linear_attn.conv1d.weight": "model-00003-of-00004.safetensors",
|
| 320 |
+
"model.layers.20.linear_attn.dt_bias": "model-00003-of-00004.safetensors",
|
| 321 |
+
"model.layers.20.linear_attn.in_proj_a.weight": "model-00003-of-00004.safetensors",
|
| 322 |
+
"model.layers.20.linear_attn.in_proj_b.weight": "model-00003-of-00004.safetensors",
|
| 323 |
+
"model.layers.20.linear_attn.in_proj_qkv.weight": "model-00003-of-00004.safetensors",
|
| 324 |
+
"model.layers.20.linear_attn.in_proj_z.weight": "model-00003-of-00004.safetensors",
|
| 325 |
+
"model.layers.20.linear_attn.norm.weight": "model-00003-of-00004.safetensors",
|
| 326 |
+
"model.layers.20.linear_attn.out_proj.weight": "model-00003-of-00004.safetensors",
|
| 327 |
+
"model.layers.20.mlp.experts.down_proj": "model-00003-of-00004.safetensors",
|
| 328 |
+
"model.layers.20.mlp.experts.gate_up_proj": "model-00003-of-00004.safetensors",
|
| 329 |
+
"model.layers.20.mlp.gate.weight": "model-00003-of-00004.safetensors",
|
| 330 |
+
"model.layers.20.mlp.shared_expert.down_proj.weight": "model-00003-of-00004.safetensors",
|
| 331 |
+
"model.layers.20.mlp.shared_expert.gate_proj.weight": "model-00003-of-00004.safetensors",
|
| 332 |
+
"model.layers.20.mlp.shared_expert.up_proj.weight": "model-00003-of-00004.safetensors",
|
| 333 |
+
"model.layers.20.mlp.shared_expert_gate.weight": "model-00003-of-00004.safetensors",
|
| 334 |
+
"model.layers.20.mlp_hyper_connection.block_inject_weight.weight": "model-00003-of-00004.safetensors",
|
| 335 |
+
"model.layers.20.mlp_hyper_connection.hc_norm.weight": "model-00003-of-00004.safetensors",
|
| 336 |
+
"model.layers.20.mlp_hyper_connection.input_mix_weight_down.weight": "model-00003-of-00004.safetensors",
|
| 337 |
+
"model.layers.20.mlp_hyper_connection.input_mix_weight_up.weight": "model-00003-of-00004.safetensors",
|
| 338 |
+
"model.layers.21.attn_hyper_connection.block_inject_weight.weight": "model-00003-of-00004.safetensors",
|
| 339 |
+
"model.layers.21.attn_hyper_connection.hc_norm.weight": "model-00003-of-00004.safetensors",
|
| 340 |
+
"model.layers.21.attn_hyper_connection.input_mix_weight_down.weight": "model-00003-of-00004.safetensors",
|
| 341 |
+
"model.layers.21.attn_hyper_connection.input_mix_weight_up.weight": "model-00003-of-00004.safetensors",
|
| 342 |
+
"model.layers.21.linear_attn.A_log": "model-00003-of-00004.safetensors",
|
| 343 |
+
"model.layers.21.linear_attn.conv1d.weight": "model-00003-of-00004.safetensors",
|
| 344 |
+
"model.layers.21.linear_attn.dt_bias": "model-00003-of-00004.safetensors",
|
| 345 |
+
"model.layers.21.linear_attn.in_proj_a.weight": "model-00003-of-00004.safetensors",
|
| 346 |
+
"model.layers.21.linear_attn.in_proj_b.weight": "model-00003-of-00004.safetensors",
|
| 347 |
+
"model.layers.21.linear_attn.in_proj_qkv.weight": "model-00003-of-00004.safetensors",
|
| 348 |
+
"model.layers.21.linear_attn.in_proj_z.weight": "model-00003-of-00004.safetensors",
|
| 349 |
+
"model.layers.21.linear_attn.norm.weight": "model-00003-of-00004.safetensors",
|
| 350 |
+
"model.layers.21.linear_attn.out_proj.weight": "model-00003-of-00004.safetensors",
|
| 351 |
+
"model.layers.21.mlp.experts.down_proj": "model-00003-of-00004.safetensors",
|
| 352 |
+
"model.layers.21.mlp.experts.gate_up_proj": "model-00003-of-00004.safetensors",
|
| 353 |
+
"model.layers.21.mlp.gate.weight": "model-00003-of-00004.safetensors",
|
| 354 |
+
"model.layers.21.mlp.shared_expert.down_proj.weight": "model-00003-of-00004.safetensors",
|
| 355 |
+
"model.layers.21.mlp.shared_expert.gate_proj.weight": "model-00003-of-00004.safetensors",
|
| 356 |
+
"model.layers.21.mlp.shared_expert.up_proj.weight": "model-00003-of-00004.safetensors",
|
| 357 |
+
"model.layers.21.mlp.shared_expert_gate.weight": "model-00003-of-00004.safetensors",
|
| 358 |
+
"model.layers.21.mlp_hyper_connection.block_inject_weight.weight": "model-00003-of-00004.safetensors",
|
| 359 |
+
"model.layers.21.mlp_hyper_connection.hc_norm.weight": "model-00003-of-00004.safetensors",
|
| 360 |
+
"model.layers.21.mlp_hyper_connection.input_mix_weight_down.weight": "model-00003-of-00004.safetensors",
|
| 361 |
+
"model.layers.21.mlp_hyper_connection.input_mix_weight_up.weight": "model-00003-of-00004.safetensors",
|
| 362 |
+
"model.layers.22.attn_hyper_connection.block_inject_weight.weight": "model-00003-of-00004.safetensors",
|
| 363 |
+
"model.layers.22.attn_hyper_connection.hc_norm.weight": "model-00003-of-00004.safetensors",
|
| 364 |
+
"model.layers.22.attn_hyper_connection.input_mix_weight_down.weight": "model-00003-of-00004.safetensors",
|
| 365 |
+
"model.layers.22.attn_hyper_connection.input_mix_weight_up.weight": "model-00003-of-00004.safetensors",
|
| 366 |
+
"model.layers.22.linear_attn.A_log": "model-00003-of-00004.safetensors",
|
| 367 |
+
"model.layers.22.linear_attn.conv1d.weight": "model-00003-of-00004.safetensors",
|
| 368 |
+
"model.layers.22.linear_attn.dt_bias": "model-00003-of-00004.safetensors",
|
| 369 |
+
"model.layers.22.linear_attn.in_proj_a.weight": "model-00003-of-00004.safetensors",
|
| 370 |
+
"model.layers.22.linear_attn.in_proj_b.weight": "model-00003-of-00004.safetensors",
|
| 371 |
+
"model.layers.22.linear_attn.in_proj_qkv.weight": "model-00003-of-00004.safetensors",
|
| 372 |
+
"model.layers.22.linear_attn.in_proj_z.weight": "model-00003-of-00004.safetensors",
|
| 373 |
+
"model.layers.22.linear_attn.norm.weight": "model-00003-of-00004.safetensors",
|
| 374 |
+
"model.layers.22.linear_attn.out_proj.weight": "model-00003-of-00004.safetensors",
|
| 375 |
+
"model.layers.22.mlp.experts.down_proj": "model-00003-of-00004.safetensors",
|
| 376 |
+
"model.layers.22.mlp.experts.gate_up_proj": "model-00003-of-00004.safetensors",
|
| 377 |
+
"model.layers.22.mlp.gate.weight": "model-00003-of-00004.safetensors",
|
| 378 |
+
"model.layers.22.mlp.shared_expert.down_proj.weight": "model-00003-of-00004.safetensors",
|
| 379 |
+
"model.layers.22.mlp.shared_expert.gate_proj.weight": "model-00003-of-00004.safetensors",
|
| 380 |
+
"model.layers.22.mlp.shared_expert.up_proj.weight": "model-00003-of-00004.safetensors",
|
| 381 |
+
"model.layers.22.mlp.shared_expert_gate.weight": "model-00003-of-00004.safetensors",
|
| 382 |
+
"model.layers.22.mlp_hyper_connection.block_inject_weight.weight": "model-00003-of-00004.safetensors",
|
| 383 |
+
"model.layers.22.mlp_hyper_connection.hc_norm.weight": "model-00003-of-00004.safetensors",
|
| 384 |
+
"model.layers.22.mlp_hyper_connection.input_mix_weight_down.weight": "model-00003-of-00004.safetensors",
|
| 385 |
+
"model.layers.22.mlp_hyper_connection.input_mix_weight_up.weight": "model-00003-of-00004.safetensors",
|
| 386 |
+
"model.layers.23.attn_hyper_connection.block_inject_weight.weight": "model-00003-of-00004.safetensors",
|
| 387 |
+
"model.layers.23.attn_hyper_connection.hc_norm.weight": "model-00003-of-00004.safetensors",
|
| 388 |
+
"model.layers.23.attn_hyper_connection.input_mix_weight_down.weight": "model-00003-of-00004.safetensors",
|
| 389 |
+
"model.layers.23.attn_hyper_connection.input_mix_weight_up.weight": "model-00003-of-00004.safetensors",
|
| 390 |
+
"model.layers.23.mlp.experts.down_proj": "model-00003-of-00004.safetensors",
|
| 391 |
+
"model.layers.23.mlp.experts.gate_up_proj": "model-00003-of-00004.safetensors",
|
| 392 |
+
"model.layers.23.mlp.gate.weight": "model-00003-of-00004.safetensors",
|
| 393 |
+
"model.layers.23.mlp.shared_expert.down_proj.weight": "model-00003-of-00004.safetensors",
|
| 394 |
+
"model.layers.23.mlp.shared_expert.gate_proj.weight": "model-00003-of-00004.safetensors",
|
| 395 |
+
"model.layers.23.mlp.shared_expert.up_proj.weight": "model-00003-of-00004.safetensors",
|
| 396 |
+
"model.layers.23.mlp.shared_expert_gate.weight": "model-00003-of-00004.safetensors",
|
| 397 |
+
"model.layers.23.mlp_hyper_connection.block_inject_weight.weight": "model-00003-of-00004.safetensors",
|
| 398 |
+
"model.layers.23.mlp_hyper_connection.hc_norm.weight": "model-00003-of-00004.safetensors",
|
| 399 |
+
"model.layers.23.mlp_hyper_connection.input_mix_weight_down.weight": "model-00003-of-00004.safetensors",
|
| 400 |
+
"model.layers.23.mlp_hyper_connection.input_mix_weight_up.weight": "model-00003-of-00004.safetensors",
|
| 401 |
+
"model.layers.23.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
|
| 402 |
+
"model.layers.23.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
|
| 403 |
+
"model.layers.23.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
|
| 404 |
+
"model.layers.23.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
|
| 405 |
+
"model.layers.23.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
|
| 406 |
+
"model.layers.23.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
|
| 407 |
+
"model.layers.24.attn_hyper_connection.block_inject_weight.weight": "model-00003-of-00004.safetensors",
|
| 408 |
+
"model.layers.24.attn_hyper_connection.hc_norm.weight": "model-00003-of-00004.safetensors",
|
| 409 |
+
"model.layers.24.attn_hyper_connection.input_mix_weight_down.weight": "model-00003-of-00004.safetensors",
|
| 410 |
+
"model.layers.24.attn_hyper_connection.input_mix_weight_up.weight": "model-00003-of-00004.safetensors",
|
| 411 |
+
"model.layers.24.linear_attn.A_log": "model-00003-of-00004.safetensors",
|
| 412 |
+
"model.layers.24.linear_attn.conv1d.weight": "model-00003-of-00004.safetensors",
|
| 413 |
+
"model.layers.24.linear_attn.dt_bias": "model-00003-of-00004.safetensors",
|
| 414 |
+
"model.layers.24.linear_attn.in_proj_a.weight": "model-00003-of-00004.safetensors",
|
| 415 |
+
"model.layers.24.linear_attn.in_proj_b.weight": "model-00003-of-00004.safetensors",
|
| 416 |
+
"model.layers.24.linear_attn.in_proj_qkv.weight": "model-00003-of-00004.safetensors",
|
| 417 |
+
"model.layers.24.linear_attn.in_proj_z.weight": "model-00003-of-00004.safetensors",
|
| 418 |
+
"model.layers.24.linear_attn.norm.weight": "model-00003-of-00004.safetensors",
|
| 419 |
+
"model.layers.24.linear_attn.out_proj.weight": "model-00003-of-00004.safetensors",
|
| 420 |
+
"model.layers.24.mlp.experts.down_proj": "model-00003-of-00004.safetensors",
|
| 421 |
+
"model.layers.24.mlp.experts.gate_up_proj": "model-00003-of-00004.safetensors",
|
| 422 |
+
"model.layers.24.mlp.gate.weight": "model-00003-of-00004.safetensors",
|
| 423 |
+
"model.layers.24.mlp.shared_expert.down_proj.weight": "model-00003-of-00004.safetensors",
|
| 424 |
+
"model.layers.24.mlp.shared_expert.gate_proj.weight": "model-00003-of-00004.safetensors",
|
| 425 |
+
"model.layers.24.mlp.shared_expert.up_proj.weight": "model-00003-of-00004.safetensors",
|
| 426 |
+
"model.layers.24.mlp.shared_expert_gate.weight": "model-00003-of-00004.safetensors",
|
| 427 |
+
"model.layers.24.mlp_hyper_connection.block_inject_weight.weight": "model-00003-of-00004.safetensors",
|
| 428 |
+
"model.layers.24.mlp_hyper_connection.hc_norm.weight": "model-00003-of-00004.safetensors",
|
| 429 |
+
"model.layers.24.mlp_hyper_connection.input_mix_weight_down.weight": "model-00003-of-00004.safetensors",
|
| 430 |
+
"model.layers.24.mlp_hyper_connection.input_mix_weight_up.weight": "model-00003-of-00004.safetensors",
|
| 431 |
+
"model.layers.25.attn_hyper_connection.block_inject_weight.weight": "model-00003-of-00004.safetensors",
|
| 432 |
+
"model.layers.25.attn_hyper_connection.hc_norm.weight": "model-00003-of-00004.safetensors",
|
| 433 |
+
"model.layers.25.attn_hyper_connection.input_mix_weight_down.weight": "model-00003-of-00004.safetensors",
|
| 434 |
+
"model.layers.25.attn_hyper_connection.input_mix_weight_up.weight": "model-00003-of-00004.safetensors",
|
| 435 |
+
"model.layers.25.linear_attn.A_log": "model-00003-of-00004.safetensors",
|
| 436 |
+
"model.layers.25.linear_attn.conv1d.weight": "model-00003-of-00004.safetensors",
|
| 437 |
+
"model.layers.25.linear_attn.dt_bias": "model-00003-of-00004.safetensors",
|
| 438 |
+
"model.layers.25.linear_attn.in_proj_a.weight": "model-00003-of-00004.safetensors",
|
| 439 |
+
"model.layers.25.linear_attn.in_proj_b.weight": "model-00003-of-00004.safetensors",
|
| 440 |
+
"model.layers.25.linear_attn.in_proj_qkv.weight": "model-00003-of-00004.safetensors",
|
| 441 |
+
"model.layers.25.linear_attn.in_proj_z.weight": "model-00003-of-00004.safetensors",
|
| 442 |
+
"model.layers.25.linear_attn.norm.weight": "model-00003-of-00004.safetensors",
|
| 443 |
+
"model.layers.25.linear_attn.out_proj.weight": "model-00003-of-00004.safetensors",
|
| 444 |
+
"model.layers.25.mlp.experts.down_proj": "model-00003-of-00004.safetensors",
|
| 445 |
+
"model.layers.25.mlp.experts.gate_up_proj": "model-00004-of-00004.safetensors",
|
| 446 |
+
"model.layers.25.mlp.gate.weight": "model-00004-of-00004.safetensors",
|
| 447 |
+
"model.layers.25.mlp.shared_expert.down_proj.weight": "model-00004-of-00004.safetensors",
|
| 448 |
+
"model.layers.25.mlp.shared_expert.gate_proj.weight": "model-00004-of-00004.safetensors",
|
| 449 |
+
"model.layers.25.mlp.shared_expert.up_proj.weight": "model-00004-of-00004.safetensors",
|
| 450 |
+
"model.layers.25.mlp.shared_expert_gate.weight": "model-00004-of-00004.safetensors",
|
| 451 |
+
"model.layers.25.mlp_hyper_connection.block_inject_weight.weight": "model-00004-of-00004.safetensors",
|
| 452 |
+
"model.layers.25.mlp_hyper_connection.hc_norm.weight": "model-00004-of-00004.safetensors",
|
| 453 |
+
"model.layers.25.mlp_hyper_connection.input_mix_weight_down.weight": "model-00004-of-00004.safetensors",
|
| 454 |
+
"model.layers.25.mlp_hyper_connection.input_mix_weight_up.weight": "model-00004-of-00004.safetensors",
|
| 455 |
+
"model.layers.26.attn_hyper_connection.block_inject_weight.weight": "model-00004-of-00004.safetensors",
|
| 456 |
+
"model.layers.26.attn_hyper_connection.hc_norm.weight": "model-00004-of-00004.safetensors",
|
| 457 |
+
"model.layers.26.attn_hyper_connection.input_mix_weight_down.weight": "model-00004-of-00004.safetensors",
|
| 458 |
+
"model.layers.26.attn_hyper_connection.input_mix_weight_up.weight": "model-00004-of-00004.safetensors",
|
| 459 |
+
"model.layers.26.linear_attn.A_log": "model-00004-of-00004.safetensors",
|
| 460 |
+
"model.layers.26.linear_attn.conv1d.weight": "model-00004-of-00004.safetensors",
|
| 461 |
+
"model.layers.26.linear_attn.dt_bias": "model-00004-of-00004.safetensors",
|
| 462 |
+
"model.layers.26.linear_attn.in_proj_a.weight": "model-00004-of-00004.safetensors",
|
| 463 |
+
"model.layers.26.linear_attn.in_proj_b.weight": "model-00004-of-00004.safetensors",
|
| 464 |
+
"model.layers.26.linear_attn.in_proj_qkv.weight": "model-00004-of-00004.safetensors",
|
| 465 |
+
"model.layers.26.linear_attn.in_proj_z.weight": "model-00004-of-00004.safetensors",
|
| 466 |
+
"model.layers.26.linear_attn.norm.weight": "model-00004-of-00004.safetensors",
|
| 467 |
+
"model.layers.26.linear_attn.out_proj.weight": "model-00004-of-00004.safetensors",
|
| 468 |
+
"model.layers.26.mlp.experts.down_proj": "model-00004-of-00004.safetensors",
|
| 469 |
+
"model.layers.26.mlp.experts.gate_up_proj": "model-00004-of-00004.safetensors",
|
| 470 |
+
"model.layers.26.mlp.gate.weight": "model-00004-of-00004.safetensors",
|
| 471 |
+
"model.layers.26.mlp.shared_expert.down_proj.weight": "model-00004-of-00004.safetensors",
|
| 472 |
+
"model.layers.26.mlp.shared_expert.gate_proj.weight": "model-00004-of-00004.safetensors",
|
| 473 |
+
"model.layers.26.mlp.shared_expert.up_proj.weight": "model-00004-of-00004.safetensors",
|
| 474 |
+
"model.layers.26.mlp.shared_expert_gate.weight": "model-00004-of-00004.safetensors",
|
| 475 |
+
"model.layers.26.mlp_hyper_connection.block_inject_weight.weight": "model-00004-of-00004.safetensors",
|
| 476 |
+
"model.layers.26.mlp_hyper_connection.hc_norm.weight": "model-00004-of-00004.safetensors",
|
| 477 |
+
"model.layers.26.mlp_hyper_connection.input_mix_weight_down.weight": "model-00004-of-00004.safetensors",
|
| 478 |
+
"model.layers.26.mlp_hyper_connection.input_mix_weight_up.weight": "model-00004-of-00004.safetensors",
|
| 479 |
+
"model.layers.27.attn_hyper_connection.block_inject_weight.weight": "model-00004-of-00004.safetensors",
|
| 480 |
+
"model.layers.27.attn_hyper_connection.hc_norm.weight": "model-00004-of-00004.safetensors",
|
| 481 |
+
"model.layers.27.attn_hyper_connection.input_mix_weight_down.weight": "model-00004-of-00004.safetensors",
|
| 482 |
+
"model.layers.27.attn_hyper_connection.input_mix_weight_up.weight": "model-00004-of-00004.safetensors",
|
| 483 |
+
"model.layers.27.mlp.experts.down_proj": "model-00004-of-00004.safetensors",
|
| 484 |
+
"model.layers.27.mlp.experts.gate_up_proj": "model-00004-of-00004.safetensors",
|
| 485 |
+
"model.layers.27.mlp.gate.weight": "model-00004-of-00004.safetensors",
|
| 486 |
+
"model.layers.27.mlp.shared_expert.down_proj.weight": "model-00004-of-00004.safetensors",
|
| 487 |
+
"model.layers.27.mlp.shared_expert.gate_proj.weight": "model-00004-of-00004.safetensors",
|
| 488 |
+
"model.layers.27.mlp.shared_expert.up_proj.weight": "model-00004-of-00004.safetensors",
|
| 489 |
+
"model.layers.27.mlp.shared_expert_gate.weight": "model-00004-of-00004.safetensors",
|
| 490 |
+
"model.layers.27.mlp_hyper_connection.block_inject_weight.weight": "model-00004-of-00004.safetensors",
|
| 491 |
+
"model.layers.27.mlp_hyper_connection.hc_norm.weight": "model-00004-of-00004.safetensors",
|
| 492 |
+
"model.layers.27.mlp_hyper_connection.input_mix_weight_down.weight": "model-00004-of-00004.safetensors",
|
| 493 |
+
"model.layers.27.mlp_hyper_connection.input_mix_weight_up.weight": "model-00004-of-00004.safetensors",
|
| 494 |
+
"model.layers.27.self_attn.k_norm.weight": "model-00004-of-00004.safetensors",
|
| 495 |
+
"model.layers.27.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
|
| 496 |
+
"model.layers.27.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
|
| 497 |
+
"model.layers.27.self_attn.q_norm.weight": "model-00004-of-00004.safetensors",
|
| 498 |
+
"model.layers.27.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
|
| 499 |
+
"model.layers.27.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
|
| 500 |
+
"model.layers.3.attn_hyper_connection.block_inject_weight.weight": "model-00001-of-00004.safetensors",
|
| 501 |
+
"model.layers.3.attn_hyper_connection.hc_norm.weight": "model-00001-of-00004.safetensors",
|
| 502 |
+
"model.layers.3.attn_hyper_connection.input_mix_weight_down.weight": "model-00001-of-00004.safetensors",
|
| 503 |
+
"model.layers.3.attn_hyper_connection.input_mix_weight_up.weight": "model-00001-of-00004.safetensors",
|
| 504 |
+
"model.layers.3.mlp.experts.down_proj": "model-00001-of-00004.safetensors",
|
| 505 |
+
"model.layers.3.mlp.experts.gate_up_proj": "model-00001-of-00004.safetensors",
|
| 506 |
+
"model.layers.3.mlp.gate.weight": "model-00001-of-00004.safetensors",
|
| 507 |
+
"model.layers.3.mlp.shared_expert.down_proj.weight": "model-00001-of-00004.safetensors",
|
| 508 |
+
"model.layers.3.mlp.shared_expert.gate_proj.weight": "model-00001-of-00004.safetensors",
|
| 509 |
+
"model.layers.3.mlp.shared_expert.up_proj.weight": "model-00001-of-00004.safetensors",
|
| 510 |
+
"model.layers.3.mlp.shared_expert_gate.weight": "model-00001-of-00004.safetensors",
|
| 511 |
+
"model.layers.3.mlp_hyper_connection.block_inject_weight.weight": "model-00001-of-00004.safetensors",
|
| 512 |
+
"model.layers.3.mlp_hyper_connection.hc_norm.weight": "model-00001-of-00004.safetensors",
|
| 513 |
+
"model.layers.3.mlp_hyper_connection.input_mix_weight_down.weight": "model-00001-of-00004.safetensors",
|
| 514 |
+
"model.layers.3.mlp_hyper_connection.input_mix_weight_up.weight": "model-00001-of-00004.safetensors",
|
| 515 |
+
"model.layers.3.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
|
| 516 |
+
"model.layers.3.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
|
| 517 |
+
"model.layers.3.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
|
| 518 |
+
"model.layers.3.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
|
| 519 |
+
"model.layers.3.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
|
| 520 |
+
"model.layers.3.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
|
| 521 |
+
"model.layers.4.attn_hyper_connection.block_inject_weight.weight": "model-00001-of-00004.safetensors",
|
| 522 |
+
"model.layers.4.attn_hyper_connection.hc_norm.weight": "model-00001-of-00004.safetensors",
|
| 523 |
+
"model.layers.4.attn_hyper_connection.input_mix_weight_down.weight": "model-00001-of-00004.safetensors",
|
| 524 |
+
"model.layers.4.attn_hyper_connection.input_mix_weight_up.weight": "model-00001-of-00004.safetensors",
|
| 525 |
+
"model.layers.4.linear_attn.A_log": "model-00001-of-00004.safetensors",
|
| 526 |
+
"model.layers.4.linear_attn.conv1d.weight": "model-00001-of-00004.safetensors",
|
| 527 |
+
"model.layers.4.linear_attn.dt_bias": "model-00001-of-00004.safetensors",
|
| 528 |
+
"model.layers.4.linear_attn.in_proj_a.weight": "model-00001-of-00004.safetensors",
|
| 529 |
+
"model.layers.4.linear_attn.in_proj_b.weight": "model-00001-of-00004.safetensors",
|
| 530 |
+
"model.layers.4.linear_attn.in_proj_qkv.weight": "model-00001-of-00004.safetensors",
|
| 531 |
+
"model.layers.4.linear_attn.in_proj_z.weight": "model-00001-of-00004.safetensors",
|
| 532 |
+
"model.layers.4.linear_attn.norm.weight": "model-00001-of-00004.safetensors",
|
| 533 |
+
"model.layers.4.linear_attn.out_proj.weight": "model-00001-of-00004.safetensors",
|
| 534 |
+
"model.layers.4.mlp.experts.down_proj": "model-00001-of-00004.safetensors",
|
| 535 |
+
"model.layers.4.mlp.experts.gate_up_proj": "model-00001-of-00004.safetensors",
|
| 536 |
+
"model.layers.4.mlp.gate.weight": "model-00001-of-00004.safetensors",
|
| 537 |
+
"model.layers.4.mlp.shared_expert.down_proj.weight": "model-00001-of-00004.safetensors",
|
| 538 |
+
"model.layers.4.mlp.shared_expert.gate_proj.weight": "model-00001-of-00004.safetensors",
|
| 539 |
+
"model.layers.4.mlp.shared_expert.up_proj.weight": "model-00001-of-00004.safetensors",
|
| 540 |
+
"model.layers.4.mlp.shared_expert_gate.weight": "model-00001-of-00004.safetensors",
|
| 541 |
+
"model.layers.4.mlp_hyper_connection.block_inject_weight.weight": "model-00001-of-00004.safetensors",
|
| 542 |
+
"model.layers.4.mlp_hyper_connection.hc_norm.weight": "model-00001-of-00004.safetensors",
|
| 543 |
+
"model.layers.4.mlp_hyper_connection.input_mix_weight_down.weight": "model-00001-of-00004.safetensors",
|
| 544 |
+
"model.layers.4.mlp_hyper_connection.input_mix_weight_up.weight": "model-00001-of-00004.safetensors",
|
| 545 |
+
"model.layers.5.attn_hyper_connection.block_inject_weight.weight": "model-00001-of-00004.safetensors",
|
| 546 |
+
"model.layers.5.attn_hyper_connection.hc_norm.weight": "model-00001-of-00004.safetensors",
|
| 547 |
+
"model.layers.5.attn_hyper_connection.input_mix_weight_down.weight": "model-00001-of-00004.safetensors",
|
| 548 |
+
"model.layers.5.attn_hyper_connection.input_mix_weight_up.weight": "model-00001-of-00004.safetensors",
|
| 549 |
+
"model.layers.5.linear_attn.A_log": "model-00001-of-00004.safetensors",
|
| 550 |
+
"model.layers.5.linear_attn.conv1d.weight": "model-00001-of-00004.safetensors",
|
| 551 |
+
"model.layers.5.linear_attn.dt_bias": "model-00001-of-00004.safetensors",
|
| 552 |
+
"model.layers.5.linear_attn.in_proj_a.weight": "model-00001-of-00004.safetensors",
|
| 553 |
+
"model.layers.5.linear_attn.in_proj_b.weight": "model-00001-of-00004.safetensors",
|
| 554 |
+
"model.layers.5.linear_attn.in_proj_qkv.weight": "model-00001-of-00004.safetensors",
|
| 555 |
+
"model.layers.5.linear_attn.in_proj_z.weight": "model-00001-of-00004.safetensors",
|
| 556 |
+
"model.layers.5.linear_attn.norm.weight": "model-00001-of-00004.safetensors",
|
| 557 |
+
"model.layers.5.linear_attn.out_proj.weight": "model-00001-of-00004.safetensors",
|
| 558 |
+
"model.layers.5.mlp.experts.down_proj": "model-00001-of-00004.safetensors",
|
| 559 |
+
"model.layers.5.mlp.experts.gate_up_proj": "model-00001-of-00004.safetensors",
|
| 560 |
+
"model.layers.5.mlp.gate.weight": "model-00001-of-00004.safetensors",
|
| 561 |
+
"model.layers.5.mlp.shared_expert.down_proj.weight": "model-00001-of-00004.safetensors",
|
| 562 |
+
"model.layers.5.mlp.shared_expert.gate_proj.weight": "model-00001-of-00004.safetensors",
|
| 563 |
+
"model.layers.5.mlp.shared_expert.up_proj.weight": "model-00001-of-00004.safetensors",
|
| 564 |
+
"model.layers.5.mlp.shared_expert_gate.weight": "model-00001-of-00004.safetensors",
|
| 565 |
+
"model.layers.5.mlp_hyper_connection.block_inject_weight.weight": "model-00001-of-00004.safetensors",
|
| 566 |
+
"model.layers.5.mlp_hyper_connection.hc_norm.weight": "model-00001-of-00004.safetensors",
|
| 567 |
+
"model.layers.5.mlp_hyper_connection.input_mix_weight_down.weight": "model-00001-of-00004.safetensors",
|
| 568 |
+
"model.layers.5.mlp_hyper_connection.input_mix_weight_up.weight": "model-00001-of-00004.safetensors",
|
| 569 |
+
"model.layers.6.attn_hyper_connection.block_inject_weight.weight": "model-00001-of-00004.safetensors",
|
| 570 |
+
"model.layers.6.attn_hyper_connection.hc_norm.weight": "model-00001-of-00004.safetensors",
|
| 571 |
+
"model.layers.6.attn_hyper_connection.input_mix_weight_down.weight": "model-00001-of-00004.safetensors",
|
| 572 |
+
"model.layers.6.attn_hyper_connection.input_mix_weight_up.weight": "model-00001-of-00004.safetensors",
|
| 573 |
+
"model.layers.6.linear_attn.A_log": "model-00001-of-00004.safetensors",
|
| 574 |
+
"model.layers.6.linear_attn.conv1d.weight": "model-00001-of-00004.safetensors",
|
| 575 |
+
"model.layers.6.linear_attn.dt_bias": "model-00001-of-00004.safetensors",
|
| 576 |
+
"model.layers.6.linear_attn.in_proj_a.weight": "model-00001-of-00004.safetensors",
|
| 577 |
+
"model.layers.6.linear_attn.in_proj_b.weight": "model-00001-of-00004.safetensors",
|
| 578 |
+
"model.layers.6.linear_attn.in_proj_qkv.weight": "model-00001-of-00004.safetensors",
|
| 579 |
+
"model.layers.6.linear_attn.in_proj_z.weight": "model-00001-of-00004.safetensors",
|
| 580 |
+
"model.layers.6.linear_attn.norm.weight": "model-00001-of-00004.safetensors",
|
| 581 |
+
"model.layers.6.linear_attn.out_proj.weight": "model-00001-of-00004.safetensors",
|
| 582 |
+
"model.layers.6.mlp.experts.down_proj": "model-00001-of-00004.safetensors",
|
| 583 |
+
"model.layers.6.mlp.experts.gate_up_proj": "model-00001-of-00004.safetensors",
|
| 584 |
+
"model.layers.6.mlp.gate.weight": "model-00001-of-00004.safetensors",
|
| 585 |
+
"model.layers.6.mlp.shared_expert.down_proj.weight": "model-00001-of-00004.safetensors",
|
| 586 |
+
"model.layers.6.mlp.shared_expert.gate_proj.weight": "model-00001-of-00004.safetensors",
|
| 587 |
+
"model.layers.6.mlp.shared_expert.up_proj.weight": "model-00001-of-00004.safetensors",
|
| 588 |
+
"model.layers.6.mlp.shared_expert_gate.weight": "model-00001-of-00004.safetensors",
|
| 589 |
+
"model.layers.6.mlp_hyper_connection.block_inject_weight.weight": "model-00001-of-00004.safetensors",
|
| 590 |
+
"model.layers.6.mlp_hyper_connection.hc_norm.weight": "model-00001-of-00004.safetensors",
|
| 591 |
+
"model.layers.6.mlp_hyper_connection.input_mix_weight_down.weight": "model-00001-of-00004.safetensors",
|
| 592 |
+
"model.layers.6.mlp_hyper_connection.input_mix_weight_up.weight": "model-00001-of-00004.safetensors",
|
| 593 |
+
"model.layers.7.attn_hyper_connection.block_inject_weight.weight": "model-00001-of-00004.safetensors",
|
| 594 |
+
"model.layers.7.attn_hyper_connection.hc_norm.weight": "model-00001-of-00004.safetensors",
|
| 595 |
+
"model.layers.7.attn_hyper_connection.input_mix_weight_down.weight": "model-00001-of-00004.safetensors",
|
| 596 |
+
"model.layers.7.attn_hyper_connection.input_mix_weight_up.weight": "model-00001-of-00004.safetensors",
|
| 597 |
+
"model.layers.7.mlp.experts.down_proj": "model-00001-of-00004.safetensors",
|
| 598 |
+
"model.layers.7.mlp.experts.gate_up_proj": "model-00002-of-00004.safetensors",
|
| 599 |
+
"model.layers.7.mlp.gate.weight": "model-00002-of-00004.safetensors",
|
| 600 |
+
"model.layers.7.mlp.shared_expert.down_proj.weight": "model-00002-of-00004.safetensors",
|
| 601 |
+
"model.layers.7.mlp.shared_expert.gate_proj.weight": "model-00002-of-00004.safetensors",
|
| 602 |
+
"model.layers.7.mlp.shared_expert.up_proj.weight": "model-00002-of-00004.safetensors",
|
| 603 |
+
"model.layers.7.mlp.shared_expert_gate.weight": "model-00002-of-00004.safetensors",
|
| 604 |
+
"model.layers.7.mlp_hyper_connection.block_inject_weight.weight": "model-00002-of-00004.safetensors",
|
| 605 |
+
"model.layers.7.mlp_hyper_connection.hc_norm.weight": "model-00002-of-00004.safetensors",
|
| 606 |
+
"model.layers.7.mlp_hyper_connection.input_mix_weight_down.weight": "model-00002-of-00004.safetensors",
|
| 607 |
+
"model.layers.7.mlp_hyper_connection.input_mix_weight_up.weight": "model-00002-of-00004.safetensors",
|
| 608 |
+
"model.layers.7.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
|
| 609 |
+
"model.layers.7.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
|
| 610 |
+
"model.layers.7.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
|
| 611 |
+
"model.layers.7.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
|
| 612 |
+
"model.layers.7.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
|
| 613 |
+
"model.layers.7.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
|
| 614 |
+
"model.layers.8.attn_hyper_connection.block_inject_weight.weight": "model-00002-of-00004.safetensors",
|
| 615 |
+
"model.layers.8.attn_hyper_connection.hc_norm.weight": "model-00002-of-00004.safetensors",
|
| 616 |
+
"model.layers.8.attn_hyper_connection.input_mix_weight_down.weight": "model-00002-of-00004.safetensors",
|
| 617 |
+
"model.layers.8.attn_hyper_connection.input_mix_weight_up.weight": "model-00002-of-00004.safetensors",
|
| 618 |
+
"model.layers.8.linear_attn.A_log": "model-00002-of-00004.safetensors",
|
| 619 |
+
"model.layers.8.linear_attn.conv1d.weight": "model-00002-of-00004.safetensors",
|
| 620 |
+
"model.layers.8.linear_attn.dt_bias": "model-00002-of-00004.safetensors",
|
| 621 |
+
"model.layers.8.linear_attn.in_proj_a.weight": "model-00002-of-00004.safetensors",
|
| 622 |
+
"model.layers.8.linear_attn.in_proj_b.weight": "model-00002-of-00004.safetensors",
|
| 623 |
+
"model.layers.8.linear_attn.in_proj_qkv.weight": "model-00002-of-00004.safetensors",
|
| 624 |
+
"model.layers.8.linear_attn.in_proj_z.weight": "model-00002-of-00004.safetensors",
|
| 625 |
+
"model.layers.8.linear_attn.norm.weight": "model-00002-of-00004.safetensors",
|
| 626 |
+
"model.layers.8.linear_attn.out_proj.weight": "model-00002-of-00004.safetensors",
|
| 627 |
+
"model.layers.8.mlp.experts.down_proj": "model-00002-of-00004.safetensors",
|
| 628 |
+
"model.layers.8.mlp.experts.gate_up_proj": "model-00002-of-00004.safetensors",
|
| 629 |
+
"model.layers.8.mlp.gate.weight": "model-00002-of-00004.safetensors",
|
| 630 |
+
"model.layers.8.mlp.shared_expert.down_proj.weight": "model-00002-of-00004.safetensors",
|
| 631 |
+
"model.layers.8.mlp.shared_expert.gate_proj.weight": "model-00002-of-00004.safetensors",
|
| 632 |
+
"model.layers.8.mlp.shared_expert.up_proj.weight": "model-00002-of-00004.safetensors",
|
| 633 |
+
"model.layers.8.mlp.shared_expert_gate.weight": "model-00002-of-00004.safetensors",
|
| 634 |
+
"model.layers.8.mlp_hyper_connection.block_inject_weight.weight": "model-00002-of-00004.safetensors",
|
| 635 |
+
"model.layers.8.mlp_hyper_connection.hc_norm.weight": "model-00002-of-00004.safetensors",
|
| 636 |
+
"model.layers.8.mlp_hyper_connection.input_mix_weight_down.weight": "model-00002-of-00004.safetensors",
|
| 637 |
+
"model.layers.8.mlp_hyper_connection.input_mix_weight_up.weight": "model-00002-of-00004.safetensors",
|
| 638 |
+
"model.layers.9.attn_hyper_connection.block_inject_weight.weight": "model-00002-of-00004.safetensors",
|
| 639 |
+
"model.layers.9.attn_hyper_connection.hc_norm.weight": "model-00002-of-00004.safetensors",
|
| 640 |
+
"model.layers.9.attn_hyper_connection.input_mix_weight_down.weight": "model-00002-of-00004.safetensors",
|
| 641 |
+
"model.layers.9.attn_hyper_connection.input_mix_weight_up.weight": "model-00002-of-00004.safetensors",
|
| 642 |
+
"model.layers.9.linear_attn.A_log": "model-00002-of-00004.safetensors",
|
| 643 |
+
"model.layers.9.linear_attn.conv1d.weight": "model-00002-of-00004.safetensors",
|
| 644 |
+
"model.layers.9.linear_attn.dt_bias": "model-00002-of-00004.safetensors",
|
| 645 |
+
"model.layers.9.linear_attn.in_proj_a.weight": "model-00002-of-00004.safetensors",
|
| 646 |
+
"model.layers.9.linear_attn.in_proj_b.weight": "model-00002-of-00004.safetensors",
|
| 647 |
+
"model.layers.9.linear_attn.in_proj_qkv.weight": "model-00002-of-00004.safetensors",
|
| 648 |
+
"model.layers.9.linear_attn.in_proj_z.weight": "model-00002-of-00004.safetensors",
|
| 649 |
+
"model.layers.9.linear_attn.norm.weight": "model-00002-of-00004.safetensors",
|
| 650 |
+
"model.layers.9.linear_attn.out_proj.weight": "model-00002-of-00004.safetensors",
|
| 651 |
+
"model.layers.9.mlp.experts.down_proj": "model-00002-of-00004.safetensors",
|
| 652 |
+
"model.layers.9.mlp.experts.gate_up_proj": "model-00002-of-00004.safetensors",
|
| 653 |
+
"model.layers.9.mlp.gate.weight": "model-00002-of-00004.safetensors",
|
| 654 |
+
"model.layers.9.mlp.shared_expert.down_proj.weight": "model-00002-of-00004.safetensors",
|
| 655 |
+
"model.layers.9.mlp.shared_expert.gate_proj.weight": "model-00002-of-00004.safetensors",
|
| 656 |
+
"model.layers.9.mlp.shared_expert.up_proj.weight": "model-00002-of-00004.safetensors",
|
| 657 |
+
"model.layers.9.mlp.shared_expert_gate.weight": "model-00002-of-00004.safetensors",
|
| 658 |
+
"model.layers.9.mlp_hyper_connection.block_inject_weight.weight": "model-00002-of-00004.safetensors",
|
| 659 |
+
"model.layers.9.mlp_hyper_connection.hc_norm.weight": "model-00002-of-00004.safetensors",
|
| 660 |
+
"model.layers.9.mlp_hyper_connection.input_mix_weight_down.weight": "model-00002-of-00004.safetensors",
|
| 661 |
+
"model.layers.9.mlp_hyper_connection.input_mix_weight_up.weight": "model-00002-of-00004.safetensors"
|
| 662 |
+
}
|
| 663 |
+
}
|
sft/Modujo-4B-A0.66B/qwen4_exp_dense_compat.py
ADDED
|
@@ -0,0 +1,62 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Compatibility support for pre-QSA dense Qwen4-Exp checkpoints.
|
| 2 |
+
|
| 3 |
+
Transformers 5.16 accepts ``None`` QSA fields in the configuration but still
|
| 4 |
+
constructs and calls the QSA indexer. A pre-QSA checkpoint needs the same
|
| 5 |
+
attention module and parameter names, with the indexer absent and the ordinary
|
| 6 |
+
causal mask passed directly to the core attention operation.
|
| 7 |
+
"""
|
| 8 |
+
|
| 9 |
+
from __future__ import annotations
|
| 10 |
+
|
| 11 |
+
from typing import Any
|
| 12 |
+
|
| 13 |
+
import torch
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
_PATCHED = False
|
| 17 |
+
|
| 18 |
+
|
| 19 |
+
def qsa_is_enabled(config: Any) -> bool:
|
| 20 |
+
fields = (
|
| 21 |
+
"indexer_n_heads",
|
| 22 |
+
"indexer_kv_heads",
|
| 23 |
+
"indexer_head_dim",
|
| 24 |
+
"indexer_budget",
|
| 25 |
+
"indexer_compress_ratio",
|
| 26 |
+
)
|
| 27 |
+
values = [getattr(config, name, None) for name in fields]
|
| 28 |
+
if any(value is None for value in values) and any(value is not None for value in values):
|
| 29 |
+
raise ValueError("QSA indexer fields must either all be set or all be None")
|
| 30 |
+
return all(value is not None for value in values)
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
def enable_dense_qwen4_exp_attention() -> None:
|
| 34 |
+
"""Make null-indexer Qwen4-Exp attention execute as dense attention."""
|
| 35 |
+
global _PATCHED
|
| 36 |
+
if _PATCHED:
|
| 37 |
+
return
|
| 38 |
+
|
| 39 |
+
from transformers.models.qwen4_exp import modeling_qwen4_exp as modeling
|
| 40 |
+
|
| 41 |
+
original_init = modeling.Qwen4ExpTextQSAIndexer.__init__
|
| 42 |
+
original_forward = modeling.Qwen4ExpTextQSAIndexer.forward
|
| 43 |
+
|
| 44 |
+
def patched_init(self, config, layer_idx):
|
| 45 |
+
if qsa_is_enabled(config):
|
| 46 |
+
original_init(self, config, layer_idx)
|
| 47 |
+
self.qsa_enabled = True
|
| 48 |
+
else:
|
| 49 |
+
torch.nn.Module.__init__(self)
|
| 50 |
+
self.layer_idx = layer_idx
|
| 51 |
+
self.qsa_enabled = False
|
| 52 |
+
|
| 53 |
+
def patched_forward(self, hidden_states, position_embeddings, attention_mask, past_key_values):
|
| 54 |
+
if self.qsa_enabled:
|
| 55 |
+
return original_forward(self, hidden_states, position_embeddings, attention_mask, past_key_values)
|
| 56 |
+
if attention_mask.dtype == torch.bool:
|
| 57 |
+
return torch.ones_like(attention_mask, dtype=torch.bool)
|
| 58 |
+
return torch.zeros_like(attention_mask)
|
| 59 |
+
|
| 60 |
+
modeling.Qwen4ExpTextQSAIndexer.__init__ = patched_init
|
| 61 |
+
modeling.Qwen4ExpTextQSAIndexer.forward = patched_forward
|
| 62 |
+
_PATCHED = True
|
sft/Modujo-4B-A0.66B/release.json
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"repository": "Modujo-AI/ModujoMoE",
|
| 3 |
+
"subdirectory": "sft/Modujo-4B-A0.66B",
|
| 4 |
+
"stage": "short SFT",
|
| 5 |
+
"step": 36000,
|
| 6 |
+
"planned_stage_steps": 40017,
|
| 7 |
+
"teacher_stage_started": false,
|
| 8 |
+
"total_parameters_approx": 3963169632,
|
| 9 |
+
"active_parameters_approx": 660155232,
|
| 10 |
+
"stored_dtype": "float32",
|
| 11 |
+
"evaluated_dtype": "bfloat16",
|
| 12 |
+
"versions": {
|
| 13 |
+
"torch": "2.13.0+cu130",
|
| 14 |
+
"transformers": "5.16.1",
|
| 15 |
+
"accelerate": "1.15.0",
|
| 16 |
+
"huggingface_hub": "1.31.0",
|
| 17 |
+
"safetensors": "0.8.0"
|
| 18 |
+
}
|
| 19 |
+
}
|
sft/Modujo-4B-A0.66B/tested-environment.json
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"torch": "2.13.0+cu130",
|
| 3 |
+
"transformers": "5.16.1",
|
| 4 |
+
"accelerate": "1.15.0",
|
| 5 |
+
"huggingface_hub": "1.31.0",
|
| 6 |
+
"safetensors": "0.8.0"
|
| 7 |
+
}
|
sft/Modujo-4B-A0.66B/tokenizer.json
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523
|
| 3 |
+
size 19989325
|
sft/Modujo-4B-A0.66B/tokenizer_config.json
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"add_prefix_space": false,
|
| 3 |
+
"audio_bos_token": "<|audio_start|>",
|
| 4 |
+
"audio_eos_token": "<|audio_end|>",
|
| 5 |
+
"audio_token": "<|audio_pad|>",
|
| 6 |
+
"backend": "tokenizers",
|
| 7 |
+
"bos_token": null,
|
| 8 |
+
"clean_up_tokenization_spaces": false,
|
| 9 |
+
"eos_token": "<|im_end|>",
|
| 10 |
+
"errors": "replace",
|
| 11 |
+
"image_token": "<|image_pad|>",
|
| 12 |
+
"is_local": true,
|
| 13 |
+
"local_files_only": false,
|
| 14 |
+
"model_max_length": 262144,
|
| 15 |
+
"model_specific_special_tokens": {
|
| 16 |
+
"audio_bos_token": "<|audio_start|>",
|
| 17 |
+
"audio_eos_token": "<|audio_end|>",
|
| 18 |
+
"audio_token": "<|audio_pad|>",
|
| 19 |
+
"image_token": "<|image_pad|>",
|
| 20 |
+
"video_token": "<|video_pad|>",
|
| 21 |
+
"vision_bos_token": "<|vision_start|>",
|
| 22 |
+
"vision_eos_token": "<|vision_end|>"
|
| 23 |
+
},
|
| 24 |
+
"pad_token": "<|endoftext|>",
|
| 25 |
+
"pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
|
| 26 |
+
"split_special_tokens": false,
|
| 27 |
+
"tokenizer_class": "Qwen2Tokenizer",
|
| 28 |
+
"unk_token": null,
|
| 29 |
+
"video_token": "<|video_pad|>",
|
| 30 |
+
"vision_bos_token": "<|vision_start|>",
|
| 31 |
+
"vision_eos_token": "<|vision_end|>"
|
| 32 |
+
}
|