Add files using upload-large-folder tool

README.md

@@ -0,0 +1,72 @@
+---
+license: apache-2.0
+base_model: tencent/Hy3
+base_model_relation: quantized
+tags:
+- mlx
+- hunyuan
+- hy3
+- moe
+pipeline_tag: text-generation
+library_name: mlx
+---
+
+# Hy3-4bit (MLX)
+
+A clean 4-bit MLX quantization of [tencent/Hy3](https://huggingface.co/tencent/Hy3) (Hunyuan 3.0, Apache-2.0), converted with `mlx-lm` and verified running on Apple Silicon.
+
+Hy3 is a 295B-parameter Mixture-of-Experts model with **21B active parameters**, 80 layers, 192 routed experts (top-8) plus one shared expert, a native MTP (multi-token-prediction) head, and 256K context.
+
+## Why this quant is clean
+
+Naive uniform 4-bit quantization degrades MoE models badly, because the router that picks which experts fire is precision-sensitive. This build follows a mixed-precision recipe:
+
+- **4-bit** (group size 64, affine) for attention and expert weights
+- **8-bit** for every MoE router gate (`*.mlp.router.gate`)
+- MTP head preserved
+
+That router protection is the difference between a coherent model and mush, and it matches the recipe used by the reference `mlx-community/Hy3-preview-4bit`.
+
+## Footprint
+
+- Weights: ~166 GB
+- Fits: two 128GB Apple Silicon machines (or one 192GB+ Mac), does not fit a single 128GB machine at 4-bit.
+
+## How it was made
+
+```python
+from mlx_lm.convert import convert
+
+def hy3_predicate(path, module, config=None):
+    if path.endswith("mlp.router.gate"):
+        return {"group_size": 64, "bits": 8}
+    return {"group_size": 64, "bits": 4}
+
+convert(
+    hf_path="tencent/Hy3",
+    mlx_path="Hy3-4bit",
+    quantize=True, q_bits=4, q_group_size=64, q_mode="affine",
+    quant_predicate=hy3_predicate,
+)
+```
+
+`hy_v3` architecture support comes from [mlx-lm PR #1211](https://github.com/ml-explore/mlx-lm/pull/1211).
+
+## Benchmarks (2x M5 Max, Thunderbolt RDMA)
+
+Measured on two M5 Max (128GB each), pipeline-parallel over Thunderbolt with Apple's `jaccl` RDMA backend (`MLX_METAL_FAST_SYNCH=1`). Single-stream decode.
+
+| Metric | Value |
+|---|---|
+| Decode (generation) | **36.91 tok/s** |
+| Prompt (prefill) | 8.6 tok/s |
+| Peak memory / node | 84.4 GB |
+| Backend | jaccl (RDMA), pipeline-parallel |
+
+Notes: at 4-bit (~166GB) the model does not fit a single 128GB machine, so this is a genuine 2-node cluster run. The `ring`/TCP backend trips the macOS Metal command-buffer watchdog on the cross-node fence wait; RDMA (`jaccl`) is required for stable single-stream decode.
+
+## Credits
+
+- Base model: Tencent Hunyuan (`tencent/Hy3`, Apache-2.0)
+- Architecture support: mlx-lm PR #1211
+- Quantization + Apple Silicon cluster benchmark: bicVanYonk

chat_template.jinja

@@ -0,0 +1,222 @@
+{#- ----------‑‑‑ special token variables ‑‑‑---------- -#}
+{%- set HYTK = ':opensource' %}
+{%- set eos_token = '<｜hy_eos{}｜>'.format(HYTK) %}
+{%- set bos_token = '<｜hy_begin_of_sentence{}｜>'.format(HYTK) %}
+{%- set pad_token = '<｜hy_pad{}｜>'.format(HYTK) %}
+{%- set user_token = '<｜hy_User{}｜>'.format(HYTK) %}
+{%- set assistant_token = '<｜hy_Assistant{}｜>'.format(HYTK) %}
+{%- set think_begin_token = '<think{}>'.format(HYTK) %}
+{%- set think_end_token = '</think{}>'.format(HYTK) %}
+{%- set toolcalls_begin_token = '<tool_calls{}>'.format(HYTK) %}
+{%- set toolcalls_end_token = '</tool_calls{}>'.format(HYTK) %}
+{%- set toolcall_begin_token = '<tool_call{}>'.format(HYTK) %}
+{%- set toolcall_end_token = '</tool_call{}>'.format(HYTK) %}
+{%- set toolsep_token = '<tool_sep{}>'.format(HYTK) %}
+{%- set argkey_begin_token = '<arg_key{}>'.format(HYTK) %}
+{%- set argkey_end_token = '</arg_key{}>'.format(HYTK) %}
+{%- set argvalue_begin_token = '<arg_value{}>'.format(HYTK) %}
+{%- set argvalue_end_token = '</arg_value{}>'.format(HYTK) %}
+{%- set toolresponses_begin_token = '<tool_responses{}>'.format(HYTK) %}
+{%- set toolresponses_end_token = '</tool_responses{}>'.format(HYTK) %}
+{%- set toolresponse_begin_token = '<tool_response{}>'.format(HYTK) %}
+{%- set toolresponse_end_token = '</tool_response{}>'.format(HYTK) %}
+{%- set reasoning_mode_token = '<｜reasoning_mode{}｜>'.format(HYTK) %}
+
+{#- ----------‑‑‑ hyperparameters variables ‑‑‑---------- -#}
+{%- if not add_generation_prompt is defined %}
+    {%- set add_generation_prompt = false %}
+{%- endif %}
+{%- if not preserved_thinking is defined %}
+    {%- if not tools %}
+        {%- set preserved_thinking = false %}
+    {%- else %}
+        {%- set preserved_thinking = true %}
+    {%- endif %}
+{%- endif %}
+{%- if not is_training is defined %}
+    {%- set is_training = false %}
+{%- endif %}
+
+{%- if not reasoning_effort is defined %}
+    {%- set reasoning_effort = 'no_think' %}
+{%- elif reasoning_effort not in ['high', 'low', 'no_think'] %}
+    {%- if reasoning_effort is none %}
+        {{- raise_exception('reasoning_effort error : None, should be no_think/low/high') }}
+    {%- else %}
+        {{- raise_exception('reasoning_effort error : ' + reasoning_effort + ', should be no_think/low/high') }}
+    {%- endif %}
+{%- endif %}
+
+{%- if fallback_strategy is defined and fallback_strategy == 'reasoning_toolcall_retry' %}
+    {%- set reasoning_effort = 'high' %}
+    {%- set add_generation_prompt = false %}
+{%- endif %}
+{%- if not raw_last_assistant is defined %}
+    {%- set raw_last_assistant = false %}
+{%- endif %}
+
+{%- macro visible_text(content) -%}
+    {%- if content is string -%}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping -%}
+        {%- for item in content -%}
+            {%- if item is mapping and item.type == 'text' -%}
+                {{- item.text }}
+            {%- elif item is string -%}
+                {{- item }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- elif content is none -%}
+        {{- '' }}
+    {%- else -%}
+        {{- content }}
+    {%- endif -%}
+{%- endmacro -%}
+
+{%- set ns = namespace(last_user_index=-1) %}
+{%- set sp_ns = namespace(system_prompt='', is_first_sp=true) %}
+{%- for message in messages %}
+    {%- if message['role'] == 'system' %}
+        {%- set sp_ns.system_prompt = sp_ns.system_prompt + visible_text(message['content']) %}
+    {%- endif %}
+    {%- if message['role'] == 'user' %}
+        {%- set ns.last_user_index = loop.index0 %}
+    {%- endif %}
+{%- endfor %}
+{%- if reasoning_effort is defined and reasoning_effort is string and reasoning_effort != '' and not tools %}
+    {%- set sp_ns.system_prompt = sp_ns.system_prompt + reasoning_mode_token + 'reasoning_effort:' + reasoning_effort %}
+{%- endif %}
+{{- bos_token }}
+{{- sp_ns.system_prompt }}
+{%- if tools %}
+    {%- if sp_ns.system_prompt != '' %}
+        {{- '\n\n# Tools\n\nYou may call one or more functions to assist with the user query.' }}
+    {%- else %}
+        {{- '# Tools\n\nYou may call one or more functions to assist with the user query.' }}
+    {%- endif %}
+    {{- '\n\nYou are provided with function signatures within <tools></tools> XML tags:' }}
+    {{- '\n<tools>\n' }}
+    {%- for tool in tools %}
+        {%- if loop.index0 > 0 %}
+            {{- '\n' }}
+        {%- endif %}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- '\n</tools>\n\n' }}
+    {{- 'For function call returns, you should first print ' + toolcalls_begin_token + '\n' }}
+    {{- 'For each function call, you should return object like:\n' }}
+    {{- toolcall_begin_token + '{function-name}' + toolsep_token + '\n' }}
+    {{- argkey_begin_token + '{arg-key-1}' + argkey_end_token + '\n' }}
+    {{- argvalue_begin_token + '{arg-value-1}' + argvalue_end_token + '\n' }}
+    {{- argkey_begin_token + '{arg-key-2}' + argkey_end_token + '\n' }}
+    {{- argvalue_begin_token + '{arg-value-2}' + argvalue_end_token + '\n' }}
+    {{- '...\n' }}
+    {{- toolcall_end_token + '\n' }}
+    {%- if reasoning_effort is defined and reasoning_effort is string and reasoning_effort != '' %}
+        {{- 'At the end of function call returns, you should print ' + toolcalls_end_token + reasoning_mode_token + 'reasoning_effort:' + reasoning_effort }}
+    {%- else %}
+        {{- 'At the end of function call returns, you should print ' + toolcalls_end_token }}
+    {%- endif %}
+{%- endif %}
+
+{%- set prev_ns = namespace(is_tool=false, is_tool_first=true) %}
+{%- set last_ns = namespace(last_is_assistant=false) %}
+{%- for message in messages %}
+    {%- if message['role'] == 'user' %}
+        {%- if prev_ns.is_tool %}
+            {{- toolresponses_end_token }}
+        {%- endif %}
+        {{- user_token + visible_text(message['content']) }}
+        {%- set prev_ns.is_tool = false %}
+    {%- endif %}
+    {%- if message['role'] == 'assistant' %}
+        {%- if is_training %}
+            {%- if 'reasoning_content' in message and message['reasoning_content'] is string %}
+                {%- set rc = message['reasoning_content'] %}
+            {%- elif 'reasoning' in message and message['reasoning'] is string %}
+                {%- set rc = message['reasoning'] %}
+            {%- else %}
+                {%- set rc = none %}
+            {%- endif %}
+            {%- if rc is not none %}
+                {%- set content = think_begin_token + rc + think_end_token + visible_text(message['content']) %}
+            {%- else %}
+                {%- set content = think_begin_token + think_end_token + visible_text(message['content']) %}
+            {%- endif %}
+        {%- else %}
+            {%- if ((preserved_thinking is defined and preserved_thinking) or loop.index0 > ns.last_user_index) %}
+                {%- if 'reasoning_content' in message and message['reasoning_content'] is string %}
+                    {%- set rc = message['reasoning_content'] %}
+                {%- elif 'reasoning' in message and message['reasoning'] is string %}
+                    {%- set rc = message['reasoning'] %}
+                {%- else %}
+                    {%- set rc = none %}
+                {%- endif %}
+                {%- if rc is not none %}
+                    {%- set content = think_begin_token + rc + think_end_token + visible_text(message['content']) %}
+                {%- else %}
+                    {%- set content = think_begin_token + think_end_token + visible_text(message['content']) %}
+                {%- endif %}
+            {%- else %}
+                {%- set content = think_begin_token + think_end_token + visible_text(message['content']) %}
+            {%- endif %}
+        {%- endif %}
+        {%- if prev_ns.is_tool %}
+            {{- toolresponses_end_token }}
+        {%- endif %}
+        {{- assistant_token }}
+        {%- if message['tool_calls'] is defined and message['tool_calls'] %}
+            {%- set prev_ns.is_tool_first = true %}
+            {{- content }}
+            {{- toolcalls_begin_token + '\n' }}
+            {%- for tool in message['tool_calls'] %}
+                {%- set arguments = tool['function']['arguments'] %}
+                {{- toolcall_begin_token + tool['function']['name'] + toolsep_token + '\n' }}
+                {%- for key, value in arguments.items() %}
+                    {{- argkey_begin_token + key + argkey_end_token + '\n' }}
+                    {%- if value is not string %}
+                        {%- set value = value | tojson(ensure_ascii=False) %}
+                    {%- endif %}
+                    {{- argvalue_begin_token + value + argvalue_end_token + '\n' }}
+                {%- endfor %}
+                {{- toolcall_end_token + '\n' }}
+            {%- endfor %}
+            {{- toolcalls_end_token + eos_token }}
+        {%- else %}
+            {%- if loop.last and raw_last_assistant %}
+                {{- visible_text(message['content']) }}
+            {%- elif not loop.last or is_training %}
+                {{- content + eos_token }}
+            {%- else %}
+                {{- content }}
+            {%- endif %}
+        {%- endif %}
+        {%- set prev_ns.is_tool = false %}
+    {%- endif %}
+    {%- if message['role'] == 'tool' %}
+        {%- set prev_ns.is_tool = true %}
+        {%- if prev_ns.is_tool_first %}
+            {{- toolresponses_begin_token + '\n' }}
+            {%- set prev_ns.is_tool_first = false %}
+        {%- endif %}
+        {{- toolresponse_begin_token + '\n' + visible_text(message['content']) + '\n' + toolresponse_end_token + '\n' }}
+    {%- endif %}
+    {%- if loop.last and message['role'] == 'assistant' %}
+        {%- set last_ns.last_is_assistant = true %}
+    {%- endif %}
+
+{%- endfor %}
+{%- if prev_ns.is_tool %}
+    {{- toolresponses_end_token }}
+{%- endif %}
+{%- if add_generation_prompt %}
+    {%- if not last_ns.last_is_assistant %}
+        {%- if reasoning_effort is defined and reasoning_effort in ['low', 'high'] %}
+            {{- assistant_token + think_begin_token }}
+        {%- elif reasoning_effort is defined and reasoning_effort == 'no_think' %}
+            {{- assistant_token + think_begin_token + think_end_token }}
+        {%- else %}
+            {{- assistant_token }}
+        {%- endif %}
+    {%- endif %}
+{%- endif %}

generation_config.json

@@ -0,0 +1,10 @@
+{
+    "bos_token_id": 120000,
+    "do_sample": true,
+    "eos_token_id": 120025,
+    "pad_token_id": 120002,
+    "temperature": 0.9,
+    "top_k": -1,
+    "top_p": 1,
+    "transformers_version": "5.6.0"
+}

model-00001-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:93db49e2c938d02ec216259d1e5f600a4256c4a2f130f19608c7c95c831456fb
+size 5319591251

model-00002-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:145fbc7a1ba5ebea79e748b128e5d29284284109ff6972a7c1b0de971bbb1a40
+size 4864224955

model-00003-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:61e6ccaa612058a5b644ee2fb2abbcff9b3433e632ce5cf3edf2e481d4ffaba6
+size 4918165689

model-00004-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d7e943443b0225d8dc6e71e5389d51523799ea9cfeabc16cb1579a1f3dc82c61
+size 4864224960

model-00005-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3ec450b0cd36244d07c2c6c2a80ea93f86fc8e2b2f2c713a6e4257a8846801e5
+size 4864225040

model-00006-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e84b81cf1656e20a7fad47702b9b3651b8351cfbf017e71e6a6a346d9903065a
+size 4918165735

model-00007-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fcc90c961f9ab8a1f4ca46baeef434e59a3f51840f5c05f9074f4dd574b64085
+size 4864224956

model-00008-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9b3ad75432c855795bab48276cfadd480860aff925e97f6b95d8ebfc6a428836
+size 4864225074

model-00009-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2fde069599aa3aedc1d490ae559d59da4189950e7bd52ad6d9540b3dd4f2ef36
+size 4918165767

model-00010-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a81b5146583c63a8cd9addf8fcf248baa7f84a53743ef9cd7d5d416ed7bbfbaa
+size 4864224972

model-00011-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7eece748e979c85ea0e0849d6a3fbe61060e8cfd5c69218d92b4e65cd83a7348
+size 4864225022

model-00012-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0ab04c2bc1ca49a2a3310d413a7b287b7ba64e9da5450dca9c77af476939734b
+size 4918165825

model-00013-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5cafb057550d9712124e161cb6df755079eb38bad9bb05c81f7f04ae14dd3810
+size 4864225016

model-00014-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:14662e08e488144243219d771774b3a0a9b5fd09943fa9510cbb753f77b00aec
+size 4864225000

model-00015-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:29980a555172eef8bb066dde5be42b7a5b19331531d6761e903ff21a576ef3f0
+size 4918165779

model-00016-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:344cd38d328e4f656ce73ac5be19fde14658e92f67a6d009bc9a515761590c29
+size 4864225006

model-00017-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7648fa638ec31408305b95a428ce56986f94caeb9fd9b0d71bad92bf0c2f8153
+size 4864225042

model-00018-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:80f52b4d59dcf3ea78d3f91d210e88a82fedd46166a99770a267675a8106ff83
+size 4918165767

model-00019-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:87da2121b0df121eac73e3dc641ec9f4ee792cda432beb7f941a03a30e89654f
+size 4864224990

model-00020-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2bc38ca906c51321382251519303b6f225ffdad4a3df4b20941eb7d7b24712a2
+size 4864225000

model-00021-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e0f151a46feab5c88caba0fba4adb3f4091e3de96c56967f74c0284404b1b786
+size 4918165807

model-00022-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c4e4a91bc04049f7dd93c993c3b65767e6f9377349ef3d7ba60befb5c60d288e
+size 4864225044

model-00023-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cb981af947cc9eb65175a79d2fc3faf1d4eeffc92cb569a1721c847b6e59e4de
+size 4864225034

model-00024-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cd76f15ba7b7a4a3a67032529eda41df174bb4efa159bb7e87663654278f2245
+size 4918165781

model-00025-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c892c08214855a01223ddf3dee79c5256b9fd024a5ce9c6b4be846b5bb4da4f5
+size 4864225022

model-00026-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2e5326050ab201096bbcdcea68c56e4afaeea66dabc54026e54bc7e89f48554b
+size 4864225082

model-00027-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7f3f104c63c94659738c71e85ca3c19acd5a95576332aa9e85d6610fa1eb66d3
+size 4918165785

model-00028-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:592170abc29c663ed28d4a0e26d72591947ddce17bac96d219dc26668cf6f232
+size 4864225046

model-00029-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b333f7f02143b7f012d6935a096c010a51a125e6a9171faada83cf79b2aaea9f
+size 4864225076

model-00030-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:440b9d090f87a1d7906cf84a98e0de71a688eeb3c2fe4e9ad8e5742e3e02d731
+size 4918165811

model-00031-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:38fd7fb3021a16b44b8de60041a5aead92dae87629a4198648cabca272c82553
+size 4864225050

model-00032-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b32772653b94c0df8a0f0d4aac5773e9d8ce91368f2a5a18c0aab847d32583d
+size 4864225032

model-00033-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7f017251e0cc516e092c53e7a9c8beaec05c607350a8044727f8bca4ae3f570f
+size 4918165797

model-00034-of-00034.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:abd1527edf60cbb717d1aff415230c0983665bc76b637cbff93e98db0e437b4b
+size 4420683411

tokenizer_config.json

@@ -0,0 +1,12 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<｜hy_begin_of_sentence:opensource｜>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<｜hy_eos:opensource｜>",
+  "is_local": true,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<｜hy_pad:opensource｜>",
+  "token_suffix": ":opensource",
+  "tokenizer_class": "TokenizersBackend",
+  "tool_parser_type": "hy_v3_opensource"
+}