Text Generation
MLX
Safetensors
modilify_mk2
diffusion
mixture-of-experts
custom-code
modilify-mk2
conversational
Instructions to use modilify/Modilify-Mk2-mlx with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use modilify/Modilify-Mk2-mlx with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("modilify/Modilify-Mk2-mlx") prompt = "Write a story about Einstein" messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True ) text = generate(model, tokenizer, prompt=prompt, verbose=True) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use modilify/Modilify-Mk2-mlx with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "modilify/Modilify-Mk2-mlx"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "modilify/Modilify-Mk2-mlx" } ] } } }Run Pi
# Start Pi in your project directory: pi
- MLX LM
How to use modilify/Modilify-Mk2-mlx with MLX LM:
Generate or start a chat session
# Install MLX LM uv tool install mlx-lm # Interactive chat REPL mlx_lm.chat --model "modilify/Modilify-Mk2-mlx"
Run an OpenAI-compatible server
# Install MLX LM uv tool install mlx-lm # Start the server mlx_lm.server --model "modilify/Modilify-Mk2-mlx" # Calling the OpenAI-compatible server with curl curl -X POST "http://localhost:8080/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "modilify/Modilify-Mk2-mlx", "messages": [ {"role": "user", "content": "Hello"} ] }' - Hermes Agent
How to use modilify/Modilify-Mk2-mlx with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "modilify/Modilify-Mk2-mlx"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default modilify/Modilify-Mk2-mlx
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use modilify/Modilify-Mk2-mlx with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "modilify/Modilify-Mk2-mlx"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "modilify/Modilify-Mk2-mlx" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Publish Modilify Mk2 MLX step 4400
Browse files- .gitattributes +1 -34
- .gitignore +7 -0
- LICENSE +167 -0
- LICENSE.apache-2.0 +204 -0
- NOTICE.md +7 -0
- README.md +49 -0
- chat_template.jinja +387 -0
- config.json +114 -0
- export_manifest.json +97 -0
- generation_config.json +48 -0
- inference.py +35 -0
- model-00001-of-00016.safetensors +3 -0
- model-00002-of-00016.safetensors +3 -0
- model-00003-of-00016.safetensors +3 -0
- model-00004-of-00016.safetensors +3 -0
- model-00005-of-00016.safetensors +3 -0
- model-00006-of-00016.safetensors +3 -0
- model-00007-of-00016.safetensors +3 -0
- model-00008-of-00016.safetensors +3 -0
- model-00009-of-00016.safetensors +3 -0
- model-00010-of-00016.safetensors +3 -0
- model-00011-of-00016.safetensors +3 -0
- model-00012-of-00016.safetensors +3 -0
- model-00013-of-00016.safetensors +3 -0
- model-00014-of-00016.safetensors +3 -0
- model-00015-of-00016.safetensors +3 -0
- model-00016-of-00016.safetensors +3 -0
- model.safetensors.index.json +0 -0
- modilify_mk2_mlx/__init__.py +5 -0
- modilify_mk2_mlx/model.py +709 -0
- modilify_mk2_mlx/runtime.py +379 -0
- requirements.txt +5 -0
- tokenizer.json +3 -0
- tokenizer_config.json +96 -0
.gitattributes
CHANGED
|
@@ -1,35 +1,2 @@
|
|
| 1 |
-
*.7z filter=lfs diff=lfs merge=lfs -text
|
| 2 |
-
*.arrow filter=lfs diff=lfs merge=lfs -text
|
| 3 |
-
*.bin filter=lfs diff=lfs merge=lfs -text
|
| 4 |
-
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
| 5 |
-
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
| 6 |
-
*.ftz filter=lfs diff=lfs merge=lfs -text
|
| 7 |
-
*.gz filter=lfs diff=lfs merge=lfs -text
|
| 8 |
-
*.h5 filter=lfs diff=lfs merge=lfs -text
|
| 9 |
-
*.joblib filter=lfs diff=lfs merge=lfs -text
|
| 10 |
-
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
| 11 |
-
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
| 12 |
-
*.model filter=lfs diff=lfs merge=lfs -text
|
| 13 |
-
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
| 14 |
-
*.npy filter=lfs diff=lfs merge=lfs -text
|
| 15 |
-
*.npz filter=lfs diff=lfs merge=lfs -text
|
| 16 |
-
*.onnx filter=lfs diff=lfs merge=lfs -text
|
| 17 |
-
*.ot filter=lfs diff=lfs merge=lfs -text
|
| 18 |
-
*.parquet filter=lfs diff=lfs merge=lfs -text
|
| 19 |
-
*.pb filter=lfs diff=lfs merge=lfs -text
|
| 20 |
-
*.pickle filter=lfs diff=lfs merge=lfs -text
|
| 21 |
-
*.pkl filter=lfs diff=lfs merge=lfs -text
|
| 22 |
-
*.pt filter=lfs diff=lfs merge=lfs -text
|
| 23 |
-
*.pth filter=lfs diff=lfs merge=lfs -text
|
| 24 |
-
*.rar filter=lfs diff=lfs merge=lfs -text
|
| 25 |
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
| 26 |
-
|
| 27 |
-
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
| 28 |
-
*.tar filter=lfs diff=lfs merge=lfs -text
|
| 29 |
-
*.tflite filter=lfs diff=lfs merge=lfs -text
|
| 30 |
-
*.tgz filter=lfs diff=lfs merge=lfs -text
|
| 31 |
-
*.wasm filter=lfs diff=lfs merge=lfs -text
|
| 32 |
-
*.xz filter=lfs diff=lfs merge=lfs -text
|
| 33 |
-
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
-
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
-
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
| 2 |
+
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
.gitignore
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
.DS_Store
|
| 2 |
+
__pycache__/
|
| 3 |
+
*.py[cod]
|
| 4 |
+
.venv/
|
| 5 |
+
.cache/
|
| 6 |
+
.env
|
| 7 |
+
/.upload-fast.sh
|
LICENSE
ADDED
|
@@ -0,0 +1,167 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Modilify Non-Commercial Model License 1.0
|
| 2 |
+
|
| 3 |
+
Copyright 2026 Modilify
|
| 4 |
+
|
| 5 |
+
1. Definitions
|
| 6 |
+
|
| 7 |
+
"Model" means the weights, configuration, inference code, tokenizer, processor,
|
| 8 |
+
architecture definitions, and documentation distributed under or with this license.
|
| 9 |
+
"Derivative Model" means any modified, fine-tuned, distilled, merged, quantized,
|
| 10 |
+
extended, or otherwise adapted version of the Model.
|
| 11 |
+
|
| 12 |
+
"Output" means any text, image, audio, video, 3D asset, latent state, synthetic
|
| 13 |
+
data, or other content generated by executing, querying, or running the Model
|
| 14 |
+
or a Derivative Model.
|
| 15 |
+
|
| 16 |
+
"Non-Commercial Use" means any use, exercise of rights, or deployment conducted
|
| 17 |
+
strictly for personal learning, educational purposes, academic research,
|
| 18 |
+
non-profit scientific investigation, or non-commercial testing and evaluation,
|
| 19 |
+
which does not directly or indirectly generate revenue, commercial advantage,
|
| 20 |
+
or monetary compensation.
|
| 21 |
+
|
| 22 |
+
"You" means the individual or legal entity exercising permissions under this
|
| 23 |
+
license.
|
| 24 |
+
|
| 25 |
+
"High-Risk Use" means a use that can materially affect a person's safety,
|
| 26 |
+
liberty, access to essential services, employment, housing, credit, education,
|
| 27 |
+
legal rights, or medical care, or that controls critical infrastructure,
|
| 28 |
+
weapons, or large-scale biometric surveillance.
|
| 29 |
+
|
| 30 |
+
2. Copyright Grant
|
| 31 |
+
|
| 32 |
+
Subject to the terms and conditions of this license, Modilify grants You a
|
| 33 |
+
worldwide, perpetual, non-exclusive, royalty-free, non-transferable, revocable
|
| 34 |
+
(as provided herein) copyright license to use, reproduce, prepare derivative
|
| 35 |
+
works of, publicly display, and distribute the Model and Derivative Models
|
| 36 |
+
solely for Non-Commercial Use.
|
| 37 |
+
|
| 38 |
+
3. Commercial Restrictions, Outputs, and Distillation
|
| 39 |
+
|
| 40 |
+
a. Prohibition on Commercial Use: You shall not use, deploy, host, serve,
|
| 41 |
+
reproduce, sublicense, or distribute the Model or any Derivative Model for
|
| 42 |
+
any commercial purpose, including but not limited to:
|
| 43 |
+
i. operating the Model or Derivative Models as a paid or revenue-generating
|
| 44 |
+
service (including API or cloud-hosted services);
|
| 45 |
+
ii. incorporating the Model or Derivative Models into commercial products,
|
| 46 |
+
enterprise workflows, or profit-seeking internal operations; or
|
| 47 |
+
iii. selling, leasing, or monetizing the Model or Derivative Models in any
|
| 48 |
+
manner.
|
| 49 |
+
|
| 50 |
+
b. Output Restrictions: You shall not use any Output generated by the Model or
|
| 51 |
+
Derivative Models for direct commercial monetization or commercial products.
|
| 52 |
+
|
| 53 |
+
c. Distillation and Competitive Training Prohibition: You shall not use the Model,
|
| 54 |
+
Derivative Models, or any Outputs generated by them (including synthetic data,
|
| 55 |
+
distillation targets, or reasoning traces) to train, fine-tune, distill,
|
| 56 |
+
align, improve, or evaluate any machine learning model that is intended for
|
| 57 |
+
commercial use or that competes with the Model.
|
| 58 |
+
|
| 59 |
+
d. Commercial Inquiries: Any commercial use of the Model or its derivatives
|
| 60 |
+
requires a separate written commercial agreement from Modilify. Inquiries
|
| 61 |
+
regarding commercial licensing may be submitted through the official project
|
| 62 |
+
repository or Modilify's public channels.
|
| 63 |
+
|
| 64 |
+
4. Patent Grant
|
| 65 |
+
|
| 66 |
+
Each contributor grants You a worldwide, perpetual, non-exclusive, royalty-free,
|
| 67 |
+
irrevocable patent license, except as stated in this section, to make, have made,
|
| 68 |
+
use, and otherwise transfer the Model solely for Non-Commercial Use, where the
|
| 69 |
+
license applies only to those patent claims licensable by that contributor that
|
| 70 |
+
are necessarily infringed by that contributor's contribution alone or in
|
| 71 |
+
combination with the Model. If You institute patent litigation alleging that the
|
| 72 |
+
Model or a contribution constitutes patent infringement, patent licenses granted
|
| 73 |
+
to You under this license terminate as of the filing date.
|
| 74 |
+
|
| 75 |
+
5. Conditions on Redistribution and Share-Alike
|
| 76 |
+
|
| 77 |
+
If You distribute the Model or a Derivative Model, You must:
|
| 78 |
+
|
| 79 |
+
a. provide recipients a copy of this license;
|
| 80 |
+
b. retain all copyright, patent, trademark, and attribution notices;
|
| 81 |
+
c. state clearly that You modified the Model and document material modifications;
|
| 82 |
+
d. Share-Alike Requirement: license and distribute any Derivative Model solely
|
| 83 |
+
under this exact Modilify Non-Commercial Model License 1.0, without imposing
|
| 84 |
+
any additional restrictions or granting any broader rights than those provided
|
| 85 |
+
herein; and
|
| 86 |
+
e. publish with the distributed model a reasonably accessible impact statement
|
| 87 |
+
(such as within a public model card or repository documentation) describing
|
| 88 |
+
intended uses, material limitations, evaluation scope, known safety risks,
|
| 89 |
+
and risk mitigations for the Derivative Model.
|
| 90 |
+
|
| 91 |
+
The impact statement may be maintained in a public model card or equivalent
|
| 92 |
+
document. You are not required to submit it separately to Modilify.
|
| 93 |
+
|
| 94 |
+
6. Responsible Use and High-Risk Uses
|
| 95 |
+
|
| 96 |
+
You must not use the Model or a Derivative Model:
|
| 97 |
+
|
| 98 |
+
a. to develop, operate, or materially facilitate weapons, autonomous targeting,
|
| 99 |
+
or systems intended to cause physical harm;
|
| 100 |
+
b. for unlawful mass surveillance, biometric identification without lawful
|
| 101 |
+
authority and appropriate safeguards, or social scoring that determines
|
| 102 |
+
access to rights or essential services;
|
| 103 |
+
c. to exploit children or vulnerable persons, facilitate human trafficking, or
|
| 104 |
+
generate non-consensual intimate content;
|
| 105 |
+
d. to impersonate a person or deceptively represent machine output as an
|
| 106 |
+
authentic human communication where the deception is reasonably likely to
|
| 107 |
+
cause material harm; or
|
| 108 |
+
e. to make a final decision in a High-Risk Use without meaningful qualified
|
| 109 |
+
human review, proportionate testing, monitoring, appeal or correction paths,
|
| 110 |
+
and compliance with applicable law.
|
| 111 |
+
|
| 112 |
+
Before deploying the Model in a High-Risk Use, You must perform safety and impact
|
| 113 |
+
due diligence proportionate to foreseeable harm. At minimum, document the use
|
| 114 |
+
context, evaluate relevant failure modes and affected groups, apply reasonable
|
| 115 |
+
technical and organizational safeguards, monitor material incidents, and update
|
| 116 |
+
or suspend the deployment when its residual risk is not reasonable. Research,
|
| 117 |
+
testing, auditing, and defensive safety work are permitted when conducted with
|
| 118 |
+
appropriate safeguards.
|
| 119 |
+
|
| 120 |
+
7. Trademarks
|
| 121 |
+
|
| 122 |
+
This license does not grant permission to use the trade names, trademarks,
|
| 123 |
+
service marks, or product names of Modilify or any contributor, except as needed
|
| 124 |
+
for reasonable and customary attribution or to describe the origin of the Model.
|
| 125 |
+
|
| 126 |
+
8. Third-Party Components
|
| 127 |
+
|
| 128 |
+
If the Model incorporates or links to any third-party components or libraries,
|
| 129 |
+
those components remain subject to their respective applicable licenses and
|
| 130 |
+
terms.
|
| 131 |
+
|
| 132 |
+
9. Termination and Reinstatement
|
| 133 |
+
|
| 134 |
+
Your rights terminate automatically if You materially violate this license and
|
| 135 |
+
do not cure the violation within 30 days after becoming aware of it. Rights are
|
| 136 |
+
reinstated upon timely cure unless a rights holder provides written notice of a
|
| 137 |
+
substantially similar repeated violation. Sections intended by their nature to
|
| 138 |
+
survive termination remain effective.
|
| 139 |
+
|
| 140 |
+
10. Disclaimer of Warranty
|
| 141 |
+
|
| 142 |
+
THE MODEL IS PROVIDED "AS IS," WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
|
| 143 |
+
EXPRESS OR IMPLIED, INCLUDING WARRANTIES OF TITLE, NON-INFRINGEMENT,
|
| 144 |
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, ACCURACY, OR SAFETY. YOU ARE
|
| 145 |
+
SOLELY RESPONSIBLE FOR DETERMINING THE APPROPRIATENESS OF USING OR REDISTRIBUTING
|
| 146 |
+
THE MODEL AND ASSUME ALL RISKS ASSOCIATED WITH YOUR EXERCISE OF PERMISSIONS.
|
| 147 |
+
|
| 148 |
+
11. Limitation of Liability
|
| 149 |
+
|
| 150 |
+
TO THE MAXIMUM EXTENT PERMITTED BY LAW, NO COPYRIGHT HOLDER OR CONTRIBUTOR SHALL
|
| 151 |
+
BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
|
| 152 |
+
CONSEQUENTIAL DAMAGES ARISING FROM THIS LICENSE OR THE USE OR INABILITY TO USE
|
| 153 |
+
THE MODEL, HOWEVER CAUSED AND UNDER ANY THEORY OF LIABILITY, EVEN IF ADVISED OF
|
| 154 |
+
THE POSSIBILITY OF SUCH DAMAGES.
|
| 155 |
+
|
| 156 |
+
12. Governing Law
|
| 157 |
+
|
| 158 |
+
To the maximum extent permitted by applicable law, this license is governed by
|
| 159 |
+
and construed in accordance with applicable laws, excluding conflict-of-law
|
| 160 |
+
rules.
|
| 161 |
+
|
| 162 |
+
13. Entire License; Severability
|
| 163 |
+
|
| 164 |
+
This document states the complete Modilify license for the Model. If any
|
| 165 |
+
provision is held to be invalid or unenforceable, that provision will be limited
|
| 166 |
+
or eliminated to the minimum extent necessary, and the remaining provisions
|
| 167 |
+
of this license will remain in full force and effect.
|
LICENSE.apache-2.0
ADDED
|
@@ -0,0 +1,204 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Copyright 2018- The Hugging Face team. All rights reserved.
|
| 2 |
+
|
| 3 |
+
Apache License
|
| 4 |
+
Version 2.0, January 2004
|
| 5 |
+
http://www.apache.org/licenses/
|
| 6 |
+
|
| 7 |
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
| 8 |
+
|
| 9 |
+
1. Definitions.
|
| 10 |
+
|
| 11 |
+
"License" shall mean the terms and conditions for use, reproduction,
|
| 12 |
+
and distribution as defined by Sections 1 through 9 of this document.
|
| 13 |
+
|
| 14 |
+
"Licensor" shall mean the copyright owner or entity authorized by
|
| 15 |
+
the copyright owner that is granting the License.
|
| 16 |
+
|
| 17 |
+
"Legal Entity" shall mean the union of the acting entity and all
|
| 18 |
+
other entities that control, are controlled by, or are under common
|
| 19 |
+
control with that entity. For the purposes of this definition,
|
| 20 |
+
"control" means (i) the power, direct or indirect, to cause the
|
| 21 |
+
direction or management of such entity, whether by contract or
|
| 22 |
+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
| 23 |
+
outstanding shares, or (iii) beneficial ownership of such entity.
|
| 24 |
+
|
| 25 |
+
"You" (or "Your") shall mean an individual or Legal Entity
|
| 26 |
+
exercising permissions granted by this License.
|
| 27 |
+
|
| 28 |
+
"Source" form shall mean the preferred form for making modifications,
|
| 29 |
+
including but not limited to software source code, documentation
|
| 30 |
+
source, and configuration files.
|
| 31 |
+
|
| 32 |
+
"Object" form shall mean any form resulting from mechanical
|
| 33 |
+
transformation or translation of a Source form, including but
|
| 34 |
+
not limited to compiled object code, generated documentation,
|
| 35 |
+
and conversions to other media types.
|
| 36 |
+
|
| 37 |
+
"Work" shall mean the work of authorship, whether in Source or
|
| 38 |
+
Object form, made available under the License, as indicated by a
|
| 39 |
+
copyright notice that is included in or attached to the work
|
| 40 |
+
(an example is provided in the Appendix below).
|
| 41 |
+
|
| 42 |
+
"Derivative Works" shall mean any work, whether in Source or Object
|
| 43 |
+
form, that is based on (or derived from) the Work and for which the
|
| 44 |
+
editorial revisions, annotations, elaborations, or other modifications
|
| 45 |
+
represent, as a whole, an original work of authorship. For the purposes
|
| 46 |
+
of this License, Derivative Works shall not include works that remain
|
| 47 |
+
separable from, or merely link (or bind by name) to the interfaces of,
|
| 48 |
+
the Work and Derivative Works thereof.
|
| 49 |
+
|
| 50 |
+
"Contribution" shall mean any work of authorship, including
|
| 51 |
+
the original version of the Work and any modifications or additions
|
| 52 |
+
to that Work or Derivative Works thereof, that is intentionally
|
| 53 |
+
submitted to Licensor for inclusion in the Work by the copyright owner
|
| 54 |
+
or by an individual or Legal Entity authorized to submit on behalf of
|
| 55 |
+
the copyright owner. For the purposes of this definition, "submitted"
|
| 56 |
+
means any form of electronic, verbal, or written communication sent
|
| 57 |
+
to the Licensor or its representatives, including but not limited to
|
| 58 |
+
communication on electronic mailing lists, source code control systems,
|
| 59 |
+
and issue tracking systems that are managed by, or on behalf of, the
|
| 60 |
+
Licensor for the purpose of discussing and improving the Work, but
|
| 61 |
+
excluding communication that is conspicuously marked or otherwise
|
| 62 |
+
designated in writing by the copyright owner as "Not a Contribution."
|
| 63 |
+
|
| 64 |
+
"Contributor" shall mean Licensor and any individual or Legal Entity
|
| 65 |
+
on behalf of whom a Contribution has been received by Licensor and
|
| 66 |
+
subsequently incorporated within the Work.
|
| 67 |
+
|
| 68 |
+
2. Grant of Copyright License. Subject to the terms and conditions of
|
| 69 |
+
this License, each Contributor hereby grants to You a perpetual,
|
| 70 |
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
| 71 |
+
copyright license to reproduce, prepare Derivative Works of,
|
| 72 |
+
publicly display, publicly perform, sublicense, and distribute the
|
| 73 |
+
Work and such Derivative Works in Source or Object form.
|
| 74 |
+
|
| 75 |
+
3. Grant of Patent License. Subject to the terms and conditions of
|
| 76 |
+
this License, each Contributor hereby grants to You a perpetual,
|
| 77 |
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
| 78 |
+
(except as stated in this section) patent license to make, have made,
|
| 79 |
+
use, offer to sell, sell, import, and otherwise transfer the Work,
|
| 80 |
+
where such license applies only to those patent claims licensable
|
| 81 |
+
by such Contributor that are necessarily infringed by their
|
| 82 |
+
Contribution(s) alone or by combination of their Contribution(s)
|
| 83 |
+
with the Work to which such Contribution(s) was submitted. If You
|
| 84 |
+
institute patent litigation against any entity (including a
|
| 85 |
+
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
| 86 |
+
or a Contribution incorporated within the Work constitutes direct
|
| 87 |
+
or contributory patent infringement, then any patent licenses
|
| 88 |
+
granted to You under this License for that Work shall terminate
|
| 89 |
+
as of the date such litigation is filed.
|
| 90 |
+
|
| 91 |
+
4. Redistribution. You may reproduce and distribute copies of the
|
| 92 |
+
Work or Derivative Works thereof in any medium, with or without
|
| 93 |
+
modifications, and in Source or Object form, provided that You
|
| 94 |
+
meet the following conditions:
|
| 95 |
+
|
| 96 |
+
(a) You must give any other recipients of the Work or
|
| 97 |
+
Derivative Works a copy of this License; and
|
| 98 |
+
|
| 99 |
+
(b) You must cause any modified files to carry prominent notices
|
| 100 |
+
stating that You changed the files; and
|
| 101 |
+
|
| 102 |
+
(c) You must retain, in the Source form of any Derivative Works
|
| 103 |
+
that You distribute, all copyright, patent, trademark, and
|
| 104 |
+
attribution notices from the Source form of the Work,
|
| 105 |
+
excluding those notices that do not pertain to any part of
|
| 106 |
+
the Derivative Works; and
|
| 107 |
+
|
| 108 |
+
(d) If the Work includes a "NOTICE" text file as part of its
|
| 109 |
+
distribution, then any Derivative Works that You distribute must
|
| 110 |
+
include a readable copy of the attribution notices contained
|
| 111 |
+
within such NOTICE file, excluding those notices that do not
|
| 112 |
+
pertain to any part of the Derivative Works, in at least one
|
| 113 |
+
of the following places: within a NOTICE text file distributed
|
| 114 |
+
as part of the Derivative Works; within the Source form or
|
| 115 |
+
documentation, if provided along with the Derivative Works; or,
|
| 116 |
+
within a display generated by the Derivative Works, if and
|
| 117 |
+
wherever such third-party notices normally appear. The contents
|
| 118 |
+
of the NOTICE file are for informational purposes only and
|
| 119 |
+
do not modify the License. You may add Your own attribution
|
| 120 |
+
notices within Derivative Works that You distribute, alongside
|
| 121 |
+
or as an addendum to the NOTICE text from the Work, provided
|
| 122 |
+
that such additional attribution notices cannot be construed
|
| 123 |
+
as modifying the License.
|
| 124 |
+
|
| 125 |
+
You may add Your own copyright statement to Your modifications and
|
| 126 |
+
may provide additional or different license terms and conditions
|
| 127 |
+
for use, reproduction, or distribution of Your modifications, or
|
| 128 |
+
for any such Derivative Works as a whole, provided Your use,
|
| 129 |
+
reproduction, and distribution of the Work otherwise complies with
|
| 130 |
+
the conditions stated in this License.
|
| 131 |
+
|
| 132 |
+
5. Submission of Contributions. Unless You explicitly state otherwise,
|
| 133 |
+
any Contribution intentionally submitted for inclusion in the Work
|
| 134 |
+
by You to the Licensor shall be under the terms and conditions of
|
| 135 |
+
this License, without any additional terms or conditions.
|
| 136 |
+
Notwithstanding the above, nothing herein shall supersede or modify
|
| 137 |
+
the terms of any separate license agreement you may have executed
|
| 138 |
+
with Licensor regarding such Contributions.
|
| 139 |
+
|
| 140 |
+
6. Trademarks. This License does not grant permission to use the trade
|
| 141 |
+
names, trademarks, service marks, or product names of the Licensor,
|
| 142 |
+
except as required for reasonable and customary use in describing the
|
| 143 |
+
origin of the Work and reproducing the content of the NOTICE file.
|
| 144 |
+
|
| 145 |
+
7. Disclaimer of Warranty. Unless required by applicable law or
|
| 146 |
+
agreed to in writing, Licensor provides the Work (and each
|
| 147 |
+
Contributor provides its Contributions) on an "AS IS" BASIS,
|
| 148 |
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
| 149 |
+
implied, including, without limitation, any warranties or conditions
|
| 150 |
+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
| 151 |
+
PARTICULAR PURPOSE. You are solely responsible for determining the
|
| 152 |
+
appropriateness of using or redistributing the Work and assume any
|
| 153 |
+
risks associated with Your exercise of permissions under this License.
|
| 154 |
+
|
| 155 |
+
8. Limitation of Liability. In no event and under no legal theory,
|
| 156 |
+
whether in tort (including negligence), contract, or otherwise,
|
| 157 |
+
unless required by applicable law (such as deliberate and grossly
|
| 158 |
+
negligent acts) or agreed to in writing, shall any Contributor be
|
| 159 |
+
liable to You for damages, including any direct, indirect, special,
|
| 160 |
+
incidental, or consequential damages of any character arising as a
|
| 161 |
+
result of this License or out of the use or inability to use the
|
| 162 |
+
Work (including but not limited to damages for loss of goodwill,
|
| 163 |
+
work stoppage, computer failure or malfunction, or any and all
|
| 164 |
+
other commercial damages or losses), even if such Contributor
|
| 165 |
+
has been advised of the possibility of such damages.
|
| 166 |
+
|
| 167 |
+
9. Accepting Warranty or Additional Liability. While redistributing
|
| 168 |
+
the Work or Derivative Works thereof, You may choose to offer,
|
| 169 |
+
and charge a fee for, acceptance of support, warranty, indemnity,
|
| 170 |
+
or other liability obligations and/or rights consistent with this
|
| 171 |
+
License. However, in accepting such obligations, You may act only
|
| 172 |
+
on Your own behalf and on Your sole responsibility, not on behalf
|
| 173 |
+
of any other Contributor, and only if You agree to indemnify,
|
| 174 |
+
defend, and hold each Contributor harmless for any liability
|
| 175 |
+
incurred by, or claims asserted against, such Contributor by reason
|
| 176 |
+
of your accepting any such warranty or additional liability.
|
| 177 |
+
|
| 178 |
+
END OF TERMS AND CONDITIONS
|
| 179 |
+
|
| 180 |
+
APPENDIX: How to apply the Apache License to your work.
|
| 181 |
+
|
| 182 |
+
To apply the Apache License to your work, attach the following
|
| 183 |
+
boilerplate notice, with the fields enclosed by brackets "[]"
|
| 184 |
+
replaced with your own identifying information. (Don't include
|
| 185 |
+
the brackets!) The text should be enclosed in the appropriate
|
| 186 |
+
comment syntax for the file format. We also recommend that a
|
| 187 |
+
file or class name and description of purpose be included on the
|
| 188 |
+
same "printed page" as the copyright notice for easier
|
| 189 |
+
identification within third-party archives.
|
| 190 |
+
|
| 191 |
+
Copyright [yyyy] [name of copyright owner]
|
| 192 |
+
|
| 193 |
+
Licensed under the Apache License, Version 2.0 (the "License");
|
| 194 |
+
you may not use this file except in compliance with the License.
|
| 195 |
+
You may obtain a copy of the License at
|
| 196 |
+
|
| 197 |
+
http://www.apache.org/licenses/LICENSE-2.0
|
| 198 |
+
|
| 199 |
+
Unless required by applicable law or agreed to in writing, software
|
| 200 |
+
distributed under the License is distributed on an "AS IS" BASIS,
|
| 201 |
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 202 |
+
See the License for the specific language governing permissions and
|
| 203 |
+
limitations under the License.
|
| 204 |
+
|
NOTICE.md
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Attribution
|
| 2 |
+
|
| 3 |
+
Modilify Mk2 uses the text backbone of Google DeepMind's [DiffusionGemma 26B-A4B-it](https://huggingface.co/google/diffusiongemma-26B-A4B-it), released under Apache 2.0. The Apache license text is reproduced in [LICENSE.apache-2.0](LICENSE.apache-2.0).
|
| 4 |
+
|
| 5 |
+
This Apple Silicon inference release uses the weights from Modilify Mk2 step 4400. The Modilify architecture, rolling-canvas generation, trajectory memory, and dense and expert adapters modify the upstream model. Modilify's contributions use the [Modilify Non-Commercial Model License 1.0](LICENSE). Upstream rights and terms continue to apply.
|
| 6 |
+
|
| 7 |
+
The runtime uses Apple MLX, MLX LM, MLX VLM, and Hugging Face Transformers as installed dependencies. These libraries retain their own licenses.
|
README.md
ADDED
|
@@ -0,0 +1,49 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Modilify Mk2 MLX
|
| 2 |
+
|
| 3 |
+
Modilify Mk2 MLX is a self-contained Apple Silicon inference release of Modilify Mk2 step 4400. The text-only DiffusionGemma 26B-A4B backbone revises a rolling 256-token canvas and commits a prefix at each denoising step when its confidence and entropy meet the commit policy. Working GDN2 memory follows canvas revisions. Persistent GDN2 memory updates when tokens are committed.
|
| 4 |
+
|
| 5 |
+
| Property | Value |
|
| 6 |
+
| --- | --- |
|
| 7 |
+
| Architecture | schema25 `compact_gdn2_v2` |
|
| 8 |
+
| Weights | 16 Safetensors shards, about 48.23 GiB |
|
| 9 |
+
| Precision | 1,297 BF16 tensors and 26 FP32 tensors |
|
| 10 |
+
| Runtime | MLX Metal on Apple Silicon |
|
| 11 |
+
|
| 12 |
+
## Run locally
|
| 13 |
+
|
| 14 |
+
Use Python 3.12 on Apple Silicon. Allow unified memory for the weights, prompt cache, rolling state and generation workspace.
|
| 15 |
+
|
| 16 |
+
```bash
|
| 17 |
+
cd ~/Modilify-Mk2-mlx
|
| 18 |
+
python -m pip install -r requirements.txt
|
| 19 |
+
python inference.py --prompt "Why is the sky blue?" --stream
|
| 20 |
+
```
|
| 21 |
+
|
| 22 |
+
The CLI loads the model and tokenizer from the script directory by default. Move the entire directory together to use it elsewhere. `--model /path/to/Modilify-Mk2-mlx` selects a local copy of this release. The loader verifies the manifest and all 16 shard checksums before generation. After dependency installation, loading and inference work offline.
|
| 23 |
+
|
| 24 |
+
`--no-thinking` disables the thinking prompt. `--seed` defaults to 42. `--max-new-tokens` and `--max-denoising-steps` both default to 256. Reaching either limit can leave a response unfinished. Streaming emits text as tokens are committed.
|
| 25 |
+
|
| 26 |
+
The runtime computes router probabilities and expert aggregation in FP32, then returns the aggregate in BF16 before the next backbone operation. Seeds reproduce results within this runtime version. The precision correction can change generated tokens relative to the earlier MLX release.
|
| 27 |
+
|
| 28 |
+
## Python
|
| 29 |
+
|
| 30 |
+
```python
|
| 31 |
+
from modilify_mk2_mlx import load
|
| 32 |
+
|
| 33 |
+
model = load()
|
| 34 |
+
result = model.generate(
|
| 35 |
+
[{"role": "user", "content": "Explain how rainbows form."}],
|
| 36 |
+
enable_thinking=False,
|
| 37 |
+
max_new_tokens=256,
|
| 38 |
+
max_denoising_steps=256,
|
| 39 |
+
seed=42,
|
| 40 |
+
)
|
| 41 |
+
print(result.text)
|
| 42 |
+
print(result.stop_reason, result.denoise_steps)
|
| 43 |
+
```
|
| 44 |
+
|
| 45 |
+
Pass system, user, and assistant messages in conversation order. `result.token_ids` contains the committed response tokens. To stream, pass `on_text=callback`. The callback receives text increments that concatenate to `result.text`. A seed makes requests reproducible in the same MLX environment. The MLX and PyTorch releases use separate random number streams.
|
| 46 |
+
|
| 47 |
+
## License
|
| 48 |
+
|
| 49 |
+
This model is available for non-commercial research and experimentation under the [Modilify Non-Commercial Model License 1.0](LICENSE). See [NOTICE.md](NOTICE.md) for attribution and upstream terms.
|
chat_template.jinja
ADDED
|
@@ -0,0 +1,387 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{#
|
| 2 |
+
Template: Google Gemma 4 Canonical Chat Template
|
| 3 |
+
Author: Google Gemma Engineering Team
|
| 4 |
+
Published: 2026-07-09
|
| 5 |
+
Context: Fixed tool-calling loops, turn closures, and thinking content-ordering.
|
| 6 |
+
#}
|
| 7 |
+
{%- macro format_parameters(properties, required, filter_keys=false) -%}
|
| 8 |
+
{%- set standard_keys = ['description', 'type', 'properties', 'required', 'nullable'] -%}
|
| 9 |
+
{%- set ns = namespace(found_first=false) -%}
|
| 10 |
+
{%- for key, value in properties | dictsort -%}
|
| 11 |
+
{%- set add_comma = false -%}
|
| 12 |
+
{%- if not filter_keys or key not in standard_keys -%}
|
| 13 |
+
{%- if ns.found_first %},{% endif -%}
|
| 14 |
+
{%- set ns.found_first = true -%}
|
| 15 |
+
{{ key }}:{
|
| 16 |
+
{%- if value['description'] -%}
|
| 17 |
+
description:<|"|>{{ value['description'] }}<|"|>
|
| 18 |
+
{%- set add_comma = true -%}
|
| 19 |
+
{%- endif -%}
|
| 20 |
+
{%- if value['type'] | upper == 'STRING' -%}
|
| 21 |
+
{%- if value['enum'] -%}
|
| 22 |
+
{%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}
|
| 23 |
+
enum:{{ format_argument(value['enum']) }}
|
| 24 |
+
{%- endif -%}
|
| 25 |
+
{%- elif value['type'] | upper == 'ARRAY' -%}
|
| 26 |
+
{%- if value['items'] is mapping and value['items'] -%}
|
| 27 |
+
{%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}
|
| 28 |
+
items:{
|
| 29 |
+
{%- set ns_items = namespace(found_first=false) -%}
|
| 30 |
+
{%- for item_key, item_value in value['items'] | dictsort -%}
|
| 31 |
+
{%- if item_value is not none -%}
|
| 32 |
+
{%- if ns_items.found_first %},{% endif -%}
|
| 33 |
+
{%- set ns_items.found_first = true -%}
|
| 34 |
+
{%- if item_key == 'properties' -%}
|
| 35 |
+
properties:{
|
| 36 |
+
{%- if item_value is mapping -%}
|
| 37 |
+
{{- format_parameters(item_value, value['items']['required'] | default([])) -}}
|
| 38 |
+
{%- endif -%}
|
| 39 |
+
}
|
| 40 |
+
{%- elif item_key == 'required' -%}
|
| 41 |
+
required:[
|
| 42 |
+
{%- for req_item in item_value -%}
|
| 43 |
+
<|"|>{{- req_item -}}<|"|>
|
| 44 |
+
{%- if not loop.last %},{% endif -%}
|
| 45 |
+
{%- endfor -%}
|
| 46 |
+
]
|
| 47 |
+
{%- elif item_key == 'type' -%}
|
| 48 |
+
{%- if item_value is string -%}
|
| 49 |
+
type:{{ format_argument(item_value | upper) }}
|
| 50 |
+
{%- else -%}
|
| 51 |
+
type:{{ format_argument(item_value | map('upper') | list) }}
|
| 52 |
+
{%- endif -%}
|
| 53 |
+
{%- else -%}
|
| 54 |
+
{{ item_key }}:{{ format_argument(item_value) }}
|
| 55 |
+
{%- endif -%}
|
| 56 |
+
{%- endif -%}
|
| 57 |
+
{%- endfor -%}
|
| 58 |
+
}
|
| 59 |
+
{%- endif -%}
|
| 60 |
+
{%- endif -%}
|
| 61 |
+
{%- if value['nullable'] %}
|
| 62 |
+
{%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}
|
| 63 |
+
nullable:true
|
| 64 |
+
{%- endif -%}
|
| 65 |
+
{%- if value['type'] | upper == 'OBJECT' -%}
|
| 66 |
+
{%- if value['properties'] is defined and value['properties'] is mapping -%}
|
| 67 |
+
{%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}
|
| 68 |
+
properties:{
|
| 69 |
+
{{- format_parameters(value['properties'], value['required'] | default([])) -}}
|
| 70 |
+
}
|
| 71 |
+
{%- elif value is mapping -%}
|
| 72 |
+
{%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}
|
| 73 |
+
properties:{
|
| 74 |
+
{{- format_parameters(value, value['required'] | default([]), filter_keys=true) -}}
|
| 75 |
+
}
|
| 76 |
+
{%- endif -%}
|
| 77 |
+
{%- if value['required'] -%}
|
| 78 |
+
{%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}
|
| 79 |
+
required:[
|
| 80 |
+
{%- for item in value['required'] | default([]) -%}
|
| 81 |
+
<|"|>{{- item -}}<|"|>
|
| 82 |
+
{%- if not loop.last %},{% endif -%}
|
| 83 |
+
{%- endfor -%}
|
| 84 |
+
]
|
| 85 |
+
{%- endif -%}
|
| 86 |
+
{%- endif -%}
|
| 87 |
+
{%- if add_comma %},{%- else -%} {%- set add_comma = true -%} {% endif -%}
|
| 88 |
+
type:<|"|>{{ value['type'] | upper }}<|"|>}
|
| 89 |
+
{%- endif -%}
|
| 90 |
+
{%- endfor -%}
|
| 91 |
+
{%- endmacro -%}
|
| 92 |
+
{%- macro format_function_declaration(tool_data) -%}
|
| 93 |
+
declaration:{{- tool_data['function']['name'] -}}{description:<|"|>{{- tool_data['function']['description'] -}}<|"|>
|
| 94 |
+
{%- set params = tool_data['function']['parameters'] -%}
|
| 95 |
+
{%- if params -%}
|
| 96 |
+
,parameters:{
|
| 97 |
+
{%- if params['properties'] -%}
|
| 98 |
+
properties:{ {{- format_parameters(params['properties'], params['required']) -}} },
|
| 99 |
+
{%- endif -%}
|
| 100 |
+
{%- if params['required'] -%}
|
| 101 |
+
required:[
|
| 102 |
+
{%- for item in params['required'] -%}
|
| 103 |
+
<|"|>{{- item -}}<|"|>
|
| 104 |
+
{{- ',' if not loop.last -}}
|
| 105 |
+
{%- endfor -%}
|
| 106 |
+
],
|
| 107 |
+
{%- endif -%}
|
| 108 |
+
{%- if params['type'] -%}
|
| 109 |
+
type:<|"|>{{- params['type'] | upper -}}<|"|>}
|
| 110 |
+
{%- endif -%}
|
| 111 |
+
{%- endif -%}
|
| 112 |
+
{%- if 'response' in tool_data['function'] -%}
|
| 113 |
+
{%- set response_declaration = tool_data['function']['response'] -%}
|
| 114 |
+
,response:{
|
| 115 |
+
{%- if response_declaration['description'] -%}
|
| 116 |
+
description:<|"|>{{- response_declaration['description'] -}}<|"|>,
|
| 117 |
+
{%- endif -%}
|
| 118 |
+
{%- if response_declaration['type'] | upper == 'OBJECT' -%}
|
| 119 |
+
type:<|"|>{{- response_declaration['type'] | upper -}}<|"|>}
|
| 120 |
+
{%- endif -%}
|
| 121 |
+
{%- endif -%}
|
| 122 |
+
}
|
| 123 |
+
{%- endmacro -%}
|
| 124 |
+
{%- macro format_argument(argument, escape_keys=True) -%}
|
| 125 |
+
{%- if argument is none -%}
|
| 126 |
+
{{- 'null' -}}
|
| 127 |
+
{%- elif argument is string -%}
|
| 128 |
+
{{- '<|"|>' + argument + '<|"|>' -}}
|
| 129 |
+
{%- elif argument is boolean -%}
|
| 130 |
+
{{- 'true' if argument else 'false' -}}
|
| 131 |
+
{%- elif argument is mapping -%}
|
| 132 |
+
{{- '{' -}}
|
| 133 |
+
{%- set ns = namespace(found_first=false) -%}
|
| 134 |
+
{%- for key, value in argument | dictsort -%}
|
| 135 |
+
{%- if ns.found_first %},{% endif -%}
|
| 136 |
+
{%- set ns.found_first = true -%}
|
| 137 |
+
{%- if escape_keys -%}
|
| 138 |
+
{{- '<|"|>' + key + '<|"|>' -}}
|
| 139 |
+
{%- else -%}
|
| 140 |
+
{{- key -}}
|
| 141 |
+
{%- endif -%}
|
| 142 |
+
:{{- format_argument(value, escape_keys=escape_keys) -}}
|
| 143 |
+
{%- endfor -%}
|
| 144 |
+
{{- '}' -}}
|
| 145 |
+
{%- elif argument is sequence -%}
|
| 146 |
+
{{- '[' -}}
|
| 147 |
+
{%- for item in argument -%}
|
| 148 |
+
{{- format_argument(item, escape_keys=escape_keys) -}}
|
| 149 |
+
{%- if not loop.last %},{% endif -%}
|
| 150 |
+
{%- endfor -%}
|
| 151 |
+
{{- ']' -}}
|
| 152 |
+
{%- else -%}
|
| 153 |
+
{{- argument -}}
|
| 154 |
+
{%- endif -%}
|
| 155 |
+
{%- endmacro -%}
|
| 156 |
+
{%- macro strip_thinking(text) -%}
|
| 157 |
+
{%- set ns = namespace(result='') -%}
|
| 158 |
+
{%- for part in text.split('<channel|>') -%}
|
| 159 |
+
{%- if '<|channel>' in part -%}
|
| 160 |
+
{%- set ns.result = ns.result + part.split('<|channel>')[0] -%}
|
| 161 |
+
{%- else -%}
|
| 162 |
+
{%- set ns.result = ns.result + part -%}
|
| 163 |
+
{%- endif -%}
|
| 164 |
+
{%- endfor -%}
|
| 165 |
+
{{- ns.result | trim -}}
|
| 166 |
+
{%- endmacro -%}
|
| 167 |
+
|
| 168 |
+
{%- macro format_tool_response_block(tool_name, response) -%}
|
| 169 |
+
{{- '<|tool_response>' -}}
|
| 170 |
+
{%- if response is mapping -%}
|
| 171 |
+
{{- 'response:' + tool_name + '{' -}}
|
| 172 |
+
{%- for key, value in response | dictsort -%}
|
| 173 |
+
{{- key -}}:{{- format_argument(value, escape_keys=False) -}}
|
| 174 |
+
{%- if not loop.last %},{% endif -%}
|
| 175 |
+
{%- endfor -%}
|
| 176 |
+
{{- '}' -}}
|
| 177 |
+
{%- else -%}
|
| 178 |
+
{{- 'response:' + tool_name + '{value:' + format_argument(response, escape_keys=False) + '}' -}}
|
| 179 |
+
{%- endif -%}
|
| 180 |
+
{{- '<tool_response|>' -}}
|
| 181 |
+
{%- endmacro -%}
|
| 182 |
+
|
| 183 |
+
{#- ===== SETUP ===== -#}
|
| 184 |
+
{%- set ns = namespace(prev_message_type=None, prev_non_tool_role=None) -%}
|
| 185 |
+
{%- set loop_messages = messages -%}
|
| 186 |
+
{%- set enable_thinking = enable_thinking | default(false) -%}
|
| 187 |
+
{%- set preserve_thinking = preserve_thinking | default(false) -%}
|
| 188 |
+
{{- bos_token -}}
|
| 189 |
+
{#- Handle System/Tool Definitions Block -#}
|
| 190 |
+
{%- if enable_thinking or tools or (messages and messages[0]['role'] in ['system', 'developer']) -%}
|
| 191 |
+
{{- '<|turn>system\n' -}}
|
| 192 |
+
{#- Inject Thinking token at the very top of the FIRST system turn -#}
|
| 193 |
+
{%- if enable_thinking -%}
|
| 194 |
+
{{- '<|think|>\n' -}}
|
| 195 |
+
{%- set ns.prev_message_type = 'think' -%}
|
| 196 |
+
{%- endif -%}
|
| 197 |
+
{%- if messages and messages[0]['role'] in ['system', 'developer'] -%}
|
| 198 |
+
{%- if messages[0]['content'] is string -%}
|
| 199 |
+
{{- messages[0]['content'] | trim -}}
|
| 200 |
+
{%- elif messages[0]['content'] is sequence -%}
|
| 201 |
+
{%- for item in messages[0]['content'] -%}
|
| 202 |
+
{{- item['text'] | trim + ' '-}}
|
| 203 |
+
{%- endfor -%}
|
| 204 |
+
{%- endif -%}
|
| 205 |
+
{%- set loop_messages = messages[1:] -%}
|
| 206 |
+
{%- endif -%}
|
| 207 |
+
{%- if tools -%}
|
| 208 |
+
{%- for tool in tools %}
|
| 209 |
+
{{- '<|tool>' -}}
|
| 210 |
+
{{- format_function_declaration(tool) | trim -}}
|
| 211 |
+
{{- '<tool|>' -}}
|
| 212 |
+
{%- endfor %}
|
| 213 |
+
{%- set ns.prev_message_type = 'tool' -%}
|
| 214 |
+
{%- endif -%}
|
| 215 |
+
{{- '<turn|>\n' -}}
|
| 216 |
+
{%- endif %}
|
| 217 |
+
|
| 218 |
+
{#- Pre-scan: find last user message index for reasoning guard -#}
|
| 219 |
+
{%- set ns_turn = namespace(last_user_idx=-1) -%}
|
| 220 |
+
{%- for i in range(loop_messages | length) -%}
|
| 221 |
+
{%- if loop_messages[i]['role'] == 'user' -%}
|
| 222 |
+
{%- set ns_turn.last_user_idx = i -%}
|
| 223 |
+
{%- endif -%}
|
| 224 |
+
{%- endfor -%}
|
| 225 |
+
|
| 226 |
+
{#- Loop through messages -#}
|
| 227 |
+
{%- for message in loop_messages -%}
|
| 228 |
+
{%- if message['role'] != 'tool' -%}
|
| 229 |
+
{%- set ns.prev_message_type = None -%}
|
| 230 |
+
{%- set role = 'model' if message['role'] == 'assistant' else message['role'] -%}
|
| 231 |
+
{#- Detect continuation using tracked state — O(1) instead of O(n) backward scan -#}
|
| 232 |
+
{%- set continue_same_model_turn = (role == 'model' and ns.prev_non_tool_role == 'assistant') -%}
|
| 233 |
+
{%- if not continue_same_model_turn -%}
|
| 234 |
+
{{- '<|turn>' + role + '\n' }}
|
| 235 |
+
|
| 236 |
+
{%- endif -%}
|
| 237 |
+
|
| 238 |
+
{#- Render reasoning/reasoning_content as thinking channel -#}
|
| 239 |
+
{%- set thinking_text = message.get('reasoning') or message.get('reasoning_content') -%}
|
| 240 |
+
{%- set thinking_gate = (loop.index0 > ns_turn.last_user_idx) or (preserve_thinking and message.get('tool_calls')) -%}
|
| 241 |
+
{%- if thinking_text and thinking_gate -%}
|
| 242 |
+
{{- '<|channel>thought\n' + thinking_text + '\n<channel|>' -}}
|
| 243 |
+
{%- endif -%}
|
| 244 |
+
|
| 245 |
+
{%- if message.get('tool_calls') -%}
|
| 246 |
+
{%- for tool_call in message.get('tool_calls') -%}
|
| 247 |
+
{%- set function = tool_call['function'] -%}
|
| 248 |
+
{{- '<|tool_call>call:' + function['name'] + '{' -}}
|
| 249 |
+
{%- if function['arguments'] is mapping -%}
|
| 250 |
+
{%- set ns_args = namespace(found_first=false) -%}
|
| 251 |
+
{%- for key, value in function['arguments'] | dictsort -%}
|
| 252 |
+
{%- if ns_args.found_first %},{% endif -%}
|
| 253 |
+
{%- set ns_args.found_first = true -%}
|
| 254 |
+
{{- key -}}:{{- format_argument(value, escape_keys=False) -}}
|
| 255 |
+
{%- endfor -%}
|
| 256 |
+
{%- elif function['arguments'] is none -%}
|
| 257 |
+
{%- else -%}
|
| 258 |
+
{{- raise_exception(
|
| 259 |
+
"chat_template: tool_calls[].function.arguments must be a "
|
| 260 |
+
"JSON object (mapping), not a string. Deserialize arguments "
|
| 261 |
+
"before passing to the template."
|
| 262 |
+
) -}}
|
| 263 |
+
{%- endif -%}
|
| 264 |
+
{{- '}<tool_call|>' -}}
|
| 265 |
+
{%- endfor -%}
|
| 266 |
+
{%- set ns.prev_message_type = 'tool_call' -%}
|
| 267 |
+
{%- endif -%}
|
| 268 |
+
|
| 269 |
+
{%- set ns_tr_out = namespace(flag=false) -%}
|
| 270 |
+
{%- if message.get('tool_responses') -%}
|
| 271 |
+
{#- Legacy: tool_responses embedded on the assistant message (Google/Gemma native) -#}
|
| 272 |
+
{%- for tool_response in message.get('tool_responses') -%}
|
| 273 |
+
{{- format_tool_response_block(tool_response['name'] | default('unknown', true), tool_response['response']) -}}
|
| 274 |
+
{%- set ns_tr_out.flag = true -%}
|
| 275 |
+
{%- set ns.prev_message_type = 'tool_response' -%}
|
| 276 |
+
{%- endfor -%}
|
| 277 |
+
{%- elif message.get('tool_calls') -%}
|
| 278 |
+
{#- OpenAI Chat Completions: forward-scan consecutive role:tool messages -#}
|
| 279 |
+
{%- set ns_tool_scan = namespace(stopped=false) -%}
|
| 280 |
+
{%- for k in range(loop.index0 + 1, loop_messages | length) -%}
|
| 281 |
+
{%- if ns_tool_scan.stopped -%}
|
| 282 |
+
{%- elif loop_messages[k]['role'] != 'tool' -%}
|
| 283 |
+
{%- set ns_tool_scan.stopped = true -%}
|
| 284 |
+
{%- else -%}
|
| 285 |
+
{%- set follow = loop_messages[k] -%}
|
| 286 |
+
{#- Resolve tool_call_id to function name -#}
|
| 287 |
+
{%- set ns_tname = namespace(name=follow.get('name') or 'unknown') -%}
|
| 288 |
+
{%- for tc in message.get('tool_calls') -%}
|
| 289 |
+
{%- if tc.get('id') == follow.get('tool_call_id') -%}
|
| 290 |
+
{%- set ns_tname.name = tc['function']['name'] -%}
|
| 291 |
+
{%- endif -%}
|
| 292 |
+
{%- endfor -%}
|
| 293 |
+
{#- Handle content as string or content-parts array -#}
|
| 294 |
+
{%- set tool_body = follow.get('content') -%}
|
| 295 |
+
{%- if tool_body is string -%}
|
| 296 |
+
{{- format_tool_response_block(ns_tname.name, tool_body) -}}
|
| 297 |
+
{%- elif tool_body is sequence and tool_body is not string -%}
|
| 298 |
+
{%- set ns_txt = namespace(s='') -%}
|
| 299 |
+
{%- for part in tool_body -%}
|
| 300 |
+
{%- if part.get('type') == 'text' -%}
|
| 301 |
+
{%- set ns_txt.s = ns_txt.s + (part.get('text') | default('')) -%}
|
| 302 |
+
{%- endif -%}
|
| 303 |
+
{%- endfor -%}
|
| 304 |
+
{{- format_tool_response_block(ns_tname.name, ns_txt.s) -}}
|
| 305 |
+
{%- for part in tool_body -%}
|
| 306 |
+
{%- if part.get('type') in ['image', 'image_url'] -%}
|
| 307 |
+
{{- '<|image|>' -}}
|
| 308 |
+
{%- elif part.get('type') in ['audio', 'input_audio'] -%}
|
| 309 |
+
{{- '<|audio|>' -}}
|
| 310 |
+
{%- elif part.get('type') == 'video' -%}
|
| 311 |
+
{{- '<|video|>' -}}
|
| 312 |
+
{%- endif -%}
|
| 313 |
+
{%- endfor -%}
|
| 314 |
+
{%- else -%}
|
| 315 |
+
{{- format_tool_response_block(ns_tname.name, tool_body) -}}
|
| 316 |
+
{%- endif -%}
|
| 317 |
+
{%- set ns_tr_out.flag = true -%}
|
| 318 |
+
{%- set ns.prev_message_type = 'tool_response' -%}
|
| 319 |
+
{%- endif -%}
|
| 320 |
+
{%- endfor -%}
|
| 321 |
+
{%- endif -%}
|
| 322 |
+
|
| 323 |
+
{%- set captured_content -%}
|
| 324 |
+
{%- if message.get('content') is string -%}
|
| 325 |
+
{%- if role == 'model' -%}
|
| 326 |
+
{{- strip_thinking(message['content']) -}}
|
| 327 |
+
{%- else -%}
|
| 328 |
+
{{- message['content'] | trim -}}
|
| 329 |
+
{%- endif -%}
|
| 330 |
+
{%- elif message.get('content') is sequence -%}
|
| 331 |
+
{%- for item in message['content'] -%}
|
| 332 |
+
{%- if item.get('type') == 'text' -%}
|
| 333 |
+
{%- if role == 'model' -%}
|
| 334 |
+
{{- strip_thinking(item['text']) -}}
|
| 335 |
+
{%- else -%}
|
| 336 |
+
{{- item['text'] | trim -}}
|
| 337 |
+
{%- endif -%}
|
| 338 |
+
{%- elif item.get('type') in ['image', 'image_url'] -%}
|
| 339 |
+
{{- '<|image|>' -}}
|
| 340 |
+
{%- elif item.get('type') in ['audio', 'input_audio'] -%}
|
| 341 |
+
{{- '<|audio|>' -}}
|
| 342 |
+
{%- elif item.get('type') == 'video' -%}
|
| 343 |
+
{{- '<|video|>' -}}
|
| 344 |
+
{%- endif -%}
|
| 345 |
+
{%- endfor -%}
|
| 346 |
+
{%- endif -%}
|
| 347 |
+
{%- endset -%}
|
| 348 |
+
|
| 349 |
+
{{- captured_content -}}
|
| 350 |
+
{%- set has_content = captured_content | trim | length > 0 -%}
|
| 351 |
+
|
| 352 |
+
{#- Forward-scan: find next non-tool message role for continuation detection -#}
|
| 353 |
+
{%- set next_nt = namespace(role=None, found=false) -%}
|
| 354 |
+
{%- for j in range(loop.index0 + 1, loop_messages | length) -%}
|
| 355 |
+
{%- if not next_nt.found -%}
|
| 356 |
+
{%- if loop_messages[j]['role'] != 'tool' -%}
|
| 357 |
+
{%- set next_nt.role = loop_messages[j]['role'] -%}
|
| 358 |
+
{%- set next_nt.found = true -%}
|
| 359 |
+
{%- endif -%}
|
| 360 |
+
{%- endif -%}
|
| 361 |
+
{%- endfor -%}
|
| 362 |
+
|
| 363 |
+
{%- set continues_into_next = (
|
| 364 |
+
role == 'model'
|
| 365 |
+
and next_nt.role == 'assistant'
|
| 366 |
+
and (not message.get('tool_calls') or ns_tr_out.flag)
|
| 367 |
+
) -%}
|
| 368 |
+
|
| 369 |
+
{%- if ns.prev_message_type == 'tool_call' and not ns_tr_out.flag -%}
|
| 370 |
+
{{- '<|tool_response>' -}}
|
| 371 |
+
{%- elif continues_into_next -%}
|
| 372 |
+
{%- elif not (ns_tr_out.flag and not has_content and not next_nt.found) -%}
|
| 373 |
+
{{- '<turn|>\n' -}}
|
| 374 |
+
{%- endif -%}
|
| 375 |
+
{%- endif -%}
|
| 376 |
+
|
| 377 |
+
{#- Track previous non-tool role for next iteration (avoids O(n) backward scan) -#}
|
| 378 |
+
{%- set ns.prev_non_tool_role = message['role'] -%}
|
| 379 |
+
{%- endfor -%}
|
| 380 |
+
|
| 381 |
+
{%- if add_generation_prompt -%}
|
| 382 |
+
{%- if ns.prev_message_type != 'tool_response' and ns.prev_message_type != 'tool_call' -%}
|
| 383 |
+
{{- '<|turn>model\n' -}}
|
| 384 |
+
{%- elif ns.prev_message_type == 'tool_response' and enable_thinking -%}
|
| 385 |
+
{{- '<|channel>thought\n' -}}
|
| 386 |
+
{%- endif -%}
|
| 387 |
+
{%- endif -%}
|
config.json
ADDED
|
@@ -0,0 +1,114 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"canvas_length": 256,
|
| 3 |
+
"checkpoint_step": 4400,
|
| 4 |
+
"commit_confidence_power": 1,
|
| 5 |
+
"commit_entropy_weight": 1,
|
| 6 |
+
"commit_failure_budget": 0.2,
|
| 7 |
+
"commit_min_p": 0.05,
|
| 8 |
+
"commit_sequence_dim": 1024,
|
| 9 |
+
"commit_target_confidence": 0.5,
|
| 10 |
+
"commit_top_k": 40,
|
| 11 |
+
"dtype": "bfloat16",
|
| 12 |
+
"latent_dim": 2816,
|
| 13 |
+
"latent_ffn_dim": 7168,
|
| 14 |
+
"latent_history_kv_rank": 1024,
|
| 15 |
+
"latent_local_attention_window": 128,
|
| 16 |
+
"latent_num_heads": 16,
|
| 17 |
+
"latent_num_layers": 4,
|
| 18 |
+
"latent_tape_probes": 4,
|
| 19 |
+
"latent_working_last_block_global": true,
|
| 20 |
+
"lora_config": {
|
| 21 |
+
"alpha": 16,
|
| 22 |
+
"expert_alpha": 8,
|
| 23 |
+
"expert_r": 8,
|
| 24 |
+
"r": 16,
|
| 25 |
+
"target_modules": [
|
| 26 |
+
"q_proj",
|
| 27 |
+
"k_proj",
|
| 28 |
+
"v_proj",
|
| 29 |
+
"o_proj",
|
| 30 |
+
"gate_proj",
|
| 31 |
+
"up_proj",
|
| 32 |
+
"down_proj",
|
| 33 |
+
"proj"
|
| 34 |
+
]
|
| 35 |
+
},
|
| 36 |
+
"memory_architecture": "compact_gdn2_v2",
|
| 37 |
+
"memory_scheme": "dual_timescale_gdn2_trajectory_memory",
|
| 38 |
+
"model_type": "modilify_mk2",
|
| 39 |
+
"persistent_memory_bus": true,
|
| 40 |
+
"state_schema_version": 25,
|
| 41 |
+
"text_config": {
|
| 42 |
+
"attention_bias": false,
|
| 43 |
+
"attention_dropout": 0,
|
| 44 |
+
"bos_token_id": 2,
|
| 45 |
+
"dtype": "bfloat16",
|
| 46 |
+
"eos_token_id": 1,
|
| 47 |
+
"final_logit_softcapping": 30,
|
| 48 |
+
"global_head_dim": 512,
|
| 49 |
+
"head_dim": 256,
|
| 50 |
+
"hidden_activation": "gelu_pytorch_tanh",
|
| 51 |
+
"hidden_size": 2816,
|
| 52 |
+
"initializer_range": 0.02,
|
| 53 |
+
"intermediate_size": 2112,
|
| 54 |
+
"layer_types": [
|
| 55 |
+
"sliding_attention",
|
| 56 |
+
"sliding_attention",
|
| 57 |
+
"sliding_attention",
|
| 58 |
+
"sliding_attention",
|
| 59 |
+
"sliding_attention",
|
| 60 |
+
"full_attention",
|
| 61 |
+
"sliding_attention",
|
| 62 |
+
"sliding_attention",
|
| 63 |
+
"sliding_attention",
|
| 64 |
+
"sliding_attention",
|
| 65 |
+
"sliding_attention",
|
| 66 |
+
"full_attention",
|
| 67 |
+
"sliding_attention",
|
| 68 |
+
"sliding_attention",
|
| 69 |
+
"sliding_attention",
|
| 70 |
+
"sliding_attention",
|
| 71 |
+
"sliding_attention",
|
| 72 |
+
"full_attention",
|
| 73 |
+
"sliding_attention",
|
| 74 |
+
"sliding_attention",
|
| 75 |
+
"sliding_attention",
|
| 76 |
+
"sliding_attention",
|
| 77 |
+
"sliding_attention",
|
| 78 |
+
"full_attention",
|
| 79 |
+
"sliding_attention",
|
| 80 |
+
"sliding_attention",
|
| 81 |
+
"sliding_attention",
|
| 82 |
+
"sliding_attention",
|
| 83 |
+
"sliding_attention",
|
| 84 |
+
"full_attention"
|
| 85 |
+
],
|
| 86 |
+
"max_position_embeddings": 262144,
|
| 87 |
+
"model_type": "modilify_mk2_text",
|
| 88 |
+
"moe_intermediate_size": 704,
|
| 89 |
+
"num_attention_heads": 16,
|
| 90 |
+
"num_experts": 128,
|
| 91 |
+
"num_global_key_value_heads": 2,
|
| 92 |
+
"num_hidden_layers": 30,
|
| 93 |
+
"num_key_value_heads": 8,
|
| 94 |
+
"pad_token_id": 0,
|
| 95 |
+
"rms_norm_eps": 0.000001,
|
| 96 |
+
"rope_parameters": {
|
| 97 |
+
"full_attention": {
|
| 98 |
+
"partial_rotary_factor": 0.25,
|
| 99 |
+
"rope_theta": 1000000,
|
| 100 |
+
"rope_type": "proportional"
|
| 101 |
+
},
|
| 102 |
+
"sliding_attention": {
|
| 103 |
+
"rope_theta": 10000,
|
| 104 |
+
"rope_type": "default"
|
| 105 |
+
}
|
| 106 |
+
},
|
| 107 |
+
"sliding_window": 1024,
|
| 108 |
+
"tie_word_embeddings": true,
|
| 109 |
+
"top_k_experts": 8,
|
| 110 |
+
"use_bidirectional_attention": null,
|
| 111 |
+
"vocab_size": 262144
|
| 112 |
+
},
|
| 113 |
+
"working_memory_bus": true
|
| 114 |
+
}
|
export_manifest.json
ADDED
|
@@ -0,0 +1,97 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"global_step": 4400,
|
| 3 |
+
"state_schema_version": 25,
|
| 4 |
+
"memory_scheme": "dual_timescale_gdn2_trajectory_memory",
|
| 5 |
+
"memory_architecture": "compact_gdn2_v2",
|
| 6 |
+
"base_model": "google/diffusiongemma-26B-A4B-it",
|
| 7 |
+
"precision_policy": "gdn2_small_fp32_v1",
|
| 8 |
+
"total_tensor_count": 1323,
|
| 9 |
+
"shards": [
|
| 10 |
+
{
|
| 11 |
+
"file": "model-00001-of-00016.safetensors",
|
| 12 |
+
"bytes": 3898256004,
|
| 13 |
+
"sha256": "d3aa3fc0393a51acec27161569d566774030e0306a74748dd2a7a11fd8b535ec"
|
| 14 |
+
},
|
| 15 |
+
{
|
| 16 |
+
"file": "model-00002-of-00016.safetensors",
|
| 17 |
+
"bytes": 3798073636,
|
| 18 |
+
"sha256": "89339fd6f0521977fff9d4ae498e485ff409dbb8716cc924c8667c8bceaa19cd"
|
| 19 |
+
},
|
| 20 |
+
{
|
| 21 |
+
"file": "model-00003-of-00016.safetensors",
|
| 22 |
+
"bytes": 3290562732,
|
| 23 |
+
"sha256": "b9351e59c7c604b26712960b5d9f3460f5c1aace9ccb348999a2334d93fb613d"
|
| 24 |
+
},
|
| 25 |
+
{
|
| 26 |
+
"file": "model-00004-of-00016.safetensors",
|
| 27 |
+
"bytes": 3319472948,
|
| 28 |
+
"sha256": "1c89ba68a1f56e986dbf43175a7a7a6f2d046b04ac993c2837024f71b5d5923d"
|
| 29 |
+
},
|
| 30 |
+
{
|
| 31 |
+
"file": "model-00005-of-00016.safetensors",
|
| 32 |
+
"bytes": 3290562732,
|
| 33 |
+
"sha256": "4e42f45a5c409649c0ac8a262b26a0f644d10619d5f857568db855d12a865183"
|
| 34 |
+
},
|
| 35 |
+
{
|
| 36 |
+
"file": "model-00006-of-00016.safetensors",
|
| 37 |
+
"bytes": 3290562772,
|
| 38 |
+
"sha256": "da6e7d99f9801cb1b7007044f3baa66e767412d4350211a637174321eacee401"
|
| 39 |
+
},
|
| 40 |
+
{
|
| 41 |
+
"file": "model-00007-of-00016.safetensors",
|
| 42 |
+
"bytes": 3319473028,
|
| 43 |
+
"sha256": "ce1f462e324e162f9c0f41b1db3945290eb06c520da3d8ff0f582cde79a6fa96"
|
| 44 |
+
},
|
| 45 |
+
{
|
| 46 |
+
"file": "model-00008-of-00016.safetensors",
|
| 47 |
+
"bytes": 3290562812,
|
| 48 |
+
"sha256": "a163f6df13b9a3fbe1df187e42700b74d71fa08b331c521be568f71dec77a8f0"
|
| 49 |
+
},
|
| 50 |
+
{
|
| 51 |
+
"file": "model-00009-of-00016.safetensors",
|
| 52 |
+
"bytes": 3290562812,
|
| 53 |
+
"sha256": "eacc2e9ac62fa45f4fd1d391bfff490e327c19df3f70e7835e556287fc9926bd"
|
| 54 |
+
},
|
| 55 |
+
{
|
| 56 |
+
"file": "model-00010-of-00016.safetensors",
|
| 57 |
+
"bytes": 3319473028,
|
| 58 |
+
"sha256": "5fa9c8080a9c3954985226496dee261938f3d05cf81e4436932de593e5a01ae1"
|
| 59 |
+
},
|
| 60 |
+
{
|
| 61 |
+
"file": "model-00011-of-00016.safetensors",
|
| 62 |
+
"bytes": 3290562812,
|
| 63 |
+
"sha256": "7df1df1cea73133a94aec88056e2514e04a3467d42c0dd55ad0b922af2e5e986"
|
| 64 |
+
},
|
| 65 |
+
{
|
| 66 |
+
"file": "model-00012-of-00016.safetensors",
|
| 67 |
+
"bytes": 3290562812,
|
| 68 |
+
"sha256": "797078b4703d602cc3a323dba676fa38bb959b70d455bdc8c673663261eee2d4"
|
| 69 |
+
},
|
| 70 |
+
{
|
| 71 |
+
"file": "model-00013-of-00016.safetensors",
|
| 72 |
+
"bytes": 3319473028,
|
| 73 |
+
"sha256": "35923e8f76bc78ae89b3f83d1cdd175649f64cc898f1862baf543d7cdfbcd0e1"
|
| 74 |
+
},
|
| 75 |
+
{
|
| 76 |
+
"file": "model-00014-of-00016.safetensors",
|
| 77 |
+
"bytes": 3290562812,
|
| 78 |
+
"sha256": "105ceea4ed17c811fb79036c199db1f7d2392a5cc4c05003085f7217e8a938bb"
|
| 79 |
+
},
|
| 80 |
+
{
|
| 81 |
+
"file": "model-00015-of-00016.safetensors",
|
| 82 |
+
"bytes": 3290562812,
|
| 83 |
+
"sha256": "d3569ec55bc4f7e89de00efaedfbca3df44096526291731ec14bd7fc245c3b39"
|
| 84 |
+
},
|
| 85 |
+
{
|
| 86 |
+
"file": "model-00016-of-00016.safetensors",
|
| 87 |
+
"bytes": 1202380654,
|
| 88 |
+
"sha256": "23c9c00fc04b475c744458f498178a6e9a20fd673c36af981dd22e963b7ca49b"
|
| 89 |
+
}
|
| 90 |
+
],
|
| 91 |
+
"format": "modilify_mk2_mlx_inference_v1",
|
| 92 |
+
"source": "Modilify/Modilify-Mk2",
|
| 93 |
+
"tensor_dtypes": {
|
| 94 |
+
"bfloat16": 1297,
|
| 95 |
+
"float32": 26
|
| 96 |
+
}
|
| 97 |
+
}
|
generation_config.json
ADDED
|
@@ -0,0 +1,48 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"bos_token_id": 2,
|
| 3 |
+
"confidence_threshold": null,
|
| 4 |
+
"eos_token_id": [
|
| 5 |
+
1,
|
| 6 |
+
50
|
| 7 |
+
],
|
| 8 |
+
"jump_on_no_progress_after": 12,
|
| 9 |
+
"max_denoising_steps": 256,
|
| 10 |
+
"max_new_tokens": 256,
|
| 11 |
+
"max_ponder_steps": 64,
|
| 12 |
+
"min_trajectory_progress": 0.005,
|
| 13 |
+
"pad_token_id": 0,
|
| 14 |
+
"repetition_penalty": 1.0,
|
| 15 |
+
"repetition_penalty_exclude_token_ids": [
|
| 16 |
+
2,
|
| 17 |
+
1,
|
| 18 |
+
3,
|
| 19 |
+
0,
|
| 20 |
+
4,
|
| 21 |
+
258881,
|
| 22 |
+
256000,
|
| 23 |
+
255999,
|
| 24 |
+
258883,
|
| 25 |
+
101,
|
| 26 |
+
258882,
|
| 27 |
+
106,
|
| 28 |
+
52,
|
| 29 |
+
49,
|
| 30 |
+
47,
|
| 31 |
+
51,
|
| 32 |
+
258880,
|
| 33 |
+
100,
|
| 34 |
+
105,
|
| 35 |
+
48,
|
| 36 |
+
46,
|
| 37 |
+
50,
|
| 38 |
+
98,
|
| 39 |
+
258884
|
| 40 |
+
],
|
| 41 |
+
"return_dict_in_generate": true,
|
| 42 |
+
"sampler_config": null,
|
| 43 |
+
"stability_threshold": null,
|
| 44 |
+
"t_max": 0.8,
|
| 45 |
+
"t_min": 0.8,
|
| 46 |
+
"transformers_version": "5.14.1",
|
| 47 |
+
"turn_end_token_id": 106
|
| 48 |
+
}
|
inference.py
ADDED
|
@@ -0,0 +1,35 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Run the published Modilify Mk2 model with Apple Silicon MLX."""
|
| 2 |
+
|
| 3 |
+
from __future__ import annotations
|
| 4 |
+
|
| 5 |
+
import argparse
|
| 6 |
+
from pathlib import Path
|
| 7 |
+
|
| 8 |
+
|
| 9 |
+
def main() -> None:
|
| 10 |
+
parser = argparse.ArgumentParser(description=__doc__)
|
| 11 |
+
parser.add_argument("--model", type=Path, default=Path(__file__).resolve().parent)
|
| 12 |
+
parser.add_argument("--prompt", default="你好,请介绍一下自己。")
|
| 13 |
+
parser.add_argument("--max-new-tokens", type=int, default=256)
|
| 14 |
+
parser.add_argument("--max-denoising-steps", type=int, default=256)
|
| 15 |
+
parser.add_argument("--seed", type=int, default=42)
|
| 16 |
+
parser.add_argument("--no-thinking", action="store_true")
|
| 17 |
+
parser.add_argument("--stream", action="store_true")
|
| 18 |
+
args = parser.parse_args()
|
| 19 |
+
if min(args.max_new_tokens, args.max_denoising_steps) < 1:
|
| 20 |
+
parser.error("Generation budgets must be positive.")
|
| 21 |
+
|
| 22 |
+
from modilify_mk2_mlx import load
|
| 23 |
+
|
| 24 |
+
runtime = load(args.model)
|
| 25 |
+
result = runtime.generate(
|
| 26 |
+
[{"role": "user", "content": args.prompt}],
|
| 27 |
+
enable_thinking=not args.no_thinking, seed=args.seed,
|
| 28 |
+
max_new_tokens=args.max_new_tokens, max_denoising_steps=args.max_denoising_steps,
|
| 29 |
+
on_text=(lambda chunk: print(chunk, end="", flush=True)) if args.stream else None,
|
| 30 |
+
)
|
| 31 |
+
print("" if args.stream else result.text)
|
| 32 |
+
|
| 33 |
+
|
| 34 |
+
if __name__ == "__main__":
|
| 35 |
+
main()
|
model-00001-of-00016.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d3aa3fc0393a51acec27161569d566774030e0306a74748dd2a7a11fd8b535ec
|
| 3 |
+
size 3898256004
|
model-00002-of-00016.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:89339fd6f0521977fff9d4ae498e485ff409dbb8716cc924c8667c8bceaa19cd
|
| 3 |
+
size 3798073636
|
model-00003-of-00016.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b9351e59c7c604b26712960b5d9f3460f5c1aace9ccb348999a2334d93fb613d
|
| 3 |
+
size 3290562732
|
model-00004-of-00016.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:1c89ba68a1f56e986dbf43175a7a7a6f2d046b04ac993c2837024f71b5d5923d
|
| 3 |
+
size 3319472948
|
model-00005-of-00016.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4e42f45a5c409649c0ac8a262b26a0f644d10619d5f857568db855d12a865183
|
| 3 |
+
size 3290562732
|
model-00006-of-00016.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:da6e7d99f9801cb1b7007044f3baa66e767412d4350211a637174321eacee401
|
| 3 |
+
size 3290562772
|
model-00007-of-00016.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ce1f462e324e162f9c0f41b1db3945290eb06c520da3d8ff0f582cde79a6fa96
|
| 3 |
+
size 3319473028
|
model-00008-of-00016.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a163f6df13b9a3fbe1df187e42700b74d71fa08b331c521be568f71dec77a8f0
|
| 3 |
+
size 3290562812
|
model-00009-of-00016.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:eacc2e9ac62fa45f4fd1d391bfff490e327c19df3f70e7835e556287fc9926bd
|
| 3 |
+
size 3290562812
|
model-00010-of-00016.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5fa9c8080a9c3954985226496dee261938f3d05cf81e4436932de593e5a01ae1
|
| 3 |
+
size 3319473028
|
model-00011-of-00016.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7df1df1cea73133a94aec88056e2514e04a3467d42c0dd55ad0b922af2e5e986
|
| 3 |
+
size 3290562812
|
model-00012-of-00016.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:797078b4703d602cc3a323dba676fa38bb959b70d455bdc8c673663261eee2d4
|
| 3 |
+
size 3290562812
|
model-00013-of-00016.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:35923e8f76bc78ae89b3f83d1cdd175649f64cc898f1862baf543d7cdfbcd0e1
|
| 3 |
+
size 3319473028
|
model-00014-of-00016.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:105ceea4ed17c811fb79036c199db1f7d2392a5cc4c05003085f7217e8a938bb
|
| 3 |
+
size 3290562812
|
model-00015-of-00016.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d3569ec55bc4f7e89de00efaedfbca3df44096526291731ec14bd7fc245c3b39
|
| 3 |
+
size 3290562812
|
model-00016-of-00016.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:23c9c00fc04b475c744458f498178a6e9a20fd673c36af981dd22e963b7ca49b
|
| 3 |
+
size 1202380654
|
model.safetensors.index.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
modilify_mk2_mlx/__init__.py
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Public API for the Modilify Mk2 MLX release."""
|
| 2 |
+
|
| 3 |
+
from .runtime import GenerationResult, Runtime, load
|
| 4 |
+
|
| 5 |
+
__all__ = ["GenerationResult", "Runtime", "load"]
|
modilify_mk2_mlx/model.py
ADDED
|
@@ -0,0 +1,709 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Schema25 MLX model, trajectory memory, and strict release loading."""
|
| 2 |
+
|
| 3 |
+
from __future__ import annotations
|
| 4 |
+
|
| 5 |
+
from collections import Counter, defaultdict
|
| 6 |
+
from dataclasses import dataclass, replace
|
| 7 |
+
import hashlib
|
| 8 |
+
import json
|
| 9 |
+
import math
|
| 10 |
+
from pathlib import Path
|
| 11 |
+
from types import SimpleNamespace
|
| 12 |
+
from typing import Any
|
| 13 |
+
import weakref
|
| 14 |
+
|
| 15 |
+
import mlx.core as mx
|
| 16 |
+
from mlx import nn
|
| 17 |
+
from mlx.utils import tree_flatten
|
| 18 |
+
from mlx_lm.tuner.lora import LoRALinear, LoRASwitchLinear
|
| 19 |
+
from mlx_vlm.models.diffusion_gemma.config import ModelConfig
|
| 20 |
+
from mlx_vlm.models.diffusion_gemma.diffusion_gemma import Model
|
| 21 |
+
from mlx_vlm.models.diffusion_gemma.language import DecoderLayer, Router
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def _l2(value: mx.array) -> mx.array:
|
| 25 |
+
return value * mx.rsqrt(mx.maximum(mx.sum(mx.square(value), axis=-1, keepdims=True), 1e-12))
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
class _GateProjection(nn.Module):
|
| 29 |
+
def __init__(self, source: int, target: int, rank: int) -> None:
|
| 30 |
+
super().__init__()
|
| 31 |
+
self.down = nn.Linear(source, rank, bias=False)
|
| 32 |
+
self.up = nn.Linear(rank, target, bias=False)
|
| 33 |
+
|
| 34 |
+
def __call__(self, source: mx.array) -> mx.array:
|
| 35 |
+
return self.up(self.down(source))
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
class GDN2Memory(nn.Module):
|
| 39 |
+
def __init__(self, input_dim: int, heads: int, key_dim: int, value_dim: int,
|
| 40 |
+
*, observation_dim: int | None = None) -> None:
|
| 41 |
+
super().__init__()
|
| 42 |
+
if min(input_dim, heads, key_dim, value_dim) <= 0:
|
| 43 |
+
raise ValueError("GDN2 dimensions must be positive.")
|
| 44 |
+
self.heads, self.key_dim, self.value_dim = heads, key_dim, value_dim
|
| 45 |
+
observation_dim = input_dim if observation_dim is None else observation_dim
|
| 46 |
+
if observation_dim <= 0:
|
| 47 |
+
raise ValueError("GDN2 observation width must be positive.")
|
| 48 |
+
self.q_proj = nn.Linear(input_dim, heads * key_dim, bias=False)
|
| 49 |
+
self.k_proj = nn.Linear(observation_dim, heads * key_dim, bias=False)
|
| 50 |
+
self.v_proj = nn.Linear(observation_dim, heads * value_dim, bias=False)
|
| 51 |
+
self.f_proj = _GateProjection(observation_dim, heads * key_dim, min(observation_dim, key_dim))
|
| 52 |
+
self.b_proj = nn.Linear(observation_dim, heads * key_dim, bias=False)
|
| 53 |
+
self.w_proj = nn.Linear(observation_dim, heads * value_dim, bias=False)
|
| 54 |
+
self.g_proj = _GateProjection(input_dim, heads * value_dim, min(input_dim, value_dim))
|
| 55 |
+
self.o_proj = nn.Linear(heads * value_dim, input_dim, bias=False)
|
| 56 |
+
self.a_log = mx.zeros((heads,), mx.float32)
|
| 57 |
+
self.dt_bias = mx.full((heads, key_dim), -6.906255, mx.float32)
|
| 58 |
+
|
| 59 |
+
def empty(self, *leading: int) -> mx.array:
|
| 60 |
+
return mx.zeros((*leading, self.heads, self.key_dim, self.value_dim), mx.float32)
|
| 61 |
+
|
| 62 |
+
def _projections(self, source: mx.array):
|
| 63 |
+
normalized = source.astype(mx.float32)
|
| 64 |
+
source = (normalized * mx.rsqrt(mx.mean(mx.square(normalized), axis=-1, keepdims=True) + 1e-6)).astype(source.dtype)
|
| 65 |
+
shape = source.shape[:-1]
|
| 66 |
+
key_shape = (*shape, self.heads, self.key_dim)
|
| 67 |
+
value_shape = (*shape, self.heads, self.value_dim)
|
| 68 |
+
k = _l2(nn.silu(self.k_proj(source).astype(mx.float32)).reshape(key_shape))
|
| 69 |
+
v = nn.silu(self.v_proj(source).astype(mx.float32)).reshape(value_shape)
|
| 70 |
+
head_rate = mx.exp(self.a_log.astype(mx.float32)).reshape(
|
| 71 |
+
*((1,) * (source.ndim - 1)), self.heads, 1)
|
| 72 |
+
decay = mx.exp(-head_rate * nn.softplus(
|
| 73 |
+
self.f_proj(source).astype(mx.float32).reshape(key_shape) + self.dt_bias.astype(mx.float32)))
|
| 74 |
+
erase = mx.sigmoid(self.b_proj(source).astype(mx.float32).reshape(key_shape))
|
| 75 |
+
write = mx.sigmoid(self.w_proj(source).astype(mx.float32).reshape(value_shape))
|
| 76 |
+
return k, v, decay, erase, write
|
| 77 |
+
|
| 78 |
+
def _query(self, source: mx.array) -> mx.array:
|
| 79 |
+
return _l2(nn.silu(self.q_proj(source).astype(mx.float32)).reshape(
|
| 80 |
+
*source.shape[:-1], self.heads, self.key_dim))
|
| 81 |
+
|
| 82 |
+
def _output(self, value: mx.array, source: mx.array) -> mx.array:
|
| 83 |
+
gate = nn.silu(self.g_proj(source).astype(mx.float32).reshape(value.shape))
|
| 84 |
+
value = value * mx.rsqrt(mx.mean(mx.square(value), axis=-1, keepdims=True) + 1e-6) * gate
|
| 85 |
+
return self.o_proj(value.reshape(*source.shape[:-1], -1).astype(source.dtype))
|
| 86 |
+
|
| 87 |
+
def read(self, state: mx.array, source: mx.array) -> mx.array:
|
| 88 |
+
if state.shape != (*source.shape[:-1], self.heads, self.key_dim, self.value_dim):
|
| 89 |
+
raise ValueError("GDN2 state and query leading dimensions differ.")
|
| 90 |
+
value = mx.sum(self._query(source)[..., None] * state.astype(mx.float32), axis=-2)
|
| 91 |
+
return self._output(value, source)
|
| 92 |
+
|
| 93 |
+
def read_shared(self, state: mx.array, source: mx.array) -> mx.array:
|
| 94 |
+
if source.ndim != 3 or state.shape != (source.shape[0], self.heads, self.key_dim, self.value_dim):
|
| 95 |
+
raise ValueError("Shared GDN2 read requires [batch, canvas, width] queries.")
|
| 96 |
+
query = self._query(source).transpose(0, 2, 1, 3)
|
| 97 |
+
value = (query @ state.astype(mx.float32)).transpose(0, 2, 1, 3)
|
| 98 |
+
return self._output(value, source)
|
| 99 |
+
|
| 100 |
+
@staticmethod
|
| 101 |
+
def _transition(state, k, v, decay, erase, write, valid):
|
| 102 |
+
decayed = state.astype(mx.float32) * decay[..., None]
|
| 103 |
+
old = mx.sum((erase * k)[..., None] * decayed, axis=-2)
|
| 104 |
+
candidate = decayed + k[..., None] * (write * v - old)[..., None, :]
|
| 105 |
+
return candidate if valid is None else mx.where(
|
| 106 |
+
valid[..., None, None, None], candidate, state.astype(mx.float32))
|
| 107 |
+
|
| 108 |
+
def transition(self, state: mx.array, source: mx.array,
|
| 109 |
+
valid: mx.array | None = None) -> mx.array:
|
| 110 |
+
if state.shape != (*source.shape[:-1], self.heads, self.key_dim, self.value_dim):
|
| 111 |
+
raise ValueError("GDN2 state and observation leading dimensions differ.")
|
| 112 |
+
k, v, decay, erase, write = self._projections(source)
|
| 113 |
+
if valid is not None:
|
| 114 |
+
if valid.shape != source.shape[:-1]:
|
| 115 |
+
raise ValueError("GDN2 valid mask must match observation rows.")
|
| 116 |
+
return self._transition(state, k, v, decay, erase, write, valid)
|
| 117 |
+
|
| 118 |
+
def write_sequence(self, state: mx.array, source: mx.array,
|
| 119 |
+
valid: mx.array) -> mx.array:
|
| 120 |
+
if source.ndim != 3 or valid.shape != source.shape[:2]:
|
| 121 |
+
raise ValueError("GDN2 sequence and mask must share [batch, length].")
|
| 122 |
+
if state.shape != (source.shape[0], self.heads, self.key_dim, self.value_dim):
|
| 123 |
+
raise ValueError("GDN2 sequence state shape differs.")
|
| 124 |
+
projected = self._projections(source)
|
| 125 |
+
for index in range(source.shape[1]):
|
| 126 |
+
state = self._transition(state, *(part[:, index] for part in projected), valid[:, index])
|
| 127 |
+
return state
|
| 128 |
+
|
| 129 |
+
|
| 130 |
+
@dataclass(frozen=True)
|
| 131 |
+
class GDN2TrajectoryState:
|
| 132 |
+
cells: mx.array
|
| 133 |
+
row: mx.array
|
| 134 |
+
persistent: mx.array
|
| 135 |
+
seen: mx.array
|
| 136 |
+
|
| 137 |
+
def clear_refill(self, lengths: mx.array, head: mx.array) -> "GDN2TrajectoryState":
|
| 138 |
+
batch, canvas = self.seen.shape
|
| 139 |
+
if lengths.shape != (batch,) or head.shape != (batch,):
|
| 140 |
+
raise ValueError("Commit lengths and canvas heads must be per row.")
|
| 141 |
+
physical = mx.arange(canvas)[None, :]
|
| 142 |
+
overwritten = ((physical - head[:, None]) % canvas) < lengths[:, None]
|
| 143 |
+
return GDN2TrajectoryState(
|
| 144 |
+
mx.where(overwritten[..., None, None, None], 0.0, self.cells),
|
| 145 |
+
self.row, self.persistent, self.seen & ~overwritten,
|
| 146 |
+
)
|
| 147 |
+
|
| 148 |
+
|
| 149 |
+
class GDN2TrajectoryMemory(nn.Module):
|
| 150 |
+
def __init__(self, width: int, *, probes: int = 4,
|
| 151 |
+
working_heads: int = 16, working_key: int = 64,
|
| 152 |
+
working_value: int = 64, persistent_heads: int = 16,
|
| 153 |
+
persistent_key: int = 128, persistent_value: int = 128,
|
| 154 |
+
persistent_observation_dim: int | None = None) -> None:
|
| 155 |
+
super().__init__()
|
| 156 |
+
if probes <= 0:
|
| 157 |
+
raise ValueError("Probe count must be positive.")
|
| 158 |
+
self.probes = probes
|
| 159 |
+
self.cell = GDN2Memory(width, working_heads, working_key, working_value)
|
| 160 |
+
self.row = GDN2Memory(width, working_heads, working_key, working_value)
|
| 161 |
+
self.persistent = GDN2Memory(width, persistent_heads, persistent_key, persistent_value,
|
| 162 |
+
observation_dim=persistent_observation_dim)
|
| 163 |
+
self.probe_embed = nn.Embedding(probes, width)
|
| 164 |
+
self.probe_embed.weight = mx.random.normal((probes, width)) * 0.02
|
| 165 |
+
|
| 166 |
+
def read(self, state: GDN2TrajectoryState, query: mx.array) -> mx.array:
|
| 167 |
+
result = (self.cell.read(state.cells, query)
|
| 168 |
+
+ self.row.read_shared(state.row, query)
|
| 169 |
+
+ self.persistent.read_shared(state.persistent, query))
|
| 170 |
+
return mx.where(state.seen[..., None], result, 0.0)
|
| 171 |
+
|
| 172 |
+
def observe(self, state: GDN2TrajectoryState, observation: mx.array,
|
| 173 |
+
live: mx.array, head: mx.array) -> GDN2TrajectoryState:
|
| 174 |
+
batch, canvas, width = observation.shape
|
| 175 |
+
if live.shape != (batch, canvas) or head.shape != (batch,):
|
| 176 |
+
raise ValueError("Observation mask and head have incorrect shapes.")
|
| 177 |
+
cells = self.cell.transition(state.cells, observation, live)
|
| 178 |
+
seen = state.seen | live
|
| 179 |
+
logical_idx = (head[:, None] + mx.arange(canvas)[None, :]) % canvas
|
| 180 |
+
logical = mx.take_along_axis(observation,
|
| 181 |
+
mx.broadcast_to(logical_idx[..., None], observation.shape), axis=1)
|
| 182 |
+
logical_live = mx.take_along_axis(live, logical_idx, axis=1)
|
| 183 |
+
row_state = state.row
|
| 184 |
+
for probe in range(self.probes):
|
| 185 |
+
lo = canvas * probe // self.probes
|
| 186 |
+
hi = canvas * (probe + 1) // self.probes
|
| 187 |
+
selected = logical_live[:, lo:hi]
|
| 188 |
+
count = mx.sum(selected.astype(mx.float32), axis=1, keepdims=True)
|
| 189 |
+
pooled = mx.sum(logical[:, lo:hi].astype(mx.float32)
|
| 190 |
+
* selected[..., None], axis=1)
|
| 191 |
+
pooled = (pooled / mx.maximum(count, 1.0)).astype(observation.dtype)
|
| 192 |
+
pooled = pooled + self.probe_embed.weight[probe].astype(observation.dtype)
|
| 193 |
+
row_state = self.row.transition(row_state, pooled, count[:, 0] > 0)
|
| 194 |
+
return GDN2TrajectoryState(cells, row_state, state.persistent, seen)
|
| 195 |
+
|
| 196 |
+
|
| 197 |
+
|
| 198 |
+
def fp32_attention(
|
| 199 |
+
query: mx.array, keys: mx.array, values: mx.array,
|
| 200 |
+
*, attn_mask: mx.array | None = None,
|
| 201 |
+
) -> mx.array:
|
| 202 |
+
"""Small memory-attention reductions in FP32 with model-dtype output."""
|
| 203 |
+
|
| 204 |
+
dtype = query.dtype
|
| 205 |
+
if query.ndim == 4 and keys.ndim == 4 and keys.shape[-2] >= 64:
|
| 206 |
+
# The working bus attends over the full 256-token canvas. Metal's
|
| 207 |
+
# fused attention avoids materializing its FP32 score/probability grid
|
| 208 |
+
# while retaining FP32 accumulation.
|
| 209 |
+
mask = attn_mask
|
| 210 |
+
if mask is not None:
|
| 211 |
+
mask = (mx.where(mask, 0.0, -1.0e9)
|
| 212 |
+
if mask.dtype == mx.bool_ else mask.astype(mx.float32))
|
| 213 |
+
return mx.fast.scaled_dot_product_attention(
|
| 214 |
+
query.astype(mx.float32), keys.astype(mx.float32),
|
| 215 |
+
values.astype(mx.float32),
|
| 216 |
+
scale=1.0 / math.sqrt(max(query.shape[-1], 1)), mask=mask,
|
| 217 |
+
).astype(dtype)
|
| 218 |
+
scores = mx.matmul(query.astype(mx.float32), mx.swapaxes(keys.astype(mx.float32), -1, -2))
|
| 219 |
+
scores = scores / math.sqrt(max(query.shape[-1], 1))
|
| 220 |
+
if attn_mask is not None:
|
| 221 |
+
if attn_mask.dtype == mx.bool_:
|
| 222 |
+
scores = mx.where(attn_mask, scores, -1.0e9)
|
| 223 |
+
else:
|
| 224 |
+
scores = scores + attn_mask.astype(mx.float32)
|
| 225 |
+
probabilities = mx.nan_to_num(mx.softmax(scores, axis=-1), nan=0.0)
|
| 226 |
+
return mx.matmul(probabilities, values.astype(mx.float32)).astype(dtype)
|
| 227 |
+
|
| 228 |
+
|
| 229 |
+
class RMSNorm(nn.Module):
|
| 230 |
+
def __init__(self, dim: int, eps: float = 1.0e-6) -> None:
|
| 231 |
+
super().__init__()
|
| 232 |
+
self.weight = mx.ones((dim,))
|
| 233 |
+
self.eps = eps
|
| 234 |
+
|
| 235 |
+
def __call__(self, hidden: mx.array) -> mx.array:
|
| 236 |
+
value = hidden.astype(mx.float32)
|
| 237 |
+
scale = mx.rsqrt(mx.mean(mx.square(value), axis=-1, keepdims=True) + self.eps)
|
| 238 |
+
return (value * scale * self.weight.astype(mx.float32)).astype(hidden.dtype)
|
| 239 |
+
|
| 240 |
+
|
| 241 |
+
class SwiGLU(nn.Module):
|
| 242 |
+
def __init__(self, dim: int, hidden: int) -> None:
|
| 243 |
+
super().__init__()
|
| 244 |
+
self.gate = nn.Linear(dim, hidden, bias=False)
|
| 245 |
+
self.up = nn.Linear(dim, hidden, bias=False)
|
| 246 |
+
self.down = nn.Linear(hidden, dim, bias=False)
|
| 247 |
+
|
| 248 |
+
def __call__(self, hidden: mx.array) -> mx.array:
|
| 249 |
+
return self.down(nn.silu(self.gate(hidden)) * self.up(hidden))
|
| 250 |
+
|
| 251 |
+
|
| 252 |
+
class RankAttention(nn.Module):
|
| 253 |
+
def __init__(self, dim: int, num_heads: int, kv_rank: int) -> None:
|
| 254 |
+
super().__init__()
|
| 255 |
+
if kv_rank % num_heads:
|
| 256 |
+
raise ValueError("Attention K/V rank must divide heads.")
|
| 257 |
+
self.num_heads, self.kv_rank = num_heads, kv_rank
|
| 258 |
+
self.head_dim = kv_rank // num_heads
|
| 259 |
+
self.q_proj = nn.Linear(dim, kv_rank, bias=False)
|
| 260 |
+
self.k_proj = nn.Linear(dim, kv_rank, bias=False)
|
| 261 |
+
self.v_proj = nn.Linear(dim, kv_rank, bias=False)
|
| 262 |
+
self.o_proj = nn.Linear(kv_rank, dim, bias=False)
|
| 263 |
+
self.q_norm = RMSNorm(dim)
|
| 264 |
+
self.k_norm = RMSNorm(dim)
|
| 265 |
+
|
| 266 |
+
def __call__(self, query: mx.array, keys: mx.array, values: mx.array,
|
| 267 |
+
attn_mask: mx.array | None = None) -> mx.array:
|
| 268 |
+
batch, queries, _ = query.shape
|
| 269 |
+
key_len = keys.shape[1]
|
| 270 |
+
heads, head_dim = self.num_heads, self.head_dim
|
| 271 |
+
reshape_q = lambda x: x.reshape(batch, queries, heads, head_dim).transpose(0, 2, 1, 3)
|
| 272 |
+
reshape_k = lambda x: x.reshape(batch, key_len, heads, head_dim).transpose(0, 2, 1, 3)
|
| 273 |
+
q = reshape_q(self.q_proj(self.q_norm(query)))
|
| 274 |
+
k = reshape_k(self.k_proj(self.k_norm(keys)))
|
| 275 |
+
v = reshape_k(self.v_proj(values))
|
| 276 |
+
mask = attn_mask
|
| 277 |
+
if mask is not None and mask.ndim == 2:
|
| 278 |
+
mask = mask[None, None, :, :]
|
| 279 |
+
elif mask is not None and mask.ndim == 3:
|
| 280 |
+
mask = mask[:, None]
|
| 281 |
+
context = fp32_attention(q, k, v, attn_mask=mask)
|
| 282 |
+
return self.o_proj(context.transpose(0, 2, 1, 3).reshape(batch, queries, self.kv_rank))
|
| 283 |
+
|
| 284 |
+
|
| 285 |
+
class _WorkingBus(nn.Module):
|
| 286 |
+
def __init__(self, width: int, heads: int, readers: int, rank: int, canvas: int) -> None:
|
| 287 |
+
super().__init__()
|
| 288 |
+
self.num_readers, self.heads, self.rank = readers, heads, rank
|
| 289 |
+
self.head_dim = rank // heads
|
| 290 |
+
self.memory_norm = RMSNorm(width)
|
| 291 |
+
# Retain the published address norm tensor in the weight topology.
|
| 292 |
+
self.address_norm = RMSNorm(width)
|
| 293 |
+
self.k_proj = nn.Linear(width, rank, bias=False)
|
| 294 |
+
self.v_proj = nn.Linear(width, rank, bias=False)
|
| 295 |
+
self.q_norm = RMSNorm(width)
|
| 296 |
+
self.q_proj = [nn.Linear(width, rank, bias=False) for _ in range(readers)]
|
| 297 |
+
self.o_proj = [nn.Linear(rank, width, bias=False) for _ in range(readers)]
|
| 298 |
+
self.alpha = mx.zeros((max(readers, 1), heads))
|
| 299 |
+
self.rel_bias = mx.zeros((heads, 2 * canvas - 1))
|
| 300 |
+
|
| 301 |
+
def prepare_kv(self, working: mx.array, seen: mx.array):
|
| 302 |
+
if not self.num_readers:
|
| 303 |
+
return None
|
| 304 |
+
normed = self.memory_norm(working)
|
| 305 |
+
batch, canvas, _ = working.shape
|
| 306 |
+
shape = (batch, canvas, self.heads, self.head_dim)
|
| 307 |
+
keys = self.k_proj(normed).reshape(shape).transpose(0, 2, 1, 3)
|
| 308 |
+
values = self.v_proj(normed).reshape(shape).transpose(0, 2, 1, 3)
|
| 309 |
+
positions = mx.arange(canvas)
|
| 310 |
+
relative = positions[:, None] - positions[None, :] + canvas - 1
|
| 311 |
+
bias = mx.take(self.rel_bias, relative, axis=1)[None].astype(mx.float32)
|
| 312 |
+
bias = bias + mx.where(seen[:, None, None, :], 0.0, -1e9).astype(mx.float32)
|
| 313 |
+
return keys, values, bias, seen
|
| 314 |
+
|
| 315 |
+
def read(self, hidden, reader, keys, values, bias, seen):
|
| 316 |
+
batch, canvas, _ = hidden.shape
|
| 317 |
+
query = self.q_proj[reader](self.q_norm(hidden))
|
| 318 |
+
query = query.reshape(batch, canvas, self.heads, self.head_dim).transpose(0, 2, 1, 3)
|
| 319 |
+
context = fp32_attention(query, keys, values, attn_mask=bias)
|
| 320 |
+
context = context * mx.tanh(self.alpha[reader]).astype(hidden.dtype)[None, :, None, None]
|
| 321 |
+
context = context.transpose(0, 2, 1, 3).reshape(batch, canvas, self.rank)
|
| 322 |
+
return mx.where(seen[..., None], hidden + self.o_proj[reader](context), hidden)
|
| 323 |
+
|
| 324 |
+
|
| 325 |
+
class _CanvasBlock(nn.Module):
|
| 326 |
+
def __init__(self, width: int, heads: int, rank: int, ffn: int,
|
| 327 |
+
window: int, global_attention: bool) -> None:
|
| 328 |
+
super().__init__()
|
| 329 |
+
self.norm = RMSNorm(width)
|
| 330 |
+
self.attn = RankAttention(width, heads, rank)
|
| 331 |
+
self.ff_norm = RMSNorm(width)
|
| 332 |
+
self.ff = SwiGLU(width, ffn)
|
| 333 |
+
self.window = window
|
| 334 |
+
self.global_attention = global_attention
|
| 335 |
+
|
| 336 |
+
def __call__(self, hidden: mx.array, seen: mx.array,
|
| 337 |
+
offsets: mx.array) -> mx.array:
|
| 338 |
+
allowed = mx.broadcast_to(seen[:, None, :], (hidden.shape[0], hidden.shape[1], hidden.shape[1]))
|
| 339 |
+
if not self.global_attention:
|
| 340 |
+
allowed = allowed & (mx.abs(offsets[:, :, None] - offsets[:, None, :]) < self.window)
|
| 341 |
+
mask = mx.where(allowed, 0.0, -1e9).astype(mx.float32)
|
| 342 |
+
normed = self.norm(hidden)
|
| 343 |
+
hidden = hidden + self.attn(normed, normed, normed, mask)
|
| 344 |
+
hidden = hidden + self.ff(self.ff_norm(hidden))
|
| 345 |
+
return mx.where(seen[..., None], hidden, mx.zeros_like(hidden))
|
| 346 |
+
|
| 347 |
+
|
| 348 |
+
|
| 349 |
+
|
| 350 |
+
class _PersistentBus(nn.Module):
|
| 351 |
+
def __init__(self, memory: nn.Module, readers: int) -> None:
|
| 352 |
+
super().__init__()
|
| 353 |
+
object.__setattr__(self, "_memory_ref", weakref.ref(memory))
|
| 354 |
+
self.num_readers = readers
|
| 355 |
+
self.alpha = mx.zeros((max(readers, 1), 1))
|
| 356 |
+
|
| 357 |
+
def prepare_kv(self, memory: mx.array,
|
| 358 |
+
seen: mx.array | None = None):
|
| 359 |
+
if self.num_readers <= 0:
|
| 360 |
+
return None
|
| 361 |
+
return memory, seen
|
| 362 |
+
|
| 363 |
+
def read(self, hidden: mx.array, reader_index: int,
|
| 364 |
+
memory: mx.array, seen: mx.array | None) -> mx.array:
|
| 365 |
+
delta = self._memory_ref().read_shared(memory, hidden)
|
| 366 |
+
if seen is not None:
|
| 367 |
+
delta = delta * seen[..., None].astype(delta.dtype)
|
| 368 |
+
return hidden + mx.tanh(self.alpha[reader_index]).astype(hidden.dtype) * delta
|
| 369 |
+
|
| 370 |
+
|
| 371 |
+
class LatentDeliberationTransformer(nn.Module):
|
| 372 |
+
def __init__(self, *, hidden_size: int,
|
| 373 |
+
latent_dim: int, ffn_dim: int,
|
| 374 |
+
num_layers: int, num_heads: int, local_attention_window: int,
|
| 375 |
+
tape_probes: int,
|
| 376 |
+
history_kv_rank: int, num_working_readers: int,
|
| 377 |
+
num_persistent_readers: int,
|
| 378 |
+
working_last_block_global: bool,
|
| 379 |
+
|
| 380 |
+
commit_sequence_dim: int | None = None,
|
| 381 |
+
max_canvas_length: int = 256) -> None:
|
| 382 |
+
super().__init__()
|
| 383 |
+
if hidden_size != latent_dim:
|
| 384 |
+
raise ValueError("Schema25 GDN2 requires hidden_size == latent_dim.")
|
| 385 |
+
packet_dim = int(commit_sequence_dim or history_kv_rank)
|
| 386 |
+
self.trajectory = GDN2TrajectoryMemory(
|
| 387 |
+
hidden_size, probes=tape_probes, persistent_observation_dim=packet_dim,
|
| 388 |
+
)
|
| 389 |
+
self.blocks = [
|
| 390 |
+
_CanvasBlock(hidden_size, num_heads, history_kv_rank, ffn_dim,
|
| 391 |
+
local_attention_window,
|
| 392 |
+
bool(working_last_block_global and i == num_layers - 1))
|
| 393 |
+
for i in range(num_layers)
|
| 394 |
+
]
|
| 395 |
+
self.output_norm = RMSNorm(hidden_size)
|
| 396 |
+
self.working_memory_bus = _WorkingBus(
|
| 397 |
+
hidden_size, num_heads, num_working_readers, history_kv_rank, max_canvas_length,
|
| 398 |
+
)
|
| 399 |
+
self.persistent_memory_bus = _PersistentBus(
|
| 400 |
+
self.trajectory.persistent, num_persistent_readers,
|
| 401 |
+
)
|
| 402 |
+
self.experience_in = nn.Linear(hidden_size * 3, packet_dim, bias=False)
|
| 403 |
+
self.reason_embed = nn.Embedding(6, packet_dim)
|
| 404 |
+
|
| 405 |
+
def __call__(self, token_embeddings: mx.array,
|
| 406 |
+
memory: GDN2TrajectoryState, canvas_head: mx.array) -> mx.array:
|
| 407 |
+
hidden = self.trajectory.read(memory, token_embeddings)
|
| 408 |
+
offsets = (mx.arange(memory.seen.shape[1])[None, :] - canvas_head[:, None]) % memory.seen.shape[1]
|
| 409 |
+
for block in self.blocks:
|
| 410 |
+
hidden = block(hidden, memory.seen, offsets)
|
| 411 |
+
return self.output_norm(hidden) * memory.seen[..., None].astype(hidden.dtype)
|
| 412 |
+
|
| 413 |
+
def commit_write(self, *, memory: mx.array, working_state: mx.array,
|
| 414 |
+
heavy_hidden: mx.array,
|
| 415 |
+
committed_token_embeddings: mx.array,
|
| 416 |
+
canvas_head: mx.array) -> mx.array:
|
| 417 |
+
batch, canvas, width = working_state.shape
|
| 418 |
+
count = committed_token_embeddings.shape[1]
|
| 419 |
+
index = (mx.arange(count)[None, :] + canvas_head[:, None]) % canvas
|
| 420 |
+
gather = mx.broadcast_to(index[..., None], (batch, count, width))
|
| 421 |
+
selected_working = mx.take_along_axis(working_state, gather, axis=1)
|
| 422 |
+
selected_heavy = mx.take_along_axis(heavy_hidden, gather, axis=1)
|
| 423 |
+
|
| 424 |
+
def normalize(value):
|
| 425 |
+
fp32 = value.astype(mx.float32)
|
| 426 |
+
return (fp32 * mx.rsqrt(mx.mean(mx.square(fp32), axis=-1, keepdims=True) + 1.0)).astype(value.dtype)
|
| 427 |
+
|
| 428 |
+
roles = tuple(normalize(value) for value in (
|
| 429 |
+
selected_heavy, selected_working, committed_token_embeddings,
|
| 430 |
+
))
|
| 431 |
+
packet = self.experience_in(mx.concatenate(roles, axis=-1))
|
| 432 |
+
# Schema25 maps both continuing commit reasons to embedding row 5.
|
| 433 |
+
packet = packet + self.reason_embed.weight[5]
|
| 434 |
+
valid = mx.ones((batch, count), mx.bool_)
|
| 435 |
+
return self.trajectory.persistent.write_sequence(memory, packet, valid)
|
| 436 |
+
|
| 437 |
+
|
| 438 |
+
def create_mlx_latent(config: Any) -> LatentDeliberationTransformer:
|
| 439 |
+
readers = sum(kind == "full_attention" for kind in config.text_config.layer_types)
|
| 440 |
+
return LatentDeliberationTransformer(
|
| 441 |
+
hidden_size=config.text_config.hidden_size,
|
| 442 |
+
latent_dim=config.latent_dim,
|
| 443 |
+
ffn_dim=config.latent_ffn_dim,
|
| 444 |
+
num_layers=config.latent_num_layers,
|
| 445 |
+
num_heads=config.latent_num_heads,
|
| 446 |
+
local_attention_window=config.latent_local_attention_window,
|
| 447 |
+
tape_probes=config.latent_tape_probes,
|
| 448 |
+
history_kv_rank=config.latent_history_kv_rank,
|
| 449 |
+
num_working_readers=readers if config.working_memory_bus else 0,
|
| 450 |
+
num_persistent_readers=readers if config.persistent_memory_bus else 0,
|
| 451 |
+
working_last_block_global=config.latent_working_last_block_global,
|
| 452 |
+
commit_sequence_dim=config.commit_sequence_dim,
|
| 453 |
+
max_canvas_length=config.canvas_length,
|
| 454 |
+
)
|
| 455 |
+
|
| 456 |
+
|
| 457 |
+
@dataclass
|
| 458 |
+
class MLXCanvasOutput:
|
| 459 |
+
heavy_hidden: mx.array
|
| 460 |
+
working_state: mx.array
|
| 461 |
+
|
| 462 |
+
|
| 463 |
+
class MLXModilifyMk2(nn.Module):
|
| 464 |
+
"""Shared DiffusionGemma trunk plus native commit-only trajectory module."""
|
| 465 |
+
|
| 466 |
+
def __init__(self, backbone: Any, config: Any) -> None:
|
| 467 |
+
super().__init__()
|
| 468 |
+
if int(config.state_schema_version) != 25:
|
| 469 |
+
raise RuntimeError("Native MLX ModilifyMk2 requires schema25.")
|
| 470 |
+
self.model = backbone.model
|
| 471 |
+
self.latent_deliberation = create_mlx_latent(config)
|
| 472 |
+
self.config = config
|
| 473 |
+
|
| 474 |
+
def _merge_context(self, token_embeddings: mx.array,
|
| 475 |
+
context: mx.array) -> mx.array:
|
| 476 |
+
"""Frozen self-conditioning bridge with the schema25 residual cap."""
|
| 477 |
+
|
| 478 |
+
mapper = self.model.decoder.self_conditioning
|
| 479 |
+
normed = mapper.pre_norm(context.astype(token_embeddings.dtype))
|
| 480 |
+
mapped = mapper.down_proj(
|
| 481 |
+
nn.gelu_approx(mapper.gate_proj(normed)) * mapper.up_proj(normed)
|
| 482 |
+
)
|
| 483 |
+
mapped_fp32 = mapped.astype(mx.float32)
|
| 484 |
+
energy = mx.mean(mx.square(mapped_fp32), axis=-1, keepdims=True)
|
| 485 |
+
token_rms = mx.sqrt(mx.mean(mx.square(token_embeddings.astype(mx.float32)),
|
| 486 |
+
axis=-1, keepdims=True))
|
| 487 |
+
cap = 0.5 * token_rms
|
| 488 |
+
scale = cap / mx.sqrt(energy + mx.square(cap) + 1.0e-12)
|
| 489 |
+
combined = token_embeddings + (mapped_fp32 * scale).astype(mapped.dtype)
|
| 490 |
+
return mapper.post_norm(combined)
|
| 491 |
+
|
| 492 |
+
def forward_canvas(self, canvas: mx.array, *, prefix_cache: list[Any],
|
| 493 |
+
state: GDN2TrajectoryState,
|
| 494 |
+
canvas_head: mx.array, first_step: bool = False,
|
| 495 |
+
has_commits: bool = True) -> MLXCanvasOutput:
|
| 496 |
+
"""Denoise physical ring state through logical-order trunk positions."""
|
| 497 |
+
|
| 498 |
+
batch, length = canvas.shape
|
| 499 |
+
decoder = self.model.decoder
|
| 500 |
+
token_embeddings = decoder.embed_tokens(canvas) * decoder.embed_scale
|
| 501 |
+
working = (mx.zeros_like(token_embeddings) if first_step
|
| 502 |
+
else self.latent_deliberation(token_embeddings, state, canvas_head))
|
| 503 |
+
physical = (canvas_head[:, None] + mx.arange(length)[None, :]) % length
|
| 504 |
+
hidden_width = token_embeddings.shape[-1]
|
| 505 |
+
gather = mx.broadcast_to(physical[:, :, None], (batch, length, hidden_width))
|
| 506 |
+
logical_tokens = mx.take_along_axis(token_embeddings, gather, axis=1)
|
| 507 |
+
logical_working = mx.take_along_axis(working, gather, axis=1)
|
| 508 |
+
hidden = self._merge_context(logical_tokens, logical_working)
|
| 509 |
+
prefix = int(prefix_cache[0].offset)
|
| 510 |
+
mask = mx.ones((batch, prefix + length), mx.bool_)
|
| 511 |
+
masks = decoder._make_decoder_masks(hidden, prefix_cache, mask)
|
| 512 |
+
offset = prefix_cache[0].offset
|
| 513 |
+
working_bus = self.latent_deliberation.working_memory_bus
|
| 514 |
+
persistent_bus = self.latent_deliberation.persistent_memory_bus
|
| 515 |
+
logical_seen = mx.take_along_axis(state.seen, physical, axis=1)
|
| 516 |
+
working_kv = None if first_step else working_bus.prepare_kv(logical_working, logical_seen)
|
| 517 |
+
persistent_kv = persistent_bus.prepare_kv(state.persistent, logical_seen) if has_commits else None
|
| 518 |
+
reader = 0
|
| 519 |
+
for layer, layer_cache in zip(decoder.layers, prefix_cache):
|
| 520 |
+
hidden = layer(hidden, masks[layer.layer_type], layer_cache,
|
| 521 |
+
decoder=True, offset=offset)
|
| 522 |
+
if layer.layer_type == "full_attention":
|
| 523 |
+
if working_kv is not None and reader < working_bus.num_readers:
|
| 524 |
+
hidden = working_bus.read(hidden, reader, *working_kv)
|
| 525 |
+
if persistent_kv is not None and reader < persistent_bus.num_readers:
|
| 526 |
+
hidden = persistent_bus.read(hidden, reader, *persistent_kv)
|
| 527 |
+
reader += 1
|
| 528 |
+
hidden = decoder.norm(hidden)
|
| 529 |
+
inverse = (mx.arange(length)[None, :] - canvas_head[:, None]) % length
|
| 530 |
+
heavy = mx.take_along_axis(hidden, mx.broadcast_to(inverse[:, :, None], hidden.shape), axis=1)
|
| 531 |
+
return MLXCanvasOutput(heavy, working)
|
| 532 |
+
|
| 533 |
+
|
| 534 |
+
def _install_sliding_encoder_masks(encoder: Any) -> None:
|
| 535 |
+
"""Align mlx-vlm's full-length masks with trimmed sliding KV chunks."""
|
| 536 |
+
|
| 537 |
+
if getattr(encoder, "_modilify_sliding_masks", False):
|
| 538 |
+
return
|
| 539 |
+
original = encoder._make_encoder_masks
|
| 540 |
+
window = int(encoder.text_config.sliding_window)
|
| 541 |
+
|
| 542 |
+
def make_masks(h, cache, attention_mask=None, mm_token_type_ids=None):
|
| 543 |
+
masks = original(h, cache, attention_mask, mm_token_type_ids)
|
| 544 |
+
if isinstance(masks, dict):
|
| 545 |
+
return masks
|
| 546 |
+
query_length = int(h.shape[1])
|
| 547 |
+
maximum = window - 1 + query_length
|
| 548 |
+
return [
|
| 549 |
+
mask[..., -maximum:] if (
|
| 550 |
+
layer.layer_type == "sliding_attention"
|
| 551 |
+
and isinstance(mask, mx.array) and mask.shape[-1] > maximum
|
| 552 |
+
) else mask
|
| 553 |
+
for layer, mask in zip(encoder.decoder.layers, masks)
|
| 554 |
+
]
|
| 555 |
+
|
| 556 |
+
encoder._make_encoder_masks = make_masks
|
| 557 |
+
encoder._modilify_sliding_masks = True
|
| 558 |
+
|
| 559 |
+
|
| 560 |
+
def _read_json(path: Path) -> dict:
|
| 561 |
+
return json.loads(path.read_text(encoding="utf-8"))
|
| 562 |
+
|
| 563 |
+
|
| 564 |
+
class _ReleaseRouter(Router):
|
| 565 |
+
"""Route with the published model's FP32 probabilities and weights."""
|
| 566 |
+
|
| 567 |
+
def __call__(self, x: mx.array) -> tuple[mx.array, mx.array]:
|
| 568 |
+
normalized = mx.fast.rms_norm(x.astype(mx.float32), None, self.eps).astype(x.dtype)
|
| 569 |
+
scores = self.proj(normalized * self.scale * self._root_size)
|
| 570 |
+
probabilities = mx.softmax(scores.astype(mx.float32), axis=-1, precise=True)
|
| 571 |
+
k = self.config.top_k_experts
|
| 572 |
+
indices = mx.argpartition(-probabilities, kth=k - 1, axis=-1)[..., :k]
|
| 573 |
+
weights = mx.take_along_axis(probabilities, indices, axis=-1)
|
| 574 |
+
weights = weights / mx.maximum(weights.sum(axis=-1, keepdims=True), 1.17549435e-38)
|
| 575 |
+
return indices, weights * self.per_expert_scale[indices]
|
| 576 |
+
|
| 577 |
+
|
| 578 |
+
class _ReleaseLayer(DecoderLayer):
|
| 579 |
+
"""Keep expert weighting in FP32 and return the sum in activation dtype."""
|
| 580 |
+
|
| 581 |
+
def __call__(self, x, mask=None, cache=None, *, decoder=False, offset=None, layer_scalar=None):
|
| 582 |
+
attention = self.self_attn(self.input_layernorm(x), mask, cache, decoder=decoder, offset=offset)
|
| 583 |
+
residual = x + self.post_attention_layernorm(attention)
|
| 584 |
+
dense = self.post_feedforward_layernorm_1(self.mlp(self.pre_feedforward_layernorm(residual)))
|
| 585 |
+
flat = residual.reshape(-1, residual.shape[-1])
|
| 586 |
+
indices, weights = self.router(flat)
|
| 587 |
+
routed = self.experts(self.pre_feedforward_layernorm_2(flat), indices)
|
| 588 |
+
routed = (routed * weights[..., None]).sum(axis=-2).astype(x.dtype).reshape(x.shape)
|
| 589 |
+
routed = self.post_feedforward_layernorm_2(routed)
|
| 590 |
+
output = residual + self.post_feedforward_layernorm(dense + routed)
|
| 591 |
+
return output * (self.layer_scalar if layer_scalar is None else layer_scalar)
|
| 592 |
+
|
| 593 |
+
|
| 594 |
+
def _create_model(config: dict) -> MLXModilifyMk2:
|
| 595 |
+
text_config = dict(config["text_config"], model_type="diffusion_gemma_text")
|
| 596 |
+
backbone = Model(ModelConfig.from_dict({
|
| 597 |
+
"model_type": "diffusion_gemma", "text_config": text_config,
|
| 598 |
+
"vision_config": None,
|
| 599 |
+
}))
|
| 600 |
+
lora = config["lora_config"]
|
| 601 |
+
targets = set(lora["target_modules"])
|
| 602 |
+
for layer in backbone.model.decoder.layers:
|
| 603 |
+
layer.__class__ = _ReleaseLayer
|
| 604 |
+
layer.router = _ReleaseRouter(layer.router.config)
|
| 605 |
+
for parent in (layer.self_attn, layer.mlp):
|
| 606 |
+
for name, module in list(parent.named_modules()):
|
| 607 |
+
if name in targets:
|
| 608 |
+
setattr(parent, name, LoRALinear.from_base(
|
| 609 |
+
module, r=lora["r"], scale=lora["alpha"] / lora["r"],
|
| 610 |
+
))
|
| 611 |
+
for name in ("gate_up_proj", "down_proj"):
|
| 612 |
+
module = getattr(layer.experts, name)
|
| 613 |
+
setattr(layer.experts, name, LoRASwitchLinear.from_base(
|
| 614 |
+
module, r=lora["expert_r"], scale=lora["expert_alpha"] / lora["expert_r"],
|
| 615 |
+
))
|
| 616 |
+
settings = SimpleNamespace(**dict(config, text_config=SimpleNamespace(**text_config)))
|
| 617 |
+
model = MLXModilifyMk2(backbone, settings)
|
| 618 |
+
_install_sliding_encoder_masks(model.model.encoder)
|
| 619 |
+
model.eval()
|
| 620 |
+
model.freeze()
|
| 621 |
+
return model
|
| 622 |
+
|
| 623 |
+
|
| 624 |
+
def _published_name(native: str) -> tuple[str, bool]:
|
| 625 |
+
"""Return the release key and whether it requires dense LoRA transposition."""
|
| 626 |
+
if not native.startswith("model.decoder.layers."):
|
| 627 |
+
return native, False
|
| 628 |
+
if ".experts." in native:
|
| 629 |
+
for projection, role in (("gate_up_proj", "gate_up"), ("down_proj", "down")):
|
| 630 |
+
prefix = f".experts.{projection}."
|
| 631 |
+
if prefix in native:
|
| 632 |
+
if native.endswith(".linear.weight"):
|
| 633 |
+
return native.replace(prefix + "linear.weight", f".experts.{projection}"), False
|
| 634 |
+
return native.replace(prefix + "lora_", f".experts.lora_{role}_"), False
|
| 635 |
+
return native.replace(".linear.weight", ".base.weight"), ".lora_" in native
|
| 636 |
+
|
| 637 |
+
|
| 638 |
+
def _expected_dtype(name: str) -> mx.Dtype:
|
| 639 |
+
if name.startswith("latent_deliberation.") and (
|
| 640 |
+
name.endswith((".dt_bias", ".a_log"))
|
| 641 |
+
or (name.endswith(".weight") and "norm" in name.rsplit(".", 2)[-2])
|
| 642 |
+
):
|
| 643 |
+
return mx.float32
|
| 644 |
+
return mx.bfloat16
|
| 645 |
+
|
| 646 |
+
|
| 647 |
+
def _verify_shard(path: Path, info: dict) -> None:
|
| 648 |
+
if path.stat().st_size != info["bytes"]:
|
| 649 |
+
raise ValueError(f"Incorrect shard size: {path.name}")
|
| 650 |
+
with path.open("rb") as source:
|
| 651 |
+
if hashlib.file_digest(source, "sha256").hexdigest() != info["sha256"]:
|
| 652 |
+
raise ValueError(f"Incorrect shard SHA-256: {path.name}")
|
| 653 |
+
|
| 654 |
+
|
| 655 |
+
def load_model(root: Path) -> MLXModilifyMk2:
|
| 656 |
+
config = _read_json(root / "config.json")
|
| 657 |
+
manifest = _read_json(root / "export_manifest.json")
|
| 658 |
+
protocol = {"state_schema_version": 25, "memory_architecture": "compact_gdn2_v2",
|
| 659 |
+
"memory_scheme": "dual_timescale_gdn2_trajectory_memory"}
|
| 660 |
+
if (manifest.get("format") != "modilify_mk2_mlx_inference_v1"
|
| 661 |
+
or any(config.get(k) != v or manifest.get(k) != v for k, v in protocol.items())
|
| 662 |
+
or config.get("checkpoint_step") != 4400 or manifest.get("global_step") != 4400
|
| 663 |
+
or config.get("canvas_length") != 256 or config.get("dtype") != "bfloat16"
|
| 664 |
+
or manifest.get("precision_policy") != "gdn2_small_fp32_v1"):
|
| 665 |
+
raise ValueError("Expected the step 4400 schema25 compact_gdn2_v2 MLX release.")
|
| 666 |
+
if not mx.metal.is_available():
|
| 667 |
+
raise RuntimeError("Modilify Mk2 MLX requires an Apple Silicon Metal device.")
|
| 668 |
+
model = _create_model(config)
|
| 669 |
+
weight_map = _read_json(root / "model.safetensors.index.json")["weight_map"]
|
| 670 |
+
tensors = {}
|
| 671 |
+
for native, value in tree_flatten(model.parameters()):
|
| 672 |
+
name, transpose = _published_name(native)
|
| 673 |
+
if name in tensors:
|
| 674 |
+
raise ValueError(f"Duplicate release weight mapping: {name}")
|
| 675 |
+
tensors[name] = (native, tuple(value.shape), _expected_dtype(name), transpose)
|
| 676 |
+
if set(tensors) != set(weight_map) or len(tensors) != manifest.get("total_tensor_count") or len(tensors) != 1323:
|
| 677 |
+
raise ValueError(
|
| 678 |
+
f"Weight index mismatch. Missing: {sorted(set(tensors) - set(weight_map))[:8]}. "
|
| 679 |
+
f"Unexpected: {sorted(set(weight_map) - set(tensors))[:8]}."
|
| 680 |
+
)
|
| 681 |
+
by_shard = defaultdict(set)
|
| 682 |
+
for name, shard in weight_map.items():
|
| 683 |
+
by_shard[shard].add(name)
|
| 684 |
+
shards = {entry["file"]: entry for entry in manifest["shards"]}
|
| 685 |
+
expected_files = {f"model-{i:05d}-of-00016.safetensors" for i in range(1, 17)}
|
| 686 |
+
if set(shards) != expected_files or set(by_shard) != expected_files or len(manifest["shards"]) != 16:
|
| 687 |
+
raise ValueError("Release manifest and index must describe all 16 shards.")
|
| 688 |
+
counts = Counter()
|
| 689 |
+
for shard in sorted(by_shard):
|
| 690 |
+
path = root / shard
|
| 691 |
+
_verify_shard(path, shards[shard])
|
| 692 |
+
raw = mx.load(str(path))
|
| 693 |
+
if set(raw) != by_shard[shard]:
|
| 694 |
+
raise ValueError(f"Weight index and shard disagree: {shard}")
|
| 695 |
+
updates = []
|
| 696 |
+
for name, value in raw.items():
|
| 697 |
+
native, shape, dtype, transpose = tensors[name]
|
| 698 |
+
if transpose:
|
| 699 |
+
value = value.T
|
| 700 |
+
if value.shape != shape or value.dtype != dtype:
|
| 701 |
+
raise ValueError(f"Invalid shape or dtype for {name}: {value.shape}, {value.dtype}")
|
| 702 |
+
updates.append((native, value))
|
| 703 |
+
counts["float32" if dtype == mx.float32 else "bfloat16"] += 1
|
| 704 |
+
model.load_weights(updates, strict=False)
|
| 705 |
+
mx.eval(*(value for _, value in updates))
|
| 706 |
+
del raw, updates
|
| 707 |
+
if counts != {"bfloat16": 1297, "float32": 26} or counts != manifest.get("tensor_dtypes"):
|
| 708 |
+
raise ValueError(f"Unexpected release precision counts: {dict(counts)}")
|
| 709 |
+
return model
|
modilify_mk2_mlx/runtime.py
ADDED
|
@@ -0,0 +1,379 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Text generation and commit policy for the schema25 MLX release."""
|
| 2 |
+
|
| 3 |
+
from __future__ import annotations
|
| 4 |
+
|
| 5 |
+
from collections.abc import Callable, Sequence
|
| 6 |
+
from dataclasses import dataclass, replace
|
| 7 |
+
import math
|
| 8 |
+
from pathlib import Path
|
| 9 |
+
from types import SimpleNamespace
|
| 10 |
+
|
| 11 |
+
import mlx.core as mx
|
| 12 |
+
from tokenizers.decoders import DecodeStream
|
| 13 |
+
from transformers import GemmaTokenizer
|
| 14 |
+
|
| 15 |
+
from .model import GDN2TrajectoryState, MLXModilifyMk2, _read_json, load_model
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
JUMP_FAILURE_BUDGET = 2.0
|
| 19 |
+
FUSED_EPS = 1.0e-6
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
def commit_target_confidence_bias(
|
| 23 |
+
target_confidence: float | None,
|
| 24 |
+
failure_budget: float = 0.2,
|
| 25 |
+
budget_safety_ratio: float = 0.85,
|
| 26 |
+
) -> float:
|
| 27 |
+
if target_confidence is None or target_confidence <= 0.0 or target_confidence >= 1.0:
|
| 28 |
+
return 0.0
|
| 29 |
+
target_failure = min(budget_safety_ratio * failure_budget, 1.0 - target_confidence)
|
| 30 |
+
target_failure = max(target_failure, 1.0e-4)
|
| 31 |
+
target_conf = 1.0 - target_failure
|
| 32 |
+
logit_c = math.log(target_conf / (1.0 - target_conf))
|
| 33 |
+
logit_p = math.log(target_confidence / (1.0 - target_confidence))
|
| 34 |
+
return max(logit_c - logit_p, 0.0)
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
def fused_commit_confidence(
|
| 38 |
+
proposal_confidence: mx.array,
|
| 39 |
+
token_entropy: mx.array,
|
| 40 |
+
*,
|
| 41 |
+
eps: float = FUSED_EPS,
|
| 42 |
+
entropy_weight: float = 1.0,
|
| 43 |
+
confidence_power: float = 2.0,
|
| 44 |
+
top_k: int | None = None,
|
| 45 |
+
min_p: float | None = None,
|
| 46 |
+
target_confidence: float | None = None,
|
| 47 |
+
failure_budget: float = 0.2,
|
| 48 |
+
) -> mx.array:
|
| 49 |
+
"""Match the schema25 excess-entropy sigmoid and checkpoint power."""
|
| 50 |
+
|
| 51 |
+
p = mx.clip(mx.nan_to_num(proposal_confidence.astype(mx.float32), nan=0.5), eps, 1.0 - eps)
|
| 52 |
+
entropy = mx.maximum(mx.nan_to_num(token_entropy.astype(mx.float32), nan=0.0), 0.0)
|
| 53 |
+
binary_entropy = -p * mx.log(p) - (1.0 - p) * mx.log1p(-p)
|
| 54 |
+
excess = mx.maximum(entropy - binary_entropy, 0.0)
|
| 55 |
+
k_eff = None
|
| 56 |
+
if top_k is not None and top_k > 0:
|
| 57 |
+
k_eff = mx.full(p.shape, float(top_k), dtype=mx.float32)
|
| 58 |
+
if min_p is not None and min_p > 0:
|
| 59 |
+
thresh = mx.maximum(float(min_p) * p, 1.0e-6)
|
| 60 |
+
k_min_p = mx.maximum((1.0 - p) / thresh, 1.0)
|
| 61 |
+
k_eff = k_min_p if k_eff is None else mx.minimum(k_eff, k_min_p)
|
| 62 |
+
if k_eff is not None:
|
| 63 |
+
max_excess = (1.0 - p) * mx.log(k_eff)
|
| 64 |
+
excess = mx.minimum(excess, max_excess)
|
| 65 |
+
bias = commit_target_confidence_bias(target_confidence, failure_budget=failure_budget)
|
| 66 |
+
fused_logit = mx.log(p) - mx.log1p(-p) + bias - entropy_weight * excess
|
| 67 |
+
return mx.clip(mx.sigmoid(fused_logit) ** confidence_power, eps, 1.0 - eps)
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
def fused_commit_failure_rate(
|
| 71 |
+
proposal_confidence: mx.array, token_entropy: mx.array, **kwargs: object
|
| 72 |
+
) -> mx.array:
|
| 73 |
+
return 1.0 - fused_commit_confidence(proposal_confidence, token_entropy, **kwargs)
|
| 74 |
+
|
| 75 |
+
|
| 76 |
+
def _terminal_mask(tokens: mx.array, stop_token_id: int | Sequence[int]) -> mx.array:
|
| 77 |
+
ids = (stop_token_id,) if isinstance(stop_token_id, int) else tuple(dict.fromkeys(stop_token_id))
|
| 78 |
+
if not ids:
|
| 79 |
+
raise ValueError("At least one stop token ID is required.")
|
| 80 |
+
matches = tokens == int(ids[0])
|
| 81 |
+
for value in ids[1:]:
|
| 82 |
+
matches = matches | (tokens == int(value))
|
| 83 |
+
return matches
|
| 84 |
+
|
| 85 |
+
|
| 86 |
+
def _prefix_length(risk: mx.array, valid: mx.array, budget: float) -> mx.array:
|
| 87 |
+
# Nonnegative risk and contiguous validity make the allowed prefix monotonic.
|
| 88 |
+
allowed = (mx.cumsum(mx.clip(risk, 0.0, 1.0) * valid, axis=-1) < budget) & valid
|
| 89 |
+
return mx.sum(allowed, axis=-1).astype(mx.int32)
|
| 90 |
+
|
| 91 |
+
|
| 92 |
+
def _commit(
|
| 93 |
+
sampled, risk, previous_risk, greedy, greedy_risk, *,
|
| 94 |
+
remaining, stops, budget, ponder, stagnation, generation,
|
| 95 |
+
):
|
| 96 |
+
positions = mx.arange(sampled.shape[-1])[None, :]
|
| 97 |
+
valid = positions < remaining
|
| 98 |
+
|
| 99 |
+
def bounded(ids, rates, limit):
|
| 100 |
+
length = _prefix_length(rates, valid, limit)
|
| 101 |
+
terminal = _terminal_mask(ids, stops) & (positions < length[:, None])
|
| 102 |
+
first = mx.min(mx.where(terminal, positions + 1, sampled.shape[-1]), axis=-1)
|
| 103 |
+
return mx.minimum(length, first)
|
| 104 |
+
|
| 105 |
+
normal = bounded(sampled, risk, budget)
|
| 106 |
+
previous = _prefix_length(previous_risk, valid, budget)
|
| 107 |
+
frontier = mx.minimum(mx.maximum(previous, normal) + 1, mx.sum(valid, axis=-1))
|
| 108 |
+
progress = mx.sum((previous_risk - risk) * (positions < frontier[:, None]), axis=-1)
|
| 109 |
+
progress = progress / mx.maximum(frontier, 1)
|
| 110 |
+
next_ponder = mx.where(normal > 0, 0, ponder + 1)
|
| 111 |
+
next_stagnation = mx.where(normal > 0, 0, stagnation + 1)
|
| 112 |
+
jump = (normal == 0) & (
|
| 113 |
+
((next_stagnation >= generation.jump_on_no_progress_after)
|
| 114 |
+
& (progress <= generation.min_trajectory_progress))
|
| 115 |
+
| (next_ponder >= generation.max_ponder_steps)
|
| 116 |
+
)
|
| 117 |
+
lengths = mx.where(jump, bounded(greedy, greedy_risk, JUMP_FAILURE_BUDGET), normal)
|
| 118 |
+
return (lengths, mx.where(jump[:, None], greedy, sampled),
|
| 119 |
+
mx.where(lengths > 0, 0, next_ponder), mx.where(lengths > 0, 0, next_stagnation))
|
| 120 |
+
|
| 121 |
+
|
| 122 |
+
def seed_key(seed: int, salt: int) -> mx.array:
|
| 123 |
+
return mx.random.key((seed + salt) % (2**32))
|
| 124 |
+
|
| 125 |
+
|
| 126 |
+
def proposal_statistics(
|
| 127 |
+
hidden: mx.array, weight: mx.array, *, seed: int, step: int,
|
| 128 |
+
softcap: float, temperature: float, top_k: int, min_p: float,
|
| 129 |
+
repetition_penalty: float, repetition_seen: set[int], chunk_size: int = 32768,
|
| 130 |
+
) -> tuple[mx.array, ...]:
|
| 131 |
+
canvas = hidden.shape[1]
|
| 132 |
+
flat = hidden[0]
|
| 133 |
+
log_z = mx.full((canvas,), -mx.inf, mx.float32)
|
| 134 |
+
greedy_score = log_z
|
| 135 |
+
greedy = mx.zeros((canvas,), mx.int32)
|
| 136 |
+
moment_sum = mx.zeros_like(log_z)
|
| 137 |
+
moment_weighted = mx.zeros_like(log_z)
|
| 138 |
+
candidates = []
|
| 139 |
+
candidate_ids = []
|
| 140 |
+
if repetition_penalty != 1:
|
| 141 |
+
seen = mx.zeros((weight.shape[0],), mx.bool_)
|
| 142 |
+
if repetition_seen:
|
| 143 |
+
seen[mx.array(sorted(repetition_seen), mx.int32)] = True
|
| 144 |
+
|
| 145 |
+
for start in range(0, weight.shape[0], chunk_size):
|
| 146 |
+
stop = min(start + chunk_size, weight.shape[0])
|
| 147 |
+
score = mx.tanh((flat @ weight[start:stop].T).astype(mx.float32) / softcap) * softcap
|
| 148 |
+
if repetition_penalty != 1:
|
| 149 |
+
penalized = mx.where(score < 0, score * repetition_penalty, score / repetition_penalty)
|
| 150 |
+
score = mx.where(seen[start:stop], penalized, score)
|
| 151 |
+
score = score / temperature
|
| 152 |
+
log_z = mx.logaddexp(log_z, mx.logsumexp(score, axis=-1))
|
| 153 |
+
local_max = mx.max(score, axis=-1)
|
| 154 |
+
greedy = mx.where(local_max > greedy_score, mx.argmax(score, axis=-1) + start, greedy)
|
| 155 |
+
next_max = mx.maximum(greedy_score, local_max)
|
| 156 |
+
previous_scale = mx.exp(greedy_score - next_max)
|
| 157 |
+
local_weights = mx.exp(score - next_max[:, None])
|
| 158 |
+
moment_sum = moment_sum * previous_scale + mx.sum(local_weights, axis=-1)
|
| 159 |
+
moment_weighted = moment_weighted * previous_scale + mx.sum(local_weights * score, axis=-1)
|
| 160 |
+
greedy_score = next_max
|
| 161 |
+
|
| 162 |
+
k = min(top_k, stop - start)
|
| 163 |
+
indices = mx.argpartition(-score, kth=k - 1, axis=-1)[:, :k]
|
| 164 |
+
candidates.append(mx.take_along_axis(score, indices, axis=-1))
|
| 165 |
+
candidate_ids.append(indices + start)
|
| 166 |
+
mx.async_eval(log_z, greedy, greedy_score, moment_sum, moment_weighted,
|
| 167 |
+
candidates[-1], candidate_ids[-1])
|
| 168 |
+
|
| 169 |
+
all_scores = mx.concatenate(candidates, axis=-1)
|
| 170 |
+
all_ids = mx.concatenate(candidate_ids, axis=-1)
|
| 171 |
+
k = min(top_k, all_scores.shape[-1])
|
| 172 |
+
indices = mx.argpartition(-all_scores, kth=k - 1, axis=-1)[:, :k]
|
| 173 |
+
scores = mx.take_along_axis(all_scores, indices, axis=-1)
|
| 174 |
+
ids = mx.take_along_axis(all_ids, indices, axis=-1)
|
| 175 |
+
eligible = mx.where(scores >= greedy_score[:, None] + math.log(min_p), scores, -mx.inf)
|
| 176 |
+
uniform = mx.random.uniform(shape=(canvas, k), key=seed_key(seed, 0xC09A + step * 1000003))
|
| 177 |
+
uniform = mx.clip(uniform, 1.17549435e-38, 1.0 - 1.19209290e-7)
|
| 178 |
+
chosen = mx.argmax(eligible - mx.log(-mx.log(uniform)), axis=-1, keepdims=True)
|
| 179 |
+
token_ids = mx.take_along_axis(ids, chosen, axis=-1)[:, 0]
|
| 180 |
+
chosen_score = mx.take_along_axis(scores, chosen, axis=-1)[:, 0]
|
| 181 |
+
confidence = mx.clip(mx.exp(chosen_score - log_z), 0.0, 1.0)
|
| 182 |
+
greedy_confidence = mx.clip(mx.exp(greedy_score - log_z), 0.0, 1.0)
|
| 183 |
+
entropy = log_z - moment_weighted / moment_sum
|
| 184 |
+
return tuple(value[None] for value in (token_ids, confidence, entropy, greedy, greedy_confidence))
|
| 185 |
+
|
| 186 |
+
|
| 187 |
+
@dataclass(frozen=True)
|
| 188 |
+
class GenerationResult:
|
| 189 |
+
text: str
|
| 190 |
+
token_ids: tuple[int, ...]
|
| 191 |
+
denoise_steps: int
|
| 192 |
+
stop_reason: str
|
| 193 |
+
|
| 194 |
+
|
| 195 |
+
def _positive_int(value: int, name: str) -> None:
|
| 196 |
+
if type(value) is not int or value < 1:
|
| 197 |
+
raise ValueError(f"{name} must be a positive integer.")
|
| 198 |
+
|
| 199 |
+
|
| 200 |
+
class Runtime:
|
| 201 |
+
def __init__(self, model: MLXModilifyMk2, tokenizer, generation: dict):
|
| 202 |
+
self.model = model
|
| 203 |
+
self.tokenizer = tokenizer
|
| 204 |
+
self.generation = SimpleNamespace(**generation)
|
| 205 |
+
|
| 206 |
+
def generate(
|
| 207 |
+
self, messages: Sequence[dict[str, str]], *, enable_thinking: bool = True,
|
| 208 |
+
seed: int = 42, max_new_tokens: int = 256, max_denoising_steps: int = 256,
|
| 209 |
+
on_text: Callable[[str], None] | None = None,
|
| 210 |
+
) -> GenerationResult:
|
| 211 |
+
"""Generate one text response and optionally emit committed text increments."""
|
| 212 |
+
_positive_int(max_new_tokens, "max_new_tokens")
|
| 213 |
+
_positive_int(max_denoising_steps, "max_denoising_steps")
|
| 214 |
+
if type(seed) is not int:
|
| 215 |
+
raise ValueError("seed must be an integer.")
|
| 216 |
+
if not messages or any(
|
| 217 |
+
not isinstance(message, dict)
|
| 218 |
+
or message.get("role") not in ("system", "user", "assistant")
|
| 219 |
+
or not isinstance(message.get("content"), str)
|
| 220 |
+
for message in messages
|
| 221 |
+
):
|
| 222 |
+
raise ValueError("Expected a non-empty sequence of system, user or assistant text messages.")
|
| 223 |
+
prompt_ids = self.tokenizer.apply_chat_template(
|
| 224 |
+
list(messages), tokenize=True, add_generation_prompt=True,
|
| 225 |
+
enable_thinking=enable_thinking, return_dict=True,
|
| 226 |
+
)["input_ids"]
|
| 227 |
+
max_positions = self.model.config.text_config.max_position_embeddings
|
| 228 |
+
if not prompt_ids or len(prompt_ids) + max_new_tokens > max_positions:
|
| 229 |
+
raise ValueError("Prompt and response budget must fit within the model position limit.")
|
| 230 |
+
with mx.stream(mx.gpu):
|
| 231 |
+
return self._generate(prompt_ids, seed, max_new_tokens, max_denoising_steps, on_text)
|
| 232 |
+
|
| 233 |
+
def _generate(
|
| 234 |
+
self, prompt_ids: list[int], seed: int, max_new_tokens: int,
|
| 235 |
+
max_denoising_steps: int, on_text: Callable[[str], None] | None,
|
| 236 |
+
) -> GenerationResult:
|
| 237 |
+
config = self.model.config
|
| 238 |
+
decoder = self.model.model.decoder
|
| 239 |
+
encoder = self.model.model.encoder
|
| 240 |
+
cache = encoder.make_cache()
|
| 241 |
+
for start in range(0, len(prompt_ids), 256):
|
| 242 |
+
_, cache = encoder(mx.array([prompt_ids[start:start + 256]], mx.int32), cache=cache)
|
| 243 |
+
mx.eval(*(value for layer in cache for value in (layer.keys, layer.values)
|
| 244 |
+
if isinstance(value, mx.array)))
|
| 245 |
+
|
| 246 |
+
generation = self.generation
|
| 247 |
+
vocab = config.text_config.vocab_size
|
| 248 |
+
canvas = config.canvas_length
|
| 249 |
+
positions = mx.arange(canvas)[None, :]
|
| 250 |
+
unknown_entropy = math.log(vocab)
|
| 251 |
+
memory = GDN2TrajectoryState(
|
| 252 |
+
mx.zeros((1, canvas, 16, 64, 64), mx.float32),
|
| 253 |
+
mx.zeros((1, 16, 64, 64), mx.float32),
|
| 254 |
+
mx.zeros((1, 16, 128, 128), mx.float32),
|
| 255 |
+
mx.zeros((1, canvas), mx.bool_),
|
| 256 |
+
)
|
| 257 |
+
confidence = mx.zeros((1, canvas), mx.float32)
|
| 258 |
+
entropy = mx.full((1, canvas), unknown_entropy, mx.float32)
|
| 259 |
+
clocks = mx.zeros((1,), mx.int32)
|
| 260 |
+
ponder = stagnation = head = clocks
|
| 261 |
+
|
| 262 |
+
def noise(step: int) -> mx.array:
|
| 263 |
+
return mx.random.randint(0, vocab, (1, canvas), key=seed_key(seed, 0x51A7 + step * 1000003))
|
| 264 |
+
|
| 265 |
+
tokens = mx.where(positions < max_new_tokens, noise(0), generation.pad_token_id)
|
| 266 |
+
turn_end = generation.turn_end_token_id
|
| 267 |
+
eos = generation.eos_token_id
|
| 268 |
+
stops = tuple(dict.fromkeys((turn_end, *(eos if isinstance(eos, list) else [eos]))))
|
| 269 |
+
excluded = set(generation.repetition_penalty_exclude_token_ids)
|
| 270 |
+
excluded.update((generation.pad_token_id, generation.bos_token_id, *stops))
|
| 271 |
+
seen = set(prompt_ids) - excluded
|
| 272 |
+
generated = []
|
| 273 |
+
stream = DecodeStream(skip_special_tokens=True) if on_text is not None else None
|
| 274 |
+
emitted = ""
|
| 275 |
+
watchdog = max_new_tokens * generation.max_ponder_steps
|
| 276 |
+
|
| 277 |
+
def failure(confidence: mx.array, entropy: mx.array) -> mx.array:
|
| 278 |
+
return fused_commit_failure_rate(
|
| 279 |
+
confidence, entropy, entropy_weight=config.commit_entropy_weight,
|
| 280 |
+
confidence_power=config.commit_confidence_power, top_k=config.commit_top_k,
|
| 281 |
+
min_p=config.commit_min_p, target_confidence=config.commit_target_confidence,
|
| 282 |
+
failure_budget=config.commit_failure_budget,
|
| 283 |
+
)
|
| 284 |
+
|
| 285 |
+
limit = min(max_denoising_steps, watchdog)
|
| 286 |
+
for step in range(1, limit + 1):
|
| 287 |
+
output = self.model.forward_canvas(
|
| 288 |
+
tokens, prefix_cache=cache, state=memory, canvas_head=head,
|
| 289 |
+
first_step=step == 1, has_commits=bool(generated),
|
| 290 |
+
)
|
| 291 |
+
sampled, next_confidence, next_entropy, greedy, greedy_confidence = proposal_statistics(
|
| 292 |
+
output.heavy_hidden, decoder.embed_tokens.weight, seed=seed, step=step - 1,
|
| 293 |
+
softcap=config.text_config.final_logit_softcapping, temperature=generation.t_min,
|
| 294 |
+
top_k=config.commit_top_k, min_p=config.commit_min_p,
|
| 295 |
+
repetition_penalty=generation.repetition_penalty, repetition_seen=seen,
|
| 296 |
+
)
|
| 297 |
+
remaining = max_new_tokens - len(generated)
|
| 298 |
+
logical_order = (head[:, None] + positions) % canvas
|
| 299 |
+
physical_offsets = (positions - head[:, None]) % canvas
|
| 300 |
+
|
| 301 |
+
def logical(values: mx.array) -> mx.array:
|
| 302 |
+
return mx.take_along_axis(values, logical_order, axis=1)
|
| 303 |
+
|
| 304 |
+
lengths, selected, ponder, stagnation = _commit(
|
| 305 |
+
logical(sampled), logical(failure(next_confidence, next_entropy)),
|
| 306 |
+
logical(failure(confidence, entropy)),
|
| 307 |
+
logical(greedy), logical(failure(greedy_confidence, next_entropy)),
|
| 308 |
+
ponder=ponder, stagnation=stagnation, remaining=remaining,
|
| 309 |
+
budget=config.commit_failure_budget, stops=stops, generation=generation,
|
| 310 |
+
)
|
| 311 |
+
mx.eval(lengths, selected)
|
| 312 |
+
count = int(lengths.item())
|
| 313 |
+
block = selected[0, :count].tolist()
|
| 314 |
+
generated.extend(block)
|
| 315 |
+
seen.update(set(block) - excluded)
|
| 316 |
+
if stream is not None and block:
|
| 317 |
+
delta = stream.step(self.tokenizer.backend_tokenizer, block)
|
| 318 |
+
if delta:
|
| 319 |
+
on_text(delta)
|
| 320 |
+
emitted += delta
|
| 321 |
+
|
| 322 |
+
if turn_end in block:
|
| 323 |
+
reason = "turn_end"
|
| 324 |
+
break
|
| 325 |
+
if any(token in stops for token in block):
|
| 326 |
+
reason = "eos"
|
| 327 |
+
break
|
| 328 |
+
if len(generated) >= max_new_tokens:
|
| 329 |
+
reason = "max_new_tokens"
|
| 330 |
+
break
|
| 331 |
+
if step == limit:
|
| 332 |
+
continue
|
| 333 |
+
|
| 334 |
+
memory = self.model.latent_deliberation.trajectory.observe(
|
| 335 |
+
memory, output.heavy_hidden + output.working_state, physical_offsets < remaining, head,
|
| 336 |
+
)
|
| 337 |
+
tokens, confidence, entropy = sampled, next_confidence, next_entropy
|
| 338 |
+
if count:
|
| 339 |
+
committed_ids = selected[:, :count]
|
| 340 |
+
persistent = self.model.latent_deliberation.commit_write(
|
| 341 |
+
memory=memory.persistent, working_state=output.working_state,
|
| 342 |
+
heavy_hidden=output.heavy_hidden,
|
| 343 |
+
committed_token_embeddings=decoder.embed_tokens(committed_ids) * decoder.embed_scale,
|
| 344 |
+
canvas_head=head,
|
| 345 |
+
)
|
| 346 |
+
memory = replace(memory.clear_refill(lengths, head), persistent=persistent)
|
| 347 |
+
_, cache = encoder(committed_ids, cache=cache)
|
| 348 |
+
head = (head + count) % canvas
|
| 349 |
+
refill = physical_offsets < count
|
| 350 |
+
tail = mx.take_along_axis(noise(step), physical_offsets, axis=1)
|
| 351 |
+
pad = ((positions - head[:, None]) % canvas) >= remaining - count
|
| 352 |
+
tokens = mx.where(refill, mx.where(pad, generation.pad_token_id, tail), tokens)
|
| 353 |
+
confidence = mx.where(refill, 0.0, confidence)
|
| 354 |
+
entropy = mx.where(refill, unknown_entropy, entropy)
|
| 355 |
+
mx.async_eval(tokens, confidence, entropy, memory.cells, memory.row, memory.persistent, memory.seen,
|
| 356 |
+
*(value for layer in cache for value in (layer.keys, layer.values)
|
| 357 |
+
if isinstance(value, mx.array)))
|
| 358 |
+
else:
|
| 359 |
+
reason = "max_denoising_steps" if max_denoising_steps <= watchdog else "episode_watchdog"
|
| 360 |
+
|
| 361 |
+
text = self.tokenizer.backend_tokenizer.decode(generated, skip_special_tokens=True)
|
| 362 |
+
if on_text is not None and emitted != text:
|
| 363 |
+
if not text.startswith(emitted):
|
| 364 |
+
raise RuntimeError("Tokenizer streaming output diverged from complete decode.")
|
| 365 |
+
on_text(text[len(emitted):])
|
| 366 |
+
return GenerationResult(text, tuple(generated), step, reason)
|
| 367 |
+
|
| 368 |
+
|
| 369 |
+
def load(model_dir: str | Path | None = None) -> Runtime:
|
| 370 |
+
"""Open a local release directory with model computation on Metal."""
|
| 371 |
+
root = Path(model_dir).expanduser().resolve() if model_dir is not None else Path(__file__).resolve().parents[1]
|
| 372 |
+
generation = _read_json(root / "generation_config.json")
|
| 373 |
+
# Keep the released Gemma tokenizer regex as stored in tokenizer.json.
|
| 374 |
+
tokenizer = GemmaTokenizer.from_pretrained(
|
| 375 |
+
root, local_files_only=True, fix_mistral_regex=False,
|
| 376 |
+
)
|
| 377 |
+
with mx.stream(mx.gpu):
|
| 378 |
+
model = load_model(root)
|
| 379 |
+
return Runtime(model, tokenizer, generation)
|
requirements.txt
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
mlx==0.32.4
|
| 2 |
+
mlx-lm==0.32.2
|
| 3 |
+
mlx-vlm==0.7.12
|
| 4 |
+
transformers==5.14.1
|
| 5 |
+
tokenizers==0.22.2
|
tokenizer.json
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:cc8d3a0ce36466ccc1278bf987df5f71db1719b9ca6b4118264f45cb627bfe0f
|
| 3 |
+
size 32169626
|
tokenizer_config.json
ADDED
|
@@ -0,0 +1,96 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"audio_token": "<|audio|>",
|
| 3 |
+
"backend": "tokenizers",
|
| 4 |
+
"boa_token": "<|audio>",
|
| 5 |
+
"boi_token": "<|image>",
|
| 6 |
+
"bos_token": "<bos>",
|
| 7 |
+
"eoa_token": "<audio|>",
|
| 8 |
+
"eoc_token": "<channel|>",
|
| 9 |
+
"eoi_token": "<image|>",
|
| 10 |
+
"eos_token": "<eos>",
|
| 11 |
+
"eot_token": "<turn|>",
|
| 12 |
+
"escape_token": "<|\"|>",
|
| 13 |
+
"etc_token": "<tool_call|>",
|
| 14 |
+
"etd_token": "<tool|>",
|
| 15 |
+
"etr_token": "<tool_response|>",
|
| 16 |
+
"extra_special_tokens": [
|
| 17 |
+
"<|video|>"
|
| 18 |
+
],
|
| 19 |
+
"image_token": "<|image|>",
|
| 20 |
+
"is_local": true,
|
| 21 |
+
"local_files_only": true,
|
| 22 |
+
"mask_token": "<mask>",
|
| 23 |
+
"model_max_length": 1000000000000000019884624838656,
|
| 24 |
+
"model_specific_special_tokens": {
|
| 25 |
+
"audio_token": "<|audio|>",
|
| 26 |
+
"boa_token": "<|audio>",
|
| 27 |
+
"boi_token": "<|image>",
|
| 28 |
+
"eoa_token": "<audio|>",
|
| 29 |
+
"eoc_token": "<channel|>",
|
| 30 |
+
"eoi_token": "<image|>",
|
| 31 |
+
"eot_token": "<turn|>",
|
| 32 |
+
"escape_token": "<|\"|>",
|
| 33 |
+
"etc_token": "<tool_call|>",
|
| 34 |
+
"etd_token": "<tool|>",
|
| 35 |
+
"etr_token": "<tool_response|>",
|
| 36 |
+
"image_token": "<|image|>",
|
| 37 |
+
"soc_token": "<|channel>",
|
| 38 |
+
"sot_token": "<|turn>",
|
| 39 |
+
"stc_token": "<|tool_call>",
|
| 40 |
+
"std_token": "<|tool>",
|
| 41 |
+
"str_token": "<|tool_response>",
|
| 42 |
+
"think_token": "<|think|>"
|
| 43 |
+
},
|
| 44 |
+
"pad_token": "<pad>",
|
| 45 |
+
"padding_side": "left",
|
| 46 |
+
"processor_class": "Gemma4Processor",
|
| 47 |
+
"response_schema": {
|
| 48 |
+
"properties": {
|
| 49 |
+
"content": {
|
| 50 |
+
"type": "string"
|
| 51 |
+
},
|
| 52 |
+
"role": {
|
| 53 |
+
"const": "assistant"
|
| 54 |
+
},
|
| 55 |
+
"thinking": {
|
| 56 |
+
"type": "string"
|
| 57 |
+
},
|
| 58 |
+
"tool_calls": {
|
| 59 |
+
"items": {
|
| 60 |
+
"properties": {
|
| 61 |
+
"function": {
|
| 62 |
+
"properties": {
|
| 63 |
+
"arguments": {
|
| 64 |
+
"additionalProperties": {},
|
| 65 |
+
"type": "object",
|
| 66 |
+
"x-parser": "gemma4-tool-call"
|
| 67 |
+
},
|
| 68 |
+
"name": {
|
| 69 |
+
"type": "string"
|
| 70 |
+
}
|
| 71 |
+
},
|
| 72 |
+
"type": "object",
|
| 73 |
+
"x-regex": "call\\:(?P<name>\\w+)(?P<arguments>\\{.*\\})"
|
| 74 |
+
},
|
| 75 |
+
"type": {
|
| 76 |
+
"const": "function"
|
| 77 |
+
}
|
| 78 |
+
},
|
| 79 |
+
"type": "object"
|
| 80 |
+
},
|
| 81 |
+
"type": "array",
|
| 82 |
+
"x-regex-iterator": "<\\|tool_call>(.*?)<tool_call\\|>"
|
| 83 |
+
}
|
| 84 |
+
},
|
| 85 |
+
"type": "object",
|
| 86 |
+
"x-regex": "(\\<\\|channel\\>thought\\n(?P<thinking>.*?)\\<channel\\|\\>)?(?P<tool_calls>\\<\\|tool_call\\>.*\\<tool_call\\|\\>)?(?P<content>(?:(?!\\<turn\\|\\>)(?!\\<\\|tool_response\\>).)+)?(?:\\<turn\\|\\>|\\<\\|tool_response\\>)?"
|
| 87 |
+
},
|
| 88 |
+
"soc_token": "<|channel>",
|
| 89 |
+
"sot_token": "<|turn>",
|
| 90 |
+
"stc_token": "<|tool_call>",
|
| 91 |
+
"std_token": "<|tool>",
|
| 92 |
+
"str_token": "<|tool_response>",
|
| 93 |
+
"think_token": "<|think|>",
|
| 94 |
+
"tokenizer_class": "GemmaTokenizer",
|
| 95 |
+
"unk_token": "<unk>"
|
| 96 |
+
}
|