Text Generation
GGUF
qwen4_exp
Mixture of Experts
mixture-of-experts
deltanet
sparse-attention
hyper-connections
qwen3.8
qwen3.8-flash-next
cpu
efficient
ternary
bitnet
jirack
routing
tool-call
robotics
ternary-weights
ternary-transformer
quantization
qat
conversational
Instructions to use CMSManhattan/JiRackDeltaNet_180b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use CMSManhattan/JiRackDeltaNet_180b with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf CMSManhattan/JiRackDeltaNet_180b:Q3_K_M # Run inference directly in the terminal: llama cli -hf CMSManhattan/JiRackDeltaNet_180b:Q3_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf CMSManhattan/JiRackDeltaNet_180b:Q3_K_M # Run inference directly in the terminal: llama cli -hf CMSManhattan/JiRackDeltaNet_180b:Q3_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf CMSManhattan/JiRackDeltaNet_180b:Q3_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf CMSManhattan/JiRackDeltaNet_180b:Q3_K_M
Use Docker
docker model run hf.co/CMSManhattan/JiRackDeltaNet_180b:Q3_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use CMSManhattan/JiRackDeltaNet_180b with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "CMSManhattan/JiRackDeltaNet_180b" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "CMSManhattan/JiRackDeltaNet_180b", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/CMSManhattan/JiRackDeltaNet_180b:Q3_K_M
- Ollama
How to use CMSManhattan/JiRackDeltaNet_180b with Ollama:
ollama run hf.co/CMSManhattan/JiRackDeltaNet_180b:Q3_K_M
- Unsloth Desktop
- Pi
How to use CMSManhattan/JiRackDeltaNet_180b with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf CMSManhattan/JiRackDeltaNet_180b:Q3_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "CMSManhattan/JiRackDeltaNet_180b:Q3_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use CMSManhattan/JiRackDeltaNet_180b with Docker Model Runner:
docker model run hf.co/CMSManhattan/JiRackDeltaNet_180b:Q3_K_M
- Lemonade
How to use CMSManhattan/JiRackDeltaNet_180b with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull CMSManhattan/JiRackDeltaNet_180b:Q3_K_M
Run and chat with the model
lemonade run user.JiRackDeltaNet_180b-Q3_K_M
List all available models
lemonade list
- Hermes Agent
How to use CMSManhattan/JiRackDeltaNet_180b with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf CMSManhattan/JiRackDeltaNet_180b:Q3_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default CMSManhattan/JiRackDeltaNet_180b:Q3_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use CMSManhattan/JiRackDeltaNet_180b with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf CMSManhattan/JiRackDeltaNet_180b:Q3_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "CMSManhattan/JiRackDeltaNet_180b:Q3_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Commit ·
84bc44f
0
Parent(s):
JiRack DeltaNet 180B
Browse filesThis view is limited to 50 files because it contains too many changes. See raw diff
- .gitattributes +47 -0
- JiRackDeltaNet_180b-00001-of-00007.gguf +3 -0
- JiRackDeltaNet_180b-00002-of-00007.gguf +3 -0
- JiRackDeltaNet_180b-00003-of-00007.gguf +3 -0
- JiRackDeltaNet_180b-00004-of-00007.gguf +3 -0
- JiRackDeltaNet_180b-00005-of-00007.gguf +3 -0
- JiRackDeltaNet_180b-00006-of-00007.gguf +3 -0
- JiRackDeltaNet_180b-00007-of-00007.gguf +3 -0
- JiRackDeltaNet_180b.py +1512 -0
- JiRackDeltaNet_180b_Q3_K_M.gguf +3 -0
- JiRackDeltaNet_180b_Q4_K_M.gguf +3 -0
- LICENSE-CODE +26 -0
- LICENSE-QWEN +16 -0
- README.md +299 -0
- chat_jirack_180b.py +65 -0
- chat_template.jinja +170 -0
- checkpoint-00001-of-00043.safetensors +3 -0
- checkpoint-00002-of-00043.safetensors +3 -0
- checkpoint-00003-of-00043.safetensors +3 -0
- checkpoint-00004-of-00043.safetensors +3 -0
- checkpoint-00005-of-00043.safetensors +3 -0
- checkpoint-00006-of-00043.safetensors +3 -0
- checkpoint-00007-of-00043.safetensors +3 -0
- checkpoint-00008-of-00043.safetensors +3 -0
- checkpoint-00009-of-00043.safetensors +3 -0
- checkpoint-00010-of-00043.safetensors +3 -0
- checkpoint-00011-of-00043.safetensors +3 -0
- checkpoint-00012-of-00043.safetensors +3 -0
- checkpoint-00013-of-00043.safetensors +3 -0
- checkpoint-00014-of-00043.safetensors +3 -0
- checkpoint-00015-of-00043.safetensors +3 -0
- checkpoint-00016-of-00043.safetensors +3 -0
- checkpoint-00017-of-00043.safetensors +3 -0
- checkpoint-00018-of-00043.safetensors +3 -0
- checkpoint-00019-of-00043.safetensors +3 -0
- checkpoint-00020-of-00043.safetensors +3 -0
- checkpoint-00021-of-00043.safetensors +3 -0
- checkpoint-00022-of-00043.safetensors +3 -0
- checkpoint-00023-of-00043.safetensors +3 -0
- checkpoint-00024-of-00043.safetensors +3 -0
- checkpoint-00025-of-00043.safetensors +3 -0
- checkpoint-00026-of-00043.safetensors +3 -0
- checkpoint-00027-of-00043.safetensors +3 -0
- checkpoint-00028-of-00043.safetensors +3 -0
- checkpoint-00029-of-00043.safetensors +3 -0
- checkpoint-00030-of-00043.safetensors +3 -0
- checkpoint-00031-of-00043.safetensors +3 -0
- checkpoint-00032-of-00043.safetensors +3 -0
- checkpoint-00033-of-00043.safetensors +3 -0
- checkpoint-00034-of-00043.safetensors +3 -0
.gitattributes
ADDED
|
@@ -0,0 +1,47 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
*.7z filter=lfs diff=lfs merge=lfs -text
|
| 2 |
+
*.arrow filter=lfs diff=lfs merge=lfs -text
|
| 3 |
+
*.bin filter=lfs diff=lfs merge=lfs -text
|
| 4 |
+
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
| 5 |
+
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
| 6 |
+
*.ftz filter=lfs diff=lfs merge=lfs -text
|
| 7 |
+
*.gz filter=lfs diff=lfs merge=lfs -text
|
| 8 |
+
*.h5 filter=lfs diff=lfs merge=lfs -text
|
| 9 |
+
*.joblib filter=lfs diff=lfs merge=lfs -text
|
| 10 |
+
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
| 11 |
+
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
| 12 |
+
*.model filter=lfs diff=lfs merge=lfs -text
|
| 13 |
+
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
| 14 |
+
*.npy filter=lfs diff=lfs merge=lfs -text
|
| 15 |
+
*.npz filter=lfs diff=lfs merge=lfs -text
|
| 16 |
+
*.onnx filter=lfs diff=lfs merge=lfs -text
|
| 17 |
+
*.ot filter=lfs diff=lfs merge=lfs -text
|
| 18 |
+
*.parquet filter=lfs diff=lfs merge=lfs -text
|
| 19 |
+
*.pb filter=lfs diff=lfs merge=lfs -text
|
| 20 |
+
*.pickle filter=lfs diff=lfs merge=lfs -text
|
| 21 |
+
*.pkl filter=lfs diff=lfs merge=lfs -text
|
| 22 |
+
*.pt filter=lfs diff=lfs merge=lfs -text
|
| 23 |
+
*.pth filter=lfs diff=lfs merge=lfs -text
|
| 24 |
+
*.rar filter=lfs diff=lfs merge=lfs -text
|
| 25 |
+
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
| 26 |
+
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
| 27 |
+
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
| 28 |
+
*.tar filter=lfs diff=lfs merge=lfs -text
|
| 29 |
+
*.tflite filter=lfs diff=lfs merge=lfs -text
|
| 30 |
+
*.tgz filter=lfs diff=lfs merge=lfs -text
|
| 31 |
+
*.wasm filter=lfs diff=lfs merge=lfs -text
|
| 32 |
+
*.xz filter=lfs diff=lfs merge=lfs -text
|
| 33 |
+
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
+
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
+
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
JiRackDeltaNet_180b-00001-of-00007.gguf filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
JiRackDeltaNet_180b-00002-of-00007.gguf filter=lfs diff=lfs merge=lfs -text
|
| 38 |
+
JiRackDeltaNet_180b-00003-of-00007.gguf filter=lfs diff=lfs merge=lfs -text
|
| 39 |
+
JiRackDeltaNet_180b-00004-of-00007.gguf filter=lfs diff=lfs merge=lfs -text
|
| 40 |
+
JiRackDeltaNet_180b-00005-of-00007.gguf filter=lfs diff=lfs merge=lfs -text
|
| 41 |
+
JiRackDeltaNet_180b-00006-of-00007.gguf filter=lfs diff=lfs merge=lfs -text
|
| 42 |
+
JiRackDeltaNet_180b-00007-of-00007.gguf filter=lfs diff=lfs merge=lfs -text
|
| 43 |
+
JiRackDeltaNet_180b_Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
| 44 |
+
JiRackDeltaNet_180b_Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
|
| 45 |
+
__pycache__/JiRackDeltaNet_180b.cpython-312.pyc filter=lfs diff=lfs merge=lfs -text
|
| 46 |
+
checkpoint.index.json filter=lfs diff=lfs merge=lfs -text
|
| 47 |
+
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
JiRackDeltaNet_180b-00001-of-00007.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:bfece217b3b399240f27daf70159f49e776d34bdcb61f39645bd972ca4119db2
|
| 3 |
+
size 11716396896
|
JiRackDeltaNet_180b-00002-of-00007.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:869679f9c35224ce467f6243f31cf15d9a207e1dfe8553b924b505c572cbfc82
|
| 3 |
+
size 102400491712
|
JiRackDeltaNet_180b-00003-of-00007.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:832a71227d550e797665ab24d50264ec8a9c6e58c25324f38ce681ba5651313a
|
| 3 |
+
size 48502374688
|
JiRackDeltaNet_180b-00004-of-00007.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f9c84a60b525c8b29bb04be73b445c9fb75ac2ce133a00e73f9c1de9dbf5c9c1
|
| 3 |
+
size 48367899136
|
JiRackDeltaNet_180b-00005-of-00007.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f8d651facdb3d1865e23d225b4d05e82359f478c47a7fb90ecbed028995744bf
|
| 3 |
+
size 48525443808
|
JiRackDeltaNet_180b-00006-of-00007.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a7043a64a50e12f5883c50266323dc5895042eac0933547a9ccb806c2d4affd9
|
| 3 |
+
size 48354794208
|
JiRackDeltaNet_180b-00007-of-00007.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ae5162ea097024caa4ee73ba387514e30b7dc2eeb5974cf40aed2f5bb1b1790a
|
| 3 |
+
size 46162529152
|
JiRackDeltaNet_180b.py
ADDED
|
@@ -0,0 +1,1512 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# =============================================================================
|
| 2 |
+
# JiRackDeltaNet_180b.py - JiRack DeltaNet 180B
|
| 3 |
+
# Copyright (c) 2025-2026 Konstantin Vladimirovich Grabko (CMSManhattan).
|
| 4 |
+
# MIT License, see LICENSE-CODE. The model weights are licensed separately, see README.md.#
|
| 5 |
+
# [PATENT PENDING
|
| 6 |
+
# =============================================================================
|
| 7 |
+
|
| 8 |
+
""
|
| 9 |
+
|
| 10 |
+
import glob
|
| 11 |
+
import json
|
| 12 |
+
import math
|
| 13 |
+
import mmap
|
| 14 |
+
import os
|
| 15 |
+
import re
|
| 16 |
+
import struct
|
| 17 |
+
from dataclasses import asdict, dataclass, fields
|
| 18 |
+
|
| 19 |
+
import torch
|
| 20 |
+
import torch.nn as nn
|
| 21 |
+
import torch.nn.functional as F
|
| 22 |
+
import torch.utils.checkpoint
|
| 23 |
+
|
| 24 |
+
@dataclass(frozen=True)
|
| 25 |
+
class FlashArch:
|
| 26 |
+
hidden_size: int = 2560
|
| 27 |
+
vocab_size: int = 248320
|
| 28 |
+
num_layers: int = 48
|
| 29 |
+
rms_norm_eps: float = 1e-6
|
| 30 |
+
full_attention_interval: int = 4
|
| 31 |
+
layer_types_override: tuple = None
|
| 32 |
+
|
| 33 |
+
linear_num_key_heads: int = 16
|
| 34 |
+
linear_key_head_dim: int = 128
|
| 35 |
+
linear_num_value_heads: int = 48
|
| 36 |
+
linear_value_head_dim: int = 128
|
| 37 |
+
linear_conv_kernel_dim: int = 4
|
| 38 |
+
delta_chunk_size: int = 64
|
| 39 |
+
output_gate_type: str = "sigmoid"
|
| 40 |
+
|
| 41 |
+
num_attention_heads: int = 24
|
| 42 |
+
num_key_value_heads: int = 2
|
| 43 |
+
head_dim: int = 256
|
| 44 |
+
partial_rotary_factor: float = 0.25
|
| 45 |
+
rope_theta: float = 1e7
|
| 46 |
+
mrope_section: tuple = (11, 11, 10)
|
| 47 |
+
attention_bias: bool = False
|
| 48 |
+
indexer_n_heads: int = 4
|
| 49 |
+
indexer_kv_heads: int = 1
|
| 50 |
+
indexer_head_dim: int = 128
|
| 51 |
+
indexer_budget: int = 2048
|
| 52 |
+
indexer_compress_ratio: int = 4
|
| 53 |
+
|
| 54 |
+
num_experts: int = 512
|
| 55 |
+
num_experts_per_tok: int = 10
|
| 56 |
+
moe_intermediate_size: int = 640
|
| 57 |
+
shared_expert_intermediate_size: int = 640
|
| 58 |
+
norm_topk_prob: bool = True
|
| 59 |
+
|
| 60 |
+
hc_count: int = 4
|
| 61 |
+
hc_lowrank: int = 320
|
| 62 |
+
|
| 63 |
+
ple_layer_ids: tuple = (2,)
|
| 64 |
+
ple_embed_dim: int = 2560
|
| 65 |
+
ple_conv_kernel_size: int = 4
|
| 66 |
+
ngram_size: int = 3
|
| 67 |
+
heads_per_ngram: int = 8
|
| 68 |
+
ngram_vocab_size_base: int = 20_000_000
|
| 69 |
+
make_ngram_vocab_size_divisible_by: int = 128
|
| 70 |
+
split_ngram_parts: int = 128
|
| 71 |
+
seed: int = 1234
|
| 72 |
+
eos_token_id: int = 248044
|
| 73 |
+
|
| 74 |
+
mtp_num_hidden_layers: int = 1
|
| 75 |
+
|
| 76 |
+
@property
|
| 77 |
+
def layer_types(self):
|
| 78 |
+
if self.layer_types_override is not None:
|
| 79 |
+
return tuple(self.layer_types_override)
|
| 80 |
+
return tuple(
|
| 81 |
+
"linear_attention" if (i + 1) % self.full_attention_interval else "full_attention"
|
| 82 |
+
for i in range(self.num_layers)
|
| 83 |
+
)
|
| 84 |
+
|
| 85 |
+
@property
|
| 86 |
+
def delta_key_dim(self):
|
| 87 |
+
return self.linear_num_key_heads * self.linear_key_head_dim
|
| 88 |
+
|
| 89 |
+
@property
|
| 90 |
+
def delta_value_dim(self):
|
| 91 |
+
return self.linear_num_value_heads * self.linear_value_head_dim
|
| 92 |
+
|
| 93 |
+
@property
|
| 94 |
+
def delta_conv_dim(self):
|
| 95 |
+
return self.delta_key_dim * 2 + self.delta_value_dim
|
| 96 |
+
|
| 97 |
+
@property
|
| 98 |
+
def hc_hidden_size(self):
|
| 99 |
+
return self.hc_count * self.hidden_size
|
| 100 |
+
|
| 101 |
+
@classmethod
|
| 102 |
+
def from_dict(cls, d: dict) -> "FlashArch":
|
| 103 |
+
known = {f.name for f in fields(cls)}
|
| 104 |
+
kw = {k: v for k, v in d.items() if k in known}
|
| 105 |
+
for k in ("mrope_section", "ple_layer_ids", "layer_types_override"):
|
| 106 |
+
if kw.get(k) is not None:
|
| 107 |
+
kw[k] = tuple(kw[k])
|
| 108 |
+
return cls(**kw)
|
| 109 |
+
|
| 110 |
+
@classmethod
|
| 111 |
+
def from_hf_text_config(cls, cfg) -> "FlashArch":
|
| 112 |
+
""
|
| 113 |
+
g = (lambda k, d=None: cfg.get(k, d)) if isinstance(cfg, dict) else (lambda k, d=None: getattr(cfg, k, d))
|
| 114 |
+
act = g("hidden_act", "silu")
|
| 115 |
+
if act != "silu":
|
| 116 |
+
raise ValueError(f"JiRack Flash supports only hidden_act=silu, the config has {act!r}")
|
| 117 |
+
rope = g("rope_parameters", None) or {}
|
| 118 |
+
eos = g("eos_token_id", 248044)
|
| 119 |
+
eos = eos[0] if isinstance(eos, (list, tuple)) else eos
|
| 120 |
+
layer_types = g("layer_types", None)
|
| 121 |
+
interval = g("full_attention_interval", 4) or 4
|
| 122 |
+
if layer_types is not None:
|
| 123 |
+
layer_types = tuple("full_attention" if t in ("full_attention", "indexed_attention") else t
|
| 124 |
+
for t in layer_types)
|
| 125 |
+
default_pattern = tuple("linear_attention" if (i + 1) % interval else "full_attention"
|
| 126 |
+
for i in range(len(layer_types)))
|
| 127 |
+
if layer_types == default_pattern:
|
| 128 |
+
layer_types = None
|
| 129 |
+
return cls(
|
| 130 |
+
hidden_size=g("hidden_size"),
|
| 131 |
+
vocab_size=g("vocab_size"),
|
| 132 |
+
num_layers=g("num_hidden_layers"),
|
| 133 |
+
rms_norm_eps=g("rms_norm_eps", 1e-6),
|
| 134 |
+
full_attention_interval=interval,
|
| 135 |
+
layer_types_override=layer_types,
|
| 136 |
+
linear_num_key_heads=g("linear_num_key_heads"),
|
| 137 |
+
linear_key_head_dim=g("linear_key_head_dim"),
|
| 138 |
+
linear_num_value_heads=g("linear_num_value_heads"),
|
| 139 |
+
linear_value_head_dim=g("linear_value_head_dim"),
|
| 140 |
+
linear_conv_kernel_dim=g("linear_conv_kernel_dim", 4),
|
| 141 |
+
output_gate_type=g("output_gate_type", None) or act,
|
| 142 |
+
num_attention_heads=g("num_attention_heads"),
|
| 143 |
+
num_key_value_heads=g("num_key_value_heads"),
|
| 144 |
+
head_dim=g("head_dim"),
|
| 145 |
+
partial_rotary_factor=rope.get("partial_rotary_factor", g("partial_rotary_factor", 1.0)),
|
| 146 |
+
rope_theta=float(rope.get("rope_theta", g("rope_theta", 1e7))),
|
| 147 |
+
mrope_section=tuple(rope.get("mrope_section", (11, 11, 10))),
|
| 148 |
+
attention_bias=g("attention_bias", False),
|
| 149 |
+
indexer_n_heads=g("indexer_n_heads"),
|
| 150 |
+
indexer_kv_heads=g("indexer_kv_heads"),
|
| 151 |
+
indexer_head_dim=g("indexer_head_dim"),
|
| 152 |
+
indexer_budget=g("indexer_budget"),
|
| 153 |
+
indexer_compress_ratio=g("indexer_compress_ratio"),
|
| 154 |
+
num_experts=g("num_experts"),
|
| 155 |
+
num_experts_per_tok=g("num_experts_per_tok"),
|
| 156 |
+
moe_intermediate_size=g("moe_intermediate_size"),
|
| 157 |
+
shared_expert_intermediate_size=g("shared_expert_intermediate_size"),
|
| 158 |
+
norm_topk_prob=g("norm_topk_prob", True),
|
| 159 |
+
hc_count=g("hc_count", 4),
|
| 160 |
+
hc_lowrank=g("hc_lowrank", 320),
|
| 161 |
+
ple_layer_ids=tuple(sorted(set(g("ple_layer_ids", None) or ()))),
|
| 162 |
+
ple_embed_dim=g("ple_embed_dim", None) or g("hidden_size"),
|
| 163 |
+
ple_conv_kernel_size=g("ple_conv_kernel_size", 4),
|
| 164 |
+
ngram_size=g("ngram_size", 3),
|
| 165 |
+
heads_per_ngram=g("heads_per_ngram", 8),
|
| 166 |
+
ngram_vocab_size_base=g("ngram_vocab_size_base", 20_000_000),
|
| 167 |
+
make_ngram_vocab_size_divisible_by=g("make_ngram_vocab_size_divisible_by", 128),
|
| 168 |
+
split_ngram_parts=g("split_ngram_parts", 512),
|
| 169 |
+
seed=g("seed", 1234),
|
| 170 |
+
eos_token_id=eos,
|
| 171 |
+
mtp_num_hidden_layers=g("mtp_num_hidden_layers", 1) or 1,
|
| 172 |
+
)
|
| 173 |
+
|
| 174 |
+
@classmethod
|
| 175 |
+
def from_config_json(cls, path: str) -> "FlashArch":
|
| 176 |
+
with open(path) as f:
|
| 177 |
+
d = json.load(f)
|
| 178 |
+
return cls.from_hf_text_config(d.get("text_config", d))
|
| 179 |
+
|
| 180 |
+
DEFAULT_ARCH = FlashArch()
|
| 181 |
+
|
| 182 |
+
HIDDEN_SIZE = DEFAULT_ARCH.hidden_size
|
| 183 |
+
VOCAB_SIZE = DEFAULT_ARCH.vocab_size
|
| 184 |
+
NUM_LAYERS = DEFAULT_ARCH.num_layers
|
| 185 |
+
HC_COUNT = DEFAULT_ARCH.hc_count
|
| 186 |
+
LAYER_TYPES = list(DEFAULT_ARCH.layer_types)
|
| 187 |
+
PLE_LAYER_INDICES = [i - 1 for i in DEFAULT_ARCH.ple_layer_ids]
|
| 188 |
+
|
| 189 |
+
ARCH_SUMMARY = f"""
|
| 190 |
+
Qwen3.8-Flash-Next (qwen4_exp), hand-rolled JiRack TransformerBlock, text-only.
|
| 191 |
+
hidden_size={HIDDEN_SIZE} x hc_count={HC_COUNT} streams, vocab={VOCAB_SIZE}, layers={NUM_LAYERS}
|
| 192 |
+
Interleaving (interval=4): 3x linear_attention (Gated DeltaNet), 1x full_attention (Gated Attn + QSA)
|
| 193 |
+
MLP: MoE {DEFAULT_ARCH.num_experts} experts, top-{DEFAULT_ARCH.num_experts_per_tok}, +1 shared
|
| 194 |
+
PLE (n-gram embedding) on layers.{PLE_LAYER_INDICES}; MTP: 1 layer (full_attention + MoE)
|
| 195 |
+
"""
|
| 196 |
+
|
| 197 |
+
KEPT_FULL_PRECISION = """
|
| 198 |
+
Always kept in full precision (not a GEMM, or too sensitive):
|
| 199 |
+
- dt_bias, A_log (raw Parameters, DeltaNet decay)
|
| 200 |
+
- linear_attn.in_proj_b/in_proj_a (by default; --include_gate_proj)
|
| 201 |
+
- conv1d (DeltaNet and PLE) (depthwise conv, not a GEMM)
|
| 202 |
+
- all norms (RMSNormGated, q/k_norm, hc_norm, norm_key/query/conv, ...)
|
| 203 |
+
- mlp.gate (MoE router), mlp.shared_expert_gate
|
| 204 |
+
(a routing error = a different expert,
|
| 205 |
+
not a "slightly different" output);
|
| 206 |
+
ALSO FROZEN (requires_grad=False,
|
| 207 |
+
freeze_routers) -- not trained during QAT
|
| 208 |
+
- *_hyper_connection.*, hyper_connection_mixer
|
| 209 |
+
(low-rank stream-mixing gates;
|
| 210 |
+
tiny and extremely sensitive)
|
| 211 |
+
- self_attn.indexer.* (QSA block selection; by default,
|
| 212 |
+
--include_indexer)
|
| 213 |
+
- ple.key_proj / ple.value_proj (by default; --include_ple_proj)
|
| 214 |
+
- ple.ple_embedding.ngram_embedding.* (embedding table, 51B)
|
| 215 |
+
- embed_tokens, lm_head
|
| 216 |
+
- mtp.* (the whole MTP head) (draft layer for speculative decoding)
|
| 217 |
+
"""
|
| 218 |
+
|
| 219 |
+
class BitLinear(nn.Linear):
|
| 220 |
+
""
|
| 221 |
+
|
| 222 |
+
def __init__(self, in_features, out_features, bias=False):
|
| 223 |
+
super().__init__(in_features, out_features, bias=bias)
|
| 224 |
+
self.eps = 1e-5
|
| 225 |
+
self.register_buffer("lambda_", torch.zeros(()), persistent=True)
|
| 226 |
+
|
| 227 |
+
@classmethod
|
| 228 |
+
def from_linear(cls, lin: nn.Linear) -> "BitLinear":
|
| 229 |
+
""
|
| 230 |
+
with torch.device("meta"):
|
| 231 |
+
bl = cls(lin.in_features, lin.out_features, bias=lin.bias is not None)
|
| 232 |
+
bl.weight = lin.weight
|
| 233 |
+
if lin.bias is not None:
|
| 234 |
+
bl.bias = lin.bias
|
| 235 |
+
bl.lambda_ = torch.zeros((), device=lin.weight.device)
|
| 236 |
+
return bl
|
| 237 |
+
|
| 238 |
+
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
| 239 |
+
if not self.training and float(self.lambda_) < 1e-6:
|
| 240 |
+
return F.linear(x, self.weight, self.bias)
|
| 241 |
+
|
| 242 |
+
lam = self.lambda_.to(x.dtype)
|
| 243 |
+
w = self.weight
|
| 244 |
+
gamma = w.float().abs().mean().clamp(min=self.eps).to(w.dtype)
|
| 245 |
+
w_quant = torch.clamp(torch.round(w / gamma), -1.0, 1.0) * gamma
|
| 246 |
+
w_effective = w + lam * (w_quant - w).detach()
|
| 247 |
+
|
| 248 |
+
x_scale = 127.0 / x.abs().max(dim=-1, keepdim=True).values.clamp(min=self.eps)
|
| 249 |
+
x_quant = torch.clamp(torch.round(x * x_scale), -128.0, 127.0) / x_scale
|
| 250 |
+
x_effective = x + lam * (x_quant - x).detach()
|
| 251 |
+
|
| 252 |
+
return F.linear(x_effective, w_effective, self.bias)
|
| 253 |
+
|
| 254 |
+
def set_lambda(self, v: float):
|
| 255 |
+
self.lambda_.fill_(v)
|
| 256 |
+
|
| 257 |
+
def ternary_weight(self) -> torch.Tensor:
|
| 258 |
+
""
|
| 259 |
+
w = self.weight
|
| 260 |
+
gamma = w.float().abs().mean().clamp(min=self.eps).to(w.dtype)
|
| 261 |
+
return torch.clamp(torch.round(w / gamma), -1.0, 1.0) * gamma
|
| 262 |
+
|
| 263 |
+
TERNARIZE_PATTERNS = [
|
| 264 |
+
r"linear_attn\.in_proj_qkv$",
|
| 265 |
+
r"linear_attn\.in_proj_z$",
|
| 266 |
+
r"linear_attn\.out_proj$",
|
| 267 |
+
r"self_attn\.q_proj$",
|
| 268 |
+
r"self_attn\.k_proj$",
|
| 269 |
+
r"self_attn\.v_proj$",
|
| 270 |
+
r"self_attn\.o_proj$",
|
| 271 |
+
r"mlp\.experts\.\d+\.(gate|up|down)_proj$",
|
| 272 |
+
r"mlp\.shared_expert\.(gate|up|down)_proj$",
|
| 273 |
+
]
|
| 274 |
+
|
| 275 |
+
GATE_PROJ_PATTERNS = [
|
| 276 |
+
r"linear_attn\.in_proj_b$",
|
| 277 |
+
r"linear_attn\.in_proj_a$",
|
| 278 |
+
]
|
| 279 |
+
|
| 280 |
+
INDEXER_PATTERNS = [r"self_attn\.indexer\.index_qk_proj$"]
|
| 281 |
+
PLE_PROJ_PATTERNS = [r"ple\.key_proj$", r"ple\.value_proj$"]
|
| 282 |
+
|
| 283 |
+
FROZEN_PATTERNS = [
|
| 284 |
+
r"mlp\.gate\.weight$",
|
| 285 |
+
r"mlp\.shared_expert_gate\.weight$",
|
| 286 |
+
]
|
| 287 |
+
|
| 288 |
+
def router_parameters(model: nn.Module, include_mtp: bool = True):
|
| 289 |
+
""
|
| 290 |
+
compiled = [re.compile(p) for p in FROZEN_PATTERNS]
|
| 291 |
+
return [(n, p) for n, p in model.named_parameters()
|
| 292 |
+
if (include_mtp or not n.startswith("mtp")) and any(c.search(n) for c in compiled)]
|
| 293 |
+
|
| 294 |
+
def set_routers_trainable(model: nn.Module, trainable: bool, include_mtp: bool = True):
|
| 295 |
+
""
|
| 296 |
+
params = router_parameters(model, include_mtp)
|
| 297 |
+
for _, p in params:
|
| 298 |
+
p.requires_grad_(trainable)
|
| 299 |
+
return [n for n, _ in params]
|
| 300 |
+
|
| 301 |
+
def freeze_routers(model: nn.Module, include_mtp: bool = True):
|
| 302 |
+
""
|
| 303 |
+
return set_routers_trainable(model, False, include_mtp)
|
| 304 |
+
|
| 305 |
+
def assert_routers_frozen(model: nn.Module, optimizer=None):
|
| 306 |
+
""
|
| 307 |
+
compiled = [re.compile(p) for p in FROZEN_PATTERNS]
|
| 308 |
+
router_params = {name: p for name, p in model.named_parameters() if any(c.search(name) for c in compiled)}
|
| 309 |
+
if not router_params:
|
| 310 |
+
raise AssertionError("No routers found -- is this really JiRack 180B?")
|
| 311 |
+
|
| 312 |
+
for name, module in model.named_modules():
|
| 313 |
+
if isinstance(module, BitLinear) and (name.endswith(".mlp.gate") or name.endswith(".mlp.shared_expert_gate")):
|
| 314 |
+
raise AssertionError(f"Router is quantized (BitLinear): {name}")
|
| 315 |
+
|
| 316 |
+
trainable = [n for n, p in router_params.items() if p.requires_grad]
|
| 317 |
+
if trainable:
|
| 318 |
+
raise AssertionError(f"{len(trainable)} routers are trainable (requires_grad=True), first: {trainable[:3]}. "
|
| 319 |
+
f"Call freeze_routers(model).")
|
| 320 |
+
|
| 321 |
+
if optimizer is not None:
|
| 322 |
+
ids = {id(p): n for n, p in router_params.items()}
|
| 323 |
+
leaked = [ids[id(p)] for g in optimizer.param_groups for p in g["params"] if id(p) in ids]
|
| 324 |
+
if leaked:
|
| 325 |
+
raise AssertionError(f"{len(leaked)} routers ended up in the optimizer, first: {leaked[:3]}. "
|
| 326 |
+
f"Build the optimizer from [p for p in model.parameters() if p.requires_grad].")
|
| 327 |
+
print(f"✅ Routers: {len(router_params)} -- not quantized, frozen"
|
| 328 |
+
+ (", not in the optimizer" if optimizer is not None else ""))
|
| 329 |
+
return len(router_params)
|
| 330 |
+
|
| 331 |
+
def apply_bitlinear_patch(model: nn.Module, include_gate_proj: bool = False,
|
| 332 |
+
include_indexer: bool = False, include_ple_proj: bool = False):
|
| 333 |
+
""
|
| 334 |
+
patterns = list(TERNARIZE_PATTERNS)
|
| 335 |
+
if include_gate_proj:
|
| 336 |
+
patterns += GATE_PROJ_PATTERNS
|
| 337 |
+
if include_indexer:
|
| 338 |
+
patterns += INDEXER_PATTERNS
|
| 339 |
+
if include_ple_proj:
|
| 340 |
+
patterns += PLE_PROJ_PATTERNS
|
| 341 |
+
compiled = [re.compile(p) for p in patterns]
|
| 342 |
+
|
| 343 |
+
patched = []
|
| 344 |
+
for name, module in list(model.named_modules()):
|
| 345 |
+
if not isinstance(module, nn.Linear) or isinstance(module, BitLinear):
|
| 346 |
+
continue
|
| 347 |
+
|
| 348 |
+
if name.startswith("mtp"):
|
| 349 |
+
continue
|
| 350 |
+
if any(p.search(name) for p in compiled):
|
| 351 |
+
parent_name, _, child_name = name.rpartition(".")
|
| 352 |
+
parent = model.get_submodule(parent_name) if parent_name else model
|
| 353 |
+
setattr(parent, child_name, BitLinear.from_linear(module))
|
| 354 |
+
patched.append(name)
|
| 355 |
+
return patched
|
| 356 |
+
|
| 357 |
+
class RMSNorm(nn.Module):
|
| 358 |
+
""
|
| 359 |
+
|
| 360 |
+
def __init__(self, dim: int, eps: float = 1e-6, group_size: int = None):
|
| 361 |
+
super().__init__()
|
| 362 |
+
self.eps = eps
|
| 363 |
+
self.group_size = group_size
|
| 364 |
+
if group_size is not None and dim % group_size != 0:
|
| 365 |
+
raise ValueError(f"dim ({dim}) must be divisible by group_size ({group_size}).")
|
| 366 |
+
self.weight = nn.Parameter(torch.zeros(dim))
|
| 367 |
+
|
| 368 |
+
def _norm(self, x: torch.Tensor) -> torch.Tensor:
|
| 369 |
+
if self.group_size is not None:
|
| 370 |
+
x = x.reshape(*x.shape[:-1], -1, self.group_size)
|
| 371 |
+
out = x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
|
| 372 |
+
return out.flatten(-2) if self.group_size is not None else out
|
| 373 |
+
|
| 374 |
+
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
| 375 |
+
output = self._norm(x.float())
|
| 376 |
+
output = output * (1.0 + self.weight.float())
|
| 377 |
+
return output.type_as(x)
|
| 378 |
+
|
| 379 |
+
_GATE_ACT = {"sigmoid": torch.sigmoid, "silu": F.silu}
|
| 380 |
+
|
| 381 |
+
class RMSNormGated(nn.Module):
|
| 382 |
+
""
|
| 383 |
+
|
| 384 |
+
def __init__(self, dim: int, eps: float = 1e-6, activation: str = "sigmoid"):
|
| 385 |
+
super().__init__()
|
| 386 |
+
self.weight = nn.Parameter(torch.ones(dim))
|
| 387 |
+
self.eps = eps
|
| 388 |
+
self.activation = activation
|
| 389 |
+
self._act = _GATE_ACT[activation]
|
| 390 |
+
|
| 391 |
+
def forward(self, x: torch.Tensor, gate: torch.Tensor) -> torch.Tensor:
|
| 392 |
+
dtype = x.dtype
|
| 393 |
+
x = x.to(torch.float32)
|
| 394 |
+
var = x.pow(2).mean(-1, keepdim=True)
|
| 395 |
+
x = x * torch.rsqrt(var + self.eps)
|
| 396 |
+
x = self.weight * x.to(dtype)
|
| 397 |
+
x = x * self._act(gate.to(torch.float32))
|
| 398 |
+
return x.to(dtype)
|
| 399 |
+
|
| 400 |
+
def rotate_half(x: torch.Tensor) -> torch.Tensor:
|
| 401 |
+
x1 = x[..., : x.shape[-1] // 2]
|
| 402 |
+
x2 = x[..., x.shape[-1] // 2:]
|
| 403 |
+
return torch.cat((-x2, x1), dim=-1)
|
| 404 |
+
|
| 405 |
+
def apply_rotary_pos_emb(q, k=None, cos=None, sin=None, unsqueeze_dim=1):
|
| 406 |
+
""
|
| 407 |
+
cos = cos.unsqueeze(unsqueeze_dim)
|
| 408 |
+
sin = sin.unsqueeze(unsqueeze_dim)
|
| 409 |
+
rotary_dim = cos.shape[-1]
|
| 410 |
+
q_rope, q_nope = q[..., :rotary_dim], q[..., rotary_dim:]
|
| 411 |
+
q_rope = (q_rope * cos) + (rotate_half(q_rope) * sin)
|
| 412 |
+
q_rotated = torch.cat([q_rope, q_nope], dim=-1)
|
| 413 |
+
if k is None:
|
| 414 |
+
return q_rotated
|
| 415 |
+
k_rope, k_nope = k[..., :rotary_dim], k[..., rotary_dim:]
|
| 416 |
+
k_rope = (k_rope * cos) + (rotate_half(k_rope) * sin)
|
| 417 |
+
return q_rotated, torch.cat([k_rope, k_nope], dim=-1)
|
| 418 |
+
|
| 419 |
+
class TextRotaryEmbedding(nn.Module):
|
| 420 |
+
""
|
| 421 |
+
|
| 422 |
+
def __init__(self, arch: FlashArch = DEFAULT_ARCH):
|
| 423 |
+
super().__init__()
|
| 424 |
+
self.rotary_dim = int(arch.head_dim * arch.partial_rotary_factor)
|
| 425 |
+
self.theta = arch.rope_theta
|
| 426 |
+
self.mrope_section = list(arch.mrope_section)
|
| 427 |
+
self.register_buffer("inv_freq", self._compute_inv_freq(), persistent=False)
|
| 428 |
+
|
| 429 |
+
def _compute_inv_freq(self, device=None):
|
| 430 |
+
dim = self.rotary_dim
|
| 431 |
+
return 1.0 / (self.theta ** (torch.arange(0, dim, 2, dtype=torch.float, device=device) / dim))
|
| 432 |
+
|
| 433 |
+
def reset_derived_buffers(self, device="cpu"):
|
| 434 |
+
self.inv_freq = self._compute_inv_freq(device)
|
| 435 |
+
|
| 436 |
+
def _recompose(self, freq: torch.Tensor) -> torch.Tensor:
|
| 437 |
+
freqs_thw = freq[0]
|
| 438 |
+
for dim, offset in enumerate((1, 2), start=1):
|
| 439 |
+
length = self.mrope_section[dim] * 3
|
| 440 |
+
idx = slice(offset, length, 3)
|
| 441 |
+
freqs_thw[..., idx] = freq[dim, ..., idx]
|
| 442 |
+
return torch.cat((freqs_thw, freqs_thw), dim=-1)
|
| 443 |
+
|
| 444 |
+
@torch.no_grad()
|
| 445 |
+
def forward(self, x: torch.Tensor, position_ids: torch.LongTensor):
|
| 446 |
+
if position_ids.ndim == 2:
|
| 447 |
+
position_ids = position_ids[None, ...].expand(3, position_ids.shape[0], -1)
|
| 448 |
+
inv = self.inv_freq[None, None, :, None].float().expand(3, position_ids.shape[1], -1, 1)
|
| 449 |
+
pos = position_ids[:, :, None, :].float()
|
| 450 |
+
freqs = (inv.float() @ pos.float()).transpose(2, 3)
|
| 451 |
+
cos = self._recompose(freqs.cos())
|
| 452 |
+
sin = self._recompose(freqs.sin())
|
| 453 |
+
return cos.to(dtype=x.dtype), sin.to(dtype=x.dtype)
|
| 454 |
+
|
| 455 |
+
def l2norm(x: torch.Tensor, dim: int = -1, eps: float = 1e-6):
|
| 456 |
+
return x * torch.rsqrt((x * x).sum(dim=dim, keepdim=True) + eps)
|
| 457 |
+
|
| 458 |
+
def causal_conv1d(hidden_states: torch.Tensor, weight: torch.Tensor, activation: str = "silu"):
|
| 459 |
+
""
|
| 460 |
+
channels, seq_len = hidden_states.shape[1], hidden_states.shape[2]
|
| 461 |
+
padding = weight.shape[-1] - 1
|
| 462 |
+
out = F.conv1d(hidden_states.to(weight.dtype), weight=weight.unsqueeze(1), bias=None,
|
| 463 |
+
padding=padding, groups=channels)[:, :, :seq_len]
|
| 464 |
+
if activation == "silu":
|
| 465 |
+
out = F.silu(out)
|
| 466 |
+
return out.to(hidden_states.dtype)
|
| 467 |
+
|
| 468 |
+
def chunked_gated_delta_rule(query, key, value, g, beta, chunk_size=64):
|
| 469 |
+
""
|
| 470 |
+
initial_dtype = query.dtype
|
| 471 |
+
batch_size, sequence_length, _, k_head_dim = key.shape
|
| 472 |
+
num_v_heads, v_head_dim = value.shape[-2:]
|
| 473 |
+
padded_output_shape = (batch_size, num_v_heads, -1, v_head_dim)
|
| 474 |
+
decay = g
|
| 475 |
+
|
| 476 |
+
query, key, value, beta, decay = [
|
| 477 |
+
x.transpose(1, 2).to(torch.float32, memory_format=torch.contiguous_format)
|
| 478 |
+
for x in (query, key, value, beta, decay)
|
| 479 |
+
]
|
| 480 |
+
query = l2norm(query, dim=-1, eps=1e-6)
|
| 481 |
+
key = l2norm(key, dim=-1, eps=1e-6)
|
| 482 |
+
query = query * (query.shape[-1] ** -0.5)
|
| 483 |
+
|
| 484 |
+
pad_size = (chunk_size - sequence_length % chunk_size) % chunk_size
|
| 485 |
+
query, key, value = (F.pad(x, (0, 0, 0, pad_size)) for x in (query, key, value))
|
| 486 |
+
beta, decay = (F.pad(x, (0, pad_size)) for x in (beta, decay))
|
| 487 |
+
num_chunks = (sequence_length + pad_size) // chunk_size
|
| 488 |
+
|
| 489 |
+
v_beta = value * beta.unsqueeze(-1)
|
| 490 |
+
k_beta = key * beta.unsqueeze(-1)
|
| 491 |
+
query, key, k_beta, v_beta = [
|
| 492 |
+
x.reshape(x.shape[0], x.shape[1], -1, chunk_size, x.shape[-1]) for x in (query, key, k_beta, v_beta)
|
| 493 |
+
]
|
| 494 |
+
decay = decay.reshape(decay.shape[0], decay.shape[1], -1, chunk_size)
|
| 495 |
+
|
| 496 |
+
strictly_upper_mask = torch.ones(chunk_size, chunk_size, dtype=torch.bool, device=query.device).triu(1)
|
| 497 |
+
cum_decay = decay.cumsum(dim=3)
|
| 498 |
+
pairwise_decay = cum_decay.unsqueeze(4) - cum_decay.unsqueeze(3)
|
| 499 |
+
pairwise_decay = pairwise_decay.masked_fill(strictly_upper_mask, float("-inf")).exp()
|
| 500 |
+
|
| 501 |
+
ut_system = (k_beta @ key.transpose(-1, -2)) * pairwise_decay
|
| 502 |
+
intra_chunk_attn = (query @ key.transpose(-1, -2)) * pairwise_decay
|
| 503 |
+
decayed_k_beta = k_beta * cum_decay.exp().unsqueeze(-1)
|
| 504 |
+
|
| 505 |
+
new_values = torch.linalg.solve_triangular(ut_system, v_beta, upper=False, unitriangular=True)
|
| 506 |
+
k_cumdecay = torch.linalg.solve_triangular(ut_system, decayed_k_beta, upper=False, unitriangular=True)
|
| 507 |
+
|
| 508 |
+
last_recurrent_state = torch.zeros((batch_size, num_v_heads, k_head_dim, v_head_dim),
|
| 509 |
+
dtype=new_values.dtype, device=new_values.device)
|
| 510 |
+
core_attn_out = torch.zeros_like(new_values)
|
| 511 |
+
|
| 512 |
+
query = query * cum_decay.exp().unsqueeze(-1)
|
| 513 |
+
key = key * (cum_decay[..., -1:] - cum_decay).exp().unsqueeze(-1)
|
| 514 |
+
chunk_decay = cum_decay[..., -1].exp()[..., None, None]
|
| 515 |
+
|
| 516 |
+
for i in range(num_chunks):
|
| 517 |
+
v_new = new_values[:, :, i] - k_cumdecay[:, :, i] @ last_recurrent_state
|
| 518 |
+
inter_chunk_attn = query[:, :, i] @ last_recurrent_state
|
| 519 |
+
core_attn_out[:, :, i] = inter_chunk_attn + intra_chunk_attn[:, :, i] @ v_new
|
| 520 |
+
last_recurrent_state = last_recurrent_state * chunk_decay[:, :, i] + key[:, :, i].transpose(-1, -2) @ v_new
|
| 521 |
+
|
| 522 |
+
core_attn_out = core_attn_out.reshape(padded_output_shape)[:, :, :sequence_length]
|
| 523 |
+
return core_attn_out.transpose(1, 2).to(initial_dtype, memory_format=torch.contiguous_format)
|
| 524 |
+
|
| 525 |
+
class GatedDeltaNetBlock(nn.Module):
|
| 526 |
+
""
|
| 527 |
+
|
| 528 |
+
def __init__(self, arch: FlashArch = DEFAULT_ARCH):
|
| 529 |
+
super().__init__()
|
| 530 |
+
self.arch = arch
|
| 531 |
+
conv_dim = arch.delta_conv_dim
|
| 532 |
+
self.conv1d = nn.Conv1d(conv_dim, conv_dim, bias=False, kernel_size=arch.linear_conv_kernel_dim,
|
| 533 |
+
groups=conv_dim, padding=arch.linear_conv_kernel_dim - 1)
|
| 534 |
+
self.dt_bias = nn.Parameter(torch.ones(arch.linear_num_value_heads))
|
| 535 |
+
self.A_log = nn.Parameter(torch.log(torch.empty(arch.linear_num_value_heads).uniform_(0.01, 16)))
|
| 536 |
+
self.norm = RMSNormGated(arch.linear_value_head_dim, eps=arch.rms_norm_eps,
|
| 537 |
+
activation=arch.output_gate_type)
|
| 538 |
+
self.out_proj = nn.Linear(arch.delta_value_dim, arch.hidden_size, bias=False)
|
| 539 |
+
self.in_proj_qkv = nn.Linear(arch.hidden_size, conv_dim, bias=False)
|
| 540 |
+
self.in_proj_z = nn.Linear(arch.hidden_size, arch.delta_value_dim, bias=False)
|
| 541 |
+
self.in_proj_b = nn.Linear(arch.hidden_size, arch.linear_num_value_heads, bias=False)
|
| 542 |
+
self.in_proj_a = nn.Linear(arch.hidden_size, arch.linear_num_value_heads, bias=False)
|
| 543 |
+
|
| 544 |
+
def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
|
| 545 |
+
a_ = self.arch
|
| 546 |
+
bsz, seq_len, _ = hidden_states.shape
|
| 547 |
+
|
| 548 |
+
mixed_qkv = self.in_proj_qkv(hidden_states).transpose(1, 2)
|
| 549 |
+
z = self.in_proj_z(hidden_states).reshape(bsz, seq_len, -1, a_.linear_value_head_dim)
|
| 550 |
+
b = self.in_proj_b(hidden_states)
|
| 551 |
+
a = self.in_proj_a(hidden_states)
|
| 552 |
+
|
| 553 |
+
mixed_qkv = causal_conv1d(mixed_qkv, self.conv1d.weight.squeeze(1), activation="silu")
|
| 554 |
+
mixed_qkv = mixed_qkv.transpose(1, 2)
|
| 555 |
+
|
| 556 |
+
query, key, value = torch.split(mixed_qkv, [a_.delta_key_dim, a_.delta_key_dim, a_.delta_value_dim], dim=-1)
|
| 557 |
+
query = query.reshape(bsz, seq_len, -1, a_.linear_key_head_dim)
|
| 558 |
+
key = key.reshape(bsz, seq_len, -1, a_.linear_key_head_dim)
|
| 559 |
+
value = value.reshape(bsz, seq_len, -1, a_.linear_value_head_dim)
|
| 560 |
+
|
| 561 |
+
beta = b.sigmoid()
|
| 562 |
+
g = -self.A_log.float().exp() * F.softplus(a.float() + self.dt_bias)
|
| 563 |
+
rep = a_.linear_num_value_heads // a_.linear_num_key_heads
|
| 564 |
+
if rep > 1:
|
| 565 |
+
query = query.repeat_interleave(rep, dim=2)
|
| 566 |
+
key = key.repeat_interleave(rep, dim=2)
|
| 567 |
+
|
| 568 |
+
core_attn_out = chunked_gated_delta_rule(query, key, value, g, beta, chunk_size=a_.delta_chunk_size)
|
| 569 |
+
|
| 570 |
+
core_attn_out = core_attn_out.reshape(-1, a_.linear_value_head_dim)
|
| 571 |
+
z = z.reshape(-1, a_.linear_value_head_dim)
|
| 572 |
+
core_attn_out = self.norm(core_attn_out, z)
|
| 573 |
+
core_attn_out = core_attn_out.reshape(bsz, seq_len, -1)
|
| 574 |
+
return self.out_proj(core_attn_out)
|
| 575 |
+
|
| 576 |
+
class QSAIndexer(nn.Module):
|
| 577 |
+
""
|
| 578 |
+
|
| 579 |
+
def __init__(self, arch: FlashArch = DEFAULT_ARCH):
|
| 580 |
+
super().__init__()
|
| 581 |
+
if arch.indexer_kv_heads != 1:
|
| 582 |
+
raise ValueError("QSA requires indexer_kv_heads=1")
|
| 583 |
+
self.n_heads = arch.indexer_n_heads
|
| 584 |
+
self.head_dim = arch.indexer_head_dim
|
| 585 |
+
self.compress_ratio = arch.indexer_compress_ratio
|
| 586 |
+
self.block_topk = arch.indexer_budget // arch.indexer_compress_ratio
|
| 587 |
+
self.index_qk_proj = nn.Linear(arch.hidden_size, (self.n_heads + 1) * self.head_dim, bias=False)
|
| 588 |
+
self.q_layernorm = RMSNorm(self.head_dim, eps=arch.rms_norm_eps)
|
| 589 |
+
self.k_layernorm = RMSNorm(self.head_dim, eps=arch.rms_norm_eps)
|
| 590 |
+
|
| 591 |
+
@torch.no_grad()
|
| 592 |
+
def forward(self, hidden_states, cos, sin, query_chunk: int = 256):
|
| 593 |
+
bsz, seq_len, _ = hidden_states.shape
|
| 594 |
+
cr, kb = self.compress_ratio, self.block_topk
|
| 595 |
+
num_blocks = seq_len // cr
|
| 596 |
+
if num_blocks <= kb:
|
| 597 |
+
return None
|
| 598 |
+
|
| 599 |
+
qk = self.index_qk_proj(hidden_states)
|
| 600 |
+
q, token_k = torch.split(qk, [self.n_heads * self.head_dim, self.head_dim], dim=-1)
|
| 601 |
+
q = q.reshape(bsz, seq_len, self.n_heads, self.head_dim)
|
| 602 |
+
raw_keys = token_k.reshape(bsz, seq_len, self.head_dim)
|
| 603 |
+
q = self.q_layernorm(q)
|
| 604 |
+
q = apply_rotary_pos_emb(q, cos=cos, sin=sin, unsqueeze_dim=2)
|
| 605 |
+
|
| 606 |
+
key_groups = raw_keys[:, : num_blocks * cr].reshape(bsz, num_blocks, cr, self.head_dim)
|
| 607 |
+
pooled = key_groups.float().mean(dim=2).to(raw_keys.dtype)
|
| 608 |
+
pooled = self.k_layernorm(pooled)
|
| 609 |
+
starts = torch.arange(num_blocks, device=hidden_states.device) * cr
|
| 610 |
+
block_keys = apply_rotary_pos_emb(pooled.unsqueeze(2), cos=cos[:, starts], sin=sin[:, starts],
|
| 611 |
+
unsqueeze_dim=2).squeeze(2)
|
| 612 |
+
block_keys_t = block_keys.float().transpose(-1, -2)
|
| 613 |
+
|
| 614 |
+
selected = torch.zeros(bsz, seq_len, seq_len, dtype=torch.bool, device=hidden_states.device)
|
| 615 |
+
pos = torch.arange(seq_len, device=hidden_states.device)
|
| 616 |
+
|
| 617 |
+
selected |= (pos[None, :] <= pos[:, None])[None]
|
| 618 |
+
|
| 619 |
+
first_q = (kb + 1) * cr - 1
|
| 620 |
+
inv_sqrt = math.sqrt(self.head_dim)
|
| 621 |
+
for q0 in range(first_q, seq_len, query_chunk):
|
| 622 |
+
q1 = min(seq_len, q0 + query_chunk)
|
| 623 |
+
qf = q[:, q0:q1].float()
|
| 624 |
+
scores = torch.matmul(qf, block_keys_t.unsqueeze(1))
|
| 625 |
+
scores = torch.relu(scores).sum(dim=2) / inv_sqrt
|
| 626 |
+
|
| 627 |
+
i = q0
|
| 628 |
+
while i < q1:
|
| 629 |
+
nb_i = (i + 1) // cr
|
| 630 |
+
j = min(q1, (nb_i + 1) * cr - 1)
|
| 631 |
+
rows = scores[:, i - q0:j - q0, :nb_i]
|
| 632 |
+
top = rows.topk(kb, dim=-1).indices
|
| 633 |
+
blk = torch.zeros(bsz, j - i, num_blocks, dtype=torch.bool, device=hidden_states.device)
|
| 634 |
+
blk.scatter_(-1, top, True)
|
| 635 |
+
tok = torch.zeros(bsz, j - i, seq_len, dtype=torch.bool, device=hidden_states.device)
|
| 636 |
+
tok[..., : num_blocks * cr] = blk.repeat_interleave(cr, dim=-1)
|
| 637 |
+
tail_start = nb_i * cr
|
| 638 |
+
qi = torch.arange(i, j, device=hidden_states.device)
|
| 639 |
+
tail = (pos[None, :] >= tail_start) & (pos[None, :] <= qi[:, None])
|
| 640 |
+
selected[:, i:j] = tok | tail[None]
|
| 641 |
+
i = j
|
| 642 |
+
return selected
|
| 643 |
+
|
| 644 |
+
class AttentionBlock(nn.Module):
|
| 645 |
+
""
|
| 646 |
+
|
| 647 |
+
def __init__(self, arch: FlashArch = DEFAULT_ARCH):
|
| 648 |
+
super().__init__()
|
| 649 |
+
self.arch = arch
|
| 650 |
+
self.head_dim = arch.head_dim
|
| 651 |
+
self.num_key_value_groups = arch.num_attention_heads // arch.num_key_value_heads
|
| 652 |
+
self.scaling = arch.head_dim ** -0.5
|
| 653 |
+
H, nh, nkv, hd = arch.hidden_size, arch.num_attention_heads, arch.num_key_value_heads, arch.head_dim
|
| 654 |
+
self.q_proj = nn.Linear(H, nh * hd * 2, bias=arch.attention_bias)
|
| 655 |
+
self.k_proj = nn.Linear(H, nkv * hd, bias=arch.attention_bias)
|
| 656 |
+
self.v_proj = nn.Linear(H, nkv * hd, bias=arch.attention_bias)
|
| 657 |
+
self.o_proj = nn.Linear(nh * hd, H, bias=arch.attention_bias)
|
| 658 |
+
self.q_norm = RMSNorm(hd, eps=arch.rms_norm_eps)
|
| 659 |
+
self.k_norm = RMSNorm(hd, eps=arch.rms_norm_eps)
|
| 660 |
+
self.indexer = QSAIndexer(arch)
|
| 661 |
+
|
| 662 |
+
def forward(self, hidden_states, cos, sin, causal_mask):
|
| 663 |
+
""
|
| 664 |
+
bsz, seq_len, _ = hidden_states.shape
|
| 665 |
+
hidden_shape = (bsz, seq_len, -1, self.head_dim)
|
| 666 |
+
|
| 667 |
+
selected = self.indexer(hidden_states, cos, sin)
|
| 668 |
+
allowed = causal_mask[None, None] if selected is None else (causal_mask[None] & selected)[:, None]
|
| 669 |
+
|
| 670 |
+
query_states, gate = torch.chunk(
|
| 671 |
+
self.q_proj(hidden_states).view(bsz, seq_len, -1, self.head_dim * 2), 2, dim=-1)
|
| 672 |
+
gate = gate.reshape(bsz, seq_len, -1)
|
| 673 |
+
|
| 674 |
+
query_states = self.q_norm(query_states.view(hidden_shape)).transpose(1, 2)
|
| 675 |
+
key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
|
| 676 |
+
value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)
|
| 677 |
+
|
| 678 |
+
query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
|
| 679 |
+
|
| 680 |
+
rep = self.num_key_value_groups
|
| 681 |
+
key_states = key_states.repeat_interleave(rep, dim=1)
|
| 682 |
+
value_states = value_states.repeat_interleave(rep, dim=1)
|
| 683 |
+
|
| 684 |
+
attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) * self.scaling
|
| 685 |
+
attn_weights = attn_weights.masked_fill(~allowed, float("-inf"))
|
| 686 |
+
attn_weights = F.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query_states.dtype)
|
| 687 |
+
attn_output = torch.matmul(attn_weights, value_states)
|
| 688 |
+
attn_output = attn_output.transpose(1, 2).contiguous().reshape(bsz, seq_len, -1)
|
| 689 |
+
attn_output = attn_output * torch.sigmoid(gate)
|
| 690 |
+
return self.o_proj(attn_output)
|
| 691 |
+
|
| 692 |
+
class MLPBlock(nn.Module):
|
| 693 |
+
def __init__(self, hidden_size: int, intermediate_size: int):
|
| 694 |
+
super().__init__()
|
| 695 |
+
self.gate_proj = nn.Linear(hidden_size, intermediate_size, bias=False)
|
| 696 |
+
self.up_proj = nn.Linear(hidden_size, intermediate_size, bias=False)
|
| 697 |
+
self.down_proj = nn.Linear(intermediate_size, hidden_size, bias=False)
|
| 698 |
+
|
| 699 |
+
def forward(self, x):
|
| 700 |
+
return self.down_proj(F.silu(self.gate_proj(x)) * self.up_proj(x))
|
| 701 |
+
|
| 702 |
+
class SparseMoeBlock(nn.Module):
|
| 703 |
+
""
|
| 704 |
+
|
| 705 |
+
def __init__(self, arch: FlashArch = DEFAULT_ARCH):
|
| 706 |
+
super().__init__()
|
| 707 |
+
self.num_experts = arch.num_experts
|
| 708 |
+
self.top_k = arch.num_experts_per_tok
|
| 709 |
+
self.norm_topk_prob = arch.norm_topk_prob
|
| 710 |
+
self.gate = nn.Linear(arch.hidden_size, arch.num_experts, bias=False)
|
| 711 |
+
self.experts = nn.ModuleList(
|
| 712 |
+
[MLPBlock(arch.hidden_size, arch.moe_intermediate_size) for _ in range(arch.num_experts)])
|
| 713 |
+
self.shared_expert = MLPBlock(arch.hidden_size, arch.shared_expert_intermediate_size)
|
| 714 |
+
self.shared_expert_gate = nn.Linear(arch.hidden_size, 1, bias=False)
|
| 715 |
+
|
| 716 |
+
def route(self, x: torch.Tensor):
|
| 717 |
+
router_logits = F.linear(x, self.gate.weight)
|
| 718 |
+
router_probs = F.softmax(router_logits, dtype=torch.float, dim=-1)
|
| 719 |
+
top_value, top_index = torch.topk(router_probs, self.top_k, dim=-1)
|
| 720 |
+
if self.norm_topk_prob:
|
| 721 |
+
top_value /= top_value.sum(dim=-1, keepdim=True)
|
| 722 |
+
return router_logits, top_value.to(router_logits.dtype), top_index
|
| 723 |
+
|
| 724 |
+
def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
|
| 725 |
+
""
|
| 726 |
+
bsz, seq_len, hidden_dim = hidden_states.shape
|
| 727 |
+
x = hidden_states.view(-1, hidden_dim)
|
| 728 |
+
num_tokens = x.shape[0]
|
| 729 |
+
shared_out = self.shared_expert(x)
|
| 730 |
+
_, top_w, top_i = self.route(x)
|
| 731 |
+
|
| 732 |
+
per_k = torch.zeros(num_tokens, self.top_k, hidden_dim, dtype=x.dtype, device=x.device)
|
| 733 |
+
with torch.no_grad():
|
| 734 |
+
expert_mask = F.one_hot(top_i, num_classes=self.num_experts + 1).permute(2, 1, 0)
|
| 735 |
+
expert_hit = torch.greater(expert_mask.sum(dim=(-1, -2)), 0).nonzero()
|
| 736 |
+
for e in expert_hit:
|
| 737 |
+
e = int(e[0])
|
| 738 |
+
if e == self.num_experts:
|
| 739 |
+
continue
|
| 740 |
+
top_k_pos, token_idx = torch.where(expert_mask[e])
|
| 741 |
+
h = self.experts[e](x[token_idx])
|
| 742 |
+
per_k[token_idx, top_k_pos] = (h * top_w[token_idx, top_k_pos, None]).to(x.dtype)
|
| 743 |
+
out = per_k.sum(dim=1).to(x.dtype)
|
| 744 |
+
|
| 745 |
+
shared_out = torch.sigmoid(self.shared_expert_gate(x)) * shared_out
|
| 746 |
+
return (out + shared_out).reshape(bsz, seq_len, hidden_dim)
|
| 747 |
+
|
| 748 |
+
class GatedResidual(nn.Module):
|
| 749 |
+
""
|
| 750 |
+
|
| 751 |
+
def __init__(self, arch: FlashArch = DEFAULT_ARCH, use_combine: bool = True):
|
| 752 |
+
super().__init__()
|
| 753 |
+
self.hc_count = arch.hc_count
|
| 754 |
+
self.hidden_size = arch.hidden_size
|
| 755 |
+
hc_hidden = arch.hc_hidden_size
|
| 756 |
+
self.hc_norm = RMSNorm(hc_hidden, eps=arch.rms_norm_eps, group_size=arch.hidden_size)
|
| 757 |
+
self.input_mix_weight_down = nn.Linear(hc_hidden, arch.hc_lowrank, bias=False)
|
| 758 |
+
self.input_mix_weight_up = nn.Linear(arch.hc_lowrank, hc_hidden, bias=False)
|
| 759 |
+
self.block_inject_weight = nn.Linear(hc_hidden, arch.hc_count, bias=False) if use_combine else None
|
| 760 |
+
|
| 761 |
+
def forward(self, hyper_input: torch.Tensor):
|
| 762 |
+
normed = self.hc_norm(hyper_input)
|
| 763 |
+
mix_w = F.silu(self.input_mix_weight_down(normed) / self.hc_count)
|
| 764 |
+
mix_w = torch.sigmoid(self.input_mix_weight_up(mix_w))
|
| 765 |
+
mix_w = mix_w.unflatten(-1, (self.hc_count, self.hidden_size))
|
| 766 |
+
mixed_input = (mix_w * normed.unflatten(-1, (self.hc_count, self.hidden_size))).mean(dim=-2)
|
| 767 |
+
if self.block_inject_weight is None:
|
| 768 |
+
return mixed_input
|
| 769 |
+
injection_weights = 2 * torch.sigmoid(self.block_inject_weight(normed) / self.hc_count)
|
| 770 |
+
return mixed_input, hyper_input, injection_weights
|
| 771 |
+
|
| 772 |
+
def hc_inject(hyper_input, block_output, injection_weights):
|
| 773 |
+
""
|
| 774 |
+
injection = block_output.unsqueeze(-2) * injection_weights.unsqueeze(-1)
|
| 775 |
+
return hyper_input + injection.flatten(-2)
|
| 776 |
+
|
| 777 |
+
_MASK64 = (1 << 64) - 1
|
| 778 |
+
_SPLITMIX_GAMMA = 0x9E3779B97F4A7C15
|
| 779 |
+
_SPLITMIX_M1 = 0xBF58476D1CE4E5B9
|
| 780 |
+
_SPLITMIX_M2 = 0x94D049BB133111EB
|
| 781 |
+
_PRIME_1 = 10007
|
| 782 |
+
|
| 783 |
+
def _splitmix64(value: int) -> int:
|
| 784 |
+
value = (value + _SPLITMIX_GAMMA) & _MASK64
|
| 785 |
+
value = ((value ^ (value >> 30)) * _SPLITMIX_M1) & _MASK64
|
| 786 |
+
value = ((value ^ (value >> 27)) * _SPLITMIX_M2) & _MASK64
|
| 787 |
+
return (value ^ (value >> 31)) & _MASK64
|
| 788 |
+
|
| 789 |
+
def _build_layer_multipliers(unigram_vocab_size, ngram_size, ple_layer_index, seed):
|
| 790 |
+
max_long = (1 << 63) - 1
|
| 791 |
+
half_bound = max(1, (max_long // max(unigram_vocab_size, 1)) // 2)
|
| 792 |
+
base_seed = seed + _PRIME_1 * ple_layer_index
|
| 793 |
+
out = []
|
| 794 |
+
for index in range(ngram_size):
|
| 795 |
+
value = (base_seed + _SPLITMIX_GAMMA * (index + 1)) & _MASK64
|
| 796 |
+
out.append(2 * (_splitmix64(value) % half_bound) + 1)
|
| 797 |
+
return out
|
| 798 |
+
|
| 799 |
+
def _is_prime(value: int) -> bool:
|
| 800 |
+
if value < 2:
|
| 801 |
+
return False
|
| 802 |
+
if value % 2 == 0:
|
| 803 |
+
return value == 2
|
| 804 |
+
for divisor in range(3, math.isqrt(value) + 1, 2):
|
| 805 |
+
if value % divisor == 0:
|
| 806 |
+
return False
|
| 807 |
+
return True
|
| 808 |
+
|
| 809 |
+
def _find_nth_prime_after(start: int, count: int) -> int:
|
| 810 |
+
prime = start
|
| 811 |
+
for _ in range(count):
|
| 812 |
+
prime += 1
|
| 813 |
+
while not _is_prime(prime):
|
| 814 |
+
prime += 1
|
| 815 |
+
return prime
|
| 816 |
+
|
| 817 |
+
class ShardedEmbedding(nn.Module):
|
| 818 |
+
""
|
| 819 |
+
|
| 820 |
+
def __init__(self, num_rows: int, dim: int, num_shards: int):
|
| 821 |
+
super().__init__()
|
| 822 |
+
if num_rows % num_shards:
|
| 823 |
+
raise ValueError(f"num_rows={num_rows} is not divisible by num_shards={num_shards}")
|
| 824 |
+
self.num_shards = num_shards
|
| 825 |
+
self.rows_per_shard = num_rows // num_shards
|
| 826 |
+
self.dim = dim
|
| 827 |
+
for k in range(num_shards):
|
| 828 |
+
self.add_module(f"shard_{k}", nn.Embedding(self.rows_per_shard, dim))
|
| 829 |
+
|
| 830 |
+
def forward(self, ids: torch.Tensor) -> torch.Tensor:
|
| 831 |
+
flat = ids.reshape(-1)
|
| 832 |
+
w0 = self.shard_0.weight
|
| 833 |
+
out = torch.empty(flat.numel(), self.dim, dtype=w0.dtype, device=flat.device)
|
| 834 |
+
shard_idx = torch.div(flat, self.rows_per_shard, rounding_mode="floor")
|
| 835 |
+
local = flat - shard_idx * self.rows_per_shard
|
| 836 |
+
for k in torch.unique(shard_idx).tolist():
|
| 837 |
+
m = shard_idx == k
|
| 838 |
+
w = getattr(self, f"shard_{k}").weight
|
| 839 |
+
out[m] = F.embedding(local[m].to(w.device), w).to(flat.device)
|
| 840 |
+
return out.view(*ids.shape, self.dim)
|
| 841 |
+
|
| 842 |
+
class NGramEmbedding(nn.Module):
|
| 843 |
+
""
|
| 844 |
+
|
| 845 |
+
def __init__(self, arch: FlashArch, ple_layer_index: int):
|
| 846 |
+
super().__init__()
|
| 847 |
+
self.arch = arch
|
| 848 |
+
self.ngram_size = arch.ngram_size
|
| 849 |
+
self.context_len = arch.ngram_size - 1
|
| 850 |
+
self.heads_per_ngram = arch.heads_per_ngram
|
| 851 |
+
self.ngram_heads = (arch.ngram_size - 1) * arch.heads_per_ngram
|
| 852 |
+
self.ple_layer_index = ple_layer_index
|
| 853 |
+
self.eos_token_id = arch.eos_token_id
|
| 854 |
+
head_dim_per_ngram = arch.ple_embed_dim // self.ngram_heads
|
| 855 |
+
|
| 856 |
+
self.head_vocab_sizes, self.head_offsets, total = [], [], 0
|
| 857 |
+
for h in range(self.ngram_heads):
|
| 858 |
+
size = _find_nth_prime_after(arch.ngram_vocab_size_base - 1, ple_layer_index * self.ngram_heads + h + 1)
|
| 859 |
+
self.head_vocab_sizes.append(size)
|
| 860 |
+
self.head_offsets.append(total)
|
| 861 |
+
total += size
|
| 862 |
+
self.total_vocab_size = total
|
| 863 |
+
div = arch.make_ngram_vocab_size_divisible_by
|
| 864 |
+
self.padded_vocab_size = math.ceil(total / div) * div
|
| 865 |
+
|
| 866 |
+
self.register_buffer("layer_multipliers", torch.zeros(self.ngram_size, dtype=torch.long), persistent=False)
|
| 867 |
+
self.register_buffer("ngram_heads_vocab_sizes", torch.zeros(self.ngram_heads, dtype=torch.long), persistent=False)
|
| 868 |
+
self.register_buffer("ngram_heads_offsets", torch.zeros(self.ngram_heads, dtype=torch.long), persistent=False)
|
| 869 |
+
self.ngram_embedding = ShardedEmbedding(self.padded_vocab_size, head_dim_per_ngram, arch.split_ngram_parts)
|
| 870 |
+
if self.layer_multipliers.device.type != "meta":
|
| 871 |
+
self.reset_derived_buffers(self.layer_multipliers.device)
|
| 872 |
+
|
| 873 |
+
def reset_derived_buffers(self, device="cpu"):
|
| 874 |
+
a = self.arch
|
| 875 |
+
self.layer_multipliers = torch.tensor(
|
| 876 |
+
_build_layer_multipliers(a.vocab_size, self.ngram_size, self.ple_layer_index, a.seed),
|
| 877 |
+
dtype=torch.long, device=device)
|
| 878 |
+
self.ngram_heads_vocab_sizes = torch.tensor(self.head_vocab_sizes, dtype=torch.long, device=device)
|
| 879 |
+
self.ngram_heads_offsets = torch.tensor(self.head_offsets, dtype=torch.long, device=device)
|
| 880 |
+
|
| 881 |
+
def _shift_right_ignore_eos(self, token_ids: torch.Tensor, shift: int) -> torch.Tensor:
|
| 882 |
+
if shift == 0:
|
| 883 |
+
return token_ids
|
| 884 |
+
batch_size, seq_len = token_ids.shape
|
| 885 |
+
positions = torch.arange(seq_len, device=token_ids.device, dtype=torch.long)
|
| 886 |
+
eos_positions = torch.where(token_ids == self.eos_token_id, positions, -1)
|
| 887 |
+
prev_incl = torch.cummax(eos_positions, dim=1).values
|
| 888 |
+
prev_eos = torch.cat([eos_positions.new_full((batch_size, 1), -1), prev_incl[:, :-1]], dim=1)
|
| 889 |
+
position_in_segment = positions.unsqueeze(0) - (prev_eos + 1)
|
| 890 |
+
source_positions = positions - shift
|
| 891 |
+
gather_positions = source_positions.clamp_min(0).unsqueeze(0).expand(batch_size, -1)
|
| 892 |
+
shifted = token_ids.gather(dim=1, index=gather_positions)
|
| 893 |
+
valid = (position_in_segment >= shift) & (source_positions.unsqueeze(0) >= 0)
|
| 894 |
+
return torch.where(valid, shifted, token_ids.new_full((), self.eos_token_id))
|
| 895 |
+
|
| 896 |
+
def forward(self, input_ids: torch.Tensor) -> torch.Tensor:
|
| 897 |
+
input_ids = input_ids.long()
|
| 898 |
+
previous_context = input_ids.new_full((input_ids.shape[0], self.context_len), self.eos_token_id)
|
| 899 |
+
token_history = torch.cat([previous_context, input_ids], dim=-1)
|
| 900 |
+
shifted = [self._shift_right_ignore_eos(token_history, s) for s in range(self.ngram_size)]
|
| 901 |
+
|
| 902 |
+
blocks = []
|
| 903 |
+
for ngram in range(2, self.ngram_size + 1):
|
| 904 |
+
s0 = (ngram - 2) * self.heads_per_ngram
|
| 905 |
+
s1 = s0 + self.heads_per_ngram
|
| 906 |
+
mixed = shifted[0] * self.layer_multipliers[0]
|
| 907 |
+
for p in range(1, ngram):
|
| 908 |
+
mixed = torch.bitwise_xor(mixed, shifted[p] * self.layer_multipliers[p])
|
| 909 |
+
vs = self.ngram_heads_vocab_sizes[s0:s1]
|
| 910 |
+
offs = self.ngram_heads_offsets[s0:s1]
|
| 911 |
+
ids = torch.remainder(mixed.unsqueeze(-1), vs.view(1, 1, -1))
|
| 912 |
+
blocks.append(ids + offs.view(1, 1, -1))
|
| 913 |
+
|
| 914 |
+
ngram_ids = torch.cat(blocks, dim=-1)[:, -input_ids.shape[1]:]
|
| 915 |
+
return self.ngram_embedding(ngram_ids).flatten(-2)
|
| 916 |
+
|
| 917 |
+
class PLELayer(nn.Module):
|
| 918 |
+
""
|
| 919 |
+
|
| 920 |
+
def __init__(self, arch: FlashArch, ple_layer_index: int):
|
| 921 |
+
super().__init__()
|
| 922 |
+
self.hidden_size = arch.hidden_size
|
| 923 |
+
self.hc_count = arch.hc_count
|
| 924 |
+
hc_hidden = arch.hc_hidden_size
|
| 925 |
+
self.ple_embedding = NGramEmbedding(arch, ple_layer_index)
|
| 926 |
+
self.short_conv_state_len = (arch.ple_conv_kernel_size - 1) * arch.ngram_size
|
| 927 |
+
self.key_proj = nn.Linear(arch.ple_embed_dim, hc_hidden, bias=False)
|
| 928 |
+
self.value_proj = nn.Linear(arch.ple_embed_dim, arch.hidden_size, bias=False)
|
| 929 |
+
self.norm_key = RMSNorm(hc_hidden, eps=arch.rms_norm_eps, group_size=arch.hidden_size)
|
| 930 |
+
self.norm_query = RMSNorm(hc_hidden, eps=arch.rms_norm_eps, group_size=arch.hidden_size)
|
| 931 |
+
self.norm_conv = RMSNorm(hc_hidden, eps=arch.rms_norm_eps, group_size=arch.hidden_size)
|
| 932 |
+
self.conv1d = nn.Conv1d(hc_hidden, hc_hidden, kernel_size=arch.ple_conv_kernel_size,
|
| 933 |
+
groups=hc_hidden, dilation=arch.ngram_size, bias=False)
|
| 934 |
+
|
| 935 |
+
def _short_conv(self, x: torch.Tensor) -> torch.Tensor:
|
| 936 |
+
x = x.transpose(1, 2)
|
| 937 |
+
x = F.pad(x, (self.short_conv_state_len, 0))
|
| 938 |
+
x = F.silu(self.conv1d(x))
|
| 939 |
+
return x.transpose(1, 2)
|
| 940 |
+
|
| 941 |
+
def forward(self, hidden_states: torch.Tensor, input_ids: torch.Tensor) -> torch.Tensor:
|
| 942 |
+
emb = self.ple_embedding(input_ids)
|
| 943 |
+
key_n = self.norm_key(self.key_proj(emb)).unflatten(-1, (self.hc_count, self.hidden_size))
|
| 944 |
+
value = self.value_proj(emb)
|
| 945 |
+
query_n = self.norm_query(hidden_states).unflatten(-1, (self.hc_count, self.hidden_size))
|
| 946 |
+
gate = (key_n * query_n).sum(dim=-1, keepdim=True) / math.sqrt(self.hidden_size)
|
| 947 |
+
gate = gate.abs().clamp_min(1e-6).sqrt() * gate.sign()
|
| 948 |
+
gated_value = torch.sigmoid(gate) * value.unsqueeze(-2)
|
| 949 |
+
gated_value_normed = self.norm_conv(gated_value.flatten(-2))
|
| 950 |
+
gated_value = gated_value.flatten(-2)
|
| 951 |
+
return gated_value + self._short_conv(gated_value_normed)
|
| 952 |
+
|
| 953 |
+
class TransformerBlock(nn.Module):
|
| 954 |
+
""
|
| 955 |
+
|
| 956 |
+
def __init__(self, layer_idx: int, layer_type: str = None, arch: FlashArch = DEFAULT_ARCH,
|
| 957 |
+
allow_ple: bool = True):
|
| 958 |
+
super().__init__()
|
| 959 |
+
self.layer_idx = layer_idx
|
| 960 |
+
self.block_type = layer_type if layer_type is not None else arch.layer_types[layer_idx]
|
| 961 |
+
if self.block_type == "linear_attention":
|
| 962 |
+
self.linear_attn = GatedDeltaNetBlock(arch)
|
| 963 |
+
else:
|
| 964 |
+
self.self_attn = AttentionBlock(arch)
|
| 965 |
+
self.mlp = SparseMoeBlock(arch)
|
| 966 |
+
ple_ids = list(arch.ple_layer_ids)
|
| 967 |
+
if allow_ple and (layer_idx + 1) in ple_ids:
|
| 968 |
+
if self.block_type != "linear_attention":
|
| 969 |
+
raise ValueError("PLE is supported only on linear_attention layers")
|
| 970 |
+
self.ple = PLELayer(arch, ple_ids.index(layer_idx + 1))
|
| 971 |
+
else:
|
| 972 |
+
self.ple = None
|
| 973 |
+
self.attn_hyper_connection = GatedResidual(arch)
|
| 974 |
+
self.mlp_hyper_connection = GatedResidual(arch)
|
| 975 |
+
|
| 976 |
+
def forward(self, hidden_states, cos, sin, causal_mask, input_ids=None):
|
| 977 |
+
if self.ple is not None:
|
| 978 |
+
hidden_states = hidden_states + self.ple(hidden_states, input_ids)
|
| 979 |
+
|
| 980 |
+
x, hyper_input, inj = self.attn_hyper_connection(hidden_states)
|
| 981 |
+
if self.block_type == "linear_attention":
|
| 982 |
+
x = self.linear_attn(x)
|
| 983 |
+
else:
|
| 984 |
+
x = self.self_attn(x, cos, sin, causal_mask)
|
| 985 |
+
hidden_states = hc_inject(hyper_input, x, inj)
|
| 986 |
+
|
| 987 |
+
x, hyper_input, inj = self.mlp_hyper_connection(hidden_states)
|
| 988 |
+
x = self.mlp(x)
|
| 989 |
+
return hc_inject(hyper_input, x, inj)
|
| 990 |
+
|
| 991 |
+
class MTPBlock(nn.Module):
|
| 992 |
+
""
|
| 993 |
+
|
| 994 |
+
def __init__(self, arch: FlashArch = DEFAULT_ARCH):
|
| 995 |
+
super().__init__()
|
| 996 |
+
self.hc_count = arch.hc_count
|
| 997 |
+
self.hidden_size = arch.hidden_size
|
| 998 |
+
self.pre_fc_norm_embedding = RMSNorm(arch.hidden_size, eps=arch.rms_norm_eps)
|
| 999 |
+
self.pre_fc_norm_hidden = RMSNorm(arch.hc_hidden_size, eps=arch.rms_norm_eps)
|
| 1000 |
+
self.fc_embedding = nn.Linear(arch.hidden_size, arch.hidden_size, bias=False)
|
| 1001 |
+
self.fc_hidden = nn.Linear(arch.hidden_size, arch.hidden_size, bias=False)
|
| 1002 |
+
self.layers = nn.ModuleList([
|
| 1003 |
+
TransformerBlock(arch.num_layers + i, layer_type="full_attention", arch=arch, allow_ple=False)
|
| 1004 |
+
for i in range(arch.mtp_num_hidden_layers)
|
| 1005 |
+
])
|
| 1006 |
+
self.hyper_connection_mixer = GatedResidual(arch, use_combine=False)
|
| 1007 |
+
|
| 1008 |
+
def forward(self, inputs_embeds, previous_multistream, cos, sin, causal_mask, step: int = 0):
|
| 1009 |
+
e = self.fc_embedding(self.pre_fc_norm_embedding(inputs_embeds))
|
| 1010 |
+
h = self.pre_fc_norm_hidden(previous_multistream).unflatten(-1, (self.hc_count, self.hidden_size))
|
| 1011 |
+
h = self.fc_hidden(h)
|
| 1012 |
+
h = (e.unsqueeze(-2) + h).flatten(-2)
|
| 1013 |
+
h = self.layers[step % len(self.layers)](h, cos, sin, causal_mask)
|
| 1014 |
+
return self.hyper_connection_mixer(h), h
|
| 1015 |
+
|
| 1016 |
+
class JiRackQwen38FlashModel(nn.Module):
|
| 1017 |
+
def __init__(self, arch: FlashArch = DEFAULT_ARCH):
|
| 1018 |
+
super().__init__()
|
| 1019 |
+
self.arch = arch
|
| 1020 |
+
self.embed_tokens = nn.Embedding(arch.vocab_size, arch.hidden_size)
|
| 1021 |
+
self.layers = nn.ModuleList([TransformerBlock(i, arch=arch) for i in range(arch.num_layers)])
|
| 1022 |
+
self.rotary_emb = TextRotaryEmbedding(arch)
|
| 1023 |
+
self.hyper_connection_mixer = GatedResidual(arch, use_combine=False)
|
| 1024 |
+
|
| 1025 |
+
self.gradient_checkpointing = False
|
| 1026 |
+
|
| 1027 |
+
def prepare(self, input_ids: torch.LongTensor, dtype):
|
| 1028 |
+
bsz, seq_len = input_ids.shape
|
| 1029 |
+
position_ids = torch.arange(seq_len, device=input_ids.device).unsqueeze(0).expand(bsz, -1)
|
| 1030 |
+
dummy = torch.empty(0, dtype=dtype, device=input_ids.device)
|
| 1031 |
+
cos, sin = self.rotary_emb(dummy, position_ids)
|
| 1032 |
+
causal_mask = torch.ones(seq_len, seq_len, dtype=torch.bool, device=input_ids.device).tril()
|
| 1033 |
+
return cos, sin, causal_mask
|
| 1034 |
+
|
| 1035 |
+
def forward(self, input_ids: torch.LongTensor, return_multistream: bool = False):
|
| 1036 |
+
inputs_embeds = self.embed_tokens(input_ids)
|
| 1037 |
+
cos, sin, causal_mask = self.prepare(input_ids, inputs_embeds.dtype)
|
| 1038 |
+
|
| 1039 |
+
hidden_states = inputs_embeds.repeat(1, 1, self.arch.hc_count)
|
| 1040 |
+
use_ckpt = self.gradient_checkpointing and self.training and torch.is_grad_enabled()
|
| 1041 |
+
for layer in self.layers:
|
| 1042 |
+
if use_ckpt:
|
| 1043 |
+
hidden_states = torch.utils.checkpoint.checkpoint(
|
| 1044 |
+
layer, hidden_states, cos, sin, causal_mask, input_ids, use_reentrant=False)
|
| 1045 |
+
else:
|
| 1046 |
+
hidden_states = layer(hidden_states, cos, sin, causal_mask, input_ids)
|
| 1047 |
+
|
| 1048 |
+
out = self.hyper_connection_mixer(hidden_states)
|
| 1049 |
+
if return_multistream:
|
| 1050 |
+
return out, hidden_states, (cos, sin, causal_mask)
|
| 1051 |
+
return out
|
| 1052 |
+
|
| 1053 |
+
_ST_DTYPES = {
|
| 1054 |
+
"BF16": torch.bfloat16, "F16": torch.float16, "F32": torch.float32, "F64": torch.float64,
|
| 1055 |
+
"I64": torch.int64, "I32": torch.int32, "I16": torch.int16, "I8": torch.int8,
|
| 1056 |
+
"U8": torch.uint8, "BOOL": torch.bool,
|
| 1057 |
+
}
|
| 1058 |
+
_FP8_TAGS = ("F8_E4M3", "F8_E5M2")
|
| 1059 |
+
|
| 1060 |
+
def mmap_safetensors(path: str, keys=None) -> dict:
|
| 1061 |
+
""
|
| 1062 |
+
with open(path, "rb") as f:
|
| 1063 |
+
header_len = struct.unpack("<Q", f.read(8))[0]
|
| 1064 |
+
header = json.loads(f.read(header_len))
|
| 1065 |
+
mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_COPY)
|
| 1066 |
+
base = 8 + header_len
|
| 1067 |
+
out = {}
|
| 1068 |
+
for name, info in header.items():
|
| 1069 |
+
if name == "__metadata__" or (keys is not None and name not in keys):
|
| 1070 |
+
continue
|
| 1071 |
+
tag = info["dtype"]
|
| 1072 |
+
if tag in _FP8_TAGS:
|
| 1073 |
+
raise ValueError(f"{path}: FP8 weights ({name}). The BF16 repository "
|
| 1074 |
+
f"Qwen/Qwen3.8-Flash-Next is required, not -FP8.")
|
| 1075 |
+
dtype = _ST_DTYPES[tag]
|
| 1076 |
+
start, end = info["data_offsets"]
|
| 1077 |
+
shape = info["shape"]
|
| 1078 |
+
itemsize = torch.empty((), dtype=dtype).element_size()
|
| 1079 |
+
count = (end - start) // itemsize
|
| 1080 |
+
if count == 0:
|
| 1081 |
+
out[name] = torch.empty(shape, dtype=dtype)
|
| 1082 |
+
else:
|
| 1083 |
+
out[name] = torch.frombuffer(mm, dtype=dtype, count=count, offset=base + start).view(shape)
|
| 1084 |
+
return out
|
| 1085 |
+
|
| 1086 |
+
def list_hf_dir_keys(hf_dir: str) -> dict:
|
| 1087 |
+
""
|
| 1088 |
+
index_path = os.path.join(hf_dir, "model.safetensors.index.json")
|
| 1089 |
+
if os.path.exists(index_path):
|
| 1090 |
+
with open(index_path) as f:
|
| 1091 |
+
wm = json.load(f)["weight_map"]
|
| 1092 |
+
return {k: os.path.join(hf_dir, v) for k, v in wm.items()}
|
| 1093 |
+
out = {}
|
| 1094 |
+
for shard in sorted(glob.glob(os.path.join(hf_dir, "*.safetensors"))):
|
| 1095 |
+
with open(shard, "rb") as f:
|
| 1096 |
+
n = struct.unpack("<Q", f.read(8))[0]
|
| 1097 |
+
for k in json.loads(f.read(n)):
|
| 1098 |
+
if k != "__metadata__":
|
| 1099 |
+
out[k] = shard
|
| 1100 |
+
return out
|
| 1101 |
+
|
| 1102 |
+
_DERIVED_BUFFER_SUFFIXES = ("layer_multipliers", "ngram_heads_vocab_sizes", "ngram_heads_offsets",
|
| 1103 |
+
"inv_freq", "original_inv_freq")
|
| 1104 |
+
|
| 1105 |
+
def normalize_hf_key(k: str):
|
| 1106 |
+
""
|
| 1107 |
+
for p in ("model.language_model.", "language_model.model.", "language_model."):
|
| 1108 |
+
if k.startswith(p):
|
| 1109 |
+
k = "model." + k[len(p):]
|
| 1110 |
+
break
|
| 1111 |
+
if k.startswith("model.mtp."):
|
| 1112 |
+
k = k[len("model."):]
|
| 1113 |
+
if k.startswith(("model.visual.", "visual.", "mtp.shared_head.head.", "model.shared_head.head.")):
|
| 1114 |
+
return None
|
| 1115 |
+
if k.endswith(_DERIVED_BUFFER_SUFFIXES):
|
| 1116 |
+
return None
|
| 1117 |
+
return k
|
| 1118 |
+
|
| 1119 |
+
_FUSED_GATE_UP = re.compile(r"^(.*\.mlp\.experts)\.gate_up_proj$")
|
| 1120 |
+
_FUSED_DOWN = re.compile(r"^(.*\.mlp\.experts)\.down_proj$")
|
| 1121 |
+
_NGRAM_FULL = re.compile(r"^(.*\.ngram_embedding)\.weight$")
|
| 1122 |
+
|
| 1123 |
+
def expand_hf_tensors(items, arch: FlashArch, dtype=None):
|
| 1124 |
+
""
|
| 1125 |
+
for k, v in items:
|
| 1126 |
+
k = normalize_hf_key(k)
|
| 1127 |
+
if k is None:
|
| 1128 |
+
continue
|
| 1129 |
+
if dtype is not None and v.is_floating_point() and v.dtype != dtype:
|
| 1130 |
+
v = v.to(dtype)
|
| 1131 |
+
m = _FUSED_GATE_UP.match(k)
|
| 1132 |
+
if m:
|
| 1133 |
+
inter = v.shape[1] // 2
|
| 1134 |
+
for e in range(v.shape[0]):
|
| 1135 |
+
yield f"{m.group(1)}.{e}.gate_proj.weight", v[e, :inter]
|
| 1136 |
+
yield f"{m.group(1)}.{e}.up_proj.weight", v[e, inter:]
|
| 1137 |
+
continue
|
| 1138 |
+
m = _FUSED_DOWN.match(k)
|
| 1139 |
+
if m and v.dim() == 3:
|
| 1140 |
+
for e in range(v.shape[0]):
|
| 1141 |
+
yield f"{m.group(1)}.{e}.down_proj.weight", v[e]
|
| 1142 |
+
continue
|
| 1143 |
+
m = _NGRAM_FULL.match(k)
|
| 1144 |
+
if m:
|
| 1145 |
+
parts = arch.split_ngram_parts
|
| 1146 |
+
rows = v.shape[0] // parts
|
| 1147 |
+
if rows * parts != v.shape[0]:
|
| 1148 |
+
raise ValueError(f"{k}: {v.shape[0]} rows are not divisible into {parts} shards")
|
| 1149 |
+
for s in range(parts):
|
| 1150 |
+
yield f"{m.group(1)}.shard_{s}.weight", v.narrow(0, s * rows, rows)
|
| 1151 |
+
continue
|
| 1152 |
+
yield k, v
|
| 1153 |
+
|
| 1154 |
+
class JiRackQwen38FlashForCausalLM(nn.Module):
|
| 1155 |
+
""
|
| 1156 |
+
|
| 1157 |
+
def __init__(self, arch: FlashArch = DEFAULT_ARCH):
|
| 1158 |
+
super().__init__()
|
| 1159 |
+
self.arch = arch
|
| 1160 |
+
self.model = JiRackQwen38FlashModel(arch)
|
| 1161 |
+
self.lm_head = nn.Linear(arch.hidden_size, arch.vocab_size, bias=False)
|
| 1162 |
+
self.mtp = MTPBlock(arch)
|
| 1163 |
+
|
| 1164 |
+
def forward(self, input_ids: torch.LongTensor) -> torch.Tensor:
|
| 1165 |
+
return self.lm_head(self.model(input_ids))
|
| 1166 |
+
|
| 1167 |
+
def mtp_forward(self, input_ids: torch.LongTensor):
|
| 1168 |
+
""
|
| 1169 |
+
out, multistream, (cos, sin, causal) = self.model(input_ids, return_multistream=True)
|
| 1170 |
+
logits_t1 = self.lm_head(out)
|
| 1171 |
+
shifted = torch.cat([input_ids[:, 1:], input_ids[:, -1:]], dim=1)
|
| 1172 |
+
emb_next = self.model.embed_tokens(shifted)
|
| 1173 |
+
h2, _ = self.mtp(emb_next, multistream, cos, sin, causal)
|
| 1174 |
+
return logits_t1, self.lm_head(h2)
|
| 1175 |
+
|
| 1176 |
+
@classmethod
|
| 1177 |
+
def build_empty(cls, arch: FlashArch = DEFAULT_ARCH) -> "JiRackQwen38FlashForCausalLM":
|
| 1178 |
+
""
|
| 1179 |
+
with torch.device("meta"):
|
| 1180 |
+
return cls(arch)
|
| 1181 |
+
|
| 1182 |
+
def reset_derived_buffers(self, device="cpu"):
|
| 1183 |
+
for m in self.modules():
|
| 1184 |
+
if m is not self and hasattr(m, "reset_derived_buffers"):
|
| 1185 |
+
m.reset_derived_buffers(device)
|
| 1186 |
+
|
| 1187 |
+
def meta_tensors(self, ignore_prefixes=()):
|
| 1188 |
+
bad = [n for n, t in list(self.named_parameters()) + list(self.named_buffers())
|
| 1189 |
+
if t.is_meta and not n.startswith(tuple(ignore_prefixes))]
|
| 1190 |
+
return bad
|
| 1191 |
+
|
| 1192 |
+
def assert_materialized(self, ignore_prefixes=()):
|
| 1193 |
+
bad = self.meta_tensors(ignore_prefixes)
|
| 1194 |
+
if bad:
|
| 1195 |
+
raise RuntimeError(f"{len(bad)} tensors were never loaded (still on meta). "
|
| 1196 |
+
f"First 10: {bad[:10]}")
|
| 1197 |
+
|
| 1198 |
+
def _load_items(self, items, dtype=None, verbose=True):
|
| 1199 |
+
own = dict(self.state_dict(keep_vars=True))
|
| 1200 |
+
sd, unexpected, shape_bad = {}, [], []
|
| 1201 |
+
for k, v in expand_hf_tensors(items, self.arch, dtype):
|
| 1202 |
+
if k not in own:
|
| 1203 |
+
unexpected.append(k)
|
| 1204 |
+
continue
|
| 1205 |
+
if tuple(own[k].shape) != tuple(v.shape):
|
| 1206 |
+
shape_bad.append((k, tuple(v.shape), tuple(own[k].shape)))
|
| 1207 |
+
continue
|
| 1208 |
+
sd[k] = v
|
| 1209 |
+
if shape_bad:
|
| 1210 |
+
raise ValueError(f"Shape mismatch ({len(shape_bad)}), first 5: {shape_bad[:5]}")
|
| 1211 |
+
self.load_state_dict(sd, strict=False, assign=True)
|
| 1212 |
+
if verbose and unexpected:
|
| 1213 |
+
print(f"⚠️ {len(unexpected)} unexpected keys (first 10): {unexpected[:10]}")
|
| 1214 |
+
return set(sd), unexpected
|
| 1215 |
+
|
| 1216 |
+
def load_hf_state_dict(self, state_dict: dict, strict: bool = False, dtype=None):
|
| 1217 |
+
""
|
| 1218 |
+
loaded, unexpected = self._load_items(state_dict.items(), dtype)
|
| 1219 |
+
own_keys = [k for k in self.state_dict() if not k.endswith("lambda_")]
|
| 1220 |
+
missing = [k for k in own_keys if k not in loaded]
|
| 1221 |
+
real_missing = [k for k in missing if not k.startswith("mtp.")]
|
| 1222 |
+
if real_missing:
|
| 1223 |
+
print(f"⚠️ {len(real_missing)} missing keys (first 10): {real_missing[:10]}")
|
| 1224 |
+
mtp_missing = len(missing) - len(real_missing)
|
| 1225 |
+
if mtp_missing:
|
| 1226 |
+
print(f"ℹ️ mtp.*: {mtp_missing} tensors are not in this state_dict "
|
| 1227 |
+
f"(HF does not keep them) -- load them with load_mtp_from_hf_dir().")
|
| 1228 |
+
if strict and (real_missing or unexpected):
|
| 1229 |
+
raise RuntimeError("strict=True: there are missing/unexpected keys")
|
| 1230 |
+
self.reset_derived_buffers(self.lm_head.weight.device if not self.lm_head.weight.is_meta else "cpu")
|
| 1231 |
+
return missing, unexpected
|
| 1232 |
+
|
| 1233 |
+
@classmethod
|
| 1234 |
+
def from_hf_state_dict(cls, state_dict: dict, arch: FlashArch = DEFAULT_ARCH, dtype=None):
|
| 1235 |
+
model = cls.build_empty(arch)
|
| 1236 |
+
model.load_hf_state_dict(state_dict, dtype=dtype)
|
| 1237 |
+
model.assert_materialized(ignore_prefixes=("mtp.",))
|
| 1238 |
+
return model
|
| 1239 |
+
|
| 1240 |
+
@classmethod
|
| 1241 |
+
def from_hf_dir(cls, hf_dir: str, dtype=torch.bfloat16, arch: FlashArch = None, include_mtp: bool = True):
|
| 1242 |
+
""
|
| 1243 |
+
cfg_path = os.path.join(hf_dir, "config.json")
|
| 1244 |
+
if arch is None:
|
| 1245 |
+
arch = FlashArch.from_config_json(cfg_path) if os.path.exists(cfg_path) else DEFAULT_ARCH
|
| 1246 |
+
if arch != DEFAULT_ARCH:
|
| 1247 |
+
print("⚠️ config.json differs from the built-in FlashArch constants -- using config.json")
|
| 1248 |
+
model = cls.build_empty(arch)
|
| 1249 |
+
key_map = list_hf_dir_keys(hf_dir)
|
| 1250 |
+
by_shard = {}
|
| 1251 |
+
for k, shard in key_map.items():
|
| 1252 |
+
nk = normalize_hf_key(k)
|
| 1253 |
+
if nk is None or (not include_mtp and nk.startswith("mtp.")):
|
| 1254 |
+
continue
|
| 1255 |
+
by_shard.setdefault(shard, []).append(k)
|
| 1256 |
+
n = 0
|
| 1257 |
+
for i, (shard, keys) in enumerate(sorted(by_shard.items())):
|
| 1258 |
+
tensors = mmap_safetensors(shard, keys=set(keys))
|
| 1259 |
+
loaded, _ = model._load_items(tensors.items(), dtype=dtype, verbose=True)
|
| 1260 |
+
n += len(loaded)
|
| 1261 |
+
if (i + 1) % 20 == 0 or i + 1 == len(by_shard):
|
| 1262 |
+
print(f" ... {i + 1}/{len(by_shard)} shards, {n} tensors")
|
| 1263 |
+
model.reset_derived_buffers("cpu")
|
| 1264 |
+
model.assert_materialized(ignore_prefixes=() if include_mtp else ("mtp.",))
|
| 1265 |
+
return model
|
| 1266 |
+
|
| 1267 |
+
def load_mtp_from_hf_dir(self, hf_dir: str):
|
| 1268 |
+
""
|
| 1269 |
+
key_map = {k: s for k, s in list_hf_dir_keys(hf_dir).items()
|
| 1270 |
+
if (normalize_hf_key(k) or "").startswith("mtp.")}
|
| 1271 |
+
if not key_map:
|
| 1272 |
+
raise FileNotFoundError(
|
| 1273 |
+
f"No mtp.* tensors found in {hf_dir}. The ORIGINAL "
|
| 1274 |
+
f"Qwen3.8-Flash-Next checkpoint is required (not a JiRack export).")
|
| 1275 |
+
ref = self.lm_head.weight
|
| 1276 |
+
dtype = None if ref.is_meta else ref.dtype
|
| 1277 |
+
by_shard = {}
|
| 1278 |
+
for k, s in key_map.items():
|
| 1279 |
+
by_shard.setdefault(s, []).append(k)
|
| 1280 |
+
n = 0
|
| 1281 |
+
for shard, keys in by_shard.items():
|
| 1282 |
+
loaded, _ = self._load_items(mmap_safetensors(shard, keys=set(keys)).items(), dtype=dtype)
|
| 1283 |
+
n += len(loaded)
|
| 1284 |
+
print(f"✅ MTP: loaded {n} tensors from {len(by_shard)} shards ({hf_dir})")
|
| 1285 |
+
return n
|
| 1286 |
+
|
| 1287 |
+
def save_checkpoint(self, path: str, lam: float, global_step: int = 0, patched_linear_names=None):
|
| 1288 |
+
torch.save({
|
| 1289 |
+
"model": self.state_dict(),
|
| 1290 |
+
"arch": asdict(self.arch),
|
| 1291 |
+
"lambda": lam,
|
| 1292 |
+
"global_step": global_step,
|
| 1293 |
+
"patched_linear_names": patched_linear_names or [],
|
| 1294 |
+
}, path)
|
| 1295 |
+
|
| 1296 |
+
@classmethod
|
| 1297 |
+
def load_checkpoint(cls, path: str, dtype=torch.bfloat16, include_gate_proj: bool = False, mmap_load: bool = True,
|
| 1298 |
+
freeze_router: bool = True):
|
| 1299 |
+
import zipfile
|
| 1300 |
+
if mmap_load and zipfile.is_zipfile(path):
|
| 1301 |
+
try:
|
| 1302 |
+
ckpt = torch.load(path, map_location="cpu", weights_only=False, mmap=True)
|
| 1303 |
+
except RuntimeError as e:
|
| 1304 |
+
|
| 1305 |
+
raise RuntimeError(
|
| 1306 |
+
f"torch.load(mmap=True) could not map {path}: {e}\n"
|
| 1307 |
+
"Likely cause: torch maps with MAP_PRIVATE (writable), and with "
|
| 1308 |
+
"vm.overcommit_memory=0 the kernel refuses a mapping larger than RAM+swap. "
|
| 1309 |
+
"Fix: python convert_pt_checkpoint_to_safetensors.py <this.pt> <directory> "
|
| 1310 |
+
"and load the directory (load_checkpoint_auto)."
|
| 1311 |
+
) from e
|
| 1312 |
+
else:
|
| 1313 |
+
|
| 1314 |
+
ckpt = torch.load(path, map_location="cpu", weights_only=False)
|
| 1315 |
+
arch = FlashArch.from_dict(ckpt["arch"]) if "arch" in ckpt else DEFAULT_ARCH
|
| 1316 |
+
model = cls.build_empty(arch)
|
| 1317 |
+
|
| 1318 |
+
patched_names = ckpt.get("patched_linear_names", [])
|
| 1319 |
+
if patched_names:
|
| 1320 |
+
for name in patched_names:
|
| 1321 |
+
parent_name, _, leaf = name.rpartition(".")
|
| 1322 |
+
parent = model.get_submodule(parent_name) if parent_name else model
|
| 1323 |
+
old = getattr(parent, leaf)
|
| 1324 |
+
if not isinstance(old, BitLinear):
|
| 1325 |
+
setattr(parent, leaf, BitLinear.from_linear(old))
|
| 1326 |
+
else:
|
| 1327 |
+
apply_bitlinear_patch(model, include_gate_proj=include_gate_proj)
|
| 1328 |
+
|
| 1329 |
+
sd = ckpt["model"]
|
| 1330 |
+
has_mtp = any(k.startswith("mtp.") for k in sd)
|
| 1331 |
+
model.load_state_dict(sd, strict=False, assign=True)
|
| 1332 |
+
model.reset_derived_buffers("cpu")
|
| 1333 |
+
|
| 1334 |
+
for m in model.modules():
|
| 1335 |
+
if isinstance(m, BitLinear) and m.lambda_.is_meta:
|
| 1336 |
+
m.lambda_ = torch.zeros(())
|
| 1337 |
+
model.assert_materialized(ignore_prefixes=() if has_mtp else ("mtp.",))
|
| 1338 |
+
model = model.to(dtype)
|
| 1339 |
+
|
| 1340 |
+
if not has_mtp:
|
| 1341 |
+
print("⚠️ This .pt has NO mtp.* weights. Before exporting, call "
|
| 1342 |
+
"model.load_mtp_from_hf_dir('<original HF checkpoint folder>').")
|
| 1343 |
+
|
| 1344 |
+
lam = ckpt.get("lambda", 0.0)
|
| 1345 |
+
for m in model.modules():
|
| 1346 |
+
if isinstance(m, BitLinear):
|
| 1347 |
+
m.set_lambda(lam)
|
| 1348 |
+
n_frozen = len(freeze_routers(model)) if freeze_router else 0
|
| 1349 |
+
print(f"✅ Loaded. global_step={ckpt.get('global_step', 0)}, lambda={lam}, "
|
| 1350 |
+
f"routers frozen: {n_frozen}")
|
| 1351 |
+
return model
|
| 1352 |
+
|
| 1353 |
+
def save_checkpoint_safetensors(self, out_dir: str, lam: float, global_step: int = 0,
|
| 1354 |
+
patched_linear_names=None, max_shard_bytes: int = 8 * 1024 ** 3,
|
| 1355 |
+
save_dtype=None):
|
| 1356 |
+
""
|
| 1357 |
+
from safetensors.torch import save_file
|
| 1358 |
+
os.makedirs(out_dir, exist_ok=True)
|
| 1359 |
+
|
| 1360 |
+
sd = self.state_dict()
|
| 1361 |
+
|
| 1362 |
+
def out_bytes(v):
|
| 1363 |
+
if save_dtype is not None and v.is_floating_point():
|
| 1364 |
+
return v.numel() * torch.empty((), dtype=save_dtype).element_size()
|
| 1365 |
+
return v.numel() * v.element_size()
|
| 1366 |
+
|
| 1367 |
+
shards, cur, cur_bytes = [], {}, 0
|
| 1368 |
+
for k, v in sd.items():
|
| 1369 |
+
nbytes = out_bytes(v)
|
| 1370 |
+
if cur and cur_bytes + nbytes > max_shard_bytes:
|
| 1371 |
+
shards.append(cur)
|
| 1372 |
+
cur, cur_bytes = {}, 0
|
| 1373 |
+
cur[k] = v
|
| 1374 |
+
cur_bytes += nbytes
|
| 1375 |
+
if cur:
|
| 1376 |
+
shards.append(cur)
|
| 1377 |
+
|
| 1378 |
+
total_size = sum(out_bytes(v) for v in sd.values())
|
| 1379 |
+
weight_map = {}
|
| 1380 |
+
n_shards = len(shards)
|
| 1381 |
+
for i, shard in enumerate(shards):
|
| 1382 |
+
name = f"checkpoint-{i + 1:05d}-of-{n_shards:05d}.safetensors"
|
| 1383 |
+
|
| 1384 |
+
data = {}
|
| 1385 |
+
for k, v in shard.items():
|
| 1386 |
+
v = v.detach().cpu()
|
| 1387 |
+
if save_dtype is not None and v.is_floating_point() and v.dtype != save_dtype:
|
| 1388 |
+
v = v.to(save_dtype)
|
| 1389 |
+
data[k] = v if v.is_contiguous() else v.contiguous()
|
| 1390 |
+
save_file(data, os.path.join(out_dir, name), metadata={"format": "pt"})
|
| 1391 |
+
del data
|
| 1392 |
+
for k in shard:
|
| 1393 |
+
weight_map[k] = name
|
| 1394 |
+
print(f" ... shard {i + 1}/{n_shards} written ({name})")
|
| 1395 |
+
|
| 1396 |
+
with open(os.path.join(out_dir, "checkpoint.index.json"), "w") as f:
|
| 1397 |
+
json.dump({"metadata": {"total_size": total_size}, "weight_map": weight_map}, f)
|
| 1398 |
+
with open(os.path.join(out_dir, "checkpoint_meta.json"), "w") as f:
|
| 1399 |
+
json.dump({
|
| 1400 |
+
"arch": asdict(self.arch),
|
| 1401 |
+
"lambda": lam,
|
| 1402 |
+
"global_step": global_step,
|
| 1403 |
+
"patched_linear_names": patched_linear_names or [],
|
| 1404 |
+
}, f)
|
| 1405 |
+
print(f"✅ Saved sharded safetensors checkpoint: {out_dir} "
|
| 1406 |
+
f"({n_shards} shards, {total_size / 1e9:.2f} GB)")
|
| 1407 |
+
return out_dir
|
| 1408 |
+
|
| 1409 |
+
@classmethod
|
| 1410 |
+
def load_checkpoint_safetensors(cls, ckpt_dir: str, dtype=torch.bfloat16,
|
| 1411 |
+
include_gate_proj: bool = False, freeze_router: bool = True):
|
| 1412 |
+
""
|
| 1413 |
+
with open(os.path.join(ckpt_dir, "checkpoint_meta.json")) as f:
|
| 1414 |
+
meta = json.load(f)
|
| 1415 |
+
arch = FlashArch.from_dict(meta["arch"])
|
| 1416 |
+
model = cls.build_empty(arch)
|
| 1417 |
+
|
| 1418 |
+
patched_names = meta.get("patched_linear_names", [])
|
| 1419 |
+
if patched_names:
|
| 1420 |
+
for name in patched_names:
|
| 1421 |
+
parent_name, _, leaf = name.rpartition(".")
|
| 1422 |
+
parent = model.get_submodule(parent_name) if parent_name else model
|
| 1423 |
+
old = getattr(parent, leaf)
|
| 1424 |
+
if not isinstance(old, BitLinear):
|
| 1425 |
+
setattr(parent, leaf, BitLinear.from_linear(old))
|
| 1426 |
+
else:
|
| 1427 |
+
apply_bitlinear_patch(model, include_gate_proj=include_gate_proj)
|
| 1428 |
+
|
| 1429 |
+
with open(os.path.join(ckpt_dir, "checkpoint.index.json")) as f:
|
| 1430 |
+
index = json.load(f)
|
| 1431 |
+
by_shard = {}
|
| 1432 |
+
for k, shard in index["weight_map"].items():
|
| 1433 |
+
by_shard.setdefault(shard, []).append(k)
|
| 1434 |
+
|
| 1435 |
+
sd = {}
|
| 1436 |
+
n_shards = len(by_shard)
|
| 1437 |
+
for i, (shard, keys) in enumerate(sorted(by_shard.items())):
|
| 1438 |
+
sd.update(mmap_safetensors(os.path.join(ckpt_dir, shard), keys=set(keys)))
|
| 1439 |
+
if (i + 1) % 10 == 0 or i + 1 == n_shards:
|
| 1440 |
+
print(f" ... {i + 1}/{n_shards} shards mapped, {len(sd)} tensors")
|
| 1441 |
+
|
| 1442 |
+
has_mtp = any(k.startswith("mtp.") for k in sd)
|
| 1443 |
+
model.load_state_dict(sd, strict=False, assign=True)
|
| 1444 |
+
model.reset_derived_buffers("cpu")
|
| 1445 |
+
for m in model.modules():
|
| 1446 |
+
if isinstance(m, BitLinear) and m.lambda_.is_meta:
|
| 1447 |
+
m.lambda_ = torch.zeros(())
|
| 1448 |
+
model.assert_materialized(ignore_prefixes=() if has_mtp else ("mtp.",))
|
| 1449 |
+
model = model.to(dtype)
|
| 1450 |
+
|
| 1451 |
+
if not has_mtp:
|
| 1452 |
+
print("⚠️ This checkpoint has NO mtp.* weights. Before exporting, call "
|
| 1453 |
+
"model.load_mtp_from_hf_dir('<original HF checkpoint folder>').")
|
| 1454 |
+
|
| 1455 |
+
lam = meta.get("lambda", 0.0)
|
| 1456 |
+
for m in model.modules():
|
| 1457 |
+
if isinstance(m, BitLinear):
|
| 1458 |
+
m.set_lambda(lam)
|
| 1459 |
+
n_frozen = len(freeze_routers(model)) if freeze_router else 0
|
| 1460 |
+
print(f"✅ Loaded (safetensors, lazy mmap). global_step={meta.get('global_step', 0)}, "
|
| 1461 |
+
f"lambda={lam}, routers frozen: {n_frozen}")
|
| 1462 |
+
return model
|
| 1463 |
+
|
| 1464 |
+
@classmethod
|
| 1465 |
+
def load_checkpoint_auto(cls, path: str, dtype=torch.bfloat16, **kwargs):
|
| 1466 |
+
""
|
| 1467 |
+
if os.path.isdir(path) or os.path.exists(os.path.join(path, "checkpoint.index.json")):
|
| 1468 |
+
return cls.load_checkpoint_safetensors(path, dtype=dtype, **kwargs)
|
| 1469 |
+
if os.path.isfile(path):
|
| 1470 |
+
return cls.load_checkpoint(path, dtype=dtype, **kwargs)
|
| 1471 |
+
raise FileNotFoundError(f"Neither a checkpoint directory nor a file was found: {path}")
|
| 1472 |
+
|
| 1473 |
+
TINY_ARCH = FlashArch(
|
| 1474 |
+
hidden_size=64, vocab_size=512, num_layers=4,
|
| 1475 |
+
linear_num_key_heads=2, linear_key_head_dim=16, linear_num_value_heads=4, linear_value_head_dim=16,
|
| 1476 |
+
num_attention_heads=4, num_key_value_heads=2, head_dim=32, mrope_section=(2, 1, 1),
|
| 1477 |
+
indexer_n_heads=2, indexer_head_dim=16, indexer_budget=8, indexer_compress_ratio=4,
|
| 1478 |
+
num_experts=8, num_experts_per_tok=2, moe_intermediate_size=32, shared_expert_intermediate_size=32,
|
| 1479 |
+
hc_count=4, hc_lowrank=8,
|
| 1480 |
+
ple_layer_ids=(2,), ple_embed_dim=64, heads_per_ngram=2, ngram_vocab_size_base=1000,
|
| 1481 |
+
make_ngram_vocab_size_divisible_by=8, split_ngram_parts=4, eos_token_id=1,
|
| 1482 |
+
)
|
| 1483 |
+
|
| 1484 |
+
def _count(model, pred):
|
| 1485 |
+
return sum(p.numel() for n, p in model.named_parameters() if pred(n))
|
| 1486 |
+
|
| 1487 |
+
if __name__ == "__main__":
|
| 1488 |
+
print(ARCH_SUMMARY)
|
| 1489 |
+
print(KEPT_FULL_PRECISION)
|
| 1490 |
+
|
| 1491 |
+
print("🔧 Skeleton of the full model on meta (no weights allocated) ...")
|
| 1492 |
+
model = JiRackQwen38FlashForCausalLM.build_empty()
|
| 1493 |
+
total = _count(model, lambda n: True)
|
| 1494 |
+
ngram = _count(model, lambda n: "ngram_embedding" in n)
|
| 1495 |
+
mtp = _count(model, lambda n: n.startswith("mtp."))
|
| 1496 |
+
experts = _count(model, lambda n: ".mlp.experts." in n and not n.startswith("mtp."))
|
| 1497 |
+
print(f" total: {total/1e9:.2f}B | n-gram: {ngram/1e9:.2f}B | MTP: {mtp/1e9:.2f}B | "
|
| 1498 |
+
f"trunk without n-gram: {(total-ngram-mtp)/1e9:.2f}B (of which routed experts {experts/1e9:.2f}B)")
|
| 1499 |
+
patched = apply_bitlinear_patch(model)
|
| 1500 |
+
print(f"✅ Replaced with BitLinear: {len(patched)} modules; examples: {patched[:3]} ... {patched[-2:]}")
|
| 1501 |
+
|
| 1502 |
+
print("\n🧪 Tiny model (random weights) -- checking shapes and the forward pass ...")
|
| 1503 |
+
torch.manual_seed(0)
|
| 1504 |
+
tiny = JiRackQwen38FlashForCausalLM(TINY_ARCH).eval()
|
| 1505 |
+
for n, p in tiny.named_parameters():
|
| 1506 |
+
p.data.add_(torch.randn_like(p) * 0.02)
|
| 1507 |
+
ids = torch.randint(0, TINY_ARCH.vocab_size, (2, 40))
|
| 1508 |
+
with torch.no_grad():
|
| 1509 |
+
logits = tiny(ids)
|
| 1510 |
+
l1, l2 = tiny.mtp_forward(ids)
|
| 1511 |
+
print(f" logits {tuple(logits.shape)}, mtp {tuple(l2.shape)}, "
|
| 1512 |
+
f"|Δ(forward vs mtp_forward.t1)|={float((logits - l1).abs().max()):.2e}")
|
JiRackDeltaNet_180b_Q3_K_M.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d8e9f996c78e33d6f8cffaf0473e5f09aa2c0f6f384cf98fb6de45d166f6be1f
|
| 3 |
+
size 94137092576
|
JiRackDeltaNet_180b_Q4_K_M.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:80986ff36ed56aaecbffb04b8c1ee7cb5d06f7ba899038a4b79b7179a37746e5
|
| 3 |
+
size 119150722016
|
LICENSE-CODE
ADDED
|
@@ -0,0 +1,26 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
MIT License
|
| 2 |
+
|
| 3 |
+
Copyright (c) 2025-2026 Konstantin Vladimirovich Grabko, CMS Manhattan JiRack Technology
|
| 4 |
+
|
| 5 |
+
This license applies to the JiRack source code in this repository
|
| 6 |
+
(JiRackDeltaNet_180b.py and the conversion, QAT training, export and test
|
| 7 |
+
scripts). It does NOT apply to the model weights, which are a derivative of
|
| 8 |
+
Qwen/Qwen3.8-Flash-Next and remain under the Qwen Community License 1.0.
|
| 9 |
+
|
| 10 |
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
| 11 |
+
of this software and associated documentation files (the "Software"), to deal
|
| 12 |
+
in the Software without restriction, including without limitation the rights
|
| 13 |
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
| 14 |
+
copies of the Software, and to permit persons to whom the Software is
|
| 15 |
+
furnished to do so, subject to the following conditions:
|
| 16 |
+
|
| 17 |
+
The above copyright notice and this permission notice shall be included in all
|
| 18 |
+
copies or substantial portions of the Software.
|
| 19 |
+
|
| 20 |
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
| 21 |
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
| 22 |
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
| 23 |
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
| 24 |
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
| 25 |
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
| 26 |
+
SOFTWARE.
|
LICENSE-QWEN
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Qwen Community License 1.0
|
| 2 |
+
|
| 3 |
+
Copyright (c) 2026 Qwen
|
| 4 |
+
|
| 5 |
+
Permission is hereby granted, free of charge, to any person obtaining a copy of this software, including the model weights, parameters, configuration files, inference code and associated documentation files (collectively, the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, sell, deploy, host, fine-tune, and create derivative works from (collectively, "Use" or "Using") copies of the Software; and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
|
| 6 |
+
|
| 7 |
+
1. The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. If the Software (or any derivative works thereof) is Used for any of the licensee's commercial products or services that have more than 100,000,000 monthly active users or US$ 20,000,000 (or equivalent in other currencies) monthly revenue, respective model name must be prominently displayed on the user interface of such product or service; and,
|
| 8 |
+
|
| 9 |
+
2. If the licensee or any of its affiliates conducts a Model as a Service or AI Work Assistant business, the licensee shall obtain a separate license from Qwen before Using the Software or its derivative works for any commercial purpose. The foregoing requirement shall not apply to the licensee's internal Use of the Software, provided that such Use does not make the Software, its outputs, or its underlying model capabilities available to any third party.
|
| 10 |
+
|
| 11 |
+
"Model as a Service" means giving a third party access to language model inference or fine-tuning (e.g., via API or a hosted endpoint) in a manner that allows such third parties to exercise meaningful control over the inputs, parameters, or training data. This does not include the mere relaying of requests to models hosted by other third parties.
|
| 12 |
+
“AI Work Assistant” means an independent AI-powered product primarily designed for AI-assisted coding or office productivity (e.g., Qoder and QwenWork). It does not include: (a) a single-purpose AI tool (such as an AI translation tool); (b) an AI assistant primarily designed for a domain other than coding or office productivity (such as Taobao AI Shopping Assistant or AMap AI Chat); or (c) an AI assistant that is a feature of a product whose primary purpose is not AI-assisted coding or office productivity.
|
| 13 |
+
|
| 14 |
+
THE SOFTWARE AND ANY OUTPUT AND RESULTS THEREFROM ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL QWEN, ITS AFFILIATES OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. THE USE OF THE SOFTWARE MUST COMPLY WITH APPLICABLE LAWS AND REGULATIONS, AND MUST NOT INFRINGE THE INTELLECTUAL PROPERTY RIGHTS OF ANY THIRD PARTY.
|
| 15 |
+
|
| 16 |
+
For any questions regarding this license, please contact model-business@notice.qwencloud.com.
|
README.md
ADDED
|
@@ -0,0 +1,299 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language:
|
| 3 |
+
- en
|
| 4 |
+
- zh
|
| 5 |
+
- ja
|
| 6 |
+
- ko
|
| 7 |
+
- fr
|
| 8 |
+
- es
|
| 9 |
+
- pt
|
| 10 |
+
- de
|
| 11 |
+
- it
|
| 12 |
+
- ru
|
| 13 |
+
- ar
|
| 14 |
+
- vi
|
| 15 |
+
- th
|
| 16 |
+
tags:
|
| 17 |
+
- text-generation
|
| 18 |
+
- moe
|
| 19 |
+
- mixture-of-experts
|
| 20 |
+
- deltanet
|
| 21 |
+
- sparse-attention
|
| 22 |
+
- hyper-connections
|
| 23 |
+
- qwen3.8
|
| 24 |
+
- qwen3.8-flash-next
|
| 25 |
+
- gguf
|
| 26 |
+
- cpu
|
| 27 |
+
- efficient
|
| 28 |
+
- ternary
|
| 29 |
+
- bitnet
|
| 30 |
+
- jirack
|
| 31 |
+
- routing
|
| 32 |
+
- tool-call
|
| 33 |
+
- robotics
|
| 34 |
+
- ternary-weights
|
| 35 |
+
- ternary-transformer
|
| 36 |
+
- quantization
|
| 37 |
+
- qat
|
| 38 |
+
|
| 39 |
+
pipeline_tag: text-generation
|
| 40 |
+
license: other
|
| 41 |
+
license_name: qwen-community-1.0
|
| 42 |
+
license_link: https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/LICENSE
|
| 43 |
+
---
|
| 44 |
+
|
| 45 |
+
# JiRack Ternary Architecture
|
| 46 |
+
- A 180B-parameter Mixture-of-Experts model with only **6B parameters active per token**, which makes high-quality CPU inference realistic at this scale
|
| 47 |
+
- Migrated to the JiRack ternary architecture (BitLinear, per-tensor absmean ternary weights) with a lambda-warmup QAT pipeline, targeting TQ2_0 CPU inference on llama.cpp and Ollama
|
| 48 |
+
- Planned with QAT and JiRack fine-tuning: robotics, routing, coding and advanced tool calling via the JiRack tool-call workflow
|
| 49 |
+
- The Qwen3.8-Flash-Next base model understands images and video; the JiRack build is currently text-only (see [Current status](#current-status))
|
| 50 |
+
|
| 51 |
+
**CMS Manhattan is a member of the NVIDIA Inception Program.**
|
| 52 |
+
|
| 53 |
+
# JiRack DeltaNet 180B (CPU)
|
| 54 |
+
A 180B MoE model built on the Qwen3.8-Flash-Next architecture: Gated DeltaNet linear attention interleaved with Qwen Sparse Attention (QSA), 512 experts with top-10 routing plus a shared expert, gated residual (hyper-connection) streams, and a 51B-parameter hashed n-gram embedding table. Only ~6B parameters are computed per token, so throughput on CPU is closer to a 6B dense model while quality is that of a much larger one.
|
| 55 |
+
- JiRack is a cloud-ready model that helps save money on cloud infrastructure. It can be used as an expert model in RAG deployments, with the ONNX JiRack Java server as an alternative.
|
| 56 |
+
|
| 57 |
+
## Current status
|
| 58 |
+
|
| 59 |
+
| Stage | Status |
|
| 60 |
+
|---|---|
|
| 61 |
+
| Hand-rolled JiRack port of the architecture (`JiRackDeltaNet_180b.py`) | ✅ done |
|
| 62 |
+
| Conversion from Qwen/Qwen3.8-Flash-Next and numerical verification against the reference | ✅ done — next-token prediction matches the reference on 100% of positions; logit differences are within the model's own bf16 noise floor |
|
| 63 |
+
| BitLinear patch (74,028 layers) at lambda = 0 | ✅ done — bit-exact passthrough (max logit difference 0.000000) |
|
| 64 |
+
| Export to Hugging Face layout and llama.cpp GGUF (BF16, Q4_K_M, Q3_K_M) | ✅ done — export verified key-for-key against the original; GGUF files published (see [GGUF files](#gguf-files)) |
|
| 65 |
+
| Ternary QAT (lambda warmup 0 → 1) | ⏳ not started — requires a machine with ≥ 640 GB RAM |
|
| 66 |
+
| TQ2_0 GGUF (after QAT), Ollama and Docker builds | ⏳ not yet published |
|
| 67 |
+
|
| 68 |
+
**Important:** until QAT is complete, the JiRack 180B weights are numerically identical to the base model (lambda = 0 means BitLinear is a pure passthrough). The ternary benefits described on this card apply after QAT.
|
| 69 |
+
|
| 70 |
+
# JiRack service options
|
| 71 |
+
- If you need custom compression or fine-tuning, please write to me and I'll perform QAT from your dataset, tailored specifically to your task.
|
| 72 |
+
- Plus double QAT via ONNX QAT.
|
| 73 |
+
- Adapt train process to avoid catastrophic forgetting with NDA
|
| 74 |
+
- Adapt train process to avoid fast plateau in training with NDA
|
| 75 |
+
- MoE-aware QAT for mixture-of-experts models
|
| 76 |
+
- Adapts to agentic or instruct models for tool calling, using the JiRack tokenizer to enable high-quality tool calling — built as a domain-specific tool expert.
|
| 77 |
+
- Deployment and scale
|
| 78 |
+
|
| 79 |
+
---
|
| 80 |
+
|
| 81 |
+
## DEMO Chat
|
| 82 |
+
- https://huggingface.co/spaces/CMSManhattan/JiRackDeltaNet-180B-demo
|
| 83 |
+
|
| 84 |
+
## JiRack Coding Agent IDE
|
| 85 |
+
- An agentic coding IDE for JiRack models, running them through Ollama on a home PC
|
| 86 |
+
- An alternative to agentic coding IDEs such as Cursor, Windsurf or Devin, with a safer workflow: it asks you to review and apply every change.
|
| 87 |
+
- Web site https://www.jirack.com
|
| 88 |
+
- Eclipse plugin site https://marketplace.eclipse.org/content/jirack-coding-agent
|
| 89 |
+
- JetBrains plugin site https://plugins.jetbrains.com/plugin/35073-jirack-agent
|
| 90 |
+
- VS Code plugin site https://marketplace.visualstudio.com/items?itemName=JiRackautomationCodingAgentIDE.jirack-agent
|
| 91 |
+
|
| 92 |
+
## JiRack Coding Agent IDE files
|
| 93 |
+
- Eclipse final release https://huggingface.co/CMSManhattan/JiRackDeltaNet_27b/resolve/main/jirack_ide_final.zip
|
| 94 |
+
- Microsoft VS Code plugin https://huggingface.co/spaces/CMSManhattan/README/blob/main/jirack-agent-0.9.0.vsix
|
| 95 |
+
- JetBrains plugin https://huggingface.co/spaces/CMSManhattan/README/resolve/main/jirack-jetbrains-0.11.0.zip
|
| 96 |
+
- JiRack Desktop UI https://huggingface.co/spaces/CMSManhattan/README/resolve/main/windows_v11.zip
|
| 97 |
+
|
| 98 |
+
---
|
| 99 |
+
|
| 100 |
+
# Ollama support
|
| 101 |
+
- 180B Ollama builds are planned after QAT. Ollama needs a llama.cpp version that supports the `qwen4exp` architecture.
|
| 102 |
+
- Qwen3.8-Flash-Next thinks by default (`<think>...</think>`); JiRack builds will ship with reasoning disabled by default, as for JiRack 27B.
|
| 103 |
+
- Follow https://ollama.com/cmsmanhattan for releases.
|
| 104 |
+
|
| 105 |
+
# Spring Boot AI tool calls examples for JiRack DeltaNet series
|
| 106 |
+
- Tool call library on java for Enterprise https://github.com/alibaba/spring-ai-alibaba
|
| 107 |
+
|
| 108 |
+
# GoEx AI tool calls examples for JiRack DeltaNet series
|
| 109 |
+
- Tool call library on python https://github.com/ShishirPatil/gorilla
|
| 110 |
+
|
| 111 |
+
# JiRack RoboTech
|
| 112 |
+
- Advanced Tokenizer with Robotics & Routing & Tool calls Tokenizer and other
|
| 113 |
+
- [CMSManhattan/JiRackDeltaNetTokenizer](https://huggingface.co/CMSManhattan/JiRackDeltaNetTokenizer)
|
| 114 |
+
- The 180B build currently uses the original Qwen3.8 tokenizer (vocabulary 248,320).
|
| 115 |
+
|
| 116 |
+
## GGUF files
|
| 117 |
+
|
| 118 |
+
Published in this repository. Before QAT all of them are quantizations of the unchanged base-model weights.
|
| 119 |
+
|
| 120 |
+
| File | Size | RAM (file + context) | Description |
|
| 121 |
+
|------|------|----------------------|-------------|
|
| 122 |
+
| `JiRackDeltaNet_180b-00001-of-00007.gguf` … `-00007-of-00007.gguf` | 354.1 GB (7 parts) | 384 GB+ | BF16, full-precision reference; load the first part, llama.cpp reads the rest |
|
| 123 |
+
| `JiRackDeltaNet_180b_Q4_K_M.gguf` | 119.2 GB | 160–192 GB | Recommended balance |
|
| 124 |
+
| `JiRackDeltaNet_180b_Q3_K_M.gguf` | 94.1 GB | 128 GB | Smaller, good quality / size trade-off |
|
| 125 |
+
|
| 126 |
+
Planned: TQ2_0 after QAT (~65–95 GB, estimate; depends on how the 51B n-gram table is stored).
|
| 127 |
+
|
| 128 |
+
### Run with llama.cpp
|
| 129 |
+
Needs a llama.cpp build with the `qwen4exp` architecture (the one used to make these files).
|
| 130 |
+
```bash
|
| 131 |
+
llama-server -m JiRackDeltaNet_180b_Q4_K_M.gguf -c 8192 --jinja --reasoning-budget 0 --port 8080
|
| 132 |
+
```
|
| 133 |
+
`--reasoning-budget 0` turns the base model's default thinking off. The server is OpenAI-compatible (`/v1/chat/completions`).
|
| 134 |
+
|
| 135 |
+
## Quick Start
|
| 136 |
+
|
| 137 |
+
## Live chat with other models
|
| 138 |
+
|
| 139 |
+
- https://huggingface.co/spaces/CMSManhattan/JiRack-16B-Chat
|
| 140 |
+
- https://huggingface.co/spaces/CMSManhattan/JiRack-8B-Chat
|
| 141 |
+
- https://huggingface.co/spaces/CMSManhattan/JiRack-2B-Chat
|
| 142 |
+
|
| 143 |
+
### Run the JiRack checkpoint directly (PyTorch, CPU)
|
| 144 |
+
The JiRack checkpoint is a directory of safetensors shards that is memory-mapped lazily, so it runs even when the model is larger than RAM (slower: pages are read from disk as experts are used).
|
| 145 |
+
```bash
|
| 146 |
+
python chat_jirack_180b.py # interactive chat
|
| 147 |
+
```
|
| 148 |
+
|
| 149 |
+
### Build a GGUF
|
| 150 |
+
The GGUF files listed above are ready to use. For other quantizations (or TQ2_0 after QAT), contact us.
|
| 151 |
+
|
| 152 |
+
### Docker
|
| 153 |
+
- 180B Docker images with the JiRack UI will be provided after QAT, or by request.
|
| 154 |
+
|
| 155 |
+
### Ollama (planned, after QAT)
|
| 156 |
+
- ollama run cmsmanhattan/JiRackDeltaNet_180b-q4
|
| 157 |
+
- ollama run cmsmanhattan/JiRackDeltaNet_180b-q3
|
| 158 |
+
|
| 159 |
+
## Ollama quick start (planned)
|
| 160 |
+
|
| 161 |
+
These commands will work once the 180B builds are published at https://ollama.com/cmsmanhattan; until then use the GGUF files with llama.cpp (above).
|
| 162 |
+
|
| 163 |
+
### Run with Ollama (reasoning off by default)
|
| 164 |
+
```bash
|
| 165 |
+
ollama pull JiRackDeltaNet_180b-q4
|
| 166 |
+
ollama run JiRackDeltaNet_180b-q4 "What is the capital of France?"
|
| 167 |
+
```
|
| 168 |
+
|
| 169 |
+
|
| 170 |
+
## Recommended sampling parameters
|
| 171 |
+
From the base model card:
|
| 172 |
+
- **Thinking mode:** `temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0`
|
| 173 |
+
- **Instruct (non-thinking) mode:** `temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0`
|
| 174 |
+
|
| 175 |
+
## Access the UI
|
| 176 |
+
Once a JiRack container is running, open your browser at `http://localhost:7869`. This opens the JiRack UI — a clean web interface. The port can be changed from the **Settings** panel.
|
| 177 |
+
|
| 178 |
+
## Licensing
|
| 179 |
+
|
| 180 |
+
This repository is dual-licensed: the model weights and the JiRack code are licensed separately.
|
| 181 |
+
|
| 182 |
+
| Component | License |
|
| 183 |
+
|---|---|
|
| 184 |
+
| **Model weights** (safetensors, GGUF) — a derivative of [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) | **Qwen Community License 1.0** — [LICENSE](https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/LICENSE), Copyright 2026 Qwen |
|
| 185 |
+
| **JiRack code and tools** (`JiRackDeltaNet_180b.py`, `chat_jirack_180b.py`) | **MIT License** — © 2025-2026 Konstantin Vladimirovich Grabko, CMS Manhattan JiRack Technology |
|
| 186 |
+
| **JiRack Docker images with UI, pre-built Ollama quantizations, JiRack UI clients** | Commercial license (see below) |
|
| 187 |
+
|
| 188 |
+
- **Weights:** the Qwen Community License continues to apply to the weights, including the JiRack-converted and quantized versions. Keep the Qwen copyright and license notice with all copies. Commercial Model-as-a-Service and AI work-assistant (coding / office) products built on these weights require a separate license from Qwen — review the license terms before such use.
|
| 189 |
+
- **Code:** the MIT License covers the JiRack source code only. It does not change the license of the weights.
|
| 190 |
+
- The Docker image with UI and pre-built Ollama quantizations are separate commercial products.
|
| 191 |
+
- All JiRack UI clients are provided under a commercial license. The UI clients can be used for free together with the official JiRack Docker containers, as long as they are not redistributed separately.
|
| 192 |
+
|
| 193 |
+
For commercial licensing, cluster deployment, or enterprise use of JiRack models, please contact us.
|
| 194 |
+
- **JiRack MS Windows 11 Desktop Client (with Ollama API):**
|
| 195 |
+
https://huggingface.co/kgrabko/JiRackTernary_1b/resolve/main/jirack-chat.zip
|
| 196 |
+
- **Live email chat with the model:** support@cmsmanhattan.com
|
| 197 |
+
|
| 198 |
+
## Hardware Recommendations
|
| 199 |
+
|
| 200 |
+
### Recommended hardware for JiRack DeltaNet 180B
|
| 201 |
+
| Use Case | CPU | RAM | Recommended Quant | Notes |
|
| 202 |
+
|----------|-----|-----|-------------------|-------|
|
| 203 |
+
| Recommended | High-core server CPU (Xeon / EPYC) | 160–192 GB | Q4_K_M (119.2 GB), or TQ2_0 after QAT | Only ~6B active parameters per token |
|
| 204 |
+
| High Performance | Dual-socket server | 384 GB+ | BF16 (354.1 GB) | Reference quality |
|
| 205 |
+
| Low Memory | Modern 16+ core CPU + fast NVMe | 128 GB | Q3_K_M (94.1 GB) | Usable |
|
| 206 |
+
| PyTorch checkpoint | Server CPU + NVMe | 256 GB+ | BF16 (lazy mmap) | Works below model size, slower |
|
| 207 |
+
|
| 208 |
+
## Important Memory Notes
|
| 209 |
+
- The whole model must be resident (or memory-mapped) even though only ~6B parameters are computed per token: the router can pick any of the 512 experts in every layer.
|
| 210 |
+
- 36 of the 48 layers use Gated DeltaNet with a fixed-size recurrent state, so long contexts need far less KV cache than a full-attention model of this size.
|
| 211 |
+
- Leave headroom for runtime buffers and the attention KV cache of the 12 full-attention layers.
|
| 212 |
+
|
| 213 |
+
## Architecture Notes
|
| 214 |
+
- **Qwen3.8-Flash-Next architecture** (`qwen4_exp` in Hugging Face, `qwen4exp` in llama.cpp)
|
| 215 |
+
- **Parameters:** 125B core + 51B n-gram embedding + 4B MTP (≈180B stored), **6B activated** per token
|
| 216 |
+
- **Layers:** 48 = 12 × (3 × Gated DeltaNet → MoE, 1 × Gated Attention with QSA → MoE)
|
| 217 |
+
- **Hidden size** 2560, carried as **4 parallel hyper-connection streams** (gated residual replaces the usual pre/post norms)
|
| 218 |
+
- **MoE:** 512 experts (intermediate 640), top-10 routing, plus 1 shared expert with a sigmoid gate
|
| 219 |
+
- **Gated DeltaNet:** 16 key heads × 128, 48 value heads × 128, conv kernel 4, sigmoid output gate
|
| 220 |
+
- **Gated Attention:** 24 query heads, 2 KV heads, head dim 256, partial rotary 0.25, interleaved mRoPE (11/11/10), θ = 10,000,000
|
| 221 |
+
- **QSA (Qwen Sparse Attention):** keys pooled in blocks of 4 tokens; each query attends to the top 2048 tokens' blocks plus the local tail
|
| 222 |
+
- **PLE n-gram embedding** on layer 2: hashed bi- and trigrams, 16 heads, 320M-row table (51B parameters) in 128 shards
|
| 223 |
+
- **MTP head:** 1 layer for speculative decoding
|
| 224 |
+
- RMSNorm ε = 1e-6, vocabulary 248,320, context 262,144 tokens natively (extensible to 1M in the base model)
|
| 225 |
+
|
| 226 |
+
### JiRack ternary scheme
|
| 227 |
+
- **Ternarized (BitLinear):** DeltaNet `in_proj_qkv` / `in_proj_z` / `out_proj`, attention `q/k/v/o_proj`, all routed experts and the shared expert — 74,028 layers
|
| 228 |
+
- **Always full precision:** MoE routers and the shared-expert gate (never quantized), hyper-connections, all norms, conv1d, `A_log` / `dt_bias`, QSA indexer, PLE projections and n-gram table, embeddings, `lm_head`, MTP head
|
| 229 |
+
- **Weights:** per-tensor absmean ternary `{-γ, 0, +γ}`; **activations** during training: per-token int8
|
| 230 |
+
- **QAT:** continuous lambda warmup from full precision (λ = 0) to fully ternary (λ = 1) with a straight-through estimator
|
| 231 |
+
|
| 232 |
+
## Benchmarks
|
| 233 |
+
JiRack DeltaNet 180B is built on Qwen3.8-Flash-Next. The tables below reproduce the published base-model results from [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) for reference — they reflect the upstream base model's capabilities, not JiRack-specific QAT or quantization results.
|
| 234 |
+
|
| 235 |
+
### Language
|
| 236 |
+
|
| 237 |
+
| | Qwen3.8-Flash-Next | Qwen3.8-27B | Qwen3.7-Plus | DeepSeek-V4-Flash-0731 | Claude-Opus-4.6 (Max) |
|
| 238 |
+
|----------------------------------------------------------|--------------------|-------------|--------------|------------------------|-----------------------|
|
| 239 |
+
| # Params | 125B | 27B | 397B | 284B | -- |
|
| 240 |
+
| # Activated params | 6B | 27B | 17B | 13B | -- |
|
| 241 |
+
| # N-gram embedding params | 51B | -- | -- | -- | -- |
|
| 242 |
+
| **Coding** | | | | | |
|
| 243 |
+
| Agentic coding — DeepSWE 1.1 | **58.7** | 42.2 | 16.5 | 54.4 | -- |
|
| 244 |
+
| Agentic coding — SWE-bench Pro | **62.5** | 61.7 | 55.8 | 56.0 | 53.4 |
|
| 245 |
+
| Multilingual software engineering — SWE-bench Multilingual | **81.0** | 73.8 | 75.8 | -- | 77.5 |
|
| 246 |
+
| Repo-level code generation — NL2Repo-Bench | 48.1 | 42.3 | 41.1 | **54.2** | 47.6 |
|
| 247 |
+
| **Agent** | | | | | |
|
| 248 |
+
| Long-horizon office work — CoWorkBench | **73.9** | 70.7 | 65.1 | 45.1 | 68.2 |
|
| 249 |
+
| Professional job tasks — JobBench | **55.7** | 33.4 | 27.6 | 41.3 | 36.6 |
|
| 250 |
+
| Frontier agentic tasks — Agents' Last Exam (Pass@1 / Score) | 24.3 / **51.2** | 20.4 / 42.9 | 13.2 / 33.6 | **25.2** / -- | -- |
|
| 251 |
+
| Real-world tool use — Toolathlon Verified (Pass@1) | **73.5** | 67.1 | 50.6 | 70.3 | -- |
|
| 252 |
+
| **General** | | | | | |
|
| 253 |
+
| Instruction following — IFBench | **81.3** | 79.5 | 79.1 | 79.2 | 62.5 |
|
| 254 |
+
| Scientific reasoning — GPQA Diamond | **91.7** | 89.2 | 90.3 | 90.8 | 91.3 |
|
| 255 |
+
| Multidisciplinary reasoning — HLE | 35.9 | 30.8 | 34.7 | 33.8 | **40.0** |
|
| 256 |
+
| Competitive coding — LiveCodeBench v6 | **91.9** | 90.3 | 89.6 | 90.6 | 88.8 |
|
| 257 |
+
|
| 258 |
+
### Vision-Language (base model)
|
| 259 |
+
|
| 260 |
+
| | Qwen3.8-Flash-Next | Qwen3.8-27B | Qwen3.7-Plus | Claude-Opus-4.6 (Max) |
|
| 261 |
+
|----------------------------------------------------------|--------------------|-------------|--------------|-----------------------|
|
| 262 |
+
| **Agentic Multimodal Intelligence** | | | | |
|
| 263 |
+
| Multimodal tool use — ClawEval-MM (Pass@3 / Average) | **64.4 / 60.4** | 57.4 / 56.9 | 57.4 / 60.1 | 52.5 / 54.7 |
|
| 264 |
+
| Application recreation — RecreationBench | **49.9** | 47.1 | 30.2 | -- |
|
| 265 |
+
| Mobile use — AndroidWorld | **84.5** | 81.9 | 81.0 | 62.0 |
|
| 266 |
+
| Computer use — OSWorld 2.0 (Binary / Partial) | **19.4 / 52.3** | 19.4 / 48.0 | 2.8 / 21.5 | -- |
|
| 267 |
+
| Visual web development — Vision2Web | **64.0** | 62.9 | 42.1 | -- |
|
| 268 |
+
| **General Multimodal Intelligence** | | | | |
|
| 269 |
+
| Embodied intelligence — ERQA | **72.3** | 65.5 | 69.8 | 40.8 |
|
| 270 |
+
| Long video understanding — LVBench | **76.6** | 72.4 | 76.2 | 63.0 |
|
| 271 |
+
| Real-world perception — RealWorldQA | **88.5** | 85.9 | 86.9 | 73.9 |
|
| 272 |
+
| Visual math — MathVision (w/o CI / w/ CI) | **90.6 / 95.7** | 90.0 / 94.6 | 90.3 / 88.7 | 65.5 / -- |
|
| 273 |
+
| Scientific chart analysis — CharXiv (RQ) (w/o CI / w/ CI) | 84.6 / **90.6** | 83.7 / 90.2 | **85.8** / 85.9 | 66.0 / -- |
|
| 274 |
+
|
| 275 |
+
*Source: [Qwen/Qwen3.8-Flash-Next model card](https://huggingface.co/Qwen/Qwen3.8-Flash-Next). Best result in each row is bolded. Empty cells (--) indicate results not available or not applicable. See the source card for evaluation harnesses, settings and footnotes.*
|
| 276 |
+
|
| 277 |
+
## Citation
|
| 278 |
+
```bibtex
|
| 279 |
+
@techreport{qwen2026design,
|
| 280 |
+
title = {On the Design of {Qwen3.8-Next} Architecture: Evaluation, Efficiency, and Training Stability},
|
| 281 |
+
author = {{Qwen Team}},
|
| 282 |
+
institution = {Alibaba Group},
|
| 283 |
+
month = {August},
|
| 284 |
+
year = {2026}
|
| 285 |
+
}
|
| 286 |
+
```
|
| 287 |
+
|
| 288 |
+
## 📧 Contact & Licensing
|
| 289 |
+
For joint venture opportunities, hardware integration, or licensing inquiries:
|
| 290 |
+
- **Email:** grabko@cmsmanhattan.com
|
| 291 |
+
- **Phone:** +1 (516) 777-0945
|
| 292 |
+
- **Location:** New York, USA
|
| 293 |
+
|
| 294 |
+
## License
|
| 295 |
+
- **Model weights:** Qwen Community License 1.0 — see [LICENSE](https://huggingface.co/Qwen/Qwen3.8-Flash-Next/blob/main/LICENSE). Copyright 2026 Qwen.
|
| 296 |
+
- **JiRack code and tools:** MIT License — see `LICENSE-CODE`. © 2025-2026 Konstantin Vladimirovich Grabko, CMS Manhattan JiRack Technology.
|
| 297 |
+
|
| 298 |
+
|
| 299 |
+

|
chat_jirack_180b.py
ADDED
|
@@ -0,0 +1,65 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python
|
| 2 |
+
# =============================================================================
|
| 3 |
+
# chat_jirack_180b.py - JiRack DeltaNet 180B
|
| 4 |
+
# Copyright (c) 2025-2026 Konstantin Vladimirovich Grabko (CMSManhattan).
|
| 5 |
+
# MIT License, see LICENSE-CODE. The model weights are licensed separately, see README.md.
|
| 6 |
+
# =============================================================================
|
| 7 |
+
|
| 8 |
+
import torch
|
| 9 |
+
from transformers import AutoTokenizer
|
| 10 |
+
|
| 11 |
+
try:
|
| 12 |
+
torch.backends.nnpack.set_flags(False)
|
| 13 |
+
except Exception:
|
| 14 |
+
pass
|
| 15 |
+
|
| 16 |
+
from JiRackDeltaNet_180b import JiRackQwen38FlashForCausalLM
|
| 17 |
+
|
| 18 |
+
MODEL_PATH = "."
|
| 19 |
+
TOKENIZER_DIR = "."
|
| 20 |
+
MAX_NEW = 256
|
| 21 |
+
ENABLE_THINKING = False
|
| 22 |
+
|
| 23 |
+
device = "cpu"
|
| 24 |
+
print(f"Loading on {device}...")
|
| 25 |
+
|
| 26 |
+
model = JiRackQwen38FlashForCausalLM.load_checkpoint_auto(MODEL_PATH, dtype=torch.bfloat16)
|
| 27 |
+
model.eval()
|
| 28 |
+
|
| 29 |
+
tokenizer = AutoTokenizer.from_pretrained(TOKENIZER_DIR, trust_remote_code=True)
|
| 30 |
+
stop_ids = {i for i in (tokenizer.eos_token_id,
|
| 31 |
+
tokenizer.convert_tokens_to_ids("<|im_end|>"),
|
| 32 |
+
tokenizer.convert_tokens_to_ids("<|endoftext|>"))
|
| 33 |
+
if isinstance(i, int) and i >= 0}
|
| 34 |
+
|
| 35 |
+
print("Ready! Type your message:")
|
| 36 |
+
history = []
|
| 37 |
+
|
| 38 |
+
while True:
|
| 39 |
+
user_input = input("\nYou: ").strip()
|
| 40 |
+
if not user_input:
|
| 41 |
+
continue
|
| 42 |
+
if user_input.lower() in ["exit", "quit"]:
|
| 43 |
+
break
|
| 44 |
+
|
| 45 |
+
history.append({"role": "user", "content": user_input})
|
| 46 |
+
input_ids = tokenizer.apply_chat_template(
|
| 47 |
+
history, add_generation_prompt=True, return_tensors="pt",
|
| 48 |
+
enable_thinking=ENABLE_THINKING,
|
| 49 |
+
)
|
| 50 |
+
if not torch.is_tensor(input_ids):
|
| 51 |
+
input_ids = input_ids["input_ids"]
|
| 52 |
+
input_ids = input_ids.to(device)
|
| 53 |
+
prompt_len = input_ids.shape[1]
|
| 54 |
+
|
| 55 |
+
with torch.no_grad():
|
| 56 |
+
for _ in range(MAX_NEW):
|
| 57 |
+
logits = model(input_ids)
|
| 58 |
+
next_token = logits[:, -1, :].argmax(dim=-1, keepdim=True)
|
| 59 |
+
input_ids = torch.cat([input_ids, next_token], dim=1)
|
| 60 |
+
if next_token.item() in stop_ids:
|
| 61 |
+
break
|
| 62 |
+
|
| 63 |
+
reply = tokenizer.decode(input_ids[0, prompt_len:], skip_special_tokens=True)
|
| 64 |
+
print(f"\nAssistant: {reply}")
|
| 65 |
+
history.append({"role": "assistant", "content": reply})
|
chat_template.jinja
ADDED
|
@@ -0,0 +1,170 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{%- set image_count = namespace(value=0) %}
|
| 2 |
+
{%- set video_count = namespace(value=0) %}
|
| 3 |
+
{%- macro render_content(content, do_vision_count, is_system_content=false) %}
|
| 4 |
+
{%- if content is string %}
|
| 5 |
+
{{- content }}
|
| 6 |
+
{%- elif content is iterable and content is not mapping %}
|
| 7 |
+
{%- for item in content %}
|
| 8 |
+
{%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
|
| 9 |
+
{%- if is_system_content %}
|
| 10 |
+
{{- raise_exception('System message cannot contain images.') }}
|
| 11 |
+
{%- endif %}
|
| 12 |
+
{%- if do_vision_count %}
|
| 13 |
+
{%- set image_count.value = image_count.value + 1 %}
|
| 14 |
+
{%- endif %}
|
| 15 |
+
{%- if add_vision_id %}
|
| 16 |
+
{{- 'Picture ' ~ image_count.value ~ ': ' }}
|
| 17 |
+
{%- endif %}
|
| 18 |
+
{{- '<|vision_start|><|image_pad|><|vision_end|>' }}
|
| 19 |
+
{%- elif 'video' in item or item.type == 'video' %}
|
| 20 |
+
{%- if is_system_content %}
|
| 21 |
+
{{- raise_exception('System message cannot contain videos.') }}
|
| 22 |
+
{%- endif %}
|
| 23 |
+
{%- if do_vision_count %}
|
| 24 |
+
{%- set video_count.value = video_count.value + 1 %}
|
| 25 |
+
{%- endif %}
|
| 26 |
+
{%- if add_vision_id %}
|
| 27 |
+
{{- 'Video ' ~ video_count.value ~ ': ' }}
|
| 28 |
+
{%- endif %}
|
| 29 |
+
{{- '<|vision_start|><|video_pad|><|vision_end|>' }}
|
| 30 |
+
{%- elif 'text' in item %}
|
| 31 |
+
{{- item.text }}
|
| 32 |
+
{%- else %}
|
| 33 |
+
{{- raise_exception('Unexpected item type in content.') }}
|
| 34 |
+
{%- endif %}
|
| 35 |
+
{%- endfor %}
|
| 36 |
+
{%- elif content is none or content is undefined %}
|
| 37 |
+
{{- '' }}
|
| 38 |
+
{%- else %}
|
| 39 |
+
{{- raise_exception('Unexpected content type.') }}
|
| 40 |
+
{%- endif %}
|
| 41 |
+
{%- endmacro %}
|
| 42 |
+
{%- if not messages %}
|
| 43 |
+
{{- raise_exception('No messages provided.') }}
|
| 44 |
+
{%- endif %}
|
| 45 |
+
{%- set reasoning_instructions = '' %}
|
| 46 |
+
{%- if enable_thinking is undefined or enable_thinking is true %}
|
| 47 |
+
{%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}
|
| 48 |
+
{%- if resolved_reasoning_effort not in ('xhigh', 'medium', 'low') %}
|
| 49 |
+
{{- raise_exception('Unexpected reasoning effort ' ~ reasoning_effort ~ '. Supported types are xhigh (default), medium, and low.') }}
|
| 50 |
+
{%- endif %}
|
| 51 |
+
{%- if resolved_reasoning_effort == 'xhigh' %}
|
| 52 |
+
{%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
|
| 53 |
+
{%- elif resolved_reasoning_effort == 'low' %}
|
| 54 |
+
{%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}
|
| 55 |
+
{%- endif %}
|
| 56 |
+
{%- endif %}
|
| 57 |
+
{%- if tools and tools is iterable and tools is not mapping %}
|
| 58 |
+
{{- '<|im_start|>system\n' }}
|
| 59 |
+
{%- if reasoning_instructions %}
|
| 60 |
+
{{- reasoning_instructions + '\n\n' }}
|
| 61 |
+
{%- endif %}
|
| 62 |
+
{{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
|
| 63 |
+
{%- for tool in tools %}
|
| 64 |
+
{{- "\n" }}
|
| 65 |
+
{{- tool | tojson }}
|
| 66 |
+
{%- endfor %}
|
| 67 |
+
{{- "\n</tools>" }}
|
| 68 |
+
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
|
| 69 |
+
{%- if messages[0].role == 'system' %}
|
| 70 |
+
{%- set content = render_content(messages[0].content, false, true)|trim %}
|
| 71 |
+
{%- if content %}
|
| 72 |
+
{{- '\n\n' + content }}
|
| 73 |
+
{%- endif %}
|
| 74 |
+
{%- endif %}
|
| 75 |
+
{{- '<|im_end|>\n' }}
|
| 76 |
+
{%- else %}
|
| 77 |
+
{%- if messages[0].role == 'system' %}
|
| 78 |
+
{%- set content = render_content(messages[0].content, false, true)|trim %}
|
| 79 |
+
{%- if content %}
|
| 80 |
+
{{- '<|im_start|>system\n' + (reasoning_instructions + '\n\n' if reasoning_instructions else '') + content + '<|im_end|>\n' }}
|
| 81 |
+
{%- elif reasoning_instructions %}
|
| 82 |
+
{{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
|
| 83 |
+
{%- endif %}
|
| 84 |
+
{%- elif reasoning_instructions %}
|
| 85 |
+
{{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
|
| 86 |
+
{%- endif %}
|
| 87 |
+
{%- endif %}
|
| 88 |
+
{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
|
| 89 |
+
{%- for message in messages[::-1] %}
|
| 90 |
+
{%- set index = (messages|length - 1) - loop.index0 %}
|
| 91 |
+
{%- if ns.multi_step_tool and message.role == "user" %}
|
| 92 |
+
{%- set content = render_content(message.content, false)|trim %}
|
| 93 |
+
{%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
|
| 94 |
+
{%- set ns.multi_step_tool = false %}
|
| 95 |
+
{%- set ns.last_query_index = index %}
|
| 96 |
+
{%- endif %}
|
| 97 |
+
{%- endif %}
|
| 98 |
+
{%- endfor %}
|
| 99 |
+
{%- if ns.multi_step_tool %}
|
| 100 |
+
{{- raise_exception('No user query found in messages.') }}
|
| 101 |
+
{%- endif %}
|
| 102 |
+
{%- for message in messages %}
|
| 103 |
+
{%- set content = render_content(message.content, true)|trim %}
|
| 104 |
+
{%- if message.role == "system" %}
|
| 105 |
+
{%- if not loop.first %}
|
| 106 |
+
{{- raise_exception('System message must be at the beginning.') }}
|
| 107 |
+
{%- endif %}
|
| 108 |
+
{%- elif message.role == "user" %}
|
| 109 |
+
{{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
|
| 110 |
+
{%- elif message.role == "assistant" %}
|
| 111 |
+
{%- set reasoning_content = '' %}
|
| 112 |
+
{%- if message.reasoning_content is string %}
|
| 113 |
+
{%- set reasoning_content = message.reasoning_content %}
|
| 114 |
+
{%- endif %}
|
| 115 |
+
{%- set reasoning_content = reasoning_content|trim %}
|
| 116 |
+
{%- if preserve_thinking is undefined or preserve_thinking is true or loop.index0 > ns.last_query_index %}
|
| 117 |
+
{{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
|
| 118 |
+
{%- else %}
|
| 119 |
+
{{- '<|im_start|>' + message.role + '\n' + content }}
|
| 120 |
+
{%- endif %}
|
| 121 |
+
{%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
|
| 122 |
+
{%- for tool_call in message.tool_calls %}
|
| 123 |
+
{%- if tool_call.function is defined %}
|
| 124 |
+
{%- set tool_call = tool_call.function %}
|
| 125 |
+
{%- endif %}
|
| 126 |
+
{%- if loop.first %}
|
| 127 |
+
{%- if content|trim %}
|
| 128 |
+
{{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
| 129 |
+
{%- else %}
|
| 130 |
+
{{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
| 131 |
+
{%- endif %}
|
| 132 |
+
{%- else %}
|
| 133 |
+
{{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
|
| 134 |
+
{%- endif %}
|
| 135 |
+
{%- if tool_call.arguments is defined and tool_call.arguments != '' %}
|
| 136 |
+
{%- for args_name, args_value in tool_call.arguments|items %}
|
| 137 |
+
{{- '<parameter=' + args_name + '>\n' }}
|
| 138 |
+
{%- set args_value = args_value | string if args_value is string else args_value | tojson | safe %}
|
| 139 |
+
{{- args_value }}
|
| 140 |
+
{{- '\n</parameter>\n' }}
|
| 141 |
+
{%- endfor %}
|
| 142 |
+
{%- endif %}
|
| 143 |
+
{{- '</function>\n</tool_call>' }}
|
| 144 |
+
{%- endfor %}
|
| 145 |
+
{%- endif %}
|
| 146 |
+
{{- '<|im_end|>\n' }}
|
| 147 |
+
{%- elif message.role == "tool" %}
|
| 148 |
+
{%- if loop.previtem and loop.previtem.role != "tool" %}
|
| 149 |
+
{{- '<|im_start|>user' }}
|
| 150 |
+
{%- endif %}
|
| 151 |
+
{{- '\n<tool_response>\n' }}
|
| 152 |
+
{{- content }}
|
| 153 |
+
{{- '\n</tool_response>' }}
|
| 154 |
+
{%- if not loop.last and loop.nextitem.role != "tool" %}
|
| 155 |
+
{{- '<|im_end|>\n' }}
|
| 156 |
+
{%- elif loop.last %}
|
| 157 |
+
{{- '<|im_end|>\n' }}
|
| 158 |
+
{%- endif %}
|
| 159 |
+
{%- else %}
|
| 160 |
+
{{- raise_exception('Unexpected message role.') }}
|
| 161 |
+
{%- endif %}
|
| 162 |
+
{%- endfor %}
|
| 163 |
+
{%- if add_generation_prompt %}
|
| 164 |
+
{{- '<|im_start|>assistant\n' }}
|
| 165 |
+
{%- if enable_thinking is defined and enable_thinking is false %}
|
| 166 |
+
{{- '<think>\n\n</think>\n\n' }}
|
| 167 |
+
{%- else %}
|
| 168 |
+
{{- '<think>\n' }}
|
| 169 |
+
{%- endif %}
|
| 170 |
+
{%- endif %}
|
checkpoint-00001-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ec1a85d6874a69848a71260b0495f7650f028bb0b6360bdc5a808703686466f1
|
| 3 |
+
size 8590380332
|
checkpoint-00002-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:93b9385fdfcdbf02ca96867815bc54fe9d05d43752225bebac00d8f4082da36e
|
| 3 |
+
size 7831294156
|
checkpoint-00003-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9256bafce60aa12fe369163049c66881d8ff2791a1c3643538db2e44217af9d9
|
| 3 |
+
size 8000039880
|
checkpoint-00004-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:fcacf1ed13c2f24b8f644f0efeb03943b0186f41ebd0b60a29af224383fd86b7
|
| 3 |
+
size 8000039880
|
checkpoint-00005-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8c21cbdf48da0f44b53fc5fed9b985924558856d35ebe3f24511f42f173488f3
|
| 3 |
+
size 8000039880
|
checkpoint-00006-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a87de80244d150eeb2db7ddff0e21091f843e2d94d64b8a464032407da59f2cb
|
| 3 |
+
size 8000039880
|
checkpoint-00007-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:de07acf63c0338e9ad0f9d49f92daedab177d3d476ea2f4b11795a3d7a93d321
|
| 3 |
+
size 8000039880
|
checkpoint-00008-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2cc5d85659291cf09911578dfbcdf47af991500232cd287a0bdc7a329fe15600
|
| 3 |
+
size 8000039880
|
checkpoint-00009-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a76560b883ea8ca38471c00810680341190691498735f3d0577cd59209ced67b
|
| 3 |
+
size 8000039880
|
checkpoint-00010-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6a1085ec240a96b44bd24b204ff95fa7c82d79f79f1fe40c6590e01bffe66e79
|
| 3 |
+
size 8000039880
|
checkpoint-00011-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:095549a992e8af23f07adbe9855131f394070c5881a9d2a12798eeb086571492
|
| 3 |
+
size 8000039880
|
checkpoint-00012-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:38272f121f015ce3a06c2f4fb0c8a18f55654d183e497bae7214b3c220c3cf00
|
| 3 |
+
size 8000039888
|
checkpoint-00013-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2d325b837b1c6a233677b38fd99fa72b94ee3636d57ec22ac740c79282b0be12
|
| 3 |
+
size 8000039896
|
checkpoint-00014-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:1d57b42be6931376c4655baf51a7cf00c7010a449ba671c90b650ed126874bf2
|
| 3 |
+
size 8000039896
|
checkpoint-00015-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ce330681500b37d9f601a287b3f874776e29f84018be8a7296a30d0e8255d843
|
| 3 |
+
size 8588435468
|
checkpoint-00016-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ad6062c339f3c52f0840d98e75727d8bd1a6d0b5801740798119eb7d2b5ba986
|
| 3 |
+
size 8589907880
|
checkpoint-00017-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:073adcad12abb2cf3a8d79a38047681c9dc2909db28c2d7e1824130b15d3becb
|
| 3 |
+
size 8587406136
|
checkpoint-00018-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4f604d1b618a357fa1b8db10cd09560e292153ef348012b325d0c10a1704b84c
|
| 3 |
+
size 8587490060
|
checkpoint-00019-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:32bbd7661f79fbc071512396aba7f92323319f93c842a2e03402cfc8d9394d50
|
| 3 |
+
size 8589907880
|
checkpoint-00020-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0322ba7321de8c4753c7fe40d91cfc639709e8197f5a9182c0e19a605f9d2736
|
| 3 |
+
size 8587495172
|
checkpoint-00021-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:cc17e0fbccbeeb6d5704c89a4f051a27b6b65b7c6d57eddb71f22b727f75647b
|
| 3 |
+
size 8587411064
|
checkpoint-00022-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4c1cd2d5696881d0aa284c29e916b6268f271921848a13ccab5ea05d018f7d7a
|
| 3 |
+
size 8589913056
|
checkpoint-00023-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2c138b2c9a6b2ff21cf69fb37551e53f8db3d09cdec6ddb41b418d1fe2591166
|
| 3 |
+
size 8587495292
|
checkpoint-00024-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:097ad486f89cfc1895e4df8c87c6f858bbc94f946f511f1bfdd174c345321565
|
| 3 |
+
size 8587411064
|
checkpoint-00025-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:16ee14c1154a9d3cbe38d17957a48b143c9c9d65371a9873539fbe736e6ed49d
|
| 3 |
+
size 8589997164
|
checkpoint-00026-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:cba0e740690b0b7ebbd28d9c380c8c0da6d39c818d68bc33ed97cc2a1a845e10
|
| 3 |
+
size 8587411200
|
checkpoint-00027-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:bb52d61a085527984487aa3588f7395bb4f6f940f9281b9d369bfa843694d09a
|
| 3 |
+
size 8587495188
|
checkpoint-00028-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b49ad2872f4e0be2f75c4c14617d979140a53cab8ae8bdaf239fd0f451dac06e
|
| 3 |
+
size 8589913040
|
checkpoint-00029-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:484f9fdba214c645ee237a8db4b9f445bcf0a65082cb1ec168efb8691b3be08f
|
| 3 |
+
size 8587411200
|
checkpoint-00030-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f6aa3420eb84cd4e15a538b477d2306d805c6241bb04a926c089cdf0216b778c
|
| 3 |
+
size 8587495292
|
checkpoint-00031-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7d9710c0425ff986d0838fe0c6ca9716561f8f4e07a17d5fabee4b724b1631de
|
| 3 |
+
size 8589912936
|
checkpoint-00032-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d248e928c45b16f9fe06bd2dd657b11e8870adaf8c3e5273821af44c4ea3f77c
|
| 3 |
+
size 8587495308
|
checkpoint-00033-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5dfcfae6ec6a569bd432ba12936ca94ebd9fdb69b5da08dfb0f1fa3214ded250
|
| 3 |
+
size 8587411280
|
checkpoint-00034-of-00043.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5fb1ce65a5f0d03adac60f424159bd1b4789d88cfe3cfe524cd5e5af5fca8e8f
|
| 3 |
+
size 8589912840
|