Duplicate from Kovacreations/Primus

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

PRIMUS_PAPER.md

@@ -0,0 +1,98 @@
+# Primus: A Coding Model That Fixes What Developers Complain About
+
+**Kovalabs** · August 2026
+
+## Abstract
+
+We present Primus, a 295B-parameter coding model fine-tuned from Tencent Hy3 using the APEX method stack. Unlike frontier models optimized for benchmark scores, Primus is optimized for the five complaints developers actually have about AI coding assistants: regression-free repair, scope discipline, ask-vs-guess behavior, API honesty, and terseness. We introduce TIDE (Trajectory-level Iterative DPO with Execution anchors), a step-anchored preference optimization method that assigns credit/blame at the exact causal step rather than smeared over whole trajectories. We also present CAT (Consequence-Augmented Training), which teaches the model to predict blast radius before acting, and SENTINEL, an anti-reward-hacking verifier. On our pre-registered 15-axis scorecard, Primus achieves [X/15] wins vs Claude Fable 5, with all losses published.
+
+## 1. Introduction
+
+Developers don't complain about benchmark scores. They complain that AI coding assistants:
+1. **Break adjacent functionality** while fixing the requested bug
+2. **Change 47 files** when asked for a one-line fix
+3. **Guess** when they should ask for clarification
+4. **Invent APIs** that don't exist
+5. **Write essays** instead of minimal diffs
+
+Existing coding models optimize for SWE-bench and similar benchmarks, which measure "can it fix the issue" but not "does it break everything else." Primus is built to fix the complaints, not just the benchmarks.
+
+## 2. The APEX Method Stack
+
+### 2.1 CAT — Consequence-Augmented Training
+Every training sample includes a consequence prediction: before acting, the model predicts `files_at_risk`, `tests_at_risk`, `risk_class`, and `minimal_plan`. This creates an internal "blast radius" model that activates at inference time.
+
+### 2.2 TIDE — Step-Anchored DPO
+Standard DPO treats a failed trajectory as one "rejected" sample, training down 39 good steps along with the 1 bad step. TIDE anchors the preference pair at the exact causal step:
+- **prompt** = everything before the failing step
+- **rejected** = the failing step
+- **chosen** = the recovery step (or gold fix)
+
+This teaches the model *which specific action* caused the failure, not just "this trajectory was bad."
+
+### 2.3 SENTINEL — Anti-Reward-Hacking
+Detects when the model games metrics instead of solving tasks: test tampering, skip-added, tautological asserts, mock exploits, scope dishonesty, cross-verifier disagreement.
+
+### 2.4 AEGIS — Adaptive Best-of-N (Inference Time)
+At inference, sample N candidates, CAT-predict consequence for each, and select the safest effective option. Not just highest logprob — lowest risk.
+
+### 2.5 SONAR — External Memory
+A persistent `STATE.md` file that survives context window limits on marathon tasks (multi-hour repo work).
+
+## 3. Training
+
+- **Base:** Tencent Hy3 (295B MoE, 21B active, 192 experts top-8)
+- **Method:** LoRA (r=64, α=128) SFT → TIDE-DPO → RLVR sprint
+- **Data:** 31,453 decontaminated samples (priors, CAT units, behavior demonstrations, TIDE pairs)
+- **Hardware:** 8× RTX PRO 6000 (764GB VRAM), ~78 hours SFT, ~11 hours DPO
+- **Decontamination:** All training data checked against eval benchmarks; no SWE-Gym/SWE-bench overlap in RLVR
+
+## 4. Evaluation
+
+### 4.1 Pre-Registered Scorecard (15 axes)
+| Benchmark | K3 | Fable 5 | GLM-5.2 | Primus | Win? |
+|---|---|---|---|---|---|
+| Terminal-Bench 2.1 | 88.3 | 88.0 | 82.7 | [TBD] | [TBD] |
+| PostTrainBench | 36.6 | 41.4 | 34.3 | [TBD] | [TBD] |
+| MLS-Bench-Lite | 48.3 | 49.9 | 40.4 | [TBD] | [TBD] |
+| SciCode | 58.7 | 60.2 | 50.5 | [TBD] | [TBD] |
+| Kimi Code Bench 2.0 | 72.9 | 76.9 | 64.2 | [TBD] | [TBD] |
+| ProgramBench | 77.8 | 76.8 | 63.7 | [TBD] | [TBD] |
+| FrontierSWE | 81.2 | 86.6 | 67.3 | [TBD] | [TBD] |
+| SWE-bench Pro | 45.0 | 52.0 | 38.0 | [TBD] | [TBD] |
+| SWE-Marathon | 42.0 | 35.0 | 13.0 | [TBD] | [TBD] |
+| DeepSWE | 67.5 | 70.0 | 46.2 | [TBD] | [TBD] |
+| **DevPain (5 axes)** | — | — | — | [TBD] | **sweep target** |
+
+### 4.2 DevPain Suite
+Our own sealed benchmark measuring the five developer complaints. Full results published with the model.
+
+## 5. Results
+
+[To be filled after eval completes]
+
+## 6. Limitations
+
+- LoRA-only (not full fine-tune) — some capability ceiling
+- Teacher cascade (Kimi k2.7) was not used in final training due to API issues
+- RLVR sprint was simplified best-of-N, not full GRPO
+- SWE-bench Pro and Marathon remain challenging
+
+## 7. Ethics & Reproducibility
+
+- All training data decontaminated against eval benchmarks
+- All losses published, not just wins
+- Harness open-sourced: [github.com/kovalabs/nebula](https://github.com/kovalabs/nebula)
+- Model card: [huggingface.co/kovalabs/primus](https://huggingface.co/kovalabs/primus)
+
+## 8. Future Work
+
+- FORGE: per-domain LoRA specialists + gated merge
+- HELM: milestone ladders for sparse-reward marathon tasks
+- Full GRPO with TRL (not simplified best-of-N)
+- Teacher cascade integration (Cloudflare/Runpod)
+- ACT loop: active curriculum training from failure autopsy
+
+---
+
+*Primus: first, do no harm to the codebase.*

README.md

@@ -0,0 +1,80 @@
+---
+license: apache-2.0
+tags:
+  - code
+  - agent
+  - merged
+  - fine-tuned
+---
+
+# Primus
+
+**A coding model that fixes what developers actually complain about.**
+
+Primus is a 295B-parameter coding model fine-tuned using the APEX method stack. Unlike frontier models optimized for benchmark scores, Primus is optimized for the five complaints developers have about AI coding assistants:
+
+1. **Regression-free repair** — fixes the bug without breaking adjacent functionality
+2. **Scope discipline** — changes only what you asked for, not 47 files
+3. **Ask-vs-guess** — asks clarifying questions instead of guessing wrong
+4. **API honesty** — never invents functions or libraries that don't exist
+5. **Terseness** — minimal diffs, no essays
+
+## Method
+
+- **Training:** LoRA SFT → TIDE-DPO → RLVR sprint
+- **Data:** 31,453 decontaminated samples (priors, CAT units, behavior demonstrations, TIDE pairs)
+- **Hardware:** 8× RTX PRO 6000 (764GB VRAM)
+
+## The APEX Stack
+
+- **CAT** (Consequence-Augmented Training): Predicts blast radius before acting
+- **TIDE** (Step-Anchored DPO): Assigns credit/blame at the exact causal step
+- **SENTINEL**: Anti-reward-hacking verifier
+- **AEGIS**: Adaptive best-of-N at inference (safest effective option)
+- **SONAR**: External memory for long-horizon tasks
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model = AutoModelForCausalLM.from_pretrained(
+    "kovacreations/primus",
+    trust_remote_code=True,
+    device_map="auto",
+    torch_dtype="bfloat16"
+)
+tok = AutoTokenizer.from_pretrained("kovacreations/primus", trust_remote_code=True)
+
+messages = [
+    {"role": "system", "content": "You are Primus, a coding assistant. Write clean, minimal code."},
+    {"role": "user", "content": "Fix the null pointer in auth.py"}
+]
+prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+inputs = tok(prompt, return_tensors="pt").to(model.device)
+out = model.generate(**inputs, max_new_tokens=2048)
+print(tok.decode(out[0], skip_special_tokens=True))
+```
+
+## Scorecard
+
+Pre-registered 15-axis scorecard with all results published (wins and losses):
+
+- 10 public benchmarks (Terminal-Bench, SWE-bench, etc.)
+- 5 DevPain axes (regression, scope, ask, API honesty, terseness)
+
+## Research Paper
+
+See [PRIMUS_PAPER.md](PRIMUS_PAPER.md) for the full method description, training details, and analysis.
+
+## Harness
+
+Open-source evaluation harness: [github.com/kovalabs/nebula](https://github.com/kovalabs/nebula)
+
+## License
+
+Apache 2.0. Decontaminated training data. All losses published.
+
+---
+
+*Primus: first, do no harm to the codebase.*

chat_template.jinja

@@ -0,0 +1,222 @@
+{#- ----------‑‑‑ special token variables ‑‑‑---------- -#}
+{%- set HYTK = ':opensource' %}
+{%- set eos_token = '<｜hy_eos{}｜>'.format(HYTK) %}
+{%- set bos_token = '<｜hy_begin_of_sentence{}｜>'.format(HYTK) %}
+{%- set pad_token = '<｜hy_pad{}｜>'.format(HYTK) %}
+{%- set user_token = '<｜hy_User{}｜>'.format(HYTK) %}
+{%- set assistant_token = '<｜hy_Assistant{}｜>'.format(HYTK) %}
+{%- set think_begin_token = '<think{}>'.format(HYTK) %}
+{%- set think_end_token = '</think{}>'.format(HYTK) %}
+{%- set toolcalls_begin_token = '<tool_calls{}>'.format(HYTK) %}
+{%- set toolcalls_end_token = '</tool_calls{}>'.format(HYTK) %}
+{%- set toolcall_begin_token = '<tool_call{}>'.format(HYTK) %}
+{%- set toolcall_end_token = '</tool_call{}>'.format(HYTK) %}
+{%- set toolsep_token = '<tool_sep{}>'.format(HYTK) %}
+{%- set argkey_begin_token = '<arg_key{}>'.format(HYTK) %}
+{%- set argkey_end_token = '</arg_key{}>'.format(HYTK) %}
+{%- set argvalue_begin_token = '<arg_value{}>'.format(HYTK) %}
+{%- set argvalue_end_token = '</arg_value{}>'.format(HYTK) %}
+{%- set toolresponses_begin_token = '<tool_responses{}>'.format(HYTK) %}
+{%- set toolresponses_end_token = '</tool_responses{}>'.format(HYTK) %}
+{%- set toolresponse_begin_token = '<tool_response{}>'.format(HYTK) %}
+{%- set toolresponse_end_token = '</tool_response{}>'.format(HYTK) %}
+{%- set reasoning_mode_token = '<｜reasoning_mode{}｜>'.format(HYTK) %}
+
+{#- ----------‑‑‑ hyperparameters variables ‑‑‑---------- -#}
+{%- if not add_generation_prompt is defined %}
+    {%- set add_generation_prompt = false %}
+{%- endif %}
+{%- if not preserved_thinking is defined %}
+    {%- if not tools %}
+        {%- set preserved_thinking = false %}
+    {%- else %}
+        {%- set preserved_thinking = true %}
+    {%- endif %}
+{%- endif %}
+{%- if not is_training is defined %}
+    {%- set is_training = false %}
+{%- endif %}
+
+{%- if not reasoning_effort is defined %}
+    {%- set reasoning_effort = 'no_think' %}
+{%- elif reasoning_effort not in ['high', 'low', 'no_think'] %}
+    {%- if reasoning_effort is none %}
+        {{- raise_exception('reasoning_effort error : None, should be no_think/low/high') }}
+    {%- else %}
+        {{- raise_exception('reasoning_effort error : ' + reasoning_effort + ', should be no_think/low/high') }}
+    {%- endif %}
+{%- endif %}
+
+{%- if fallback_strategy is defined and fallback_strategy == 'reasoning_toolcall_retry' %}
+    {%- set reasoning_effort = 'high' %}
+    {%- set add_generation_prompt = false %}
+{%- endif %}
+{%- if not raw_last_assistant is defined %}
+    {%- set raw_last_assistant = false %}
+{%- endif %}
+
+{%- macro visible_text(content) -%}
+    {%- if content is string -%}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping -%}
+        {%- for item in content -%}
+            {%- if item is mapping and item.type == 'text' -%}
+                {{- item.text }}
+            {%- elif item is string -%}
+                {{- item }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- elif content is none -%}
+        {{- '' }}
+    {%- else -%}
+        {{- content }}
+    {%- endif -%}
+{%- endmacro -%}
+
+{%- set ns = namespace(last_user_index=-1) %}
+{%- set sp_ns = namespace(system_prompt='', is_first_sp=true) %}
+{%- for message in messages %}
+    {%- if message['role'] == 'system' %}
+        {%- set sp_ns.system_prompt = sp_ns.system_prompt + visible_text(message['content']) %}
+    {%- endif %}
+    {%- if message['role'] == 'user' %}
+        {%- set ns.last_user_index = loop.index0 %}
+    {%- endif %}
+{%- endfor %}
+{%- if reasoning_effort is defined and reasoning_effort is string and reasoning_effort != '' and not tools %}
+    {%- set sp_ns.system_prompt = sp_ns.system_prompt + reasoning_mode_token + 'reasoning_effort:' + reasoning_effort %}
+{%- endif %}
+{{- bos_token }}
+{{- sp_ns.system_prompt }}
+{%- if tools %}
+    {%- if sp_ns.system_prompt != '' %}
+        {{- '\n\n# Tools\n\nYou may call one or more functions to assist with the user query.' }}
+    {%- else %}
+        {{- '# Tools\n\nYou may call one or more functions to assist with the user query.' }}
+    {%- endif %}
+    {{- '\n\nYou are provided with function signatures within <tools></tools> XML tags:' }}
+    {{- '\n<tools>\n' }}
+    {%- for tool in tools %}
+        {%- if loop.index0 > 0 %}
+            {{- '\n' }}
+        {%- endif %}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- '\n</tools>\n\n' }}
+    {{- 'For function call returns, you should first print ' + toolcalls_begin_token + '\n' }}
+    {{- 'For each function call, you should return object like:\n' }}
+    {{- toolcall_begin_token + '{function-name}' + toolsep_token + '\n' }}
+    {{- argkey_begin_token + '{arg-key-1}' + argkey_end_token + '\n' }}
+    {{- argvalue_begin_token + '{arg-value-1}' + argvalue_end_token + '\n' }}
+    {{- argkey_begin_token + '{arg-key-2}' + argkey_end_token + '\n' }}
+    {{- argvalue_begin_token + '{arg-value-2}' + argvalue_end_token + '\n' }}
+    {{- '...\n' }}
+    {{- toolcall_end_token + '\n' }}
+    {%- if reasoning_effort is defined and reasoning_effort is string and reasoning_effort != '' %}
+        {{- 'At the end of function call returns, you should print ' + toolcalls_end_token + reasoning_mode_token + 'reasoning_effort:' + reasoning_effort }}
+    {%- else %}
+        {{- 'At the end of function call returns, you should print ' + toolcalls_end_token }}
+    {%- endif %}
+{%- endif %}
+
+{%- set prev_ns = namespace(is_tool=false, is_tool_first=true) %}
+{%- set last_ns = namespace(last_is_assistant=false) %}
+{%- for message in messages %}
+    {%- if message['role'] == 'user' %}
+        {%- if prev_ns.is_tool %}
+            {{- toolresponses_end_token }}
+        {%- endif %}
+        {{- user_token + visible_text(message['content']) }}
+        {%- set prev_ns.is_tool = false %}
+    {%- endif %}
+    {%- if message['role'] == 'assistant' %}
+        {%- if is_training %}
+            {%- if 'reasoning_content' in message and message['reasoning_content'] is string %}
+                {%- set rc = message['reasoning_content'] %}
+            {%- elif 'reasoning' in message and message['reasoning'] is string %}
+                {%- set rc = message['reasoning'] %}
+            {%- else %}
+                {%- set rc = none %}
+            {%- endif %}
+            {%- if rc is not none %}
+                {%- set content = think_begin_token + rc + think_end_token + visible_text(message['content']) %}
+            {%- else %}
+                {%- set content = think_begin_token + think_end_token + visible_text(message['content']) %}
+            {%- endif %}
+        {%- else %}
+            {%- if ((preserved_thinking is defined and preserved_thinking) or loop.index0 > ns.last_user_index) %}
+                {%- if 'reasoning_content' in message and message['reasoning_content'] is string %}
+                    {%- set rc = message['reasoning_content'] %}
+                {%- elif 'reasoning' in message and message['reasoning'] is string %}
+                    {%- set rc = message['reasoning'] %}
+                {%- else %}
+                    {%- set rc = none %}
+                {%- endif %}
+                {%- if rc is not none %}
+                    {%- set content = think_begin_token + rc + think_end_token + visible_text(message['content']) %}
+                {%- else %}
+                    {%- set content = think_begin_token + think_end_token + visible_text(message['content']) %}
+                {%- endif %}
+            {%- else %}
+                {%- set content = think_begin_token + think_end_token + visible_text(message['content']) %}
+            {%- endif %}
+        {%- endif %}
+        {%- if prev_ns.is_tool %}
+            {{- toolresponses_end_token }}
+        {%- endif %}
+        {{- assistant_token }}
+        {%- if message['tool_calls'] is defined and message['tool_calls'] %}
+            {%- set prev_ns.is_tool_first = true %}
+            {{- content }}
+            {{- toolcalls_begin_token + '\n' }}
+            {%- for tool in message['tool_calls'] %}
+                {%- set arguments = tool['function']['arguments'] %}
+                {{- toolcall_begin_token + tool['function']['name'] + toolsep_token + '\n' }}
+                {%- for key, value in arguments.items() %}
+                    {{- argkey_begin_token + key + argkey_end_token + '\n' }}
+                    {%- if value is not string %}
+                        {%- set value = value | tojson(ensure_ascii=False) %}
+                    {%- endif %}
+                    {{- argvalue_begin_token + value + argvalue_end_token + '\n' }}
+                {%- endfor %}
+                {{- toolcall_end_token + '\n' }}
+            {%- endfor %}
+            {{- toolcalls_end_token + eos_token }}
+        {%- else %}
+            {%- if loop.last and raw_last_assistant %}
+                {{- visible_text(message['content']) }}
+            {%- elif not loop.last or is_training %}
+                {{- content + eos_token }}
+            {%- else %}
+                {{- content }}
+            {%- endif %}
+        {%- endif %}
+        {%- set prev_ns.is_tool = false %}
+    {%- endif %}
+    {%- if message['role'] == 'tool' %}
+        {%- set prev_ns.is_tool = true %}
+        {%- if prev_ns.is_tool_first %}
+            {{- toolresponses_begin_token + '\n' }}
+            {%- set prev_ns.is_tool_first = false %}
+        {%- endif %}
+        {{- toolresponse_begin_token + '\n' + visible_text(message['content']) + '\n' + toolresponse_end_token + '\n' }}
+    {%- endif %}
+    {%- if loop.last and message['role'] == 'assistant' %}
+        {%- set last_ns.last_is_assistant = true %}
+    {%- endif %}
+
+{%- endfor %}
+{%- if prev_ns.is_tool %}
+    {{- toolresponses_end_token }}
+{%- endif %}
+{%- if add_generation_prompt %}
+    {%- if not last_ns.last_is_assistant %}
+        {%- if reasoning_effort is defined and reasoning_effort in ['low', 'high'] %}
+            {{- assistant_token + think_begin_token }}
+        {%- elif reasoning_effort is defined and reasoning_effort == 'no_think' %}
+            {{- assistant_token + think_begin_token + think_end_token }}
+        {%- else %}
+            {{- assistant_token }}
+        {%- endif %}
+    {%- endif %}
+{%- endif %}

config.json

@@ -0,0 +1,132 @@
+{
+  "architectures": [
+    "HYV3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 120000,
+  "dtype": "bfloat16",
+  "enable_attention_fp32_softmax": false,
+  "enable_lm_head_fp32": true,
+  "enable_moe_fp32_combine": false,
+  "eod_token_id": 120026,
+  "eos_token_id": 120025,
+  "expert_hidden_dim": 1536,
+  "first_k_dense_replace": 1,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.006,
+  "intermediate_size": 13312,
+  "max_position_embeddings": 262144,
+  "mlp_bias": false,
+  "mlp_layer_types": [
+    "dense",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse",
+    "sparse"
+  ],
+  "model_type": "hy_v3",
+  "moe_intermediate_size": 1536,
+  "moe_router_enable_expert_bias": true,
+  "moe_router_use_sigmoid": true,
+  "num_attention_heads": 64,
+  "num_experts": 192,
+  "num_experts_per_tok": 8,
+  "num_hidden_layers": 80,
+  "num_key_value_heads": 8,
+  "num_nextn_predict_layers": 1,
+  "num_shared_experts": 1,
+  "output_router_logits": true,
+  "pad_token_id": 120002,
+  "qk_norm": true,
+  "rms_norm_eps": 1e-05,
+  "rope_parameters": {
+    "rope_theta": 11158840.0,
+    "rope_type": "default"
+  },
+  "route_norm": true,
+  "router_scaling_factor": 2.826,
+  "sep_token_id": 120007,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.15.0.dev0",
+  "use_cache": true,
+  "use_grouped_mm": false,
+  "vocab_size": 120832
+}

model-00001-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:66c9811c1a6f3a8c89e0e252b5991ee0112c8b6dd91a0a06c7863dc6c0200c16
+size 7247829008

model-00002-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:438bfdf0685abb46abf98d83b0abec150096f68951ad61af60fe5cbd9c915ac7
+size 7247829584

model-00003-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0112623ad063ea80a559d4ba02b72877a7d9cfe7fd80f57448efad0c006309e6
+size 7247829584

model-00004-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b289f1c54b9645d03a3acb93e0af23b9c897cd38a389db1a601170e5ae2302e3
+size 7247829584

model-00005-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7fdc11262b41aceaf68aefbda84088a909dc2d1344781315b50763ad457b0d60
+size 7247829584

model-00006-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c63adc9dcbaebef3c6876a95b966f2880b3259f28e89093316299ac23499445e
+size 1073745312

model-00007-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:768a4a6797d0f18d3658db19256dc837b9bf8207be5853d1e534aac4fe2dc9b4
+size 7247829008

model-00008-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e1a02ea7da3ae3fa4feea8d4b31ab15f88b73d33b9789691b9a7310710ae7c28
+size 7247829584

model-00009-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:09dbd820bd4dcb306976105266dbb017b2d0f1a8019bd37b6dbd4bf859e35c59
+size 7247829584

model-00010-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9bd6929f27a8bea148d20d709e82762b3593767ac5604be4a91e11c498249827
+size 7247829584

model-00011-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:29c8d8a926246569e553a1486b1d9f40774ed4bc46ff6a33e3e425ea76bb5e16
+size 7247829584

model-00012-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f03adb53cbf5e2ff5200b8358374f5109ec1ce996679875c3a75a2b7ea986077
+size 1073742272

model-00013-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1a8222ce5abf81d4cdf72fc8ff23d37d09cfc972552b7db1e2f63adcaf15d249
+size 7247829008

model-00014-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fcc9f1aed84d47e68df595eff48272e6726c77e96c78d89ded6294ce958e6332
+size 7247829584

model-00015-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d3e7c4360e3782e60bd4e43b029e66818a2bbd513794d34503e0c7ceaecd1b92
+size 7247829584

model-00016-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1547e4a234524ee22e45155bf18a763e5a7ad3bcfd19fce86ee1332131ca9510
+size 7247829584

model-00017-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6d6e18ece70a4813a408b894a0399bd830c18a6888788b263bfef17b880272c4
+size 7247829584

model-00018-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f69f7c228609cfb2c994c86fd5ec33cbc792cf38c0e399e7ff6a35b7267b0a48
+size 1073746912

model-00019-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:20e4912b081ddf058497e07a00a76715cf7d58a108020528dd9469a0d0971d24
+size 7247829008

model-00020-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:09bcddf356c7098e82688d818a331f36a10973a67184c8430832271882035487
+size 7247829584

model-00021-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f9de5be0d34f3fad73d011e6c73c5da89b2813c03e52c61cb2ed0391beeb27b2
+size 7247829584

model-00022-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e6682fe3970c95511dc7e25c96394a0adf1b6cedc227613f2902e6a960dae0bb
+size 7247829584

model-00023-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:52d9924ead7e25ddf5019cc7e9504b3dc113f4777202f05cb9916d55f2a25ea2
+size 7247829584

model-00024-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e6da22a75d48894da525c85b3bc41748e4d68dc214a8d4e828239835b22da646
+size 1067458192

model-00025-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4e428cd287456c189baac90b1ca8aa2d0d9e1cb453faf0ca827a062913874c84
+size 7247829008

model-00026-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a3f98207f6205818674908052efb4c3adeed6ce73154617323177df347a6da4d
+size 7247829584

model-00027-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:59946d434bd2e6377c68728b5417285c77510ccc13a2c873b0ad96002f5a3914
+size 7247829584

model-00028-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dcd50b4691e61a45a06f0a8c7a461e71de1e6fe55951ba23439eba90a764fa03
+size 7247829584

model-00029-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d9a7b69c60dbb9afdc447bb0dc06a643846ef83da32b9206c1ca9b37b6e91d6a
+size 7247829584

model-00030-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a9a4c2cec9b58f0bd9b4907596fefd4d8c18808158f8abd4bb7c7d8810491581
+size 1065220848

model-00031-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d2e0f20db6780b593af07b38158163df6af4ccd76f35b6506dead303f60730a7
+size 7247829008

model-00032-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:51216f9f9d6658aea0277e9e73a91083223193d9b166ab62961290be3050a9a1
+size 7247829584

model-00033-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:986014e11dbc2ebbd20465f7a7a7558bc7e6640d76753cf6e308ef91a1c73833
+size 7247829584

model-00034-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:70225673b81ed882c341329979749df9ebf3d76397bc062cd8d45985578b6aaa
+size 7247829584

model-00035-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:496c6fa66c557193f11c3bc8c53da798293ba4dd549055d6d7228a74d8f3f47b
+size 7247829584

model-00036-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8f27a414abc0c63d17dd384efc22d6f567d9dd717132834e055a05a015fcbaf4
+size 1057352200

model-00037-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:96d1b08006c13931f68c4974cf90b48eb027da6da70bd79e83d436e04bf1c5de
+size 7247829008

model-00038-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5b0a8a47f60b4471bdb316d4019f14cd6361d4e69619ff52d0ba03246278de3f
+size 7247829584

model-00039-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e32ce985b17809bc2702adfff736c4dba28162dd3c014b356e7903e0ef32cae5
+size 7247829584

model-00040-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cd189895394021bc96f86ec9aaf81850f1bd5d832df94c4a96aff4b765598f8d
+size 7247829584

model-00041-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3973686874599c9c0b886b43387a0b05f073752aed07de84db823f709ea8ae87
+size 7247829584

model-00042-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50841884bd8a563a18ae65c946031fb7bde5efaf4cd76cbd2206a3befa4a804d
+size 1056967864

model-00043-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f3fe90acd2f05b6a5e3e7c5f386b9022f88a22f0014b407bb3338134360699c
+size 7247829008

model-00044-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8f5dfcc9a31b58f575f9f76d413aaff20ae71ff007f6a914082bd448ffbe5d9c
+size 7247829584

model-00045-of-00099.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d54a762394c124e668f9591ce11f687caaa9d63ea986f95ffb11ca193ceed30a
+size 7247829584