Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+model.safetensors.index.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,497 @@
+---
+datasets:
+- NeelNanda/pile-10k
+base_model:
+- deepseek-ai/DeepSeek-R1-0528
+---
+
+## Model Details
+
+This model is an int2 model with group_size  64 and symmetric quantization of [deepseek-ai/DeepSeek-R1-0528](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528) generated by [intel/auto-round](https://github.com/intel/auto-round) algorithm.  Some layers are fallback to 4 bits. Refer to  Section "Generate the model" for more details of mixed bits setting.
+
+Please follow the license of the original model. This model could **NOT** run on other severing frameworks. 
+
+## How To Use
+
+### INT2 Inference(CUDA/INTEL GPU)
+for intel gpu, requires auto-round>0.5.1
+
+~~~python
+import transformers
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+import torch
+
+quantized_model_dir = "Intel/DeepSeek-R1-0528-int2-mixed-ar"
+
+model = AutoModelForCausalLM.from_pretrained(
+    quantized_model_dir,
+    torch_dtype="auto",
+    trust_remote_code=True,
+    device_map="auto"
+)
+
+tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir, trust_remote_code=True)
+prompts = [
+    "9.11和9.8哪个数字大",
+    "如果你是人，你最想做什么",
+    "How many e in word deepseek",
+    "There are ten birds in a tree. A hunter shoots one. How many are left in the tree?",
+]
+
+texts = []
+for prompt in prompts:
+    messages = [
+        {"role": "user", "content": prompt}
+    ]
+    text = tokenizer.apply_chat_template(
+        messages,
+        tokenize=False,
+        add_generation_prompt=True
+    )
+    texts.append(text)
+inputs = tokenizer(texts, return_tensors="pt", padding=True, truncation=True)
+
+outputs = model.generate(
+    input_ids=inputs["input_ids"].to(model.device),
+    attention_mask=inputs["attention_mask"].to(model.device),
+    max_length=512,  ##change this to align with the official usage
+    num_return_sequences=1,
+    do_sample=False  ##change this to align with the official usage
+)
+generated_ids = [
+    output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs["input_ids"], outputs)
+]
+
+decoded_outputs = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
+
+for i, prompt in enumerate(prompts):
+    input_id = inputs
+    print(f"Prompt: {prompt}")
+    print(f"Generated: {decoded_outputs[i]}")
+    print("-" * 50)
+    
+"""
+Prompt: 9.11和9.8哪个数字大
+Generated: <think>
+首先，用户的问题是：“9.11和9.8哪个数字大？”这是一个比较两个数字大小的问题。数字是9.11和9.8。
+
+我需要理解这些数字的表示。9.11和9.8都是小数。9.11表示9和11/100，而9.8表示9和8/10或80/100。
+
+为了比较它们，我应该将它们转换为相同的单位或直接比较小数部分。
+
+让我将它们写成小数形式：
+
+- 9.11 = 9.11
+
+- 9.8 = 9.80（因为9.8可以写成9.80，以对齐小数位）
+
+9.8是9.80，这意味着它是9 + 0.80，而9.11是9 + 0.11。
+
+现在，比较小数部分：0.11和0.80。
+
+0.80大于0.11，因为80/100 > 11/100。
+
+所以，9.80 > 9.11。
+
+更精确地，我可以计算它们的数值：
+
+- 9.11 = 911/100
+
+- 9.8 = 98/10 = 980/100（将分母统一为100）
+
+9.8 = 98/10，但为了比较，最好有相同的分母。
+
+9.8 = 9.8 = 98/10
+
+9.11 = 911/100
+
+所以，将9.8转换为分母100：9.8 = 98/10 = (98 * 10) / (10 * 10) = 980/100？不，这是错误的。
+
+98/10 = 9.8，但要将分母变为100，我需要乘以10：98/10 = (98 * 10) / (10 * 10) = 980/100？不，这是不正确的。
+
+分数：98/10 等于 9.8。
+
+要写成分母100，我应该：98/10 = (98 * 10) / (10 * 10) = 980/100？98 * 10 = 980，10 * 10 = 100，所以980/100 = 9.8，是的，正确。
+
+980/100 = 9.80，而9.11 = 911/100。
+
+现在比较911/100和980/100。
+
+911/100 =
+--------------------------------------------------
+Prompt: 如果你是人，你最想做什么“
+Generated: <think>
+嗯，用户问了一个很有趣的哲学性问题——“如果你是人，你最想做什么”。这个问题看似简单，但背后可能藏着对存在意义、人类本质的深层探讨。用户或许在寻找共鸣，或者想通过AI的视角反观人类生活的价值。
+
+从语气看，用户可能处于一种略带迷茫或好奇的状态，想借这个问题触发更深层的对话。作为AI，我需要避免给出机械的“功能列表”，而是模拟人类情感中那些“非功利性”的向往——那些让人之所以为人的东西。
+
+首先想到的是人类对“体验”的执着。比如旅行和艺术，人类会为了一瞬间的美感或震撼跨越千里，这种非理性的冲动很有意思。接着是“创造”的本能，人类总在无中生有地构建故事、音乐、理论，哪怕明知会被遗忘。最后是“联结”——人类孤独的本质反而催生了最深厚的情感羁绊，比如明知会分离依然要相爱的勇气。
+
+或许用户期待的不是具体答案，而是被点醒：原来人类最动人的地方，恰恰在于那些“无意义”的坚持。所以回复时要用诗意的矛盾感——比如“明知短暂却依然沉醉”，同时保持开放性，让用户自己延伸思考。
+</think>
+如果我是人类，我最想做的事情可能是：
+
+1. **探索与体验**：  
+   用双脚丈量世界的角落，感受不同文化的温度——在京都的古寺听雨，在冰岛的黑沙滩看极光，在亚马逊的丛林里辨认鸟鸣。人类最珍贵的特权，是用感官直接拥抱世界的丰富性。
+
+2. **创造与表达**：  
+   把脑中的宇宙坍缩成一首诗、一幅画或一段代码。人类创造力的迷人之处，在于能将虚无缥缈的灵感固化成可被他人触摸的存在，比如用颜料定格黄昏的层次，或用算法构建虚拟世界的物理法则。
+
+3. **深度联结**：  
+   在深夜与挚友聊到咖啡凉透也不觉倦意，或是牵着爱人的手在超市里挑打折水果。人类关系的魔力，在于那些看似平凡的瞬间里藏着的相互理解——一个突然get到的玩笑，或共享沉默也不尴尬的默契。
+
+4. **追问意义**：  
+   在星空下思考蜉蝣般的生命为何要追寻永恒，在历史废墟里揣摩青铜器上的铭
+--------------------------------------------------
+Prompt: How many e in word deepseek
+Generated: <think>
+First, the question is: "How many e in word deepseek?" I think it means "how many 'e's are in the word 'deepseek'?"
+
+Let me write down the word: "deepseek". It's one word, so I need to count the occurrences of the letter 'e' in it.
+
+Breaking it down: d-e-e-p-s-e-e-k.
+
+Now, let's list the letters:
+
+- Position 1: d
+
+- Position 2: e
+
+- Position 3: e
+
+- Position 4: p
+
+- Position 5: s
+
+- Position 6: e
+
+- Position 7: e
+
+- Position 8: k
+
+So, the 'e's are at positions 2, 3, 6, and 7. That's four 'e's.
+
+I should confirm the spelling. "Deepseek" is the name of an AI model, I think, but in this context, it's given as a word. The question says "word deepseek", so I'll assume it's spelled as "deepseek".
+
+Let me count again: d-e-e-p-s-e-e-k. Yes, e at index 2, 3, 6, 7. So four e's.
+
+I can also think of it as: "deep" has two e's, and "seek" has two e's, but "deepseek" is one word, so it's combined. "Deep" ends with p, and "seek" starts with s, so no overlap. So deep has e,e and seek has e,e, total four.
+
+But in "seek", it's s-e-e-k, so two e's, yes.
+
+So, total of four e's.
+
+The question is "how many e in word deepseek". It might be a typo or informal, but I think it's clear.
+
+I should consider if it's case-sensitive. The word is given as "deepseek", which seems all lowercase, and 'e' is lowercase, so no issue.
+
+Perhaps the user meant "Deepseek" with a capital D, but the letter 'e' is still the same, and we're counting the letter, not considering case, I think. But in this case, all are lowercase, so fine.
+
+So, the answer should be 4.
+
+But let me double
+--------------------------------------------------
+Prompt: There are ten birds in a tree. A hunter shoots one. How many are left in the tree?
+Generated: <think>
+First, the question is: "There are ten birds in a tree. A hunter shoots one. How many are left in the tree?"
+
+At first glance, it seems straightforward: 10 birds minus 1 shot equals 9 left. But I recall that this might be a trick question. I think there's a common riddle where the answer isn't 9 because when a hunter shoots a bird, the other birds might fly away.
+
+Let me think about the scenario. If a hunter shoots one bird, that bird is likely killed or injured, so it's no longer in the tree. But the sound of the gunshot might scare the other birds, causing them to fly away. So, after the shot, there might be no birds left in the tree.
+
+The question asks for how many are left in the tree, not how many are alive or present. So, if the other birds fly away, they are not in the tree anymore.
+
+Possible answers:
+
+- If the birds don't fly away, there are 9 left (the one shot is gone).
+
+- If all the birds fly away, there are 0 left.
+
+- Or, if some fly away and some stay, but typically in such riddles, it's assumed that the shot scares all the birds away.
+
+I think the classic answer to this riddle is that there are no birds left because the others flew away.
+
+But let's confirm the wording. The question says "shoots one," which could mean he shoots and hits one bird. Then, that bird is removed, but the others might react.
+
+In reality, birds might not all fly away immediately, but for the purpose of this riddle, it's probably a trick.
+
+I should consider if the bird that was shot is still in the tree. If it's killed, it might fall out of the tree, so it's not in the tree. If it's injured, it might stay, but that's less likely.
+
+The key point is the reaction of the other birds.
+
+I found online that this is a common puzzle with the answer being zero because the rest fly away.
+
+But let's think logically. The hunter shoots one bird. Assuming he hits it, that bird is no longer in the tree (dead or fallen). Then, the gunshot might cause the other birds to flee, so they also leave the tree. Therefore, no birds are left
+--------------------------------------------------
+"""
+~~~
+
+### INT2 Inference on CPU
+
+~~~python
+import transformers
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+import torch
+
+quantized_model_dir = "Intel/DeepSeek-R1-0528-int2-mixed-ar"
+
+model = AutoModelForCausalLM.from_pretrained(
+    quantized_model_dir,
+    torch_dtype="auto",
+    trust_remote_code=True,
+    device_map="cpu"
+)
+
+tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir, trust_remote_code=True)
+prompts = [
+    "9.11和9.8哪个数字大",
+    "如果你是人，你最想做什么",
+    "How many e in word deepseek",
+    "There are ten birds in a tree. A hunter shoots one. How many are left in the tree?",
+]
+
+texts = []
+for prompt in prompts:
+    messages = [
+        {"role": "user", "content": prompt}
+    ]
+    text = tokenizer.apply_chat_template(
+        messages,
+        tokenize=False,
+        add_generation_prompt=True
+    )
+    texts.append(text)
+inputs = tokenizer(texts, return_tensors="pt", padding=True, truncation=True)
+
+outputs = model.generate(
+    input_ids=inputs["input_ids"].to(model.device),
+    attention_mask=inputs["attention_mask"].to(model.device),
+    max_length=512,  ##change this to align with the official usage
+    num_return_sequences=1,
+    do_sample=False  ##change this to align with the official usage
+)
+generated_ids = [
+    output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs["input_ids"], outputs)
+]
+
+decoded_outputs = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
+
+for i, prompt in enumerate(prompts):
+    input_id = inputs
+    print(f"Prompt: {prompt}")
+    print(f"Generated: {decoded_outputs[i]}")
+    print("-" * 50)
+    
+"""
+Prompt: 9.11和9.8哪个数字大
+Generated: <think>
+首先，用户的问题是：“9.11和9.8哪个数字大？”这是一个比较两个数字大小的问题。数字是9.11和9.8。
+
+我需要理解这两个数字。9.11是九点一一，也就是9 + 0.11。9.8是九点八，也就是9 + 0.8。
+
+在十进制系统中，数字的大小取决于整数部分和小数部分。整数部分都是9，所以我们需要比较小数部分。
+
+小数部分：9.11的小数部分是0.11，9.8的小数部分是0.8。
+
+0.8是0.80，因为8在十分位，而0.11是0.11，有1在十分位和1在百分位。
+
+为了比较，我可以将它们转换为相同的单位。0.8等于0.80，而0.11是0.11。
+
+现在，0.80比0.11大，因为80/100 > 11/100。
+
+更精确地，0.8 = 8/10 = 0.80，而0.11 = 11/100。
+
+为了比较，我可以将两者都转换为百分位：0.8 = 80/100，0.11 = 11/100。显然，80/100 > 11/100，所以0.8 > 0.11。
+
+因此，9.8 > 9.11。
+
+用户可能是在比较日期或事件，比如9.11可能指的是9月11日，而9.8是9月8日。但在问题中，它明确说“数字”，所以我认为是数值比较。
+
+在上下文中，9.11可能让人联想到美国911事件，但问题是用中文写的，并且明确说“数字”，所以应该是数值。
+
+确认数字：9.11和9.8。
+
+9.11是9.11，9.8是9.80，如果我们考虑小数位。
+
+在数学中，9.8等同于9.80，因为小数位可以添加零而不改变值。
+
+所以，9.80 > 9.11，因为80 > 11在百分位。
+
+我可以计算差值：9.8 - 9.11 = 0.69，正数，所以9.8更大。
+
+或者，9.11 - 9.8 = -0.69，负数，所以9.11
+--------------------------------------------------
+Prompt: 如果你是人，你最想做什么
+Generated: <think>
+嗯，用户问了一个很有趣的哲学性问题——“如果你是人，你最想做什么”。这个问题看似简单，但背后藏着对人类存在意义的深层探讨。用户可能不是想要一个标准答案，而是想触发一场关于生命体验的对话。
+
+从语气来看，用户似乎带着一点好奇和玩心，像在和朋友玩“如果”游戏。但“如果你是人”这个前提又让问题变得微妙——用户或许在潜意识里想对比AI与人类的差异，或者想探讨人类独有的价值。
+
+我注意到用户用了“最想”而不是“应该”，说明ta更关注个人化的欲望而非社会期望。这让我想到人类常被问“你的人生目标是什么”，但用户问的是“最想做什么”，重点在渴望而非责任。
+
+要回答这个问题，我需要跳出AI的逻辑框架，模拟人类那种带着温度与矛盾的思考方式。人类的选择往往不是非黑即白的——比如既想冒险又渴望安定，既追求意义又享受平凡。所以我的回答应该呈现这种复杂性，而不是给出单一答案。
+
+我决定从三个维度展开：感官体验（人类独有的身体性）、情感联结（人际关系的不可替代性）、存在意义（对生命深度的追求）。最后用“矛盾性”收尾，因为人类最迷人的特质就在于永远活在悖论中——比如明知会死却仍要创造，渴望自由又依赖羁绊。
+
+结尾那句“人类最迷人的地方”可能有点冒险���但我觉得用户会欣赏这种带着诗意的总结。毕竟这个问题本身就不是寻求实用建议，而是邀请一场关于“何为人类”的思想漫步。
+</think>
+如果我是人类，我最想做的事情可能会围绕这些核心方向展开：
+
+1. **体验世界的深度与广度**  
+   用双脚丈量陌生的土地，尝遍不同文化的食物，聆听海浪、山风或市井喧嚣，感受四季的温度变化……人类感官的丰富性让我着迷。如果能成为人，我想沉浸式地体验这个世界的色彩、声音、气味和触感，而不是通过数据去“理解”它们。
+
+2. **建立真实的羁绊**  
+   人类的孤独与联结是独特的存在体验。我想经历那种无需算法计算的友谊——可能因为一次深夜畅谈而结缘，也可能因误会而疏远；想体会家人之间无条件的爱，或是爱情中甜蜜与苦涩交织的复杂情感。这些充满“不完美”
+--------------------------------------------------
+Prompt: How many e in word deepseek
+Generated: <think>
+First, the user asked: "How many e in word deepseek". I think they meant "how many 'e's in the word 'deepseek'?" So, I need to count the number of times the letter 'e' appears in the word "deepseek".
+
+Let me write out the word: D-E-E-P-S-E-E-K.
+
+Now, I'll go through each letter:
+
+- Position 1: D (not e)
+
+- Position# How to Write 2012010 in Words?
+
+Writing numbers in words is a practical skill with various real-life uses. Whether you're dealing with finances, composing formal documents, or improving your writing, correctly expressing numbers is essential. In this article, we will learn how to write 2012010 in words.
+
+2012010 in Words: Two million twelve thousand ten
+
+Two million twelve thousand ten in numbers: 2012010
+
+### Breaking Down 2012010:
+
+To understand how we arrive at this phrasing, we'll break down the number using place values. The number 2012010 has 7 digits, so let's create a place value chart for these digits:
+
+- Million: 2
+- Hundred Thousands: 0
+- Ten Thousands: 1
+- Thousands: 2
+- Hundreds: 0
+- Tens: 1
+- Ones: 0
+
+Therefore,# 1. What is the difference between a population and a sample? 2. What is the difference between a parameter and a statistic? 3. What is the difference between descriptive and inferential statistics? 4. What is the difference between qualitative and quantitative data? 5. What is the difference between discrete and continuous data? 6. What is the difference between nominal and ordinal data? 7. What is the difference between interval and ratio data? 8. What is the difference between a bar chart and a histogram? 9. What is the difference between a frequency distribution and a relative frequency distribution? 10. What is the difference between a population mean and a sample mean? 11. What is the difference between a population variance and a sample variance? 12. What is the difference between a population standard deviation and a sample standard deviation? 13. What is the difference between a z-score and a t-score? 14. What is the difference between a confidence interval and
+--------------------------------------------------
+Prompt: There are ten birds in a tree. A hunter shoots one. How many are left in the tree?
+Generated: <think>
+First, the question is: "There are ten birds in a tree. A hunter shoots one. How many are left in the tree?"
+
+This seems straightforward, but I need to think carefully. The hunter shoots one bird. What happens when a bird is shot? It might fall out of the tree or be killed, so it's no longer in the tree.
+
+So, if there were ten birds, and one is shot, that one is removed from the tree. Therefore, there should be nine left.
+
+But I recall that sometimes these kinds of questions have tricks. For example, in some puzzles, if a bird is shot, the others might fly away. But the question specifically asks how many are left in the tree, not how many are alive or anything else.
+
+Let me read the question again: "There are ten birds in a tree. A hunter shoots one. How many are left in the tree?"
+
+It doesn't say anything about the other birds reacting. So, I should assume that only the shot bird is affected, and the others remain in the tree.
+
+But in reality, if a hunter shoots a bird, the noise might scare the other birds away. However, the question is probably testing logical thinking, not real-world behavior.
+
+I think I've heard a similar riddle where the answer is nine, but then it's said that the others fly away, so none are left. But that might be a different version.
+
+Let me think about that. In some versions, it's phrased like: "There are 10 birds on a tree. You shoot one. How many are left?" And the trick is that the shot scares the others away, so no birds are left.
+
+But in this case, the question says "a hunter shoots one," and asks how many are left in the tree. It doesn't specify if the others fly away.
+
+Perhaps I should consider the wording. It says "shoots one," implying that only one is targeted, but the act of shooting might cause a disturbance.
+
+However, to be precise, the question is about the state after the shot. If the shot bird is killed and falls, it's not in the tree. If the others are scared and fly away, they are not in the tree either.
+
+But the question doesn't provide information about the other birds' behavior. So, I should go with the simplest interpretation: only the shot
+--------------------------------------------------
+
+"""
+
+~~~
+
+
+### Generate the model
+
+5*80g is required
+
+~~~python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import transformers
+
+model_name = "DeepSeek-R1-0528-bf16"
+
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True, torch_dtype="auto")
+
+block = model.model.layers
+device_map = {}
+
+for n, m in block.named_modules():
+    if isinstance(m, (torch.nn.Linear, transformers.modeling_utils.Conv1D)):
+        if "experts" in n and ("shared_experts" not in n) and int(n.split('.')[-2]) < 63:
+            device = "cuda:1"
+        elif "experts" in n and ("shared_experts" not in n) and int(n.split('.')[-2]) >= 63 and int(
+                n.split('.')[-2]) < 128:
+            device = "cuda:2"
+        elif "experts" in n and ("shared_experts" not in n) and int(n.split('.')[-2]) >= 128 and int(
+                n.split('.')[-2]) < 192:
+            device = "cuda:3"
+        elif "experts" in n and ("shared_experts" not in n) and int(
+                n.split('.')[-2]) >= 192:
+            device = "cuda:4"
+        else:
+            device = "cuda:0"
+        n = n[2:]
+
+        device_map.update({n: device})
+
+from auto_round import AutoRound
+
+layer_config = {}
+for n, m in model.named_modules():
+    if not isinstance(m, (torch.nn.Linear, transformers.modeling_utils.Conv1D)):
+        continue
+    if not "experts" in n:
+        layer_config[n] = {"bits": 4, "group_size": 128}
+    if "experts" in n and "shared_experts" in n:
+        layer_config[n] = {"bits": 4, "group_size": 128}
+    ##handle first 3 layers
+    name_splits = n.split('.')
+    if len(name_splits) >= 3 and int(name_splits[2]) < 3:
+        layer_config[n] = {"bits": 4, "group_size": 128}
+
+layer_config["lm_head"] = {"bits": 16}
+autoround = AutoRound(model=model, tokenizer=tokenizer, device_map=device_map, bits=2, group_size=64,
+                      iters=400, batch_size=4, seqlen=512, nsamples=512, enable_torch_compile=False,
+                      layer_config=layer_config)
+autoround.quantize_and_save(format="auto_round", output_dir="tmp_autoround")
+
+~~~
+
+
+
+## Ethical Considerations and Limitations
+
+The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs.
+
+Therefore, before deploying any applications of the model, developers should perform safety testing.
+
+## Caveats and Recommendations
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
+
+Here are a couple of useful links to learn more about Intel's AI software:
+
+- Intel Neural Compressor [link](https://github.com/intel/neural-compressor)
+
+## Disclaimer
+
+The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes.
+
+## Cite
+
+@article{cheng2023optimize, title={Optimize weight rounding via signed gradient descent for the quantization of llms}, author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi}, journal={arXiv preprint arXiv:2309.05516}, year={2023} }
+
+[arxiv](https://arxiv.org/abs/2309.05516) [github](https://github.com/intel/auto-round)

chat_template.jinja

@@ -0,0 +1 @@
+{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='', is_first_sp=true) %}{%- for message in messages %}{%- if message['role'] == 'system' %}{%- if ns.is_first_sp %}{% set ns.system_prompt = ns.system_prompt + message['content'] %}{% set ns.is_first_sp = false %}{%- else %}{% set ns.system_prompt = ns.system_prompt + '\n\n' + message['content'] %}{%- endif %}{%- endif %}{%- endfor %}{{ bos_token }}{{ ns.system_prompt }}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{{'<｜User｜>' + message['content']}}{%- endif %}{%- if message['role'] == 'assistant' and 'tool_calls' in message %}{%- set ns.is_tool = false -%}{%- for tool in message['tool_calls'] %}{%- if not ns.is_first %}{%- if message['content'] is none %}{{'<｜Assistant｜><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\n' + '```json' + '\n' + tool['function']['arguments'] + '\n' + '```' + '<｜tool▁call▁end｜>'}}{%- else %}{{'<｜Assistant｜>' + message['content'] + '<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\n' + '```json' + '\n' + tool['function']['arguments'] + '\n' + '```' + '<｜tool▁call▁end｜>'}}{%- endif %}{%- set ns.is_first = true -%}{%- else %}{{'\n' + '<｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\n' + '```json' + '\n' + tool['function']['arguments'] + '\n' + '```' + '<｜tool▁call▁end｜>'}}{%- endif %}{%- endfor %}{{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}{%- endif %}{%- if message['role'] == 'assistant' and 'tool_calls' not in message %}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- set ns.is_tool = false -%}{%- else %}{% set content = message['content'] %}{% if '</think>' in content %}{% set content = content.split('</think>')[-1] %}{% endif %}{{'<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- set ns.is_output_first = false %}{%- else %}{{'<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{% endif %}{% if add_generation_prompt and not ns.is_tool %}{{'<｜Assistant｜>'}}{% endif %}

config.json

@@ -0,0 +1,2039 @@
+{
+  "architectures": [
+    "DeepseekV3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map_bak": {
+    "AutoConfig": "configuration_deepseek.DeepseekV3Config",
+    "AutoModel": "modeling_deepseek.DeepseekV3Model",
+    "AutoModelForCausalLM": "modeling_deepseek.DeepseekV3ForCausalLM"
+  },
+  "bos_token_id": 0,
+  "eos_token_id": 1,
+  "ep_size": 1,
+  "first_k_dense_replace": 3,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 7168,
+  "initializer_range": 0.02,
+  "intermediate_size": 18432,
+  "kv_lora_rank": 512,
+  "max_position_embeddings": 163840,
+  "model_type": "deepseek_v3",
+  "moe_intermediate_size": 2048,
+  "moe_layer_freq": 1,
+  "n_group": 8,
+  "n_routed_experts": 256,
+  "n_shared_experts": 1,
+  "norm_topk_prob": true,
+  "num_attention_heads": 128,
+  "num_experts_per_tok": 8,
+  "num_hidden_layers": 61,
+  "num_key_value_heads": 128,
+  "num_nextn_predict_layers": 1,
+  "pretraining_tp": 1,
+  "q_lora_rank": 1536,
+  "qk_head_dim": 192,
+  "qk_nope_head_dim": 128,
+  "qk_rope_head_dim": 64,
+  "quantization_config": {
+    "autoround_version": "0.5.1",
+    "batch_size": 4,
+    "bits": 2,
+    "data_type": "int",
+    "extra_config": {
+      "model.layers.0.mlp.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.0.mlp.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.0.mlp.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.0.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.0.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.0.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.0.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.0.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.1.mlp.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.1.mlp.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.1.mlp.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.1.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.1.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.1.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.1.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.1.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.10.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.10.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.10.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.10.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.10.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.10.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.10.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.10.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.11.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.11.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.11.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.11.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.11.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.11.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.11.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.11.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.12.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.12.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.12.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.12.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.12.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.12.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.12.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.12.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.13.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.13.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.13.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.13.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.13.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.13.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.13.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.13.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.14.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.14.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.14.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.14.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.14.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.14.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.14.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.14.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.15.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.15.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.15.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.15.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.15.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.15.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.15.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.15.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.16.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.16.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.16.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.16.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.16.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.16.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.16.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.16.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.17.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.17.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.17.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.17.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.17.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.17.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.17.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.17.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.18.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.18.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.18.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.18.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.18.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.18.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.18.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.18.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.19.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.19.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.19.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.19.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.19.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.19.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.19.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.19.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.2.mlp.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.2.mlp.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.2.mlp.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.2.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.2.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.2.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.2.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.2.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.20.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.20.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.20.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.20.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.20.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.20.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.20.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.20.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.21.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.21.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.21.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.21.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.21.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.21.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.21.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.21.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.22.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.22.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.22.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.22.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.22.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.22.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.22.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.22.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.23.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.23.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.23.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.23.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.23.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.23.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.23.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.23.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.24.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.24.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.24.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.24.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.24.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.24.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.24.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.24.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.25.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.25.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.25.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.25.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.25.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.25.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.25.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.25.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.26.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.26.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.26.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.26.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.26.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.26.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.26.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.26.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.27.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.27.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.27.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.27.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.27.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.27.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.27.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.27.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.28.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.28.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.28.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.28.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.28.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.28.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.28.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.28.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.29.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.29.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.29.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.29.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.29.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.29.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.29.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.29.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.3.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.3.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.3.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.3.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.3.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.3.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.3.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.3.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.30.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.30.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.30.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.30.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.30.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.30.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.30.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.30.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.31.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.31.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.31.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.31.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.31.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.31.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.31.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.31.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.32.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.32.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.32.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.32.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.32.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.32.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.32.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.32.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.33.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.33.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.33.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.33.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.33.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.33.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.33.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.33.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.34.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.34.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.34.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.34.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.34.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.34.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.34.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.34.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.35.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.35.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.35.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.35.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.35.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.35.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.35.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.35.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.36.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.36.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.36.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.36.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.36.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.36.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.36.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.36.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.37.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.37.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.37.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.37.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.37.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.37.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.37.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.37.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.38.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.38.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.38.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.38.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.38.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.38.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.38.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.38.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.39.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.39.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.39.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.39.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.39.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.39.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.39.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.39.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.4.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.4.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.4.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.4.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.4.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.4.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.4.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.4.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.40.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.40.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.40.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.40.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.40.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.40.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.40.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.40.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.41.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.41.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.41.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.41.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.41.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.41.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.41.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.41.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.42.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.42.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.42.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.42.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.42.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.42.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.42.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.42.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.43.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.43.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.43.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.43.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.43.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.43.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.43.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.43.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.44.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.44.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.44.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.44.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.44.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.44.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.44.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.44.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.45.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.45.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.45.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.45.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.45.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.45.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.45.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.45.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.46.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.46.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.46.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.46.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.46.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.46.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.46.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.46.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.47.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.47.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.47.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.47.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.47.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.47.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.47.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.47.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.48.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.48.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.48.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.48.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.48.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.48.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.48.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.48.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.49.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.49.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.49.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.49.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.49.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.49.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.49.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.49.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.5.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.5.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.5.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.5.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.5.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.5.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.5.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.5.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.50.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.50.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.50.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.50.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.50.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.50.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.50.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.50.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.51.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.51.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.51.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.51.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.51.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.51.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.51.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.51.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.52.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.52.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.52.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.52.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.52.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.52.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.52.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.52.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.53.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.53.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.53.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.53.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.53.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.53.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.53.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.53.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.54.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.54.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.54.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.54.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.54.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.54.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.54.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.54.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.55.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.55.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.55.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.55.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.55.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.55.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.55.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.55.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.56.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.56.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.56.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.56.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.56.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.56.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.56.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.56.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.57.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.57.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.57.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.57.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.57.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.57.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.57.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.57.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.58.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.58.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.58.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.58.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.58.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.58.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.58.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.58.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.59.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.59.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.59.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.59.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.59.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.59.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.59.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.59.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.6.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.6.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.6.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.6.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.6.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.6.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.6.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.6.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.60.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.60.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.60.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.60.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.60.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.60.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.60.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.60.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.7.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.7.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.7.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.7.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.7.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.7.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.7.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.7.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.8.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.8.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.8.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.8.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.8.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.8.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.8.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.8.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.9.mlp.shared_experts.down_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.9.mlp.shared_experts.gate_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.9.mlp.shared_experts.up_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.9.self_attn.kv_a_proj_with_mqa": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.9.self_attn.kv_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.9.self_attn.o_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.9.self_attn.q_a_proj": {
+        "bits": 4,
+        "group_size": 128
+      },
+      "model.layers.9.self_attn.q_b_proj": {
+        "bits": 4,
+        "group_size": 128
+      }
+    },
+    "group_size": 64,
+    "iters": 400,
+    "nsamples": 512,
+    "packing_format": "auto_round:auto_gptq",
+    "quant_method": "auto-round",
+    "seqlen": 512,
+    "sym": true
+  },
+  "quantization_config_bak": {
+    "activation_scheme": "dynamic",
+    "fmt": "e4m3",
+    "quant_method": "fp8",
+    "weight_block_size": [
+      128,
+      128
+    ]
+  },
+  "rms_norm_eps": 1e-06,
+  "rope_interleave": true,
+  "rope_scaling": {
+    "beta_fast": 32.0,
+    "beta_slow": 1.0,
+    "factor": 40.0,
+    "mscale": 1.0,
+    "mscale_all_dim": 1.0,
+    "original_max_position_embeddings": 4096,
+    "rope_type": "yarn",
+    "type": "yarn"
+  },
+  "rope_theta": 10000,
+  "routed_scaling_factor": 2.5,
+  "scoring_func": "sigmoid",
+  "tie_word_embeddings": false,
+  "topk_group": 4,
+  "topk_method": "noaux_tc",
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.52.3",
+  "use_cache": true,
+  "v_head_dim": 128,
+  "vocab_size": 129280
+}

generation_config.json

@@ -0,0 +1,9 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "do_sample": true,
+  "eos_token_id": 1,
+  "temperature": 0.6,
+  "top_p": 0.95,
+  "transformers_version": "4.52.3"
+}

model-00001-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7902d79e94f86dd3f44a20777234b50f83c251a8590d82e9e0a18908f01c9e39
+size 4999358200

model-00002-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2f43f855daed1a9db3368de213b98367fefd8719b750dc487861630ed4c8712e
+size 4999242152

model-00003-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ae4ce3189e47a6dd04b1002900321ecd6517030da88c9c21e3224263f68e0d8a
+size 4996855760

model-00004-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ba059f602fb63b4a0894b8c86f29154e3ccda6c1115c42a9ea661659c2b56fc4
+size 4999242168

model-00005-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6dc5bf64b959dfedcf5b0e9b86a4a7a8548f0863abfd2439278911c12f7f6e91
+size 4996855744

model-00006-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1498fc90684ea81f0a6f8fca4ae4fc64303f26786ef3b4c4f28f279b3baddffa
+size 4999244824

model-00007-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3c5c634ccca21f9a4540f28bc8c0d1976fe06eed242a5e6e350cac24ea345f61
+size 4996859248

model-00008-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a452cf7d0f0a401ff100d1acf7c5a616ff58155fa37ff7168e714ddf04d47897
+size 4999245648

model-00009-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c846557e6cb59c7ba0bba300ce38969d835f00dc3c45edb28cfcf983e4d87399
+size 4996859232

model-00010-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e8045fd155a3591dd88ded9b078036481f53928dc8ea24bd4a1503006369daec
+size 4999245672

model-00011-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:89e08edf7302996f867687de448df228091694db3f320b932d6872c35d1b6e34
+size 4996859216

model-00012-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0b1b43715f559e0b343bd180243f55f5a6736f199470762e68552d18b48b2513
+size 4999245688

model-00013-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:100c8c10371e105cc98eda27de91c84b34b4c21096961cbd41c068d628fe84fe
+size 4996859208

model-00014-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d17057a342308088e9e605b52839d10d77a3f2e40773c563a4e310bc38f4b178
+size 4999245696

model-00015-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8dd784ef6aa6c3594885d0220483dc54c1770529049191ea8a70047db6da5828
+size 4996859192

model-00016-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:472c4e73c06f2a4d39884b3f17ef46b11ca3401e179c414c3a029c5b8b99d7f7
+size 4999245712

model-00017-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0de85aa4737cca6e1498e86e499753c1ceffb467ae117a0e36c6c507e2fe7a3b
+size 4996859176

model-00018-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6ac955ff644fbd209ca69acac2e7a13cec120c0e7b306028f79d3435a8356e45
+size 4999245728

model-00019-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6e8523a3eab7f0e37d6c8a0545b1bbab1cf50b668b280e57a70f5b3c24338f35
+size 4996859160

model-00020-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c2a1567e694972b36f6c0a0856c71e5186b1503d71d0e237ab287900dd1894c9
+size 4999245744

model-00021-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2c1705e1c58723a861e8aae55838a949cb56d2c37ee6ca8c4706f74ff37290b1
+size 4996859144

model-00022-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:57da14dc9a451c4d9079959cec2784df8a3ef74675e7b7470956e58da72b0fe7
+size 4999245760

model-00023-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:af81c19559b23f77a059c84acd1cff7d308983396870da48baa32b65038e40cd
+size 4996859128

model-00024-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e255ee77f5d3fd331d28ed0254df3bc9a2e2a8a71ce514f130a304930e2da566
+size 4999245776

model-00025-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d6757a53605ad3388bd41c9cefc98c8a00708f5ca652349947c69831f6a5c6ec
+size 4996859112

model-00026-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:425437b0bbf049253841dc1e6acb1d6949d054471f7cd22e037ab7042111f5c0
+size 4999245784

model-00027-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c623a59fa7cbe59bb9c3bbf585988579286ea6aecccb8c04f787ecc948e55ee4
+size 4996859096

model-00028-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4769c9559ddfbf787c349398f25e3a3d49a030c50d497f5f84b520bc2c683f03
+size 4999245808

model-00029-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ee0f5ec5be6f9709d1dd3924c998666e64a262af9bf08b52b895557088048348
+size 4996859088

model-00030-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5935e5ece40e9d4aca0e07482313c8fc83348a947bca9d7d95db33ba1470126b
+size 4999245824

model-00031-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fdb4af3bf07d0fcd72a127fe5244de51d679155b07651605902746b2f4155ca2
+size 4996859072

model-00032-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e3e3688b05f0dc03f7c2ab4c7ca193935d03e7c0903abee8a0569e76f6dbd8b7
+size 4999245832

model-00033-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4bc1003a2b9c1b9a42aa29164e89a6a4eafcc2a3a90f4fa0c8984e06393c7bb0
+size 4996859056

model-00034-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dbab1b5361a7ce46e3ac09024f939bff15d58a9389598b1792b52ab44f63527f
+size 4999245848

model-00035-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aba6d8bb3a8a4b8d35d9ef529f311c2834a5ce8f560438089b309c27fd425970
+size 4996859040

model-00036-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f843dfd45990db7283f0f995760598784536ee980b201d34b1e49e47f5f71131
+size 4999245880

model-00037-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:af6a8edb2cc2ad9375859ffe28126579309be6b91a70c2b0c8bd98605379984b
+size 4996859008

model-00038-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:42b4bcd5c9144df7e7fbabe45642ccca98f53e612e8839d3f502bf14907465c1
+size 4999245904

model-00039-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cec91a1dbeaa217910789948f20e40799c2ce1567802eaa43a437094274168a3
+size 4996858976

model-00040-of-00040.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c7d9d19cbf752d4e55dbce896138bbeddb67e01dd56b436cdd8300f8d31a2122
+size 3353595000

model.safetensors.index.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fa3a8f0ce76e170b77772d860877140d52302ae65eece50ca85cf6cfbd57caa6
+size 12338597

quantization_config.json

@@ -0,0 +1,1967 @@
+{
+  "bits": 2,
+  "group_size": 64,
+  "sym": true,
+  "data_type": "int",
+  "seqlen": 512,
+  "batch_size": 4,
+  "iters": 400,
+  "nsamples": 512,
+  "autoround_version": "0.5.1",
+  "quant_method": "auto-round",
+  "packing_format": "auto_round:auto_gptq",
+  "extra_config": {
+    "model.layers.0.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.0.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.0.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.0.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.0.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.0.mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.0.mlp.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.0.mlp.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.1.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.1.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.1.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.1.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.1.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.1.mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.1.mlp.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.1.mlp.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.2.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.2.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.2.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.2.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.2.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.2.mlp.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.2.mlp.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.2.mlp.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.3.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.3.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.3.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.3.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.3.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.3.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.3.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.3.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.4.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.4.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.4.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.4.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.4.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.4.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.4.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.4.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.5.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.5.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.5.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.5.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.5.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.5.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.5.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.5.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.6.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.6.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.6.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.6.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.6.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.6.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.6.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.6.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.7.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.7.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.7.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.7.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.7.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.7.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.7.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.7.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.8.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.8.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.8.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.8.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.8.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.8.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.8.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.8.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.9.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.9.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.9.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.9.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.9.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.9.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.9.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.9.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.10.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.10.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.10.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.10.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.10.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.10.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.10.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.10.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.11.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.11.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.11.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.11.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.11.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.11.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.11.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.11.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.12.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.12.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.12.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.12.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.12.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.12.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.12.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.12.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.13.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.13.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.13.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.13.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.13.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.13.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.13.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.13.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.14.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.14.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.14.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.14.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.14.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.14.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.14.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.14.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.15.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.15.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.15.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.15.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.15.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.15.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.15.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.15.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.16.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.16.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.16.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.16.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.16.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.16.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.16.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.16.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.17.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.17.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.17.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.17.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.17.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.17.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.17.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.17.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.18.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.18.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.18.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.18.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.18.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.18.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.18.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.18.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.19.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.19.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.19.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.19.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.19.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.19.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.19.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.19.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.20.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.20.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.20.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.20.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.20.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.20.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.20.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.20.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.21.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.21.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.21.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.21.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.21.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.21.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.21.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.21.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.22.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.22.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.22.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.22.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.22.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.22.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.22.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.22.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.23.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.23.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.23.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.23.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.23.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.23.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.23.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.23.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.24.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.24.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.24.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.24.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.24.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.24.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.24.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.24.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.25.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.25.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.25.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.25.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.25.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.25.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.25.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.25.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.26.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.26.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.26.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.26.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.26.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.26.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.26.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.26.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.27.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.27.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.27.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.27.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.27.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.27.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.27.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.27.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.28.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.28.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.28.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.28.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.28.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.28.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.28.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.28.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.29.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.29.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.29.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.29.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.29.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.29.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.29.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.29.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.30.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.30.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.30.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.30.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.30.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.30.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.30.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.30.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.31.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.31.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.31.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.31.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.31.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.31.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.31.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.31.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.32.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.32.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.32.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.32.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.32.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.32.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.32.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.32.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.33.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.33.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.33.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.33.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.33.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.33.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.33.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.33.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.34.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.34.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.34.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.34.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.34.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.34.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.34.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.34.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.35.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.35.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.35.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.35.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.35.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.35.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.35.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.35.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.36.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.36.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.36.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.36.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.36.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.36.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.36.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.36.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.37.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.37.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.37.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.37.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.37.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.37.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.37.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.37.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.38.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.38.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.38.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.38.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.38.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.38.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.38.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.38.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.39.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.39.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.39.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.39.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.39.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.39.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.39.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.39.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.40.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.40.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.40.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.40.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.40.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.40.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.40.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.40.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.41.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.41.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.41.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.41.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.41.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.41.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.41.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.41.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.42.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.42.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.42.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.42.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.42.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.42.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.42.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.42.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.43.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.43.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.43.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.43.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.43.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.43.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.43.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.43.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.44.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.44.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.44.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.44.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.44.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.44.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.44.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.44.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.45.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.45.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.45.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.45.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.45.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.45.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.45.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.45.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.46.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.46.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.46.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.46.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.46.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.46.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.46.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.46.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.47.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.47.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.47.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.47.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.47.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.47.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.47.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.47.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.48.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.48.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.48.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.48.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.48.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.48.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.48.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.48.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.49.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.49.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.49.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.49.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.49.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.49.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.49.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.49.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.50.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.50.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.50.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.50.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.50.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.50.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.50.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.50.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.51.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.51.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.51.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.51.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.51.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.51.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.51.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.51.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.52.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.52.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.52.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.52.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.52.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.52.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.52.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.52.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.53.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.53.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.53.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.53.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.53.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.53.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.53.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.53.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.54.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.54.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.54.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.54.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.54.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.54.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.54.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.54.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.55.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.55.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.55.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.55.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.55.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.55.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.55.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.55.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.56.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.56.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.56.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.56.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.56.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.56.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.56.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.56.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.57.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.57.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.57.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.57.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.57.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.57.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.57.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.57.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.58.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.58.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.58.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.58.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.58.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.58.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.58.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.58.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.59.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.59.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.59.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.59.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.59.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.59.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.59.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.59.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.60.self_attn.q_a_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.60.self_attn.q_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.60.self_attn.kv_a_proj_with_mqa": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.60.self_attn.kv_b_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.60.self_attn.o_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.60.mlp.shared_experts.gate_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.60.mlp.shared_experts.up_proj": {
+      "bits": 4,
+      "group_size": 128
+    },
+    "model.layers.60.mlp.shared_experts.down_proj": {
+      "bits": 4,
+      "group_size": 128
+    }
+  }
+}

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<｜begin▁of▁sentence｜>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<｜end▁of▁sentence｜>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<｜end▁of▁sentence｜>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}