Initial upload: base + CS-chatbot LoRA + supervisor LoRA merged

README.md

@@ -0,0 +1,140 @@
+---
+library_name: transformers
+license: mit
+base_model: microsoft/Phi-4-reasoning
+tags:
+  - customer-service
+  - supervisor
+  - cycleinstruct
+  - lg-electronics
+  - phi
+  - fine-tuned
+language:
+  - ko
+  - en
+  - de
+  - fr
+  - es
+pipeline_tag: text-generation
+---
+
+# cycleinstruct-phi4-supervisor
+
+Fully merged **microsoft/Phi-4-reasoning** (14.66 B) fine-tuned in two
+stages for the LG-Electronics customer-service **quality-supervisor** task.
+Given a `(Category, Conversation Transcript, Retrieved Document)` triplet,
+the model emits
+
+```
+<think>
+[Query-Document Alignment] …
+[Response-Document Consistency] …
+[Response Completeness] …
+</think>
+{"label": "correct" | "incorrect", "reason": "…"}
+```
+
+This repo contains a **single-file, ready-to-use** checkpoint — no adapter
+merging required at load time.
+
+## Training pipeline (CycleInstruct-motivated, two-stage SFT)
+
+Following the [CycleInstruct paper](https://arxiv.org/abs/2508.09551)
+(EMNLP 2025) as the augmentation strategy motivator:
+
+1. **Stage 1 — CS-chatbot SFT** on 9,868 natural `(question, answer)`
+   pairs built from LG feedback + general-inquiry data. LoRA r=16 α=32,
+   Muon @ lr=2e-3, seed=1337, 8 epochs.
+2. **Stage 2 — Supervisor SFT** on 3,771 human-annotated supervisor
+   judgements. Stage-1 LoRA is merged into the base first, then a fresh
+   LoRA r=16 α=32 is added and trained with Muon @ lr=1e-3, seed=42,
+   7 epochs on 4,096-token sequences.
+
+The uploaded checkpoint is the result of merging **both** LoRA stages into
+the base weights and re-saving with `save_pretrained`.
+
+## Metrics — 199-item held-out supervisor test set (T=0, `max_new_tokens=1200`)
+
+| Metric | Stage-1 only | **This model (full merged)** |
+|---|---|---|
+| Parse-fail rate | 95.98 % | **0.00 %** |
+| Accuracy | 1.01 % | **68.84 %** |
+| Macro-F1 | 0.033 | **0.615** |
+| chrF | 6.55 | **40.92** |
+| ROUGE-L | 0.062 | **0.885** |
+| BLEU-4 | 0.37 | **22.41** |
+| BERTScore-F1 | 0.826 | **0.901** |
+| SBERT-cos (multi-mpnet) | 0.437 | **0.830** |
+
+Per-class:
+
+| Class | Precision | Recall | F1 | Support |
+|---|---|---|---|---|
+| correct   | 0.417 | 0.481 | 0.446 | 52 |
+| incorrect | 0.806 | 0.762 | 0.783 | 147 |
+
+## Loading
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+
+REPO = "shareit/cycleinstruct-phi4-supervisor"
+
+tok   = AutoTokenizer.from_pretrained(REPO)
+model = AutoModelForCausalLM.from_pretrained(
+    REPO, torch_dtype=torch.bfloat16,
+    attn_implementation="sdpa", device_map="auto").eval()
+
+SYSTEM = "당신은 전자제품 CS 챗봇의 품질을 평가하는 수퍼바이저입니다."
+USER   = "[Category] W/M\n[Conversation Transcript] …\n[Retrieved Document] …"
+
+# Phi-4-reasoning ChatML with our clean system prompt (skip default Thought scaffold)
+prompt = (
+    f"<|im_start|>system<|im_sep|>{SYSTEM}<|im_end|>"
+    f"<|im_start|>user<|im_sep|>{USER}<|im_end|>"
+    f"<|im_start|>assistant<|im_sep|>"
+)
+out = model.generate(
+    **tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device),
+    do_sample=False, max_new_tokens=1200,
+    pad_token_id=tok.pad_token_id,
+)
+print(tok.decode(out[0], skip_special_tokens=False))
+```
+
+`max_new_tokens=1200` matters — the `<think>` block usually consumes
+500-900 tokens before the final JSON verdict.
+
+## Training details (stage 2, on top of stage-1-merged base)
+
+- **PEFT**: LoRA r=16, α=32, dropout 0.05, `target_modules=all-linear`, bias='none'
+- **Optimizer**: Muon on 2D matrices (Newton-Schulz orthogonalisation) + AdamW on 1D params
+- **LR**: 1e-3 (matrix) / 1e-4 (aux), cosine decay with 3 % warmup, grad-clip 1.0
+- **Batch**: per-device 1 × grad-accum 16 (effective 16)
+- **Seq len**: 4096 (user text char-clipped if exceeds; assistant always preserved)
+- **Seed**: 42, **Epochs**: 7
+- **Attention**: SDPA (bf16 native on H200)
+- **Wall clock**: 5h48m on a half-H200 (48 GB active)
+
+## Data
+
+- Stage-1 train: 9,868 `(q, a)` pairs from `data/processed/train_pairs.jsonl`
+  (multilingual, mostly English, ~50 % English, ~15 % German, then FR/ES/IT/JA/ZH…)
+- Stage-2 train: 3,771 supervisor-annotated rows
+  `{"conversations": [{"from":"system", …}, {"from":"user", …}, {"from":"assistant", …}]}`
+  with the assistant response being a `<think>…</think>{"label":…,"reason":…}` judgement.
+- Test: 199 held-out supervisor rows (unseen during either stage).
+
+## Intended use / limitations
+
+- Intended for research reproduction of CycleInstruct-style continuation
+  training on labeled downstream tasks.
+- The `correct` class has substantially lower F1 (0.446) than `incorrect`
+  (0.783), reflecting the 39/61 % class imbalance in the training data.
+  Class-weighted loss or balanced sampling would likely help.
+- The `<think>` reasoning is Korean; input transcripts may be any language.
+
+## License
+
+MIT (inherits from the `microsoft/Phi-4-reasoning` base model).

chat_template.jinja

@@ -0,0 +1 @@
+<|im_start|>system<|im_sep|>You are Phi, a language model trained by Microsoft to help users. Your role as an assistant involves thoroughly exploring questions through a systematic thinking process before providing the final precise and accurate solutions. This requires engaging in a comprehensive cycle of analysis, summarizing, exploration, reassessment, reflection, backtracing, and iteration to develop well-considered thinking process. Please structure your response into two main sections: Thought and Solution using the specified format: <think> {Thought section} </think> {Solution section}. In the Thought section, detail your reasoning process in steps. Each step should include detailed considerations such as analysing questions, summarizing relevant findings, brainstorming new ideas, verifying the accuracy of the current steps, refining any errors, and revisiting previous steps. In the Solution section, based on various attempts, explorations, and reflections from the Thought section, systematically present the final solution that you deem correct. The Solution section should be logical, accurate, and concise and detail necessary steps needed to reach the conclusion. Now, try to solve the following question through the above guidelines:<|im_end|>{% for message in messages %}{% if (message['role'] == 'user') %}{{'<|im_start|>user<|im_sep|>' + message['content'] + '<|im_end|>'}}{% elif (message['role'] == 'assistant') %}{{'<|im_start|>assistant<|im_sep|>'}}{% generation %}{{message['content'] + '<|im_end|>'}}{% endgeneration %}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant<|im_sep|>' }}{% endif %}

config.json

@@ -0,0 +1,35 @@
+{
+  "architectures": [
+    "Phi3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 100257,
+  "dtype": "bfloat16",
+  "embd_pdrop": 0.0,
+  "eos_token_id": 100265,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "intermediate_size": 17920,
+  "max_position_embeddings": 32768,
+  "model_type": "phi3",
+  "num_attention_heads": 40,
+  "num_hidden_layers": 40,
+  "num_key_value_heads": 10,
+  "original_max_position_embeddings": 32768,
+  "pad_token_id": 100349,
+  "partial_rotary_factor": 1.0,
+  "resid_pdrop": 0.0,
+  "rms_norm_eps": 1e-05,
+  "rope_parameters": {
+    "partial_rotary_factor": 1.0,
+    "rope_theta": 500000,
+    "rope_type": "default"
+  },
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.12.1",
+  "use_cache": true,
+  "vocab_size": 100352
+}

generation_config.json

@@ -0,0 +1,11 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 100257,
+  "do_sample": true,
+  "eos_token_id": 100265,
+  "pad_token_id": 100349,
+  "temperature": 0.8,
+  "top_k": 50,
+  "top_p": 0.95,
+  "transformers_version": "5.12.1"
+}

model-00001-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:688e3f5e94051936e54a789e5683b567b8c0663a19f78d53eec6b4e1645b2908
+size 3968923696

model-00002-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:294d5d5a4c92ccb9f79288ff212293342e428fd260060335d110d726d7566185
+size 3722561416

model-00003-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2dce113162389f7a3593d032eaca63cd58c4f67a4018999841f11152a17540bf
+size 3906072600

model-00004-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ba8a77cc1a9d753b061f42e43a11e54ed112f20044cf162a75596ee6ae15afb5
+size 3958490944

model-00005-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:678b294053ba8bc90690203f9bde3052b0d6224d6a34d6bae737a0291a4316d9
+size 3722561448

model-00006-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:47abfb5db3d59683d27ad7bfd734a0ae641468b2507ff588b8c7c523c11a9bd1
+size 3906072624

model-00007-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:867284d5a00fa3b26f6d2988970ee93762f5792219af1d0a2caa4b5cae8b2432
+size 3958490944

model-00008-of-00008.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8808010d5843d3bfc2a4f7d9f0a110ae179cd6add557321bebf03e9a73a79a4f
+size 2175869336

model.safetensors.index.json

@@ -0,0 +1,251 @@
+{
+  "metadata": {
+    "total_parameters": 14659507200,
+    "total_size": 29319014400
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00001-of-00008.safetensors",
+    "model.embed_tokens.weight": "model-00001-of-00008.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-00001-of-00008.safetensors",
+    "model.layers.0.mlp.down_proj.weight": "model-00001-of-00008.safetensors",
+    "model.layers.0.mlp.gate_up_proj.weight": "model-00001-of-00008.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00008.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00008.safetensors",
+    "model.layers.0.self_attn.qkv_proj.weight": "model-00001-of-00008.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-00001-of-00008.safetensors",
+    "model.layers.1.mlp.down_proj.weight": "model-00001-of-00008.safetensors",
+    "model.layers.1.mlp.gate_up_proj.weight": "model-00001-of-00008.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00008.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00008.safetensors",
+    "model.layers.1.self_attn.qkv_proj.weight": "model-00001-of-00008.safetensors",
+    "model.layers.10.input_layernorm.weight": "model-00003-of-00008.safetensors",
+    "model.layers.10.mlp.down_proj.weight": "model-00003-of-00008.safetensors",
+    "model.layers.10.mlp.gate_up_proj.weight": "model-00003-of-00008.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00003-of-00008.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-00003-of-00008.safetensors",
+    "model.layers.10.self_attn.qkv_proj.weight": "model-00003-of-00008.safetensors",
+    "model.layers.11.input_layernorm.weight": "model-00003-of-00008.safetensors",
+    "model.layers.11.mlp.down_proj.weight": "model-00003-of-00008.safetensors",
+    "model.layers.11.mlp.gate_up_proj.weight": "model-00003-of-00008.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00003-of-00008.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-00003-of-00008.safetensors",
+    "model.layers.11.self_attn.qkv_proj.weight": "model-00003-of-00008.safetensors",
+    "model.layers.12.input_layernorm.weight": "model-00003-of-00008.safetensors",
+    "model.layers.12.mlp.down_proj.weight": "model-00003-of-00008.safetensors",
+    "model.layers.12.mlp.gate_up_proj.weight": "model-00003-of-00008.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00003-of-00008.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00003-of-00008.safetensors",
+    "model.layers.12.self_attn.qkv_proj.weight": "model-00003-of-00008.safetensors",
+    "model.layers.13.input_layernorm.weight": "model-00003-of-00008.safetensors",
+    "model.layers.13.mlp.down_proj.weight": "model-00003-of-00008.safetensors",
+    "model.layers.13.mlp.gate_up_proj.weight": "model-00003-of-00008.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00003-of-00008.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00003-of-00008.safetensors",
+    "model.layers.13.self_attn.qkv_proj.weight": "model-00003-of-00008.safetensors",
+    "model.layers.14.input_layernorm.weight": "model-00003-of-00008.safetensors",
+    "model.layers.14.mlp.down_proj.weight": "model-00004-of-00008.safetensors",
+    "model.layers.14.mlp.gate_up_proj.weight": "model-00004-of-00008.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00004-of-00008.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00004-of-00008.safetensors",
+    "model.layers.14.self_attn.qkv_proj.weight": "model-00004-of-00008.safetensors",
+    "model.layers.15.input_layernorm.weight": "model-00004-of-00008.safetensors",
+    "model.layers.15.mlp.down_proj.weight": "model-00004-of-00008.safetensors",
+    "model.layers.15.mlp.gate_up_proj.weight": "model-00004-of-00008.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00004-of-00008.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00004-of-00008.safetensors",
+    "model.layers.15.self_attn.qkv_proj.weight": "model-00004-of-00008.safetensors",
+    "model.layers.16.input_layernorm.weight": "model-00004-of-00008.safetensors",
+    "model.layers.16.mlp.down_proj.weight": "model-00004-of-00008.safetensors",
+    "model.layers.16.mlp.gate_up_proj.weight": "model-00004-of-00008.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00004-of-00008.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00004-of-00008.safetensors",
+    "model.layers.16.self_attn.qkv_proj.weight": "model-00004-of-00008.safetensors",
+    "model.layers.17.input_layernorm.weight": "model-00004-of-00008.safetensors",
+    "model.layers.17.mlp.down_proj.weight": "model-00004-of-00008.safetensors",
+    "model.layers.17.mlp.gate_up_proj.weight": "model-00004-of-00008.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00004-of-00008.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00004-of-00008.safetensors",
+    "model.layers.17.self_attn.qkv_proj.weight": "model-00004-of-00008.safetensors",
+    "model.layers.18.input_layernorm.weight": "model-00004-of-00008.safetensors",
+    "model.layers.18.mlp.down_proj.weight": "model-00004-of-00008.safetensors",
+    "model.layers.18.mlp.gate_up_proj.weight": "model-00004-of-00008.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00004-of-00008.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00004-of-00008.safetensors",
+    "model.layers.18.self_attn.qkv_proj.weight": "model-00004-of-00008.safetensors",
+    "model.layers.19.input_layernorm.weight": "model-00004-of-00008.safetensors",
+    "model.layers.19.mlp.down_proj.weight": "model-00004-of-00008.safetensors",
+    "model.layers.19.mlp.gate_up_proj.weight": "model-00004-of-00008.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00004-of-00008.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00005-of-00008.safetensors",
+    "model.layers.19.self_attn.qkv_proj.weight": "model-00005-of-00008.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-00001-of-00008.safetensors",
+    "model.layers.2.mlp.down_proj.weight": "model-00001-of-00008.safetensors",
+    "model.layers.2.mlp.gate_up_proj.weight": "model-00001-of-00008.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00008.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00002-of-00008.safetensors",
+    "model.layers.2.self_attn.qkv_proj.weight": "model-00002-of-00008.safetensors",
+    "model.layers.20.input_layernorm.weight": "model-00005-of-00008.safetensors",
+    "model.layers.20.mlp.down_proj.weight": "model-00005-of-00008.safetensors",
+    "model.layers.20.mlp.gate_up_proj.weight": "model-00005-of-00008.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00005-of-00008.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00005-of-00008.safetensors",
+    "model.layers.20.self_attn.qkv_proj.weight": "model-00005-of-00008.safetensors",
+    "model.layers.21.input_layernorm.weight": "model-00005-of-00008.safetensors",
+    "model.layers.21.mlp.down_proj.weight": "model-00005-of-00008.safetensors",
+    "model.layers.21.mlp.gate_up_proj.weight": "model-00005-of-00008.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00005-of-00008.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00005-of-00008.safetensors",
+    "model.layers.21.self_attn.qkv_proj.weight": "model-00005-of-00008.safetensors",
+    "model.layers.22.input_layernorm.weight": "model-00005-of-00008.safetensors",
+    "model.layers.22.mlp.down_proj.weight": "model-00005-of-00008.safetensors",
+    "model.layers.22.mlp.gate_up_proj.weight": "model-00005-of-00008.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-00005-of-00008.safetensors",
+    "model.layers.22.self_attn.o_proj.weight": "model-00005-of-00008.safetensors",
+    "model.layers.22.self_attn.qkv_proj.weight": "model-00005-of-00008.safetensors",
+    "model.layers.23.input_layernorm.weight": "model-00005-of-00008.safetensors",
+    "model.layers.23.mlp.down_proj.weight": "model-00005-of-00008.safetensors",
+    "model.layers.23.mlp.gate_up_proj.weight": "model-00005-of-00008.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-00005-of-00008.safetensors",
+    "model.layers.23.self_attn.o_proj.weight": "model-00005-of-00008.safetensors",
+    "model.layers.23.self_attn.qkv_proj.weight": "model-00005-of-00008.safetensors",
+    "model.layers.24.input_layernorm.weight": "model-00005-of-00008.safetensors",
+    "model.layers.24.mlp.down_proj.weight": "model-00005-of-00008.safetensors",
+    "model.layers.24.mlp.gate_up_proj.weight": "model-00005-of-00008.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-00005-of-00008.safetensors",
+    "model.layers.24.self_attn.o_proj.weight": "model-00005-of-00008.safetensors",
+    "model.layers.24.self_attn.qkv_proj.weight": "model-00005-of-00008.safetensors",
+    "model.layers.25.input_layernorm.weight": "model-00005-of-00008.safetensors",
+    "model.layers.25.mlp.down_proj.weight": "model-00005-of-00008.safetensors",
+    "model.layers.25.mlp.gate_up_proj.weight": "model-00006-of-00008.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-00006-of-00008.safetensors",
+    "model.layers.25.self_attn.o_proj.weight": "model-00006-of-00008.safetensors",
+    "model.layers.25.self_attn.qkv_proj.weight": "model-00006-of-00008.safetensors",
+    "model.layers.26.input_layernorm.weight": "model-00006-of-00008.safetensors",
+    "model.layers.26.mlp.down_proj.weight": "model-00006-of-00008.safetensors",
+    "model.layers.26.mlp.gate_up_proj.weight": "model-00006-of-00008.safetensors",
+    "model.layers.26.post_attention_layernorm.weight": "model-00006-of-00008.safetensors",
+    "model.layers.26.self_attn.o_proj.weight": "model-00006-of-00008.safetensors",
+    "model.layers.26.self_attn.qkv_proj.weight": "model-00006-of-00008.safetensors",
+    "model.layers.27.input_layernorm.weight": "model-00006-of-00008.safetensors",
+    "model.layers.27.mlp.down_proj.weight": "model-00006-of-00008.safetensors",
+    "model.layers.27.mlp.gate_up_proj.weight": "model-00006-of-00008.safetensors",
+    "model.layers.27.post_attention_layernorm.weight": "model-00006-of-00008.safetensors",
+    "model.layers.27.self_attn.o_proj.weight": "model-00006-of-00008.safetensors",
+    "model.layers.27.self_attn.qkv_proj.weight": "model-00006-of-00008.safetensors",
+    "model.layers.28.input_layernorm.weight": "model-00006-of-00008.safetensors",
+    "model.layers.28.mlp.down_proj.weight": "model-00006-of-00008.safetensors",
+    "model.layers.28.mlp.gate_up_proj.weight": "model-00006-of-00008.safetensors",
+    "model.layers.28.post_attention_layernorm.weight": "model-00006-of-00008.safetensors",
+    "model.layers.28.self_attn.o_proj.weight": "model-00006-of-00008.safetensors",
+    "model.layers.28.self_attn.qkv_proj.weight": "model-00006-of-00008.safetensors",
+    "model.layers.29.input_layernorm.weight": "model-00006-of-00008.safetensors",
+    "model.layers.29.mlp.down_proj.weight": "model-00006-of-00008.safetensors",
+    "model.layers.29.mlp.gate_up_proj.weight": "model-00006-of-00008.safetensors",
+    "model.layers.29.post_attention_layernorm.weight": "model-00006-of-00008.safetensors",
+    "model.layers.29.self_attn.o_proj.weight": "model-00006-of-00008.safetensors",
+    "model.layers.29.self_attn.qkv_proj.weight": "model-00006-of-00008.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-00002-of-00008.safetensors",
+    "model.layers.3.mlp.down_proj.weight": "model-00002-of-00008.safetensors",
+    "model.layers.3.mlp.gate_up_proj.weight": "model-00002-of-00008.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00002-of-00008.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00002-of-00008.safetensors",
+    "model.layers.3.self_attn.qkv_proj.weight": "model-00002-of-00008.safetensors",
+    "model.layers.30.input_layernorm.weight": "model-00006-of-00008.safetensors",
+    "model.layers.30.mlp.down_proj.weight": "model-00006-of-00008.safetensors",
+    "model.layers.30.mlp.gate_up_proj.weight": "model-00006-of-00008.safetensors",
+    "model.layers.30.post_attention_layernorm.weight": "model-00006-of-00008.safetensors",
+    "model.layers.30.self_attn.o_proj.weight": "model-00006-of-00008.safetensors",
+    "model.layers.30.self_attn.qkv_proj.weight": "model-00006-of-00008.safetensors",
+    "model.layers.31.input_layernorm.weight": "model-00006-of-00008.safetensors",
+    "model.layers.31.mlp.down_proj.weight": "model-00007-of-00008.safetensors",
+    "model.layers.31.mlp.gate_up_proj.weight": "model-00007-of-00008.safetensors",
+    "model.layers.31.post_attention_layernorm.weight": "model-00007-of-00008.safetensors",
+    "model.layers.31.self_attn.o_proj.weight": "model-00007-of-00008.safetensors",
+    "model.layers.31.self_attn.qkv_proj.weight": "model-00007-of-00008.safetensors",
+    "model.layers.32.input_layernorm.weight": "model-00007-of-00008.safetensors",
+    "model.layers.32.mlp.down_proj.weight": "model-00007-of-00008.safetensors",
+    "model.layers.32.mlp.gate_up_proj.weight": "model-00007-of-00008.safetensors",
+    "model.layers.32.post_attention_layernorm.weight": "model-00007-of-00008.safetensors",
+    "model.layers.32.self_attn.o_proj.weight": "model-00007-of-00008.safetensors",
+    "model.layers.32.self_attn.qkv_proj.weight": "model-00007-of-00008.safetensors",
+    "model.layers.33.input_layernorm.weight": "model-00007-of-00008.safetensors",
+    "model.layers.33.mlp.down_proj.weight": "model-00007-of-00008.safetensors",
+    "model.layers.33.mlp.gate_up_proj.weight": "model-00007-of-00008.safetensors",
+    "model.layers.33.post_attention_layernorm.weight": "model-00007-of-00008.safetensors",
+    "model.layers.33.self_attn.o_proj.weight": "model-00007-of-00008.safetensors",
+    "model.layers.33.self_attn.qkv_proj.weight": "model-00007-of-00008.safetensors",
+    "model.layers.34.input_layernorm.weight": "model-00007-of-00008.safetensors",
+    "model.layers.34.mlp.down_proj.weight": "model-00007-of-00008.safetensors",
+    "model.layers.34.mlp.gate_up_proj.weight": "model-00007-of-00008.safetensors",
+    "model.layers.34.post_attention_layernorm.weight": "model-00007-of-00008.safetensors",
+    "model.layers.34.self_attn.o_proj.weight": "model-00007-of-00008.safetensors",
+    "model.layers.34.self_attn.qkv_proj.weight": "model-00007-of-00008.safetensors",
+    "model.layers.35.input_layernorm.weight": "model-00007-of-00008.safetensors",
+    "model.layers.35.mlp.down_proj.weight": "model-00007-of-00008.safetensors",
+    "model.layers.35.mlp.gate_up_proj.weight": "model-00007-of-00008.safetensors",
+    "model.layers.35.post_attention_layernorm.weight": "model-00007-of-00008.safetensors",
+    "model.layers.35.self_attn.o_proj.weight": "model-00007-of-00008.safetensors",
+    "model.layers.35.self_attn.qkv_proj.weight": "model-00007-of-00008.safetensors",
+    "model.layers.36.input_layernorm.weight": "model-00007-of-00008.safetensors",
+    "model.layers.36.mlp.down_proj.weight": "model-00007-of-00008.safetensors",
+    "model.layers.36.mlp.gate_up_proj.weight": "model-00007-of-00008.safetensors",
+    "model.layers.36.post_attention_layernorm.weight": "model-00007-of-00008.safetensors",
+    "model.layers.36.self_attn.o_proj.weight": "model-00008-of-00008.safetensors",
+    "model.layers.36.self_attn.qkv_proj.weight": "model-00008-of-00008.safetensors",
+    "model.layers.37.input_layernorm.weight": "model-00008-of-00008.safetensors",
+    "model.layers.37.mlp.down_proj.weight": "model-00008-of-00008.safetensors",
+    "model.layers.37.mlp.gate_up_proj.weight": "model-00008-of-00008.safetensors",
+    "model.layers.37.post_attention_layernorm.weight": "model-00008-of-00008.safetensors",
+    "model.layers.37.self_attn.o_proj.weight": "model-00008-of-00008.safetensors",
+    "model.layers.37.self_attn.qkv_proj.weight": "model-00008-of-00008.safetensors",
+    "model.layers.38.input_layernorm.weight": "model-00008-of-00008.safetensors",
+    "model.layers.38.mlp.down_proj.weight": "model-00008-of-00008.safetensors",
+    "model.layers.38.mlp.gate_up_proj.weight": "model-00008-of-00008.safetensors",
+    "model.layers.38.post_attention_layernorm.weight": "model-00008-of-00008.safetensors",
+    "model.layers.38.self_attn.o_proj.weight": "model-00008-of-00008.safetensors",
+    "model.layers.38.self_attn.qkv_proj.weight": "model-00008-of-00008.safetensors",
+    "model.layers.39.input_layernorm.weight": "model-00008-of-00008.safetensors",
+    "model.layers.39.mlp.down_proj.weight": "model-00008-of-00008.safetensors",
+    "model.layers.39.mlp.gate_up_proj.weight": "model-00008-of-00008.safetensors",
+    "model.layers.39.post_attention_layernorm.weight": "model-00008-of-00008.safetensors",
+    "model.layers.39.self_attn.o_proj.weight": "model-00008-of-00008.safetensors",
+    "model.layers.39.self_attn.qkv_proj.weight": "model-00008-of-00008.safetensors",
+    "model.layers.4.input_layernorm.weight": "model-00002-of-00008.safetensors",
+    "model.layers.4.mlp.down_proj.weight": "model-00002-of-00008.safetensors",
+    "model.layers.4.mlp.gate_up_proj.weight": "model-00002-of-00008.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00002-of-00008.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00002-of-00008.safetensors",
+    "model.layers.4.self_attn.qkv_proj.weight": "model-00002-of-00008.safetensors",
+    "model.layers.5.input_layernorm.weight": "model-00002-of-00008.safetensors",
+    "model.layers.5.mlp.down_proj.weight": "model-00002-of-00008.safetensors",
+    "model.layers.5.mlp.gate_up_proj.weight": "model-00002-of-00008.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00002-of-00008.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-00002-of-00008.safetensors",
+    "model.layers.5.self_attn.qkv_proj.weight": "model-00002-of-00008.safetensors",
+    "model.layers.6.input_layernorm.weight": "model-00002-of-00008.safetensors",
+    "model.layers.6.mlp.down_proj.weight": "model-00002-of-00008.safetensors",
+    "model.layers.6.mlp.gate_up_proj.weight": "model-00002-of-00008.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00002-of-00008.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-00002-of-00008.safetensors",
+    "model.layers.6.self_attn.qkv_proj.weight": "model-00002-of-00008.safetensors",
+    "model.layers.7.input_layernorm.weight": "model-00002-of-00008.safetensors",
+    "model.layers.7.mlp.down_proj.weight": "model-00002-of-00008.safetensors",
+    "model.layers.7.mlp.gate_up_proj.weight": "model-00002-of-00008.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00002-of-00008.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00002-of-00008.safetensors",
+    "model.layers.7.self_attn.qkv_proj.weight": "model-00002-of-00008.safetensors",
+    "model.layers.8.input_layernorm.weight": "model-00002-of-00008.safetensors",
+    "model.layers.8.mlp.down_proj.weight": "model-00002-of-00008.safetensors",
+    "model.layers.8.mlp.gate_up_proj.weight": "model-00003-of-00008.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00003-of-00008.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00003-of-00008.safetensors",
+    "model.layers.8.self_attn.qkv_proj.weight": "model-00003-of-00008.safetensors",
+    "model.layers.9.input_layernorm.weight": "model-00003-of-00008.safetensors",
+    "model.layers.9.mlp.down_proj.weight": "model-00003-of-00008.safetensors",
+    "model.layers.9.mlp.gate_up_proj.weight": "model-00003-of-00008.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00003-of-00008.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-00003-of-00008.safetensors",
+    "model.layers.9.self_attn.qkv_proj.weight": "model-00003-of-00008.safetensors",
+    "model.norm.weight": "model-00008-of-00008.safetensors"
+  }
+}

tokenizer_config.json

@@ -0,0 +1,13 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 32768,
+  "pad_token": "<|dummy_85|>",
+  "padding_side": "left",
+  "tokenizer_class": "TokenizersBackend"
+}