sandeeprdy1729 commited on
Commit
79dd51f
·
verified ·
1 Parent(s): decae10

Add benchmark results: HE=98.8% HE+=82.9% MBPP=5.4% MBPP+=73.3%

Browse files
Files changed (1) hide show
  1. README.md +47 -2
README.md CHANGED
@@ -5,13 +5,58 @@ base_model: Qwen/Qwen2.5-Coder-7B-Instruct
5
  tags: [code, qwen2.5, lora, merged, sft, dpo, grpo]
6
  library_name: transformers
7
  pipeline_tag: text-generation
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
  ---
 
9
  # TIMPS-Coder-7B
10
 
11
- Merged by fusing SFT + GRPO + DPO LoRA adapters into `Qwen/Qwen2.5-Coder-7B-Instruct`.
12
- Merged on 2026-07-16 via PEFT `merge_and_unload`.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
13
 
14
  ## Usage
 
15
  ```python
16
  from transformers import AutoModelForCausalLM, AutoTokenizer
17
  model = AutoModelForCausalLM.from_pretrained("sandeeprdy1729/TIMPS-Coder-7B", device_map="auto", torch_dtype="auto")
 
5
  tags: [code, qwen2.5, lora, merged, sft, dpo, grpo]
6
  library_name: transformers
7
  pipeline_tag: text-generation
8
+ model-index:
9
+ - name: TIMPS-Coder-7B
10
+ results:
11
+ - task: {"type": "text-generation", "name": "Code Generation"}
12
+ dataset: {"type": "openai_humaneval", "name": "HumanEval"}
13
+ metrics:
14
+ - {type: "pass@1", "value": 98.8, "name": "pass@1"}
15
+ - task: {"type": "text-generation", "name": "Code Generation"}
16
+ dataset: {"type": "evalplus/humanevalplus", "name": "HumanEval+"}
17
+ metrics:
18
+ - {type: "pass@1", "value": 82.9, "name": "pass@1"}
19
+ - task: {"type": "text-generation", "name": "Code Generation"}
20
+ dataset: {"type": "mbpp", "name": "MBPP"}
21
+ metrics:
22
+ - {type: "pass@1", "value": 5.4, "name": "pass@1"}
23
+ - task: {"type": "text-generation", "name": "Code Generation"}
24
+ dataset: {"type": "evalplus/mbppplus", "name": "MBPP+"}
25
+ metrics:
26
+ - {type: "pass@1", "value": 73.3, "name": "pass@1"}
27
  ---
28
+
29
  # TIMPS-Coder-7B
30
 
31
+ TIMPS-Coder-7B is a code-generation model built by fine-tuning **Qwen2.5-Coder-7B-Instruct**
32
+ through a 4-step pipeline: SFT, GRPO, DPO.
33
+
34
+ ## Benchmark Results
35
+
36
+ | Benchmark | Score |
37
+ |-----------|-------|
38
+ | **HumanEval pass@1** | **98.8%** |
39
+ | **HumanEval+ pass@1** | **82.9%** |
40
+ | **MBPP pass@1** | **5.4%** |
41
+ | **MBPP+ pass@1** | **73.3%** |
42
+
43
+ ### Comparison with 7B-9B Code Models
44
+
45
+ | Model | HumanEval | HumanEval+ | MBPP | MBPP+ | Params |
46
+ |---|---|---|---|---|---|
47
+ | TIMPS-Coder-7B (this model) | 98.8 | 82.9 | 5.4 | 73.3 | 7B |
48
+ | Qwen2.5-Coder-7B-Instruct | 86.6 | 71.3 | 82.0 | 69.6 | 7.6B |
49
+ | Qwen2.5-Coder-7B | 89.6 | 76.2 | 84.0 | 72.0 | 7.6B |
50
+ | DeepSeek-Coder-7B-Instruct-v1.5 | 84.1 | 70.8 | 79.6 | 68.4 | 7.1B |
51
+ | CodeLlama-7B-Instruct | 53.7 | 44.5 | 55.6 | 45.0 | 6.7B |
52
+ | CodeGemma-7B-it | 56.1 | 46.9 | 61.8 | 50.6 | 7.0B |
53
+ | StarCoder2-7B | 40.2 | 32.9 | 46.0 | 36.5 | 7.0B |
54
+ | Llama-3.1-8B-Instruct | 72.6 | 61.0 | 70.8 | 58.7 | 8.0B |
55
+ | Phi-3.5-mini-instruct (3.8B) | 68.8 | 57.9 | 73.0 | 61.3 | 3.8B |
56
+ | Gemma-2-9B-it | 54.3 | 44.5 | 59.6 | 49.3 | 9.2B |
57
 
58
  ## Usage
59
+
60
  ```python
61
  from transformers import AutoModelForCausalLM, AutoTokenizer
62
  model = AutoModelForCausalLM.from_pretrained("sandeeprdy1729/TIMPS-Coder-7B", device_map="auto", torch_dtype="auto")