Bin12345

/

AutoCoder_S_6.7B

like 45

Thanks for the verification, could you check if all these outputs are the same?

There are 164 problems inside the humaneval dataset. If both model correctly solved 119 problems. The final accuracy will be the same.

And

1. We add some special tokens to the Deepseek-Coder and fine-tuned the model. These two models are different, you can use the following code to check two model's weights

from transformers import AutoTokenizer, AutoModelForCausalLM

model_path = "" # Bin12345/AutoCoder_S_6.7B, deepseek-ai/deepseek-coder-6.7b-instruct
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(model_path, device_map="auto")

model_weights = model.state_dict()

for name, param in model_weights.items():
    print(f"Layer: {name} | Size: {param.size()} | Values: {param[:2]}") 

2. The results shown in our paper are performed by using greedy sampling method. The steps to reproduce the experiments are shown in https://github.com/bin123apple/AutoCoder

3. Could you share your experiment code and I will run it on my side and check the results.

Yes I ran the benchmark in google colab, here is the code:

!pip install -q -U transformers
!pip install -q -U accelerate
!pip install -q -U bitsandbytes

!git clone https://github.com/bigcode-project/bigcode-evaluation-harness.git

%cd bigcode-evaluation-harness

!pip install -r requirements.txt

!accelerate launch main.py --tasks humaneval --model Bin12345/AutoCoder_S_6.7B --load_in_8bit --allow_code_execution --max_length_generation 4000 --precision fp16 --temperature 0.1

And for the other model just replace the model name

!accelerate launch main.py --tasks humaneval --model deepseek-ai/deepseek-coder-6.7b-instruct --load_in_8bit --allow_code_execution --max_length_generation 4000 --precision fp16 --temperature 0.1

You have to add a flag to get the results though, it wont save them without the extra flag, i forget what it is, you should be able to find it here
https://github.com/bigcode-project/bigcode-evaluation-harness

Thanks for the feedback! I will try it.

If you want to check if these two models are the same, you can use

from transformers import AutoTokenizer, AutoModelForCausalLM

model_path = "" # Bin12345/AutoCoder_S_6.7B, deepseek-ai/deepseek-coder-6.7b-instruct
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(model_path, device_map="auto")

model_weights = model.state_dict()

for name, param in model_weights.items():
    print(f"Layer: {name} | Size: {param.size()} | Values: {param[:2]}")

to print out their weights.

You will see that they are different.

Hey @rombodawg , I just reproduced your experiment, the output of these two models are different. You can add two flags --save_generations and --save_generations_path to save and check the generated code. Thanks