prompts for humaneval
Hi,
In the paper I see that StarCoder-Prompted is better than StarCoder on HumanEval dataset. Could you please let me know where can I find these prompt examples? Thank you.
The prompt is mentioned in the paper section 6.1.1. We basically added this prefix to the HumanEval prompts <filename>solutions/solution_1.py\n# Here is the correct implementation of the code exercise\n before feeding them to the model.
Thank you @loubnabnl . But, where can I find the solution1.py? and does the code internally loads the contents of this solution and adds that as a prefix to the original prompt?
That's actually just text that we add at the beginning of each problem since we conditionned on file paths during pre-training. So the model tends to give better completions when we indicate that the code comes from a file with the path solutions/solution_1.py
Well, this is interesting, it's only that " solutions/solution_1.py\n# Here is the correct implementation of the code exercise\n" as the prefix and that increases the accuracy of the model from 34% to 40%. It feels quite counterintuitive. I feel like the model should produce the same result with/without this input (if the model understands something about the question). did you guys further explore this direction and found out something why you get these results? Thank you.