import os from openai import OpenAI import json import os import string import regex import time from collections import Counter import joblib from tqdm import tqdm import pandas as pd import pandas as pd import socket HOST = '127.0.0.1' PORT = 50007 client = OpenAI( api_key=os.environ.get("DEEPSEEK_API_KEY", ""), base_url='https://api.deepseek.com/v1', ) import re def extract_between_asterisks(text): # 使用正则表达式匹配 `**` 之间的内容 pattern = r'\*\*(.*?)\*\*' matches = re.findall(pattern, text) return matches ans_ls = [] pred_ls= [] query_ls = [] acc_ls = [] acc = 0 def eval_em(p_ls, g_ls): assert len(p_ls) == len(g_ls) cnt = 0 for idx in range(len(p_ls)): pred = p_ls[idx] gold = g_ls[idx] if gold in pred: cnt += 1 return cnt/len(p_ls) def excute(data_path,start_idx): data = pd.read_csv(data_path) for k, example in data.iterrows(): if k < start_idx: continue print(k) # example = json.loads(example) q = example['question'] answer = example['answer'] query_ls.append(q) ans_ls.append(answer) feedback_tmp_ls = [] round_count = 0 message_keys_list = [{"role": "user", "content": """Construct a global reasoning chain for this complex [Question] : " {} " You should generate a query to the search engine based on what you already know at each step of the reasoning chain, starting with [Query]. If you know the answer for [Query], generate it starting with [Answer]. You can try to generate the final answer for the [Question] by referring to the [Query]-[Answer] pairs, starting with [Final Content]. The final answer for the [Question] can only be yes or no, marked with **. If you don't know the answer, generate a query to search engine based on what you already know and do not know, starting with [Unsolved Query]. For example: [Question]: "Where do greyhound buses that are in the birthplace of Spirit If...'s performer leave from? " [Query 1]: Who is the performer of Spirit If... ? If you don't know the answer: [Unsolved Query]: Who is the performer of Spirit If... ? If you know the answer: [Answer 1]: The performer of Spirit If... is Kevin Drew. [Query 2]: Where was Kevin Drew born? If you don't know the answer: [Unsolved Query]: Where was Kevin Drew born? If you know the answer: [Answer 2]: Toronto. [Query 3]: Where do greyhound buses in Toronto leave from? If you don't know the answer: [Unsolved Query]: Where do greyhound buses in Toronto leave from? If you know the answer: [Answer 3]: Toronto Coach Terminal. [Final Content]: The performer of Spirit If... is Kevin Drew [1]. Kevin Drew was born in Toronto [2]. Greyhound buses in Toronto leave from Toronto Coach Terminal [3]. So the final answer is Toronto Coach Terminal. [Question]:"Which magazine was started first Arthur’s Magazine or First for Women?" [Query 1]: When was Arthur’s Magazine started? [Answer 1]: 1844. [Query 2]: When was First for Women started? [Answer 2]: 1989 [Final Content]: Arthur’s Magazine started in 1844 [1]. First for Women started in 1989 [2]. So Arthur’s Magazine was started first. So the answer is Arthur’s Magazi [Question]: {} """.format(q,q)}] feedback_answer = 'continue' predict_answer = '' sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock.connect((HOST, PORT)) while round_count < 5 and not feedback_answer == 'end': print('round is {}'.format(round_count)) try: rsp = client.chat.completions.create( model='deepseek-chat', messages=message_keys_list ) round_count += 1 input_str = rsp.choices[0].message.content message_keys_list.append({"role": "assistant", "content": input_str}) print('solving......') predict_answer += input_str sock.send(input_str.encode()) print('send message {}'.format(input_str)) feedback = sock.recv(10240).decode() print('feedback is '+feedback) if feedback == 'end': break #[Query]:xxxx[Answer]:xxxx[Reference]:xxxx feedback_list = feedback.split('') if not 'Unsolved Query' in feedback: new_prompt = """ According to this Reference, the answer for "{}" should be "{}", you can change your answer based on the Reference and continue constructing the reasoning chain to give the final answer for [Question]:{} Reference: {} """.format(feedback_list[0],feedback_list[1],q,feedback_list[2]) else: new_prompt = """ According to this Reference, the answer for "{}" should be "{}", you can give your answer based on the Reference and continue constructing the reasoning chain to give the final answer for [Question]：{} Reference: {} """.format(feedback_list[0],feedback_list[1],q,feedback_list[2]) message_keys_list.append({"role": "user", "content":new_prompt}) feedback_tmp_ls.append((feedback_list[0], feedback_list[2])) except: print('start_idx is {}'.format(k)) sock.send('end'.encode()) sock.close() return k if not feedback_answer == 'end': sock.send('end'.encode()) sock.close() print(message_keys_list) last_assistant = None for entry in reversed(message_keys_list): if entry['role'] == 'assistant': last_assistant = entry break pred_ans = last_assistant['content'].split('[Final Content]')[-1].lower() pred_ls.append(pred_ans.split('.')[-2]) if answer is True: answer = 'yes' else: answer = 'no' if answer in extract_between_asterisks(pred_ans): global acc acc += 1 acc_ls.append(1) else: acc_ls.append(0) return -1 if __name__ == '__main__': start_idx = 0 excute(os.path.join(os.path.dirname(os.path.abspath(__file__)), "data", "strategyqa_dev.csv"), start_idx=start_idx) # while not start_idx == -1: # start_idx = excute( # os.path.join(os.path.dirname(os.path.abspath(__file__)), "data", "strategyqa_dev.csv"), # start_idx=start_idx) # print(acc / len(ans_ls)) # # print(eval_em(pred_ls, ans_ls)) # data = { # "query": query_ls, # "ans": ans_ls, # "pred" : pred_ls, # "acc" : acc_ls # } # df = pd.DataFrame(data) # df.to_csv("/home/icml01/multi_rag/RAG/Search-in-the-Chain/output/strategyQA_log1.csv", index=False, encoding="utf-8")