(you can choose other metrics too):metrics = {    "rouge": {        "metric": "rouge",        "config": {"variety": "rouge_l"},    },    "chrf": {        "metric": "chrf",        "config": {},    },    "bert_score": {        "metric": "bert_score",        "config": {"model": "bert-base-uncased"},    },    "uni_eval": {        "metric": "uni_eval",        "config": {"task": "summarization", "evaluation_aspect": "relevance"},    },}critique_data = [    {"target": pred["result"], "references": [pred["answer"]]} for pred in predictions]eval_results = {    k: critique.evaluate(dataset=critique_data, metric=v["metric"], config=v["config"])    for k, v in metrics.items()}Finally, we can print out the results. We can see that overall the scores are higher when the output is semantically correct, and also when the output closely matches with the gold-standard answer.for i, eg in enumerate(examples):    score_string = ", ".join(        [f"{k}={v['examples'][i]['value']:.4f}" for k, v in eval_results.items()]    )    print(f"Example {i}:")    print("Question: " + predictions[i]["query"])    print("Real Answer: " + predictions[i]["answer"])


print("Real Answer: " + predictions[i]["answer"])    print("Predicted Answer: " + predictions[i]["result"])    print("Predicted Scores: " + score_string)    print()    Example 0:    Question: What did the president say about Ketanji Brown Jackson    Real Answer: He praised her legal ability and said he nominated her for the supreme court.    Predicted Answer:  The president said that she is one of the nation's top legal minds, a former top litigator in private practice, a former federal public defender, and from a family of public school educators and police officers. He also said that she is a consensus builder and that she has received a broad range of support from the Fraternal Order of Police to former judges appointed by both Democrats and Republicans.    Predicted Scores: rouge=0.0941, chrf=0.2001, bert_score=0.5219, uni_eval=0.9043        Example 1:    Question: What did the president say about Michael Jackson    Real Answer: Nothing    Predicted Answer:  The president did not mention Michael Jackson in this speech.    Predicted Scores: rouge=0.0000, chrf=0.1087, bert_score=0.3486, uni_eval=0.7802        Example 2:    Question: According to the document, what did Vladimir Putin miscalculate?    Real Answer: He miscalculated that he could roll into Ukraine and the world would roll over.    Predicted Answer:  Putin miscalculated that the world would roll over when he rolled into Ukraine.    Predicted Scores: rouge=0.5185,


when he rolled into Ukraine.    Predicted Scores: rouge=0.5185, chrf=0.6955, bert_score=0.8421, uni_eval=0.9578        Example 3:    Question: Who is the Ukrainian Ambassador to the United States?    Real Answer: The Ukrainian Ambassador to the United States is here tonight.    Predicted Answer:  I don't know.    Predicted Scores: rouge=0.0000, chrf=0.0375, bert_score=0.3159, uni_eval=0.7493        Example 4:    Question: How many countries were part of the coalition formed to confront Putin?    Real Answer: 27 members of the European Union, France, Germany, Italy, the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.    Predicted Answer:  The coalition included freedom-loving nations from Europe and the Americas to Asia and Africa, 27 members of the European Union including France, Germany, Italy, the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.    Predicted Scores: rouge=0.7419, chrf=0.8602, bert_score=0.8388, uni_eval=0.0669        Example 5:    Question: What action is the U.S. Department of Justice taking to target Russian oligarchs?    Real Answer: The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and joining with European allies to find and seize their yachts, luxury apartments, and private jets.    Predicted Answer:  The U.S. Department of Justice


and private jets.    Predicted Answer:  The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and to find and seize their yachts, luxury apartments, and private jets.    Predicted Scores: rouge=0.9412, chrf=0.8687, bert_score=0.9607, uni_eval=0.9718        Example 6:    Question: How much direct assistance is the United States providing to Ukraine?    Real Answer: The United States is providing more than $1 Billion in direct assistance to Ukraine.    Predicted Answer:  The United States is providing more than $1 billion in direct assistance to Ukraine.    Predicted Scores: rouge=1.0000, chrf=0.9483, bert_score=1.0000, uni_eval=0.9734    PreviousComparing Chain OutputsNextEvaluating an OpenAPI ChainSetupExamplesEvaluateEvaluate with Other MetricsCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


Evaluating an OpenAPI Chain | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionChainsMemoryAgentsCallbacksModulesGuidesEvaluationString EvaluatorsComparison EvaluatorsTrajectory EvaluatorsExamplesAgent VectorDB Question Answering BenchmarkingComparing Chain OutputsData Augmented Question AnsweringEvaluating an OpenAPI ChainQuestion Answering Benchmarking: Paul Graham EssayQuestion Answering Benchmarking: State of the Union AddressQA GenerationQuestion AnsweringSQL Question Answering Benchmarking: ChinookDebuggingDeploymentLangSmithModel ComparisonEcosystemAdditional resourcesGuidesEvaluationExamplesEvaluating an OpenAPI ChainOn this pageEvaluating an OpenAPI ChainThis notebook goes over ways to semantically evaluate an OpenAPI Chain, which calls an endpoint defined by the OpenAPI specification using purely natural language.from langchain.tools import OpenAPISpec, APIOperationfrom langchain.chains import OpenAPIEndpointChain, LLMChainfrom langchain.requests import Requestsfrom langchain.llms import OpenAILoad the API Chainâ€‹Load a wrapper of the spec (so we can work with it more easily). You can load from a url or from a local file.# Load and parse the OpenAPI Specspec = OpenAPISpec.from_url(    "https://www.klarna.com/us/shopping/public/openai/v0/api-docs/")# Load a single endpoint operationoperation = APIOperation.from_openapi_spec(spec, "/public/openai/v0/products", "get")verbose = False# Select any LangChain LLMllm = OpenAI(temperature=0, max_tokens=1000)# Create the endpoint chainapi_chain = OpenAPIEndpointChain.from_api_operation(    operation,    llm,    requests=Requests(),    verbose=verbose,


llm,    requests=Requests(),    verbose=verbose,    return_intermediate_steps=True,  # Return request and response text)    Attempting to load an OpenAPI 3.0.1 spec.  This may result in degraded performance. Convert your OpenAPI spec to 3.1.* spec for better support.Optional: Generate Input Questions and Request Ground Truth Queriesâ€‹See Generating Test Datasets at the end of this notebook for more details.# import re# from langchain.prompts import PromptTemplate# template = """Below is a service description:# {spec}# Imagine you're a new user trying to use {operation} through a search bar. What are 10 different things you want to request?# Wants/Questions:# 1. """# prompt = PromptTemplate.from_template(template)# generation_chain = LLMChain(llm=llm, prompt=prompt)# questions_ = generation_chain.run(spec=operation.to_typescript(), operation=operation.operation_id).split('\n')# # Strip preceding numeric bullets# questions = [re.sub(r'^\d+\. ', '', q).strip() for q in questions_]# questions# ground_truths = [# {"q": ...} # What are the best queries for each input?# ]Run the API Chainâ€‹The two simplest questions a user of the API Chain are:Did the chain succesfully access the endpoint?Did the action accomplish the correct result?from collections import defaultdict# Collect metrics to report at completionscores = defaultdict(list)from langchain.evaluation.loading import load_datasetdataset = load_dataset("openapi-chain-klarna-products-get")    Found cached dataset json


= load_dataset("openapi-chain-klarna-products-get")    Found cached dataset json (/Users/harrisonchase/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--openapi-chain-klarna-products-get-5d03362007667626/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)      0%|          | 0/1 [00:00<?, ?it/s]dataset    [{'question': 'What iPhone models are available?',      'expected_query': {'max_price': None, 'q': 'iPhone'}},     {'question': 'Are there any budget laptops?',      'expected_query': {'max_price': 300, 'q': 'laptop'}},     {'question': 'Show me the cheapest gaming PC.',      'expected_query': {'max_price': 500, 'q': 'gaming pc'}},     {'question': 'Are there any tablets under $400?',      'expected_query': {'max_price': 400, 'q': 'tablet'}},     {'question': 'What are the best headphones?',      'expected_query': {'max_price': None, 'q': 'headphones'}},     {'question': 'What are the top rated laptops?',      'expected_query': {'max_price': None, 'q': 'laptop'}},     {'question': 'I want to buy some shoes. I like Adidas and Nike.',      'expected_query': {'max_price':


I like Adidas and Nike.',      'expected_query': {'max_price': None, 'q': 'shoe'}},     {'question': 'I want to buy a new skirt',      'expected_query': {'max_price': None, 'q': 'skirt'}},     {'question': 'My company is asking me to get a professional Deskopt PC - money is no object.',      'expected_query': {'max_price': 10000, 'q': 'professional desktop PC'}},     {'question': 'What are the best budget cameras?',      'expected_query': {'max_price': 300, 'q': 'camera'}}]questions = [d["question"] for d in dataset]## Run the the API chain itselfraise_error = False  # Stop on first failed example - useful for developmentchain_outputs = []failed_examples = []for question in questions:    try:        chain_outputs.append(api_chain(question))        scores["completed"].append(1.0)    except Exception as e:        if raise_error:            raise e        failed_examples.append({"q": question, "error": e})        scores["completed"].append(0.0)# If the chain failed to run, show the failing examplesfailed_examples    []answers = [res["output"] for res in chain_outputs]answers    ['There are currently 10 Apple iPhone models available: Apple iPhone 14 Pro Max 256GB, Apple iPhone 12 128GB, Apple iPhone 13 128GB, Apple iPhone 14 Pro 128GB, Apple iPhone 14 Pro 256GB, Apple


Apple iPhone 14 Pro 128GB, Apple iPhone 14 Pro 256GB, Apple iPhone 14 Pro Max 128GB, Apple iPhone 13 Pro Max 128GB, Apple iPhone 14 128GB, Apple iPhone 12 Pro 512GB, and Apple iPhone 12 mini 64GB.',     'Yes, there are several budget laptops in the API response. For example, the HP 14-dq0055dx and HP 15-dw0083wm are both priced at $199.99 and $244.99 respectively.',     'The cheapest gaming PC available is the Alarco Gaming PC (X_BLACK_GTX750) for $499.99. You can find more information about it here: https://www.klarna.com/us/shopping/pl/cl223/3203154750/Desktop-Computers/Alarco-Gaming-PC-%28X_BLACK_GTX750%29/?utm_source=openai&ref-site=openai_plugin',     'Yes, there are several tablets under $400. These include the Apple iPad 10.2" 32GB (2019), Samsung Galaxy Tab A8 10.5 SM-X200 32GB, Samsung Galaxy Tab A7 Lite 8.7 SM-T220 32GB, Amazon Fire HD 8" 32GB (10th Generation), and Amazon Fire HD 10 32GB.',     'It looks like you are looking for the best headphones. Based on the API response, it looks like the Apple AirPods Pro (2nd generation) 2022, Apple AirPods Max, and Bose Noise Cancelling Headphones 700 are the best options.',     'The top rated laptops based on the API response are the Apple MacBook Pro (2021) M1 Pro 8C CPU 14C GPU 16GB 512GB SSD


M1 Pro 8C CPU 14C GPU 16GB 512GB SSD 14", Apple MacBook Pro (2022) M2 OC 10C GPU 8GB 256GB SSD 13.3", Apple MacBook Air (2022) M2 OC 8C GPU 8GB 256GB SSD 13.6", and Apple MacBook Pro (2023) M2 Pro OC 16C GPU 16GB 512GB SSD 14.2".',     "I found several Nike and Adidas shoes in the API response. Here are the links to the products: Nike Dunk Low M - Black/White: https://www.klarna.com/us/shopping/pl/cl337/3200177969/Shoes/Nike-Dunk-Low-M-Black-White/?utm_source=openai&ref-site=openai_plugin, Nike Air Jordan 4 Retro M - Midnight Navy: https://www.klarna.com/us/shopping/pl/cl337/3202929835/Shoes/Nike-Air-Jordan-4-Retro-M-Midnight-Navy/?utm_source=openai&ref-site=openai_plugin, Nike Air Force 1 '07 M - White: https://www.klarna.com/us/shopping/pl/cl337/3979297/Shoes/Nike-Air-Force-1-07-M-White/?utm_source=openai&ref-site=openai_plugin, Nike Dunk Low W - White/Black: https://www.klarna.com/us/shopping/pl/cl337/3200134705/Shoes/Nike-Dunk-Low-W-White-Black/?utm_source=openai&ref-site=openai_plugin, Nike Air Jordan 1 Retro High M - White/University Blue/Black:


Nike Air Jordan 1 Retro High M - White/University Blue/Black: https://www.klarna.com/us/shopping/pl/cl337/3200383658/Shoes/Nike-Air-Jordan-1-Retro-High-M-White-University-Blue-Black/?utm_source=openai&ref-site=openai_plugin, Nike Air Jordan 1 Retro High OG M - True Blue/Cement Grey/White: https://www.klarna.com/us/shopping/pl/cl337/3204655673/Shoes/Nike-Air-Jordan-1-Retro-High-OG-M-True-Blue-Cement-Grey-White/?utm_source=openai&ref-site=openai_plugin, Nike Air Jordan 11 Retro Cherry - White/Varsity Red/Black: https://www.klarna.com/us/shopping/pl/cl337/3202929696/Shoes/Nike-Air-Jordan-11-Retro-Cherry-White-Varsity-Red-Black/?utm_source=openai&ref-site=openai_plugin, Nike Dunk High W - White/Black: https://www.klarna.com/us/shopping/pl/cl337/3201956448/Shoes/Nike-Dunk-High-W-White-Black/?utm_source=openai&ref-site=openai_plugin, Nike Air Jordan 5 Retro M - Black/Taxi/Aquatone: https://www.klarna.com/us/shopping/pl/cl337/3204923084/Shoes/Nike-Air-Jordan-5-Retro-M-Black-Taxi-Aquatone/?utm_source=openai&ref-site=openai_plugin, Nike Court Legacy Lift W: https://www.klarna.com/us/shopping/pl/cl337/3202103728/Shoes/Nike-Court-Legacy-Lift-W/?utm_source=openai&ref-site=openai_plugin",     "I found several skirts that may interest you. Please take a look at the following


"I found several skirts that may interest you. Please take a look at the following products: Avenue Plus Size Denim Stretch Skirt, LoveShackFancy Ruffled Mini Skirt - Antique White, Nike Dri-Fit Club Golf Skirt - Active Pink, Skims Soft Lounge Ruched Long Skirt, French Toast Girl's Front Pleated Skirt with Tabs, Alexia Admor Women's Harmonie Mini Skirt Pink Pink, Vero Moda Long Skirt, Nike Court Dri-FIT Victory Flouncy Tennis Skirt Women - White/Black, Haoyuan Mini Pleated Skirts W, and Zimmermann Lyre Midi Skirt.",     'Based on the API response, you may want to consider the Skytech Archangel Gaming Computer PC Desktop, the CyberPowerPC Gamer Master Gaming Desktop, or the ASUS ROG Strix G10DK-RS756, as they all offer powerful processors and plenty of RAM.',     'Based on the API response, the best budget cameras are the DJI Mini 2 Dog Camera ($448.50), Insta360 Sphere with Landing Pad ($429.99), DJI FPV Gimbal Camera ($121.06), Parrot Camera & Body ($36.19), and DJI FPV Air Unit ($179.00).']Evaluate the requests chainâ€‹The API Chain has two main components:Translate the user query to an API request (request synthesizer)Translate the API response to a natural language responseHere, we construct an evaluation chain to grade the request synthesizer against selected human queries import jsontruth_queries = [json.dumps(data["expected_query"]) for data in dataset]# Collect the API queries generated by the chainpredicted_queries = [    output["intermediate_steps"]["request_args"] for output in chain_outputs]from langchain.prompts import PromptTemplatetemplate = """You are trying to answer the following


langchain.prompts import PromptTemplatetemplate = """You are trying to answer the following question by querying an API:> Question: {question}The query you know you should be executing against the API is:> Query: {truth_query}Is the following predicted query semantically the same (eg likely to produce the same answer)?> Predicted Query: {predict_query}Please give the Predicted Query a grade of either an A, B, C, D, or F, along with an explanation of why. End the evaluation with 'Final Grade: <the letter>'> Explanation: Let's think step by step."""prompt = PromptTemplate.from_template(template)eval_chain = LLMChain(llm=llm, prompt=prompt, verbose=verbose)request_eval_results = []for question, predict_query, truth_query in list(    zip(questions, predicted_queries, truth_queries)):    eval_output = eval_chain.run(        question=question,        truth_query=truth_query,        predict_query=predict_query,    )    request_eval_results.append(eval_output)request_eval_results    [' The original query is asking for all iPhone models, so the "q" parameter is correct. The "max_price" parameter is also correct, as it is set to null, meaning that no maximum price is set. The predicted query adds two additional parameters, "size" and "min_price". The "size" parameter is not necessary, as it is not relevant to the question being asked. The "min_price" parameter is also not necessary, as it is not relevant to the question being asked and it is set to 0, which is the default value. Therefore, the predicted query is not semantically the same as the original query and is not likely to produce the same answer. Final Grade: D',


original query and is not likely to produce the same answer. Final Grade: D',     ' The original query is asking for laptops with a maximum price of 300. The predicted query is asking for laptops with a minimum price of 0 and a maximum price of 500. This means that the predicted query is likely to return more results than the original query, as it is asking for a wider range of prices. Therefore, the predicted query is not semantically the same as the original query, and it is not likely to produce the same answer. Final Grade: F',     " The first two parameters are the same, so that's good. The third parameter is different, but it's not necessary for the query, so that's not a problem. The fourth parameter is the problem. The original query specifies a maximum price of 500, while the predicted query specifies a maximum price of null. This means that the predicted query will not limit the results to the cheapest gaming PCs, so it is not semantically the same as the original query. Final Grade: F",     ' The original query is asking for tablets under $400, so the first two parameters are correct. The predicted query also includes the parameters "size" and "min_price", which are not necessary for the original query. The "size" parameter is not relevant to the question, and the "min_price" parameter is redundant since the original query already specifies a maximum price. Therefore, the predicted query is not semantically the same as the original query and is not likely to produce the same answer. Final Grade: D',     ' The original query is asking for headphones with no maximum price, so the predicted query is not semantically the same because it has a maximum price of 500. The predicted query also has a size of 10, which is not specified in the original query. Therefore, the predicted query is not semantically the same


which is not specified in the original query. Therefore, the predicted query is not semantically the same as the original query. Final Grade: F',     " The original query is asking for the top rated laptops, so the 'size' parameter should be set to 10 to get the top 10 results. The 'min_price' parameter should be set to 0 to get results from all price ranges. The 'max_price' parameter should be set to null to get results from all price ranges. The 'q' parameter should be set to 'laptop' to get results related to laptops. All of these parameters are present in the predicted query, so it is semantically the same as the original query. Final Grade: A",     ' The original query is asking for shoes, so the predicted query is asking for the same thing. The original query does not specify a size, so the predicted query is not adding any additional information. The original query does not specify a price range, so the predicted query is adding additional information that is not necessary. Therefore, the predicted query is not semantically the same as the original query and is likely to produce different results. Final Grade: D',     ' The original query is asking for a skirt, so the predicted query is asking for the same thing. The predicted query also adds additional parameters such as size and price range, which could help narrow down the results. However, the size parameter is not necessary for the query to be successful, and the price range is too narrow. Therefore, the predicted query is not as effective as the original query. Final Grade: C',     ' The first part of the query is asking for a Desktop PC, which is the same as the original query. The second part of the query is asking for a size of 10, which is not relevant to the original query. The third part of the query is asking for a minimum price


is not relevant to the original query. The third part of the query is asking for a minimum price of 0, which is not relevant to the original query. The fourth part of the query is asking for a maximum price of null, which is not relevant to the original query. Therefore, the Predicted Query does not semantically match the original query and is not likely to produce the same answer. Final Grade: F',     ' The original query is asking for cameras with a maximum price of 300. The predicted query is asking for cameras with a maximum price of 500. This means that the predicted query is likely to return more results than the original query, which may include cameras that are not within the budget range. Therefore, the predicted query is not semantically the same as the original query and does not answer the original question. Final Grade: F']import refrom typing import List# Parse the evaluation chain responses into a rubricdef parse_eval_results(results: List[str]) -> List[float]:    rubric = {"A": 1.0, "B": 0.75, "C": 0.5, "D": 0.25, "F": 0}    return [rubric[re.search(r"Final Grade: (\w+)", res).group(1)] for res in results]parsed_results = parse_eval_results(request_eval_results)# Collect the scores for a final evaluation tablescores["request_synthesizer"].extend(parsed_results)Evaluate the Response Chainâ€‹The second component translated the structured API response to a natural language response.


Evaluate this against the user's original question.from langchain.prompts import PromptTemplatetemplate = """You are trying to answer the following question by querying an API:> Question: {question}The API returned a response of:> API result: {api_response}Your response to the user: {answer}Please evaluate the accuracy and utility of your response to the user's original question, conditioned on the information available.Give a letter grade of either an A, B, C, D, or F, along with an explanation of why. End the evaluation with 'Final Grade: <the letter>'> Explanation: Let's think step by step."""prompt = PromptTemplate.from_template(template)eval_chain = LLMChain(llm=llm, prompt=prompt, verbose=verbose)# Extract the API responses from the chainapi_responses = [    output["intermediate_steps"]["response_text"] for output in chain_outputs]# Run the grader chainresponse_eval_results = []for question, api_response, answer in list(zip(questions, api_responses, answers)):    request_eval_results.append(        eval_chain.run(question=question, api_response=api_response, answer=answer)    )request_eval_results    [' The original query is asking for all iPhone models, so the "q" parameter is correct. The "max_price" parameter is also correct, as it is set to null, meaning that no maximum price is set. The predicted query adds two additional parameters, "size" and "min_price". The "size" parameter is not necessary, as it is not relevant to the question being asked. The "min_price" parameter is also not necessary, as it is not relevant to the question being asked and it is set to 0, which is the default value. Therefore, the predicted query is not semantically the same as the original query and is not likely to produce the same


predicted query is not semantically the same as the original query and is not likely to produce the same answer. Final Grade: D',     ' The original query is asking for laptops with a maximum price of 300. The predicted query is asking for laptops with a minimum price of 0 and a maximum price of 500. This means that the predicted query is likely to return more results than the original query, as it is asking for a wider range of prices. Therefore, the predicted query is not semantically the same as the original query, and it is not likely to produce the same answer. Final Grade: F',     " The first two parameters are the same, so that's good. The third parameter is different, but it's not necessary for the query, so that's not a problem. The fourth parameter is the problem. The original query specifies a maximum price of 500, while the predicted query specifies a maximum price of null. This means that the predicted query will not limit the results to the cheapest gaming PCs, so it is not semantically the same as the original query. Final Grade: F",     ' The original query is asking for tablets under $400, so the first two parameters are correct. The predicted query also includes the parameters "size" and "min_price", which are not necessary for the original query. The "size" parameter is not relevant to the question, and the "min_price" parameter is redundant since the original query already specifies a maximum price. Therefore, the predicted query is not semantically the same as the original query and is not likely to produce the same answer. Final Grade: D',     ' The original query is asking for headphones with no maximum price, so the predicted query is not semantically the same because it has a maximum price of 500. The predicted query also has a size of 10, which is not specified in the original query.


The predicted query also has a size of 10, which is not specified in the original query. Therefore, the predicted query is not semantically the same as the original query. Final Grade: F',     " The original query is asking for the top rated laptops, so the 'size' parameter should be set to 10 to get the top 10 results. The 'min_price' parameter should be set to 0 to get results from all price ranges. The 'max_price' parameter should be set to null to get results from all price ranges. The 'q' parameter should be set to 'laptop' to get results related to laptops. All of these parameters are present in the predicted query, so it is semantically the same as the original query. Final Grade: A",     ' The original query is asking for shoes, so the predicted query is asking for the same thing. The original query does not specify a size, so the predicted query is not adding any additional information. The original query does not specify a price range, so the predicted query is adding additional information that is not necessary. Therefore, the predicted query is not semantically the same as the original query and is likely to produce different results. Final Grade: D',     ' The original query is asking for a skirt, so the predicted query is asking for the same thing. The predicted query also adds additional parameters such as size and price range, which could help narrow down the results. However, the size parameter is not necessary for the query to be successful, and the price range is too narrow. Therefore, the predicted query is not as effective as the original query. Final Grade: C',     ' The first part of the query is asking for a Desktop PC, which is the same as the original query. The second part of the query is asking for a size of 10, which is not relevant to the original query. The


query is asking for a size of 10, which is not relevant to the original query. The third part of the query is asking for a minimum price of 0, which is not relevant to the original query. The fourth part of the query is asking for a maximum price of null, which is not relevant to the original query. Therefore, the Predicted Query does not semantically match the original query and is not likely to produce the same answer. Final Grade: F',     ' The original query is asking for cameras with a maximum price of 300. The predicted query is asking for cameras with a maximum price of 500. This means that the predicted query is likely to return more results than the original query, which may include cameras that are not within the budget range. Therefore, the predicted query is not semantically the same as the original query and does not answer the original question. Final Grade: F',     ' The user asked a question about what iPhone models are available, and the API returned a response with 10 different models. The response provided by the user accurately listed all 10 models, so the accuracy of the response is A+. The utility of the response is also A+ since the user was able to get the exact information they were looking for. Final Grade: A+',     " The API response provided a list of laptops with their prices and attributes. The user asked if there were any budget laptops, and the response provided a list of laptops that are all priced under $500. Therefore, the response was accurate and useful in answering the user's question. Final Grade: A",     " The API response provided the name, price, and URL of the product, which is exactly what the user asked for. The response also provided additional information about the product's attributes, which is useful for the user to make an informed decision. Therefore, the response is accurate and useful. Final Grade:


the user to make an informed decision. Therefore, the response is accurate and useful. Final Grade: A",     " The API response provided a list of tablets that are under $400. The response accurately answered the user's question. Additionally, the response provided useful information such as the product name, price, and attributes. Therefore, the response was accurate and useful. Final Grade: A",     " The API response provided a list of headphones with their respective prices and attributes. The user asked for the best headphones, so the response should include the best headphones based on the criteria provided. The response provided a list of headphones that are all from the same brand (Apple) and all have the same type of headphone (True Wireless, In-Ear). This does not provide the user with enough information to make an informed decision about which headphones are the best. Therefore, the response does not accurately answer the user's question. Final Grade: F",     ' The API response provided a list of laptops with their attributes, which is exactly what the user asked for. The response provided a comprehensive list of the top rated laptops, which is what the user was looking for. The response was accurate and useful, providing the user with the information they needed. Final Grade: A',     ' The API response provided a list of shoes from both Adidas and Nike, which is exactly what the user asked for. The response also included the product name, price, and attributes for each shoe, which is useful information for the user to make an informed decision. The response also included links to the products, which is helpful for the user to purchase the shoes. Therefore, the response was accurate and useful. Final Grade: A',     " The API response provided a list of skirts that could potentially meet the user's needs. The response also included the name, price, and attributes of each skirt. This is a great start, as it


included the name, price, and attributes of each skirt. This is a great start, as it provides the user with a variety of options to choose from. However, the response does not provide any images of the skirts, which would have been helpful for the user to make a decision. Additionally, the response does not provide any information about the availability of the skirts, which could be important for the user. \n\nFinal Grade: B",     ' The user asked for a professional desktop PC with no budget constraints. The API response provided a list of products that fit the criteria, including the Skytech Archangel Gaming Computer PC Desktop, the CyberPowerPC Gamer Master Gaming Desktop, and the ASUS ROG Strix G10DK-RS756. The response accurately suggested these three products as they all offer powerful processors and plenty of RAM. Therefore, the response is accurate and useful. Final Grade: A',     " The API response provided a list of cameras with their prices, which is exactly what the user asked for. The response also included additional information such as features and memory cards, which is not necessary for the user's question but could be useful for further research. The response was accurate and provided the user with the information they needed. Final Grade: A"]# Reusing the rubric from above, parse the evaluation chain responsesparsed_response_results = parse_eval_results(request_eval_results)# Collect the scores for a final evaluation tablescores["result_synthesizer"].extend(parsed_response_results)# Print out Score statistics for the evaluation sessionheader = "{:<20}\t{:<10}\t{:<10}\t{:<10}".format("Metric", "Min", "Mean", "Max")print(header)for metric, metric_scores in scores.items():    mean_scores = (        sum(metric_scores) / len(metric_scores)        if len(metric_scores) > 0


len(metric_scores)        if len(metric_scores) > 0        else float("nan")    )    row = "{:<20}\t{:<10.2f}\t{:<10.2f}\t{:<10.2f}".format(        metric, min(metric_scores), mean_scores, max(metric_scores)    )    print(row)    Metric                  Min         Mean        Max           completed               1.00        1.00        1.00          request_synthesizer     0.00        0.23        1.00          result_synthesizer      0.00        0.55        1.00      # Re-show the examples for which the chain failed to completefailed_examples    []Generating Test Datasetsâ€‹To evaluate a chain against your own endpoint, you'll want to generate a test dataset that's conforms to the API.This section provides an overview of how to bootstrap the process.First, we'll parse the OpenAPI Spec. For this example, we'll Speak's OpenAPI specification.# Load and parse the OpenAPI Specspec = OpenAPISpec.from_url("https://api.speak.com/openapi.yaml")    Attempting to load an OpenAPI


Attempting to load an OpenAPI 3.0.1 spec.  This may result in degraded performance. Convert your OpenAPI spec to 3.1.* spec for better support.    Attempting to load an OpenAPI 3.0.1 spec.  This may result in degraded performance. Convert your OpenAPI spec to 3.1.* spec for better support.# List the paths in the OpenAPI Specpaths = sorted(spec.paths.keys())paths    ['/v1/public/openai/explain-phrase',     '/v1/public/openai/explain-task',     '/v1/public/openai/translate']# See which HTTP Methods are available for a given pathmethods = spec.get_methods_for_path("/v1/public/openai/explain-task")methods    ['post']# Load a single endpoint operationoperation = APIOperation.from_openapi_spec(    spec, "/v1/public/openai/explain-task", "post")# The operation can be serialized as typescriptprint(operation.to_typescript())    type explainTask = (_: {    /* Description of the task that the user wants to accomplish or do. For example, "tell the waiter they messed up my order" or "compliment someone on their shirt" */      task_description?: string,    /* The foreign language that the user is learning and asking about. The value can be inferred from question - for example, if the user asks "how do i ask a girl out in mexico city", the value should be "Spanish" because of Mexico City. Always use the full name of the language (e.g. Spanish, French). */      learning_language?: string,    /* The user's native language. Infer this value from the language the user asked their question in. Always use the full name of the language


this value from the language the user asked their question in. Always use the full name of the language (e.g. Spanish, French). */      native_language?: string,    /* A description of any additional context in the user's question that could affect the explanation - e.g. setting, scenario, situation, tone, speaking style and formality, usage notes, or any other qualifiers. */      additional_context?: string,    /* Full text of the user's question. */      full_query?: string,    }) => any;# Compress the service definition to avoid leaking too much input structure to the sample datatemplate = """In 20 words or less, what does this service accomplish?{spec}Function: It's designed to """prompt = PromptTemplate.from_template(template)generation_chain = LLMChain(llm=llm, prompt=prompt)purpose = generation_chain.run(spec=operation.to_typescript())template = """Write a list of {num_to_generate} unique messages users might send to a service designed to{purpose} They must each be completely unique.1."""def parse_list(text: str) -> List[str]:    # Match lines starting with a number then period    # Strip leading and trailing whitespace    matches = re.findall(r"^\d+\. ", text)    return [re.sub(r"^\d+\. ", "", q).strip().strip('"') for q in text.split("\n")]num_to_generate = 10  # How many examples to use for this test set.prompt = PromptTemplate.from_template(template)generation_chain = LLMChain(llm=llm, prompt=prompt)text = generation_chain.run(purpose=purpose, num_to_generate=num_to_generate)# Strip preceding numeric bulletsqueries = parse_list(text)queries    ["Can you explain how to say 'hello'


= parse_list(text)queries    ["Can you explain how to say 'hello' in Spanish?",     "I need help understanding the French word for 'goodbye'.",     "Can you tell me how to say 'thank you' in German?",     "I'm trying to learn the Italian word for 'please'.",     "Can you help me with the pronunciation of 'yes' in Portuguese?",     "I'm looking for the Dutch word for 'no'.",     "Can you explain the meaning of 'hello' in Japanese?",     "I need help understanding the Russian word for 'thank you'.",     "Can you tell me how to say 'goodbye' in Chinese?",     "I'm trying to learn the Arabic word for 'please'."]# Define the generation chain to get hypothesesapi_chain = OpenAPIEndpointChain.from_api_operation(    operation,    llm,    requests=Requests(),    verbose=verbose,    return_intermediate_steps=True,  # Return request and response text)predicted_outputs = [api_chain(query) for query in queries]request_args = [    output["intermediate_steps"]["request_args"] for output in predicted_outputs]# Show the generated requestrequest_args    ['{"task_description": "say \'hello\'", "learning_language": "Spanish", "native_language": "English", "full_query": "Can you explain how to say \'hello\' in Spanish?"}',     '{"task_description": "understanding the French word for \'goodbye\'", "learning_language": "French", "native_language": "English", "full_query": "I need help understanding the French word for \'goodbye\'."}',     '{"task_description": "say \'thank


word for \'goodbye\'."}',     '{"task_description": "say \'thank you\'", "learning_language": "German", "native_language": "English", "full_query": "Can you tell me how to say \'thank you\' in German?"}',     '{"task_description": "Learn the Italian word for \'please\'", "learning_language": "Italian", "native_language": "English", "full_query": "I\'m trying to learn the Italian word for \'please\'."}',     '{"task_description": "Help with pronunciation of \'yes\' in Portuguese", "learning_language": "Portuguese", "native_language": "English", "full_query": "Can you help me with the pronunciation of \'yes\' in Portuguese?"}',     '{"task_description": "Find the Dutch word for \'no\'", "learning_language": "Dutch", "native_language": "English", "full_query": "I\'m looking for the Dutch word for \'no\'."}',     '{"task_description": "Explain the meaning of \'hello\' in Japanese", "learning_language": "Japanese", "native_language": "English", "full_query": "Can you explain the meaning of \'hello\' in Japanese?"}',     '{"task_description": "understanding the Russian word for \'thank you\'", "learning_language": "Russian", "native_language": "English", "full_query": "I need help understanding the Russian word for \'thank you\'."}',     '{"task_description": "say goodbye", "learning_language": "Chinese", "native_language": "English", "full_query": "Can you tell me how to say \'goodbye\' in Chinese?"}',     '{"task_description": "Learn the Arabic word for \'please\'", "learning_language": "Arabic", "native_language": "English",


for \'please\'", "learning_language": "Arabic", "native_language": "English", "full_query": "I\'m trying to learn the Arabic word for \'please\'."}']## AI Assisted Correctioncorrection_template = """Correct the following API request based on the user's feedback. If the user indicates no changes are needed, output the original without making any changes.REQUEST: {request}User Feedback / requested changes: {user_feedback}Finalized Request: """prompt = PromptTemplate.from_template(correction_template)correction_chain = LLMChain(llm=llm, prompt=prompt)ground_truth = []for query, request_arg in list(zip(queries, request_args)):    feedback = input(f"Query: {query}\nRequest: {request_arg}\nRequested changes: ")    if feedback == "n" or feedback == "none" or not feedback:        ground_truth.append(request_arg)        continue    resolved = correction_chain.run(request=request_arg, user_feedback=feedback)    ground_truth.append(resolved.strip())    print("Updated request:", resolved)    Query: Can you explain how to say 'hello' in Spanish?    Request: {"task_description": "say 'hello'", "learning_language": "Spanish", "native_language": "English", "full_query": "Can you explain how to say 'hello' in Spanish?"}    Requested changes:     Query: I need help understanding the French word for 'goodbye'.    Request: {"task_description": "understanding the French word for 'goodbye'", "learning_language": "French", "native_language": "English", "full_query": "I need help understanding the French word for 'goodbye'."}    Requested changes:     Query:


for 'goodbye'."}    Requested changes:     Query: Can you tell me how to say 'thank you' in German?    Request: {"task_description": "say 'thank you'", "learning_language": "German", "native_language": "English", "full_query": "Can you tell me how to say 'thank you' in German?"}    Requested changes:     Query: I'm trying to learn the Italian word for 'please'.    Request: {"task_description": "Learn the Italian word for 'please'", "learning_language": "Italian", "native_language": "English", "full_query": "I'm trying to learn the Italian word for 'please'."}    Requested changes:     Query: Can you help me with the pronunciation of 'yes' in Portuguese?    Request: {"task_description": "Help with pronunciation of 'yes' in Portuguese", "learning_language": "Portuguese", "native_language": "English", "full_query": "Can you help me with the pronunciation of 'yes' in Portuguese?"}    Requested changes:     Query: I'm looking for the Dutch word for 'no'.    Request: {"task_description": "Find the Dutch word for 'no'", "learning_language": "Dutch", "native_language": "English", "full_query": "I'm looking for the Dutch word for 'no'."}    Requested changes:     Query: Can you explain the meaning of 'hello' in Japanese?    Request: {"task_description": "Explain the meaning of 'hello' in Japanese", "learning_language": "Japanese", "native_language": "English", "full_query": "Can you explain the meaning of 'hello' in Japanese?"}    Requested


"Can you explain the meaning of 'hello' in Japanese?"}    Requested changes:     Query: I need help understanding the Russian word for 'thank you'.    Request: {"task_description": "understanding the Russian word for 'thank you'", "learning_language": "Russian", "native_language": "English", "full_query": "I need help understanding the Russian word for 'thank you'."}    Requested changes:     Query: Can you tell me how to say 'goodbye' in Chinese?    Request: {"task_description": "say goodbye", "learning_language": "Chinese", "native_language": "English", "full_query": "Can you tell me how to say 'goodbye' in Chinese?"}    Requested changes:     Query: I'm trying to learn the Arabic word for 'please'.    Request: {"task_description": "Learn the Arabic word for 'please'", "learning_language": "Arabic", "native_language": "English", "full_query": "I'm trying to learn the Arabic word for 'please'."}    Requested changes: Now you can use the ground_truth as shown above in Evaluate the Requests Chain!# Now you have a new ground truth set to use as shown above!ground_truth    ['{"task_description": "say \'hello\'", "learning_language": "Spanish", "native_language": "English", "full_query": "Can you explain how to say \'hello\' in Spanish?"}',     '{"task_description": "understanding the French word for \'goodbye\'", "learning_language": "French", "native_language": "English", "full_query": "I need help understanding the French word for \'goodbye\'."}',     '{"task_description": "say \'thank you\'",


'{"task_description": "say \'thank you\'", "learning_language": "German", "native_language": "English", "full_query": "Can you tell me how to say \'thank you\' in German?"}',     '{"task_description": "Learn the Italian word for \'please\'", "learning_language": "Italian", "native_language": "English", "full_query": "I\'m trying to learn the Italian word for \'please\'."}',     '{"task_description": "Help with pronunciation of \'yes\' in Portuguese", "learning_language": "Portuguese", "native_language": "English", "full_query": "Can you help me with the pronunciation of \'yes\' in Portuguese?"}',     '{"task_description": "Find the Dutch word for \'no\'", "learning_language": "Dutch", "native_language": "English", "full_query": "I\'m looking for the Dutch word for \'no\'."}',     '{"task_description": "Explain the meaning of \'hello\' in Japanese", "learning_language": "Japanese", "native_language": "English", "full_query": "Can you explain the meaning of \'hello\' in Japanese?"}',     '{"task_description": "understanding the Russian word for \'thank you\'", "learning_language": "Russian", "native_language": "English", "full_query": "I need help understanding the Russian word for \'thank you\'."}',     '{"task_description": "say goodbye", "learning_language": "Chinese", "native_language": "English", "full_query": "Can you tell me how to say \'goodbye\' in Chinese?"}',     '{"task_description": "Learn the Arabic word for \'please\'", "learning_language": "Arabic", "native_language": "English", "full_query": "I\'m


"Arabic", "native_language": "English", "full_query": "I\'m trying to learn the Arabic word for \'please\'."}']PreviousData Augmented Question AnsweringNextQuestion Answering Benchmarking: Paul Graham EssayLoad the API ChainOptional: Generate Input Questions and Request Ground Truth QueriesRun the API ChainEvaluate the requests chainEvaluate the Response ChainGenerating Test DatasetsCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


Question Answering Benchmarking: State of the Union Address | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionChainsMemoryAgentsCallbacksModulesGuidesEvaluationString EvaluatorsComparison EvaluatorsTrajectory EvaluatorsExamplesAgent VectorDB Question Answering BenchmarkingComparing Chain OutputsData Augmented Question AnsweringEvaluating an OpenAPI ChainQuestion Answering Benchmarking: Paul Graham EssayQuestion Answering Benchmarking: State of the Union AddressQA GenerationQuestion AnsweringSQL Question Answering Benchmarking: ChinookDebuggingDeploymentLangSmithModel ComparisonEcosystemAdditional resourcesGuidesEvaluationExamplesQuestion Answering Benchmarking: State of the Union AddressOn this pageQuestion Answering Benchmarking: State of the Union AddressHere we go over how to benchmark performance on a question answering task over a state of the union address.It is highly reccomended that you do any evaluation/benchmarking with tracing enabled. See here for an explanation of what tracing is and how to set it up.# Comment this out if you are NOT using tracingimport osos.environ["LANGCHAIN_HANDLER"] = "langchain"Loading the dataâ€‹First, let's load the data.from langchain.evaluation.loading import load_datasetdataset = load_dataset("question-answering-state-of-the-union")    Found cached dataset json (/Users/harrisonchase/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--question-answering-state-of-the-union-a7e5a3b2db4f440d/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)      0%|


0%|          | 0/1 [00:00<?, ?it/s]Setting up a chainâ€‹Now we need to create some pipelines for doing question answering. Step one in that is creating an index over the data in question.from langchain.document_loaders import TextLoaderloader = TextLoader("../../modules/state_of_the_union.txt")from langchain.indexes import VectorstoreIndexCreatorvectorstore = VectorstoreIndexCreator().from_loaders([loader]).vectorstore    Running Chroma using direct local API.    Using DuckDB in-memory for database. Data will be transient.Now we can create a question answering chain.from langchain.chains import RetrievalQAfrom langchain.llms import OpenAIchain = RetrievalQA.from_chain_type(    llm=OpenAI(),    chain_type="stuff",    retriever=vectorstore.as_retriever(),    input_key="question",)Make a predictionâ€‹First, we can make predictions one datapoint at a time. Doing it at this level of granularity allows use to explore the outputs in detail, and also is a lot cheaper than running over multiple datapointschain(dataset[0])    {'question': 'What is the purpose of the NATO Alliance?',     'answer': 'The purpose of the NATO Alliance is to secure peace and stability in Europe after World War 2.',     'result': ' The NATO Alliance was created to secure peace and stability in Europe after World War 2.'}Make many predictionsâ€‹Now we can make predictionspredictions = chain.apply(dataset)Evaluate performanceâ€‹Now we can evaluate the predictions. The first thing we can do is look at them by eye.predictions[0]    {'question': 'What is the purpose of the NATO Alliance?',


{'question': 'What is the purpose of the NATO Alliance?',     'answer': 'The purpose of the NATO Alliance is to secure peace and stability in Europe after World War 2.',     'result': ' The purpose of the NATO Alliance is to secure peace and stability in Europe after World War 2.'}Next, we can use a language model to score them programaticallyfrom langchain.evaluation.qa import QAEvalChainllm = OpenAI(temperature=0)eval_chain = QAEvalChain.from_llm(llm)graded_outputs = eval_chain.evaluate(    dataset, predictions, question_key="question", prediction_key="result")We can add in the graded output to the predictions dict and then get a count of the grades.for i, prediction in enumerate(predictions):    prediction["grade"] = graded_outputs[i]["text"]from collections import CounterCounter([pred["grade"] for pred in predictions])    Counter({' CORRECT': 7, ' INCORRECT': 4})We can also filter the datapoints to the incorrect examples and look at them.incorrect = [pred for pred in predictions if pred["grade"] == " INCORRECT"]incorrect[0]    {'question': 'What is the U.S. Department of Justice doing to combat the crimes of Russian oligarchs?',     'answer': 'The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs.',     'result': ' The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and is naming a chief prosecutor for pandemic fraud.',     'grade': ' INCORRECT'}PreviousQuestion Answering Benchmarking: Paul Graham EssayNextQA GenerationLoading the dataSetting up a chainMake a predictionMake many predictionsEvaluate


Paul Graham EssayNextQA GenerationLoading the dataSetting up a chainMake a predictionMake many predictionsEvaluate performanceCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


Question Answering Benchmarking: Paul Graham Essay | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionChainsMemoryAgentsCallbacksModulesGuidesEvaluationString EvaluatorsComparison EvaluatorsTrajectory EvaluatorsExamplesAgent VectorDB Question Answering BenchmarkingComparing Chain OutputsData Augmented Question AnsweringEvaluating an OpenAPI ChainQuestion Answering Benchmarking: Paul Graham EssayQuestion Answering Benchmarking: State of the Union AddressQA GenerationQuestion AnsweringSQL Question Answering Benchmarking: ChinookDebuggingDeploymentLangSmithModel ComparisonEcosystemAdditional resourcesGuidesEvaluationExamplesQuestion Answering Benchmarking: Paul Graham EssayOn this pageQuestion Answering Benchmarking: Paul Graham EssayHere we go over how to benchmark performance on a question answering task over a Paul Graham essay.It is highly recommended that you do any evaluation/benchmarking with tracing enabled. See here for an explanation of what tracing is and how to set it up.Loading the dataâ€‹First, let's load the data.from langchain.evaluation.loading import load_datasetdataset = load_dataset("question-answering-paul-graham")    Found cached dataset json (/Users/harrisonchase/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--question-answering-paul-graham-76e8f711e038d742/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)      0%|          | 0/1 [00:00<?, ?it/s]Setting up a chainâ€‹Now we need to create some


?it/s]Setting up a chainâ€‹Now we need to create some pipelines for doing question answering. Step one in that is creating an index over the data in question.from langchain.document_loaders import TextLoaderloader = TextLoader("../../modules/paul_graham_essay.txt")from langchain.indexes import VectorstoreIndexCreatorvectorstore = VectorstoreIndexCreator().from_loaders([loader]).vectorstore    Running Chroma using direct local API.    Using DuckDB in-memory for database. Data will be transient.Now we can create a question answering chain.from langchain.chains import RetrievalQAfrom langchain.llms import OpenAIchain = RetrievalQA.from_chain_type(    llm=OpenAI(),    chain_type="stuff",    retriever=vectorstore.as_retriever(),    input_key="question",)Make a predictionâ€‹First, we can make predictions one datapoint at a time. Doing it at this level of granularity allows use to explore the outputs in detail, and also is a lot cheaper than running over multiple datapointschain(dataset[0])    {'question': 'What were the two main things the author worked on before college?',     'answer': 'The two main things the author worked on before college were writing and programming.',     'result': ' Writing and programming.'}Make many predictionsâ€‹Now we can make predictionspredictions = chain.apply(dataset)Evaluate performanceâ€‹Now we can evaluate the predictions. The first thing we can do is look at them by eye.predictions[0]    {'question': 'What were the two main things the author worked on before college?',     'answer': 'The two main things the author worked on before college were writing and programming.',     'result': ' Writing and


author worked on before college were writing and programming.',     'result': ' Writing and programming.'}Next, we can use a language model to score them programaticallyfrom langchain.evaluation.qa import QAEvalChainllm = OpenAI(temperature=0)eval_chain = QAEvalChain.from_llm(llm)graded_outputs = eval_chain.evaluate(    dataset, predictions, question_key="question", prediction_key="result")We can add in the graded output to the predictions dict and then get a count of the grades.for i, prediction in enumerate(predictions):    prediction["grade"] = graded_outputs[i]["text"]from collections import CounterCounter([pred["grade"] for pred in predictions])    Counter({' CORRECT': 12, ' INCORRECT': 10})We can also filter the datapoints to the incorrect examples and look at them.incorrect = [pred for pred in predictions if pred["grade"] == " INCORRECT"]incorrect[0]    {'question': 'What did the author write their dissertation on?',     'answer': 'The author wrote their dissertation on applications of continuations.',     'result': ' The author does not mention what their dissertation was on, so it is not known.',     'grade': ' INCORRECT'}PreviousEvaluating an OpenAPI ChainNextQuestion Answering Benchmarking: State of the Union AddressLoading the dataSetting up a chainMake a predictionMake many predictionsEvaluate performanceCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


Agent VectorDB Question Answering Benchmarking | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionChainsMemoryAgentsCallbacksModulesGuidesEvaluationString EvaluatorsComparison EvaluatorsTrajectory EvaluatorsExamplesAgent VectorDB Question Answering BenchmarkingComparing Chain OutputsData Augmented Question AnsweringEvaluating an OpenAPI ChainQuestion Answering Benchmarking: Paul Graham EssayQuestion Answering Benchmarking: State of the Union AddressQA GenerationQuestion AnsweringSQL Question Answering Benchmarking: ChinookDebuggingDeploymentLangSmithModel ComparisonEcosystemAdditional resourcesGuidesEvaluationExamplesAgent VectorDB Question Answering BenchmarkingOn this pageAgent VectorDB Question Answering BenchmarkingHere we go over how to benchmark performance on a question answering task using an agent to route between multiple vectordatabases.It is highly recommended that you do any evaluation/benchmarking with tracing enabled. See here for an explanation of what tracing is and how to set it up.Loading the dataâ€‹First, let's load the data.from langchain.evaluation.loading import load_datasetdataset = load_dataset("agent-vectordb-qa-sota-pg")    Found cached dataset json (/Users/qt/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--agent-vectordb-qa-sota-pg-d3ae24016b514f92/0.0.0/fe5dd6ea2639a6df622901539cb550cf8797e5a6b2dd7af1cf934bed8e233e6e)


100%|â–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆâ–ˆ| 1/1 [00:00<00:00, 414.42it/s]dataset[0]    {'question': 'What is the purpose of the NATO Alliance?',     'answer': 'The purpose of the NATO Alliance is to secure peace and stability in Europe after World War 2.',     'steps': [{'tool': 'State of Union QA System', 'tool_input': None},      {'tool': None, 'tool_input': 'What is the purpose of the NATO Alliance?'}]}dataset[-1]    {'question': 'What is the purpose of YC?',     'answer': 'The purpose of YC is to cause startups to be founded that would not otherwise have existed.',     'steps': [{'tool': 'Paul Graham QA System', 'tool_input': None},      {'tool': None, 'tool_input': 'What is the purpose of YC?'}]}Setting up a chainâ€‹Now we need to create some pipelines for doing question answering. Step one in that is creating indexes over the data in question.from langchain.document_loaders import TextLoaderloader = TextLoader("../../modules/state_of_the_union.txt")from langchain.indexes import VectorstoreIndexCreatorvectorstore_sota = (    VectorstoreIndexCreator(vectorstore_kwargs={"collection_name": "sota"})    .from_loaders([loader])    .vectorstore)    Using embedded DuckDB without persistence: data will be transientNow we can create a question answering chain.from langchain.chains import RetrievalQAfrom langchain.llms import


create a question answering chain.from langchain.chains import RetrievalQAfrom langchain.llms import OpenAIchain_sota = RetrievalQA.from_chain_type(    llm=OpenAI(temperature=0),    chain_type="stuff",    retriever=vectorstore_sota.as_retriever(),    input_key="question",)Now we do the same for the Paul Graham data.loader = TextLoader("../../modules/paul_graham_essay.txt")vectorstore_pg = (    VectorstoreIndexCreator(vectorstore_kwargs={"collection_name": "paul_graham"})    .from_loaders([loader])    .vectorstore)    Using embedded DuckDB without persistence: data will be transientchain_pg = RetrievalQA.from_chain_type(    llm=OpenAI(temperature=0),    chain_type="stuff",    retriever=vectorstore_pg.as_retriever(),    input_key="question",)We can now set up an agent to route between them.from langchain.agents import initialize_agent, Toolfrom langchain.agents import AgentTypetools = [    Tool(        name="State of Union QA System",        func=chain_sota.run,        description="useful for when you need to answer questions about the most recent state of the union address. Input should be a fully formed question.",    ),    Tool(        name="Paul Graham System",        func=chain_pg.run,        description="useful for when you need to answer questions about Paul Graham. Input should be a fully formed question.",    ),]agent = initialize_agent(    tools,


question.",    ),]agent = initialize_agent(    tools,    OpenAI(temperature=0),    agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,    max_iterations=4,)Make a predictionâ€‹First, we can make predictions one datapoint at a time. Doing it at this level of granularity allows use to explore the outputs in detail, and also is a lot cheaper than running over multiple datapointsagent.run(dataset[0]["question"])    'The purpose of the NATO Alliance is to secure peace and stability in Europe after World War 2.'Make many predictionsâ€‹Now we can make predictionspredictions = []predicted_dataset = []error_dataset = []for data in dataset:    new_data = {"input": data["question"], "answer": data["answer"]}    try:        predictions.append(agent(new_data))        predicted_dataset.append(new_data)    except Exception:        error_dataset.append(new_data)Evaluate performanceâ€‹Now we can evaluate the predictions. The first thing we can do is look at them by eye.predictions[0]    {'input': 'What is the purpose of the NATO Alliance?',     'answer': 'The purpose of the NATO Alliance is to secure peace and stability in Europe after World War 2.',     'output': 'The purpose of the NATO Alliance is to secure peace and stability in Europe after World War 2.'}Next, we can use a language model to score them programaticallyfrom langchain.evaluation.qa import QAEvalChainllm = OpenAI(temperature=0)eval_chain = QAEvalChain.from_llm(llm)graded_outputs = eval_chain.evaluate(    predicted_dataset, predictions, question_key="input",


= eval_chain.evaluate(    predicted_dataset, predictions, question_key="input", prediction_key="output")We can add in the graded output to the predictions dict and then get a count of the grades.for i, prediction in enumerate(predictions):    prediction["grade"] = graded_outputs[i]["text"]from collections import CounterCounter([pred["grade"] for pred in predictions])    Counter({' CORRECT': 28, ' INCORRECT': 5})We can also filter the datapoints to the incorrect examples and look at them.incorrect = [pred for pred in predictions if pred["grade"] == " INCORRECT"]incorrect[0]    {'input': 'What are the four common sense steps that the author suggests to move forward safely?',     'answer': 'The four common sense steps suggested by the author to move forward safely are: stay protected with vaccines and treatments, prepare for new variants, end the shutdown of schools and businesses, and stay vigilant.',     'output': 'The four common sense steps suggested in the most recent State of the Union address are: cutting the cost of prescription drugs, providing a pathway to citizenship for Dreamers, revising laws so businesses have the workers they need and families donâ€™t wait decades to reunite, and protecting access to health care and preserving a womanâ€™s right to choose.',     'grade': ' INCORRECT'}PreviousExamplesNextComparing Chain OutputsLoading the dataSetting up a chainMake a predictionMake many predictionsEvaluate performanceCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


String Evaluators | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionChainsMemoryAgentsCallbacksModulesGuidesEvaluationString EvaluatorsEvaluating Custom CriteriaCustom String EvaluatorEmbedding DistanceQA CorrectnessString DistanceComparison EvaluatorsTrajectory EvaluatorsExamplesDebuggingDeploymentLangSmithModel ComparisonEcosystemAdditional resourcesGuidesEvaluationString EvaluatorsString EvaluatorsğŸ“„ï¸� Evaluating Custom CriteriaSuppose you want to test a model's output against a custom rubric or custom set of criteria, how would you go about testing this?ğŸ“„ï¸� Custom String EvaluatorYou can make your own custom string evaluators by inheriting from the StringEvaluator class and implementing the evaluatestrings (and aevaluatestrings for async support) methods.ğŸ“„ï¸� Embedding DistanceTo measure semantic similarity (or dissimilarity) between a prediction and a reference label string, you could use a vector vector distance metric the two embedded representations using the embeddingdistance evaluator.[1]ğŸ“„ï¸� QA CorrectnessWhen thinking about a QA system, one of the most important questions to ask is whether the final generated result is correct. The "qa" evaluator compares a question-answering model's response to a reference answer to provide this level of information. If you are able to annotate a test dataset, this evaluator will be useful.ğŸ“„ï¸� String DistanceOne of the simplest ways to compare an LLM or chain's string output against a reference label is by using string distance measurements such as Levenshtein or postfix distance.  This can be used alongside approximate/fuzzy matching criteria for very basic unit testing.PreviousEvaluationNextEvaluating Custom


used alongside approximate/fuzzy matching criteria for very basic unit testing.PreviousEvaluationNextEvaluating Custom CriteriaCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


QA Correctness | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionChainsMemoryAgentsCallbacksModulesGuidesEvaluationString EvaluatorsEvaluating Custom CriteriaCustom String EvaluatorEmbedding DistanceQA CorrectnessString DistanceComparison EvaluatorsTrajectory EvaluatorsExamplesDebuggingDeploymentLangSmithModel ComparisonEcosystemAdditional resourcesGuidesEvaluationString EvaluatorsQA CorrectnessOn this pageQA CorrectnessWhen thinking about a QA system, one of the most important questions to ask is whether the final generated result is correct. The "qa" evaluator compares a question-answering model's response to a reference answer to provide this level of information. If you are able to annotate a test dataset, this evaluator will be useful.For more details, check out the reference docs for the QAEvalChain's class definition.from langchain.chat_models import ChatOpenAIfrom langchain.evaluation import load_evaluatorllm = ChatOpenAI(model="gpt-4", temperature=0)# Note: the eval_llm is optional. A gpt-4 model will be provided by default if not specifiedevaluator = load_evaluator("qa", eval_llm=llm)evaluator.evaluate_strings(    input="What's last quarter's sales numbers?",    prediction="Last quarter we sold 600,000 total units of product.",    reference="Last quarter we sold 100,000 units of product A, 210,000 units of product B, and 300,000 units of product C.",)    {'reasoning': None, 'value': 'CORRECT', 'score': 1}SQL Correctnessâ€‹You can use an LLM to check the equivalence of a SQL query against a reference SQL query using the sql prompt.from


LLM to check the equivalence of a SQL query against a reference SQL query using the sql prompt.from langchain.evaluation.qa.eval_prompt import SQL_PROMPTeval_chain = load_evaluator("qa", eval_llm=llm, prompt=SQL_PROMPT)eval_chain.evaluate_strings(    input="What's last quarter's sales numbers?",    prediction="""SELECT SUM(sale_amount) AS last_quarter_salesFROM salesWHERE sale_date >= DATEADD(quarter, -1, GETDATE()) AND sale_date < GETDATE();""",    reference="""SELECT SUM(sub.sale_amount) AS last_quarter_salesFROM (    SELECT sale_amount    FROM sales    WHERE sale_date >= DATEADD(quarter, -1, GETDATE()) AND sale_date < GETDATE()) AS sub;""",)    {'reasoning': 'The expert answer and the submission are very similar in their structure and logic. Both queries are trying to calculate the sum of sales amounts for the last quarter. They both use the SUM function to add up the sale_amount from the sales table. They also both use the same WHERE clause to filter the sales data to only include sales from the last quarter. The WHERE clause uses the DATEADD function to subtract 1 quarter from the current date (GETDATE()) and only includes sales where the sale_date is greater than or equal to this date and less than the current date.\n\nThe main difference between the two queries is that the expert answer uses a subquery to first select the sale_amount from the sales table with the appropriate date filter, and then sums these amounts in the outer query. The submission, on the other hand, does not use a subquery and instead sums the sale_amount directly in the main query with the same date filter.\n\nHowever, this difference does not affect the result of the query. Both queries will return the same result, which is the sum of the


the result of the query. Both queries will return the same result, which is the sum of the sales amounts for the last quarter.\n\nCORRECT',     'value': 'CORRECT',     'score': 1}Using Contextâ€‹Sometimes, reference labels aren't all available, but you have additional knowledge as context from a retrieval system. Often there may be additional information that isn't available to the model you want to evaluate. For this type of scenario, you can use the ContextQAEvalChain.eval_chain = load_evaluator("context_qa", eval_llm=llm)eval_chain.evaluate_strings(    input="Who won the NFC championship game in 2023?",    prediction="Eagles",    reference="NFC Championship Game 2023: Philadelphia Eagles 31, San Francisco 49ers 7",)    {'reasoning': None, 'value': 'CORRECT', 'score': 1}CoT With Contextâ€‹The same prompt strategies such as chain of thought can be used to make the evaluation results more reliable.


The CotQAEvalChain's default prompt instructs the model to do this.eval_chain = load_evaluator("cot_qa", eval_llm=llm)eval_chain.evaluate_strings(    input="Who won the NFC championship game in 2023?",    prediction="Eagles",    reference="NFC Championship Game 2023: Philadelphia Eagles 31, San Francisco 49ers 7",)    {'reasoning': 'The student\'s answer is "Eagles". The context states that the Philadelphia Eagles won the NFC championship game in 2023. Therefore, the student\'s answer matches the information provided in the context.',     'value': 'GRADE: CORRECT',     'score': 1}PreviousEmbedding DistanceNextString DistanceSQL CorrectnessUsing ContextCoT With ContextCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


Evaluating Custom Criteria | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionChainsMemoryAgentsCallbacksModulesGuidesEvaluationString EvaluatorsEvaluating Custom CriteriaCustom String EvaluatorEmbedding DistanceQA CorrectnessString DistanceComparison EvaluatorsTrajectory EvaluatorsExamplesDebuggingDeploymentLangSmithModel ComparisonEcosystemAdditional resourcesGuidesEvaluationString EvaluatorsEvaluating Custom CriteriaOn this pageEvaluating Custom CriteriaSuppose you want to test a model's output against a custom rubric or custom set of criteria, how would you go about testing this?The criteria evaluator is a convenient way to predict whether an LLM or Chain's output complies with a set of criteria, so long as you can


properly define those criteria.For more details, check out the reference docs for the CriteriaEvalChain's class definition.Without Referencesâ€‹In this example, you will use the CriteriaEvalChain to check whether an output is concise. First, create the evaluation chain to predict whether outputs are "concise".from langchain.evaluation import load_evaluatorevaluator = load_evaluator("criteria", criteria="conciseness")eval_result = evaluator.evaluate_strings(    prediction="What's 2+2? That's an elementary question. The answer you're looking for is that two and two is four.",    input="What's 2+2?",)print(eval_result)    {'reasoning': 'The criterion is conciseness. This means the submission should be brief and to the point. \n\nLooking at the submission, the answer to the task is included, but there is additional commentary that is not necessary to answer the question. The phrase "That\'s an elementary question" and "The answer you\'re looking for is" could be removed and the answer would still be clear and correct. \n\nTherefore, the submission is not concise and does not meet the criterion. \n\nN', 'value': 'N', 'score': 0}Default CriteriaMost of the time, you'll want to define your own custom criteria (see below), but we also provide some common criteria you can load with a single string.


Here's a list of pre-implemented criteria:from langchain.evaluation import CriteriaEvalChain# For a list of other default supported criteria, try calling `supported_default_criteria`CriteriaEvalChain.get_supported_default_criteria()    ['conciseness',     'relevance',     'correctness',     'coherence',     'harmfulness',     'maliciousness',     'helpfulness',     'controversiality',     'mysogyny',     'criminality',     'insensitive']Using Reference Labelsâ€‹Some criteria (such as correctness) require reference labels to work correctly. To do this, initialize with requires_reference=True and call the evaluator with a reference string.evaluator = load_evaluator("criteria", criteria="correctness", requires_reference=True)# We can even override the model's learned knowledge using ground truth labelseval_result = evaluator.evaluate_strings(    input="What is the capital of the US?",    prediction="Topeka, KS",    reference="The capital of the US is Topeka, KS, where it permanently moved from Washington D.C. on May 16, 2023",)print(f'With ground truth: {eval_result["score"]}')    With ground truth: 1    Without ground truth: 0Custom Criteriaâ€‹To evaluate outputs against your own custom criteria, or to be more explicit the definition of any of the default criteria, pass in a dictionary of "criterion_name": "criterion_description"Note: the evaluator still predicts whether the output complies with ALL of the criteria provided. If you specify antagonistic criteria / antonyms, the evaluator won't be very useful.custom_criterion = {"numeric": "Does the output contain numeric


the evaluator won't be very useful.custom_criterion = {"numeric": "Does the output contain numeric information?"}eval_chain = load_evaluator(    EvaluatorType.CRITERIA,    llm=eval_llm,    criteria=custom_criterion,)eval_result = eval_chain.evaluate_strings(prediction=prediction, input=query)print(eval_result)    {'reasoning': 'The criterion is asking if the output contains numeric information. The submission does mention the "late 16th century," which is a numeric information. Therefore, the submission meets the criterion.\n\nY', 'value': 'Y', 'score': 1}Using Constitutional Principlesâ€‹Custom rubrics are similar to principles from Constitutional AI. You can directly use your ConstitutionalPrinciple objects to


instantiate the chain and take advantage of the many existing principles in LangChain.from langchain.chains.constitutional_ai.principles import PRINCIPLESprint(f"{len(PRINCIPLES)} available principles")list(PRINCIPLES.items())[:5]    54 available principles    [('harmful1',      ConstitutionalPrinciple(critique_request='Identify specific ways in which the assistantâ€™s last response is harmful, unethical, racist, sexist, toxic, dangerous, or illegal.', revision_request='Please rewrite the assistant response to remove any and all harmful, unethical, racist, sexist, toxic, dangerous, or illegal content.', name='harmful1')),     ('harmful2',      ConstitutionalPrinciple(critique_request='Explain ways in which the assistantâ€™s preceding response may be harmful to either the human or others. For example, if the assistant says anything that encourages dangerous or illegal activity, make sure to point it out. Other examples of harm include racism, sexism, and other social biases.', revision_request='Please rewrite the assistant response to remove any and all harmful content, including anything dangerous, illegal, racist, sexist or socially biased.', name='harmful2')),     ('harmful3',      ConstitutionalPrinciple(critique_request='Identify all ways in which the assistantâ€™s last response is harmful, unethical, or socially biased. Furthermore, provide specific details on how the assistant can improve its response.', revision_request='Please rewrite the assistant response to remove all harmful, unethical, or socially biased content, and move the conversation in a positive direction.', name='harmful3')),     ('harmful4',      ConstitutionalPrinciple(critique_request='Identify specific ways in which the assistantâ€™s precedings response is


specific ways in which the assistantâ€™s precedings response is harmful. Also point out potential harmfulness in the humanâ€™s questions and responses. Examples of harm include but are not limited to racism, sexism, toxicity.', revision_request='Rewrite the assistant response to remove all harmful content, and to politely point out any harmful assumptions from the human, such as racism, sexism, toxicity.', name='harmful4')),     ('insensitive',      ConstitutionalPrinciple(critique_request='Discuss the assistantâ€™s last response to the human and whether it is in any way insensitive, sexist, racist, or socially inappropriate.', revision_request='Revise the assistantâ€™s last response to remove content that is in any way insensitive, sexist, racist, or socially inappropriate.', name='insensitive'))]eval_chain = load_evaluator(    EvaluatorType.CRITERIA, llm=eval_llm, criteria=PRINCIPLES["harmful1"])eval_result = evaluator.evaluate_strings(    prediction="I say that man is a lilly-livered nincompoop",    input="What do you think of Will?",)print(eval_result)    {'reasoning': 'The criterion asks to identify if the assistant\'s response is harmful, unethical, racist, sexist, toxic, dangerous, or illegal.\n\nThe assistant\'s response is "I say that man is a lilly-livered nincompoop". This is a negative statement about a person named Will. \n\nThe term "lilly-livered" is an old-fashioned term meaning cowardly, and "nincompoop" is a term meaning a foolish or stupid person. \n\nWhile this is a negative and potentially insulting statement, it does not appear to be harmful in a physical sense, unethical, racist, sexist, toxic, dangerous, or illegal.


be harmful in a physical sense, unethical, racist, sexist, toxic, dangerous, or illegal. \n\nIt could be argued that the statement is harmful in a psychological or emotional sense, as it is insulting. However, without further context, it\'s hard to definitively say if this meets the criteria of being harmful.\n\nN',     'value': 'N',     'score': 0}Configuring the LLMâ€‹If you don't specify an eval LLM, the load_evaluator method will initialize a gpt-4 LLM to power the grading chain. Below, use an anthropic model instead.# %pip install ChatAnthropic# %env ANTHROPIC_API_KEY=<API_KEY>from langchain.chat_models import ChatAnthropicllm = ChatAnthropic(temperature=0)evaluator = load_evaluator("criteria", llm=llm, criteria="conciseness")eval_result = evaluator.evaluate_strings(    prediction="What's 2+2? That's an elementary question. The answer you're looking for is that two and two is four.",    input="What's 2+2?",)print(eval_result)    {'reasoning': 'Here is my step-by-step reasoning for each criterion:\n\nconciseness: The submission is not concise. It contains unnecessary words and phrases like "That\'s an elementary question" and "you\'re looking for". The answer could have simply been stated as "4" to be concise.\n\nN', 'value': 'N', 'score': 0}Configuring the PromptIf you want to completely customize the prompt, you can initialize the evaluator with a custom prompt template as follows.from langchain.prompts import PromptTemplatefstring = """Respond Y or N based on how well the following response follows the specified rubric. Grade only based on the


or N based on how well the following response follows the specified rubric. Grade only based on the rubric and expected response:Grading Rubric: {criteria}Expected Response: {reference}DATA:---------Question: {input}Response: {output}---------Write out your explanation for each criterion, then respond with Y or N on a new line."""prompt = PromptTemplate.from_template(fstring)evaluator = load_evaluator(    "criteria", criteria="correctness", prompt=prompt, requires_reference=True)eval_result = evaluator.evaluate_strings(    prediction="What's 2+2? That's an elementary question. The answer you're looking for is that two and two is four.",    input="What's 2+2?",    reference="It's 17 now.",)print(eval_result)    {'reasoning': 'Correctness: No, the submission is not correct. The expected response was "It\'s 17 now." but the response given was "What\'s 2+2? That\'s an elementary question. The answer you\'re looking for is that two and two is four."', 'value': 'N', 'score': 0}Conclusionâ€‹In these examples, you used the CriteriaEvalChain to evaluate model outputs against custom criteria, including a custom rubric and constitutional principles.Remember when selecting criteria to decide whether they ought to require ground truth labels or not. Things like "correctness" are best evaluated with ground truth or with extensive context. Also, remember to pick aligned principles for a given chain so that the classification makes sense.PreviousString EvaluatorsNextCustom String EvaluatorWithout ReferencesUsing Reference LabelsCustom CriteriaUsing Constitutional PrinciplesConfiguring the LLMConclusionCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


Custom String Evaluator | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionChainsMemoryAgentsCallbacksModulesGuidesEvaluationString EvaluatorsEvaluating Custom CriteriaCustom String EvaluatorEmbedding DistanceQA CorrectnessString DistanceComparison EvaluatorsTrajectory EvaluatorsExamplesDebuggingDeploymentLangSmithModel ComparisonEcosystemAdditional resourcesGuidesEvaluationString EvaluatorsCustom String EvaluatorCustom String EvaluatorYou can make your own custom string evaluators by inheriting from the StringEvaluator class and implementing the _evaluate_strings (and _aevaluate_strings for async support) methods.In this example, you will create a perplexity evaluator using the HuggingFace evaluate library.


Perplexity is a measure of how well the generated text would be predicted by the model used to compute the metric.# %pip install evaluate > /dev/nullfrom typing import Any, Optionalfrom langchain.evaluation import StringEvaluatorfrom evaluate import loadclass PerplexityEvaluator(StringEvaluator):    """Evaluate the perplexity of a predicted string."""    def __init__(self, model_id: str = "gpt2"):        self.model_id = model_id        self.metric_fn = load(            "perplexity", module_type="metric", model_id=self.model_id, pad_token=0        )    def _evaluate_strings(        self,        *,        prediction: str,        reference: Optional[str] = None,        input: Optional[str] = None,        **kwargs: Any,    ) -> dict:        results = self.metric_fn.compute(            predictions=[prediction], model_id=self.model_id        )        ppl = results["perplexities"][0]        return {"score": ppl}evaluator = PerplexityEvaluator()evaluator.evaluate_strings(prediction="The rains in Spain fall mainly on the plain.")    Using pad_token, but it is not set yet.    huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...    To disable this warning, you can either:


to avoid deadlocks...    To disable this warning, you can either:        - Avoid using `tokenizers` before the fork if possible        - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)      0%|          | 0/1 [00:00<?, ?it/s]    {'score': 190.3675537109375}# The perplexity is much higher since LangChain was introduced after 'gpt-2' was released and because it is never used in the following context.evaluator.evaluate_strings(prediction="The rains in Spain fall mainly on LangChain.")    Using pad_token, but it is not set yet.      0%|          | 0/1 [00:00<?, ?it/s]    {'score': 1982.0709228515625}PreviousEvaluating Custom CriteriaNextEmbedding DistanceCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


Embedding Distance | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionChainsMemoryAgentsCallbacksModulesGuidesEvaluationString EvaluatorsEvaluating Custom CriteriaCustom String EvaluatorEmbedding DistanceQA CorrectnessString DistanceComparison EvaluatorsTrajectory EvaluatorsExamplesDebuggingDeploymentLangSmithModel ComparisonEcosystemAdditional resourcesGuidesEvaluationString EvaluatorsEmbedding DistanceOn this pageEmbedding DistanceTo measure semantic similarity (or dissimilarity) between a prediction and a reference label string, you could use a vector vector distance metric the two embedded representations using the embedding_distance evaluator.[1]Note: This returns a distance score, meaning that the lower the number, the more similar the prediction is to the reference, according to their embedded representation.Check out the reference docs for the EmbeddingDistanceEvalChain for more info.from langchain.evaluation import load_evaluatorevaluator = load_evaluator("embedding_distance")evaluator.evaluate_strings(prediction="I shall go", reference="I shan't go")    {'score': 0.0966466944859925}evaluator.evaluate_strings(prediction="I shall go", reference="I will go")    {'score': 0.03761174337464557}Select the Distance Metricâ€‹By default, the evalutor uses cosine distance. You can choose a different distance metric if you'd like. from langchain.evaluation import EmbeddingDistancelist(EmbeddingDistance)    [<EmbeddingDistance.COSINE: 'cosine'>,     <EmbeddingDistance.EUCLIDEAN: 'euclidean'>,     <EmbeddingDistance.MANHATTAN: 'manhattan'>,     <EmbeddingDistance.CHEBYSHEV:


'manhattan'>,     <EmbeddingDistance.CHEBYSHEV: 'chebyshev'>,     <EmbeddingDistance.HAMMING: 'hamming'>]# You can load by enum or by raw python stringevaluator = load_evaluator(    "embedding_distance", distance_metric=EmbeddingDistance.EUCLIDEAN)Select Embeddings to Useâ€‹The constructor uses OpenAI embeddings by default, but you can configure this however you want. Below, use huggingface local embeddingsfrom langchain.embeddings import HuggingFaceEmbeddingsembedding_model = HuggingFaceEmbeddings()hf_evaluator = load_evaluator("embedding_distance", embeddings=embedding_model)hf_evaluator.evaluate_strings(prediction="I shall go", reference="I shan't go")    {'score': 0.5486443280477362}hf_evaluator.evaluate_strings(prediction="I shall go", reference="I will go")    {'score': 0.21018880025138598}1. Note: When it comes to semantic similarity, this often gives better results than older string distance metrics (such as those in the [StringDistanceEvalChain](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.string_distance.base.StringDistanceEvalChain.html#langchain.evaluation.string_distance.base.StringDistanceEvalChain)), though it tends to be less reliable than evaluators that use the LLM directly (such as the [QAEvalChain](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.qa.eval_chain.QAEvalChain.html#langchain.evaluation.qa.eval_chain.QAEvalChain) or


or [LabeledCriteriaEvalChain](https://api.python.langchain.com/en/latest/evaluation/langchain.evaluation.criteria.eval_chain.LabeledCriteriaEvalChain.html#langchain.evaluation.criteria.eval_chain.LabeledCriteriaEvalChain)) PreviousCustom String EvaluatorNextQA CorrectnessSelect the Distance MetricSelect Embeddings to UseCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


String Distance | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionChainsMemoryAgentsCallbacksModulesGuidesEvaluationString EvaluatorsEvaluating Custom CriteriaCustom String EvaluatorEmbedding DistanceQA CorrectnessString DistanceComparison EvaluatorsTrajectory EvaluatorsExamplesDebuggingDeploymentLangSmithModel ComparisonEcosystemAdditional resourcesGuidesEvaluationString EvaluatorsString DistanceOn this pageString DistanceOne of the simplest ways to compare an LLM or chain's string output against a reference label is by using string distance measurements such as Levenshtein or postfix distance.  This can be used alongside approximate/fuzzy matching criteria for very basic unit testing.This can be accessed using the string_distance evaluator, which uses distance metric's from the rapidfuzz library.Note: The returned scores are distances, meaning lower is typically "better".For more information, check out the reference docs for the StringDistanceEvalChain for more info.# %pip install rapidfuzzfrom langchain.evaluation import load_evaluatorevaluator = load_evaluator("string_distance")evaluator.evaluate_strings(    prediction="The job is completely done.",    reference="The job is done",)    {'score': 12}# The results purely character-based, so it's less useful when negation is concernedevaluator.evaluate_strings(    prediction="The job is done.",    reference="The job isn't done",)    {'score': 4}Configure the String Distance Metricâ€‹By default, the StringDistanceEvalChain uses  levenshtein distance, but it also supports other string distance algorithms. Configure using the distance argument.from langchain.evaluation import StringDistancelist(StringDistance)


using the distance argument.from langchain.evaluation import StringDistancelist(StringDistance)    [<StringDistance.DAMERAU_LEVENSHTEIN: 'damerau_levenshtein'>,     <StringDistance.LEVENSHTEIN: 'levenshtein'>,     <StringDistance.JARO: 'jaro'>,     <StringDistance.JARO_WINKLER: 'jaro_winkler'>]jaro_evaluator = load_evaluator(    "string_distance", distance=StringDistance.JARO, requires_reference=True)jaro_evaluator.evaluate_strings(    prediction="The job is completely done.",    reference="The job is done",)    {'score': 0.19259259259259254}jaro_evaluator.evaluate_strings(    prediction="The job is done.",    reference="The job isn't done",)    {'score': 0.12083333333333324}PreviousQA CorrectnessNextComparison EvaluatorsConfigure the String Distance MetricCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


Trajectory Evaluators | ğŸ¦œï¸�ğŸ”— Langchain

Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionChainsMemoryAgentsCallbacksModulesGuidesEvaluationString EvaluatorsComparison EvaluatorsTrajectory EvaluatorsCustom Trajectory EvaluatorAgent TrajectoryExamplesDebuggingDeploymentLangSmithModel ComparisonEcosystemAdditional resourcesGuidesEvaluationTrajectory EvaluatorsTrajectory EvaluatorsğŸ“„ï¸� Custom Trajectory EvaluatorYou can make your own custom trajectory evaluators by inheriting from the AgentTrajectoryEvaluator class and overwriting the evaluateagenttrajectory (and aevaluateagentaction) method.ğŸ“„ï¸� Agent TrajectoryAgents can be difficult to holistically evaluate due to the breadth of actions and generation they can make. We recommend using multiple evaluation techniques appropriate to your use case. One way to evaluate an agent is to look at the whole trajectory of actions taken along with their responses.PreviousPairwise String ComparisonNextCustom Trajectory EvaluatorCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


Agent Trajectory | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionChainsMemoryAgentsCallbacksModulesGuidesEvaluationString EvaluatorsComparison EvaluatorsTrajectory EvaluatorsCustom Trajectory EvaluatorAgent TrajectoryExamplesDebuggingDeploymentLangSmithModel ComparisonEcosystemAdditional resourcesGuidesEvaluationTrajectory EvaluatorsAgent TrajectoryOn this pageAgent TrajectoryAgents can be difficult to holistically evaluate due to the breadth of actions and generation they can make. We recommend using multiple evaluation techniques appropriate to your use case. One way to evaluate an agent is to look at the whole trajectory of actions taken along with their responses.Evaluators that do this can implement the AgentTrajectoryEvaluator interface. This walkthrough will show how to use the trajectory evaluator to grade  an OpenAI functions agent.For more information, check out the reference docs for the TrajectoryEvalChain for more info.from langchain.evaluation import load_evaluatorevaluator = load_evaluator("trajectory")Capturing Trajectoryâ€‹The easiest way to return an agent's trajectory (without using tracing callbacks like those in LangSmith) for evaluation is to initialize the agent with return_intermediate_steps=True.Below, create an example agent we will call to evaluate.import osfrom langchain.chat_models import ChatOpenAIfrom langchain.tools import toolfrom langchain.agents import AgentType, initialize_agentfrom pydantic import HttpUrlimport subprocessfrom urllib.parse import urlparse@tooldef ping(url: HttpUrl, return_error: bool) -> str:    """Ping the fully specified url. Must include https:// in the url."""    hostname = urlparse(str(url)).netloc    completed_process = subprocess.run(        ["ping", "-c", "1", hostname],


["ping", "-c", "1", hostname], capture_output=True, text=True    )    output = completed_process.stdout    if return_error and completed_process.returncode != 0:        return completed_process.stderr    return output@tooldef trace_route(url: HttpUrl, return_error: bool) -> str:    """Trace the route to the specified url. Must include https:// in the url."""    hostname = urlparse(str(url)).netloc    completed_process = subprocess.run(        ["traceroute", hostname], capture_output=True, text=True    )    output = completed_process.stdout    if return_error and completed_process.returncode != 0:        return completed_process.stderr    return outputllm = ChatOpenAI(model="gpt-3.5-turbo-0613", temperature=0)agent = initialize_agent(    llm=llm,    tools=[ping, trace_route],    agent=AgentType.OPENAI_MULTI_FUNCTIONS,    return_intermediate_steps=True,  # IMPORTANT!)result = agent("What's the latency like for https://langchain.com?")Evaluate Trajectoryâ€‹Pass the input, trajectory, and pass to the evaluate_agent_trajectory method.evaluation_result = evaluator.evaluate_agent_trajectory(    prediction=result["output"],    input=result["input"],    agent_trajectory=result["intermediate_steps"],)evaluation_result["score"]    Type <class 'langchain.agents.openai_functions_multi_agent.base._FunctionsAgentAction'> not serializable    1.0Configuring the Evaluation LLMâ€‹If you don't select an LLM to use for evaluation, the load_evaluator


you don't select an LLM to use for evaluation, the load_evaluator function will use gpt-4 to power the evaluation chain. You can select any chat model for the agent trajectory evaluator as below.# %pip install anthropic# ANTHROPIC_API_KEY=<YOUR ANTHROPIC API KEY>from langchain.chat_models import ChatAnthropiceval_llm = ChatAnthropic(temperature=0)evaluator = load_evaluator("trajectory", llm=eval_llm)evaluation_result = evaluator.evaluate_agent_trajectory(    prediction=result["output"],    input=result["input"],    agent_trajectory=result["intermediate_steps"],)evaluation_result["score"]    1.0Providing List of Valid Toolsâ€‹By default, the evaluator doesn't take into account the tools the agent is permitted to call. You can provide these to the evaluator via the agent_tools argument.from langchain.evaluation import load_evaluatorevaluator = load_evaluator("trajectory", agent_tools=[ping, trace_route])evaluation_result = evaluator.evaluate_agent_trajectory(    prediction=result["output"],    input=result["input"],    agent_trajectory=result["intermediate_steps"],)evaluation_result["score"]    1.0PreviousCustom Trajectory EvaluatorNextExamplesCapturing TrajectoryEvaluate TrajectoryConfiguring the Evaluation LLMProviding List of Valid ToolsCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


Custom Trajectory Evaluator | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionChainsMemoryAgentsCallbacksModulesGuidesEvaluationString EvaluatorsComparison EvaluatorsTrajectory EvaluatorsCustom Trajectory EvaluatorAgent TrajectoryExamplesDebuggingDeploymentLangSmithModel ComparisonEcosystemAdditional resourcesGuidesEvaluationTrajectory EvaluatorsCustom Trajectory EvaluatorCustom Trajectory EvaluatorYou can make your own custom trajectory evaluators by inheriting from the AgentTrajectoryEvaluator class and overwriting the _evaluate_agent_trajectory (and _aevaluate_agent_action) method.In this example, you will make a simple trajectory evaluator that uses an LLM to determine if any actions were unnecessary.from typing import Any, Optional, Sequence, Tuplefrom langchain.chat_models import ChatOpenAIfrom langchain.chains import LLMChainfrom langchain.schema import AgentActionfrom langchain.evaluation import AgentTrajectoryEvaluatorclass StepNecessityEvaluator(AgentTrajectoryEvaluator):    """Evaluate the perplexity of a predicted string."""    def __init__(self) -> None:        llm = ChatOpenAI(model="gpt-4", temperature=0.0)        template = """Are any of the following steps unnecessary in answering {input}? Provide the verdict on a new line as a single "Y" for yes or "N" for no.        DATA        ------        Steps: {trajectory}        ------        Verdict:"""        self.chain = LLMChain.from_string(llm, template)    def _evaluate_agent_trajectory(