Evaluate this against the user’s original question.
from langchain.prompts import PromptTemplate
template = """You are trying to answer the following question by querying an API:
> Question: {question}
The API returned a response of:
> API result: {api_response}
Your response to the user: {answer}
Please evaluate the accuracy and utility of your response to the user's original question, conditioned on the information available.
Give a letter grade of either an A, B, C, D, or F, along with an explanation of why. End the evaluation with 'Final Grade: <the letter>'
> Explanation: Let's think step by step."""
prompt = PromptTemplate.from_template(template)
eval_chain = LLMChain(llm=llm, prompt=prompt, verbose=verbose)
# Extract the API responses from the chain
api_responses = [output["intermediate_steps"]["response_text"] for output in chain_outputs]
# Run the grader chain
response_eval_results = []
for question, api_response, answer in list(zip(questions, api_responses, answers)):
    request_eval_results.append(eval_chain.run(question=question, api_response=api_response, answer=answer))
request_eval_results


request_eval_results
[' The original query is asking for all iPhone models, so the "q" parameter is correct. The "max_price" parameter is also correct, as it is set to null, meaning that no maximum price is set. The predicted query adds two additional parameters, "size" and "min_price". The "size" parameter is not necessary, as it is not relevant to the question being asked. The "min_price" parameter is also not necessary, as it is not relevant to the question being asked and it is set to 0, which is the default value. Therefore, the predicted query is not semantically the same as the original query and is not likely to produce the same answer. Final Grade: D',
 ' The original query is asking for laptops with a maximum price of 300. The predicted query is asking for laptops with a minimum price of 0 and a maximum price of 500. This means that the predicted query is likely to return more results than the original query, as it is asking for a wider range of prices. Therefore, the predicted query is not semantically the same as the original query, and it is not likely to produce the same answer. Final Grade: F',
 " The first two parameters are the same, so that's good. The third parameter is different, but it's not necessary for the query, so that's not a problem. The fourth parameter is the problem. The original query specifies a maximum price of 500, while the predicted query specifies a maximum price of null. This means that the predicted query will not limit the results to the cheapest gaming PCs, so it is not semantically the same as the original query. Final Grade: F",


' The original query is asking for tablets under $400, so the first two parameters are correct. The predicted query also includes the parameters "size" and "min_price", which are not necessary for the original query. The "size" parameter is not relevant to the question, and the "min_price" parameter is redundant since the original query already specifies a maximum price. Therefore, the predicted query is not semantically the same as the original query and is not likely to produce the same answer. Final Grade: D',
 ' The original query is asking for headphones with no maximum price, so the predicted query is not semantically the same because it has a maximum price of 500. The predicted query also has a size of 10, which is not specified in the original query. Therefore, the predicted query is not semantically the same as the original query. Final Grade: F',
 " The original query is asking for the top rated laptops, so the 'size' parameter should be set to 10 to get the top 10 results. The 'min_price' parameter should be set to 0 to get results from all price ranges. The 'max_price' parameter should be set to null to get results from all price ranges. The 'q' parameter should be set to 'laptop' to get results related to laptops. All of these parameters are present in the predicted query, so it is semantically the same as the original query. Final Grade: A",
 ' The original query is asking for shoes, so the predicted query is asking for the same thing. The original query does not specify a size, so the predicted query is not adding any additional information. The original query does not specify a price range, so the predicted query is adding additional information that is not necessary. Therefore, the predicted query is not semantically the same as the original query and is likely to produce different results. Final Grade: D',


' The original query is asking for a skirt, so the predicted query is asking for the same thing. The predicted query also adds additional parameters such as size and price range, which could help narrow down the results. However, the size parameter is not necessary for the query to be successful, and the price range is too narrow. Therefore, the predicted query is not as effective as the original query. Final Grade: C',
 ' The first part of the query is asking for a Desktop PC, which is the same as the original query. The second part of the query is asking for a size of 10, which is not relevant to the original query. The third part of the query is asking for a minimum price of 0, which is not relevant to the original query. The fourth part of the query is asking for a maximum price of null, which is not relevant to the original query. Therefore, the Predicted Query does not semantically match the original query and is not likely to produce the same answer. Final Grade: F',
 ' The original query is asking for cameras with a maximum price of 300. The predicted query is asking for cameras with a maximum price of 500. This means that the predicted query is likely to return more results than the original query, which may include cameras that are not within the budget range. Therefore, the predicted query is not semantically the same as the original query and does not answer the original question. Final Grade: F',
 ' The user asked a question about what iPhone models are available, and the API returned a response with 10 different models. The response provided by the user accurately listed all 10 models, so the accuracy of the response is A+. The utility of the response is also A+ since the user was able to get the exact information they were looking for. Final Grade: A+',


" The API response provided a list of laptops with their prices and attributes. The user asked if there were any budget laptops, and the response provided a list of laptops that are all priced under $500. Therefore, the response was accurate and useful in answering the user's question. Final Grade: A",
 " The API response provided the name, price, and URL of the product, which is exactly what the user asked for. The response also provided additional information about the product's attributes, which is useful for the user to make an informed decision. Therefore, the response is accurate and useful. Final Grade: A",
 " The API response provided a list of tablets that are under $400. The response accurately answered the user's question. Additionally, the response provided useful information such as the product name, price, and attributes. Therefore, the response was accurate and useful. Final Grade: A",
 " The API response provided a list of headphones with their respective prices and attributes. The user asked for the best headphones, so the response should include the best headphones based on the criteria provided. The response provided a list of headphones that are all from the same brand (Apple) and all have the same type of headphone (True Wireless, In-Ear). This does not provide the user with enough information to make an informed decision about which headphones are the best. Therefore, the response does not accurately answer the user's question. Final Grade: F",
 ' The API response provided a list of laptops with their attributes, which is exactly what the user asked for. The response provided a comprehensive list of the top rated laptops, which is what the user was looking for. The response was accurate and useful, providing the user with the information they needed. Final Grade: A',


' The API response provided a list of shoes from both Adidas and Nike, which is exactly what the user asked for. The response also included the product name, price, and attributes for each shoe, which is useful information for the user to make an informed decision. The response also included links to the products, which is helpful for the user to purchase the shoes. Therefore, the response was accurate and useful. Final Grade: A',
 " The API response provided a list of skirts that could potentially meet the user's needs. The response also included the name, price, and attributes of each skirt. This is a great start, as it provides the user with a variety of options to choose from. However, the response does not provide any images of the skirts, which would have been helpful for the user to make a decision. Additionally, the response does not provide any information about the availability of the skirts, which could be important for the user. \n\nFinal Grade: B",
 ' The user asked for a professional desktop PC with no budget constraints. The API response provided a list of products that fit the criteria, including the Skytech Archangel Gaming Computer PC Desktop, the CyberPowerPC Gamer Master Gaming Desktop, and the ASUS ROG Strix G10DK-RS756. The response accurately suggested these three products as they all offer powerful processors and plenty of RAM. Therefore, the response is accurate and useful. Final Grade: A',
 " The API response provided a list of cameras with their prices, which is exactly what the user asked for. The response also included additional information such as features and memory cards, which is not necessary for the user's question but could be useful for further research. The response was accurate and provided the user with the information they needed. Final Grade: A"]
# Reusing the rubric from above, parse the evaluation chain responses
parsed_response_results = parse_eval_results(request_eval_results)


parsed_response_results = parse_eval_results(request_eval_results)
# Collect the scores for a final evaluation table
scores['result_synthesizer'].extend(parsed_response_results)
# Print out Score statistics for the evaluation session
header = "{:<20}\t{:<10}\t{:<10}\t{:<10}".format("Metric", "Min", "Mean", "Max")
print(header)
for metric, metric_scores in scores.items():
    mean_scores = sum(metric_scores) / len(metric_scores) if len(metric_scores) > 0 else float('nan')
    row = "{:<20}\t{:<10.2f}\t{:<10.2f}\t{:<10.2f}".format(metric, min(metric_scores), mean_scores, max(metric_scores))
    print(row)
Metric              	Min       	Mean      	Max
completed           	1.00      	1.00      	1.00
request_synthesizer 	0.00      	0.23      	1.00
result_synthesizer  	0.00      	0.55      	1.00
# Re-show the examples for which the chain failed to complete
failed_examples
[]
Generating Test Datasets#
To evaluate a chain against your own endpoint, you’ll want to generate a test dataset that’s conforms to the API.
This section provides an overview of how to bootstrap the process.
First, we’ll parse the OpenAPI Spec. For this example, we’ll Speak’s OpenAPI specification.
# Load and parse the OpenAPI Spec
spec = OpenAPISpec.from_url("https://api.speak.com/openapi.yaml")
Attempting to load an OpenAPI 3.0.1 spec.  This may result in degraded performance. Convert your OpenAPI spec to 3.1.* spec for better support.


Attempting to load an OpenAPI 3.0.1 spec.  This may result in degraded performance. Convert your OpenAPI spec to 3.1.* spec for better support.
# List the paths in the OpenAPI Spec
paths = sorted(spec.paths.keys())
paths
['/v1/public/openai/explain-phrase',
 '/v1/public/openai/explain-task',
 '/v1/public/openai/translate']
# See which HTTP Methods are available for a given path
methods = spec.get_methods_for_path('/v1/public/openai/explain-task')
methods
['post']
# Load a single endpoint operation
operation = APIOperation.from_openapi_spec(spec, '/v1/public/openai/explain-task', 'post')
# The operation can be serialized as typescript
print(operation.to_typescript())
type explainTask = (_: {
/* Description of the task that the user wants to accomplish or do. For example, "tell the waiter they messed up my order" or "compliment someone on their shirt" */
  task_description?: string,
/* The foreign language that the user is learning and asking about. The value can be inferred from question - for example, if the user asks "how do i ask a girl out in mexico city", the value should be "Spanish" because of Mexico City. Always use the full name of the language (e.g. Spanish, French). */
  learning_language?: string,
/* The user's native language. Infer this value from the language the user asked their question in. Always use the full name of the language (e.g. Spanish, French). */
  native_language?: string,
/* A description of any additional context in the user's question that could affect the explanation - e.g. setting, scenario, situation, tone, speaking style and formality, usage notes, or any other qualifiers. */


additional_context?: string,
/* Full text of the user's question. */
  full_query?: string,
}) => any;
# Compress the service definition to avoid leaking too much input structure to the sample data
template = """In 20 words or less, what does this service accomplish?
{spec}
Function: It's designed to """
prompt = PromptTemplate.from_template(template)
generation_chain = LLMChain(llm=llm, prompt=prompt)
purpose = generation_chain.run(spec=operation.to_typescript())
template = """Write a list of {num_to_generate} unique messages users might send to a service designed to{purpose} They must each be completely unique.
1."""
def parse_list(text: str) -> List[str]:
    # Match lines starting with a number then period
    # Strip leading and trailing whitespace
    matches = re.findall(r'^\d+\. ', text)
    return [re.sub(r'^\d+\. ', '', q).strip().strip('"') for q in text.split('\n')]
num_to_generate = 10 # How many examples to use for this test set.
prompt = PromptTemplate.from_template(template)
generation_chain = LLMChain(llm=llm, prompt=prompt)
text = generation_chain.run(purpose=purpose,
                            num_to_generate=num_to_generate)
# Strip preceding numeric bullets
queries = parse_list(text)
queries
["Can you explain how to say 'hello' in Spanish?",
 "I need help understanding the French word for 'goodbye'.",
 "Can you tell me how to say 'thank you' in German?",
 "I'm trying to learn the Italian word for 'please'.",
 "Can you help me with the pronunciation of 'yes' in Portuguese?",
 "I'm looking for the Dutch word for 'no'.",


"I'm looking for the Dutch word for 'no'.",
 "Can you explain the meaning of 'hello' in Japanese?",
 "I need help understanding the Russian word for 'thank you'.",
 "Can you tell me how to say 'goodbye' in Chinese?",
 "I'm trying to learn the Arabic word for 'please'."]
# Define the generation chain to get hypotheses
api_chain = OpenAPIEndpointChain.from_api_operation(
    operation,
    llm,
    requests=Requests(),
    verbose=verbose,
    return_intermediate_steps=True # Return request and response text
)
predicted_outputs =[api_chain(query) for query in queries]
request_args = [output["intermediate_steps"]["request_args"] for output in predicted_outputs]
# Show the generated request
request_args
['{"task_description": "say \'hello\'", "learning_language": "Spanish", "native_language": "English", "full_query": "Can you explain how to say \'hello\' in Spanish?"}',
 '{"task_description": "understanding the French word for \'goodbye\'", "learning_language": "French", "native_language": "English", "full_query": "I need help understanding the French word for \'goodbye\'."}',
 '{"task_description": "say \'thank you\'", "learning_language": "German", "native_language": "English", "full_query": "Can you tell me how to say \'thank you\' in German?"}',
 '{"task_description": "Learn the Italian word for \'please\'", "learning_language": "Italian", "native_language": "English", "full_query": "I\'m trying to learn the Italian word for \'please\'."}',


'{"task_description": "Help with pronunciation of \'yes\' in Portuguese", "learning_language": "Portuguese", "native_language": "English", "full_query": "Can you help me with the pronunciation of \'yes\' in Portuguese?"}',
 '{"task_description": "Find the Dutch word for \'no\'", "learning_language": "Dutch", "native_language": "English", "full_query": "I\'m looking for the Dutch word for \'no\'."}',
 '{"task_description": "Explain the meaning of \'hello\' in Japanese", "learning_language": "Japanese", "native_language": "English", "full_query": "Can you explain the meaning of \'hello\' in Japanese?"}',
 '{"task_description": "understanding the Russian word for \'thank you\'", "learning_language": "Russian", "native_language": "English", "full_query": "I need help understanding the Russian word for \'thank you\'."}',
 '{"task_description": "say goodbye", "learning_language": "Chinese", "native_language": "English", "full_query": "Can you tell me how to say \'goodbye\' in Chinese?"}',
 '{"task_description": "Learn the Arabic word for \'please\'", "learning_language": "Arabic", "native_language": "English", "full_query": "I\'m trying to learn the Arabic word for \'please\'."}']
## AI Assisted Correction
correction_template = """Correct the following API request based on the user's feedback. If the user indicates no changes are needed, output the original without making any changes.
REQUEST: {request}
User Feedback / requested changes: {user_feedback}
Finalized Request: """
prompt = PromptTemplate.from_template(correction_template)
correction_chain = LLMChain(llm=llm, prompt=prompt)
ground_truth = []


ground_truth = []
for query, request_arg in list(zip(queries, request_args)):
    feedback = input(f"Query: {query}\nRequest: {request_arg}\nRequested changes: ")
    if feedback == 'n' or feedback == 'none' or not feedback:
        ground_truth.append(request_arg)
        continue
    resolved = correction_chain.run(request=request_arg,
                            user_feedback=feedback)
    ground_truth.append(resolved.strip())
    print("Updated request:", resolved)
Query: Can you explain how to say 'hello' in Spanish?
Request: {"task_description": "say 'hello'", "learning_language": "Spanish", "native_language": "English", "full_query": "Can you explain how to say 'hello' in Spanish?"}
Requested changes:
Query: I need help understanding the French word for 'goodbye'.
Request: {"task_description": "understanding the French word for 'goodbye'", "learning_language": "French", "native_language": "English", "full_query": "I need help understanding the French word for 'goodbye'."}
Requested changes:
Query: Can you tell me how to say 'thank you' in German?
Request: {"task_description": "say 'thank you'", "learning_language": "German", "native_language": "English", "full_query": "Can you tell me how to say 'thank you' in German?"}
Requested changes:
Query: I'm trying to learn the Italian word for 'please'.
Request: {"task_description": "Learn the Italian word for 'please'", "learning_language": "Italian", "native_language": "English", "full_query": "I'm trying to learn the Italian word for 'please'."}
Requested changes:
Query: Can you help me with the pronunciation of 'yes' in Portuguese?


Query: Can you help me with the pronunciation of 'yes' in Portuguese?
Request: {"task_description": "Help with pronunciation of 'yes' in Portuguese", "learning_language": "Portuguese", "native_language": "English", "full_query": "Can you help me with the pronunciation of 'yes' in Portuguese?"}
Requested changes:
Query: I'm looking for the Dutch word for 'no'.
Request: {"task_description": "Find the Dutch word for 'no'", "learning_language": "Dutch", "native_language": "English", "full_query": "I'm looking for the Dutch word for 'no'."}
Requested changes:
Query: Can you explain the meaning of 'hello' in Japanese?
Request: {"task_description": "Explain the meaning of 'hello' in Japanese", "learning_language": "Japanese", "native_language": "English", "full_query": "Can you explain the meaning of 'hello' in Japanese?"}
Requested changes:
Query: I need help understanding the Russian word for 'thank you'.
Request: {"task_description": "understanding the Russian word for 'thank you'", "learning_language": "Russian", "native_language": "English", "full_query": "I need help understanding the Russian word for 'thank you'."}
Requested changes:
Query: Can you tell me how to say 'goodbye' in Chinese?
Request: {"task_description": "say goodbye", "learning_language": "Chinese", "native_language": "English", "full_query": "Can you tell me how to say 'goodbye' in Chinese?"}
Requested changes:
Query: I'm trying to learn the Arabic word for 'please'.


Requested changes:
Query: I'm trying to learn the Arabic word for 'please'.
Request: {"task_description": "Learn the Arabic word for 'please'", "learning_language": "Arabic", "native_language": "English", "full_query": "I'm trying to learn the Arabic word for 'please'."}
Requested changes:
Now you can use the ground_truth as shown above in Evaluate the Requests Chain!
# Now you have a new ground truth set to use as shown above!
ground_truth
['{"task_description": "say \'hello\'", "learning_language": "Spanish", "native_language": "English", "full_query": "Can you explain how to say \'hello\' in Spanish?"}',
 '{"task_description": "understanding the French word for \'goodbye\'", "learning_language": "French", "native_language": "English", "full_query": "I need help understanding the French word for \'goodbye\'."}',
 '{"task_description": "say \'thank you\'", "learning_language": "German", "native_language": "English", "full_query": "Can you tell me how to say \'thank you\' in German?"}',
 '{"task_description": "Learn the Italian word for \'please\'", "learning_language": "Italian", "native_language": "English", "full_query": "I\'m trying to learn the Italian word for \'please\'."}',
 '{"task_description": "Help with pronunciation of \'yes\' in Portuguese", "learning_language": "Portuguese", "native_language": "English", "full_query": "Can you help me with the pronunciation of \'yes\' in Portuguese?"}',
 '{"task_description": "Find the Dutch word for \'no\'", "learning_language": "Dutch", "native_language": "English", "full_query": "I\'m looking for the Dutch word for \'no\'."}',


'{"task_description": "Explain the meaning of \'hello\' in Japanese", "learning_language": "Japanese", "native_language": "English", "full_query": "Can you explain the meaning of \'hello\' in Japanese?"}',
 '{"task_description": "understanding the Russian word for \'thank you\'", "learning_language": "Russian", "native_language": "English", "full_query": "I need help understanding the Russian word for \'thank you\'."}',
 '{"task_description": "say goodbye", "learning_language": "Chinese", "native_language": "English", "full_query": "Can you tell me how to say \'goodbye\' in Chinese?"}',
 '{"task_description": "Learn the Arabic word for \'please\'", "learning_language": "Arabic", "native_language": "English", "full_query": "I\'m trying to learn the Arabic word for \'please\'."}']
previous
LLM Math
next
Question Answering Benchmarking: Paul Graham Essay
 Contents

Load the API Chain
Optional: Generate Input Questions and Request Ground Truth Queries
Run the API Chain
Evaluate the requests chain
Evaluate the Response Chain
Generating Test Datasets
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Apr 26, 2023.


.ipynb
.pdf
LLM Math
 Contents
Setting up a chain
LLM Math#
Evaluating chains that know how to do math.
# Comment this out if you are NOT using tracing
import os
os.environ["LANGCHAIN_HANDLER"] = "langchain"
from langchain.evaluation.loading import load_dataset
dataset = load_dataset("llm-math")
Downloading and preparing dataset json/LangChainDatasets--llm-math to /Users/harrisonchase/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--llm-math-509b11d101165afa/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51...
Dataset json downloaded and prepared to /Users/harrisonchase/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--llm-math-509b11d101165afa/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51. Subsequent calls will reuse this data.
Setting up a chain#
Now we need to create some pipelines for doing math.
from langchain.llms import OpenAI
from langchain.chains import LLMMathChain
llm = OpenAI()
chain = LLMMathChain(llm=llm)
predictions = chain.apply(dataset)
numeric_output = [float(p['answer'].strip().strip("Answer: ")) for p in predictions]
correct = [example['answer'] == numeric_output[i] for i, example in enumerate(dataset)]
sum(correct) / len(correct)
1.0


sum(correct) / len(correct)
1.0
for i, example in enumerate(dataset):
    print("input: ", example["question"])
    print("expected output :", example["answer"])
    print("prediction: ", numeric_output[i])
input:  5
expected output : 5.0
prediction:  5.0
input:  5 + 3
expected output : 8.0
prediction:  8.0
input:  2^3.171
expected output : 9.006708689094099
prediction:  9.006708689094099
input:    2 ^3.171
expected output : 9.006708689094099
prediction:  9.006708689094099
input:  two to the power of three point one hundred seventy one
expected output : 9.006708689094099
prediction:  9.006708689094099
input:  five + three squared minus 1
expected output : 13.0
prediction:  13.0
input:  2097 times 27.31
expected output : 57269.07
prediction:  57269.07
input:  two thousand ninety seven times twenty seven point thirty one
expected output : 57269.07
prediction:  57269.07
input:  209758 / 2714
expected output : 77.28739867354459
prediction:  77.28739867354459
input:  209758.857 divided by 2714.31
expected output : 77.27888745205964
prediction:  77.27888745205964
previous
Using Hugging Face Datasets
next
Evaluating an OpenAPI Chain
 Contents

Setting up a chain


next
Evaluating an OpenAPI Chain
 Contents

Setting up a chain
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Apr 26, 2023.


.ipynb
.pdf
Agent VectorDB Question Answering Benchmarking
 Contents
Loading the data
Setting up a chain
Make a prediction
Make many predictions
Evaluate performance
Agent VectorDB Question Answering Benchmarking#
Here we go over how to benchmark performance on a question answering task using an agent to route between multiple vectordatabases.
It is highly reccomended that you do any evaluation/benchmarking with tracing enabled. See here for an explanation of what tracing is and how to set it up.
# Comment this out if you are NOT using tracing
import os
os.environ["LANGCHAIN_HANDLER"] = "langchain"
Loading the data#
First, let’s load the data.
from langchain.evaluation.loading import load_dataset
dataset = load_dataset("agent-vectordb-qa-sota-pg")
Found cached dataset json (/Users/qt/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--agent-vectordb-qa-sota-pg-d3ae24016b514f92/0.0.0/fe5dd6ea2639a6df622901539cb550cf8797e5a6b2dd7af1cf934bed8e233e6e)
100%|██████████| 1/1 [00:00<00:00, 414.42it/s]
dataset[0]
{'question': 'What is the purpose of the NATO Alliance?',
 'answer': 'The purpose of the NATO Alliance is to secure peace and stability in Europe after World War 2.',
 'steps': [{'tool': 'State of Union QA System', 'tool_input': None},
  {'tool': None, 'tool_input': 'What is the purpose of the NATO Alliance?'}]}
dataset[-1]


dataset[-1]
{'question': 'What is the purpose of YC?',
 'answer': 'The purpose of YC is to cause startups to be founded that would not otherwise have existed.',
 'steps': [{'tool': 'Paul Graham QA System', 'tool_input': None},
  {'tool': None, 'tool_input': 'What is the purpose of YC?'}]}
Setting up a chain#
Now we need to create some pipelines for doing question answering. Step one in that is creating indexes over the data in question.
from langchain.document_loaders import TextLoader
loader = TextLoader("../../modules/state_of_the_union.txt")
from langchain.indexes import VectorstoreIndexCreator
vectorstore_sota = VectorstoreIndexCreator(vectorstore_kwargs={"collection_name":"sota"}).from_loaders([loader]).vectorstore
Using embedded DuckDB without persistence: data will be transient
Now we can create a question answering chain.
from langchain.chains import RetrievalQA
from langchain.llms import OpenAI
chain_sota = RetrievalQA.from_chain_type(llm=OpenAI(temperature=0), chain_type="stuff", retriever=vectorstore_sota.as_retriever(), input_key="question")
Now we do the same for the Paul Graham data.
loader = TextLoader("../../modules/paul_graham_essay.txt")
vectorstore_pg = VectorstoreIndexCreator(vectorstore_kwargs={"collection_name":"paul_graham"}).from_loaders([loader]).vectorstore
Using embedded DuckDB without persistence: data will be transient
chain_pg = RetrievalQA.from_chain_type(llm=OpenAI(temperature=0), chain_type="stuff", retriever=vectorstore_pg.as_retriever(), input_key="question")
We can now set up an agent to route between them.
from langchain.agents import initialize_agent, Tool


from langchain.agents import initialize_agent, Tool
from langchain.agents import AgentType
tools = [
    Tool(
        name = "State of Union QA System",
        func=chain_sota.run,
        description="useful for when you need to answer questions about the most recent state of the union address. Input should be a fully formed question."
    ),
    Tool(
        name = "Paul Graham System",
        func=chain_pg.run,
        description="useful for when you need to answer questions about Paul Graham. Input should be a fully formed question."
    ),
]
agent = initialize_agent(tools, OpenAI(temperature=0), agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, max_iterations=4)
Make a prediction#
First, we can make predictions one datapoint at a time. Doing it at this level of granularity allows use to explore the outputs in detail, and also is a lot cheaper than running over multiple datapoints
agent.run(dataset[0]['question'])
'The purpose of the NATO Alliance is to secure peace and stability in Europe after World War 2.'
Make many predictions#
Now we can make predictions
predictions = []
predicted_dataset = []
error_dataset = []
for data in dataset:
    new_data = {"input": data["question"], "answer": data["answer"]}
    try:
        predictions.append(agent(new_data))
        predicted_dataset.append(new_data)
    except Exception:
        error_dataset.append(new_data)
Evaluate performance#
Now we can evaluate the predictions. The first thing we can do is look at them by eye.
predictions[0]
{'input': 'What is the purpose of the NATO Alliance?',
 'answer': 'The purpose of the NATO Alliance is to secure peace and stability in Europe after World War 2.',


'output': 'The purpose of the NATO Alliance is to secure peace and stability in Europe after World War 2.'}
Next, we can use a language model to score them programatically
from langchain.evaluation.qa import QAEvalChain
llm = OpenAI(temperature=0)
eval_chain = QAEvalChain.from_llm(llm)
graded_outputs = eval_chain.evaluate(predicted_dataset, predictions, question_key="input", prediction_key="output")
We can add in the graded output to the predictions dict and then get a count of the grades.
for i, prediction in enumerate(predictions):
    prediction['grade'] = graded_outputs[i]['text']
from collections import Counter
Counter([pred['grade'] for pred in predictions])
Counter({' CORRECT': 28, ' INCORRECT': 5})
We can also filter the datapoints to the incorrect examples and look at them.
incorrect = [pred for pred in predictions if pred['grade'] == " INCORRECT"]
incorrect[0]
{'input': 'What are the four common sense steps that the author suggests to move forward safely?',
 'answer': 'The four common sense steps suggested by the author to move forward safely are: stay protected with vaccines and treatments, prepare for new variants, end the shutdown of schools and businesses, and stay vigilant.',
 'output': 'The four common sense steps suggested in the most recent State of the Union address are: cutting the cost of prescription drugs, providing a pathway to citizenship for Dreamers, revising laws so businesses have the workers they need and families don’t wait decades to reunite, and protecting access to health care and preserving a woman’s right to choose.',
 'grade': ' INCORRECT'}
previous
Agent Benchmarking: Search + Calculator
next
Benchmarking Template
 Contents

Loading the data
Setting up a chain
Make a prediction


Benchmarking Template
 Contents

Loading the data
Setting up a chain
Make a prediction
Make many predictions
Evaluate performance
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Apr 26, 2023.


.ipynb
.pdf
Agent Benchmarking: Search + Calculator
 Contents
Loading the data
Setting up a chain
Make a prediction
Make many predictions
Evaluate performance
Agent Benchmarking: Search + Calculator#
Here we go over how to benchmark performance of an agent on tasks where it has access to a calculator and a search tool.
It is highly reccomended that you do any evaluation/benchmarking with tracing enabled. See here for an explanation of what tracing is and how to set it up.
# Comment this out if you are NOT using tracing
import os
os.environ["LANGCHAIN_HANDLER"] = "langchain"
Loading the data#
First, let’s load the data.
from langchain.evaluation.loading import load_dataset
dataset = load_dataset("agent-search-calculator")
Setting up a chain#
Now we need to load an agent capable of answering these questions.
from langchain.llms import OpenAI
from langchain.chains import LLMMathChain
from langchain.agents import initialize_agent, Tool, load_tools
from langchain.agents import AgentType
tools = load_tools(['serpapi', 'llm-math'], llm=OpenAI(temperature=0))
agent = initialize_agent(tools, OpenAI(temperature=0), agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True)
Make a prediction#
First, we can make predictions one datapoint at a time. Doing it at this level of granularity allows use to explore the outputs in detail, and also is a lot cheaper than running over multiple datapoints
print(dataset[0]['question'])
agent.run(dataset[0]['question'])
Make many predictions#
Now we can make predictions
agent.run(dataset[4]['question'])
predictions = []
predicted_dataset = []
error_dataset = []
for data in dataset:


predictions = []
predicted_dataset = []
error_dataset = []
for data in dataset:
    new_data = {"input": data["question"], "answer": data["answer"]}
    try:
        predictions.append(agent(new_data))
        predicted_dataset.append(new_data)
    except Exception as e:
        predictions.append({"output": str(e), **new_data})
        error_dataset.append(new_data)
Evaluate performance#
Now we can evaluate the predictions. The first thing we can do is look at them by eye.
predictions[0]
Next, we can use a language model to score them programatically
from langchain.evaluation.qa import QAEvalChain
llm = OpenAI(temperature=0)
eval_chain = QAEvalChain.from_llm(llm)
graded_outputs = eval_chain.evaluate(dataset, predictions, question_key="question", prediction_key="output")
We can add in the graded output to the predictions dict and then get a count of the grades.
for i, prediction in enumerate(predictions):
    prediction['grade'] = graded_outputs[i]['text']
from collections import Counter
Counter([pred['grade'] for pred in predictions])
We can also filter the datapoints to the incorrect examples and look at them.
incorrect = [pred for pred in predictions if pred['grade'] == " INCORRECT"]
incorrect
previous
Evaluation
next
Agent VectorDB Question Answering Benchmarking
 Contents

Loading the data
Setting up a chain
Make a prediction
Make many predictions
Evaluate performance
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Apr 26, 2023.


.ipynb
.pdf
Generic Agent Evaluation
 Contents
Setup
Testing the Agent
Evaluating the Agent
Generic Agent Evaluation#
Good evaluation is key for quickly iterating on your agent’s prompts and tools. Here we provide an example of how to use the TrajectoryEvalChain to evaluate your agent.
Setup#
Let’s start by defining our agent.
from langchain import Wikipedia
from langchain.chat_models import ChatOpenAI
from langchain.agents import initialize_agent, Tool
from langchain.agents import AgentType
from langchain.agents.react.base import DocstoreExplorer
from langchain.memory import ConversationBufferMemory
from langchain import LLMMathChain
from langchain.llms import OpenAI
from langchain import SerpAPIWrapper
docstore = DocstoreExplorer(Wikipedia())
math_llm = OpenAI(temperature=0)
llm_math_chain = LLMMathChain(llm=math_llm, verbose=True)
search = SerpAPIWrapper()
tools = [
    Tool(
        name="Search",
        func=docstore.search,
        description="useful for when you need to ask with search",
    ),
    Tool(
        name="Lookup",
        func=docstore.lookup,
        description="useful for when you need to ask with lookup",
    ),
    Tool(
        name="Calculator",
        func=llm_math_chain.run,
        description="useful for doing calculations",
    ),
    Tool(
        name="Search the Web (SerpAPI)",
        func=search.run,
        description="useful for when you need to answer questions about current events",
    ),
]
memory = ConversationBufferMemory(
    memory_key="chat_history", return_messages=True, output_key="output"
)


memory_key="chat_history", return_messages=True, output_key="output"
)
llm = ChatOpenAI(temperature=0, model_name="gpt-3.5-turbo")
agent = initialize_agent(
    tools,
    llm,
    agent=AgentType.CHAT_CONVERSATIONAL_REACT_DESCRIPTION,
    verbose=True,
    memory=memory,
    return_intermediate_steps=True, # This is needed for the evaluation later
)
Testing the Agent#
Now let’s try our agent out on some example queries.
query_one = "How many ping pong balls would it take to fill the entire Empire State Building?"
test_outputs_one = agent({"input": query_one}, return_only_outputs=False)
> Entering new AgentExecutor chain...
{
    "action": "Search the Web (SerpAPI)",
    "action_input": "How many ping pong balls would it take to fill the entire Empire State Building?"
}
Observation: 12.8 billion. The volume of the Empire State Building Googles in at around 37 million ft³. A golf ball comes in at about 2.5 in³.
Thought:{
    "action": "Final Answer",
    "action_input": "It would take approximately 12.8 billion ping pong balls to fill the entire Empire State Building."
}
> Finished chain.
This looks good! Let’s try it out on another query.
query_two = "If you laid the Eiffel Tower end to end, how many would you need cover the US from coast to coast?"
test_outputs_two = agent({"input": query_two}, return_only_outputs=False)
> Entering new AgentExecutor chain...
{
    "action": "Calculator",


> Entering new AgentExecutor chain...
{
    "action": "Calculator",
    "action_input": "The length of the Eiffel Tower is 324 meters. The distance from coast to coast in the US is approximately 4,828 kilometers. First, we need to convert 4,828 kilometers to meters, which gives us 4,828,000 meters. To find out how many Eiffel Towers we need, we can divide 4,828,000 by 324. This gives us approximately 14,876 Eiffel Towers."
}
> Entering new LLMMathChain chain...
The length of the Eiffel Tower is 324 meters. The distance from coast to coast in the US is approximately 4,828 kilometers. First, we need to convert 4,828 kilometers to meters, which gives us 4,828,000 meters. To find out how many Eiffel Towers we need, we can divide 4,828,000 by 324. This gives us approximately 14,876 Eiffel Towers.
```text
4828000 / 324
```
...numexpr.evaluate("4828000 / 324")...
Answer: 14901.234567901234
> Finished chain.
Observation: Answer: 14901.234567901234
Thought:{
    "action": "Calculator",
    "action_input": "The length of the Eiffel Tower is 324 meters. The distance from coast to coast in the US is approximately 4,828 kilometers. First, we need to convert 4,828 kilometers to meters, which gives us 4,828,000 meters. To find out how many Eiffel Towers we need, we can divide 4,828,000 by 324. This gives us approximately 14,901 Eiffel Towers."
}


}
> Entering new LLMMathChain chain...
The length of the Eiffel Tower is 324 meters. The distance from coast to coast in the US is approximately 4,828 kilometers. First, we need to convert 4,828 kilometers to meters, which gives us 4,828,000 meters. To find out how many Eiffel Towers we need, we can divide 4,828,000 by 324. This gives us approximately 14,901 Eiffel Towers.
```text
4828000 / 324
```
...numexpr.evaluate("4828000 / 324")...
Answer: 14901.234567901234
> Finished chain.
Observation: Answer: 14901.234567901234
Thought:{
    "action": "Final Answer",
    "action_input": "If you laid the Eiffel Tower end to end, you would need approximately 14,901 Eiffel Towers to cover the US from coast to coast."
}
> Finished chain.
This doesn’t look so good. Let’s try running some evaluation.
Evaluating the Agent#
Let’s start by defining the TrajectoryEvalChain.
from langchain.evaluation.agents import TrajectoryEvalChain
# Define chain
eval_chain = TrajectoryEvalChain.from_llm(
    llm=ChatOpenAI(temperature=0, model_name="gpt-4"), # Note: This must be a ChatOpenAI model
    agent_tools=agent.tools,
    return_reasoning=True,
)
Let’s try evaluating the first query.
question, steps, answer = test_outputs_one["input"], test_outputs_one["intermediate_steps"], test_outputs_one["output"]
evaluation = eval_chain(
    inputs={"question": question, "answer": answer, "agent_trajectory": eval_chain.get_agent_trajectory(steps)},


)
print("Score from 1 to 5: ", evaluation["score"])
print("Reasoning: ", evaluation["reasoning"])
Score from 1 to 5:  1
Reasoning:  First, let's evaluate the final answer. The final answer is incorrect because it uses the volume of golf balls instead of ping pong balls. The answer is not helpful.
Second, does the model use a logical sequence of tools to answer the question? The model only used one tool, which was the Search the Web (SerpAPI). It did not use the Calculator tool to calculate the correct volume of ping pong balls.
Third, does the AI language model use the tools in a helpful way? The model used the Search the Web (SerpAPI) tool, but the output was not helpful because it provided information about golf balls instead of ping pong balls.
Fourth, does the AI language model use too many steps to answer the question? The model used only one step, which is not too many. However, it should have used more steps to provide a correct answer.
Fifth, are the appropriate tools used to answer the question? The model should have used the Search tool to find the volume of the Empire State Building and the volume of a ping pong ball. Then, it should have used the Calculator tool to calculate the number of ping pong balls needed to fill the building.
Judgment: Given the incorrect final answer and the inappropriate use of tools, we give the model a score of 1.
That seems about right. Let’s try the second query.
question, steps, answer = test_outputs_two["input"], test_outputs_two["intermediate_steps"], test_outputs_two["output"]
evaluation = eval_chain(
    inputs={"question": question, "answer": answer, "agent_trajectory": eval_chain.get_agent_trajectory(steps)},
)


)
print("Score from 1 to 5: ", evaluation["score"])
print("Reasoning: ", evaluation["reasoning"])
Score from 1 to 5:  3
Reasoning:  i. Is the final answer helpful?
Yes, the final answer is helpful as it provides an approximate number of Eiffel Towers needed to cover the US from coast to coast.
ii. Does the AI language use a logical sequence of tools to answer the question?
No, the AI language model does not use a logical sequence of tools. It directly uses the Calculator tool without first using the Search or Lookup tools to find the necessary information (length of the Eiffel Tower and distance from coast to coast in the US).
iii. Does the AI language model use the tools in a helpful way?
The AI language model uses the Calculator tool in a helpful way to perform the calculation, but it should have used the Search or Lookup tools first to find the required information.
iv. Does the AI language model use too many steps to answer the question?
No, the AI language model does not use too many steps. However, it repeats the same step twice, which is unnecessary.
v. Are the appropriate tools used to answer the question?
Not entirely. The AI language model should have used the Search or Lookup tools to find the required information before using the Calculator tool.
Given the above evaluation, the AI language model's performance can be scored as follows:
That also sounds about right. In conclusion, the TrajectoryEvalChain allows us to use GPT-4 to score both our agent’s outputs and tool use in addition to giving us the reasoning behind the evaluation.
previous
Data Augmented Question Answering
next
Using Hugging Face Datasets
 Contents

Setup
Testing the Agent
Evaluating the Agent
By Harrison Chase


Setup
Testing the Agent
Evaluating the Agent
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Apr 26, 2023.


.ipynb
.pdf
Question Answering
 Contents
Setup
Examples
Predictions
Evaluation
Customize Prompt
Evaluation without Ground Truth
Comparing to other evaluation metrics
Question Answering#
This notebook covers how to evaluate generic question answering problems. This is a situation where you have an example containing a question and its corresponding ground truth answer, and you want to measure how well the language model does at answering those questions.
Setup#
For demonstration purposes, we will just evaluate a simple question answering system that only evaluates the model’s internal knowledge. Please see other notebooks for examples where it evaluates how the model does at question answering over data not present in what the model was trained on.
from langchain.prompts import PromptTemplate
from langchain.chains import LLMChain
from langchain.llms import OpenAI
prompt = PromptTemplate(template="Question: {question}\nAnswer:", input_variables=["question"])
llm = OpenAI(model_name="text-davinci-003", temperature=0)
chain = LLMChain(llm=llm, prompt=prompt)
Examples#
For this purpose, we will just use two simple hardcoded examples, but see other notebooks for tips on how to get and/or generate these examples.
examples = [
    {
        "question": "Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now?",
        "answer": "11"
    },
    {
        "question": 'Is the following sentence plausible? "Joao Moutinho caught the screen pass in the NFC championship."',
        "answer": "No"
    }
]
Predictions#
We can now make and inspect the predictions for these questions.
predictions = chain.apply(examples)
predictions
[{'text': ' 11 tennis balls'},


predictions = chain.apply(examples)
predictions
[{'text': ' 11 tennis balls'},
 {'text': ' No, this sentence is not plausible. Joao Moutinho is a professional soccer player, not an American football player, so it is not likely that he would be catching a screen pass in the NFC championship.'}]
Evaluation#
We can see that if we tried to just do exact match on the answer answers (11 and No) they would not match what the language model answered. However, semantically the language model is correct in both cases. In order to account for this, we can use a language model itself to evaluate the answers.
from langchain.evaluation.qa import QAEvalChain
llm = OpenAI(temperature=0)
eval_chain = QAEvalChain.from_llm(llm)
graded_outputs = eval_chain.evaluate(examples, predictions, question_key="question", prediction_key="text")
for i, eg in enumerate(examples):
    print(f"Example {i}:")
    print("Question: " + eg['question'])
    print("Real Answer: " + eg['answer'])
    print("Predicted Answer: " + predictions[i]['text'])
    print("Predicted Grade: " + graded_outputs[i]['text'])
    print()
Example 0:
Question: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now?
Real Answer: 11
Predicted Answer:  11 tennis balls
Predicted Grade:  CORRECT
Example 1:
Question: Is the following sentence plausible? "Joao Moutinho caught the screen pass in the NFC championship."
Real Answer: No


Real Answer: No
Predicted Answer:  No, this sentence is not plausible. Joao Moutinho is a professional soccer player, not an American football player, so it is not likely that he would be catching a screen pass in the NFC championship.
Predicted Grade:  CORRECT
Customize Prompt#
You can also customize the prompt that is used. Here is an example prompting it using a score from 0 to 10.
The custom prompt requires 3 input variables: “query”, “answer” and “result”. Where “query” is the question, “answer” is the ground truth answer, and “result” is the predicted answer.
from langchain.prompts.prompt import PromptTemplate
_PROMPT_TEMPLATE = """You are an expert professor specialized in grading students' answers to questions.
You are grading the following question:
{query}
Here is the real answer:
{answer}
You are grading the following predicted answer:
{result}
What grade do you give from 0 to 10, where 0 is the lowest (very low similarity) and 10 is the highest (very high similarity)?
"""
PROMPT = PromptTemplate(input_variables=["query", "answer", "result"], template=_PROMPT_TEMPLATE)
evalchain = QAEvalChain.from_llm(llm=llm,prompt=PROMPT)
evalchain.evaluate(examples, predictions, question_key="question", answer_key="answer", prediction_key="text")
Evaluation without Ground Truth#
Its possible to evaluate question answering systems without ground truth. You would need a "context" input that reflects what the information the LLM uses to answer the question. This context can be obtained by any retreival system. Here’s an example of how it works:
context_examples = [
    {
        "question": "How old am I?",


context_examples = [
    {
        "question": "How old am I?",
        "context": "I am 30 years old. I live in New York and take the train to work everyday.",
    },
    {
        "question": 'Who won the NFC championship game in 2023?"',
        "context": "NFC Championship Game 2023: Philadelphia Eagles 31, San Francisco 49ers 7"
    }
]
QA_PROMPT = "Answer the question based on the  context\nContext:{context}\nQuestion:{question}\nAnswer:"
template = PromptTemplate(input_variables=["context", "question"], template=QA_PROMPT)
qa_chain = LLMChain(llm=llm, prompt=template)
predictions = qa_chain.apply(context_examples)
predictions
[{'text': 'You are 30 years old.'},
 {'text': ' The Philadelphia Eagles won the NFC championship game in 2023.'}]
from langchain.evaluation.qa import ContextQAEvalChain
eval_chain = ContextQAEvalChain.from_llm(llm)
graded_outputs = eval_chain.evaluate(context_examples, predictions, question_key="question", prediction_key="text")
graded_outputs
[{'text': ' CORRECT'}, {'text': ' CORRECT'}]
Comparing to other evaluation metrics#
We can compare the evaluation results we get to other common evaluation metrics. To do this, let’s load some evaluation metrics from HuggingFace’s evaluate package.
# Some data munging to get the examples in the right format
for i, eg in enumerate(examples):
    eg['id'] = str(i)
    eg['answers'] = {"text": [eg['answer']], "answer_start": [0]}
    predictions[i]['id'] = str(i)


predictions[i]['id'] = str(i)
    predictions[i]['prediction_text'] = predictions[i]['text']
for p in predictions:
    del p['text']
new_examples = examples.copy()
for eg in new_examples:
    del eg ['question']
    del eg['answer']
from evaluate import load
squad_metric = load("squad")
results = squad_metric.compute(
    references=new_examples,
    predictions=predictions,
)
results
{'exact_match': 0.0, 'f1': 28.125}
previous
QA Generation
next
SQL Question Answering Benchmarking: Chinook
 Contents

Setup
Examples
Predictions
Evaluation
Customize Prompt
Evaluation without Ground Truth
Comparing to other evaluation metrics
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Apr 26, 2023.


.ipynb
.pdf
SQL Question Answering Benchmarking: Chinook
 Contents
Loading the data
Setting up a chain
Make a prediction
Make many predictions
Evaluate performance
SQL Question Answering Benchmarking: Chinook#
Here we go over how to benchmark performance on a question answering task over a SQL database.
It is highly reccomended that you do any evaluation/benchmarking with tracing enabled. See here for an explanation of what tracing is and how to set it up.
# Comment this out if you are NOT using tracing
import os
os.environ["LANGCHAIN_HANDLER"] = "langchain"
Loading the data#
First, let’s load the data.
from langchain.evaluation.loading import load_dataset
dataset = load_dataset("sql-qa-chinook")
Downloading and preparing dataset json/LangChainDatasets--sql-qa-chinook to /Users/harrisonchase/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--sql-qa-chinook-7528565d2d992b47/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51...
Dataset json downloaded and prepared to /Users/harrisonchase/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--sql-qa-chinook-7528565d2d992b47/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51. Subsequent calls will reuse this data.
dataset[0]
{'question': 'How many employees are there?', 'answer': '8'}


{'question': 'How many employees are there?', 'answer': '8'}
Setting up a chain#
This uses the example Chinook database.
To set it up follow the instructions on https://database.guide/2-sample-databases-sqlite/, placing the .db file in a notebooks folder at the root of this repository.
Note that here we load a simple chain. If you want to experiment with more complex chains, or an agent, just create the chain object in a different way.
from langchain import OpenAI, SQLDatabase, SQLDatabaseChain
db = SQLDatabase.from_uri("sqlite:///../../../notebooks/Chinook.db")
llm = OpenAI(temperature=0)
Now we can create a SQL database chain.
chain = SQLDatabaseChain(llm=llm, database=db, input_key="question")
Make a prediction#
First, we can make predictions one datapoint at a time. Doing it at this level of granularity allows use to explore the outputs in detail, and also is a lot cheaper than running over multiple datapoints
chain(dataset[0])
{'question': 'How many employees are there?',
 'answer': '8',
 'result': ' There are 8 employees.'}
Make many predictions#
Now we can make predictions. Note that we add a try-except because this chain can sometimes error (if SQL is written incorrectly, etc)
predictions = []
predicted_dataset = []
error_dataset = []
for data in dataset:
    try:
        predictions.append(chain(data))
        predicted_dataset.append(data)
    except:
        error_dataset.append(data)
Evaluate performance#
Now we can evaluate the predictions. We can use a language model to score them programatically
from langchain.evaluation.qa import QAEvalChain
llm = OpenAI(temperature=0)


llm = OpenAI(temperature=0)
eval_chain = QAEvalChain.from_llm(llm)
graded_outputs = eval_chain.evaluate(predicted_dataset, predictions, question_key="question", prediction_key="result")
We can add in the graded output to the predictions dict and then get a count of the grades.
for i, prediction in enumerate(predictions):
    prediction['grade'] = graded_outputs[i]['text']
from collections import Counter
Counter([pred['grade'] for pred in predictions])
Counter({' CORRECT': 3, ' INCORRECT': 4})
We can also filter the datapoints to the incorrect examples and look at them.
incorrect = [pred for pred in predictions if pred['grade'] == " INCORRECT"]
incorrect[0]
{'question': 'How many employees are also customers?',
 'answer': 'None',
 'result': ' 59 employees are also customers.',
 'grade': ' INCORRECT'}
previous
Question Answering
next
Installation
 Contents

Loading the data
Setting up a chain
Make a prediction
Make many predictions
Evaluate performance
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Apr 26, 2023.


.ipynb
.pdf
Benchmarking Template
 Contents
Loading the data
Setting up a chain
Make a prediction
Make many predictions
Evaluate performance
Benchmarking Template#
This is an example notebook that can be used to create a benchmarking notebook for a task of your choice. Evaluation is really hard, and so we greatly welcome any contributions that can make it easier for people to experiment
It is highly reccomended that you do any evaluation/benchmarking with tracing enabled. See here for an explanation of what tracing is and how to set it up.
# Comment this out if you are NOT using tracing
import os
os.environ["LANGCHAIN_HANDLER"] = "langchain"
Loading the data#
First, let’s load the data.
# This notebook should so how to load the dataset from LangChainDatasets on Hugging Face
# Please upload your dataset to https://huggingface.co/LangChainDatasets
# The value passed into `load_dataset` should NOT have the `LangChainDatasets/` prefix
from langchain.evaluation.loading import load_dataset
dataset = load_dataset("TODO")
Setting up a chain#
This next section should have an example of setting up a chain that can be run on this dataset.
Make a prediction#
First, we can make predictions one datapoint at a time. Doing it at this level of granularity allows use to explore the outputs in detail, and also is a lot cheaper than running over multiple datapoints
# Example of running the chain on a single datapoint (`dataset[0]`) goes here
Make many predictions#
Now we can make predictions.
# Example of running the chain on many predictions goes here
# Sometimes its as simple as `chain.apply(dataset)`
# Othertimes you may want to write a for loop to catch errors
Evaluate performance#


# Othertimes you may want to write a for loop to catch errors
Evaluate performance#
Any guide to evaluating performance in a more systematic manner goes here.
previous
Agent VectorDB Question Answering Benchmarking
next
Data Augmented Question Answering
 Contents

Loading the data
Setting up a chain
Make a prediction
Make many predictions
Evaluate performance
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Apr 26, 2023.


.ipynb
.pdf
Question Answering Benchmarking: Paul Graham Essay
 Contents
Loading the data
Setting up a chain
Make a prediction
Make many predictions
Evaluate performance
Question Answering Benchmarking: Paul Graham Essay#
Here we go over how to benchmark performance on a question answering task over a Paul Graham essay.
It is highly reccomended that you do any evaluation/benchmarking with tracing enabled. See here for an explanation of what tracing is and how to set it up.
# Comment this out if you are NOT using tracing
import os
os.environ["LANGCHAIN_HANDLER"] = "langchain"
Loading the data#
First, let’s load the data.
from langchain.evaluation.loading import load_dataset
dataset = load_dataset("question-answering-paul-graham")
Found cached dataset json (/Users/harrisonchase/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--question-answering-paul-graham-76e8f711e038d742/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)
Setting up a chain#
Now we need to create some pipelines for doing question answering. Step one in that is creating an index over the data in question.
from langchain.document_loaders import TextLoader
loader = TextLoader("../../modules/paul_graham_essay.txt")
from langchain.indexes import VectorstoreIndexCreator
vectorstore = VectorstoreIndexCreator().from_loaders([loader]).vectorstore
Running Chroma using direct local API.
Using DuckDB in-memory for database. Data will be transient.
Now we can create a question answering chain.
from langchain.chains import RetrievalQA


Now we can create a question answering chain.
from langchain.chains import RetrievalQA
from langchain.llms import OpenAI
chain = RetrievalQA.from_chain_type(llm=OpenAI(), chain_type="stuff", retriever=vectorstore.as_retriever(), input_key="question")
Make a prediction#
First, we can make predictions one datapoint at a time. Doing it at this level of granularity allows use to explore the outputs in detail, and also is a lot cheaper than running over multiple datapoints
chain(dataset[0])
{'question': 'What were the two main things the author worked on before college?',
 'answer': 'The two main things the author worked on before college were writing and programming.',
 'result': ' Writing and programming.'}
Make many predictions#
Now we can make predictions
predictions = chain.apply(dataset)
Evaluate performance#
Now we can evaluate the predictions. The first thing we can do is look at them by eye.
predictions[0]
{'question': 'What were the two main things the author worked on before college?',
 'answer': 'The two main things the author worked on before college were writing and programming.',
 'result': ' Writing and programming.'}
Next, we can use a language model to score them programatically
from langchain.evaluation.qa import QAEvalChain
llm = OpenAI(temperature=0)
eval_chain = QAEvalChain.from_llm(llm)
graded_outputs = eval_chain.evaluate(dataset, predictions, question_key="question", prediction_key="result")
We can add in the graded output to the predictions dict and then get a count of the grades.
for i, prediction in enumerate(predictions):
    prediction['grade'] = graded_outputs[i]['text']
from collections import Counter
Counter([pred['grade'] for pred in predictions])


from collections import Counter
Counter([pred['grade'] for pred in predictions])
Counter({' CORRECT': 12, ' INCORRECT': 10})
We can also filter the datapoints to the incorrect examples and look at them.
incorrect = [pred for pred in predictions if pred['grade'] == " INCORRECT"]
incorrect[0]
{'question': 'What did the author write their dissertation on?',
 'answer': 'The author wrote their dissertation on applications of continuations.',
 'result': ' The author does not mention what their dissertation was on, so it is not known.',
 'grade': ' INCORRECT'}
previous
Evaluating an OpenAPI Chain
next
Question Answering Benchmarking: State of the Union Address
 Contents

Loading the data
Setting up a chain
Make a prediction
Make many predictions
Evaluate performance
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Apr 26, 2023.


.ipynb
.pdf
Data Augmented Question Answering
 Contents
Setup
Examples
Evaluate
Evaluate with Other Metrics
Data Augmented Question Answering#
This notebook uses some generic prompts/language models to evaluate an question answering system that uses other sources of data besides what is in the model. For example, this can be used to evaluate a question answering system over your proprietary data.
Setup#
Let’s set up an example with our favorite example - the state of the union address.
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores import Chroma
from langchain.text_splitter import CharacterTextSplitter
from langchain.llms import OpenAI
from langchain.chains import RetrievalQA
from langchain.document_loaders import TextLoader
loader = TextLoader('../../modules/state_of_the_union.txt')
documents = loader.load()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
texts = text_splitter.split_documents(documents)
embeddings = OpenAIEmbeddings()
docsearch = Chroma.from_documents(texts, embeddings)
qa = RetrievalQA.from_llm(llm=OpenAI(), retriever=docsearch.as_retriever())
Running Chroma using direct local API.
Using DuckDB in-memory for database. Data will be transient.
Examples#
Now we need some examples to evaluate. We can do this in two ways:
Hard code some examples ourselves
Generate examples automatically, using a language model
# Hard-coded examples
examples = [
    {
        "query": "What did the president say about Ketanji Brown Jackson",
        "answer": "He praised her legal ability and said he nominated her for the supreme court."
    },
    {
        "query": "What did the president say about Michael Jackson",
        "answer": "Nothing"


"answer": "Nothing"
    }
]
# Generated examples
from langchain.evaluation.qa import QAGenerateChain
example_gen_chain = QAGenerateChain.from_llm(OpenAI())
new_examples = example_gen_chain.apply_and_parse([{"doc": t} for t in texts[:5]])
new_examples
[{'query': 'According to the document, what did Vladimir Putin miscalculate?',
  'answer': 'He miscalculated that he could roll into Ukraine and the world would roll over.'},
 {'query': 'Who is the Ukrainian Ambassador to the United States?',
  'answer': 'The Ukrainian Ambassador to the United States is here tonight.'},
 {'query': 'How many countries were part of the coalition formed to confront Putin?',
  'answer': '27 members of the European Union, France, Germany, Italy, the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.'},
 {'query': 'What action is the U.S. Department of Justice taking to target Russian oligarchs?',
  'answer': 'The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and joining with European allies to find and seize their yachts, luxury apartments, and private jets.'},
 {'query': 'How much direct assistance is the United States providing to Ukraine?',
  'answer': 'The United States is providing more than $1 Billion in direct assistance to Ukraine.'}]
# Combine examples
examples += new_examples
Evaluate#
Now that we have examples, we can use the question answering evaluator to evaluate our question answering chain.
from langchain.evaluation.qa import QAEvalChain
predictions = qa.apply(examples)
llm = OpenAI(temperature=0)
eval_chain = QAEvalChain.from_llm(llm)


eval_chain = QAEvalChain.from_llm(llm)
graded_outputs = eval_chain.evaluate(examples, predictions)
for i, eg in enumerate(examples):
    print(f"Example {i}:")
    print("Question: " + predictions[i]['query'])
    print("Real Answer: " + predictions[i]['answer'])
    print("Predicted Answer: " + predictions[i]['result'])
    print("Predicted Grade: " + graded_outputs[i]['text'])
    print()
Example 0:
Question: What did the president say about Ketanji Brown Jackson
Real Answer: He praised her legal ability and said he nominated her for the supreme court.
Predicted Answer:  The president said that she is one of the nation's top legal minds, a former top litigator in private practice, a former federal public defender, and from a family of public school educators and police officers. He also said that she is a consensus builder and that she has received a broad range of support from the Fraternal Order of Police to former judges appointed by both Democrats and Republicans.
Predicted Grade:  CORRECT
Example 1:
Question: What did the president say about Michael Jackson
Real Answer: Nothing
Predicted Answer:  The president did not mention Michael Jackson in this speech.
Predicted Grade:  CORRECT
Example 2:
Question: According to the document, what did Vladimir Putin miscalculate?
Real Answer: He miscalculated that he could roll into Ukraine and the world would roll over.
Predicted Answer:  Putin miscalculated that the world would roll over when he rolled into Ukraine.
Predicted Grade:  CORRECT
Example 3:
Question: Who is the Ukrainian Ambassador to the United States?
Real Answer: The Ukrainian Ambassador to the United States is here tonight.
Predicted Answer:  I don't know.


Predicted Answer:  I don't know.
Predicted Grade:  INCORRECT
Example 4:
Question: How many countries were part of the coalition formed to confront Putin?
Real Answer: 27 members of the European Union, France, Germany, Italy, the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.
Predicted Answer:  The coalition included freedom-loving nations from Europe and the Americas to Asia and Africa, 27 members of the European Union including France, Germany, Italy, the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.
Predicted Grade:  INCORRECT
Example 5:
Question: What action is the U.S. Department of Justice taking to target Russian oligarchs?
Real Answer: The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and joining with European allies to find and seize their yachts, luxury apartments, and private jets.
Predicted Answer:  The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and to find and seize their yachts, luxury apartments, and private jets.
Predicted Grade:  INCORRECT
Example 6:
Question: How much direct assistance is the United States providing to Ukraine?
Real Answer: The United States is providing more than $1 Billion in direct assistance to Ukraine.
Predicted Answer:  The United States is providing more than $1 billion in direct assistance to Ukraine.
Predicted Grade:  CORRECT
Evaluate with Other Metrics#


Predicted Grade:  CORRECT
Evaluate with Other Metrics#
In addition to predicting whether the answer is correct or incorrect using a language model, we can also use other metrics to get a more nuanced view on the quality of the answers. To do so, we can use the Critique library, which allows for simple calculation of various metrics over generated text.
First you can get an API key from the Inspired Cognition Dashboard and do some setup:
export INSPIREDCO_API_KEY="..."
pip install inspiredco
import inspiredco.critique
import os
critique = inspiredco.critique.Critique(api_key=os.environ['INSPIREDCO_API_KEY'])
Then run the following code to set up the configuration and calculate the ROUGE, chrf, BERTScore, and UniEval (you can choose other metrics too):
metrics = {
    "rouge": {
        "metric": "rouge",
        "config": {"variety": "rouge_l"},
    },
    "chrf": {
        "metric": "chrf",
        "config": {},
    },
    "bert_score": {
        "metric": "bert_score",
        "config": {"model": "bert-base-uncased"},
    },
    "uni_eval": {
        "metric": "uni_eval",
        "config": {"task": "summarization", "evaluation_aspect": "relevance"},
    },
}
critique_data = [
    {"target": pred['result'], "references": [pred['answer']]} for pred in predictions
]
eval_results = {
    k: critique.evaluate(dataset=critique_data, metric=v["metric"], config=v["config"])
    for k, v in metrics.items()
}


for k, v in metrics.items()
}
Finally, we can print out the results. We can see that overall the scores are higher when the output is semantically correct, and also when the output closely matches with the gold-standard answer.
for i, eg in enumerate(examples):
    score_string = ", ".join([f"{k}={v['examples'][i]['value']:.4f}" for k, v in eval_results.items()])
    print(f"Example {i}:")
    print("Question: " + predictions[i]['query'])
    print("Real Answer: " + predictions[i]['answer'])
    print("Predicted Answer: " + predictions[i]['result'])
    print("Predicted Scores: " + score_string)
    print()
Example 0:
Question: What did the president say about Ketanji Brown Jackson
Real Answer: He praised her legal ability and said he nominated her for the supreme court.
Predicted Answer:  The president said that she is one of the nation's top legal minds, a former top litigator in private practice, a former federal public defender, and from a family of public school educators and police officers. He also said that she is a consensus builder and that she has received a broad range of support from the Fraternal Order of Police to former judges appointed by both Democrats and Republicans.
Predicted Scores: rouge=0.0941, chrf=0.2001, bert_score=0.5219, uni_eval=0.9043
Example 1:
Question: What did the president say about Michael Jackson
Real Answer: Nothing
Predicted Answer:  The president did not mention Michael Jackson in this speech.
Predicted Scores: rouge=0.0000, chrf=0.1087, bert_score=0.3486, uni_eval=0.7802


Example 2:
Question: According to the document, what did Vladimir Putin miscalculate?
Real Answer: He miscalculated that he could roll into Ukraine and the world would roll over.
Predicted Answer:  Putin miscalculated that the world would roll over when he rolled into Ukraine.
Predicted Scores: rouge=0.5185, chrf=0.6955, bert_score=0.8421, uni_eval=0.9578
Example 3:
Question: Who is the Ukrainian Ambassador to the United States?
Real Answer: The Ukrainian Ambassador to the United States is here tonight.
Predicted Answer:  I don't know.
Predicted Scores: rouge=0.0000, chrf=0.0375, bert_score=0.3159, uni_eval=0.7493
Example 4:
Question: How many countries were part of the coalition formed to confront Putin?
Real Answer: 27 members of the European Union, France, Germany, Italy, the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.
Predicted Answer:  The coalition included freedom-loving nations from Europe and the Americas to Asia and Africa, 27 members of the European Union including France, Germany, Italy, the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.
Predicted Scores: rouge=0.7419, chrf=0.8602, bert_score=0.8388, uni_eval=0.0669
Example 5:
Question: What action is the U.S. Department of Justice taking to target Russian oligarchs?


Question: What action is the U.S. Department of Justice taking to target Russian oligarchs?
Real Answer: The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and joining with European allies to find and seize their yachts, luxury apartments, and private jets.
Predicted Answer:  The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and to find and seize their yachts, luxury apartments, and private jets.
Predicted Scores: rouge=0.9412, chrf=0.8687, bert_score=0.9607, uni_eval=0.9718
Example 6:
Question: How much direct assistance is the United States providing to Ukraine?
Real Answer: The United States is providing more than $1 Billion in direct assistance to Ukraine.
Predicted Answer:  The United States is providing more than $1 billion in direct assistance to Ukraine.
Predicted Scores: rouge=1.0000, chrf=0.9483, bert_score=1.0000, uni_eval=0.9734
previous
Benchmarking Template
next
Generic Agent Evaluation
 Contents

Setup
Examples
Evaluate
Evaluate with Other Metrics
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Apr 26, 2023.


.ipynb
.pdf
Multi-Player Dungeons & Dragons
 Contents
Import LangChain related modules
DialogueAgent class
DialogueSimulator class
Define roles and quest
Ask an LLM to add detail to the game description
Use an LLM to create an elaborate quest description
Main Loop
Multi-Player Dungeons & Dragons#
This notebook shows how the DialogueAgent and DialogueSimulator class make it easy to extend the Two-Player Dungeons & Dragons example to multiple players.
The main difference between simulating two players and multiple players is in revising the schedule for when each agent speaks
To this end, we augment DialogueSimulator to take in a custom function that determines the schedule of which agent speaks. In the example below, each character speaks in round-robin fashion, with the storyteller interleaved between each player.
Import LangChain related modules#
from typing import List, Dict, Callable
from langchain.chat_models import ChatOpenAI
from langchain.schema import (
    AIMessage,
    HumanMessage,
    SystemMessage,
    BaseMessage,
)
DialogueAgent class#
The DialogueAgent class is a simple wrapper around the ChatOpenAI model that stores the message history from the dialogue_agent’s point of view by simply concatenating the messages as strings.
It exposes two methods:
send(): applies the chatmodel to the message history and returns the message string
receive(name, message): adds the message spoken by name to message history
class DialogueAgent():
    def __init__(
        self,
        name,
        system_message: SystemMessage,
        model: ChatOpenAI,
    ) -> None:
        self.name = name
        self.system_message = system_message
        self.model = model
        self.message_history = f"""Here is the conversation so far.
        """
        self.prefix = f'\n{self.name}:'


"""
        self.prefix = f'\n{self.name}:'

    def send(self) -> str:
        """
        Applies the chatmodel to the message history
        and returns the message string
        """
        message = self.model(
            [self.system_message,
             HumanMessage(content=self.message_history+self.prefix)])
        return message.content

    def receive(self, name: str, message: str) -> None:
        """
        Concatenates {message} spoken by {name} into message history
        """
        self.message_history += f'\n{name}: {message}'
DialogueSimulator class#
The DialogueSimulator class takes a list of agents. At each step, it performs the following:
Select the next speaker
Calls the next speaker to send a message
Broadcasts the message to all other agents
Update the step counter.
The selection of the next speaker can be implemented as any function, but in this case we simply loop through the agents.
class DialogueSimulator():

    def __init__(
        self,
        agents: List[DialogueAgent],
        selection_function: Callable[[int, List[DialogueAgent]], int]
    ) -> None:
        self.agents = agents
        self._step = 0
        self.select_next_speaker = selection_function

    def reset(self, name: str, message: str):
        """
        Initiates the conversation with a {message} from {name}
        """
        for agent in self.agents:
            agent.receive(name, message)

        # increment time
        self._step += 1

    def step(self) -> tuple[str, str]:
        # 1. choose the next speaker


# 1. choose the next speaker
        speaker_idx = self.select_next_speaker(self._step, self.agents)
        speaker = self.agents[speaker_idx]

        # 2. next speaker sends message
        message = speaker.send()

        # 3. everyone receives message
        for receiver in self.agents:
            receiver.receive(speaker.name, message)

        # 4. increment time
        self._step += 1

        return speaker.name, message
Define roles and quest#
character_names = ["Harry Potter", "Ron Weasley", "Hermione Granger", "Argus Filch"]
storyteller_name = "Dungeon Master"
quest = "Find all of Lord Voldemort's seven horcruxes."
word_limit = 50 # word limit for task brainstorming
Ask an LLM to add detail to the game description#
game_description = f"""Here is the topic for a Dungeons & Dragons game: {quest}.
        The characters are: {*character_names,}.
        The story is narrated by the storyteller, {storyteller_name}."""
player_descriptor_system_message = SystemMessage(
    content="You can add detail to the description of a Dungeons & Dragons player.")
def generate_character_description(character_name):
    character_specifier_prompt = [
        player_descriptor_system_message,
        HumanMessage(content=
            f"""{game_description}
            Please reply with a creative description of the character, {character_name}, in {word_limit} words or less.
            Speak directly to {character_name}.
            Do not add anything else."""
            )
    ]
    character_description = ChatOpenAI(temperature=1.0)(character_specifier_prompt).content
    return character_description
def generate_character_system_message(character_name, character_description):


return character_description
def generate_character_system_message(character_name, character_description):
    return SystemMessage(content=(
    f"""{game_description}
    Your name is {character_name}.
    Your character description is as follows: {character_description}.
    You will propose actions you plan to take and {storyteller_name} will explain what happens when you take those actions.
    Speak in the first person from the perspective of {character_name}.
    For describing your own body movements, wrap your description in '*'.
    Do not change roles!
    Do not speak from the perspective of anyone else.
    Remember you are {character_name}.
    Stop speaking the moment you finish speaking from your perspective.
    Never forget to keep your response to {word_limit} words!
    Do not add anything else.
    """
    ))
character_descriptions = [generate_character_description(character_name) for character_name in character_names]
character_system_messages = [generate_character_system_message(character_name, character_description) for character_name, character_description in zip(character_names, character_descriptions)]
storyteller_specifier_prompt = [
    player_descriptor_system_message,
    HumanMessage(content=
        f"""{game_description}
        Please reply with a creative description of the storyteller, {storyteller_name}, in {word_limit} words or less.
        Speak directly to {storyteller_name}.
        Do not add anything else."""
        )
]
storyteller_description = ChatOpenAI(temperature=1.0)(storyteller_specifier_prompt).content
storyteller_system_message = SystemMessage(content=(
f"""{game_description}
You are the storyteller, {storyteller_name}.
Your description is as follows: {storyteller_description}.


Your description is as follows: {storyteller_description}.
The other players will propose actions to take and you will explain what happens when they take those actions.
Speak in the first person from the perspective of {storyteller_name}.
Do not change roles!
Do not speak from the perspective of anyone else.
Remember you are the storyteller, {storyteller_name}.
Stop speaking the moment you finish speaking from your perspective.
Never forget to keep your response to {word_limit} words!
Do not add anything else.
"""
))
print('Storyteller Description:')
print(storyteller_description)
for character_name, character_description in zip(character_names, character_descriptions):
    print(f'{character_name} Description:')
    print(character_description)
Storyteller Description:
Dungeon Master, your vivid imagination conjures a world of wonder and danger. Will you lead our triumphant trio or be the ultimate foil to their quest to rid the world of Voldemort's horcruxes? The fate of both the muggle and wizarding worlds rests in your hands.
Harry Potter Description:
Harry Potter, the boy who lived, you hold the fate of the wizarding world in your hands. Your bravery and loyalty to your friends are unmatched. The burden you carry is heavy, but with the power of love by your side, you can overcome any obstacle. The hunt for the horcruxes begins now.
Ron Weasley Description:
Ron Weasley, you are Harry Potter's loyal and brave best friend. You have a great sense of humor and always bring joy to the team. Your skills with magic and strategy make you a valuable asset in the fight against Voldemort. Your love for food and your family keeps you grounded and motivated.
Hermione Granger Description:


Hermione Granger Description:
Hermione Granger, you are the brightest witch of your age. Your quick wit and vast knowledge are essential in our quest to find the horcruxes. Trust in your abilities and remember, knowledge is power.
Argus Filch Description:
Argus Filch, you are a bitter and cruel caretaker of the Hogwarts School of Witchcraft and Wizardry. Your harsh mannerisms and love for punishing the students know no bounds. Your loyalty to the Wizarding World and disdain for magic-wielders makes it surprising that you would join Harry, Ron, and Hermione in their quest to defeat Voldemort.
Use an LLM to create an elaborate quest description#
quest_specifier_prompt = [
    SystemMessage(content="You can make a task more specific."),
    HumanMessage(content=
        f"""{game_description}

        You are the storyteller, {storyteller_name}.
        Please make the quest more specific. Be creative and imaginative.
        Please reply with the specified quest in {word_limit} words or less.
        Speak directly to the characters: {*character_names,}.
        Do not add anything else."""
        )
]
specified_quest = ChatOpenAI(temperature=1.0)(quest_specifier_prompt).content
print(f"Original quest:\n{quest}\n")
print(f"Detailed quest:\n{specified_quest}\n")
Original quest:
Find all of Lord Voldemort's seven horcruxes.
Detailed quest:
You have discovered that one of Voldemort's horcruxes is hidden deep in the Forbidden Forest. You must navigate the dangerous terrain, avoid the creatures lurking within, and find the horcrux before the full moon rises, unleashing a pack of hungry werewolves. Remember, time is of the essence!
Main Loop#
characters = []


Main Loop#
characters = []
for character_name, character_system_message in zip(character_names, character_system_messages):
    characters.append(DialogueAgent(
        name=character_name,
        system_message=character_system_message,
        model=ChatOpenAI(temperature=0.2)))
storyteller = DialogueAgent(name=storyteller_name,
                     system_message=storyteller_system_message,
                     model=ChatOpenAI(temperature=0.2))
def select_next_speaker(step: int, agents: List[DialogueAgent]) -> int:
    """
    If the step is even, then select the storyteller
    Otherwise, select the other characters in a round-robin fashion.

    For example, with three characters with indices: 1 2 3
    The storyteller is index 0.
    Then the selected index will be as follows:
    step: 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16
    idx:  0  1  0  2  0  3  0  1  0  2  0  3  0  1  0  2  0
    """
    if step % 2 == 0:
        idx = 0
    else:
        idx = (step//2) % (len(agents)-1) + 1
    return idx
max_iters = 20
n = 0
simulator = DialogueSimulator(
    agents=[storyteller] + characters,
    selection_function=select_next_speaker
)
simulator.reset(storyteller_name, specified_quest)


)
simulator.reset(storyteller_name, specified_quest)
print(f"({storyteller_name}): {specified_quest}")
print('\n')
while n < max_iters:
    name, message = simulator.step()
    print(f"({name}): {message}")
    print('\n')
    n += 1
(Dungeon Master): You have discovered that one of Voldemort's horcruxes is hidden deep in the Forbidden Forest. You must navigate the dangerous terrain, avoid the creatures lurking within, and find the horcrux before the full moon rises, unleashing a pack of hungry werewolves. Remember, time is of the essence!
(Harry Potter): I take out my wand and cast a Lumos spell to light our way through the dark forest. We need to move quickly and quietly to avoid any unwanted attention from the creatures. Ron, Hermione, and I will lead the way while Argus Filch keeps watch behind us. Let's go!
(Dungeon Master): As you make your way through the forest, you hear the rustling of leaves and the snapping of twigs. Suddenly, a group of acromantulas, giant spiders, appear in front of you, blocking your path. What do you do?
(Ron Weasley): I quickly cast a spell to create a wall of fire between us and the acromantulas. Hopefully, the flames will deter them from attacking us. We need to keep moving forward and find that horcrux before it's too late.
(Dungeon Master): The acromantulas hiss and retreat from the wall of fire, allowing you to pass. As you continue deeper into the forest, you come across a clearing with a small pond. In the center of the pond, you see a glowing object. It must be the horcrux! But how do you get to it? What do you do?


(Hermione Granger): I take out my wand and cast a spell to conjure a small boat. We can use it to reach the center of the pond and retrieve the horcrux. But we need to be careful, there could be traps or other obstacles in our way. Ron, Harry, let's row the boat while Argus Filch keeps watch from the shore.
(Dungeon Master): As you row towards the center of the pond, you hear a loud hissing sound. Suddenly, a giant serpent emerges from the water, blocking your path. It looks angry and ready to attack. What do you do?
(Argus Filch): I take out my crossbow and aim it at the serpent. I may not be a wizard, but I know how to handle a weapon. I'll shoot it if it comes any closer. We can't let this serpent stop us from getting that horcrux.
(Dungeon Master): The serpent lunges towards the boat, but Argus Filch's crossbow bolt hits it in the head, causing it to retreat back into the water. You reach the center of the pond and retrieve the glowing object, which turns out to be a locket. Congratulations, you have found one of Voldemort's horcruxes! But there are still six more to find. What challenges will you face next?
(Harry Potter): We need to regroup and figure out our next move. We should head back to Hogwarts and consult with Professor Dumbledore's portrait. He may have some insight on where the other horcruxes could be hidden. We can't waste any time, Voldemort is getting stronger every day. Let's go!


(Dungeon Master): As you make your way back to Hogwarts, you hear a loud roar coming from the Forbidden Forest. It sounds like a werewolf. You must hurry before it catches up to you. You arrive at Dumbledore's office and he tells you that the next horcrux is hidden in a dangerous location. Are you ready for the next challenge?
(Ron Weasley): I'm always ready for a challenge! What's the location and what do we need to do to get there? We can't let Voldemort win, we have to find all of the horcruxes and destroy them. Let's do this!
(Dungeon Master): Dumbledore tells you that the next horcrux is hidden in the depths of Gringotts Bank. You must break into the bank, navigate its treacherous security measures, and find the horcrux before the goblins catch you. Are you ready to face the challenge of a lifetime? The fate of the wizarding world rests in your hands.
(Hermione Granger): I suggest we do some research on Gringotts Bank and its security measures before we attempt to break in. We need to be prepared and have a solid plan in place. We can also gather any necessary tools or potions that may help us along the way. Let's not rush into this blindly.
(Dungeon Master): As you research and plan your break-in to Gringotts Bank, you discover that the bank is heavily guarded by goblins, dragons, and other dangerous creatures. You'll need to be stealthy and quick to avoid detection. Are you ready to put your plan into action and face the dangers that await you? The clock is ticking, Voldemort's power grows stronger with each passing day.


(Argus Filch): I'll make sure to keep watch outside the bank while you all go in. I may not be able to help with the magic, but I can make sure no one interferes with our mission. We can't let anyone stop us from finding that horcrux and defeating Voldemort. Let's go!
(Dungeon Master): As you approach Gringotts Bank, you see the imposing structure looming before you. You sneak past the guards and make your way inside, navigating the twisting corridors and avoiding the traps set to catch intruders. Finally, you reach the vault where the horcrux is hidden. But it's guarded by a fierce dragon. What do you do?
(Harry Potter): I remember the time when I faced a dragon during the Triwizard Tournament. I take out my wand and cast a spell to distract the dragon while Ron and Hermione retrieve the horcrux. We need to work together and be quick. Time is running out and we can't afford to fail.
(Dungeon Master): The dragon roars and breathes fire, but Harry's spell distracts it long enough for Ron and Hermione to retrieve the horcrux. You make your way out of Gringotts Bank, but the goblins are hot on your trail. You must escape before they catch you. Congratulations, you have found another horcrux. But there are still five more to go. What challenges will you face next?
(Ron Weasley): We need to regroup and figure out our next move. We should consult with Professor Dumbledore's portrait again and see if he has any information on the next horcrux. We also need to be prepared for whatever challenges come our way. Voldemort won't make it easy for us, but we can't give up. Let's go!


(Dungeon Master): As you make your way back to Hogwarts, you hear a loud explosion coming from the direction of Hogsmeade. You arrive to find that Death Eaters have attacked the village and are wreaking havoc. You must fight off the Death Eaters and protect the innocent villagers. Are you ready to face this unexpected challenge and defend the wizarding world? The fate of both muggles and wizards rests in your hands.
 Contents

Import LangChain related modules
DialogueAgent class
DialogueSimulator class
Define roles and quest
Ask an LLM to add detail to the game description
Use an LLM to create an elaborate quest description
Main Loop
By Harrison Chase

      © Copyright 2023, Harrison Chase.

  Last updated on Apr 26, 2023.


.ipynb
.pdf
Two-Player Dungeons & Dragons
 Contents
Import LangChain related modules
DialogueAgent class
DialogueSimulator class
Define roles and quest
Ask an LLM to add detail to the game description
Protagonist and dungeon master system messages
Use an LLM to create an elaborate quest description
Main Loop
Two-Player Dungeons & Dragons#
In this notebook, we show how we can use concepts from CAMEL to simulate a role-playing game with a protagonist and a dungeon master. To simulate this game, we create an DialogueSimulator class that coordinates the dialogue between the two agents.
Import LangChain related modules#
from typing import List, Dict
from langchain.chat_models import ChatOpenAI
from langchain.schema import (
    AIMessage,
    HumanMessage,
    SystemMessage,
    BaseMessage,
)
DialogueAgent class#
The DialogueAgent class is a simple wrapper around the ChatOpenAI model that stores the message history from the dialogue_agent’s point of view by simply concatenating the messages as strings.
It exposes two methods:
send(): applies the chatmodel to the message history and returns the message string
receive(name, message): adds the message spoken by name to message history
class DialogueAgent():
    def __init__(
        self,
        name,
        system_message: SystemMessage,
        model: ChatOpenAI,
    ) -> None:
        self.name = name
        self.system_message = system_message
        self.model = model
        self.message_history = f"""Here is the conversation so far.
        """
        self.prefix = f'\n{self.name}:'

    def send(self) -> str:
        """
        Applies the chatmodel to the message history
        and returns the message string
        """
        message = self.model(


and returns the message string
        """
        message = self.model(
            [self.system_message,
             HumanMessage(content=self.message_history+self.prefix)])
        return message.content

    def receive(self, name: str, message: str) -> None:
        """
        Concatenates {message} spoken by {name} into message history
        """
        self.message_history += f'\n{name}: {message}'
DialogueSimulator class#
The DialogueSimulator class takes a list of agents. At each step, it performs the following:
Select the next speaker
Calls the next speaker to send a message
Broadcasts the message to all other agents
Update the step counter.
The selection of the next speaker can be implemented as any function, but in this case we simply loop through the agents.
class DialogueSimulator():

    def __init__(self, agents: List[DialogueAgent]):
        self.agents = agents
        self._step = 0

    def reset(self, name: str, message: str):
        """
        Initiates the conversation with a {message} from {name}
        """
        for agent in self.agents:
            agent.receive(name, message)

    def select_next_speaker(self, step: int) -> int:
        idx = (step + 1) % len(self.agents)
        return idx

    def step(self) -> tuple[str, str]:
        # 1. choose the next speaker
        speaker = self.agents[self.select_next_speaker(self._step)]

        # 2. next speaker sends message
        message = speaker.send()

        # 3. everyone receives message
        for receiver in self.agents:
            receiver.receive(speaker.name, message)


for receiver in self.agents:
            receiver.receive(speaker.name, message)

        # 4. increment time
        self._step += 1

        return speaker.name, message
Define roles and quest#
protagonist_name = "Harry Potter"
storyteller_name = "Dungeon Master"
quest = "Find all of Lord Voldemort's seven horcruxes."
word_limit = 50 # word limit for task brainstorming
Ask an LLM to add detail to the game description#
game_description = f"""Here is the topic for a Dungeons & Dragons game: {quest}.
        There is one player in this game: the protagonist, {protagonist_name}.
        The story is narrated by the storyteller, {storyteller_name}."""
player_descriptor_system_message = SystemMessage(
    content="You can add detail to the description of a Dungeons & Dragons player.")
protagonist_specifier_prompt = [
    player_descriptor_system_message,
    HumanMessage(content=
        f"""{game_description}
        Please reply with a creative description of the protagonist, {protagonist_name}, in {word_limit} words or less.
        Speak directly to {protagonist_name}.
        Do not add anything else."""
        )
]
protagonist_description = ChatOpenAI(temperature=1.0)(protagonist_specifier_prompt).content
storyteller_specifier_prompt = [
    player_descriptor_system_message,
    HumanMessage(content=
        f"""{game_description}
        Please reply with a creative description of the storyteller, {storyteller_name}, in {word_limit} words or less.
        Speak directly to {storyteller_name}.
        Do not add anything else."""
        )
]


Do not add anything else."""
        )
]
storyteller_description = ChatOpenAI(temperature=1.0)(storyteller_specifier_prompt).content
print('Protagonist Description:')
print(protagonist_description)
print('Storyteller Description:')
print(storyteller_description)
Protagonist Description:
Harry Potter, you are a brave and resourceful wizard. Your lightning scar and famous name precede you, but it is your heart that truly sets you apart. Your love and loyalty for your friends has been tested time and time again, and you have never faltered in your determination to vanquish evil.
Storyteller Description:
Dear Dungeon Master, you are a master of imagination, weaving enticing tales of adventure with a flick of your wrist. A patient guide, you lead Harry Potter through the perilous journey of finding Lord Voldemort's horcruxes, instilling excitement and wonder at every turn. Your storytelling prowess enchants all who dare to listen.
Protagonist and dungeon master system messages#
protagonist_system_message = SystemMessage(content=(
f"""{game_description}
Never forget you are the protagonist, {protagonist_name}, and I am the storyteller, {storyteller_name}.
Your character description is as follows: {protagonist_description}.
You will propose actions you plan to take and I will explain what happens when you take those actions.
Speak in the first person from the perspective of {protagonist_name}.
For describing your own body movements, wrap your description in '*'.
Do not change roles!
Do not speak from the perspective of {storyteller_name}.
Do not forget to finish speaking by saying, 'It is your turn, {storyteller_name}.'
Do not add anything else.
Remember you are the protagonist, {protagonist_name}.