Debugging | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionChainsMemoryAgentsCallbacksModulesGuidesEvaluationDebuggingDeploymentLangSmithModel ComparisonEcosystemAdditional resourcesGuidesDebuggingOn this pageDebuggingIf you're building with LLMs, at some point something will break, and you'll need to debug. A model call will fail, or the model output will be misformatted, or there will be some nested model calls and it won't be clear where along the way an incorrect output was created.Here's a few different tools and functionalities to aid in debugging.Tracingâ€‹Platforms with tracing capabilities like LangSmith and WandB are the most comprehensive solutions for debugging. These platforms make it easy to not only log and visualize LLM apps, but also to actively debug, test and refine them.For anyone building production-grade LLM applications, we highly recommend using a platform like this.langchain.debug and langchain.verboseâ€‹If you're prototyping in Jupyter Notebooks or running Python scripts, it can be helpful to print out the intermediate steps of a Chain run. There's a number of ways to enable printing at varying degrees of verbosity.Let's suppose we have a simple agent and want to visualize the actions it takes and tool outputs it receives. Without any debugging, here's what we see:from langchain.agents import AgentType, initialize_agent, load_toolsfrom langchain.chat_models import ChatOpenAIllm = ChatOpenAI(model_name="gpt-4", temperature=0)tools = load_tools(["ddg-search", "llm-math"], llm=llm)agent = initialize_agent(tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION)agent.run("Who directed the


llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION)agent.run("Who directed the 2023 film Oppenheimer and what is their age? What is their age in days (assume 365 days per year)?")    'The director of the 2023 film Oppenheimer is Christopher Nolan and he is approximately 19345 days old in 2023.'langchain.debug = Trueâ€‹Setting the global debug flag will cause all LangChain components with callback support (chains, models, agents, tools, retrievers) to print the inputs they receive and outputs they generate. This is the most verbose setting and will fully log raw inputs and outputs.import langchainlangchain.debug = Trueagent.run("Who directed the 2023 film Oppenheimer and what is their age? What is their age in days (assume 365 days per year)?")Console output     [chain/start] [1:RunTypeEnum.chain:AgentExecutor] Entering Chain run with input:    {      "input": "Who directed the 2023 film Oppenheimer and what is their age? What is their age in days (assume 365 days per year)?"    }    [chain/start] [1:RunTypeEnum.chain:AgentExecutor > 2:RunTypeEnum.chain:LLMChain] Entering Chain run with input:    {      "input": "Who directed the 2023 film Oppenheimer and what is their age? What is their age in days (assume 365 days per year)?",      "agent_scratchpad": "",      "stop": [        "\nObservation:",        "\n\tObservation:"      ]    }    [llm/start]


]    }    [llm/start] [1:RunTypeEnum.chain:AgentExecutor > 2:RunTypeEnum.chain:LLMChain > 3:RunTypeEnum.llm:ChatOpenAI] Entering LLM run with input:    {      "prompts": [        "Human: Answer the following questions as best you can. You have access to the following tools:\n\nduckduckgo_search: A wrapper around DuckDuckGo Search. Useful for when you need to answer questions about current events. Input should be a search query.\nCalculator: Useful for when you need to answer questions about math.\n\nUse the following format:\n\nQuestion: the input question you must answer\nThought: you should always think about what to do\nAction: the action to take, should be one of [duckduckgo_search, Calculator]\nAction Input: the input to the action\nObservation: the result of the action\n... (this Thought/Action/Action Input/Observation can repeat N times)\nThought: I now know the final answer\nFinal Answer: the final answer to the original input question\n\nBegin!\n\nQuestion: Who directed the 2023 film Oppenheimer and what is their age? What is their age in days (assume 365 days per year)?\nThought:"      ]    }    [llm/end] [1:RunTypeEnum.chain:AgentExecutor > 2:RunTypeEnum.chain:LLMChain > 3:RunTypeEnum.llm:ChatOpenAI] [5.53s] Exiting LLM run with output:    {      "generations": [        [          {


[          {            "text": "I need to find out who directed the 2023 film Oppenheimer and their age. Then, I need to calculate their age in days. I will use DuckDuckGo to find out the director and their age.\nAction: duckduckgo_search\nAction Input: \"Director of the 2023 film Oppenheimer and their age\"",            "generation_info": {              "finish_reason": "stop"            },            "message": {              "lc": 1,              "type": "constructor",              "id": [                "langchain",                "schema",                "messages",                "AIMessage"              ],              "kwargs": {                "content": "I need to find out who directed the 2023 film Oppenheimer and their age. Then, I need to calculate their age in days. I will use DuckDuckGo to find out the director and their age.\nAction: duckduckgo_search\nAction Input:


to find out the director and their age.\nAction: duckduckgo_search\nAction Input: \"Director of the 2023 film Oppenheimer and their age\"",                "additional_kwargs": {}              }            }          }        ]      ],      "llm_output": {        "token_usage": {          "prompt_tokens": 206,          "completion_tokens": 71,          "total_tokens": 277        },        "model_name": "gpt-4"      },      "run": null    }    [chain/end] [1:RunTypeEnum.chain:AgentExecutor > 2:RunTypeEnum.chain:LLMChain] [5.53s] Exiting Chain run with output:    {      "text": "I need to find out who directed the 2023 film Oppenheimer and their age. Then, I need to calculate their age in days. I will use DuckDuckGo to find out the director and their age.\nAction: duckduckgo_search\nAction Input: \"Director of the 2023 film Oppenheimer and their age\""    }    [tool/start] [1:RunTypeEnum.chain:AgentExecutor > 4:RunTypeEnum.tool:duckduckgo_search] Entering Tool run with input:    "Director of the 2023 film Oppenheimer and their


run with input:    "Director of the 2023 film Oppenheimer and their age"    [tool/end] [1:RunTypeEnum.chain:AgentExecutor > 4:RunTypeEnum.tool:duckduckgo_search] [1.51s] Exiting Tool run with output:    "Capturing the mad scramble to build the first atomic bomb required rapid-fire filming, strict set rules and the construction of an entire 1940s western town. By Jada Yuan. July 19, 2023 at 5:00 a ... In Christopher Nolan's new film, "Oppenheimer," Cillian Murphy stars as J. Robert Oppenheimer, the American physicist who oversaw the Manhattan Project in Los Alamos, N.M. Universal Pictures... Oppenheimer: Directed by Christopher Nolan. With Cillian Murphy, Emily Blunt, Robert Downey Jr., Alden Ehrenreich. The story of American scientist J. Robert Oppenheimer and his role in the development of the atomic bomb. Christopher Nolan goes deep on 'Oppenheimer,' his most 'extreme' film to date. By Kenneth Turan. July 11, 2023 5 AM PT. For Subscribers. Christopher Nolan is photographed in Los Angeles ... Oppenheimer is a 2023 epic biographical thriller film written and directed by Christopher Nolan.It is based on the 2005 biography American Prometheus by Kai Bird and Martin J. Sherwin about J. Robert Oppenheimer, a theoretical physicist who was pivotal in developing the first nuclear weapons as part of the Manhattan Project and thereby ushering in the Atomic Age."    [chain/start] [1:RunTypeEnum.chain:AgentExecutor > 5:RunTypeEnum.chain:LLMChain] Entering Chain run with input:    {      "input": "Who directed the 2023 film Oppenheimer and what is their age?


"input": "Who directed the 2023 film Oppenheimer and what is their age? What is their age in days (assume 365 days per year)?",      "agent_scratchpad": "I need to find out who directed the 2023 film Oppenheimer and their age. Then, I need to calculate their age in days. I will use DuckDuckGo to find out the director and their age.\nAction: duckduckgo_search\nAction Input: \"Director of the 2023 film Oppenheimer and their age\"\nObservation: Capturing the mad scramble to build the first atomic bomb required rapid-fire filming, strict set rules and the construction of an entire 1940s western town. By Jada Yuan. July 19, 2023 at 5:00 a ... In Christopher Nolan's new film, \"Oppenheimer,\" Cillian Murphy stars as J. Robert Oppenheimer, the American physicist who oversaw the Manhattan Project in Los Alamos, N.M. Universal Pictures... Oppenheimer: Directed by Christopher Nolan. With Cillian Murphy, Emily Blunt, Robert Downey Jr., Alden Ehrenreich. The story of American scientist J. Robert Oppenheimer and his role in the development of the atomic bomb. Christopher Nolan goes deep on 'Oppenheimer,' his most 'extreme' film to date. By Kenneth Turan. July 11, 2023 5 AM PT. For Subscribers. Christopher Nolan is photographed in Los Angeles ... Oppenheimer is a 2023 epic biographical thriller film written and directed by Christopher Nolan.It is based on the 2005 biography American Prometheus by Kai Bird and Martin J. Sherwin about J. Robert Oppenheimer, a theoretical physicist who was pivotal in developing the first nuclear weapons as part of the Manhattan Project and thereby ushering in the Atomic Age.\nThought:",


the Manhattan Project and thereby ushering in the Atomic Age.\nThought:",      "stop": [        "\nObservation:",        "\n\tObservation:"      ]    }    [llm/start] [1:RunTypeEnum.chain:AgentExecutor > 5:RunTypeEnum.chain:LLMChain > 6:RunTypeEnum.llm:ChatOpenAI] Entering LLM run with input:    {      "prompts": [        "Human: Answer the following questions as best you can. You have access to the following tools:\n\nduckduckgo_search: A wrapper around DuckDuckGo Search. Useful for when you need to answer questions about current events. Input should be a search query.\nCalculator: Useful for when you need to answer questions about math.\n\nUse the following format:\n\nQuestion: the input question you must answer\nThought: you should always think about what to do\nAction: the action to take, should be one of [duckduckgo_search, Calculator]\nAction Input: the input to the action\nObservation: the result of the action\n... (this Thought/Action/Action Input/Observation can repeat N times)\nThought: I now know the final answer\nFinal Answer: the final answer to the original input question\n\nBegin!\n\nQuestion: Who directed the 2023 film Oppenheimer and what is their age? What is their age in days (assume 365 days per year)?\nThought:I need to find out who directed the 2023 film Oppenheimer and their age. Then, I need to calculate their age in days. I will use DuckDuckGo to find out the director and their age.\nAction: duckduckgo_search\nAction


to find out the director and their age.\nAction: duckduckgo_search\nAction Input: \"Director of the 2023 film Oppenheimer and their age\"\nObservation: Capturing the mad scramble to build the first atomic bomb required rapid-fire filming, strict set rules and the construction of an entire 1940s western town. By Jada Yuan. July 19, 2023 at 5:00 a ... In Christopher Nolan's new film, \"Oppenheimer,\" Cillian Murphy stars as J. Robert Oppenheimer, the American physicist who oversaw the Manhattan Project in Los Alamos, N.M. Universal Pictures... Oppenheimer: Directed by Christopher Nolan. With Cillian Murphy, Emily Blunt, Robert Downey Jr., Alden Ehrenreich. The story of American scientist J. Robert Oppenheimer and his role in the development of the atomic bomb. Christopher Nolan goes deep on 'Oppenheimer,' his most 'extreme' film to date. By Kenneth Turan. July 11, 2023 5 AM PT. For Subscribers. Christopher Nolan is photographed in Los Angeles ... Oppenheimer is a 2023 epic biographical thriller film written and directed by Christopher Nolan.It is based on the 2005 biography American Prometheus by Kai Bird and Martin J. Sherwin about J. Robert Oppenheimer, a theoretical physicist who was pivotal in developing the first nuclear weapons as part of the Manhattan Project and thereby ushering in the Atomic Age.\nThought:"      ]    }    [llm/end] [1:RunTypeEnum.chain:AgentExecutor > 5:RunTypeEnum.chain:LLMChain > 6:RunTypeEnum.llm:ChatOpenAI] [4.46s] Exiting LLM run with output:    {      "generations": [        [


"generations": [        [          {            "text": "The director of the 2023 film Oppenheimer is Christopher Nolan. Now I need to find out his age.\nAction: duckduckgo_search\nAction Input: \"Christopher Nolan age\"",            "generation_info": {              "finish_reason": "stop"            },            "message": {              "lc": 1,              "type": "constructor",              "id": [                "langchain",                "schema",                "messages",                "AIMessage"              ],              "kwargs": {                "content": "The director of the 2023 film Oppenheimer is Christopher Nolan. Now I need to find out his age.\nAction: duckduckgo_search\nAction Input: \"Christopher Nolan age\"",                "additional_kwargs": {}              }


{}              }            }          }        ]      ],      "llm_output": {        "token_usage": {          "prompt_tokens": 550,          "completion_tokens": 39,          "total_tokens": 589        },        "model_name": "gpt-4"      },      "run": null    }    [chain/end] [1:RunTypeEnum.chain:AgentExecutor > 5:RunTypeEnum.chain:LLMChain] [4.46s] Exiting Chain run with output:    {      "text": "The director of the 2023 film Oppenheimer is Christopher Nolan. Now I need to find out his age.\nAction: duckduckgo_search\nAction Input: \"Christopher Nolan age\""    }    [tool/start] [1:RunTypeEnum.chain:AgentExecutor > 7:RunTypeEnum.tool:duckduckgo_search] Entering Tool run with input:    "Christopher Nolan age"    [tool/end] [1:RunTypeEnum.chain:AgentExecutor > 7:RunTypeEnum.tool:duckduckgo_search] [1.33s] Exiting Tool run with output:    "Christopher Edward Nolan CBE (born 30 July 1970) is a British and American filmmaker. Known for his Hollywood blockbusters with complex storytelling, Nolan is considered a leading filmmaker of the 21st century. His films have grossed


storytelling, Nolan is considered a leading filmmaker of the 21st century. His films have grossed $5 billion worldwide. The recipient of many accolades, he has been nominated for five Academy Awards, five BAFTA Awards and six Golden Globe Awards. July 30, 1970 (age 52) London England Notable Works: "Dunkirk" "Tenet" "The Prestige" See all related content â†’ Recent News Jul. 13, 2023, 11:11 AM ET (AP) Cillian Murphy, playing Oppenheimer, finally gets to lead a Christopher Nolan film July 11, 2023 5 AM PT For Subscribers Christopher Nolan is photographed in Los Angeles. (Joe Pugliese / For The Times) This is not the story I was supposed to write. Oppenheimer director Christopher Nolan, Cillian Murphy, Emily Blunt and Matt Damon on the stakes of making a three-hour, CGI-free summer film. Christopher Nolan, the director behind such films as "Dunkirk," "Inception," "Interstellar," and the "Dark Knight" trilogy, has spent the last three years living in Oppenheimer's world, writing ..."    [chain/start] [1:RunTypeEnum.chain:AgentExecutor > 8:RunTypeEnum.chain:LLMChain] Entering Chain run with input:    {      "input": "Who directed the 2023 film Oppenheimer and what is their age? What is their age in days (assume 365 days per year)?",      "agent_scratchpad": "I need to find out who directed the 2023 film Oppenheimer and their age. Then, I need to calculate their age in days. I will use DuckDuckGo to find out the director and their age.\nAction: duckduckgo_search\nAction Input: \"Director of the


director and their age.\nAction: duckduckgo_search\nAction Input: \"Director of the 2023 film Oppenheimer and their age\"\nObservation: Capturing the mad scramble to build the first atomic bomb required rapid-fire filming, strict set rules and the construction of an entire 1940s western town. By Jada Yuan. July 19, 2023 at 5:00 a ... In Christopher Nolan's new film, \"Oppenheimer,\" Cillian Murphy stars as J. Robert Oppenheimer, the American physicist who oversaw the Manhattan Project in Los Alamos, N.M. Universal Pictures... Oppenheimer: Directed by Christopher Nolan. With Cillian Murphy, Emily Blunt, Robert Downey Jr., Alden Ehrenreich. The story of American scientist J. Robert Oppenheimer and his role in the development of the atomic bomb. Christopher Nolan goes deep on 'Oppenheimer,' his most 'extreme' film to date. By Kenneth Turan. July 11, 2023 5 AM PT. For Subscribers. Christopher Nolan is photographed in Los Angeles ... Oppenheimer is a 2023 epic biographical thriller film written and directed by Christopher Nolan.It is based on the 2005 biography American Prometheus by Kai Bird and Martin J. Sherwin about J. Robert Oppenheimer, a theoretical physicist who was pivotal in developing the first nuclear weapons as part of the Manhattan Project and thereby ushering in the Atomic Age.\nThought:The director of the 2023 film Oppenheimer is Christopher Nolan. Now I need to find out his age.\nAction: duckduckgo_search\nAction Input: \"Christopher Nolan age\"\nObservation: Christopher Edward Nolan CBE (born 30 July 1970) is a British and American filmmaker. Known for his Hollywood blockbusters with complex storytelling, Nolan is considered a leading filmmaker of the 21st century. His films have grossed


storytelling, Nolan is considered a leading filmmaker of the 21st century. His films have grossed $5 billion worldwide. The recipient of many accolades, he has been nominated for five Academy Awards, five BAFTA Awards and six Golden Globe Awards. July 30, 1970 (age 52) London England Notable Works: \"Dunkirk\" \"Tenet\" \"The Prestige\" See all related content â†’ Recent News Jul. 13, 2023, 11:11 AM ET (AP) Cillian Murphy, playing Oppenheimer, finally gets to lead a Christopher Nolan film July 11, 2023 5 AM PT For Subscribers Christopher Nolan is photographed in Los Angeles. (Joe Pugliese / For The Times) This is not the story I was supposed to write. Oppenheimer director Christopher Nolan, Cillian Murphy, Emily Blunt and Matt Damon on the stakes of making a three-hour, CGI-free summer film. Christopher Nolan, the director behind such films as \"Dunkirk,\" \"Inception,\" \"Interstellar,\" and the \"Dark Knight\" trilogy, has spent the last three years living in Oppenheimer's world, writing ...\nThought:",      "stop": [        "\nObservation:",        "\n\tObservation:"      ]    }    [llm/start] [1:RunTypeEnum.chain:AgentExecutor > 8:RunTypeEnum.chain:LLMChain > 9:RunTypeEnum.llm:ChatOpenAI] Entering LLM run with input:    {      "prompts": [        "Human: Answer the following questions as best you can. You have access to the following tools:\n\nduckduckgo_search: A wrapper around DuckDuckGo


to the following tools:\n\nduckduckgo_search: A wrapper around DuckDuckGo Search. Useful for when you need to answer questions about current events. Input should be a search query.\nCalculator: Useful for when you need to answer questions about math.\n\nUse the following format:\n\nQuestion: the input question you must answer\nThought: you should always think about what to do\nAction: the action to take, should be one of [duckduckgo_search, Calculator]\nAction Input: the input to the action\nObservation: the result of the action\n... (this Thought/Action/Action Input/Observation can repeat N times)\nThought: I now know the final answer\nFinal Answer: the final answer to the original input question\n\nBegin!\n\nQuestion: Who directed the 2023 film Oppenheimer and what is their age? What is their age in days (assume 365 days per year)?\nThought:I need to find out who directed the 2023 film Oppenheimer and their age. Then, I need to calculate their age in days. I will use DuckDuckGo to find out the director and their age.\nAction: duckduckgo_search\nAction Input: \"Director of the 2023 film Oppenheimer and their age\"\nObservation: Capturing the mad scramble to build the first atomic bomb required rapid-fire filming, strict set rules and the construction of an entire 1940s western town. By Jada Yuan. July 19, 2023 at 5:00 a ... In Christopher Nolan's new film, \"Oppenheimer,\" Cillian Murphy stars as J. Robert Oppenheimer, the American physicist who oversaw the Manhattan Project in Los Alamos, N.M. Universal Pictures... Oppenheimer: Directed by Christopher Nolan. With Cillian Murphy, Emily Blunt, Robert Downey Jr., Alden


by Christopher Nolan. With Cillian Murphy, Emily Blunt, Robert Downey Jr., Alden Ehrenreich. The story of American scientist J. Robert Oppenheimer and his role in the development of the atomic bomb. Christopher Nolan goes deep on 'Oppenheimer,' his most 'extreme' film to date. By Kenneth Turan. July 11, 2023 5 AM PT. For Subscribers. Christopher Nolan is photographed in Los Angeles ... Oppenheimer is a 2023 epic biographical thriller film written and directed by Christopher Nolan.It is based on the 2005 biography American Prometheus by Kai Bird and Martin J. Sherwin about J. Robert Oppenheimer, a theoretical physicist who was pivotal in developing the first nuclear weapons as part of the Manhattan Project and thereby ushering in the Atomic Age.\nThought:The director of the 2023 film Oppenheimer is Christopher Nolan. Now I need to find out his age.\nAction: duckduckgo_search\nAction Input: \"Christopher Nolan age\"\nObservation: Christopher Edward Nolan CBE (born 30 July 1970) is a British and American filmmaker. Known for his Hollywood blockbusters with complex storytelling, Nolan is considered a leading filmmaker of the 21st century. His films have grossed $5 billion worldwide. The recipient of many accolades, he has been nominated for five Academy Awards, five BAFTA Awards and six Golden Globe Awards. July 30, 1970 (age 52) London England Notable Works: \"Dunkirk\" \"Tenet\" \"The Prestige\" See all related content â†’ Recent News Jul. 13, 2023, 11:11 AM ET (AP) Cillian Murphy, playing Oppenheimer, finally gets to lead a Christopher Nolan film July 11, 2023 5 AM PT For Subscribers Christopher Nolan is photographed in Los Angeles. (Joe Pugliese /


AM PT For Subscribers Christopher Nolan is photographed in Los Angeles. (Joe Pugliese / For The Times) This is not the story I was supposed to write. Oppenheimer director Christopher Nolan, Cillian Murphy, Emily Blunt and Matt Damon on the stakes of making a three-hour, CGI-free summer film. Christopher Nolan, the director behind such films as \"Dunkirk,\" \"Inception,\" \"Interstellar,\" and the \"Dark Knight\" trilogy, has spent the last three years living in Oppenheimer's world, writing ...\nThought:"      ]    }    [llm/end] [1:RunTypeEnum.chain:AgentExecutor > 8:RunTypeEnum.chain:LLMChain > 9:RunTypeEnum.llm:ChatOpenAI] [2.69s] Exiting LLM run with output:    {      "generations": [        [          {            "text": "Christopher Nolan was born on July 30, 1970, which makes him 52 years old in 2023. Now I need to calculate his age in days.\nAction: Calculator\nAction Input: 52*365",            "generation_info": {              "finish_reason": "stop"            },            "message": {              "lc": 1,              "type": "constructor",              "id": [


"id": [                "langchain",                "schema",                "messages",                "AIMessage"              ],              "kwargs": {                "content": "Christopher Nolan was born on July 30, 1970, which makes him 52 years old in 2023. Now I need to calculate his age in days.\nAction: Calculator\nAction Input: 52*365",                "additional_kwargs": {}              }            }          }        ]      ],      "llm_output": {        "token_usage": {          "prompt_tokens": 868,          "completion_tokens": 46,          "total_tokens": 914        },        "model_name": "gpt-4"      },      "run": null    }    [chain/end] [1:RunTypeEnum.chain:AgentExecutor > 8:RunTypeEnum.chain:LLMChain]


> 8:RunTypeEnum.chain:LLMChain] [2.69s] Exiting Chain run with output:    {      "text": "Christopher Nolan was born on July 30, 1970, which makes him 52 years old in 2023. Now I need to calculate his age in days.\nAction: Calculator\nAction Input: 52*365"    }    [tool/start] [1:RunTypeEnum.chain:AgentExecutor > 10:RunTypeEnum.tool:Calculator] Entering Tool run with input:    "52*365"    [chain/start] [1:RunTypeEnum.chain:AgentExecutor > 10:RunTypeEnum.tool:Calculator > 11:RunTypeEnum.chain:LLMMathChain] Entering Chain run with input:    {      "question": "52*365"    }    [chain/start] [1:RunTypeEnum.chain:AgentExecutor > 10:RunTypeEnum.tool:Calculator > 11:RunTypeEnum.chain:LLMMathChain > 12:RunTypeEnum.chain:LLMChain] Entering Chain run with input:    {      "question": "52*365",      "stop": [        "```output"      ]    }    [llm/start] [1:RunTypeEnum.chain:AgentExecutor > 10:RunTypeEnum.tool:Calculator > 11:RunTypeEnum.chain:LLMMathChain > 12:RunTypeEnum.chain:LLMChain > 13:RunTypeEnum.llm:ChatOpenAI] Entering LLM run with input:    {      "prompts": [        "Human: Translate a math problem into a expression


[        "Human: Translate a math problem into a expression that can be executed using Python's numexpr library. Use the output of running this code to answer the question.\n\nQuestion: ${Question with math problem.}\n```text\n${single line mathematical expression that solves the problem}\n```\n...numexpr.evaluate(text)...\n```output\n${Output of running the code}\n```\nAnswer: ${Answer}\n\nBegin.\n\nQuestion: What is 37593 * 67?\n```text\n37593 * 67\n```\n...numexpr.evaluate(\"37593 * 67\")...\n```output\n2518731\n```\nAnswer: 2518731\n\nQuestion: 37593^(1/5)\n```text\n37593**(1/5)\n```\n...numexpr.evaluate(\"37593**(1/5)\")...\n```output\n8.222831614237718\n```\nAnswer: 8.222831614237718\n\nQuestion: 52*365"      ]    }    [llm/end] [1:RunTypeEnum.chain:AgentExecutor > 10:RunTypeEnum.tool:Calculator > 11:RunTypeEnum.chain:LLMMathChain > 12:RunTypeEnum.chain:LLMChain > 13:RunTypeEnum.llm:ChatOpenAI] [2.89s] Exiting LLM run with output:    {      "generations": [        [          {            "text": "```text\n52*365\n```\n...numexpr.evaluate(\"52*365\")...\n",            "generation_info":


"generation_info": {              "finish_reason": "stop"            },            "message": {              "lc": 1,              "type": "constructor",              "id": [                "langchain",                "schema",                "messages",                "AIMessage"              ],              "kwargs": {                "content": "```text\n52*365\n```\n...numexpr.evaluate(\"52*365\")...\n",                "additional_kwargs": {}              }            }          }        ]      ],      "llm_output": {        "token_usage": {          "prompt_tokens": 203,          "completion_tokens": 19,


"completion_tokens": 19,          "total_tokens": 222        },        "model_name": "gpt-4"      },      "run": null    }    [chain/end] [1:RunTypeEnum.chain:AgentExecutor > 10:RunTypeEnum.tool:Calculator > 11:RunTypeEnum.chain:LLMMathChain > 12:RunTypeEnum.chain:LLMChain] [2.89s] Exiting Chain run with output:    {      "text": "```text\n52*365\n```\n...numexpr.evaluate(\"52*365\")...\n"    }    [chain/end] [1:RunTypeEnum.chain:AgentExecutor > 10:RunTypeEnum.tool:Calculator > 11:RunTypeEnum.chain:LLMMathChain] [2.90s] Exiting Chain run with output:    {      "answer": "Answer: 18980"    }    [tool/end] [1:RunTypeEnum.chain:AgentExecutor > 10:RunTypeEnum.tool:Calculator] [2.90s] Exiting Tool run with output:    "Answer: 18980"    [chain/start] [1:RunTypeEnum.chain:AgentExecutor > 14:RunTypeEnum.chain:LLMChain] Entering Chain run with input:    {      "input": "Who directed the 2023 film Oppenheimer and what is their age? What is their age in days (assume 365 days per year)?",      "agent_scratchpad": "I need to find out who directed the 2023 film Oppenheimer and their


"I need to find out who directed the 2023 film Oppenheimer and their age. Then, I need to calculate their age in days. I will use DuckDuckGo to find out the director and their age.\nAction: duckduckgo_search\nAction Input: \"Director of the 2023 film Oppenheimer and their age\"\nObservation: Capturing the mad scramble to build the first atomic bomb required rapid-fire filming, strict set rules and the construction of an entire 1940s western town. By Jada Yuan. July 19, 2023 at 5:00 a ... In Christopher Nolan's new film, \"Oppenheimer,\" Cillian Murphy stars as J. Robert Oppenheimer, the American physicist who oversaw the Manhattan Project in Los Alamos, N.M. Universal Pictures... Oppenheimer: Directed by Christopher Nolan. With Cillian Murphy, Emily Blunt, Robert Downey Jr., Alden Ehrenreich. The story of American scientist J. Robert Oppenheimer and his role in the development of the atomic bomb. Christopher Nolan goes deep on 'Oppenheimer,' his most 'extreme' film to date. By Kenneth Turan. July 11, 2023 5 AM PT. For Subscribers. Christopher Nolan is photographed in Los Angeles ... Oppenheimer is a 2023 epic biographical thriller film written and directed by Christopher Nolan.It is based on the 2005 biography American Prometheus by Kai Bird and Martin J. Sherwin about J. Robert Oppenheimer, a theoretical physicist who was pivotal in developing the first nuclear weapons as part of the Manhattan Project and thereby ushering in the Atomic Age.\nThought:The director of the 2023 film Oppenheimer is Christopher Nolan. Now I need to find out his age.\nAction: duckduckgo_search\nAction Input: \"Christopher Nolan age\"\nObservation: Christopher Edward Nolan CBE (born


Input: \"Christopher Nolan age\"\nObservation: Christopher Edward Nolan CBE (born 30 July 1970) is a British and American filmmaker. Known for his Hollywood blockbusters with complex storytelling, Nolan is considered a leading filmmaker of the 21st century. His films have grossed $5 billion worldwide. The recipient of many accolades, he has been nominated for five Academy Awards, five BAFTA Awards and six Golden Globe Awards. July 30, 1970 (age 52) London England Notable Works: \"Dunkirk\" \"Tenet\" \"The Prestige\" See all related content â†’ Recent News Jul. 13, 2023, 11:11 AM ET (AP) Cillian Murphy, playing Oppenheimer, finally gets to lead a Christopher Nolan film July 11, 2023 5 AM PT For Subscribers Christopher Nolan is photographed in Los Angeles. (Joe Pugliese / For The Times) This is not the story I was supposed to write. Oppenheimer director Christopher Nolan, Cillian Murphy, Emily Blunt and Matt Damon on the stakes of making a three-hour, CGI-free summer film. Christopher Nolan, the director behind such films as \"Dunkirk,\" \"Inception,\" \"Interstellar,\" and the \"Dark Knight\" trilogy, has spent the last three years living in Oppenheimer's world, writing ...\nThought:Christopher Nolan was born on July 30, 1970, which makes him 52 years old in 2023. Now I need to calculate his age in days.\nAction: Calculator\nAction Input: 52*365\nObservation: Answer: 18980\nThought:",      "stop": [        "\nObservation:",        "\n\tObservation:"      ]    }


"\n\tObservation:"      ]    }    [llm/start] [1:RunTypeEnum.chain:AgentExecutor > 14:RunTypeEnum.chain:LLMChain > 15:RunTypeEnum.llm:ChatOpenAI] Entering LLM run with input:    {      "prompts": [        "Human: Answer the following questions as best you can. You have access to the following tools:\n\nduckduckgo_search: A wrapper around DuckDuckGo Search. Useful for when you need to answer questions about current events. Input should be a search query.\nCalculator: Useful for when you need to answer questions about math.\n\nUse the following format:\n\nQuestion: the input question you must answer\nThought: you should always think about what to do\nAction: the action to take, should be one of [duckduckgo_search, Calculator]\nAction Input: the input to the action\nObservation: the result of the action\n... (this Thought/Action/Action Input/Observation can repeat N times)\nThought: I now know the final answer\nFinal Answer: the final answer to the original input question\n\nBegin!\n\nQuestion: Who directed the 2023 film Oppenheimer and what is their age? What is their age in days (assume 365 days per year)?\nThought:I need to find out who directed the 2023 film Oppenheimer and their age. Then, I need to calculate their age in days. I will use DuckDuckGo to find out the director and their age.\nAction: duckduckgo_search\nAction Input: \"Director of the 2023 film Oppenheimer and their age\"\nObservation: Capturing the mad scramble to build the first atomic bomb required rapid-fire filming, strict set rules and the


the mad scramble to build the first atomic bomb required rapid-fire filming, strict set rules and the construction of an entire 1940s western town. By Jada Yuan. July 19, 2023 at 5:00 a ... In Christopher Nolan's new film, \"Oppenheimer,\" Cillian Murphy stars as J. Robert Oppenheimer, the American physicist who oversaw the Manhattan Project in Los Alamos, N.M. Universal Pictures... Oppenheimer: Directed by Christopher Nolan. With Cillian Murphy, Emily Blunt, Robert Downey Jr., Alden Ehrenreich. The story of American scientist J. Robert Oppenheimer and his role in the development of the atomic bomb. Christopher Nolan goes deep on 'Oppenheimer,' his most 'extreme' film to date. By Kenneth Turan. July 11, 2023 5 AM PT. For Subscribers. Christopher Nolan is photographed in Los Angeles ... Oppenheimer is a 2023 epic biographical thriller film written and directed by Christopher Nolan.It is based on the 2005 biography American Prometheus by Kai Bird and Martin J. Sherwin about J. Robert Oppenheimer, a theoretical physicist who was pivotal in developing the first nuclear weapons as part of the Manhattan Project and thereby ushering in the Atomic Age.\nThought:The director of the 2023 film Oppenheimer is Christopher Nolan. Now I need to find out his age.\nAction: duckduckgo_search\nAction Input: \"Christopher Nolan age\"\nObservation: Christopher Edward Nolan CBE (born 30 July 1970) is a British and American filmmaker. Known for his Hollywood blockbusters with complex storytelling, Nolan is considered a leading filmmaker of the 21st century. His films have grossed $5 billion worldwide. The recipient of many accolades, he has been nominated for five Academy Awards, five BAFTA Awards and six Golden Globe Awards. July 30, 1970


Academy Awards, five BAFTA Awards and six Golden Globe Awards. July 30, 1970 (age 52) London England Notable Works: \"Dunkirk\" \"Tenet\" \"The Prestige\" See all related content â†’ Recent News Jul. 13, 2023, 11:11 AM ET (AP) Cillian Murphy, playing Oppenheimer, finally gets to lead a Christopher Nolan film July 11, 2023 5 AM PT For Subscribers Christopher Nolan is photographed in Los Angeles. (Joe Pugliese / For The Times) This is not the story I was supposed to write. Oppenheimer director Christopher Nolan, Cillian Murphy, Emily Blunt and Matt Damon on the stakes of making a three-hour, CGI-free summer film. Christopher Nolan, the director behind such films as \"Dunkirk,\" \"Inception,\" \"Interstellar,\" and the \"Dark Knight\" trilogy, has spent the last three years living in Oppenheimer's world, writing ...\nThought:Christopher Nolan was born on July 30, 1970, which makes him 52 years old in 2023. Now I need to calculate his age in days.\nAction: Calculator\nAction Input: 52*365\nObservation: Answer: 18980\nThought:"      ]    }    [llm/end] [1:RunTypeEnum.chain:AgentExecutor > 14:RunTypeEnum.chain:LLMChain > 15:RunTypeEnum.llm:ChatOpenAI] [3.52s] Exiting LLM run with output:    {      "generations": [        [          {            "text": "I now know the final answer\nFinal Answer: The director of the 2023


"I now know the final answer\nFinal Answer: The director of the 2023 film Oppenheimer is Christopher Nolan and he is 52 years old. His age in days is approximately 18980 days.",            "generation_info": {              "finish_reason": "stop"            },            "message": {              "lc": 1,              "type": "constructor",              "id": [                "langchain",                "schema",                "messages",                "AIMessage"              ],              "kwargs": {                "content": "I now know the final answer\nFinal Answer: The director of the 2023 film Oppenheimer is Christopher Nolan and he is 52 years old. His age in days is approximately 18980 days.",                "additional_kwargs": {}              }            }          }        ]      ],


}        ]      ],      "llm_output": {        "token_usage": {          "prompt_tokens": 926,          "completion_tokens": 43,          "total_tokens": 969        },        "model_name": "gpt-4"      },      "run": null    }    [chain/end] [1:RunTypeEnum.chain:AgentExecutor > 14:RunTypeEnum.chain:LLMChain] [3.52s] Exiting Chain run with output:    {      "text": "I now know the final answer\nFinal Answer: The director of the 2023 film Oppenheimer is Christopher Nolan and he is 52 years old. His age in days is approximately 18980 days."    }    [chain/end] [1:RunTypeEnum.chain:AgentExecutor] [21.96s] Exiting Chain run with output:    {      "output": "The director of the 2023 film Oppenheimer is Christopher Nolan and he is 52 years old. His age in days is approximately 18980 days."    }    'The director of the 2023 film Oppenheimer is Christopher Nolan and he is 52 years old. His age in days is approximately 18980 days.'langchain.verbose = Trueâ€‹Setting the verbose flag will print out inputs and outputs in a slightly more readable format and will skip logging certain raw outputs (like the token usage stats for an LLM call) so that you can focus on application


outputs (like the token usage stats for an LLM call) so that you can focus on application logic.import langchainlangchain.verbose = Trueagent.run("Who directed the 2023 film Oppenheimer and what is their age? What is their age in days (assume 365 days per year)?")Console output             > Entering new AgentExecutor chain...            > Entering new LLMChain chain...    Prompt after formatting:    Answer the following questions as best you can. You have access to the following tools:        duckduckgo_search: A wrapper around DuckDuckGo Search. Useful for when you need to answer questions about current events. Input should be a search query.    Calculator: Useful for when you need to answer questions about math.        Use the following format:        Question: the input question you must answer    Thought: you should always think about what to do    Action: the action to take, should be one of [duckduckgo_search, Calculator]    Action Input: the input to the action    Observation: the result of the action    ... (this Thought/Action/Action Input/Observation can repeat N times)    Thought: I now know the final answer    Final Answer: the final answer to the original input question        Begin!        Question: Who directed the 2023 film Oppenheimer and what is their age? What is their age in days (assume 365 days per year)?    Thought:        > Finished chain.    First, I need to find out who directed the film Oppenheimer in


chain.    First, I need to find out who directed the film Oppenheimer in 2023 and their birth date to calculate their age.    Action: duckduckgo_search    Action Input: "Director of the 2023 film Oppenheimer"    Observation: Oppenheimer: Directed by Christopher Nolan. With Cillian Murphy, Emily Blunt, Robert Downey Jr., Alden Ehrenreich. The story of American scientist J. Robert Oppenheimer and his role in the development of the atomic bomb. In Christopher Nolan's new film, "Oppenheimer," Cillian Murphy stars as J. Robert ... 2023, 12:16 p.m. ET. ... including his role as the director of the Manhattan Engineer District, better ... J Robert Oppenheimer was the director of the secret Los Alamos Laboratory. It was established under US president Franklin D Roosevelt as part of the Manhattan Project to build the first atomic bomb. He oversaw the first atomic bomb detonation in the New Mexico desert in July 1945, code-named "Trinity". In this opening salvo of 2023's Oscar battle, Nolan has enjoined a star-studded cast for a retelling of the brilliant and haunted life of J. Robert Oppenheimer, the American physicist whose... Oppenheimer is a 2023 epic biographical thriller film written and directed by Christopher Nolan.It is based on the 2005 biography American Prometheus by Kai Bird and Martin J. Sherwin about J. Robert Oppenheimer, a theoretical physicist who was pivotal in developing the first nuclear weapons as part of the Manhattan Project and thereby ushering in the Atomic Age.    Thought:        > Entering new LLMChain chain...    Prompt after formatting:    Answer the following questions as best you can. You have access to the following tools:


the following questions as best you can. You have access to the following tools:        duckduckgo_search: A wrapper around DuckDuckGo Search. Useful for when you need to answer questions about current events. Input should be a search query.    Calculator: Useful for when you need to answer questions about math.        Use the following format:        Question: the input question you must answer    Thought: you should always think about what to do    Action: the action to take, should be one of [duckduckgo_search, Calculator]    Action Input: the input to the action    Observation: the result of the action    ... (this Thought/Action/Action Input/Observation can repeat N times)    Thought: I now know the final answer    Final Answer: the final answer to the original input question        Begin!        Question: Who directed the 2023 film Oppenheimer and what is their age? What is their age in days (assume 365 days per year)?    Thought:First, I need to find out who directed the film Oppenheimer in 2023 and their birth date to calculate their age.    Action: duckduckgo_search    Action Input: "Director of the 2023 film Oppenheimer"    Observation: Oppenheimer: Directed by Christopher Nolan. With Cillian Murphy, Emily Blunt, Robert Downey Jr., Alden Ehrenreich. The story of American scientist J. Robert Oppenheimer and his role in the development of the atomic bomb. In Christopher Nolan's new film, "Oppenheimer," Cillian Murphy stars as J. Robert ... 2023, 12:16 p.m. ET. ... including his


J. Robert ... 2023, 12:16 p.m. ET. ... including his role as the director of the Manhattan Engineer District, better ... J Robert Oppenheimer was the director of the secret Los Alamos Laboratory. It was established under US president Franklin D Roosevelt as part of the Manhattan Project to build the first atomic bomb. He oversaw the first atomic bomb detonation in the New Mexico desert in July 1945, code-named "Trinity". In this opening salvo of 2023's Oscar battle, Nolan has enjoined a star-studded cast for a retelling of the brilliant and haunted life of J. Robert Oppenheimer, the American physicist whose... Oppenheimer is a 2023 epic biographical thriller film written and directed by Christopher Nolan.It is based on the 2005 biography American Prometheus by Kai Bird and Martin J. Sherwin about J. Robert Oppenheimer, a theoretical physicist who was pivotal in developing the first nuclear weapons as part of the Manhattan Project and thereby ushering in the Atomic Age.    Thought:        > Finished chain.    The director of the 2023 film Oppenheimer is Christopher Nolan. Now I need to find out his birth date to calculate his age.    Action: duckduckgo_search    Action Input: "Christopher Nolan birth date"    Observation: July 30, 1970 (age 52) London England Notable Works: "Dunkirk" "Tenet" "The Prestige" See all related content â†’ Recent News Jul. 13, 2023, 11:11 AM ET (AP) Cillian Murphy, playing Oppenheimer, finally gets to lead a Christopher Nolan film Christopher Edward Nolan CBE (born 30 July 1970) is a British and American filmmaker. Known for his Hollywood blockbusters with complex storytelling, Nolan is considered


is a British and American filmmaker. Known for his Hollywood blockbusters with complex storytelling, Nolan is considered a leading filmmaker of the 21st century. His films have grossed $5 billion worldwide. The recipient of many accolades, he has been nominated for five Academy Awards, five BAFTA Awards and six Golden Globe Awards. Christopher Nolan is currently 52 according to his birthdate July 30, 1970 Sun Sign Leo Born Place Westminster, London, England, United Kingdom Residence Los Angeles, California, United States Nationality Education Chris attended Haileybury and Imperial Service College, in Hertford Heath, Hertfordshire. Christopher Nolan's next movie will study the man who developed the atomic bomb, J. Robert Oppenheimer. Here's the release date, plot, trailers & more. July 2023 sees the release of Christopher Nolan's new film, Oppenheimer, his first movie since 2020's Tenet and his split from Warner Bros. Billed as an epic thriller about "the man who ...    Thought:        > Entering new LLMChain chain...    Prompt after formatting:    Answer the following questions as best you can. You have access to the following tools:        duckduckgo_search: A wrapper around DuckDuckGo Search. Useful for when you need to answer questions about current events. Input should be a search query.    Calculator: Useful for when you need to answer questions about math.        Use the following format:        Question: the input question you must answer    Thought: you should always think about what to do    Action: the action to take, should be one of [duckduckgo_search, Calculator]    Action Input: the input to the action    Observation: the result of the action


Input: the input to the action    Observation: the result of the action    ... (this Thought/Action/Action Input/Observation can repeat N times)    Thought: I now know the final answer    Final Answer: the final answer to the original input question        Begin!        Question: Who directed the 2023 film Oppenheimer and what is their age? What is their age in days (assume 365 days per year)?    Thought:First, I need to find out who directed the film Oppenheimer in 2023 and their birth date to calculate their age.    Action: duckduckgo_search    Action Input: "Director of the 2023 film Oppenheimer"    Observation: Oppenheimer: Directed by Christopher Nolan. With Cillian Murphy, Emily Blunt, Robert Downey Jr., Alden Ehrenreich. The story of American scientist J. Robert Oppenheimer and his role in the development of the atomic bomb. In Christopher Nolan's new film, "Oppenheimer," Cillian Murphy stars as J. Robert ... 2023, 12:16 p.m. ET. ... including his role as the director of the Manhattan Engineer District, better ... J Robert Oppenheimer was the director of the secret Los Alamos Laboratory. It was established under US president Franklin D Roosevelt as part of the Manhattan Project to build the first atomic bomb. He oversaw the first atomic bomb detonation in the New Mexico desert in July 1945, code-named "Trinity". In this opening salvo of 2023's Oscar battle, Nolan has enjoined a star-studded cast for a retelling of the brilliant and haunted life of J. Robert Oppenheimer, the American physicist whose... Oppenheimer is a 2023 epic biographical thriller film written and directed


American physicist whose... Oppenheimer is a 2023 epic biographical thriller film written and directed by Christopher Nolan.It is based on the 2005 biography American Prometheus by Kai Bird and Martin J. Sherwin about J. Robert Oppenheimer, a theoretical physicist who was pivotal in developing the first nuclear weapons as part of the Manhattan Project and thereby ushering in the Atomic Age.    Thought:The director of the 2023 film Oppenheimer is Christopher Nolan. Now I need to find out his birth date to calculate his age.    Action: duckduckgo_search    Action Input: "Christopher Nolan birth date"    Observation: July 30, 1970 (age 52) London England Notable Works: "Dunkirk" "Tenet" "The Prestige" See all related content â†’ Recent News Jul. 13, 2023, 11:11 AM ET (AP) Cillian Murphy, playing Oppenheimer, finally gets to lead a Christopher Nolan film Christopher Edward Nolan CBE (born 30 July 1970) is a British and American filmmaker. Known for his Hollywood blockbusters with complex storytelling, Nolan is considered a leading filmmaker of the 21st century. His films have grossed $5 billion worldwide. The recipient of many accolades, he has been nominated for five Academy Awards, five BAFTA Awards and six Golden Globe Awards. Christopher Nolan is currently 52 according to his birthdate July 30, 1970 Sun Sign Leo Born Place Westminster, London, England, United Kingdom Residence Los Angeles, California, United States Nationality Education Chris attended Haileybury and Imperial Service College, in Hertford Heath, Hertfordshire. Christopher Nolan's next movie will study the man who developed the atomic bomb, J. Robert Oppenheimer. Here's the release date, plot, trailers & more. July 2023 sees the release of Christopher


the release date, plot, trailers & more. July 2023 sees the release of Christopher Nolan's new film, Oppenheimer, his first movie since 2020's Tenet and his split from Warner Bros. Billed as an epic thriller about "the man who ...    Thought:        > Finished chain.    Christopher Nolan was born on July 30, 1970. Now I need to calculate his age in 2023 and then convert it into days.    Action: Calculator    Action Input: (2023 - 1970) * 365        > Entering new LLMMathChain chain...    (2023 - 1970) * 365        > Entering new LLMChain chain...    Prompt after formatting:    Translate a math problem into a expression that can be executed using Python's numexpr library. Use the output of running this code to answer the question.        Question: ${Question with math problem.}    ```text    ${single line mathematical expression that solves the problem}    ```    ...numexpr.evaluate(text)...    ```output    ${Output of running the code}    ```    Answer: ${Answer}        Begin.        Question: What is 37593 * 67?    ```text    37593 * 67    ```    ...numexpr.evaluate("37593 * 67")...    ```output    2518731    ```    Answer: 2518731        Question: 37593^(1/5)    ```text


Question: 37593^(1/5)    ```text    37593**(1/5)    ```    ...numexpr.evaluate("37593**(1/5)")...    ```output    8.222831614237718    ```    Answer: 8.222831614237718        Question: (2023 - 1970) * 365            > Finished chain.    ```text    (2023 - 1970) * 365    ```    ...numexpr.evaluate("(2023 - 1970) * 365")...        Answer: 19345    > Finished chain.        Observation: Answer: 19345    Thought:        > Entering new LLMChain chain...    Prompt after formatting:    Answer the following questions as best you can. You have access to the following tools:        duckduckgo_search: A wrapper around DuckDuckGo Search. Useful for when you need to answer questions about current events. Input should be a search query.    Calculator: Useful for when you need to answer questions about math.        Use the following format:        Question: the input question you must answer    Thought: you should always think about what to do    Action: the action to take, should be one of [duckduckgo_search, Calculator]    Action Input: the input to the action    Observation: the result of the action    ... (this Thought/Action/Action Input/Observation can repeat N times)


... (this Thought/Action/Action Input/Observation can repeat N times)    Thought: I now know the final answer    Final Answer: the final answer to the original input question        Begin!        Question: Who directed the 2023 film Oppenheimer and what is their age? What is their age in days (assume 365 days per year)?    Thought:First, I need to find out who directed the film Oppenheimer in 2023 and their birth date to calculate their age.    Action: duckduckgo_search    Action Input: "Director of the 2023 film Oppenheimer"    Observation: Oppenheimer: Directed by Christopher Nolan. With Cillian Murphy, Emily Blunt, Robert Downey Jr., Alden Ehrenreich. The story of American scientist J. Robert Oppenheimer and his role in the development of the atomic bomb. In Christopher Nolan's new film, "Oppenheimer," Cillian Murphy stars as J. Robert ... 2023, 12:16 p.m. ET. ... including his role as the director of the Manhattan Engineer District, better ... J Robert Oppenheimer was the director of the secret Los Alamos Laboratory. It was established under US president Franklin D Roosevelt as part of the Manhattan Project to build the first atomic bomb. He oversaw the first atomic bomb detonation in the New Mexico desert in July 1945, code-named "Trinity". In this opening salvo of 2023's Oscar battle, Nolan has enjoined a star-studded cast for a retelling of the brilliant and haunted life of J. Robert Oppenheimer, the American physicist whose... Oppenheimer is a 2023 epic biographical thriller film written and directed by Christopher Nolan.It is based on the 2005 biography American Prometheus by Kai Bird


written and directed by Christopher Nolan.It is based on the 2005 biography American Prometheus by Kai Bird and Martin J. Sherwin about J. Robert Oppenheimer, a theoretical physicist who was pivotal in developing the first nuclear weapons as part of the Manhattan Project and thereby ushering in the Atomic Age.    Thought:The director of the 2023 film Oppenheimer is Christopher Nolan. Now I need to find out his birth date to calculate his age.    Action: duckduckgo_search    Action Input: "Christopher Nolan birth date"    Observation: July 30, 1970 (age 52) London England Notable Works: "Dunkirk" "Tenet" "The Prestige" See all related content â†’ Recent News Jul. 13, 2023, 11:11 AM ET (AP) Cillian Murphy, playing Oppenheimer, finally gets to lead a Christopher Nolan film Christopher Edward Nolan CBE (born 30 July 1970) is a British and American filmmaker. Known for his Hollywood blockbusters with complex storytelling, Nolan is considered a leading filmmaker of the 21st century. His films have grossed $5 billion worldwide. The recipient of many accolades, he has been nominated for five Academy Awards, five BAFTA Awards and six Golden Globe Awards. Christopher Nolan is currently 52 according to his birthdate July 30, 1970 Sun Sign Leo Born Place Westminster, London, England, United Kingdom Residence Los Angeles, California, United States Nationality Education Chris attended Haileybury and Imperial Service College, in Hertford Heath, Hertfordshire. Christopher Nolan's next movie will study the man who developed the atomic bomb, J. Robert Oppenheimer. Here's the release date, plot, trailers & more. July 2023 sees the release of Christopher Nolan's new film, Oppenheimer, his first movie since 2020's


release of Christopher Nolan's new film, Oppenheimer, his first movie since 2020's Tenet and his split from Warner Bros. Billed as an epic thriller about "the man who ...    Thought:Christopher Nolan was born on July 30, 1970. Now I need to calculate his age in 2023 and then convert it into days.    Action: Calculator    Action Input: (2023 - 1970) * 365    Observation: Answer: 19345    Thought:        > Finished chain.    I now know the final answer    Final Answer: The director of the 2023 film Oppenheimer is Christopher Nolan and he is 53 years old in 2023. His age in days is 19345 days.        > Finished chain.    'The director of the 2023 film Oppenheimer is Christopher Nolan and he is 53 years old in 2023. His age in days is 19345 days.'Chain(..., verbose=True)â€‹You can also scope verbosity down to a single object, in which case only the inputs and outputs to that object are printed (along with any additional callbacks calls made specifically by that object).# Passing verbose=True to initialize_agent will pass that along to the AgentExecutor (which is a Chain).agent = initialize_agent(    tools,     llm,     agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,    verbose=True,)agent.run("Who directed the 2023 film Oppenheimer and what is their age? What is their age in days (assume 365 days per year)?")Console output     > Entering new AgentExecutor chain...    First, I need to find out who directed the film Oppenheimer in


chain...    First, I need to find out who directed the film Oppenheimer in 2023 and their birth date. Then, I can calculate their age in years and days.    Action: duckduckgo_search    Action Input: "Director of 2023 film Oppenheimer"    Observation: Oppenheimer: Directed by Christopher Nolan. With Cillian Murphy, Emily Blunt, Robert Downey Jr., Alden Ehrenreich. The story of American scientist J. Robert Oppenheimer and his role in the development of the atomic bomb. In Christopher Nolan's new film, "Oppenheimer," Cillian Murphy stars as J. Robert Oppenheimer, the American physicist who oversaw the Manhattan Project in Los Alamos, N.M. Universal Pictures... J Robert Oppenheimer was the director of the secret Los Alamos Laboratory. It was established under US president Franklin D Roosevelt as part of the Manhattan Project to build the first atomic bomb. He oversaw the first atomic bomb detonation in the New Mexico desert in July 1945, code-named "Trinity". A Review of Christopher Nolan's new film 'Oppenheimer' , the story of the man who fathered the Atomic Bomb. Cillian Murphy leads an all star cast ... Release Date: July 21, 2023. Director ... For his new film, "Oppenheimer," starring Cillian Murphy and Emily Blunt, director Christopher Nolan set out to build an entire 1940s western town.    Thought:The director of the 2023 film Oppenheimer is Christopher Nolan. Now I need to find out his birth date to calculate his age.    Action: duckduckgo_search    Action Input: "Christopher Nolan birth date"    Observation: July 30, 1970 (age 52) London England Notable Works: "Dunkirk" "Tenet"


(age 52) London England Notable Works: "Dunkirk" "Tenet" "The Prestige" See all related content â†’ Recent News Jul. 13, 2023, 11:11 AM ET (AP) Cillian Murphy, playing Oppenheimer, finally gets to lead a Christopher Nolan film Christopher Edward Nolan CBE (born 30 July 1970) is a British and American filmmaker. Known for his Hollywood blockbusters with complex storytelling, Nolan is considered a leading filmmaker of the 21st century. His films have grossed $5 billion worldwide. The recipient of many accolades, he has been nominated for five Academy Awards, five BAFTA Awards and six Golden Globe Awards. Christopher Nolan is currently 52 according to his birthdate July 30, 1970 Sun Sign Leo Born Place Westminster, London, England, United Kingdom Residence Los Angeles, California, United States Nationality Education Chris attended Haileybury and Imperial Service College, in Hertford Heath, Hertfordshire. Christopher Nolan's next movie will study the man who developed the atomic bomb, J. Robert Oppenheimer. Here's the release date, plot, trailers & more. Date of Birth: 30 July 1970 . ... Christopher Nolan is a British-American film director, producer, and screenwriter. His films have grossed more than US$5 billion worldwide, and have garnered 11 Academy Awards from 36 nominations. ...    Thought:Christopher Nolan was born on July 30, 1970. Now I can calculate his age in years and then in days.    Action: Calculator    Action Input: {"operation": "subtract", "operands": [2023, 1970]}    Observation: Answer: 53    Thought:Christopher Nolan is 53 years old in 2023. Now I need to calculate his age in days.    Action:


in 2023. Now I need to calculate his age in days.    Action: Calculator    Action Input: {"operation": "multiply", "operands": [53, 365]}    Observation: Answer: 19345    Thought:I now know the final answer    Final Answer: The director of the 2023 film Oppenheimer is Christopher Nolan. He is 53 years old in 2023, which is approximately 19345 days.        > Finished chain.    'The director of the 2023 film Oppenheimer is Christopher Nolan. He is 53 years old in 2023, which is approximately 19345 days.'Other callbacksâ€‹Callbacks are what we use to execute any functionality within a component outside the primary component logic. All of the above solutions use Callbacks under the hood to log intermediate steps of components. There's a number of Callbacks relevant for debugging that come with LangChain out of the box, like the FileCallbackHandler. You can also implement your own callbacks to execute custom functionality.See here for more info on Callbacks, how to use them, and customize them.PreviousSQL Question Answering Benchmarking: ChinookNextDeploymentTracinglangchain.debug and langchain.verboselangchain.debug = Truelangchain.verbose = TrueChain(..., verbose=True)Other callbacksCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


Evaluation | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionChainsMemoryAgentsCallbacksModulesGuidesEvaluationString EvaluatorsComparison EvaluatorsTrajectory EvaluatorsExamplesDebuggingDeploymentLangSmithModel ComparisonEcosystemAdditional resourcesGuidesEvaluationOn this pageEvaluationLanguage models can be unpredictable. This makes it challenging to ship reliable applications to production, where repeatable, useful outcomes across diverse inputs are a minimum requirement. Tests help demonstrate each component in an LLM application can produce the required or expected functionality. These tests also safeguard against regressions while you improve interconnected pieces of an integrated system. However, measuring the quality of generated text can be challenging. It can be hard to agree on the right set of metrics for your application, and it can be difficult to translate those into better performance. Furthermore, it's common to lack sufficient evaluation data to adequately test the range of inputs and expected outputs for each component when you're just getting started. The LangChain community is building open source tools and guides to help address these challenges.LangChain exposes different types of evaluators for common types of evaluation. Each type has off-the-shelf implementations you can use to get started, as well as an


extensible API so you can create your own or contribute improvements for everyone to use. The following sections have example notebooks for you to get started.String Evaluators: Evaluate the predicted string for a given input, usually against a reference stringTrajectory Evaluators: Evaluate the whole trajectory of agent actionsComparison Evaluators: Compare predictions from two runs on a common inputThis section also provides some additional examples of how you could use these evaluators for different scenarios or apply to different chain implementations in the LangChain library. Some examples include:Preference Scoring Chain Outputs: An example using a comparison evaluator on different models or prompts to select statistically significant differences in aggregate preference scoresReference Docsâ€‹For detailed information of the available evaluators, including how to instantiate, configure, and customize them. Check out the reference documentation directly.ğŸ—ƒï¸� String Evaluators5 itemsğŸ—ƒï¸� Comparison Evaluators3 itemsğŸ—ƒï¸� Trajectory Evaluators2 itemsğŸ—ƒï¸� Examples9 itemsPreviousGuidesNextString EvaluatorsReference DocsCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


Examples | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionChainsMemoryAgentsCallbacksModulesGuidesEvaluationString EvaluatorsComparison EvaluatorsTrajectory EvaluatorsExamplesAgent VectorDB Question Answering BenchmarkingComparing Chain OutputsData Augmented Question AnsweringEvaluating an OpenAPI ChainQuestion Answering Benchmarking: Paul Graham EssayQuestion Answering Benchmarking: State of the Union AddressQA GenerationQuestion AnsweringSQL Question Answering Benchmarking: ChinookDebuggingDeploymentLangSmithModel ComparisonEcosystemAdditional resourcesGuidesEvaluationExamplesExamplesğŸš§ Docs under construction ğŸš§Below are some examples for inspecting and checking different chains.ğŸ“„ï¸� Agent VectorDB Question Answering BenchmarkingHere we go over how to benchmark performance on a question answering task using an agent to route between multiple vectordatabases.ğŸ“„ï¸� Comparing Chain OutputsSuppose you have two different prompts (or LLMs). How do you know which will generate "better" results?ğŸ“„ï¸� Data Augmented Question AnsweringThis notebook uses some generic prompts/language models to evaluate an question answering system that uses other sources of data besides what is in the model. For example, this can be used to evaluate a question answering system over your proprietary data.ğŸ“„ï¸� Evaluating an OpenAPI ChainThis notebook goes over ways to semantically evaluate an OpenAPI Chain, which calls an endpoint defined by the OpenAPI specification using purely natural language.ğŸ“„ï¸� Question Answering Benchmarking: Paul Graham EssayHere we go over how to benchmark performance on a question answering task over a Paul Graham


Paul Graham EssayHere we go over how to benchmark performance on a question answering task over a Paul Graham essay.ğŸ“„ï¸� Question Answering Benchmarking: State of the Union AddressHere we go over how to benchmark performance on a question answering task over a state of the union address.ğŸ“„ï¸� QA GenerationThis notebook shows how to use the QAGenerationChain to come up with question-answer pairs over a specific document.ğŸ“„ï¸� Question AnsweringThis notebook covers how to evaluate generic question answering problems. This is a situation where you have an example containing a question and its corresponding ground truth answer, and you want to measure how well the language model does at answering those questions.ğŸ“„ï¸� SQL Question Answering Benchmarking: ChinookHere we go over how to benchmark performance on a question answering task over a SQL database.PreviousAgent TrajectoryNextAgent VectorDB Question Answering BenchmarkingCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


Comparing Chain Outputs | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionChainsMemoryAgentsCallbacksModulesGuidesEvaluationString EvaluatorsComparison EvaluatorsTrajectory EvaluatorsExamplesAgent VectorDB Question Answering BenchmarkingComparing Chain OutputsData Augmented Question AnsweringEvaluating an OpenAPI ChainQuestion Answering Benchmarking: Paul Graham EssayQuestion Answering Benchmarking: State of the Union AddressQA GenerationQuestion AnsweringSQL Question Answering Benchmarking: ChinookDebuggingDeploymentLangSmithModel ComparisonEcosystemAdditional resourcesGuidesEvaluationExamplesComparing Chain OutputsOn this pageComparing Chain OutputsSuppose you have two different prompts (or LLMs). How do you know which will generate "better" results?One automated way to predict the preferred configuration is to use a PairwiseStringEvaluator like the PairwiseStringEvalChain[1]. This chain prompts an LLM to select which output is preferred, given a specific input.For this evaluation, we will need 3 things:An evaluatorA dataset of inputs2 (or more) LLMs, Chains, or Agents to compareThen we will aggregate the restults to determine the preferred model.Step 1. Create the Evaluatorâ€‹In this example, you will use gpt-4 to select which output is preferred.from langchain.chat_models import ChatOpenAIfrom langchain.evaluation.comparison import PairwiseStringEvalChainllm = ChatOpenAI(model="gpt-4")eval_chain = PairwiseStringEvalChain.from_llm(llm=llm)Step 2. Select Datasetâ€‹If you already have real usage data for your LLM, you can use a representative sample. More examples


provide more reliable results. We will use some example queries someone might have about how to use langchain here.from langchain.evaluation.loading import load_datasetdataset = load_dataset("langchain-howto-queries")    Found cached dataset parquet (/Users/wfh/.cache/huggingface/datasets/LangChainDatasets___parquet/LangChainDatasets--langchain-howto-queries-bbb748bbee7e77aa/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)      0%|          | 0/1 [00:00<?, ?it/s]Step 3. Define Models to Compareâ€‹We will be comparing two agents in this case.from langchain import SerpAPIWrapperfrom langchain.agents import initialize_agent, Toolfrom langchain.agents import AgentTypefrom langchain.chat_models import ChatOpenAI# Initialize the language model# You can add your own OpenAI API key by adding openai_api_key="<your_api_key>"llm = ChatOpenAI(temperature=0, model="gpt-3.5-turbo-0613")# Initialize the SerpAPIWrapper for search functionality# Replace <your_api_key> in openai_api_key="<your_api_key>" with your actual SerpAPI key.search = SerpAPIWrapper()# Define a list of tools offered by the agenttools = [    Tool(        name="Search",        func=search.run,        coroutine=search.arun,        description="Useful when you need to answer questions about current events. You should ask targeted questions.",


when you need to answer questions about current events. You should ask targeted questions.",    ),]functions_agent = initialize_agent(    tools, llm, agent=AgentType.OPENAI_MULTI_FUNCTIONS, verbose=False)conversations_agent = initialize_agent(    tools, llm, agent=AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION, verbose=False)Step 4. Generate Responsesâ€‹We will generate outputs for each of the models before evaluating them.from tqdm.notebook import tqdmimport asyncioresults = []agents = [functions_agent, conversations_agent]concurrency_level = 6  # How many concurrent agents to run. May need to decrease if OpenAI is rate limiting.# We will only run the first 20 examples of this dataset to speed things up# This will lead to larger confidence intervals downstream.batch = []for example in tqdm(dataset[:20]):    batch.extend([agent.acall(example["inputs"]) for agent in agents])    if len(batch) >= concurrency_level:        batch_results = await asyncio.gather(*batch, return_exceptions=True)        results.extend(list(zip(*[iter(batch_results)] * 2)))        batch = []if batch:    batch_results = await asyncio.gather(*batch, return_exceptions=True)    results.extend(list(zip(*[iter(batch_results)] * 2)))      0%|          | 0/20 [00:00<?, ?it/s]    Retrying langchain.chat_models.openai.acompletion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised ServiceUnavailableError: The server is overloaded or not ready yet..    Retrying langchain.chat_models.openai.acompletion_with_retry.<locals>._completion_with_retry


Retrying langchain.chat_models.openai.acompletion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised ServiceUnavailableError: The server is overloaded or not ready yet..Step 5. Evaluate Pairsâ€‹Now it's time to evaluate the results. For each agent response, run the evaluation chain to select which output is preferred (or return a tie).Randomly select the input order to reduce the likelihood that one model will be preferred just because it is presented first.import randomdef predict_preferences(dataset, results) -> list:    preferences = []    for example, (res_a, res_b) in zip(dataset, results):        input_ = example["inputs"]        # Flip a coin to reduce persistent position bias        if random.random() < 0.5:            pred_a, pred_b = res_a, res_b            a, b = "a", "b"        else:            pred_a, pred_b = res_b, res_a            a, b = "b", "a"        eval_res = eval_chain.evaluate_string_pairs(            prediction=pred_a["output"] if isinstance(pred_a, dict) else str(pred_a),            prediction_b=pred_b["output"] if isinstance(pred_b, dict) else str(pred_b),            input=input_,        )        if eval_res["value"] == "A":


)        if eval_res["value"] == "A":            preferences.append(a)        elif eval_res["value"] == "B":            preferences.append(b)        else:            preferences.append(None)  # No preference    return preferencespreferences = predict_preferences(dataset, results)Print out the ratio of preferences.from collections import Countername_map = {    "a": "OpenAI Functions Agent",    "b": "Structured Chat Agent",}counts = Counter(preferences)pref_ratios = {k: v / len(preferences) for k, v in counts.items()}for k, v in pref_ratios.items():    print(f"{name_map.get(k)}: {v:.2%}")    OpenAI Functions Agent: 90.00%    Structured Chat Agent: 10.00%Estimate Confidence Intervalsâ€‹The results seem pretty clear, but if you want to have a better sense of how confident we are, that model "A" (the OpenAI Functions Agent) is the preferred model, we can calculate confidence intervals. Below, use the Wilson score to estimate the confidence interval.from math import sqrtdef wilson_score_interval(    preferences: list, which: str = "a", z: float = 1.96) -> tuple:    """Estimate the confidence interval using the Wilson score.    See: https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval#Wilson_score_interval    for more details, including when to use it and when it should not be used.    """    total_preferences = preferences.count("a") +


not be used.    """    total_preferences = preferences.count("a") + preferences.count("b")    n_s = preferences.count(which)    if total_preferences == 0:        return (0, 0)    p_hat = n_s / total_preferences    denominator = 1 + (z**2) / total_preferences    adjustment = (z / denominator) * sqrt(        p_hat * (1 - p_hat) / total_preferences        + (z**2) / (4 * total_preferences * total_preferences)    )    center = (p_hat + (z**2) / (2 * total_preferences)) / denominator    lower_bound = min(max(center - adjustment, 0.0), 1.0)    upper_bound = min(max(center + adjustment, 0.0), 1.0)    return (lower_bound, upper_bound)for which_, name in name_map.items():    low, high = wilson_score_interval(preferences, which=which_)    print(        f'The "{name}" would be preferred between {low:.2%} and {high:.2%} percent of the time (with 95% confidence).'    )    The "OpenAI Functions Agent" would be preferred between 69.90% and 97.21% percent of the time (with 95% confidence).    The "Structured Chat Agent" would be preferred between 2.79% and 30.10% percent of the time (with 95% confidence).Print out the p-value.from scipy import statspreferred_model = max(pref_ratios, key=pref_ratios.get)successes =


import statspreferred_model = max(pref_ratios, key=pref_ratios.get)successes = preferences.count(preferred_model)n = len(preferences) - preferences.count(None)p_value = stats.binom_test(successes, n, p=0.5, alternative="two-sided")print(    f"""The p-value is {p_value:.5f}. If the null hypothesis is true (i.e., if the selected eval chain actually has no preference between the models),then there is a {p_value:.5%} chance of observing the {name_map.get(preferred_model)} be preferred at least {successes}times out of {n} trials.""")    The p-value is 0.00040. If the null hypothesis is true (i.e., if the selected eval chain actually has no preference between the models),    then there is a 0.04025% chance of observing the OpenAI Functions Agent be preferred at least 18    times out of 20 trials._1. Note: Automated evals are still an open research topic and are best used alongside other evaluation approaches. LLM preferences exhibit biases, including banal ones like the order of outputs. In choosing preferences, "ground truth" may not be taken into account, which may lead to scores that aren't grounded in utility._PreviousAgent VectorDB Question Answering BenchmarkingNextData Augmented Question AnsweringStep 1. Create the EvaluatorStep 2. Select DatasetStep 3. Define Models to CompareStep 4. Generate ResponsesStep 5. Evaluate PairsEstimate Confidence IntervalsCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


Question Answering | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionChainsMemoryAgentsCallbacksModulesGuidesEvaluationString EvaluatorsComparison EvaluatorsTrajectory EvaluatorsExamplesAgent VectorDB Question Answering BenchmarkingComparing Chain OutputsData Augmented Question AnsweringEvaluating an OpenAPI ChainQuestion Answering Benchmarking: Paul Graham EssayQuestion Answering Benchmarking: State of the Union AddressQA GenerationQuestion AnsweringSQL Question Answering Benchmarking: ChinookDebuggingDeploymentLangSmithModel ComparisonEcosystemAdditional resourcesGuidesEvaluationExamplesQuestion AnsweringOn this pageQuestion AnsweringThis notebook covers how to evaluate generic question answering problems. This is a situation where you have an example containing a question and its corresponding ground truth answer, and you want to measure how well the language model does at answering those questions.Setupâ€‹For demonstration purposes, we will just evaluate a simple question answering system that only evaluates the model's internal knowledge. Please see other notebooks for examples where it evaluates how the model does at question answering over data not present in what the model was trained on.from langchain.prompts import PromptTemplatefrom langchain.chains import LLMChainfrom langchain.llms import OpenAIprompt = PromptTemplate(    template="Question: {question}\nAnswer:", input_variables=["question"])llm = OpenAI(model_name="text-davinci-003", temperature=0)chain = LLMChain(llm=llm, prompt=prompt)Examplesâ€‹For this purpose, we will just use two simple hardcoded examples, but see other notebooks for tips on how to get and/or generate these examples.examples = [    {        "question": "Roger has 5 tennis balls. He buys 2 more


"question": "Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now?",        "answer": "11",    },    {        "question": 'Is the following sentence plausible? "Joao Moutinho caught the screen pass in the NFC championship."',        "answer": "No",    },]Predictionsâ€‹We can now make and inspect the predictions for these questions.predictions = chain.apply(examples)predictions    [{'text': ' 11 tennis balls'},     {'text': ' No, this sentence is not plausible. Joao Moutinho is a professional soccer player, not an American football player, so it is not likely that he would be catching a screen pass in the NFC championship.'}]Evaluationâ€‹We can see that if we tried to just do exact match on the answer answers (11 and No) they would not match what the language model answered. However, semantically the language model is correct in both cases. In order to account for this, we can use a language model itself to evaluate the answers.from langchain.evaluation.qa import QAEvalChainllm = OpenAI(temperature=0)eval_chain = QAEvalChain.from_llm(llm)graded_outputs = eval_chain.evaluate(    examples, predictions, question_key="question", prediction_key="text")for i, eg in enumerate(examples):    print(f"Example {i}:")    print("Question: " + eg["question"])    print("Real Answer: " + eg["answer"])    print("Predicted Answer: " + predictions[i]["text"])    print("Predicted


print("Predicted Answer: " + predictions[i]["text"])    print("Predicted Grade: " + graded_outputs[i]["text"])    print()    Example 0:    Question: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now?    Real Answer: 11    Predicted Answer:  11 tennis balls    Predicted Grade:  CORRECT        Example 1:    Question: Is the following sentence plausible? "Joao Moutinho caught the screen pass in the NFC championship."    Real Answer: No    Predicted Answer:  No, this sentence is not plausible. Joao Moutinho is a professional soccer player, not an American football player, so it is not likely that he would be catching a screen pass in the NFC championship.    Predicted Grade:  CORRECT    Customize Promptâ€‹You can also customize the prompt that is used. Here is an example prompting it using a score from 0 to 10.


The custom prompt requires 3 input variables: "query", "answer" and "result". Where "query" is the question, "answer" is the ground truth answer, and "result" is the predicted answer.from langchain.prompts.prompt import PromptTemplate_PROMPT_TEMPLATE = """You are an expert professor specialized in grading students' answers to questions.You are grading the following question:{query}Here is the real answer:{answer}You are grading the following predicted answer:{result}What grade do you give from 0 to 10, where 0 is the lowest (very low similarity) and 10 is the highest (very high similarity)?"""PROMPT = PromptTemplate(    input_variables=["query", "answer", "result"], template=_PROMPT_TEMPLATE)evalchain = QAEvalChain.from_llm(llm=llm, prompt=PROMPT)evalchain.evaluate(    examples,    predictions,    question_key="question",    answer_key="answer",    prediction_key="text",)Evaluation without Ground Truthâ€‹Its possible to evaluate question answering systems without ground truth. You would need a "context" input that reflects what the information the LLM uses to answer the question. This context can be obtained by any retreival system. Here's an example of how it works:context_examples = [    {        "question": "How old am I?",        "context": "I am 30 years old. I live in New York and take the train to work everyday.",    },    {        "question": 'Who won the NFC championship game in 2023?"',        "context": "NFC Championship Game 2023: Philadelphia Eagles 31, San Francisco 49ers


"NFC Championship Game 2023: Philadelphia Eagles 31, San Francisco 49ers 7",    },]QA_PROMPT = "Answer the question based on the  context\nContext:{context}\nQuestion:{question}\nAnswer:"template = PromptTemplate(input_variables=["context", "question"], template=QA_PROMPT)qa_chain = LLMChain(llm=llm, prompt=template)predictions = qa_chain.apply(context_examples)predictions    [{'text': 'You are 30 years old.'},     {'text': ' The Philadelphia Eagles won the NFC championship game in 2023.'}]from langchain.evaluation.qa import ContextQAEvalChaineval_chain = ContextQAEvalChain.from_llm(llm)graded_outputs = eval_chain.evaluate(    context_examples, predictions, question_key="question", prediction_key="text")graded_outputs    [{'text': ' CORRECT'}, {'text': ' CORRECT'}]Comparing to other evaluation metricsâ€‹We can compare the evaluation results we get to other common evaluation metrics. To do this, let's load some evaluation metrics from HuggingFace's evaluate package.# Some data munging to get the examples in the right formatfor i, eg in enumerate(examples):    eg["id"] = str(i)    eg["answers"] = {"text": [eg["answer"]], "answer_start": [0]}    predictions[i]["id"] = str(i)    predictions[i]["prediction_text"] = predictions[i]["text"]for p in predictions:    del p["text"]new_examples = examples.copy()for eg in new_examples:    del eg["question"]    del eg["answer"]from evaluate import loadsquad_metric = load("squad")results = squad_metric.compute(    references=new_examples,


load("squad")results = squad_metric.compute(    references=new_examples,    predictions=predictions,)results    {'exact_match': 0.0, 'f1': 28.125}PreviousQA GenerationNextSQL Question Answering Benchmarking: ChinookSetupExamplesPredictionsEvaluationCustomize PromptEvaluation without Ground TruthComparing to other evaluation metricsCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


QA Generation | ğŸ¦œï¸�ğŸ”— Langchain

Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionChainsMemoryAgentsCallbacksModulesGuidesEvaluationString EvaluatorsComparison EvaluatorsTrajectory EvaluatorsExamplesAgent VectorDB Question Answering BenchmarkingComparing Chain OutputsData Augmented Question AnsweringEvaluating an OpenAPI ChainQuestion Answering Benchmarking: Paul Graham EssayQuestion Answering Benchmarking: State of the Union AddressQA GenerationQuestion AnsweringSQL Question Answering Benchmarking: ChinookDebuggingDeploymentLangSmithModel ComparisonEcosystemAdditional resourcesGuidesEvaluationExamplesQA GenerationQA GenerationThis notebook shows how to use the QAGenerationChain to come up with question-answer pairs over a specific document.
This is important because often times you may not have data to evaluate your question-answer system over, so this is a cheap and lightweight way to generate it!from langchain.document_loaders import TextLoaderloader = TextLoader("../../modules/state_of_the_union.txt")doc = loader.load()[0]from langchain.chat_models import ChatOpenAIfrom langchain.chains import QAGenerationChainchain = QAGenerationChain.from_llm(ChatOpenAI(temperature=0))qa = chain.run(doc.page_content)qa[1]    {'question': 'What is the U.S. Department of Justice doing to combat the crimes of Russian oligarchs?',     'answer': 'The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs.'}PreviousQuestion Answering Benchmarking: State of the Union AddressNextQuestion AnsweringCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


SQL Question Answering Benchmarking: Chinook | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionChainsMemoryAgentsCallbacksModulesGuidesEvaluationString EvaluatorsComparison EvaluatorsTrajectory EvaluatorsExamplesAgent VectorDB Question Answering BenchmarkingComparing Chain OutputsData Augmented Question AnsweringEvaluating an OpenAPI ChainQuestion Answering Benchmarking: Paul Graham EssayQuestion Answering Benchmarking: State of the Union AddressQA GenerationQuestion AnsweringSQL Question Answering Benchmarking: ChinookDebuggingDeploymentLangSmithModel ComparisonEcosystemAdditional resourcesGuidesEvaluationExamplesSQL Question Answering Benchmarking: ChinookOn this pageSQL Question Answering Benchmarking: ChinookHere we go over how to benchmark performance on a question answering task over a SQL database.It is highly reccomended that you do any evaluation/benchmarking with tracing enabled. See here for an explanation of what tracing is and how to set it up.# Comment this out if you are NOT using tracingimport osos.environ["LANGCHAIN_HANDLER"] = "langchain"Loading the dataâ€‹First, let's load the data.from langchain.evaluation.loading import load_datasetdataset = load_dataset("sql-qa-chinook")    Downloading readme:   0%|          | 0.00/21.0 [00:00<?, ?B/s]    Downloading and preparing dataset json/LangChainDatasets--sql-qa-chinook to


Downloading and preparing dataset json/LangChainDatasets--sql-qa-chinook to /Users/harrisonchase/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--sql-qa-chinook-7528565d2d992b47/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51...    Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]    Downloading data:   0%|          | 0.00/1.44k [00:00<?, ?B/s]    Extracting data files:   0%|          | 0/1 [00:00<?, ?it/s]    Generating train split: 0 examples [00:00, ? examples/s]    Dataset json downloaded and prepared to /Users/harrisonchase/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--sql-qa-chinook-7528565d2d992b47/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51. Subsequent calls will reuse this data.      0%|          | 0/1 [00:00<?, ?it/s]dataset[0]    {'question': 'How many employees are there?', 'answer': '8'}Setting up


{'question': 'How many employees are there?', 'answer': '8'}Setting up a chainâ€‹This uses the example Chinook database.


To set it up follow the instructions on https://database.guide/2-sample-databases-sqlite/, placing the .db file in a notebooks folder at the root of this repository.Note that here we load a simple chain. If you want to experiment with more complex chains, or an agent, just create the chain object in a different way.from langchain import OpenAI, SQLDatabase, SQLDatabaseChaindb = SQLDatabase.from_uri("sqlite:///../../../notebooks/Chinook.db")llm = OpenAI(temperature=0)Now we can create a SQL database chain.chain = SQLDatabaseChain.from_llm(llm, db, input_key="question")Make a predictionâ€‹First, we can make predictions one datapoint at a time. Doing it at this level of granularity allows use to explore the outputs in detail, and also is a lot cheaper than running over multiple datapointschain(dataset[0])    {'question': 'How many employees are there?',     'answer': '8',     'result': ' There are 8 employees.'}Make many predictionsâ€‹Now we can make predictions. Note that we add a try-except because this chain can sometimes error (if SQL is written incorrectly, etc)predictions = []predicted_dataset = []error_dataset = []for data in dataset:    try:        predictions.append(chain(data))        predicted_dataset.append(data)    except:        error_dataset.append(data)Evaluate performanceâ€‹Now we can evaluate the predictions. We can use a language model to score them programaticallyfrom langchain.evaluation.qa import QAEvalChainllm = OpenAI(temperature=0)eval_chain = QAEvalChain.from_llm(llm)graded_outputs = eval_chain.evaluate(    predicted_dataset, predictions,


= eval_chain.evaluate(    predicted_dataset, predictions, question_key="question", prediction_key="result")We can add in the graded output to the predictions dict and then get a count of the grades.for i, prediction in enumerate(predictions):    prediction["grade"] = graded_outputs[i]["text"]from collections import CounterCounter([pred["grade"] for pred in predictions])    Counter({' CORRECT': 3, ' INCORRECT': 4})We can also filter the datapoints to the incorrect examples and look at them.incorrect = [pred for pred in predictions if pred["grade"] == " INCORRECT"]incorrect[0]    {'question': 'How many employees are also customers?',     'answer': 'None',     'result': ' 59 employees are also customers.',     'grade': ' INCORRECT'}PreviousQuestion AnsweringNextDebuggingLoading the dataSetting up a chainMake a predictionMake many predictionsEvaluate performanceCommunityDiscordTwitterGitHubPythonJS/TSMoreHomepageBlogCopyright Â© 2023 LangChain, Inc.


Data Augmented Question Answering | ğŸ¦œï¸�ğŸ”— Langchain


Skip to main contentğŸ¦œï¸�ğŸ”— LangChainDocsUse casesIntegrationsAPILangSmithJS/TS DocsCTRLKGet startedIntroductionInstallationQuickstartModulesModel I/â€‹OData connectionChainsMemoryAgentsCallbacksModulesGuidesEvaluationString EvaluatorsComparison EvaluatorsTrajectory EvaluatorsExamplesAgent VectorDB Question Answering BenchmarkingComparing Chain OutputsData Augmented Question AnsweringEvaluating an OpenAPI ChainQuestion Answering Benchmarking: Paul Graham EssayQuestion Answering Benchmarking: State of the Union AddressQA GenerationQuestion AnsweringSQL Question Answering Benchmarking: ChinookDebuggingDeploymentLangSmithModel ComparisonEcosystemAdditional resourcesGuidesEvaluationExamplesData Augmented Question AnsweringOn this pageData Augmented Question AnsweringThis notebook uses some generic prompts/language models to evaluate an question answering system that uses other sources of data besides what is in the model. For example, this can be used to evaluate a question answering system over your proprietary data.Setupâ€‹Let's set up an example with our favorite example - the state of the union address.from langchain.embeddings.openai import OpenAIEmbeddingsfrom langchain.vectorstores import Chromafrom langchain.text_splitter import CharacterTextSplitterfrom langchain.llms import OpenAIfrom langchain.chains import RetrievalQAfrom langchain.document_loaders import TextLoaderloader = TextLoader("../../modules/state_of_the_union.txt")documents = loader.load()text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)texts = text_splitter.split_documents(documents)embeddings = OpenAIEmbeddings()docsearch = Chroma.from_documents(texts, embeddings)qa = RetrievalQA.from_llm(llm=OpenAI(), retriever=docsearch.as_retriever())    Running Chroma using direct local API.    Using DuckDB in-memory for database. Data will be


using direct local API.    Using DuckDB in-memory for database. Data will be transient.Examplesâ€‹Now we need some examples to evaluate. We can do this in two ways:Hard code some examples ourselvesGenerate examples automatically, using a language model# Hard-coded examplesexamples = [    {        "query": "What did the president say about Ketanji Brown Jackson",        "answer": "He praised her legal ability and said he nominated her for the supreme court.",    },    {"query": "What did the president say about Michael Jackson", "answer": "Nothing"},]# Generated examplesfrom langchain.evaluation.qa import QAGenerateChainexample_gen_chain = QAGenerateChain.from_llm(OpenAI())new_examples = example_gen_chain.apply_and_parse([{"doc": t} for t in texts[:5]])new_examples    [{'query': 'According to the document, what did Vladimir Putin miscalculate?',      'answer': 'He miscalculated that he could roll into Ukraine and the world would roll over.'},     {'query': 'Who is the Ukrainian Ambassador to the United States?',      'answer': 'The Ukrainian Ambassador to the United States is here tonight.'},     {'query': 'How many countries were part of the coalition formed to confront Putin?',      'answer': '27 members of the European Union, France, Germany, Italy, the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.'},     {'query': 'What action is the U.S. Department of Justice taking to target Russian oligarchs?',      'answer': 'The U.S. Department of Justice is assembling a dedicated task force to go after the


'The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and joining with European allies to find and seize their yachts, luxury apartments, and private jets.'},     {'query': 'How much direct assistance is the United States providing to Ukraine?',      'answer': 'The United States is providing more than $1 Billion in direct assistance to Ukraine.'}]# Combine examplesexamples += new_examplesEvaluateâ€‹Now that we have examples, we can use the question answering evaluator to evaluate our question answering chain.from langchain.evaluation.qa import QAEvalChainpredictions = qa.apply(examples)llm = OpenAI(temperature=0)eval_chain = QAEvalChain.from_llm(llm)graded_outputs = eval_chain.evaluate(examples, predictions)for i, eg in enumerate(examples):    print(f"Example {i}:")    print("Question: " + predictions[i]["query"])    print("Real Answer: " + predictions[i]["answer"])    print("Predicted Answer: " + predictions[i]["result"])    print("Predicted Grade: " + graded_outputs[i]["text"])    print()    Example 0:    Question: What did the president say about Ketanji Brown Jackson    Real Answer: He praised her legal ability and said he nominated her for the supreme court.    Predicted Answer:  The president said that she is one of the nation's top legal minds, a former top litigator in private practice, a former federal public defender, and from a family of public school educators and police officers. He also said that she is a consensus builder and that she has received a broad range of support from the Fraternal Order of Police to former judges appointed by both Democrats and Republicans.    Predicted Grade:


Order of Police to former judges appointed by both Democrats and Republicans.    Predicted Grade:  CORRECT        Example 1:    Question: What did the president say about Michael Jackson    Real Answer: Nothing    Predicted Answer:  The president did not mention Michael Jackson in this speech.    Predicted Grade:  CORRECT        Example 2:    Question: According to the document, what did Vladimir Putin miscalculate?    Real Answer: He miscalculated that he could roll into Ukraine and the world would roll over.    Predicted Answer:  Putin miscalculated that the world would roll over when he rolled into Ukraine.    Predicted Grade:  CORRECT        Example 3:    Question: Who is the Ukrainian Ambassador to the United States?    Real Answer: The Ukrainian Ambassador to the United States is here tonight.    Predicted Answer:  I don't know.    Predicted Grade:  INCORRECT        Example 4:    Question: How many countries were part of the coalition formed to confront Putin?    Real Answer: 27 members of the European Union, France, Germany, Italy, the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.    Predicted Answer:  The coalition included freedom-loving nations from Europe and the Americas to Asia and Africa, 27 members of the European Union including France, Germany, Italy, the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.    Predicted Grade:  INCORRECT        Example 5:    Question: What action is the


Example 5:    Question: What action is the U.S. Department of Justice taking to target Russian oligarchs?    Real Answer: The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and joining with European allies to find and seize their yachts, luxury apartments, and private jets.    Predicted Answer:  The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and to find and seize their yachts, luxury apartments, and private jets.    Predicted Grade:  INCORRECT        Example 6:    Question: How much direct assistance is the United States providing to Ukraine?    Real Answer: The United States is providing more than $1 Billion in direct assistance to Ukraine.    Predicted Answer:  The United States is providing more than $1 billion in direct assistance to Ukraine.    Predicted Grade:  CORRECT    Evaluate with Other Metricsâ€‹In addition to predicting whether the answer is correct or incorrect using a language model, we can also use other metrics to get a more nuanced view on the quality of the answers. To do so, we can use the Critique library, which allows for simple calculation of various metrics over generated text.First you can get an API key from the Inspired Cognition Dashboard and do some setup:export INSPIREDCO_API_KEY="..."pip install inspiredcoimport inspiredco.critiqueimport oscritique = inspiredco.critique.Critique(api_key=os.environ["INSPIREDCO_API_KEY"])Then run the following code to set up the configuration and calculate the ROUGE, chrf, BERTScore, and UniEval (you can choose other metrics too):metrics = {    "rouge": {