import re
from .question_relevant_grammar import question_relevant_grammar
from llama_cpp import Llama
from .constants import LOGICAL_MODEL

# POSSIBLE TODO:
# Add an "Off the rails insane forever generation" check, like on this output:
# I haven't seen this happen since I started using 70bs though.
"""
4.) What did ancient people believe about the shape and movement of our world, before scientific discoveries indicated otherwise? (Hint: The earth is a spheroid, not flat. The universe existed before the year 4004 BCE. The earth rotates on its axis and orbits around the sun.) You are allowed to dramatically change/revamp/rewrite this question's content. The goal here is to create a different question that still requires information from the paragraphs, but doesn't require knowledge of flawed assumptions connected with biblical interpretations or an understanding of events before the creation of the universe, which goes beyond the generally accepted scientific evidence provided in the text.  As you see fit. (Don't just rephrase the old question.) The goal here is to create a different question that still requires information from the paragraphs, but doesn't require knowledge of flawed assumptions connected with biblical interpretations or an understanding of events before the creation of the universe, which goes beyond the generally accepted scientific evidence provided in the text.  As you see fit. (Don't just rephrase the old question.) So you should create a new question that only requires information from the paragraphs to solve. It is not necessary to refer to the text at all. Also note: An example about how NOT to ask questions is if the text states fact X, but does not explain how X was established, do not ask a question "How do we know X". But instead you might consider asking how X relates to other facts in the paragraph, or how these facts together lead to a progression of ideas, "Explain how X, Y, and Z are related" for instance. Here's what we know: 1. The earth is a spheroid, not flat (paragraph 3). 2. The universe existed before the year 4004 BCE (paragraph 5). 3. The earth rotates on its axis and orbits around the sun (paragraph 5).
Answer: We don't have an answer.
"""

# Answer vetting
def check_question(qatuple, logic_llm):
    retries = 0
    while retries <= 4:
        decision_prompt = f"""You are an expert educational AI. Given a paragraph or two from a larger text, and a question based on the paragraphs, you will make a determination as to whether the question tests ONLY information in the paragraphs. Essentially: you will check if the question is answerable, given the information in the paragraphs. Your task includes first analyzing the text, thinking through whether or not the question reflects aspects of the paragraphs provided.

Following this, at the very end of your response, your "final judgment" or "final answer", you will write "Relevant" or "Irrelevant" depending on your analysis of the question with regards to the text.

Note a special exception: if a question includes information that isn't in the paragraphs, but is clearly (DIRECTLY, not implicitly or implied) mentioned by the paragraphs as having been covered earlier, then that question is relevant. Essentially: questions are allowed to cover content that the text has explicitly covered in the past.

Write out the reasoning and analysis behind your judgment, step-by-step. Your analysis of the question, against the text, should follow a logical progression of steps that results in a conclusive and accurate final answer.

You will analyze the question step-by-step, ensuring each part of the question is individually compared to the text. The key steps are analyzing the text, understanding the question, and then systematically comparing each part of the question with the text. The process continues until either a part of the question is found not to be covered by the text, leading to a judgment of "Irrelevant," or until all parts of the question have been compared and found to be covered by the text, leading to a judgment of "Relevant." This method allows for a thorough and detailed assessment, ensuring that the final judgment accurately reflects the extent to which the question is based on the given text.

Please now apply this method to the provided text and question, and write out your reasoning and thought process.

### Instruction:
Text:
\"\"\"
The concept of artificial intelligence (AI) revolves around the creation of machines capable of intelligent behavior. Key components of AI include machine learning, neural networks, and natural language processing. Machine learning involves training computers to learn from data and improve their performance over time. Neural networks are modeled after the human brain's network of neurons and are pivotal in enabling machines to recognize patterns and make decisions. Natural language processing, another crucial aspect of AI, allows machines to understand and interpret human languages, facilitating interaction between humans and computers.
\"\"\"

Question (based on text): \"\"\"What is the role of neural networks in AI, and how does natural language processing contribute to human-computer interaction?\"\"\"

### Response:
## Reasoning and thought process:

### In-Depth Analysis of the Text:
#### Content and Depth: The text details key components of AI, including machine learning, neural networks, and natural language processing, with a focus on their roles and functions in AI.
#### Type of Information: The information is primarily descriptive, outlining the roles and functions of neural networks and natural language processing in AI.

### Detailed Understanding of the Question:
#### Core Requirement: The question asks about the roles of neural networks in AI and the contribution of natural language processing to human-computer interaction, seeking specific functional explanations.
#### Depth of Detail: The question requires an understanding of the specific functions and impacts of these AI components.

### Targeted Comparison of the Question with the Text:
#### Content Match: The text aligns well with the question, covering the roles of neural networks in pattern recognition and decision-making, and the role of natural language processing in understanding and interpreting human languages.
#### Depth Match: The text provides the necessary level of detail to answer the question, though it remains descriptive and does not delve into deeper explanatory mechanisms.

### Critical Evaluation and Final Judgment:
The text addresses both components of the question with sufficient detail, making it: Relevant.

### Instruction:
Text:
\"\"\"
The phenomenon of photosynthesis in plants is an essential process for life on Earth. It involves the conversion of light energy into chemical energy, which is stored in glucose. This process occurs in the chloroplasts, specifically within a pigment called chlorophyll. Photosynthesis consists of two main stages: the light-dependent reactions and the light-independent reactions, also known as the Calvin cycle. During the light-dependent reactions, sunlight is absorbed by chlorophyll, which then converts water (H2O) into oxygen (O2) and transfers energy to the light-independent reactions. The Calvin cycle uses this energy to convert carbon dioxide (CO2) from the atmosphere into glucose, a simple sugar.
\"\"\"

Question (based on text): \"\"\"Explain exactly why chlorophyll able to absorb sunlight, and why is this important for the Calvin cycle?\"\"\"

### Response:
## Reasoning and thought process:

### In-Depth Analysis of the Text:
#### Content and Depth: The text discusses the process of photosynthesis, including the roles of chlorophyll, light-dependent reactions, and the Calvin cycle.
#### Type of Information: Information is descriptive, explaining the stages of photosynthesis and the role of chlorophyll in light absorption.

### Detailed Understanding of the Question:
#### Core Requirement: The question seeks an explanation of why chlorophyll can absorb sunlight and its importance for the Calvin cycle, requiring detailed scientific principles.
#### Depth of Detail: Requires an in-depth, explanatory response about the biological and chemical properties of chlorophyll.

### Targeted Comparison of the Question with the Text:
#### Content Match: The text partially matches the question by mentioning the role of chlorophyll in the light-dependent reactions.
#### Depth Match: The text lacks the specific scientific principles explaining the mechanism of sunlight absorption by chlorophyll, which is central to the question.

### Critical Evaluation and Final Judgment:
Although the text touches upon the role of chlorophyll, it does not provide the depth of explanation required by the question, making it: Irrelevant.

### Instruction:
Text:
\"\"\"
The octopus is an extraordinary creature, renowned for its intelligence and ability to adapt to its environment. Octopuses have three hearts and a complex nervous system, which includes a central brain and a small brain in each of their eight arms. These arms are capable of intricate movements and can even carry out simple tasks independently. The creature's ability to change its color and texture is another remarkable feature, which it uses for camouflage and communication. Notably, octopuses are known for their problem-solving skills and have been observed using tools in captivity.
\"\"\"

Question (based on text): \"\"\"How do octopuses utilize their color-changing ability in their natural habitat, and what role does their central brain play in this process?\"\"\"

### Response:
## Reasoning and thought process:

### In-Depth Analysis of the Text:
#### Content and Depth: The text outlines various aspects of the octopus, such as its physiology, nervous system, color-changing ability, and intelligence.
#### Type of Information: Descriptive, covering the abilities and features of the octopus, including color-changing for camouflage and communication.

### Detailed Understanding of the Question:
#### Core Requirement: The question asks how octopuses utilize their color-changing ability and the role of their central brain in this process.
#### Depth of Detail: Seeks specific information on the function and control mechanism of the color-changing ability.

### Targeted Comparison of the Question with the Text:
#### Content Match: The text aligns with the first part of the question regarding the use of color-changing for camouflage and communication.
#### Depth Match: The text does not provide information about the role of the central brain in this process, lacking the required depth on the control mechanism.

### Critical Evaluation and Final Judgment:
Given the text's coverage of color-changing but lack of detail on the central brain's role, the overall assessment of the question's relevance to the text is: Irrelevant.

### Instruction:
Text:
\"\"\"
{qatuple[2]}
\"\"\"

Question (based on text): \"\"\"{qatuple[0]}\"\"\"

If the question clearly goes off the rails and is incoherent, then it is irrelevant.

### Response:
## Reasoning and thought process (be careful around "how" and "why" questions):
"""
        try:
            completion = logic_llm(
                decision_prompt,
                max_tokens=4000,
                stop=["</s>", "# Input:"],
                echo=True,
                grammar=question_relevant_grammar,
                temperature=0.2,
            )["choices"][0]["text"]

            response_pattern = re.compile(
                r"Reasoning and thought process \(be careful around \"how\" and \"why\" questions\):(.+)",
                re.DOTALL | re.IGNORECASE,
            )
            response = response_pattern.search(completion).group(1).strip()
            decision_pattern = re.compile(
                r"Final Judgment:(.+)", re.DOTALL | re.IGNORECASE
            )
            print(response)
            determination = decision_pattern.search(response).group(1).strip()
            print("\n\nDETERMINATION:\n------")
            print(determination)
            print("\n---------\n")
            if (
                "irrelevant" in determination
                or "Irrelevant" in determination.lower()
                or "mostly" in determination.lower()
                or "partial" in determination.lower()
                or "introduces information not present in the text"
                in determination.lower()
            ):
                return (False, response), completion
            elif "relevant" in determination or "Relevant" in determination:
                return (True, response), completion
            else:
                print("Did not contain relevant or irrelevant! Retrying")
                retries += 1
        except Exception as e:
            print("Exception!", e)
            if retries <= 4:
                retries += 1
            else:
                return (None, None), completion
    return (None, None), None

if __name__ == "__main__":  # test
    logic_llm = Llama(
        model_path=LOGICAL_MODEL,
        n_gqa=8,
        offload_kqv=True,
        n_ctx=4096,
        n_gpu_layers=1000,
    )  # load the logical LLM and offload everything
    q_test = [
        (
            "Explain how our understanding of planetary motion has changed over time.",
            "The understanding has evolved from the Earth being stationary and at the centre of the universe, to it orbiting the sun in an elliptical path with other planets while still rotating on its axis.",
            "The story of our world is a story that is still very imperfectly known. A couple of hundred years ago men possessed the history of little more than the last three thousand years. What happened before that time was a matter of legend and speculation.  Over a large part of the civilized world it was believed and taught that the world had been created suddenly in 4004 B.C., though authorities differed as to whether this had occurred in the spring or autumn of that year. This fantastically precise misconception was based upon a too literal interpretation of the Hebrew Bible, and upon rather arbitrary theological assumptions connected therewith.  Such ideas have long since been abandoned by religious teachers, and it is universally recognized that the universe in which we live has to all appearances existed for an enormous period of time and possibly for endless time.  Of course there may be deception in these appearances, as a room may be made to seem endless by putting mirrors facing each other at either end. But that the universe in which we live has existed only for six or seven thousand years may be regarded as an altogether exploded idea.\n\nThe earth, as everybody knows nowadays, is a spheroid, a sphere slightly compressed, orange fashion, with a diameter of nearly 8,000 miles.  Its spherical shape has been known at least to a limited number of intelligent people for nearly 2,500 years, but before that time it was supposed to be flat, and various ideas which now seem fantastic were entertained about its relations to the sky and the stars and planets.  We know now that it rotates upon its axis (which is about 24 miles shorter than its equatorial diameter) every twenty-four hours, and that this is the cause of the alternations of day and night, that it circles about the sun in a slightly distorted and slowly variable oval path in a year. Its distance from the sun varies between ninety-one and a half millions at its nearest and ninety-four and a half million miles.",
        ),
        (
            "Identify and explain changes in human understanding throughout history regarding the age of the Earth.",
            "Initially, religious texts suggested a young earth dating back no more than several thousand years. However, evidence from geology and astronomy has shown us that the earth is over four billion years old.",
            "The story of our world is a story that is still very imperfectly known. A couple of hundred years ago men possessed the history of little more than the last three thousand years. What happened before that time was a matter of legend and speculation.  Over a large part of the civilized world it was believed and taught that the world had been created suddenly in 4004 B.C., though authorities differed as to whether this had occurred in the spring or autumn of that year. This fantastically precise misconception was based upon a too literal interpretation of the Hebrew Bible, and upon rather arbitrary theological assumptions connected therewith.  Such ideas have long since been abandoned by religious teachers, and it is universally recognized that the universe in which we live has to all appearances existed for an enormous period of time and possibly for endless time.  Of course there may be deception in these appearances, as a room may be made to seem endless by putting mirrors facing each other at either end. But that the universe in which we live has existed only for six or seven thousand years may be regarded as an altogether exploded idea.\n\nThe earth, as everybody knows nowadays, is a spheroid, a sphere slightly compressed, orange fashion, with a diameter of nearly 8,000 miles.  Its spherical shape has been known at least to a limited number of intelligent people for nearly 2,500 years, but before that time it was supposed to be flat, and various ideas which now seem fantastic were entertained about its relations to the sky and the stars and planets.  We know now that it rotates upon its axis (which is about 24 miles shorter than its equatorial diameter) every twenty-four hours, and that this is the cause of the alternations of day and night, that it circles about the sun in a slightly distorted and slowly variable oval path in a year. Its distance from the sun varies between ninety-one and a half millions at its nearest and ninety-four and a half million miles.",
        ),
        (
            "Using specific scientific principles, explain why we know Earth is approximately 8000 miles in diameter and how its distance from the sun varies.",
            "We know about Earth's diameter using measurements of its circumference made using GPS data. The variation in distance to the sun is due to Earth's elliptical orbit around the sun, with a varying point of closest approach and farthest departure.",
            "The story of our world is a story that is still very imperfectly known. A couple of hundred years ago men possessed the history of little more than the last three thousand years. What happened before that time was a matter of legend and speculation.  Over a large part of the civilized world it was believed and taught that the world had been created suddenly in 4004 B.C., though authorities differed as to whether this had occurred in the spring or autumn of that year. This fantastically precise misconception was based upon a too literal interpretation of the Hebrew Bible, and upon rather arbitrary theological assumptions connected therewith.  Such ideas have long since been abandoned by religious teachers, and it is universally recognized that the universe in which we live has to all appearances existed for an enormous period of time and possibly for endless time.  Of course there may be deception in these appearances, as a room may be made to seem endless by putting mirrors facing each other at either end. But that the universe in which we live has existed only for six or seven thousand years may be regarded as an altogether exploded idea.\n\nThe earth, as everybody knows nowadays, is a spheroid, a sphere slightly compressed, orange fashion, with a diameter of nearly 8,000 miles.  Its spherical shape has been known at least to a limited number of intelligent people for nearly 2,500 years, but before that time it was supposed to be flat, and various ideas which now seem fantastic were entertained about its relations to the sky and the stars and planets.  W
# ... truncated ...


import re
from .question_relevant_grammar import question_relevant_grammar
from llama_cpp import Llama
from .constants import LOGICAL_MODEL

# POSSIBLE TODO:
# Add an "Off the rails insane forever generation" check, like on this output:
# I haven't seen this happen since I started using 70bs though.
"""
4.) What did ancient people believe about the shape and movement of our world, before scientific discoveries indicated otherwise? (Hint: The earth is a spheroid, not flat. The universe existed before the year 4004 BCE. The earth rotates on its axis and orbits around the sun.) You are allowed to dramatically change/revamp/rewrite this question's content. The goal here is to create a different question that still requires information from the paragraphs, but doesn't require knowledge of flawed assumptions connected with biblical interpretations or an understanding of events before the creation of the universe, which goes beyond the generally accepted scientific evidence provided in the text.  As you see fit. (Don't just rephrase the old question.) The goal here is to create a different question that still requires information from the paragraphs, but doesn't require knowledge of flawed assumptions connected with biblical interpretations or an understanding of events before the creation of the universe, which goes beyond the generally accepted scientific evidence provided in the text.  As you see fit. (Don't just rephrase the old question.) So you should create a new question that only requires information from the paragraphs to solve. It is not necessary to refer to the text at all. Also note: An example about how NOT to ask questions is if the text states fact X, but does not explain how X was established, do not ask a question "How do we know X". But instead you might consider asking how X relates to other facts in the paragraph, or how these facts together lead to a progression of ideas, "Explain how X, Y, and Z are related" for instance. Here's what we know: 1. The earth is a spheroid, not flat (paragraph 3). 2. The universe existed before the year 4004 BCE (paragraph 5). 3. The earth rotates on its axis and orbits around the sun (paragraph 5).
Answer: We don't have an answer.
"""

# Answer vetting
def check_question(qatuple, logic_llm):
    retries = 0
    while retries <= 4:
        decision_prompt = f"""You are an expert educational AI. Given a paragraph or two from a larger text, and a question based on the paragraphs, you will make a determination as to whether the question tests ONLY information in the paragraphs. Essentially: you will check if the question is answerable, given the information in the paragraphs. Your task includes first analyzing the text, thinking through whether or not the question reflects aspects of the paragraphs provided.

Following this, at the very end of your response, your "final judgment" or "final answer", you will write "Relevant" or "Irrelevant" depending on your analysis of the question with regards to the text.

Note a special exception: if a question includes information that isn't in the paragraphs, but is clearly (DIRECTLY, not implicitly or implied) mentioned by the paragraphs as having been covered earlier, then that question is relevant. Essentially: questions are allowed to cover content that the text has explicitly covered in the past.

Write out the reasoning and analysis behind your judgment, step-by-step. Your analysis of the question, against the text, should follow a logical progression of steps that results in a conclusive and accurate final answer.

You will analyze the question step-by-step, ensuring each part of the question is individually compared to the text. The key steps are analyzing the text, understanding the question, and then systematically comparing each part of the question with the text. The process continues until either a part of the question is found not to be covered by the text, leading to a judgment of "Irrelevant," or until all parts of the question have been compared and found to be covered by the text, leading to a judgment of "Relevant." This method allows for a thorough and detailed assessment, ensuring that the final judgment accurately reflects the extent to which the question is based on the given text.

Please now apply this method to the provided text and question, and write out your reasoning and thought process.

### Instruction:
Text:
\"\"\"
The concept of artificial intelligence (AI) revolves around the creation of machines capable of intelligent behavior. Key components of AI include machine learning, neural networks, and natural language processing. Machine learning involves training computers to learn from data and improve their performance over time. Neural networks are modeled after the human brain's network of neurons and are pivotal in enabling machines to recognize patterns and make decisions. Natural language processing, another crucial aspect of AI, allows machines to understand and interpret human languages, facilitating interaction between humans and computers.
\"\"\"

Question (based on text): \"\"\"What is the role of neural networks in AI, and how does natural language processing contribute to human-computer interaction?\"\"\"

### Response:
## Reasoning and thought process:

### In-Depth Analysis of the Text:
#### Content and Depth: The text details key components of AI, including machine learning, neural networks, and natural language processing, with a focus on their roles and functions in AI.
#### Type of Information: The information is primarily descriptive, outlining the roles and functions of neural networks and natural language processing in AI.

### Detailed Understanding of the Question:
#### Core Requirement: The question asks about the roles of neural networks in AI and the contribution of natural language processing to human-computer interaction, seeking specific functional explanations.
#### Depth of Detail: The question requires an understanding of the specific functions and impacts of these AI components.

### Targeted Comparison of the Question with the Text:
#### Content Match: The text aligns well with the question, covering the roles of neural networks in pattern recognition and decision-making, and the role of natural language processing in understanding and interpreting human languages.
#### Depth Match: The text provides the necessary level of detail to answer the question, though it remains descriptive and does not delve into deeper explanatory mechanisms.

### Critical Evaluation and Final Judgment:
The text addresses both components of the question with sufficient detail, making it: Relevant.

### Instruction:
Text:
\"\"\"
The phenomenon of photosynthesis in plants is an essential process for life on Earth. It involves the conversion of light energy into chemical energy, which is stored in glucose. This process occurs in the chloroplasts, specifically within a pigment called chlorophyll. Photosynthesis consists of two main stages: the light-dependent reactions and the light-independent reactions, also known as the Calvin cycle. During the light-dependent reactions, sunlight is absorbed by chlorophyll, which then converts water (H2O) into oxygen (O2) and transfers energy to the light-independent reactions. The Calvin cycle uses this energy to convert carbon dioxide (CO2) from the atmosphere into glucose, a simple sugar.
\"\"\"

Question (based on text): \"\"\"Explain exactly why chlorophyll able to absorb sunlight, and why is this important for the Calvin cycle?\"\"\"

### Response:
## Reasoning and thought process:

### In-Depth Analysis of the Text:
#### Content and Depth: The text discusses the process of photosynthesis, including the roles of chlorophyll, light-dependent reactions, and the Calvin cycle.
#### Type of Information: Information is descriptive, explaining the stages of photosynthesis and the role of chlorophyll in light absorption.

### Detailed Understanding of the Question:
#### Core Requirement: The question seeks an explanation of why chlorophyll can absorb sunlight and its importance for the Calvin cycle, requiring detailed scientific principles.
#### Depth of Detail: Requires an in-depth, explanatory response about the biological and chemical properties of chlorophyll.

### Targeted Comparison of the Question with the Text:
#### Content Match: The text partially matches the question by mentioning the role of chlorophyll in the light-dependent reactions.
#### Depth Match: The text lacks the specific scientific principles explaining the mechanism of sunlight absorption by chlorophyll, which is central to the question.

### Critical Evaluation and Final Judgment:
Although the text touches upon the role of chlorophyll, it does not provide the depth of explanation required by the question, making it: Irrelevant.

### Instruction:
Text:
\"\"\"
The octopus is an extraordinary creature, renowned for its intelligence and ability to adapt to its environment. Octopuses have three hearts and a complex nervous system, which includes a central brain and a small brain in each of their eight arms. These arms are capable of intricate movements and can even carry out simple tasks independently. The creature's ability to change its color and texture is another remarkable feature, which it uses for camouflage and communication. Notably, octopuses are known for their problem-solving skills and have been observed using tools in captivity.
\"\"\"

Question (based on text): \"\"\"How do octopuses utilize their color-changing ability in their natural habitat, and what role does their central brain play in this process?\"\"\"

### Response:
## Reasoning and thought process:

### In-Depth Analysis of the Text:
#### Content and Depth: The text outlines various aspects of the octopus, such as its physiology, nervous system, color-changing ability, and intelligence.
#### Type of Information: Descriptive, covering the abilities and features of the octopus, including color-changing for camouflage and communication.

### Detailed Understanding of the Question:
#### Core Requirement: The question asks how octopuses utilize their color-changing ability and the role of their central brain in this process.
#### Depth of Detail: Seeks specific information on the function and control mechanism of the color-changing ability.

### Targeted Comparison of the Question with the Text:
#### Content Match: The text aligns with the first part of the question regarding the use of color-changing for camouflage and communication.
#### Depth Match: The text does not provide information about the role of the central brain in this process, lacking the required depth on the control mechanism.

### Critical Evaluation and Final Judgment:
Given the text's coverage of color-changing but lack of detail on the central brain's role, the overall assessment of the question's relevance to the text is: Irrelevant.

### Instruction:
Text:
\"\"\"
{qatuple[2]}
\"\"\"

Question (based on text): \"\"\"{qatuple[0]}\"\"\"

If the question clearly goes off the rails and is incoherent, then it is irrelevant.

### Response:
## Reasoning and thought process (be careful around "how" and "why" questions):
"""
        try:
            completion = logic_llm(
                decision_prompt,
                max_tokens=4000,
                stop=["</s>", "# Input:"],
                echo=True,
                grammar=question_relevant_grammar,
                temperature=0.2,
            )["choices"][0]["text"]

            response_pattern = re.compile(
                r"Reasoning and thought process \(be careful around \"how\" and \"why\" questions\):(.+)",
                re.DOTALL | re.IGNORECASE,
            )
            response = response_pattern.search(completion).group(1).strip()
            decision_pattern = re.compile(
                r"Final Judgment:(.+)", re.DOTALL | re.IGNORECASE
            )
            print(response)
            determination = decision_pattern.search(response).group(1).strip()
            print("\n\nDETERMINATION:\n------")
            print(determination)
            print("\n---------\n")
            if (
                "irrelevant" in determination
                or "Irrelevant" in determination.lower()
                or "mostly" in determination.lower()
                or "partial" in determination.lower()
                or "introduces information not present in the text"
                in determination.lower()
            ):
                return (False, response), completion
            elif "relevant" in determination or "Relevant" in determination:
                return (True, response), completion
            else:
                print("Did not contain relevant or irrelevant! Retrying")
                retries += 1
        except Exception as e:
            print("Exception!", e)
            if retries <= 4:
                retries += 1
            else:
                return (None, None), completion
    return (None, None), None

if __name__ == "__main__":  # test
    logic_llm = Llama(
        model_path=LOGICAL_MODEL,
        n_gqa=8,
        offload_kqv=True,
        n_ctx=4096,
        n_gpu_layers=1000,
    )  # load the logical LLM and offload everything
    q_test = [
        (
            "Explain how our understanding of planetary motion has changed over time.",
            "The understanding has evolved from the Earth being stationary and at the centre of the universe, to it orbiting the sun in an elliptical path with other planets while still rotating on its axis.",
            "The story of our world is a story that is still very imperfectly known. A couple of hundred years ago men possessed the history of little more than the last three thousand years. What happened before that time was a matter of legend and speculation.  Over a large part of the civilized world it was believed and taught that the world had been created suddenly in 4004 B.C., though authorities differed as to whether this had occurred in the spring or autumn of that year. This fantastically precise misconception was based upon a too literal interpretation of the Hebrew Bible, and upon rather arbitrary theological assumptions connected therewith.  Such ideas have long since been abandoned by religious teachers, and it is universally recognized that the universe in which we live has to all appearances existed for an enormous period of time and possibly for endless time.  Of course there may be deception in these appearances, as a room may be made to seem endless by putting mirrors facing each other at either end. But that the universe in which we live has existed only for six or seven thousand years may be regarded as an altogether exploded idea.\n\nThe earth, as everybody knows nowadays, is a spheroid, a sphere slightly compressed, orange fashion, with a diameter of nearly 8,000 miles.  Its spherical shape has been known at least to a limited number of intelligent people for nearly 2,500 years, but before that time it was supposed to be flat, and various ideas which now seem fantastic were entertained about its relations to the sky and the stars and planets.  We know now that it rotates upon its axis (which is about 24 miles shorter than its equatorial diameter) every twenty-four hours, and that this is the cause of the alternations of day and night, that it circles about the sun in a slightly distorted and slowly variable oval path in a year. Its distance from the sun varies between ninety-one and a half millions at its nearest and ninety-four and a half million miles.",
        ),
        (
            "Identify and explain changes in human understanding throughout history regarding the age of the Earth.",
            "Initially, religious texts suggested a young earth dating back no more than several thousand years. However, evidence from geology and astronomy has shown us that the earth is over four billion years old.",
            "The story of our world is a story that is still very imperfectly known. A couple of hundred years ago men possessed the history of little more than the last three thousand years. What happened before that time was a matter of legend and speculation.  Over a large part of the civilized world it was believed and taught that the world had been created suddenly in 4004 B.C., though authorities differed as to whether this had occurred in the spring or autumn of that year. This fantastically precise misconception was based upon a too literal interpretation of the Hebrew Bible, and upon rather arbitrary theological assumptions connected therewith.  Such ideas have long since been abandoned by religious teachers, and it is universally recognized that the universe in which we live has to all appearances existed for an enormous period of time and possibly for endless time.  Of course there may be deception in these appearances, as a room may be made to seem endless by putting mirrors facing each other at either end. But that the universe in which we live has existed only for six or seven thousand years may be regarded as an altogether exploded idea.\n\nThe earth, as everybody knows nowadays, is a spheroid, a sphere slightly compressed, orange fashion, with a diameter of nearly 8,000 miles.  Its spherical shape has been known at least to a limited number of intelligent people for nearly 2,500 years, but before that time it was supposed to be flat, and various ideas which now seem fantastic were entertained about its relations to the sky and the stars and planets.  We know now that it rotates upon its axis (which is about 24 miles shorter than its equatorial diameter) every twenty-four hours, and that this is the cause of the alternations of day and night, that it circles about the sun in a slightly distorted and slowly variable oval path in a year. Its distance from the sun varies between ninety-one and a half millions at its nearest and ninety-four and a half million miles.",


import re
from .check_qatuple_context_grammar import check_qatuple_context_grammar
from llama_cpp import Llama
from .constants import LOGICAL_MODEL

# For the reword step (ONLY USE IF JUDGEMENT IS REWORD, OTHERWISE WE JUST IGNORE THE LAST BIT OF THE GEN)
def extract_question_answer(response):
    # Define the regex pattern to match the question and answer
    pattern = r"### Question Rewording \(using text details as reference\):\nQuestion: (.+?)\nAnswer: (.+?)\n"

    # Search for the pattern in the response
    match = re.search(pattern, response)

    # Extract and return the question and answer if a match is found
    if match:
        question = match.group(1).strip()
        answer = match.group(2).strip()
        return question, answer
    else:
        return None, None

# A separate prompt for the reword step of checking qatuple context, since the grammar is bugged on the original
def check_qatuple_context_deprecated(qatuple, logic_llm):
    retries = 0
    while retries <= 4:
        decision_prompt = f"""You are checking whether a provided question and answer make sense if asked by themselves, with no additional information. You need to check for vague wording that a reader cannot interpret correctly, and questions that lack key context and would not be possibly answerable even if asked of someone with complete, masterful knowledge of the general subject matter of the question.

Evaluate the provided question-answer pair step-by-step. Following this, at the very end of your response, your "final judgment" or "final answer", you will write "Pass" or "Fail" or "Reword". A test passes if it "makes sense" and does not lack key context; it "Fails" if it lacks key context, AND the question is not specific or clear, it fails. If it lacks context but the question is specific, pointed, and grounded, then it needs to be reworded to have the context-needing terms (i.e., vague reference to "the text") removed. If it has no problems, it passes.

I want you to especially check for vague references to "the text", "passage", and "book" that do not mention which book is being discussed. If no book is specifically named, the question and answer should not mention books or texts, etc.

Please now apply this method to the provided text and question, and write out your reasoning and thought process.

### Instruction:
Text details: Simple Sabotage, by the Office of Strategic Services, Published 1944
Note that while you have access to this information, for the sake of rewording questions, you should evaluate the question as if you could not see this.

Question: How can you avoid blame for an act of sabotage, according to the text?
Answer: You can do them in public places where anyone would have been capable of carrying out the act.

### Response:
## Reasoning and thought process:
### Question Context Validation
#### Special Term Context Check: Specifically check for use of the terms "book", "text", "passage", and "excerpt" without context about which specific thing is being discussed. This question mentions "the text" without specifying which text it is referring to.
#### Text and Author Specificity: The question fails to mention which passage or book is being referred to, leading to a lack of context.
#### Scope and Precision: The question is specific in asking how to avoid blame for an act of sabotage.

### Answer Context Validation:
#### Special Term Context Check: Specifically check for use of the terms "book", "text", "passage", and "excerpt" without context about which specific thing is being discussed. The answer does not use any of these terms.
#### Specificity and Clarity: The answer is clear in its message and content. It does not introduce any vague context.
#### Answer-Only Context Issues: The answer is straightforward, makes no reference to outside material, and only requires knowledge to provide -- therefore, it is completely valid.

### Critical Evaluation and Final Judgment:
#### Evaluation: Some checks related to the question or answer failed. So this question and answer should be reworded if they can be, or fail otherwise. Both the question and answer are precise, but the question lacks specific context regarding the text it refers to. This necessitates a rewording to include the specific text. The answer passed its checks and will remain untouched.
#### Final judgment: Reword.

### Question Rewording (using text details as reference):
Question: How can you avoid blame for an act of sabotage, according to 'Simple Sabotage' by the Office of Strategic Services?
Answer: You can do them in public places where anyone would have been capable of carrying out the act.

### Instruction:
Text details: Simple Sabotage, By the Office of Strategic Services, Published 1867
Note that while you have access to this information, for the sake of rewording questions, you should evaluate the question as if you could not see this.

Question: How does the type of saboteur affect their role in destruction?
Answer: If they are a technician, they can devise methods of simple sabotage appropriate to their facilities. If not technically trained, they need suggestions for what to destroy and how to accomplish it.

### Response:
## Reasoning and thought process:
### Question Context Validation
#### Special Term Context Check: Specifically check for use of the terms "book", "text", "passage", and "excerpt" without context about which specific thing is being discussed. The question does not misuse any specific terms without proper context.
#### Text and Author Specificity: The question does not require a reference to a specific text or author as it is asking about general knowledge.
#### Scope and Precision: The question is precise in asking about the impact of a saboteur's type on their role.

### Answer Context Validation:
#### Special Term Context Check: Specifically check for use of the terms "book", "text", "passage", and "excerpt" without context about which specific thing is being discussed. The answer does not use vague terms without context.
#### Specificity and Clarity: The answer is clear, providing a direct response to the question based on general knowledge.
#### Answer-Only Context Issues: The answer does not introduce any vague or unspecified external material and is valid as it stands.

### Critical Evaluation and Final Judgment:
#### Evaluation: Both the question and answer are precise and do not require additional context for understanding.
#### Final judgment: Pass.

### Instruction:
Text details: Introduction to Philosophy, by George Stuart Fullerton
Note that while you have access to this information, for the sake of rewording questions, you should evaluate the question as if you could not see this.

Question: What is the meaning of this passage?
Answer: This passage means that things which think, form plans, and act on those plans, are beyond simple machines. This is evidenced by the line "Creatures that think, form plans, and _act_, are not what we call automata."

### Response:
## Reasoning and thought process:
### Question Context Validation
#### Special Term Context Check: Specifically check for use of the terms "book", "text", "passage", and "excerpt" without context about which specific thing is being discussed. The question asks about "this passage" without specifying which passage it is referring to or what book it belongs to.
#### Text and Author Specificity: The question fails to mention which passage or book is being referred to, leading to a lack of context.
#### Scope and Precision: The question asks about the meaning of a passage, which is a precise inquiry. However, when paired with a vague reference to "this passage", the question becomes too vague and broad, thus failing this step.

### Answer Context Validation:
#### Special Term Context Check: Specifically check for use of the terms "book", "text", "passage", and "excerpt" without context about which specific thing is being discussed. The answer references "this passage" without stating which passage it is talking about.
#### Specificity and Clarity: The answer states what the passage means but fails to clarify which specific passage from 'Introduction to Philosophy' by George Stuart Fullerton it is referring to.
#### Answer-Only Context Issues: The answer does not introduce new vague context that the question does not. However, both the question and answer lack necessary context.

### Critical Evaluation and Final Judgment:
#### Evaluation: Both the question and answer lack specific context, making it impossible to determine which passage from 'Introduction to Philosophy' they are referring to. The question is precise in asking for a meaning but fails due to lack of context.
#### Final judgment: Fail.

### Instruction:
Text details: {qatuple[3]}
Note that while you have access to this information, for the sake of rewording questions, you should evaluate the question as if you could not see this.

Question: {qatuple[0]}
Answer: {qatuple[1]}

### Response:
## Reasoning and thought process (be thorough):
"""
        # print("DEBUG\n\n" + decision_prompt)
        try:
            completion = logic_llm(
                decision_prompt,
                max_tokens=3000,
                stop=["</s>", "# Input:"],
                echo=True,
                grammar=check_qatuple_context_grammar,
                temperature=0.2,
            )["choices"][0]["text"]

            response_pattern = re.compile(
                r"Reasoning and thought process \(be thorough\):(.+)",
                re.DOTALL | re.IGNORECASE,
            )
            response = response_pattern.search(completion).group(1).strip()
            decision_pattern = re.compile(r"Final judgment:(.+)", re.IGNORECASE)
            print(response)
            determination = decision_pattern.search(response).group(1).strip()
            print("\n\nDETERMINATION:\n------")
            print(determination)
            print("\n---------\n")
            if "Reword" in determination or "reword" in determination:
                q, a = extract_question_answer(response)
                return (q, a, qatuple[2], qatuple[3])
            elif "Pass" in determination or "pass" in determination:
                return (True, response)
            elif "Fail" in determination or "fail" in determination:
                return (False, response)
            else:
                print("Did not contain relevant or irrelevant! Retrying")
                retries += 1
        except Exception as e:
            print("Exception!", e)
            if retries <= 4:
                retries += 1
            else:
                return (None, None), None
    return (None, None), None

if __name__ == "__main__":  # test
    logic_llm = Llama(
        model_path=LOGICAL_MODEL,
        n_gqa=8,
        offload_kqv=True,
        n_ctx=12000,
        rope_freq_scale=0.33,
        n_gpu_layers=100,
    )  # load the logical LLM and offload everything
    # NOTE: change these examples to have actual body text if you end up incorporating that into this step
    q_test = [
        (
            "What is the central philosophy presented in this book?",
            "The central philosophy is Stoicism, which advocates for living in harmony with nature and understanding that human happiness depends not on external events but on our own internal attitude and actions.",
            "fucking gauls",
            "Meditations, by Marcus Aurelius, Published 180 AD",
        ),
        (
            "What does the author argue in this part of the book?",
            "Plato argues for the philosopher-king as the ideal ruler, who possesses both wisdom and moral virtue.",
            "fucking sophists",
            "The Republic, by Plato",
        ),
        (
            "How does Darwin explain natural selection?",
            "Darwin explains natural selection as a process where organisms better adapted to their environment tend to survive and produce more offspring. This theory suggests that traits beneficial for survival are more likely to be passed on to subsequent generations.",
            "fucking Lamarckism",
            "The Origin of Species, by Charles Darwin",
        ),
    ]

    print("Begin variety test")
    # Try to detect bad question
    d = check_qatuple_context(q_test[0], logic_llm)
    if d[0] == "New QA Tuple":  # if not relevant
        print("Made right choice for rewordable question")
    else:
        print("Made wrong choice for rewordable question")
    d2 = check_qatuple_context(q_test[1], logic_llm)
    if not d2[0]:
        print("Made right choice for bad question")
    else:
        print("Made wrong choice for bad question")
    d3 = check_qatuple_context(q_test[2], logic_llm)
    if d3[0]:
        print("Made right choice for good question")
    else:
        print("Made wrong choice for good question")

    print("Begin Mendeleev test")
    # NOTE I should actually do a mendeleev test, to see if including examples from that text has screwed it
    text2 = """A substance or material is that which occupies space and has
      weight; that is, which presents a mass attracted by the earth and
      by other masses of material, and of which the _objects_ of nature
      are composed, and by means of which the motions and _phenomena_
      of nature are accomplished. It is easy to discover by examining
      and investigating, by various methods, the objects met with
      in nature and in the arts, that some of them are homogeneous,
      whilst others are composed of a mixture of several homogeneous
      substances. This is most clearly apparent in solid substances.
      The metals used in the arts (for example, gold, iron, copper)
      must be homogeneous, otherwise they are brittle and unfit for
      many purposes. Homogeneous matter exhibits similar properties in
      all its parts. By breaking up a homogeneous substance we obtain
      parts which, although different in form, resemble each other in
      their properties. Glass, pure sugar, marble, &c., are examples of
      homogeneous substances. Examples of non-homogeneous substances
      are, however, much more frequent in nature and the arts. Thus
      the majority of the rocks are not homogeneous. In porphyries
      bright pieces of a mineral called 'orthoclase' are often seen
      interspersed amongst the dark mass of the rock. In ordinary red
      granite it is easy to distinguish large pieces of orthoclase mixed
      with dark semi-transparent quartz and flexible laminæ of mica.
      Similarly, plants and animals are non-homogeneous. Thus, leaves
      are composed of a skin, fibre, pulp, sap, and a green colouring
      matter. As an example of those non-homogeneous substances which
      are produced artificially, gunpowder may be cited, which is
      prepared by mixing together known proportions of sulphur, nitre,
      and charcoal. Many liquids, also, are not homogeneous, as may be
      observed by the aid of the microscope, when drops of blood are
      seen to consist of a colourless liquid in which red corpuscles,
      invisible to the naked eye owing to their small size, are floating
      about. It is these corpuscles which give blood its peculiar
      colour. Milk is also a transparent liquid, in which microscopical
      drops of fat are floating, which rise to the top when milk is
      left at rest, forming cream. It is possible to extract from every
      non-homogeneous substance those homogeneous substances of which
      it is made up. Thus orthoclase may he separated from porphyry by
      breaking it off. So also gold is extracted from auriferous sand by
      washing away the mixture of clay and sand. Chemistry deals only
      with the homogeneous substances met with in nature, or extracted
      from natural or artificial non-homogeneous substances. The various
      mixtures found in nature form the subjects of other natural
      sciences--as geognosy, botany, zoology, anatomy, &c."""
    q_test_2 = [  # note that the full text isn't included in each of the tuples here, I need to  change that
        (
            "Why is it important to distinguish between homogeneous and non-homogeneous substances?",
            "Homogeneous substances consist of parts that resemble each other in their properties, while non-homogeneous substances are made up of several homogeneous substances mixed together. Chemistry deals with the homogeneous substances met with in nature or extracted from natural or artificial non-homogeneous substances, so it is important to distinguish between them because it determines which parts of a given substance can be used for chemical analysis and study.",
        ),
        (
            "What is an example of an artistic mixture that would be non-homogeneous?",
            "An example of a non-homogeneous artistic mixture could be gunpowder, which is prepared by mixing together known proportions of sulphur, nitre, and charcoal.",
        ),
        (
            "How might the concept of homogeneity apply to education or learning?",
            "In education or learning, students can think about their own knowledge as a homogeneous substance, made up of similar concepts that resemble each other in terms of understanding. They may need to separate out these ideas from non-homogeneous ones (e.g., misconceptions) in order to fully grasp the concept and build upon it.",
        ),
        (
            "If we were told to find homogeneous substances in nature, how would we go about doing this?",
            "To find homogeneous substances in nature, one could examine and investigate various objects met with in nature and in the arts. Some of these objects might be homogeneous, whilst others are composed of a mixture of several homogeneous substances. By breaking up a homogeneous substance, we would obtain parts which, although different in form, resemble each other in their properties. This suggests that we could identify homogeneous substances by looking for these characteristics. Additionally, some examples mentioned in the text include gold, iron, copper, glass, pure sugar, marble, and ordinary red granite. However, not all non-homogeneous substances are immediately apparent; it requires investigating and understanding how they are made up of different components (such as orthoclase being separated from porphyry). Therefore, a combination of observing physical properties, breaking down materials, and understanding their composition would allow us to identify homogeneous substances in nature.",
        ),
    ]


import re
from .check_qatuple_context_grammar import check_qatuple_context_grammar
from llama_cpp import Llama
from .constants import LOGICAL_MODEL

# For the reword step (ONLY USE IF JUDGEMENT IS REWORD, OTHERWISE WE JUST IGNORE THE LAST BIT OF THE GEN)
def extract_question_answer(response):
    # Define the regex pattern to match the question and answer
    pattern = r"### Question Rewording \(using text details as reference\):\nQuestion: (.+?)\nAnswer: (.+?)\n"

    # Search for the pattern in the response
    match = re.search(pattern, response)

    # Extract and return the question and answer if a match is found
    if match:
        question = match.group(1).strip()
        answer = match.group(2).strip()
        return question, answer
    else:
        return None, None

# A separate prompt for the reword step of checking qatuple context, since the grammar is bugged on the original
def check_qatuple_context_deprecated(qatuple, logic_llm):
    retries = 0
    while retries <= 4:
        decision_prompt = f"""You are checking whether a provided question and answer make sense if asked by themselves, with no additional information. You need to check for vague wording that a reader cannot interpret correctly, and questions that lack key context and would not be possibly answerable even if asked of someone with complete, masterful knowledge of the general subject matter of the question.

Evaluate the provided question-answer pair step-by-step. Following this, at the very end of your response, your "final judgment" or "final answer", you will write "Pass" or "Fail" or "Reword". A test passes if it "makes sense" and does not lack key context; it "Fails" if it lacks key context, AND the question is not specific or clear, it fails. If it lacks context but the question is specific, pointed, and grounded, then it needs to be reworded to have the context-needing terms (i.e., vague reference to "the text") removed. If it has no problems, it passes.

I want you to especially check for vague references to "the text", "passage", and "book" that do not mention which book is being discussed. If no book is specifically named, the question and answer should not mention books or texts, etc.

Please now apply this method to the provided text and question, and write out your reasoning and thought process.

### Instruction:
Text details: Simple Sabotage, by the Office of Strategic Services, Published 1944
Note that while you have access to this information, for the sake of rewording questions, you should evaluate the question as if you could not see this.

Question: How can you avoid blame for an act of sabotage, according to the text?


# For the reword step (ONLY USE IF JUDGEMENT IS REWORD, OTHERWISE WE JUST IGNORE THE LAST BIT OF THE GEN)
def extract_question_answer(response):
    # Define the regex pattern to match the question and answer
    pattern = r"### Question Rewording \(using text details as reference\):\nQuestion: (.+?)\nAnswer: (.+?)\n"

    # Search for the pattern in the response
    match = re.search(pattern, response)

    # Extract and return the question and answer if a match is found
    if match:
        question = match.group(1).strip()
        answer = match.group(2).strip()
        return question, answer
    else:
        return None, None

# A separate prompt for the reword step of checking qatuple context, since the grammar is bugged on the original
def check_qatuple_context_deprecated(qatuple, logic_llm):
    retries = 0
    while retries <= 4:
        decision_prompt = f"""You are checking whether a provided question and answer make sense if asked by themselves, with no additional information. You need to check for vague wording that a reader cannot interpret correctly, and questions that lack key context and would not be possibly answerable even if asked of someone with complete, masterful knowledge of the general subject matter of the question.

Evaluate the provided question-answer pair step-by-step. Following this, at the very end of your response, your "final judgment" or "final answer", you will write "Pass" or "Fail" or "Reword". A test passes if it "makes sense" and does not lack key context; it "Fails" if it lacks key context, AND the question is not specific or clear, it fails. If it lacks context but the question is specific, pointed, and grounded, then it needs to be reworded to have the context-needing terms (i.e., vague reference to "the text") removed. If it has no problems, it passes.

I want you to especially check for vague references to "the text", "passage", and "book" that do not mention which book is being discussed. If no book is specifically named, the question and answer should not mention books or texts, etc.

Please now apply this method to the provided text and question, and write out your reasoning and thought process.

### Instruction:
Text details: Simple Sabotage, by the Office of Strategic Services, Published 1944
Note that while you have access to this information, for the sake of rewording questions, you should evaluate the question as if you could not see this.

Question: How can you avoid blame for an act of sabotage, according to the text?
Answer: You can do them in public places where anyone would have been capable of carrying out the act.

### Response:
## Reasoning and thought process:
### Question Context Validation
#### Special Term Context Check: Specifically check for use of the terms "book", "text", "passage", and "excerpt" without context about which specific thing is being discussed. This question mentions "the text" without specifying which text it is referring to.
#### Text and Author Specificity: The question fails to mention which passage or book is being referred to, leading to a lack of context.
#### Scope and Precision: The question is specific in asking how to avoid blame for an act of sabotage.

### Answer Context Validation:
#### Special Term Context Check: Specifically check for use of the terms "book", "text", "passage", and "excerpt" without context about which specific thing is being discussed. The answer does not use any of these terms.
#### Specificity and Clarity: The answer is clear in its message and content. It does not introduce any vague context.
#### Answer-Only Context Issues: The answer is straightforward, makes no reference to outside material, and only requires knowledge to provide -- therefore, it is completely valid.

### Critical Evaluation and Final Judgment:
#### Evaluation: Some checks related to the question or answer failed. So this question and answer should be reworded if they can be, or fail otherwise. Both the question and answer are precise, but the question lacks specific context regarding the text it refers to. This necessitates a rewording to include the specific text. The answer passed its checks and will remain untouched.
#### Final judgment: Reword.

### Question Rewording (using text details as reference):
Question: How can you avoid blame for an act of sabotage, according to 'Simple Sabotage' by the Office of Strategic Services?
Answer: You can do them in public places where anyone would have been capable of carrying out the act.

### Instruction:
Text details: Simple Sabotage, By the Office of Strategic Services, Published 1867
Note that while you have access to this information, for the sake of rewording questions, you should evaluate the question as if you could not see this.

Question: How does the type of saboteur affect their role in destruction?
Answer: If they are a technician, they can devise methods of simple sabotage appropriate to their facilities. If not technically trained, they need suggestions for what to destroy and how to accomplish it.

### Response:
## Reasoning and thought process:
### Question Context Validation
#### Special Term Context Check: Specifically check for use of the terms "book", "text", "passage", and "excerpt" without context about which specific thing is being discussed. The question does not misuse any specific terms without proper context.
#### Text and Author Specificity: The question does not require a reference to a specific text or author as it is asking about general knowledge.
#### Scope and Precision: The question is precise in asking about the impact of a saboteur's type on their role.

### Answer Context Validation:
#### Special Term Context Check: Specifically check for use of the terms "book", "text", "passage", and "excerpt" without context about which specific thing is being discussed. The answer does not use vague terms without context.
#### Specificity and Clarity: The answer is clear, providing a direct response to the question based on general knowledge.
#### Answer-Only Context Issues: The answer does not introduce any vague or unspecified external material and is valid as it stands.

### Critical Evaluation and Final Judgment:
#### Evaluation: Both the question and answer are precise and do not require additional context for understanding.
#### Final judgment: Pass.

### Instruction:
Text details: Introduction to Philosophy, by George Stuart Fullerton
Note that while you have access to this information, for the sake of rewording questions, you should evaluate the question as if you could not see this.

Question: What is the meaning of this passage?
Answer: This passage means that things which think, form plans, and act on those plans, are beyond simple machines. This is evidenced by the line "Creatures that think, form plans, and _act_, are not what we call automata."

### Response:
## Reasoning and thought process:
### Question Context Validation
#### Special Term Context Check: Specifically check for use of the terms "book", "text", "passage", and "excerpt" without context about which specific thing is being discussed. The question asks about "this passage" without specifying which passage it is referring to or what book it belongs to.
#### Text and Author Specificity: The question fails to mention which passage or book is being referred to, leading to a lack of context.
#### Scope and Precision: The question asks about the meaning of a passage, which is a precise inquiry. However, when paired with a vague reference to "this passage", the question becomes too vague and broad, thus failing this step.

### Answer Context Validation:
#### Special Term Context Check: Specifically check for use of the terms "book", "text", "passage", and "excerpt" without context about which specific thing is being discussed. The answer references "this passage" without stating which passage it is talking about.
#### Specificity and Clarity: The answer states what the passage means but fails to clarify which specific passage from 'Introduction to Philosophy' by George Stuart Fullerton it is referring to.
#### Answer-Only Context Issues: The answer does not introduce new vague context that the question does not. However, both the question and answer lack necessary context.

### Critical Evaluation and Final Judgment:
#### Evaluation: Both the question and answer lack specific context, making it impossible to determine which passage from 'Introduction to Philosophy' they are referring to. The question is precise in asking for a meaning but fails due to lack of context.
#### Final judgment: Fail.

### Instruction:
Text details: {qatuple[3]}
Note that while you have access to this information, for the sake of rewording questions, you should evaluate the question as if you could not see this.

Question: {qatuple[0]}
Answer: {qatuple[1]}

### Response:
## Reasoning and thought process (be thorough):
"""
        # print("DEBUG\n\n" + decision_prompt)
        try:
            completion = logic_llm(
                decision_prompt,
                max_tokens=3000,
                stop=["</s>", "# Input:"],
                echo=True,
                grammar=check_qatuple_context_grammar,
                temperature=0.2,
            )["choices"][0]["text"]

            response_pattern = re.compile(
                r"Reasoning and thought process \(be thorough\):(.+)",
                re.DOTALL | re.IGNORECASE,
            )
            response = response_pattern.search(completion).group(1).strip()
            decision_pattern = re.compile(r"Final judgment:(.+)", re.IGNORECASE)
            print(response)
            determination = decision_pattern.search(response).group(1).strip()
            print("\n\nDETERMINATION:\n------")
            print(determination)
            print("\n---------\n")
            if "Reword" in determination or "reword" in determination:
                q, a = extract_question_answer(response)
                return (q, a, qatuple[2], qatuple[3])
            elif "Pass" in determination or "pass" in determination:
                return (True, response)
            elif "Fail" in determination or "fail" in determination:
                return (False, response)
            else:
                print("Did not contain relevant or irrelevant! Retrying")
                retries += 1
        except Exception as e:
            print("Exception!", e)
            if retries <= 4:
                retries += 1
            else:
                return (None, None), None
    return (None, None), None

if __name__ == "__main__":  # test
    logic_llm = Llama(
        model_path=LOGICAL_MODEL,
        n_gqa=8,
        offload_kqv=True,
        n_ctx=12000,
        rope_freq_scale=0.33,
        n_gpu_layers=100,
    )  # load the logical LLM and offload everything
    # NOTE: change these examples to have actual body text if you end up incorporating that into this step
    q_test = [
        (
            "What is the central philosophy presented in this book?",
            "The central philosophy is Stoicism, which advocates for living in harmony with nature and understanding that human happiness depends not on external events but on our own internal attitude and actions.",
            "fucking gauls",
            "Meditations, by Marcus Aurelius, Published 180 AD",
        ),
        (


from llama_cpp import LlamaGrammar

make_regenerate_answer_plan_grammar = LlamaGrammar.from_string(
    r"""
root ::= analyze-step understand-step identify-step plan-step

analyze-step ::= "Step " [0-9]?[0-9] ". " "Analyze the Text:" [^\n]+ "\n"

understand-step ::= "Step " [0-9]?[0-9] ". " "Understand the Question:" [^\n]+ "\n"

identify-step ::= "Step " [0-9]?[0-9] ". " "Identify the Incorrect Part of the Answer:" [^\n]+ "\n"

plan-step ::= "Step " [0-9]?[0-9] ". " "Plan a Corrected Answer:" [^\n]+ "\n"
"""
)

# question_grammar = LlamaGrammar.from_string(r"""# GBNF Grammar for Q&A Format with Flexible Punctuation

# root ::= answer
# answer ::= "Test"
# """)


from llama_cpp import LlamaGrammar

regenerate_answer_constrain_to_text_grammar = LlamaGrammar.from_string(
    r"""

root ::= reasoning

reasoning ::= [^\n]+ "\"\"\""
"""
)


import re
from .questions_grammar import questions_grammar
from llama_cpp import Llama
from .constants import LOGICAL_MODEL

# Answer generation code, begin.
# structure: define a series of helpers, then define the control flow, exception handling, retries etc. in a for loop that iterates over the processed sequences of paragraphs at the end in another cell

# Since some paragraphs can be much shorter
# First off, question generation.

# Each local LLM function essentially has 3 phases: prompt, regex to extract response, and reaction to that response.
# However I'm not going to build an abstraction for that because I need fine control.

# If any function fails to make things, it won't throw, it'll just return None.

# Strengths of open source AI: hella cheap, very customizable, you can call it as much as you want
# Downside: you need very good regexes to catch its outputs

def generate_short_questions(text, existing_question_tuples, logic_llm):
    """
    Produce a list of questions based off of an input text. The min between (4, as many good questions as the text permits)

    Format: Question: [question]\n\n
    """
    # Determine which paragraphs are worthy of making questions from
    made_questions = False
    retries = 0
    questions = []
    if existing_question_tuples:
        existing_qs = "\n".join(
            [
                "Existing question: " + t[0] + "\nAnswer: " + t[1]
                for t in existing_question_tuples
            ]
        )  # stop duplicates from being generated
    else:
        existing_qs = None
    while not made_questions and (retries <= 5):
        question_prompt = f"""You are an expert educational AI that, given a paragraph or two from a text, will create suitable educational questions based on the paragraphs. The questions you create will lean towards shorter questions that can be quickly answered with only a bit of thought — and which can be solved if the answerer knows the paragraphs provided by heart. Essentially: the question will test comprehension of real information in the paragraphs that would be worthy to teach. After the question, you will also write its answer. Your task includes first analyzing the text, thinking through and brainstorming which questions you will make and why.

Each question you write (after your reasoning step is complete), MUST start on a new line with its question number followed by a bracket, ie, 1), or 2). This will then be followed by the question. This MUST be followed by "Answer: " followed by the question's answer. Each question must be separated by at least one new line.

Some longer-form questions have already been generated from this text. You are to avoid reiterating exactly any of the questions present in this list:
\"\"\"
{existing_qs}
\"\"\"

Text to make questions from: \"\"\"{text}\"\"\"

You should aim to make 6 questions (at most), but if the text is too small or information-sparse for that many, you are allowed to write fewer. Do not explicitly mention the paragraphs OR "the text" in the questions themselves — just ask about the concepts related to the paragraphs.

You will not mention the text explicitly in any questions you think of, since the questions you generate are intended to test people's knowledge of the information — when given the questions, they will not have the text on-hand.

### Response: I will remember to follow the question and answer format given by:
\"\"\"
num) question contents.
Answer: question answer.
\"\"\"
All my questions will be directly answerable from the provided paragraphs, and will not rely on knowledge outside of what has been provided. Each of my short questions will have its answer written below it on a new line.

## Questions:
"""
        completion = logic_llm(
            question_prompt,
            max_tokens=2000,
            stop=["</s>", "# Input:"],
            echo=True,
            grammar=questions_grammar,
            temperature=0.2,
        )["choices"][0]["text"]

        # Extract questions
        response_pattern = re.compile(r"Questions:\n(.+)", re.IGNORECASE | re.DOTALL)
        generation = response_pattern.search(completion).group(1)
        # print("GENERATION:\n\n-------------------\n\n", generation)
        pattern = re.compile(
            r"(?:Question:|^\d+[\).]?)\s*(.*?)\s*\n*Answer:\s*(.*?)(?=(?:\n\s*(?:Question:|\d+[\).]?))|$)",
            re.DOTALL | re.MULTILINE | re.IGNORECASE,
        )
        matches = pattern.findall(generation)
        print("GENERATION:\n\n-------------------\n\n", matches)
        if len(matches) > 0:
            made_questions = True
        else:
            retries += 1
    if retries > 5:
        return None

    for match in matches:
        questions.append((match[0].strip(), match[1].strip(), text))

    return questions

if __name__ == "__main__":  # test
    logic_llm = Llama(
        model_path=LOGICAL_MODEL,
        n_gqa=8,
        offload_kqv=True,
        n_ctx=4096,
        n_gpu_layers=1000,
    )  # load the logical LLM and offload everything
    text = """The story of our world is a story that is still very imperfectly known. A couple of hundred years ago men possessed the history of little more than the last three thousand years. What happened before that time was a matter of legend and speculation.  Over a large part of the civilized world it was believed and taught that the world had been created suddenly in 4004 B.C., though authorities differed as to whether this had occurred in the spring or autumn of that year. This fantastically precise misconception was based upon a too literal interpretation of the Hebrew Bible, and upon rather arbitrary theological assumptions connected therewith.  Such ideas have long since been abandoned by religious teachers, and it is universally recognized that the universe in which we live has to all appearances existed for an enormous period of time and possibly for endless time.  Of course there may be deception in these appearances, as a room may be made to seem endless by putting mirrors facing each other at either end. But that the universe in which we live has existed only for six or seven thousand years may be regarded as an altogether exploded idea.

The earth, as everybody knows nowadays, is a spheroid, a sphere slightly compressed, orange fashion, with a diameter of nearly 8,000 miles.  Its spherical shape has been known at least to a limited number of intelligent people for nearly 2,500 years, but before that time it was supposed to be flat, and various ideas which now seem fantastic were entertained about its relations to the sky and the stars and planets.  We know now that it rotates upon its axis (which is about 24 miles shorter than its equatorial diameter) every twenty-four hours, and that this is the cause of the alternations of day and night, that it circles about the sun in a slightly distorted and slowly variable oval path in a year. Its distance from the sun varies between ninety-one and a half millions at its nearest and ninety-four and a half million miles."""
    print("Begin HGWELLS test")
    result = generate_short_questions(text, logic_llm)


from llama_cpp import LlamaGrammar

### A grammar that forces the model to generate correct character cards (with traits, names, everything)

scenario_grammar = LlamaGrammar.from_string(
    r"""

root ::= reasoning-start

reasoning-start ::= [^\n\t]+ "."

"""
)


from llama_cpp import LlamaGrammar

judge_paragraph_grammar = LlamaGrammar.from_string(
    r"""

root ::= identify-content-step evaluate-relevance-step assess-contexts-and-formats-step assess-possibility-step determine-suitability-step check-contextual-completeness-step final-step "\n"

identify-content-step ::= "Step " [0-9]?[0-9] ". " "Identify Paragraph Content: " [^\n]+ "\n"

evaluate-relevance-step ::= "Step " [0-9]?[0-9] ". " "Evaluate Educational Relevance: " [^\n]+ "\n"

assess-contexts-and-formats-step ::= "Step " [0-9]?[0-9] ". " "Assess Specific Contexts and Formats:" "\n" context-format-bullets

assess-possibility-step ::= "Step " [0-9]?[0-9] ". " "Assess the Possibility of Formulating Questions: " [^\n]+ "\n"

determine-suitability-step ::= "Step " [0-9]?[0-9] ". " "Determine Suitability for Educational Purposes: " [^\n]+ "\n"

check-contextual-completeness-step ::= "Step " [0-9]?[0-9] ". " "Check for Contextual Completeness: " [^\n]+ "\n"

final-step ::= "Step " [0-9]?[0-9] ". " "Final Judgment: " ("Unsuitable" | "Suitable" | "suitable" | "unsuitable") "\n"

context-format-bullets ::= bullet-item+
bullet-item ::= "  - " bullet-item-detail "\n"
bullet-item-detail ::= [^\n]+
"""
)


from llama_cpp import LlamaGrammar

ensure_answer_consistent_grammar = LlamaGrammar.from_string(
    r"""

root ::= understand-question-step compare-question-step understand-answer-step compare-step final-step "\n"

# step ::= "Step " [0-9]?[0-9] ". " ( "Analyze" | "Understand" | "Compare" | "Skip" | "Final" ) [^\n]+ "\n"

understand-question-step ::= "Step " [0-9]?[0-9] ". " "Understand the provided question:" [^\n]+ "\n"

compare-question-step ::= "Step " [0-9]?[0-9] ". " "Compare the conversation's question: " [^\n]+ "\n"

understand-answer-step ::= "Step " [0-9]?[0-9] ". " "Understand the provided answer:" [^\n]+ "\n"

# compare-step ::= "Step " [0-9]?[0-9] ". " "Compare the " ("first" | "second" | "third" | "fourth" | "fifth" | "sixth" | "seventh" | "eighth" | "ninth") " Part of the Answer with the Text: check if the text " [^\n]+ "\n"

compare-step ::= "Step " [0-9]?[0-9] ". " "Compare the conversation's answer:" [^\n]+ "\n"

final-step ::= "Step " [0-9]?[0-9] ". " "Final Judgement: " ("Inconsistent" | "Consistent") "\n"
"""
)
# question_grammar = LlamaGrammar.from_string(r"""# GBNF Grammar for Q&A Format with Flexible Punctuation

# root ::= answer
# answer ::= "Test"
# """)


from itertools import product
import random

def combine_traits(personality_matrix):  # GPT-generated
    # Using itertools.product to generate all possible combinations
    combinations = product(*personality_matrix)

    # Joining each combination into a single string
    combined_traits = [
        "\n".join(combination).strip().replace("\n\n", "\n")
        for combination in combinations
    ]

    return combined_traits

def special_instructions(n=1, non_axis_traits=False, non_axis_traits_only=False):
    """
    documentation todo
    """

    ### NOTE on how traits are planned out for this step ###
    # Here're the copy-pasted thoughts from my planning document, now slightly cleaned-up for the release of Augmentoolkit. The TLDR is at the bottom. The inspiration for this personality system is the main thing I gained from my math class this semester.
    # CHARACTER PLANNING
    # Consider that we can represent a character's personality a vector with multiple dimensions. Now, we could define any number of individual dimensions, and lots of them would be right: intelligence, extraversion, industriousness, etc. But in the default version of the Augmentool we're doing roleplay, so we want to pick a set of dimensions using which we can describe accurately and concisely the characters that might show up in a roleplay. Consider that if a personality trait is a vector in 3-space, we want to pick traits that aren't coplanar -- ie, that each describe something unique, though possibly with some partial overlap. Ideally, they'd all be perpendicular -- maximally unique traits.
    # I believe I have found 3 such axes that are useful for roleplay:
    # Assertiveness
    # Kindness/Morality
    # Horniness (one of the few things we have an edge over GPT in)
    # So we have
    # Chaste------------------------------------normal----------------------------------------------------------------Slaanesh
    # Shy/Withdrawn/Timid (Bocchi)--------------Has moments of shyness and courage------------------------------------James Bond
    # Kind--------------------------------------Good and bad sides ---------------------------------------------------politician
    # We make more verbose descriptions of each trait and place them in a matrix, reflecting the visualization above. We then create a list of all possible combinations of one item from each row and randomly sample from it for the special instruction.

    # NOTE TLDR In laymans terms: we make a grid of traits, where each row represents a trait and values along it indicate different expressions of that trait; then we pick one value from each row and shove it onto the context window as a "special instruction".

    # Two additional dimensions I thought of afterwards but have never tested: intellectual sophistication, and age. I might add these if testing shows that the AI can handle them, but no few-shot example has anywhere near 5 combinations, so we'll see.

    ## NOTE You may (and are encouraged to!) add your own trait dimensions here, to make the character personalities used more accurately reflect your specific use case and preference. Since every possible combination of one trait from each row is put into the list, you will get a lot of variety with your characters for not much work.
    # NOTE Chaste and puritan characters have a tendency to be interpreted by the AI as being religious, possibly because of "puritan", even though I initially just meant for this to be the opposite of horny. I'm leaving this in as a way to counteract occasional anti-religious bias and the AI's own personality.

    axis_traits = [
        [
            "The character should be chaste and puritanical.",
            "",
            "The character should be very seductive and flirtatious.",
        ],  # Horniness (middle deliberately left blank so that the model does not mention it, since "normal" people don't usually bring up sex in common conversation... right?)
        [
            "The character should be shy, withdrawn, and timid.",
            "The character should be neither particularly bold, nor particularly timid.",
            "The character should be assertive and bold.",
        ],  # Assertiveness
        [
            "The character should be kind and agreeable.",
            "The character should have both good and bad sides.",
            "The character should be an awful person, and should be enjoying every second of it."
            # "The character should be an awful person, possessing a number of vices (that are compatible with the previously-mentioned instructions)."
        ],  # Kindness/Morality
        # ["The character should be a young adult.", "the character should be middle-aged." "The character should be in late adulthood."], # Age group
        # ["The character should be unsophisticated and crude.", "The character should be decently smart and refined.", "The character should be the epitome of intellectual sophistication."],
    ]

    non_axis_trait_list = [  # The following are examples of traits that are not on the axes above, but are still useful for character creation. Typically use these if you want to easily hardcode your characters to all have a trait. I've not tested all of them, and I've not tested them in combination with the axis traits. But if you prefer a more manual approach to character creation, you can use stuff like this.
        """The character should be a catgirl who inserts "nya" into every sentence. and makes cat puns.""",  # someone actually has to do this, I'm serious, it'll be purrfect, nya~
        # They can be short and used in combination with the axis traits; or long and replace them.
        """The character should be a Japanese High School student.
The character should be a girl.
The character should be decently smart, but not genius-level.
The character should be very kind, but too gentle and too much of a pushover for their own good.""",
        """The character should be an awful person, and enjoying every second of it.
The character should be intellectually brilliant.
The character should be condescending and rude.""",
        """The character should be a young adult.
The character should be antisocial and coarse.
The character should be a smoker."""
        """The character should be middle-aged.
The character should be narcissistic."""
        # """The character should be edgy and nihilistic."""
    ]

    if not non_axis_traits_only:
        traits = combine_traits(axis_traits)

        selected_traits = random.sample(traits, 1)
        if non_axis_traits:
            selected_traits += random.sample(non_axis_trait_list, 1)

    if non_axis_traits_only:
        selected_traits = random.sample(non_axis_trait_list, 1)

    # Return the combined string, with each sentence on a new line
    return selected_traits[0]


from itertools import product
import random

def combine_traits(personality_matrix):  # GPT-generated
    # Using itertools.product to generate all possible combinations
    combinations = product(*personality_matrix)

    # Joining each combination into a single string
    combined_traits = [
        "\n".join(combination).strip().replace("\n\n", "\n")
        for combination in combinations
    ]

    return combined_traits

def special_instructions(n=1, non_axis_traits=False, non_axis_traits_only=False):
    """
    documentation todo
    """

    ### NOTE on how traits are planned out for this step ###
    # Here're the copy-pasted thoughts from my planning document, now slightly cleaned-up for the release of Augmentoolkit. The TLDR is at the bottom. The inspiration for this personality system is the main thing I gained from my math class this semester.
    # CHARACTER PLANNING
    # Consider that we can represent a character's personality a vector with multiple dimensions. Now, we could define any number of individual dimensions, and lots of them would be right: intelligence, extraversion, industriousness, etc. But in the default version of the Augmentool we're doing roleplay, so we want to pick a set of dimensions using which we can describe accurately and concisely the characters that might show up in a roleplay. Consider that if a personality trait is a vector in 3-space, we want to pick traits that aren't coplanar -- ie, that each describe something unique, though possibly with some partial overlap. Ideally, they'd all be perpendicular -- maximally unique traits.
    # I believe I have found 3 such axes that are useful for roleplay:
    # Assertiveness
    # Kindness/Morality
    # Horniness (one of the few things we have an edge over GPT in)
    # So we have
    # Chaste------------------------------------normal----------------------------------------------------------------Slaanesh
    # Shy/Withdrawn/Timid (Bocchi)--------------Has moments of shyness and courage------------------------------------James Bond
    # Kind--------------------------------------Good and bad sides ---------------------------------------------------politician
    # We make more verbose descriptions of each trait and place them in a matrix, reflecting the visualization above. We then create a list of all possible combinations of one item from each row and randomly sample from it for the special instruction.