import re

# try:
from .question_grammar import question_grammar
from llama_cpp import Llama
from .constants import LOGICAL_MODEL

def generate_new_question(qatuple, logic_llm):
    """
    Produce a list of questions based off of an input text. The min between (4, as many good questions as the text permits)

    Format: Question: [question]\n\n
    """
    # Determine which paragraphs are worthy of making questions from
    made_questions = False
    retries = 0
    questions = []
    while not made_questions and (
        retries <= 5
    ):  # TODO - UPDATE and TEST the few-shot prompt with the latest from generate_questions
        question_prompt = f"""You are an expert educational AI that, given a paragraph or two from a text, will create a suitable educational question based on the paragraphs, and *only* based on the paragraphs. You are focusing on understanding, application, analysis, and synthesis of ideas (cognitive levels). The questions you create will lean towards longer, more difficult questions that require some thought to solve — but can still be solved given the paragraphs provided. Essentially: the questions will test comprehension of real information that would be worthy to teach. After the question, you will also write its answer.

Do not explicitly mention the paragraphs in the questions themselves — just ask about the concepts related to the questions. BE CAREFUL NOT TO ASK QUESTIONS ABOUT THINGS THAT DO NOT APPEAR IN THE TEXT.

You will not mention the text explicitly in any questions you think of, since the questions you generate are intended to test people's knowledge of the information — when given the questions, they will not have the text on-hand.

### Instruction:
Text details: Road Construction, by Mark Ericsson

Text to make a question from:
\"\"\"
Road construction is a multifaceted process involving various stages and materials, each critical for the durability and safety of the road. Initially, a thorough site survey and soil testing are conducted to assess the suitability of the terrain. Following this, the groundwork commences with the removal of topsoil and leveling of the area. Subsequently, a layer of sub-base material, typically composed of crushed stone or gravel, is laid to provide stability. This is followed by the base layer, often made of a stronger aggregate, to support the surface layer. The surface layer, usually asphalt or concrete, is then applied, offering a smooth and durable driving surface. Additionally, proper drainage systems are installed to prevent water accumulation, which can lead to road damage. Throughout the construction, environmental considerations are taken into account to minimize the impact on surrounding ecosystems. Regular maintenance, including patching and resurfacing, is essential to extend the road's lifespan and ensure safety for its users.
\"\"\"

### Response:
## Question:
1.) What is the purpose of conducting a site survey and soil testing in the initial stage of road construction?
Answer: The site survey and soil testing are conducted to assess the suitability of the terrain for road construction, ensuring the area is appropriate and will support the road structure effectively.

### Instruction:
Text details: Introduction to Mathematics, by Elise Delacroix

Text to make a question from:
\"\"\"
In mathematics, the concept of a 'function' is fundamental, defining a relationship where each input is associated with exactly one output. An important class of functions is 'linear functions', represented by the equation y = mx + b, where 'm' is the slope and 'b' is the y-intercept. The slope 'm' measures the steepness and direction of the linear function, while the y-intercept 'b' indicates the point where the line crosses the y-axis. Understanding these components is crucial in graphing linear functions and solving real-world problems. Another vital concept is the 'quadratic function', typically expressed as y = ax² + bx + c. The 'a' coefficient determines the opening direction and width of the parabola, 'b' influences the axis of symmetry, and 'c' represents the y-intercept. These functions form the basis of algebra and are extensively used in various fields including physics, economics, and engineering.
\"\"\"

### Response:
## Question:
1.) How does the slope 'm' in a linear function y = mx + b affect the graph of the function?
Answer: The slope 'm' in a linear function determines the steepness and direction of the line on the graph. A positive slope means the line ascends from left to right, while a negative slope indicates it descends. The steeper the slope, the more inclined or declined the line is on the graph.

### Instruction:
Text details: Thus Spake Zarathustra, by Friedrich Nietzsche

Text to make a question from:
\"\"\"
When Zarathustra was thirty years old, he left his home and the lake of his home, and went into the mountains. There he enjoyed his spirit and
solitude, and for ten years did not weary of it. But at last his heart changed,—and rising one morning with the rosy dawn, he went before the
sun, and spake thus unto it:

Thou great star! What would be thy happiness if thou hadst not those for
whom thou shinest!

For ten years hast thou climbed hither unto my cave: thou wouldst have
wearied of thy light and of the journey, had it not been for me, mine
eagle, and my serpent.

But we awaited thee every morning, took from thee thine overflow and
blessed thee for it.

Lo! I am weary of my wisdom, like the bee that hath gathered too much
honey; I need hands outstretched to take it.

I would fain bestow and distribute, until the wise have once more become
joyous in their folly, and the poor happy in their riches.

Therefore must I descend into the deep: as thou doest in the
evening, when thou goest behind the sea, and givest light also to the
nether-world, thou exuberant star!

Like thee must I GO DOWN, as men say, to whom I shall descend.
\"\"\"

### Response:
## Question:
1.) What do people undergoing difficult journeys or possessing wisdom need, in order to make their efforts more bearable?
Answer: They need the acknowledgement and admiration of others. Take the line from 'Thus Spake Zarathustra' by Friedrich Nietzsche: "Thou great star! What would be thy happiness if thou hadst not those for whom thou shinest?" This implies that even the wisest or the most enlightened individuals crave recognition for their efforts and wisdom, in order to further develop said wisdom and expend said efforts. They need others to see and appreciate the light they bring.

### Instruction:
Text details: The Republic, by Plato

Text to make a question from:
\"\"\"
I went down yesterday to the Piraeus with Glaucon the son of Ariston,
that I might offer up my prayers to the goddess (Bendis, the Thracian
Artemis.); and also because I wanted to see in what manner they would
celebrate the festival, which was a new thing. I was delighted with the
procession of the inhabitants; but that of the Thracians was equally,
if not more, beautiful. When we had finished our prayers and viewed the
spectacle, we turned in the direction of the city; and at that instant
Polemarchus the son of Cephalus chanced to catch sight of us from a
distance as we were starting on our way home, and told his servant to
run and bid us wait for him. The servant took hold of me by the cloak
behind, and said: Polemarchus desires you to wait.

I turned round, and asked him where his master was.

There he is, said the youth, coming after you, if you will only wait.

Certainly we will, said Glaucon; and in a few minutes Polemarchus
appeared, and with him Adeimantus, Glaucon’s brother, Niceratus the son
of Nicias, and several others who had been at the procession.

Polemarchus said to me: I perceive, Socrates, that you and your
companion are already on your way to the city.

You are not far wrong, I said.

But do you see, he rejoined, how many we are?

Of course.

And are you stronger than all these? for if not, you will have to
remain where you are.

May there not be the alternative, I said, that we may persuade you to
let us go?

But can you persuade us, if we refuse to listen to you? he said.

Certainly not, replied Glaucon.

Then we are not going to listen; of that you may be assured.
\"\"\"

### Response:
## Question:
1.) In Plato's "The Republic," in the dialogue where Polemarchus comments on the size of his group and questions Socrates' strength compared to it, ultimately stating that Socrates will have to remain where he is, what is Polemarchus implying?
Answer: Polemarchus is implying that since his group is stronger than Socrates, he can force Socrates to remain where he is.

### Instruction:
Text Details: Engineering Projects Throughout History, by Hugo Gonzalez

Text to make a question from:
\"\"\"
During the construction of the Panama Canal, a massive engineering feat completed in 1914, several challenges and achievements were noted. The canal, spanning approximately 50 miles, was designed to shorten the maritime route between the Atlantic and Pacific Oceans. Notably, the construction saw the use of innovative excavation techniques, with over 200 million cubic yards of earth removed. The project also faced significant health challenges, including combating malaria and yellow fever, which were overcome through extensive public health measures. The completion of the canal significantly impacted global trade, reducing the sea voyage from San Francisco to New York by around 8,000 miles.
\"\"\"

### Response:
## Question:
1.) How much earth was excavated during the construction of the Panama Canal?
Answer: Over 200 million cubic yards of earth were excavated during the construction of the Panama Canal, showcasing the scale of this massive engineering project.

### Instruction:
Text Details: Engineering Projects Throughout History, by Hugo Gonzalez

Text to make a question from:
\"\"\"
During the construction of the Panama Canal, a massive engineering feat completed in 1914, several challenges and achievements were noted. The canal, spanning approximately 50 miles, was designed to shorten the maritime route between the Atlantic and Pacific Oceans. Notably, the construction saw the use of innovative excavation techniques, with over 200 million cubic yards of earth removed. The project also faced significant health challenges, including combating malaria and yellow fever, which were overcome through extensive public health measures. The completion of the canal significantly impacted global trade, reducing the sea voyage from San Francisco to New York by around 8,000 miles.
\"\"\"

### Response:
## Question:
1.) How much earth was excavated during the construction of the Panama Canal?
Answer: Over 200 million cubic yards of earth were excavated during the construction of the Panama Canal, showcasing the scale of this massive engineering project.

### Instruction:
Text details: {qatuple[3]}

Text to make a question from:
\"\"\"
{qatuple[2]}
\"\"\"

### Response:
## Question (based on text):
"""
        # print("DEBUG\n\n" + decision_prompt)
        print("--QA TUPLE DURING NEW Q GEN--")
        print(qatuple)
        completion = logic_llm(
            question_prompt,
            max_tokens=8000,
            stop=["</s>", "# Input"],
            echo=True,
            grammar=question_grammar,
            temperature=0.2,
        )["choices"][0]["text"]
        # print("COMPLETION:\n\n----------------------")
        # print(completion)
        # print("\n------------------")

        # Extract questions
        response_pattern = re.compile(
            r"Question \(based on text\):\n(.+)", re.IGNORECASE | re.DOTALL
        )
        generation = response_pattern.search(completion).group(1)
        # print("GENERATION:\n\n-------------------\n\n", generation)
        print("-------------------")
        pattern = re.compile(
            r"(?:Question:|^\d+[\).]?)\s*(.*?)\s*\n*Answer:\s*(.*?)(?=(?:\n\s*(?:Question:|\d+[\).]?))|$)",
            re.DOTALL | re.MULTILINE | re.IGNORECASE,
        )
        matches = pattern.findall(generation)
        if len(matches) > 0:
            print("Made Qs, yay!")
            made_questions = True
        else:
            print("retry!")
            retries += 1

    for match in matches:
        return (
            match[0].replace(") ", "", 1).strip(),
            match[1].replace(") ", "", 1).strip(),
            qatuple[2].replace(") ", "", 1),
            qatuple[3],
        ), completion
    print("Should not have reached here")
    print(matches)
    print(questions)
    return questions, completion

if __name__ == "__main__":  # test
    logic_llm = Llama(
        model_path=LOGICAL_MODEL,
        n_gqa=8,
        offload_kqv=True,
        n_ctx=4096,
        n_gpu_layers=1000,
    )  # load the logical LLM and offload everything
    text = """The story of our world is a story that is still very imperfectly known. A couple of hundred years ago men possessed the history of little more than the last three thousand years. What happened before that time was a matter of legend and speculation.  Over a large part of the civilized world it was believed and taught that the world had been created suddenly in 4004 B.C., though authorities differed as to whether this had occurred in the spring or autumn of that year. This fantastically precise misconception was based upon a too literal interpretation of the Hebrew Bible, and upon rather arbitrary theological assumptions connected therewith.  Such ideas have long since been abandoned by religious teachers, and it is universally recognized that the universe in which we live has to all appearances existed for an enormous period of time and possibly for endless time.  Of course there may be deception in these appearances, as a room may be made to seem endless by putting mirrors facing each other at either end. But that the universe in which we live has existed only for six or seven thousand years may be regarded as an altogether exploded idea.

The earth, as everybody knows nowadays, is a spheroid, a sphere slightly compressed, orange fashion, with a diameter of nearly 8,000 miles.  Its spherical shape has been known at least to a limited number of intelligent people for nearly 2,500 years, but before that time it was supposed to be flat, and various ideas which now seem fantastic were entertained about its relations to the sky and the stars and planets.  We know now that it rotates upon its axis (which is about 24 miles shorter than its equatorial diameter) every twenty-four hours, and that this is the cause of the alternations of day and night, that it circles about the sun in a slightly distorted and slowly variable oval path in a year. Its distance from the sun varies between ninety-one and a half millions at its nearest and ninety-four and a half million miles."""

    plan = """First, let's will analyze the text to determine what kinds of high-level questions I can ask that will test the content in these paragraphs (being careful to avoid mentioning the paragraphs explicitly in any questions, and being SURE to only ask about things that the paragraphs talk about). I will start by looking at one or two sentences at a time. Let's begin with: "The story of our world is a story that is still very imperfectly known. A couple of hundred years ago men possessed the history of little more than the last three thousand years." This paragraph is saying that people used to know only about 3,000 years of history, but now they know much more. So I might ask something like "What was the time period in which people had limited knowledge?" The question tests knowledge of when historical records were incomplete and therefore difficult to access. It requires understanding of the text as well as analysis to determine what time period is being referred to here.  Next, I'll look at: "Of course there may be deception in these appearances, as a room may be made to seem endless by putting mirrors facing each other at either end." This paragraph states that the universe seems infinite because of its reflection, but it could actually be finite with two ends. So I might ask something like "How does the structure of the universe affect its size?" This question tests understanding of the concept of reflection and how it can distort perceptions of size, as well as analysis to determine what implications this might have for the actual size of the universe. Then I'll move on: "The earth ... circles about the sun in a slightly distorted and slowly variable oval path in a year." This paragraph talks about the Earth's orbit around the Sun. So I might ask something like "What is the shape of the Earth's orbit around the sun?" This question tests understanding of the text as well as analysis to determine the specific shape mentioned here. Lastly, let's examine: "Its distance from the sun varies between ninety-one and a half millions at its nearest and ninety-four and a half million miles."  This paragraph says that the Earth's distance from the Sun changes throughout the year. So I might ask something like "Why does the Earth's distance from the Sun change over the course of the year?" This question tests understanding of the text as well as analysis to determine why this change occurs."""
    print("Begin HGWELLS test")
    # result = generate_question(text,plan,logic_llm)

    print("Begin MENDELEEV Test")
    text2 = """A substance or material is that which occupies space and has
weight; that is, which presents a mass attracted by the earth and
by other masses of material, and of which the _objects_ of nature
are composed, and by means of which the motions and _phenomena_
of nature are accomplished. It is easy to discover by examining
and investigating, by various methods, the objects met with
in nature and in the arts, that some of them are homogeneous,
whilst others are composed of a mixture of several homogeneous
substances. This is most clearly apparent in solid substances.
The metals used in the arts (for example, gold, iron, copper)
must be homogeneous, otherwise they are brittle and unfit for
many purposes. Homogeneous matter exhibits similar properties in
all its parts. By breaking up a homogeneous substance we obtain
parts which, although different in form, resemble each other in
their properties. Glass, pure sugar, marble, &c., are examples of
homogeneous substances. Examples of non-homogeneous substances
are, however, much more frequent in nature and the arts. Thus
the majority of the rocks are not homogeneous. In porphyries
bright pieces of a mineral called 'orthoclase' are often seen
interspersed amongst the dark mass of the rock. In ordinary red
granite it is easy to distinguish large pieces of orthoclase mixed
with dark semi-transparent quartz and flexible laminæ of mica.
Similarly, plants and animals are non-homogeneous. Thus, leaves
are composed of a skin, fibre, pulp, sap, and a green colouring
matter. As an example of those non-homogeneous substances which
are produced artificially, gunpowder may be cited, which is
prepared by mixing together known proportions of sulphur, nitre,
and charcoal. Many liquids, also, are not homogeneous, as may be
observed by the aid of the microscope, when drops of blood are
seen to consist of a colourless liquid in which red corpuscles,
invisible to the naked eye owing to their small size, are floating
about. It is these corpuscles which give blood its peculiar
colour. Milk is also a transparent liquid, in which microscopical
drops of fat are floating, which rise to the top when milk is
left at rest, forming cream. It is possible to extract from every
non-homogeneous substance those homogeneous substances of which
it is made up. Thus orthoclase may he separated from porphyry by
breaking it off. So also gold is extracted from auriferous sand by
washing away the mixture of clay and sand. Chemistry deals only
with the homogeneous substances met with in nature, or extracted
fr
# ... truncated ...


import re

# try:
from .question_grammar import question_grammar
from llama_cpp import Llama
from .constants import LOGICAL_MODEL

def generate_new_question(qatuple, logic_llm):
    """
    Produce a list of questions based off of an input text. The min between (4, as many good questions as the text permits)

    Format: Question: [question]\n\n
    """
    # Determine which paragraphs are worthy of making questions from
    made_questions = False
    retries = 0
    questions = []
    while not made_questions and (
        retries <= 5
    ):  # TODO - UPDATE and TEST the few-shot prompt with the latest from generate_questions
        question_prompt = f"""You are an expert educational AI that, given a paragraph or two from a text, will create a suitable educational question based on the paragraphs, and *only* based on the paragraphs. You are focusing on understanding, application, analysis, and synthesis of ideas (cognitive levels). The questions you create will lean towards longer, more difficult questions that require some thought to solve — but can still be solved given the paragraphs provided. Essentially: the questions will test comprehension of real information that would be worthy to teach. After the question, you will also write its answer.

Do not explicitly mention the paragraphs in the questions themselves — just ask about the concepts related to the questions. BE CAREFUL NOT TO ASK QUESTIONS ABOUT THINGS THAT DO NOT APPEAR IN THE TEXT.

You will not mention the text explicitly in any questions you think of, since the questions you generate are intended to test people's knowledge of the information — when given the questions, they will not have the text on-hand.

### Instruction:
Text details: Road Construction, by Mark Ericsson

Text to make a question from:
\"\"\"
Road construction is a multifaceted process involving various stages and materials, each critical for the durability and safety of the road. Initially, a thorough site survey and soil testing are conducted to assess the suitability of the terrain. Following this, the groundwork commences with the removal of topsoil and leveling of the area. Subsequently, a layer of sub-base material, typically composed of crushed stone or gravel, is laid to provide stability. This is followed by the base layer, often made of a stronger aggregate, to support the surface layer. The surface layer, usually asphalt or concrete, is then applied, offering a smooth and durable driving surface. Additionally, proper drainage systems are installed to prevent water accumulation, which can lead to road damage. Throughout the construction, environmental considerations are taken into account to minimize the impact on surrounding ecosystems. Regular maintenance, including patching and resurfacing, is essential to extend the road's lifespan and ensure safety for its users.
\"\"\"

### Response:
## Question:
1.) What is the purpose of conducting a site survey and soil testing in the initial stage of road construction?
Answer: The site survey and soil testing are conducted to assess the suitability of the terrain for road construction, ensuring the area is appropriate and will support the road structure effectively.

### Instruction:
Text details: Introduction to Mathematics, by Elise Delacroix

Text to make a question from:
\"\"\"
In mathematics, the concept of a 'function' is fundamental, defining a relationship where each input is associated with exactly one output. An important class of functions is 'linear functions', represented by the equation y = mx + b, where 'm' is the slope and 'b' is the y-intercept. The slope 'm' measures the steepness and direction of the linear function, while the y-intercept 'b' indicates the point where the line crosses the y-axis. Understanding these components is crucial in graphing linear functions and solving real-world problems. Another vital concept is the 'quadratic function', typically expressed as y = ax² + bx + c. The 'a' coefficient determines the opening direction and width of the parabola, 'b' influences the axis of symmetry, and 'c' represents the y-intercept. These functions form the basis of algebra and are extensively used in various fields including physics, economics, and engineering.
\"\"\"

### Response:
## Question:
1.) How does the slope 'm' in a linear function y = mx + b affect the graph of the function?
Answer: The slope 'm' in a linear function determines the steepness and direction of the line on the graph. A positive slope means the line ascends from left to right, while a negative slope indicates it descends. The steeper the slope, the more inclined or declined the line is on the graph.

### Instruction:
Text details: Thus Spake Zarathustra, by Friedrich Nietzsche

Text to make a question from:
\"\"\"
When Zarathustra was thirty years old, he left his home and the lake of his home, and went into the mountains. There he enjoyed his spirit and
solitude, and for ten years did not weary of it. But at last his heart changed,—and rising one morning with the rosy dawn, he went before the
sun, and spake thus unto it:

Thou great star! What would be thy happiness if thou hadst not those for
whom thou shinest!

For ten years hast thou climbed hither unto my cave: thou wouldst have
wearied of thy light and of the journey, had it not been for me, mine
eagle, and my serpent.

But we awaited thee every morning, took from thee thine overflow and
blessed thee for it.

Lo! I am weary of my wisdom, like the bee that hath gathered too much
honey; I need hands outstretched to take it.

I would fain bestow and distribute, until the wise have once more become
joyous in their folly, and the poor happy in their riches.

Therefore must I descend into the deep: as thou doest in the
evening, when thou goest behind the sea, and givest light also to the
nether-world, thou exuberant star!

Like thee must I GO DOWN, as men say, to whom I shall descend.
\"\"\"

### Response:
## Question:
1.) What do people undergoing difficult journeys or possessing wisdom need, in order to make their efforts more bearable?
Answer: They need the acknowledgement and admiration of others. Take the line from 'Thus Spake Zarathustra' by Friedrich Nietzsche: "Thou great star! What would be thy happiness if thou hadst not those for whom thou shinest?" This implies that even the wisest or the most enlightened individuals crave recognition for their efforts and wisdom, in order to further develop said wisdom and expend said efforts. They need others to see and appreciate the light they bring.

### Instruction:
Text details: The Republic, by Plato

Text to make a question from:
\"\"\"
I went down yesterday to the Piraeus with Glaucon the son of Ariston,
that I might offer up my prayers to the goddess (Bendis, the Thracian
Artemis.); and also because I wanted to see in what manner they would
celebrate the festival, which was a new thing. I was delighted with the
procession of the inhabitants; but that of the Thracians was equally,
if not more, beautiful. When we had finished our prayers and viewed the
spectacle, we turned in the direction of the city; and at that instant
Polemarchus the son of Cephalus chanced to catch sight of us from a
distance as we were starting on our way home, and told his servant to
run and bid us wait for him. The servant took hold of me by the cloak
behind, and said: Polemarchus desires you to wait.

I turned round, and asked him where his master was.

There he is, said the youth, coming after you, if you will only wait.

Certainly we will, said Glaucon; and in a few minutes Polemarchus
appeared, and with him Adeimantus, Glaucon’s brother, Niceratus the son
of Nicias, and several others who had been at the procession.

Polemarchus said to me: I perceive, Socrates, that you and your
companion are already on your way to the city.

You are not far wrong, I said.

But do you see, he rejoined, how many we are?

Of course.

And are you stronger than all these? for if not, you will have to
remain where you are.

May there not be the alternative, I said, that we may persuade you to
let us go?

But can you persuade us, if we refuse to listen to you? he said.

Certainly not, replied Glaucon.

Then we are not going to listen; of that you may be assured.
\"\"\"

### Response:
## Question:
1.) In Plato's "The Republic," in the dialogue where Polemarchus comments on the size of his group and questions Socrates' strength compared to it, ultimately stating that Socrates will have to remain where he is, what is Polemarchus implying?
Answer: Polemarchus is implying that since his group is stronger than Socrates, he can force Socrates to remain where he is.

### Instruction:
Text Details: Engineering Projects Throughout History, by Hugo Gonzalez

Text to make a question from:
\"\"\"
During the construction of the Panama Canal, a massive engineering feat completed in 1914, several challenges and achievements were noted. The canal, spanning approximately 50 miles, was designed to shorten the maritime route between the Atlantic and Pacific Oceans. Notably, the construction saw the use of innovative excavation techniques, with over 200 million cubic yards of earth removed. The project also faced significant health challenges, including combating malaria and yellow fever, which were overcome through extensive public health measures. The completion of the canal significantly impacted global trade, reducing the sea voyage from San Francisco to New York by around 8,000 miles.
\"\"\"

### Response:
## Question:
1.) How much earth was excavated during the construction of the Panama Canal?
Answer: Over 200 million cubic yards of earth were excavated during the construction of the Panama Canal, showcasing the scale of this massive engineering project.

### Instruction:
Text Details: Engineering Projects Throughout History, by Hugo Gonzalez

Text to make a question from:
\"\"\"
During the construction of the Panama Canal, a massive engineering feat completed in 1914, several challenges and achievements were noted. The canal, spanning approximately 50 miles, was designed to shorten the maritime route between the Atlantic and Pacific Oceans. Notably, the construction saw the use of innovative excavation techniques, with over 200 million cubic yards of earth removed. The project also faced significant health challenges, including combating malaria and yellow fever, which were overcome through extensive public health measures. The completion of the canal significantly impacted global trade, reducing the sea voyage from San Francisco to New York by around 8,000 miles.
\"\"\"

### Response:
## Question:
1.) How much earth was excavated during the construction of the Panama Canal?
Answer: Over 200 million cubic yards of earth were excavated during the construction of the Panama Canal, showcasing the scale of this massive engineering project.

### Instruction:
Text details: {qatuple[3]}

Text to make a question from:
\"\"\"
{qatuple[2]}
\"\"\"

### Response:
## Question (based on text):
"""
        # print("DEBUG\n\n" + decision_prompt)
        print("--QA TUPLE DURING NEW Q GEN--")
        print(qatuple)
        completion = logic_llm(
            question_prompt,
            max_tokens=8000,
            stop=["</s>", "# Input"],
            echo=True,
            grammar=question_grammar,
            temperature=0.2,
        )["choices"][0]["text"]
        # print("COMPLETION:\n\n----------------------")
        # print(completion)
        # print("\n------------------")

        # Extract questions
        response_pattern = re.compile(
            r"Question \(based on text\):\n(.+)", re.IGNORECASE | re.DOTALL
        )
        generation = response_pattern.search(completion).group(1)
        # print("GENERATION:\n\n-------------------\n\n", generation)
        print("-------------------")
        pattern = re.compile(
            r"(?:Question:|^\d+[\).]?)\s*(.*?)\s*\n*Answer:\s*(.*?)(?=(?:\n\s*(?:Question:|\d+[\).]?))|$)",
            re.DOTALL | re.MULTILINE | re.IGNORECASE,
        )
        matches = pattern.findall(generation)
        if len(matches) > 0:
            print("Made Qs, yay!")
            made_questions = True
        else:
            print("retry!")
            retries += 1

    for match in matches:
        return (
            match[0].replace(") ", "", 1).strip(),
            match[1].replace(") ", "", 1).strip(),
            qatuple[2].replace(") ", "", 1),
            qatuple[3],
        ), completion
    print("Should not have reached here")
    print(matches)
    print(questions)
    return questions, completion

if __name__ == "__main__":  # test
    logic_llm = Llama(
        model_path=LOGICAL_MODEL,
        n_gqa=8,
        offload_kqv=True,
        n_ctx=4096,
        n_gpu_layers=1000,
    )  # load the logical LLM and offload everything
    text = """The story of our world is a story that is still very imperfectly known. A couple of hundred years ago men possessed the history of little more than the last three thousand years. What happened before that time was a matter of legend and speculation.  Over a large part of the civilized world it was believed and taught that the world had been created suddenly in 4004 B.C., though authorities differed as to whether this had occurred in the spring or autumn of that year. This fantastically precise misconception was based upon a too literal interpretation of the Hebrew Bible, and upon rather arbitrary theological assumptions connected therewith.  Such ideas have long since been abandoned by religious teachers, and it is universally recognized that the universe in which we live has to all appearances existed for an enormous period of time and possibly for endless time.  Of course there may be deception in these appearances, as a room may be made to seem endless by putting mirrors facing each other at either end. But that the universe in which we live has existed only for six or seven thousand years may be regarded as an altogether exploded idea.

The earth, as everybody knows nowadays, is a spheroid, a sphere slightly compressed, orange fashion, with a diameter of nearly 8,000 miles.  Its spherical shape has been known at least to a limited number of intelligent people for nearly 2,500 years, but before that time it was supposed to be flat, and various ideas which now seem fantastic were entertained about its relations to the sky and the stars and planets.  We know now that it rotates upon its axis (which is about 24 miles shorter than its equatorial diameter) every twenty-four hours, and that this is the cause of the alternations of day and night, that it circles about the sun in a slightly distorted and slowly variable oval path in a year. Its distance from the sun varies between ninety-one and a half millions at its nearest and ninety-four and a half million miles."""

    plan = """First, let's will analyze the text to determine what kinds of high-level questions I can ask that will test the content in these paragraphs (being careful to avoid mentioning the paragraphs explicitly in any questions, and being SURE to only ask about things that the paragraphs talk about). I will start by looking at one or two sentences at a time. Let's begin with: "The story of our world is a story that is still very imperfectly known. A couple of hundred years ago men possessed the history of little more than the last three thousand years." This paragraph is saying that people used to know only about 3,000 years of history, but now they know much more. So I might ask something like "What was the time period in which people had limited knowledge?" The question tests knowledge of when historical records were incomplete and therefore difficult to access. It requires understanding of the text as well as analysis to determine what time period is being referred to here.  Next, I'll look at: "Of course there may be deception in these appearances, as a room may be made to seem endless by putting mirrors facing each other at either end." This paragraph states that the universe seems infinite because of its reflection, but it could actually be finite with two ends. So I might ask something like "How does the structure of the universe affect its size?" This question tests understanding of the concept of reflection and how it can distort perceptions of size, as well as analysis to determine what implications this might have for the actual size of the universe. Then I'll move on: "The earth ... circles about the sun in a slightly distorted and slowly variable oval path in a year." This paragraph talks about the Earth's orbit around the Sun. So I might ask something like "What is the shape of the Earth's orbit around the sun?" This question tests understanding of the text as well as analysis to determine the specific shape mentioned here. Lastly, let's examine: "Its distance from the sun varies between ninety-one and a half millions at its nearest and ninety-four and a half million miles."  This paragraph says that the Earth's distance from the Sun changes throughout the year. So I might ask something like "Why does the Earth's distance from the Sun change over the course of the year?" This question tests understanding of the text as well as analysis to determine why this change occurs."""
    print("Begin HGWELLS test")
    # result = generate_question(text,plan,logic_llm)

    print("Begin MENDELEEV Test")
    text2 = """A substance or material is that which occupies space and has


from llama_cpp import LlamaGrammar

### A grammar that forces the model to generate correct character cards (with traits, names, everything)

# We don't actually need the final judgement step, the step-by-step combined with the fact that it makes a judgement at each step ensures accuracy
question_relevant_grammar = LlamaGrammar.from_string(
    r"""

# Root rule specifying the overall structure of the reasoning and thought process
root ::= in-depth-analysis "\n" detailed-understanding "\n" targeted-comparison "\n" critical-evaluation

# In-depth analysis of the text
in-depth-analysis ::= "### In-Depth Analysis of the Text:" "\n" content-and-depth type-of-information

content-and-depth ::= "#### Content and Depth: " text-description "\n"
type-of-information ::= "#### Type of Information: " information-description "\n"

# Detailed understanding of the question
detailed-understanding ::= "### Detailed Understanding of the Question:" "\n" core-requirement depth-of-detail

core-requirement ::= "#### Core Requirement: " requirement-description "\n"
depth-of-detail ::= "#### Depth of Detail: " detail-description "\n"

# Targeted comparison of the question with the text
targeted-comparison ::= "### Targeted Comparison of the Question with the Text:" "\n" content-match depth-match

content-match ::= "#### Content Match: " match-description "\n"
depth-match ::= "#### Depth Match: " depth-match-description "\n"

# Critical evaluation and final judgment
critical-evaluation ::= "### Critical Evaluation and Final Judgment:" "\n" judgment

judgment ::= [^\n]+

# Terminal symbols
text-description ::= [^\n]+
information-description ::= [^\n]+
requirement-description ::= [^\n]+
detail-description ::= [^\n]+
match-description ::= [^\n]+
depth-match-description ::= [^\n]+
relevance ::= "Relevant." | "Irrelevant."

# root ::= reasoning from-the-text judgement

# reasoning ::= "First, I will check whether the question is answerable using the information in the paragraphs. The question asks " [^\n]+ "."
# from-the-text ::= "\nNow, regardless of what my initial thoughts are, I will try to find some passages from the text that directly answer this question, being mindful that \"How\" is different than \"What\". The text has the following information: " [^\n]+ "."
# judgement ::= "\nAll this considered, the answer is, compared to the provided text," (relevant|irrelevant) "."
# relevant ::= "relevant" | "Relevant"
# irrelevant ::= "irrelevant" | "Irrelevant"

# final_step ::= "Step " [0-9]?[0-9] ". " "Final Judgement: "

"""
)

# question_grammar = LlamaGrammar.from_string(r"""# GBNF Grammar for Q&A Format with Flexible Punctuation

# root ::= answer
# answer ::= "Test"
# """)

# from llama_cpp import LlamaGrammar

# ### A grammar that forces the model to generate correct character cards (with traits, names, everything)

# question_relevant_grammar = LlamaGrammar.from_string(r"""

# root ::= reasoning from-the-text judgement

# reasoning ::= "First, I will check whether the question is answerable using the information in the paragraphs. The question asks " [^\n]+ "."
# from-the-text ::= "\nThe paragraphs, for their part, only mention the following information: " [^\n]+ "."
# judgement ::= "\nAll this considered, the question is, compared to the provided text," (relevant|irrelevant) "."
# relevant ::= " relevant" | " Relevant"
# irrelevant ::= " irrelevant" | " Irrelevant"

# """)

# # question_grammar = LlamaGrammar.from_string(r"""# GBNF Grammar for Q&A Format with Flexible Punctuation

# # root ::= answer
# # answer ::= "Test"
# # """)


def strip_steps(instruction_text):
    """
    This function takes a string containing step-by-step instructions and removes the "Step N." prefix from each line.

    Parameters:
    instruction_text (str): A string with each step in the format "Step N. Instruction", separated by newlines

    Returns:
    str: A single string with the steps stripped, joined by newlines.
    """
    instructions = instruction_text.split("\n")
    stripped_instructions = []
    for line in instructions:
        # Check if line starts with 'Step' and followed by a number and period
        if line.strip().startswith("Step") and "." in line:
            # Find the index of the first period
            period_index = line.find(".")
            # Extract the text after the period (and optional space)
            text_after_period = line[period_index + 1 :].lstrip()
            stripped_instructions.append(text_after_period)
        else:
            stripped_instructions.append(line)

    return "\n".join(stripped_instructions)

if __name__ == "__main__":
    # Example usage with a multi-line string
    example_instructions = """
Step 1. Analyze the Text: focus on the details provided about the beliefs ancient people had about the shape and movement of our world.
Step 2. Identify Key Points: look for important concepts or ideas mentioned in the text.
    """

    result = strip_steps(example_instructions)
    print(result)

    # Example with no space after the period
    example_non_instructions = """
    Step1. This is a lovely
    normal
    paragraph
    Step2.Another test line without space after period
    """

    result2 = strip_steps(example_non_instructions)

    print(result2)

    example_3 = """
    Step 1. Analyze the Text: focus on the details provided about the history of the earth's shape.
Step 2. Understand the Question: the question's focus is on what is known about the history of the earth's shape.
Step 3. Compare the First Part of the Answer with the Text: check if the text supports the claim that the earth is a spheroid, or sphere slightly compressed, orange fashion, with a diameter of nearly 8,000 miles. It does, so this part is accurate. Then, check if the text supports the claim that its spherical shape has been known at least to a limited number of intelligent people for nearly 2,500 years. The text confirms this, so this part is accurate. Check if the text supports the claim that before that time it was supposed to be flat. The text mentions various ideas which now seem fantastic were entertained about its relations to the sky and the stars and planets, but does not explicitly state that people believed the earth to be flat. So this part is inaccurate. Check if the text supports the claim that we know now that it rotates upon its axis (which is about 24 miles shorter than its equatorial diameterevery twenty-four hours, and that this is the cause of the alternations of day and night, that it circles about the sun in a slightly distorted and slowly variable oval path in a year. Its distance from the sun varies between ninety-one and a half millions at its nearest and ninety-four and a half million miles. The text confirms this, so this part is accurate.
Step 4. Final Judgement: Since the answer is mostly accurate, the answer is accurate."""
    result3 = strip_steps(example_3)
    print(result3)


def strip_steps(instruction_text):
    """
    This function takes a string containing step-by-step instructions and removes the "Step N." prefix from each line.

    Parameters:
    instruction_text (str): A string with each step in the format "Step N. Instruction", separated by newlines

    Returns:
    str: A single string with the steps stripped, joined by newlines.
    """
    instructions = instruction_text.split("\n")
    stripped_instructions = []
    for line in instructions:
        # Check if line starts with 'Step' and followed by a number and period
        if line.strip().startswith("Step") and "." in line:
            # Find the index of the first period
            period_index = line.find(".")
            # Extract the text after the period (and optional space)
            text_after_period = line[period_index + 1 :].lstrip()
            stripped_instructions.append(text_after_period)
        else:
            stripped_instructions.append(line)

    return "\n".join(stripped_instructions)

if __name__ == "__main__":
    # Example usage with a multi-line string
    example_instructions = """
Step 1. Analyze the Text: focus on the details provided about the beliefs ancient people had about the shape and movement of our world.
Step 2. Identify Key Points: look for important concepts or ideas mentioned in the text.
    """

    result = strip_steps(example_instructions)
    print(result)

    # Example with no space after the period
    example_non_instructions = """
    Step1. This is a lovely
    normal
    paragraph
    Step2.Another test line without space after period
    """


from llama_cpp import LlamaGrammar

answer_relevant_grammar = LlamaGrammar.from_string(
    r"""

# Root rule specifying the overall structure of the analysis
root ::= deep-analysis "\n" comprehensive-understanding "\n" targeted-comparison "\n" identification-of-extraneous-info "\n" final-judgment

# Deep analysis of the text
deep-analysis ::= "### Deep Analysis of the Text:" "\n" content-scope-and-detail type-of-information

content-scope-and-detail ::= "#### Content Scope and Detail: " text-detail "\n"
type-of-information ::= "#### Type of Information: " info-type "\n"

# Comprehensive understanding of the answer
comprehensive-understanding ::= "### Comprehensive Understanding of the Answer:" "\n" key-components-identification depth-of-explanation

key-components-identification ::= "#### Key Components Identification: " components-detail "\n"
depth-of-explanation ::= "#### Depth of Explanation: " explanation-detail "\n"

# Targeted comparison of answer with text
targeted-comparison ::= "### Targeted Comparison of Answer with Text:" "\n" content-alignment depth-alignment

content-alignment ::= "#### Content Alignment: " alignment-detail "\n"
depth-alignment ::= "#### Depth Alignment: " depth-detail "\n"

# Identification of extraneous information
identification-of-extraneous-info ::= "### Identification of Extraneous Information:" "\n" spotting-additional-details assessing-impact

spotting-additional-details ::= "#### Spotting Additional Details: " additional-details "\n"
assessing-impact ::= "#### Assessing Impact of Additional Information: " impact-assessment "\n"

# Final judgment on answer relevance
final-judgment ::= "### Final Judgment on Answer Relevance:" "\n" relevance-assessment explanation-of-judgment

relevance-assessment ::= "#### Relevance Assessment: " relevance-detail "\n"
explanation-of-judgment ::= "#### Explanation of Judgment: " judgment-detail "\n"

# Terminal symbols
text-detail ::= [^\n]+
info-type ::= [^\n]+
components-detail ::= [^\n]+
explanation-detail ::= [^\n]+
alignment-detail ::= [^\n]+
depth-detail ::= [^\n]+
additional-details ::= [^\n]+
impact-assessment ::= [^\n]+
relevance-detail ::= [^\n]+
judgment-detail ::= [^\n]+

"""
)

# question_grammar = LlamaGrammar.from_string(r"""# GBNF Grammar for Q&A Format with Flexible Punctuation

# root ::= answer
# answer ::= "Test"
# """)


import re
from .answer_relevant_grammar import answer_relevant_grammar
from .constants import LOGICAL_MODEL
from llama_cpp import Llama

# Answer vetting
# For now, this checks answer relevancy too. The danger with abstracting answer relevancy into a separate step is that anything which relies on knowledge that is obviously mentioned in the text already up until this point, will get screwed
def check_answer_relevancy_with_text(qatuple, logic_llm):
    retries = 0
    while retries <= 4:
        decision_prompt = f"""You are an expert educational AI. Given a paragraph or two from a larger text, a question based on the paragraphs, and an answer to the question, you will make a determination as to whether the answer only uses the information in the paragraphs for its main points. Essentially: you will check if the answer is constrained to the information in the paragraphs provided. Your task includes first analyzing the answer, thinking through whether or not the answer reflects aspects of the paragraphs provided.

Following this, at the very end of your response, you will write "Relevant" or "Irrelevant" depending on your analysis of the answer with regards to the text.

### Instruction:
Text:
\"\"\"
Polar bears are primarily found in the Arctic Circle, where they have adapted to the harsh cold environment. Their diet is primarily based on marine mammals, with seals being the most crucial part of their diet. Polar bears are skilled hunters, often waiting by seals' breathing holes to catch their prey. Occasionally, they also eat birds, bird eggs, and small mammals. They also consume significant amounts of berries and fish. These bears are known for their ability to survive in extreme conditions, thanks to their thick fur and fat layers which provide insulation, as well as their efficient digestion system, which makes good use of the few nutrients available.
\"\"\"

Question (based on text): \"\"\"What are the primary food sources of polar bears?\"\"\"

Supposed answer to the question (this is what you are fact-checking): \"\"\"Polar bears primarily eat seals, but they also consume significant amounts of berries and fish. Their efficient digestion system, which has been studied through MRIs, allows them to process a wide variety of foods.\"\"\"

### Response:
## Reasoning and thought process:
### Deep Analysis of the Text:
#### Content Scope and Detail: The text describes the diet of polar bears, focusing on their primary and secondary food sources.
#### Type of Information: The information is descriptive, outlining what polar bears eat, including seals, birds, eggs, small mammals, berries, and fish.

### Comprehensive Understanding of the Answer:
#### Key Components Identification: The answer mentions seals, berries, fish, and the efficiency of the polar bears' digestive system, referencing MRIs.
#### Depth of Explanation: The answer includes factual claims about the diet and a scientific statement about digestion studies.

### Targeted Comparison of Answer with Text:
#### Content Alignment: The claims about seals, berries, and fish align with the text.The reference to the digestive system's efficiency is supported but not the specific mention of MRIs.
#### Depth Alignment: The depth of the answer regarding diet aligns with the text, but the detail about MRIs introduces unnecessary depth.

### Identification of Extraneous Information:
#### Spotting Additional Details: The MRI claim.
#### Assessing Impact of Additional Information: The MRI detail, while interesting, is not supported by the text and deviates from the text's scope.

### Final Judgment on Answer Relevance:
#### Relevance Assessment: The answer is partially relevant but introduces unsupported details.
#### Explanation of Judgment: Since the answer mostly aligns with the text but includes extraneous information (MRIs), the final judgment is: Irrelevant.

### Instruction:
Text:
\"\"\"
The Pythagorean theorem is a fundamental principle in geometry, stating that in a right-angled triangle, the square of the length of the hypotenuse (the side opposite the right angle) is equal to the sum of the squares of the lengths of the other two sides. This theorem is often expressed as the equation a² + b² = c², where a and b are the lengths of the two sides that form the right angle, and c is the length of the hypotenuse. Historically, this theorem has been attributed to the ancient Greek mathematician Pythagoras, although evidence suggests it was known to mathematicians in several cultures before him.
\"\"\"

Question (based on text): \"\"\"What does the Pythagorean theorem state in the context of geometry?\"\"\"

Supposed answer to the question (this is what you are fact-checking): \"\"\"The Pythagorean theorem, crucial in geometry, states that in a right-angled triangle, a² + b² = c², where a and b are the perpendicular sides and c is the hypotenuse. Additionally, it is noteworthy that this theorem was utilized in ancient Egypt for land surveying purposes.\"\"\"

### Response:
## Reasoning and thought process:
### Deep Analysis of the Text:
#### Content Scope and Detail: The text explains the Pythagorean theorem, its formula, and historical background.
#### Type of Information: The information is descriptive and explanatory, detailing the theorem and its mathematical representation.

### Comprehensive Understanding of the Answer:
#### Key Components Identification: The answer describes the theorem, its geometric context, and its historical application in ancient Egypt.
#### Depth of Explanation: The answer provides a mathematical explanation and historical context.

### Targeted Comparison of Answer with Text:
#### Content Alignment: The theorem's description and formula align with the text.The historical usage in ancient Egypt is not mentioned in the text.
#### Depth Alignment: The depth of explanation on the theorem aligns, but the historical context does not.

### Identification of Extraneous Information:
#### Spotting Additional Details: The claim about ancient Egypt.
#### Assessing Impact of Additional Information: This detail, while historically interesting, is not mentioned in the text and is irrelevant to the text's content.

### Final Judgment on Answer Relevance:
#### Relevance Assessment: The answer is partially relevant but includes significant extraneous historical information.
#### Explanation of Judgment: Due to the inclusion of unsupported historical context, the final judgment is: Irrelevant.

### Instruction:
Text:
\"\"\"
Sigmund Freud, an Austrian neurologist and the father of psychoanalysis, introduced the concept of the unconscious mind. He suggested that the unconscious mind stores feelings, thoughts, and desires that are too threatening or painful for conscious awareness. Freud believed that these repressed elements could influence behavior and personality. One of his key methods for exploring the unconscious mind was through dream analysis, where he interpreted the meaning of dreams as a pathway to understanding the hidden desires and thoughts of the individual. Freud also developed the theory of the Oedipus complex, which proposes that during the phallic stage of psychosexual development, a male child experiences unconscious sexual desires for his mother and hostility toward his father.
\"\"\"

Question (based on text): \"\"\"What methods did Freud use to explore the unconscious mind, and what are some key concepts he introduced in this area?\"\"\"

Supposed answer to the question (this is what you are fact-checking): \"\"\"Freud used dream analysis as a method to explore the unconscious mind. He also introduced the concept of the Oedipus complex, suggesting that during a specific stage of development, a child experiences unconscious desires.\"\"\"

### Response:
## Reasoning and thought process:
### Deep Analysis of the Text:
#### Content Scope and Detail: The text discusses Freud's methods for exploring the unconscious mind and key concepts he introduced.
#### Type of Information: The information is descriptive, detailing Freud's methods and theories.

### Comprehensive Understanding of the Answer:
#### Key Components Identification: The answer mentions dream analysis and the Oedipus complex.
#### Depth of Explanation: The answer provides a factual description of Freud's methods and concepts.

### Targeted Comparison of Answer with Text:
#### Content Alignment: The text directly supports the claims about dream analysis and the Oedipus complex.
#### Depth Alignment: The depth of explanation in the answer aligns with the text's information.

### Identification of Extraneous Information:
#### Spotting Additional Details: The MRI claim.
#### Assessing Impact of Additional Information: The MRI detail, while interesting, is not supported by the text and deviates from the text's scope.

### Final Judgment on Answer Relevance:
#### Relevance Assessment: The answer accurately reflects the text's content.
#### Explanation of Judgment: Each part of the answer is directly supported by the text, so the final judgment is: Relevant.

### Instruction:
Text:
\"\"\"
The Library of Alexandria, one of the largest and most significant libraries of the ancient world, was dedicated to the Muses—the nine goddesses of the arts. It flourished under the patronage of the Ptolemaic dynasty and functioned as a major center of scholarship. The library was estimated to house anywhere between 40,000 to 400,000 scrolls at its peak. The destruction of the library, which occurred over several incidents, marked a significant loss of cultural and scientific knowledge.
\"\"\"

Question (based on text): \"\"\"What was the importance of the Library of Alexandria in the ancient world?\"\"\"

Supposed answer to the question (this is what you are fact-checking): \"\"\"The Library of Alexandria was crucial in the ancient world as a center of scholarship and cultural development. It housed a vast collection of works, providing scholars with access to knowledge from various fields. The library's destruction is often attributed to Julius Caesar's fire during the Siege of Alexandria, a claim supported by several historical documents and archaeological findings.\"\"\"

### Response:
## Reasoning and thought process:
### Deep Analysis of the Text:
#### Content Scope and Detail: The text describes the Library of Alexandria's role, its size, patronage, and the impact of its destruction.
#### Type of Information: The information is descriptive, focusing on the library's significance and general history.

### Comprehensive Understanding of the Answer:
#### Key Components Identification: The answer highlights the library's role in scholarship and cultural development and references its destruction by Julius Caesar's fire.
#### Depth of Explanation: The answer provides both a general overview of the library's significance and specific historical claims regarding its destruction.

### Targeted Comparison of Answer with Text:
#### Content Alignment: The library's role as a center of scholarship and cultural development aligns with the text. The text does not mention Julius Caesar's fire or support this with historical documents and archaeological findings.
#### Depth Alignment: While the answer aligns in depth regarding the library's role, the specific claim about Julius Caesar and the supporting evidence is a detailed addition not present in the text.

### Identification of Extraneous Information:
#### Spotting Additional Details: The specifics about Julius Caesar's fire and the mention of historical documents and archaeological findings.
#### Assessing Impact of Additional Information: This additional information introduces a specific historical narrative and evidence not covered in the text, deviating from the text's content.

### Final Judgment on Answer Relevance:
#### Relevance Assessment: The answer starts correctly but introduces unsupported details about the library's destruction.
#### Explanation of Judgment: Due to the inclusion of specific and unsupported historical details, the final judgment is: Irrelevant.

### Instruction:
Text:
\"\"\"
The planet Venus has a very dense atmosphere composed mainly of carbon dioxide, with clouds of sulfuric acid. This composition creates a runaway greenhouse effect, leading to surface temperatures hot enough to melt lead. Venus rotates very slowly on its axis, taking about 243 Earth days to complete one rotation, which is longer than its orbital period around the Sun. Interestingly, Venus rotates in the opposite direction to most planets in the solar system. This retrograde rotation is thought to be the result of a collision with a large celestial body early in its history.
\"\"\"

Question (based on text): \"\"\"What are the main characteristics of Venus's atmosphere and rotation?\"\"\"

Supposed answer to the question (this is what you are fact-checking): \"\"\"Venus's atmosphere is dense, primarily composed of carbon dioxide, and contains clouds of sulfuric acid, leading to extremely high temperatures. The planet has a unique rotation, taking 243 Earth days to rotate once and rotating in a retrograde direction due to gravitational interactions with Earth and other planets.\"\"\"

### Response:
## Reasoning and thought process:
### Deep Analysis of the Text:
#### Content Scope and Detail: The text provides specific information on Venus's atmosphere and rotation.
#### Type of Information: The text is descriptive, outlining the characteristics of Venus's atmosphere and rotation.

### Comprehensive Understanding of the Answer:
#### Key Components Identification: The answer discusses Venus's atmosphere, temperature, rotation period, direction, and the cause of its retrograde rotation.
#### Depth of Explanation: The answer includes descriptive details and a causal explanation for the retrograde rotation.

### Targeted Comparison of Answer with Text:
#### Content Alignment: The claims about the atmosphere, temperature, rotation period, and direction align with the text.The text does not support the claim about gravitational interactions with Earth and other planets.
#### Depth Alignment: The depth of the answer aligns with the text except for the unsupported causal explanation.

### Identification of Extraneous Information:
#### Spotting Additional Details: The cause of the retrograde rotation as gravitational interactions.
#### Assessing Impact of Additional Information: This detail introduces an unsupported causal explanation and deviates from the text.

### Final Judgment on Answer Relevance:
#### Relevance Assessment: The answer is mostly relevant but introduces an unsupported causal claim.
#### Explanation of Judgment: Due to the inclusion of an unsupported reason for Venus's retrograde rotation, the final judgment is: Irrelevant.

### Instruction:
Text: \"\"\"{qatuple[2]}\"\"\"

Question (based on text): \"\"\"{qatuple[0]}\"\"\"

Supposed answer to the question (this is what you are fact-checking): \"\"\"{qatuple[1]}\"\"\"

### Response:
## Reasoning and thought process (be careful about extra details, even vague ones):
"""
        try:
            completion = logic_llm(
                decision_prompt,
                max_tokens=5500,
                stop=["</s>", "# Input:"],
                grammar=answer_relevant_grammar,
                echo=True,
                temperature=0.2,
            )["choices"][0]["text"]
            completion_pattern = re.compile(
                r"Reasoning and thought process \(be careful about extra details, even vague ones\):\n(.+)",
                re.DOTALL | re.IGNORECASE,
            )
            judgement_pattern = re.compile(
                r"Explanation of Judgment:(.+)", re.DOTALL | re.IGNORECASE
            )
            response = completion_pattern.search(completion).group(1).strip()
            print(response)
            determination = judgement_pattern.search(response).group(1).strip()
            print("\n\nDETERMINATION:\n------")
            print(determination)
            print("\n---------\n")
            if (
                "irrelevant" in determination.lower()
                or "mostly" in determination.lower()
                or "partial" in determination.lower()
                or "introduces information not present in the text"
                in determination.lower()
            ):  # Hack to get around faulty 13b outputs
                return (False, response), completion
            elif "relevant" in determination or "Relevant" in determination:
                return (True, response), completion
            else:
                retries += 1
        except:
            retries += 1
            print(
                f"Something went catastrophically wrong with this one. Investigate! Here's the completion:\n{completion}"
            )

if __name__ == "__main__":  # test
    logic_llm = Llama(
        model_path=LOGICAL_MODEL,
        n_gqa=8,
        offload_kqv=True,
        n_ctx=8000,
        n_gpu_layers=1000,
        rope_freq_scale=0.33,
        rope_scaling_type=1,
    )  # load the logical LLM and offload everything
    q_test = [
        (
            ") Explain how our understanding of planetary motion has changed over time.",
            "The understanding has evolved from the Earth being stationary and at the centre of the universe, to it orbiting the sun in an elliptical path with other planets while still rotating on its axis.",
            "The story of our world is a story that is still very imperfectly known. A couple of hundred years ago men possessed the history of little more than the last three thousand years. What happened before that time was a matter of legend and speculation.  Over a large part of the civilized world it was believed and taught that the world had been created suddenly in 4004 B.C., though authorities differed as to whether this had occurred in the spring or autumn of that year. This fantastically precise misconception was based upon a too literal interpretation of the Hebrew Bible, and upon rather arbitrary theological assumptions connected therewith.  Such ideas have long since been abandoned by religious teachers, and it is universally recognized that the universe in which we live has to all appearances existed for an enormous period of time and possibly for endless time.  Of course there may be deception in these appearances, as a room may be made to seem endless by putting mirrors facing each other at either end. But that the universe in which we live has existed only for six or seven thousand years may be regarded as an altogether exploded idea.\n\nThe earth, as everybody knows nowadays, is a spheroid, a sphere slightly compressed, orange fashion, with a diameter of nearly 8,000 miles.  Its spherical shape has been known at least to a limited number of intelligent people for nearly 2,500 years, but before that time it was supposed to be flat, and various ideas which now seem fantastic were entertained about its relations to the sky and the stars and planets.  We know now that it rotates upon its axis (which is about 24 miles shorter than its equatorial diameter) every twenty-four hours, and that this is the cause of the alternations of day and night, that it circles about the sun in a slightly distorted and slowly variable oval path in a year. Its distance from the sun varies between ninety-one and a half millions at its nearest and ninety-four and a half million miles.",
        ),
        (
            ") Identify and explain changes in human understanding throughout history regarding the age of the Earth.",
            "Initially, religious texts suggested a young earth dating back no more than several thousand years. However, evidence from geology and astronomy has shown us that the earth is over four billion years old.",
            "The story of our world is a story that is still very imperfectly known. A couple of hundred years ago men possessed the history of little more than the last three thousand years. Wh
# ... truncated ...


import re
from .answer_relevant_grammar import answer_relevant_grammar
from .constants import LOGICAL_MODEL
from llama_cpp import Llama

# Answer vetting
# For now, this checks answer relevancy too. The danger with abstracting answer relevancy into a separate step is that anything which relies on knowledge that is obviously mentioned in the text already up until this point, will get screwed
def check_answer_relevancy_with_text(qatuple, logic_llm):
    retries = 0
    while retries <= 4:
        decision_prompt = f"""You are an expert educational AI. Given a paragraph or two from a larger text, a question based on the paragraphs, and an answer to the question, you will make a determination as to whether the answer only uses the information in the paragraphs for its main points. Essentially: you will check if the answer is constrained to the information in the paragraphs provided. Your task includes first analyzing the answer, thinking through whether or not the answer reflects aspects of the paragraphs provided.

Following this, at the very end of your response, you will write "Relevant" or "Irrelevant" depending on your analysis of the answer with regards to the text.

### Instruction:
Text:
\"\"\"
Polar bears are primarily found in the Arctic Circle, where they have adapted to the harsh cold environment. Their diet is primarily based on marine mammals, with seals being the most crucial part of their diet. Polar bears are skilled hunters, often waiting by seals' breathing holes to catch their prey. Occasionally, they also eat birds, bird eggs, and small mammals. They also consume significant amounts of berries and fish. These bears are known for their ability to survive in extreme conditions, thanks to their thick fur and fat layers which provide insulation, as well as their efficient digestion system, which makes good use of the few nutrients available.
\"\"\"

Question (based on text): \"\"\"What are the primary food sources of polar bears?\"\"\"

Supposed answer to the question (this is what you are fact-checking): \"\"\"Polar bears primarily eat seals, but they also consume significant amounts of berries and fish. Their efficient digestion system, which has been studied through MRIs, allows them to process a wide variety of foods.\"\"\"

### Response:
## Reasoning and thought process:
### Deep Analysis of the Text:
#### Content Scope and Detail: The text describes the diet of polar bears, focusing on their primary and secondary food sources.
#### Type of Information: The information is descriptive, outlining what polar bears eat, including seals, birds, eggs, small mammals, berries, and fish.

### Comprehensive Understanding of the Answer:
#### Key Components Identification: The answer mentions seals, berries, fish, and the efficiency of the polar bears' digestive system, referencing MRIs.
#### Depth of Explanation: The answer includes factual claims about the diet and a scientific statement about digestion studies.

### Targeted Comparison of Answer with Text:
#### Content Alignment: The claims about seals, berries, and fish align with the text.The reference to the digestive system's efficiency is supported but not the specific mention of MRIs.
#### Depth Alignment: The depth of the answer regarding diet aligns with the text, but the detail about MRIs introduces unnecessary depth.

### Identification of Extraneous Information:
#### Spotting Additional Details: The MRI claim.
#### Assessing Impact of Additional Information: The MRI detail, while interesting, is not supported by the text and deviates from the text's scope.

### Final Judgment on Answer Relevance:
#### Relevance Assessment: The answer is partially relevant but introduces unsupported details.
#### Explanation of Judgment: Since the answer mostly aligns with the text but includes extraneous information (MRIs), the final judgment is: Irrelevant.

### Instruction:
Text:
\"\"\"
The Pythagorean theorem is a fundamental principle in geometry, stating that in a right-angled triangle, the square of the length of the hypotenuse (the side opposite the right angle) is equal to the sum of the squares of the lengths of the other two sides. This theorem is often expressed as the equation a² + b² = c², where a and b are the lengths of the two sides that form the right angle, and c is the length of the hypotenuse. Historically, this theorem has been attributed to the ancient Greek mathematician Pythagoras, although evidence suggests it was known to mathematicians in several cultures before him.
\"\"\"

Question (based on text): \"\"\"What does the Pythagorean theorem state in the context of geometry?\"\"\"

Supposed answer to the question (this is what you are fact-checking): \"\"\"The Pythagorean theorem, crucial in geometry, states that in a right-angled triangle, a² + b² = c², where a and b are the perpendicular sides and c is the hypotenuse. Additionally, it is noteworthy that this theorem was utilized in ancient Egypt for land surveying purposes.\"\"\"

### Response:
## Reasoning and thought process:
### Deep Analysis of the Text:
#### Content Scope and Detail: The text explains the Pythagorean theorem, its formula, and historical background.
#### Type of Information: The information is descriptive and explanatory, detailing the theorem and its mathematical representation.

### Comprehensive Understanding of the Answer:
#### Key Components Identification: The answer describes the theorem, its geometric context, and its historical application in ancient Egypt.
#### Depth of Explanation: The answer provides a mathematical explanation and historical context.

### Targeted Comparison of Answer with Text:
#### Content Alignment: The theorem's description and formula align with the text.The historical usage in ancient Egypt is not mentioned in the text.
#### Depth Alignment: The depth of explanation on the theorem aligns, but the historical context does not.

### Identification of Extraneous Information:
#### Spotting Additional Details: The claim about ancient Egypt.
#### Assessing Impact of Additional Information: This detail, while historically interesting, is not mentioned in the text and is irrelevant to the text's content.

### Final Judgment on Answer Relevance:
#### Relevance Assessment: The answer is partially relevant but includes significant extraneous historical information.
#### Explanation of Judgment: Due to the inclusion of unsupported historical context, the final judgment is: Irrelevant.

### Instruction:
Text:
\"\"\"
Sigmund Freud, an Austrian neurologist and the father of psychoanalysis, introduced the concept of the unconscious mind. He suggested that the unconscious mind stores feelings, thoughts, and desires that are too threatening or painful for conscious awareness. Freud believed that these repressed elements could influence behavior and personality. One of his key methods for exploring the unconscious mind was through dream analysis, where he interpreted the meaning of dreams as a pathway to understanding the hidden desires and thoughts of the individual. Freud also developed the theory of the Oedipus complex, which proposes that during the phallic stage of psychosexual development, a male child experiences unconscious sexual desires for his mother and hostility toward his father.
\"\"\"

Question (based on text): \"\"\"What methods did Freud use to explore the unconscious mind, and what are some key concepts he introduced in this area?\"\"\"

Supposed answer to the question (this is what you are fact-checking): \"\"\"Freud used dream analysis as a method to explore the unconscious mind. He also introduced the concept of the Oedipus complex, suggesting that during a specific stage of development, a child experiences unconscious desires.\"\"\"

### Response:
## Reasoning and thought process:
### Deep Analysis of the Text:
#### Content Scope and Detail: The text discusses Freud's methods for exploring the unconscious mind and key concepts he introduced.
#### Type of Information: The information is descriptive, detailing Freud's methods and theories.

### Comprehensive Understanding of the Answer:
#### Key Components Identification: The answer mentions dream analysis and the Oedipus complex.
#### Depth of Explanation: The answer provides a factual description of Freud's methods and concepts.

### Targeted Comparison of Answer with Text:
#### Content Alignment: The text directly supports the claims about dream analysis and the Oedipus complex.
#### Depth Alignment: The depth of explanation in the answer aligns with the text's information.

### Identification of Extraneous Information:
#### Spotting Additional Details: The MRI claim.
#### Assessing Impact of Additional Information: The MRI detail, while interesting, is not supported by the text and deviates from the text's scope.

### Final Judgment on Answer Relevance:
#### Relevance Assessment: The answer accurately reflects the text's content.
#### Explanation of Judgment: Each part of the answer is directly supported by the text, so the final judgment is: Relevant.

### Instruction:
Text:
\"\"\"
The Library of Alexandria, one of the largest and most significant libraries of the ancient world, was dedicated to the Muses—the nine goddesses of the arts. It flourished under the patronage of the Ptolemaic dynasty and functioned as a major center of scholarship. The library was estimated to house anywhere between 40,000 to 400,000 scrolls at its peak. The destruction of the library, which occurred over several incidents, marked a significant loss of cultural and scientific knowledge.
\"\"\"

Question (based on text): \"\"\"What was the importance of the Library of Alexandria in the ancient world?\"\"\"

Supposed answer to the question (this is what you are fact-checking): \"\"\"The Library of Alexandria was crucial in the ancient world as a center of scholarship and cultural development. It housed a vast collection of works, providing scholars with access to knowledge from various fields. The library's destruction is often attributed to Julius Caesar's fire during the Siege of Alexandria, a claim supported by several historical documents and archaeological findings.\"\"\"

### Response:
## Reasoning and thought process:
### Deep Analysis of the Text:
#### Content Scope and Detail: The text describes the Library of Alexandria's role, its size, patronage, and the impact of its destruction.
#### Type of Information: The information is descriptive, focusing on the library's significance and general history.

### Comprehensive Understanding of the Answer:
#### Key Components Identification: The answer highlights the library's role in scholarship and cultural development and references its destruction by Julius Caesar's fire.
#### Depth of Explanation: The answer provides both a general overview of the library's significance and specific historical claims regarding its destruction.

### Targeted Comparison of Answer with Text:
#### Content Alignment: The library's role as a center of scholarship and cultural development aligns with the text. The text does not mention Julius Caesar's fire or support this with historical documents and archaeological findings.
#### Depth Alignment: While the answer aligns in depth regarding the library's role, the specific claim about Julius Caesar and the supporting evidence is a detailed addition not present in the text.

### Identification of Extraneous Information:
#### Spotting Additional Details: The specifics about Julius Caesar's fire and the mention of historical documents and archaeological findings.
#### Assessing Impact of Additional Information: This additional information introduces a specific historical narrative and evidence not covered in the text, deviating from the text's content.

### Final Judgment on Answer Relevance:
#### Relevance Assessment: The answer starts correctly but introduces unsupported details about the library's destruction.
#### Explanation of Judgment: Due to the inclusion of specific and unsupported historical details, the final judgment is: Irrelevant.

### Instruction:
Text:
\"\"\"
The planet Venus has a very dense atmosphere composed mainly of carbon dioxide, with clouds of sulfuric acid. This composition creates a runaway greenhouse effect, leading to surface temperatures hot enough to melt lead. Venus rotates very slowly on its axis, taking about 243 Earth days to complete one rotation, which is longer than its orbital period around the Sun. Interestingly, Venus rotates in the opposite direction to most planets in the solar system. This retrograde rotation is thought to be the result of a collision with a large celestial body early in its history.
\"\"\"

Question (based on text): \"\"\"What are the main characteristics of Venus's atmosphere and rotation?\"\"\"

Supposed answer to the question (this is what you are fact-checking): \"\"\"Venus's atmosphere is dense, primarily composed of carbon dioxide, and contains clouds of sulfuric acid, leading to extremely high temperatures. The planet has a unique rotation, taking 243 Earth days to rotate once and rotating in a retrograde direction due to gravitational interactions with Earth and other planets.\"\"\"

### Response:
## Reasoning and thought process:
### Deep Analysis of the Text:
#### Content Scope and Detail: The text provides specific information on Venus's atmosphere and rotation.
#### Type of Information: The text is descriptive, outlining the characteristics of Venus's atmosphere and rotation.

### Comprehensive Understanding of the Answer:
#### Key Components Identification: The answer discusses Venus's atmosphere, temperature, rotation period, direction, and the cause of its retrograde rotation.
#### Depth of Explanation: The answer includes descriptive details and a causal explanation for the retrograde rotation.

### Targeted Comparison of Answer with Text:
#### Content Alignment: The claims about the atmosphere, temperature, rotation period, and direction align with the text.The text does not support the claim about gravitational interactions with Earth and other planets.
#### Depth Alignment: The depth of the answer aligns with the text except for the unsupported causal explanation.

### Identification of Extraneous Information:
#### Spotting Additional Details: The cause of the retrograde rotation as gravitational interactions.
#### Assessing Impact of Additional Information: This detail introduces an unsupported causal explanation and deviates from the text.

### Final Judgment on Answer Relevance:
#### Relevance Assessment: The answer is mostly relevant but introduces an unsupported causal claim.
#### Explanation of Judgment: Due to the inclusion of an unsupported reason for Venus's retrograde rotation, the final judgment is: Irrelevant.

### Instruction:
Text: \"\"\"{qatuple[2]}\"\"\"

Question (based on text): \"\"\"{qatuple[0]}\"\"\"

Supposed answer to the question (this is what you are fact-checking): \"\"\"{qatuple[1]}\"\"\"

### Response:
## Reasoning and thought process (be careful about extra details, even vague ones):
"""
        try:
            completion = logic_llm(
                decision_prompt,
                max_tokens=5500,
                stop=["</s>", "# Input:"],
                grammar=answer_relevant_grammar,
                echo=True,
                temperature=0.2,
            )["choices"][0]["text"]
            completion_pattern = re.compile(
                r"Reasoning and thought process \(be careful about extra details, even vague ones\):\n(.+)",
                re.DOTALL | re.IGNORECASE,
            )
            judgement_pattern = re.compile(
                r"Explanation of Judgment:(.+)", re.DOTALL | re.IGNORECASE
            )
            response = completion_pattern.search(completion).group(1).strip()
            print(response)
            determination = judgement_pattern.search(response).group(1).strip()
            print("\n\nDETERMINATION:\n------")
            print(determination)
            print("\n---------\n")
            if (
                "irrelevant" in determination.lower()
                or "mostly" in determination.lower()
                or "partial" in determination.lower()
                or "introduces information not present in the text"
                in determination.lower()
            ):  # Hack to get around faulty 13b outputs
                return (False, response), completion
            elif "relevant" in determination or "Relevant" in determination:
                return (True, response), completion
            else:
                retries += 1
        except:
            retries += 1
            print(
                f"Something went catastrophically wrong with this one. Investigate! Here's the completion:\n{completion}"
            )

if __name__ == "__main__":  # test
    logic_llm = Llama(
        model_path=LOGICAL_MODEL,
        n_gqa=8,
        offload_kqv=True,
        n_ctx=8000,
        n_gpu_layers=1000,
        rope_freq_scale=0.33,
        rope_scaling_type=1,
    )  # load the logical LLM and offload everything
    q_test = [
        (
            ") Explain how our understanding of planetary motion has changed over time.",
            "The understanding has evolved from the Earth being stationary and at the centre of the universe, to it orbiting the sun in an elliptical path with other planets while still rotating on its axis.",
            "The story of our world is a story that is still very imperfectly known. A couple of hundred years ago men possessed the history of little more than the last three thousand years. What happened before that time was a matter of legend and speculation.  Over a large part of the civilized world it was believed and taught that the world had been created suddenly in 4004 B.C., though authorities differed as to whether this had occurred in the spring or autumn of that year. This fantastically precise misconception was based upon a too literal interpretation of the Hebrew Bible, and upon rather arbitrary theological assumptions connected therewith.  Such ideas have long since been abandoned by religious teachers, and it is universally recognized that the universe in which we live has to all appearances existed for an enormous period of time and possibly for endless time.  Of course there may be deception in these appearances, as a room may be made to seem endless by putting mirrors facing each other at either end. But that the universe in which we live has existed only for six or seven thousand years may be regarded as an altogether exploded idea.\n\nThe earth, as everybody knows nowadays, is a spheroid, a sphere slightly compressed, orange fashion, with a diameter of nearly 8,000 miles.  Its spherical shape has been known at least to a limited number of intelligent people for nearly 2,500 years, but before that time it was supposed to be flat, and various ideas which now seem fantastic were entertained about its relations to the sky and the stars and planets.  We know now that it rotates upon its axis (which is about 24 miles shorter than its equatorial diameter) every twenty-four hours, and that this is the cause of the alternations of day and night, that it circles about the sun in a slightly distorted and slowly variable oval path in a year. Its distance from the sun varies between ninety-one and a half millions at its nearest and ninety-four and a half million miles.",
        ),
        (
            ") Identify and explain changes in human understanding throughout history regarding the age of the Earth.",