ladsjklf

.gitattributes

@@ -33,3 +33,10 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+data/categorizer filter=lfs diff=lfs merge=lfs -text
+models/categorizer filter=lfs diff=lfs merge=lfs -text
+__pycache__/QAPipeline.cpython-38.pyc filter=lfs diff=lfs merge=lfs -text
+__pycache__/QBModelConfig.cpython-38.pyc filter=lfs diff=lfs merge=lfs -text
+__pycache__/QBModelWrapper.cpython-38.pyc filter=lfs diff=lfs merge=lfs -text
+__pycache__/qbmodel.cpython-38.pyc filter=lfs diff=lfs merge=lfs -text
+__pycache__/tfidf_model.cpython-38.pyc filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,5 @@
+*.json
+<<<<<<< HEAD
+*.pkl
+=======
+>>>>>>> 8f336f8225704206d8ba2ab4e229f71676bdcf0e

README.md

@@ -1,199 +1,343 @@
+<<<<<<< HEAD
+=======
 ---
-library_name: transformers
-tags: []
+license: mit
+language:
+- en
+pipeline_tag: question-answering
 ---
+>>>>>>> 8f336f8225704206d8ba2ab4e229f71676bdcf0e
+The evaluation of this project is to answer trivia questions.  You do
+not need to do well at this task, but you should submit a system that
+completes the task or create adversarial questions in that setting.  This will help the whole class share data and
+resources.
 
-# Model Card for Model ID
+If you focus on something other than predicting answers, *that's fine*!  
 
-<!-- Provide a quick summary of what the model is/does. -->
+About the Data
+==============
 
+Quiz bowl is an academic competition between schools in
+English-speaking countries; hundreds of teams compete in dozens of
+tournaments each year. Quiz bowl is different from Jeopardy, a recent
+application area.  While Jeopardy also uses signaling devices, these
+are only usable after a question is completed (interrupting Jeopardy's
+questions would make for bad television).  Thus, Jeopardy is rapacious
+classification followed by a race---among those who know the
+answer---to punch a button first.
 
+Here's an example of a quiz bowl question:
 
-## Model Details
+Expanding on a 1908 paper by Smoluchowski, he derived a formula for
+the intensity of scattered light in media fluctuating densities that
+reduces to Rayleigh's law for ideal gases in The Theory of the
+Opalescence of Homogenous Fluids and Liquid Mixtures near the Critical
+State.  That research supported his theories of matter first developed
+when he calculated the diffusion constant in terms of fundamental
+parameters of the particles of a gas undergoing Brownian Motion.  In
+that same year, 1905, he also published On a Heuristic Point of View
+Concerning the Production and Transformation of Light.  That
+explication of the photoelectric effect won him 1921 Nobel in Physics.
+For ten points, name this German physicist best known for his theory
+of Relativity.
 
-### Model Description
+*ANSWER*: Albert _Einstein_
 
-<!-- Provide a longer summary of what this model is. -->
+Two teams listen to the same question. Teams interrupt the question at
+any point by "buzzing in"; if the answer is correct, the team gets
+points and the next question is read.  Otherwise, the team loses
+points and the other team can answer.
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+You are welcome to use any *automatic* method to choose an answer.  It
+need not be similar nor build on our provided systems.  In addition to
+the data we provide, you are welcome to use any external data *except*
+our test quiz bowl questions (i.e., don't hack our server!).  You are
+welcome (an encouraged) to use any publicly available software, but
+you may want to check on Piazza for suggestions as many tools are
+better (or easier to use) than others.
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+If you don't like the interruptability of questions, you can also just answer entire questions.  However, you must also output a confidence.
 
-### Model Sources [optional]
+Competition
+==================
+We will use Dynabech website (https://dynabench.org/tasks/qa). If you remember the past workshop about Dynabench submission, this is the way to do it. The specific task name is "Grounded QA". Here, with the help of the video tutorial, you submit your QA model and assess how your QA model did compared to others. The assessment will take place by testing your QA model on several QA test datasets and the results of yours and your competitors will be visible on the leaderboard. Your goal is to rank the highest in terms of expected wins: you buzz in with probability proportional to your confidence, and if you're more right than the competition, you win. 
 
-<!-- Provide the basic links for the model. -->
+Writing Questions
+==================
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+Alternatively, you can also *write* 50 adversarial questions that
+challenge modern NLP systems. These questions must be diverse in the
+subjects asked about, the skills computers need to answer the
+questions, and the entities in those questions. Remember that your questions should be *factual* and
+*specific* enough for humans to answer, because your task is to stump
+the computers relative to humans!
 
-## Uses
+In addition to the raw questions, you will also need to create citations describing:
+* Why the question is difficult for computers: include citations from the NLP/AI/ML literature
+* Why the information in the question is correct: include citations from the sources you drew on the write the question
+* Why the question is interesting: include scholarly / popular culture artifacts to prove that people care about this
+* Why the question is pyramidal: discuss why your first clues are harder than your later clues
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+**Category**
 
-### Direct Use
+We want questions from many domains such as Art, Literature, Geography, History,
+Science, TV and Film, Music, Lifestyle, and Sport. The questions
+should be written using all topics above (5 questions for each
+category and 5 more for the remaining categories). Indicate in your
+writeup which category you chose to write on for each question.
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
 
-[More Information Needed]
+Art:
 
-### Downstream Use [optional]
+* Questions about works: Mona Lisa, Raft of the Medussa
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+* Questions about forms: color, contour, texture
 
-[More Information Needed]
+* Questions about artists: Picasso, Monet, Leonardo da Vinci
 
-### Out-of-Scope Use
+* Questions about context: Renaissance, post-modernism, expressionism, surrealism
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
 
-[More Information Needed]
+Literature: 
 
-## Bias, Risks, and Limitations
+*	Questions about works: novels (1984), plays (The Lion and the Jewel), poems (Rubaiyat), criticism (Poetics)
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+*	Questions about major characters or events in literature: The Death of Anna Karenina, Noboru Wataya, the Marriage of Hippolyta and Theseus
 
-[More Information Needed]
+*	Questions about literary movements (Sturm und Drang)
 
-### Recommendations
+*	Questions about translations
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+*	Cross-cutting questions (appearances of Overcoats in novels)
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+*	Common link questions (the literary output of a country/region)
 
-## How to Get Started with the Model
 
-Use the code below to get started with the model.
+Geography: 
 
-[More Information Needed]
+*	Questions about location: names of capital, state, river
 
-## Training Details
+*	Questions about the place: temperature, wind flow, humidity
 
-### Training Data
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+History: 
 
-[More Information Needed]
+*	When: When did the First World war start? 
 
-### Training Procedure
+*	Who: Who is called Napoleon of Iran? 
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+*	Where: Where was the first Summer Olympics held?
 
-#### Preprocessing [optional]
+*	Which: Which is the oldest civilization in the world?
 
-[More Information Needed]
 
+Science: 
 
-#### Training Hyperparameters
+*	Questions about terminology: The concept of gravity was discovered by which famous physicist?
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+*	Questions about the experiment
 
-#### Speeds, Sizes, Times [optional]
+*	Questions about theory: The social action theory believes that individuals are influenced by this theory.
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
 
-[More Information Needed]
+TV and Film: 
 
-## Evaluation
+*	Quotes: What are the dying words of Charles Foster Kane in Citizen Kane?
 
-<!-- This section describes the evaluation protocols and provides the results. -->
+*	Title: What 1927 musical was the first "talkie"?
 
-### Testing Data, Factors & Metrics
+*	Plot: In The Matrix, does Neo take the blue pill or the red pill?
 
-#### Testing Data
 
-<!-- This should link to a Dataset Card if possible. -->
+Music: 
 
-[More Information Needed]
+*	Singer: What singer has had a Billboard No. 1 hit in each of the last four decades?
 
-#### Factors
+*	Band: Before Bleachers and fun., Jack Antonoff fronted what band?
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+*	Title: What was Madonna's first top 10 hit?
 
-[More Information Needed]
+*	History: Which classical composer was deaf?
 
-#### Metrics
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+Lifestyle: 
 
-[More Information Needed]
+*	Clothes: What clothing company, founded by a tennis player, has an alligator logo?
 
-### Results
+*	Decoration: What was the first perfume sold by Coco Chanel?
 
-[More Information Needed]
 
-#### Summary
+Sport: 
 
+*	Known facts: What sport is best known as the ‘king of sports’?
 
+*	Nationality: What’s the national sport of Canada?
 
-## Model Examination [optional]
+*	Sport player: The classic 1980 movie called Raging Bull is about which real-life boxer?
 
-<!-- Relevant interpretability work for the model goes here -->
+*	Country: What country has competed the most times in the Summer Olympics yet hasn’t won any kind of medal?
 
-[More Information Needed]
 
-## Environmental Impact
+**Diversity** 
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+Other than category diversity, if you find an ingenious way of writing questions about underrepresented countries, you will get bonus points (indicate which questions you included the diversity component in your writeup). You may decide which are underrepresented countries with your own reasonable reason (etc., less population may indicate underrepresented), but make sure to articulate this in your writeup. 
 
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+* Run state of the art QA systems on the questions to show they struggle, give individual results for each question and a summary over all questions
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
+For an example of what the writeup for a single question should look like, see the adversarial HW:
+https://github.com/Pinafore/nlp-hw/blob/master/adversarial/question.tex
 
-## Technical Specifications [optional]
+Proposal
+==================
 
-### Model Architecture and Objective
+The project proposal is a one page PDF document that describes:
 
-[More Information Needed]
+* Who is on your team (team sizes can be between three and six
+  students, but six is really too big to be effective; my suggestion
+  is that most groups should be between four or five).
 
-### Compute Infrastructure
+* What techniques you will explore 
 
-[More Information Needed]
+* Your timeline for completing the project (be realistic; you should
+  have your first submission in a week or two)
 
-#### Hardware
+Submit the proposal on Gradescope, but make sure to include all group
+members.  If all group members are not included, you will lose points.  Late days cannot be used on this
+assignment.
 
-[More Information Needed]
+Milestone 1
+====================== 
 
-#### Software
+You'll have to update how things are going: what's
+working, what isn't, and how does it change your timeline?  How does it change your division of labor?
 
-[More Information Needed]
+*Question Writing*: You'll need to have answers selected for all of
+your questions and first drafts of at least 15 questions.  This must
+be submitted as a JSON file so that we run computer QA systems on it.
 
-## Citation [optional]
+*Project*: You'll need to have made a submission to the leaderboard with something that satisfies the API.
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+Submit a PDF updating on your progress to Gradescope.  If all team
+members are not on the submission, you will lose points.
 
-**BibTeX:**
+Milestone 2
+===================
 
-[More Information Needed]
+As before, provide an updated timeline / division of labor, provide your intermediary results.  
 
-**APA:**
+*Question Writing*: You'll need to have reflected the feedback from the first questions and completed a first draft of at least 30 questions.  You'll also need machine results to your questions and an overall evaluation of your human/computer accuracy.
 
-[More Information Needed]
+*Project*: You'll need to have a made a submission to the leaderboard with a working system (e.g., not just obey the API, but actually get reasonable answers).
 
-## Glossary [optional]
+Submit a PDF updating on your progress.
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+Final Presentation
+======================
 
-[More Information Needed]
+The final presentation will be virtual (uploading a video).  In
+the final presentation you will:
 
-## More Information [optional]
+* Explain what you did
 
-[More Information Needed]
+* Who did what.  For example, for the question writing project a team of five people might write: A wrote the first draft of questions.  B and C verified they were initially answerable by a human.  B ran computer systems to verify they were challenging to a computer.  C edited the questions and increased the computer difficulty.  D and E verified that the edited questions were still answerable by a human.  D and E checked all of the questions for factual accuracy and created citations and the writeup.
 
-## Model Card Authors [optional]
+* What challenges you had
 
-[More Information Needed]
+* Review how well you did (based on the competition or your own metrics).  If you do not use the course infrastructure to evaluate your project's work, you should talk about what alternative evaluations you used, why they're appropriate/fair, and how well you did on them.
 
-## Model Card Contact
+* Provide an error analysis.  An error analysis must contain examples from the
+  development set that you get wrong.  You should show those sentences
+  and explain why (in terms of features or the model) they have the
+  wrong answer.  You should have been doing this all along as you
+  derive new features, but this is your final inspection of
+  your errors. The feature or model problems you discover should not
+  be trivial features you could add easily.  Instead, these should be
+  features or models that are difficult to correct.  An error analysis
+  is not the same thing as simply presenting the error matrix, as it
+  does not inspect any individual examples.  If you're writing questions, talk about examples of questions that didn't work out as intended.
 
-[More Information Needed]
+* The linguistic motivation for your features / how your wrote the questions.  This is a
+  computational linguistics class, so you should give precedence to
+  features / techniques that we use in this class (e.g., syntax,
+  morphology, part of speech, word sense, etc.).  Given two features
+  that work equally well and one that is linguistically motivated,
+  we'll prefer the linguistically motivated one.
+
+* Presumably you did many different things; how did they each
+  individually contribute to your final result?
+
+Each group has 10 minutes to deliver their presentation. Please record the video, and upload it to Google Drive, and include the link in your writeup submission.
+
+Final Question Submission
+======================
+
+Because we need to get the questions ready for the systems, upload your raw questions on May 10.  This doesn't include the citations or other parts of the writeup.
+
+System Submission
+======================
+
+You must submit a version of your system by May 12. It may not be perfect, but this what the question writing teams will use to test their results.
+
+Your system should be sent directly to the professor and TAs in zip files, including the correct dependencies and a working inference code. Your inference code should run successfully in the root folder (extracted from zip folder) directory with the command:
+
+```
+> python3 inference.py --data=evaluation_set.json 
+
+```
+
+The input will be in the form of a .json file () in the same format as the file the adversarial question writing team submits. The output format should also be in string.
+
+If you have any notes or comments that we should be aware of while running your code, please include them in the folder as a .txt file. Also, dependency information should be included as a .txt file. 
+
+Please prepend your email title with [2024-CMSC 470 System Submission].
+
+Project Writeup and JSON file
+======================
+
+By May 17, submit your project writeup explaining what
+you did and what results you achieved.  This document should
+make it clear:
+
+* Why this is a good idea
+* What you did
+* Who did what
+* Whether your technique worked or not
+
+For systems, please do not go over 2500 words unless you have a really good reason.
+Images are a much better use of space than words, usually (there's no
+limit on including images, but use judgement and be selective).
+
+For question writing, you have one page (single spaced, two column) per question plus a two page summary of results. Talk about how you organized the question writing, how you evaluated the questions, and a summary of the results.  Along with your writeup, turn in a json including the raw text of the question and answer and category. The json file is included in this directory. Make sure your json file is in the correct format and is callable via below code. Your submission will not be graded if it does not follow the format of the example json file.
+
+```
+with open('path to your json file', 'r') as f:
+    data = json.load(f)
+```
+
+
+
+Grade
+======================
+
+The grade will be out of 25 points, broken into five areas:
+
+* _Presentation_: For your oral presentation, do you highlight what
+  you did and make people care?  Did you use time well during the
+  presentation?
+
+* _Writeup_: Does the writeup explain what you did in a way that is
+  clear and effective?
+
+The final three areas are different between the system and the questions.
+
+|    |      System      |  Questions |
+|----------|:-------------:|------:|
+| _Technical Soundness_ |  Did you use the right tools for the job, and did you use them correctly?  Were they relevant to this class? | Were your questions correct and accurately cited. |
+| _Effort_ |  Did you do what you say you would, and was it the right ammount of effort.  | Are the questions well-written, interesting, and thoroughly edited? |
+| _Performance_ | How did your techniques perform in terms of accuracy, recall, etc.? | Is the human accuracy substantially higher than the computer accuracy? |
+
+<<<<<<< HEAD
+All members of the group will receive the same grade.  It's impossible for the course staff to adjudicate Rashomon-style accounts of who did what, and the goal of a group project is for all team members to work together to create a cohesive project that works well together.  While it makes sense to divide the work into distinct areas of responsibility, at grading time we have now way to know who really did what, so it's the groups responsibility to create a piece of output that reflects well on the whole group.
+=======
+All members of the group will receive the same grade.  It's impossible for the course staff to adjudicate Rashomon-style accounts of who did what, and the goal of a group project is for all team members to work together to create a cohesive project that works well together.  While it makes sense to divide the work into distinct areas of responsibility, at grading time we have now way to know who really did what, so it's the groups responsibility to create a piece of output that reflects well on the whole group.
+>>>>>>> 8f336f8225704206d8ba2ab4e229f71676bdcf0e

config.json

@@ -1,5 +1,4 @@
 {
-  "_name_or_path": "backedman/TriviaAnsweringMachine8",
   "architectures": [
     "QBModelWrapper"
   ],
@@ -7,18 +6,7 @@
     "AutoConfig": "QBModelConfig.QBModelConfig",
     "AutoModelForQuestionAnswering": "QBModelWrapper.QBModelWrapper"
   },
-  "custom_pipelines": {
-    "demo-qa": {
-      "impl": "QAPipeline.QApipeline",
-      "pt": [
-        "AutoModelForQuestionAnswering"
-      ],
-      "tf": [
-        "TFAutoModelForQuestionAnswering"
-      ]
-    }
-  },
   "model_type": "TFIDF-QA",
-  "torch_dtype": "float16",
+  "torch_dtype": "torch.float16",
   "transformers_version": "4.40.1"
-}
+}

custom_tokenizer.py

@@ -0,0 +1,7 @@
+class CustomTokenizer:
+    def __init__(self):
+        pass
+
+    def __call__(self, text):
+        # No tokenization, return input text as-is
+        return text

data/categorizer

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2c88b6bbd839788571ec706c83dc8dc51f3dcd3e2ba1569bc2d8f5525565b441
+size 26940976

data_collection.py

@@ -0,0 +1,188 @@
+import json
+from bs4 import BeautifulSoup
+import re
+from tqdm import tqdm  # Import tqdm for progress tracking
+import sys
+import question_categorizer as qc
+import numpy as np
+from question_categorizer import TextClassificationModel
+
+qc_model = qc.TextClassificationModel.load_model("models/categorizer")
+
+categories = ['Geography', 'Religion', 'Philosophy', 'Trash','Mythology', 'Literature','Science', 'Social Science', 'History', 'Current Events', 'Fine Arts']
+
+def remove_newline(string):
+    return re.sub('\n+', ' ', string)
+
+def clean_text(text, answer):
+    # Remove HTML tags
+    text = re.sub(r'<.*?>', '', text)
+    
+    #text = re.sub(r'?','.',text)
+    text = text.replace('?','.')
+    
+    # Clean the text further
+    text = re.sub(r'[^a-zA-Z.\s-]', '', text)
+    
+    
+    
+    # Remove answer from text
+    try:
+        # Preprocess the answer to replace underscores with spaces
+        processed_answer = answer.replace('_', ' ')
+        
+        # Remove parentheses from the processed answer
+        processed_answer = re.sub(r'\([^)]*\)', '', processed_answer)
+        
+        # Replace all instances of the processed answer with an empty string, ignoring case
+        text = re.sub(re.escape(processed_answer), '', text, flags=re.IGNORECASE)
+    except Exception as e:
+        print("An error occurred during text cleaning:", e)
+        print("Text:", text)
+        print("Answer:", answer)
+    
+    # Remove extra whitespaces
+    text = re.sub(r'\s+', ' ', text)
+    
+    return text.strip()
+
+def process_data():
+    #with open("data/JEOPARDY_QUESTIONS1.json", "r") as f:
+    #    jeopardy_data = json.load(f)
+    jeopardy_data = []
+
+    wiki_files = [
+    ]
+    
+    question_files = [
+        "qadata.json"]
+    
+    wiki_data = []
+    question_data = []
+    
+    for file_path in wiki_files:
+        with open('data/' + file_path, "r") as f:
+            wiki_data.extend(json.load(f))
+            
+    for file_path in question_files:
+        with open('data/' + file_path, "r") as f:
+            question_data.extend(json.load(f))
+            
+    #print(question_data)
+
+    with open("data/training_data.json", "w") as f:
+        training_data = []
+
+        # Process Jeopardy data
+        print("Processing Jeopardy data...")
+        for entry in tqdm(jeopardy_data):
+            question = entry["question"]
+            answer = str(entry["answer"])
+
+            # Preprocess the text
+            soup = BeautifulSoup(question, 'html.parser')
+            clean_question = ''.join(soup.findAll(text=True, recursive=False))
+            
+            question_category = []
+            
+            # Get category from qc_model
+            prediction = qc_model.predict(question)
+            predictions = np.argwhere(prediction >= 1.5)[1]
+            
+            for prediction_ind in predictions:
+                # Store data in array with respective index
+                question_category.append(categories[prediction_ind])
+                
+            question_category.append('ALL')
+            
+            
+
+            training_entry = {
+                "text": clean_question,
+                "answer": answer,#,
+                # Mohit, put categorizing code here
+                "category": question_category
+            }
+
+            training_data.append(training_entry)
+
+        # Process Wikipedia data
+        print("Processing Wikipedia data...")
+        for entry in tqdm(wiki_data):
+            page = str(entry["page"])
+            text = entry["text"]
+            
+            if(text == ""):
+                continue
+            
+            text = remove_newline(text)
+            text = clean_text(text, page)
+            
+            question_category = []
+            
+            # Get category from qc_model
+            prediction = qc_model.predict(text)
+            predictions = np.argwhere(prediction >= 1.5)[1]
+            
+            for prediction_ind in predictions:
+                # Store data in array with respective index
+                question_category.append(categories[prediction_ind])
+                
+            question_category.append('ALL')
+            
+
+
+            training_entry = {
+                "text": text,
+                "answer": page,
+                # Mohit, put categorizing code here
+                "category": question_category
+            }
+
+            training_data.append(training_entry)
+
+        print("Processing Misc data...")
+        for entry in tqdm(question_data):
+            
+            answer = str(entry["answer"])
+            text = entry["text"]
+            
+            if(text == "" or answer == ""):
+                continue
+            
+            text = remove_newline(text)
+            text = clean_text(text, answer)
+            
+            question_category = []
+            
+            # Get category from qc_model
+            try:
+              prediction = qc_model.predict(text)
+              predictions = np.argwhere(prediction >= 1.5)[1]
+            except:
+              print("answer: " + str(answer))
+              print("text:" + str(text))
+              continue
+            
+            for prediction_ind in predictions:
+                # Store data in array with respective index
+                question_category.append(categories[prediction_ind])
+                
+            question_category.append('ALL')
+            
+
+
+            training_entry = {
+                "text": text,
+                "answer": answer,
+                # Mohit, put categorizing code here
+                "category": question_category
+            }
+            
+            training_data.append(training_entry)
+            
+            
+
+        json.dump(training_data, f, indent=4)
+        
+process_data()

models/categorizer

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2c88b6bbd839788571ec706c83dc8dc51f3dcd3e2ba1569bc2d8f5525565b441
+size 26940976

qbmodel.py

@@ -0,0 +1,255 @@
+from typing import List, Tuple
+import nltk
+import sklearn
+import question_categorizer as qc
+from question_categorizer import TextClassificationModel
+from tfidf_model import NLPModel
+import tfidf_model
+import transformers
+import numpy as np
+import pandas as pd
+import json
+from tqdm import tqdm
+from collections import defaultdict
+
+
+
+
+
+class QuizBowlModel:
+
+    def __init__(self, clear = False):
+        """
+        Load your model(s) and whatever else you need in this function.
+
+        Do NOT load your model or resources in the guess_and_buzz() function, 
+        as it will increase latency severely. 
+        """
+        
+        self.categories = ['Geography', 'Religion', 'Philosophy', 'Trash','Mythology', 'Literature','Science', 'Social Science', 'History', 'Current Events', 'Fine Arts', 'ALL']
+        self.tfidf_models = [None for _ in range(len(self.categories))]
+        self.qc_model = qc.TextClassificationModel.load_model("models/categorizer")
+                
+        self.load_tfidf_models(clear=clear)
+
+
+
+        
+
+    def guess_and_buzz(self, question_text: List[str]) -> List[Tuple[str, bool]]:
+        """
+        This function accepts a list of question strings, and returns a list of tuples containing
+        strings representing the guess and corresponding booleans representing 
+        whether or not to buzz. 
+
+        So, guess_and_buzz(["This is a question"]) should return [("answer", False)]
+
+        If you are using a deep learning model, try to use batched prediction instead of 
+        iterating using a for loop.
+        """
+        
+        guesses = []
+        curr_question = ""
+        
+        for question in question_text:
+            curr_question += question + "."
+            
+            confidence,answer = self.predict(curr_question)
+            
+            confidence = True if confidence > 0.5 else False
+
+            guesses.append((confidence,answer))
+            
+        return guesses
+    
+    def load_tfidf_models(self, clear=False):
+        
+        print("loading tfidf models")
+        
+        # Create respective model if not exist
+        if not clear:
+            for category in range(len(self.categories)):
+                if self.tfidf_models[category] is None:
+                    self.tfidf_models[category] = NLPModel().load(f"models/{self.categories[category]}_tfidf.pkl")
+                
+            self.tfidf_models[-1] = NLPModel().load(f"models/{'ALL'}_tfidf.pkl")
+        else:
+            for category in range(len(self.categories)):
+                if self.tfidf_models[category] is None:
+                    self.tfidf_models[category] = NLPModel()
+                    
+            print(self.tfidf_models)
+                    
+        
+    
+    def train(self, data):
+                
+        # Create n empty lists, each index associated with the index of the category
+        training_data = [[] for _ in range(len(self.categories))]
+        
+        with tqdm(total=len(data)) as pbar:
+            for data_point in data:
+                text = data_point["text"]
+                answer = data_point["answer"]
+                categories = data_point["category"]
+                
+                for category in categories:
+                    
+                    category_ind = self.categories.index(category)
+                            
+                    training_data[category_ind].append({"text": text, "answer": answer})
+                    
+                pbar.update(1)
+                    
+
+        for ind,data in enumerate(training_data):
+            
+            self.tfidf_models[ind].process_data(data)
+            
+            # Train model
+            self.tfidf_models[ind].train_model()
+                
+            # Save model
+            self.tfidf_models[ind].save(f"models/{self.categories[ind]}_tfidf.pkl")
+            self.tfidf_models[ind] = None
+                
+                        
+            training_data[ind] = []
+            
+            #Update progress bar
+            #pbar.update(1)
+        
+        print("TRAINING DATA")
+        '''with tqdm(total=len(self.categories)) as pbar:
+            for category in range(len(self.categories)):
+                
+                # Train model
+                self.tfidf_models[category].train_model()
+                
+                # Save model
+                self.tfidf_models[category].save(f"models/{self.categories[category]}_tfidf.pkl")
+                
+                # Unload model
+                #print(f'category {self.categories[category]} gets unloaded')
+                self.tfidf_models[category] = None
+                training_data[category] = None
+                
+                pbar.update(1)'''
+                
+        print("Training complete.")
+
+    
+        
+    def predict(self, input_data, confidence_threshold=1.5):
+        # Get category confidence scores from qc_model
+        category_confidences = self.qc_model.predict(input_data)
+        #print("Category confidences:", category_confidences)
+        
+        # Find the indices of categories with confidence scores above the threshold
+        confident_indices = (category_confidences > confidence_threshold).nonzero()[:,1]
+        
+        #print(confident_indices)
+        
+        max_confidence = 0
+        max_answer = None
+        max_category = 0
+        for category in confident_indices:
+            #print(category)
+            confidence,answer = self.tfidf_models[category].predict(input_data)
+            
+            if(confidence > max_confidence):
+                max_confidence = confidence
+                max_answer = answer
+                max_category = category
+            
+        #max_confidence, max_answer = selected_model.predict(input_data)
+        #print("Prediction for category", self.categories[category], ":", max_answer, "with confidence", max_confidence)
+        
+        return (np.tanh(max_confidence), max_answer)
+        
+    def evaluate(self, input_data):
+        correct = 0
+        count = 0
+        
+        with tqdm(total=len(input_data)) as pbar: 
+          for data_point in input_data:
+              print(count % 10)
+              count += 1
+              text = data_point["text"]
+              answer = data_point["answer"]
+          
+              answer_predict = self.predict(text)[1]
+              
+              if(answer == answer_predict):
+                  correct += 1
+                  print(correct)
+                
+              if(count % 10 == 0):
+                  average = float(correct)/count
+                  print(f'rolling average: {average}')
+                
+              pbar.update(1)
+        
+          
+          accuracy = correct/len(input_data)
+          
+          return accuracy
+        
+            
+        
+
+        
+        
+        
+        
+        
+        
+if __name__ == "__main__":
+    # Train a simple model on QB data, save it to a file
+    import argparse
+    parser = argparse.ArgumentParser()
+
+    parser.add_argument('--data', type=str)
+    parser.add_argument('--model', type=str)
+    parser.add_argument('--predict', type=str)
+    parser.add_argument('--clear', action='store_const', const=True, default=False)
+    parser.add_argument('--evaluate', type=str)
+
+    flags = parser.parse_args()
+    model = None
+    
+    print(flags.clear)
+
+    if flags.clear:
+
+        model = QuizBowlModel(clear=True)
+        
+    else:
+
+        model = QuizBowlModel()
+
+        
+
+    if flags.data:
+        
+        data_json = []
+        
+        for data in flags.data:
+            with open(flags.data, 'r') as data_file:
+                data_json.extend(json.load(data_file))
+
+                model.train(data_json)
+            #print(model.predict("My name is bobby, bobby newport. your name is jeff?"))
+            #model.save("model.pkl")
+
+    if flags.model:
+        model.load(flags.model)
+    
+    if flags.predict:
+        print(model.predict(flags.predict))
+        
+    if flags.evaluate:
+        with open(flags.evaluate, 'r') as data_file:
+          data_json = json.load(data_file)
+          print(f'accuracy: {model.evaluate(data_json)}')
+        

question_categorizer.py

@@ -0,0 +1,272 @@
+import time
+from torch.utils.data.dataset import random_split
+from torchtext.data.functional import to_map_style_dataset
+import torch
+import gzip
+import json
+import numpy as np
+import nltk
+from nltk.corpus import stopwords
+from nltk.tokenize import word_tokenize, sent_tokenize
+from nltk.stem import PorterStemmer, WordNetLemmatizer
+from torchtext.data.utils import get_tokenizer
+from torchtext.vocab import build_vocab_from_iterator
+from torch.utils.data import DataLoader
+import argparse
+from torch import nn
+import json
+
+nltk.download('punkt')
+nltk.download('stopwords')
+nltk.download('averaged_perceptron_tagger')
+nltk.download('maxent_ne_chunker')
+nltk.download('words')
+nltk.download('wordnet')
+
+
+class TextClassificationModel(nn.Module):
+    def __init__(self, vocab_size, embed_dim, num_class, vocab):
+        self.model = super(TextClassificationModel, self).__init__()
+        self.embedding = nn.EmbeddingBag(vocab_size, embed_dim, sparse=False)
+        self.fc = nn.Linear(embed_dim, num_class)
+        self.init_weights()
+        self.vocab_size = vocab_size
+        self.emsize = embed_dim
+        self.num_class = num_class
+        self.vocab = vocab
+        self.text_pipeline = self.tokenizer
+        self.tokenizer_convert = get_tokenizer("basic_english")
+
+
+    def tokenizer(self, text):
+        return self.vocab(self.tokenizer_convert(text))
+
+    def init_weights(self):
+        initrange = 0.5
+        self.embedding.weight.data.uniform_(-initrange, initrange)
+        self.fc.weight.data.uniform_(-initrange, initrange)
+        self.fc.bias.data.zero_()
+
+    def forward(self, text, offsets):
+        embedded = self.embedding(text, offsets)
+        return self.fc(embedded)
+
+    def train_model(self, train_dataloader, valid_dataloader):
+
+        total_accu = None
+        for epoch in range(1, EPOCHS + 1):
+          epoch_start_time = time.time()
+          
+          self.train()
+          total_acc, total_count = 0, 0
+          log_interval = 500
+          start_time = time.time()
+
+          for idx, (label, text, offsets) in enumerate(train_dataloader):
+              optimizer.zero_grad()
+              predicted_label = self(text, offsets)
+              loss = criterion(predicted_label, label)
+              loss.backward()
+              torch.nn.utils.clip_grad_norm_(model.parameters(), 0.1)
+              optimizer.step()
+              total_acc += (predicted_label.argmax(1) == label).sum().item()
+              total_count += label.size(0)
+              if idx % log_interval == 0 and idx > 0:
+                  elapsed = time.time() - start_time
+                  print(
+                      "| epoch {:3d} | {:5d}/{:5d} batches "
+                      "| accuracy {:8.3f}".format(
+                          epoch, idx, len(train_dataloader), total_acc / total_count
+                      )
+                  )
+                  total_acc, total_count = 0, 0
+                  start_time = time.time()
+
+
+          accu_val = self.evaluate(valid_dataloader)
+          if total_accu is not None and total_accu > accu_val:
+              scheduler.step()
+          else:
+              total_accu = accu_val
+          print("-" * 59)
+          print(
+              "| end of epoch {:3d} | time: {:5.2f}s | "
+              "valid accuracy {:8.3f} ".format(
+                  epoch, time.time() - epoch_start_time, accu_val
+              )
+          )
+          print("-" * 59)
+
+
+    #TODO: FIX THE LOADING MODEL
+    def save_model(self, file_path):
+        model_state = {
+            'state_dict': self.state_dict(),
+            'vocab_size': self.vocab_size,
+            'embed_dim': self.emsize,
+            'num_class': self.num_class,
+            'vocab': self.vocab
+        }
+        torch.save(model_state, file_path)
+        print("Model saved successfully.")
+
+    @classmethod
+    def load_model(self, file_path):
+        model_state = torch.load(file_path, map_location=torch.device('cpu'))
+        #print(model_state)
+        vocab_size = model_state['vocab_size']
+        embed_dim = model_state['embed_dim']
+        num_class = model_state['num_class']
+        vocab = model_state['vocab']
+
+        model = TextClassificationModel(vocab_size, embed_dim, num_class, vocab)
+        model.load_state_dict(model_state['state_dict'])
+        model.eval()
+        print("Model loaded successfully.")
+        return model
+
+    def evaluate(self, dataloader):
+      self.eval()
+      total_acc, total_count = 0, 0
+
+      with torch.no_grad():
+          for idx, (label, text, offsets) in enumerate(dataloader):
+              predicted_label = self(text, offsets)
+              loss = criterion(predicted_label, label)
+              total_acc += (predicted_label.argmax(1) == label).sum().item()
+              total_count += label.size(0)
+      return total_acc / total_count
+
+    def predict(self, text):
+      with torch.no_grad():
+          text = torch.tensor(self.text_pipeline(text))
+          #print(text)
+          #print(torch.tensor([0]))
+          output = self(text, torch.tensor([0]))
+          #print(output)
+          return output
+      
+    @staticmethod
+    def read_gz_json(file_path):
+        with gzip.open(file_path, 'rt', encoding='utf-8') as f:
+            data = json.load(f)
+            for obj in data:
+                yield obj['text'], obj['category']
+
+    @staticmethod
+    def preprocess_text(text):
+        sentences = sent_tokenize(text)
+        return sentences
+
+    @staticmethod
+    def data_iter(file_paths, categories):
+
+        categories = np.array(categories)
+
+        for path in file_paths:
+            for text, category in TextClassificationModel.read_gz_json(path):
+                sentences = TextClassificationModel.preprocess_text(text)
+
+                for sentence in sentences:
+                    yield np.where(categories == category)[0][0], sentence
+    @staticmethod
+    def collate_batch(batch):
+        label_list, text_list, offsets = [], [], [0]
+        for _label, _text in batch:
+            label_list.append(label_pipeline(_label))
+            processed_text = torch.tensor(text_pipeline(_text), dtype=torch.int64)
+            text_list.append(processed_text)
+            offsets.append(processed_text.size(0))
+        label_list = torch.tensor(label_list, dtype=torch.int64)
+        offsets = torch.tensor(offsets[:-1]).cumsum(dim=0)
+        text_list = torch.cat(text_list)
+        return label_list.to(device), text_list.to(device), offsets.to(device)
+
+
+def parse_arguments():
+    parser = argparse.ArgumentParser(description="Text Classification Model")
+    parser.add_argument("--train_path", type=str, nargs='+', required=True, help="Path to the training data")
+    parser.add_argument("--test_path", type=str, nargs='+', required=True, help="Path to the test data")
+    parser.add_argument("--epochs", type=int, default=5, help="Number of epochs for training")
+    parser.add_argument("--lr", type=float, default=3, help="Learning rate")
+    parser.add_argument("--batch_size", type=int, default=64, help="Batch size for training")
+    return parser.parse_args()
+
+if __name__ == '__main__':
+
+    args = parse_arguments()
+
+    categories = ['Geography', 'Religion', 'Philosophy', 'Trash', 'Mythology', 'Literature', 'Science', 'Social Science', 'History', 'Current Events', 'Fine Arts']
+
+    test_path = args.test_path
+    train_path = args.train_path
+
+    tokenizer = get_tokenizer("basic_english")
+    train_iter = iter(TextClassificationModel.data_iter(train_path, categories))
+
+    def yield_tokens(data_iter):
+        for _, text in data_iter:
+            yield tokenizer(text)
+
+
+    vocab = build_vocab_from_iterator(yield_tokens(train_iter), specials=["<unk>"])
+    vocab.set_default_index(vocab["<unk>"])
+
+    text_pipeline = lambda x: vocab(tokenizer(x))
+    label_pipeline = lambda x: int(x)
+
+    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+
+    dataloader = DataLoader(
+        train_iter, batch_size=8, shuffle=False, collate_fn=TextClassificationModel.collate_batch
+    )
+
+    train_iter = iter(TextClassificationModel.data_iter(train_path, categories))
+    classes = set([label for (label, text) in train_iter])
+    num_class = len(classes)
+    print(num_class)
+    vocab_size = len(vocab)
+    emsize = 64
+    model = TextClassificationModel(vocab_size, emsize, num_class).to(device)
+    print(model)
+
+
+
+    # Hyperparameters
+    EPOCHS = args.epochs  # epoch
+    LR = args.lr  # learning rate
+    BATCH_SIZE = args.batch_size  # batch size for training
+
+    criterion = torch.nn.CrossEntropyLoss()
+    optimizer = torch.optim.SGD(model.parameters(), lr=LR)
+    scheduler = torch.optim.lr_scheduler.StepLR(optimizer, 1.0, gamma=0.1)
+    total_accu = None
+    train_iter = iter(TextClassificationModel.data_iter(train_path, categories))
+    test_iter = iter(TextClassificationModel.data_iter(test_path, categories))
+    train_dataset = to_map_style_dataset(train_iter)
+    test_dataset = to_map_style_dataset(test_iter)
+    num_train = int(len(train_dataset) * 0.95)
+    split_train_, split_valid_ = random_split(
+        train_dataset, [num_train, len(train_dataset) - num_train]
+    )
+
+    train_dataloader = DataLoader(
+        split_train_, batch_size=BATCH_SIZE, shuffle=True, collate_fn=TextClassificationModel.collate_batch
+    )
+    valid_dataloader = DataLoader(
+        split_valid_, batch_size=BATCH_SIZE, shuffle=True, collate_fn=TextClassificationModel.collate_batch
+    )
+    test_dataloader = DataLoader(
+        test_dataset, batch_size=BATCH_SIZE, shuffle=True, collate_fn=TextClassificationModel.collate_batch
+    )
+
+    model.train_model(train_dataloader,valid_dataloader)
+
+    print("Checking the results of test dataset.")
+    accu_test = model.evaluate(test_dataloader)
+    print("test accuracy {:8.3f}".format(accu_test))
+
+    model.save_model("text_classification_model.pth")
+
+

rand.py

@@ -0,0 +1,20 @@
+from constituent_treelib import ConstituentTree
+
+# First, we have to provide a sentence that should be parsed
+sentence = "I've got a machine learning task involving a large amount of text data."
+
+# Then, we define the language that should be considered with respect to the underlying models 
+language = ConstituentTree.Language.English
+
+# You can also specify the desired model for the language ("Small" is selected by default)
+spacy_model_size = ConstituentTree.SpacyModelSize.Medium
+
+# Next, we must create the neccesary NLP pipeline. 
+# If you wish, you can instruct the library to download and install the models automatically
+nlp = ConstituentTree.create_pipeline(language, spacy_model_size) # , download_models=True
+
+# Now, we can instantiate a ConstituentTree object and pass it the sentence and the NLP pipeline
+tree = ConstituentTree(sentence, nlp)
+
+# Finally, we can extract the phrases
+tree.extract_all_phrases()

requirements.txt

@@ -0,0 +1,225 @@
+accelerate==0.29.3
+alabaster==0.7.8
+altair==5.3.0
+annotated-types==0.6.0
+anyascii==0.3.2
+anyio==4.2.0
+asttokens==2.4.1
+attrs==19.3.0
+Automat==0.8.0
+Babel==2.6.0
+backcall==0.2.0
+beautifulsoup4==4.12.3
+benepar==0.2.0
+blinker==1.7.0
+blis==0.7.11
+bs4==0.0.2
+cachetools==5.3.3
+catalogue==2.0.10
+certifi==2019.11.28
+chardet==3.0.4
+charset-normalizer==3.3.2
+click==8.1.7
+cloud-init==23.4.4
+cloudpathlib==0.16.0
+colorama==0.4.3
+command-not-found==0.3
+confection==0.1.4
+configobj==5.0.6
+constantly==15.1.0
+constituent-treelib==0.0.7
+contractions==0.1.73
+cryptography==2.8
+cymem==2.0.8
+dbus-python==1.2.16
+decorator==5.1.1
+distro==1.9.0
+distro-info==0.23+ubuntu1.1
+docopt==0.6.2
+docutils==0.16
+entrypoints==0.3
+exceptiongroup==1.2.0
+executing==2.0.1
+falcon==3.1.3
+feedparser==5.2.1
+filelock==3.13.1
+flask==3.0.2
+fsspec==2023.12.2
+gensim==4.3.2
+gevent==23.9.1
+gitdb==4.0.11
+GitPython==3.1.43
+gitsome==0.8.0
+greenlet==3.0.3
+h11==0.14.0
+httpcore==1.0.2
+httplib2==0.14.0
+httpx==0.26.0
+huggingface-hub==0.22.2
+huspacy==0.11.0
+hyperlink==19.0.0
+idna==2.8
+ijson==3.2.3
+imagesize==1.2.0
+importlib-metadata==7.1.0
+importlib-resources==6.4.0
+incremental==16.10.1
+iniconfig==2.0.0
+install==1.3.5
+ipython==8.12.3
+itsdangerous==2.1.2
+jedi==0.19.1
+Jinja2==3.1.3
+joblib==1.3.2
+jsonpatch==1.22
+jsonpointer==2.0
+jsonschema==3.2.0
+keyring==18.0.1
+langcodes==3.3.0
+langid==1.1.6
+language-selector==0.1
+launchpadlib==1.10.13
+lazr.restfulclient==0.14.2
+lazr.uri==1.0.3
+lxml==5.1.0
+Mako==1.3.1
+markdown-it-py==3.0.0
+MarkupSafe==2.1.5
+matplotlib-inline==0.1.7
+mdurl==0.1.2
+more-itertools==4.2.0
+mpmath==1.3.0
+murmurhash==1.0.10
+netifaces==0.10.4
+networkx==3.1
+nltk==3.8.1
+numpy==1.24.4
+numpydoc==0.7.0
+nvidia-cublas-cu12==12.1.3.1
+nvidia-cuda-cupti-cu12==12.1.105
+nvidia-cuda-nvrtc-cu12==12.1.105
+nvidia-cuda-runtime-cu12==12.1.105
+nvidia-cudnn-cu12==8.9.2.26
+nvidia-cufft-cu12==11.0.2.54
+nvidia-curand-cu12==10.3.2.106
+nvidia-cusolver-cu12==11.4.5.107
+nvidia-cusparse-cu12==12.1.0.106
+nvidia-nccl-cu12==2.19.3
+nvidia-nvjitlink-cu12==12.3.101
+nvidia-nvtx-cu12==12.1.105
+oauthlib==3.1.0
+olefile==0.46
+openai==1.9.0
+packaging==20.3
+pandas==2.0.0
+parso==0.8.4
+pdfkit==1.0.0
+pexpect==4.6.0
+pickleshare==0.7.5
+pillow==10.3.0
+pluggy==1.5.0
+ply==3.11
+portalocker==2.8.2
+preshed==3.0.9
+prompt-toolkit==3.0.43
+protobuf==3.20.3
+psutil==5.9.8
+psycopg2-binary==2.9.6
+pure-eval==0.2.2
+pyahocorasick==2.1.0
+pyarrow==16.0.0
+pyasn1==0.4.2
+pyasn1-modules==0.2.1
+pydantic==2.5.3
+pydantic-core==2.14.6
+pydeck==0.9.0b1
+pygments==2.17.2
+PyGObject==3.36.0
+PyHamcrest==1.9.0
+pyinotify==0.9.6
+PyJWT==1.7.1
+pymacaroons==0.13.0
+PyNaCl==1.3.0
+pyOpenSSL==19.0.0
+pyparsing==2.4.6
+pyrsistent==0.15.5
+pyserial==3.4
+pytest==8.2.0
+python-apt==2.0.1+ubuntu0.20.4.1
+python-baseconv==1.2.2
+python-dateutil==2.8.2
+python-debian==0.1.36+ubuntu1.1
+python-dotenv==0.10.5
+pytz==2023.3
+PyYAML==5.3.1
+regex==2023.12.25
+requests==2.31.0
+requests-unixsocket==0.2.0
+rich==13.7.1
+roman==2.0.0
+safetensors==0.4.3
+scikit-learn==1.3.2
+scipy==1.10.1
+SecretStorage==2.3.1
+sentencepiece==0.2.0
+service-identity==18.1.0
+setproctitle==1.1.10
+simplejson==3.16.0
+six==1.14.0
+smart-open==6.4.0
+smmap==5.0.1
+sniffio==1.3.0
+sos==4.5.6
+soupsieve==2.5
+spacy==3.7.2
+spacy-legacy==3.0.12
+spacy-loggers==1.0.5
+Sphinx==1.8.5
+srsly==2.4.8
+ssh-import-id==5.10
+stack-data==0.6.3
+streamlit==1.33.0
+svgling==0.4.0
+svgwrite==1.4.3
+sympy==1.12
+systemd-python==234
+tenacity==8.2.3
+textblob==0.18.0.post0
+textsearch==0.0.24
+thinc==8.2.2
+threadpoolctl==3.2.0
+tokenizers==0.19.1
+toml==0.10.2
+tomli==2.0.1
+toolz==0.12.1
+torch==2.2.2
+torch-struct==0.5
+torchtext==0.17.2
+torchvision==0.16.2
+tornado==6.4
+tqdm==4.66.1
+traitlets==5.14.3
+transformers==4.40.1
+triton==2.2.0
+Twisted==18.9.0
+typer==0.9.0
+typing-extensions==4.9.0
+tzdata==2023.3
+ubuntu-pro-client==8001
+ufw==0.36
+unattended-upgrades==0.1
+Unidecode==1.3.8
+urllib3==1.26.9
+wadllib==1.3.3
+Wand==0.6.13
+wasabi==1.1.2
+watchdog==4.0.0
+wcwidth==0.1.8
+weasel==0.3.4
+werkzeug==3.0.2
+xonsh==0.9.13
+zimply==1.1.4
+zipp==3.18.1
+zope.event==5.0
+zope.interface==4.7.1
+zstandard==0.22.0

submission.py

@@ -0,0 +1,49 @@
+from QBModelConfig import QBModelConfig
+from QBModelWrapper import QBModelWrapper
+from transformers import AutoConfig, AutoModel, AutoModelForQuestionAnswering
+import torch 
+import numpy as np
+from transformers import QuestionAnsweringPipeline
+from transformers import PretrainedConfig
+from transformers.pipelines import PIPELINE_REGISTRY
+from transformers import AutoModelForQuestionAnswering, TFAutoModelForQuestionAnswering
+from transformers import pipeline
+from QAPipeline import QApipeline
+
+AutoConfig.register("TFIDF-QA", QBModelConfig)
+AutoModel.register(QBModelConfig, QBModelWrapper)
+AutoModelForQuestionAnswering.register(QBModelConfig, QBModelWrapper)
+
+QBModelConfig.register_for_auto_class()
+QBModelWrapper.register_for_auto_class("AutoModel")
+QBModelWrapper.register_for_auto_class("AutoModelForQuestionAnswering")
+
+
+
+#qbmodel_config.save_pretrained("model-config")
+#qbmodel.save_pretrained(save_directory='TriviaAnsweringMachine8', safe_serialization= False, push_to_hub=True)
+#print(qbmodel.config.torch_dtype.split(".")[1])
+from huggingface_hub import Repository
+
+repo = Repository("/mnt/c/Users/backe/Documents/GitHub/TriviaAnsweringMachine/")
+repo.push_to_hub("TriviaAnsweringMachine10")
+
+#qbmodel_config = QBModelConfig()
+#qbmodel = QBModelWrapper(qbmodel_config)
+
+#qbmodel.push_to_hub("TriviaAnsweringMachine10", safe_serialization=False)
+
+#model = AutoModelForQuestionAnswering.from_pretrained("backedman/TriviaAnsweringMachine6", config=QBModelConfig(), trust_remote_code = True)
+#tokenizer = AutoTokenizer.from_pretrained(model_name)
+
+
+PIPELINE_REGISTRY.register_pipeline(
+     "demo-qa",
+     pipeline_class=QApipeline,
+     pt_model=AutoModelForQuestionAnswering,
+     tf_model=TFAutoModelForQuestionAnswering,
+)
+
+qa_pipe = pipeline("demo-qa", model="backedman/TriviaAnsweringMachine10", tokenizer="backedman/TriviaAnsweringMachine10")
+qa_pipe.push_to_hub("TriviaAnsweringMachineREAL", safe_serialization=False)
+#qa_pipe("test")

tfidf_model.py

@@ -0,0 +1,275 @@
+import json
+import nltk
+from nltk.corpus import stopwords
+from nltk.tokenize import word_tokenize, sent_tokenize
+from nltk.stem import PorterStemmer, WordNetLemmatizer
+from sklearn.feature_extraction.text import TfidfVectorizer
+from sklearn.metrics.pairwise import cosine_similarity
+import numpy as np
+import os
+import math
+import pickle
+import joblib
+import multiprocessing
+from concurrent.futures import ProcessPoolExecutor
+from tqdm import tqdm  # Import tqdm for progress tracking
+from collections import defaultdict
+
+
+nltk.download('punkt')
+nltk.download('stopwords')
+nltk.download('averaged_perceptron_tagger')
+nltk.download('maxent_ne_chunker')
+nltk.download('words')
+nltk.download('wordnet')
+nltk.download('omw-1.4')
+
+
+# Helper function to map NLTK POS tags to WordNet POS tags
+def get_wordnet_pos(treebank_tag):
+    if treebank_tag.startswith('J'):
+        return nltk.corpus.wordnet.ADJ
+    elif treebank_tag.startswith('V'):
+        return nltk.corpus.wordnet.VERB
+    elif treebank_tag.startswith('N'):
+        return nltk.corpus.wordnet.NOUN
+    elif treebank_tag.startswith('R'):
+        return nltk.corpus.wordnet.ADV
+    else:
+        return nltk.corpus.wordnet.NOUN
+
+class NLPModel:
+    def __init__(self):  # Initialize the model with necessary parameters
+        # Initialize model components (preprocessing, training, etc.)
+        #self.model
+
+        self.tfidf = TfidfVectorizer(tokenizer=self.tokenize, lowercase=False)
+        
+        self.training_tfidf = None
+        
+        #self.manager = multiprocessing.Manager()
+        
+        self.flattened_sentences = []
+        self.training_tagged = []
+        self.answers = []
+        
+        
+        
+    def tokenize(self, text):
+        # Your tokenization logic goes here
+        return text  # No tokenization needed, return the input as-is
+
+    def preprocess_text(self, text):
+        # Tokenization
+        sentences = sent_tokenize(text)
+        
+        preprocessed_sentences = []
+        batch_size = 50  # Adjust the batch size based on your system's capabilities
+        for i in range(0, len(sentences), batch_size):
+            batch_sentences = sentences[i:i + batch_size]
+            batch_words = [word_tokenize(sentence) for sentence in batch_sentences]
+
+            # Filtering Stop Words
+            stop_words = set(stopwords.words('english'))
+            filtered_words = [[word for word in words if word.lower() not in stop_words] for words in batch_words]
+
+            # Stemming
+            stemmer = PorterStemmer()
+            stemmed_words = [[stemmer.stem(word) for word in words] for words in filtered_words]
+
+            # Tagging Parts of Speech
+            pos_tags = [nltk.pos_tag(words) for words in stemmed_words]
+
+            # Lemmatizing
+            lemmatizer = WordNetLemmatizer()
+            lemmatized_words = [[lemmatizer.lemmatize(word, pos=get_wordnet_pos(tag)) for word, tag in pos] for pos in pos_tags]
+
+            preprocessed_sentences.extend(lemmatized_words)
+
+        return preprocessed_sentences
+    
+    def process_data(self, data_json):
+        #print("Processing data in parallel...")
+        batch_size = 10000  # Experiment with different batch sizes
+        num_processes = int(multiprocessing.cpu_count()/2)  # Utilize more processes
+
+        batches = [data_json[i:i + batch_size] for i in range(0, len(data_json), batch_size)]
+        
+        #print('batches')
+
+        #training_tagged = []  # Initialize or clear self.training_tagged
+        sentence_answers = []
+
+        with ProcessPoolExecutor(max_workers=num_processes) as executor:
+            results = list(tqdm(executor.map(self.process_data_batch, batches), total=len(batches)))
+            
+        #with multiprocessing.Pool() as pool:
+        #results = []
+        #for batch in batches:
+        #results.append(self.process_data_batch(batch))
+
+        for batch_result in results:
+            for result in batch_result:
+                sentence_answers.extend(result)
+            #print("here")
+            
+        # Create a dictionary to group sentences by answer
+        answer_groups = defaultdict(list)
+
+        # Iterate through each (sentence, answer) pair in batch_results
+        for sentence, answer in sentence_answers:
+            answer_groups[answer].extend(sentence)
+            
+        #print(list(answer_groups.items())[0])
+
+        # Create a new list with sentences grouped by answer
+        sentence_answers.extend([(sentence,answer) for answer, sentence in answer_groups.items()])
+
+        self.flattened_sentences.extend([x[0] for x in sentence_answers])
+        self.training_tagged.extend([x[1] for x in sentence_answers])
+        
+        
+        
+        #print("Data processing complete.")
+
+    def process_data_batch(self, batch):
+        batch_results = []
+        
+        
+        
+        for data in batch:
+            text = data["text"]
+            answer = data["answer"]
+            preprocessed_sentences = self.preprocess_text(text)
+            training_tagged = [(sentence, answer) for sentence in preprocessed_sentences]
+            
+            
+            
+            #print(training_tagged)
+            batch_results.append(training_tagged)
+            
+        #create another list where instead, the "sentence" of elements with the same answer are appended with each other
+            
+        return batch_results
+
+    def train_model(self):
+        # Fit and transform the TF-IDF vectorizer
+        
+        #print(self.flattened_sentences)
+        if(self.flattened_sentences):
+            self.training_tfidf = self.tfidf.fit_transform(self.flattened_sentences)
+            self.flattened_sentences = []
+            #self.
+            
+        #print(self.training_tfidf)
+        #print(self.training_tagged)
+
+            
+
+
+    def save(self, file_path):
+        model_data = {
+            'training_tagged': list(self.training_tagged),
+            'tfidf': self.tfidf,
+            'training_tfidf': self.training_tfidf
+        }
+        #print(model_data)
+        with open(file_path, 'wb') as f:
+            joblib.dump(model_data, f)
+
+    def load(self, file_path):
+        
+        if os.path.exists(file_path):
+            with open(file_path, 'rb') as f:
+                print(os.path.exists(file_path))
+                model_data = joblib.load(file_path)
+                self.training_tagged = list(model_data['training_tagged'])
+                self.tfidf = model_data['tfidf']
+                print(self.tfidf)
+                self.training_tfidf = model_data['training_tfidf']
+            
+        return self
+
+    def predict(self, input_data):
+        # Preprocess input data
+        new_text_processed = self.preprocess_text(input_data)
+        new_text_processed_tfidf = self.tfidf.transform(new_text_processed)
+        training_tfidf = self.training_tfidf
+    
+        # Calculate sentence similarities
+        sentence_similarities = cosine_similarity(new_text_processed_tfidf, training_tfidf)
+    
+        # Initialize data structures
+        similarities_max = {}
+        answers = []
+    
+        # Iterate over sentence similarities
+        for similarity_row in sentence_similarities:
+            for answer, similarity in zip(self.training_tagged, similarity_row):
+                if isinstance(answer, list):
+                    continue
+                # Update similarities_max only when the new similarity is greater
+                if answer not in similarities_max or similarity > similarities_max[answer]:
+                    similarities_max[answer] = similarity
+    
+            if not answers:
+                answers.extend(similarities_max.keys())
+    
+        # Calculate total similarity for each answer and find the maximum similarity and its index
+        total_similarities = np.array([similarities_max[answer] for answer in answers])
+        closest_index = np.argmax(total_similarities)
+        closest_answer = answers[closest_index]
+    
+        return total_similarities[closest_index], closest_answer
+
+
+
+
+        
+        
+                        
+    #return (sentences.max(),self.training_tagged[closest_index])
+
+
+
+        
+    
+    def evaluate(self, test_data, labels):
+        # Evaluate the performance of the model on test data
+        # Return evaluation metrics
+        pass
+    
+    # Additional functions for model tuning, hyperparameter optimization, etc.
+
+if __name__ == "__main__":
+    # Train a simple model on QB data, save it to a file
+    import argparse
+    parser = argparse.ArgumentParser()
+
+    parser.add_argument('--data', type=str)
+    parser.add_argument('--model', type=str)
+    parser.add_argument('--predict', type=str)
+
+    flags = parser.parse_args()
+
+    model = NLPModel()
+
+    if flags.data:
+        with open(flags.data, 'r') as data_file:
+            data_json = json.load(data_file)
+
+            model.process_data(data_json)
+            model.train_model()
+            print(model.predict("My name is bobby, bobby newport. your name is jeff?"))
+            model.save("model.pkl")
+
+    if flags.model:
+        model.load(flags.model)
+    
+    if flags.predict:
+        print(model.predict(flags.predict))
+
+
+
+
+