additional files/README.md

Tamil MCQ Generator Model

An advanced Natural Language Processing (NLP) system designed to automatically generate high-quality Multiple Choice Questions (MCQs) from Tamil text.

🚀 Overview

This project automates the creation of MCQs by understanding sentence structures, identifying key entities, and mathematically determining plausible distractors using state-of-the-art machine learning models specifically tailored for Tamil.

🧠 The Pipeline

The architecture is broken down into four major steps:

1. Preprocessing (preprocessing/)

   • Tokenization: Text is parsed and split into sentences and words using Tamil-specific rules.
   • Cleaning: Removes whitespace and non-Tamil characters to sanitize input.
   • Validation: Filters out short sentences or those beginning with dependent pronouns (e.g., "இது" - This).

2. Language Understanding (nlp/)

   • POS Tagging: Uses stanza to tag words with their grammatical role (Noun, Verb, etc.).
   • Named Entity Recognition (NER): Uses Hugging Face transformers to identify core entities (Locations, Persons, Concepts) within the text to use as the true answer.

3. Knowledge Base & Clustering (Root Scripts)

   • FastText Embeddings: An unsupervised fasttext model trained on a large Tamil corpus maps words into mathematical vectors.
   • HDBSCAN Clustering: Groups semantically similar words offline to build a fast-lookup map for question generation.

4. Question Generation (question_generation/)

   • Masking: Blank lines replace the chosen target word in the sentence.
   • Domain Inference: The system detects the domain (e.g., Biology, Physics) based on vocabulary.
   • Distractor Selection: Uses Cosine Similarity to select distractors from the FastText vectors. It strictly matches POS tags to ensure the options are linguistically accurate. If the strict similarity fails, it falls back to broader domain relationships to ensure 4 unique options are provided.