Upload docs/PROJECT_REPORT_SKELETON.md with huggingface_hub

docs/PROJECT_REPORT_SKELETON.md

@@ -1,128 +1,128 @@
-# Project Report Skeleton for MVM²
-
-This document serves as a structured template for the final project report. It combines Markdown structure with standard LaTeX placeholders where applicable for formal formatting.
-
----
-
-# 1. Introduction
-
-## 1.1 Problem Statement
-The verification of mathematical reasoning generated by Large Language Models (LLMs) faces two distinct challenges:
-1.  **Hallucination:** LLMs often produce "step-by-step" reasoning that appears plausible but contains logical gaps or contradictions.
-2.  **Multimodal Noise:** When the input source is an image (handwritten or printed), Optical Character Recognition (OCR) errors introduce uncertainty (e.g., misinterpreting symbols like $\int$ vs $S$). Existing pipelines typically treat transcribed text as ground truth, leading to catastrophic error propagation.
-
-**Objective:** To develop *MVM²*, a multimodal verification system that integrates OCR confidence scores, symbolic execution, and multi-agent consensus to robustly verify mathematical solutions.
-
-## 1.2 Motivation
-- Requirement for trusted AI in education (automated grading, tutoring).
-- Limitations of "black box" verifiers (GPT-4) which lack explainability.
-- Need for formally verifiable metrics (SymPy) combined with semantic understanding (LLMs).
-
----
-
-# 2. Literature Review
-
-| Paper / Tool | Key Contribution | Limitation Addressed by MVM² |
-|---|---|---|
-| **Math-Verify (HuggingFace)** | Rule-based answer extraction & equivalency. | Lacks semantic logic checking; purely symbolic. |
-| **MathVerse (ECCV 2024)** | Multimodal benchmark for visual math. | Focuses on evaluation, not the *verification algorithm* itself. |
-| **Self-Consistency (Wang et al.)** | Majority voting for LLMs. | Computationally expensive; doesn't handle visual uncertainty. |
-
----
-
-# 3. Methodology & System Architecture
-
-## 3.1 Overview
-The system adopts a modular microservice architecture consisting of seven key components:
-1.  **Input Receiver:** Validates multimodal inputs.
-2.  **Preprocessing:** Image binarization and noise reduction (OpenCV).
-3.  **OCR Service:** Hybrid extraction (Tesseract + Handwritten CNN).
-4.  **Representation:** Canonicalization to Intermediate Representation (IR).
-5.  **Verification Engine:** SymPy (Symbolic) + Multi-Agent LLMs (Logical).
-6.  **Classifier:** Weighted Consensus Scoring.
-7.  **Reporting:** Explainable feedback generation.
-
-## 3.2 Formal Methods
-
-### 3.2.1 OCR-Aware Confidence Propagation
-We propose a novel method to discount verification confidence based on visual uncertainty. Let $C_{ocr}$ be the OCR confidence score. The final confidence $C_{final}$ is calibrated as:
-
-$$
-C_{final} = S_{weighted} \times (\lambda + (1-\lambda)C_{ocr})
-$$
-
-Where $\lambda=0.9$ ensures a high floor for legibility but penalizes ambiguity.
-
-### 3.2.2 Hybrid Scoring Function
-The validity score $S_{weighted}$ is computed from three independent signals:
-
-$$
-S_{weighted} = \alpha S_{sym} + \beta S_{log} + \gamma S_{clf}
-$$
-
-**Parameters:**
-- $\alpha = 0.40$ (Symbolic Accuracy)
-- $\beta = 0.35$ (Logical Consistency)
-- $\gamma = 0.25$ (Classifier Consensus)
-
-## 3.3 Multi-Agent Consensus
-We utilize three agents with distinct prompts:
-1.  **Solver:** Independently solves the problem.
-2.  **Critic:** Reviews the provided steps for logical fallacies.
-3.  **Verifier:** Compares the Solver and User steps.
-
-**Hallucination Rate ($H$)** is defined as the fraction of steps where agents fail to reach consensus (similarity threshold $< 0.7$).
-
----
-
-# 4. Experiments
-
-## 4.1 Dataset
-- **Sources:** Hand-curated samples, modified GSM8K subset.
-- **Types:** Text-only, Clean Images, Noisy Images (Gaussian noise added).
-
-## 4.2 Ablation Studies
-We evaluated four configurations to quantify the contribution of each component:
-
-| Mode | Description |
-|---|---|
-| **M1: Baseline** | Single LLM (Gemini-Pro) without SymPy or OCR weighting. |
-| **M2: Hybrid** | Single LLM + SymPy verification. |
-| **M3: Consensus** | Multi-Agent LLM + SymPy (No OCR calibration). |
-| **M4: Full MVM²** | Full pipeline with OCR-aware confidence. |
-
----
-
-# 5. Results & Analysis
-
-## 5.1 Quantitative Results via `run_evaluation.py`
-
-*(Insert table from `evaluation_results.csv` here)*
-
-| Mode | Accuracy | Hallucination Rate | Avg Latency |
-|---|---|---|---|
-| M1 (Baseline) | Low | High | Low |
-| M4 (Full) | **High** | **Low** | Moderate |
-
-**Key Finding:** The multi-agent approach reduced the hallucination rate by **X%** compared to the baseline.
-
-## 5.2 Case Studies
-- **Case A (Ambiguous Handwriting):** M4 correctly flagged "Low Confidence" due to OCR uncertainty, whereas M1 confidently marked it incorrect based on bad transcription.
-- **Case B (Algebraic Error):** SymPy component ($S_{sym}$) detected a subtle sign error that the LLM ($S_{log}$) missed.
-
----
-
-# 6. Limitations & Future Work
-
-## 6.1 Limitations
-- **Latency:** Multi-agent calls increase response time (~3-4s).
-- **OCR Dependency:** Extremely poor handwriting still fails early in the pipeline.
-
-## 6.2 Future Work
-- **Fine-tuning:** Train a dedicated small model (SLM) for the Critic role to reduce latency.
-- **Visual-LLM Integration:** Feed images directly to Gemini 1.5 Pro to bypass OCR for complex geometry problems.
-
----
-
-# 7. Conclusion
-MVM² successfully demonstrates that integrating formal symbolic methods with probabilistic LLM reasoning—calibrated by visual uncertainty—significantly improves the reliability of mathematical verification systems.
+# Project Report Skeleton for MVM²
+
+This document serves as a structured template for the final project report. It combines Markdown structure with standard LaTeX placeholders where applicable for formal formatting.
+
+---
+
+# 1. Introduction
+
+## 1.1 Problem Statement
+The verification of mathematical reasoning generated by Large Language Models (LLMs) faces two distinct challenges:
+1.  **Hallucination:** LLMs often produce "step-by-step" reasoning that appears plausible but contains logical gaps or contradictions.
+2.  **Multimodal Noise:** When the input source is an image (handwritten or printed), Optical Character Recognition (OCR) errors introduce uncertainty (e.g., misinterpreting symbols like $\int$ vs $S$). Existing pipelines typically treat transcribed text as ground truth, leading to catastrophic error propagation.
+
+**Objective:** To develop *MVM²*, a multimodal verification system that integrates OCR confidence scores, symbolic execution, and multi-agent consensus to robustly verify mathematical solutions.
+
+## 1.2 Motivation
+- Requirement for trusted AI in education (automated grading, tutoring).
+- Limitations of "black box" verifiers (GPT-4) which lack explainability.
+- Need for formally verifiable metrics (SymPy) combined with semantic understanding (LLMs).
+
+---
+
+# 2. Literature Review
+
+| Paper / Tool | Key Contribution | Limitation Addressed by MVM² |
+|---|---|---|
+| **Math-Verify (HuggingFace)** | Rule-based answer extraction & equivalency. | Lacks semantic logic checking; purely symbolic. |
+| **MathVerse (ECCV 2024)** | Multimodal benchmark for visual math. | Focuses on evaluation, not the *verification algorithm* itself. |
+| **Self-Consistency (Wang et al.)** | Majority voting for LLMs. | Computationally expensive; doesn't handle visual uncertainty. |
+
+---
+
+# 3. Methodology & System Architecture
+
+## 3.1 Overview
+The system adopts a modular microservice architecture consisting of seven key components:
+1.  **Input Receiver:** Validates multimodal inputs.
+2.  **Preprocessing:** Image binarization and noise reduction (OpenCV).
+3.  **OCR Service:** Hybrid extraction (Tesseract + Handwritten CNN).
+4.  **Representation:** Canonicalization to Intermediate Representation (IR).
+5.  **Verification Engine:** SymPy (Symbolic) + Multi-Agent LLMs (Logical).
+6.  **Classifier:** Weighted Consensus Scoring.
+7.  **Reporting:** Explainable feedback generation.
+
+## 3.2 Formal Methods
+
+### 3.2.1 OCR-Aware Confidence Propagation
+We propose a novel method to discount verification confidence based on visual uncertainty. Let $C_{ocr}$ be the OCR confidence score. The final confidence $C_{final}$ is calibrated as:
+
+$$
+C_{final} = S_{weighted} \times (\lambda + (1-\lambda)C_{ocr})
+$$
+
+Where $\lambda=0.9$ ensures a high floor for legibility but penalizes ambiguity.
+
+### 3.2.2 Hybrid Scoring Function
+The validity score $S_{weighted}$ is computed from three independent signals:
+
+$$
+S_{weighted} = \alpha S_{sym} + \beta S_{log} + \gamma S_{clf}
+$$
+
+**Parameters:**
+- $\alpha = 0.40$ (Symbolic Accuracy)
+- $\beta = 0.35$ (Logical Consistency)
+- $\gamma = 0.25$ (Classifier Consensus)
+
+## 3.3 Multi-Agent Consensus
+We utilize three agents with distinct prompts:
+1.  **Solver:** Independently solves the problem.
+2.  **Critic:** Reviews the provided steps for logical fallacies.
+3.  **Verifier:** Compares the Solver and User steps.
+
+**Hallucination Rate ($H$)** is defined as the fraction of steps where agents fail to reach consensus (similarity threshold $< 0.7$).
+
+---
+
+# 4. Experiments
+
+## 4.1 Dataset
+- **Sources:** Hand-curated samples, modified GSM8K subset.
+- **Types:** Text-only, Clean Images, Noisy Images (Gaussian noise added).
+
+## 4.2 Ablation Studies
+We evaluated four configurations to quantify the contribution of each component:
+
+| Mode | Description |
+|---|---|
+| **M1: Baseline** | Single LLM (Gemini-Pro) without SymPy or OCR weighting. |
+| **M2: Hybrid** | Single LLM + SymPy verification. |
+| **M3: Consensus** | Multi-Agent LLM + SymPy (No OCR calibration). |
+| **M4: Full MVM²** | Full pipeline with OCR-aware confidence. |
+
+---
+
+# 5. Results & Analysis
+
+## 5.1 Quantitative Results via `run_evaluation.py`
+
+*(Insert table from `evaluation_results.csv` here)*
+
+| Mode | Accuracy | Hallucination Rate | Avg Latency |
+|---|---|---|---|
+| M1 (Baseline) | Low | High | Low |
+| M4 (Full) | **High** | **Low** | Moderate |
+
+**Key Finding:** The multi-agent approach reduced the hallucination rate by **X%** compared to the baseline.
+
+## 5.2 Case Studies
+- **Case A (Ambiguous Handwriting):** M4 correctly flagged "Low Confidence" due to OCR uncertainty, whereas M1 confidently marked it incorrect based on bad transcription.
+- **Case B (Algebraic Error):** SymPy component ($S_{sym}$) detected a subtle sign error that the LLM ($S_{log}$) missed.
+
+---
+
+# 6. Limitations & Future Work
+
+## 6.1 Limitations
+- **Latency:** Multi-agent calls increase response time (~3-4s).
+- **OCR Dependency:** Extremely poor handwriting still fails early in the pipeline.
+
+## 6.2 Future Work
+- **Fine-tuning:** Train a dedicated small model (SLM) for the Critic role to reduce latency.
+- **Visual-LLM Integration:** Feed images directly to Gemini 1.5 Pro to bypass OCR for complex geometry problems.
+
+---
+
+# 7. Conclusion
+MVM² successfully demonstrates that integrating formal symbolic methods with probabilistic LLM reasoning—calibrated by visual uncertainty—significantly improves the reliability of mathematical verification systems.