Add writing_pattern_classifier package for live demo

writing_pattern_classifier/README.md

@@ -0,0 +1,163 @@
+# Dyslexic Writing-Pattern Classifier (Sinhala)
+
+This module implements an **interpretable, rule-based dyslexic writing-pattern classifier** for Sinhala text.
+
+Unlike traditional machine-learning classifiers, this component focuses on **pattern inference and explainability**, rather than predictive accuracy.  
+It is designed to analyze _how_ dyslexic writing manifests, not merely _whether_ dyslexia is present.
+
+---
+
+## Purpose
+
+- Identify **dominant dyslexic writing patterns** in Sinhala text
+- Provide **explainable, linguistically grounded analysis**
+- Support educational and research-oriented dyslexia-aware systems
+
+This module is executed **only after** an essay has been identified as dyslexic by the Binary Dyslexia Detector.
+
+---
+
+## Core Design Principle
+
+> Dyslexia is expressed through **consistent patterns of surface-level writing errors**, not isolated mistakes.
+
+Therefore, this classifier infers patterns using **rule-based dominance of error signals**, rather than supervised learning.
+
+---
+
+## Writing Patterns Identified
+
+The system currently identifies the following dyslexic writing patterns:
+
+- **Orthographic Instability**  
+  Frequent character omissions, additions, or diacritic loss
+
+- **Phonetic Confusion**  
+  Character substitutions reflecting phonetic similarity
+
+- **Mixed Dyslexic Pattern**  
+  Co-occurrence of multiple dominant error types
+
+- **No Dominant Pattern**  
+  Absence of consistent dyslexic error behavior
+
+- **Word Boundary Confusion** (when applicable)  
+  Spacing and word segmentation errors
+
+These patterns are derived from dyslexia-related literature and adapted for Sinhala writing.
+
+---
+
+## Processing Pipeline
+
+### 1. Sentence-Level Analysis
+
+For each sentence:
+
+- Clean and dyslexic versions are compared
+- Surface error features are extracted:
+  - Character addition
+  - Character omission
+  - Character substitution
+  - Diacritic loss
+  - Spacing issues
+- A **rule-based inference engine** assigns a sentence-level writing pattern
+
+### 2. Essay-Level Aggregation
+
+Because the dataset does not provide explicit essay boundaries:
+
+- Essays are approximated using **fixed-size sentence windows** (pseudo-essays)
+- Sentence-level patterns are aggregated per essay
+
+### 3. Dominant Pattern Classification
+
+For each essay:
+
+- The most frequent pattern is selected as the **dominant pattern**
+- A **confidence score** is computed as:
+
+\[
+Confidence = \frac{\text{Number of sentences supporting dominant pattern}}
+{\text{Total number of sentences in essay}}
+\]
+
+- Dominance strength is categorized as:
+  - Strong Dominance
+  - Moderate Dominance
+  - Weak / Mixed
+
+---
+
+## Outputs
+
+For each essay, the classifier produces:
+
+- Dominant dyslexic writing pattern
+- Pattern dominance confidence
+- Dominance strength label
+- Sentence-level pattern breakdown (for explainability)
+
+### Example Output
+
+```json
+{
+  "dominant_pattern": "Orthographic Instability",
+  "confidence": 0.6,
+  "dominance_strength": "Strong Dominance"
+}
+
+---
+
+## Evaluation Strategy
+
+This component does not use supervised evaluation metrics such as accuracy or F1-score.
+
+Reason:
+
+- Essay-level pattern labels are inferred, not manually annotated
+
+- Reporting accuracy would result in label leakage
+
+Instead, evaluation is performed using:
+
+- Pattern distribution analysis
+
+- Confidence distribution statistics
+
+- Qualitative case studies with sentence-level evidence
+
+This approach aligns with best practices in dyslexia-related linguistic analysis.
+
+## Notebooks
+
+notebooks/
+├── 01_surface_feature_extraction_and_pattern_inference_v3.ipynb
+└── 02_essay_level_dyslexic_pattern_profiling.ipynb
+
+These notebooks document the full development and validation process.
+
+## Limitations
+
+Essay boundaries are approximated using fixed-size sentence windows
+
+The system does not perform clinical diagnosis
+
+Pattern definitions may evolve with expert validation
+
+## Role in the Overall System
+
+(Binary Dyslexia Detector)
+          ↓
+Dyslexic Essay
+          ↓
+Writing-Pattern Classifier
+          ↓
+Pattern Profile + Confidence
+
+## Disclaimer
+
+This module is intended for research and educational purposes only and should not be used for clinical diagnosis.
+
+Generated CSV artifacts are intentionally excluded from version control and can be reproduced by executing the notebooks or pipeline.
+```

writing_pattern_classifier/artifacts/pattern_taxonomy.md

@@ -0,0 +1,79 @@
+# Sinhala Dyslexic Writing Pattern Taxonomy
+
+This document defines the interpretable dyslexic writing-pattern taxonomy used in this project.
+
+The taxonomy is derived from surface-level orthographic and phonetic deviations observed in Sinhala dyslexic writing.
+
+---
+
+## 1. Orthographic Instability
+
+**Definition:**  
+Inconsistent or incorrect written forms of characters without strong phonetic substitution.
+
+**Surface Signals:**
+
+- Character omission
+- Character addition
+- Diacritic loss
+- Inconsistent spelling
+
+**Example:**
+
+- Clean: රුපියල් දෙදාහක් තියෙනවා
+- Dyslexic: රුපියල් දෙදාහක් තියනව
+
+---
+
+## 2. Phonetic Confusion
+
+**Definition:**  
+Errors that reflect confusion between phonologically similar sounds.
+
+**Surface Signals:**
+
+- Character substitution
+- Phonetically similar replacements
+
+**Example:**
+
+- Clean: ගණිත
+- Dyslexic: ගනිත
+
+---
+
+## 3. Word Boundary Confusion
+
+**Definition:**  
+Difficulty maintaining correct word segmentation.
+
+**Surface Signals:**
+
+- Word merges
+- Extra spaces
+- Missing spaces
+
+---
+
+## 4. Mixed Dyslexic Pattern
+
+**Definition:**  
+Presence of multiple dyslexic patterns within the same sentence or essay.
+
+**Criteria:**
+
+- More than one dominant surface error type
+
+---
+
+## 5. No Dominant Pattern
+
+**Definition:**  
+No consistent dyslexic pattern detected or very low error density.
+
+---
+
+## Notes
+
+- Patterns are assigned using rule-based dominance logic.
+- This system prioritizes explainability over raw accuracy.

writing_pattern_classifier/src/essay_profile.py

@@ -0,0 +1,86 @@
+"""
+Essay-level dyslexic writing pattern profiling.
+
+This module aggregates sentence-level dyslexic writing patterns
+into dominance-based essay profiles.
+"""
+
+import pandas as pd
+
+
+def assign_essay_ids(df: pd.DataFrame, essay_size: int = 5) -> pd.DataFrame:
+    """
+    Assign essay IDs to sentence-level data using fixed-size grouping.
+
+    Parameters
+    ----------
+    df : pd.DataFrame
+        DataFrame containing sentence-level patterns.
+    essay_size : int
+        Number of sentences per essay abstraction.
+
+    Returns
+    -------
+    pd.DataFrame
+        DataFrame with an added 'essay_id' column.
+    """
+    df = df.copy()
+    df["essay_id"] = df.index // essay_size
+    return df
+
+
+def profile_essays(df: pd.DataFrame) -> pd.DataFrame:
+    """
+    Aggregate sentence-level patterns into essay-level dominance profiles.
+
+    Parameters
+    ----------
+    df : pd.DataFrame
+        DataFrame containing 'essay_id' and 'writing_pattern'.
+
+    Returns
+    -------
+    pd.DataFrame
+        Essay-level pattern profiles with dominance and confidence.
+    """
+
+    # Count patterns per essay
+    pattern_counts = (
+        df
+        .groupby("essay_id")["writing_pattern"]
+        .value_counts()
+        .unstack(fill_value=0)
+    )
+
+    essay_summary = pattern_counts.copy()
+
+    # Dominant pattern
+    essay_summary["dominant_pattern"] = essay_summary.idxmax(axis=1)
+
+    # Compute dominance metrics
+    pattern_columns = pattern_counts.columns
+    essay_summary["max_count"] = essay_summary[pattern_columns].max(axis=1)
+    essay_summary["total_sentences"] = essay_summary[pattern_columns].sum(axis=1)
+
+    essay_summary["confidence"] = (
+        essay_summary["max_count"] / essay_summary["total_sentences"]
+    )
+
+    # Dominance strength categorization
+    essay_summary["dominance_strength"] = essay_summary["confidence"].apply(
+        dominance_strength
+    )
+
+    return essay_summary.reset_index()
+
+
+def dominance_strength(confidence: float) -> str:
+    """
+    Categorize dominance strength based on confidence score.
+    """
+    if confidence >= 0.6:
+        return "Strong"
+    elif confidence >= 0.4:
+        return "Moderate"
+    else:
+        return "Weak / Mixed"

writing_pattern_classifier/src/feature_extraction.py

@@ -0,0 +1,75 @@
+"""
+Sentence-level surface feature extraction for Sinhala dyslexic writing analysis.
+
+This module computes interpretable surface-level error signals
+by comparing clean and dyslexic sentence pairs.
+"""
+
+import difflib
+
+# Sinhala diacritic characters
+SINHALA_DIACRITICS = set([
+    "ා", "ැ", "ෑ", "ි", "ී", "ු", "ූ", "ෘ", "ෙ", "ේ", "ො", "ෝ", "ං", "ඃ"
+])
+
+
+def char_level_diff(clean: str, dyslexic: str) -> dict:
+    """
+    Compute character-level edit operations between clean and dyslexic sentences.
+    """
+    matcher = difflib.SequenceMatcher(None, clean, dyslexic)
+
+    additions = omissions = substitutions = 0
+
+    for tag, i1, i2, j1, j2 in matcher.get_opcodes():
+        if tag == "insert":
+            additions += (j2 - j1)
+        elif tag == "delete":
+            omissions += (i2 - i1)
+        elif tag == "replace":
+            substitutions += max(i2 - i1, j2 - j1)
+
+    return {
+        "char_addition": additions,
+        "char_omission": omissions,
+        "char_substitution": substitutions,
+        "has_addition": additions > 0,
+        "has_omission": omissions > 0,
+        "has_substitution": substitutions > 0,
+    }
+
+
+def spacing_diff(clean: str, dyslexic: str) -> dict:
+    """
+    Detect word boundary (spacing) inconsistencies.
+    """
+    diff = abs(len(clean.split()) - len(dyslexic.split()))
+    return {
+        "word_count_diff": diff,
+        "has_spacing_issue": diff > 0,
+    }
+
+
+def diacritic_loss(clean: str, dyslexic: str) -> dict:
+    """
+    Detect diacritic loss in dyslexic writing.
+    """
+    clean_count = sum(1 for c in clean if c in SINHALA_DIACRITICS)
+    dys_count = sum(1 for c in dyslexic if c in SINHALA_DIACRITICS)
+
+    return {
+        "has_diacritic_loss": clean_count > dys_count
+    }
+
+
+def extract_surface_features(clean_sentence: str, dyslexic_sentence: str) -> dict:
+    """
+    Extract all sentence-level surface features.
+    """
+    features = {}
+
+    features.update(char_level_diff(clean_sentence, dyslexic_sentence))
+    features.update(spacing_diff(clean_sentence, dyslexic_sentence))
+    features.update(diacritic_loss(clean_sentence, dyslexic_sentence))
+
+    return features

writing_pattern_classifier/src/pattern_rules.py

@@ -0,0 +1,46 @@
+"""
+Rule-based sentence-level dyslexic writing pattern inference.
+
+This module implements dominance-aware, interpretable rules
+for identifying dyslexic writing patterns from surface features.
+"""
+
+
+def infer_pattern(features: dict) -> str:
+    """
+    Infer the dominant dyslexic writing pattern for a sentence
+    using surface-level error signals.
+
+    Parameters
+    ----------
+    features : dict
+        Dictionary containing extracted surface features.
+
+    Returns
+    -------
+    str
+        One of the predefined dyslexic writing pattern labels.
+    """
+
+    # Priority 1: Word boundary confusion
+    if features.get("has_spacing_issue"):
+        return "Word Boundary Confusion"
+
+    has_sub = features.get("has_substitution", False)
+    has_omit = features.get("has_omission", False)
+    has_diacritic = features.get("has_diacritic_loss", False)
+
+    # Priority 2: Mixed dyslexic pattern
+    if has_sub and has_omit:
+        return "Mixed Dyslexic Pattern"
+
+    # Priority 3: Phonetic confusion
+    if has_sub:
+        return "Phonetic Confusion"
+
+    # Priority 4: Orthographic instability
+    if has_omit or has_diacritic:
+        return "Orthographic Instability"
+
+    # Fallback
+    return "No Dominant Pattern"

writing_pattern_classifier/src/pipeline.py

@@ -0,0 +1,66 @@
+"""
+End-to-end pipeline for Sinhala dyslexic writing pattern analysis.
+
+This module orchestrates sentence-level feature extraction,
+pattern inference, and essay-level profiling.
+"""
+
+import pandas as pd
+
+from .feature_extraction import extract_surface_features
+from .pattern_rules import infer_pattern
+from .essay_profile import assign_essay_ids, profile_essays
+
+
+def run_pattern_analysis(
+    df: pd.DataFrame,
+    essay_size: int = 5
+) -> tuple[pd.DataFrame, pd.DataFrame]:
+    """
+    Run the complete dyslexic writing pattern analysis pipeline.
+
+    Parameters
+    ----------
+    df : pd.DataFrame
+        Input DataFrame containing:
+        - 'clean_sentence'
+        - 'dyslexic_sentence'
+    essay_size : int
+        Number of sentences per essay abstraction.
+
+    Returns
+    -------
+    tuple (sentence_df, essay_df)
+        sentence_df : pd.DataFrame
+            Sentence-level features and inferred patterns.
+        essay_df : pd.DataFrame
+            Essay-level dominance profiles.
+    """
+
+    df = df.copy()
+
+    # --- Sentence-level feature extraction ---
+    surface_features = df.apply(
+        lambda row: extract_surface_features(
+            row["clean_sentence"],
+            row["dyslexic_sentence"]
+        ),
+        axis=1
+    )
+
+    feature_df = pd.concat(
+        [df.reset_index(drop=True), surface_features.apply(pd.Series)],
+        axis=1
+    )
+
+    # --- Sentence-level pattern inference ---
+    feature_df["writing_pattern"] = feature_df.apply(
+        lambda row: infer_pattern(row),
+        axis=1
+    )
+
+    # --- Essay-level profiling ---
+    feature_df = assign_essay_ids(feature_df, essay_size=essay_size)
+    essay_df = profile_essays(feature_df)
+
+    return feature_df, essay_df