Update src/summarization/README.md

src/summarization/README.md

@@ -1,116 +1,102 @@
-# Summarization Module 📝
-
-## Responsibility
-This module handles **text summarization and conversion to structured study notes**.
-
-## Functionality
-1. Receive transcribed text from videos.
-2. Use a **local mT5 model** (map-reduce pipeline) to analyze text and generate structured JSON notes.
-3. Produce clean Markdown output with:
-   - Source & Duration header
-   - Overall Summary
-   - Chronological Timeline (3-7 segments with Key Insight + Why It Matters)
-   - Conclusion
-
-## Pipeline (Map-Reduce)
-1. **Map** — `TranscriptSegmenter.segment_text_by_words()` splits the transcript into
-   fixed-size word chunks (default 350 words, safely under mT5's 512-token limit).
-   Each chunk is summarized independently by the local model, producing one
-   `SegmentSchema` entry per chunk.
-2. **Reduce** — All chunk-level summaries are concatenated and summarized once more
-   by the same model, producing the single overall `summary` field.
-3. **Validate** — The assembled result is passed through
-   `SummarySchema.model_validate()`. If the chunk count falls outside the
-   schema's allowed range (3-7), segments are automatically merged or padded
-   before validation. If validation still fails, a safe schema-valid fallback
-   is returned instead.
-
-## Files
-
-### 1. `schemas.py`
-- **Purpose:** Single source of truth for all Pydantic data models.
-- **Key Classes:**
-  - `SummarySchema` — Full structured output (title, detected_language, summary, segments, suggested_category, conclusion, topics).
-  - `SegmentSchema` — A timeline section (title, summary, key_insight, why_it_matters).
-
-### 2. `note_generator.py`
-- **Purpose:** Generate notes using a local mT5 model with chunk-based map-reduce and schema validation.
-- **Main Class:** `NoteGenerator`
-- **Key Methods:**
-  - `generateSummary(transcript, title)` — Runs the map-reduce pipeline and returns a `SummarySchema`-validated dict.
-  - `format_notes_to_markdown(json_notes)` — Converts JSON to clean Markdown.
-  - `format_final_notes(notes, title, url, duration)` — Wraps Markdown with Source/Duration header.
-
-### 3. `segmenter.py`
-- **Purpose:** Split long texts into smaller segments for preprocessing.
-- **Main Class:** `TranscriptSegmenter`
-- **Key Methods:**
-  - `segment_text_by_words()` — Split text into fixed-size word chunks (used by the mT5 map step).
-  - `segment_by_time()` — Split by time intervals.
-  - `segment_by_topic()` — Split by paragraph/topic boundaries.
-  - `clean_text()` — Remove filler words.
-
-## JSON Output Structure
-```json
-{
-    "title": "...",
-    "detected_language": "Arabic",
-    "summary": "Overall summary (3-5 sentences)",
-    "segments": [
-        {
-            "title": "Segment title",
-            "summary": "What this section covers",
-            "key_insight": "Most important point",
-            "why_it_matters": "Why this is valuable"
-        }
-    ],
-    "suggested_category": "General",
-    "conclusion": "Final takeaway",
-    "topics": ["Topic1", "Topic2"]
-}
+# Summarization Module
+
+Part of the `Aldea-Server` project (`src/summarization`). This module turns a raw YouTube video transcript into a structured, validated set of study notes: a general summary, a chronological timeline of segments, key insights, a conclusion, and extracted topics — in either Arabic or English, auto-detected from the transcript.
+
+## Evolution of this module
+
+The summarization approach went through three phases before reaching its current design:
+
+| Phase | Approach | Outcome |
+|---|---|---|
+| 1 | Third-party LLM API only | Fluent output, but fully dependent on an external service (cost, latency, no offline mode) |
+| 2 | Local **mT5** model | **Discontinued** — output quality (coherence, accuracy) was too weak compared to the API, especially on longer/Arabic transcripts |
+| 3 (current) | Hybrid: **Groq API** (Llama-3.3-70B) **or** local **Qwen2.5-0.5B-Instruct** | Best of both — high quality via the cloud path, or fully local/offline inference when data privacy or no-internet constraints apply |
+
+Switching between the Groq and local-Qwen backends is controlled by a single `INFERENCE_MODE` flag (see `model_loader.py` in `src/utils/`, shared across the project). Both backends share the exact same prompts, chunking logic, and output schema — the rest of the app never needs to know which one produced a given summary.
+
+> ⚠️ Evaluation note: comparisons between the three phases have been **qualitative/manual** so far (no ROUGE/BLEU scores were collected). See *Limitations* below.
+
+## Architecture (Map-Reduce pipeline)
+
+```
+Transcript ─▶ TranscriptSegmenter ─▶ MAP (summarize each chunk)
+                                            │
+                                            ▼
+                                   REDUCE (combine into one summary)
+                                            │
+                                            ▼
+                              Validate against SummarySchema (Pydantic)
+                                            │
+                                            ▼
+                          Markdown formatting (timeline + conclusion)
 ```
-> **Note:** `topics` and `suggested_category` are hidden metadata — not rendered in markdown, used by downstream modules only.
-
-## Markdown Output Order
-1. **Source** — video URL
-2. **Duration** — video length
-3. **Overall Summary** — one concise summary
-4. **Timeline** — chronological segments (3-7), each with Key Insight + Why It Matters
-5. **Conclusion** — final takeaway
-
-## Labels (Localized)
-| Key | English | Arabic |
-|-----|---------|--------|
-| source | Source | المصدر |
-| duration | Duration | المدة |
-| summary | Overall Summary | الملخص العام |
-| timeline | Timeline | التسلسل الزمني |
-| insight | Key Insight | أهم نقطة |
-| why | Why It Matters | لماذا يهم؟ |
-| conclusion | Conclusion | الخلاصة |
-
-## Testing
+
+## Files in this module
+
+| File | Responsibility |
+|---|---|
+| `note_generator.py` | `NoteGenerator` — orchestrates the Map-Reduce pipeline, language detection, schema validation, fallback handling, Markdown formatting, and the "chat with this note" Q&A feature |
+| `segmenter.py` | `TranscriptSegmenter` — splits transcripts into chunks (by word count, by time, or by topic heuristic) and cleans filler words |
+| `schemas.py` | Pydantic models (`SummarySchema`, `SegmentSchema`) that define and enforce the output contract |
+| `__init__.py` | Module entry point |
+
+> `model_loader.py` (which exposes `generate_text`, `INFERENCE_MODE`, `get_groq_client`, `_generate_text_local`) lives in `src/utils/` since it's shared by other parts of the server, not just summarization.
+
+## Usage
+
 ```python
 from src.summarization.note_generator import NoteGenerator
 
-generator = NoteGenerator()
-transcript = "Here is the complete video transcript..."
-title = "Introduction to Python"
+generator = NoteGenerator(chunk_size_words=200)
+
+result = generator.generateSummary(transcript_text, video_title)
+# result is a dict validated against SummarySchema
+
+markdown_body = generator.format_notes_to_markdown(result)
+final_note = generator.format_final_notes(
+    markdown_body,
+    video_title=video_title,
+    video_url=video_url,
+    duration=duration_seconds,
+    detected_language=result["detected_language"],
+)
 
-# Generate notes
-summary_json = generator.generateSummary(transcript, title)
-notes_md = generator.format_notes_to_markdown(summary_json)
-print(notes_md)
+# Optional: ask a follow-up question grounded in the generated note
+answer = generator.chat_with_note(final_note, "What's the main takeaway?")
 ```
 
-## Libraries Used
-- `transformers` — Load and run the local mT5 model (HuggingFace).
-- `sentencepiece` — Tokenizer backend required by mT5.
-- `langdetect` — Automatic language detection for multilingual support.
-- `torch` — PyTorch runtime for model inference.
-- `pydantic` — Data validation and schema enforcement.
-
-## Environment Variables
-| Variable | Default | Description |
-|----------|---------|-------------|
-| `MT5_MODEL_NAME` | `csebuetnlp/mT5_multilingual_XLSum` | HuggingFace model ID to load (fine-tuned for multilingual summarization, including Arabic) |
+## Output schema
+
+**`SummarySchema`**
+
+| Field | Type | Notes |
+|---|---|---|
+| `title` | `str \| None` | Carried through for formatting |
+| `detected_language` | `str \| None` | `"Arabic"` or `"English"` |
+| `summary` | `str` | 3–5 sentence overview |
+| `segments` | `list[SegmentSchema]` | 3–7 chronological sections |
+| `suggested_category` | `str` | 1–2 word category, always English |
+| `conclusion` | `str \| None` | Closing takeaway |
+| `topics` | `list[str]` | Extracted topics (min 1) |
+
+**`SegmentSchema`**: `title`, `summary`, `key_insight`, `why_it_matters`
+
+## Configuration
+
+- **`INFERENCE_MODE`** — `"groq"` to use the Groq Cloud API (Llama-3.3-70B-Versatile), anything else to use the local Qwen2.5-0.5B-Instruct path.
+- **`chunk_size_words`** (`NoteGenerator.__init__`) — default `200`. Controls how many words go into each Map-step chunk.
+- Requires a valid Groq API key/credentials when `INFERENCE_MODE = "groq"` (handled by `get_groq_client()` in `model_loader.py`).
+- `langdetect` is optional — if not installed, language detection falls back to an Arabic-character-ratio heuristic.
+
+## Limitations
+
+- No quantitative benchmark (ROUGE/BLEU) exists yet across the API / mT5 / Qwen phases — decisions so far were based on manual review.
+- The local Qwen2.5-0.5B-Instruct model is small by design (CPU-friendly), so it's less capable than the cloud Llama-3.3-70B path on long or technical transcripts.
+- Fixed word-count chunking doesn't guarantee chunks align with real topic boundaries.
+- `langdetect` can misclassify very short transcripts.
+
+## Possible next steps
+
+- Add a quantitative evaluation pipeline (ROUGE-1/2/L or a small human-rated set) to formally compare phases and catch regressions.
+- Try light fine-tuning of the local Qwen model on Arabic/English video-summary pairs.
+- Move from fixed-size chunking to a topic/semantic-aware segmentation strategy.