Spaces:

bayan10
/

bayan-api

Running

App Files Files Community

bayan-api / archive /phase_reports /PHASE_4_DOCUMENT_MANAGEMENT.md

youssefreda9

Cleanup: Archive legacy scripts + phase reports, delete 35 orphaned files

78a516a about 1 month ago

preview code

Raw

History Blame Contribute Delete

21.7 kB

Phase 4 — Document Management

Status: Architecture Plan (Pre-Implementation)
Date: June 15, 2026
Goal: Transform Bayan from a text editor into a document editor with import/export
Prerequisites: Phase 1 editor engine complete; Phase 2 theme system in place

Executive Summary
Architecture Plan
Required Libraries
File Structure Changes
Feature Specifications
UI Design
Implementation Roadmap
Risk Analysis
Testing Checklist
Approval Gate

1. Executive Summary

Phase 4 adds client-side document I/O without changing the Phase 1 highlight pipeline. All import paths produce plain text loaded through existing selection.js APIs. All export paths read plain text via getEditorText(). The offset-based renderer, cursor preservation, and analyze flow remain untouched.

Format	Direction	Library	Complexity
`.txt`	Import + Export	Native `FileReader` + Blob	Low
`.docx`	Import	Mammoth.js	Medium
`.docx`	Export	docx.js	Medium–High (RTL)
`.pdf`	Export	html2pdf.js (recommended)	Medium–High (Arabic)

Out of scope for Phase 4: Server-side conversion, cloud storage, auth, database persistence, .doc (legacy binary), rich formatting preservation, images/tables in DOCX.

2. Architecture Plan

2.1 Current Editor Data Flow (unchanged)

User types → contenteditable #editor-container
           → getEditorText()                    [selection.js]
           → /api/analyze
           → render({ text, suggestions })        [renderer.js]
           → setEditorHTML(html)                [selection.js]
           → restoreSelection()                 [selection.js]

Phase 4 inserts a parallel I/O layer that only talks to the editor through two public seams:

Seam	Function	Owner
Read	`getEditorText()`	`selection.js` — do not modify
Write	`setEditorHTML(escapeHtml(text))` + trigger analysis	via new `loadDocumentText()` in `editor.js`

2.2 Proposed Document Layer

┌─────────────────────────────────────────────────────────────┐
│  UI: Import button · Export dropdown · hidden <input type=file>│
└──────────────────────────┬──────────────────────────────────┘
                           │
              ┌────────────▼────────────┐
              │   documents.js (facade)  │
              │   initDocuments()        │
              └────────────┬────────────┘
           ┌───────────────┼───────────────┐
           │               │               │
    ┌──────▼──────┐ ┌──────▼──────┐ ┌─────▼─────┐
    │  import.js  │ │  export.js  │ │  utils.js │
    │  · importTxt│ │  · exportTxt│ │  · download│
    │  · importDocx│ │ · exportDocx│ │  · filename│
    └──────┬──────┘ │  · exportPdf│ └───────────┘
           │        └──────┬──────┘
           │               │
           └───────┬───────┘
                   ▼
         loadDocumentText(plainText)     getEditorText()
                   │                         ▲
                   ▼                         │
              editor.js ─────────────────────┘
         (new wrapper only — no renderer changes)
                   │
                   ▼
              selection.js  ·  renderer.js  (UNTOUCHED)

2.3 `loadDocumentText()` Contract (new in `editor.js`)

Single entry point for all imports. Keeps renderer/selection logic isolated.

/**
 * Load plain text into editor — used by document import only.
 * @param {string} text - UTF-8 plain text
 * @param {object} options - { analyze: true, filename: 'doc.txt' }
 */
function loadDocumentText(text, options = {}) {
  const normalized = normalizeImportedText(text); // CRLF → LF, trim BOM
  setEditorHTML(escapeHtml(normalized));
  window.currentSuggestions = [];
  updatePlaceholder();
  updateEditorStats();
  updateSuggestionCounts(0, 0, 0);
  updateWritingScore(0, 0, 0);
  updateSuggestionsList([]);
  hideTooltip();
  if (options.analyze !== false) {
    analyzeTextDelayed();
  }
}

Rules:

Always pass through escapeHtml() before setEditorHTML() (XSS safety)
Never call render() directly from import layer
After load, debounced analyzeText() runs as if user typed

2.4 Paragraph Model

Internal representation remains plain text with \n paragraph breaks:

Source	Paragraph handling
TXT	Preserve `\n` as-is
DOCX import (Mammoth)	`mammoth.extractRawText()` — paragraphs separated by `\n\n`
DOCX export (docx.js)	Split on `\n\n` or `\n` → one `Paragraph` per block
PDF export	Render editor DOM or text block with `dir="rtl"`

2.5 Backend Constraint

/api/analyze enforces MAX_TEXT_LENGTH = 5000 characters (app.py). Imports may load longer documents for editing/export, but analysis will only process what the API accepts. Phase 4 should:

Show a non-blocking warning if imported text exceeds 5000 chars
Still load full text into editor (export works on full content)
Optionally analyze first 5000 chars only — defer to Phase 4.1 if needed

3. Required Libraries

3.1 Recommended Versions (CDN for Vanilla JS)

Library	Version	Purpose	CDN
Mammoth.js	1.8.x	DOCX → text	`https://cdn.jsdelivr.net/npm/mammoth@1.8.0/mammoth.browser.min.js`
docx	8.5.x	Build DOCX	`https://unpkg.com/docx@8.5.0/build/index.umd.js`
file-saver	2.0.5	Trigger download	`https://cdn.jsdelivr.net/npm/file-saver@2.0.5/dist/FileSaver.min.js`
html2pdf.js	0.10.2	PDF via HTML render	`https://cdnjs.cloudflare.com/ajax/libs/html2pdf.js/0.10.2/html2pdf.bundle.min.js`

3.2 Library Evaluation

Mammoth.js (DOCX Import) — Selected

mammoth.extractRawText({ arrayBuffer })
  .then(result => result.value); // plain text, paragraphs as \n\n

Ignores styling, tables, images by design (extractRawText)
Works in browser with FileReader.readAsArrayBuffer
No server required

docx.js (DOCX Export) — Selected

const doc = new docx.Document({
  sections: [{
    properties: { rightToLeft: true },
    children: paragraphs
  }]
});
docx.Packer.toBlob(doc).then(blob => saveAs(blob, 'document.docx'));

Native RTL section support via rightToLeft: true
Paragraph-level Arabic text
Requires mapping \n splits to docx.Paragraph + docx.TextRun

PDF Export — html2pdf.js (recommended over raw jsPDF)

Approach	Arabic support	Searchable text	Effort
jsPDF alone	Poor (no Arabic glyphs)	Yes	High (embed Amiri font)
jsPDF + custom Arabic font	Good	Yes	High (~500KB font)
html2pdf.js (html2canvas + jsPDF)	Good (browser renders Cairo)	No (image-based)	Medium
Browser print dialog	Good	Yes	Low UX

Recommendation: Start with html2pdf.js rendering a cloned off-screen RTL div with editor plain text + Cairo font. Upgrade to jsPDF + Amiri font in Phase 4.1 if searchable PDF is required.

3.3 No New Python Dependencies

All conversion is client-side. Flask continues to serve static files only.

4. File Structure Changes

src/
├── js/
│   ├── renderer.js          (unchanged)
│   ├── selection.js         (unchanged)
│   ├── editor.js            (+ loadDocumentText, export hook)
│   ├── documents/
│   │   ├── documents.js     # init, UI event wiring
│   │   ├── import.js        # importTxt, importDocx
│   │   ├── export.js        # exportTxt, exportDocx, exportPdf
│   │   └── doc-utils.js     # normalizeText, downloadBlob, defaultFilename
│   ├── theme.js
│   └── ui.js
├── css/
│   └── components.css       (+ .doc-dropdown, .doc-menu-item)
└── index.html               (+ script tags, toolbar UI, hidden file input)

4.1 New Files

File	~Lines	Responsibility
`documents/documents.js`	80	`initDocuments()`, wire buttons
`documents/import.js`	120	TXT + DOCX import
`documents/export.js`	180	TXT + DOCX + PDF export
`documents/doc-utils.js`	60	Shared helpers

4.2 Modified Files

File	Change
`editor.js`	Add `loadDocumentText()` only (~25 lines)
`index.html`	Toolbar UI, CDN scripts, `initDocuments()` call
`components.css`	Dropdown + import button styles (~80 lines)

4.3 Untouched Files

renderer.js — no changes
selection.js — no changes
app.py — no changes (optional future: raise MAX_TEXT_LENGTH)

5. Feature Specifications

5.1 TXT Import

function importTxt(file) {
  const reader = new FileReader();
  reader.onload = (e) => loadDocumentText(e.target.result, { filename: file.name });
  reader.readAsText(file, 'UTF-8');
}

Requirement	Implementation
Upload `.txt`	Hidden `<input accept=".txt,text/plain">`
FileReader	`readAsText(file, 'UTF-8')`
Load into editor	`loadDocumentText()`
Preserve architecture	No renderer/selection changes
BOM handling	Strip `\uFEFF` prefix in `normalizeImportedText()`

5.2 TXT Export

function exportTxt() {
  const text = getEditorText();
  const blob = new Blob([text], { type: 'text/plain;charset=utf-8' });
  downloadBlob(blob, defaultFilename('txt'));
}

Requirement	Implementation
UTF-8	`Blob` with `charset=utf-8`
Arabic preserved	UTF-8 encoding (no `escape` on export)
Download	`file-saver` `saveAs()` or `<a download>` + `URL.createObjectURL`

5.3 DOCX Import

async function importDocx(file) {
  const arrayBuffer = await file.arrayBuffer();
  const result = await mammoth.extractRawText({ arrayBuffer });
  if (result.messages.length) console.warn('Mammoth warnings:', result.messages);
  loadDocumentText(result.value, { filename: file.name });
}

Requirement	Implementation
Mammoth.js	`extractRawText` only
Text only	No `convertToHtml`
Ignore tables/images	Default Mammoth raw text behavior
Paragraphs	Normalize `\n\n` → preserve

Accept attribute: .docx,application/vnd.openxmlformats-officedocument.wordprocessingml.document

5.4 DOCX Export

async function exportDocx() {
  const text = getEditorText();
  const paragraphs = text.split(/\n\s*\n/).filter(Boolean).map(block =>
    new docx.Paragraph({
      bidirectional: true,
      children: [new docx.TextRun({ text: block, rightToLeft: true, font: 'Arial' })],
      alignment: docx.AlignmentType.RIGHT
    })
  );
  const doc = new docx.Document({
    sections: [{ properties: { rightToLeft: true }, children: paragraphs }]
  });
  const blob = await docx.Packer.toBlob(doc);
  saveAs(blob, defaultFilename('docx'));
}

Requirement	Implementation
docx.js	`Document`, `Paragraph`, `TextRun`, `Packer`
Paragraphs	Split on double newline (fallback: single `\n`)
RTL	`rightToLeft: true` on section + `bidirectional: true` on paragraphs
Font	Arial or Traditional Arabic (Word-safe)

5.5 PDF Export

async function exportPdf() {
  const text = getEditorText();
  const el = buildPdfExportNode(text); // off-screen div, dir=rtl, Cairo font
  await html2pdf().set({
    margin: 15,
    filename: defaultFilename('pdf'),
    html2canvas: { scale: 2, useCORS: true },
    jsPDF: { unit: 'mm', format: 'a4', orientation: 'portrait' }
  }).from(el).save();
}

Requirement	Implementation
Arabic rendering	Clone styled RTL node with Cairo font
Content source	`getEditorText()` (plain text, not highlighted HTML)
Theme	Export node uses light theme tokens for readability

6. UI Design

6.1 Toolbar Layout (proposed)

┌──────────────────────────────────────────────────────────────────┐
│ [كتابة] [تلخيص]  │  [↑ استيراد]  [↓ تصدير ▾]  │  جاري... │ ١٢٣ كلمة │
└──────────────────────────────────────────────────────────────────┘

Import button (btn-ghost):

Label: استيراد
Icon: upload arrow
Opens file picker (.txt,.docx)
aria-label="استيراد ملف نصي أو Word"

Export dropdown (btn-ghost + menu):

تصدير ▾
  ├── ملف نصي (.txt)
  ├── Word (.docx)
  └── PDF (.pdf)

6.2 Design System Compliance

Element	Classes
Import button	`.btn-ghost`, `.doc-btn`
Export trigger	`.btn-ghost`, `.doc-dropdown__trigger`
Menu panel	`.surface-card`, `.doc-dropdown__menu`
Menu items	`.doc-dropdown__item`
Focus	`:focus-visible` via existing tokens
Theme	`var(--color-surface)`, `var(--color-border)`, `var(--color-text-primary)`

6.3 Mobile

Import + Export move to editor footer alongside مسح الكل / نسخ النص
Export dropdown opens as bottom sheet on <640px (reuse Phase 2 pattern)

6.4 User Feedback

Event	Feedback
Import success	Toast: `تم تحميل الملف`
Import error	Toast: `تعذر قراءة الملف`
Export success	Browser download (implicit)
File too large for analysis	Banner: `النص أطول من ٥٠٠٠ حرف — التحليل على الجزء الأول فقط`
Empty export	Disable export items when editor empty

7. Implementation Roadmap

Phase 4.1 — Foundation (1 day)

Task	Verify
Create `documents/doc-utils.js`	Unit: `normalizeImportedText`, `downloadBlob`
Add `loadDocumentText()` to `editor.js`	Import "مرحبا" → editor shows text, analyze runs
Add CDN scripts to `index.html`	Libraries load without console errors

Phase 4.2 — TXT I/O (0.5 day)

Task	Verify
`importTxt()` + file input	Arabic .txt round-trips correctly
`exportTxt()`	Downloaded file opens in Notepad with correct Arabic

Phase 4.3 — DOCX Import (1 day)

Task	Verify
Integrate Mammoth.js	Sample .docx loads as plain text
Error handling	Corrupt file shows user message
Paragraph breaks	Multi-paragraph doc preserves structure

Phase 4.4 — DOCX Export (1 day)

Task	Verify
Integrate docx.js	File opens in Word
RTL verification	Arabic aligns right in Word
Paragraph mapping	Blank lines → new paragraphs

Phase 4.5 — PDF Export (1–2 days)

Task	Verify
html2pdf.js integration	PDF downloads
Arabic visual check	-glyphs render correctly
Multi-page	Long text paginates

Phase 4.6 — UI + Polish (1 day)

Task	Verify
Toolbar import/export buttons	Matches design system both themes
Mobile layout	Footer placement works
Empty state / loading states	Disabled when appropriate
Toast notifications	Success/error feedback

Phase 4.7 — QA (0.5 day)

Task	Verify
Full testing checklist	All pass
`node test_renderer.js`	Still passes (no regression)
Manual Phase 1 tests	Duplicate highlights, cursor, XSS

Total estimate: 6–7 working days

8. Risk Analysis

#	Risk	Severity	Likelihood	Mitigation
R1	Arabic PDF garbled	High	Medium	Use html2pdf with Cairo font on RTL clone; fallback Phase 4.1: jsPDF + Amiri
R2	DOCX export RTL broken in Word	Medium	Medium	Test `rightToLeft` section + `bidirectional` paragraphs; use Word-safe font
R3	Mammoth loses complex DOCX structure	Low	High	Expected — document "text only" scope; show warning if messages
R4	Import XSS	High	Low	Always `escapeHtml()` before `setEditorHTML()`
R5	Import breaks cursor/offsets	Medium	Low	Use `loadDocumentText()` → `setEditorHTML` + fresh analyze (same as clear+type)
R6	File > 5000 chars breaks analyze	Medium	High	Warn user; full text still editable/exportable
R7	CDN library unavailable offline	Medium	Medium	Vendor copies in `src/js/vendor/` for graduation demo
R8	docx.js UMD global name mismatch	Low	Medium	Verify `window.docx` after script load in dev
R9	Large file memory	Low	Low	Client-side limit ~2MB; reject with message
R10	Phase 1 regression	High	Low	Do not touch `renderer.js` / `selection.js`; run existing tests

8.1 Critical Path

TXT import/export (low risk, validates architecture)
    → DOCX import (Mammoth)
    → DOCX export (RTL)
    → PDF export (Arabic rendering)  ← highest risk, schedule last

9. Testing Checklist

9.1 TXT

Import UTF-8 Arabic .txt — text appears correctly in editor
Import UTF-8 with BOM — BOM stripped, text correct
Import Windows-1256 file — detect mojibake or reject gracefully
Export .txt — re-import round-trip preserves content
Export empty editor — button disabled or shows error
Import triggers analyzeTextDelayed() — highlights appear

9.2 DOCX Import

Import simple Arabic .docx — text loads
Multi-paragraph document — paragraph breaks preserved
DOCX with images/tables — text extracted, no crash
Invalid/corrupt file — error toast, editor unchanged
.doc (legacy) — rejected by accept filter

9.3 DOCX Export

Export opens in Microsoft Word without error
Export opens in LibreOffice Writer
Arabic text is right-aligned
Paragraph breaks match editor
Empty lines preserved as empty paragraphs
Re-import exported DOCX — content matches (round-trip via Mammoth)

9.4 PDF Export

PDF downloads with correct filename
Arabic glyphs render correctly (visual inspection)
Multi-page document paginates
Long Arabic text (2000+ chars) exports without truncation
Empty editor — export disabled

9.5 UI / Theme

Import button visible in dark theme
Import button visible in light theme
Export dropdown readable in both themes
Keyboard: Tab reaches import/export controls
Mobile: buttons accessible in footer
aria-label on import and export controls

9.6 Phase 1 Regression (must pass)

node test_renderer.js — all tests pass
Type in editor after import — cursor preserved on analyze
Duplicate word highlights (ذهبو ×3) — independent spans
Click highlight → popover → apply correction
<script> in imported text — escaped, not executed
Sidebar suggestions update after import + analyze

9.7 Edge Cases

Import 10,000 char file — loads; warning shown for analyze limit
Import while analysis in progress — AbortController cancels prior request
Switch theme during export — export still works
Arabic filename in download — document.txt default used if problematic

10. Approval Gate

No implementation until this plan is reviewed.

Please confirm:

PDF approach: html2pdf.js (visual, easier Arabic) vs jsPDF + embedded Amiri font (searchable, harder)?
Analyze limit: Warn only, or truncate analysis to first 5000 chars automatically?
CDN vs vendored libs: CDN for dev, copy to src/js/vendor/ for offline demo?
Default export filename: بيان-مستند.txt or bayan-document.txt?
Proceed with implementation after approval?

Document	Role
`PHASE_1_COMPLETE_VERIFICATION.md`	Editor engine constraints
`PHASE_2_STATUS.md`	UI / theme context
`THEME_SYSTEM.md`	Design tokens for new UI