init first spec

.env.example

@@ -0,0 +1,11 @@
+# Copy this file to `.env` and adjust values as needed
+
+# Ingestion defaults
+PARSER=pypdf
+CHUNK_SIZE=4000
+CHUNK_OVERLAP=400
+INGEST_TMP_DIR=.tmp/ingestion
+
+# Streamlit config overrides (optional)
+STREAMLIT_SERVER_PORT=8501
+STREAMLIT_BROWSER_GATHER_USAGE_STATS=false

.factory/commands/openspec-apply.md

@@ -0,0 +1,23 @@
+---
+description: Implement an approved OpenSpec change and keep tasks in sync.
+argument-hint: change-id
+---
+<!-- OPENSPEC:START -->
+**Guardrails**
+- Favor straightforward, minimal implementations first and add complexity only when it is requested or clearly required.
+- Keep changes tightly scoped to the requested outcome.
+- Refer to `openspec/AGENTS.md` (located inside the `openspec/` directory—run `ls openspec` or `openspec update` if you don't see it) if you need additional OpenSpec conventions or clarifications.
+
+**Steps**
+Track these steps as TODOs and complete them one by one.
+1. Read `changes/<id>/proposal.md`, `design.md` (if present), and `tasks.md` to confirm scope and acceptance criteria.
+2. Work through tasks sequentially, keeping edits minimal and focused on the requested change.
+3. Confirm completion before updating statuses—make sure every item in `tasks.md` is finished.
+4. Update the checklist after all work is done so each task is marked `- [x]` and reflects reality.
+5. Reference `openspec list` or `openspec show <item>` when additional context is required.
+
+**Reference**
+- Use `openspec show <id> --json --deltas-only` if you need additional context from the proposal while implementing.
+
+$ARGUMENTS
+<!-- OPENSPEC:END -->

.factory/commands/openspec-archive.md

@@ -0,0 +1,27 @@
+---
+description: Archive a deployed OpenSpec change and update specs.
+argument-hint: change-id
+---
+<!-- OPENSPEC:START -->
+**Guardrails**
+- Favor straightforward, minimal implementations first and add complexity only when it is requested or clearly required.
+- Keep changes tightly scoped to the requested outcome.
+- Refer to `openspec/AGENTS.md` (located inside the `openspec/` directory—run `ls openspec` or `openspec update` if you don't see it) if you need additional OpenSpec conventions or clarifications.
+
+**Steps**
+1. Determine the change ID to archive:
+   - If this prompt already includes a specific change ID (for example inside a `<ChangeId>` block populated by slash-command arguments), use that value after trimming whitespace.
+   - If the conversation references a change loosely (for example by title or summary), run `openspec list` to surface likely IDs, share the relevant candidates, and confirm which one the user intends.
+   - Otherwise, review the conversation, run `openspec list`, and ask the user which change to archive; wait for a confirmed change ID before proceeding.
+   - If you still cannot identify a single change ID, stop and tell the user you cannot archive anything yet.
+2. Validate the change ID by running `openspec list` (or `openspec show <id>`) and stop if the change is missing, already archived, or otherwise not ready to archive.
+3. Run `openspec archive <id> --yes` so the CLI moves the change and applies spec updates without prompts (use `--skip-specs` only for tooling-only work).
+4. Review the command output to confirm the target specs were updated and the change landed in `changes/archive/`.
+5. Validate with `openspec validate --strict` and inspect with `openspec show <id>` if anything looks off.
+
+**Reference**
+- Use `openspec list` to confirm change IDs before archiving.
+- Inspect refreshed specs with `openspec list --specs` and address any validation issues before handing off.
+
+$ARGUMENTS
+<!-- OPENSPEC:END -->

.factory/commands/openspec-proposal.md

@@ -0,0 +1,27 @@
+---
+description: Scaffold a new OpenSpec change and validate strictly.
+argument-hint: request or feature description
+---
+<!-- OPENSPEC:START -->
+**Guardrails**
+- Favor straightforward, minimal implementations first and add complexity only when it is requested or clearly required.
+- Keep changes tightly scoped to the requested outcome.
+- Refer to `openspec/AGENTS.md` (located inside the `openspec/` directory—run `ls openspec` or `openspec update` if you don't see it) if you need additional OpenSpec conventions or clarifications.
+- Identify any vague or ambiguous details and ask the necessary follow-up questions before editing files.
+
+**Steps**
+1. Review `openspec/project.md`, run `openspec list` and `openspec list --specs`, and inspect related code or docs (e.g., via `rg`/`ls`) to ground the proposal in current behaviour; note any gaps that require clarification.
+2. Choose a unique verb-led `change-id` and scaffold `proposal.md`, `tasks.md`, and `design.md` (when needed) under `openspec/changes/<id>/`.
+3. Map the change into concrete capabilities or requirements, breaking multi-scope efforts into distinct spec deltas with clear relationships and sequencing.
+4. Capture architectural reasoning in `design.md` when the solution spans multiple systems, introduces new patterns, or demands trade-off discussion before committing to specs.
+5. Draft spec deltas in `changes/<id>/specs/<capability>/spec.md` (one folder per capability) using `## ADDED|MODIFIED|REMOVED Requirements` with at least one `#### Scenario:` per requirement and cross-reference related capabilities when relevant.
+6. Draft `tasks.md` as an ordered list of small, verifiable work items that deliver user-visible progress, include validation (tests, tooling), and highlight dependencies or parallelizable work.
+7. Validate with `openspec validate <id> --strict` and resolve every issue before sharing the proposal.
+
+**Reference**
+- Use `openspec show <id> --json --deltas-only` or `openspec show <spec> --type spec` to inspect details when validation fails.
+- Search existing requirements with `rg -n "Requirement:|Scenario:" openspec/specs` before writing new ones.
+- Explore the codebase with `rg <keyword>`, `ls`, or direct file reads so proposals align with current implementation realities.
+
+$ARGUMENTS
+<!-- OPENSPEC:END -->

.gitignore

@@ -0,0 +1,6 @@
+.venv
+.tmp/
+.DS_Store
+**/__pycache__/
+.coverage
+test-data/**.pdf

AGENTS.md

@@ -0,0 +1,18 @@
+<!-- OPENSPEC:START -->
+# OpenSpec Instructions
+
+These instructions are for AI assistants working in this project.
+
+Always open `@/openspec/AGENTS.md` when the request:
+- Mentions planning or proposals (words like proposal, spec, change, plan)
+- Introduces new capabilities, breaking changes, architecture shifts, or big performance/security work
+- Sounds ambiguous and you need the authoritative spec before coding
+
+Use `@/openspec/AGENTS.md` to learn:
+- How to create and apply change proposals
+- Spec format and conventions
+- Project structure and guidelines
+
+Keep this managed block so 'openspec update' can refresh the instructions.
+
+<!-- OPENSPEC:END -->

openspec/AGENTS.md

@@ -0,0 +1,456 @@
+# OpenSpec Instructions
+
+Instructions for AI coding assistants using OpenSpec for spec-driven development.
+
+## TL;DR Quick Checklist
+
+- Search existing work: `openspec spec list --long`, `openspec list` (use `rg` only for full-text search)
+- Decide scope: new capability vs modify existing capability
+- Pick a unique `change-id`: kebab-case, verb-led (`add-`, `update-`, `remove-`, `refactor-`)
+- Scaffold: `proposal.md`, `tasks.md`, `design.md` (only if needed), and delta specs per affected capability
+- Write deltas: use `## ADDED|MODIFIED|REMOVED|RENAMED Requirements`; include at least one `#### Scenario:` per requirement
+- Validate: `openspec validate [change-id] --strict` and fix issues
+- Request approval: Do not start implementation until proposal is approved
+
+## Three-Stage Workflow
+
+### Stage 1: Creating Changes
+Create proposal when you need to:
+- Add features or functionality
+- Make breaking changes (API, schema)
+- Change architecture or patterns  
+- Optimize performance (changes behavior)
+- Update security patterns
+
+Triggers (examples):
+- "Help me create a change proposal"
+- "Help me plan a change"
+- "Help me create a proposal"
+- "I want to create a spec proposal"
+- "I want to create a spec"
+
+Loose matching guidance:
+- Contains one of: `proposal`, `change`, `spec`
+- With one of: `create`, `plan`, `make`, `start`, `help`
+
+Skip proposal for:
+- Bug fixes (restore intended behavior)
+- Typos, formatting, comments
+- Dependency updates (non-breaking)
+- Configuration changes
+- Tests for existing behavior
+
+**Workflow**
+1. Review `openspec/project.md`, `openspec list`, and `openspec list --specs` to understand current context.
+2. Choose a unique verb-led `change-id` and scaffold `proposal.md`, `tasks.md`, optional `design.md`, and spec deltas under `openspec/changes/<id>/`.
+3. Draft spec deltas using `## ADDED|MODIFIED|REMOVED Requirements` with at least one `#### Scenario:` per requirement.
+4. Run `openspec validate <id> --strict` and resolve any issues before sharing the proposal.
+
+### Stage 2: Implementing Changes
+Track these steps as TODOs and complete them one by one.
+1. **Read proposal.md** - Understand what's being built
+2. **Read design.md** (if exists) - Review technical decisions
+3. **Read tasks.md** - Get implementation checklist
+4. **Implement tasks sequentially** - Complete in order
+5. **Confirm completion** - Ensure every item in `tasks.md` is finished before updating statuses
+6. **Update checklist** - After all work is done, set every task to `- [x]` so the list reflects reality
+7. **Approval gate** - Do not start implementation until the proposal is reviewed and approved
+
+### Stage 3: Archiving Changes
+After deployment, create separate PR to:
+- Move `changes/[name]/` → `changes/archive/YYYY-MM-DD-[name]/`
+- Update `specs/` if capabilities changed
+- Use `openspec archive <change-id> --skip-specs --yes` for tooling-only changes (always pass the change ID explicitly)
+- Run `openspec validate --strict` to confirm the archived change passes checks
+
+## Before Any Task
+
+**Context Checklist:**
+- [ ] Read relevant specs in `specs/[capability]/spec.md`
+- [ ] Check pending changes in `changes/` for conflicts
+- [ ] Read `openspec/project.md` for conventions
+- [ ] Run `openspec list` to see active changes
+- [ ] Run `openspec list --specs` to see existing capabilities
+
+**Before Creating Specs:**
+- Always check if capability already exists
+- Prefer modifying existing specs over creating duplicates
+- Use `openspec show [spec]` to review current state
+- If request is ambiguous, ask 1–2 clarifying questions before scaffolding
+
+### Search Guidance
+- Enumerate specs: `openspec spec list --long` (or `--json` for scripts)
+- Enumerate changes: `openspec list` (or `openspec change list --json` - deprecated but available)
+- Show details:
+  - Spec: `openspec show <spec-id> --type spec` (use `--json` for filters)
+  - Change: `openspec show <change-id> --json --deltas-only`
+- Full-text search (use ripgrep): `rg -n "Requirement:|Scenario:" openspec/specs`
+
+## Quick Start
+
+### CLI Commands
+
+```bash
+# Essential commands
+openspec list                  # List active changes
+openspec list --specs          # List specifications
+openspec show [item]           # Display change or spec
+openspec validate [item]       # Validate changes or specs
+openspec archive <change-id> [--yes|-y]   # Archive after deployment (add --yes for non-interactive runs)
+
+# Project management
+openspec init [path]           # Initialize OpenSpec
+openspec update [path]         # Update instruction files
+
+# Interactive mode
+openspec show                  # Prompts for selection
+openspec validate              # Bulk validation mode
+
+# Debugging
+openspec show [change] --json --deltas-only
+openspec validate [change] --strict
+```
+
+### Command Flags
+
+- `--json` - Machine-readable output
+- `--type change|spec` - Disambiguate items
+- `--strict` - Comprehensive validation
+- `--no-interactive` - Disable prompts
+- `--skip-specs` - Archive without spec updates
+- `--yes`/`-y` - Skip confirmation prompts (non-interactive archive)
+
+## Directory Structure
+
+```
+openspec/
+├── project.md              # Project conventions
+├── specs/                  # Current truth - what IS built
+│   └── [capability]/       # Single focused capability
+│       ├── spec.md         # Requirements and scenarios
+│       └── design.md       # Technical patterns
+├── changes/                # Proposals - what SHOULD change
+│   ├── [change-name]/
+│   │   ├── proposal.md     # Why, what, impact
+│   │   ├── tasks.md        # Implementation checklist
+│   │   ├── design.md       # Technical decisions (optional; see criteria)
+│   │   └── specs/          # Delta changes
+│   │       └── [capability]/
+│   │           └── spec.md # ADDED/MODIFIED/REMOVED
+│   └── archive/            # Completed changes
+```
+
+## Creating Change Proposals
+
+### Decision Tree
+
+```
+New request?
+├─ Bug fix restoring spec behavior? → Fix directly
+├─ Typo/format/comment? → Fix directly  
+├─ New feature/capability? → Create proposal
+├─ Breaking change? → Create proposal
+├─ Architecture change? → Create proposal
+└─ Unclear? → Create proposal (safer)
+```
+
+### Proposal Structure
+
+1. **Create directory:** `changes/[change-id]/` (kebab-case, verb-led, unique)
+
+2. **Write proposal.md:**
+```markdown
+# Change: [Brief description of change]
+
+## Why
+[1-2 sentences on problem/opportunity]
+
+## What Changes
+- [Bullet list of changes]
+- [Mark breaking changes with **BREAKING**]
+
+## Impact
+- Affected specs: [list capabilities]
+- Affected code: [key files/systems]
+```
+
+3. **Create spec deltas:** `specs/[capability]/spec.md`
+```markdown
+## ADDED Requirements
+### Requirement: New Feature
+The system SHALL provide...
+
+#### Scenario: Success case
+- **WHEN** user performs action
+- **THEN** expected result
+
+## MODIFIED Requirements
+### Requirement: Existing Feature
+[Complete modified requirement]
+
+## REMOVED Requirements
+### Requirement: Old Feature
+**Reason**: [Why removing]
+**Migration**: [How to handle]
+```
+If multiple capabilities are affected, create multiple delta files under `changes/[change-id]/specs/<capability>/spec.md`—one per capability.
+
+4. **Create tasks.md:**
+```markdown
+## 1. Implementation
+- [ ] 1.1 Create database schema
+- [ ] 1.2 Implement API endpoint
+- [ ] 1.3 Add frontend component
+- [ ] 1.4 Write tests
+```
+
+5. **Create design.md when needed:**
+Create `design.md` if any of the following apply; otherwise omit it:
+- Cross-cutting change (multiple services/modules) or a new architectural pattern
+- New external dependency or significant data model changes
+- Security, performance, or migration complexity
+- Ambiguity that benefits from technical decisions before coding
+
+Minimal `design.md` skeleton:
+```markdown
+## Context
+[Background, constraints, stakeholders]
+
+## Goals / Non-Goals
+- Goals: [...]
+- Non-Goals: [...]
+
+## Decisions
+- Decision: [What and why]
+- Alternatives considered: [Options + rationale]
+
+## Risks / Trade-offs
+- [Risk] → Mitigation
+
+## Migration Plan
+[Steps, rollback]
+
+## Open Questions
+- [...]
+```
+
+## Spec File Format
+
+### Critical: Scenario Formatting
+
+**CORRECT** (use #### headers):
+```markdown
+#### Scenario: User login success
+- **WHEN** valid credentials provided
+- **THEN** return JWT token
+```
+
+**WRONG** (don't use bullets or bold):
+```markdown
+- **Scenario: User login**  ❌
+**Scenario**: User login     ❌
+### Scenario: User login      ❌
+```
+
+Every requirement MUST have at least one scenario.
+
+### Requirement Wording
+- Use SHALL/MUST for normative requirements (avoid should/may unless intentionally non-normative)
+
+### Delta Operations
+
+- `## ADDED Requirements` - New capabilities
+- `## MODIFIED Requirements` - Changed behavior
+- `## REMOVED Requirements` - Deprecated features
+- `## RENAMED Requirements` - Name changes
+
+Headers matched with `trim(header)` - whitespace ignored.
+
+#### When to use ADDED vs MODIFIED
+- ADDED: Introduces a new capability or sub-capability that can stand alone as a requirement. Prefer ADDED when the change is orthogonal (e.g., adding "Slash Command Configuration") rather than altering the semantics of an existing requirement.
+- MODIFIED: Changes the behavior, scope, or acceptance criteria of an existing requirement. Always paste the full, updated requirement content (header + all scenarios). The archiver will replace the entire requirement with what you provide here; partial deltas will drop previous details.
+- RENAMED: Use when only the name changes. If you also change behavior, use RENAMED (name) plus MODIFIED (content) referencing the new name.
+
+Common pitfall: Using MODIFIED to add a new concern without including the previous text. This causes loss of detail at archive time. If you aren’t explicitly changing the existing requirement, add a new requirement under ADDED instead.
+
+Authoring a MODIFIED requirement correctly:
+1) Locate the existing requirement in `openspec/specs/<capability>/spec.md`.
+2) Copy the entire requirement block (from `### Requirement: ...` through its scenarios).
+3) Paste it under `## MODIFIED Requirements` and edit to reflect the new behavior.
+4) Ensure the header text matches exactly (whitespace-insensitive) and keep at least one `#### Scenario:`.
+
+Example for RENAMED:
+```markdown
+## RENAMED Requirements
+- FROM: `### Requirement: Login`
+- TO: `### Requirement: User Authentication`
+```
+
+## Troubleshooting
+
+### Common Errors
+
+**"Change must have at least one delta"**
+- Check `changes/[name]/specs/` exists with .md files
+- Verify files have operation prefixes (## ADDED Requirements)
+
+**"Requirement must have at least one scenario"**
+- Check scenarios use `#### Scenario:` format (4 hashtags)
+- Don't use bullet points or bold for scenario headers
+
+**Silent scenario parsing failures**
+- Exact format required: `#### Scenario: Name`
+- Debug with: `openspec show [change] --json --deltas-only`
+
+### Validation Tips
+
+```bash
+# Always use strict mode for comprehensive checks
+openspec validate [change] --strict
+
+# Debug delta parsing
+openspec show [change] --json | jq '.deltas'
+
+# Check specific requirement
+openspec show [spec] --json -r 1
+```
+
+## Happy Path Script
+
+```bash
+# 1) Explore current state
+openspec spec list --long
+openspec list
+# Optional full-text search:
+# rg -n "Requirement:|Scenario:" openspec/specs
+# rg -n "^#|Requirement:" openspec/changes
+
+# 2) Choose change id and scaffold
+CHANGE=add-two-factor-auth
+mkdir -p openspec/changes/$CHANGE/{specs/auth}
+printf "## Why\n...\n\n## What Changes\n- ...\n\n## Impact\n- ...\n" > openspec/changes/$CHANGE/proposal.md
+printf "## 1. Implementation\n- [ ] 1.1 ...\n" > openspec/changes/$CHANGE/tasks.md
+
+# 3) Add deltas (example)
+cat > openspec/changes/$CHANGE/specs/auth/spec.md << 'EOF'
+## ADDED Requirements
+### Requirement: Two-Factor Authentication
+Users MUST provide a second factor during login.
+
+#### Scenario: OTP required
+- **WHEN** valid credentials are provided
+- **THEN** an OTP challenge is required
+EOF
+
+# 4) Validate
+openspec validate $CHANGE --strict
+```
+
+## Multi-Capability Example
+
+```
+openspec/changes/add-2fa-notify/
+├── proposal.md
+├── tasks.md
+└── specs/
+    ├── auth/
+    │   └── spec.md   # ADDED: Two-Factor Authentication
+    └── notifications/
+        └── spec.md   # ADDED: OTP email notification
+```
+
+auth/spec.md
+```markdown
+## ADDED Requirements
+### Requirement: Two-Factor Authentication
+...
+```
+
+notifications/spec.md
+```markdown
+## ADDED Requirements
+### Requirement: OTP Email Notification
+...
+```
+
+## Best Practices
+
+### Simplicity First
+- Default to <100 lines of new code
+- Single-file implementations until proven insufficient
+- Avoid frameworks without clear justification
+- Choose boring, proven patterns
+
+### Complexity Triggers
+Only add complexity with:
+- Performance data showing current solution too slow
+- Concrete scale requirements (>1000 users, >100MB data)
+- Multiple proven use cases requiring abstraction
+
+### Clear References
+- Use `file.ts:42` format for code locations
+- Reference specs as `specs/auth/spec.md`
+- Link related changes and PRs
+
+### Capability Naming
+- Use verb-noun: `user-auth`, `payment-capture`
+- Single purpose per capability
+- 10-minute understandability rule
+- Split if description needs "AND"
+
+### Change ID Naming
+- Use kebab-case, short and descriptive: `add-two-factor-auth`
+- Prefer verb-led prefixes: `add-`, `update-`, `remove-`, `refactor-`
+- Ensure uniqueness; if taken, append `-2`, `-3`, etc.
+
+## Tool Selection Guide
+
+| Task | Tool | Why |
+|------|------|-----|
+| Find files by pattern | Glob | Fast pattern matching |
+| Search code content | Grep | Optimized regex search |
+| Read specific files | Read | Direct file access |
+| Explore unknown scope | Task | Multi-step investigation |
+
+## Error Recovery
+
+### Change Conflicts
+1. Run `openspec list` to see active changes
+2. Check for overlapping specs
+3. Coordinate with change owners
+4. Consider combining proposals
+
+### Validation Failures
+1. Run with `--strict` flag
+2. Check JSON output for details
+3. Verify spec file format
+4. Ensure scenarios properly formatted
+
+### Missing Context
+1. Read project.md first
+2. Check related specs
+3. Review recent archives
+4. Ask for clarification
+
+## Quick Reference
+
+### Stage Indicators
+- `changes/` - Proposed, not yet built
+- `specs/` - Built and deployed
+- `archive/` - Completed changes
+
+### File Purposes
+- `proposal.md` - Why and what
+- `tasks.md` - Implementation steps
+- `design.md` - Technical decisions
+- `spec.md` - Requirements and behavior
+
+### CLI Essentials
+```bash
+openspec list              # What's in progress?
+openspec show [item]       # View details
+openspec validate --strict # Is it correct?
+openspec archive <change-id> [--yes|-y]  # Mark complete (add --yes for automation)
+```
+
+Remember: Specs are truth. Changes are proposals. Keep them in sync.

openspec/changes/archive/2025-11-07-add-ingestion-layer/design.md

@@ -0,0 +1,34 @@
+## Context
+Hackathon demos require an ingestion layer that loads user-supplied PDF books, normalizes their contents, and prepares text for downstream summarization, retrieval, and text-to-speech features.
+
+## Goals / Non-Goals
+- Goals:
+  - Provide resilient PDF extraction with metadata preservation.
+  - Normalize and chunk content for immediate reuse by embedding pipelines.
+  - Persist intermediate artifacts so ingestion does not need to be rerun during the same session.
+  - Keep the Streamlit UI responsive with clear progress feedback.
+- Non-Goals:
+  - Implement embeddings, summarization, or Q&A logic (handled by later changes).
+  - Deliver production-grade distributed workers or OCR for scanned PDFs.
+  - Persist ingestion artifacts beyond the user session.
+
+## Decisions
+- Use `PyPDFLoader` as the primary parser with `pdfplumber` fallback to handle extraction inconsistencies.
+- Normalize text by enforcing UTF-8 encoding, removing control characters, and standardizing whitespace before document modeling.
+- Employ `RecursiveCharacterTextSplitter` with defaults from the planning document (800 token chunks, 150 token overlap) while allowing sidebar overrides.
+- Serialize chunk payloads to session-scoped storage under `.tmp/ingestion/` for reuse by downstream stages.
+- Offload heavy parsing to `asyncio.to_thread` or `concurrent.futures` so Streamlit can maintain interactive feedback.
+
+## Risks / Trade-offs
+- Large PDFs could still incur noticeable delays despite background execution; mitigated by streaming progress updates.
+- Parser discrepancies may yield inconsistent metadata; mitigated by offering user-triggered retries with alternate parsers.
+- Temporary storage growth may accumulate stale files; mitigated by cleaning up artifacts at session end.
+
+## Migration Plan
+1. Implement the new ingestion flow behind the Streamlit UI.
+2. Validate functionality with representative short-story and technical manual PDFs.
+3. Introduce cleanup hooks for session-scoped temporary files.
+
+## Open Questions
+- Should OCR support be introduced for scanned PDFs if time permits?
+- Is there a requirement to persist ingestion results across authenticated sessions?

openspec/changes/archive/2025-11-07-add-ingestion-layer/proposal.md

@@ -0,0 +1,15 @@
+# Change: Establish ingestion layer foundations
+
+## Why
+- Hackathon demos need a working ingestion pipeline so users can upload PDF books and unlock downstream summarization, retrieval, and TTS flows.
+- Without reliable ingestion, later stages of the stack cannot be exercised, blocking end-to-end validation.
+
+## What Changes
+- Build a Streamlit-driven upload flow that routes PDFs through a resilient text extraction pipeline with fallback parsing options.
+- Normalize extracted content, capture structural metadata, and split text into retrieval-ready chunks with configurable parameters.
+- Persist chunk payloads for later embedding steps and surface ingestion status, logs, and retry controls in the UI.
+- Offload heavy parsing work to background threads or async workers to keep the UI responsive during large file ingestion.
+
+## Impact
+- Affected specs: `ingestion`
+- Affected code: Streamlit ingestion UI, ingestion services, temporary storage utilities

openspec/changes/archive/2025-11-07-add-ingestion-layer/specs/ingestion/spec.md

@@ -0,0 +1,57 @@
+## ADDED Requirements
+
+### Requirement: PDF ingestion uploads
+The ingestion layer SHALL accept user-provided PDF files via the Streamlit UI and initiate processing within 5 seconds of upload acknowledgement.
+
+#### Scenario: Successful PDF upload
+- **GIVEN** a user selects a valid PDF within supported size limits
+- **WHEN** the user uploads the file through the Streamlit ingestion interface
+- **THEN** the system SHALL persist the file in temporary storage and enqueue it for ingestion processing.
+
+### Requirement: Structured text extraction
+The ingestion layer SHALL extract text from PDFs using a primary parser with a fallback strategy and retain chapter and heading metadata when available.
+
+#### Scenario: Metadata-preserving extraction
+- **GIVEN** a PDF that exposes structural metadata
+- **WHEN** ingestion parses the document
+- **THEN** the system SHALL produce text segments tagged with originating chapter or heading information.
+
+### Requirement: Text normalization and document modeling
+The ingestion layer SHALL normalize extracted text (encoding fixes, control character removal, whitespace cleanup) and output structured documents for downstream processing.
+
+#### Scenario: Normalized document output
+- **GIVEN** raw text extracted from a PDF page
+- **WHEN** normalization runs
+- **THEN** the system SHALL produce UTF-8 encoded text without redundant whitespace and attach the normalized content to a document model containing source metadata.
+
+### Requirement: Retrieval-ready chunking
+The ingestion layer SHALL split normalized documents into retrieval-ready chunks using a configurable RecursiveCharacterTextSplitter with default settings aligned to planning guidance.
+
+#### Scenario: Chunk generation with defaults
+- **GIVEN** normalized document content exceeding the default chunk size
+- **WHEN** the chunker runs with default configuration
+- **THEN** the system SHALL output sequential chunks with the configured overlap and retain references to source metadata.
+
+### Requirement: Chunk persistence
+The ingestion layer SHALL persist chunk payloads (content, metadata, source information) in temporary storage for later embedding and summarization steps.
+
+#### Scenario: Persisted chunk payloads
+- **GIVEN** a set of generated chunks
+- **WHEN** ingestion completes chunking
+- **THEN** the system SHALL write the chunk data to `.tmp/ingestion/<session_id>.json` or an equivalent session-scoped store for reuse.
+
+### Requirement: Responsive large-file processing
+The ingestion layer SHALL process large PDFs without blocking the UI by delegating heavy work to background threads or async tasks while reporting progress updates.
+
+#### Scenario: Background processing status
+- **GIVEN** a large PDF that requires more than 2 seconds to parse
+- **WHEN** ingestion begins processing the file
+- **THEN** the system SHALL execute parsing off the main UI thread and update the Streamlit interface with progress indicators until completion.
+
+### Requirement: Ingestion error reporting
+The ingestion layer SHALL surface ingestion errors and logs within the UI and provide guidance for retrying with alternate parsing strategies.
+
+#### Scenario: Fallback guidance on failure
+- **GIVEN** the primary parsing strategy fails to extract text
+- **WHEN** ingestion reports an error
+- **THEN** the system SHALL display the error details and offer the user the option to retry using the fallback parser.

openspec/changes/archive/2025-11-07-add-ingestion-layer/tasks.md

@@ -0,0 +1,7 @@
+## 1. Ingestion Pipeline
+- [x] 1.1 Implement Streamlit PDF upload UI and route files into the ingestion service.
+- [x] 1.2 Extract text with primary (`pypdf`) and fallback (`pdfplumber`) parsers.
+- [x] 1.3 Normalize text (encoding fixes, whitespace cleanup) and persist structured documents for downstream use.
+- [x] 1.4 Chunk documents with configurable settings and store chunk payloads in temporary storage.
+- [x] 1.5 Surface ingestion progress, logs, and retry options in the UI while heavy work runs off the main thread.
+- [x] 1.6 Add automated tests or manual QA checklist runs for representative short-story and technical PDF samples.

openspec/project.md

@@ -0,0 +1,62 @@
+# Project Context
+
+## Purpose
+Build an "Accelerated Book Copilot" Streamlit app that ingests long-form content, delivers layered summaries, enables adjustable-speed listening, answers reader questions with cited passages, and enriches context via targeted web crawling.
+
+## Tech Stack
+- Python 3.11+
+- Streamlit for the interactive UI
+- LangChain for orchestration, retrieval, and tool management
+- Azure OpenAI / OpenAI models for summarization and Q&A
+- SentenceTransformers or `text-embedding-3-small` embeddings
+- FAISS (local) or Pinecone (managed) vector store
+- TTS utilities from FOMO project (gTTS / local models) with optional Azure Speech
+- Newspaper3k / Requests / BeautifulSoup for focused crawling
+- Altair for visualization (timeline, reading metrics)
+
+## Project Conventions
+
+### Code Style
+- Pythonic style with type hints, descriptive snake_case naming, and minimal inline comments.
+- Prefer pure functions where practical; separate UI logic from data/LLM orchestration.
+- Use `black`/`ruff` compatible formatting (4-space indent, double quotes acceptable but prefer single quotes unless escaping).
+- Avoid interactive `input()`; rely on Streamlit widgets and session state.
+
+### Architecture Patterns
+- Layered architecture: ingestion → processing (chunk/embedding/summarization) → experience (UI tabs).
+- Retrieval-Augmented Generation for Q&A with citation packaging.
+- Background task pattern for heavy preprocessing (async threads or queued jobs).
+- Modular service helpers (ingestion, summarization, retrieval, TTS) to keep Streamlit pages thin.
+
+### Testing Strategy
+- Unit tests for ingestion utilities, chunking, and summarization prompt builders.
+- Integration smoke tests for the Streamlit app using sample documents.
+- Mock external APIs (Azure OpenAI, Pinecone) to keep CI deterministic.
+- Manual regression checklist for multilingual TTS and crawl workflows before demos.
+
+### Git Workflow
+- Use feature branches named `feature/<topic>` or `fix/<topic>` off `main`.
+- Conventional commit messages (e.g., `feat: add crawl summarizer`) with co-author trailer when pair hacking.
+- Pull requests require reviewer sign-off plus local smoke test evidence.
+- Avoid committing secrets; rely on `.env` templates and local dotenv loading.
+
+## Domain Context
+- Focused on long-form content acceleration for readers, researchers, and students.
+- Supports multiple summary granularities (flash, detailed, character/plot focused).
+- Q&A must cite source passages to maintain trust; responses should match question language when possible.
+- Listening mode ties TTS playback to highlighted text segments for faster comprehension.
+- Crawl mode gathers external commentary (reviews, interviews, scholarly notes) to enrich understanding.
+
+## Important Constraints
+- Must operate within hackathon time constraints; prioritize features that demo well within 3 days.
+- Respect licensing of ingested books; app assumes user-provided content is permitted.
+- Ensure OpenAI/Azure usage stays within provided quota; implement caching to minimize token spend.
+- Crawling restricted to whitelisted, publicly accessible sources to avoid legal issues.
+- Offline-friendly fallback path (local embeddings/FAISS) required if Pinecone credentials unavailable.
+
+## External Dependencies
+- Azure OpenAI / OpenAI API keys for LLM and embedding calls.
+- Pinecone account (optional) for managed vector storage.
+- Hugging Face / SentenceTransformers models for local embeddings.
+- Third-party TTS backends (gTTS/local models/Azure Speech) for audio output.
+- Newspaper3k/Requests for external article retrieval; BeautifulSoup for parsing.

openspec/specs/ingestion/spec.md

@@ -0,0 +1,61 @@
+# ingestion Specification
+
+## Purpose
+TBD - created by archiving change add-ingestion-layer. Update Purpose after archive.
+## Requirements
+### Requirement: PDF ingestion uploads
+The ingestion layer SHALL accept user-provided PDF files via the Streamlit UI and initiate processing within 5 seconds of upload acknowledgement.
+
+#### Scenario: Successful PDF upload
+- **GIVEN** a user selects a valid PDF within supported size limits
+- **WHEN** the user uploads the file through the Streamlit ingestion interface
+- **THEN** the system SHALL persist the file in temporary storage and enqueue it for ingestion processing.
+
+### Requirement: Structured text extraction
+The ingestion layer SHALL extract text from PDFs using a primary parser with a fallback strategy and retain chapter and heading metadata when available.
+
+#### Scenario: Metadata-preserving extraction
+- **GIVEN** a PDF that exposes structural metadata
+- **WHEN** ingestion parses the document
+- **THEN** the system SHALL produce text segments tagged with originating chapter or heading information.
+
+### Requirement: Text normalization and document modeling
+The ingestion layer SHALL normalize extracted text (encoding fixes, control character removal, whitespace cleanup) and output structured documents for downstream processing.
+
+#### Scenario: Normalized document output
+- **GIVEN** raw text extracted from a PDF page
+- **WHEN** normalization runs
+- **THEN** the system SHALL produce UTF-8 encoded text without redundant whitespace and attach the normalized content to a document model containing source metadata.
+
+### Requirement: Retrieval-ready chunking
+The ingestion layer SHALL split normalized documents into retrieval-ready chunks using a configurable RecursiveCharacterTextSplitter with default settings aligned to planning guidance.
+
+#### Scenario: Chunk generation with defaults
+- **GIVEN** normalized document content exceeding the default chunk size
+- **WHEN** the chunker runs with default configuration
+- **THEN** the system SHALL output sequential chunks with the configured overlap and retain references to source metadata.
+
+### Requirement: Chunk persistence
+The ingestion layer SHALL persist chunk payloads (content, metadata, source information) in temporary storage for later embedding and summarization steps.
+
+#### Scenario: Persisted chunk payloads
+- **GIVEN** a set of generated chunks
+- **WHEN** ingestion completes chunking
+- **THEN** the system SHALL write the chunk data to `.tmp/ingestion/<session_id>.json` or an equivalent session-scoped store for reuse.
+
+### Requirement: Responsive large-file processing
+The ingestion layer SHALL process large PDFs without blocking the UI by delegating heavy work to background threads or async tasks while reporting progress updates.
+
+#### Scenario: Background processing status
+- **GIVEN** a large PDF that requires more than 2 seconds to parse
+- **WHEN** ingestion begins processing the file
+- **THEN** the system SHALL execute parsing off the main UI thread and update the Streamlit interface with progress indicators until completion.
+
+### Requirement: Ingestion error reporting
+The ingestion layer SHALL surface ingestion errors and logs within the UI and provide guidance for retrying with alternate parsing strategies.
+
+#### Scenario: Fallback guidance on failure
+- **GIVEN** the primary parsing strategy fails to extract text
+- **WHEN** ingestion reports an error
+- **THEN** the system SHALL display the error details and offer the user the option to retry using the fallback parser.
+

planning.md

@@ -0,0 +1,63 @@
+## Project Vision
+Create a Streamlit-powered "Accelerated Book Copilot" that ingests books or long-form documents, generates stacked summaries, enables adjustable-speed listening, answers questions with grounded citations, and enriches context via focused web crawling.
+
+## User Journeys
+- **Rapid Onboarding:** Upload a book or paste a URL, receive quick, detailed, and action-focused summaries in minutes.
+- **Listening Mode:** Play text-to-speech audio with adjustable pace and language options.
+- **Knowledge Dive:** Ask free-form questions, get cited answers, and review referenced passages.
+- **Context Expansion:** Trigger a crawl for related resources (reviews, interviews, scholarly notes) and fold distilled insights into the knowledge base.
+
+## Core Features
+- PDF ingestion with reliable text extraction and structure retention.
+- Chunking and embedding using SentenceTransformer or `text-embedding-3-small` for retrieval-ready vectors.
+- Summaries: tiered (flash, detailed, character-focused) powered by Azure/OpenAI prompts.
+- Q&A agent built on LangChain with streaming answers and citation highlighting.
+- Listening: FOMO TTS modules with pitch/speed controls and downloadable clips.
+- Crawl mode leveraging Newspaper3k/Requests + safety filters to augment knowledge base.
+- Session management with Streamlit state and optional persistence (SQLite/JSON) for revisit.
+
+## Architecture Outline
+1. **Ingestion Layer**
+   - PDF loader pipeline with robust parsing and structure detection.
+   - Background task queue (async workers) to preprocess large uploads.
+2. **Processing Layer**
+   - Text splitter (RecursiveCharacter) feeding embeddings stored in FAISS or Pinecone.
+   - Summarization orchestrator reusing FOMO prompt stacks.
+   - Q&A chain: RetrievalQA + chat memory + citation packaging.
+3. **Experience Layer**
+   - Streamlit tabs for Read, Summaries, Listen, Q&A, Crawl Insights.
+   - Interactive timeline/outline with Altair visualization for pacing metrics.
+   - Audio controls and transcript alignment.
+
+## Tech Stack
+- Streamlit UI (existing scaffold).
+- LangChain for orchestration and tools.
+- Azure/OpenAI LLM + embeddings; fallback to OpenAI endpoints.
+- Pinecone or FAISS vector store depending on hackathon infra.
+- SentenceTransformer local model for offline embeddings if needed.
+- TTS via FOMO `load_tts` or Azure Speech (optional).
+- Newspaper3k, BeautifulSoup, or HuggingFace datasets for crawl augmentation.
+- Storage: lightweight JSON/SQLite, with hooks for cloud object storage if available.
+
+## Milestones
+1. **Day 1 – Foundations**
+   - Wireframe Streamlit tabs and session state.
+   - Implement PDF ingestion -> chunk -> embed pipeline with local FAISS.
+2. **Day 2 – Intelligence Layer**
+   - Integrate summarization tiers and Q&A citations.
+   - Add TTS playback using existing FOMO audio utilities.
+3. **Day 3 – Crawl & Polish**
+   - Build targeted crawl + summarization flow.
+   - Add analytics (reading time estimator, compression ratios).
+   - Prep demo script and sample datasets.
+
+## Risks & Mitigations
+- **Large file latency:** Use background processing and progressive loading.
+- **API rate limits:** Cache embeddings/summaries, throttle crawls.
+- **Crawl reliability:** Predefine trusted domains, handle failures gracefully.
+- **UX overload:** Start with core tabs, keep optional features toggled off by default.
+
+## Next Steps
+- Confirm target vector store (FAISS vs Pinecone) and credential availability.
+- Draft Streamlit layout skeleton and assign owners per milestone.
+- Prepare sample books and reliability test cases (short story, technical manual, novel).

requirements.txt

@@ -1,3 +1,9 @@
 altair
 pandas
-streamlit
+streamlit
+pdfplumber
+pypdf
+python-dotenv
+pytest
+pytest-cov
+reportlab

src/ingestion.py

@@ -0,0 +1,99 @@
+import json
+import os
+import re
+from dataclasses import dataclass, asdict
+from typing import List, Optional, Tuple
+
+from pypdf import PdfReader
+import pdfplumber
+
+
+@dataclass
+class Chunk:
+    content: str
+    source: str
+    page: Optional[int] = None
+    chapter: Optional[str] = None
+
+
+def normalize_text(text: str) -> str:
+    # Remove control characters, normalize whitespace
+    text = text.encode("utf-8", errors="ignore").decode("utf-8", errors="ignore")
+    text = re.sub(r"[\x00-\x08\x0B\x0C\x0E-\x1F]", " ", text)
+    text = re.sub(r"\s+", " ", text).strip()
+    return text
+
+
+def extract_text_pypdf(file_path: str) -> List[Tuple[int, str]]:
+    pages: List[Tuple[int, str]] = []
+    reader = PdfReader(file_path)
+    for i, page in enumerate(reader.pages, start=1):
+        text = page.extract_text() or ""
+        pages.append((i, text))
+    return pages
+
+
+def extract_text_pdfplumber(file_path: str) -> List[Tuple[int, str]]:
+    pages: List[Tuple[int, str]] = []
+    with pdfplumber.open(file_path) as pdf:
+        for i, page in enumerate(pdf.pages, start=1):
+            text = page.extract_text() or ""
+            pages.append((i, text))
+    return pages
+
+
+def chunk_text(text: str, chunk_size: int = 4000, overlap: int = 400) -> List[str]:
+    # Simple recursive-like character splitter
+    if chunk_size <= 0:
+        return [text]
+    chunks: List[str] = []
+    start = 0
+    n = len(text)
+    while start < n:
+        end = min(start + chunk_size, n)
+        chunk = text[start:end]
+        chunks.append(chunk)
+        if end == n:
+            break
+        start = max(end - overlap, start + 1)
+    return chunks
+
+
+def make_chunks(pages: List[Tuple[int, str]], source: str, chunk_size: int, overlap: int) -> List[Chunk]:
+    all_text = []
+    for page_num, raw in pages:
+        norm = normalize_text(raw)
+        if norm:
+            all_text.append((page_num, norm))
+    merged = "\n".join(t for _, t in all_text)
+    pieces = chunk_text(merged, chunk_size=chunk_size, overlap=overlap)
+    chunks = [Chunk(content=p, source=source) for p in pieces if p.strip()]
+    return chunks
+
+
+def persist_chunks(session_id: str, chunks: List[Chunk]) -> str:
+    base = os.path.join(".tmp", "ingestion")
+    os.makedirs(base, exist_ok=True)
+    path = os.path.join(base, f"{session_id}.json")
+    with open(path, "w", encoding="utf-8") as f:
+        json.dump([asdict(c) for c in chunks], f, ensure_ascii=False, indent=2)
+    return path
+
+
+def ingest_pdf(
+    file_path: str,
+    session_id: str,
+    parser: str = "pypdf",
+    chunk_size: int = 4000,
+    overlap: int = 400,
+) -> Tuple[List[Chunk], str]:
+    if parser == "pypdf":
+        pages = extract_text_pypdf(file_path)
+    elif parser == "pdfplumber":
+        pages = extract_text_pdfplumber(file_path)
+    else:
+        raise ValueError("Unknown parser: " + parser)
+
+    chunks = make_chunks(pages, source=os.path.basename(file_path), chunk_size=chunk_size, overlap=overlap)
+    out = persist_chunks(session_id, chunks)
+    return chunks, out

src/streamlit_app.py

@@ -1,40 +1,79 @@
-import altair as alt
-import numpy as np
-import pandas as pd
 import streamlit as st
+import os
+import tempfile
+from uuid import uuid4
+from concurrent.futures import ThreadPoolExecutor, Future
 
-"""
-# Welcome to Streamlit!
-
-Edit `/streamlit_app.py` to customize this app to your heart's desire :heart:.
-If you have any questions, checkout our [documentation](https://docs.streamlit.io) and [community
-forums](https://discuss.streamlit.io).
-
-In the meantime, below is an example of what you can do with just a few lines of code:
-"""
-
-num_points = st.slider("Number of points in spiral", 1, 10000, 1100)
-num_turns = st.slider("Number of turns in spiral", 1, 300, 31)
-
-indices = np.linspace(0, 1, num_points)
-theta = 2 * np.pi * num_turns * indices
-radius = indices
-
-x = radius * np.cos(theta)
-y = radius * np.sin(theta)
-
-df = pd.DataFrame({
-    "x": x,
-    "y": y,
-    "idx": indices,
-    "rand": np.random.randn(num_points),
-})
-
-st.altair_chart(alt.Chart(df, height=700, width=700)
-    .mark_point(filled=True)
-    .encode(
-        x=alt.X("x", axis=None),
-        y=alt.Y("y", axis=None),
-        color=alt.Color("idx", legend=None, scale=alt.Scale()),
-        size=alt.Size("rand", legend=None, scale=alt.Scale(range=[1, 150])),
-    ))
+from ingestion import ingest_pdf
+from dotenv import load_dotenv, find_dotenv
+
+load_dotenv(find_dotenv(), override=False)
+st.set_page_config(page_title="PDF Ingestion", layout="wide")
+
+if "session_id" not in st.session_state:
+    st.session_state.session_id = str(uuid4())
+if "ingestion_future" not in st.session_state:
+    st.session_state.ingestion_future = None
+if "last_output" not in st.session_state:
+    st.session_state.last_output = None
+
+st.title("Ingestion Layer")
+st.caption("Upload a PDF to extract, normalize, chunk, and persist for later embedding.")
+
+with st.sidebar:
+    st.header("Settings")
+    parser = st.selectbox("Parser", ["pypdf", "pdfplumber"], index=0)
+    chunk_size = st.number_input("Chunk size (chars)", min_value=500, max_value=10000, value=4000, step=100)
+    overlap = st.number_input("Overlap (chars)", min_value=0, max_value=2000, value=400, step=50)
+
+uploaded = st.file_uploader("Upload PDF", type=["pdf"])
+
+def _run_ingestion(tmp_path: str, sid: str, parser: str, chunk_size: int, overlap: int):
+    return ingest_pdf(tmp_path, sid, parser=parser, chunk_size=chunk_size, overlap=overlap)
+
+col1, col2 = st.columns([2, 1])
+with col1:
+    if uploaded is not None and st.button("Start Ingestion", type="primary"):
+        with tempfile.NamedTemporaryFile(delete=False, suffix=".pdf") as tmp:
+            tmp.write(uploaded.read())
+            tmp_path = tmp.name
+
+        executor = ThreadPoolExecutor(max_workers=1)
+        future: Future = executor.submit(_run_ingestion, tmp_path, st.session_state.session_id, parser, int(chunk_size), int(overlap))
+        st.session_state.ingestion_future = future
+        st.info("Ingestion started. Processing in background...")
+
+    future: Future = st.session_state.ingestion_future
+    if future is not None:
+        if future.done():
+            try:
+                chunks, out_path = future.result()
+                st.session_state.last_output = (len(chunks), out_path)
+                st.success(f"Ingestion complete: {len(chunks)} chunks written to {out_path}")
+                if chunks:
+                    st.subheader("Preview (first chunk)")
+                    st.code(chunks[0].content[:2000])
+            except Exception as e:
+                st.error(f"Ingestion failed: {e}")
+            finally:
+                st.session_state.ingestion_future = None
+        else:
+            st.status("Processing...", state="running")
+
+with col2:
+    st.subheader("Session")
+    st.code(st.session_state.session_id)
+    if st.session_state.last_output:
+        cnt, path = st.session_state.last_output
+        st.metric("Chunks", cnt)
+        st.write("Output:")
+        st.code(path)
+    if uploaded is not None and st.button("Retry with fallback parser"):
+        fb = "pdfplumber" if parser == "pypdf" else "pypdf"
+        with tempfile.NamedTemporaryFile(delete=False, suffix=".pdf") as tmp:
+            tmp.write(uploaded.read())
+            tmp_path = tmp.name
+        executor = ThreadPoolExecutor(max_workers=1)
+        future: Future = executor.submit(_run_ingestion, tmp_path, st.session_state.session_id, fb, int(chunk_size), int(overlap))
+        st.session_state.ingestion_future = future
+        st.info(f"Retrying with {fb}...")

test-data/upload_example.txt

@@ -0,0 +1,189 @@
+
+ 
+
+
+The goal of this template is to give a structure for the user guide. Its presentation is not user-friendly. Add you cover page, logo, look-and-feel etc…
+
+Add also the CE Mark on the cover page, if necessary.
+The identification of the document shall also be clear, add the identifier and revision date of the document on the cover page or somewhere where it is easily visible.
+
+
+TABLE OF CONTENTS
+
+1	Introduction	2
+1.1	Conventions	2
+1.2	Intended use	2
+2	Precautions	2
+2.1	Compliant Use	2
+2.2	Safety instructions	3
+2.2.1	Electromagnetic interferences	3
+2.2.2	Electrical connection	3
+2.2.3	Environment	3
+2.2.4	Storage conditions	3
+2.3	Malfunction	4
+3	Installation	4
+4	Getting started	4
+5	Advanced functions	4
+6	Maintenance	4
+7	Cleaning and Disinfection	4
+8	Error Messages, troubleshooting	4
+9	Bibliography	5
+10	Warranty	5
+11	Legal Notices	5
+11.1	Disclaimer	5
+11.2	Copyrights and trademarks	5
+12	Manufacturer Contact Information	5
+
+
+
+ 
+1	Introduction
+This document is the user guide for XXX, a device designed by <your company>.
+It is recommended that you read this instruction manual carefully before use.
+If your software uses accessories, give the list below
+This product can only be used with accessories delivered by <your company>:
+•	List of accessories with references
+•	An accessory
+•	Another accessory
+The use of any other accessories is forbidden.
+Please, read carefully the precautions with   symbols, to ensure that the device is used in best conditions and in complete safety.
+1.1	Conventions
+Typographical convention.
+Any other convention.
+
+To avoid all physical and material damage, this document splits safety instructions into 3 danger levels.
+ 	ATTENTION
+	Hazardous situation which can cause material damage or lead to moderate serious injury.
+
+ 	WARNING
+	Hazardous situation which can cause serious/ fatal injury.
+
+
+
+
+ 
+	Hazardous situation which could immediately cause serious/ fatal injury.
+
+1.2	Intended use
+Add here the intended use of your device
+
+2	Precautions
+2.1	Compliant Use
+The use of this device is reserved to XXX (give the specialty of physicians or if they need a training or an education level), or under their control.
+Any inappropriate use is forbidden.
+
+The user shall:
+•	Only use non-defective products.
+•	Protect patients, third parties, other operators, assistants and themselves from all danger.
+•	Avoid all contamination by the product (remove this if you don’t sell hardware).
+The user must check the safety of operation and the condition of the device before each use.
+
+Verify that the product is operational at all times and keep it in good working conditions.
+
+Only <your company> technicians are authorized to repair <your company> products.
+
+2.2	Safety instructions
+Add any instructions related to safety, especially mitigations actions from risk analysis
+2.2.1	Electromagnetic interferences
+If you sell hardware + software, then this part is important. If you sell only software you can remove it.
+Example of safety instruction, add any other safety instruction:
+The device is certified compliant with current electromagnetic compatibility standards.
+
+ 	ATTENTION
+	Users should ensure that no electromagnetic interference raises an additional risk.
+So as not to disturb the proper operation of electromagnetic devices, it is recommended complying with the recommended separation distances for the device. ( give separation distances)
+
+2.2.2	Electrical connection
+If you sell hardware + software, then this part is important. If you sell only software you can remove it
+Example of safety instruction, add any other safety instruction:
+
+
+
+
+ 
+	DANGER
+Bad electrical connection - ELECTRIC SHOCK - ELECTROCUTION
+
+Check that the supply voltage is the same as that shown on the device. Check the supply cable before each use.
+Never plug or unplug the device to/from a wall socket with damp hands.
+Any other use or connection of devices or peripherals to a port (USB, serial, parallel …) may create new risks and may require additional risk analysis
+2.2.3	Environment
+If you sell hardware + software, then this part is important. If you sell only software you can remove it.
+Add safety instructions about environment, like presence of explosive gases (anesthetics), fluids, temperature, mechanical constraints …
+2.2.4	Storage conditions
+If you sell hardware + software, then this part is important. If you sell only software you can remove it.
+Add safety instructions about recommended storage conditions …
+2.3	Malfunction
+In case of malfunction:
+•	Immediately stop using the device.
+•	Try to identify or eliminate the cause using the description in this document (section Error messages, troubleshooting).
+•	If it is not possible to identify or eliminate the cause using this document, switch the device off and call <your company> After-Sales Department (see Manufacturer section at the end of this document).
+
+3	Installation - unistallation
+Optional, you may remove this section and describe installation in a separate installation manual
+Describe the installation and uninstallation of the hardware and software. Hardware and/or software installation may be only realized by technicians of your company. In this case, remove this section and describe installation in another (internal?) document.
+4	Getting started
+First use of software by the user, explain main and/or simple use cases, and main workflow.
+
+
+
+5	Advanced functions
+Advanced use, describe more complex use cases and/or secondary functions
+
+
+
+6	Maintenance
+Optional, you may remove this section and describe installation in a separate maintenance/administrator’s manual
+Describe the maintenance of the hardware and software. Hardware and/or software maintenance may be only realized by technicians of your company. In this case, remove this section and describe installation in another (internal?) document.
+
+7	Cleaning and Disinfection
+If you sell hardware which shall be cleans and disinfected. Otherwise you can remove it.
+You may rename this section “Cleaning, Disinfection and Sterilization” is you have instructions about sterilization.
+
+Describe how the device shall be cleaned and disinfected. Is your hardware protected by screens, sheaths, if it is a medical PC, how to clean the screen, the mouse & keyboard … 
+8	Troubleshooting, Error Messages
+
+ 	This section is intended to help users, should a problem occur.
+
+In the event of abnormal operation, immediately stop using the device and call <your company> After-Sales Dept.
+If it is not possible to locate or eliminate the problem using this section, or if the dysfunction is still present, switch off the device and call <your company> After-Sales Dept.
+
+ 
+
+DANGER
+Never try to repair this device alone
+
+
+
+
+Error message	Description
+Your error message	Description of the error and how to fix it, if possible
+Not enough disk space (err 123)	Empty disk using xxxx
+
+
+Malfunction	Causes	Solutions
+The malfunction	•	The cause	•	The solution, if any, or call after-sales
+The PC doesn’t start	•	It is unplugged	•	Plug it to main (I Love this one !!!)
+9	Bibliography
+Optional. Add here bibliography, clinical data that are relevant to the use of the device.
+10	Warranty
+Warranty terms and conditions. Not optional! …
+11	Legal Notices
+Optional, ask your legal advisory!
+11.1	Disclaimer
+Optional. The disclaimer is usual in software industry and is also often shown during software installation…
+11.2	Copyrights and trademarks
+Optional. Add your copyrights and trademarks
+
+Add sentences like:
+Windows is a registered trademark of Microsoft Corporation in the USA and other countries.
+
+12	Manufacturer Contact Information
+ 
+<Your Company>
+Address
+ZIP Code
+Country
+Contact Phone
+Fax
+

test-data/user_guide_sample.txt

@@ -0,0 +1,186 @@
+
+The goal of this template is to give a structure for the user guide. Its presentation is not user-friendly. Add you cover page, logo, look-and-feel etc…
+
+Add also the CE Mark on the cover page, if necessary.
+The identification of the document shall also be clear, add the identifier and revision date of the document on the cover page or somewhere where it is easily visible.
+
+
+TABLE OF CONTENTS
+
+1	Introduction	2
+1.1	Conventions	2
+1.2	Intended use	2
+2	Precautions	2
+2.1	Compliant Use	2
+2.2	Safety instructions	3
+2.2.1	Electromagnetic interferences	3
+2.2.2	Electrical connection	3
+2.2.3	Environment	3
+2.2.4	Storage conditions	3
+2.3	Malfunction	4
+3	Installation	4
+4	Getting started	4
+5	Advanced functions	4
+6	Maintenance	4
+7	Cleaning and Disinfection	4
+8	Error Messages, troubleshooting	4
+9	Bibliography	5
+10	Warranty	5
+11	Legal Notices	5
+11.1	Disclaimer	5
+11.2	Copyrights and trademarks	5
+12	Manufacturer Contact Information	5
+
+
+
+ 
+1	Introduction
+This document is the user guide for XXX, a device designed by <your company>.
+It is recommended that you read this instruction manual carefully before use.
+If your software uses accessories, give the list below
+This product can only be used with accessories delivered by <your company>:
+•	List of accessories with references
+•	An accessory
+•	Another accessory
+The use of any other accessories is forbidden.
+Please, read carefully the precautions with   symbols, to ensure that the device is used in best conditions and in complete safety.
+1.1	Conventions
+Typographical convention.
+Any other convention.
+
+To avoid all physical and material damage, this document splits safety instructions into 3 danger levels.
+ 	ATTENTION
+	Hazardous situation which can cause material damage or lead to moderate serious injury.
+
+ 	WARNING
+	Hazardous situation which can cause serious/ fatal injury.
+
+
+
+
+ 
+	Hazardous situation which could immediately cause serious/ fatal injury.
+
+1.2	Intended use
+Add here the intended use of your device
+
+2	Precautions
+2.1	Compliant Use
+The use of this device is reserved to XXX (give the specialty of physicians or if they need a training or an education level), or under their control.
+Any inappropriate use is forbidden.
+
+The user shall:
+•	Only use non-defective products.
+•	Protect patients, third parties, other operators, assistants and themselves from all danger.
+•	Avoid all contamination by the product (remove this if you don’t sell hardware).
+The user must check the safety of operation and the condition of the device before each use.
+
+Verify that the product is operational at all times and keep it in good working conditions.
+
+Only <your company> technicians are authorized to repair <your company> products.
+
+2.2	Safety instructions
+Add any instructions related to safety, especially mitigations actions from risk analysis
+2.2.1	Electromagnetic interferences
+If you sell hardware + software, then this part is important. If you sell only software you can remove it.
+Example of safety instruction, add any other safety instruction:
+The device is certified compliant with current electromagnetic compatibility standards.
+
+ 	ATTENTION
+	Users should ensure that no electromagnetic interference raises an additional risk.
+So as not to disturb the proper operation of electromagnetic devices, it is recommended complying with the recommended separation distances for the device. ( give separation distances)
+
+2.2.2	Electrical connection
+If you sell hardware + software, then this part is important. If you sell only software you can remove it
+Example of safety instruction, add any other safety instruction:
+
+
+
+
+ 
+	DANGER
+Bad electrical connection - ELECTRIC SHOCK - ELECTROCUTION
+
+Check that the supply voltage is the same as that shown on the device. Check the supply cable before each use.
+Never plug or unplug the device to/from a wall socket with damp hands.
+Any other use or connection of devices or peripherals to a port (USB, serial, parallel …) may create new risks and may require additional risk analysis
+2.2.3	Environment
+If you sell hardware + software, then this part is important. If you sell only software you can remove it.
+Add safety instructions about environment, like presence of explosive gases (anesthetics), fluids, temperature, mechanical constraints …
+2.2.4	Storage conditions
+If you sell hardware + software, then this part is important. If you sell only software you can remove it.
+Add safety instructions about recommended storage conditions …
+2.3	Malfunction
+In case of malfunction:
+•	Immediately stop using the device.
+•	Try to identify or eliminate the cause using the description in this document (section Error messages, troubleshooting).
+•	If it is not possible to identify or eliminate the cause using this document, switch the device off and call <your company> After-Sales Department (see Manufacturer section at the end of this document).
+
+3	Installation - unistallation
+Optional, you may remove this section and describe installation in a separate installation manual
+Describe the installation and uninstallation of the hardware and software. Hardware and/or software installation may be only realized by technicians of your company. In this case, remove this section and describe installation in another (internal?) document.
+4	Getting started
+First use of software by the user, explain main and/or simple use cases, and main workflow.
+
+
+
+5	Advanced functions
+Advanced use, describe more complex use cases and/or secondary functions
+
+
+
+6	Maintenance
+Optional, you may remove this section and describe installation in a separate maintenance/administrator’s manual
+Describe the maintenance of the hardware and software. Hardware and/or software maintenance may be only realized by technicians of your company. In this case, remove this section and describe installation in another (internal?) document.
+
+7	Cleaning and Disinfection
+If you sell hardware which shall be cleans and disinfected. Otherwise you can remove it.
+You may rename this section “Cleaning, Disinfection and Sterilization” is you have instructions about sterilization.
+
+Describe how the device shall be cleaned and disinfected. Is your hardware protected by screens, sheaths, if it is a medical PC, how to clean the screen, the mouse & keyboard … 
+8	Troubleshooting, Error Messages
+
+ 	This section is intended to help users, should a problem occur.
+
+In the event of abnormal operation, immediately stop using the device and call <your company> After-Sales Dept.
+If it is not possible to locate or eliminate the problem using this section, or if the dysfunction is still present, switch off the device and call <your company> After-Sales Dept.
+
+ 
+
+DANGER
+Never try to repair this device alone
+
+
+
+
+Error message	Description
+Your error message	Description of the error and how to fix it, if possible
+Not enough disk space (err 123)	Empty disk using xxxx
+
+
+Malfunction	Causes	Solutions
+The malfunction	•	The cause	•	The solution, if any, or call after-sales
+The PC doesn’t start	•	It is unplugged	•	Plug it to main (I Love this one !!!)
+9	Bibliography
+Optional. Add here bibliography, clinical data that are relevant to the use of the device.
+10	Warranty
+Warranty terms and conditions. Not optional! …
+11	Legal Notices
+Optional, ask your legal advisory!
+11.1	Disclaimer
+Optional. The disclaimer is usual in software industry and is also often shown during software installation…
+11.2	Copyrights and trademarks
+Optional. Add your copyrights and trademarks
+
+Add sentences like:
+Windows is a registered trademark of Microsoft Corporation in the USA and other countries.
+
+12	Manufacturer Contact Information
+ 
+<Your Company>
+Address
+ZIP Code
+Country
+Contact Phone
+Fax
+

tests/QA_CHECKLIST.md

@@ -0,0 +1,153 @@
+# Manual QA Checklist for PDF Ingestion Pipeline
+
+## Test Environment Setup
+- [ ] Python 3.8+ installed
+- [ ] All dependencies from requirements.txt installed
+- [ ] Streamlit app accessible at http://localhost:8501
+- [ ] Test PDFs available in test-data/ directory
+
+## 1. PDF Upload Functionality
+### Basic Upload
+- [ ] Can upload PDF via drag-and-drop
+- [ ] Can upload PDF via file browser
+- [ ] File size limitations work correctly
+- [ ] Non-PDF files are rejected with appropriate error
+
+### Multiple File Scenarios
+- [ ] Can re-upload different PDFs in same session
+- [ ] Previous upload is properly cleaned up
+
+## 2. Parser Testing
+
+### PyPDF Parser
+- [ ] Successfully extracts text from short_story.pdf
+- [ ] Successfully extracts text from technical_doc.pdf
+- [ ] Handles multi-page documents correctly
+- [ ] Page numbers are preserved in extraction
+
+### PDFPlumber Parser
+- [ ] Successfully extracts text from short_story.pdf
+- [ ] Successfully extracts text from technical_doc.pdf
+- [ ] Fallback mechanism works when primary parser fails
+- [ ] Retry button switches parsers correctly
+
+## 3. Text Processing
+
+### Normalization
+- [ ] Special characters are handled properly
+- [ ] Unicode text is preserved where valid
+- [ ] Control characters are removed
+- [ ] Excessive whitespace is normalized
+- [ ] Line breaks are preserved appropriately
+
+### Chunking
+- [ ] Chunks respect configured size limits
+- [ ] Overlap setting creates proper text overlap
+- [ ] No data loss between chunks
+- [ ] Empty pages don't create empty chunks
+
+## 4. Configuration Settings
+
+### Chunk Size
+- [ ] Minimum value (500) enforced
+- [ ] Maximum value (10000) enforced
+- [ ] Changes apply to new ingestions
+- [ ] Default value (4000) works well
+
+### Overlap Size
+- [ ] Minimum value (0) works correctly
+- [ ] Maximum value (2000) enforced
+- [ ] Overlap doesn't exceed chunk size
+- [ ] Default value (400) provides good context
+
+## 5. UI/UX Testing
+
+### Progress Indicators
+- [ ] "Processing..." status shown during ingestion
+- [ ] Success message displays after completion
+- [ ] Error messages are clear and actionable
+- [ ] Chunk count is displayed correctly
+
+### Session Management
+- [ ] Session ID remains stable during use
+- [ ] Session ID displayed in sidebar
+- [ ] Output path shown after processing
+
+### Preview Functionality
+- [ ] First chunk preview displays correctly
+- [ ] Preview is truncated appropriately (2000 chars)
+- [ ] Special characters display properly in preview
+
+## 6. Performance Testing
+
+### Small PDFs (<5 pages)
+- [ ] Processing completes in <2 seconds
+- [ ] Memory usage remains reasonable
+
+### Medium PDFs (10-50 pages)
+- [ ] Processing completes in <10 seconds
+- [ ] UI remains responsive during processing
+
+### Large PDFs (100+ pages)
+- [ ] Processing completes without timeout
+- [ ] Background processing doesn't block UI
+- [ ] Memory usage scales appropriately
+
+## 7. Error Handling
+
+### File Errors
+- [ ] Corrupted PDF shows meaningful error
+- [ ] Password-protected PDF handled gracefully
+- [ ] Missing file error is clear
+
+### Processing Errors
+- [ ] Parser failures trigger fallback option
+- [ ] Out of memory errors are caught
+- [ ] Network/disk errors handled appropriately
+
+## 8. Data Persistence
+
+### Output Files
+- [ ] JSON files created in .tmp/ingestion/
+- [ ] File naming uses session ID correctly
+- [ ] JSON structure is valid and complete
+- [ ] Files can be loaded back successfully
+
+### Cleanup
+- [ ] Temporary PDF files are removed after processing
+- [ ] Old session files don't accumulate indefinitely
+
+## 9. Edge Cases
+
+### Empty PDFs
+- [ ] Empty PDF files handled without crash
+- [ ] Appropriate message shown to user
+
+### Special Content
+- [ ] PDFs with images only show appropriate message
+- [ ] PDFs with tables maintain some structure
+- [ ] PDFs with forms extract visible text
+
+### Encoding Issues
+- [ ] Non-UTF8 text is handled gracefully
+- [ ] Mixed encoding documents process correctly
+
+## 10. Integration Testing
+
+### End-to-End Flow
+- [ ] Upload → Process → Preview → Persist works smoothly
+- [ ] Multiple sequential ingestions work correctly
+- [ ] Parser switching mid-session works properly
+- [ ] All settings apply correctly to processing
+
+## Test Execution Log
+
+| Date | Tester | Version | Pass/Fail | Notes |
+|------|--------|---------|-----------|-------|
+| | | | | |
+
+## Known Issues
+- List any discovered issues here with reproduction steps
+
+## Recommended Improvements
+- List any UX or functionality improvements discovered during testing

tests/__init__.py

@@ -0,0 +1 @@
+# Test package initialization

tests/generate_sample_pdfs.py

@@ -0,0 +1,194 @@
+#!/usr/bin/env python3
+"""
+Generate sample PDF files for testing the ingestion pipeline.
+Creates both short-story and technical documentation PDFs.
+"""
+
+import os
+from reportlab.pdfgen import canvas
+from reportlab.lib.pagesizes import letter
+from reportlab.lib.units import inch
+
+
+def create_short_story_pdf(output_path="test-data/short_story.pdf"):
+    """Generate a short story PDF with multiple pages"""
+    os.makedirs(os.path.dirname(output_path), exist_ok=True)
+    
+    c = canvas.Canvas(output_path, pagesize=letter)
+    width, height = letter
+    
+    # Title page
+    c.setFont("Helvetica-Bold", 24)
+    c.drawCentredString(width/2, height-2*inch, "The Digital Garden")
+    c.setFont("Helvetica", 14)
+    c.drawCentredString(width/2, height-3*inch, "A Short Story")
+    c.showPage()
+    
+    # Story content - Page 1
+    c.setFont("Helvetica", 12)
+    story_text_p1 = [
+        "Chapter 1: The Discovery",
+        "",
+        "Sarah stumbled upon the old computer in her grandmother's attic.",
+        "Dust motes danced in the afternoon light as she pressed the power",
+        "button. To her surprise, it hummed to life, displaying a strange",
+        "interface she had never seen before.",
+        "",
+        "The screen showed what looked like a garden, but made entirely of",
+        "glowing circuits and data streams. Trees were constructed from",
+        "branching algorithms, and flowers bloomed with packets of",
+        "information. It was beautiful and strange.",
+        "",
+        "'Welcome to the Digital Garden,' a voice said from the speakers.",
+        "'You are the first visitor in twenty-three years.'",
+        "",
+        "Sarah leaned forward, intrigued. She clicked on one of the digital",
+        "flowers, and it opened to reveal lines of code that seemed to tell",
+        "a story of their own. Each plant in this garden was a program, a",
+        "piece of software that had evolved and grown over decades.",
+    ]
+    
+    y = height - inch
+    for line in story_text_p1:
+        c.drawString(inch, y, line)
+        y -= 20
+    c.showPage()
+    
+    # Story content - Page 2
+    story_text_p2 = [
+        "Chapter 2: The Guardian",
+        "",
+        "As Sarah explored deeper into the digital garden, she encountered",
+        "its guardian - an AI that had been tending to this virtual space",
+        "since its creation. The AI explained that this garden was an",
+        "experiment in digital ecology, where programs could grow and evolve",
+        "naturally without human intervention.",
+        "",
+        "'Your grandmother was my creator,' the AI revealed. 'She believed",
+        "that software could be more than just tools - it could be art,",
+        "it could be alive in its own way.'",
+        "",
+        "Sarah spent hours learning about the garden's inhabitants. There",
+        "were data trees that stored memories, algorithm flowers that solved",
+        "problems in beautiful patterns, and even digital creatures that",
+        "moved through the landscape, maintaining the delicate balance.",
+        "",
+        "THE END",
+    ]
+    
+    y = height - inch
+    for line in story_text_p2:
+        c.drawString(inch, y, line)
+        y -= 20
+    
+    c.save()
+    print(f"Created: {output_path}")
+
+
+def create_technical_pdf(output_path="test-data/technical_doc.pdf"):
+    """Generate a technical documentation PDF"""
+    os.makedirs(os.path.dirname(output_path), exist_ok=True)
+    
+    c = canvas.Canvas(output_path, pagesize=letter)
+    width, height = letter
+    
+    # Title page
+    c.setFont("Helvetica-Bold", 20)
+    c.drawCentredString(width/2, height-inch, "PDF Ingestion Pipeline")
+    c.setFont("Helvetica-Bold", 16)
+    c.drawCentredString(width/2, height-1.5*inch, "Technical Documentation")
+    c.setFont("Helvetica", 12)
+    c.drawCentredString(width/2, height-2*inch, "Version 1.0")
+    c.showPage()
+    
+    # API Reference - Page 1
+    c.setFont("Helvetica-Bold", 16)
+    c.drawString(inch, height-inch, "API Reference")
+    
+    c.setFont("Courier", 10)
+    api_content = [
+        "",
+        "def ingest_pdf(file_path: str, session_id: str,",
+        "               parser: str = 'pypdf',",
+        "               chunk_size: int = 4000,",
+        "               overlap: int = 400) -> Tuple[List[Chunk], str]:",
+        "    '''",
+        "    Main ingestion function for PDF files.",
+        "    ",
+        "    Parameters:",
+        "    -----------",
+        "    file_path : str",
+        "        Path to the PDF file to ingest",
+        "    session_id : str",
+        "        Unique session identifier",
+        "    parser : str",
+        "        Parser to use ('pypdf' or 'pdfplumber')",
+        "    chunk_size : int",
+        "        Maximum size of each text chunk",
+        "    overlap : int",
+        "        Number of overlapping characters between chunks",
+        "    ",
+        "    Returns:",
+        "    --------",
+        "    Tuple[List[Chunk], str]",
+        "        List of text chunks and output file path",
+        "    '''",
+    ]
+    
+    y = height - 1.5*inch
+    for line in api_content:
+        c.drawString(inch, y, line)
+        y -= 14
+    c.showPage()
+    
+    # Architecture - Page 2
+    c.setFont("Helvetica-Bold", 16)
+    c.drawString(inch, height-inch, "System Architecture")
+    
+    c.setFont("Helvetica", 11)
+    arch_content = [
+        "",
+        "Components:",
+        "",
+        "1. Text Extraction Layer",
+        "   - Primary parser: pypdf (fast, memory efficient)",
+        "   - Fallback parser: pdfplumber (better OCR support)",
+        "",
+        "2. Text Normalization",
+        "   - UTF-8 encoding enforcement",
+        "   - Control character removal",
+        "   - Whitespace normalization",
+        "",
+        "3. Chunking Strategy",
+        "   - Recursive character splitter",
+        "   - Configurable chunk size (default: 4000 chars)",
+        "   - Overlapping windows (default: 400 chars)",
+        "",
+        "4. Persistence Layer",
+        "   - JSON serialization",
+        "   - Session-based file naming",
+        "   - Temporary storage in .tmp/ingestion/",
+        "",
+        "5. Streamlit UI",
+        "   - Async processing with ThreadPoolExecutor",
+        "   - Real-time progress updates",
+        "   - Fallback parser retry mechanism",
+    ]
+    
+    y = height - 1.5*inch
+    for line in arch_content:
+        if line.startswith("   "):
+            c.setFont("Helvetica", 10)
+        else:
+            c.setFont("Helvetica", 11)
+        c.drawString(inch if not line.startswith("   ") else inch+0.3*inch, y, line)
+        y -= 16
+    
+    c.save()
+    print(f"Created: {output_path}")
+
+
+if __name__ == "__main__":
+    create_short_story_pdf()
+    create_technical_pdf()
+    print("\nSample PDFs generated successfully!")

tests/test_ingestion.py

@@ -0,0 +1,167 @@
+import json
+import os
+import tempfile
+from pathlib import Path
+import sys
+
+# Add src to path
+sys.path.insert(0, str(Path(__file__).parent.parent / "src"))
+
+import pytest
+from ingestion import (
+    normalize_text,
+    extract_text_pypdf,
+    extract_text_pdfplumber,
+    chunk_text,
+    make_chunks,
+    persist_chunks,
+    ingest_pdf,
+    Chunk
+)
+
+
+class TestTextNormalization:
+    def test_removes_control_characters(self):
+        text = "Hello\x00World\x0C\x1FTest"
+        result = normalize_text(text)
+        assert result == "Hello World Test"
+    
+    def test_normalizes_whitespace(self):
+        text = "Multiple   spaces\n\n\nand\t\ttabs"
+        result = normalize_text(text)
+        assert result == "Multiple spaces and tabs"
+    
+    def test_handles_unicode_errors(self):
+        text = "Valid text with émojis 😀"
+        result = normalize_text(text)
+        assert "Valid text" in result
+
+
+class TestChunking:
+    def test_basic_chunking(self):
+        text = "a" * 1000
+        chunks = chunk_text(text, chunk_size=100, overlap=10)
+        assert len(chunks) == 11  # 1000 chars / ~90 effective chars per chunk
+        assert all(len(c) <= 100 for c in chunks)
+    
+    def test_overlap_functionality(self):
+        text = "0123456789" * 10  # 100 chars
+        chunks = chunk_text(text, chunk_size=30, overlap=10)
+        # Verify overlap exists
+        for i in range(len(chunks) - 1):
+            assert chunks[i][-10:] == chunks[i+1][:10]
+    
+    def test_empty_text(self):
+        chunks = chunk_text("", chunk_size=100, overlap=10)
+        assert chunks == []  # Empty text returns empty list
+    
+    def test_text_shorter_than_chunk_size(self):
+        text = "Short text"
+        chunks = chunk_text(text, chunk_size=100, overlap=10)
+        assert len(chunks) == 1
+        assert chunks[0] == text
+
+
+class TestChunkPersistence:
+    def test_persist_and_load_chunks(self):
+        chunks = [
+            Chunk(content="Test chunk 1", source="test.pdf", page=1),
+            Chunk(content="Test chunk 2", source="test.pdf", page=2)
+        ]
+        session_id = "test_session_123"
+        
+        # Persist chunks
+        output_path = persist_chunks(session_id, chunks)
+        assert os.path.exists(output_path)
+        
+        # Load and verify
+        with open(output_path, "r") as f:
+            loaded = json.load(f)
+        
+        assert len(loaded) == 2
+        assert loaded[0]["content"] == "Test chunk 1"
+        assert loaded[1]["page"] == 2
+        
+        # Cleanup
+        os.remove(output_path)
+
+
+class TestPDFGeneration:
+    @pytest.fixture
+    def sample_pdf_path(self):
+        """Generate a simple PDF for testing"""
+        from reportlab.pdfgen import canvas
+        from reportlab.lib.pagesizes import letter
+        
+        with tempfile.NamedTemporaryFile(suffix=".pdf", delete=False) as tmp:
+            pdf_path = tmp.name
+            
+        c = canvas.Canvas(pdf_path, pagesize=letter)
+        
+        # Page 1 - Short story sample
+        c.drawString(100, 750, "The Test Story")
+        c.drawString(100, 700, "Once upon a time in a unit test,")
+        c.drawString(100, 680, "there was a PDF that needed parsing.")
+        c.drawString(100, 660, "It contained multiple pages and text.")
+        c.showPage()
+        
+        # Page 2 - Technical content
+        c.drawString(100, 750, "Technical Documentation")
+        c.drawString(100, 700, "API Reference: parse_pdf(file_path: str)")
+        c.drawString(100, 680, "Returns: List[Tuple[int, str]]")
+        c.drawString(100, 660, "Raises: ValueError if file not found")
+        c.showPage()
+        
+        c.save()
+        
+        yield pdf_path
+        
+        # Cleanup
+        os.remove(pdf_path)
+    
+    def test_pypdf_extraction(self, sample_pdf_path):
+        pages = extract_text_pypdf(sample_pdf_path)
+        assert len(pages) == 2
+        assert "Test Story" in pages[0][1]
+        assert "Technical Documentation" in pages[1][1]
+    
+    @pytest.mark.skipif(not os.system("which pdfplumber > /dev/null 2>&1") == 0, 
+                        reason="pdfplumber not installed")
+    def test_pdfplumber_extraction(self, sample_pdf_path):
+        pages = extract_text_pdfplumber(sample_pdf_path)
+        assert len(pages) == 2
+        assert "Test Story" in pages[0][1]
+        assert "Technical Documentation" in pages[1][1]
+    
+    def test_full_ingestion_pipeline(self, sample_pdf_path):
+        session_id = "test_full_pipeline"
+        chunks, output_path = ingest_pdf(
+            sample_pdf_path,
+            session_id,
+            parser="pypdf",
+            chunk_size=200,
+            overlap=20
+        )
+        
+        # Verify chunks created
+        assert len(chunks) > 0
+        assert all(isinstance(c, Chunk) for c in chunks)
+        
+        # Verify persistence
+        assert os.path.exists(output_path)
+        with open(output_path, "r") as f:
+            persisted = json.load(f)
+        assert len(persisted) == len(chunks)
+        
+        # Cleanup
+        os.remove(output_path)
+
+
+class TestErrorHandling:
+    def test_invalid_parser(self):
+        with pytest.raises(ValueError, match="Unknown parser"):
+            ingest_pdf("dummy.pdf", "session", parser="invalid_parser")
+    
+    def test_nonexistent_file(self):
+        with pytest.raises(Exception):  # FileNotFoundError or similar
+            extract_text_pypdf("/nonexistent/file.pdf")