Spaces:
Sleeping
Sleeping
File size: 4,008 Bytes
bbb4f78 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 | # `app/utils` – Utility Modules for Summarization, Export, and Embeddings
This directory includes reusable tools that support the backend application with:
- Chat summarization for display/export
- Document extraction and cleanup
- File export to TXT, DOCX, and PDF formats
- File upload validation
- Persona-specific vector DB with ChromaDB
These modules are loosely coupled and used across core routes, RAG logic, and export endpoints.
---
## `chat_summary.py` – Conversation Summarization
This module provides summarization of past conversations using the LLM client.
### Key Functions
- `generate_summary_from_messages(messages, llm, max_tokens)` – Generates a formatted, bullet-style summary
- `format_summary_for_text_export(summary_text)` – Cleans summary for export to PDF/DOCX/TXT
- `parse_summary_to_blocks(summary_text)` – Converts summary to structured blocks (headings, lists, paragraphs)
### Format Guidelines
Summaries follow a markdown-style format with:
- `**Section Name:**` for headings
- `* Bullet Points` for insights and recommendations
- Auto-trimming and line breaks for export formatting
---
## `chroma_client.py` – Persona-Specific Knowledge Store
A minimal ChromaDB wrapper used to store and query persona-specific documents or embeddings.
### Functions
- `add_persona_doc(text, persona, doc_id)` – Add a new chunk/document for a persona
- `query_persona_knowledge(query, persona)` – Query ChromaDB for a persona-specific response
### Notes
- Uses `./chroma_storage` as the default persistent path
- Uses the local embedding model via `get_embedding()` from `embedding_client.py`
---
## `document_extractor.py` – File Text Extraction
Supports extracting raw text from uploaded documents.
### Supported Formats
| Format | Content Type |
|--------|---------------|
| PDF | `application/pdf` |
| DOCX | `application/vnd.openxmlformats-officedocument.wordprocessingml.document` |
| TXT | `text/plain` |
### Key Function
```python
extract_text_from_file(file_bytes: bytes, content_type: str) -> str
```
Uses:
- `PyPDF2` for PDFs
- `docx2txt` for Word documents (via temp file)
- UTF-8 decoding for plain text
---
## `file_export.py` – Export Chat & Summaries
Exports content (chat logs or summaries) to the following formats:
- `.txt`
- `.docx` (Word)
- `.pdf` (ReportLab)
### Key Functions
- `export_chat_as_file(content, format)` – Unified export method (calls generate_*)
- `prepare_export_response()` – Returns a `StreamingResponse` with correct content-disposition
### Formatting Functions
- `generate_txt_file()` – Simple UTF-8 stream
- `generate_docx_file()` – Paragraph-based Word file using `python-docx`
- `generate_pdf_file()` – Uses ReportLab’s Platypus for chat-style layout
- `generate_pdf_file_from_blocks()` – Used for structured summaries (heading, lists, etc.)
All formats apply automatic cleanup and styling via:
- `_clean_text_for_pdf()` and `_render_rich_text()`
---
## `file_limits.py` – Upload Size Checks
Used to prevent users from uploading excessively large files in a session.
### Configurable Limit
```python
MAX_TOTAL_UPLOAD_MB = 10
```
### Function
- `is_within_upload_limit(session_id, new_file_bytes, session_context)` – Returns `True` if upload is within session cap
Used by routes handling document uploads.
---
## Dependencies
These modules are used in:
| Module | Depends On |
|--------|------------|
| `rag_manager.py` | `document_extractor`, `file_limits` |
| `chat_summary.py` | `llm_client` |
| `routes/documents.py` | `document_extractor`, `file_limits` |
| `routes/export.py` | `file_export`, `chat_summary` |
---
## Example Workflow
```text
Upload File → document_extractor.py → raw text
↓
file_limits.py → check quota
Chat History → chat_summary.py → formatted summary
↓
file_export.py → TXT, DOCX, PDF
Persona Notes → chroma_client.py → embedded in ChromaDB
``` |