mbudisic commited on
Commit
f672221
Β·
1 Parent(s): 9440955

feat: Introduce asynchronous loading system in DEVELOPER.md with new methods and usage examples

Browse files
Files changed (1) hide show
  1. docs/DEVELOPER.md +44 -1
docs/DEVELOPER.md CHANGED
@@ -97,6 +97,15 @@ pip install -e ".[dev,web]" # Core + dev + web server
97
  - **`chunk_transcripts()`**: Async semantic chunking with timestamp preservation
98
  - **Custom embedding models**: Fine-tuned embeddings for PsTuts domain
99
 
 
 
 
 
 
 
 
 
 
100
  #### πŸ” Evaluation System
101
  - **`evaluator_utils.py`**: RAG evaluation utilities using RAGAS framework
102
  - **Notebook-based evaluation**: `evaluate_rag.ipynb` for systematic testing
@@ -164,4 +173,38 @@ ipdb # Available for interactive debugging
164
  - [LangChain Documentation](https://python.langchain.com/docs/get_started/introduction) 🦜
165
  - [LangGraph Documentation](https://langchain-ai.github.io/langgraph/) πŸ•ΈοΈ
166
  - [Qdrant Documentation](https://qdrant.tech/documentation/) πŸ”
167
- - [RAGAS Documentation](https://docs.ragas.io/) πŸ“Š
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
97
  - **`chunk_transcripts()`**: Async semantic chunking with timestamp preservation
98
  - **Custom embedding models**: Fine-tuned embeddings for PsTuts domain
99
 
100
+ #### ⚑ Asynchronous Loading System
101
+ - **`DatastoreManager.loading_complete`**: AsyncIO Event object that's set when data loading completes
102
+ - **`DatastoreManager.is_ready()`**: Convenience method to check if loading is complete
103
+ - **`DatastoreManager.wait_for_loading(timeout)`**: Async method to wait for loading completion with optional timeout
104
+ - **`DatastoreManager.add_completion_callback(callback)`**: Register callbacks (sync or async) to be called when loading completes
105
+ - **Non-blocking startup**: Vector database loading runs in background threads to prevent UI blocking
106
+ - **Background processing**: `asyncio.create_task()` used for concurrent data loading during application startup
107
+ - **Event-driven notifications**: Hook into loading completion for reactive programming patterns
108
+
109
  #### πŸ” Evaluation System
110
  - **`evaluator_utils.py`**: RAG evaluation utilities using RAGAS framework
111
  - **Notebook-based evaluation**: `evaluate_rag.ipynb` for systematic testing
 
173
  - [LangChain Documentation](https://python.langchain.com/docs/get_started/introduction) 🦜
174
  - [LangGraph Documentation](https://langchain-ai.github.io/langgraph/) πŸ•ΈοΈ
175
  - [Qdrant Documentation](https://qdrant.tech/documentation/) πŸ”
176
+ - [RAGAS Documentation](https://docs.ragas.io/) πŸ“Š
177
+
178
+ ### πŸ”„ Usage Examples
179
+
180
+ #### Event-Based Loading with Callbacks
181
+ ```python
182
+ # Option 1: Custom callback passed to startup
183
+ async def my_completion_handler():
184
+ print("βœ… Database is ready for queries!")
185
+ await notify_users("System ready")
186
+
187
+ datastore = await startup(
188
+ config=my_config,
189
+ on_loading_complete=my_completion_handler
190
+ )
191
+
192
+ # Option 2: Register callbacks after initialization
193
+ datastore = await startup(config=my_config)
194
+
195
+ # Add additional callbacks
196
+ def on_complete():
197
+ print("βœ… Loading finished!")
198
+
199
+ async def on_complete_async():
200
+ await send_notification("Database ready")
201
+
202
+ datastore.add_completion_callback(on_complete)
203
+ datastore.add_completion_callback(on_complete_async)
204
+
205
+ # Option 3: Wait for completion with timeout
206
+ if await datastore.wait_for_loading(timeout=60):
207
+ print("Loading completed within timeout")
208
+ else:
209
+ print("Loading timed out")
210
+ ```