Spaces:
Sleeping
Sleeping
feat: Introduce asynchronous loading system in DEVELOPER.md with new methods and usage examples
Browse files- docs/DEVELOPER.md +44 -1
docs/DEVELOPER.md
CHANGED
|
@@ -97,6 +97,15 @@ pip install -e ".[dev,web]" # Core + dev + web server
|
|
| 97 |
- **`chunk_transcripts()`**: Async semantic chunking with timestamp preservation
|
| 98 |
- **Custom embedding models**: Fine-tuned embeddings for PsTuts domain
|
| 99 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 100 |
#### π Evaluation System
|
| 101 |
- **`evaluator_utils.py`**: RAG evaluation utilities using RAGAS framework
|
| 102 |
- **Notebook-based evaluation**: `evaluate_rag.ipynb` for systematic testing
|
|
@@ -164,4 +173,38 @@ ipdb # Available for interactive debugging
|
|
| 164 |
- [LangChain Documentation](https://python.langchain.com/docs/get_started/introduction) π¦
|
| 165 |
- [LangGraph Documentation](https://langchain-ai.github.io/langgraph/) πΈοΈ
|
| 166 |
- [Qdrant Documentation](https://qdrant.tech/documentation/) π
|
| 167 |
-
- [RAGAS Documentation](https://docs.ragas.io/) π
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 97 |
- **`chunk_transcripts()`**: Async semantic chunking with timestamp preservation
|
| 98 |
- **Custom embedding models**: Fine-tuned embeddings for PsTuts domain
|
| 99 |
|
| 100 |
+
#### β‘ Asynchronous Loading System
|
| 101 |
+
- **`DatastoreManager.loading_complete`**: AsyncIO Event object that's set when data loading completes
|
| 102 |
+
- **`DatastoreManager.is_ready()`**: Convenience method to check if loading is complete
|
| 103 |
+
- **`DatastoreManager.wait_for_loading(timeout)`**: Async method to wait for loading completion with optional timeout
|
| 104 |
+
- **`DatastoreManager.add_completion_callback(callback)`**: Register callbacks (sync or async) to be called when loading completes
|
| 105 |
+
- **Non-blocking startup**: Vector database loading runs in background threads to prevent UI blocking
|
| 106 |
+
- **Background processing**: `asyncio.create_task()` used for concurrent data loading during application startup
|
| 107 |
+
- **Event-driven notifications**: Hook into loading completion for reactive programming patterns
|
| 108 |
+
|
| 109 |
#### π Evaluation System
|
| 110 |
- **`evaluator_utils.py`**: RAG evaluation utilities using RAGAS framework
|
| 111 |
- **Notebook-based evaluation**: `evaluate_rag.ipynb` for systematic testing
|
|
|
|
| 173 |
- [LangChain Documentation](https://python.langchain.com/docs/get_started/introduction) π¦
|
| 174 |
- [LangGraph Documentation](https://langchain-ai.github.io/langgraph/) πΈοΈ
|
| 175 |
- [Qdrant Documentation](https://qdrant.tech/documentation/) π
|
| 176 |
+
- [RAGAS Documentation](https://docs.ragas.io/) π
|
| 177 |
+
|
| 178 |
+
### π Usage Examples
|
| 179 |
+
|
| 180 |
+
#### Event-Based Loading with Callbacks
|
| 181 |
+
```python
|
| 182 |
+
# Option 1: Custom callback passed to startup
|
| 183 |
+
async def my_completion_handler():
|
| 184 |
+
print("β
Database is ready for queries!")
|
| 185 |
+
await notify_users("System ready")
|
| 186 |
+
|
| 187 |
+
datastore = await startup(
|
| 188 |
+
config=my_config,
|
| 189 |
+
on_loading_complete=my_completion_handler
|
| 190 |
+
)
|
| 191 |
+
|
| 192 |
+
# Option 2: Register callbacks after initialization
|
| 193 |
+
datastore = await startup(config=my_config)
|
| 194 |
+
|
| 195 |
+
# Add additional callbacks
|
| 196 |
+
def on_complete():
|
| 197 |
+
print("β
Loading finished!")
|
| 198 |
+
|
| 199 |
+
async def on_complete_async():
|
| 200 |
+
await send_notification("Database ready")
|
| 201 |
+
|
| 202 |
+
datastore.add_completion_callback(on_complete)
|
| 203 |
+
datastore.add_completion_callback(on_complete_async)
|
| 204 |
+
|
| 205 |
+
# Option 3: Wait for completion with timeout
|
| 206 |
+
if await datastore.wait_for_loading(timeout=60):
|
| 207 |
+
print("Loading completed within timeout")
|
| 208 |
+
else:
|
| 209 |
+
print("Loading timed out")
|
| 210 |
+
```
|