docs/weaviate_database_setup.md

Weaviate Database Setup

This project uses Weaviate Cloud to store retrieval chunks and vectors. The application generates embeddings through OpenRouter openai/text-embedding-3-small and stores them as self-provided vectors.

Installation steps

1. Create a new python virtual environment using python -m venv venv, activate the environment via source venv/bin/activate, install the needed requirements from the requirements.txt file if you haven't done it already.
2. Configure WEAVIATE_CLUSTER_URL, WEAVIATE_API_KEY, and OPEN_ROUTER_API_KEY.
3. With the python environment activated, initialize the collections with python main.py --weaviate init. Inspect the logs to check whether collection creation was successful.

If you've managed to setup the database and create the collections, the installation process is finished and the database is accessible from the other parts of the program.

Managing the database

To manage the state of the database directly, multiple useful scripts were developed. The scripts can be called via the weaviate.py using the following arguments:

• -cc or --create_collections: initializes separate collections for different language contents.
• -dc or --delete_collections: deletes all collections and their contents from the database.
• -rc or --redo_collections: deletes the collections and creates them again.
• -ch or --checkhealth: checks the connection to the database and existence of the content collections.
• -cb or --create_backup: creates a backup of the current state of the database.
• -rb pr --restore_backup: restores the state of the database from the provided backup_id.

When changing embedding model, tokenizer, or vector dimensions, rebuild the collections and re-import content:

python main.py --weaviate redo
python main.py --scrape full

Run python main.py --imports ... afterward for any local documents that are part of the knowledge base.

Data properties

Embeddings are stored in the corresponding language collection with a set of properties that define chunk metadata:

• body (TEXT): text content of the stored embedding.
• chunk_id (TEXT): ID of the chunk defined by the data processor.
• document_id (TEXT): ID of the document from which the chunk was derived (also defined by the data processor).
• programs (TEXT_ARRAY): list of the EMBA programs that were identified in the document of the derived information.
• source (TEXT): source of the information (name of the document or the url).
• date (DATE): date when the data chunk was prepared for the insertion.

WeaviateService

The WeaviateService class manages the connection and interaction with Weaviate Cloud.