README.md

metadata

title: BGE-M3 Multilingual Embedding Server
emoji: 🚀
colorFrom: blue
colorTo: indigo
sdk: docker
pinned: false

BGE-M3 Multilingual Embedding Server

A high-performance, robust, and production-ready embedding server utilizing the BGE-M3 model. This server is designed to provide dense, multilingual embeddings with features like batch processing, automatic GPU/CPU detection, and strict security guards.

---

🚀 Key Features

• State-of-the-Art Model: Uses BAAI/bge-m3 for high-quality multilingual embeddings. U can Change it also. with any huggigface model name just make sure it gives 1024 size vector embedding for consistency.
• FastAPI Powered: Asynchronous, high-performance API endpoints.
• Intelligent Device Management: Automatically detects NVIDIA GPUs (CUDA) or falls back to CPU.
• Robust Batching: Efficiently processes large lists of text with automatic chunking and OOM (Out of Memory) recovery.
• Strict Security: Built-in guard for API Key enforcement.
• Premium Logging: Beautiful, colorized, and structured logs for easy debugging and monitoring.
• Reliability: Integrated retry logic for model loading and processing.

---

🛠️ Tech Stack

• Language: Python 3.11+
• Web Framework: FastAPI
• Deep Learning: PyTorch & HuggingFace Transformers
• Validation: Pydantic V2 & Pydantic Settings
• Logging: Colorlog
• Process Manager: Uvicorn

---

📥 Installation

It is highly recommended to use a virtual environment to keep dependencies local to the project.

1. Clone the project and navigate to the directory.
2. Create a virtual environment:

   python -m venv venv
   

3. Activate the environment:
   • Windows: venv\Scripts\activate
   • Linux/Mac: source venv/bin/activate
4. Install dependencies:

   pip install -r requirements.txt
   

---

⚙️ Configuration

Copy the .env.example file to .env and configure your settings. The application uses these variables to control everything from security to model performance.

# -----------------------------
# Security / Access
# -----------------------------
EMBED_API_KEY=your_secret_key_here
HUGGING_FACE_TOKEN=your_free_token_with_repo_read_access

# -----------------------------
# Model & Device Settings
# -----------------------------
MODEL_NAME=BAAI/bge-m3       # Best if u copy this repo to your personal repo and use updated model name
DEVICE=                      # Leave empty for auto-detection, or use "cuda" / "cpu"
BATCH_SIZE=8                 # Adjust based on your VRAM/RAM
MAX_LENGTH=1024              # Max token length

# -----------------------------
# Server & Reliability
# -----------------------------
RETRY_ATTEMPTS=3             # Retries for model loading/processing
RETRY_BACKOFF_SECONDS=2.0    # Wait time between retries
HOST=0.0.0.0
PORT=8000
WORKERS=1                    # Number of Uvicorn workers
LOG_LEVEL=INFO               # DEBUG, INFO, WARNING, ERROR, CRITICAL

---

🏁 Running Locally

Start the server using Python's module runner:

python -m app.main

The server will automatically log the detected device (GPU/CPU) and start listening on the configured port.

---

🐳 Docker Deployment

You can deploy the server easily using Docker and Docker Compose. This is the recommended way for production.

1. Build and Run

docker-compose up --build -d

This will start the server on the port defined in your .env (default 7860).

Note: After building the image if u want to run the image without code just use this command

docker run -d --restart unless-stopped --name embed-server -p 7860:7860 --env-file .env vector-embedding-server-embed-server:latest

here vector-embedding-server-embed-server:latest is the image name which is build

.env should have the Configuration specified in the .env.example file

2. GPU Acceleration (NVIDIA)

To use your NVIDIA GPU with Docker, you must have the NVIDIA Container Toolkit installed on your host system.

1. Open docker-compose.yml.
2. Uncomment the deploy section at the bottom of the file:

   deploy:
     resources:
       reservations:
         devices:
           - driver: nvidia
             count: all
             capabilities: [gpu]
   

3. Restart the container:

   docker-compose up -d --force-recreate
   

The server logs will confirm if cuda is being used.

---

🚀 Run Docker Image (Without Source Code)

After building the image, run the container using:

docker run -d --restart unless-stopped --name embed-server -p 7860:7860 --env-file .env vector-embedding-server-embed-server:latest

Requirements

• The Docker image vector-embedding-server-embed-server:latest must already be built.
• A .env file must exist in the same directory.

Environment Setup

Create your .env file based on:

.env.example

Example:

cp .env.example .env

Update the values inside .env before running the container.

Check Running Container

docker ps

You should see embed-server listed as running.

---

📡 API Endpoints

1. Health Check

GET /embed/health
Returns the server status and whether the model is loaded.

2. Single Embedding

POST /embed/single
Generates an embedding for a single text string.

3. Batch Embedding

POST /embed/batch
Generates embeddings for a list of strings efficiently.

---

🧪 Documentation & Testing

Once the server is running, you can access the interactive API documentation at:

• Swagger UI: http://localhost:7860/docs
• Redoc: http://localhost:7860/redoc

Test with CURL

Note: The server requires x-api-key header as configured in your .env.

Single Embedding Request:

curl -X POST http://localhost:7860/embed/single \
     -H "Content-Type: application/json" \
     -H "x-api-key: MY_SECURE_KEY" \
     -d '{"text": "Hello world, this is a test."}'

Batch Embedding Request:

curl -X POST http://localhost:7860/embed/batch \
     -H "Content-Type: application/json" \
     -H "x-api-key: MY_SECURE_KEY" \
     -d '{
       "texts": [
         "First sentence to embed.",
         "BGE-M3 handles multilingual text very well."
       ]
     }'

---

🔧 Troubleshooting

OSError: Could not create safetensors conversion PR

If you see this error in your logs, it means the transformers library is trying to auto-convert your PyTorch model to SafeTensors format but failing (often due to repo permissions or structure).

Solution:

We have included a script to manually convert and upload the SafeTensors model to your Hugging Face repository.

Prerequisites:

1. Write Access: You must have write access to the Hugging Face repository defined in MODEL_NAME.
2. Environment Variable: Ensure your HUGGING_FACE_TOKEN in .env has write permissions.

Instructions: Run this script ONLY ONCE and ONLY if you are facing the error:

python -m scripts.convert_safetensors

This will:

1. Load your current model.
2. Convert it to SafeTensors format locally.
3. Verify the converted model loads correctly.
4. Upload the model.safetensors file to your repository.

Once done, the "safetensors not found" error will disappear, and your model loading speed will improve.

---

👨‍💻 Developed By

FAYSAL AHMMED