README.md

---

title: BGE-M3 Multilingual Embedding Server
emoji: 🚀
colorFrom: blue
colorTo: indigo
sdk: docker
pinned: false
---


# BGE-M3 Multilingual Embedding Server

A high-performance, robust, and production-ready embedding server utilizing the **BGE-M3** model. This server is designed to provide dense, multilingual embeddings with features like batch processing, automatic GPU/CPU detection, and strict security guards.

---

## 🚀 Key Features

- **State-of-the-Art Model**: Uses `BAAI/bge-m3` for high-quality multilingual embeddings. U can Change it also. with any huggigface model name just make sure it gives 1024 size vector embedding for consistency.
- **FastAPI Powered**: Asynchronous, high-performance API endpoints.
- **Intelligent Device Management**: Automatically detects NVIDIA GPUs (CUDA) or falls back to CPU.
- **Robust Batching**: Efficiently processes large lists of text with automatic chunking and OOM (Out of Memory) recovery.
- **Strict Security**: Built-in guard for API Key enforcement.
- **Premium Logging**: Beautiful, colorized, and structured logs for easy debugging and monitoring.
- **Reliability**: Integrated retry logic for model loading and processing.

---

## 🛠️ Tech Stack

- **Language**: Python 3.11+
- **Web Framework**: FastAPI
- **Deep Learning**: PyTorch & HuggingFace Transformers
- **Validation**: Pydantic V2 & Pydantic Settings
- **Logging**: Colorlog
- **Process Manager**: Uvicorn

---

## 📥 Installation

It is highly recommended to use a virtual environment to keep dependencies local to the project.

1.  **Clone the project** and navigate to the directory.
2.  **Create a virtual environment**:
    ```cmd

    python -m venv venv

    ```

3.  **Activate the environment**:

    - Windows: `venv\Scripts\activate`

    - Linux/Mac: `source venv/bin/activate`

4.  **Install dependencies**:

    ```cmd

    pip install -r requirements.txt

    ```


---

## ⚙️ Configuration

Copy the `.env.example` file to `.env` and configure your settings. The application uses these variables to control everything from security to model performance.

```env

# -----------------------------

# Security / Access

# -----------------------------

EMBED_API_KEY=your_secret_key_here

HUGGING_FACE_TOKEN=your_free_token_with_repo_read_access



# -----------------------------

# Model & Device Settings

# -----------------------------

MODEL_NAME=BAAI/bge-m3       # Best if u copy this repo to your personal repo and use updated model name

DEVICE=                      # Leave empty for auto-detection, or use "cuda" / "cpu"

BATCH_SIZE=8                 # Adjust based on your VRAM/RAM

MAX_LENGTH=1024              # Max token length



# -----------------------------

# Server & Reliability

# -----------------------------

RETRY_ATTEMPTS=3             # Retries for model loading/processing

RETRY_BACKOFF_SECONDS=2.0    # Wait time between retries

HOST=0.0.0.0

PORT=8000

WORKERS=1                    # Number of Uvicorn workers

LOG_LEVEL=INFO               # DEBUG, INFO, WARNING, ERROR, CRITICAL

```

---

## 🏁 Running Locally

Start the server using Python's module runner:

```cmd

python -m app.main

```

The server will automatically log the detected device (GPU/CPU) and start listening on the configured port.

---

## 🐳 Docker Deployment

You can deploy the server easily using Docker and Docker Compose. This is the recommended way for production.

### 1. Build and Run

```bash

docker-compose up --build -d

```

This will start the server on the port defined in your `.env` (default `7860`).
# Note: After building the image if u want to run the image without code just use this command
# docker run -d --restart unless-stopped --name embed-server -p 7860:7860 --env-file .env vector-embedding-server-embed-server:latest
# here vector-embedding-server-embed-server:latest is the image name which is build
# .env should have the Configuration specified in the .env.example file

### 2. GPU Acceleration (NVIDIA)

To use your NVIDIA GPU with Docker, you must have the **NVIDIA Container Toolkit** installed on your host system.

1.  Open `docker-compose.yml`.
2.  Uncomment the `deploy` section at the bottom of the file:
    ```yaml

    deploy:

      resources:

        reservations:

          devices:

            - driver: nvidia

              count: all

              capabilities: [gpu]

    ```

3.  Restart the container:

    ```bash

    docker-compose up -d --force-recreate

    ```


The server logs will confirm if `cuda` is being used.

---

## 🚀 Run Docker Image (Without Source Code)

After building the image, run the container using:
```

docker run -d --restart unless-stopped --name embed-server -p 7860:7860 --env-file .env vector-embedding-server-embed-server:latest

```
# Requirements

- The Docker image vector-embedding-server-embed-server:latest must already be built.
- A .env file must exist in the same directory.

# Environment Setup

Create your .env file based on:
```

.env.example

```
Example:
```

cp .env.example .env

```
Update the values inside .env before running the container.

# Check Running Container
```

docker ps

```
You should see embed-server listed as running.

---

## 📡 API Endpoints

### 1. Health Check

`GET /embed/health`  
Returns the server status and whether the model is loaded.

### 2. Single Embedding

`POST /embed/single`  
Generates an embedding for a single text string.

### 3. Batch Embedding

`POST /embed/batch`  
Generates embeddings for a list of strings efficiently.

---

## 🧪 Documentation & Testing

Once the server is running, you can access the interactive API documentation at:

- **Swagger UI**: `http://localhost:7860/docs`
- **Redoc**: `http://localhost:7860/redoc`

### Test with CURL

**Note**: The server requires `x-api-key` header as configured in your `.env`.

#### Single Embedding Request:

```bash

curl -X POST http://localhost:7860/embed/single \

     -H "Content-Type: application/json" \

     -H "x-api-key: MY_SECURE_KEY" \

     -d '{"text": "Hello world, this is a test."}'

```

#### Batch Embedding Request:

```bash

curl -X POST http://localhost:7860/embed/batch \

     -H "Content-Type: application/json" \

     -H "x-api-key: MY_SECURE_KEY" \

     -d '{

       "texts": [

         "First sentence to embed.",

         "BGE-M3 handles multilingual text very well."

       ]

     }'

```

---

## 🔧 Troubleshooting

### `OSError: Could not create safetensors conversion PR`

If you see this error in your logs, it means the `transformers` library is trying to auto-convert your PyTorch model to SafeTensors format but failing (often due to repo permissions or structure).

**Solution:**

We have included a script to manually convert and upload the SafeTensors model to your Hugging Face repository.

**Prerequisites:**
1.  **Write Access**: You must have **write access** to the Hugging Face repository defined in `MODEL_NAME`.
2.  **Environment Variable**: Ensure your `HUGGING_FACE_TOKEN` in `.env` has **write permissions**.

**Instructions:**
Run this script **ONLY ONCE** and **ONLY** if you are facing the error:

```bash

python -m scripts.convert_safetensors

```

This will:
1.  Load your current model.
2.  Convert it to SafeTensors format locally.
3.  Verify the converted model loads correctly.
4.  Upload the `model.safetensors` file to your repository.

Once done, the "safetensors not found" error will disappear, and your model loading speed will improve.

---

## 👨‍💻 Developed By

**FAYSAL AHMMED**