README.md

---
title: BGE-M3 Multilingual Embedding Server
emoji: 🚀
colorFrom: blue
colorTo: indigo
sdk: docker
pinned: false
---

# BGE-M3 Multilingual Embedding Server

A high-performance, robust, and production-ready embedding server utilizing the **BGE-M3** model. This server is designed to provide dense, multilingual embeddings with features like batch processing, automatic GPU/CPU detection, and strict security guards.

---

## 🚀 Key Features

- **State-of-the-Art Model**: Uses `BAAI/bge-m3` for high-quality multilingual embeddings. U can Change it also. with any huggigface model name just make sure it gives 1024 size vector embedding for consistency.
- **FastAPI Powered**: Asynchronous, high-performance API endpoints.
- **Intelligent Device Management**: Automatically detects NVIDIA GPUs (CUDA) or falls back to CPU.
- **Robust Batching**: Efficiently processes large lists of text with automatic chunking and OOM (Out of Memory) recovery.
- **Strict Security**: Built-in guard for API Key enforcement.
- **Premium Logging**: Beautiful, colorized, and structured logs for easy debugging and monitoring.
- **Reliability**: Integrated retry logic for model loading and processing.

---

## 🛠️ Tech Stack

- **Language**: Python 3.11+
- **Web Framework**: FastAPI
- **Deep Learning**: PyTorch & HuggingFace Transformers
- **Validation**: Pydantic V2 & Pydantic Settings
- **Logging**: Colorlog
- **Process Manager**: Uvicorn

---

## 📥 Installation

It is highly recommended to use a virtual environment to keep dependencies local to the project.

1.  **Clone the project** and navigate to the directory.
2.  **Create a virtual environment**:
    ```cmd
    python -m venv venv
    ```
3.  **Activate the environment**:
    - Windows: `venv\Scripts\activate`
    - Linux/Mac: `source venv/bin/activate`
4.  **Install dependencies**:
    ```cmd
    pip install -r requirements.txt
    ```

---

## ⚙️ Configuration

Copy the `.env.example` file to `.env` and configure your settings. The application uses these variables to control everything from security to model performance.

```env
# -----------------------------
# Security / Access
# -----------------------------
EMBED_API_KEY=your_secret_key_here
HUGGING_FACE_TOKEN=your_free_token_with_repo_read_access

# -----------------------------
# Model & Device Settings
# -----------------------------
MODEL_NAME=BAAI/bge-m3       # Best if u copy this repo to your personal repo and use updated model name
DEVICE=                      # Leave empty for auto-detection, or use "cuda" / "cpu"
BATCH_SIZE=8                 # Adjust based on your VRAM/RAM
MAX_LENGTH=1024              # Max token length

# -----------------------------
# Server & Reliability
# -----------------------------
RETRY_ATTEMPTS=3             # Retries for model loading/processing
RETRY_BACKOFF_SECONDS=2.0    # Wait time between retries
HOST=0.0.0.0
PORT=8000
WORKERS=1                    # Number of Uvicorn workers
LOG_LEVEL=INFO               # DEBUG, INFO, WARNING, ERROR, CRITICAL
```

---

## 🏁 Running Locally

Start the server using Python's module runner:

```cmd
python -m app.main
```

The server will automatically log the detected device (GPU/CPU) and start listening on the configured port.

---

## 🐳 Docker Deployment

You can deploy the server easily using Docker and Docker Compose. This is the recommended way for production.

### 1. Build and Run

```bash
docker-compose up --build -d
```

This will start the server on the port defined in your `.env` (default `7860`).
# Note: After building the image if u want to run the image without code just use this command
# docker run -d --restart unless-stopped --name embed-server -p 7860:7860 --env-file .env vector-embedding-server-embed-server:latest
# here vector-embedding-server-embed-server:latest is the image name which is build
# .env should have the Configuration specified in the .env.example file

### 2. GPU Acceleration (NVIDIA)

To use your NVIDIA GPU with Docker, you must have the **NVIDIA Container Toolkit** installed on your host system.

1.  Open `docker-compose.yml`.
2.  Uncomment the `deploy` section at the bottom of the file:
    ```yaml
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    ```
3.  Restart the container:
    ```bash
    docker-compose up -d --force-recreate
    ```

The server logs will confirm if `cuda` is being used.

---

## 🚀 Run Docker Image (Without Source Code)

After building the image, run the container using:
```
docker run -d --restart unless-stopped --name embed-server -p 7860:7860 --env-file .env vector-embedding-server-embed-server:latest
```
# Requirements

- The Docker image vector-embedding-server-embed-server:latest must already be built.
- A .env file must exist in the same directory.

# Environment Setup

Create your .env file based on:
```
.env.example
```
Example:
```
cp .env.example .env
```
Update the values inside .env before running the container.

# Check Running Container
```
docker ps
```
You should see embed-server listed as running.

---

## 📡 API Endpoints

### 1. Health Check

`GET /embed/health`  
Returns the server status and whether the model is loaded.

### 2. Single Embedding

`POST /embed/single`  
Generates an embedding for a single text string.

### 3. Batch Embedding

`POST /embed/batch`  
Generates embeddings for a list of strings efficiently.

---

## 🧪 Documentation & Testing

Once the server is running, you can access the interactive API documentation at:

- **Swagger UI**: `http://localhost:7860/docs`
- **Redoc**: `http://localhost:7860/redoc`

### Test with CURL

**Note**: The server requires `x-api-key` header as configured in your `.env`.

#### Single Embedding Request:

```bash
curl -X POST http://localhost:7860/embed/single \
     -H "Content-Type: application/json" \
     -H "x-api-key: MY_SECURE_KEY" \
     -d '{"text": "Hello world, this is a test."}'
```

#### Batch Embedding Request:

```bash
curl -X POST http://localhost:7860/embed/batch \
     -H "Content-Type: application/json" \
     -H "x-api-key: MY_SECURE_KEY" \
     -d '{
       "texts": [
         "First sentence to embed.",
         "BGE-M3 handles multilingual text very well."
       ]
     }'
```

---

## 🔧 Troubleshooting

### `OSError: Could not create safetensors conversion PR`

If you see this error in your logs, it means the `transformers` library is trying to auto-convert your PyTorch model to SafeTensors format but failing (often due to repo permissions or structure).

**Solution:**

We have included a script to manually convert and upload the SafeTensors model to your Hugging Face repository.

**Prerequisites:**
1.  **Write Access**: You must have **write access** to the Hugging Face repository defined in `MODEL_NAME`.
2.  **Environment Variable**: Ensure your `HUGGING_FACE_TOKEN` in `.env` has **write permissions**.

**Instructions:**
Run this script **ONLY ONCE** and **ONLY** if you are facing the error:

```bash
python -m scripts.convert_safetensors
```

This will:
1.  Load your current model.
2.  Convert it to SafeTensors format locally.
3.  Verify the converted model loads correctly.
4.  Upload the `model.safetensors` file to your repository.

Once done, the "safetensors not found" error will disappear, and your model loading speed will improve.

---

## 👨‍💻 Developed By

**FAYSAL AHMMED**