docs/CIP_5_Large_Batch_Handling_Improvements.md

# CIP-5: Large Batch Handling Improvements Proposal

## Status

Current Status: `Under Discussion`

## **Motivation**

As users start putting Chroma in its paces and storing ever-increasing datasets, we must ensure that errors
related to significant and potentially expensive batches are handled gracefully. This CIP proposes to add a new
setting, `max_batch_size` API, on the local segment API and use it to split large batches into smaller ones.

## **Public Interfaces**

The following interfaces are impacted:

- New Server API endpoint - `/pre-flight-checks`
- New `max_batch_size` property on the `API` interface
- Updated `_add`, `_update` and `_upsert` methods on `chromadb.api.segment.SegmentAPI`
- Updated `_add`, `_update` and `_upsert` methods on `chromadb.api.fastapi.FastAPI`
- New utility library `batch_utils.py`
- New exception raised when batch size exceeds `max_batch_size`

## **Proposed Changes**

We propose the following changes:

- The new `max_batch_size` property is now available in the `API` interface. The property relies on the
  underlying `Producer` class
  to fetch the actual value. The property will be implemented by both `chromadb.api.segment.SegmentAPI`
  and `chromadb.api.fastapi.FastAPI`
- `chromadb.api.segment.SegmentAPI` will implement the `max_batch_size` property by fetching the value from the
  `Producer` class.
- `chromadb.api.fastapi.FastAPI` will implement the `max_batch_size` by fetching it from a new `/pre-flight-checks`
  endpoint on the Server.
- New `/pre-flight-checks` endpoint on the Server will return a dictionary with pre-flight checks the client must
  fulfil to integrate with the server side. For now, we propose using this only for `max_batch_size`, but we can
  add more checks in the future. The pre-flight checks will be only fetched once per client and cached for the duration
  of the client's lifetime.
- Updated `_add`, `_update` and `_upsert` method on `chromadb.api.segment.SegmentAPI` to validate batch size.
- Updated `_add`, `_update` and `_upsert` method on `chromadb.api.fastapi.FastAPI`  to validate batch size (client-side
  validation)
- New utility library `batch_utils.py` will contain the logic for splitting batches into smaller ones.

## **Compatibility, Deprecation, and Migration Plan**

The change will be fully compatible with existing implementations. The changes will be transparent to the user.

## **Test Plan**

New tests:

- Batch splitting tests for `chromadb.api.segment.SegmentAPI`
- Batch splitting tests for `chromadb.api.fastapi.FastAPI`
- Tests for `/pre-flight-checks` endpoint

## **Rejected Alternatives**

N/A