Spaces:
Sleeping
Sleeping
Commit ·
9519893
1
Parent(s): 86105c9
perf(ingest): KI-125 — bump MPS embedding batch 64→128 for bulk re-ingest
Browse filesDoubles batch size on Mac M-series MPS device. ~100 MB per batch (was ~50 MB),
still well within unified-memory budget. Halves GPU kernel launches when
ingesting all 206 policies + 18 regulatory docs from scratch (motivated by
the 2026-05-15 re-ingest). CPU path unchanged at batch_size=32 to keep peak
RSS bounded on machines without a GPU.
backend/providers/local_embeddings.py
CHANGED
|
@@ -73,8 +73,11 @@ class LocalEmbeddings(EmbeddingsProvider):
|
|
| 73 |
texts = [f"Represent this sentence for searching relevant passages: {t}" for t in texts]
|
| 74 |
# Batch size scales by device: MPS / CUDA throughput benefits from
|
| 75 |
# bigger batches; CPU prefers smaller to avoid memory pressure on M1.
|
| 76 |
-
#
|
| 77 |
-
|
|
|
|
|
|
|
|
|
|
| 78 |
vectors = self.model.encode(
|
| 79 |
texts,
|
| 80 |
batch_size=batch,
|
|
|
|
| 73 |
texts = [f"Represent this sentence for searching relevant passages: {t}" for t in texts]
|
| 74 |
# Batch size scales by device: MPS / CUDA throughput benefits from
|
| 75 |
# bigger batches; CPU prefers smaller to avoid memory pressure on M1.
|
| 76 |
+
# KI-125 (2026-05-15): bumped MPS 64→128 to halve the number of GPU
|
| 77 |
+
# kernel launches during bulk re-ingest. 800-token chunks at 128 ≈
|
| 78 |
+
# 100 MB per batch — still well within Mac M-series unified memory.
|
| 79 |
+
# CPU stays at 32 to keep peak RSS bounded on machines without a GPU.
|
| 80 |
+
batch = 128 if self.device in ("mps", "cuda") else 32
|
| 81 |
vectors = self.model.encode(
|
| 82 |
texts,
|
| 83 |
batch_size=batch,
|