rohitsar567 commited on
Commit
9519893
·
1 Parent(s): 86105c9

perf(ingest): KI-125 — bump MPS embedding batch 64→128 for bulk re-ingest

Browse files

Doubles batch size on Mac M-series MPS device. ~100 MB per batch (was ~50 MB),
still well within unified-memory budget. Halves GPU kernel launches when
ingesting all 206 policies + 18 regulatory docs from scratch (motivated by
the 2026-05-15 re-ingest). CPU path unchanged at batch_size=32 to keep peak
RSS bounded on machines without a GPU.

backend/providers/local_embeddings.py CHANGED
@@ -73,8 +73,11 @@ class LocalEmbeddings(EmbeddingsProvider):
73
  texts = [f"Represent this sentence for searching relevant passages: {t}" for t in texts]
74
  # Batch size scales by device: MPS / CUDA throughput benefits from
75
  # bigger batches; CPU prefers smaller to avoid memory pressure on M1.
76
- # 800-token chunks at batch_size=64 is ~50 MB which fits 8GB M1 fine.
77
- batch = 64 if self.device in ("mps", "cuda") else 32
 
 
 
78
  vectors = self.model.encode(
79
  texts,
80
  batch_size=batch,
 
73
  texts = [f"Represent this sentence for searching relevant passages: {t}" for t in texts]
74
  # Batch size scales by device: MPS / CUDA throughput benefits from
75
  # bigger batches; CPU prefers smaller to avoid memory pressure on M1.
76
+ # KI-125 (2026-05-15): bumped MPS 64→128 to halve the number of GPU
77
+ # kernel launches during bulk re-ingest. 800-token chunks at 128
78
+ # 100 MB per batch — still well within Mac M-series unified memory.
79
+ # CPU stays at 32 to keep peak RSS bounded on machines without a GPU.
80
+ batch = 128 if self.device in ("mps", "cuda") else 32
81
  vectors = self.model.encode(
82
  texts,
83
  batch_size=batch,