atakan Claude Sonnet 5 commited on
Commit
f971ecf
·
1 Parent(s): c01ad45

perf: Build llama.cpp for AVX2/FMA instead of the generic fallback target

Browse files

GGML_NATIVE=OFF (added to fix the build OOM) compiles a lowest-common-
denominator binary with no SIMD acceleration, which made CPU inference
markedly slower than necessary. Pin explicitly to AVX2+FMA instead --
universally supported on the x86-64 server CPUs Spaces runs on -- to
recover most of that speed without reintroducing the OOM (that was
caused by unbounded build parallelism, not by the AVX2 codegen itself)
or the portability risk of a full native/AVX-512 build.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Files changed (1) hide show
  1. Dockerfile +6 -1
Dockerfile CHANGED
@@ -26,9 +26,14 @@ WORKDIR /home/user/app
26
  # Install Python dependencies. llama-cpp-python compiles from source (no
27
  # prebuilt wheel is published for recent versions); cap build parallelism so
28
  # the compiler doesn't spawn enough jobs to OOM-kill the Spaces build machine.
 
 
 
 
 
29
  COPY requirements.txt .
30
  ENV CMAKE_BUILD_PARALLEL_LEVEL=1 \
31
- CMAKE_ARGS="-DGGML_NATIVE=OFF"
32
  RUN pip install --no-cache-dir --upgrade pip && \
33
  pip install --no-cache-dir -r requirements.txt
34
 
 
26
  # Install Python dependencies. llama-cpp-python compiles from source (no
27
  # prebuilt wheel is published for recent versions); cap build parallelism so
28
  # the compiler doesn't spawn enough jobs to OOM-kill the Spaces build machine.
29
+ # Pin to a fixed AVX2+FMA target instead of NATIVE (avoids depending on the
30
+ # build machine's exact CPU) or full auto-detection (which was the extra
31
+ # compile cost that caused the OOM) -- AVX2/FMA are present on essentially
32
+ # every x86-64 server CPU Spaces runs on, so this keeps inference fast
33
+ # without the crash risk of a mismatched AVX-512 build.
34
  COPY requirements.txt .
35
  ENV CMAKE_BUILD_PARALLEL_LEVEL=1 \
36
+ CMAKE_ARGS="-DGGML_NATIVE=OFF -DGGML_AVX=ON -DGGML_AVX2=ON -DGGML_FMA=ON -DGGML_AVX512=OFF"
37
  RUN pip install --no-cache-dir --upgrade pip && \
38
  pip install --no-cache-dir -r requirements.txt
39