Spaces:
Running on Zero
Running on Zero
atakan Claude Sonnet 5 commited on
Commit ·
f971ecf
1
Parent(s): c01ad45
perf: Build llama.cpp for AVX2/FMA instead of the generic fallback target
Browse filesGGML_NATIVE=OFF (added to fix the build OOM) compiles a lowest-common-
denominator binary with no SIMD acceleration, which made CPU inference
markedly slower than necessary. Pin explicitly to AVX2+FMA instead --
universally supported on the x86-64 server CPUs Spaces runs on -- to
recover most of that speed without reintroducing the OOM (that was
caused by unbounded build parallelism, not by the AVX2 codegen itself)
or the portability risk of a full native/AVX-512 build.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Dockerfile +6 -1
Dockerfile
CHANGED
|
@@ -26,9 +26,14 @@ WORKDIR /home/user/app
|
|
| 26 |
# Install Python dependencies. llama-cpp-python compiles from source (no
|
| 27 |
# prebuilt wheel is published for recent versions); cap build parallelism so
|
| 28 |
# the compiler doesn't spawn enough jobs to OOM-kill the Spaces build machine.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 29 |
COPY requirements.txt .
|
| 30 |
ENV CMAKE_BUILD_PARALLEL_LEVEL=1 \
|
| 31 |
-
CMAKE_ARGS="-DGGML_NATIVE=OFF"
|
| 32 |
RUN pip install --no-cache-dir --upgrade pip && \
|
| 33 |
pip install --no-cache-dir -r requirements.txt
|
| 34 |
|
|
|
|
| 26 |
# Install Python dependencies. llama-cpp-python compiles from source (no
|
| 27 |
# prebuilt wheel is published for recent versions); cap build parallelism so
|
| 28 |
# the compiler doesn't spawn enough jobs to OOM-kill the Spaces build machine.
|
| 29 |
+
# Pin to a fixed AVX2+FMA target instead of NATIVE (avoids depending on the
|
| 30 |
+
# build machine's exact CPU) or full auto-detection (which was the extra
|
| 31 |
+
# compile cost that caused the OOM) -- AVX2/FMA are present on essentially
|
| 32 |
+
# every x86-64 server CPU Spaces runs on, so this keeps inference fast
|
| 33 |
+
# without the crash risk of a mismatched AVX-512 build.
|
| 34 |
COPY requirements.txt .
|
| 35 |
ENV CMAKE_BUILD_PARALLEL_LEVEL=1 \
|
| 36 |
+
CMAKE_ARGS="-DGGML_NATIVE=OFF -DGGML_AVX=ON -DGGML_AVX2=ON -DGGML_FMA=ON -DGGML_AVX512=OFF"
|
| 37 |
RUN pip install --no-cache-dir --upgrade pip && \
|
| 38 |
pip install --no-cache-dir -r requirements.txt
|
| 39 |
|