gemma4-llama-cpp / README.md
ArGrigorov's picture
Upload folder using huggingface_hub
bced3e1 verified
|
Raw History Blame Contribute Delete
1.83 kB
# llama.cpp (fork) β€” Gemma 4 + audio (mtmd)
Source snapshot of the private llama.cpp fork used by AGIWS inference workers.
## What is inside
- `src/models/gemma4.cpp`, `gemma4-assistant.cpp` β€” text models `gemma4`, `gemma4-assistant`.
- `tools/mtmd/clip.cpp` β€” graphs `clip_graph_gemma4a` / `clip_graph_gemma4ua` (audio encoders).
- `tools/mtmd/mtmd-audio.cpp` β€” preprocessors `gemma4a` / `gemma4ua`, mel filters, streaming.
- `tools/mtmd/mtmd.h` β€” `mtmd_support_audio`, `mtmd_bitmap_init_from_audio`.
- `tools/server` β€” OpenAI-compatible server accepting `input_audio` content parts.
- `conversion/gemma.py` β€” `Gemma4Model` GGUF converter (`Gemma4ForConditionalGeneration`).
## Files
- `src/llama-cpp-src.tar.gz` β€” full source tree (no `.git`, no build artifacts).
- `lib/*.so` β€” prebuilt CUDA libraries and `llama-server` for T4 (sm_75), uploaded by the build job.
- `lib/BUILD_INFO.txt` β€” compiler, CUDA version, git-less snapshot hash, sha256 of every artifact.
## Build (Colab T4)
```bash
tar -xzf src/llama-cpp-src.tar.gz -C /content/llama-cpp-fork
cmake -S /content/llama-cpp-fork -B /content/llama-cpp-fork/build \
-DCMAKE_BUILD_TYPE=Release \
-DGGML_CUDA=ON \
-DGGML_NATIVE=OFF \
-DCMAKE_CUDA_ARCHITECTURES=75 \
-DBUILD_SHARED_LIBS=ON \
-DLLAMA_CURL=OFF \
-DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_EXAMPLES=OFF
cmake --build /content/llama-cpp-fork/build -j"$(nproc)" --target llama-server
```
Artifacts needed at runtime: `llama-server` plus `libllama.so*`, `libmtmd.so*`, `libggml*.so*`
(`libggml-base`, `libggml-cpu`, `libggml-cuda`, `libggml`).
Runtime flags for Gemma audio:
```bash
LD_LIBRARY_PATH=/content/llama/lib \
/content/llama/lib/llama-server -m .../gemma-4-12b-it-Q4_K_M.gguf \
--mmproj .../mmproj-F16.gguf -ngl 99 -c 4096 --host 127.0.0.1 --port 8091
```