README.md

---
license: other
license_name: exaone
license_link: https://huggingface.co/LGAI-EXAONE/EXAONE-Deep-2.4B/blob/main/LICENSE
library_name: llama.cpp
tags:
- gguf
- exaone
- quantized
- llama-cpp
- korean
base_model: LGAI-EXAONE/EXAONE-Deep-32B-Instruct
model_type: exaone
---

# Calvin806/EXAONE-Deep-32B-GGUF

GGUF quantizations for **EXAONE-Deep-32B**.

## Contents
This folder typically contains:
- `EXAONE-Deep-32B.F16.gguf`
- `EXAONE-Deep-32B.Q4_K_M.gguf`
- `EXAONE-Deep-32B.Q5_K_M.gguf`
- `EXAONE-Deep-32B.Q8_0.gguf` (optional)

---

## 🔧 llama.cpp patch (EXAONE GGUF quantize compatibility)

EXAONE GGUF 변환/양자화 과정에서 일부 모델(예: **2.4B / 7.8B**) 간 **KV key 네이밍 불일치**가 발견되었습니다.

- 어떤 GGUF는 `exaone.attention.layer_norm_epsilon`만 존재
- 어떤 GGUF는 `exaone.attention.layer_norm_rms_epsilon`만 존재

이 상태에서 vanilla llama.cpp의 `llama-quantize`가 특정 키를 찾지 못해 실패할 수 있어,
**llama.cpp의 model loader에서 gguf key lookup에 fallback을 추가하는 패치**를 적용했습니다.

### What was patched

`src/llama-model-loader.cpp`에서 `gguf_find_key()` lookup에 다음 fallback을 수행하도록 수정:

- key가 `exaone.attention.layer_norm_epsilon`이고 찾지 못하면 → `exaone.attention.layer_norm_rms_epsilon`로 재시도
- key가 `exaone.attention.layer_norm_rms_epsilon`이고 찾지 못하면 → `exaone.attention.layer_norm_epsilon`로 재시도

이 패치를 통해 **EXAONE 3.5 / EXAONE-Deep 2.4B, 7.8B, 32B** 계열을 동일 파이프라인으로 GGUF+quantize할 수 있습니다.

### Patch note (minimal diff summary)
- Added a fallback wrapper/hook for `gguf_find_key()` inside `llama-model-loader.cpp`
- Ensured all lookups in that translation unit route through the fallback

This repo includes:
- `exaone-gguf-fallback.patch`

### Tested llama.cpp commit
- `021cc28bef4dd7d0bf9c91dbbd0803caa6cb15f2`

---

## Build (CUDA)
```bash
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git apply ../exaone-gguf-fallback.patch

rm -rf build
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON
cmake --build build -j
```

## Convert / Quantize
```bash
# Convert HF snapshot -> GGUF(F16)
python3 llama.cpp/convert_hf_to_gguf.py <LOCAL_SNAPSHOT_DIR> --outtype f16 --outfile model.F16.gguf

# Quantize (example: Q4_K_M)
llama.cpp/build/bin/llama-quantize model.F16.gguf model.Q4_K_M.gguf Q4_K_M
```