fine_tuning/GGUF_CONVERSION.md

Convert Fine-Tuned Model To GGUF

Run these steps on the GPU/training machine or another Linux machine with enough disk space.

1. Clone llama.cpp

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
pip install -r requirements.txt
cmake -B build
cmake --build build --config Release -j

2. Merge LoRA Into The Base Model

From the Advisor repo:

python fine_tuning/scripts/merge_lora.py \
  --model_id openbmb/MiniCPM5-1B \
  --adapter_dir fine_tuning/out/advisor-minicpm-lora \
  --output_dir fine_tuning/out/advisor-minicpm-merged

3. Convert To FP16 GGUF

From llama.cpp:

python convert_hf_to_gguf.py /path/to/Advisor/fine_tuning/out/advisor-minicpm-merged \
  --outfile /path/to/advisor-minicpm-f16.gguf \
  --outtype f16

4. Quantize

./build/bin/llama-quantize \
  /path/to/advisor-minicpm-f16.gguf \
  /path/to/advisor-minicpm-q4_k_m.gguf \
  Q4_K_M

On Windows, the executable path/name may differ.

5. Upload To Hugging Face

Create a repo such as:

your-user/advisor-minicpm-finetuned-gguf

Upload:

advisor-minicpm-q4_k_m.gguf

Then update the app environment:

LLAMA_HF_REPO=your-user/advisor-minicpm-finetuned-gguf
LLAMA_HF_FILENAME=advisor-minicpm-q4_k_m.gguf