Xylolabs

GLM-4.7-Flash GGUF

Base model Formats License Code

GGUF quants of zai-org/GLM-4.7-Flash for llama.cpp, on CUDA, Metal or CPU.

  • Files: IQ4_XS and IQ4_XS-imat.
  • IQ4_XS: KL 0.0868 from BF16, top-1 87.80 %.
  • Which one to use: the smallest file whose KL you accept; see Files.

Files

  • GLM-4.7-Flash-<QUANT>.gguf is one file per quant.
  • Contents: Each file holds the full language model plus the original tokenizer and chat template. It also carries the MTP module.
Quant Size Mean KL vs BF16 Same top-1 vs BF16
IQ4_XS 16.88 GB 0.0868 87.80%
IQ4_XS-imat 16.71 GB 0.0845 88.34%

How we measured:

  • Reference: our BF16 pass in transformers.
  • KL and top-1: 16,288 positions of English and Korean text.

Mean KL and same top-1 against BF16 by file size

How to run

Use llama.cpp commit cd959bae99ff4bed22bef2604f45241c09bbb8c8 or later.

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout cd959bae99ff4bed22bef2604f45241c09bbb8c8
cmake -B build -DCMAKE_BUILD_TYPE=Release        # add -DGGML_CUDA=ON for NVIDIA GPUs
cmake --build build -j --target llama-cli llama-server

./build/bin/llama-cli -m GLM-4.7-Flash-IQ4_XS.gguf --jinja -c 8192 -ngl 99

llama-server takes the same flags.

How these were made

  1. Convert the original checkpoint to a BF16 GGUF with convert_hf_to_gguf.py.
  2. Quantize each type with llama-quantize.
  3. The -imat files use an importance matrix.

License and notices

  • License: MIT License, copied in LICENSE.
  • What we changed: we converted the weights to GGUF and quantized them.
  • Notices: NOTICE names the source model and its copyright holder.
  • Xylolabs is not affiliated with or endorsed by the original model's authors.

Citation

Cite the original GLM-4.7-Flash model card and its linked GLM technical report.

Downloads last month
1,196
GGUF
Model size
31B params
Architecture
deepseek2
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Xylolabs/GLM-4.7-Flash-GGUF

Quantized
(115)
this model

Collection including Xylolabs/GLM-4.7-Flash-GGUF