Buckets:
| # GGUF | |
| [GGUF](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md) is a single-file format used to store models for inference with [GGML](https://github.com/ggerganov/ggml), containing the model metadata and tensors. It supports many quantized data types (refer to the [quantization type table](https://hf.co/docs/hub/en/gguf#quantization-types)), which saves a significant amount of memory. | |
| ## Load GGUF models | |
| Install [kernels](https://huggingface.co/docs/kernels/index), otherwise the packed path falls back to full [dequantization](#dequantize) at load. | |
| ```bash | |
| pip install kernels | |
| ``` | |
| Weights stay packed when the Hub kernel [ggml-org/ggml-quantization](https://huggingface.co/ggml-org/ggml-quantization) is available. The loader defaults to MPS when that kernel is present and runs matmuls directly on the packed blocks. If the kernel isn't available, the model is dequantized at load. | |
| ```py | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| model_id = "unsloth/Qwen3.5-4B-GGUF" | |
| filename = "Qwen3.5-4B-Q4_K_M.gguf" | |
| model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename) | |
| tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename) | |
| ``` | |
| The same load works for Qwen3.5 MoE. | |
| ```py | |
| moe_model_id = "unsloth/Qwen3.5-35B-A3B-GGUF" | |
| moe_filename = "Qwen3.5-35B-A3B-Q4_K_M.gguf" | |
| moe_model = AutoModelForCausalLM.from_pretrained(moe_model_id, gguf_file=moe_filename) | |
| moe_tokenizer = AutoTokenizer.from_pretrained(moe_model_id, gguf_file=moe_filename) | |
| ``` | |
| The packed path currently supports Qwen3.5 and Qwen3.5 MoE. Packed loads use float32 automatically because it's faster on MPS. If you set another `dtype`, the loader returns a warning. Other architectures go through the legacy loader. | |
| ## Attention | |
| On MPS, with kernels installed, [ggml-attn](https://huggingface.co/kernels/ggml-org/ggml-attn) is the default attention when loading a GGUF file: the same flash-attention kernel llama.cpp uses for decode and prefill. If the kernel cannot be fetched, the model keeps its attention implementation. Passing `attn_implementation` (for example `"sdpa"`) always takes precedence. | |
| You can also pass it explicitly: | |
| ```py | |
| model = AutoModelForCausalLM.from_pretrained( | |
| model_id, gguf_file=filename, attn_implementation="ggml-org/ggml-attn" | |
| ) | |
| ``` | |
| ## Dequantize | |
| Dequantizing unpacks every weight at load time and gives back a plain dense model. It is the fallback whenever the fast, compressed path doesn't apply. You can also ask for it explicitly with [GgufConfig](/docs/transformers/pr_36708/en/main_classes/quantization#transformers.GgufConfig). | |
| ```py | |
| import torch | |
| from transformers import AutoModelForCausalLM, GgufConfig | |
| quantization_config = GgufConfig(dequantize=True) | |
| model = AutoModelForCausalLM.from_pretrained( | |
| model_id, gguf_file=filename, quantization_config=quantization_config, dtype=torch.bfloat16 | |
| ) | |
| ``` | |
| You get a regular dense model in the `dtype` you passed. | |
| Architectures other than Qwen3.5 and Qwen3.5 MoE go through the legacy loader, which always dequantizes. | |
| > [!TIP] | |
| > The legacy loader supports Llama, Mistral, Qwen2, Qwen2Moe, Phi3, Bloom, Falcon, StableLM, GPT2, Starcoder2, and [more](https://github.com/huggingface/transformers/blob/main/src/transformers/integrations/ggml.py). | |
| ## Serve | |
| [transformers serve](../serve-cli/serving) lists each `.gguf` file in a repository as its own model. A GGUF model is named `<repo>:<file>.gguf`, since a repository holds | |
| several quantizations and the id has to say which one to load. Requests name it the same way. | |
| ```shell | |
| transformers serve unsloth/Qwen3.5-4B-GGUF:Qwen3.5-4B-Q4_K_M.gguf | |
| ``` | |
Xet Storage Details
- Size:
- 3.64 kB
- Xet hash:
- acf2a5e337a9505eea14220a582df843885079a4261a6061123eac1a8ffb1dff
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.