| --- |
| license: apache-2.0 |
| language: |
| - en |
| - zh |
| tags: |
| - text generation |
| - gguf |
| - llama.cpp |
| - code |
| - MoE |
| - 80B |
| - zen-5 |
| - zen-5-coder |
| - zen5-coder |
| - VibeManGeo |
| base_model: |
| - zenlm/Zen-5-Coder |
| --- |
| |
| # Zen-5-Coder GGUF |
|
|
| GGUF quantizations of **Zen-5-Coder 80B** for **llama.cpp** and compatible runtimes. |
|
|
| The original model was released by **Zen LM** in Hugging Face Transformers format. This repository provides converted and quantized GGUF versions optimized for local inference across a wide range of hardware configurations. |
|
|
| --- |
|
|
| ## Overview |
|
|
| | Property | Value | |
| | --------------------- | ------------------------- | |
| | Model | Zen-5-Coder | |
| | Architecture | Mixture of Experts (MoE) | |
| | Parameters | 80B | |
| | Original Format | Hugging Face Transformers | |
| | GGUF Conversion | llama.cpp | |
| | Repository Maintainer | VibeManGeo | |
|
|
| --- |
|
|
| ## Available Quantizations |
|
|
| | Quantization | Description | |
| | ------------ | --------------------------- | |
| | **Q2_K** | Lowest memory usage | |
| | **Q3_K_M** | Balanced low-memory option | |
| | **Q4_K_M** | Recommended default | |
| | **Q5_K_M** | Higher quality generation | |
| | **Q6_K** | Near-lossless experience | |
| | **Q8_0** | Maximum GGUF quality | |
| | **FP16** | Unquantized reference model | |
| |
| --- |
| |
| ## Conversion Pipeline |
| |
| All files were generated locally using the standard llama.cpp workflow: |
| |
| ```text |
| Hugging Face Transformers |
| ↓ |
| GGUF FP16 |
| ↓ |
| GGUF Quantization |
| ``` |
| |
| ### Tools Used |
| |
| * llama.cpp |
| * convert_hf_to_gguf.py |
| * llama-quantize |
| |
| --- |
| |
| ## Example Usage |
| |
| ### llama.cpp |
| |
| ```bash |
| llama-cli \ |
| -m Zen-5-Coder-Q4_K_M.gguf \ |
| -c 32768 \ |
| -ngl 999 \ |
| -p "Write a Python web server" |
| ``` |
| |
| ### llama-server |
| |
| ```bash |
| llama-server \ |
| -m Zen-5-Coder-Q4_K_M.gguf \ |
| -c 32768 \ |
| --host 127.0.0.1 \ |
| --port 8080 |
| ``` |
| |
| --- |
| |
| ## Hardware Used For Conversion |
| |
| The quantizations in this repository were generated and tested on: |
| |
| * GPU 0 NVIDIA RTX 3060 12 GB Headless |
| * GPU 1 NVIDIA Tesla P40 24 GB Headless |
| * AMD Ryzen 7 5700G |
| * 64 GB DDR-4 3200Mhz System RAM |
| * Debian Linux 13.2 |
| |
| Actual performance will depend on context size, quantization level, GPU offloading, and runtime configuration. |
| |
| --- |
| |
| ## Credits |
| |
| ### Original Model |
| |
| **Zen LM** — creators of Zen-5-Coder. |
|
|
| ### GGUF Conversion & Quantization |
|
|
| **VibeManGeo** |
|
|
| **Fun fact:** these 80B quantizations were produced before the author passed CompTIA A+ Core 1. |
|
|
| --- |
|
|
| ## Acknowledgements |
|
|
| Special thanks to the **llama.cpp** developers for providing the tools that make efficient local inference and GGUF quantization possible. |
|
|
| --- |
|
|
| ## Disclaimer |
|
|
| This repository contains converted and quantized derivatives of the original model. |
|
|
| All credit for model architecture, training, datasets, and original weights belongs to the original authors. |
|
|
| --- |
|
|
| ## Support the Original Authors |
|
|
| If these GGUF files save you the time and compute resources required for conversion and quantization, please consider supporting the original creators by visiting the original Zen-5-Coder model page. |
|
|
| ## Notes |
|
|
| These GGUF files were independently converted and quantized from the original Hugging Face release using llama.cpp. |
|
|
| The goal of this repository is to make Zen-5-Coder immediately accessible to the local inference community without requiring users to perform the conversion process themselves. |
|
|
