| --- |
| base_model: Kwaipilot/KAT-Dev |
| base_model_relation: quantized |
| quantized_by: ArtusDev |
| language: |
| - multilingual |
| license: other |
| license_name: kwaipilot-license |
| license_link: LICENSE |
| tags: |
| - exl3 |
| --- |
| <style> |
| .container-dark { |
| font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, "Helvetica Neue", Arial, sans-serif; |
| line-height: 1.6; |
| color: #d4d4d4; |
| } |
| a { |
| color: #569cd6; |
| text-decoration: none; |
| font-weight: 600; |
| } |
| a:hover { |
| text-decoration: underline; |
| } |
| .card-dark { |
| background-color: #252526; |
| border-radius: 12px; |
| padding: 24px; |
| margin-bottom: 20px; |
| box-shadow: 0 4px 12px rgba(0,0,0,0.3); |
| border: 1px solid #3c3c3c; |
| } |
| .card-dark h1 { |
| font-size: 2.2em; |
| color: #ffffff; |
| text-align: center; |
| margin-bottom: 10px; |
| } |
| .card-dark.card-dark-title h1 { |
| font-size: 1.5em; |
| } |
| .card-dark .subtitle { |
| text-align: center; |
| font-size: 1.1em; |
| color: #a0a0a0; |
| } |
| .card-dark h2 { |
| font-size: 1.5em; |
| margin-top: 0; |
| padding-bottom: 10px; |
| border-bottom: 1px solid #3c3c3c; |
| color: #c586c0; |
| } |
| .card-dark h3 { |
| font-size: 1.2em; |
| color: #d4d4d4; |
| } |
| .styled-table { |
| display: table; |
| border: none; |
| width: 100%; |
| font-size: 0.95em; |
| margin-bottom: 0px; |
| } |
| .styled-table thead th { |
| background-color: #333333; |
| color: #c586c0; |
| text-align: left; |
| } |
| .styled-table th { |
| padding: 12px 15px; |
| } |
| .styled-table td { |
| padding: 0; |
| } |
| .styled-table table, .styled-table th, .styled-table td { |
| border-left: none; |
| border-right: none; |
| border-bottom: none; |
| } |
| .styled-table td { |
| border-bottom: 1px solid #3c3c3c; |
| } |
| .styled-table tbody tr { |
| transition: background-color 0.1s ease; |
| } |
| .styled-table tbody tr:hover { |
| background-color: #3a3a3a; |
| } |
| .styled-table tr:last-child td { |
| border-bottom: none; |
| } |
| .styled-table td a { |
| display: block; |
| padding: 12px 15px; |
| } |
| .styled-table td a.fake-link { |
| text-decoration:none; |
| color:inherit; |
| } |
| details { |
| margin-top: 20px; |
| border: 1px solid #3c3c3c; |
| border-radius: 8px; |
| overflow: hidden; |
| } |
| summary { |
| cursor: pointer; |
| padding: 12px 18px; |
| background-color: #6A5ACD; |
| font-weight: 600; |
| display: flex; |
| align-items: center; |
| gap: 10px; |
| justify-content: space-between; |
| list-style: none; |
| } |
| summary::-webkit-details-marker { |
| display: none; |
| } |
| summary:hover { |
| filter: brightness(1.1); |
| } |
| summary::after { |
| content: ''; |
| display: inline-block; |
| width: 8px; |
| height: 8px; |
| border-bottom: 2px solid white; |
| border-right: 2px solid white; |
| transform: rotate(45deg); |
| transition: transform 0.3s ease; |
| } |
| details[open] > summary::after { |
| transform: rotate(225deg); |
| } |
| .details-content { |
| padding: 18px; |
| } |
| .btn-purple { |
| display: inline-block; |
| background-color: #6A5ACD; |
| color: white !important; |
| padding: 12px 24px; |
| border-radius: 8px; |
| text-decoration: none; |
| font-weight: 600; |
| transition: background-color 0.3s ease, transform 0.2s ease; |
| text-align: center; |
| } |
| .btn-purple:hover { |
| background-color: #7B68EE; |
| transform: translateY(-2px); |
| } |
| </style> |
| |
| <div class="container-dark"> |
|
|
| <div class="card-dark card-dark-title"> |
| <h1>ArtusDev/Kwaipilot_KAT-Dev-EXL3</h1> |
| <p class="subtitle"> |
| EXL3 quants of <a href="https://huggingface.co/Kwaipilot/KAT-Dev" target="_blank">Kwaipilot/KAT-Dev</a> using <a href="https://github.com/turboderp-org/exllamav3/" target="_blank">exllamav3</a> for quantization. |
| </p> |
| </div> |
| |
| <div class="card-dark"> |
| <h2>Quants</h2> |
| <table class="styled-table"> |
| <thead> |
| <tr> |
| <th>Quant</th> |
| <th>BPW</th> |
| <th>Head Bits</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><a href="https://huggingface.co/ArtusDev/Kwaipilot_KAT-Dev-EXL3/tree/2.5bpw_H6" target="_blank">2.5_H6</a></td> |
| <td><a class="fake-link" href="https://huggingface.co/ArtusDev/Kwaipilot_KAT-Dev-EXL3/tree/2.5bpw_H6" target="_blank">2.5</a></td> |
| <td><a class="fake-link" href="https://huggingface.co/ArtusDev/Kwaipilot_KAT-Dev-EXL3/tree/2.5bpw_H6" target="_blank">6</a></td> |
| </tr> |
| <tr> |
| <td><a href="https://huggingface.co/ArtusDev/Kwaipilot_KAT-Dev-EXL3/tree/3.0bpw_H6" target="_blank">3.0_H6</a></td> |
| <td><a class="fake-link" href="https://huggingface.co/ArtusDev/Kwaipilot_KAT-Dev-EXL3/tree/3.0bpw_H6" target="_blank">3.0</a></td> |
| <td><a class="fake-link" href="https://huggingface.co/ArtusDev/Kwaipilot_KAT-Dev-EXL3/tree/3.0bpw_H6" target="_blank">6</a></td> |
| </tr> |
| <tr> |
| <td><a href="https://huggingface.co/ArtusDev/Kwaipilot_KAT-Dev-EXL3/tree/3.5bpw_H6" target="_blank">3.5_H6</a></td> |
| <td><a class="fake-link" href="https://huggingface.co/ArtusDev/Kwaipilot_KAT-Dev-EXL3/tree/3.5bpw_H6" target="_blank">3.5</a></td> |
| <td><a class="fake-link" href="https://huggingface.co/ArtusDev/Kwaipilot_KAT-Dev-EXL3/tree/3.5bpw_H6" target="_blank">6</a></td> |
| </tr> |
| <tr> |
| <td><a href="https://huggingface.co/ArtusDev/Kwaipilot_KAT-Dev-EXL3/tree/4.0bpw_H6" target="_blank">4.0_H6</a></td> |
| <td><a class="fake-link" href="https://huggingface.co/ArtusDev/Kwaipilot_KAT-Dev-EXL3/tree/4.0bpw_H6" target="_blank">4.0</a></td> |
| <td><a class="fake-link" href="https://huggingface.co/ArtusDev/Kwaipilot_KAT-Dev-EXL3/tree/4.0bpw_H6" target="_blank">6</a></td> |
| </tr> |
| <tr> |
| <td><a href="https://huggingface.co/ArtusDev/Kwaipilot_KAT-Dev-EXL3/tree/4.5bpw_H6" target="_blank">4.5_H6</a></td> |
| <td><a class="fake-link" href="https://huggingface.co/ArtusDev/Kwaipilot_KAT-Dev-EXL3/tree/4.5bpw_H6" target="_blank">4.5</a></td> |
| <td><a class="fake-link" href="https://huggingface.co/ArtusDev/Kwaipilot_KAT-Dev-EXL3/tree/4.5bpw_H6" target="_blank">6</a></td> |
| </tr> |
| <tr> |
| <td><a href="https://huggingface.co/ArtusDev/Kwaipilot_KAT-Dev-EXL3/tree/5.0bpw_H6" target="_blank">5.0_H6</a></td> |
| <td><a class="fake-link" href="https://huggingface.co/ArtusDev/Kwaipilot_KAT-Dev-EXL3/tree/5.0bpw_H6" target="_blank">5.0</a></td> |
| <td><a class="fake-link" href="https://huggingface.co/ArtusDev/Kwaipilot_KAT-Dev-EXL3/tree/5.0bpw_H6" target="_blank">6</a></td> |
| </tr> |
| <tr> |
| <td><a href="https://huggingface.co/ArtusDev/Kwaipilot_KAT-Dev-EXL3/tree/6.0bpw_H6" target="_blank">6.0_H6</a></td> |
| <td><a class="fake-link" href="https://huggingface.co/ArtusDev/Kwaipilot_KAT-Dev-EXL3/tree/6.0bpw_H6" target="_blank">6.0</a></td> |
| <td><a class="fake-link" href="https://huggingface.co/ArtusDev/Kwaipilot_KAT-Dev-EXL3/tree/6.0bpw_H6" target="_blank">6</a></td> |
| </tr> |
| <tr> |
| <td><a href="https://huggingface.co/ArtusDev/Kwaipilot_KAT-Dev-EXL3/tree/8.0bpw_H8" target="_blank">8.0_H8</a></td> |
| <td><a class="fake-link" href="https://huggingface.co/ArtusDev/Kwaipilot_KAT-Dev-EXL3/tree/8.0bpw_H8" target="_blank">8.0</a></td> |
| <td><a class="fake-link" href="https://huggingface.co/ArtusDev/Kwaipilot_KAT-Dev-EXL3/tree/8.0bpw_H8" target="_blank">8</a></td> |
| </tr> |
| </tbody> |
| </table> |
| </div> |
| |
| <div class="card-dark"> |
| <h2>How to Download and Use Quants</h2> |
| <p>You can download quants by targeting specific size using the Hugging Face CLI.</p> |
| <details> |
| <summary>Click for download commands</summary> |
| <div class="details-content"> |
| <b>1. Install huggingface-cli:</b> |
| <pre><code>pip install -U "huggingface_hub[cli]"</code></pre> |
| <b>2. Download a specific quant:</b> |
| <pre><code>huggingface-cli download ArtusDev/Kwaipilot_KAT-Dev-EXL3 --revision "5.0bpw_H6" --local-dir ./</code></pre> |
| </div> |
| </details> |
| <p>EXL3 quants can be run with any inference client that supports EXL3, such as <a href="https://github.com/theroyallab/tabbyapi" target="_blank"><b>TabbyAPI</b></a>. Refer to <a href="https://github.com/theroyallab/tabbyAPI/wiki/01.-Getting-Started" target="_blank">documentation</a> for set up instructions.</p> |
| </div> |
| |
| <div class="card-dark"> |
| <h2>Quant Requests</h2> |
| <div style="text-align: center; margin-top: 25px;"> |
| <a href="https://huggingface.co/ArtusDev/requests-exl/discussions/new?title=[MODEL_NAME_HERE]&description=[MODEL_HF_LINK_HERE]" class="btn-purple" target="_blank">Request EXL3 Quants</a> |
| </div> |
| <p class="subtitle"> |
| See <a href="https://huggingface.co/ArtusDev/requests-exl" target="_blank">EXL community hub</a> for request guidelines. |
| </p> |
| </div> |
| |
| <div class="card-dark"> |
| <h2>Acknowledgements</h2> |
| <p style="text-align: center; font-size: 1.1em; margin-top: 10px;"> |
| Made possible with cloud compute from <a href="https://lium.io/" target="_blank"><b>lium.io</b></a> |
| </p> |
| </div> |
| |
| </div> |
