| --- |
| title: README |
| emoji: 🔧 |
| colorFrom: indigo |
| colorTo: purple |
| sdk: static |
| pinned: false |
| --- |
| |
| # SlyBase |
|
|
| **Fast LLM inference on consumer AMD GPUs.** |
|
|
| We run large models on a single AMD Radeon AI PRO R9700 (RDNA4, 32 GB) and publish what it takes to make |
| them fast: kernels, serving configs and ready-to-run quantizations — measured, not guessed. |
|
|
| ### 🚀 [vllm-sly-radiance](https://github.com/SlyBase/vllm-sly-radiance) |
|
|
| A vLLM fork for RDNA4 (gfx1201). Hand-written W4A8 MXFP4 kernels on the fp8 matrix cores, fused |
| norm/quant, tuned decode and drafter attention, DFlash2 speculative decoding and int4 heads — Qwen3.8-27B at |
| ~130 tokens/s decode on one card, 262k context. One Docker image, every change gated on a benchmark and a |
| quality check. |
|
|
| ### 📦 Models |
|
|
| Quantizations built for hardware with native MXFP4 (RDNA4, AMD Instinct MI350), each with a model card |
| that says what it costs in quality and speed — including when you are better off with another build. |
|
|
| - [ThinkingCap-Qwen3.8-27B-MXFP4](https://huggingface.co/slybase/ThinkingCap-Qwen3.8-27B-MXFP4) — |
| BottleCap AI's thinking-efficient Qwen3.8-27B finetune, GPTQ-quantized to MXFP4. |
|
|
| ### 📏 How we measure |
|
|
| Decode by step time and speculative acceptance (not single noisy runs), quality by deterministic |
| prompt-logprob comparison plus GSM8K, all on the same card and power cap. Numbers and scripts live in the |
| fork's [docs](https://github.com/SlyBase/vllm-sly-radiance/tree/main/docs). |
|
|
| --- |
|
|
| Credit where it's due: this work stands on [vLLM](https://github.com/vllm-project/vllm), |
| [StillDeadcode/vllm-radiance](https://github.com/StillDeadcode/vllm-radiance) and ggz14's RDNA4 kernels. |
| Model weights belong to their authors and keep their licenses. |
|
|
