README.md

metadata

title: Dynamic LLM ZeroGPU Playground
emoji: 🧪
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: 5.50.0
app_file: app.py
short_description: Test Transformers and GGUF LLM quants on ZeroGPU
python_version: '3.10'
startup_duration_timeout: 1h

Dynamic LLM ZeroGPU Playground

A small, general-purpose playground for testing Hugging Face LLMs by model ID. It supports standard Transformers checkpoints and direct, non-dequantized GGUF inference through a CUDA-enabled llama.cpp backend.

How it works

1. Enter a model ID.
2. Click Inspect / list GGUF. If the repository contains GGUF, choose one quant file such as Q4_K_M, Q5_K_M, Q8_0, IQ4, or a newer type.
3. Choose Backend: Auto (recommended), or force Transformers/llama.cpp.
4. Click Download. Standard repositories use a CPU-side snapshot download; GGUF repositories download only the selected .gguf file.
5. Click Load, then chat. Use Unload before switching models and Delete from disk to remove the cached revisions/files.

Only one model is kept active by the runtime. Switching models releases the previous model with del, gc.collect(), and torch.cuda.empty_cache() before the new model is loaded. Chat templates are used whenever the tokenizer provides apply_chat_template().

Backend routing

• Auto routes GGUF to llama.cpp and Transformers weight repositories to AutoModelForCausalLM.
• AWQ, GPTQ, bitsandbytes 4/8-bit, compressed-tensors, and FP8 metadata are detected from config.json and filenames. Transformers receives the repository quantization config and uses the installed optional loaders.
• GGUF is never passed to Transformers or dequantized. llama.cpp is loaded with n_gpu_layers=-1, and the GGUF's embedded chat template/metadata is used by the Python binding for current Qwen and other supported architectures.
• Only one model/backend is active at a time. Cleanup calls del, gc.collect(), torch.cuda.empty_cache(), and llama.cpp's close method when available.

Notes

• Model downloads happen on CPU and are never triggered by the chat handler.
• A model must be downloaded before it can be loaded or used.
• Large models may exceed ZeroGPU memory or take a long time to load.
• Remote model code is disabled for safety and stability.
• A GGUF repository normally embeds its tokenizer/chat metadata, so the large companion files are not downloaded. Multimodal mmproj files are listed but are not selected as the default quant.
• Very large models can still exceed the temporary ZeroGPU memory budget; Q4/Q5 GGUF files are generally the best starting point on an A10G.
• Some Hub repositories are gated even when their file list is visible. For a gated repo, accept access on Hugging Face and add a read-scoped HF_TOKEN secret to this Space; the UI reports this requirement before download.