README.md

---
title: Dynamic LLM ZeroGPU Playground
emoji: 🧪
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: 5.50.0
app_file: app.py
short_description: Test Transformers and GGUF LLM quants on ZeroGPU
python_version: "3.10"
startup_duration_timeout: 1h
---

# Dynamic LLM ZeroGPU Playground

A small, general-purpose playground for testing Hugging Face LLMs by model ID.
It supports standard Transformers checkpoints and direct, non-dequantized GGUF
inference through a CUDA-enabled llama.cpp backend.

## How it works

1. Enter a model ID.
2. Click **Inspect / list GGUF**. If the repository contains GGUF, choose one
   quant file such as Q4_K_M, Q5_K_M, Q8_0, IQ4, or a newer type.
3. Choose **Backend: Auto** (recommended), or force Transformers/llama.cpp.
4. Click **Download**. Standard repositories use a CPU-side snapshot download;
   GGUF repositories download only the selected `.gguf` file.
5. Click **Load**, then chat. Use **Unload** before switching models and
   **Delete from disk** to remove the cached revisions/files.

Only one model is kept active by the runtime. Switching models releases the
previous model with `del`, `gc.collect()`, and `torch.cuda.empty_cache()` before
the new model is loaded. Chat templates are used whenever the tokenizer
provides `apply_chat_template()`.

## Backend routing

- `Auto` routes GGUF to llama.cpp and Transformers weight repositories to
  `AutoModelForCausalLM`.
- AWQ, GPTQ, bitsandbytes 4/8-bit, compressed-tensors, and FP8 metadata are
  detected from `config.json` and filenames. Transformers receives the
  repository quantization config and uses the installed optional loaders.
- GGUF is never passed to Transformers or dequantized. llama.cpp is loaded with
  `n_gpu_layers=-1`, and the GGUF's embedded chat template/metadata is used by
  the Python binding for current Qwen and other supported architectures.
- Only one model/backend is active at a time. Cleanup calls `del`,
  `gc.collect()`, `torch.cuda.empty_cache()`, and llama.cpp's close method when
  available.

## Notes

- Model downloads happen on CPU and are never triggered by the chat handler.
- A model must be downloaded before it can be loaded or used.
- Large models may exceed ZeroGPU memory or take a long time to load.
- Remote model code is disabled for safety and stability.
- A GGUF repository normally embeds its tokenizer/chat metadata, so the large
  companion files are not downloaded. Multimodal `mmproj` files are listed but
  are not selected as the default quant.
- Very large models can still exceed the temporary ZeroGPU memory budget; Q4/Q5
  GGUF files are generally the best starting point on an A10G.
- Some Hub repositories are gated even when their file list is visible. For a
  gated repo, accept access on Hugging Face and add a read-scoped `HF_TOKEN`
  secret to this Space; the UI reports this requirement before download.