---
# For reference on model card metadata, see the spec: https://github.com/huggingface/hub-docs/blob/main/modelcard.md?plain=1
# Doc / guide: https://huggingface.co/docs/hub/model-cards
base_model:
- nm-testing/tinysmokeqwen3
library_name: transformers
tags:
- compressed-tensors
- kv-cache
- quantization
---

# TinyQwen3 KV-Cache-Only quantized

A tiny Qwen3 model ([nm-testing/tinysmokeqwen3](https://huggingface.co/nm-testing/tinysmokeqwen3)) quantized
with [llm-compressor](https://github.com/vllm-project/llm-compressor) using **only KV cache quantization** —
the model weights are left unquantized (dense).

## Quantization config

- `kv_cache_scheme`: 8-bit float (`fp8`), `tensor` strategy, static, symmetric
- `config_groups`: `{}` (no weight quantization)
- `quantization_status`: `compressed`

Produced with `oneshot` and the recipe:

```yaml
quant_stage:
    quant_modifiers:
        QuantizationModifier:
            kv_cache_scheme:
                num_bits: 8
                type: float
                strategy: tensor
                dynamic: False
                symmetric: True
```

Calibrated on 16 samples of `open_platypus` with `max_seq_length=512`.

## Use

Intended for testing `compressed-tensors` KV-cache-only checkpoints in `transformers` CI
(see `tests/quantization/compressed_tensors_integration/test_compressed_tensors.py`).

```python
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("inference-optimization/TinyQwen3-KV-Cache-Only")
```