README.md

---
library_name: ctranslate2
base_model:
- Qwen/Qwen3-1.7B
base_model_relation: quantized
tags:
  - ctranslate2
  - chat
---
> [!NOTE]
> Bloat16 Ctranslate2 compatable version of [Qwen/Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B).

## VRAM Usage:

| Model | VRAM Usage |
|-------|------------|
| [Qwen3-32B-ct2-awq](https://huggingface.co/CTranslate2HQ/Qwen3-32B-ct2-AWQ) | ~18.3 GB |
| [Qwen3-14B-ct2-awq](https://huggingface.co/CTranslate2HQ/Qwen3-14B-ct2-AWQ) | ~9.5 GB |
| [Qwen3-8B-ct2-awq](https://huggingface.co/CTranslate2HQ/Qwen3-8B-ct2-AWQ) | ~5.8 GB |
| **👉 [Qwen3-1.7B-ct2-bfloat16](https://huggingface.co/CTranslate2HQ/Qwen3-1.7B-ct2-bfloat16)** | ~3.3 GB |
| [Qwen3-4B-ct2-awq](https://huggingface.co/CTranslate2HQ/Qwen3-4B-ct2-AWQ) | ~2.6 GB |
| [Qwen3-1.7B-ct2-awq](https://huggingface.co/CTranslate2HQ/Qwen3-1.7B-ct2-AWQ) | ~1.3 GB |
| [Qwen3-0.6B-ct2-awq](https://huggingface.co/CTranslate2HQ/Qwen3-0.6B-ct2-AWQ) | ~0.6 GB |

## Example Usage:

```python
import ctranslate2
from transformers import AutoTokenizer

MODEL_ID = "CTranslate2HQ/Qwen3-1.7B-ct2-bfloat16"

# Load model and tokenizer from Hugging Face Hub
generator = ctranslate2.Generator(MODEL_ID, device="cuda")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

# Format prompt using chat template
messages = [
    {"role": "system", "content": "You are a helpful AI assistant."},
    {"role": "user", "content": "Write a short poem about a cat."}
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False
)

# Tokenize and generate
tokens = tokenizer.convert_ids_to_tokens(tokenizer.encode(prompt))

# Do NOT use the "compute_type" parameter with AWQ models
results = generator.generate_batch(
    [tokens],
    max_length=8192,
    sampling_temperature=0.7,
    sampling_topk=50,
    compute_type="bfloat16"
)

# Decode and print response
output_ids = results[0].sequences_ids[0]
response = tokenizer.decode(output_ids, skip_special_tokens=True)
print(response)
```

**Requirements:**
```
ctranslate2
transformers
torch
huggingface_hub
```