---
title: HuatuoGPT CPU-Compatible Demo
emoji: 🧠
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: "4.27.0"
app_file: app.py
pinned: false
---

# ⚙️ Model Selection Rationale: Why I Didn't Use HuatuoGPT-o1-8B

## 🚫 Why I Did Not Use `FreedomIntelligence/HuatuoGPT-o1-8B`

My original implementation was based on `HuatuoGPT-o1-8B`, a large-scale, instruction-tuned model optimized for medical reasoning tasks. However, it has several hardware and software requirements that make it infeasible to deploy on CPU-only environments such as Hugging Face free-tier Spaces:

- The model contains **8 billion parameters**, making it extremely resource-intensive.
- It requires **4-bit quantization** via the `bitsandbytes` library to fit into memory-constrained environments.
- `bitsandbytes` depends on a **CUDA-enabled GPU**, which is not available in Spaces without an upgraded hardware tier.
- The model uses `torch_dtype=torch.bfloat16` and `device_map="auto"`, both of which assume GPU availability.

Because the Hugging Face free Space environment does **not provide GPU acceleration**, attempting to load this model leads to runtime errors such as:

```
CUDA is required but not available for bitsandbytes.
```

This makes `HuatuoGPT-o1-8B` incompatible with CPU-only deployments in its current form.

## ✅ Why I Switched to a Lightweight, CPU-Compatible Model

To ensure compatibility with CPU-only inference and to preserve interactive response times, I replaced the original model with a significantly smaller, general-purpose model:

```python
model_id = "microsoft/DialoGPT-small"
```

I selected this model because:

- It is **lightweight (~355M parameters)** and can be loaded entirely into CPU memory without quantization.
- It does **not require `bitsandbytes`**, CUDA, or other GPU-dependent optimizations.
- It supports **conversational response generation**, making it suitable for demo purposes even in non-medical contexts.
- It enables full deployment on **Hugging Face Spaces (CPU runtime)** with minimal latency and no memory constraints.

Although it is not domain-specialized for medical QA, it serves as a practical baseline for deploying the interface and application logic in a constrained compute environment.

## 📝 Summary

| Characteristic                 | `HuatuoGPT-o1-8B` (original)     | `DialoGPT-small` (deployed)      |
|-------------------------------|----------------------------------|----------------------------------|
| Parameter Count               | ~8B                              | ~355M                            |
| Intended Domain               | Medical Q&A                      | General conversational AI        |
| GPU Required                  | ✅ Yes                            | ❌ No                            |
| Quantization Needed           | ✅ 4-bit (bitsandbytes)          | ❌ None                          |
| CUDA Dependency               | ✅ Yes                            | ❌ No                            |
| CPU Runtime Compatibility     | ❌ Not feasible                  | ✅ Fully compatible              |
| Hugging Face Free Tier Ready  | ❌ No                             | ✅ Yes                           |