| --- |
| title: HuatuoGPT CPU-Compatible Demo |
| emoji: 🧠 |
| colorFrom: blue |
| colorTo: indigo |
| sdk: gradio |
| sdk_version: "4.27.0" |
| app_file: app.py |
| pinned: false |
| --- |
| |
| # ⚙️ Model Selection Rationale: Why I Didn't Use HuatuoGPT-o1-8B |
|
|
| ## 🚫 Why I Did Not Use `FreedomIntelligence/HuatuoGPT-o1-8B` |
|
|
| My original implementation was based on `HuatuoGPT-o1-8B`, a large-scale, instruction-tuned model optimized for medical reasoning tasks. However, it has several hardware and software requirements that make it infeasible to deploy on CPU-only environments such as Hugging Face free-tier Spaces: |
|
|
| - The model contains **8 billion parameters**, making it extremely resource-intensive. |
| - It requires **4-bit quantization** via the `bitsandbytes` library to fit into memory-constrained environments. |
| - `bitsandbytes` depends on a **CUDA-enabled GPU**, which is not available in Spaces without an upgraded hardware tier. |
| - The model uses `torch_dtype=torch.bfloat16` and `device_map="auto"`, both of which assume GPU availability. |
|
|
| Because the Hugging Face free Space environment does **not provide GPU acceleration**, attempting to load this model leads to runtime errors such as: |
|
|
| ``` |
| CUDA is required but not available for bitsandbytes. |
| ``` |
|
|
| This makes `HuatuoGPT-o1-8B` incompatible with CPU-only deployments in its current form. |
|
|
| ## ✅ Why I Switched to a Lightweight, CPU-Compatible Model |
|
|
| To ensure compatibility with CPU-only inference and to preserve interactive response times, I replaced the original model with a significantly smaller, general-purpose model: |
|
|
| ```python |
| model_id = "microsoft/DialoGPT-small" |
| ``` |
|
|
| I selected this model because: |
|
|
| - It is **lightweight (~355M parameters)** and can be loaded entirely into CPU memory without quantization. |
| - It does **not require `bitsandbytes`**, CUDA, or other GPU-dependent optimizations. |
| - It supports **conversational response generation**, making it suitable for demo purposes even in non-medical contexts. |
| - It enables full deployment on **Hugging Face Spaces (CPU runtime)** with minimal latency and no memory constraints. |
|
|
| Although it is not domain-specialized for medical QA, it serves as a practical baseline for deploying the interface and application logic in a constrained compute environment. |
|
|
| ## 📝 Summary |
|
|
| | Characteristic | `HuatuoGPT-o1-8B` (original) | `DialoGPT-small` (deployed) | |
| |-------------------------------|----------------------------------|----------------------------------| |
| | Parameter Count | ~8B | ~355M | |
| | Intended Domain | Medical Q&A | General conversational AI | |
| | GPU Required | ✅ Yes | ❌ No | |
| | Quantization Needed | ✅ 4-bit (bitsandbytes) | ❌ None | |
| | CUDA Dependency | ✅ Yes | ❌ No | |
| | CPU Runtime Compatibility | ❌ Not feasible | ✅ Fully compatible | |
| | Hugging Face Free Tier Ready | ❌ No | ✅ Yes | |
|
|
