README.md

metadata

title: HuatuoGPT CPU-Compatible Demo
emoji: 🧠
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: 4.27.0
app_file: app.py
pinned: false

⚙️ Model Selection Rationale: Why I Didn't Use HuatuoGPT-o1-8B

🚫 Why I Did Not Use FreedomIntelligence/HuatuoGPT-o1-8B

My original implementation was based on HuatuoGPT-o1-8B, a large-scale, instruction-tuned model optimized for medical reasoning tasks. However, it has several hardware and software requirements that make it infeasible to deploy on CPU-only environments such as Hugging Face free-tier Spaces:

• The model contains 8 billion parameters, making it extremely resource-intensive.
• It requires 4-bit quantization via the bitsandbytes library to fit into memory-constrained environments.
• bitsandbytes depends on a CUDA-enabled GPU, which is not available in Spaces without an upgraded hardware tier.
• The model uses torch_dtype=torch.bfloat16 and device_map="auto", both of which assume GPU availability.

Because the Hugging Face free Space environment does not provide GPU acceleration, attempting to load this model leads to runtime errors such as:

CUDA is required but not available for bitsandbytes.

This makes HuatuoGPT-o1-8B incompatible with CPU-only deployments in its current form.

✅ Why I Switched to a Lightweight, CPU-Compatible Model

To ensure compatibility with CPU-only inference and to preserve interactive response times, I replaced the original model with a significantly smaller, general-purpose model:

model_id = "microsoft/DialoGPT-small"

I selected this model because:

• It is lightweight (~355M parameters) and can be loaded entirely into CPU memory without quantization.
• It does not require bitsandbytes, CUDA, or other GPU-dependent optimizations.
• It supports conversational response generation, making it suitable for demo purposes even in non-medical contexts.
• It enables full deployment on Hugging Face Spaces (CPU runtime) with minimal latency and no memory constraints.

Although it is not domain-specialized for medical QA, it serves as a practical baseline for deploying the interface and application logic in a constrained compute environment.

📝 Summary

Characteristic	HuatuoGPT-o1-8B (original)	DialoGPT-small (deployed)
Parameter Count	~8B	~355M
Intended Domain	Medical Q&A	General conversational AI
GPU Required	✅ Yes	❌ No
Quantization Needed	✅ 4-bit (bitsandbytes)	❌ None
CUDA Dependency	✅ Yes	❌ No
CPU Runtime Compatibility	❌ Not feasible	✅ Fully compatible
Hugging Face Free Tier Ready	❌ No	✅ Yes