Upload Qwen3-0.6B-Comparison.md with huggingface_hub

Qwen3-0.6B-Comparison.md

@@ -0,0 +1,168 @@
+# Qwen3-0.6B Quantized Models Comparison
+
+## Summary
+
+Three quantized versions of Qwen3-0.6B were created on March 21, 2026 and uploaded to Hugging Face at:
+**https://huggingface.co/Bopalv/Qwen3-0.6B-quantized**
+
+## Models Overview
+
+| Model | Format | Quantization | File Size | Status |
+|-------|--------|--------------|-----------|--------|
+| GGUF Q4_K_M | GGUF | 4-bit K-quant | 462 MB | ✅ Complete |
+| GPTQ-Int4 | Safetensors | 4-bit GPTQ | 517 MB | ✅ Complete |
+| GPTQ-Int8 | Safetensors | 8-bit GPTQ | 727 MB | ✅ Complete |
+
+## Technical Specifications
+
+### Common Properties
+- **Base Model**: Qwen3-0.6B
+- **Parameters**: 0.6B (490M)
+- **Architecture**: Qwen3ForCausalLM
+- **Hidden Size**: 1024
+- **Layers**: 28
+- **Attention Heads**: 16
+- **KV Heads**: 8
+- **Max Context**: 40,960 tokens
+- **Vocab Size**: 151,936
+
+### Quantization Details
+
+| Model | Bits | Group Size | Symmetric | Quantizer | Pack Dtype |
+|-------|------|------------|-----------|-----------|------------|
+| GGUF Q4_K_M | 4 | N/A | Yes | llama.cpp | N/A |
+| GPTQ-Int4 | 4 | 128 | Yes | gptqmodel 4.0.0 | int32 |
+| GPTQ-Int8 | 8 | 128 | Yes | gptqmodel 2.2.0 | int32 |
+
+## File Size Comparison
+
+```
+GGUF Q4_K_M  ████████████████████ 462 MB (Smallest)
+GPTQ-Int4    ████████████████████████ 517 MB (+12%)
+GPTQ-Int8    ████████████████████████████████████ 727 MB (+57%)
+```
+
+## Theoretical Performance Analysis
+
+### Memory Usage
+- **GGUF Q4_K_M**: ~462 MB loaded
+- **GPTQ-Int4**: ~517 MB loaded + overhead
+- **GPTQ-Int8**: ~727 MB loaded + overhead
+
+### Expected Quality (Lower bits = More compression, Potentially lower quality)
+1. **GPTQ-Int8**: Best quality (8-bit precision)
+2. **GPTQ-Int4**: Good quality (4-bit with group quantization)
+3. **GGUF Q4_K_M**: Good quality (4-bit K-quant, optimized for llama.cpp)
+
+### Expected Speed (CPU-based)
+1. **GGUF Q4_K_M**: Fastest (optimized for llama.cpp, smallest size)
+2. **GPTQ-Int4**: Medium (requires dequantization overhead)
+3. **GPTQ-Int8**: Slowest (largest size, more computation)
+
+## Compatibility
+
+### GGUF Q4_K_M
+- ✅ llama.cpp
+- ✅ prima.cpp (if Qwen3 architecture is supported)
+- ✅ Ollama
+- ✅ LM Studio
+- ✅ Text Generation WebUI
+
+### GPTQ-Int4 & GPTQ-Int8
+- ✅ HuggingFace Transformers
+- ✅ AutoGPTQ
+- ✅ vLLM
+- ✅ Text Generation WebUI
+- ⚠️ llama.cpp (requires conversion)
+
+## Usage Recommendations
+
+### For CPU-only systems
+**Recommended: GGUF Q4_K_M**
+- Smallest file size
+- Optimized for CPU inference
+- Fastest loading time
+- Compatible with llama.cpp ecosystem
+
+### For GPU systems
+**Recommended: GPTQ-Int4**
+- Good balance of quality and size
+- Works with AutoGPTQ and Transformers
+- Faster than GPTQ-Int8
+- Better quality than GGUF on GPU
+
+### For Maximum Quality
+**Recommended: GPTQ-Int8**
+- Highest precision (8-bit)
+- Best output quality
+- Requires more memory
+- Slower inference
+
+## Benchmarking Notes
+
+The model-efficiency tool from `bopalvelut-prog/model-efficiency` requires:
+1. Ollama running on port 11434, OR
+2. An OpenAI-compatible API server
+
+To benchmark these models:
+
+### Option 1: Using Ollama
+```bash
+# Install Ollama
+curl -fsSL https://ollama.com/install.sh | sh
+
+# Import GGUF model
+ollama create qwen3-0.6b-gguf -f Modelfile
+
+# Run benchmark
+cd model-efficiency
+python model_efficiency_comparator.py -p "Your test prompt"
+```
+
+### Option 2: Using prima.cpp
+```bash
+# Start server with GGUF model
+/home/ma/prima.cpp/llama-server \
+  -m /home/ma/models/Qwen3-0.6B-GGUF/Qwen3-0.6B.Q4_K_M.gguf \
+  --port 8080
+
+# Test with curl
+curl http://localhost:8080/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{"messages":[{"role":"user","content":"Hello"}],"max_tokens":50}'
+```
+
+### Option 3: Using Transformers (for GPTQ models)
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import time
+
+model_path = "/home/ma/models/Qwen3-0.6B-GPTQ-Int4"
+tokenizer = AutoTokenizer.from_pretrained(model_path)
+model = AutoModelForCausalLM.from_pretrained(model_path, device_map="auto")
+
+inputs = tokenizer("Hello", return_tensors="pt")
+start = time.time()
+outputs = model.generate(**inputs, max_new_tokens=100)
+end = time.time()
+
+print(f"Time: {end-start:.2f}s")
+```
+
+## Storage Requirements
+
+| Model | File Size | Disk Space Needed | RAM Needed (Est.) |
+|-------|-----------|-------------------|-------------------|
+| GGUF Q4_K_M | 462 MB | 462 MB | ~600 MB |
+| GPTQ-Int4 | 517 MB | 517 MB | ~700 MB |
+| GPTQ-Int8 | 727 MB | 727 MB | ~900 MB |
+| **All Models** | **1.7 GB** | **1.7 GB** | **~2.2 GB** |
+
+## Conclusion
+
+- **Best for CPU/Embedded**: GGUF Q4_K_M (smallest, fastest)
+- **Best for GPU**: GPTQ-Int4 (balanced)
+- **Best Quality**: GPTQ-Int8 (highest precision)
+
+All models are available at:
+**https://huggingface.co/Bopalv/Qwen3-0.6B-quantized**