| # Project: PrettyBird BCE Basic Coder 8B | |
| ## Project Overview | |
| This project involves fine-tuning the `Qwen/Qwen2.5-Coder-7B-Instruct` model using the Unsloth library. The goal was to produce a high-performance coding assistant. Following the fine-tuning process, the model was converted into GGUF format with multiple quantization levels to optimize for various deployment scenarios. | |
| ## Methodology | |
| - **Fine-Tuning Framework**: Unsloth (LoRA adapters) | |
| - **Base Model**: Qwen/Qwen2.5-Coder-7B-Instruct | |
| - **Inference Engine**: Ollama | |
| - **Hardware**: NVIDIA A100 GPU | |
| ## Quantitative Results | |
| The models were benchmarked for inference speed (Tokens Per Second) using the `api/generate` raw mode to ensure consistent evaluation. | |
| | Model Tag | Quantization | Mean TPS | Speedup vs Baseline | | |
| | :--- | :--- | :--- | :--- | | |
| | f16 | Full Precision | ~97.0 | 1.0x | | |
| | q8_0 | 8-bit | ~141.0 | 1.45x | | |
| | q5_k_m | 5-bit | ~151.0 | 1.56x | | |
| | q4_k_m | 4-bit | ~161.0 | 1.66x | | |
| | q2_k | 2-bit | ~140.0 | 1.44x | | |
| ## Performance Analysis | |
| - **Sweet Spot**: The `q4_k_m` model demonstrated the highest throughput, achieving approximately 161 TPS. It represents the optimal balance between speed and precision, offering a ~1.7x speedup over the full-precision baseline. | |
| - **The 2-bit Anomaly**: The `q2_k` model, despite being the most compressed, performed slower than the 4-bit and 5-bit variants (~140 TPS). This counter-intuitive result is attributed to the computational overhead required to dequantize highly compressed weights on-the-fly, which creates a bottleneck on high-performance hardware like the NVIDIA A100. | |
| ## Recommendations | |
| For production deployment, the **`q4_k_m`** model is recommended as the primary candidate due to its superior throughput and efficient memory usage. The **`q5_k_m`** model serves as a high-fidelity alternative if slightly higher reasoning precision is required. | |