| --- |
| language: |
| - en |
| --- |
| |
| # Community Quantization Requests |
|
|
| This space is for requesting oQe builds. These quants utilize multi-stage calibration and Hessian-based error compensation to maintain logic stability, specifically tuned for Apple Silicon performance. |
|
|
| ### How to Request |
| Open a new Discussion for requests. To ensure a valid build, please include: |
|
|
| 1. Model Link: URL to the official Hugging Face repository. Note that I only process builds starting from original BF16 or FP16 source weights. |
| 2. Quantization Format: Specify if you need BF16 or FP16 quants. |
| * FP16 is generally recommended for M1/M2 series to utilize AMX units for faster prefill. |
| * BF16 is recommended for M3/M4 series with native support. |
| 3. Preferred Tiers: Specify the target bitrate (e.g., oQ5e, oQ4e) based on your available Unified Memory. |
|
|
| ### Guidelines |
| * Hardware: Builds are processed on a 192GB M2 Ultra. Models up to 70B parameters (standard dense) are supported. Anything significantly larger (100B+ or large MoE architectures) will exceed memory limits when loading source weights for calibration. |
| * Selection Criteria: Priority is given to base models and official instruct tunes. Experimental merges or low-epoch fine-tunes are generally excluded unless there is significant community interest. |
| * The Process: Every oQe build undergoes a 600-sample calibration pass. These are not one-pass streaming quants. |
|
|
| ### Technical Spec |
| All fulfilled requests are processed using the oMLX Enhanced Quantization process: |
| * Sensitivity Mapping: Calibration pass measures precision requirements per layer to prevent output drift. |
| * Hessian-Based Tuning: GPTQ-Hessian error compensation is applied during weight rounding. |
| * Precision Anchoring: Native BF16 for routing gates and FP16 for attention heads to maximize Apple Silicon AMX throughput. |
| * Logic Floor: The lm_head and critical early blocks are locked at 8-bit to ensure core reasoning stability. |
