Spaces:
Sleeping
Sleeping
Commit History
-
Add GPU estimator, DDG search, and cancel support
4ce42e8 -
Load vLLM from local snapshot to support default subfolders
0e2f6c4 -
Default to Gemma router and limit prefetch
6fb2aa6 -
Fix prefetch init order
2790442 -
Parallelize AWQ model prefetching
e829b15 -
Disable vLLM by default on MIG devices
63c8de5 -
Remove unsupported vLLM device kwarg
f886036 -
Adjust CUDA handling and set explicit device for vLLM
8fd14bc -
Fix UnboundLocalError: remove duplicate torch import
5ee455a
-
Improve vLLM device detection: force torch CUDA reinit
75aac04 -
Fix remaining pipeline calls to use transformers_repo
34ee4d1 -
Fix vLLM device detection and AWQ model loading
41f50c5 -
Fix AWQ model loading: point to default/ subfolder and fix tokenizer loading
a76dbfd -
Add test scripts for AWQ models on ZeroGPU Space
dd11bd9 -
Update Qwen model repo to AWQ quantized version
27234fe -
Update Qwen model to use AWQ quantized version
d02a9d8 -
Update Gemma model to use AWQ quantized version
- Change repo_id from router-gemma3-merged to router-gemma3-merged-awq - vLLM will auto-detect AWQ quantization from quantization_config.json - Weights in 'default' subfolder are handled automatically by vLLMb36a0b0 -
Lower Gemma AWQ group size to 16
f8c20fd -
Use model-specific AWQ configs (Gemma group_size=64)
98c0d4d -
Remove processor arg from oneshot to avoid tokenizer conflict
4b47dea -
Pass processor to oneshot for text-only models
56105b6 -
Fix processor error: pass tokenizer explicitly for text-only models
- Load tokenizer using AutoTokenizer.from_pretrained() - Pass tokenizer parameter to oneshot() to avoid processor initialization errors - For text-only LLMs, tokenizer serves as the processor - Fixes RuntimeError about processor initialization for text models5bf2e9f -
Add stage='default' parameter to oneshot() call
- Add stage parameter as mentioned in instructions - All parameters now correctly map to HfArgumentParser dataclasses - Recipe.from_modifiers() and Dataset.from_dict() already in place5bceece -
Fix oneshot() API: use Recipe and Dataset objects
- Import Recipe from llmcompressor.recipe and Dataset from datasets - Convert modifiers list to Recipe object using Recipe.from_modifiers() - Convert calibration_texts to Hugging Face Dataset using Dataset.from_dict() - Use use_auth_token=True instead of token parameter - All kwargs now map to ModelArguments, DatasetArguments, RecipeArguments - Fixes HfArgumentParser 'Some keys are not used' error671d7f9 -
Fix oneshot() API: use correct parameter names from documentation
- Change 'model_id' to 'model' - Change 'modifiers' to 'recipe' (list of modifiers) - Change 'calibration_data' to 'dataset' (list of strings) - Remove 'token' parameter (not needed) - Add optional 'max_seq_length' and 'num_calibration_samples' - Based on official llm-compressor documentation35d8225 -
Try alternative oneshot() API parameter names
- First try: model_id, modifiers, dataset, token - Fallback: minimal parameters (model_id, output_dir) - Need to verify correct API from llmcompressor documentatione9f4b24 -
Fix oneshot() API: use correct parameter names
- Change 'model' to 'model_id' - Change 'modifiers' to 'recipe_modifiers' - Change 'calibration_data' to 'recipe_dataset' - Change 'token' to 'hf_token' - Fixes HfArgumentParser unrecognized keys error7e31310 -
Fix cell 7: convert markdown note to Python comment
4001f22 -
Remove duplicate build_awq_modifier_config - keep existing correct version
- Remove duplicate helper function from cell 7 - Keep existing build_awq_modifier_config in cell 9 that uses QuantizationScheme objects - Existing function correctly returns config_groups and ignore separately - Quantization function already uses it correctly with AWQModifier5bf02e9 -
Add build_awq_modifier_config helper using QuantizationScheme objects
- Create helper function that uses QuantizationScheme/QuantizationArgs objects - Properly constructs QuantizationConfig with config_groups - Falls back to dict-based config if QuantizationScheme not available - Fixes ValidationError by using proper object structure instead of plain dicts - Update quantization function to use the helpercf9ed91 -
Fix quantization_config structure: use correct AWQ format
- Use dict-based config with config_groups.group_0 structure - Include targets, weights (num_bits, group_size, zero_point), etc. - Remove fallback approaches - use correct structure directly - Fixes ValidationError for AWQModifier3f08592 -
Fix modifiers initialization: ensure it's always defined
- Initialize modifiers and quant_config to None - Ensure modifiers is always created before use - Add final fallback to default AWQModifier if all config approaches failf3114ba -
Fix BaseQuantizationConfig import: add fallback approaches
- Try BaseQuantizationConfig import first - Fallback to dict-based config_groups if import fails - Fallback to default AWQModifier() if config creation fails - Handles different compressed_tensors API versionsa49281c -
Add local test script for quantization notebook validation
- Tests imports and notebook structure without requiring GPU - Handles missing llmcompressor gracefully (expected locally) - Validates configuration, calibration data, disk space functions - Ready for Colab deployment011c926 -
Fix QuantizationConfig: use config_groups with BaseQuantizationConfig
- QuantizationConfig requires config_groups dict, not direct parameters - Use BaseQuantizationConfig for default group with num_bits=4 - Fixes missing config_groups field errorecf6a69 -
Fix AWQModifier: use quantization_config with num_bits
- AWQModifier requires quantization_config parameter - Create QuantizationConfig with num_bits=4, group_size=128 - Fixes assertion error about num_bits configuration022b2da -
Add note about restarting kernel if AWQModifier errors occur
- Users may be running cached old code - Add reminder to restart kernel and rerun cells - AWQModifier() works without parameters33a1d2e -
Simplify AWQModifier usage - remove try/except wrapper
- AWQModifier() works without parameters - Remove unnecessary try/except that might cause confusion - Use default 4-bit AWQ settingse08f8c4 -
Fix AWQModifier parameters - use default configuration
- AWQModifier doesn't accept w_bit, q_group_size parameters - Use AWQModifier() with default settings instead - Default is 4-bit AWQ which matches our requirementscef8ecd -
Fix delete_revisions import with fallback cache cleanup
- Handle missing delete_revisions function gracefully - Add manual cache directory deletion as fallback - Improve error handling for cache cleanup7a2a590 -
Fix delete_revisions import - use fallback cache cleanup method
- delete_revisions may not be available in all huggingface_hub versions - Add fallback to manual cache directory deletion - Use shutil.rmtree as alternative when delete_revisions unavailable - Add helpful manual cleanup instructions4be72e0 -
Fix AWQModifier import path: use modifiers.awq instead of modifiers.quantization
- Correct import: from llmcompressor.modifiers.awq import AWQModifier - Wrong import: from llmcompressor.modifiers.quantization import AWQModifier - Updated all files with correct import path - Added upgrade suggestion in error messagef0033ab -
Fix LLM Compressor package name: llmcompressor (no hyphen)
- Change from llm-compressor to llmcompressor (correct package name) - Add fallback installation from GitHub if PyPI fails - Add import error handling with helpful error messages - Add troubleshooting tips in error messages2326498 -
Remove duplicate LLM Compressor section - now primary method
d4bc333 -
Replace AutoAWQ with LLM Compressor (vLLM native) in Colab notebook
- Use llm-compressor instead of autoawq for quantization - LLM Compressor is vLLM's native tool with better integration - Simplified quantization pipeline using oneshot() function - Updated verification to prefer vLLM over Transformers - Better compatibility with vLLM inference engineae07f77 -
Add advanced vLLM and LLM Compressor optimizations
- Enable FP8 KV cache for 50% memory reduction (compatible with AWQ) - Add FP8 quantization support (fallback if available) - Optimize SamplingParams for router plan generation - Add LLM Compressor alternative quantization method to notebook - Document advanced features: FP8, pruning, combined modifiers - Improve memory efficiency and enable longer contexts808203f -
Add disk space cleanup after quantization in Colab notebook
- Delete source model from Hugging Face cache after quantization - Free GPU memory and force garbage collection - Check disk space before/after operations - Add warnings for low disk space - Critical for Colab's limited disk space24107f3 -
Fix linter error: use %pip instead of !pip in Colab notebook
2dff966 -
Add Colab notebook for AWQ quantization of router models
- Complete quantization pipeline for Gemma3 and Qwen3 models - Uses AutoAWQ for reliable quantization - Includes verification and upload steps - Documentation with troubleshooting guidea79bc8f -
Clarify LLM Compressor optional status - vLLM has native AWQ support
- LLM Compressor is only needed for quantizing models, not loading pre-quantized AWQ - vLLM can load AWQ models natively without llm-compressor - Update import logic to try both package names - Comment out llmcompressor from requirements (optional dependency)b2bf767