| --- |
| license: mit |
| library_name: transformers |
| tags: |
| - vector-quantization |
| - vq2 |
| - moe |
| - qwen |
| - compression |
| - triton |
| - paged-streaming |
| pipeline_tag: text-generation |
| --- |
| |
| # ⚡ VQ2 & LEGION AI: Model Hub |
|
|
| Official repository for compiled binary containers of **VQ2** and **LEGION AI**. |
|
|
| * **GitHub Code & Docs**: [https://github.com/Unoalss/CompressAI](https://github.com/Unoalss/CompressAI) |
| * **Architecture**: Vector Quantization with Incoherence (RHT), Hessian LDLQ compensation, and Paged MoE Streaming. |
|
|
| --- |
|
|
| ## 📦 Hosted Models |
|
|
| ### 1. 🏆 `qwen35b_mixed_b2.vq2m` (Production Candidate MoE - 10.11 GB) |
| * **Base Model**: `Qwen3.8-35B-A3B-Distill` (35.5B parameters, 40 layers, 256 experts per layer). |
| * **Compression**: 86.0% smaller than BF16 (67 GB -> 10.11 GB / 9.41 GiB). |
| * **Quality**: **0.9974 cosine similarity** and **90.6% top-1 agreement** vs dense BF16. |
| * **Memory**: Runs in **< 1.25 GB active VRAM** and **~3.0 GB RAM** via LEGION Paged Streaming. |
| * **SHA-256**: `2ADB36E37F387842A521D2F3E07A29B77A7D9240997411372E0975586AEB8EAE` |
|
|
| ### 2. ⚡ `qwen3.5_2b_vq2_560mb.vq2r` (Ultra-compact 2B RDO - 567.9 MB) |
| * **Base Model**: `Qwen3.5-2B` (2 Billion parameters). |
| * **Compression**: 53.5% smaller than GGUF Q4_K_M (1221 MB -> 567 MB) and 84.2% smaller than BF16. |
| * **Memory**: Loads into GPU VRAM in **0.61 seconds**, occupying only **582 MiB** of active VRAM. |
| |
| ### 3. 🔑 `codebooks.pt` (4.47 MB) |
| * Shared 8-dimensional spherical codebooks for on-the-fly decompression. |
| |
| --- |
| |
| ## 📥 How to Download & Run |
| |
| Clone the engine from GitHub: |
| ```bash |
| git clone https://github.com/Unoalss/CompressAI.git |
| cd CompressAI |
| pip install -r requirements.txt |
| pip install -e . |
| ``` |
| |
| Download the models: |
| ```bash |
| # MoE 35B B2 Model (10 GB) |
| huggingface-cli download ArthurLss/CompressAI qwen35b_mixed_b2.vq2m --local-dir vq2_35b_out |
| |
| # 2B Model (567 MB) |
| huggingface-cli download ArthurLss/CompressAI qwen3.5_2b_vq2_560mb.vq2r --local-dir VQ2_MARCO_FINAL |
|
|
| # Shared Codebooks (4.47 MB) |
| huggingface-cli download ArthurLss/CompressAI codebooks.pt --local-dir vq2_out |
| ``` |
| |
| Launch the LEGION chatbot interface: |
| ```bash |
| # Windows |
| INICIAR_LEGION.bat |
|
|
| # Linux / Mac |
| python legion/server.py |
| ``` |
| Open `http://127.0.0.1:7777` in your browser! |
| |
