Instructions to use LGxNDs/Geeked-Out-Quantization-Software with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use LGxNDs/Geeked-Out-Quantization-Software with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="LGxNDs/Geeked-Out-Quantization-Software",
	filename="Qwen3.6-GeekedOutAi-35B-A3B-BF16-IQ2_M-00001-of-00002.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use LGxNDs/Geeked-Out-Quantization-Software with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf LGxNDs/Geeked-Out-Quantization-Software:IQ2_M
# Run inference directly in the terminal:
llama-cli -hf LGxNDs/Geeked-Out-Quantization-Software:IQ2_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf LGxNDs/Geeked-Out-Quantization-Software:IQ2_M
# Run inference directly in the terminal:
llama-cli -hf LGxNDs/Geeked-Out-Quantization-Software:IQ2_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf LGxNDs/Geeked-Out-Quantization-Software:IQ2_M
# Run inference directly in the terminal:
./llama-cli -hf LGxNDs/Geeked-Out-Quantization-Software:IQ2_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf LGxNDs/Geeked-Out-Quantization-Software:IQ2_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf LGxNDs/Geeked-Out-Quantization-Software:IQ2_M

Use Docker

docker model run hf.co/LGxNDs/Geeked-Out-Quantization-Software:IQ2_M

LM Studio
Jan
Ollama
How to use LGxNDs/Geeked-Out-Quantization-Software with Ollama:
```
ollama run hf.co/LGxNDs/Geeked-Out-Quantization-Software:IQ2_M
```

Unsloth Studio new

How to use LGxNDs/Geeked-Out-Quantization-Software with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for LGxNDs/Geeked-Out-Quantization-Software to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for LGxNDs/Geeked-Out-Quantization-Software to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for LGxNDs/Geeked-Out-Quantization-Software to start chatting

Pi new

How to use LGxNDs/Geeked-Out-Quantization-Software with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf LGxNDs/Geeked-Out-Quantization-Software:IQ2_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "LGxNDs/Geeked-Out-Quantization-Software:IQ2_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use LGxNDs/Geeked-Out-Quantization-Software with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf LGxNDs/Geeked-Out-Quantization-Software:IQ2_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default LGxNDs/Geeked-Out-Quantization-Software:IQ2_M

Run Hermes

hermes

Docker Model Runner
How to use LGxNDs/Geeked-Out-Quantization-Software with Docker Model Runner:
```
docker model run hf.co/LGxNDs/Geeked-Out-Quantization-Software:IQ2_M
```

Lemonade

How to use LGxNDs/Geeked-Out-Quantization-Software with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull LGxNDs/Geeked-Out-Quantization-Software:IQ2_M

Run and chat with the model

lemonade run user.Geeked-Out-Quantization-Software-IQ2_M

List all available models

lemonade list

LGxNDs commited on 21 days ago

Commit

ffa8c49

verified ·

1 Parent(s): 9086a30

Create README.md

Browse files

Files changed (1) hide show

README.md +9 -36

README.md CHANGED Viewed

@@ -1,9 +1,7 @@
 ---
-comments: |
-  This model card describes the Qwen3.6 IQ2_M 2-bit quantized model repository.
 tags:
 - gguf
@@ -14,6 +12,8 @@ tags:
 - conversational
 ---
@@ -22,8 +22,6 @@ tags:
-## Overview
 This repository contains a **Qwen3.6** large language model series that has been quantized to **IQ2_M (Intelligent Quants) 2-bit precision** using the GGUF format. The model retains high-quality performance while significantly reducing memory footprint through advanced mixed-precision quantization techniques.
@@ -52,13 +50,13 @@ This repository contains a **Qwen3.6** large language model series that has been
 The **IQ2_M** quantization scheme is part of the Intelligent Quants (IQ) family developed for efficient model inference:
-- **Mixed Precision**: Different weights receive varying bit allocations based on their sensitivity and importance
 -
-- **Block-wise Quantization**: Optimized scaling factors applied across weight blocks
 -
-- **2-Bit Compression**: Achieves extreme low-bit precision while preserving critical model capabilities
 -
-- **Smart Allocation**: Critical parameters preserved in higher precision, less important weights packed into minimal bit formats
 -
@@ -78,42 +76,17 @@ This quantized model is designed for:
 ## Usage Instructions
-To load this model locally:
-```bash
-# Using llama.cpp
-llama-server --model Qwen3.6-GeekedOutAi-35B-A3B-BF16-IQ2_M.gguf
-# Using LM Studio or compatible tools
-Load the GGUF files from this repository
-```
 ## Technical Notes
-- The model is split into two parts (00001-of-00002 and 00002-of-00002) for efficient storage
--
 - IQ2_M quantization maintains conversational capability while achieving significant size reduction
 -
-- Compatible with llama.cpp and local inference frameworks
 -
 - Uses imatrix-based calibration for optimal quantization quality
--
-## License & Attribution
-This model card documents a community-contributed quantized version of Qwen3.6 optimized for practical deployment scenarios.

 ---
+title: Qwen3.6 IQ2_M 2-Bit Quantized Model
 tags:
 - gguf
 - conversational
+license: other
 ---
 This repository contains a **Qwen3.6** large language model series that has been quantized to **IQ2_M (Intelligent Quants) 2-bit precision** using the GGUF format. The model retains high-quality performance while significantly reducing memory footprint through advanced mixed-precision quantization techniques.
 The **IQ2_M** quantization scheme is part of the Intelligent Quants (IQ) family developed for efficient model inference:
+- Mixed precision - different weights receive varying bit allocations based on their sensitivity and importance
 -
+- Block-wise quantization with optimized scaling factors applied across weight blocks
 -
+- 2-bit compression achieving extreme low-bit precision while preserving critical model capabilities
 -
+- Smart allocation where critical parameters are preserved in higher precision while less important weights are packed into minimal bit formats
 -
 ## Usage Instructions
+To load this model locally using llama.cpp or compatible inference frameworks. The GGUF files are split into two parts for efficient storage (00001-of-00002 and 00002-of-00002).
 ## Technical Notes
 - IQ2_M quantization maintains conversational capability while achieving significant size reduction
 -
+- Compatible with llama.cpp, LM Studio, Jan, and other local inference frameworks
 -
 - Uses imatrix-based calibration for optimal quantization quality