README.md

---
license: other
language:
  - en
  - fa
tags:
  - gguf
  - safetensors
  - local-llm
  - quantized
  - rtx-3060
  - consumer-hardware
  - text-generation
  - code
  - agent
  - agents
  - rag
  - embeddings
  - reranker
  - text-to-speech
  - speech-to-text
  - multimodal
  - vision
  - voice-cloning
  - persian
  - farsi
  - uncensored
  - abliterated
pipeline_tag: text-generation
---

<p align="center">
  <img src="https://img.shields.io/badge/README-AI_Generated-FF6B6B?style=flat-square&logo=openai&logoColor=white" alt="AI Generated">
</p>

> ⚠️ **Note:** This README was generated and is maintained by AI.

<div align="center">

# 🧠 Local AI Toolkit

### A Curated Collection of Locally-Runnable AI Models
**Optimized for RTX 3060 12GB VRAM + 32GB RAM**

<p>
  <img src="https://img.shields.io/badge/GPU-RTX_3060_12GB-76B900?style=for-the-badge&logo=nvidia&logoColor=white" alt="GPU">
  <img src="https://img.shields.io/badge/RAM-32GB_DDR4-4A90D9?style=for-the-badge&logo=memory&logoColor=white" alt="RAM">
  <img src="https://img.shields.io/badge/Models-37+-FF6B6B?style=for-the-badge&logo=huggingface&logoColor=white" alt="Models">
  <img src="https://img.shields.io/badge/Categories-8-F59E0B?style=for-the-badge&logo=databricks&logoColor=white" alt="Categories">
  <img src="https://img.shields.io/badge/Format-GGUF_&_SafeTensors-8B5CF6?style=for-the-badge&logo=archive&logoColor=white" alt="Format">
</p>

<p>
  <img src="https://img.shields.io/badge/🤖 LLMs-✅-success" alt="LLMs">
  <img src="https://img.shields.io/badge/💻 Coding-✅-success" alt="Coding">
  <img src="https://img.shields.io/badge/🔗 RAG-✅-success" alt="RAG">
  <img src="https://img.shields.io/badge/🎙️ TTS-✅-success" alt="TTS">
  <img src="https://img.shields.io/badge/🎤 STT-✅-success" alt="STT">
  <img src="https://img.shields.io/badge/👁️ Multimodal-✅-success" alt="Multimodal">
  <img src="https://img.shields.io/badge/🔗 Agents-✅-success" alt="Agents">
  <img src="https://img.shields.io/badge/🖼️ Image_Gen-🔜-yellow" alt="Image Gen">
  <img src="https://img.shields.io/badge/🎬 Video_Gen-🔜-yellow" alt="Video Gen">
</p>

<i>AI-assisted model selection · Quantized for consumer hardware · A suggested starting point</i>

</div>

---

## 📖 About This Toolkit

This repository is a **suggested toolkit** of open-source AI models assembled to run entirely locally on consumer-grade hardware — specifically an **NVIDIA RTX 3060 (12GB VRAM) with 32GB system RAM**. Each model was chosen to fit within these constraints while offering strong output quality for the VRAM budget.

The selection process was **AI-assisted** — I consulted several AI assistants to weigh benchmarks, quantization trade-offs, and model capabilities, then settled on the common recommendations you see here. This is **not a definitively optimal combination**: better models may already exist, and stronger ones are released regularly. Treat it as a well-reasoned starting point rather than a final answer — your own testing on your own hardware is the real benchmark.

### 🔄 Updates & Longevity

This toolkit is **periodically refreshed** as stronger or more efficient models are released. When a model gets clearly outclassed, it gets swapped out — so the lineup you see today may differ from a few months down the line.

### 🎯 Design Philosophy

Each category follows a dual-tier architecture. Both tiers target the **same 12GB VRAM + 32GB RAM machine** — the difference is *how* that budget is used:

| Tier | Purpose | What It Means |
|------|---------|---------------|
| 🏆 **Quality** | Best achievable quality | The highest-quality model that can still run within the 12GB VRAM + 32GB RAM budget, using **GPU+CPU offload** (some layers spill into system RAM). Slower, but maximizes output quality |
| ⚡ **Speed** | Fastest, no offload | A capable model that fits **entirely within 12GB VRAM** — leaving room for context too — so it runs fully on-GPU with no CPU offload overhead. Prioritizes throughput and low latency |

> **Note on Coding models:** The Coding Agents category includes two quality-tier models (Architect & Executor + Detective & Debugger). The "speed" variants in that category, while labeled for speed, are still powerful enough to serve as competent rapid-response code assistants — they simply trade some reasoning depth for significantly faster token generation.

### ⚠️ Work In Progress

This toolkit is **not yet complete**. Notably, **Image Generation** and **Video Generation** models are still missing and will be added in future updates. The current collection focuses on text, speech, and multimodal understanding capabilities. Stay tuned for expansions.

---

## 📂 Repository Structure & Category Guide

```
local-ai-toolkit/
├── 1-Universal_Foundation_LLMs/      🧠 General-purpose language models
├── 2-Coding_Agents/                  💻 Specialized code generation & debugging
├── 3-Agent_Orchestration_and_Routing/ 🔗 Multi-agent system coordination
├── 4-Document_Analysis_and_RAG/      📄 Embeddings, rerankers, and retrieval
├── 5-TTS/                            🎙️ Text-to-speech synthesis
├── 6-STT/                            🎤 Speech-to-text recognition
├── 7-Multimodal/                     👁️ Vision + language understanding
└── 8-Uncensored_Models/              🔓 Unfiltered model variants
```

---

## 1️⃣ Universal Foundation LLMs

<div align="center">
<i>The backbone of any local AI setup — general-purpose models for conversation, analysis, writing, and reasoning.</i>
</div>

These are your **go-to models** for everyday tasks: answering questions, drafting emails, summarizing documents, brainstorming ideas, creative writing, and general knowledge retrieval. They serve as the foundation upon which more specialized workflows are built.

| Tier | Model | Size | Quantization | Key Strengths |
|------|-------|------|-------------|---------------|
| 🏆 Quality | [Qwen 3.6-35B-A3B](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF) | 35B (MoE, 3B active) | Q4_K_XL | Mixture-of-Experts with only 3B active params — exceptional quality-to-VRAM ratio. Strong multilingual support, excellent reasoning |
| ⚡ Speed | [Qwen 3.5-9B](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF) | 9B | Q5_K_M | Blazing fast inference, still remarkably capable for its size. Perfect for quick Q&A, chat, and real-time interactions |

**Why these models?** The Qwen 3.x series represents a significant leap in efficient architecture. The 35B-A3B MoE variant is particularly special — it gives you the reasoning depth of a much larger model while only activating 3 billion parameters at a time, making it one of the most VRAM-efficient high-quality models available. The 9B speed variant offers near-instant responses for situations where latency matters more than depth.

<details>
<summary>📁 Folder Structure</summary>

```
1-Universal_Foundation_LLMs/
├── Quality/
│   └── Qwen3.6-35B-A3B-UD-Q4_K_XL/
│       ├── Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf
│       └── mmproj-BF16.gguf
└── Speed/
    └── Qwen3.5-9B-Q5_K_M/
        ├── Qwen3.5-9B-Q5_K_M.gguf
        └── mmproj-BF16.gguf
```
</details>

---

## 2️⃣ Coding Agents

<div align="center">
<i>Specialized models for writing, debugging, and architecting code — the core of any AI-assisted development workflow.</i>
</div>

This category contains models specifically fine-tuned for software engineering tasks. They are organized into five distinct roles that mirror a professional development team structure, from high-level architecture down to real-time inline suggestions.

| Role | Tier | Model | Size | Quantization | Purpose |
|------|------|-------|------|-------------|---------|
| 🏗️ Architect & Executor | 🏆 Quality | [Qwen3-Coder-30B-A3B](https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF) | 30B (MoE, 3B active) | Q5_K_M | Plans architecture, writes full implementations, handles complex codebases |
| 🔍 Detective & Debugger | 🏆 Quality | [DeepSeek-R1-Distill-Qwen-32B](https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-32B-GGUF) | 32B | Q4_K_M | Deep chain-of-thought reasoning for finding and fixing bugs |
| ⚡ Autocomplete | — | [Qwen2.5-Coder-7B](https://huggingface.co/EasierAI/Qwen-2.5-Coder-7B) | 7B | Q5_K_M | Ultra-fast inline code completion for IDE integration |
| 🏗️ Architect (Fast) | ⚡ Speed | [Qwen2.5-Coder-14B](https://huggingface.co/bartowski/Qwen2.5-Coder-14B-GGUF) | 14B | Q4_K_M | Quick code generation and architectural guidance |
| 🔍 Debugger (Fast) | ⚡ Speed | [DeepSeek-R1-Distill-Qwen-14B](https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-14B-GGUF) | 14B | Q4_K_M | Rapid debugging with reasoning capabilities |

**Why this structure?** Modern AI-assisted coding works best when you use **specialized models for specialized tasks**. The Qwen3-Coder excels at writing new code and understanding project structure, while the DeepSeek-R1 distill variant uses its chain-of-thought training to methodically trace through bugs. The 7B autocomplete model is lightweight enough to run alongside your IDE for real-time suggestions without competing for VRAM with your main coding model.

> **Pro tip:** Use the "speed" variants for quick prototyping and test-driven development cycles where you need rapid iteration. They're still high-quality coders — they just generate tokens faster and use less memory, leaving headroom for other tools in your pipeline.

<details>
<summary>📁 Folder Structure</summary>

```
2-Coding_Agents/
├── 1-Architect_and_Executor/
│   └── Qwen3-Coder-30B-A3B-Instruct-Q5_K_M/
│       └── Qwen3-Coder-30B-A3B-Instruct-Q5_K_M.gguf
├── 2-Detective_and_Debugger/
│   └── DeepSeek-R1-Distill-Qwen-32B-Q4_K_M/
│       └── DeepSeek-R1-Distill-Qwen-32B-Q4_K_M.gguf
├── 3-Inline_Autocomplete/
│   └── qwen2.5-coder-7b-instruct-q5_k_m/
│       └── qwen2.5-coder-7b-instruct-q5_k_m.gguf
├── 4-Architect_and_Executor_speed/
│   └── Qwen2.5_Coder_14B_Instruct_Q4_K_M/
│       └── Qwen2.5_Coder_14B_Instruct_Q4_K_M.gguf
└── 5-Detective_and_Debugger_speed/
    └── DeepSeek_R1_Distill_Qwen_14B_Q4_K_M/
        └── DeepSeek_R1_Distill_Qwen_14B_Q4_K_M.gguf
```
</details>

---

## 3️⃣ Agent Orchestration and Routing

<div align="center">
<i>The nervous system of multi-agent AI systems — models that coordinate, route, guard, and execute complex agent workflows.</i>
</div>

When you move beyond single-model interactions into **multi-agent architectures** (where different specialized models collaborate on complex tasks), you need orchestrators that can plan, delegate, monitor, and validate. This category provides the complete infrastructure for building sophisticated agentic systems locally.

### 🎯 Orchestrators & Managers

The "brains" of your agent network. These models decide which worker to call, when to call it, and how to synthesize results.

| Model | Size | Quantization | Role |
|-------|------|-------------|------|
| [Agents-A1 35B MoE](https://huggingface.co/Chungulus/Agents-A1-IQ4_XS-imatrix-gguf-fable5-calibrated) | 35B (MoE) | IQ4_XS | **CEO Agent** — high-level autonomous planning and task decomposition. Calibrated with imatrix for superior quantization fidelity |
| [GLM-4.7-Flash](https://huggingface.co/unsloth/GLM-4.7-Flash-GGUF) | Flash | Q4_K_XL | **Router** — ultra-fast model for routing decisions, intent classification, and dispatching tasks to appropriate workers |
| [Hermes 4.3 36B](https://huggingface.co/NousResearch/Hermes-4.3-36B-GGUF) | 36B | Q4_K_M | **Quality Orchestrator** — a capable model for complex multi-step reasoning, tool use, and agent coordination. Excellent function calling |

### ⚙️ Agentic Workers

Models optimized to be **called by orchestrators** — they receive specific tasks, execute them, and return structured results.

| Model | Size | Quantization | Role |
|-------|------|-------------|------|
| [Gemma 4 12B Agentic Worker v2](https://huggingface.co/yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2-GGUF) | 12B | Q4_K_M | **Worker Agent** — fine-tuned specifically for agentic task execution with Fable5 and Composer2.5 training. Designed to follow orchestrator instructions reliably |

### 🛡️ Guardrails & Safety

Models that **monitor and filter** agent inputs/outputs to prevent harmful, off-topic, or undesired behavior.

| Model | Size | Quantization | Role |
|-------|------|-------------|------|
| [Hermes 4 14B](https://huggingface.co/gabriellarson/Hermes-4-14B-GGUF) | 14B | Q4_K_M | **Speed Guardrail** — fast content moderation and output validation for real-time agent pipelines |
| [Llama Guard 3 8B](https://huggingface.co/mradermacher/Llama-Guard-3-8B-i1-GGUF) | 8B | i1-Q4_K_M | **Firewall** — Meta's safety-focused model for input/output classification and content policy enforcement |

<details>
<summary>📁 Folder Structure</summary>

```
3-Agent_Orchestration_and_Routing/
├── 1-Orchestrators_and_Managers/
│   ├── Agents_A1_35B_MoE_Calibrated_CEO/
│   │   └── Agents-A1-IQ4_XS-imatrix-gguf-fable5-calibrated.gguf
│   ├── GLM-4.7-Flash-UD-Q4_K_XL/
│   │   └── GLM-4.7-Flash-UD-Q4_K_XL.gguf
│   └── Hermes_4_3_36B_Quality_Orchestrator/
│       └── hermes-4_3_36b-Q4_K_M.gguf
├── 2-Agentic_Workers/
│   └── Gemma4_12B_Agentic_Worker_v2/
│       └── Gemma4_12B_Agentic_Worker_v2.gguf
└── 3-Guardrails_and_Safety/
    ├── Hermes_4_14B_Speed_Guardrail/
    │   └── Hermes-4-14B-Q4_K_M.gguf
    └── Llama_Guard_3_8B_i1_Firewall/
        └── Llama-Guard-3-8B.i1-Q4_K_M.gguf
```
</details>

---

## 4️⃣ Document Analysis and RAG

<div align="center">
<i>The complete pipeline for building local Retrieval-Augmented Generation systems — from embedding to generation.</i>
</div>

RAG (Retrieval-Augmented Generation) is how you give your AI **access to your own documents, knowledge bases, and data**. This category provides every component needed to build a production-quality local RAG pipeline — no cloud APIs required.

### Pipeline Overview

```
📄 Documents → 🔤 Embedding → 📊 Vector Store → 🔄 Reranking → 🧠 LLM Generation
                  BGE-M3           (your DB)        BGE-Reranker     Gemma 4 12B
                                                              ↕
                                              🕸️ GraphRAG Extractor (NuExtract)
                                                              ↕
                                              🌐 Web Distiller (Jina Reader)
```

### 🔤 1. Embeddings — [BGE-M3](https://huggingface.co/BAAI/bge-m3)

**The foundation of any RAG system.** BGE-M3 is a state-of-the-art multilingual embedding model that supports three retrieval modes simultaneously:

- **Dense retrieval** — traditional semantic similarity search
- **Sparse retrieval** — BM25-style keyword matching (great for exact term matches)
- **ColBERT retrieval** — late interaction for fine-grained token-level relevance scoring

This multi-modal retrieval approach means you combine semantic understanding and keyword precision in a single model. It supports 100+ languages out of the box.

### 🔄 2. Reranker — [BGE-Reranker-V2-M3](https://huggingface.co/Heisenberg0314/bge-reranker-v2-m3)

After initial retrieval returns candidate documents, the reranker **re-scores and re-orders** them for maximum relevance. This two-stage approach (retrieve broadly, then rerank precisely) dramatically improves RAG accuracy compared to single-stage retrieval. The M3 variant is specifically designed to complement BGE-M3 embeddings.

### 🕸️ 3. GraphRAG Extractor — [NuExtract v1.5](https://huggingface.co/bartowski/NuExtract-v1.5-GGUF)

Takes unstructured text and **extracts structured knowledge graphs** — entities, relationships, and attributes. This enables GraphRAG pipelines that can reason over the *connections* between facts, not just individual document chunks. Particularly powerful for research, legal, and technical domains where relationships matter as much as the facts themselves.

### 🌐 4. Web Distiller — [Jina Reader LM 1.5B](https://huggingface.co/bartowski/reader-lm-1.5b-GGUF)

A tiny but remarkably effective model that **cleans and structures raw web content** into clean, readable text suitable for embedding. Instead of embedding HTML noise, navigation bars, and ads, this model extracts the actual article content. At only 1.5B parameters, it runs almost instantly.

### 🧠 5. LLM Generator — [Gemma 4 12B](https://huggingface.co/unsloth/gemma-4-12b-it-GGUF)

The final stage of the RAG pipeline — a vision-capable LLM that synthesizes retrieved context into coherent, accurate answers. Includes **speculative decoding via MTP (Multi-Token Prediction)** for faster generation, and a multimodal projector for handling documents with images and diagrams.

<details>
<summary>📁 Folder Structure</summary>

```
4-Document_Analysis_and_RAG/
├── 1-Embeddings/
│   └── BGE_M3_Core_Embedding/
│       ├── config.json, tokenizer files, model weights...
│       └── colbert_linear.pt, sparse_linear.pt
├── 2-Rerankers/
│   └── BGE_Reranker_V2_M3_Core/
│       ├── config.json, model.safetensors
│       └── tokenizer files
├── 3-GraphRAG_Extractors/
│   └── NuExtract_1_5_GraphRAG_Extractor/
│       └── NuExtract-v1.5-Q8_0.gguf
├── 4-Web_Distillers/
│   └── Jina_Reader_LM_1_5B_Web_Distiller/
│       └── reader-lm-1.5b-Q8_0.gguf
└── 5-LLM_Generators/
    └── gemma-4-12b-it-UD-Q4_K_XL/
        ├── gemma-4-12b-it-UD-Q4_K_XL.gguf
        ├── mmproj-BF16.gguf
        └── MTP/
            └── mtp-gemma-4-12b-it-Q8_0.gguf
```
</details>

---

## 5️⃣ Text-to-Speech (TTS)

<div align="center">
<i>Convert text to natural-sounding speech — from multi-speaker dialogue to voice cloning and custom voice design.</i>
</div>

This category covers four distinct TTS paradigms, each suited to different use cases. Whether you need a single consistent narrator, a full cast of characters, or the ability to clone any voice from a short sample, there's a model here for it.

| Paradigm | Model | Size | Use Case |
|----------|-------|------|----------|
| 🎭 **Multi-Speaker** | [Dia 2.5 2B](https://huggingface.co/nari-labs/Dia2-2B) | 2B | Generate dialogue with multiple distinct speakers in a single pass — perfect for audiobooks, podcasts, and conversational AI |
| 🇮🇷 **Persian TTS** | [Chatterbox TTS Persian](https://huggingface.co/Thomcles/Chatterbox-TTS-Persian-Farsi) | — | High-quality Farsi text-to-speech with natural Persian prosody and pronunciation |
| 🇮🇷 **Persian TTS** | [F5-TTS Persian](https://huggingface.co/Lumos675/F5_TTS_Persian) | — | Alternative Persian TTS model — good for A/B testing voice quality on Farsi content |
| 🎨 **Voice Design** | [Qwen3-TTS 12Hz](https://huggingface.co/Jinstudio/Qwen3-TTS-12Hz-1.7B-VoiceDesign) | 1.7B | **Design entirely new voices from text descriptions** — describe the voice you want ("warm grandmotherly voice") and it generates it |
| 🎤 **Zero-Shot Cloning** | [F5-TTS v1](https://huggingface.co/TopherAU/F5-TTS-v1) | — | Clone any voice from a short audio reference sample — no fine-tuning required |

**Why so many TTS models?** Different tasks demand different approaches. Voice Design is perfect for creating brand voices or fictional characters. Zero-shot cloning excels when you need to match an existing voice. Multi-speaker Dia handles scripts with dialogue natively. And the Persian-specific models ensure high-quality Farsi output that general multilingual TTS models often struggle with.

<details>
<summary>📁 Folder Structure</summary>

```
5-TTS/
├── MULTI_SPEAKER/
│   └── Dia2_2B_Dialogue/
├── TTS_PERSIAN/
│   ├── Chatterbox_TTS_Persian/
│   │   └── t3_fa.safetensors
│   └── F5_TTS_Persian/
├── VOICE_DESIGN/
│   └── Qwen3_TTS_12Hz_VoiceDesign/
│       ├── model.safetensors
│       └── speech_tokenizer/
└── ZERO_SHOT_CLONING/
    └── F5_TTS_v1_Base/
        └── F5TTS_v1_Base/
```
</details>

---

## 6️⃣ Speech-to-Text (STT)

<div align="center">
<i>Transcribe, translate, and preprocess audio — from studio-quality multi-language recognition to specialized Persian ASR.</i>
</div>

A complete speech understanding pipeline that starts with audio preprocessing (cleaning and segmentation) and ends with either transcription or translation. Like other categories, it follows the Quality vs. Speed dual-tier design.

### 🔧 Audio Preprocessing Filters

| Filter | Model | Purpose |
|--------|-------|---------|
| 🎙️ Vocal Isolation | [Demucs V4 (ONNX)](https://huggingface.co/MrCitron/demucs-v4-onnx) | Studio-quality vocal separation — removes background music, noise, and other speakers. Essential for noisy recordings |
| 🔇 Silence/Noise Cutter | [Silero VAD (ONNX)](https://huggingface.co/istupakov/silero-vad-onnx) | Voice Activity Detection — automatically cuts silence and non-speech segments, reducing processing time and improving transcription accuracy |

### 🏆 Quality Tier — High Accuracy

| Scope | Model | Size | Highlights |
|-------|-------|------|------------|
| 🌍 Multi-Language | [Whisper Large V3](https://huggingface.co/openai/whisper-large-v3) | Large | OpenAI's flagship multilingual STT model. Supports 99 languages with state-of-the-art accuracy. The gold standard for transcription |
| 🇮🇷 Persian | [FastConformer Fa Large](https://huggingface.co/nvidia/stt_fa_fastconformer_hybrid_large) | Large | **Anti-hallucination** Persian ASR by NVIDIA. Specifically designed to avoid the common Whisper problem of fabricating text during silence — critical for Persian content |
| 🇮🇷 Persian | [Whisper Persian V4](https://huggingface.co/nezamisafa/whisper-persian-v4) | Large | Whisper fine-tuned specifically on Persian data for improved Farsi recognition accuracy |
| 🌐 Translation | [Seamless M4T V2 Large](https://huggingface.co/facebook/seamless-m4t-v2-large) | Large | Meta's **universal translation** model — translates speech-to-text across 100+ languages. Not just transcription, but actual cross-lingual translation |

### ⚡ Speed Tier — Fastest Processing

| Scope | Model | Size | Highlights |
|-------|-------|------|------------|
| 🌍 Multi-Language | [Whisper Large V3 Turbo](https://huggingface.co/openai/whisper-large-v3-turbo) | Turbo | 8x faster than Whisper Large with only minimal accuracy loss. The sweet spot for real-time or batch processing |
| 🇮🇷 Persian | [Whisper Turbo Persian](https://huggingface.co/SadeghK/whisper-large-v3-turbo) | Turbo | Turbo-speed Whisper variant for Persian language. Ideal for transcribing long Persian audio files quickly |

<details>
<summary>📁 Folder Structure</summary>

```
6-STT/
├── Audio_Preprocessing_Filters/
│   ├── Studio_Vocal_Isolation_DemucsV4/
│   │   └── Studio_Vocal_Isolation_DemucsV4.onnx
│   └── Universal_Silence_Noise_Cutter_Silero/
│       └── Universal_Silence_Noise_Cutter_Silero.onnx
├── Quality/
│   ├── Multi_Language/
│   │   └── Whisper_Large_V3_Global_FP16/
│   ├── Persian_ASR/
│   │   ├── Anti_Hallucination_FastConformer_Fa/
│   │   │   └── stt_fa_fastconformer_hybrid_large.nemo
│   │   └── Whisper_Persian_V4_FP16/
│   └── Translation/
│       └── Seamless_M4T_V2_Large_Translation/
└── Speed/
    ├── Multi_Language/
    │   └── Whisper_Turbo_Global_Speed/
    └── Persian_ASR/
        └── Whisper_Turbo_Persian_Speed/
```
</details>

---

## 7️⃣ Multimodal

<div align="center">
<i>Models that see and hear — process images, audio, and text together for rich understanding.</i>
</div>

Multimodal models are the frontier of local AI — they can analyze images, describe screenshots, read documents with visual layouts, and process audio alongside text. This enables workflows like "take a photo of this whiteboard and summarize it" or "analyze this chart and explain the trends."

| Tier | Model | Size | Quantization | Key Capabilities |
|------|-------|------|-------------|------------------|
| 🏆 Quality | [Qwen3-Omni 30B-A3B](https://huggingface.co/ggml-org/Qwen3-Omni-30B-A3B-Instruct-GGUF) | 30B (MoE, 3B active) | Q4_K_M | **Full omni-model** — understands text, images, AND audio natively. The most capable multimodal model in this collection. MoE architecture keeps VRAM usage manageable |
| ⚡ Speed | [Gemma 4 E4B](https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF) | ~4B (effective) | Q4_K_XL | Speculative decoding with MTP draft model for faster generation. Excellent for quick image analysis, screenshot understanding, and visual Q&A. Despite its small effective size, Gemma 4 punches well above its weight |

> ⚠️ **CRITICAL — Read before relying on "multimodal":** The models shipped here are **GGUF quantized** files. In their current state, tooling only supports the **vision (image)** modality from these quantized files — **audio input does not work** on the quantized versions.
>
> True text + image + audio multimodality is currently only available from the **original, uncompressed (FP16/BF16) models** — not from the quantized GGUF files in this toolkit. The quantized variants effectively behave as **vision-language models (VLM)**: they can see images and read text, but they **cannot process audio**.
>
> This is a tooling limitation, not a model defect — current inference engines (llama.cpp, Ollama, LM Studio, etc.) cannot yet route audio through quantized GGUF multimodal pipelines. If you need genuine audio understanding, download and run the **original uncompressed weights** from the source repos linked below.
>
> I learned this the hard way after downloading — the quantized files simply won't run as full multimodal. Hence this warning.

**Why Qwen3-Omni for quality?** It's one of the very few models that genuinely handles text + image + audio in a unified architecture. Most "multimodal" models only handle text and images — Qwen3-Omni adds audio understanding, making it a true all-in-one model for multimedia tasks.

<details>
<summary>📁 Folder Structure</summary>

```
7-Multimodal/
├── Quality/
│   └── Qwen3_Omni_30B_A3B_Quality/
│       ├── Qwen3-Omni-30B-A3B-Instruct-Q4_K_M.gguf
│       └── mmproj-Qwen3-Omni-30B-A3B-Instruct-Q8_0.gguf
└── Speed/
    └── Gemma4_E4B_Speed_Speculative/
        ├── gemma-4-E4B-it-UD-Q4_K_XL.gguf
        ├── mmproj-F16.gguf
        └── MTP/
            └── gemma-4-E4B-it-F16-MTP.gguf
```
</details>

---

## 8️⃣ Uncensored Models

<div align="center">
<i>Unfiltered variants for research, creative writing, and use cases requiring unrestricted model outputs.</i>
</div>

These are **abliterated** (uncensored) versions of the models found in other categories. They have had their refusal mechanisms and output filters removed through fine-tuning. They produce the same quality outputs as their censored counterparts but without refusing requests.

> ⚠️ **Disclaimer:** These models are provided for legitimate research, creative, and educational purposes. Users are responsible for ensuring their use complies with applicable laws and ethical guidelines.

| Category | Tier | Model | Size | Quantization | Notes |
|----------|------|-------|------|-------------|-------|
| 💻 Code | 🏆 Quality | [Huihui Qwen3-VL 30B (Uncensored)](https://huggingface.co/mradermacher/Huihui-Qwen3-VL-30B-A3B-Instruct-abliterated-GGUF) | 30B (MoE) | Q4_K_M | Uncensored vision-language model — can analyze code screenshots, diagrams, and visual content without restrictions |
| 🧠 General | 🏆 Quality | [Huihui Qwen3.6-35B Claude 4.7 Opus (Uncensored)](https://huggingface.co/huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-MTP-GGUF) | 35B (MoE) | Q4_K | Uncensored general-purpose model with MTP support. Tuned with Claude 4.7 Opus data for enhanced instruction following |
| 💻 Code | ⚡ Speed | [Huihui Qwen3-Coder 30B (Uncensored)](https://huggingface.co/mradermacher/Huihui-Qwen3-Coder-30B-A3B-Instruct-abliterated-i1-GGUF) | 30B (MoE) | i1-Q3_K_M | Uncensored coding specialist — importance-matrix quantized for better quality at lower bitrates |
| 🧠 General | ⚡ Speed | [Huihui Qwen3-VL 8B (Uncensored)](https://huggingface.co/mradermacher/Huihui-Qwen3-VL-8B-Instruct-abliterated-GGUF) | 8B | Q8_0 | Small, fast uncensored VLM — great for quick visual analysis tasks where you need unrestricted output |

<details>
<summary>📁 Folder Structure</summary>

```
8-Uncensored_Models/
├── Quality/
│   ├── Code/
│   │   └── Huihui_Qwen3_VL_30B_Uncensored_Q4_K_M/
│   │       └── Huihui-Qwen3-VL-30B-A3B-Instruct-abliterated.Q4_K_M.gguf
│   └── General/
│       └── Huihui_Qwen3.6_35B_Claude4.7_Opus_MTP_Uncensored_Q4_K/
│           ├── Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-ggml-model-Q4_K.gguf
│           └── mmproj-model-f16.gguf
└── Speed/
    ├── Code/
    │   └── Huihui_Qwen3_Coder_30B_Uncensored_i1_Q3_K_M/
    │       └── Huihui-Qwen3-Coder-30B-A3B-Instruct-abliterated.i1-Q3_K_M.gguf
    └── General/
        └── Huihui_Qwen3_VL_8B_Uncensored_Q8_0/
            ├── Huihui-Qwen3-VL-8B-Instruct-abliterated.Q8_0.gguf
            └── Huihui-Qwen3-VL-8B-Instruct-abliterated.mmproj-f16.gguf
```
</details>

---

## 🔗 Model Source Links

All models in this toolkit are sourced from Hugging Face. During the initial download process, direct URLs were not recorded. The original sources were **recovered by matching SHA-256 hashes** against Hugging Face repositories. While some links may point to slightly different file versions within the same quantization family, the hash-verified content is identical to what is hosted in this toolkit.

### 1 — Universal Foundation LLMs

| File | Source |
|------|--------|
| Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf | [unsloth/Qwen3.6-35B-A3B-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/blob/main/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf) |
| mmproj-BF16.gguf (Qwen3.6) | [unsloth/Qwen3.6-35B-A3B-GGUF](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/blob/main/mmproj-BF16.gguf) |
| Qwen3.5-9B-Q5_K_M.gguf | [unsloth/Qwen3.5-9B-GGUF](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF/blob/main/Qwen3.5-9B-Q5_K_M.gguf) |
| mmproj-BF16.gguf (Qwen3.5) | [unsloth/Qwen3.5-9B-GGUF](https://huggingface.co/unsloth/Qwen3.5-9B-GGUF/blob/0b11907b7136445eab40d9c30b737ecf6888ea43/mmproj-BF16.gguf) |

### 2 — Coding Agents

| File | Source |
|------|--------|
| Qwen3-Coder-30B-A3B-Instruct-Q5_K_M.gguf | [unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF](https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF/blob/7ce945e58ed3f09f9cf9c33a2122d86ac979b457/Qwen3-Coder-30B-A3B-Instruct-Q5_K_M.gguf) |
| DeepSeek-R1-Distill-Qwen-32B-Q4_K_M.gguf | [bartowski/DeepSeek-R1-Distill-Qwen-32B-GGUF](https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-32B-GGUF/blob/main/DeepSeek-R1-Distill-Qwen-32B-Q4_K_M.gguf) |
| qwen2.5-coder-7b-instruct-q5_k_m.gguf | [EasierAI/Qwen-2.5-Coder-7B](https://huggingface.co/EasierAI/Qwen-2.5-Coder-7B/blob/930efd5c3de60fde0723f262592f8551aebbd7f2/qwen2.5-coder-7b-instruct-q5_k_m.gguf) |
| Qwen2.5-Coder-14B-Q4_K_M.gguf | [bartowski/Qwen2.5-Coder-14B-GGUF](https://huggingface.co/bartowski/Qwen2.5-Coder-14B-GGUF/blob/main/Qwen2.5-Coder-14B-Q4_K_M.gguf) |
| DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf | [bartowski/DeepSeek-R1-Distill-Qwen-14B-GGUF](https://huggingface.co/bartowski/DeepSeek-R1-Distill-Qwen-14B-GGUF/blob/850859f9d3d93e1d2b19d37d15adf1d7ea7ac493/DeepSeek-R1-Distill-Qwen-14B-Q4_K_M.gguf) |

### 3 — Agent Orchestration and Routing

| File | Source |
|------|--------|
| Agents-A1-IQ4_XS-imatrix-gguf-fable5-calibrated.gguf | [Chungulus/Agents-A1-IQ4_XS-imatrix-gguf-fable5-calibrated](https://huggingface.co/Chungulus/Agents-A1-IQ4_XS-imatrix-gguf-fable5-calibrated/blob/main/Agents-A1-IQ4_XS-imatrix-gguf-fable5-calibrated.gguf) |
| GLM-4.7-Flash-UD-Q4_K_XL.gguf | [unsloth/GLM-4.7-Flash-GGUF](https://huggingface.co/unsloth/GLM-4.7-Flash-GGUF/blob/main/GLM-4.7-Flash-UD-Q4_K_XL.gguf) |
| hermes-4_3_36b-Q4_K_M.gguf | [NousResearch/Hermes-4.3-36B-GGUF](https://huggingface.co/NousResearch/Hermes-4.3-36B-GGUF/blob/e6cfb17bc2a4287a7b9982df186b8c87b6de445c/hermes-4_3_36b-Q4_K_M.gguf) |
| Gemma4_12B_Agentic_Worker_v2.gguf | [yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2-GGUF](https://huggingface.co/yuxinlu1/gemma-4-12B-agentic-fable5-composer2.5-v2-3.5x-tau2-GGUF/blob/190a31365a6b80a692349be34ccdac730cad4fe4/gemma4-v2-Q4_K_M.gguf) |
| Hermes-4-14B-Q4_K_M.gguf | [gabriellarson/Hermes-4-14B-GGUF](https://huggingface.co/gabriellarson/Hermes-4-14B-GGUF/blob/main/Hermes-4-14B-Q4_K_M.gguf) |
| Llama-Guard-3-8B.i1-Q4_K_M.gguf | [mradermacher/Llama-Guard-3-8B-i1-GGUF](https://huggingface.co/mradermacher/Llama-Guard-3-8B-i1-GGUF/blob/main/Llama-Guard-3-8B.i1-Q4_K_M.gguf) |

### 4 — Document Analysis and RAG

| File | Source |
|------|--------|
| BGE-M3 Embedding | [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3/tree/main) |
| BGE-Reranker-V2-M3 | [Heisenberg0314/bge-reranker-v2-m3](https://huggingface.co/Heisenberg0314/bge-reranker-v2-m3/tree/main) |
| NuExtract-v1.5-Q8_0.gguf | [bartowski/NuExtract-v1.5-GGUF](https://huggingface.co/bartowski/NuExtract-v1.5-GGUF/blob/24e96b9346fd75b1b56ef606cba6f42f7cd12f8a/NuExtract-v1.5-Q8_0.gguf) |
| reader-lm-1.5b-Q8_0.gguf | [bartowski/reader-lm-1.5b-GGUF](https://huggingface.co/bartowski/reader-lm-1.5b-GGUF/blob/bd0334641e23f8cb8e1555cc244a91dfeb2ffff7/reader-lm-1.5b-Q8_0.gguf) |
| gemma-4-12b-it-UD-Q4_K_XL.gguf | [unsloth/gemma-4-12b-it-GGUF](https://huggingface.co/unsloth/gemma-4-12b-it-GGUF/blob/c0e396f9ee60d3ad702248f17edc3ec22384c287/gemma-4-12b-it-UD-Q4_K_XL.gguf) |
| mmproj-BF16.gguf (Gemma 4 12B) | [unsloth/gemma-4-12b-it-GGUF](https://huggingface.co/unsloth/gemma-4-12b-it-GGUF/blob/main/mmproj-BF16.gguf) |
| mtp-gemma-4-12b-it-Q8_0.gguf | [unsloth/gemma-4-12b-it-GGUF](https://huggingface.co/unsloth/gemma-4-12b-it-GGUF/blob/main/MTP/mtp-gemma-4-12b-it-Q8_0.gguf) |

### 5 — Text-to-Speech

| File | Source |
|------|--------|
| Dia2 2B Dialogue | [nari-labs/Dia2-2B](https://huggingface.co/nari-labs/Dia2-2B/tree/main) |
| Chatterbox TTS Persian | [Thomcles/Chatterbox-TTS-Persian-Farsi](https://huggingface.co/Thomcles/Chatterbox-TTS-Persian-Farsi/blob/main/t3_fa.safetensors) |
| F5-TTS Persian | [Lumos675/F5_TTS_Persian](https://huggingface.co/Lumos675/F5_TTS_Persian/tree/main) |
| Qwen3-TTS 12Hz VoiceDesign | [Jinstudio/Qwen3-TTS-12Hz-1.7B-VoiceDesign](https://huggingface.co/Jinstudio/Qwen3-TTS-12Hz-1.7B-VoiceDesign/tree/main) |
| F5-TTS v1 Base | [TopherAU/F5-TTS-v1](https://huggingface.co/TopherAU/F5-TTS-v1/tree/main) |

### 6 — Speech-to-Text

| File | Source |
|------|--------|
| Demucs V4 ONNX | [MrCitron/demucs-v4-onnx](https://huggingface.co/MrCitron/demucs-v4-onnx/blob/main/htdemucs_ft.onnx) |
| Silero VAD ONNX | [istupakov/silero-vad-onnx](https://huggingface.co/istupakov/silero-vad-onnx/blob/main/silero_vad.onnx) |
| Whisper Large V3 | [openai/whisper-large-v3](https://huggingface.co/openai/whisper-large-v3/tree/main) |
| FastConformer Fa Large | [nvidia/stt_fa_fastconformer_hybrid_large](https://huggingface.co/nvidia/stt_fa_fastconformer_hybrid_large/blob/main/stt_fa_fastconformer_hybrid_large.nemo) |
| Whisper Persian V4 | [nezamisafa/whisper-persian-v4](https://huggingface.co/nezamisafa/whisper-persian-v4/tree/main) |
| Seamless M4T V2 Large | [facebook/seamless-m4t-v2-large](https://huggingface.co/facebook/seamless-m4t-v2-large/tree/main) |
| Whisper Large V3 Turbo | [openai/whisper-large-v3-turbo](https://huggingface.co/openai/whisper-large-v3-turbo/tree/main) |
| Whisper Turbo Persian | [SadeghK/whisper-large-v3-turbo](https://huggingface.co/SadeghK/whisper-large-v3-turbo/tree/main) |

### 7 — Multimodal

| File | Source |
|------|--------|
| Qwen3-Omni-30B-A3B-Instruct-Q4_K_M.gguf | [ggml-org/Qwen3-Omni-30B-A3B-Instruct-GGUF](https://huggingface.co/ggml-org/Qwen3-Omni-30B-A3B-Instruct-GGUF/blob/main/Qwen3-Omni-30B-A3B-Instruct-Q4_K_M.gguf) |
| mmproj-Qwen3-Omni-30B-A3B-Instruct-Q8_0.gguf | [ggml-org/Qwen3-Omni-30B-A3B-Instruct-GGUF](https://huggingface.co/ggml-org/Qwen3-Omni-30B-A3B-Instruct-GGUF/blob/main/mmproj-Qwen3-Omni-30B-A3B-Instruct-Q8_0.gguf) |
| gemma-4-E4B-it-UD-Q4_K_XL.gguf | [unsloth/gemma-4-E4B-it-GGUF](https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF/blob/0f9c09189de30e8d9a30d09fe5fb9ce9284aa326/gemma-4-E4B-it-UD-Q4_K_XL.gguf) |
| mmproj-F16.gguf (Gemma 4 E4B) | [unsloth/gemma-4-E4B-it-GGUF](https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF/blob/main/mmproj-F16.gguf) |
| gemma-4-E4B-it-F16-MTP.gguf | [unsloth/gemma-4-E4B-it-GGUF](https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF/blob/main/MTP/mtp-gemma-4-E4B-it-F16.gguf) |

### 8 — Uncensored Models

| File | Source |
|------|--------|
| Huihui-Qwen3-VL-30B-A3B-Instruct-abliterated.Q4_K_M.gguf | [mradermacher/Huihui-Qwen3-VL-30B-A3B-Instruct-abliterated-GGUF](https://huggingface.co/mradermacher/Huihui-Qwen3-VL-30B-A3B-Instruct-abliterated-GGUF/blob/main/Huihui-Qwen3-VL-30B-A3B-Instruct-abliterated.Q4_K_M.gguf) |
| Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-Q4_K.gguf | [huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-MTP-GGUF](https://huggingface.co/huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-MTP-GGUF/blob/main/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-ggml-model-Q4_K.gguf) |
| mmproj-model-f16.gguf (Huihui 35B) | [huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-MTP-GGUF](https://huggingface.co/huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-MTP-GGUF/blob/main/mmproj-model-f16.gguf) |
| Huihui-Qwen3-Coder-30B-A3B-Instruct-abliterated.i1-Q3_K_M.gguf | [mradermacher/Huihui-Qwen3-Coder-30B-A3B-Instruct-abliterated-i1-GGUF](https://huggingface.co/mradermacher/Huihui-Qwen3-Coder-30B-A3B-Instruct-abliterated-i1-GGUF/blob/main/Huihui-Qwen3-Coder-30B-A3B-Instruct-abliterated.i1-Q3_K_M.gguf) |
| Huihui-Qwen3-VL-8B-Instruct-abliterated.Q8_0.gguf | [mradermacher/Huihui-Qwen3-VL-8B-Instruct-abliterated-GGUF](https://huggingface.co/mradermacher/Huihui-Qwen3-VL-8B-Instruct-abliterated-GGUF/blob/main/Huihui-Qwen3-VL-8B-Instruct-abliterated.Q8_0.gguf) |
| Huihui-Qwen3-VL-8B-Instruct-abliterated.mmproj-f16.gguf | [mradermacher/Huihui-Qwen3-VL-8B-Instruct-abliterated-GGUF](https://huggingface.co/mradermacher/Huihui-Qwen3-VL-8B-Instruct-abliterated-GGUF/blob/main/Huihui-Qwen3-VL-8B-Instruct-abliterated.mmproj-f16.gguf) |

---

## 🛠️ Recommended Tools for Running These Models

| Tool | Best For | Link |
|------|----------|------|
| **Ollama** | Easiest setup, CLI-first, model management | [ollama.com](https://ollama.com) |
| **LM Studio** | GUI-based, beginner-friendly, model browsing | [lmstudio.ai](https://lmstudio.ai) |
| **llama.cpp** | Maximum control, server mode, API compatible | [github.com/ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp) |
| **vLLM** | High-throughput serving, OpenAI-compatible API | [github.com/vllm-project/vllm](https://github.com/vllm-project/vllm) |
| **Koboldcpp** | One-click launcher, built-in GPU support | [github.com/lostruins/koboldcpp](https://github.com/lostruins/koboldcpp) |

---

## 📋 Hardware Requirements

| Component | Minimum | Recommended | This Toolkit's Target |
|-----------|---------|-------------|-----------------------|
| **GPU VRAM** | 8 GB | 12 GB | **NVIDIA RTX 3060 12GB** |
| **System RAM** | 16 GB | 32 GB | **32 GB DDR4** |
| **Storage** | 100 GB SSD | 500 GB NVMe | SSD strongly recommended |
| **OS** | Windows/Linux/macOS | Linux (Ubuntu) | Any with CUDA support |

> 💡 **MoE Advantage:** Many models in this toolkit use Mixture-of-Experts (MoE) architecture, which activates only a fraction of total parameters during inference. This means you get the quality of much larger models while staying within 12GB VRAM constraints.

---

<div align="center">

## ⭐ Acknowledgments

All models in this toolkit are the work of their respective creators and are used in accordance with their original licenses. Special thanks to:

**Model Families:** [Qwen](https://huggingface.co/Qwen) · [DeepSeek](https://huggingface.co/deepseek-ai) · [Gemma](https://huggingface.co/google) · [Hermes/NousResearch](https://huggingface.co/NousResearch) · [Whisper/OpenAI](https://huggingface.co/openai) · [Llama/Meta](https://huggingface.co/meta-llama) · [GLM](https://huggingface.co/THUDM) · [BGE/BAAI](https://huggingface.co/BAAI)

**Quantization:** [unsloth](https://huggingface.co/unsloth) · [bartowski](https://huggingface.co/bartowski) · [mradermacher](https://huggingface.co/mradermacher) · [ggml-org](https://huggingface.co/ggml-org) · [Chungulus](https://huggingface.co/Chungulus)

**Uncensored Variants:** [huihui-ai](https://huggingface.co/huihui-ai) · [mradermacher](https://huggingface.co/mradermacher)

---

<i>Assembled with AI-assisted model selection · Curated for RTX 3060 12GB · Updated as stronger models arrive</i>

</div>