README.md

---
license: mit
---
# 🚀 Local AI Model Hub: Quantized LLMs, OCR Vision, RAG & Audio AI Ecosystem

<p align="center">
  <img src="https://shields.io" alt="Format">
  <img src="https://shields.io" alt="Tasks">
  <img src="https://shields.io" alt="Optimization">
</p>

## 📌 Overview
Welcome to the ultimate **Local AI Model Repository**. This repo provides a highly optimized, categorized collection of quantized Large Language Models (LLMs), Multimodal Vision Models, and state-of-the-art Audio AI models. 

Everything here is engineered for **offline deployment, low-VRAM hardware, edge computing, and privacy-first local workflows**.

---

## 📂 Model Recommendations & Use-Cases (Best For...)

To streamline your local deployment, models are strictly categorized by their specialized engineering strengths:

### 👁️ 1. BEST FOR: Local OCR & Document Processing (Vision-Language)
* **Ministral 3B Instruct**: `Ministral-3-3B-Instruct-2512-BF16-mmproj.gguf` & `Ministral-3-3B-Instruct-2512-Q8_0.gguf`
  * 🎯 **Best For**: **Local OCR (Optical Character Recognition)**, structured text extraction from images, document/invoice parsing, and multimodal visual analysis. *(Note: Must load the `.mmproj` file alongside the model weight to enable Vision capabilities).*
* **NuExtract 3**: (`NuExtract3-Q4_K_M.gguf` || `NuExtract3-Q8_0.gguf`) && mmproj-NuExtract3-BF16.gguf
  * 🎯 **Best For**: Converting unstructured OCR raw text or documents into strictly structured, valid JSON outputs.

### 🗄️ 2. BEST FOR: Vector DB Querying & RAG (Retrieval-Augmented Generation)
* **Llama 3 8B**: `llama-3-8b.q4_K_M.gguf`
  * 🎯 **Best For**: **Vector Database querying**, processing context windows in RAG pipelines, semantic search synthesis, and acting as the core reasoning engine for local knowledge bases (Chroma, Pinecone, Qdrant, PGVector).
* **BAAI BGE M3**: `models--BAAI--bge-m3`
  * 🎯 **Best For**: High-accuracy multilingual text embeddings to populate your local Vector DB before querying.

### 🤖 3. BEST FOR: General Chat & Standard LLM Tasks
* **Qwen 3.5 9B (Standard)**: `Qwen3.5-9B-Q5_K_S.bpw.gguf`
  * 🎯 **Best For**: General-purpose conversation, advanced multi-lingual reasoning, text summarization, and acting as a standard, well-aligned conversational assistant.
* * **Gemma 4 (Core & MTP Draft Pairs)**:
  * 📱 **The 2B Active Pair**: `gemma-4-E2B-it-qat-ud-Q4_K_XL.gguf` **must be paired with** `mtp-gemma-4-E2B-it-Q4_0.gguf` as its dedicated draft model to accelerate token generation speed.
  * 🧠 **The 4B Standalone**: `gemma-4-E4B-it-Q4_K_M.gguf` (Run independently for deeper cognitive tasks, or pair it with its corresponding 4B MTP draft file).
  * 🎯 **Best For**: Native hardware evaluation utilizing Google's latest quantized and Multi-Token Prediction (MTP) architectures for up to 2x faster local inference.

### 🔓 4. BEST FOR: Uncensored Chat & CyberSecurity Coding
* **Qwen 3.5 9B (Uncensored)**: `Uncensored-HauhauCS-Aggressive-Q4_K_M.gguf`
  * 🎯 **Best For**: Boundless, unrestricted conversational tasks, open-ended creative writing, and complex multi-turn code generation.
* **WhiteRabbitNeo**: `WhiteRabbitNeo-V3-7B-Q4_K_M.gguf`
  * 🎯 **Best For**: Specialized offensive/defensive cybersecurity operations, log analysis, and malware code debugging.

### 🎙️ 5. BEST FOR: Speech AI (Audio Processing)
* **Faster-Whisper**: `models--Systran--faster-whisper-large-v3` & `models--deepdml--faster-whisper-large-v3-turbo-ct2`
  * 🎯 **Best For**: **Automatic Speech Recognition (ASR / STT)**. Offers blazing-fast, state-of-the-art transcription for English, Vietnamese, and multilingual speech.
* **Qwen-TTS**: `models--Qwen--Qwen3-TTS-12Hz-1.7B-CustomVoice`
  * 🎯 **Best For**: Natural Text-to-Speech (TTS) synthesis and lightweight voice cloning operations.
* **NVIDIA Parakeet & Vosk**: `models--nvidia--parakeet-tdt-0.6b-v3` & `vosk_models`
  * 🎯 **Best For**: Ultra-low latency, real-time voice recognition on low-spec edge devices.

---

## 🛠️ Quick Start Guide

### 1. Running Multimodal OCR (Ministral 3B) via `llama.cpp`
To perform OCR or image-to-text natively, feed both the core GGUF file and the vision projection module:
```bash
./llama-cli -m Ministral-3-3B-Instruct-2512-Q8_0.gguf \
            --mmproj Ministral-3-3B-Instruct-2512-BF16-mmproj.gguf \
            --image path/to/your/document.png \
            -p "Perform a highly accurate OCR transcript of this document."
```

### 2. Integrating Llama 3 8B with Local Vector DB (RAG)
Below is a standard conceptual architecture snippet using LangChain to query your vector backend:
```python
from langchain_community.llms import LlamaCpp
from langchain_chains import RetrievalQA

# Load the Llama-3-8B vector database specialist
llm = LlamaCpp(model_path="./llama-3-8b.q4_K_M.gguf", n_ctx=8192)

# Querying your pre-loaded Vector Store
retriever = vector_db.as_retriever(search_kwargs={"k": 5})
qa_chain = RetrievalQA.from_chain_type(llm=llm, chain_type="stuff", retriever=retriever)

response = qa_chain.run("What does the internal database say about our 2026 Q3 policy?")
```

### 🌐 3. Hosting a Local OpenAI-Compatible API Server
You can host any GGUF model (such as the standard Qwen 3.5 9B) as a local background server using `llama-server.exe` on Windows. This opens an OpenAI-compatible endpoint that can be plugged into SillyTavern, Open WebUI, or custom apps.

Run the following command in your terminal (adjust the GGUF file path if necessary):

```cmd
./llama-server.exe -m "Qwen3.5-9B-Q5_K_S-5.10bpw.gguf" \
                   -ngl 99 \
                   -c 32768 \
                   -np 1 \
                   -mg 0 \
                   --host 127.0.0.1 \
                   --port 8001
```

For draft model:
```cmd
set MAIN_MODEL=.\gemma-4-E2B-it-qat-UD-Q4_K_XL.gguf
set DRAFT_MODEL=.\mtp-gemma-4-E2B-it-Q4_0.gguf

llama-server.exe -m "%MAIN_MODEL%" --model-draft "%DRAFT_MODEL%" --spec-type draft-mtp -ngl 99 -c 32768 -np 1 -mg 0 --host 127.0.0.1 --port 8001
```

**Parameter Breakdown:**
* `-ngl 99`: Offloads 99 layers to the GPU (ensuring full GPU acceleration).
* `-c 32768`: Expands the model's context window up to 32k tokens.
* `-np 1`: Spawns 1 parallel processing slot.
* `--host 127.0.0.1 --port 8001`: Sets up your local endpoint at `http://127.0.0.1:8001`.

---

## ⚡ 🖥️ Hardware Optimization Note (RTX 4060 8GB & Below)
> 💡 **Best For Budget GPUs**: Every model in this repository is heavily quantized and strictly optimized to **run flawlessly with full GPU acceleration on NVIDIA RTX 4060 8GB and below** (including RTX 3060, RTX 4050, and 6GB/8GB Laptop GPUs). 
> 
> By utilizing efficient architectures and GGUF/MTP pairings, you will achieve blazing-fast inference speeds without ever hit any out-of-memory (OOM) limitations on 8GB VRAM hardware.

---

## 🏷️ SEO Keywords & Tags
`local-ocr` • `llama-server-gguf` • `vector-db-querying` • `rag-llm-gguf` • `qwen3.5-gguf` • `standard-llm` • `ministral-ocr` • `llama3-rag` • `gguf-models` • `faster-whisper` • `local-ai-hub` • `uncensored-llm` • `offline-embeddings` • `low-vram-rag`