Hugging Face's logo

motgame
/
public_models

NeMo
Safetensors
GGUF
conversational
Model card Files
xet
 Community
public_models / README.md
motgame's picture
motgame
Update README.md
c502758 verified
preview code
|
Raw
History Blame Contribute Delete
7.17 kB
metadata 
license: mit

🚀 Local AI Model Hub: Quantized LLMs, OCR Vision, RAG & Audio AI Ecosystem

Format Tasks Optimization

📌 Overview

Welcome to the ultimate Local AI Model Repository. This repo provides a highly optimized, categorized collection of quantized Large Language Models (LLMs), Multimodal Vision Models, and state-of-the-art Audio AI models.

Everything here is engineered for offline deployment, low-VRAM hardware, edge computing, and privacy-first local workflows.

📂 Model Recommendations & Use-Cases (Best For...)

To streamline your local deployment, models are strictly categorized by their specialized engineering strengths:

👁️ 1. BEST FOR: Local OCR & Document Processing (Vision-Language)

  • Ministral 3B Instruct: Ministral-3-3B-Instruct-2512-BF16-mmproj.gguf & Ministral-3-3B-Instruct-2512-Q8_0.gguf
    • 🎯 Best For: Local OCR (Optical Character Recognition), structured text extraction from images, document/invoice parsing, and multimodal visual analysis. (Note: Must load the .mmproj file alongside the model weight to enable Vision capabilities).
  • NuExtract 3: (NuExtract3-Q4_K_M.gguf || NuExtract3-Q8_0.gguf) && mmproj-NuExtract3-BF16.gguf
    • 🎯 Best For: Converting unstructured OCR raw text or documents into strictly structured, valid JSON outputs.

🗄️ 2. BEST FOR: Vector DB Querying & RAG (Retrieval-Augmented Generation)

  • Llama 3 8B: llama-3-8b.q4_K_M.gguf
    • 🎯 Best For: Vector Database querying, processing context windows in RAG pipelines, semantic search synthesis, and acting as the core reasoning engine for local knowledge bases (Chroma, Pinecone, Qdrant, PGVector).
  • BAAI BGE M3: models--BAAI--bge-m3
    • 🎯 Best For: High-accuracy multilingual text embeddings to populate your local Vector DB before querying.

🤖 3. BEST FOR: General Chat & Standard LLM Tasks

  • Qwen 3.5 9B (Standard): Qwen3.5-9B-Q5_K_S.bpw.gguf
    • 🎯 Best For: General-purpose conversation, advanced multi-lingual reasoning, text summarization, and acting as a standard, well-aligned conversational assistant.
    • Gemma 4 (Core & MTP Draft Pairs):
    • 📱 The 2B Active Pair: gemma-4-E2B-it-qat-ud-Q4_K_XL.gguf must be paired with mtp-gemma-4-E2B-it-Q4_0.gguf as its dedicated draft model to accelerate token generation speed.
    • 🧠 The 4B Standalone: gemma-4-E4B-it-Q4_K_M.gguf (Run independently for deeper cognitive tasks, or pair it with its corresponding 4B MTP draft file).
    • 🎯 Best For: Native hardware evaluation utilizing Google's latest quantized and Multi-Token Prediction (MTP) architectures for up to 2x faster local inference.

🔓 4. BEST FOR: Uncensored Chat & CyberSecurity Coding

  • Qwen 3.5 9B (Uncensored): Uncensored-HauhauCS-Aggressive-Q4_K_M.gguf
    • 🎯 Best For: Boundless, unrestricted conversational tasks, open-ended creative writing, and complex multi-turn code generation.
  • WhiteRabbitNeo: WhiteRabbitNeo-V3-7B-Q4_K_M.gguf
    • 🎯 Best For: Specialized offensive/defensive cybersecurity operations, log analysis, and malware code debugging.

🎙️ 5. BEST FOR: Speech AI (Audio Processing)

  • Faster-Whisper: models--Systran--faster-whisper-large-v3 & models--deepdml--faster-whisper-large-v3-turbo-ct2
    • 🎯 Best For: Automatic Speech Recognition (ASR / STT). Offers blazing-fast, state-of-the-art transcription for English, Vietnamese, and multilingual speech.
  • Qwen-TTS: models--Qwen--Qwen3-TTS-12Hz-1.7B-CustomVoice
    • 🎯 Best For: Natural Text-to-Speech (TTS) synthesis and lightweight voice cloning operations.
  • NVIDIA Parakeet & Vosk: models--nvidia--parakeet-tdt-0.6b-v3 & vosk_models
    • 🎯 Best For: Ultra-low latency, real-time voice recognition on low-spec edge devices.

🛠️ Quick Start Guide

1. Running Multimodal OCR (Ministral 3B) via llama.cpp

To perform OCR or image-to-text natively, feed both the core GGUF file and the vision projection module:

./llama-cli -m Ministral-3-3B-Instruct-2512-Q8_0.gguf \
            --mmproj Ministral-3-3B-Instruct-2512-BF16-mmproj.gguf \
            --image path/to/your/document.png \
            -p "Perform a highly accurate OCR transcript of this document."

2. Integrating Llama 3 8B with Local Vector DB (RAG)

Below is a standard conceptual architecture snippet using LangChain to query your vector backend:

from langchain_community.llms import LlamaCpp
from langchain_chains import RetrievalQA

# Load the Llama-3-8B vector database specialist
llm = LlamaCpp(model_path="./llama-3-8b.q4_K_M.gguf", n_ctx=8192)

# Querying your pre-loaded Vector Store
retriever = vector_db.as_retriever(search_kwargs={"k": 5})
qa_chain = RetrievalQA.from_chain_type(llm=llm, chain_type="stuff", retriever=retriever)

response = qa_chain.run("What does the internal database say about our 2026 Q3 policy?")

🌐 3. Hosting a Local OpenAI-Compatible API Server

You can host any GGUF model (such as the standard Qwen 3.5 9B) as a local background server using llama-server.exe on Windows. This opens an OpenAI-compatible endpoint that can be plugged into SillyTavern, Open WebUI, or custom apps.

Run the following command in your terminal (adjust the GGUF file path if necessary):

./llama-server.exe -m "Qwen3.5-9B-Q5_K_S-5.10bpw.gguf" \
                   -ngl 99 \
                   -c 32768 \
                   -np 1 \
                   -mg 0 \
                   --host 127.0.0.1 \
                   --port 8001

For draft model:

set MAIN_MODEL=.\gemma-4-E2B-it-qat-UD-Q4_K_XL.gguf
set DRAFT_MODEL=.\mtp-gemma-4-E2B-it-Q4_0.gguf

llama-server.exe -m "%MAIN_MODEL%" --model-draft "%DRAFT_MODEL%" --spec-type draft-mtp -ngl 99 -c 32768 -np 1 -mg 0 --host 127.0.0.1 --port 8001

Parameter Breakdown:

  • -ngl 99: Offloads 99 layers to the GPU (ensuring full GPU acceleration).
  • -c 32768: Expands the model's context window up to 32k tokens.
  • -np 1: Spawns 1 parallel processing slot.
  • --host 127.0.0.1 --port 8001: Sets up your local endpoint at http://127.0.0.1:8001.

⚡ 🖥️ Hardware Optimization Note (RTX 4060 8GB & Below)

💡 Best For Budget GPUs: Every model in this repository is heavily quantized and strictly optimized to run flawlessly with full GPU acceleration on NVIDIA RTX 4060 8GB and below (including RTX 3060, RTX 4050, and 6GB/8GB Laptop GPUs).

By utilizing efficient architectures and GGUF/MTP pairings, you will achieve blazing-fast inference speeds without ever hit any out-of-memory (OOM) limitations on 8GB VRAM hardware.

🏷️ SEO Keywords & Tags

local-ocr • llama-server-gguf • vector-db-querying • rag-llm-gguf • qwen3.5-gguf • standard-llm • ministral-ocr • llama3-rag • gguf-models • faster-whisper • local-ai-hub • uncensored-llm • offline-embeddings • low-vram-rag