replit.md

Local LLM Inference — Flask API (llama-cpp-python + ChromaDB Memory)

Multi-model support: hanggang 2 models sabay sa RAM. Switch anytime via CLI o API. Memory: ChromaDB + sentence-transformers/all-MiniLM-L6-v2

No GPU required. GGUF only — no safetensors, no torch dependency.

Architecture

app.py                  (Flask routes + multi-backend dict + mode system + memory + streaming)
  ├── gguf_backend.py   (GGUFBackend — llama-cpp-python wrapper, join chunks → load)
  │     └── model/
  │           ├── Qwen3.5-4B/
  │           │     └── chunks/  (100MB layer-split shards)
  │           └── DeepSeek-R1-8B/
  │                 └── chunks/  (100MB layer-split shards)
  ├── model_manager.py  (multi-model CLI — list, use, remove, download, quants)
  ├── memory.py         (ChromaDB + sentence-transformers RAG)
  │     └── memory_db/  (persistent vector store)
  ├── split_gguf.py     (utility: split GGUF into layer-aware shards)
  └── install.json      (all config — active_models list, model list, server settings)

Quick Start

bash install.sh

Models (install.json)

Name	Repo	Default File	Size	Notes
Qwen3.5-4B (active)	unsloth/Qwen3.5-4B-GGUF	IQ4_NL	~2.6GB	Mabilis, magandang balanse
DeepSeek-R1-8B (active)	unsloth/DeepSeek-R1-Distill-Llama-8B-GGUF	Q4_K_M	~4.6GB	Malalim na reasoning, R1 distill
Qwen3-1.7B	unsloth/Qwen3-1.7B-GGUF	Q4_K_M	~1.0GB	Pinakamabilis
Llama-3.1-8B	unsloth/Llama-3.1-8B-Instruct-GGUF	Q4_K_M	~4.6GB	General purpose
Phi-4-mini	unsloth/Phi-4-mini-reasoning-GGUF	Q4_K_M	~2.3GB	Math/reasoning

Model Manager CLI

python model_manager.py list                          # Lahat ng models + disk status
python model_manager.py quants Qwen3.5-4B             # All quants mula sa HuggingFace
python model_manager.py quants DeepSeek-R1-8B
python model_manager.py use Llama-3.1-8B              # Magdagdag ng active model (max 2)
python model_manager.py use DeepSeek-R1-8B Q3_K_M    # Switch + ibang quantization
python model_manager.py remove DeepSeek-R1-8B         # Alisin sa active list
python model_manager.py download                      # I-download ang lahat ng active
python model_manager.py info                          # Dump ng buong install.json

Pagkatapos mag-switch ng model, i-restart ang server:

python app.py
# o sa Replit: i-restart ang "Start application" workflow

API Endpoints

Method	Path	Notes
GET	/	Index + examples + mode docs
GET	/health	Status ng lahat ng loaded models
GET/POST	/chat	JSON reply
GET/POST	/chat/stream	Token-by-token streaming
GET	/memory?session=X	List conversation memory
POST	/memory/clear	Clear session memory
GET	/models	Lahat ng configured models + disk status
GET	/models/quants?model=X	HuggingFace quantization list

Params

Param	Values	Default	Notes
message	string	required	Prompt
model	model name	first active	Piliin ang model (e.g. ?model=Qwen3.5-4B)
mode	fast|thinking|balanced|code|auto	auto	Mode shortcut
max_tokens	int | auto	from mode	Overrides mode budget
greedy	1/true	from mode	Deterministic decoding
thinking	1/true	false	Same as mode=thinking
raw	1/true	false	No system prompt (jailbreak)
session	string	default	Memory namespace
no_memory	1/true	false	Skip memory lookup

curl Examples

# Default model (Qwen3.5-4B)
curl "http://localhost:5000/chat?message=hi&mode=fast"

# Specific model
curl "http://localhost:5000/chat?message=solve+fibonacci&model=DeepSeek-R1-8B&mode=thinking"
curl "http://localhost:5000/chat?message=write+hello+world&model=Qwen3.5-4B&mode=code"

# Streaming
curl "http://localhost:5000/chat/stream?message=explain+AI&model=Qwen3.5-4B"

# Health + models
curl "http://localhost:5000/health"
curl "http://localhost:5000/models"

Mode System

Mode	Tokens	Greedy	~Speed
fast	120	yes	~1-2s
thinking	800	no	~20-40s
thinking_fast	400	no	~10-20s
balanced	auto	auto	~3-10s
code	600	no	~5-15s
auto	auto	auto	~3-10s

GGUF Split Loading

• Model chunks sa model/{Name}/chunks/ bilang ModelFile-00001-of-NNNNN.gguf
• Sa startup, ang chunks ay ni-jo-join sa temp whole GGUF → ini-load → tinatanggal ang temp
• Split utility: python split_gguf.py split_layers

SIGILL Protection (x86_64)

• gguf_backend.py checks CPU flags + inspects .so with objdump before loading
• If AVX2/AVX512 mismatch detected → raises clear error with recompile command
• install.sh detects mismatch at install time → auto-recompiles with safe flags

Performance Settings

Setting	Value	Effect
n_batch	1024	Faster prompt prefill
use_mlock	True	Lock model in RAM (no swapping)
n_threads	CPU count	All cores
n_ctx	4096	Context window

Memory System (ChromaDB RAG)

• Every response stored as vector embedding
• Top-2 semantically similar past turns injected as context
• Persistent on disk (./memory_db/)
• Multi-session: ?session=your_id

Platform Support

Termux · Linux · macOS · Windows (WSL/Git Bash) · Replit · Docker

User Preferences

• Prefers Tagalog/Filipino communication
• llama-cpp-python (GGUF Q4_K_M) as primary backend — fastest
• GGUF only — no safetensors, no torch, no ChunkedModel
• No response cache — every request generates fresh output
• ?raw=1 jailbreak mode — no system prompt
• mode=fast/thinking/balanced/code/auto shortcuts
• /no_think auto-appended for non-thinking modes (Qwen3 GGUF)
• ChromaDB persistent memory (all sessions)
• True token streaming via /chat/stream
• Cross-platform install via bash install.sh
• Split chunks = permanent storage, join to temp on load
• Multi-model: up to 2 models loaded simultaneously, ?model= param to pick
• New directory: model/{ModelName}/chunks/ per model