Spaces:
Sleeping
Sleeping
| # Local LLM Inference — Flask API (llama-cpp-python + ChromaDB Memory) | |
| Multi-model support: hanggang 2 models sabay sa RAM. Switch anytime via CLI o API. | |
| Memory: **ChromaDB** + **sentence-transformers/all-MiniLM-L6-v2** | |
| No GPU required. GGUF only — no safetensors, no torch dependency. | |
| ## Architecture | |
| ``` | |
| app.py (Flask routes + multi-backend dict + mode system + memory + streaming) | |
| ├── gguf_backend.py (GGUFBackend — llama-cpp-python wrapper, join chunks → load) | |
| │ └── model/ | |
| │ ├── Qwen3.5-4B/ | |
| │ │ └── chunks/ (100MB layer-split shards) | |
| │ └── DeepSeek-R1-8B/ | |
| │ └── chunks/ (100MB layer-split shards) | |
| ├── model_manager.py (multi-model CLI — list, use, remove, download, quants) | |
| ├── memory.py (ChromaDB + sentence-transformers RAG) | |
| │ └── memory_db/ (persistent vector store) | |
| ├── split_gguf.py (utility: split GGUF into layer-aware shards) | |
| └── install.json (all config — active_models list, model list, server settings) | |
| ``` | |
| ## Quick Start | |
| ```bash | |
| bash install.sh | |
| ``` | |
| ## Models (install.json) | |
| | Name | Repo | Default File | Size | Notes | | |
| |---|---|---|---|---| | |
| | **Qwen3.5-4B** *(active)* | unsloth/Qwen3.5-4B-GGUF | IQ4_NL | ~2.6GB | Mabilis, magandang balanse | | |
| | **DeepSeek-R1-8B** *(active)* | unsloth/DeepSeek-R1-Distill-Llama-8B-GGUF | Q4_K_M | ~4.6GB | Malalim na reasoning, R1 distill | | |
| | Qwen3-1.7B | unsloth/Qwen3-1.7B-GGUF | Q4_K_M | ~1.0GB | Pinakamabilis | | |
| | Llama-3.1-8B | unsloth/Llama-3.1-8B-Instruct-GGUF | Q4_K_M | ~4.6GB | General purpose | | |
| | Phi-4-mini | unsloth/Phi-4-mini-reasoning-GGUF | Q4_K_M | ~2.3GB | Math/reasoning | | |
| ## Model Manager CLI | |
| ```bash | |
| python model_manager.py list # Lahat ng models + disk status | |
| python model_manager.py quants Qwen3.5-4B # All quants mula sa HuggingFace | |
| python model_manager.py quants DeepSeek-R1-8B | |
| python model_manager.py use Llama-3.1-8B # Magdagdag ng active model (max 2) | |
| python model_manager.py use DeepSeek-R1-8B Q3_K_M # Switch + ibang quantization | |
| python model_manager.py remove DeepSeek-R1-8B # Alisin sa active list | |
| python model_manager.py download # I-download ang lahat ng active | |
| python model_manager.py info # Dump ng buong install.json | |
| ``` | |
| Pagkatapos mag-switch ng model, i-restart ang server: | |
| ```bash | |
| python app.py | |
| # o sa Replit: i-restart ang "Start application" workflow | |
| ``` | |
| ## API Endpoints | |
| | Method | Path | Notes | | |
| |--------|------|-------| | |
| | GET | `/` | Index + examples + mode docs | | |
| | GET | `/health` | Status ng lahat ng loaded models | | |
| | GET/POST | `/chat` | JSON reply | | |
| | GET/POST | `/chat/stream` | Token-by-token streaming | | |
| | GET | `/memory?session=X` | List conversation memory | | |
| | POST | `/memory/clear` | Clear session memory | | |
| | GET | `/models` | Lahat ng configured models + disk status | | |
| | GET | `/models/quants?model=X` | HuggingFace quantization list | | |
| ## Params | |
| | Param | Values | Default | Notes | | |
| |---|---|---|---| | |
| | `message` | string | required | Prompt | | |
| | `model` | model name | first active | Piliin ang model (e.g. `?model=Qwen3.5-4B`) | | |
| | `mode` | fast\|thinking\|balanced\|code\|auto | `auto` | Mode shortcut | | |
| | `max_tokens` | int \| `auto` | from mode | Overrides mode budget | | |
| | `greedy` | 1/true | from mode | Deterministic decoding | | |
| | `thinking` | 1/true | false | Same as `mode=thinking` | | |
| | `raw` | 1/true | false | No system prompt (jailbreak) | | |
| | `session` | string | `default` | Memory namespace | | |
| | `no_memory` | 1/true | false | Skip memory lookup | | |
| ## curl Examples | |
| ```bash | |
| # Default model (Qwen3.5-4B) | |
| curl "http://localhost:5000/chat?message=hi&mode=fast" | |
| # Specific model | |
| curl "http://localhost:5000/chat?message=solve+fibonacci&model=DeepSeek-R1-8B&mode=thinking" | |
| curl "http://localhost:5000/chat?message=write+hello+world&model=Qwen3.5-4B&mode=code" | |
| # Streaming | |
| curl "http://localhost:5000/chat/stream?message=explain+AI&model=Qwen3.5-4B" | |
| # Health + models | |
| curl "http://localhost:5000/health" | |
| curl "http://localhost:5000/models" | |
| ``` | |
| ## Mode System | |
| | Mode | Tokens | Greedy | ~Speed | | |
| |---|---|---|---| | |
| | `fast` | 120 | yes | ~1-2s | | |
| | `thinking` | 800 | no | ~20-40s | | |
| | `thinking_fast` | 400 | no | ~10-20s | | |
| | `balanced` | auto | auto | ~3-10s | | |
| | `code` | 600 | no | ~5-15s | | |
| | `auto` | auto | auto | ~3-10s | | |
| ## GGUF Split Loading | |
| - Model chunks sa `model/{Name}/chunks/` bilang `ModelFile-00001-of-NNNNN.gguf` | |
| - Sa startup, ang chunks ay ni-jo-join sa temp whole GGUF → ini-load → tinatanggal ang temp | |
| - Split utility: `python split_gguf.py split_layers` | |
| ## SIGILL Protection (x86_64) | |
| - `gguf_backend.py` checks CPU flags + inspects `.so` with objdump before loading | |
| - If AVX2/AVX512 mismatch detected → raises clear error with recompile command | |
| - `install.sh` detects mismatch at install time → auto-recompiles with safe flags | |
| ## Performance Settings | |
| | Setting | Value | Effect | | |
| |---|---|---| | |
| | `n_batch` | 1024 | Faster prompt prefill | | |
| | `use_mlock` | True | Lock model in RAM (no swapping) | | |
| | `n_threads` | CPU count | All cores | | |
| | `n_ctx` | 4096 | Context window | | |
| ## Memory System (ChromaDB RAG) | |
| - Every response stored as vector embedding | |
| - Top-2 semantically similar past turns injected as context | |
| - Persistent on disk (`./memory_db/`) | |
| - Multi-session: `?session=your_id` | |
| ## Platform Support | |
| Termux · Linux · macOS · Windows (WSL/Git Bash) · Replit · Docker | |
| ## User Preferences | |
| - Prefers Tagalog/Filipino communication | |
| - llama-cpp-python (GGUF Q4_K_M) as primary backend — fastest | |
| - **GGUF only** — no safetensors, no torch, no ChunkedModel | |
| - **No response cache** — every request generates fresh output | |
| - `?raw=1` jailbreak mode — no system prompt | |
| - `mode=fast/thinking/balanced/code/auto` shortcuts | |
| - `/no_think` auto-appended for non-thinking modes (Qwen3 GGUF) | |
| - ChromaDB persistent memory (all sessions) | |
| - True token streaming via `/chat/stream` | |
| - Cross-platform install via `bash install.sh` | |
| - Split chunks = permanent storage, join to temp on load | |
| - Multi-model: up to 2 models loaded simultaneously, `?model=` param to pick | |
| - New directory: `model/{ModelName}/chunks/` per model | |