# Local LLM Inference — Flask API (llama-cpp-python + ChromaDB Memory)

Multi-model support: hanggang 2 models sabay sa RAM. Switch anytime via CLI o API.
Memory: **ChromaDB** + **sentence-transformers/all-MiniLM-L6-v2**

No GPU required. GGUF only — no safetensors, no torch dependency.

## Architecture

```
app.py                  (Flask routes + multi-backend dict + mode system + memory + streaming)
  ├── gguf_backend.py   (GGUFBackend — llama-cpp-python wrapper, join chunks → load)
  │     └── model/
  │           ├── Qwen3.5-4B/
  │           │     └── chunks/  (100MB layer-split shards)
  │           └── DeepSeek-R1-8B/
  │                 └── chunks/  (100MB layer-split shards)
  ├── model_manager.py  (multi-model CLI — list, use, remove, download, quants)
  ├── memory.py         (ChromaDB + sentence-transformers RAG)
  │     └── memory_db/  (persistent vector store)
  ├── split_gguf.py     (utility: split GGUF into layer-aware shards)
  └── install.json      (all config — active_models list, model list, server settings)
```

## Quick Start

```bash
bash install.sh
```

## Models (install.json)

| Name | Repo | Default File | Size | Notes |
|---|---|---|---|---|
| **Qwen3.5-4B** *(active)* | unsloth/Qwen3.5-4B-GGUF | IQ4_NL | ~2.6GB | Mabilis, magandang balanse |
| **DeepSeek-R1-8B** *(active)* | unsloth/DeepSeek-R1-Distill-Llama-8B-GGUF | Q4_K_M | ~4.6GB | Malalim na reasoning, R1 distill |
| Qwen3-1.7B | unsloth/Qwen3-1.7B-GGUF | Q4_K_M | ~1.0GB | Pinakamabilis |
| Llama-3.1-8B | unsloth/Llama-3.1-8B-Instruct-GGUF | Q4_K_M | ~4.6GB | General purpose |
| Phi-4-mini | unsloth/Phi-4-mini-reasoning-GGUF | Q4_K_M | ~2.3GB | Math/reasoning |

## Model Manager CLI

```bash
python model_manager.py list                          # Lahat ng models + disk status
python model_manager.py quants Qwen3.5-4B             # All quants mula sa HuggingFace
python model_manager.py quants DeepSeek-R1-8B
python model_manager.py use Llama-3.1-8B              # Magdagdag ng active model (max 2)
python model_manager.py use DeepSeek-R1-8B Q3_K_M    # Switch + ibang quantization
python model_manager.py remove DeepSeek-R1-8B         # Alisin sa active list
python model_manager.py download                      # I-download ang lahat ng active
python model_manager.py info                          # Dump ng buong install.json
```

Pagkatapos mag-switch ng model, i-restart ang server:
```bash
python app.py
# o sa Replit: i-restart ang "Start application" workflow
```

## API Endpoints

| Method | Path | Notes |
|--------|------|-------|
| GET | `/` | Index + examples + mode docs |
| GET | `/health` | Status ng lahat ng loaded models |
| GET/POST | `/chat` | JSON reply |
| GET/POST | `/chat/stream` | Token-by-token streaming |
| GET | `/memory?session=X` | List conversation memory |
| POST | `/memory/clear` | Clear session memory |
| GET | `/models` | Lahat ng configured models + disk status |
| GET | `/models/quants?model=X` | HuggingFace quantization list |

## Params

| Param | Values | Default | Notes |
|---|---|---|---|
| `message` | string | required | Prompt |
| `model` | model name | first active | Piliin ang model (e.g. `?model=Qwen3.5-4B`) |
| `mode` | fast\|thinking\|balanced\|code\|auto | `auto` | Mode shortcut |
| `max_tokens` | int \| `auto` | from mode | Overrides mode budget |
| `greedy` | 1/true | from mode | Deterministic decoding |
| `thinking` | 1/true | false | Same as `mode=thinking` |
| `raw` | 1/true | false | No system prompt (jailbreak) |
| `session` | string | `default` | Memory namespace |
| `no_memory` | 1/true | false | Skip memory lookup |

## curl Examples

```bash
# Default model (Qwen3.5-4B)
curl "http://localhost:5000/chat?message=hi&mode=fast"

# Specific model
curl "http://localhost:5000/chat?message=solve+fibonacci&model=DeepSeek-R1-8B&mode=thinking"
curl "http://localhost:5000/chat?message=write+hello+world&model=Qwen3.5-4B&mode=code"

# Streaming
curl "http://localhost:5000/chat/stream?message=explain+AI&model=Qwen3.5-4B"

# Health + models
curl "http://localhost:5000/health"
curl "http://localhost:5000/models"
```

## Mode System

| Mode | Tokens | Greedy | ~Speed |
|---|---|---|---|
| `fast` | 120 | yes | ~1-2s |
| `thinking` | 800 | no | ~20-40s |
| `thinking_fast` | 400 | no | ~10-20s |
| `balanced` | auto | auto | ~3-10s |
| `code` | 600 | no | ~5-15s |
| `auto` | auto | auto | ~3-10s |

## GGUF Split Loading

- Model chunks sa `model/{Name}/chunks/` bilang `ModelFile-00001-of-NNNNN.gguf`
- Sa startup, ang chunks ay ni-jo-join sa temp whole GGUF → ini-load → tinatanggal ang temp
- Split utility: `python split_gguf.py split_layers`

## SIGILL Protection (x86_64)

- `gguf_backend.py` checks CPU flags + inspects `.so` with objdump before loading
- If AVX2/AVX512 mismatch detected → raises clear error with recompile command
- `install.sh` detects mismatch at install time → auto-recompiles with safe flags

## Performance Settings

| Setting | Value | Effect |
|---|---|---|
| `n_batch` | 1024 | Faster prompt prefill |
| `use_mlock` | True | Lock model in RAM (no swapping) |
| `n_threads` | CPU count | All cores |
| `n_ctx` | 4096 | Context window |

## Memory System (ChromaDB RAG)

- Every response stored as vector embedding
- Top-2 semantically similar past turns injected as context
- Persistent on disk (`./memory_db/`)
- Multi-session: `?session=your_id`

## Platform Support

Termux · Linux · macOS · Windows (WSL/Git Bash) · Replit · Docker

## User Preferences

- Prefers Tagalog/Filipino communication
- llama-cpp-python (GGUF Q4_K_M) as primary backend — fastest
- **GGUF only** — no safetensors, no torch, no ChunkedModel
- **No response cache** — every request generates fresh output
- `?raw=1` jailbreak mode — no system prompt
- `mode=fast/thinking/balanced/code/auto` shortcuts
- `/no_think` auto-appended for non-thinking modes (Qwen3 GGUF)
- ChromaDB persistent memory (all sessions)
- True token streaming via `/chat/stream`
- Cross-platform install via `bash install.sh`
- Split chunks = permanent storage, join to temp on load
- Multi-model: up to 2 models loaded simultaneously, `?model=` param to pick
- New directory: `model/{ModelName}/chunks/` per model