Update clean backend-only project docs and eval

README.md

@@ -19,7 +19,7 @@ base_model:
 
 # BlitzKode
 
-BlitzKode is a local AI coding assistant powered by a fine-tuned Qwen2.5-1.5B-Instruct model. It runs entirely on your machine — no external API calls, no data leaving your device.
+BlitzKode is a local API-first AI coding assistant powered by a fine-tuned Qwen2.5-1.5B-Instruct model. It runs on your machine through `llama-cpp-python` with no external model API calls.
 
 ## Tech Stack
 
@@ -30,24 +30,21 @@ BlitzKode is a local AI coding assistant powered by a fine-tuned Qwen2.5-1.5B-In
 | Training | HuggingFace Transformers + TRL |
 | Inference | llama-cpp-python (GGUF Q8_0) |
 | Backend | Python 3.11+, FastAPI, uvicorn |
-| Frontend | React 18, Vite, Phosphor Icons |
 
 ## Features
 
-- **Local-first** — inference with the bundled GGUF, no cloud dependency
-- **Real-time streaming** — SSE token-by-token via `/generate/stream`
-- **Web research mode** — DuckDuckGo search → context-augmented generation via `/generate/research`
-- **Web search API** — standalone `/search/web` endpoint for raw results
-- **React chat UI** — streaming, conversation history, copy controls, research-mode toggle
-- **Multi-language** — Python, JavaScript, Java, C++, TypeScript, SQL
-- **API key auth + rate limiting** — production-ready security middleware
-- **Docker** — multi-stage production image with frontend baked in
+- **Local-first inference** with the bundled GGUF model
+- **FastAPI backend only** with `/generate`, `/generate/stream`, `/generate/research`, `/search/web`, `/health`, and `/info`
+- **Real-time streaming** via Server-Sent Events on `/generate/stream`
+- **Web research mode** using DuckDuckGo search context before generation
+- **API key auth, request-size limits, and rate limiting** for production use
+- **Backend/model optimizations**: mmap model loading, configurable GPU layer offload, batch/thread tuning, optional prompt cache, search-result TTL caching, and efficient deque-based rate limiting
+- **Docker** runtime image without Node.js/frontend build steps
 
 ## Prerequisites
 
 - Python 3.11+
-- Node.js 20+ (for frontend dev/builds only)
-- `blitzkode.gguf` at repo root (or set `BLITZKODE_MODEL_PATH`)
+- `blitzkode.gguf` at repo root, or set `BLITZKODE_MODEL_PATH`
 - 4 GB+ RAM
 
 ## Quick Start
@@ -55,26 +52,9 @@ BlitzKode is a local AI coding assistant powered by a fine-tuned Qwen2.5-1.5B-In
 ```bash
 pip install -r requirements.txt
 python server.py
-# Open http://localhost:7860
-```
-
-## Frontend Development
-
-```bash
-cd frontend
-npm install
-npm run dev      # http://localhost:5173 — proxies /generate and /health to :7860
-```
-
-## Production Frontend Build
-
-```bash
-cd frontend && npm install && npm run build && cd ..
-python server.py
+curl http://localhost:7860/health
 ```
 
-FastAPI serves `frontend/dist/index.html` and `/assets/*` from the same port.
-
 ## Docker
 
 ```bash
@@ -104,7 +84,7 @@ curl -X POST http://localhost:7860/search/web \
   -H "Content-Type: application/json" \
   -d '{"query":"FastAPI dependency injection","max_results":3}'
 
-# Research-augmented generation (search → inject → answer)
+# Research-augmented generation
 curl -X POST http://localhost:7860/generate/research \
   -H "Content-Type: application/json" \
   -d '{"prompt":"How do I use async generators in Python 3.12?","deep_search":true}'
@@ -130,7 +110,7 @@ curl http://localhost:7860/info
 
 ### Research (`/generate/research`)
 
-Same as above, plus:
+Same as generation, plus:
 
 | Parameter | Type | Default | Description |
 |---|---|---|---|
@@ -151,100 +131,94 @@ Same as above, plus:
 | Variable | Default | Description |
 |---|---|---|
 | `BLITZKODE_MODEL_PATH` | `blitzkode.gguf` | GGUF model path |
-| `BLITZKODE_FRONTEND_PATH` | `frontend/dist/index.html` | Built frontend |
 | `BLITZKODE_HOST` | `0.0.0.0` | Server bind address |
 | `BLITZKODE_PORT` | `7860` | Server port |
-| `BLITZKODE_GPU_LAYERS` | `0` | GPU layers for llama.cpp |
+| `BLITZKODE_GPU_LAYERS` | `0` | GPU layers for llama.cpp; use `-1` to offload all supported layers |
 | `BLITZKODE_N_CTX` | `2048` | Context window |
-| `BLITZKODE_THREADS` | auto | CPU worker threads |
-| `BLITZKODE_BATCH` | `128` | Batch size |
+| `BLITZKODE_THREADS` | auto | CPU decode threads |
+| `BLITZKODE_THREADS_BATCH` | auto | CPU prompt-processing threads |
+| `BLITZKODE_BATCH` | `256` | Prompt-processing batch size |
+| `BLITZKODE_UBATCH` | `128` | llama.cpp micro-batch size |
+| `BLITZKODE_PROMPT_CACHE` | `true` | Enable llama.cpp in-memory prompt cache when supported |
+| `BLITZKODE_PROMPT_CACHE_BYTES` | `67108864` | Prompt cache capacity in bytes |
+| `BLITZKODE_USE_MMAP` | `true` | Memory-map the GGUF for faster startup and lower memory pressure |
+| `BLITZKODE_USE_MLOCK` | `false` | Try to lock model pages in RAM |
+| `BLITZKODE_OFFLOAD_KQV` | `true` | Offload K/Q/V operations when GPU layers are enabled |
 | `BLITZKODE_MAX_PROMPT_LENGTH` | `4000` | Max prompt chars |
 | `BLITZKODE_PRELOAD_MODEL` | `false` | Load model at startup |
 | `BLITZKODE_CORS_ORIGINS` | `http://localhost:7860` | CORS origins |
 | `BLITZKODE_API_KEY` | empty | Optional bearer token |
 | `BLITZKODE_WEB_SEARCH` | `true` | Enable web search endpoints |
-| `BLITZKODE_SEARCH_TIMEOUT` | `8` | Search HTTP timeout (s) |
+| `BLITZKODE_SEARCH_TIMEOUT` | `8` | Search HTTP timeout in seconds |
 | `BLITZKODE_MAX_SEARCH_RESULTS` | `5` | Max search results |
+| `BLITZKODE_SEARCH_CACHE_TTL` | `300` | Search result cache TTL in seconds |
 | `BLITZKODE_RATE_LIMIT` | `true` | Enable per-IP rate limiting |
 | `BLITZKODE_RATE_LIMIT_MAX` | `30` | Requests per IP per minute |
 | `BLITZKODE_MAX_REQUEST_BYTES` | `50000` | Request body size limit |
 
+## Model Evaluation
+
+Latest local GGUF evaluation: **2026-05-16** using `python scripts/evaluate_model.py` on CPU (`n_ctx=2048`, `threads=8`, `batch=256`, `gpu_layers=0`). Full machine-readable results are stored in `docs/evaluation_results.json`.
+
+| Eval case | Result | Notes |
+|---|---:|---|
+| Python factorial with negative-input handling | ✅ Pass | Generated a correct iterative implementation with `ValueError` for negative input. |
+| Iterative binary search | ✅ Pass | Generated a valid loop-based search returning index or `-1`. |
+| SQL top users by order count | ✅ Pass | Generated `JOIN`, `GROUP BY`, `ORDER BY`, and `LIMIT 5`. |
+| Unknown fictional API uncertainty | ❌ Fail | The raw model hallucinated a plausible signature for `imaginary_blitz_api`; the backend guard still blocks direct unknown-signature prompts on `/generate` and `/generate/stream`. |
+
+Summary: **3 / 4 passed (75%)**. Total generation time was **28.864 s** after a **0.312 s** model load. Evaluation-of-the-evaluation: this is a lightweight heuristic smoke eval, not a comprehensive benchmark; it is useful for regression tracking and quick sanity checks, but code should still be reviewed and tested. Future eval work should add executable unit tests for generated code and larger benchmark suites such as HumanEval/MBPP-style tasks.
+
 ## Training Pipeline
 
 BlitzKode was fine-tuned through a staged pipeline on an RTX 4060 (8 GB VRAM):
 
 | Stage | Script | Details |
 |---|---|---|
-| SFT v1 | `train_sft.py` | LoRA r=32 on 24 curated coding examples |
+| SFT v1 | `train_sft.py` | LoRA r=32 on curated coding examples |
 | Reward-SFT | `train_reward_sft.py` | Reward-heuristic continuation |
-| DPO | `train_dpo.py` | 10 chosen/rejected preference pairs |
-| SFT v2 | `train_available.py` | LoRA r=16, 100 steps, 99 samples (1.5B) |
+| DPO | `train_dpo.py` | Chosen/rejected preference pairs |
+| SFT v2 | `train_available.py` | LoRA r=16 resource-aware training |
 | Export | `export_production.py` | Merge → GGUF Q8_0 via llama.cpp |
 
 ### Re-train from scratch
 
 ```bash
 pip install -r requirements-training.txt
-
-# Build dataset
 python scripts/build_full_dataset.py
-
-# Train 1.5B LoRA (100 steps, ~5 min on RTX 4060)
 python scripts/train_available.py \
   --model Qwen/Qwen2.5-1.5B-Instruct \
   --quantization none \
   --dataset datasets/raw/blitzkode_full_training.json \
   --max-steps 100 --seq-len 384 --batch-size 1 --grad-accum 8
-
-# Export: merge + GGUF
 python scripts/export_production.py
 ```
 
-### Push to HuggingFace
-
-```bash
-export HF_TOKEN=hf_XXXX        # get from https://huggingface.co/settings/tokens
-python scripts/push_all_to_hub.py
-```
-
-This uploads:
-- `checkpoints/blitzkode-1.5b-lora/final` → `neuralbroker/blitzkode-1.5b-lora`
-- `checkpoints/available-lora-0.5b-full/final` → `neuralbroker/blitzkode-lora-0.5b`
-- `blitzkode.gguf` → `neuralbroker/blitzkode`
-
 ## Project Structure
 
 ```text
 BlitzKode/
   server.py                    FastAPI backend
   blitzkode.gguf               Local GGUF model (ignored by git)
-  frontend/                    React/Vite web UI
-    src/App.jsx                Chat UI with streaming + research toggle
-    src/index.css
-    vite.config.js
-  scripts/
-    train_available.py         Resource-aware LoRA training
-    build_full_dataset.py      Dataset builder
-    export_production.py       Merge LoRA → GGUF
-    push_to_hub.py             Single-adapter HF push
-    push_all_to_hub.py         Push all artifacts in one command
-    test_inference.py          Adapter smoke test
-    healthcheck.sh             Docker/Compose health probe
-  tests/test_server.py         20 backend endpoint tests (all passing)
+  scripts/                     Training, export, evaluation, and utility scripts
+  docs/evaluation_results.json Latest local model evaluation output
+  tests/test_server.py         Backend endpoint tests
   datasets/MANIFEST.md         Dataset provenance
-  docs/PROJECT_OVERVIEW.md     Architecture & roadmap
-  Dockerfile                   Multi-stage production image
+  docs/                        Architecture and production docs
+  Dockerfile                   Python runtime image
   docker-compose.yml           CPU + GPU service definitions
   requirements.txt             Serving dependencies
-  requirements-training.txt    Training dependencies (pinned)
+  requirements-training.txt    Training dependencies
 ```
 
 ## CI
 
 ```bash
-python -m pytest tests/ -v          # 20 tests, all pass
-python -m ruff check .              # lint
-npm --prefix frontend run build     # frontend build
+python -m pytest tests/ -v
+python -m ruff check .
+python -m mypy server.py --ignore-missing-imports
+python scripts/evaluate_model.py
+docker build -t blitzkode:ci .
 ```
 
 ## License