| # BlitzKode Project Overview | |
| ## What this project is | |
| BlitzKode is a local coding-assistant application built around a fine-tuned Qwen2.5-1.5B GGUF model. The repository contains: | |
| - `server.py`: FastAPI backend that exposes API endpoints and performs llama.cpp inference. | |
| - `tests/`: backend endpoint tests with a stubbed llama.cpp model. | |
| - `scripts/`: dataset, training, checkpoint smoke-test, and GGUF export utilities. | |
| - `blitzkode.gguf`: local model artifact, intentionally ignored by git. | |
| The runtime design is intentionally small and local-first: API client -> FastAPI -> `llama-cpp-python` -> local GGUF model. | |
| ## Backend context | |
| `server.py` is the central orchestration layer. | |
| ### Request lifecycle | |
| 1. `create_app()` builds the FastAPI app and stores `Settings`, `ModelService`, and `WebSearchService` on `app.state`. | |
| 2. Middleware applies CORS, request-size limits, and optional per-IP rate limiting. | |
| 3. `/generate` and `/generate/stream` validate prompts, enforce optional bearer auth, and serialize model access with an `asyncio.Lock`. | |
| 4. `ModelService` lazily loads `blitzkode.gguf` through `llama_cpp.Llama` and formats requests with a Qwen-style ChatML prompt. | |
| 5. Streaming generation runs llama.cpp in a worker thread and bridges tokens into SSE messages through a queue. | |
| ### Inference efficiency controls | |
| The following environment variables control CPU/GPU usage: | |
| - `BLITZKODE_GPU_LAYERS`: offload layers to GPU when llama.cpp was built with GPU support. | |
| - `BLITZKODE_THREADS`: CPU decode thread count. | |
| - `BLITZKODE_THREADS_BATCH`: CPU prompt-processing thread count. | |
| - `BLITZKODE_N_CTX`: context window. | |
| - `BLITZKODE_BATCH`: llama.cpp prompt-processing batch size. | |
| - `BLITZKODE_UBATCH`: llama.cpp micro-batch size. | |
| - `BLITZKODE_PROMPT_CACHE`: in-memory prompt cache for repeated prefixes. | |
| - `BLITZKODE_PRELOAD_MODEL`: load model at startup instead of first request. | |
| For CPU-only machines, keep `BLITZKODE_GPU_LAYERS=0`, set `BLITZKODE_THREADS` close to physical performance cores, and use a quantized GGUF such as Q4_K_M for lower memory pressure. For GPU machines, increase `BLITZKODE_GPU_LAYERS` gradually until VRAM is stable. | |
| ## Search and research augmentation | |
| The backend now has optional web search primitives: | |
| - `POST /search/web`: DuckDuckGo Instant Answer search. | |
| - `POST /generate/research`: searches, injects snippets into the prompt as untrusted context, and generates an answer. | |
| API clients can use `/generate/stream` for normal token streaming or `/generate/research` when retrieval-augmented output is needed. | |
| Important safety behavior: | |
| - Search context is marked as untrusted. | |
| - The system prompt tells the model not to invent APIs, citations, files, or execution results. | |
| - Research mode asks the model to cite URLs when it relies on search results. | |
| ## Training pipeline context | |
| The scripts directory currently represents several generations of training experimentation: | |
| - `build_dataset.py` and `build_full_dataset.py`: local synthetic/curated dataset builders. | |
| - `train_sft.py`: baseline LoRA SFT over curated coding examples. | |
| - `train_reward_sft.py`: reward-inspired continuation, but not true GRPO. | |
| - `train_dpo.py`: preference optimization over handcrafted chosen/rejected samples. | |
| - `train_v2.py`, `train_continue.py`, `train_max.py`: larger public-dataset/QLoRA experiments. | |
| - `export_gguf.py`: merge LoRA adapters and prepare for llama.cpp conversion. | |
| - `test_inference.py`: smoke-test adapter checkpoints. | |
| The scripts are intentionally excluded from Ruff in `pyproject.toml`, so backend CI remains focused and fast. They should be treated as operator tools rather than library code. | |
| ## Research-informed optimization notes | |
| DeepSeek-R1 emphasizes staged post-training: cold-start reasoning data, RL with GRPO, rejection sampling, further SFT, and preference/RL alignment. DeepSeek-V3 emphasizes efficient architecture and training systems: MoE, MLA, FP8, multi-token prediction, and strong post-training/distillation. | |
| For this repository and a 1.5B Qwen-family model, the practical subset is: | |
| 1. Use high-quality SFT data first; small models are very sensitive to noisy data. | |
| 2. Add preference data with DPO or ORPO to reduce low-quality or hallucinated answers. | |
| 3. If GRPO is available in the installed TRL version, use verifiable coding rewards on unit-tested tasks rather than subjective string heuristics. | |
| 4. Distill reasoning behavior from larger open models only where licensing permits, and store provenance for every dataset. | |
| 5. Use QLoRA / 4-bit loading for efficient GPU training. | |
| 6. Keep inference GGUF quantized for CPU/GPU-local deployment. | |
| Do not indiscriminately scrape “all social/public internet” data. That creates legal, privacy, quality, and contamination issues. Prefer permissively licensed datasets and retain dataset cards/provenance. | |
| ## Cleanup completed in this pass | |
| - Removed `.mypy_cache/` generated cache. | |
| - Removed the malformed empty directory `USERPROFILE.ollamamodels/`. | |
| - Added tested web search and research-generation endpoints. | |
| - Removed the React/Vite frontend and made the project API-only. | |
| - Tightened request models for message roles and history size. | |
| - Hardened request-size middleware against invalid `Content-Length`. | |
| - Added tests for search, disabled search, and research prompt injection. | |
| - Updated README API/environment documentation. | |
| ## Current verification commands | |
| - `python -m pytest tests/ -v` — 22 backend endpoint tests | |
| - `python -m ruff check .` | |
| - `python -m mypy server.py --ignore-missing-imports` | |
| Run these after backend changes. | |
| ## Production quality audit results (v2.1 — Q8_0 GGUF, 1.53 GB) | |
| All tests ran on RTX 4060 Laptop GPU, using CPU inference, load time 0.35 s. | |
| | Task | Result | Notes | | |
| |---|---|---| | |
| | Floyd cycle detection | ✅ Correct | O(n)/O(1) with explanation | | |
| | Flatten nested list | ✅ Correct | Recursive, clean | | |
| | Kadane's max subarray | ✅ Correct | With complexity analysis | | |
| | Dijkstra shortest path | ✅ Correct | Heap-based, full implementation | | |
| | Fix zero-division bug | ✅ Correct | Detected and fixed | | |
| | Refactor to context manager | ✅ Correct | One-liner | | |
| | Decorator explanation | ✅ Correct | 2-sentence, accurate | | |
| | JavaScript debounce | ✅ Correct | With explanation | | |
| | SQL top-5 users | ✅ Correct | GROUP BY + ORDER BY + LIMIT | | |
| | BFS traversal | ✅ Correct | deque-based | | |
| | TypeScript typed fetch | ⚠️ Partial | Mixes RxJS/fetch patterns | | |
| | Refactor global→class | ⚠️ Weak | Didn't complete the OOP refactor | | |
| | Non-existent API hallucination | ❌ Hallucinated | Small model limitation | | |
| **Root causes and mitigations:** | |
| 1. **TypeScript/OOP refactor gaps** — training data was Python-heavy. Adding more TypeScript and OOP refactor examples to `datasets/raw/available_training.jsonl` and re-running `train_available.py` with `--max-steps 200` would improve these. | |
| 2. **Hallucination of fictional APIs** — this is a fundamental 1.5B model limitation. The system prompt already instructs against it (`Do not invent APIs…`) but small models don't reliably follow system instructions they weren't trained on. Mitigations: add negative examples to DPO pairs, or use the `/generate/research` endpoint to ground answers in real web results. | |
| 3. **n_ctx warning** — `n_ctx_per_seq (2048) < n_ctx_train (32768)` is informational only. For most coding tasks 2048 tokens is sufficient. Increase `BLITZKODE_N_CTX` if you need longer code generation. | |
| ## Next recommended work | |
| 1. **Expand dataset** — add TypeScript, OOP refactoring, and "I don't know" (grounding) examples to `datasets/raw/available_training.jsonl`, then re-run `train_available.py --max-steps 300`. | |
| 2. **DPO for hallucination** — add preference pairs where `chosen = "I don't know, here's how to check"` and `rejected = <hallucinated answer>`, run `train_dpo.py`. | |
| 3. **API contract tests** — add tests for OpenAPI schema stability and streaming client behavior. | |
| 4. **Search cache tuning** — tune `BLITZKODE_SEARCH_CACHE_TTL` based on expected research workload. | |
| 5. **Context window** — increase `BLITZKODE_N_CTX=4096` in production for longer codebases, or `8192` with GPU offload. | |
| 6. **GRPO reward training** — use TRL `GRPOTrainer` with unit-test-verified coding rewards once a suitable test harness is available. | |