docs/PROJECT_OVERVIEW.md

# BlitzKode Project Overview

## What this project is

BlitzKode is a local coding-assistant application built around a fine-tuned Qwen2.5-1.5B GGUF model. The repository contains:

- `server.py`: FastAPI backend that exposes API endpoints and performs llama.cpp inference.
- `tests/`: backend endpoint tests with a stubbed llama.cpp model.
- `scripts/`: dataset, training, checkpoint smoke-test, and GGUF export utilities.
- `blitzkode.gguf`: local model artifact, intentionally ignored by git.

The runtime design is intentionally small and local-first: API client -> FastAPI -> `llama-cpp-python` -> local GGUF model.

## Backend context

`server.py` is the central orchestration layer.

### Request lifecycle

1. `create_app()` builds the FastAPI app and stores `Settings`, `ModelService`, and `WebSearchService` on `app.state`.
2. Middleware applies CORS, request-size limits, and optional per-IP rate limiting.
3. `/generate` and `/generate/stream` validate prompts, enforce optional bearer auth, and serialize model access with an `asyncio.Lock`.
4. `ModelService` lazily loads `blitzkode.gguf` through `llama_cpp.Llama` and formats requests with a Qwen-style ChatML prompt.
5. Streaming generation runs llama.cpp in a worker thread and bridges tokens into SSE messages through a queue.

### Inference efficiency controls

The following environment variables control CPU/GPU usage:

- `BLITZKODE_GPU_LAYERS`: offload layers to GPU when llama.cpp was built with GPU support.
- `BLITZKODE_THREADS`: CPU decode thread count.
- `BLITZKODE_THREADS_BATCH`: CPU prompt-processing thread count.
- `BLITZKODE_N_CTX`: context window.
- `BLITZKODE_BATCH`: llama.cpp prompt-processing batch size.
- `BLITZKODE_UBATCH`: llama.cpp micro-batch size.
- `BLITZKODE_PROMPT_CACHE`: in-memory prompt cache for repeated prefixes.
- `BLITZKODE_PRELOAD_MODEL`: load model at startup instead of first request.

For CPU-only machines, keep `BLITZKODE_GPU_LAYERS=0`, set `BLITZKODE_THREADS` close to physical performance cores, and use a quantized GGUF such as Q4_K_M for lower memory pressure. For GPU machines, increase `BLITZKODE_GPU_LAYERS` gradually until VRAM is stable.

## Search and research augmentation

The backend now has optional web search primitives:

- `POST /search/web`: DuckDuckGo Instant Answer search.
- `POST /generate/research`: searches, injects snippets into the prompt as untrusted context, and generates an answer.

API clients can use `/generate/stream` for normal token streaming or `/generate/research` when retrieval-augmented output is needed.

Important safety behavior:

- Search context is marked as untrusted.
- The system prompt tells the model not to invent APIs, citations, files, or execution results.
- Research mode asks the model to cite URLs when it relies on search results.



## Training pipeline context

The scripts directory currently represents several generations of training experimentation:

- `build_dataset.py` and `build_full_dataset.py`: local synthetic/curated dataset builders.
- `train_sft.py`: baseline LoRA SFT over curated coding examples.
- `train_reward_sft.py`: reward-inspired continuation, but not true GRPO.
- `train_dpo.py`: preference optimization over handcrafted chosen/rejected samples.
- `train_v2.py`, `train_continue.py`, `train_max.py`: larger public-dataset/QLoRA experiments.
- `export_gguf.py`: merge LoRA adapters and prepare for llama.cpp conversion.
- `test_inference.py`: smoke-test adapter checkpoints.

The scripts are intentionally excluded from Ruff in `pyproject.toml`, so backend CI remains focused and fast. They should be treated as operator tools rather than library code.

## Research-informed optimization notes

DeepSeek-R1 emphasizes staged post-training: cold-start reasoning data, RL with GRPO, rejection sampling, further SFT, and preference/RL alignment. DeepSeek-V3 emphasizes efficient architecture and training systems: MoE, MLA, FP8, multi-token prediction, and strong post-training/distillation.

For this repository and a 1.5B Qwen-family model, the practical subset is:

1. Use high-quality SFT data first; small models are very sensitive to noisy data.
2. Add preference data with DPO or ORPO to reduce low-quality or hallucinated answers.
3. If GRPO is available in the installed TRL version, use verifiable coding rewards on unit-tested tasks rather than subjective string heuristics.
4. Distill reasoning behavior from larger open models only where licensing permits, and store provenance for every dataset.
5. Use QLoRA / 4-bit loading for efficient GPU training.
6. Keep inference GGUF quantized for CPU/GPU-local deployment.

Do not indiscriminately scrape “all social/public internet” data. That creates legal, privacy, quality, and contamination issues. Prefer permissively licensed datasets and retain dataset cards/provenance.

## Cleanup completed in this pass

- Removed `.mypy_cache/` generated cache.
- Removed the malformed empty directory `USERPROFILE.ollamamodels/`.
- Added tested web search and research-generation endpoints.
- Removed the React/Vite frontend and made the project API-only.
- Tightened request models for message roles and history size.
- Hardened request-size middleware against invalid `Content-Length`.
- Added tests for search, disabled search, and research prompt injection.
- Updated README API/environment documentation.

## Current verification commands

- `python -m pytest tests/ -v` — 22 backend endpoint tests
- `python -m ruff check .`
- `python -m mypy server.py --ignore-missing-imports`

Run these after backend changes.

## Production quality audit results (v2.1 — Q8_0 GGUF, 1.53 GB)



All tests ran on RTX 4060 Laptop GPU, using CPU inference, load time 0.35 s.



| Task | Result | Notes |

|---|---|---|

| Floyd cycle detection | ✅ Correct | O(n)/O(1) with explanation |

| Flatten nested list | ✅ Correct | Recursive, clean |

| Kadane's max subarray | ✅ Correct | With complexity analysis |

| Dijkstra shortest path | ✅ Correct | Heap-based, full implementation |

| Fix zero-division bug | ✅ Correct | Detected and fixed |

| Refactor to context manager | ✅ Correct | One-liner |

| Decorator explanation | ✅ Correct | 2-sentence, accurate |

| JavaScript debounce | ✅ Correct | With explanation |

| SQL top-5 users | ✅ Correct | GROUP BY + ORDER BY + LIMIT |

| BFS traversal | ✅ Correct | deque-based |

| TypeScript typed fetch | ⚠️ Partial | Mixes RxJS/fetch patterns |

| Refactor global→class | ⚠️ Weak | Didn't complete the OOP refactor |

| Non-existent API hallucination | ❌ Hallucinated | Small model limitation |



**Root causes and mitigations:**



1. **TypeScript/OOP refactor gaps** — training data was Python-heavy. Adding more TypeScript and OOP refactor examples to `datasets/raw/available_training.jsonl` and re-running `train_available.py` with `--max-steps 200` would improve these.



2. **Hallucination of fictional APIs** — this is a fundamental 1.5B model limitation. The system prompt already instructs against it (`Do not invent APIs…`) but small models don't reliably follow system instructions they weren't trained on. Mitigations: add negative examples to DPO pairs, or use the `/generate/research` endpoint to ground answers in real web results.



3. **n_ctx warning** — `n_ctx_per_seq (2048) < n_ctx_train (32768)` is informational only. For most coding tasks 2048 tokens is sufficient. Increase `BLITZKODE_N_CTX` if you need longer code generation.

## Next recommended work

1. **Expand dataset** — add TypeScript, OOP refactoring, and "I don't know" (grounding) examples to `datasets/raw/available_training.jsonl`, then re-run `train_available.py --max-steps 300`.
2. **DPO for hallucination** — add preference pairs where `chosen = "I don't know, here's how to check"` and `rejected = <hallucinated answer>`, run `train_dpo.py`.
3. **API contract tests** — add tests for OpenAPI schema stability and streaming client behavior.
4. **Search cache tuning** — tune `BLITZKODE_SEARCH_CACHE_TTL` based on expected research workload.
5. **Context window** — increase `BLITZKODE_N_CTX=4096` in production for longer codebases, or `8192` with GPU offload.
6. **GRPO reward training** — use TRL `GRPOTrainer` with unit-test-verified coding rewards once a suitable test harness is available.