SevdanurGenc/LLM-Automated-Test-Generation-Agent

TestAjanı

Multilingual code explanation and self-repairing test generation with local LLM agents

English · Türkçe

TestAjanı explains Python, JavaScript/TypeScript and Java code with a locally hosted large language model and generates unit, interface (contract) and web UI tests. Every generated suite is compiled and executed by the real toolchain inside an isolated Docker sandbox; compilation errors, failing tests and uncovered lines are fed back to the agent until the suite is green and a coverage target is met. Suites are then scored by line coverage, mutation testing and hand-seeded realistic faults.

No source code leaves your machine: the model runs on Ollama, and test execution has no network access.

Contents

• Features
• How it works
• Quick start
• Usage
• Evaluation
• Project structure
• Configuration
• Development
• Limitations
• Citation
• License

Features

Languages	Python (pytest), JavaScript & TypeScript (Jest + Babel), Java (JUnit 5)
Test kinds	Unit · Interface/contract (public API, signatures, exception types) · Web UI (Playwright)
Agent loop	Plan → Implement → Verify → Repair, driven by tool calling
Verification	Real compilers and test runners in a Docker sandbox (--network none, CPU/memory limits)
Metrics	Line coverage (coverage.py, Istanbul, JaCoCo), mutation score (8 operator families), seeded-fault detection
Interface	Live agent timeline, coverage + mutant map on the source, test download, batch evaluation, history · Turkish / English switch (TR · EN) · real-time GPU monitor
Local-first	Any Ollama model with tool calling; default qwen2.5-coder:7b; compute device Auto / GPU / CPU
Cloud models (optional)	OpenAI (and OpenAI-compatible servers such as LM Studio, OpenRouter, vLLM) and Anthropic Claude; model lists sync after connecting

How it works

flowchart TD
    A[Source code + language + test kind] --> B[1 · PLAN<br/>Analyst agent: explanation, units, edge cases, test plan]
    B --> C[2 · IMPLEMENT<br/>Test-writer agent calls tools]
    C --> D[3 · VERIFY<br/>Sandbox: compile → run → coverage]
    D --> E{Accepted and<br/>coverage ≥ target?}
    E -- yes --> G[5 · EVALUATE<br/>Mutation testing + seeded faults]
    G --> M[6 · STRENGTHEN<br/>Surviving mutants → agent adds tests<br/>→ re-verify, re-measure]
    E -- no --> F{Iterations left?}
    F -- yes --> R[4 · REPAIR<br/>Feedback: compile errors, failing tests,<br/>uncovered lines + context compaction]
    R --> C
    F -- no --> G
    M --> H[Best suite, explanation, coverage, mutation score, suspected bugs]

Agent tools

Tool	Purpose
write_tests(code)	Save the complete test file
run_tests()	Compile, run and measure coverage in the sandbox
read_source()	Return the code under test with line numbers
report_bug(test_name, reason)	Flag a failing test as a genuine bug in the code (not in the test)
finish(summary)	Stop; accepted only when the last run is acceptable

Design choices that make small local models work

• Mandatory verification: if the model writes tests but does not run them, the orchestrator runs them anyway.
• Recovery layer: tool calls embedded as JSON text, or plain code blocks, are converted into tool calls.
• Context compaction: each new iteration keeps only the system prompt, task, current test file and latest result.
• Best-version selection: the delivered suite is the best verified candidate, not the last one.
• Actionable feedback: uncovered lines are sent with their source text, not just line numbers.
• Mutation-guided strengthening: after mutation testing, surviving mutants (line, original → mutated) are shown to the agent, which adds tests that should kill them. The extended suite is adopted only if it stays green on the original code, keeps every earlier test and does not lower the mutation score (same seeded mutant set). It can be switched off per run, or as the mutfix ablation.
• Energy per run: with an NVIDIA card the GPU power draw is sampled every second during a run and integrated to Wh (whole-card draw, so run experiments one at a time).

Details: docs/AJAN_MIMARISI.md (Turkish).

Quick start

The only requirement is Docker Desktop (Docker Engine on Linux). The launchers start Docker, build the sandbox images on first run, start the app, download the model on first run (~5 GB) and open http://localhost:8000.

OS	Start	Stop	GPU check	Desktop shortcut (once)
Windows	Baslat-Windows.bat (double-click)	Durdur-Windows.bat	GPU-Kontrol-Windows.bat	Kisayol-Olustur-Windows.bat
macOS	Baslat-macOS.command (double-click)	Durdur-macOS.command	GPU-Kontrol-macOS.command	Drag the file to the Dock
Linux	./baslat-linux.sh	./durdur-linux.sh	./gpu-kontrol-linux.sh	./scripts/linux-kisayol-olustur.sh

Is the model on the GPU? Ollama uses the GPU automatically; there is nothing to switch on. The GPU monitor in the top bar (left of the Ollama status) refreshes every 2 seconds: card name, utilisation, VRAM used/total and a pulsing dot while the card is busy. Click it for details — utilisation and VRAM bars, a 2-minute utilisation graph, temperature, power draw/limit, core clock, memory bandwidth, fan, power state, driver — and the share of each loaded model that sits in GPU memory (green at 100%, amber when part of it has spilled over to the CPU). On a computer with an NVIDIA card the launcher gives the app container GPU monitoring access (docker-compose.gpu.yml); without it (e.g. on a Mac) the monitor still shows the model's GPU share from Ollama. Every run also records the GPU share next to its generation speed (tokens/s). The GPU check launcher loads the model, measures the speed, checks that the running app is the latest version and explains what to do if the model is not fully on the GPU.

If the monitor does not appear after an update, the old app container is still running: run the start launcher once more (it always rebuilds the app). If it still looks old, run Onar-Windows.bat (repair): it removes old app containers, rebuilds the app without cache and checks the running version.

Options: --model qwen2.5-coder:14b, --rebuild, --no-browser (Windows: Baslat-Windows.bat -Model qwen2.5-coder:14b -Rebuild).

Windows: Docker Desktop needs WSL 2 and hardware virtualization. If Docker reports "Virtual Machine Platform not enabled", run wsl --install --no-distribution in an administrator PowerShell, restart, and check that Virtualization: Enabled appears in Task Manager → Performance → CPU (otherwise enable Intel VT-x / AMD SVM in the BIOS).

macOS: if Gatekeeper blocks the launcher, right-click it and choose Open.

If Ollama is already installed on the host, the launcher uses it (better GPU acceleration on Windows/macOS); otherwise Ollama runs in a container.

Manual start with Docker Compose

./scripts/build_sandboxes.sh                      # sandbox images, once
docker compose --profile ollama up --build -d     # app + Ollama
docker compose exec ollama ollama pull qwen2.5-coder:7b

Running without Docker (development)

./scripts/setup_local.sh        # pip, npm, JUnit 5/JaCoCo jars, Playwright Chromium
ollama serve &
ollama pull qwen2.5-coder:7b

cd backend && SANDBOX_MODE=local uvicorn app.main:app --reload --port 8000
cd frontend && npm run dev      # http://localhost:5173, proxies the API

SANDBOX_MODE=local executes code directly on your machine. Use it only with code you trust.

Usage

1. Paste code, pick a sample, or open files: a single file, several files or a project zip. With several files the app enters project mode: you choose the file to test and the others are placed next to it as helper modules (imports work; coverage and mutation are measured on the tested file only).
2. Choose the language and test kind; optionally set the model, iteration limit, coverage target and mutants.
3. Click Generate tests and follow the agent timeline.
4. Review the result tabs: explanation, generated tests (copy/download), coverage and mutant map, mutation table. Report opens a self-contained HTML report of the run (print it to save as PDF). The same detailed, colour report (verdict, gauges, round chart, coverage map, mutation breakdown, suggestions) is available for every past run in History via the HTML and PDF buttons, in the selected language.

Tab	What it does
Workspace	Single run with live timeline and results
Evaluation	Batch run over the benchmark; pause/resume; Markdown report
Compare	Two evaluations side by side over the same items: means, Wilcoxon p, A12, per-item values
Experiments	Runs the full paper protocol (quick, full or model-comparison profile, including cross-validation with an external mutation tool) with progress, ETA, pause/resume and a results zip; resumes automatically after a restart
Automation	Runs many samples unattended, one after another, and saves TR/EN HTML + PDF reports for each
History	Previous runs
Code history	All runs of the same code together: coverage/mutation trend, per-model table, best run, CSV

Workspace automation

The Automation tab tests any number of the built-in samples one after another, exactly as the Workspace would, without anyone at the computer. Pick samples (search and filter by language or test type; All / Clear), choose the same settings as in the Workspace (model, device, rounds, coverage target, mutants, mutant round) and the report languages, then press Start. After every sample the run appears in History and its detailed report is saved in Turkish and English as HTML and PDF (PDFs are rendered by the headless Chromium of the Web UI sandbox). A chip in the top bar shows progress on every tab; when the batch ends, a banner summarises it (completed, all-green, failed, time) and, if allowed, a desktop notification appears. The results table links every report; Summary opens an overview page and Download all (zip) returns everything in a dated zip. Add files… (or drag and drop) puts your own code into the list under My files: single files or a project zip, where every source file becomes a sample and the other files of the same upload are placed next to it as helpers; added files are kept for later automations. Model choice is either Single model or Random: each sample gets a model from a pool you tick (local and cloud), assigned by a seeded, balanced random draw; the model is shown per sample and the summary adds a By model table. By default the draw is weighted towards stronger models (strength is estimated from the model name — version, tier such as opus/sonnet/haiku or mini/nano, parameter count — so the strongest model gets about 4× as many samples as the weakest). Before a batch starts, every model in use is checked with one short request; models that fail (deprecated, no access, not installed) are left out, and Check models shows the result in advance. If a model fails during the run with a model-level error, it is removed, its remaining samples move to other models and the failed sample is retried once with another model. Pause / Resume / Stop are available, state is saved after each sample, and an interrupted batch resumes automatically when the app starts again.

Code history

Code history groups every run by the code it tested (language + code content), whether it came from the Workspace, Automation, Evaluation or Experiments. Built-in samples and added files are recognised by name; pasted code is named after its file name or the analyst's summary, and edited code is tracked as a new entry. For each code the page shows the number of runs and all-green runs, best coverage and mutation score, a chart of coverage and mutation score over the runs (hover for model and details), a By model table and every run with links to open it in the Workspace and to its HTML/PDF report. Download CSV exports the runs for tables in a paper.

Model providers and compute device

The plug button in the top bar opens Connections:

• Ollama (local) – status and the default compute device: Auto (Ollama decides; recommended), GPU (all layers on the graphics card; fails if VRAM is too small) or CPU (processor only). The Workspace can override the device for a single run. A "CPU mode" badge appears on the GPU indicator when CPU is the default.
• OpenAI and Claude (Anthropic) – paste an API key and press Connect and sync models. The key is checked first and saved only if the provider accepts it; the model list is then fetched and refreshed at every start (Sync models refreshes it on demand). OpenAI also accepts a custom server URL for OpenAI-compatible servers.

Connected models appear, grouped by provider, in every model picker (Workspace, Evaluation, Experiments). Model ids carry a provider prefix – openai:gpt-4o, anthropic:claude-sonnet-4-5 – so history, evaluations and comparisons record which provider produced each run (scripts/run_benchmark.py --model openai:gpt-4o works too). With a cloud model the code and tests are sent to that provider and usage is billed to your account; energy is measured for local models only. Keys are stored in the settings file (below) or read from environment variables, and are never returned to the browser in full; the API only accepts requests from pages on this computer.

Settings file (portable)

Every setting lives in one file in the app folder: ayarlar/testajani-ayarlar.json – API keys and server URLs, the default compute device, language, theme, the selected model and the Workspace, Evaluation and Experiments options. Copy that file into the ayarlar folder of the app on another computer and start the app: all settings come along (a file dropped in while the app is running is picked up on the next page load). Connections → Settings file offers the same as Download / Load. A backup is kept in the data folder, so a new version unzipped into another folder on the same computer restores the settings even if you forget to copy the file. The file holds API keys in plain text: git, release zips and the Docker image never include it – do not share it.

The TR · EN switch at the top right changes the whole interface, including the agent timeline, stop reasons and downloadable reports (?lang=en); new runs also ask the analyst to write in that language. If you switch the language after a run, the code explanation is translated once by the model and stored with the run. The sun/moon button switches between light and dark mode (by default the system setting is followed). Both choices are remembered in the browser. Runs created before version 2026.10.10 keep their original Turkish timeline titles.

Coverage and mutant map	Batch evaluation

Experiments (dark mode)	Compare	Mutation round

Evaluation

The benchmark contains 42 programs with 76 faults in three splits ("split" in benchmarks/manifest.json):

Split	Programs	Faults	Written by
dev – development	15 (Python, JS/TS, Java, web UI)	35 seeded	project authors; used while building the agent
test – held-out	7 (Python, JS/TS, Java)	21 seeded	project authors; never shown to the agent during development
bagimsiz – independent	20 (10 Python, 10 Java, distinct algorithms)	20 original one-line bugs	third parties: QuixBugs (MIT), imported by scripts/import_quixbugs.py

Your own code can be opened with Open file or dragged onto the editor (.py, .js, .ts, .java, .html; an HTML + JS pair switches to web UI tests).

Metric	Definition
Compile rate	Share of items whose test file compiles and loads
Pass rate	Passing tests on the original code / all tests
Line coverage	Executed / executable lines
Mutation score	(killed + timed-out) / (killed + survived + timed-out) mutants
Seeded-fault detection	Seeded faults that make at least one new test fail
First → final	One-shot first iteration vs. best suite after the loop

python scripts/run_benchmark.py --reference                                   # human-written reference suites (no LLM)
python scripts/run_benchmark.py --model qwen2.5-coder:7b                      # full loop → reports/LLM_SONUCLARI.md
python scripts/run_benchmark.py --model qwen2.5-coder:7b --max-iterations 1 --out reports/one_shot.md
python scripts/run_benchmark.py --model qwen2.5-coder:7b --report-lang en --out reports/llm_results_en.md  # English report
python scripts/run_benchmark.py --model qwen2.5-coder:7b --model llama3.1:8b  # model comparison
python scripts/run_benchmark.py --model qwen2.5-coder:14b --split test --repeats 3   # held-out set, 3 repeats
python scripts/run_benchmark.py --model qwen2.5-coder:14b --ablate judge prune       # ablation
python scripts/judge_accuracy.py --model qwen2.5-coder:14b --rule-mode veto           # judge precision/recall
PYNGUIN=pynguin python scripts/pynguin_baseline.py --seeds 1 2 3                      # search-based baseline
python scripts/mutation_sensitivity.py                                                # mutant sample-size check

Paper experiments in one click: open the Experiments tab, choose the Full (paper) profile and the models, and press Start experiment. It runs both models with 1 and 4 iterations (3 repeats each), all ablations (including the mutation round, mutfix) and the judge-accuracy study, shows progress and the remaining time, and saves its state after every item: if the computer sleeps or restarts, it continues from the same item when the app starts again. A Trial (~5 min) profile (first selected model, two small programs, full loop) checks the setup quickly; the Model comparison profile runs up to 8 selected models (local and cloud can be mixed) on the same programs with the same settings (max 4 rounds, 3 repetitions) and reports tokens per item. In the full and comparison profiles the agent's final test suites are also re-measured with established tools (Python: mutmut 2.5, JavaScript/TypeScript: StrykerJS 8; Java/PIT is not included): the report section Cross-validation with external mutation tools gives our score, the tool's score, the mean difference and Spearman ρ / Pearson r, and the raw values are in harici_mutasyon.csv in the zip. The PDF buttons in reports now open an A4 PDF produced by the app instead of the browser's print dialog (tables fit the page, collapsed sections are expanded, the footer has page numbers); the experiment zip also contains the reports as PDF. Items lost because the computer restarted or Ollama/Docker stopped are no longer counted as failures: the app waits for the infrastructure to return and retries; in a finished experiment, such items and external mutation measurements that failed with a tool error can be re-run with Fill in gaps (the model’s own failures are kept). On the Automation tab, Rebuild reports regenerates the HTML and PDF reports of older runs with the current template (tests are not re-run and no model is called). On first launch the Python and Node sandbox images are rebuilt once for these tools; Stop ends an experiment for good (finished steps are kept). The experiment panel also has Report and PDF buttons for a detailed colour report (highlights, configuration chart, paired statistics, per-split results, judge accuracy, per-item details) with a TR/EN report-language switch; the zip is named with the profile and date and contains both languages. Download results gives the zip with every evaluation, ANALIZ.md (means ± SD, Wilcoxon signed-rank, Vargha–Delaney A12) and the judge results. The command-line launchers (Deney-Calistir-Windows.bat, ./deney-calistir-linux.sh, profile -Profil hizli) remain available.

Measured without an LLM (reports/):

Study	Result
Human-written reference suites, 12 programs (dev + held-out)	140 tests, all green; 100% line coverage (8 programs with coverage), 87.1% pooled mutation score (276/317), 34/34 seeded faults
Mutant sample size (11 programs, all mutants)	30 mutants deviate 1.8 points on average from the all-mutant score (20 mutants: 3.8)
Pynguin 0.47, 6 Python programs × 3 seeds, 120 s	97.6% line coverage, 68.6% mutation score, only 43.1% (22/51) seeded faults detected
Judge rule stage, 72 labelled claims	rejects 19/19 false claims on deliberate validation, but also 5/32 genuine bugs

These numbers describe the reference suites, not the LLM; model results are produced by the commands above. Methodology: reports/BASARIM_RAPORU.md (Turkish).

Project structure

backend/
  app/
    agents/       orchestrator, prompts, tool schemas
    llm/          Ollama client (tool calling + recovery), scripted LLM for tests
    sandbox/      local/Docker executor; Python, JS/TS, Java and UI runners
    mutation/     mutation engine
    eval/         benchmark runner, report generator
    main.py       FastAPI (REST + SSE)
  tests/          end-to-end tests
frontend/         React + Vite + Monaco UI
benchmarks/       22 programs (dev + held-out test), seeded faults, reference suites, judge claims
sandbox/          sandbox Dockerfiles, Node toolchain
scripts/          launchers, setup, benchmark CLI
docs/             architecture document, screenshots
reports/          performance report

Configuration

Variable	Default	Description
OLLAMA_URL	http://localhost:11434	Ollama endpoint
OLLAMA_MODEL	qwen2.5-coder:7b	Default model
OLLAMA_NUM_CTX	16384	Context window
OPENAI_API_KEY / ANTHROPIC_API_KEY	—	Optional cloud provider keys (can also be entered in Connections)
OPENAI_BASE_URL	—	OpenAI-compatible server URL (e.g. http://host.docker.internal:1234/v1 for LM Studio)
ANTHROPIC_MAX_TOKENS	16000	Output token limit for Claude (lowered automatically if a model allows less)
SETTINGS_DIR	<app>/ayarlar	Folder of the portable settings file (/config in Docker, mounted from ./ayarlar)
ALLOWED_HOSTS	localhost,127.0.0.1,…	Host names the app answers to (protects against DNS rebinding); * = any
OLLAMA_DEVICE	auto	Initial default compute device (auto, gpu, cpu); changeable in Connections
OLLAMA_NUM_GPU	empty	Layers offloaded to the GPU; empty = Ollama decides (recommended), 999 = force all layers onto the GPU
LLM_TIMEOUT	600	Max wait between streamed model chunks (s); the first chunk includes model loading
LLM_TOTAL_TIMEOUT	1800	Max total time for one model response (s)
SANDBOX_MODE	local	docker (isolated) or local
MAX_ITERATIONS	4	Verify/repair iteration limit
COVERAGE_TARGET	85	Coverage target (%)
MAX_MUTANTS	30	Maximum mutants per job
RUN_TIMEOUT	90	Single test run timeout (s)
MAX_PARALLEL_JOBS	2	Concurrent jobs
AUTO_RESUME	1	Resume an interrupted experiment when the app starts (0 = off)
LTA_CHROMIUM	—	Chromium path for Playwright in local mode

Development

cd backend && python -m pytest -q     # agent loop, runners, mutation, reports, Ollama client, SSE
cd frontend && npm run build

Interactive API documentation is served at http://localhost:8000/docs.

Limitations

• Mutation operators are syntactic and equivalent mutants are not filtered, so scores are a lower bound.
• Inputs are single files without external dependencies (no mocking frameworks such as Mockito).
• Web UI tests are not measured for line coverage.
• Small local models are sensitive to sampling; repeat evaluations before drawing conclusions.

Citation

If you use TestAjanı in academic work, please cite:

@misc{testajani2026,
  title  = {Yerel B{\"u}y{\"u}k Dil Modeli Ajanlar{\i}yla {\c{C}}ok Dilli Kod Yorumlama ve
            Kendini D{\"u}zelten Otomatik Test {\"U}retimi},
  author = {Surname, Name and Surname, Name},
  year   = {2026},
  howpublished = {\url{https://github.com/<user>/<repo>}}
}

License

No license has been chosen yet. Add a LICENSE file (for example MIT or Apache-2.0) before making the repository public.