| --- |
| license: apache-2.0 |
| library_name: diffusiongemma-agent |
| base_model: google/diffusiongemma-26B-A4B-it |
| base_model_relation: quantized |
| pipeline_tag: text-generation |
| tags: |
| - gguf |
| - diffusiongemma |
| - coding-agent |
| - cuda |
| - wsl |
| --- |
| |
| # Run DiffusionGemma as a local coding agent on a 16 GB GPU |
|
|
| This is a ready-to-run DiffusionGemma stack for Windows + WSL2. It turns the |
| 26B-A4B model into a repository-aware coding agent on a single 16 GB NVIDIA |
| GPU, without requiring you to compile llama.cpp, convert weights, assemble |
| CUDA libraries, or build an agent wrapper yourself. |
|
|
| This release contains more than a quantized GGUF: |
|
|
| - an IQ3 model profile that fits entirely in 16 GB VRAM; |
| - a matched custom llama.cpp/CUDA backend for DiffusionGemma; |
| - pinned CUDA 13 runtime libraries and launch settings; |
| - repository retrieval instead of sending the whole codebase to the model; |
| - focused file editing through Aider, automatic tests, session artifacts, and |
| rollback when validation fails; |
| - a localhost OpenAI-compatible endpoint for supported coding clients. |
|
|
| ## Why use this instead of a standalone quantization? |
|
|
| | Standalone GGUF | This runtime | |
| | --- | --- | |
| | Supplies model weights | Supplies a tested model, backend, CUDA runtime, and agent workflow | |
| | Requires finding a compatible DiffusionGemma runner | Includes the exact custom backend used by the release | |
| | Requires manual offload, memory, and launch tuning | Ships a full-GPU preset tested on a 16 GB RTX 3080 Laptop GPU | |
| | Primarily provides raw generation/chat | Searches a repository, edits focused files, runs tests, and records results | |
| | Leaves integration and reproducibility to the user | Installs and operates through one versioned CLI | |
|
|
| Use another GGUF if you only need weights or want to assemble your own |
| inference stack. Use this release when you want a reproducible local coding |
| agent that is already wired together for 16 GB NVIDIA hardware. |
|
|
| ## Easiest Windows setup |
|
|
| You do not need Python, a CUDA Toolkit, a compiler, or ML setup experience. |
|
|
| 1. [Download the standalone Windows installer](https://github.com/aogavrilov/diffusiongemma-agent/releases/download/desktop-v0.1.2/DiffusionGemmaAgentSetup-0.1.2.exe). |
| 2. Open **DiffusionGemma Agent** after installation. |
| 3. Let the app check Windows, WSL2, the GPU, VRAM, disk, and download access. |
| 4. Review the model and CUDA licenses and click **Download and install**. |
| 5. Choose a Git repository, enter one concrete task, and review the generated |
| diff and test output in the app. |
|
|
| The desktop app includes its own Python runtime. It automatically reuses a |
| compatible runtime or IQ3 GGUF already present in common download and Hugging |
| Face cache folders. The separate 13.2 GB download, when needed, is resumable. |
| The current alpha installer is not code-signed, so |
| Windows may show an unknown-publisher warning; GitHub publishes its SHA-256 |
| checksum beside the download. |
|
|
| ## Command-line alternative |
|
|
| Run in Windows PowerShell: |
|
|
| ```powershell |
| python -m pip install --upgrade diffusiongemma-agent |
| dg-agent doctor |
| dg-agent install --accept-licenses |
| dg-agent status |
| ``` |
|
|
| Run a focused repository task: |
|
|
| ```powershell |
| dg-agent run --repo C:\work\repo --task "Fix src/x.py and run its tests" --file src/x.py |
| ``` |
|
|
| Stop the service and release GPU memory with `dg-agent stop`. |
|
|
| ## What a task looks like |
|
|
| ```text |
| your request |
| -> bounded repository search |
| -> focused edit session |
| -> detected tests and verification |
| -> final diff and saved session artifacts |
| -> rollback if validation fails |
| ``` |
|
|
| This works best for concrete file-level changes, focused bug fixes, and tests. |
| It is not intended to replace a large-context cloud agent for broad, |
| underspecified repository-wide work. |
|
|
| ## Hardware and storage |
|
|
| - Windows 10/11 with an initialized WSL2 Ubuntu distribution; |
| - NVIDIA GPU with at least 16 GB VRAM; |
| - current NVIDIA Windows driver with WSL CUDA support; |
| - at least 15 GiB free for download, with 30 GiB recommended for the download |
| plus the installed WSL copy; |
| - network access during first installation. |
|
|
| The tested system is an RTX 3080 Laptop GPU with 16 GB VRAM. CPU-only, native |
| Linux, macOS, AMD, and sub-16-GB GPU installations are not supported by this |
| release. |
|
|
| ## Contents |
|
|
| - IQ3 GGUF derived from `google/diffusiongemma-26B-A4B-it`; |
| - Linux x86-64 custom llama.cpp DiffusionGemma backend; |
| - private CUDA 13 runtime libraries required by that backend; |
| - checkpointed repository supervisor, bounded retrieval, Haystack, and Aider |
| adapters; |
| - service launchers for localhost ports `4100` and `8090`; |
| - `manifest.json` with file sizes and SHA-256 values; |
| - complete third-party notices under `LICENSES/`. |
|
|
| ## Limits |
|
|
| The default fast agent profile uses a 768-token effective input context and up |
| to 256 output tokens. It compensates with bounded repository retrieval; it does |
| not load an entire large repository into the model context. IQ3 quantization |
| reduces quality relative to the original model. Native model-selected tool |
| calls are disabled in the default route. |
|
|
| A short warmed probe measured approximately 19.6 words/s on the tested RTX |
| 3080 Laptop 16 GB. Full agent tasks are slower because retrieval, diffusion |
| passes, edits, and tests add latency. This is not a performance guarantee. |
|
|
| ## Status and scope |
|
|
| This is alpha, hardware-specific software. It is not an official Google, |
| NVIDIA, Hugging Face, Aider, or upstream llama.cpp distribution. |
|
|
| ## Research context |
|
|
| This repository is an engineering runtime, not the implementation artifact of |
| the papers below. For problem-first research guides on adjacent evaluation and |
| control questions, see: |
|
|
| - [How to tell whether a compressed text generator fails in the codec or the generator](https://aogavrilov.com/projects/codec-bottleneck-diagnosis/) · [Русский](https://aogavrilov.com/ru/projects/codec-bottleneck-diagnosis/) · [简体中文](https://aogavrilov.com/zh/projects/codec-bottleneck-diagnosis/) · [한국어](https://aogavrilov.com/ko/projects/codec-bottleneck-diagnosis/) |
| - [How can AI edit code without regenerating the entire program?](https://aogavrilov.com/projects/discrete-latent-generation/) · [Русский](https://aogavrilov.com/ru/projects/discrete-latent-generation/) · [简体中文](https://aogavrilov.com/zh/projects/discrete-latent-generation/) · [한국어](https://aogavrilov.com/ko/projects/discrete-latent-generation/) |
|
|
| ## Safety |
|
|
| The agent can modify files and run repository tests. Checkpointing and rollback |
| are not a security sandbox. Use a clean Git worktree, inspect generated diffs, |
| and do not expose secrets to untrusted repositories. Services bind to localhost |
| by default. |
|
|
| Source code: |
|
|
| - [agent and installer](https://github.com/aogavrilov/diffusiongemma-agent) |
| - [custom llama.cpp/CUDA backend](https://github.com/aogavrilov/diffusiongemma-llama-cpp-diffusion) |
|
|
| See the public source README for lifecycle commands, troubleshooting, update, |
| uninstall, and architecture details. |
|
|
| ## Licenses |
|
|
| Review `LICENSES/NOTICE.md` before installation or redistribution. The runtime |
| contains Apache-2.0 model-derived material, MIT-licensed llama.cpp code, and |
| NVIDIA redistributable libraries governed by the included CUDA EULA. |
|
|
