Instructions to use aogavrilov/diffusiongemma-agent-iq3-cuda13 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use aogavrilov/diffusiongemma-agent-iq3-cuda13 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M # Run inference directly in the terminal: llama cli -hf aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M # Run inference directly in the terminal: llama cli -hf aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M # Run inference directly in the terminal: ./llama-cli -hf aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama-cli -hf aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M
Use Docker
docker model run hf.co/aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M
- LM Studio
- Jan
- vLLM
How to use aogavrilov/diffusiongemma-agent-iq3-cuda13 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "aogavrilov/diffusiongemma-agent-iq3-cuda13" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "aogavrilov/diffusiongemma-agent-iq3-cuda13", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M
- Ollama
How to use aogavrilov/diffusiongemma-agent-iq3-cuda13 with Ollama:
ollama run hf.co/aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M
- Unsloth Studio
How to use aogavrilov/diffusiongemma-agent-iq3-cuda13 with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for aogavrilov/diffusiongemma-agent-iq3-cuda13 to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for aogavrilov/diffusiongemma-agent-iq3-cuda13 to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for aogavrilov/diffusiongemma-agent-iq3-cuda13 to start chatting
- Pi
How to use aogavrilov/diffusiongemma-agent-iq3-cuda13 with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @mariozechner/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent new
How to use aogavrilov/diffusiongemma-agent-iq3-cuda13 with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M
Run Hermes
hermes
- Atomic Chat new
- OpenClaw new
How to use aogavrilov/diffusiongemma-agent-iq3-cuda13 with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
- Docker Model Runner
How to use aogavrilov/diffusiongemma-agent-iq3-cuda13 with Docker Model Runner:
docker model run hf.co/aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M
- Lemonade
How to use aogavrilov/diffusiongemma-agent-iq3-cuda13 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M
Run and chat with the model
lemonade run user.diffusiongemma-agent-iq3-cuda13-Q4_K_M
List all available models
lemonade list
| [CmdletBinding()] | |
| param( | |
| [string]$WslRoot = '', | |
| [switch]$NoStart, | |
| [switch]$VerifyOnly | |
| ) | |
| $ErrorActionPreference = 'Stop' | |
| $bundle = Split-Path -Parent $PSCommandPath | |
| $model = 'diffusiongemma-26B-A4B-it-IQ3_M-from-Q4_K_M.gguf' | |
| $pythonArchive = "$bundle\payload\python\cpython-3.12.13.tar.gz" | |
| foreach ($required in @("$bundle\payload\app\server.py", "$bundle\payload\bin\llama-diffusion-gemma-visual-server", "$bundle\payload\models\$model", $pythonArchive, "$bundle\manifest.json")) { | |
| if (-not (Test-Path -LiteralPath $required)) { throw "Incomplete portable bundle: $required is missing." } | |
| } | |
| $manifest = Get-Content -LiteralPath "$bundle\manifest.json" -Raw | ConvertFrom-Json | |
| $manifestHash = (Get-FileHash -LiteralPath "$bundle\manifest.json" -Algorithm SHA256).Hash.ToLowerInvariant() | |
| $modelEntry = $manifest.files | Where-Object path -eq "payload/models/$model" | Select-Object -First 1 | |
| if (-not $modelEntry -or [int64]$modelEntry.bytes -ne (Get-Item -LiteralPath "$bundle\payload\models\$model").Length) { | |
| throw 'The bundled GGUF size does not match manifest.json.' | |
| } | |
| $modelBytes = [int64]$modelEntry.bytes | |
| if (-not (Get-Command wsl.exe -ErrorAction SilentlyContinue)) { throw 'WSL2 is required. Run wsl --install, reboot if requested, then run this command again.' } | |
| & wsl.exe --exec true | |
| if ($LASTEXITCODE -ne 0) { throw 'A default WSL distribution must be initialized once before installation.' } | |
| $bundleWsl = (& wsl.exe --exec wslpath -a $bundle).Trim() | |
| if (-not $bundleWsl) { throw 'Could not translate the bundle path for WSL.' } | |
| if (-not $WslRoot) { $WslRoot = '~/.local/share/diffusiongemma-agent' } | |
| $WslRoot = (& wsl.exe --exec python3 -c 'import os,sys; print(os.path.abspath(os.path.expandvars(os.path.expanduser(sys.argv[1]))))' $WslRoot).Trim() | |
| if (-not $WslRoot.StartsWith('/') -or $WslRoot -in @('/', '/root', '/home', '/usr', '/var')) { | |
| throw "Unsafe WSL installation path: $WslRoot" | |
| } | |
| if ($WslRoot.Contains("'")) { throw "WslRoot cannot contain an apostrophe." } | |
| $runtimeBin = "$bundleWsl/payload/bin" | |
| $ldd = & wsl.exe --exec env "LD_LIBRARY_PATH=$runtimeBin" ldd "$runtimeBin/llama-diffusion-gemma-visual-server" 2>&1 | |
| if ($LASTEXITCODE -ne 0 -or ($ldd -join "`n") -match 'not found') { throw "Portable CUDA runtime check failed:`n$($ldd -join "`n")" } | |
| if ($VerifyOnly) { Write-Output 'Portable bundle verification passed.'; exit 0 } | |
| $quotedBundle = $bundleWsl | |
| $quotedRoot = $WslRoot | |
| $setup = @" | |
| set -euo pipefail | |
| mkdir -p '$quotedRoot/runtime/bin' '$quotedRoot/models/diffusiongemma' | |
| if ! grep -Fqx '$manifestHash' '$quotedRoot/runtime/installed-manifest.sha256' 2>/dev/null; then | |
| cp -a '$quotedBundle/payload/app/.' '$quotedRoot/' | |
| cp -a '$quotedBundle/payload/bin/.' '$quotedRoot/runtime/bin/' | |
| if [ ! -f '$quotedRoot/models/diffusiongemma/$model' ] || [ `$(stat -c %s '$quotedRoot/models/diffusiongemma/$model') -ne $modelBytes ]; then | |
| cp -a '$quotedBundle/payload/models/.' '$quotedRoot/models/diffusiongemma/' | |
| fi | |
| fi | |
| if [ ! -x '$quotedRoot/python/bin/python3.12' ] || \ | |
| ! '$quotedRoot/python/bin/python3.12' -c 'import sys; raise SystemExit(0 if (3, 10) <= sys.version_info[:2] < (3, 13) else 1)' >/dev/null 2>&1; then | |
| rm -rf '$quotedRoot/python' | |
| mkdir -p '$quotedRoot/python' | |
| tar -xzf '$quotedBundle/payload/python/cpython-3.12.13.tar.gz' -C '$quotedRoot/python' | |
| fi | |
| '$quotedRoot/python/bin/python3.12' -c 'import sys; raise SystemExit(0 if (3, 10) <= sys.version_info[:2] < (3, 13) else 1)' || exit 43 | |
| if [ ! -x '$quotedRoot/.venv-runtime/bin/python' ] || \ | |
| ! '$quotedRoot/.venv-runtime/bin/python' -c 'import aider, fastapi, haystack, pydantic, uvicorn' >/dev/null 2>&1; then | |
| if ! timeout 15 bash -c 'cat < /dev/null > /dev/tcp/pypi.org/443'; then | |
| exit 42 | |
| fi | |
| rm -rf '$quotedRoot/.venv-runtime' | |
| '$quotedRoot/python/bin/python3.12' -m venv '$quotedRoot/.venv-runtime' | |
| '$quotedRoot/.venv-runtime/bin/python' -m pip install --disable-pip-version-check --upgrade 'pip==25.0.1' | |
| '$quotedRoot/.venv-runtime/bin/python' -m pip install --disable-pip-version-check --prefer-binary \ | |
| 'aider-chat==0.86.2' \ | |
| 'fastapi==0.128.8' \ | |
| 'haystack-ai==2.31.0' \ | |
| 'pydantic==2.12.5' \ | |
| 'uvicorn[standard]==0.51.0' | |
| fi | |
| chmod +x '$quotedRoot/runtime/bin/'* '$quotedRoot/scripts/'*.sh '$quotedRoot/start-runtime.sh' | |
| printf '%s\n' '$manifestHash' > '$quotedRoot/runtime/installed-manifest.sha256' | |
| "@ | |
| & wsl.exe --exec bash -lc $setup | |
| if ($LASTEXITCODE -eq 42) { throw 'WSL cannot reach PyPI. Pause or reconfigure the VPN/proxy for WSL, then retry; the downloaded model files will be reused.' } | |
| if ($LASTEXITCODE -eq 43) { throw 'The bundled Python runtime is incompatible. Download the current runtime bundle and retry.' } | |
| if ($LASTEXITCODE -ne 0) { throw "WSL runtime installation failed (exit $LASTEXITCODE). The downloaded model files were kept and will be reused." } | |
| [ordered]@{ format = 1; runtime_version = '0.1.2'; bundle = $bundle; wsl_root = $WslRoot; model = $model; installed_at = (Get-Date).ToUniversalTime().ToString('o') } | | |
| ConvertTo-Json | Set-Content -LiteralPath "$bundle\installed.json" -Encoding utf8 | |
| if (-not $NoStart) { & "$bundle\dg.ps1" -StartOnly } | |
| Write-Output "Installed. Use: & '$bundle\dg.ps1' -Repo C:\path\to\repo -Task 'fix ... and run tests'" | |