Instructions to use aogavrilov/diffusiongemma-agent-iq3-cuda13 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use aogavrilov/diffusiongemma-agent-iq3-cuda13 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M # Run inference directly in the terminal: llama cli -hf aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M # Run inference directly in the terminal: llama cli -hf aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M # Run inference directly in the terminal: ./llama-cli -hf aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama-cli -hf aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M
Use Docker
docker model run hf.co/aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M
- LM Studio
- Jan
- vLLM
How to use aogavrilov/diffusiongemma-agent-iq3-cuda13 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "aogavrilov/diffusiongemma-agent-iq3-cuda13" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "aogavrilov/diffusiongemma-agent-iq3-cuda13", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M
- Ollama
How to use aogavrilov/diffusiongemma-agent-iq3-cuda13 with Ollama:
ollama run hf.co/aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M
- Unsloth Studio
How to use aogavrilov/diffusiongemma-agent-iq3-cuda13 with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for aogavrilov/diffusiongemma-agent-iq3-cuda13 to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for aogavrilov/diffusiongemma-agent-iq3-cuda13 to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for aogavrilov/diffusiongemma-agent-iq3-cuda13 to start chatting
- Pi
How to use aogavrilov/diffusiongemma-agent-iq3-cuda13 with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @mariozechner/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent new
How to use aogavrilov/diffusiongemma-agent-iq3-cuda13 with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M
Run Hermes
hermes
- Atomic Chat new
- OpenClaw new
How to use aogavrilov/diffusiongemma-agent-iq3-cuda13 with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
- Docker Model Runner
How to use aogavrilov/diffusiongemma-agent-iq3-cuda13 with Docker Model Runner:
docker model run hf.co/aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M
- Lemonade
How to use aogavrilov/diffusiongemma-agent-iq3-cuda13 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull aogavrilov/diffusiongemma-agent-iq3-cuda13:Q4_K_M
Run and chat with the model
lemonade run user.diffusiongemma-agent-iq3-cuda13-Q4_K_M
List all available models
lemonade list
File size: 5,262 Bytes
ef2127a 76a38cb ef2127a 76a38cb ef2127a 76a38cb ef2127a ed9e13a ef2127a 76a38cb ef2127a 76a38cb ef2127a 76a38cb ef2127a 76a38cb ef2127a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 | [CmdletBinding()]
param(
[string]$WslRoot = '',
[switch]$NoStart,
[switch]$VerifyOnly
)
$ErrorActionPreference = 'Stop'
$bundle = Split-Path -Parent $PSCommandPath
$model = 'diffusiongemma-26B-A4B-it-IQ3_M-from-Q4_K_M.gguf'
$pythonArchive = "$bundle\payload\python\cpython-3.12.13.tar.gz"
foreach ($required in @("$bundle\payload\app\server.py", "$bundle\payload\bin\llama-diffusion-gemma-visual-server", "$bundle\payload\models\$model", $pythonArchive, "$bundle\manifest.json")) {
if (-not (Test-Path -LiteralPath $required)) { throw "Incomplete portable bundle: $required is missing." }
}
$manifest = Get-Content -LiteralPath "$bundle\manifest.json" -Raw | ConvertFrom-Json
$manifestHash = (Get-FileHash -LiteralPath "$bundle\manifest.json" -Algorithm SHA256).Hash.ToLowerInvariant()
$modelEntry = $manifest.files | Where-Object path -eq "payload/models/$model" | Select-Object -First 1
if (-not $modelEntry -or [int64]$modelEntry.bytes -ne (Get-Item -LiteralPath "$bundle\payload\models\$model").Length) {
throw 'The bundled GGUF size does not match manifest.json.'
}
$modelBytes = [int64]$modelEntry.bytes
if (-not (Get-Command wsl.exe -ErrorAction SilentlyContinue)) { throw 'WSL2 is required. Run wsl --install, reboot if requested, then run this command again.' }
& wsl.exe --exec true
if ($LASTEXITCODE -ne 0) { throw 'A default WSL distribution must be initialized once before installation.' }
$bundleWsl = (& wsl.exe --exec wslpath -a $bundle).Trim()
if (-not $bundleWsl) { throw 'Could not translate the bundle path for WSL.' }
if (-not $WslRoot) { $WslRoot = '~/.local/share/diffusiongemma-agent' }
$WslRoot = (& wsl.exe --exec python3 -c 'import os,sys; print(os.path.abspath(os.path.expandvars(os.path.expanduser(sys.argv[1]))))' $WslRoot).Trim()
if (-not $WslRoot.StartsWith('/') -or $WslRoot -in @('/', '/root', '/home', '/usr', '/var')) {
throw "Unsafe WSL installation path: $WslRoot"
}
if ($WslRoot.Contains("'")) { throw "WslRoot cannot contain an apostrophe." }
$runtimeBin = "$bundleWsl/payload/bin"
$ldd = & wsl.exe --exec env "LD_LIBRARY_PATH=$runtimeBin" ldd "$runtimeBin/llama-diffusion-gemma-visual-server" 2>&1
if ($LASTEXITCODE -ne 0 -or ($ldd -join "`n") -match 'not found') { throw "Portable CUDA runtime check failed:`n$($ldd -join "`n")" }
if ($VerifyOnly) { Write-Output 'Portable bundle verification passed.'; exit 0 }
$quotedBundle = $bundleWsl
$quotedRoot = $WslRoot
$setup = @"
set -euo pipefail
mkdir -p '$quotedRoot/runtime/bin' '$quotedRoot/models/diffusiongemma'
if ! grep -Fqx '$manifestHash' '$quotedRoot/runtime/installed-manifest.sha256' 2>/dev/null; then
cp -a '$quotedBundle/payload/app/.' '$quotedRoot/'
cp -a '$quotedBundle/payload/bin/.' '$quotedRoot/runtime/bin/'
if [ ! -f '$quotedRoot/models/diffusiongemma/$model' ] || [ `$(stat -c %s '$quotedRoot/models/diffusiongemma/$model') -ne $modelBytes ]; then
cp -a '$quotedBundle/payload/models/.' '$quotedRoot/models/diffusiongemma/'
fi
fi
if [ ! -x '$quotedRoot/python/bin/python3.12' ] || \
! '$quotedRoot/python/bin/python3.12' -c 'import sys; raise SystemExit(0 if (3, 10) <= sys.version_info[:2] < (3, 13) else 1)' >/dev/null 2>&1; then
rm -rf '$quotedRoot/python'
mkdir -p '$quotedRoot/python'
tar -xzf '$quotedBundle/payload/python/cpython-3.12.13.tar.gz' -C '$quotedRoot/python'
fi
'$quotedRoot/python/bin/python3.12' -c 'import sys; raise SystemExit(0 if (3, 10) <= sys.version_info[:2] < (3, 13) else 1)' || exit 43
if [ ! -x '$quotedRoot/.venv-runtime/bin/python' ] || \
! '$quotedRoot/.venv-runtime/bin/python' -c 'import aider, fastapi, haystack, pydantic, uvicorn' >/dev/null 2>&1; then
if ! timeout 15 bash -c 'cat < /dev/null > /dev/tcp/pypi.org/443'; then
exit 42
fi
rm -rf '$quotedRoot/.venv-runtime'
'$quotedRoot/python/bin/python3.12' -m venv '$quotedRoot/.venv-runtime'
'$quotedRoot/.venv-runtime/bin/python' -m pip install --disable-pip-version-check --upgrade 'pip==25.0.1'
'$quotedRoot/.venv-runtime/bin/python' -m pip install --disable-pip-version-check --prefer-binary \
'aider-chat==0.86.2' \
'fastapi==0.128.8' \
'haystack-ai==2.31.0' \
'pydantic==2.12.5' \
'uvicorn[standard]==0.51.0'
fi
chmod +x '$quotedRoot/runtime/bin/'* '$quotedRoot/scripts/'*.sh '$quotedRoot/start-runtime.sh'
printf '%s\n' '$manifestHash' > '$quotedRoot/runtime/installed-manifest.sha256'
"@
& wsl.exe --exec bash -lc $setup
if ($LASTEXITCODE -eq 42) { throw 'WSL cannot reach PyPI. Pause or reconfigure the VPN/proxy for WSL, then retry; the downloaded model files will be reused.' }
if ($LASTEXITCODE -eq 43) { throw 'The bundled Python runtime is incompatible. Download the current runtime bundle and retry.' }
if ($LASTEXITCODE -ne 0) { throw "WSL runtime installation failed (exit $LASTEXITCODE). The downloaded model files were kept and will be reused." }
[ordered]@{ format = 1; runtime_version = '0.1.2'; bundle = $bundle; wsl_root = $WslRoot; model = $model; installed_at = (Get-Date).ToUniversalTime().ToString('o') } |
ConvertTo-Json | Set-Content -LiteralPath "$bundle\installed.json" -Encoding utf8
if (-not $NoStart) { & "$bundle\dg.ps1" -StartOnly }
Write-Output "Installed. Use: & '$bundle\dg.ps1' -Repo C:\path\to\repo -Task 'fix ... and run tests'"
|