Instructions to use htunn/gemma-4-e2b-aiops-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use htunn/gemma-4-e2b-aiops-gguf with MLX:

# Make sure mlx-lm is installed
# pip install --upgrade mlx-lm

# Generate text with mlx-lm
from mlx_lm import load, generate

model, tokenizer = load("htunn/gemma-4-e2b-aiops-gguf")

prompt = "Write a story about Einstein"
messages = [{"role": "user", "content": prompt}]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True
)

text = generate(model, tokenizer, prompt=prompt, verbose=True)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use htunn/gemma-4-e2b-aiops-gguf with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf htunn/gemma-4-e2b-aiops-gguf:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf htunn/gemma-4-e2b-aiops-gguf:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf htunn/gemma-4-e2b-aiops-gguf:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf htunn/gemma-4-e2b-aiops-gguf:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf htunn/gemma-4-e2b-aiops-gguf:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf htunn/gemma-4-e2b-aiops-gguf:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf htunn/gemma-4-e2b-aiops-gguf:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf htunn/gemma-4-e2b-aiops-gguf:Q4_K_M

Use Docker

docker model run hf.co/htunn/gemma-4-e2b-aiops-gguf:Q4_K_M

LM Studio
Jan

vLLM

How to use htunn/gemma-4-e2b-aiops-gguf with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "htunn/gemma-4-e2b-aiops-gguf"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "htunn/gemma-4-e2b-aiops-gguf",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/htunn/gemma-4-e2b-aiops-gguf:Q4_K_M

Ollama
How to use htunn/gemma-4-e2b-aiops-gguf with Ollama:
```
ollama run hf.co/htunn/gemma-4-e2b-aiops-gguf:Q4_K_M
```

Unsloth Studio

How to use htunn/gemma-4-e2b-aiops-gguf with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for htunn/gemma-4-e2b-aiops-gguf to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for htunn/gemma-4-e2b-aiops-gguf to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for htunn/gemma-4-e2b-aiops-gguf to start chatting

How to use htunn/gemma-4-e2b-aiops-gguf with Pi:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "htunn/gemma-4-e2b-aiops-gguf"

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "mlx-lm": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "htunn/gemma-4-e2b-aiops-gguf"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use htunn/gemma-4-e2b-aiops-gguf with Hermes Agent:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "htunn/gemma-4-e2b-aiops-gguf"

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default htunn/gemma-4-e2b-aiops-gguf

Run Hermes

hermes

OpenClaw new

How to use htunn/gemma-4-e2b-aiops-gguf with OpenClaw:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "htunn/gemma-4-e2b-aiops-gguf"

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "htunn/gemma-4-e2b-aiops-gguf" \
  --custom-provider-id mlx-lm \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

MLX LM

How to use htunn/gemma-4-e2b-aiops-gguf with MLX LM:

Generate or start a chat session

# Install MLX LM
uv tool install mlx-lm
# Interactive chat REPL
mlx_lm.chat --model "htunn/gemma-4-e2b-aiops-gguf"

Run an OpenAI-compatible server

# Install MLX LM
uv tool install mlx-lm
# Start the server
mlx_lm.server --model "htunn/gemma-4-e2b-aiops-gguf"
# Calling the OpenAI-compatible server with curl
curl -X POST "http://localhost:8000/v1/chat/completions" \
   -H "Content-Type: application/json" \
   --data '{
     "model": "htunn/gemma-4-e2b-aiops-gguf",
     "messages": [
       {"role": "user", "content": "Hello"}
     ]
   }'

Docker Model Runner
How to use htunn/gemma-4-e2b-aiops-gguf with Docker Model Runner:
```
docker model run hf.co/htunn/gemma-4-e2b-aiops-gguf:Q4_K_M
```

Lemonade

How to use htunn/gemma-4-e2b-aiops-gguf with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull htunn/gemma-4-e2b-aiops-gguf:Q4_K_M

Run and chat with the model

lemonade run user.gemma-4-e2b-aiops-gguf-Q4_K_M

List all available models

lemonade list

Atomic Chat

gemma-4-e2b-aiops-gguf

A Q4_K_M GGUF of Gemma 4 E2B, fine-tuned on Apple Silicon (M3 Pro, 18 GB) using MLX LoRA to act as an autonomous AIOps Orchestrator Agent.

The model evaluates multi-domain infrastructure telemetry and outputs strict, execution-ready JSON commands across: Kubernetes · Nutanix · VMware ESXi · Active Directory · ADFS · PKI · Windows Server

Cross-platform inference: Trained on macOS (Apple Silicon / Metal), but the GGUF runs on macOS, Linux, and Windows via Ollama or llama.cpp (CUDA, ROCm, Vulkan, or CPU).

Model Details

Property	Value
Base model	`google/gemma-4-E2B-it`
Fine-tuning method	LoRA (MLX, 16 layers)
Training iterations	600
Trainable parameters	6.816M (0.147% of 4.63B)
Peak memory (training)	~11.3 GB on M3 Pro 18 GB
Val loss (start → end)	3.665 → 0.097
Quantization	Q4_K_M (llama.cpp)
FP16 size	9.3 GB
Q4_K_M size	3.2 GB (5.88 BPW)
Hardware	Apple M3 Pro, 18 GB Unified Memory
Training time	~5 minutes at ~2.4 it/sec

Quick Start

Ollama (recommended) — macOS · Linux · Windows

# Pull and run directly from this repo
ollama run hf.co/htunn/gemma-4-e2b-aiops-gguf:Q4_K_M

Custom Ollama Agent (with system prompt)

# Modelfile
FROM hf.co/htunn/gemma-4-e2b-aiops-gguf:Q4_K_M

SYSTEM """You are an autonomous AIOps orchestrator agent. Evaluate
telemetry across Kubernetes, Nutanix, VMware, Active Directory, ADFS, and PKI
infrastructure. Output strict, execution-ready JSON commands."""

PARAMETER temperature 0.3
PARAMETER top_p 0.9
PARAMETER top_k 40
PARAMETER repeat_penalty 1.1

ollama create aiops-orchestrator -f Modelfile
ollama run aiops-orchestrator

llama.cpp — macOS · Linux · Windows

llama-cli \
  -m gemma4-e2b-aiops-Q4_K_M.gguf \
  --temp 0.3 \
  -p "[AIOps-Agent] Node k8s-worker-03 status is NotReady. AD service account 'svc_k8s_cluster' authentication failed on ADFS."

vLLM (safetensors format)

A HuggingFace safetensors version is available at htunn/gemma-4-e2b-aiops-hf for use with vLLM or the Transformers library:

vllm serve htunn/gemma-4-e2b-aiops-hf --dtype bfloat16

from transformers import AutoTokenizer, AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("htunn/gemma-4-e2b-aiops-hf")

Example Output

Prompt:

[AIOps-Agent] Node k8s-worker-03 status is NotReady.
Active Directory service account 'svc_k8s_cluster' authentication failed on ADFS.

Response:

{
  "action": "remediate_auth",
  "target_domain": "ADFS",
  "service_account": "svc_k8s_cluster",
  "steps": ["check_pki_cert_validity", "rotate_secret_k8s"],
  "api_call": "POST /api/v1/auth/refresh"
}

Prompt:

[AIOps-Agent] PKI certificate for win-srv-2019-app01 expires in 48 hours.
Certificate Authority 'DC-CA-ROOT' is unreachable via LDAP.

Response:

{
  "action": "renew_certificate",
  "target_host": "win-srv-2019-app01",
  "ca_authority": "DC-CA-ROOT",
  "steps": ["verify_ca_health", "request_cert_renewal", "deploy_via_gpo"],
  "priority": "high"
}

Prompt:

[AIOps-Agent] Nutanix VM 'win-dc-01' CPU usage exceeds 95%.
VMware host ESXi-02 reports datastore latency.

Response:

{
  "action": "diagnose_hypervisor",
  "target_vm": "win-dc-01",
  "infrastructure": ["nutanix", "vmware"],
  "recommendation": "Live-migrate VM via Prism API to secondary host ESXi-04."
}

Training Data Format

Each training example follows the Gemma 4 chat template:

{
  "text": "<bos><start_of_turn>user\n[AIOps-Agent] <telemetry_event><end_of_turn>\n<start_of_turn>model\n{\"action\": \"...\", \"steps\": [...]}<end_of_turn>"
}

Scenarios cover:

Kubernetes node failures with AD/ADFS auth issues
Nutanix VM resource exhaustion and live migration
VMware ESXi host maintenance and cluster recovery
PKI certificate expiry and CA unreachability
Windows Server domain controller incidents
Cross-domain cascading failures

Reproduce Fine-Tuning

Source repository: htunn/aiops-gemma4

git clone https://github.com/Htunn/aiops-gemma4.git
cd aiops-gemma4

# Create Python 3.12 venv and install deps
make setup

# Fine-tune (requires HF access to google/gemma-4-E2B-it)
make train

# Fuse + convert to GGUF Q4_K_M
make quantize

# Upload GGUF to HF Hub (Ollama / llama.cpp)
make upload

# Upload safetensors to HF Hub (vLLM / Transformers)
make upload-hf

# Create Ollama model
make ollama

Known mlx-lm Compatibility Notes

Use --num-layers (not --lora-layers, removed in mlx-lm ≥ 0.28)
mlx_lm.fuse no longer needs --export-hf (HF format is now default)
Build llama.cpp with cmake: cmake -B build -DGGML_METAL=ON && cmake --build build --target llama-quantize
Pin numpy>=2.0, scipy>=1.13, transformers>=5.0 after installing llama.cpp requirements

Hardware Requirements

Task	Memory	Platform
Fine-tuning (MLX LoRA)	~11 GB	macOS Apple Silicon
Inference Q4_K_M (Ollama / llama.cpp)	~3.5 GB	macOS · Linux · Windows
Inference FP16 (vLLM / Transformers)	~9.5 GB	Linux (CUDA recommended)

License

Apache 2.0 — see LICENSE.

Base model (google/gemma-4-E2B-it) is subject to the Gemma Terms of Use.

Downloads last month: 139

GGUF

Model size

5B params

Architecture

gemma4

Hardware compatibility

4-bit

Model tree for htunn/gemma-4-e2b-aiops-gguf

Base model

google/gemma-4-E2B

Finetuned

google/gemma-4-E2B-it

Adapter

(135)

this model