brokencircuitranch/gemma4-hermes-tools-vision

Fine-tuned version of google/gemma-4-26B-A4B-it with vision/multimodal capabilities, reliable tool use, and function calling.

This is the vision-capable variant of gemma4-hermes-tools.

Training

• Base model: google/gemma-4-26B-A4B-it (Mixture of Experts, multimodal)
• Fine-tuning framework: Unsloth
• Hardware: NVIDIA A100 80GB (HuggingFace Space)
• Method: QLoRA (4-bit) → merged to 16-bit

Training Data

• NousResearch/hermes-function-calling-v1 — 1,893 examples of structured tool use and function calling in Hermes format
• teknium/OpenHermes-2.5 — 5,000 sampled examples for general instruction following and reasoning

Total: 6,893 examples, 2 epochs

Vision Integration

• LLM weights converted via llama.cpp convert_hf_to_gguf.py
• Vision encoder (mmproj) converted via llama.cpp convert_image_encoder_to_gguf.py
• LLM + mmproj merged into single GGUF for unified inference

Intended Use

Designed for agentic pipelines requiring reliable structured tool call generation with image understanding. Supports vision inputs (images, charts, screenshots) alongside tool/function calling. Tested with Ollama for local inference.

Usage

Ollama (recommended)

ollama pull brokencircuitranch/gemma4-hermes-tools-vision
ollama run brokencircuitranch/gemma4-hermes-tools-vision

Vision — Python

import ollama

response = ollama.chat(
    model=gemma4-hermes-tools-vision,
    messages=[{
        role: user,
        content: What is in this image?,
        images: [path/to/image.jpg]
    }]
)
print(response[message][content])

Tool Calling — Python

import ollama

tools = [{
    type: function,
    function: {
        name: tag_image,
        description: Analyze an image and return structured tags and description,
        parameters: {
            type: object,
            properties: {
                description: {type: string},
                objects: {type: array, items: {type: string}},
                scene: {type: string}
            },
            required: [description, objects, scene]
        }
    }
}]

response = ollama.chat(
    model=gemma4-hermes-tools-vision,
    messages=[{
        role: user,
        content: Analyze this image and return structured output.,
        images: [path/to/image.jpg]
    }],
    tools=tools
)
print(response[message][tool_calls])

Tool Calling with Vision — Hermes Format (raw)

This model uses the NousResearch Hermes prompt format for tool calls.

<|im_start|>system
You are a helpful assistant with access to tools and vision capabilities.

<tools>
[{type: function, function: {name: tag_image, description: Analyze an image and return structured tags and description, parameters: {type: object, properties: {description: {type: string, description: What is in the image}, objects: {type: array, items: {type: string}, description: List of detected objects}, scene: {type: string, description: Overall scene type}}, required: [description, objects, scene]}}}]
</tools><|im_end|>
<|im_start|>user
Analyze this image and return structured output.
<image>
path/to/image.jpg
</image><|im_end|>
<|im_start|>assistant
<tool_call>
{name: tag_image, arguments: {description: ..., objects: [...], scene: ...}}
</tool_call><|im_end|>
<|im_start|>tool
{status: ok}
<|im_end|>
<|im_start|>assistant
[Model summarizes findings based on its own analysis of the image]<|im_end|>

llama.cpp

llama-cli   --model gemma4-hermes-vision-q4-v2.gguf   --image path/to/image.jpg   --prompt Describe what you see.

Files

• gemma4-hermes-vision-q4-v2.gguf — Q4 quantized for local inference via Ollama/llama.cpp

License

Inherits Gemma Terms of Use