README.md

---
license: apache-2.0
base_model: google/gemma-4-E4B
pipeline_tag: image-text-to-image
library_name: transformers
language:
- en
tags:
- knowledge-graph
- information-extraction
- scene-graph
- image-to-graph
- structured-extraction
- opengraph
- gemma4
- lora
- peft
- mcp
datasets:
- OpenGraphAI/opengraph-image-gold-v1
metrics:
- accuracy
- f1
---

# Model Card for opengraph-image-gemma4-e4b-v1

<!-- Provide a quick summary of what the model is/does. -->

A fine-tuned **Gemma 4 E4B** that turns images into a valid, schema-faithful **knowledge graph** (`graph.json`) — objects, attributes, scene context, text spans, and the relationships between them — ready for AI agents to query and reason over. Built by [OpenGraph AI](https://github.com/OpenGraphAI/opengraph-ai) as the small, cheap, on-device alternative to calling a frontier vision API for every extraction.

## Model Details

### Model Description

The model does one thing extremely well: **image → knowledge graph, on-schema, every time.** It was trained on verified image→graph gold pairs so that its output always conforms to OpenGraph's `graph.json` contract — stable snake_case node IDs with type prefixes (`entity_`, `concept_`, `event_`, `attr_`), typed edges, and cross-image-mergeable entities. Compared to prompting a general frontier model, it is dramatically cheaper per extraction, runs on a single consumer GPU (or laptop, quantized), and produces structurally consistent output that downstream graph tooling can rely on.

- **Developed by:** OpenGraph AI
- **Model type:** Multimodal (image-text-to-image), decoder-only transformer; QLoRA fine-tune of Gemma 4 E4B
- **Language(s):** English
- **License:** Apache 2.0 (inherited from Gemma 4; use is additionally subject to Google's [Gemma terms of use](https://ai.google.dev/gemma/terms))
- **Finetuned from model:** [`google/gemma-4-E4B`](https://huggingface.co/google/gemma-4-E4B)

### Model Sources

- **Repository:** https://github.com/OpenGraphAI/opengraph-ai
- **Demo:** [placeholder] <!-- link the graph.html shareable demo when live -->

## Uses

### Direct Use

Feed the model images plus the OpenGraph extraction system prompt; it returns a complete `graph.json` — nodes for detected objects (fine-grained labels, normalized bounding boxes), one scene node, attribute nodes, transcribed text spans, and the edges wiring them together. Useful anywhere images need to become structured, queryable knowledge: visual search indexes, dataset annotation, scene understanding, and document/diagram parsing.

### Downstream Use

The model's intended home is inside the **[opengraph-image MCP server](https://github.com/OpenGraphAI/opengraph-ai)**: register it with Claude Desktop, Cursor, or any MCP-compatible agent, and the agent gains persistent, queryable visual memory — including multi-hop questions across many images ("which components appear in both photos, and what changed between them?"). It also serves as a local extraction backend for robotics scene memory, where per-frame frontier API calls are too slow and expensive.

### Out-of-Scope Use

- **Safety-critical decisions without human review** (e.g., equipment maintenance, medical, or navigation decisions made solely from the extracted graph).
- **General chat, reasoning, or text generation** — the fine-tune specializes the model for extraction; general capabilities may be degraded relative to the base model.
- **Images far outside the training distribution** (see Limitations).
- Any use prohibited by Google's [Gemma prohibited use policy](https://ai.google.dev/gemma/prohibited_use_policy).

## Bias, Risks, and Limitations

- **Extraction errors are silent.** The model can hallucinate objects, miss objects, or assign wrong relationships while still producing perfectly *valid* JSON — structural validity is not factual accuracy.
- **Distribution sensitivity.** Accuracy degrades on image types unlike the training data (domains, camera angles, lighting, languages in text spans). [More Information Needed — characterize after evaluation]
- **Inherited bias.** The model inherits biases from Gemma 4's pretraining data, from the public image datasets used for fine-tuning, and from the frontier models used to generate a portion of the training labels.
- **Schema lock-in.** Output follows OpenGraph's `graph.json` schema; it is not a general-purpose captioner and will not follow arbitrary output formats reliably.

### Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. Validate every output with the schema validators shipped in the [OpenGraph repo](https://github.com/OpenGraphAI/opengraph-ai), keep a human in the loop for consequential decisions, and spot-check extractions when applying the model to a new image domain.

## How to Get Started with the Model

Use the code below to get started with the model.

```python
# pip install -U "transformers>=5.10.1" torch torchvision accelerate

from transformers import AutoProcessor, AutoModelForImageTextToText
from PIL import Image

MODEL_ID = "OpenGraphAI/opengraph-image-gemma4-e4b-v1"

processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForImageTextToText.from_pretrained(
    MODEL_ID, dtype="auto", device_map="auto"
)

SYSTEM_PROMPT = """[More Information Needed — paste the OpenGraph extraction system prompt]"""

image = Image.open("your_image.jpg").convert("RGB")
messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": "Extract the knowledge graph from this image."},
    ]},
]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=2048)
graph_json = processor.decode(
    outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True
)
print(graph_json)  # -> valid graph.json
```

Or skip the code entirely and use it through the MCP server:

```bash
[More Information Needed — one-line MCP install command]
```

## Training Details

### Training Data

Trained on **[placeholder] verified image→`graph.json` gold pairs** ([`OpenGraphAI/opengraph-image-gold-v1`](https://huggingface.co/datasets/OpenGraphAI/opengraph-image-gold-v1)), assembled from two sources: (1) public scene-graph datasets (e.g., Visual Genome) converted programmatically into the `graph.json` schema, and (2) unannotated images labeled by two independent frontier vision models, auto-accepted where both models agreed and human-reviewed otherwise. Every pair passed the OpenGraph Pydantic schema validators before inclusion.

### Training Procedure

Supervised fine-tuning (SFT) with **QLoRA**: the base model frozen in 4-bit NF4 quantization, with LoRA adapters (rank 16, all linear layers, plus `lm_head`/`embed_tokens`) trained via Hugging Face TRL's `SFTTrainer`, following [Google's official Gemma 4 vision QLoRA guide](https://ai.google.dev/gemma/docs/core/huggingface_vision_finetune_qlora).

#### Preprocessing

Each example is formatted as a three-turn conversation (system = schema instruction, user = image + extraction request, assistant = gold `graph.json`) and templated with the official Gemma 4 chat template. Images are processed at their native aspect ratio; image tokens are masked out of the training loss.

#### Training Hyperparameters

- **Training regime:** bf16 mixed precision (4-bit NF4 quantized base, bf16 compute)
- **LoRA:** r=16, alpha=16, dropout=0.05, target_modules=all-linear
- **Epochs:** 3
- **Learning rate:** 2e-4 (constant schedule)
- **Per-device batch size:** 1
- **Max grad norm:** 0.3

#### Speeds, Sizes, Times

[More Information Needed — fill after training: total training time, adapter size, merged checkpoint size]

## Evaluation

### Testing Data, Factors & Metrics

#### Testing Data

A held-out **test split (5%)** of [`OpenGraphAI/opengraph-image-gold-v1`](https://huggingface.co/datasets/OpenGraphAI/opengraph-image-gold-v1), never seen during training.

#### Factors

Results are disaggregated by image source/domain (converted scene-graph data vs. frontier-labeled robot/inspection frames). [More Information Needed — add further factors after evaluation]

#### Metrics

- **Schema-valid extraction rate** — % of outputs that parse as JSON *and* pass the OpenGraph Pydantic validators on the first attempt. Chosen because downstream graph tooling hard-fails on invalid output; this is the reliability number that matters in production.
- **Node F1 / Edge F1** — precision and recall of predicted nodes and edges against the gold graph, matched on normalized label + type. Measures whether the *content* of the graph is right, not just its shape.
- **Cost per 1,000 images & p50 latency** — the practical case for a small fine-tune over a frontier API.

### Results

All numbers produced by the open [eval harness](https://github.com/OpenGraphAI/opengraph-ai) and reproducible from the linked script.

| Metric | This model | Base Gemma 4 E4B-it | Frontier API baseline |
|---|---|---|---|
| Schema-valid rate | [More Information Needed] | [More Information Needed] | [More Information Needed] |
| Node/Edge F1 | [More Information Needed] | [More Information Needed] | [More Information Needed] |
| $ / 1k images | [More Information Needed] | [More Information Needed] | [More Information Needed] |
| p50 latency | [More Information Needed] | [More Information Needed] | [More Information Needed] |

#### Summary

[More Information Needed — 2–3 honest sentences: where the fine-tune wins, where it still trails the frontier baseline]


## Technical Specifications

### Model Architecture and Objective

Gemma 4 E4B: a decoder-only transformer (~4.5B effective parameters, ~8B with embeddings) with a dedicated vision encoder, hybrid local/global attention, and a 128K-token context window. Fine-tuning objective: supervised next-token prediction on gold `graph.json` completions, with prompt and image tokens masked from the loss.

### Compute Infrastructure

#### Hardware

[More Information Needed — e.g., 1× NVIDIA L4 24GB (Google Colab Pro)]

#### Software

Python, PyTorch, Hugging Face `transformers>=5.10.1`, `trl`, `peft`, `bitsandbytes`, `datasets`.


## More Information

OpenGraph AI is open-source, MCP-first infrastructure for turning heterogeneous data (images, tables, text, audio, video) into semantic knowledge graphs that AI agents can query and reason over. ⭐ [Star the repo](https://github.com/OpenGraphAI/opengraph-ai) — and contribute schemas, test images, or extraction edge cases.

## Model Card Authors

OpenGraph AI team

## Model Card Contact

team@opengraphai.io · [GitHub issues](https://github.com/OpenGraphAI/opengraph-ai/issues)