Image-Text-to-Image
Transformers
PEFT
English
knowledge-graph
information-extraction
scene-graph
image-to-graph
structured-extraction
opengraph
gemma4
lora
mcp
Instructions to use OpenGraph-AI/opengraph-image-gemma4-e4b-v1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use OpenGraph-AI/opengraph-image-gemma4-e4b-v1 with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("OpenGraph-AI/opengraph-image-gemma4-e4b-v1", device_map="auto") - PEFT
How to use OpenGraph-AI/opengraph-image-gemma4-e4b-v1 with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
base_model: google/gemma-4-E4B
|
| 4 |
-
pipeline_tag: image-text-to-
|
| 5 |
library_name: transformers
|
| 6 |
language:
|
| 7 |
- en
|
|
@@ -21,32 +21,13 @@ datasets:
|
|
| 21 |
metrics:
|
| 22 |
- accuracy
|
| 23 |
- f1
|
| 24 |
-
# new_version: # leave unset — only used later to point to a v2 successor
|
| 25 |
-
# --- Eval Results (uncomment and fill once the eval harness produces numbers) ---
|
| 26 |
-
# model-index:
|
| 27 |
-
# - name: opengraph-image-gemma4-e4b-v1
|
| 28 |
-
# results:
|
| 29 |
-
# - task:
|
| 30 |
-
# type: image-text-to-text
|
| 31 |
-
# name: Image-to-Knowledge-Graph Extraction
|
| 32 |
-
# dataset:
|
| 33 |
-
# name: OpenGraph Image Gold v1 (test split)
|
| 34 |
-
# type: OpenGraphAI/opengraph-image-gold-v1
|
| 35 |
-
# split: test
|
| 36 |
-
# metrics:
|
| 37 |
-
# - type: accuracy
|
| 38 |
-
# name: Schema-valid extraction rate
|
| 39 |
-
# value: [XX.X]
|
| 40 |
-
# - type: f1
|
| 41 |
-
# name: Node/Edge F1
|
| 42 |
-
# value: [XX.X]
|
| 43 |
---
|
| 44 |
|
| 45 |
# Model Card for opengraph-image-gemma4-e4b-v1
|
| 46 |
|
| 47 |
<!-- Provide a quick summary of what the model is/does. -->
|
| 48 |
|
| 49 |
-
A fine-tuned **Gemma 4 E4B** that turns
|
| 50 |
|
| 51 |
## Model Details
|
| 52 |
|
|
@@ -55,7 +36,7 @@ A fine-tuned **Gemma 4 E4B** that turns a single image into a valid, schema-fait
|
|
| 55 |
The model does one thing extremely well: **image → knowledge graph, on-schema, every time.** It was trained on verified image→graph gold pairs so that its output always conforms to OpenGraph's `graph.json` contract — stable snake_case node IDs with type prefixes (`entity_`, `concept_`, `event_`, `attr_`), typed edges, and cross-image-mergeable entities. Compared to prompting a general frontier model, it is dramatically cheaper per extraction, runs on a single consumer GPU (or laptop, quantized), and produces structurally consistent output that downstream graph tooling can rely on.
|
| 56 |
|
| 57 |
- **Developed by:** OpenGraph AI
|
| 58 |
-
- **Model type:** Multimodal (image-text-to-
|
| 59 |
- **Language(s):** English
|
| 60 |
- **License:** Apache 2.0 (inherited from Gemma 4; use is additionally subject to Google's [Gemma terms of use](https://ai.google.dev/gemma/terms))
|
| 61 |
- **Finetuned from model:** [`google/gemma-4-E4B`](https://huggingface.co/google/gemma-4-E4B)
|
|
@@ -63,13 +44,13 @@ The model does one thing extremely well: **image → knowledge graph, on-schema,
|
|
| 63 |
### Model Sources
|
| 64 |
|
| 65 |
- **Repository:** https://github.com/OpenGraphAI/opengraph-ai
|
| 66 |
-
- **Demo:** [
|
| 67 |
|
| 68 |
## Uses
|
| 69 |
|
| 70 |
### Direct Use
|
| 71 |
|
| 72 |
-
Feed the model
|
| 73 |
|
| 74 |
### Downstream Use
|
| 75 |
|
|
@@ -143,7 +124,7 @@ Or skip the code entirely and use it through the MCP server:
|
|
| 143 |
|
| 144 |
### Training Data
|
| 145 |
|
| 146 |
-
Trained on **[
|
| 147 |
|
| 148 |
### Training Procedure
|
| 149 |
|
|
@@ -199,15 +180,6 @@ All numbers produced by the open [eval harness](https://github.com/OpenGraphAI/o
|
|
| 199 |
|
| 200 |
[More Information Needed — 2–3 honest sentences: where the fine-tune wins, where it still trails the frontier baseline]
|
| 201 |
|
| 202 |
-
## Environmental Impact
|
| 203 |
-
|
| 204 |
-
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
|
| 205 |
-
|
| 206 |
-
- **Hardware Type:** [More Information Needed — e.g., 1× NVIDIA L4 / A100]
|
| 207 |
-
- **Hours used:** [More Information Needed]
|
| 208 |
-
- **Cloud Provider:** Google Colab
|
| 209 |
-
- **Compute Region:** [More Information Needed]
|
| 210 |
-
- **Carbon Emitted:** [More Information Needed]
|
| 211 |
|
| 212 |
## Technical Specifications
|
| 213 |
|
|
@@ -225,24 +197,10 @@ Gemma 4 E4B: a decoder-only transformer (~4.5B effective parameters, ~8B with em
|
|
| 225 |
|
| 226 |
Python, PyTorch, Hugging Face `transformers>=5.10.1`, `trl`, `peft`, `bitsandbytes`, `datasets`.
|
| 227 |
|
| 228 |
-
## Citation
|
| 229 |
-
|
| 230 |
-
**BibTeX:**
|
| 231 |
-
|
| 232 |
-
```bibtex
|
| 233 |
-
@misc{opengraph2026imagegemma,
|
| 234 |
-
title = {opengraph-image-gemma4-e4b-v1: A schema-faithful image-to-knowledge-graph model},
|
| 235 |
-
author = {OpenGraph AI},
|
| 236 |
-
year = {2026},
|
| 237 |
-
url = {https://huggingface.co/OpenGraphAI/opengraph-image-gemma4-e4b-v1}
|
| 238 |
-
}
|
| 239 |
-
```
|
| 240 |
-
|
| 241 |
-
This model builds on Gemma 4 — please also cite the [Gemma 4 Technical Report](https://arxiv.org/abs/2607.02770) (Gemma Team, 2026).
|
| 242 |
|
| 243 |
## More Information
|
| 244 |
|
| 245 |
-
OpenGraph AI is open-source, MCP-first infrastructure for turning heterogeneous data (images, tables, text) into semantic knowledge graphs that AI agents can query and reason over. ⭐ [Star the repo](https://github.com/OpenGraphAI/opengraph-ai) — and contribute schemas, test images, or extraction edge cases.
|
| 246 |
|
| 247 |
## Model Card Authors
|
| 248 |
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
base_model: google/gemma-4-E4B
|
| 4 |
+
pipeline_tag: image-text-to-image
|
| 5 |
library_name: transformers
|
| 6 |
language:
|
| 7 |
- en
|
|
|
|
| 21 |
metrics:
|
| 22 |
- accuracy
|
| 23 |
- f1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 24 |
---
|
| 25 |
|
| 26 |
# Model Card for opengraph-image-gemma4-e4b-v1
|
| 27 |
|
| 28 |
<!-- Provide a quick summary of what the model is/does. -->
|
| 29 |
|
| 30 |
+
A fine-tuned **Gemma 4 E4B** that turns images into a valid, schema-faithful **knowledge graph** (`graph.json`) — objects, attributes, scene context, text spans, and the relationships between them — ready for AI agents to query and reason over. Built by [OpenGraph AI](https://github.com/OpenGraphAI/opengraph-ai) as the small, cheap, on-device alternative to calling a frontier vision API for every extraction.
|
| 31 |
|
| 32 |
## Model Details
|
| 33 |
|
|
|
|
| 36 |
The model does one thing extremely well: **image → knowledge graph, on-schema, every time.** It was trained on verified image→graph gold pairs so that its output always conforms to OpenGraph's `graph.json` contract — stable snake_case node IDs with type prefixes (`entity_`, `concept_`, `event_`, `attr_`), typed edges, and cross-image-mergeable entities. Compared to prompting a general frontier model, it is dramatically cheaper per extraction, runs on a single consumer GPU (or laptop, quantized), and produces structurally consistent output that downstream graph tooling can rely on.
|
| 37 |
|
| 38 |
- **Developed by:** OpenGraph AI
|
| 39 |
+
- **Model type:** Multimodal (image-text-to-image), decoder-only transformer; QLoRA fine-tune of Gemma 4 E4B
|
| 40 |
- **Language(s):** English
|
| 41 |
- **License:** Apache 2.0 (inherited from Gemma 4; use is additionally subject to Google's [Gemma terms of use](https://ai.google.dev/gemma/terms))
|
| 42 |
- **Finetuned from model:** [`google/gemma-4-E4B`](https://huggingface.co/google/gemma-4-E4B)
|
|
|
|
| 44 |
### Model Sources
|
| 45 |
|
| 46 |
- **Repository:** https://github.com/OpenGraphAI/opengraph-ai
|
| 47 |
+
- **Demo:** [placeholder] <!-- link the graph.html shareable demo when live -->
|
| 48 |
|
| 49 |
## Uses
|
| 50 |
|
| 51 |
### Direct Use
|
| 52 |
|
| 53 |
+
Feed the model images plus the OpenGraph extraction system prompt; it returns a complete `graph.json` — nodes for detected objects (fine-grained labels, normalized bounding boxes), one scene node, attribute nodes, transcribed text spans, and the edges wiring them together. Useful anywhere images need to become structured, queryable knowledge: visual search indexes, dataset annotation, scene understanding, and document/diagram parsing.
|
| 54 |
|
| 55 |
### Downstream Use
|
| 56 |
|
|
|
|
| 124 |
|
| 125 |
### Training Data
|
| 126 |
|
| 127 |
+
Trained on **[placeholder] verified image���`graph.json` gold pairs** ([`OpenGraphAI/opengraph-image-gold-v1`](https://huggingface.co/datasets/OpenGraphAI/opengraph-image-gold-v1)), assembled from two sources: (1) public scene-graph datasets (e.g., Visual Genome) converted programmatically into the `graph.json` schema, and (2) unannotated images labeled by two independent frontier vision models, auto-accepted where both models agreed and human-reviewed otherwise. Every pair passed the OpenGraph Pydantic schema validators before inclusion.
|
| 128 |
|
| 129 |
### Training Procedure
|
| 130 |
|
|
|
|
| 180 |
|
| 181 |
[More Information Needed — 2–3 honest sentences: where the fine-tune wins, where it still trails the frontier baseline]
|
| 182 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 183 |
|
| 184 |
## Technical Specifications
|
| 185 |
|
|
|
|
| 197 |
|
| 198 |
Python, PyTorch, Hugging Face `transformers>=5.10.1`, `trl`, `peft`, `bitsandbytes`, `datasets`.
|
| 199 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 200 |
|
| 201 |
## More Information
|
| 202 |
|
| 203 |
+
OpenGraph AI is open-source, MCP-first infrastructure for turning heterogeneous data (images, tables, text, audio, video) into semantic knowledge graphs that AI agents can query and reason over. ⭐ [Star the repo](https://github.com/OpenGraphAI/opengraph-ai) — and contribute schemas, test images, or extraction edge cases.
|
| 204 |
|
| 205 |
## Model Card Authors
|
| 206 |
|