--- license: apache-2.0 base_model: google/gemma-4-E4B pipeline_tag: image-text-to-image library_name: transformers language: - en tags: - knowledge-graph - information-extraction - scene-graph - image-to-graph - structured-extraction - opengraph - gemma4 - lora - peft - mcp datasets: - OpenGraphAI/opengraph-image-gold-v1 metrics: - accuracy - f1 --- # Model Card for opengraph-image-gemma4-e4b-v1 A fine-tuned **Gemma 4 E4B** that turns images into a valid, schema-faithful **knowledge graph** (`graph.json`) — objects, attributes, scene context, text spans, and the relationships between them — ready for AI agents to query and reason over. Built by [OpenGraph AI](https://github.com/OpenGraphAI/opengraph-ai) as the small, cheap, on-device alternative to calling a frontier vision API for every extraction. ## Model Details ### Model Description The model does one thing extremely well: **image → knowledge graph, on-schema, every time.** It was trained on verified image→graph gold pairs so that its output always conforms to OpenGraph's `graph.json` contract — stable snake_case node IDs with type prefixes (`entity_`, `concept_`, `event_`, `attr_`), typed edges, and cross-image-mergeable entities. Compared to prompting a general frontier model, it is dramatically cheaper per extraction, runs on a single consumer GPU (or laptop, quantized), and produces structurally consistent output that downstream graph tooling can rely on. - **Developed by:** OpenGraph AI - **Model type:** Multimodal (image-text-to-image), decoder-only transformer; QLoRA fine-tune of Gemma 4 E4B - **Language(s):** English - **License:** Apache 2.0 (inherited from Gemma 4; use is additionally subject to Google's [Gemma terms of use](https://ai.google.dev/gemma/terms)) - **Finetuned from model:** [`google/gemma-4-E4B`](https://huggingface.co/google/gemma-4-E4B) ### Model Sources - **Repository:** https://github.com/OpenGraphAI/opengraph-ai - **Demo:** [placeholder] ## Uses ### Direct Use Feed the model images plus the OpenGraph extraction system prompt; it returns a complete `graph.json` — nodes for detected objects (fine-grained labels, normalized bounding boxes), one scene node, attribute nodes, transcribed text spans, and the edges wiring them together. Useful anywhere images need to become structured, queryable knowledge: visual search indexes, dataset annotation, scene understanding, and document/diagram parsing. ### Downstream Use The model's intended home is inside the **[opengraph-image MCP server](https://github.com/OpenGraphAI/opengraph-ai)**: register it with Claude Desktop, Cursor, or any MCP-compatible agent, and the agent gains persistent, queryable visual memory — including multi-hop questions across many images ("which components appear in both photos, and what changed between them?"). It also serves as a local extraction backend for robotics scene memory, where per-frame frontier API calls are too slow and expensive. ### Out-of-Scope Use - **Safety-critical decisions without human review** (e.g., equipment maintenance, medical, or navigation decisions made solely from the extracted graph). - **General chat, reasoning, or text generation** — the fine-tune specializes the model for extraction; general capabilities may be degraded relative to the base model. - **Images far outside the training distribution** (see Limitations). - Any use prohibited by Google's [Gemma prohibited use policy](https://ai.google.dev/gemma/prohibited_use_policy). ## Bias, Risks, and Limitations - **Extraction errors are silent.** The model can hallucinate objects, miss objects, or assign wrong relationships while still producing perfectly *valid* JSON — structural validity is not factual accuracy. - **Distribution sensitivity.** Accuracy degrades on image types unlike the training data (domains, camera angles, lighting, languages in text spans). [More Information Needed — characterize after evaluation] - **Inherited bias.** The model inherits biases from Gemma 4's pretraining data, from the public image datasets used for fine-tuning, and from the frontier models used to generate a portion of the training labels. - **Schema lock-in.** Output follows OpenGraph's `graph.json` schema; it is not a general-purpose captioner and will not follow arbitrary output formats reliably. ### Recommendations Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. Validate every output with the schema validators shipped in the [OpenGraph repo](https://github.com/OpenGraphAI/opengraph-ai), keep a human in the loop for consequential decisions, and spot-check extractions when applying the model to a new image domain. ## How to Get Started with the Model Use the code below to get started with the model. ```python # pip install -U "transformers>=5.10.1" torch torchvision accelerate from transformers import AutoProcessor, AutoModelForImageTextToText from PIL import Image MODEL_ID = "OpenGraphAI/opengraph-image-gemma4-e4b-v1" processor = AutoProcessor.from_pretrained(MODEL_ID) model = AutoModelForImageTextToText.from_pretrained( MODEL_ID, dtype="auto", device_map="auto" ) SYSTEM_PROMPT = """[More Information Needed — paste the OpenGraph extraction system prompt]""" image = Image.open("your_image.jpg").convert("RGB") messages = [ {"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": [ {"type": "image", "image": image}, {"type": "text", "text": "Extract the knowledge graph from this image."}, ]}, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=2048) graph_json = processor.decode( outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True ) print(graph_json) # -> valid graph.json ``` Or skip the code entirely and use it through the MCP server: ```bash [More Information Needed — one-line MCP install command] ``` ## Training Details ### Training Data Trained on **[placeholder] verified image→`graph.json` gold pairs** ([`OpenGraphAI/opengraph-image-gold-v1`](https://huggingface.co/datasets/OpenGraphAI/opengraph-image-gold-v1)), assembled from two sources: (1) public scene-graph datasets (e.g., Visual Genome) converted programmatically into the `graph.json` schema, and (2) unannotated images labeled by two independent frontier vision models, auto-accepted where both models agreed and human-reviewed otherwise. Every pair passed the OpenGraph Pydantic schema validators before inclusion. ### Training Procedure Supervised fine-tuning (SFT) with **QLoRA**: the base model frozen in 4-bit NF4 quantization, with LoRA adapters (rank 16, all linear layers, plus `lm_head`/`embed_tokens`) trained via Hugging Face TRL's `SFTTrainer`, following [Google's official Gemma 4 vision QLoRA guide](https://ai.google.dev/gemma/docs/core/huggingface_vision_finetune_qlora). #### Preprocessing Each example is formatted as a three-turn conversation (system = schema instruction, user = image + extraction request, assistant = gold `graph.json`) and templated with the official Gemma 4 chat template. Images are processed at their native aspect ratio; image tokens are masked out of the training loss. #### Training Hyperparameters - **Training regime:** bf16 mixed precision (4-bit NF4 quantized base, bf16 compute) - **LoRA:** r=16, alpha=16, dropout=0.05, target_modules=all-linear - **Epochs:** 3 - **Learning rate:** 2e-4 (constant schedule) - **Per-device batch size:** 1 - **Max grad norm:** 0.3 #### Speeds, Sizes, Times [More Information Needed — fill after training: total training time, adapter size, merged checkpoint size] ## Evaluation ### Testing Data, Factors & Metrics #### Testing Data A held-out **test split (5%)** of [`OpenGraphAI/opengraph-image-gold-v1`](https://huggingface.co/datasets/OpenGraphAI/opengraph-image-gold-v1), never seen during training. #### Factors Results are disaggregated by image source/domain (converted scene-graph data vs. frontier-labeled robot/inspection frames). [More Information Needed — add further factors after evaluation] #### Metrics - **Schema-valid extraction rate** — % of outputs that parse as JSON *and* pass the OpenGraph Pydantic validators on the first attempt. Chosen because downstream graph tooling hard-fails on invalid output; this is the reliability number that matters in production. - **Node F1 / Edge F1** — precision and recall of predicted nodes and edges against the gold graph, matched on normalized label + type. Measures whether the *content* of the graph is right, not just its shape. - **Cost per 1,000 images & p50 latency** — the practical case for a small fine-tune over a frontier API. ### Results All numbers produced by the open [eval harness](https://github.com/OpenGraphAI/opengraph-ai) and reproducible from the linked script. | Metric | This model | Base Gemma 4 E4B-it | Frontier API baseline | |---|---|---|---| | Schema-valid rate | [More Information Needed] | [More Information Needed] | [More Information Needed] | | Node/Edge F1 | [More Information Needed] | [More Information Needed] | [More Information Needed] | | $ / 1k images | [More Information Needed] | [More Information Needed] | [More Information Needed] | | p50 latency | [More Information Needed] | [More Information Needed] | [More Information Needed] | #### Summary [More Information Needed — 2–3 honest sentences: where the fine-tune wins, where it still trails the frontier baseline] ## Technical Specifications ### Model Architecture and Objective Gemma 4 E4B: a decoder-only transformer (~4.5B effective parameters, ~8B with embeddings) with a dedicated vision encoder, hybrid local/global attention, and a 128K-token context window. Fine-tuning objective: supervised next-token prediction on gold `graph.json` completions, with prompt and image tokens masked from the loss. ### Compute Infrastructure #### Hardware [More Information Needed — e.g., 1× NVIDIA L4 24GB (Google Colab Pro)] #### Software Python, PyTorch, Hugging Face `transformers>=5.10.1`, `trl`, `peft`, `bitsandbytes`, `datasets`. ## More Information OpenGraph AI is open-source, MCP-first infrastructure for turning heterogeneous data (images, tables, text, audio, video) into semantic knowledge graphs that AI agents can query and reason over. ⭐ [Star the repo](https://github.com/OpenGraphAI/opengraph-ai) — and contribute schemas, test images, or extraction edge cases. ## Model Card Authors OpenGraph AI team ## Model Card Contact team@opengraphai.io · [GitHub issues](https://github.com/OpenGraphAI/opengraph-ai/issues)