Update README.md

README.md

@@ -1,3 +1,253 @@
 ---
 license: apache-2.0
+base_model: google/gemma-4-E4B
+pipeline_tag: image-text-to-text
+library_name: transformers
+language:
+- en
+tags:
+- knowledge-graph
+- information-extraction
+- scene-graph
+- image-to-graph
+- structured-extraction
+- opengraph
+- gemma4
+- lora
+- peft
+- mcp
+datasets:
+- OpenGraphAI/opengraph-image-gold-v1
+metrics:
+- accuracy
+- f1
+# new_version:            # leave unset — only used later to point to a v2 successor
+# --- Eval Results (uncomment and fill once the eval harness produces numbers) ---
+# model-index:
+# - name: opengraph-image-gemma4-e4b-v1
+#   results:
+#   - task:
+#       type: image-text-to-text
+#       name: Image-to-Knowledge-Graph Extraction
+#     dataset:
+#       name: OpenGraph Image Gold v1 (test split)
+#       type: OpenGraphAI/opengraph-image-gold-v1
+#       split: test
+#     metrics:
+#     - type: accuracy
+#       name: Schema-valid extraction rate
+#       value: [XX.X]
+#     - type: f1
+#       name: Node/Edge F1
+#       value: [XX.X]
 ---
+
+# Model Card for opengraph-image-gemma4-e4b-v1
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+A fine-tuned **Gemma 4 E4B** that turns a single image into a valid, schema-faithful **knowledge graph** (`graph.json`) — objects, attributes, scene context, text spans, and the relationships between them — ready for AI agents to query and reason over. Built by [OpenGraph AI](https://github.com/OpenGraphAI/opengraph-ai) as the small, cheap, on-device alternative to calling a frontier vision API for every extraction.
+
+## Model Details
+
+### Model Description
+
+The model does one thing extremely well: **image → knowledge graph, on-schema, every time.** It was trained on verified image→graph gold pairs so that its output always conforms to OpenGraph's `graph.json` contract — stable snake_case node IDs with type prefixes (`entity_`, `concept_`, `event_`, `attr_`), typed edges, and cross-image-mergeable entities. Compared to prompting a general frontier model, it is dramatically cheaper per extraction, runs on a single consumer GPU (or laptop, quantized), and produces structurally consistent output that downstream graph tooling can rely on.
+
+- **Developed by:** OpenGraph AI
+- **Model type:** Multimodal (image-text-to-text), decoder-only transformer; QLoRA fine-tune of Gemma 4 E4B
+- **Language(s):** English
+- **License:** Apache 2.0 (inherited from Gemma 4; use is additionally subject to Google's [Gemma terms of use](https://ai.google.dev/gemma/terms))
+- **Finetuned from model:** [`google/gemma-4-E4B`](https://huggingface.co/google/gemma-4-E4B)
+
+### Model Sources
+
+- **Repository:** https://github.com/OpenGraphAI/opengraph-ai
+- **Demo:** [More Information Needed] <!-- link the graph.html shareable demo when live -->
+
+## Uses
+
+### Direct Use
+
+Feed the model one image plus the OpenGraph extraction system prompt; it returns a complete `graph.json` — nodes for detected objects (fine-grained labels, normalized bounding boxes), one scene node, attribute nodes, transcribed text spans, and the edges wiring them together. Useful anywhere images need to become structured, queryable knowledge: visual search indexes, dataset annotation, scene understanding, and document/diagram parsing.
+
+### Downstream Use
+
+The model's intended home is inside the **[opengraph-image MCP server](https://github.com/OpenGraphAI/opengraph-ai)**: register it with Claude Desktop, Cursor, or any MCP-compatible agent, and the agent gains persistent, queryable visual memory — including multi-hop questions across many images ("which components appear in both photos, and what changed between them?"). It also serves as a local extraction backend for robotics scene memory, where per-frame frontier API calls are too slow and expensive.
+
+### Out-of-Scope Use
+
+- **Safety-critical decisions without human review** (e.g., equipment maintenance, medical, or navigation decisions made solely from the extracted graph).
+- **General chat, reasoning, or text generation** — the fine-tune specializes the model for extraction; general capabilities may be degraded relative to the base model.
+- **Images far outside the training distribution** (see Limitations).
+- Any use prohibited by Google's [Gemma prohibited use policy](https://ai.google.dev/gemma/prohibited_use_policy).
+
+## Bias, Risks, and Limitations
+
+- **Extraction errors are silent.** The model can hallucinate objects, miss objects, or assign wrong relationships while still producing perfectly *valid* JSON — structural validity is not factual accuracy.
+- **Distribution sensitivity.** Accuracy degrades on image types unlike the training data (domains, camera angles, lighting, languages in text spans). [More Information Needed — characterize after evaluation]
+- **Inherited bias.** The model inherits biases from Gemma 4's pretraining data, from the public image datasets used for fine-tuning, and from the frontier models used to generate a portion of the training labels.
+- **Schema lock-in.** Output follows OpenGraph's `graph.json` schema; it is not a general-purpose captioner and will not follow arbitrary output formats reliably.
+
+### Recommendations
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. Validate every output with the schema validators shipped in the [OpenGraph repo](https://github.com/OpenGraphAI/opengraph-ai), keep a human in the loop for consequential decisions, and spot-check extractions when applying the model to a new image domain.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+```python
+# pip install -U "transformers>=5.10.1" torch torchvision accelerate
+
+from transformers import AutoProcessor, AutoModelForImageTextToText
+from PIL import Image
+
+MODEL_ID = "OpenGraphAI/opengraph-image-gemma4-e4b-v1"
+
+processor = AutoProcessor.from_pretrained(MODEL_ID)
+model = AutoModelForImageTextToText.from_pretrained(
+    MODEL_ID, dtype="auto", device_map="auto"
+)
+
+SYSTEM_PROMPT = """[More Information Needed — paste the OpenGraph extraction system prompt]"""
+
+image = Image.open("your_image.jpg").convert("RGB")
+messages = [
+    {"role": "system", "content": SYSTEM_PROMPT},
+    {"role": "user", "content": [
+        {"type": "image", "image": image},
+        {"type": "text", "text": "Extract the knowledge graph from this image."},
+    ]},
+]
+
+inputs = processor.apply_chat_template(
+    messages, add_generation_prompt=True, tokenize=True,
+    return_dict=True, return_tensors="pt",
+).to(model.device)
+
+outputs = model.generate(**inputs, max_new_tokens=2048)
+graph_json = processor.decode(
+    outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True
+)
+print(graph_json)  # -> valid graph.json
+```
+
+Or skip the code entirely and use it through the MCP server:
+
+```bash
+[More Information Needed — one-line MCP install command]
+```
+
+## Training Details
+
+### Training Data
+
+Trained on **[More Information Needed — N] verified image→`graph.json` gold pairs** ([`OpenGraphAI/opengraph-image-gold-v1`](https://huggingface.co/datasets/OpenGraphAI/opengraph-image-gold-v1)), assembled from two sources: (1) public scene-graph datasets (e.g., Visual Genome) converted programmatically into the `graph.json` schema, and (2) unannotated images labeled by two independent frontier vision models, auto-accepted where both models agreed and human-reviewed otherwise. Every pair passed the OpenGraph Pydantic schema validators before inclusion.
+
+### Training Procedure
+
+Supervised fine-tuning (SFT) with **QLoRA**: the base model frozen in 4-bit NF4 quantization, with LoRA adapters (rank 16, all linear layers, plus `lm_head`/`embed_tokens`) trained via Hugging Face TRL's `SFTTrainer`, following [Google's official Gemma 4 vision QLoRA guide](https://ai.google.dev/gemma/docs/core/huggingface_vision_finetune_qlora).
+
+#### Preprocessing
+
+Each example is formatted as a three-turn conversation (system = schema instruction, user = image + extraction request, assistant = gold `graph.json`) and templated with the official Gemma 4 chat template. Images are processed at their native aspect ratio; image tokens are masked out of the training loss.
+
+#### Training Hyperparameters
+
+- **Training regime:** bf16 mixed precision (4-bit NF4 quantized base, bf16 compute)
+- **LoRA:** r=16, alpha=16, dropout=0.05, target_modules=all-linear
+- **Epochs:** 3
+- **Learning rate:** 2e-4 (constant schedule)
+- **Per-device batch size:** 1
+- **Max grad norm:** 0.3
+
+#### Speeds, Sizes, Times
+
+[More Information Needed — fill after training: total training time, adapter size, merged checkpoint size]
+
+## Evaluation
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+A held-out **test split (5%)** of [`OpenGraphAI/opengraph-image-gold-v1`](https://huggingface.co/datasets/OpenGraphAI/opengraph-image-gold-v1), never seen during training.
+
+#### Factors
+
+Results are disaggregated by image source/domain (converted scene-graph data vs. frontier-labeled robot/inspection frames). [More Information Needed — add further factors after evaluation]
+
+#### Metrics
+
+- **Schema-valid extraction rate** — % of outputs that parse as JSON *and* pass the OpenGraph Pydantic validators on the first attempt. Chosen because downstream graph tooling hard-fails on invalid output; this is the reliability number that matters in production.
+- **Node F1 / Edge F1** — precision and recall of predicted nodes and edges against the gold graph, matched on normalized label + type. Measures whether the *content* of the graph is right, not just its shape.
+- **Cost per 1,000 images & p50 latency** — the practical case for a small fine-tune over a frontier API.
+
+### Results
+
+All numbers produced by the open [eval harness](https://github.com/OpenGraphAI/opengraph-ai) and reproducible from the linked script.
+
+| Metric | This model | Base Gemma 4 E4B-it | Frontier API baseline |
+|---|---|---|---|
+| Schema-valid rate | [More Information Needed] | [More Information Needed] | [More Information Needed] |
+| Node/Edge F1 | [More Information Needed] | [More Information Needed] | [More Information Needed] |
+| $ / 1k images | [More Information Needed] | [More Information Needed] | [More Information Needed] |
+| p50 latency | [More Information Needed] | [More Information Needed] | [More Information Needed] |
+
+#### Summary
+
+[More Information Needed — 2–3 honest sentences: where the fine-tune wins, where it still trails the frontier baseline]
+
+## Environmental Impact
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed — e.g., 1× NVIDIA L4 / A100]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** Google Colab
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications
+
+### Model Architecture and Objective
+
+Gemma 4 E4B: a decoder-only transformer (~4.5B effective parameters, ~8B with embeddings) with a dedicated vision encoder, hybrid local/global attention, and a 128K-token context window. Fine-tuning objective: supervised next-token prediction on gold `graph.json` completions, with prompt and image tokens masked from the loss.
+
+### Compute Infrastructure
+
+#### Hardware
+
+[More Information Needed — e.g., 1× NVIDIA L4 24GB (Google Colab Pro)]
+
+#### Software
+
+Python, PyTorch, Hugging Face `transformers>=5.10.1`, `trl`, `peft`, `bitsandbytes`, `datasets`.
+
+## Citation
+
+**BibTeX:**
+
+```bibtex
+@misc{opengraph2026imagegemma,
+  title  = {opengraph-image-gemma4-e4b-v1: A schema-faithful image-to-knowledge-graph model},
+  author = {OpenGraph AI},
+  year   = {2026},
+  url    = {https://huggingface.co/OpenGraphAI/opengraph-image-gemma4-e4b-v1}
+}
+```
+
+This model builds on Gemma 4 — please also cite the [Gemma 4 Technical Report](https://arxiv.org/abs/2607.02770) (Gemma Team, 2026).
+
+## More Information
+
+OpenGraph AI is open-source, MCP-first infrastructure for turning heterogeneous data (images, tables, text) into semantic knowledge graphs that AI agents can query and reason over. ⭐ [Star the repo](https://github.com/OpenGraphAI/opengraph-ai) — and contribute schemas, test images, or extraction edge cases.
+
+## Model Card Authors
+
+OpenGraph AI team
+
+## Model Card Contact
+
+team@opengraphai.io · [GitHub issues](https://github.com/OpenGraphAI/opengraph-ai/issues)