ysharma HF Staff commited on
Commit
af888c6
·
verified ·
1 Parent(s): 6fe13ea

Workflow1111 — Automatic1111-style diffusion studio on gr.Workflow

Browse files
Files changed (7) hide show
  1. README.md +32 -8
  2. build_workflow.py +19 -16
  3. make_samples.py +18 -2
  4. nodes.py +91 -13
  5. samples/with_parameters.png +2 -2
  6. test_nodes.py +9 -3
  7. workflow.json +48 -32
README.md CHANGED
@@ -80,12 +80,12 @@ install too.
80
 
81
  ```
82
  14 references → 32 operators → 18 subjects 73 edges
83
- ├─ 26 fn 19 pure-local · 7 calling InferenceClient
84
- ├─ 4 model HF Inference Providers
85
  └─ 2 space Gradio Spaces on the Hub
86
  ```
87
 
88
- 19 of the 26 `fn` nodes are pure local Pillow/numpy — all the prompt logic,
89
  post-processing, annotators, masking, grid composition and metadata parsing —
90
  so most of the app keeps working with no token, no quota and no network. 13
91
  nodes in total leave the machine.
@@ -95,7 +95,7 @@ nodes in total leave the machine.
95
  | File | What it is |
96
  |---|---|
97
  | `app.py` | Entry point — 12 lines of actual wiring |
98
- | `nodes.py` | The 19 bound functions (the `fn` node library) |
99
  | `build_workflow.py` | **Generates + verifies** `workflow.json` |
100
  | `workflow.json` | The committed graph |
101
  | `test_nodes.py` | 53 offline unit tests (~2s) |
@@ -116,9 +116,9 @@ and no node is orphaned.
116
 
117
  ---
118
 
119
- ## Five gotchas this app is built around
120
 
121
- All five were found by probing gradio 6.22.0 / huggingface_hub 1.26.0
122
  directly, not from the docs. They are the difference between "renders on the
123
  canvas" and "actually runs".
124
 
@@ -139,8 +139,9 @@ replying "Hello! It seems like your message might be missing something").
139
 
140
  The fix is to stop using `model` nodes wherever the control surface is richer
141
  than the schema: `txt2img`, `chat_llm` and `interrogate` are `fn` nodes that
142
- call `InferenceClient` themselves. `fn` ports are never rewritten. The four
143
- remaining `model` nodes have ports exactly equal to their schema, and
 
144
  `build_workflow.py` now **refuses to build** if that ever stops being true.
145
 
146
  A useful side effect: an `fn` node can validate. `interrogate` requires its
@@ -203,6 +204,29 @@ So `_emit` returns **both**: `{"path": <temp file>, "url": <data: URI>}`.
203
  `_from_output` takes `path` first (endpoint happy), the frontend and `_img_url`
204
  take `url` first (canvas and providers happy).
205
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
206
  Reference-node *defaults* are a separate case with the opposite answer: the
207
  canvas strips `path` out of a graph default and keeps only `url`, so the sample
208
  images are referenced by their **public Hub URL** — the one form that renders
 
80
 
81
  ```
82
  14 references → 32 operators → 18 subjects 73 edges
83
+ ├─ 28 fn 19 pure-local · 9 calling InferenceClient
84
+ ├─ 2 model HF Inference Providers
85
  └─ 2 space Gradio Spaces on the Hub
86
  ```
87
 
88
+ 19 of the 28 `fn` nodes are pure local Pillow/numpy — all the prompt logic,
89
  post-processing, annotators, masking, grid composition and metadata parsing —
90
  so most of the app keeps working with no token, no quota and no network. 13
91
  nodes in total leave the machine.
 
95
  | File | What it is |
96
  |---|---|
97
  | `app.py` | Entry point — 12 lines of actual wiring |
98
+ | `nodes.py` | The 21 bound functions (the `fn` node library) |
99
  | `build_workflow.py` | **Generates + verifies** `workflow.json` |
100
  | `workflow.json` | The committed graph |
101
  | `test_nodes.py` | 53 offline unit tests (~2s) |
 
116
 
117
  ---
118
 
119
+ ## Six gotchas this app is built around
120
 
121
+ All six were found by probing gradio 6.22.0 / huggingface_hub 1.26.0
122
  directly, not from the docs. They are the difference between "renders on the
123
  canvas" and "actually runs".
124
 
 
139
 
140
  The fix is to stop using `model` nodes wherever the control surface is richer
141
  than the schema: `txt2img`, `chat_llm` and `interrogate` are `fn` nodes that
142
+ call `InferenceClient` themselves. `fn` ports are never rewritten. The two
143
+ remaining `model` nodes (`image_to_image`) have ports exactly equal to their
144
+ schema, and
145
  `build_workflow.py` now **refuses to build** if that ever stops being true.
146
 
147
  A useful side effect: an `fn` node can validate. `interrogate` requires its
 
204
  `_from_output` takes `path` first (endpoint happy), the frontend and `_img_url`
205
  take `url` first (canvas and providers happy).
206
 
207
+ ### 6. A `json` port silently destroys its value in the canvas
208
+
209
+ The canvas serializes a `json`-typed port with JavaScript's `String(obj)`
210
+ instead of `JSON.stringify`, so the receiving node gets the literal six-word
211
+ string `"[object Object]"`. Everything downstream then sees *no data*, with no
212
+ error anywhere:
213
+
214
+ - DETR detections reached `draw_detections` as `"[object Object]"` → zero boxes
215
+ → an annotated image identical to the input, and
216
+ `mask_from_detections` failing with "No detections matched" at **every**
217
+ `min_score`.
218
+ - ViT labels reached `top_labels` the same way → "No labels above the score
219
+ threshold".
220
+ - `png_info`'s field dict reached its output node as `"[object Object]"`.
221
+
222
+ Both the executor and the REST API handle `json` ports perfectly, so this is
223
+ invisible to `test_pipelines.py` *and* `test_api.py` — only the canvas is
224
+ affected. The graph therefore contains **no `json` ports at all**: structured
225
+ data travels as JSON *text*, which survives, and `_as_list` parses it back.
226
+ `detect_objects` and `classify_image` are `fn` nodes calling `InferenceClient`
227
+ for the same reason a `model` node could not be used (their output port type is
228
+ fixed by the endpoint schema — gotcha #1).
229
+
230
  Reference-node *defaults* are a separate case with the opposite answer: the
231
  canvas strips `path` out of a graph default and keeps only `url`, so the sample
232
  images are referenced by their **public Hub URL** — the one form that renders
build_workflow.py CHANGED
@@ -393,16 +393,19 @@ fn("op_clean_interrogate", "clean_prompt", COL[2], Y, label="② Tidy up",
393
  data={"max_tags": 45},
394
  outputs=[("out_0", "prompt", "text")])
395
 
396
- model("op_classify", CLASSIFY_MODEL, "image_classification", "image-classification",
397
- COL[1], Y + 260, label="③ Classify · ViT",
398
- inputs=[("image", "image", "image", True)],
399
- outputs=[("out_0", "Labels", "json")])
 
 
 
400
 
401
  fn("op_labels", "top_labels", COL[2], Y + 260, label="④ Rank labels",
402
- types={"labels": "json", "top_k": "number", "min_score": "number"},
403
  required=("labels",),
404
  data={"top_k": 5, "min_score": 0.01},
405
- outputs=[("out_0", "table", "text"), ("out_1", "rows", "json")])
406
 
407
  out("sub_interrogated", "🔍 Recovered prompt", "text", COL[3], Y)
408
  out("sub_labels", "🏷 Classification", "text", COL[3], Y + 260)
@@ -410,7 +413,7 @@ out("sub_labels", "🏷 Classification", "text", COL[3], Y + 260)
410
  link("ref_interrogate_image.out", "op_vlm.in_image")
411
  link("op_vlm.out_0", "op_clean_interrogate.in_raw")
412
  link("op_clean_interrogate.out_0", "sub_interrogated.in")
413
- link("ref_interrogate_image.out", "op_classify.image")
414
  link("op_classify.out_0", "op_labels.in_labels")
415
  link("op_labels.out_0", "sub_labels.in")
416
 
@@ -420,20 +423,20 @@ link("op_labels.out_0", "sub_labels.in")
420
  Y = 2200
421
  ref("ref_detect_image", "Image to analyse", "image", COL[0], Y, sample("detect.jpg"))
422
 
423
- model("op_detect", DETECT_MODEL, "object_detection", "object-detection", COL[1], Y,
424
- label=" Detect · DETR",
425
- inputs=[("image", "image", "image", True)],
426
- outputs=[("out_0", "Detections", "json")])
427
 
428
  fn("op_draw", "draw_detections", COL[2], Y, label="② Annotate",
429
- types={"image": "image", "detections": "json", "min_score": "number",
430
  "show_labels": "boolean"},
431
  required=("image", "detections"),
432
  data={"min_score": 0.5, "show_labels": True},
433
  outputs=[("out_0", "image", "image"), ("out_1", "summary", "text")])
434
 
435
  fn("op_mask", "mask_from_detections", COL[2], Y + 300, label="③ Build inpaint mask",
436
- types={"image": "image", "detections": "json", "min_score": "number",
437
  "feather": "number", "invert": "boolean", "preview": "boolean"},
438
  required=("image", "detections"),
439
  data={"label_filter": "", "min_score": 0.5, "feather": 8,
@@ -444,7 +447,7 @@ out("sub_detected", "📦 Detected objects", "image", COL[3], Y)
444
  out("sub_detect_summary", "📝 Detection summary", "text", COL[3], Y + 300)
445
  out("sub_mask", "🎭 Inpaint mask", "image", COL[4], Y + 300)
446
 
447
- link("ref_detect_image.out", "op_detect.image")
448
  link("ref_detect_image.out", "op_draw.in_image")
449
  link("op_detect.out_0", "op_draw.in_detections")
450
  link("ref_detect_image.out", "op_mask.in_image")
@@ -559,10 +562,10 @@ ref("ref_pnginfo_image", "PNG to inspect", "image", COL[0], Y,
559
 
560
  fn("op_pnginfo", "png_info", COL[1], Y, label="Read PNG metadata",
561
  types={"image": "image"}, required=("image",),
562
- outputs=[("out_0", "report", "text"), ("out_1", "fields", "json")])
563
 
564
  out("sub_png_report", "🧾 PNG info", "text", COL[2], Y)
565
- out("sub_png_fields", "🧮 Parsed fields", "json", COL[3], Y)
566
 
567
  link("ref_pnginfo_image.out", "op_pnginfo.in_image")
568
  link("op_pnginfo.out_0", "sub_png_report.in")
 
393
  data={"max_tags": 45},
394
  outputs=[("out_0", "prompt", "text")])
395
 
396
+ # `fn`, not `model`: a `json` output port reaches the canvas as the literal
397
+ # string "[object Object]" (JS String(obj) instead of JSON.stringify), so the
398
+ # labels never survive the edge. Text ports carrying JSON do.
399
+ fn("op_classify", "classify_image", COL[1], Y + 260, label="③ Classify · ViT",
400
+ types={"image": "image"}, required=("image",),
401
+ data={"model_id": CLASSIFY_MODEL},
402
+ outputs=[("out_0", "labels", "text")])
403
 
404
  fn("op_labels", "top_labels", COL[2], Y + 260, label="④ Rank labels",
405
+ types={"labels": "text", "top_k": "number", "min_score": "number"},
406
  required=("labels",),
407
  data={"top_k": 5, "min_score": 0.01},
408
+ outputs=[("out_0", "table", "text"), ("out_1", "rows", "text")])
409
 
410
  out("sub_interrogated", "🔍 Recovered prompt", "text", COL[3], Y)
411
  out("sub_labels", "🏷 Classification", "text", COL[3], Y + 260)
 
413
  link("ref_interrogate_image.out", "op_vlm.in_image")
414
  link("op_vlm.out_0", "op_clean_interrogate.in_raw")
415
  link("op_clean_interrogate.out_0", "sub_interrogated.in")
416
+ link("ref_interrogate_image.out", "op_classify.in_image")
417
  link("op_classify.out_0", "op_labels.in_labels")
418
  link("op_labels.out_0", "sub_labels.in")
419
 
 
423
  Y = 2200
424
  ref("ref_detect_image", "Image to analyse", "image", COL[0], Y, sample("detect.jpg"))
425
 
426
+ fn("op_detect", "detect_objects", COL[1], Y, label="① Detect · DETR",
427
+ types={"image": "image", "min_score": "number"}, required=("image",),
428
+ data={"model_id": DETECT_MODEL, "min_score": 0.0},
429
+ outputs=[("out_0", "detections", "text")])
430
 
431
  fn("op_draw", "draw_detections", COL[2], Y, label="② Annotate",
432
+ types={"image": "image", "detections": "text", "min_score": "number",
433
  "show_labels": "boolean"},
434
  required=("image", "detections"),
435
  data={"min_score": 0.5, "show_labels": True},
436
  outputs=[("out_0", "image", "image"), ("out_1", "summary", "text")])
437
 
438
  fn("op_mask", "mask_from_detections", COL[2], Y + 300, label="③ Build inpaint mask",
439
+ types={"image": "image", "detections": "text", "min_score": "number",
440
  "feather": "number", "invert": "boolean", "preview": "boolean"},
441
  required=("image", "detections"),
442
  data={"label_filter": "", "min_score": 0.5, "feather": 8,
 
447
  out("sub_detect_summary", "📝 Detection summary", "text", COL[3], Y + 300)
448
  out("sub_mask", "🎭 Inpaint mask", "image", COL[4], Y + 300)
449
 
450
+ link("ref_detect_image.out", "op_detect.in_image")
451
  link("ref_detect_image.out", "op_draw.in_image")
452
  link("op_detect.out_0", "op_draw.in_detections")
453
  link("ref_detect_image.out", "op_mask.in_image")
 
562
 
563
  fn("op_pnginfo", "png_info", COL[1], Y, label="Read PNG metadata",
564
  types={"image": "image"}, required=("image",),
565
+ outputs=[("out_0", "report", "text"), ("out_1", "fields", "text")])
566
 
567
  out("sub_png_report", "🧾 PNG info", "text", COL[2], Y)
568
+ out("sub_png_fields", "🧮 Parsed fields", "text", COL[3], Y)
569
 
570
  link("ref_pnginfo_image.out", "op_pnginfo.in_image")
571
  link("op_pnginfo.out_0", "sub_png_report.in")
make_samples.py CHANGED
@@ -99,7 +99,22 @@ def main():
99
  steps, cfg, seed, img.width, img.height,
100
  "black-forest-labs/FLUX.1-schnell")
101
  stamped = N.postprocess(N._emit(img), 1, "Lanczos", 0, 1, 1, 1, 0, 0, 0, "", info)
102
- shutil.copyfile(stamped["path"], png)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
103
  print(f" made with_parameters.png {os.path.getsize(png)//1024} KB "
104
  "— PNG Info, with a real embedded parameter block")
105
  manifest["ref_pnginfo_image"] = "with_parameters.png"
@@ -116,7 +131,8 @@ def main():
116
  labels = sorted({d["label"] for d in det[0]} if isinstance(det[0], list) else set())
117
  print(f" detect.jpg → DETR finds: {labels or 'NOTHING (bad sample)'}")
118
 
119
- report, fields = N.png_info(png)
 
120
  ok = fields.get("prompt", "").startswith("a red fox curled")
121
  print(f" with_parameters.png → embedded params readable: {ok} "
122
  f"(seed={fields.get('seed')}, size={fields.get('size')})")
 
99
  steps, cfg, seed, img.width, img.height,
100
  "black-forest-labs/FLUX.1-schnell")
101
  stamped = N.postprocess(N._emit(img), 1, "Lanczos", 0, 1, 1, 1, 0, 0, 0, "", info)
102
+ # Quantize to a 256-colour palette before shipping. This sample has to
103
+ # stay PNG (a JPEG cannot carry the `parameters` text chunk), and a
104
+ # full-colour 768×512 photo PNG is ~471 KB — slow enough over the Hub
105
+ # (~4.8s) that the reference node looks broken while it loads. The
106
+ # palette version is ~161 KB and keeps the text chunk and the
107
+ # dimensions, so the embedded `Size:` still matches the image.
108
+ from PIL import PngImagePlugin
109
+ full = Image.open(stamped["path"])
110
+ full.load()
111
+ meta = PngImagePlugin.PngInfo()
112
+ for key, value in (full.info or {}).items():
113
+ if isinstance(value, str):
114
+ meta.add_text(key, value)
115
+ quantized = full.convert("RGB").quantize(
116
+ colors=256, method=Image.MEDIANCUT, dither=Image.FLOYDSTEINBERG)
117
+ quantized.save(png, format="PNG", optimize=True, pnginfo=meta)
118
  print(f" made with_parameters.png {os.path.getsize(png)//1024} KB "
119
  "— PNG Info, with a real embedded parameter block")
120
  manifest["ref_pnginfo_image"] = "with_parameters.png"
 
131
  labels = sorted({d["label"] for d in det[0]} if isinstance(det[0], list) else set())
132
  print(f" detect.jpg → DETR finds: {labels or 'NOTHING (bad sample)'}")
133
 
134
+ report, fields_json = N.png_info(png)
135
+ fields = json.loads(fields_json)
136
  ok = fields.get("prompt", "").startswith("a red fox curled")
137
  print(f" with_parameters.png → embedded params readable: {ok} "
138
  f"(seed={fields.get('seed')}, size={fields.get('size')})")
nodes.py CHANGED
@@ -15,12 +15,15 @@ Two conventions matter, and both are load-bearing (see the module docstring in
15
  it can be a ``{"path"/"url"}`` dict, a ``data:`` URI, an ``http(s)`` URL, a
16
  ``/gradio_api/file=`` reference, or a plain path. `_load_image` normalizes
17
  all of them.
18
- 2. **Image outputs are always ``data:`` URIs.** That is the one shape that both
19
- renders in the canvas *and* chains correctly into a `model` node's
20
- ``image_to_image`` port. Returning the ``{"path", "url"}`` dict that model
21
- nodes emit would break the chain, because gradio's `_img_url()` prefers the
22
- ``url`` key and a ``/gradio_api/file=`` path means nothing to a remote
23
- provider.
 
 
 
24
 
25
  Everything returned must be JSON-serializable — that is the contract for
26
  `bind=` functions.
@@ -707,6 +710,74 @@ def interrogate(image, instruction, model_id, max_tokens,
707
  max_tokens=_num(max_tokens, 512, lo=32, hi=4096, integer=True))
708
 
709
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
710
  def top_labels(labels, top_k, min_score):
711
  """Format an image-classification payload. Returns (text, json)."""
712
  items = _as_list(labels)
@@ -725,13 +796,15 @@ def top_labels(labels, top_k, min_score):
725
  rows = rows[:k]
726
 
727
  if not rows:
728
- return "No labels above the score threshold.", []
729
  width = max(len(r["label"]) for r in rows)
730
  lines = [
731
  f"{r['label']:<{width}} {r['score'] * 100:5.1f}% {'█' * max(1, int(r['score'] * 24))}"
732
  for r in rows
733
  ]
734
- return "\n".join(lines), rows
 
 
735
 
736
 
737
  # ─────────────────────────────────────────────────────────────────────────────
@@ -1086,8 +1159,10 @@ def draw_detections(image, detections, min_score, show_labels):
1086
  found = _boxes(detections, min_score)
1087
 
1088
  draw = ImageDraw.Draw(img, "RGBA")
1089
- stroke = max(2, int(min(img.size) * 0.005))
1090
- font = _font(max(13, int(min(img.size) * 0.028)))
 
 
1091
 
1092
  for i, det in enumerate(found):
1093
  color = _PALETTE[i % len(_PALETTE)]
@@ -1209,8 +1284,9 @@ def contact_sheet(image_1, image_2, image_3, image_4, labels, columns, gap, titl
1209
  def png_info(image):
1210
  """A1111's 'PNG Info' tab: recover generation parameters from a file.
1211
 
1212
- Returns (report, fields). Reads the ``parameters`` text chunk that
1213
- `postprocess` writes, so images this app produced round-trip exactly.
 
1214
  """
1215
  img = _load_image(image)
1216
  img.load() # force chunk parsing so text metadata is populated
@@ -1260,7 +1336,7 @@ def png_info(image):
1260
  + ("Other metadata:\n" + "\n".join(extra) if extra else
1261
  "This image carries no text metadata at all.")
1262
  )
1263
- return head + body, fields
1264
 
1265
 
1266
  # What `app.py` binds onto the canvas. Keys must match the "fn" field of each
@@ -1276,6 +1352,8 @@ BIND = {
1276
  "txt2img": txt2img,
1277
  "chat_llm": chat_llm,
1278
  "interrogate": interrogate,
 
 
1279
  "top_labels": top_labels,
1280
  "postprocess": postprocess,
1281
  "prep_image": prep_image,
 
15
  it can be a ``{"path"/"url"}`` dict, a ``data:`` URI, an ``http(s)`` URL, a
16
  ``/gradio_api/file=`` reference, or a plain path. `_load_image` normalizes
17
  all of them.
18
+ 2. **Image outputs are ``{"path": <file>, "url": <data: URI>}``** see `_emit`.
19
+ The REST endpoint needs the real file, the canvas and any chained `model`
20
+ node need the URI, so the value carries both.
21
+ 3. **Structured data travels as JSON *text*, never on a ``json`` port.** The
22
+ canvas stringifies a `json` port value with JavaScript's ``String(obj)``
23
+ rather than ``JSON.stringify``, so the receiving node gets the literal text
24
+ ``"[object Object]"`` and the data is gone. `detect_objects`,
25
+ `classify_image`, `top_labels` and `png_info` therefore emit JSON strings,
26
+ and `_as_list` parses them back.
27
 
28
  Everything returned must be JSON-serializable — that is the contract for
29
  `bind=` functions.
 
710
  max_tokens=_num(max_tokens, 512, lo=32, hi=4096, integer=True))
711
 
712
 
713
+ def _image_file(image, label="image"):
714
+ """Materialize any accepted image value as a temp file path.
715
+
716
+ A **path**, not bytes: handing `InferenceClient` raw bytes makes the router
717
+ reject the call with "No content type provided and no default one
718
+ configured", whereas from a path huggingface_hub infers the MIME type.
719
+ """
720
+ img = _load_image(image, label)
721
+ path = os.path.join(tempfile.gettempdir(), f"wf1111_in_{os.urandom(8).hex()}.jpg")
722
+ _rgb(img).save(path, format="JPEG", quality=94, optimize=True)
723
+ return path
724
+
725
+
726
+ def detect_objects(image, model_id, min_score, oauth_token: Optional[OAuthToken] = None):
727
+ """Object detection, returning the detections as a **JSON string**.
728
+
729
+ An `fn` node calling `InferenceClient` rather than a `model` node, because
730
+ the canvas destroys `json`-typed port values: it stringifies them with
731
+ JavaScript's `String(obj)` instead of `JSON.stringify`, so the downstream
732
+ node receives the literal text ``"[object Object]"`` and sees zero
733
+ detections. Text ports survive intact, so the detections travel as JSON
734
+ text and `_as_list` parses them back.
735
+ """
736
+ from huggingface_hub import InferenceClient
737
+
738
+ model = _text(model_id, "facebook/detr-resnet-50")
739
+ floor = _num(min_score, 0.0, lo=0.0, hi=1.0)
740
+ client = InferenceClient(model=model, token=_hf_token(oauth_token), provider="auto")
741
+ try:
742
+ results = client.object_detection(image=_image_file(image, "image to analyse"))
743
+ except Exception as e:
744
+ raise ValueError(f"{model} failed: {str(e)[:300]}") from e
745
+
746
+ found = []
747
+ for r in results:
748
+ box = getattr(r, "box", None) or {}
749
+ get = (lambda k: getattr(box, k, None)) if not isinstance(box, dict) else box.get
750
+ try:
751
+ coords = {k: int(get(k)) for k in ("xmin", "ymin", "xmax", "ymax")}
752
+ except (TypeError, ValueError):
753
+ continue
754
+ score = float(getattr(r, "score", 0.0) or 0.0)
755
+ if score < floor:
756
+ continue
757
+ found.append({"label": str(getattr(r, "label", "object")),
758
+ "score": round(score, 4), "box": coords})
759
+ found.sort(key=lambda d: d["score"], reverse=True)
760
+ return json.dumps(found)
761
+
762
+
763
+ def classify_image(image, model_id, oauth_token: Optional[OAuthToken] = None):
764
+ """Image classification, returning the labels as a **JSON string**
765
+ (same reason as `detect_objects`)."""
766
+ from huggingface_hub import InferenceClient
767
+
768
+ model = _text(model_id, "google/vit-base-patch16-224")
769
+ client = InferenceClient(model=model, token=_hf_token(oauth_token), provider="auto")
770
+ try:
771
+ results = client.image_classification(
772
+ image=_image_file(image, "image to classify"))
773
+ except Exception as e:
774
+ raise ValueError(f"{model} failed: {str(e)[:300]}") from e
775
+
776
+ return json.dumps([{"label": str(getattr(r, "label", "?")),
777
+ "score": round(float(getattr(r, "score", 0.0) or 0.0), 5)}
778
+ for r in results])
779
+
780
+
781
  def top_labels(labels, top_k, min_score):
782
  """Format an image-classification payload. Returns (text, json)."""
783
  items = _as_list(labels)
 
796
  rows = rows[:k]
797
 
798
  if not rows:
799
+ return "No labels above the score threshold.", "[]"
800
  width = max(len(r["label"]) for r in rows)
801
  lines = [
802
  f"{r['label']:<{width}} {r['score'] * 100:5.1f}% {'█' * max(1, int(r['score'] * 24))}"
803
  for r in rows
804
  ]
805
+ # JSON *text*, not a list: a `json` port would reach the canvas as
806
+ # "[object Object]" (see `detect_objects`).
807
+ return "\n".join(lines), json.dumps(rows, indent=2)
808
 
809
 
810
  # ─────────────────────────────────────────────────────────────────────────────
 
1159
  found = _boxes(detections, min_score)
1160
 
1161
  draw = ImageDraw.Draw(img, "RGBA")
1162
+ # Sized generously on purpose: a canvas node renders a 768px image at
1163
+ # roughly a third of its size, where a hairline box is invisible.
1164
+ stroke = max(3, int(min(img.size) * 0.008))
1165
+ font = _font(max(15, int(min(img.size) * 0.034)))
1166
 
1167
  for i, det in enumerate(found):
1168
  color = _PALETTE[i % len(_PALETTE)]
 
1284
  def png_info(image):
1285
  """A1111's 'PNG Info' tab: recover generation parameters from a file.
1286
 
1287
+ Returns (report, fields-as-JSON-text). The fields are serialized rather
1288
+ than returned as a dict because a `json` port arrives in the canvas as
1289
+ "[object Object]" (see `detect_objects`).
1290
  """
1291
  img = _load_image(image)
1292
  img.load() # force chunk parsing so text metadata is populated
 
1336
  + ("Other metadata:\n" + "\n".join(extra) if extra else
1337
  "This image carries no text metadata at all.")
1338
  )
1339
+ return head + body, json.dumps(fields, indent=2, default=str)
1340
 
1341
 
1342
  # What `app.py` binds onto the canvas. Keys must match the "fn" field of each
 
1352
  "txt2img": txt2img,
1353
  "chat_llm": chat_llm,
1354
  "interrogate": interrogate,
1355
+ "detect_objects": detect_objects,
1356
+ "classify_image": classify_image,
1357
  "top_labels": top_labels,
1358
  "postprocess": postprocess,
1359
  "prep_image": prep_image,
samples/with_parameters.png CHANGED

Git LFS Details

  • SHA256: 046f00c94a4f8b695101980b9691da6c2aa68dbce83e1712ee650a99fb1fa3f7
  • Pointer size: 131 Bytes
  • Size of remote file: 483 kB

Git LFS Details

  • SHA256: 188c2b6910f1482d0e65e35aa060df21b9f90b39412a4281edfc8abcd2621485
  • Pointer size: 131 Bytes
  • Size of remote file: 168 kB
test_nodes.py CHANGED
@@ -210,10 +210,14 @@ check("clean_prompt strips LLM chatter", t_clean)
210
 
211
 
212
  def t_labels():
213
- text, rows = N.top_labels(
214
  [{"label": "tiger", "score": 0.88}, {"label": "cat", "score": 0.10},
215
  {"label": "dog", "score": 0.001}], 2, 0.05)
 
 
216
  assert rows[0]["label"] == "tiger" and len(rows) == 2
 
 
217
  assert "tiger" in text and "%" in text
218
  empty, _ = N.top_labels([], 5, 0.5)
219
  assert "No labels" in empty
@@ -348,7 +352,8 @@ def t_png_roundtrip():
348
  stamped = N.postprocess(DATA_URI, 1, "Lanczos", 0, 1, 1, 1, 0, 0, 0, "", info)
349
  assert stamped["url"].startswith("data:image/png"), "metadata must force PNG"
350
 
351
- report, fields = N.png_info(stamped)
 
352
  assert "Generation parameters" in report
353
  assert fields["prompt"] == "a fox in snow", fields.get("prompt")
354
  assert fields["negative_prompt"] == "ugly, blurry", fields.get("negative_prompt")
@@ -362,7 +367,8 @@ check("generation params survive the round trip", t_png_roundtrip)
362
 
363
 
364
  def t_png_bare():
365
- report, fields = N.png_info(DATA_URI)
 
366
  assert "No generation parameters" in report
367
  assert fields["width"] == 256 and fields["height"] == 192
368
 
 
210
 
211
 
212
  def t_labels():
213
+ text, rows_json = N.top_labels(
214
  [{"label": "tiger", "score": 0.88}, {"label": "cat", "score": 0.10},
215
  {"label": "dog", "score": 0.001}], 2, 0.05)
216
+ # second output is JSON *text* — a json port arrives as "[object Object]"
217
+ rows = __import__("json").loads(rows_json)
218
  assert rows[0]["label"] == "tiger" and len(rows) == 2
219
+ # and it must survive a round trip through a text port
220
+ assert N.top_labels(rows_json, 2, 0.05)[0].startswith("tiger")
221
  assert "tiger" in text and "%" in text
222
  empty, _ = N.top_labels([], 5, 0.5)
223
  assert "No labels" in empty
 
352
  stamped = N.postprocess(DATA_URI, 1, "Lanczos", 0, 1, 1, 1, 0, 0, 0, "", info)
353
  assert stamped["url"].startswith("data:image/png"), "metadata must force PNG"
354
 
355
+ report, fields_json = N.png_info(stamped)
356
+ fields = __import__("json").loads(fields_json)
357
  assert "Generation parameters" in report
358
  assert fields["prompt"] == "a fox in snow", fields.get("prompt")
359
  assert fields["negative_prompt"] == "ugly, blurry", fields.get("negative_prompt")
 
367
 
368
 
369
  def t_png_bare():
370
+ report, fields_json = N.png_info(DATA_URI)
371
+ fields = __import__("json").loads(fields_json)
372
  assert "No generation parameters" in report
373
  assert fields["width"] == 256 and fields["height"] == 192
374
 
workflow.json CHANGED
@@ -1260,32 +1260,37 @@
1260
  {
1261
  "id": "op_classify",
1262
  "role": "operator",
1263
- "kind": "model",
1264
- "model_id": "google/vit-base-patch16-224",
1265
- "pipeline_tag": "image-classification",
1266
- "endpoint": "image_classification",
1267
  "label": "③ Classify · ViT",
1268
  "inputs": [
1269
  {
1270
- "id": "image",
1271
  "label": "image",
1272
  "type": "image",
1273
  "required": true
 
 
 
 
 
1274
  }
1275
  ],
1276
  "outputs": [
1277
  {
1278
  "id": "out_0",
1279
- "label": "Labels",
1280
- "type": "json",
1281
  "output_index": 0
1282
  }
1283
  ],
1284
- "data": {},
 
 
1285
  "x": 514.3,
1286
  "y": 2007.2,
1287
  "width": 290,
1288
- "height": 94
1289
  },
1290
  {
1291
  "id": "op_labels",
@@ -1297,7 +1302,7 @@
1297
  {
1298
  "id": "in_labels",
1299
  "label": "labels",
1300
- "type": "json",
1301
  "required": true
1302
  },
1303
  {
@@ -1321,7 +1326,7 @@
1321
  {
1322
  "id": "out_1",
1323
  "label": "rows",
1324
- "type": "json",
1325
  "output_index": 1
1326
  }
1327
  ],
@@ -1337,32 +1342,43 @@
1337
  {
1338
  "id": "op_detect",
1339
  "role": "operator",
1340
- "kind": "model",
1341
- "model_id": "facebook/detr-resnet-50",
1342
- "pipeline_tag": "object-detection",
1343
- "endpoint": "object_detection",
1344
  "label": "① Detect · DETR",
1345
  "inputs": [
1346
  {
1347
- "id": "image",
1348
  "label": "image",
1349
  "type": "image",
1350
  "required": true
 
 
 
 
 
 
 
 
 
 
1351
  }
1352
  ],
1353
  "outputs": [
1354
  {
1355
  "id": "out_0",
1356
- "label": "Detections",
1357
- "type": "json",
1358
  "output_index": 0
1359
  }
1360
  ],
1361
- "data": {},
 
 
 
1362
  "x": 302.4,
1363
  "y": 2154.9,
1364
  "width": 290,
1365
- "height": 94
1366
  },
1367
  {
1368
  "id": "op_draw",
@@ -1380,7 +1396,7 @@
1380
  {
1381
  "id": "in_detections",
1382
  "label": "detections",
1383
- "type": "json",
1384
  "required": true
1385
  },
1386
  {
@@ -1433,7 +1449,7 @@
1433
  {
1434
  "id": "in_detections",
1435
  "label": "detections",
1436
- "type": "json",
1437
  "required": true
1438
  },
1439
  {
@@ -2102,7 +2118,7 @@
2102
  {
2103
  "id": "out_1",
2104
  "label": "fields",
2105
- "type": "json",
2106
  "output_index": 1
2107
  }
2108
  ],
@@ -2543,19 +2559,19 @@
2543
  "id": "sub_png_fields",
2544
  "role": "subject",
2545
  "label": "🧮 Parsed fields",
2546
- "asset_type": "json",
2547
  "inputs": [
2548
  {
2549
  "id": "in",
2550
  "label": "🧮 Parsed fields",
2551
- "type": "json"
2552
  }
2553
  ],
2554
  "outputs": [
2555
  {
2556
  "id": "out",
2557
  "label": "🧮 Parsed fields",
2558
- "type": "json"
2559
  }
2560
  ],
2561
  "data": {},
@@ -2875,7 +2891,7 @@
2875
  "from_node_id": "ref_interrogate_image",
2876
  "from_port_id": "out",
2877
  "to_node_id": "op_classify",
2878
- "to_port_id": "image",
2879
  "type": "image"
2880
  },
2881
  {
@@ -2884,7 +2900,7 @@
2884
  "from_port_id": "out_0",
2885
  "to_node_id": "op_labels",
2886
  "to_port_id": "in_labels",
2887
- "type": "json"
2888
  },
2889
  {
2890
  "id": "e41",
@@ -2899,7 +2915,7 @@
2899
  "from_node_id": "ref_detect_image",
2900
  "from_port_id": "out",
2901
  "to_node_id": "op_detect",
2902
- "to_port_id": "image",
2903
  "type": "image"
2904
  },
2905
  {
@@ -2916,7 +2932,7 @@
2916
  "from_port_id": "out_0",
2917
  "to_node_id": "op_draw",
2918
  "to_port_id": "in_detections",
2919
- "type": "json"
2920
  },
2921
  {
2922
  "id": "e45",
@@ -2932,7 +2948,7 @@
2932
  "from_port_id": "out_0",
2933
  "to_node_id": "op_mask",
2934
  "to_port_id": "in_detections",
2935
- "type": "json"
2936
  },
2937
  {
2938
  "id": "e47",
@@ -3148,7 +3164,7 @@
3148
  "from_port_id": "out_1",
3149
  "to_node_id": "sub_png_fields",
3150
  "to_port_id": "in",
3151
- "type": "json"
3152
  }
3153
  ]
3154
  }
 
1260
  {
1261
  "id": "op_classify",
1262
  "role": "operator",
1263
+ "kind": "fn",
1264
+ "fn": "classify_image",
 
 
1265
  "label": "③ Classify · ViT",
1266
  "inputs": [
1267
  {
1268
+ "id": "in_image",
1269
  "label": "image",
1270
  "type": "image",
1271
  "required": true
1272
+ },
1273
+ {
1274
+ "id": "in_model_id",
1275
+ "label": "model_id",
1276
+ "type": "text"
1277
  }
1278
  ],
1279
  "outputs": [
1280
  {
1281
  "id": "out_0",
1282
+ "label": "labels",
1283
+ "type": "text",
1284
  "output_index": 0
1285
  }
1286
  ],
1287
+ "data": {
1288
+ "in_model_id": "google/vit-base-patch16-224"
1289
+ },
1290
  "x": 514.3,
1291
  "y": 2007.2,
1292
  "width": 290,
1293
+ "height": 124
1294
  },
1295
  {
1296
  "id": "op_labels",
 
1302
  {
1303
  "id": "in_labels",
1304
  "label": "labels",
1305
+ "type": "text",
1306
  "required": true
1307
  },
1308
  {
 
1326
  {
1327
  "id": "out_1",
1328
  "label": "rows",
1329
+ "type": "text",
1330
  "output_index": 1
1331
  }
1332
  ],
 
1342
  {
1343
  "id": "op_detect",
1344
  "role": "operator",
1345
+ "kind": "fn",
1346
+ "fn": "detect_objects",
 
 
1347
  "label": "① Detect · DETR",
1348
  "inputs": [
1349
  {
1350
+ "id": "in_image",
1351
  "label": "image",
1352
  "type": "image",
1353
  "required": true
1354
+ },
1355
+ {
1356
+ "id": "in_model_id",
1357
+ "label": "model_id",
1358
+ "type": "text"
1359
+ },
1360
+ {
1361
+ "id": "in_min_score",
1362
+ "label": "min_score",
1363
+ "type": "number"
1364
  }
1365
  ],
1366
  "outputs": [
1367
  {
1368
  "id": "out_0",
1369
+ "label": "detections",
1370
+ "type": "text",
1371
  "output_index": 0
1372
  }
1373
  ],
1374
+ "data": {
1375
+ "in_model_id": "facebook/detr-resnet-50",
1376
+ "in_min_score": 0.0
1377
+ },
1378
  "x": 302.4,
1379
  "y": 2154.9,
1380
  "width": 290,
1381
+ "height": 154
1382
  },
1383
  {
1384
  "id": "op_draw",
 
1396
  {
1397
  "id": "in_detections",
1398
  "label": "detections",
1399
+ "type": "text",
1400
  "required": true
1401
  },
1402
  {
 
1449
  {
1450
  "id": "in_detections",
1451
  "label": "detections",
1452
+ "type": "text",
1453
  "required": true
1454
  },
1455
  {
 
2118
  {
2119
  "id": "out_1",
2120
  "label": "fields",
2121
+ "type": "text",
2122
  "output_index": 1
2123
  }
2124
  ],
 
2559
  "id": "sub_png_fields",
2560
  "role": "subject",
2561
  "label": "🧮 Parsed fields",
2562
+ "asset_type": "text",
2563
  "inputs": [
2564
  {
2565
  "id": "in",
2566
  "label": "🧮 Parsed fields",
2567
+ "type": "text"
2568
  }
2569
  ],
2570
  "outputs": [
2571
  {
2572
  "id": "out",
2573
  "label": "🧮 Parsed fields",
2574
+ "type": "text"
2575
  }
2576
  ],
2577
  "data": {},
 
2891
  "from_node_id": "ref_interrogate_image",
2892
  "from_port_id": "out",
2893
  "to_node_id": "op_classify",
2894
+ "to_port_id": "in_image",
2895
  "type": "image"
2896
  },
2897
  {
 
2900
  "from_port_id": "out_0",
2901
  "to_node_id": "op_labels",
2902
  "to_port_id": "in_labels",
2903
+ "type": "text"
2904
  },
2905
  {
2906
  "id": "e41",
 
2915
  "from_node_id": "ref_detect_image",
2916
  "from_port_id": "out",
2917
  "to_node_id": "op_detect",
2918
+ "to_port_id": "in_image",
2919
  "type": "image"
2920
  },
2921
  {
 
2932
  "from_port_id": "out_0",
2933
  "to_node_id": "op_draw",
2934
  "to_port_id": "in_detections",
2935
+ "type": "text"
2936
  },
2937
  {
2938
  "id": "e45",
 
2948
  "from_port_id": "out_0",
2949
  "to_node_id": "op_mask",
2950
  "to_port_id": "in_detections",
2951
+ "type": "text"
2952
  },
2953
  {
2954
  "id": "e47",
 
3164
  "from_port_id": "out_1",
3165
  "to_node_id": "sub_png_fields",
3166
  "to_port_id": "in",
3167
+ "type": "text"
3168
  }
3169
  ]
3170
  }