Spaces:
Sleeping
Sleeping
| # Commercial-FCI-Agent:基于闭源商业多模态图像模型的前景条件图像修复系统 | |
| **论文级技术方案与开发实现文档 v0.1** | |
| **日期:2026-06-02** | |
| --- | |
| ## 摘要 | |
| **Commercial-FCI-Agent** 是一个 training-free、model-agnostic、agentic foreground-conditioned inpainting 框架。给定前景 RGBA 图像和用户 prompt,系统首先构造前景核心保护区、边界融合环、背景编辑区和接触区域;然后由 VLM/LLM 分析前景语义、材质、姿态、光照、尺度和接触点,生成结构化 scene plan;接着调用闭源商业图像模型生成多个候选背景;之后用 alpha-aware foreground restoration 把原始前景强制复合回候选结果;最后由 VLM critic 和可计算指标共同评分,选择最佳候选,必要时根据失败标签触发 prompt repair、mask repair 或局部 boundary/contact-shadow refinement。 | |
| 核心原则: | |
| > **商业模型负责生成候选,系统负责保证约束。** | |
| 该方案适合把 GPT-image-2、Nano Banana / Gemini Image、Seedream、Firefly 等闭源商业图像模型作为黑盒生成器,用系统化约束、评估和修复机制提升 foreground preservation、spatial rationality、boundary quality 和产品级可控性。 | |
| --- | |
| ## 目录 | |
| 1. [方法定位](#1-方法定位) | |
| 2. [论文贡献点设计](#2-论文贡献点设计) | |
| 3. [系统总览](#3-系统总览) | |
| 4. [形式化定义](#4-形式化定义) | |
| 5. [模块一:Asset Normalizer](#5-模块一asset-normalizer) | |
| 6. [模块二:Foreground Analyzer Agent](#6-模块二foreground-analyzer-agent) | |
| 7. [模块三:Scene Planner Agent](#7-模块三scene-planner-agent) | |
| 8. [模块四:Mask & Layout Builder](#8-模块四mask--layout-builder) | |
| 9. [模块五:Prompt Ensemble Agent](#9-模块五prompt-ensemble-agent) | |
| 10. [模块六:Commercial Generator Router](#10-模块六commercial-generator-router) | |
| 11. [模块七:Alpha-aware Foreground Restorer](#11-模块七alpha-aware-foreground-restorer) | |
| 12. [模块八:Boundary / Contact / Shadow Refiner](#12-模块八boundary--contact--shadow-refiner) | |
| 13. [模块九:Critic & Reranker Agent](#13-模块九critic--reranker-agent) | |
| 14. [模块十:Repair Planner Agent](#14-模块十repair-planner-agent) | |
| 15. [完整算法](#15-完整算法) | |
| 16. [`report.json` 设计](#16-reportjson-设计) | |
| 17. [实验设计](#17-实验设计) | |
| 18. [工程实现建议](#18-工程实现建议) | |
| 19. [失败模式与解决策略](#19-失败模式与解决策略) | |
| 20. [推荐论文结构](#20-推荐论文结构) | |
| 21. [最小可行产品版本](#21-最小可行产品版本) | |
| 22. [开发里程碑](#22-开发里程碑) | |
| 23. [实现原则](#23-实现原则) | |
| 24. [一句话版方法](#24-一句话版方法) | |
| 25. [参考资料](#25-参考资料) | |
| --- | |
| ## 1. 方法定位 | |
| ### 1.1 任务定义 | |
| 给定一个前景主体 $F$,通常包含 RGB 与 alpha/mask,以及一个目标背景文本 prompt $p$,系统需要生成一张最终图像 $I^*$,满足: | |
| 1. **前景主体保持不变**:形状、纹理、身份、logo、文字、姿态尽可能像素级保留。 | |
| 2. **背景符合 prompt**:语义、风格、场景、材质、时间、天气、摄影风格与用户需求一致。 | |
| 3. **前景-背景关系合理**:尺度、透视、接触点、地面、阴影、反射、遮挡关系合理。 | |
| 4. **输出可开发、可评测、可复现**:保留中间产物、prompt、mask、模型参数、critic 分数和失败标签。 | |
| 5. **不训练主生成模型**:训练成本转移到 pipeline 设计、评估器、数据与 reranking 上。 | |
| ### 1.2 与普通 inpainting 的区别 | |
| 普通 text-guided inpainting 主要关心 mask 区域补全;Commercial-FCI-Agent 关心的是: | |
| > 给定一个必须保真的前景主体,如何生成与其语义、形状、尺度、光照、透视和接触关系一致的新背景。 | |
| 因此,本任务的核心指标不是 FID 或单纯美学分数,而是 foreground preservation、spatial rationality、boundary quality、lighting harmony、prompt alignment 和 human preference。 | |
| ### 1.3 与直接调用商业图像模型的区别 | |
| 直接调用 GPT-image-2 / Nano Banana 的问题是: | |
| - 模型可能改写前景。 | |
| - 模型可能重画 logo、文字、脸、手或商品细节。 | |
| - mask 或自然语言约束不一定严格。 | |
| - 结果可能好看但主体漂浮、尺度错误或阴影不自然。 | |
| - 结果缺乏可复现记录和可诊断中间产物。 | |
| Commercial-FCI-Agent 的关键区别是: | |
| 1. **先分析前景,再规划场景。** | |
| 2. **生成多个候选,而非一次生成。** | |
| 3. **最终强制复合原始前景,而非信任模型输出。** | |
| 4. **用 critic 自动评价和诊断失败。** | |
| 5. **根据失败标签进行迭代修复。** | |
| --- | |
| ## 2. 论文贡献点设计 | |
| ### Contribution 1:Black-box Commercial FCI Protocol | |
| 提出一种面向闭源商业图像模型的 foreground-conditioned inpainting 协议。该协议不依赖模型内部 latent、attention 或训练接口,只要求模型支持图像输入、文本 prompt、可选 mask 或多参考图编辑。 | |
| ### Contribution 2:Alpha-aware Foreground Restoration | |
| 提出 **foreground core / boundary ring / editable background / contact region** 四分区机制,并在生成后通过原始前景强制复合,解决闭源模型易改写前景的问题。 | |
| ### Contribution 3:Critic-guided Agentic Repair | |
| 提出面向 FCI 的结构化 VLM critic,自动诊断: | |
| - `foreground_changed` | |
| - `logo_changed` | |
| - `printed_text_changed` | |
| - `floating` | |
| - `wrong_scale` | |
| - `wrong_perspective` | |
| - `missing_shadow` | |
| - `bad_reflection` | |
| - `halo` | |
| - `prompt_mismatch` | |
| 并把错误映射到 prompt、mask、layout、模型选择或局部修复策略。 | |
| ### Contribution 4:Commercial-FCI Benchmark Protocol | |
| 提出一套可复现实验协议,对比 raw commercial model、commercial + foreground restoration、commercial + rerank、commercial + agent repair,以及开源 inpainting / foreground-conditioned 方法。 | |
| --- | |
| ## 3. 系统总览 | |
| ### 3.1 总体架构 | |
| ```text | |
| Input: | |
| foreground.png / foreground_rgba.png | |
| foreground_alpha.png optional | |
| user_prompt | |
| optional style refs / layout refs / background refs | |
| │ | |
| ▼ | |
| [1] Asset Normalizer | |
| │ | |
| ▼ | |
| [2] Foreground Analyzer Agent | |
| │ | |
| ▼ | |
| [3] Scene Planner Agent | |
| │ | |
| ▼ | |
| [4] Mask & Layout Builder | |
| │ | |
| ▼ | |
| [5] Prompt Ensemble Agent | |
| │ | |
| ▼ | |
| [6] Commercial Generator Router | |
| ├── GPT-image-2 Adapter | |
| ├── Nano Banana Adapter | |
| └── Other commercial model adapter | |
| │ | |
| ▼ | |
| [7] Alpha-aware Foreground Restorer | |
| │ | |
| ▼ | |
| [8] Boundary / Contact / Shadow Refiner | |
| │ | |
| ▼ | |
| [9] Critic & Reranker Agent | |
| │ | |
| ▼ | |
| [10] Repair Planner Agent | |
| │ | |
| └── repeat generation/refinement if needed | |
| ▼ | |
| Output: | |
| final_composite.png | |
| background.png | |
| foreground.png | |
| alpha.png | |
| contact_shadow.png optional | |
| report.json | |
| ``` | |
| ### 3.2 Agent 列表 | |
| | Agent | 输入 | 输出 | 作用 | | |
| |---|---|---|---| | |
| | Asset Normalizer | 原图、mask、RGBA | 标准化 RGBA、画布、分辨率 | 统一尺寸、色彩空间、alpha、边界 | | |
| | Foreground Analyzer | 前景图 + alpha | foreground JSON | 识别主体类别、姿态、材质、光照、接触点 | | |
| | Scene Planner | 用户 prompt + foreground JSON | scene plan JSON | 把用户 prompt 展开成可生成的背景方案 | | |
| | Mask & Layout Builder | alpha、scene plan | masks、layout guides | 构造保护区、编辑区、边界环、接触区 | | |
| | Prompt Ensemble Agent | scene plan、失败历史 | K 个 prompts | 多样化候选生成 | | |
| | Generator Router | prompts、masks、refs | raw candidates | 调用 GPT-image-2 / Nano Banana | | |
| | Foreground Restorer | raw candidate + 原前景 | restored candidate | 强制复合原始前景 | | |
| | Boundary Refiner | restored candidate + contact mask | refined candidate | 修边、接触阴影、反射 | | |
| | Critic & Reranker | candidates + prompt + foreground | scores、failure tags | 评分、排序、诊断失败 | | |
| | Repair Planner | failure tags + history | repair action | 自动决定重试、换模型、改 mask、改 prompt | | |
| --- | |
| ## 4. 形式化定义 | |
| ### 4.1 输入 | |
| \[ | |
| \mathcal{X} = \{I_f, A_f, p, b, R, L, \mathcal{B}\} | |
| \] | |
| 其中: | |
| - $I_f \in [0,1]^{H \times W \times 3}$:前景 RGB。 | |
| - $A_f \in [0,1]^{H \times W}$:前景 alpha matte。 | |
| - $p$:用户目标背景文本 prompt。 | |
| - $b$:前景在目标画布中的位置、尺度、旋转。 | |
| - $R = \{r_1, ..., r_n\}$:可选参考图,包括风格图、背景图、品牌图、材质图。 | |
| - $L$:可选 layout guide,例如地平线、地面区域、桌面、透视线、depth map、sketch。 | |
| - $\mathcal{B}$:预算约束,包括最大 API 次数、最大延迟、模型优先级、质量等级。 | |
| ### 4.2 输出 | |
| \[ | |
| \mathcal{Y} = \{I^*, B^*, F^*, A^*, S^*, Q^*, \Pi^*\} | |
| \] | |
| 其中: | |
| - $I^*$:最终合成图。 | |
| - $B^*$:生成背景层。 | |
| - $F^* = I_f$:原始前景层。 | |
| - $A^* = A_f$:前景 alpha。 | |
| - $S^*$:可选接触阴影或反射层。 | |
| - $Q^*$:最终质量评分。 | |
| - $\Pi^*$:最终 prompt、模型、mask、参数、critic 报告。 | |
| ### 4.3 候选生成 | |
| \[ | |
| Y_i = G_m(C, M, P_i, R, L) | |
| \] | |
| 其中: | |
| - $G_m$:闭源商业图像模型。 | |
| - $C$:带前景的条件画布。 | |
| - $M$:编辑 mask 或 mask visualization。 | |
| - $P_i$:第 $i$ 个 prompt variant。 | |
| ### 4.4 最终复合 | |
| \[ | |
| I_i = A_{soft} \odot I_f + (1 - A_{soft}) \odot Y_i | |
| \] | |
| 对于核心保护区: | |
| \[ | |
| I_i[x] = I_f[x], \quad \forall x \in A_{core} | |
| \] | |
| 这一步是 Commercial-FCI-Agent 的核心,不应省略。 | |
| --- | |
| ## 5. 模块一:Asset Normalizer | |
| ### 5.1 输入格式 | |
| 系统应支持: | |
| ```text | |
| case A: foreground_rgba.png | |
| case B: rgb.png + mask.png | |
| case C: product image with background | |
| case D: original image + user-selected foreground bbox | |
| case E: multi-foreground RGBA list | |
| ``` | |
| 推荐内部统一为: | |
| ```python | |
| { | |
| "foreground_rgb": "float32[H, W, 3]", # linear or sRGB, documented | |
| "alpha": "float32[H, W]", # 0-1 | |
| "canvas_size": [Hc, Wc], | |
| "bbox": [x, y, w, h], | |
| "category_hint": "optional[str]", | |
| "metadata": { | |
| "source": "...", | |
| "has_original_shadow": True, | |
| "has_logo_or_text": True | |
| } | |
| } | |
| ``` | |
| ### 5.2 alpha 获取与清理 | |
| 如果已有 RGBA: | |
| ```text | |
| A_f = alpha channel | |
| I_f = RGB channel | |
| ``` | |
| 如果没有 alpha: | |
| ```text | |
| A_f = Matting(SAM/RMBG/BiRefNet/MODNet) | |
| ``` | |
| 后处理: | |
| ```text | |
| A_bin = threshold(A_f, tau=0.5) | |
| A_core = erode(A_bin, r_core) | |
| A_dilate = dilate(A_bin, r_dilate) | |
| A_ring = A_dilate - A_core | |
| A_soft = gaussian_blur(A_f, sigma) | |
| M_bg = 1 - A_dilate | |
| ``` | |
| 推荐默认参数: | |
| | 场景 | `r_core` | `r_dilate` | `sigma` | 说明 | | |
| |---|---:|---:|---:|---| | |
| | 商品硬边 | 3 px | 8 px | 1.0 | 鞋、瓶子、电子产品 | | |
| | 人像 | 2 px | 10 px | 1.5 | 保脸、手、衣物纹理 | | |
| | 毛发/动物 | 1 px | 16 px | 2.5 | 允许边缘更软 | | |
| | 玻璃/透明物 | 1 px | 20 px | 3.0 | 避免硬切 | | |
| | logo/text 商品 | 4 px | 8 px | 0.8 | 更强前景保护 | | |
| ### 5.3 接触区域估计 | |
| 接触区域 $A_{contact}$ 很重要,因为前景漂浮感大多来自这里。 | |
| 基础 heuristic: | |
| ```python | |
| bottom_band = lower_25_percent_of_foreground_bbox | |
| contact_seed = alpha_pixels_in_bottom_band_with_high_local_density | |
| A_contact = dilate(contact_seed, radius=40_to_120_px) - A_core | |
| ``` | |
| 更强版本: | |
| 1. 用 depth / normal estimator 估计地面。 | |
| 2. 用 VLM 判断“主体应该接触哪里”。 | |
| 3. 对人像检测脚部,对商品检测底面,对车辆检测轮胎,对家具检测支撑脚。 | |
| 4. 构造地面投影椭圆区域作为 shadow candidate mask。 | |
| 输出: | |
| ```json | |
| { | |
| "contact_type": "ground | table | wall | hand-held | floating_allowed", | |
| "contact_points": [[x1, y1], [x2, y2]], | |
| "contact_region_mask": "A_contact.png", | |
| "expected_shadow": "soft oval shadow below the sole" | |
| } | |
| ``` | |
| --- | |
| ## 6. 模块二:Foreground Analyzer Agent | |
| ### 6.1 目标 | |
| Foreground Analyzer 用 VLM/多模态模型把前景变成结构化约束。它不是为了写 caption,而是为了给后续 generation、critique 和 repair 提供可执行信息。 | |
| 输出 schema: | |
| ```json | |
| { | |
| "subject": { | |
| "category": "product/shoe", | |
| "name": "white running shoe", | |
| "count": 1, | |
| "identity_sensitive": true, | |
| "has_logo_or_text": true, | |
| "must_preserve": [ | |
| "silhouette", | |
| "logo", | |
| "printed text", | |
| "texture", | |
| "laces", | |
| "sole pattern", | |
| "pose" | |
| ] | |
| }, | |
| "geometry": { | |
| "viewpoint": "side view, slightly top-down", | |
| "pose": "static product pose", | |
| "bbox": [256, 340, 512, 420], | |
| "approx_real_size": "shoe-sized object", | |
| "expected_support": "floor/table/display surface" | |
| }, | |
| "appearance": { | |
| "materials": ["mesh fabric", "rubber sole"], | |
| "dominant_colors": ["white", "gray", "blue"], | |
| "surface_finish": "matte with slight highlights" | |
| }, | |
| "lighting": { | |
| "direction": "upper-left", | |
| "softness": "softbox-like", | |
| "color_temperature": "neutral-cool", | |
| "existing_shadow": "none/minimal" | |
| }, | |
| "risk_flags": [ | |
| "logo_text_sensitive", | |
| "thin_boundary", | |
| "contact_shadow_needed" | |
| ] | |
| } | |
| ``` | |
| ### 6.2 Analyzer prompt | |
| ```text | |
| You are a foreground analysis module for foreground-conditioned inpainting. | |
| Analyze only the provided foreground subject and its alpha/mask. | |
| Do not invent background details. | |
| Return strict JSON with: | |
| - subject category and concise description | |
| - visible parts and count | |
| - materials and dominant colors | |
| - viewpoint, pose, scale prior, expected support surface | |
| - lighting direction, softness, color temperature | |
| - contact region and expected shadow | |
| - identity-sensitive regions such as face, hands, logo, printed text | |
| - failure risks for background generation | |
| The output will be used to preserve the subject exactly during image editing. | |
| ``` | |
| ### 6.3 开发建议 | |
| 不要只让 Analyzer 输出自然语言。必须输出结构化 JSON,因为后续 Prompt Ensemble、Mask Builder 和 Critic 都要消费这些字段。 | |
| --- | |
| ## 7. 模块三:Scene Planner Agent | |
| ### 7.1 目标 | |
| Scene Planner 把用户 prompt $p$ 转换为可执行的 scene plan,补足摄影、空间、光照、接触、风格和负面约束。 | |
| 输入: | |
| ```json | |
| { | |
| "user_prompt": "把这只鞋放在雨夜东京街头的橱窗前", | |
| "foreground_analysis": {...}, | |
| "optional_refs": [] | |
| } | |
| ``` | |
| 输出: | |
| ```json | |
| { | |
| "scene": { | |
| "semantic_scene": "rainy Tokyo street at night", | |
| "support_surface": "wet asphalt sidewalk", | |
| "background_elements": [ | |
| "blurred neon signs", | |
| "storefront window", | |
| "subtle rain reflections" | |
| ], | |
| "forbidden_elements": [ | |
| "extra shoes", | |
| "changed logo", | |
| "unreadable fake brand text on subject" | |
| ] | |
| }, | |
| "composition": { | |
| "camera": "low product photography angle, 50mm lens", | |
| "horizon": "slightly above the shoe", | |
| "depth_of_field": "shallow background blur", | |
| "subject_position": "center-lower third", | |
| "scale_relation": "shoe rests naturally on sidewalk" | |
| }, | |
| "lighting": { | |
| "key_light": "cool neon from upper-left", | |
| "fill_light": "soft ambient city light", | |
| "shadow": "soft contact shadow under sole", | |
| "reflection": "faint reflection on wet pavement" | |
| }, | |
| "style": { | |
| "photorealism": "high", | |
| "commercial_quality": "premium ad photography", | |
| "color_palette": "cool blue, magenta neon, wet black pavement" | |
| }, | |
| "negative_constraints": [ | |
| "do not alter the foreground shoe", | |
| "do not duplicate the shoe", | |
| "do not change logo or printed text", | |
| "no floating object", | |
| "no unrealistic scale mismatch", | |
| "no hard cutout edge" | |
| ] | |
| } | |
| ``` | |
| ### 7.2 Planner 设计原则 | |
| Scene Planner 的输出要有两个版本: | |
| 1. **Generation plan**:给生成器用,强调好看、真实、符合 prompt。 | |
| 2. **Evaluation plan**:给 critic 用,强调该检查哪些条件。 | |
| 例如: | |
| ```json | |
| { | |
| "must_satisfy": [ | |
| "foreground unchanged", | |
| "wet pavement support surface", | |
| "contact shadow below sole", | |
| "neon night city background", | |
| "no second shoe" | |
| ], | |
| "nice_to_have": [ | |
| "subtle reflection", | |
| "cinematic bokeh", | |
| "premium advertisement look" | |
| ] | |
| } | |
| ``` | |
| --- | |
| ## 8. 模块四:Mask & Layout Builder | |
| ### 8.1 四分区 mask | |
| Commercial-FCI-Agent 不使用单一 mask,而使用四个区域: | |
| ```text | |
| A_core : 前景核心保护区,最终 100% 使用原始前景 | |
| A_ring : 前景边界融合区,用于软融合、局部修边 | |
| M_bg : 背景生成区 | |
| A_contact : 接触阴影/反射修复区 | |
| ``` | |
| 逻辑关系: | |
| ```text | |
| A_core ⊂ A_f | |
| A_ring = dilate(A_f) - erode(A_f) | |
| M_bg = 1 - dilate(A_f) | |
| A_contact near lower/support region, outside A_core | |
| ``` | |
| ### 8.2 GPT-image-2 mask 构造 | |
| 对 GPT-image-2 路线,使用显式 mask edit。推荐 mask: | |
| ```text | |
| editable = M_bg + optional outer part of A_ring + A_contact | |
| protected = A_core | |
| ``` | |
| 注意:不同 API 对 mask alpha 的语义可能不同,工程上建议写一个 `MaskAdapter`,把内部 mask schema 转换为目标 API 所需格式,不要在业务代码里硬编码。 | |
| ### 8.3 Nano Banana mask visualization | |
| Nano Banana / Gemini 更适合用多参考图和语义编辑方式。可以把 mask 转成可视化 guide: | |
| ```text | |
| Image 1: foreground pasted on neutral canvas | |
| Image 2: mask visualization | |
| white = protected foreground | |
| black = editable background | |
| red outline = foreground boundary | |
| blue plane = expected support surface | |
| green dots = contact points | |
| Image 3: optional layout sketch | |
| Image 4: optional style/background reference | |
| ``` | |
| --- | |
| ## 9. 模块五:Prompt Ensemble Agent | |
| ### 9.1 为什么要 ensemble | |
| 闭源模型不可控,单次生成不稳定。Prompt Ensemble 通过多种 prompt variant 采样候选,再由 critic 选择最优。 | |
| 默认: | |
| ```text | |
| K = 4 快速预览 | |
| K = 8 标准产品模式 | |
| K = 16 论文实验模式 | |
| K = 32 离线数据合成模式 | |
| ``` | |
| ### 9.2 prompt 类型 | |
| | Variant | 目的 | | |
| |---|---| | |
| | `conservative` | 最大化前景保持,背景简单但可靠 | | |
| | `spatial` | 强调接触、尺度、透视、地面 | | |
| | `lighting` | 强调色温、光照方向、阴影 | | |
| | `aesthetic` | 强调广告级画质、构图、景深 | | |
| | `literal` | 严格遵循用户 prompt | | |
| | `reference-heavy` | 强调参考图/品牌风格 | | |
| | `minimal` | 减少额外物体,降低 hallucination | | |
| | `repair-specific` | 针对上一轮失败标签定制 | | |
| ### 9.3 GPT-image-2 prompt 模板 | |
| ```text | |
| Edit only the editable/masked background region. Keep the provided foreground subject exactly unchanged. | |
| Foreground subject: | |
| {foreground_description} | |
| Target scene: | |
| {scene.semantic_scene} | |
| Composition: | |
| {composition.camera} | |
| The subject should be placed naturally on {scene.support_surface}. | |
| Match the subject's current viewpoint, scale, and perspective. | |
| The subject must not float. | |
| Lighting: | |
| Match the foreground lighting: {lighting.direction}, {lighting.softness}, {lighting.color_temperature}. | |
| Add realistic contact shadow around {contact_region_description}. | |
| If physically plausible, add a subtle reflection or ambient occlusion. | |
| Strict preservation constraints: | |
| - Do not redraw, repaint, extend, shrink, duplicate, stylize, or deform the foreground subject. | |
| - Preserve silhouette, identity, texture, logo, printed text, pose, and all visible details. | |
| - Do not add extra copies or extra object parts. | |
| Negative constraints: | |
| {negative_constraints} | |
| Output: | |
| A photorealistic, coherent image with natural foreground-background integration. | |
| ``` | |
| ### 9.4 Nano Banana prompt 模板 | |
| ```text | |
| You are given reference images. | |
| Image 1 is the composition canvas with the foreground subject already placed. | |
| Image 2 is the mask guide: | |
| - white area means protected foreground, which must remain unchanged | |
| - black area means editable background | |
| - red outline marks the foreground boundary | |
| - blue guide marks the support surface or horizon | |
| - green dots mark expected contact points | |
| Task: | |
| Generate a new realistic background around the protected foreground subject. | |
| Strict requirements: | |
| - Preserve the foreground subject exactly: shape, identity, pose, texture, logo, printed text, and boundary. | |
| - Do not redraw, repaint, duplicate, expand, shrink, replace, stylize, or change the subject. | |
| - Only create the surrounding environment, support surface, lighting, contact shadow, and background depth. | |
| - The subject must be physically grounded and must not float. | |
| - Match perspective, scale, color temperature, and lighting direction. | |
| Target scene: | |
| {scene.semantic_scene} | |
| Camera and composition: | |
| {composition.camera} | |
| Lighting and physical effects: | |
| {lighting} | |
| Avoid: | |
| {negative_constraints} | |
| ``` | |
| ### 9.5 prompt repair 模板 | |
| ```text | |
| Previous generation failed because: | |
| {failure_tags} | |
| {critic_reason} | |
| Repair instructions: | |
| {repair_instruction} | |
| Generate a corrected version. Prioritize fixing the listed failures while preserving all constraints. | |
| ``` | |
| --- | |
| ## 10. 模块六:Commercial Generator Router | |
| ### 10.1 设计目标 | |
| Generator Router 对外暴露统一接口,对内适配不同闭源模型。 | |
| ```python | |
| class ImageGeneratorAdapter: | |
| def generate(self, request: GenerationRequest) -> GenerationResult: | |
| ... | |
| ``` | |
| ### 10.2 统一请求 schema | |
| ```python | |
| from dataclasses import dataclass | |
| from typing import Any, Dict, List, Literal, Optional, Tuple | |
| @dataclass | |
| class GenerationRequest: | |
| model_family: Literal["gpt_image", "nano_banana", "seedream", "firefly"] | |
| model_name: str | |
| canvas_path: str | |
| prompt: str | |
| mask_path: Optional[str] | |
| reference_paths: List[str] | |
| layout_paths: List[str] | |
| size: Tuple[int, int] | |
| quality: Literal["low", "medium", "high", "auto"] | |
| seed: Optional[int] | |
| metadata: Dict[str, Any] | |
| ``` | |
| ### 10.3 统一响应 schema | |
| ```python | |
| from dataclasses import dataclass | |
| from typing import Any, Dict, Optional | |
| @dataclass | |
| class GenerationResult: | |
| raw_image_path: str | |
| model_name: str | |
| request_id: str | |
| revised_prompt: Optional[str] | |
| latency_ms: int | |
| cost_estimate: Optional[float] | |
| safety_status: Optional[str] | |
| metadata: Dict[str, Any] | |
| ``` | |
| ### 10.4 GPT-image-2 Adapter | |
| GPT-image-2 路线适合显式 mask edit、商品图、局部背景替换和前景保护要求高的场景。 | |
| 伪代码: | |
| ```python | |
| def call_gpt_image_2(canvas_path, mask_path, prompt, size, quality): | |
| result = client.images.edit( | |
| model="gpt-image-2-2026-04-21", | |
| image=open(canvas_path, "rb"), | |
| mask=open(mask_path, "rb"), | |
| prompt=prompt, | |
| size=f"{size[0]}x{size[1]}", | |
| quality=quality | |
| ) | |
| return decode_and_save(result.data[0].b64_json) | |
| ``` | |
| 实际参数需要按当前 SDK 支持情况调整。 | |
| ### 10.5 Nano Banana Adapter | |
| Nano Banana 路线适合多参考图、多轮编辑、复杂指令、创意背景和广告视觉。 | |
| 伪代码: | |
| ```python | |
| def call_nano_banana(prompt, image_paths, aspect_ratio, resolution): | |
| contents = [prompt] + [Image.open(p) for p in image_paths] | |
| response = client.models.generate_content( | |
| model="gemini-3.1-flash-image", | |
| contents=contents, | |
| config=types.GenerateContentConfig( | |
| response_modalities=["TEXT", "IMAGE"], | |
| response_format={ | |
| "image": { | |
| "aspect_ratio": aspect_ratio, | |
| "image_size": resolution | |
| } | |
| } | |
| ) | |
| ) | |
| return extract_and_save_image(response) | |
| ``` | |
| 模型选择建议: | |
| | 用例 | 推荐 | | |
| |---|---| | |
| | 快速批量候选 | Nano Banana 2 / Gemini Flash Image | | |
| | 专业广告图、复杂文字、复杂构图 | Nano Banana Pro / Gemini Pro Image | | |
| | 低延迟高吞吐 | Nano Banana / Gemini Flash Image | | |
| | 前景严格保护、显式 mask | GPT-image-2 masked edit | | |
| | 多参考图、风格/品牌/人物一致性 | Nano Banana 系列 | | |
| --- | |
| ## 11. 模块七:Alpha-aware Foreground Restorer | |
| ### 11.1 核心原则 | |
| 不要直接使用商业模型输出作为最终图。商业模型输出 $Y_i$ 只是候选背景和融合提示。 | |
| 最终结果: | |
| \[ | |
| I_i = A_{hard} \odot I_f + (1 - A_{hard}) \odot Y_i | |
| \] | |
| 其中 $A_{hard}$ 在核心区域为 1,边界区域可用软 alpha。 | |
| 推荐实现: | |
| ```python | |
| def restore_foreground(raw, fg_rgb, alpha, alpha_core, alpha_soft): | |
| out = raw.copy() | |
| # core: exact copy | |
| core = alpha_core > 0.5 | |
| out[core] = fg_rgb[core] | |
| # ring: soft blend | |
| ring = ((alpha_soft > 0.01) & (alpha_core <= 0.5)) | |
| a = alpha_soft[..., None] | |
| out[ring] = a[ring] * fg_rgb[ring] + (1 - a[ring]) * raw[ring] | |
| return out | |
| ``` | |
| ### 11.2 三层输出 | |
| 建议输出: | |
| ```text | |
| background.png # raw/restored background without original foreground if recoverable | |
| foreground.png # original foreground RGB | |
| foreground_alpha.png # alpha | |
| final_composite.png # final | |
| contact_shadow.png # optional estimated shadow layer | |
| report.json # metadata | |
| ``` | |
| ### 11.3 旧阴影处理 | |
| 输入前景可能带原始阴影。如果直接贴到新背景,会造成光照冲突。开发时要区分: | |
| ```text | |
| subject_alpha # 主体 alpha | |
| shadow_alpha_old # 旧阴影,如果能分离则不要直接贴 | |
| contact_shadow_new # 新背景中生成或修复的阴影 | |
| ``` | |
| 推荐策略: | |
| | 输入类型 | 策略 | | |
| |---|---| | |
| | 干净 RGBA,无阴影 | 直接复合 | | |
| | 商品 PNG 带旧阴影 | 尽量分离旧阴影,或让用户选择“保留/去除原阴影” | | |
| | 人像毛发复杂 | 用软 alpha,边界 ring 更宽 | | |
| | 透明/反光物 | 避免完全硬贴,保留部分环境反射区域 | | |
| --- | |
| ## 12. 模块八:Boundary / Contact / Shadow Refiner | |
| ### 12.1 为什么需要第二阶段 refinement | |
| Foreground restoration 解决了前景改写,但可能带来“贴纸感”。因此需要局部修边: | |
| ```text | |
| Stage 1: background generation | |
| Stage 2: boundary/contact refinement | |
| ``` | |
| ### 12.2 contact refinement mask | |
| ```python | |
| M_refine = dilate(A_contact, r=60) + A_ring | |
| M_refine = M_refine - A_core | |
| ``` | |
| 只允许编辑主体外侧窄区域,不允许改主体内部。 | |
| ### 12.3 refinement prompt | |
| ```text | |
| Refine only the narrow region around the foreground boundary and the contact area. | |
| Add natural contact shadow, ambient occlusion, subtle reflection, and lighting transition. | |
| Do not change the foreground subject itself. | |
| Do not alter silhouette, logo, printed text, texture, face, hands, or object parts. | |
| Remove halo or cutout artifacts if present. | |
| Keep the background scene unchanged. | |
| ``` | |
| ### 12.4 是否使用商业模型做 refinement | |
| 推荐两种模式。 | |
| **Mode A:GPT-image-2 local edit** | |
| 适合显式 mask、局部接触阴影、halo 修复。 | |
| **Mode B:本地传统图像处理** | |
| 适合批量、低成本、可复现: | |
| ```text | |
| shadow = soft elliptical shadow under contact points | |
| color_match = local color transfer | |
| edge_feather = guided filter / poisson blend | |
| halo_suppression = boundary color decontamination | |
| ``` | |
| 实际产品里可以混合:先用本地方法快速修,再对高价值图用 GPT-image-2 或 Nano Banana 做二次修复。 | |
| --- | |
| ## 13. 模块九:Critic & Reranker Agent | |
| ### 13.1 总评分 | |
| 对每个候选 $I_i$,定义: | |
| \[ | |
| S_i = | |
| w_f S_f + | |
| w_t S_t + | |
| w_s S_s + | |
| w_l S_l + | |
| w_b S_b + | |
| w_a S_a - | |
| \lambda C_i | |
| \] | |
| 其中: | |
| | 分数 | 含义 | | |
| |---|---| | |
| | $S_f$ | Foreground preservation | | |
| | $S_t$ | Text / prompt alignment | | |
| | $S_s$ | Spatial rationality | | |
| | $S_l$ | Lighting / shadow harmony | | |
| | $S_b$ | Boundary quality | | |
| | $S_a$ | Aesthetic / commercial quality | | |
| | $C_i$ | 成本或延迟惩罚,可选 | | |
| 默认权重: | |
| ```json | |
| { | |
| "foreground_preservation": 0.25, | |
| "prompt_alignment": 0.20, | |
| "spatial_rationality": 0.25, | |
| "lighting_harmony": 0.15, | |
| "boundary_quality": 0.10, | |
| "aesthetic_quality": 0.05 | |
| } | |
| ``` | |
| 对电商图可提高 foreground 和 logo 权重;对创意广告可提高 aesthetic 和 prompt 权重。 | |
| ### 13.2 可计算指标 | |
| #### 前景保持 | |
| 最终图: | |
| \[ | |
| E_{fg}^{final} = \| A_{core} \odot (I_i - I_f) \|_1 | |
| \] | |
| 理论上应接近 0,因为已经强制复合。 | |
| 更重要的是 raw commercial output: | |
| \[ | |
| E_{fg}^{raw} = \| A_{core} \odot (Y_i - I_f) \|_1 | |
| \] | |
| 它衡量商业模型本身是否尊重前景。 | |
| #### 边界异常 | |
| ```text | |
| BoundaryHaloScore = mean color/brightness discontinuity across A_ring | |
| EdgeGradientMismatch = |∇foreground - ∇background| around boundary | |
| ``` | |
| #### 文本一致性 | |
| 可使用: | |
| ```text | |
| CLIPScore(image, prompt) | |
| VLM 0-5 prompt alignment score | |
| human pairwise preference | |
| ``` | |
| #### 空间合理性 | |
| 更依赖 VLM/human: | |
| ```text | |
| object grounded? | |
| scale plausible? | |
| viewpoint consistent? | |
| contact shadow plausible? | |
| support surface plausible? | |
| no duplicate subject? | |
| ``` | |
| ### 13.3 VLM critic prompt | |
| ```text | |
| You are a strict evaluator for foreground-conditioned inpainting. | |
| Inputs: | |
| 1. Original foreground subject with alpha/mask. | |
| 2. Final composite image. | |
| 3. User prompt. | |
| 4. Scene plan. | |
| Evaluate the final image according to: | |
| - foreground preservation | |
| - prompt alignment | |
| - spatial rationality | |
| - lighting and shadow consistency | |
| - boundary quality | |
| - overall commercial image quality | |
| Pay special attention to: | |
| - whether the subject floats | |
| - whether the subject scale is realistic | |
| - whether the support surface is plausible | |
| - whether the contact shadow/reflection is natural | |
| - whether the foreground silhouette, logo, text, face, hands, texture, and pose are unchanged | |
| - whether there are duplicated subjects or hallucinated parts | |
| Return strict JSON: | |
| { | |
| "scores": { | |
| "foreground_preservation": 0, | |
| "prompt_alignment": 0, | |
| "spatial_rationality": 0, | |
| "lighting_harmony": 0, | |
| "boundary_quality": 0, | |
| "aesthetic_quality": 0, | |
| "overall": 0 | |
| }, | |
| "failure_tags": [], | |
| "must_fix": [], | |
| "nice_to_fix": [], | |
| "short_reason": "", | |
| "accept": true | |
| } | |
| ``` | |
| ### 13.4 failure tags | |
| 固定标签集: | |
| ```text | |
| foreground_changed | |
| logo_changed | |
| printed_text_changed | |
| face_changed | |
| hand_changed | |
| duplicate_subject | |
| extra_parts | |
| floating | |
| wrong_scale | |
| wrong_perspective | |
| bad_contact | |
| missing_shadow | |
| wrong_shadow_direction | |
| bad_reflection | |
| halo | |
| hard_cutout | |
| background_prompt_mismatch | |
| overwhelming_background | |
| overstylized | |
| unsafe_or_policy_issue | |
| low_resolution | |
| compression_artifact | |
| ``` | |
| --- | |
| ## 14. 模块十:Repair Planner Agent | |
| ### 14.1 修复动作空间 | |
| ```python | |
| RepairAction = Literal[ | |
| "increase_foreground_protection", | |
| "expand_boundary_ring", | |
| "shrink_boundary_ring", | |
| "add_contact_shadow_refinement", | |
| "rewrite_prompt_more_literal", | |
| "rewrite_prompt_more_spatial", | |
| "reduce_background_complexity", | |
| "switch_generator_model", | |
| "use_gpt_masked_edit", | |
| "use_nano_multi_reference", | |
| "increase_candidate_count", | |
| "change_canvas_layout", | |
| "manual_review_required", | |
| "stop_accept" | |
| ] | |
| ``` | |
| ### 14.2 失败到动作映射 | |
| | failure tag | repair action | | |
| |---|---| | |
| | `foreground_changed` | `increase_foreground_protection` + 强制复合 + prompt 加重 | | |
| | `logo_changed` | 扩大 logo 区保护;prompt 显式保护文字/logo | | |
| | `printed_text_changed` | 前景硬保护;critic 单独检查文字区域 | | |
| | `floating` | `add_contact_shadow_refinement` + prompt 增加 support surface | | |
| | `wrong_scale` | prompt 加真实尺寸、参照物、摄影角度 | | |
| | `wrong_perspective` | 加地平线、相机高度、镜头、透视线 | | |
| | `missing_shadow` | 局部 contact mask 二次修复 | | |
| | `wrong_shadow_direction` | 从 foreground analyzer 继承光照方向重写 prompt | | |
| | `halo` | 调整 A_ring;做边界 color decontamination | | |
| | `duplicate_subject` | prompt 加 “no extra copies”;降低背景复杂度 | | |
| | `background_prompt_mismatch` | 改写 scene plan,减少抽象词 | | |
| | `overstylized` | 降低风格强度,使用 photorealistic/literal prompt | | |
| | `overwhelming_background` | 使用 minimal prompt variant | | |
| ### 14.3 迭代策略 | |
| ```text | |
| T = 1 demo 模式 | |
| T = 2 标准模式 | |
| T = 3 离线高质量模式 | |
| ``` | |
| 停止条件: | |
| ```python | |
| accept == True | |
| or best_score >= threshold | |
| or budget_exhausted | |
| or no_improvement_for_two_rounds | |
| or policy_or_safety_blocked | |
| ``` | |
| --- | |
| ## 15. 完整算法 | |
| ### Algorithm 1:Commercial-FCI-Agent | |
| ```python | |
| def commercial_fci_agent( | |
| input_image, | |
| input_alpha, | |
| user_prompt, | |
| reference_images=None, | |
| layout_hint=None, | |
| models=("gpt-image-2-2026-04-21", "gemini-3.1-flash-image"), | |
| K=8, | |
| T=2, | |
| budget=None | |
| ): | |
| # 1. Asset normalization | |
| asset = normalize_asset(input_image, input_alpha) | |
| masks = build_masks(asset.alpha) | |
| canvas = build_condition_canvas(asset, layout_hint) | |
| # 2. Foreground understanding | |
| fg_info = foreground_analyzer( | |
| foreground_rgb=asset.fg_rgb, | |
| alpha=asset.alpha | |
| ) | |
| # 3. Scene planning | |
| scene_plan = scene_planner( | |
| user_prompt=user_prompt, | |
| foreground_info=fg_info, | |
| references=reference_images, | |
| layout_hint=layout_hint | |
| ) | |
| history = [] | |
| best = None | |
| for t in range(T): | |
| # 4. Prompt ensemble | |
| prompt_list = make_prompt_ensemble( | |
| scene_plan=scene_plan, | |
| fg_info=fg_info, | |
| previous_failures=history, | |
| K=K | |
| ) | |
| # 5. Mask/layout preparation | |
| condition_package = make_condition_package( | |
| canvas=canvas, | |
| masks=masks, | |
| references=reference_images, | |
| layout_hint=layout_hint, | |
| scene_plan=scene_plan | |
| ) | |
| round_candidates = [] | |
| for prompt in prompt_list: | |
| # 6. Generator routing | |
| gen_request = route_model( | |
| prompt=prompt, | |
| condition_package=condition_package, | |
| models=models, | |
| scene_plan=scene_plan, | |
| budget=budget | |
| ) | |
| raw = call_commercial_generator(gen_request) | |
| # 7. Hard foreground restoration | |
| restored = restore_foreground( | |
| raw_image=raw.image, | |
| foreground_rgb=asset.fg_rgb, | |
| alpha_core=masks.alpha_core, | |
| alpha_soft=masks.alpha_soft | |
| ) | |
| # 8. Optional boundary/contact refinement | |
| refined = maybe_refine_boundary_contact( | |
| restored=restored, | |
| raw=raw.image, | |
| asset=asset, | |
| masks=masks, | |
| scene_plan=scene_plan, | |
| budget=budget | |
| ) | |
| # 9. Critic scoring | |
| score_report = critic_score( | |
| original_foreground=asset.fg_rgb, | |
| alpha=asset.alpha, | |
| raw_image=raw.image, | |
| final_image=refined, | |
| user_prompt=user_prompt, | |
| scene_plan=scene_plan, | |
| fg_info=fg_info | |
| ) | |
| candidate = { | |
| "final": refined, | |
| "raw": raw.image, | |
| "prompt": prompt, | |
| "model": raw.model_name, | |
| "score": score_report, | |
| "metadata": raw.metadata | |
| } | |
| round_candidates.append(candidate) | |
| history.append(candidate) | |
| # 10. Rerank | |
| round_candidates.sort( | |
| key=lambda c: c["score"]["scores"]["overall"], | |
| reverse=True | |
| ) | |
| if best is None or round_candidates[0]["score"]["scores"]["overall"] > best["score"]["scores"]["overall"]: | |
| best = round_candidates[0] | |
| # 11. Stop or repair | |
| if should_accept(best["score"]): | |
| break | |
| repair_action = repair_planner( | |
| best_score=best["score"], | |
| history=history, | |
| scene_plan=scene_plan, | |
| masks=masks | |
| ) | |
| scene_plan, masks, canvas, models = apply_repair_action( | |
| repair_action, | |
| scene_plan, | |
| masks, | |
| canvas, | |
| models | |
| ) | |
| # 12. Export | |
| return export_result(best, asset, masks, history) | |
| ``` | |
| --- | |
| ## 16. `report.json` 设计 | |
| 每次输出必须可复查、可复现。 | |
| ```json | |
| { | |
| "task_id": "fci_20260602_0001", | |
| "input": { | |
| "foreground_path": "foreground.png", | |
| "alpha_path": "alpha.png", | |
| "user_prompt": "rainy Tokyo street at night", | |
| "canvas_size": [1536, 1024] | |
| }, | |
| "models": [ | |
| { | |
| "name": "gpt-image-2-2026-04-21", | |
| "role": "masked_background_generator", | |
| "requests": 8 | |
| }, | |
| { | |
| "name": "gemini-3.1-flash-image", | |
| "role": "multi_reference_generator", | |
| "requests": 4 | |
| } | |
| ], | |
| "foreground_analysis": {}, | |
| "scene_plan": {}, | |
| "masks": { | |
| "alpha_core": "alpha_core.png", | |
| "alpha_soft": "alpha_soft.png", | |
| "background_edit": "mask_bg.png", | |
| "boundary_ring": "mask_ring.png", | |
| "contact": "mask_contact.png" | |
| }, | |
| "best_candidate": { | |
| "candidate_id": 5, | |
| "model": "gpt-image-2-2026-04-21", | |
| "prompt": "...", | |
| "score": { | |
| "foreground_preservation": 5, | |
| "prompt_alignment": 4, | |
| "spatial_rationality": 5, | |
| "lighting_harmony": 4, | |
| "boundary_quality": 4, | |
| "aesthetic_quality": 4, | |
| "overall": 4.55 | |
| }, | |
| "failure_tags": [] | |
| }, | |
| "all_candidates": [ | |
| { | |
| "candidate_id": 0, | |
| "model": "...", | |
| "prompt": "...", | |
| "raw_path": "...", | |
| "final_path": "...", | |
| "score": {}, | |
| "failure_tags": ["missing_shadow"] | |
| } | |
| ], | |
| "export": { | |
| "final_composite": "final.png", | |
| "background": "background.png", | |
| "foreground": "foreground.png", | |
| "alpha": "alpha.png", | |
| "contact_shadow": "shadow.png" | |
| } | |
| } | |
| ``` | |
| --- | |
| ## 17. 实验设计 | |
| ### 17.1 Baselines | |
| 必须包含以下 baseline: | |
| | Baseline | 描述 | | |
| |---|---| | |
| | `GPT-raw` | 直接 GPT-image-2 masked edit,不重贴前景 | | |
| | `GPT+Restore` | GPT-image-2 + 原前景强制复合 | | |
| | `GPT+Restore+Rerank` | 生成 K 个候选后 rerank | | |
| | `GPT+Restore+Refine` | 加 boundary/contact refinement | | |
| | `GPT-Agent` | 完整 repair loop | | |
| | `Nano-raw` | Nano Banana 多参考图直接编辑 | | |
| | `Nano+Restore` | Nano Banana + 原前景复合 | | |
| | `Nano-Agent` | Nano Banana + critic + repair | | |
| | `Hybrid-Agent` | GPT 做显式 mask edit,Nano 做复杂多参考或风格候选 | | |
| | `Open-source Inpainting` | BrushNet / PowerPaint / SD inpainting / Pinco 等 | | |
| ### 17.2 Ablation | |
| 重点消融: | |
| ```text | |
| w/o foreground restoration | |
| w/o boundary ring | |
| w/o contact refinement | |
| w/o prompt planner | |
| w/o prompt ensemble | |
| w/o VLM critic | |
| w/o repair loop | |
| GPT-only vs Nano-only vs Hybrid | |
| K=1 vs K=4 vs K=8 vs K=16 | |
| T=1 vs T=2 vs T=3 | |
| ``` | |
| ### 17.3 Benchmark 组成 | |
| 建议构造 **Commercial-FCI-Bench**: | |
| | 类别 | 样本数建议 | 难点 | | |
| |---|---:|---| | |
| | 商品硬边 | 100 | logo、文字、反射、接触阴影 | | |
| | 人像 | 100 | 脸、手、头发、衣服纹理 | | |
| | 动物 | 80 | 毛发、地面接触 | | |
| | 车辆 | 60 | 轮胎接触、尺度、透视 | | |
| | 家具 | 60 | 支撑脚、室内透视 | | |
| | 透明/反光物 | 60 | 玻璃、金属、反射 | | |
| | 复杂边界 | 60 | 羽毛、树枝、蕾丝 | | |
| | 多前景 | 80 | 相对位置、遮挡、尺度 | | |
| 每个前景配 5 个背景 prompt: | |
| ```text | |
| studio product photography | |
| outdoor natural scene | |
| night city / rainy street | |
| luxury commercial advertisement | |
| indoor table/floor scene | |
| ``` | |
| ### 17.4 主指标 | |
| | 指标 | 说明 | | |
| |---|---| | |
| | Human Preference Win Rate | 与 baseline 成对比较 | | |
| | Foreground Preservation Score | 人工/VLM + raw foreground deviation | | |
| | Prompt Alignment Score | VLM/CLIP/human | | |
| | Spatial Rationality Score | 是否漂浮、尺度、透视、接触 | | |
| | Boundary Quality Score | halo、硬边、融合自然度 | | |
| | Lighting Harmony Score | 阴影方向、色温、反射 | | |
| | Diversity | 同一前景同一 prompt 下合理候选多样性 | | |
| | Cost-quality curve | API 成本/延迟 vs 质量 | | |
| 不要只用 FID。该任务的核心不是整体图像分布,而是条件一致性、空间合理性和前景保持。 | |
| --- | |
| ## 18. 工程实现建议 | |
| ### 18.1 代码结构 | |
| ```text | |
| commercial_fci_agent/ | |
| configs/ | |
| default.yaml | |
| gpt_image_2.yaml | |
| nano_banana.yaml | |
| eval.yaml | |
| core/ | |
| asset_normalizer.py | |
| mask_builder.py | |
| canvas_builder.py | |
| compositor.py | |
| shadow_refiner.py | |
| agents/ | |
| foreground_analyzer.py | |
| scene_planner.py | |
| prompt_ensemble.py | |
| critic.py | |
| repair_planner.py | |
| generators/ | |
| base.py | |
| openai_gpt_image.py | |
| google_nano_banana.py | |
| router.py | |
| eval/ | |
| metrics.py | |
| vlm_eval.py | |
| human_eval_export.py | |
| pipelines/ | |
| run_single.py | |
| run_batch.py | |
| run_benchmark.py | |
| schemas/ | |
| request.py | |
| result.py | |
| report.py | |
| outputs/ | |
| ``` | |
| ### 18.2 配置示例 | |
| ```yaml | |
| canvas: | |
| width: 1536 | |
| height: 1024 | |
| background: neutral_gray | |
| align_to_multiple: 16 | |
| mask: | |
| threshold: 0.5 | |
| r_core: 3 | |
| r_dilate: 8 | |
| sigma: 1.0 | |
| contact_radius: 80 | |
| generation: | |
| K: 8 | |
| T: 2 | |
| models: | |
| - gpt-image-2-2026-04-21 | |
| - gemini-3.1-flash-image | |
| quality: high | |
| max_parallel_requests: 4 | |
| rerank: | |
| accept_threshold: 4.2 | |
| weights: | |
| foreground_preservation: 0.25 | |
| prompt_alignment: 0.20 | |
| spatial_rationality: 0.25 | |
| lighting_harmony: 0.15 | |
| boundary_quality: 0.10 | |
| aesthetic_quality: 0.05 | |
| repair: | |
| enable: true | |
| max_rounds: 2 | |
| enable_boundary_refine: true | |
| switch_model_on_repeated_failure: true | |
| export: | |
| save_raw_candidates: true | |
| save_masks: true | |
| save_report_json: true | |
| ``` | |
| ### 18.3 API 成本与并发 | |
| 开发时要把 candidate generation 做成异步队列: | |
| ```text | |
| Queue: | |
| task_id | |
| candidate_id | |
| model | |
| prompt | |
| mask | |
| canvas | |
| status | |
| retry_count | |
| result_path | |
| request_id | |
| ``` | |
| 推荐: | |
| ```text | |
| Preview mode: | |
| K=4, quality=medium/auto, T=1 | |
| Production mode: | |
| K=8, quality=high, T=2 | |
| Offline benchmark: | |
| K=16, quality=high, T=2~3 | |
| ``` | |
| ### 18.4 版本锁定 | |
| 尽量记录: | |
| ```text | |
| model snapshot | |
| API date | |
| SDK version | |
| prompt template version | |
| mask builder version | |
| critic prompt version | |
| ``` | |
| 闭源模型会更新。如果 API 支持 snapshot,应优先使用 snapshot 以提高实验复现性。 | |
| --- | |
| ## 19. 失败模式与解决策略 | |
| | 失败 | 原因 | 解决 | | |
| |---|---|---| | |
| | 前景被改 | 商业模型重画主体 | 强制复合;扩大 A_core;prompt 加 preserve constraints | | |
| | logo/文字变形 | 模型生成式重绘 | logo 区硬保护;critic 单独检查 | | |
| | 主体漂浮 | 缺接触阴影/地面 | contact refinement;scene plan 加 support surface | | |
| | 尺度不对 | prompt 参照不足 | 加真实尺寸、镜头、参照物 | | |
| | 透视不对 | 没有 horizon/camera | layout guide + camera prompt | | |
| | 阴影方向错 | 背景光照与前景不一致 | Foreground Analyzer 提取光照,repair prompt 指定 | | |
| | halo 白边 | alpha/mask 边界差 | A_ring 软融合;color decontamination | | |
| | 背景过乱 | prompt 过开放 | minimal/conservative prompt | | |
| | 出现复制主体 | 模型把前景当 prompt 对象 | negative constraints + mask guide + rerank | | |
| | 多轮编辑漂移 | conversation edit 累积误差 | 每轮都从原始前景和原始 canvas 重新生成,而不是连续改最终图 | | |
| --- | |
| ## 20. 推荐论文结构 | |
| ```text | |
| Title: | |
| Commercial-FCI-Agent: Training-Free Foreground-Conditioned Inpainting with Commercial Multimodal Image Models | |
| Abstract | |
| 1. Introduction | |
| - Foreground-conditioned inpainting 的实用价值 | |
| - 开源模型训练成本高,闭源模型强但不可控 | |
| - 我们提出 agentic black-box protocol | |
| 2. Related Work | |
| - Text-guided inpainting | |
| - Foreground-conditioned inpainting: Anywhere, Pinco, InpaintDPO | |
| - Commercial multimodal image models | |
| - VLM-based evaluation and agentic generation | |
| 3. Problem Formulation | |
| - 输入输出 | |
| - 前景保持、背景一致性、空间合理性 | |
| 4. Method | |
| 4.1 Asset normalization and mask decomposition | |
| 4.2 Foreground analyzer | |
| 4.3 Scene planner | |
| 4.4 Commercial generator router | |
| 4.5 Alpha-aware foreground restoration | |
| 4.6 Critic-guided reranking and repair | |
| 5. Benchmark and Metrics | |
| - Commercial-FCI-Bench | |
| - 主客观指标 | |
| 6. Experiments | |
| - Baselines | |
| - Quantitative results | |
| - Human preference | |
| - Ablation | |
| - Cost-quality tradeoff | |
| 7. Limitations | |
| - 闭源不可复现性 | |
| - API 成本 | |
| - 安全策略和可用性 | |
| - 极复杂透明/反射/多物体仍困难 | |
| 8. Conclusion | |
| ``` | |
| --- | |
| ## 21. 最小可行产品版本 | |
| MVP 只需要做: | |
| ```text | |
| 1. 输入 RGBA 前景 + prompt | |
| 2. 构造 canvas 和 background mask | |
| 3. 调 GPT-image-2 masked edit 生成 K=4 候选 | |
| 4. 用原始 alpha 强制贴回前景 | |
| 5. 用 VLM critic 打分 | |
| 6. 返回最高分 final.png + report.json | |
| ``` | |
| MVP 的价值已经很强,因为它能证明: | |
| ```text | |
| raw commercial model < commercial model + foreground restoration < commercial model + restoration + rerank | |
| ``` | |
| --- | |
| ## 22. 开发里程碑 | |
| ### Milestone 1:单模型可跑通 | |
| - 支持 RGBA 输入。 | |
| - GPT-image-2 masked edit。 | |
| - 前景强制复合。 | |
| - 输出 final 和 report。 | |
| - 人工检查 20 个样本。 | |
| ### Milestone 2:critic 和 rerank | |
| - K=8 候选。 | |
| - VLM critic JSON。 | |
| - 自动排序。 | |
| - 输出 candidate grid 和分数表。 | |
| ### Milestone 3:repair loop | |
| - failure tags。 | |
| - prompt repair。 | |
| - contact-shadow refinement。 | |
| - halo 修复。 | |
| - T=2 自动迭代。 | |
| ### Milestone 4:Nano Banana adapter | |
| - mask visualization 输入。 | |
| - 多参考图输入。 | |
| - GPT vs Nano vs Hybrid 对比。 | |
| ### Milestone 5:benchmark | |
| - 500–800 个前景。 | |
| - 每个 5 个 prompt。 | |
| - Human preference 标注。 | |
| - ablation 和 cost-quality curve。 | |
| --- | |
| ## 23. 实现原则 | |
| 1. **永远保存原始前景并最终重贴。** | |
| 不要相信闭源模型会严格保护前景。 | |
| 2. **让商业模型生成背景,而不是决定最终图。** | |
| 它是 candidate generator,不是 constraint solver。 | |
| 3. **把 foreground preservation、spatial rationality、boundary quality 分开评估。** | |
| 单一美学分数会掩盖漂浮、尺度错误、logo 改写等关键问题。 | |
| 4. **从第一天就记录 `report.json`。** | |
| 没有日志就无法写论文,也无法定位失败。 | |
| 5. **每轮 repair 从原始前景和原始 canvas 重新开始。** | |
| 不要在已漂移的图上连续编辑,否则错误会累积。 | |
| 6. **把 prompt、mask、layout 和模型版本都当作实验变量记录。** | |
| 这对复现、ablation 和成本分析都很重要。 | |
| --- | |
| ## 24. 一句话版方法 | |
| > **Commercial-FCI-Agent treats commercial multimodal image models as black-box background proposal generators, then enforces foreground preservation through alpha-aware restoration and improves spatial realism through VLM-guided reranking, failure diagnosis, and iterative repair.** | |
| 这套方法最 promising 的地方在于:它不和 GPT-image-2 / Nano Banana 比拼底层生成能力,而是利用它们的强生成能力,同时用系统设计补齐闭源模型在 **精确前景保护、空间关系可靠性、可复现评测和产品级可控性** 上的短板。 | |
| --- | |
| ## 25. 参考资料 | |
| 以下参考资料用于定位相关工作与商业模型能力。具体 API 参数、模型名称和可用能力会更新,工程实现时应以官方最新文档为准。 | |
| 1. OpenAI GPT-image-2 model documentation | |
| <https://developers.openai.com/api/docs/models/gpt-image-2> | |
| 2. OpenAI Image generation and image edit guide | |
| <https://developers.openai.com/api/docs/guides/image-generation> | |
| 3. Google Gemini image generation / Nano Banana documentation | |
| <https://ai.google.dev/gemini-api/docs/image-generation> | |
| 4. Anywhere: A Multi-Agent Framework for Reliable and Diverse Foreground-Conditioned Image Generation | |
| <https://arxiv.org/abs/2404.18598> | |
| 5. Pinco: Position-induced Consistent Adapter for Diffusion Transformer in Foreground-conditioned Inpainting | |
| <https://openaccess.thecvf.com/content/ICCV2025/html/Lu_Pinco_Position-induced_Consistent_Adapter_for_Diffusion_Transformer_in_Foreground-conditioned_Inpainting_ICCV_2025_paper.html> | |
| 6. InpaintDPO: Spatial Relationship Hallucinations in Image Inpainting | |
| <https://arxiv.org/abs/2512.15644> | |
| 7. BrushNet: A Plug-and-Play Image Inpainting Model | |
| <https://arxiv.org/abs/2403.06976> | |
| 8. PowerPaint: A Versatile Image Inpainting Model | |
| <https://powerpaint.github.io/> | |
| 9. LayerDiffuse: Transparent Image Layer Diffusion | |
| <https://arxiv.org/abs/2402.17113> | |
| 10. LASAGNA: Layered Scene Generation with Visual Effects | |
| <https://arxiv.org/abs/2601.15507> | |