LeafCat79 commited on
Commit
d0a78ed
·
verified ·
1 Parent(s): c352e81

Upgrade generation to full-quality SDXL

Browse files
Files changed (2) hide show
  1. README.md +12 -9
  2. app.py +42 -17
README.md CHANGED
@@ -84,15 +84,15 @@ If the Space has an `HF_TOKEN` secret and `USE_HF_PROMPT_PROVIDER=1`, it first u
84
  The deployed Space is designed for Hugging Face ZeroGPU and uses real neural models for generation and transparent cutout extraction:
85
 
86
  ```text
87
- segmind/SSD-1B + latent-consistency/lcm-lora-ssd-1b
88
  ZhengPeng7/BiRefNet (transparent-cutout alpha only)
89
  ```
90
 
91
- SSD-1B is an Apache-2.0 distilled SDXL model. The LCM adapter enables low-step generation. The weights run inside the Space's ZeroGPU allocation, so the app does not consume Hugging Face Inference Provider credits and does not require a paid image API.
92
 
93
- Sprites and backgrounds are generated directly from their written prompts. No procedurally drawn sprite, palette, silhouette, camera guide, or background layout is passed into the production image generator. Sprite prompts use a portrait canvas and positive single-subject composition language; the negative prompt separately rejects sheets, lineups, repeated subjects, multiple views, and surrounding environments. BiRefNet predicts only the alpha mask for transparent cutouts while preserving SSD-1B's generated RGB artwork. After neural transparency extraction, sprites with zero or multiple significant foreground components, retained scene-like alpha, implausible silhouettes, or inadequate scale are regenerated with a new seed up to `PRIMARY_SPRITE_ATTEMPTS` times. If every attempt fails, the deployed Space returns an explicit model-generation error rather than returning a contact sheet or environment-contaminated cutout. Background negative prompts reject characters and generic particle overlays.
94
 
95
- SSD-1B's CLIP text encoder has a 77-token context window. Production prompts therefore use a conservative word budget, place the camera and single-subject contract first, and compact the free-form description. This prevents those critical constraints from being silently truncated behind verbose style text.
96
 
97
  The deployed Space is fail-closed: if the primary neural model cannot run, generation stops with a visible error instead of silently returning procedural art and calling it model output. The procedural renderer remains available for local development and diagnostics only and cannot influence successful public generations.
98
 
@@ -105,16 +105,19 @@ The primary pipeline can be configured with:
105
  ```text
106
  USE_PRIMARY_IMAGE_MODEL=1
107
  REQUIRE_PRIMARY_IMAGE_MODEL=1
108
- PRIMARY_IMAGE_MODEL=segmind/SSD-1B
109
- PRIMARY_LORA_MODEL=latent-consistency/lcm-lora-ssd-1b
 
110
  FOREGROUND_MODEL_ID=ZhengPeng7/BiRefNet
111
  FOREGROUND_MODEL_REVISION=e2bf8e4460fc8fa32bba5ea4d94b3233d367b0e4
112
- PRIMARY_IMAGE_STEPS=4
113
- PRIMARY_SPRITE_STEPS=6
114
  PRIMARY_SPRITE_ATTEMPTS=3
115
- PRIMARY_GUIDANCE_SCALE=1.5
116
  ```
117
 
 
 
118
  `HF_TOKEN` is not required to download these public image models. It remains useful for the optional prompt-model provider and for the UI's token-validity check.
119
 
120
  ## Optional generation tiers
 
84
  The deployed Space is designed for Hugging Face ZeroGPU and uses real neural models for generation and transparent cutout extraction:
85
 
86
  ```text
87
+ stabilityai/stable-diffusion-xl-base-1.0 (native full-step inference)
88
  ZhengPeng7/BiRefNet (transparent-cutout alpha only)
89
  ```
90
 
91
+ SDXL Base is a 3B-parameter Open RAIL++ text-to-image model. Production generation intentionally runs without an LCM acceleration adapter: backgrounds use 30 denoising steps, sprites use 32, and classifier-free guidance is 8.0. This is slower than the earlier SSD-1B + LCM configuration, but it preserves the base model's full denoising process, makes negative prompts effective, and gives strict perspective and composition instructions substantially more influence. The weights run inside the Space's free ZeroGPU allocation, so the app does not consume Hugging Face Inference Provider credits and does not require a paid image API.
92
 
93
+ Sprites and backgrounds are generated directly from their written prompts. No procedurally drawn sprite, palette, silhouette, camera guide, or background layout is passed into the production image generator. Strict top-down sprites use a native 1024×1024 canvas, an orthographic 90-degree overhead instruction at the front of the positive prompt, and an effective negative prompt that rejects front, oblique, three-quarter, eye-level, and side-elevation views. Other perspectives use camera-appropriate native SDXL aspect ratios. BiRefNet predicts only the alpha mask for transparent cutouts while preserving SDXL's generated RGB artwork. After neural transparency extraction, sprites with zero or multiple significant foreground components, retained scene-like alpha, implausible silhouettes, or inadequate scale are regenerated with a new seed up to `PRIMARY_SPRITE_ATTEMPTS` times. If every attempt fails, the deployed Space returns an explicit model-generation error rather than returning a character sheet, environment-contaminated cutout, or obvious upright portrait. Background negative prompts reject characters and generic particle overlays.
94
 
95
+ SDXL uses two fixed text encoders with limited prompt context. Production prompts therefore retain a conservative word budget, place the camera and single-subject contract first, and compact the free-form description. This prevents critical geometry constraints from being silently truncated behind verbose style text.
96
 
97
  The deployed Space is fail-closed: if the primary neural model cannot run, generation stops with a visible error instead of silently returning procedural art and calling it model output. The procedural renderer remains available for local development and diagnostics only and cannot influence successful public generations.
98
 
 
105
  ```text
106
  USE_PRIMARY_IMAGE_MODEL=1
107
  REQUIRE_PRIMARY_IMAGE_MODEL=1
108
+ PRIMARY_IMAGE_MODEL=stabilityai/stable-diffusion-xl-base-1.0
109
+ PRIMARY_IMAGE_REVISION=
110
+ PRIMARY_LORA_MODEL=
111
  FOREGROUND_MODEL_ID=ZhengPeng7/BiRefNet
112
  FOREGROUND_MODEL_REVISION=e2bf8e4460fc8fa32bba5ea4d94b3233d367b0e4
113
+ PRIMARY_IMAGE_STEPS=30
114
+ PRIMARY_SPRITE_STEPS=32
115
  PRIMARY_SPRITE_ATTEMPTS=3
116
+ PRIMARY_GUIDANCE_SCALE=8.0
117
  ```
118
 
119
+ `PRIMARY_LORA_MODEL` remains an optional compatibility switch for an LCM acceleration adapter. When it is non-empty and the step/guidance variables are not explicitly overridden, the app automatically returns to LCM-safe defaults of 6 background steps, 8 sprite steps, and guidance 1.5. That mode is faster but is no longer the production default because quality and contract adherence take priority.
120
+
121
  `HF_TOKEN` is not required to download these public image models. It remains useful for the optional prompt-model provider and for the UI's token-validity check.
122
 
123
  ## Optional generation tiers
app.py CHANGED
@@ -74,17 +74,22 @@ class IntegrationReport:
74
 
75
 
76
  HF_TOKEN = os.environ.get("HF_TOKEN", "")
77
- PRIMARY_IMAGE_MODEL = os.environ.get("PRIMARY_IMAGE_MODEL", "segmind/SSD-1B")
78
- PRIMARY_LORA_MODEL = os.environ.get("PRIMARY_LORA_MODEL", "latent-consistency/lcm-lora-ssd-1b")
 
 
 
 
 
79
  FOREGROUND_MODEL_ID = os.environ.get("FOREGROUND_MODEL_ID", "ZhengPeng7/BiRefNet")
80
  FOREGROUND_MODEL_REVISION = os.environ.get(
81
  "FOREGROUND_MODEL_REVISION",
82
  "e2bf8e4460fc8fa32bba5ea4d94b3233d367b0e4",
83
  )
84
- PRIMARY_IMAGE_STEPS = int(os.environ.get("PRIMARY_IMAGE_STEPS", "6"))
85
- PRIMARY_SPRITE_STEPS = int(os.environ.get("PRIMARY_SPRITE_STEPS", "8"))
86
  PRIMARY_SPRITE_ATTEMPTS = max(1, int(os.environ.get("PRIMARY_SPRITE_ATTEMPTS", "3")))
87
- PRIMARY_GUIDANCE_SCALE = float(os.environ.get("PRIMARY_GUIDANCE_SCALE", "1.5"))
88
  USE_PRIMARY_IMAGE_MODEL = os.environ.get("USE_PRIMARY_IMAGE_MODEL", "1") == "1"
89
  REQUIRE_PRIMARY_IMAGE_MODEL = os.environ.get(
90
  "REQUIRE_PRIMARY_IMAGE_MODEL",
@@ -1779,7 +1784,8 @@ def diffusion_negative_prompt(spec: AssetSpec) -> str:
1779
  camera_negative = {
1780
  "top_down": (
1781
  ", sky, horizon, eye-level view, front view, front-facing facade, upright portrait, "
1782
- "standing toward camera, face looking at viewer, vanishing point"
 
1783
  ),
1784
  "isometric": ", eye-level view, front view, inconsistent perspective, horizon",
1785
  "side_view": ", overhead view, top-down view, isometric view",
@@ -1861,7 +1867,7 @@ def initialize_primary_image_model() -> None:
1861
 
1862
  try:
1863
  import torch
1864
- from diffusers import AutoPipelineForText2Image, LCMScheduler
1865
 
1866
  model_kwargs = {
1867
  "torch_dtype": torch.float16,
@@ -1870,11 +1876,16 @@ def initialize_primary_image_model() -> None:
1870
  }
1871
  if HF_TOKEN:
1872
  model_kwargs["token"] = HF_TOKEN
 
 
1873
 
1874
  text_pipe = AutoPipelineForText2Image.from_pretrained(PRIMARY_IMAGE_MODEL, **model_kwargs)
1875
- text_pipe.scheduler = LCMScheduler.from_config(text_pipe.scheduler.config)
1876
- text_pipe.load_lora_weights(PRIMARY_LORA_MODEL)
1877
- text_pipe.fuse_lora()
 
 
 
1878
  text_pipe = text_pipe.to("cuda")
1879
  if hasattr(text_pipe, "set_progress_bar_config"):
1880
  text_pipe.set_progress_bar_config(disable=True)
@@ -1947,8 +1958,9 @@ def primary_diffusion_prompt(spec: AssetSpec) -> str:
1947
  subject = (spec.group or spec.role).replace("_", " ")
1948
  if spec.camera == "top_down":
1949
  return compact_prompt_words(
1950
- f"Top-down game sprite. Camera directly above the {subject}, looking straight down. "
1951
- "Show top surfaces and a compact foreshortened silhouette, never an upright front-facing portrait. "
 
1952
  f"{description}. Exactly one complete isolated subject, centered on uniform white. "
1953
  f"{variation}Crisp game-ready details. No scene.",
1954
  55,
@@ -2633,7 +2645,7 @@ def generate_asset(spec: AssetSpec, index: int, run_id: int) -> tuple[str, str,
2633
 
2634
  if USE_PRIMARY_IMAGE_MODEL:
2635
  png_content, error = primary_diffusion_png(spec, index, run_id)
2636
- source = f"{PRIMARY_IMAGE_MODEL} + {PRIMARY_LORA_MODEL} on ZeroGPU"
2637
  if not is_background:
2638
  source += f" + {FOREGROUND_MODEL_ID} neural alpha"
2639
  if error:
@@ -2879,14 +2891,23 @@ def model_source_note(image_model: str) -> str:
2879
  if "via fal-ai" in lowered or "via replicate" in lowered:
2880
  return "routed image-to-image refinement—preserves a layout guide but consumes account inference credit"
2881
  if "zerogpu" in lowered:
 
 
2882
  return "direct locally hosted text-to-image diffusion—no procedural guide or per-image API credit; visual review required"
2883
  return "diffusion output—higher visual variety; verify subject, perspective, alpha edges, and prompt compliance"
2884
 
2885
 
 
 
 
 
 
 
 
2886
  def model_configuration_summary() -> str:
2887
  prompt_source = HF_PROMPT_MODEL if USE_HF_PROMPT_PROVIDER else "local prompt interpreter"
2888
  sprite_chain = []
2889
- primary_source = f"{PRIMARY_IMAGE_MODEL} + {PRIMARY_LORA_MODEL} on ZeroGPU"
2890
  if USE_PRIMARY_IMAGE_MODEL:
2891
  sprite_chain.append(f"direct text-to-image {primary_source} → neural alpha {FOREGROUND_MODEL_ID}")
2892
  if USE_DIFFUSION_FOR_SPRITES:
@@ -2918,7 +2939,9 @@ def model_configuration_summary() -> str:
2918
  remote_fallback = " → ".join(remote_sources) if remote_sources else "disabled"
2919
  neural_models = []
2920
  if USE_PRIMARY_IMAGE_MODEL:
2921
- neural_models.append(f"{PRIMARY_IMAGE_MODEL} + {PRIMARY_LORA_MODEL}")
 
 
2922
  neural_models.append(FOREGROUND_MODEL_ID)
2923
  if USE_DIFFUSION_FOR_SPRITES or USE_DIFFUSION_FOR_BACKGROUNDS:
2924
  neural_models.append(FREE_IMAGE_MODEL)
@@ -3045,7 +3068,8 @@ def estimate_generation_gpu_duration(
3045
  output_count = max(1, len(resolve_role_lines(html_code or "", roles or "")[0]))
3046
  except AssetManifestError:
3047
  output_count = 1
3048
- return min(120, 30 + output_count * 18)
 
3049
 
3050
 
3051
  @gpu_task(duration=estimate_generation_gpu_duration)
@@ -3143,7 +3167,8 @@ def estimate_regeneration_gpu_duration(state: dict, selected_role: str, approved
3143
  count = sum(1 for spec in specs if (spec.group or spec.role) == group)
3144
  else:
3145
  count = 1
3146
- return min(120, 27 + max(1, count) * 18)
 
3147
 
3148
 
3149
  @gpu_task(duration=estimate_regeneration_gpu_duration)
 
74
 
75
 
76
  HF_TOKEN = os.environ.get("HF_TOKEN", "")
77
+ PRIMARY_IMAGE_MODEL = os.environ.get(
78
+ "PRIMARY_IMAGE_MODEL",
79
+ "stabilityai/stable-diffusion-xl-base-1.0",
80
+ )
81
+ PRIMARY_IMAGE_REVISION = os.environ.get("PRIMARY_IMAGE_REVISION", "").strip()
82
+ PRIMARY_LORA_MODEL = os.environ.get("PRIMARY_LORA_MODEL", "").strip()
83
+ PRIMARY_USES_LCM = bool(PRIMARY_LORA_MODEL)
84
  FOREGROUND_MODEL_ID = os.environ.get("FOREGROUND_MODEL_ID", "ZhengPeng7/BiRefNet")
85
  FOREGROUND_MODEL_REVISION = os.environ.get(
86
  "FOREGROUND_MODEL_REVISION",
87
  "e2bf8e4460fc8fa32bba5ea4d94b3233d367b0e4",
88
  )
89
+ PRIMARY_IMAGE_STEPS = int(os.environ.get("PRIMARY_IMAGE_STEPS", "6" if PRIMARY_USES_LCM else "30"))
90
+ PRIMARY_SPRITE_STEPS = int(os.environ.get("PRIMARY_SPRITE_STEPS", "8" if PRIMARY_USES_LCM else "32"))
91
  PRIMARY_SPRITE_ATTEMPTS = max(1, int(os.environ.get("PRIMARY_SPRITE_ATTEMPTS", "3")))
92
+ PRIMARY_GUIDANCE_SCALE = float(os.environ.get("PRIMARY_GUIDANCE_SCALE", "1.5" if PRIMARY_USES_LCM else "8.0"))
93
  USE_PRIMARY_IMAGE_MODEL = os.environ.get("USE_PRIMARY_IMAGE_MODEL", "1") == "1"
94
  REQUIRE_PRIMARY_IMAGE_MODEL = os.environ.get(
95
  "REQUIRE_PRIMARY_IMAGE_MODEL",
 
1784
  camera_negative = {
1785
  "top_down": (
1786
  ", sky, horizon, eye-level view, front view, front-facing facade, upright portrait, "
1787
+ "standing toward camera, face looking at viewer, three-quarter view, oblique view, "
1788
+ "side elevation, low camera, vanishing point"
1789
  ),
1790
  "isometric": ", eye-level view, front view, inconsistent perspective, horizon",
1791
  "side_view": ", overhead view, top-down view, isometric view",
 
1867
 
1868
  try:
1869
  import torch
1870
+ from diffusers import AutoPipelineForText2Image
1871
 
1872
  model_kwargs = {
1873
  "torch_dtype": torch.float16,
 
1876
  }
1877
  if HF_TOKEN:
1878
  model_kwargs["token"] = HF_TOKEN
1879
+ if PRIMARY_IMAGE_REVISION:
1880
+ model_kwargs["revision"] = PRIMARY_IMAGE_REVISION
1881
 
1882
  text_pipe = AutoPipelineForText2Image.from_pretrained(PRIMARY_IMAGE_MODEL, **model_kwargs)
1883
+ if PRIMARY_LORA_MODEL:
1884
+ from diffusers import LCMScheduler
1885
+
1886
+ text_pipe.scheduler = LCMScheduler.from_config(text_pipe.scheduler.config)
1887
+ text_pipe.load_lora_weights(PRIMARY_LORA_MODEL)
1888
+ text_pipe.fuse_lora()
1889
  text_pipe = text_pipe.to("cuda")
1890
  if hasattr(text_pipe, "set_progress_bar_config"):
1891
  text_pipe.set_progress_bar_config(disable=True)
 
1958
  subject = (spec.group or spec.role).replace("_", " ")
1959
  if spec.camera == "top_down":
1960
  return compact_prompt_words(
1961
+ f"Strict orthographic top-down game sprite. Camera at 90 degrees directly above the {subject}, "
1962
+ "looking straight down. Show only top-facing surfaces and a compact foreshortened silhouette. "
1963
+ "No visible frontal face, chest, facade, or side elevation. "
1964
  f"{description}. Exactly one complete isolated subject, centered on uniform white. "
1965
  f"{variation}Crisp game-ready details. No scene.",
1966
  55,
 
2645
 
2646
  if USE_PRIMARY_IMAGE_MODEL:
2647
  png_content, error = primary_diffusion_png(spec, index, run_id)
2648
+ source = primary_model_source()
2649
  if not is_background:
2650
  source += f" + {FOREGROUND_MODEL_ID} neural alpha"
2651
  if error:
 
2891
  if "via fal-ai" in lowered or "via replicate" in lowered:
2892
  return "routed image-to-image refinement—preserves a layout guide but consumes account inference credit"
2893
  if "zerogpu" in lowered:
2894
+ if "lcm" not in lowered:
2895
+ return "full-step locally hosted diffusion—stronger prompt guidance and effective negative prompting; visual review required"
2896
  return "direct locally hosted text-to-image diffusion—no procedural guide or per-image API credit; visual review required"
2897
  return "diffusion output—higher visual variety; verify subject, perspective, alpha edges, and prompt compliance"
2898
 
2899
 
2900
+ def primary_model_source() -> str:
2901
+ model_chain = PRIMARY_IMAGE_MODEL
2902
+ if PRIMARY_LORA_MODEL:
2903
+ model_chain += f" + {PRIMARY_LORA_MODEL}"
2904
+ return f"{model_chain} on ZeroGPU"
2905
+
2906
+
2907
  def model_configuration_summary() -> str:
2908
  prompt_source = HF_PROMPT_MODEL if USE_HF_PROMPT_PROVIDER else "local prompt interpreter"
2909
  sprite_chain = []
2910
+ primary_source = primary_model_source()
2911
  if USE_PRIMARY_IMAGE_MODEL:
2912
  sprite_chain.append(f"direct text-to-image {primary_source} → neural alpha {FOREGROUND_MODEL_ID}")
2913
  if USE_DIFFUSION_FOR_SPRITES:
 
2939
  remote_fallback = " → ".join(remote_sources) if remote_sources else "disabled"
2940
  neural_models = []
2941
  if USE_PRIMARY_IMAGE_MODEL:
2942
+ neural_models.append(
2943
+ f"{PRIMARY_IMAGE_MODEL} + {PRIMARY_LORA_MODEL}" if PRIMARY_LORA_MODEL else PRIMARY_IMAGE_MODEL
2944
+ )
2945
  neural_models.append(FOREGROUND_MODEL_ID)
2946
  if USE_DIFFUSION_FOR_SPRITES or USE_DIFFUSION_FOR_BACKGROUNDS:
2947
  neural_models.append(FREE_IMAGE_MODEL)
 
3068
  output_count = max(1, len(resolve_role_lines(html_code or "", roles or "")[0]))
3069
  except AssetManifestError:
3070
  output_count = 1
3071
+ per_output = max(36, int(max(PRIMARY_IMAGE_STEPS, PRIMARY_SPRITE_STEPS) * 1.5))
3072
+ return min(300, 40 + output_count * per_output)
3073
 
3074
 
3075
  @gpu_task(duration=estimate_generation_gpu_duration)
 
3167
  count = sum(1 for spec in specs if (spec.group or spec.role) == group)
3168
  else:
3169
  count = 1
3170
+ per_output = max(36, int(max(PRIMARY_IMAGE_STEPS, PRIMARY_SPRITE_STEPS) * 1.5))
3171
+ return min(300, 35 + max(1, count) * per_output)
3172
 
3173
 
3174
  @gpu_task(duration=estimate_regeneration_gpu_duration)