resakemal commited on
Commit
a700bb2
Β·
1 Parent(s): a64427c

Update field notes

Browse files
Files changed (1) hide show
  1. field_notes.md +7 -0
field_notes.md CHANGED
@@ -250,6 +250,13 @@ Both Modal services (`modal_llm.py`, `modal_painter.py`) are deployed once with
250
 
251
  **Why four LLM backends?** This was the hardest part of shipping. `gr.Server` (for the custom frontend) needs gradio 6.x, which forces `huggingface-hub β‰₯ 1.2`, which forces `transformers β‰₯ 5.0`. But MiniCPM4.1-8B loads via `trust_remote_code`, and its remote code is incompatible with transformers 5.x: loading needs a shim, and generation then still crashes deep in attention. The two clean escapes are to run it on **Modal** with transformers pinned to 4.x, or β€” better β€” to run it through **llama.cpp** as a quantized GGUF, which uses no `transformers` at all and is small enough (~5 GB) to sit beside FLUX on a single 24 GB GPU. The llama.cpp route also earns the Llama Champion badge.
252
 
 
 
 
 
 
 
 
253
  **Engine (`engine/`):**
254
  - `mappings.py` β€” affect β†’ geometry and affect β†’ color (OKLCH)
255
  - `renderer.py` β€” geometry β†’ polygon boundary (triangle-wave spikiness, segmented smooth interpolation)
 
250
 
251
  **Why four LLM backends?** This was the hardest part of shipping. `gr.Server` (for the custom frontend) needs gradio 6.x, which forces `huggingface-hub β‰₯ 1.2`, which forces `transformers β‰₯ 5.0`. But MiniCPM4.1-8B loads via `trust_remote_code`, and its remote code is incompatible with transformers 5.x: loading needs a shim, and generation then still crashes deep in attention. The two clean escapes are to run it on **Modal** with transformers pinned to 4.x, or β€” better β€” to run it through **llama.cpp** as a quantized GGUF, which uses no `transformers` at all and is small enough (~5 GB) to sit beside FLUX on a single 24 GB GPU. The llama.cpp route also earns the Llama Champion badge.
252
 
253
+ **The deployment journey.** The four backends weren't designed up front β€” they're the fossil record of getting this to run on the hardware I actually had (a laptop with **16 GB RAM / 8 GB VRAM**, nowhere near enough for an 8B LLM *and* a 4B diffusion model at once):
254
+
255
+ 1. **Qwen3-8B on Ollama (local) + FLUX.2 Klein on Modal.** The first working setup. The LLM ran locally through Ollama (which is itself llama.cpp under the hood); the painter β€” far too heavy for 8 GB β€” was pushed to a Modal GPU endpoint. This split kept the loop fast to iterate on while the expensive image step lived in the cloud.
256
+ 2. **MiniCPM4.1-8B via llama.cpp + FLUX.2 Klein on Modal.** To target the OpenBMB prize I swapped the judge to MiniCPM4.1-8B, and moved to running it directly on **llama.cpp** (a quantized GGUF via `llama-cpp-python`) rather than Ollama β€” which also claims the Llama Champion badge. The painter is still on Modal due to previous RAM constraints.
257
+ 3. **Both on a single ZeroGPU Space β€” blocked.** The goal was to host the whole thing on one free Hugging Face **ZeroGPU** Space (llama.cpp LLM + FLUX painter together). In practice this proved very difficult: ZeroGPU attributes GPU usage via an `X-IP-Token` header that the standard Gradio UI forwards automatically, but a **custom `gr.Server` frontend doesn't**, so every GPU call is scheduled *anonymously* and immediately hits a tiny per-IP quota β€” a constant stream of **429 Too Many Requests**, while the account's real quota sits untouched. (On a campus network the anonymous quota is also shared across everyone behind the same NAT.) The solutions are a **dedicated** GPU Space β€” which isn't ZeroGPU and has no such attribution problem β€” or keeping the heavy work on Modal. Cracking ZeroGPU for a custom-frontend app remains open.
258
+ *Note: The ZeroGPU Too Many Requests assumption may be mistaken and I apologize if that is the case.*
259
+
260
  **Engine (`engine/`):**
261
  - `mappings.py` β€” affect β†’ geometry and affect β†’ color (OKLCH)
262
  - `renderer.py` β€” geometry β†’ polygon boundary (triangle-wave spikiness, segmented smooth interpolation)