makeup-extract: install + run scripts for Makeup Extraction of 3D Representation (EG2023) (single photo -> FLAME UV materials + extracted makeup layers)
Browse files- makeup-extract/README.md +102 -0
- makeup-extract/makeup_extract_install.sh +227 -0
- makeup-extract/makeup_extract_run.sh +127 -0
makeup-extract/README.md
ADDED
|
@@ -0,0 +1,102 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# makeup-extract — 3D makeup extraction from a single photo
|
| 2 |
+
|
| 3 |
+
Support scripts for [YangXingchao/makeup-extract](https://github.com/YangXingchao/makeup-extract)
|
| 4 |
+
(*Makeup Extraction of 3D Representation via Illumination-Aware Image
|
| 5 |
+
Decomposition*, Yang, Taketomi, Kanamori — Computer Graphics Forum / Eurographics 2023).
|
| 6 |
+
|
| 7 |
+
From one face photo the pipeline reconstructs a coarse FLAME face, unwraps and
|
| 8 |
+
completes the UV texture, decomposes it into materials (diffuse albedo, normal,
|
| 9 |
+
diffuse/specular shading) and extracts the **makeup as a separate layer**:
|
| 10 |
+
|
| 11 |
+
| output | meaning |
|
| 12 |
+
|---|---|
|
| 13 |
+
| `bare_skin.png` | UV albedo with makeup removed |
|
| 14 |
+
| `make_base.png` | makeup base color (UV) |
|
| 15 |
+
| `make_alpha.png` | makeup alpha/opacity matte (UV, white = no makeup) |
|
| 16 |
+
| `make_blend.png` | premultiplied makeup layer (`base * alpha`, UV) |
|
| 17 |
+
| `albe_refine.png`, `norm_refine.png`, `diff_refine.png`, `rnsr_refine.png`, `tex_refine.png`, `completion.png` | refined UV materials / completed texture |
|
| 18 |
+
| `img_compose.png`, `img_compose_with_bg.png` | face re-rendered from the decomposed materials (sanity check — should resemble the aligned input) |
|
| 19 |
+
|
| 20 |
+
## Install
|
| 21 |
+
|
| 22 |
+
```bash
|
| 23 |
+
./makeup_extract_install.sh
|
| 24 |
+
```
|
| 25 |
+
|
| 26 |
+
One-shot, idempotent, installs into the **native** python env (no venv; a pip
|
| 27 |
+
constraints file protects torch/numpy/etc. from being touched). Tested on
|
| 28 |
+
Ubuntu 24.04, python 3.12, torch 2.8.0+cu128, H100 (image `plx1029/comfyui-qwen`).
|
| 29 |
+
|
| 30 |
+
What it does:
|
| 31 |
+
|
| 32 |
+
- apt-installs GL/EGL dev headers (`libglvnd-dev`, mesa dev packages) as a
|
| 33 |
+
safety net for nvdiffrast's GL path. Note: current nvdiffrast (2026)
|
| 34 |
+
deprecates `RasterizeGLContext` and silently maps it to the CUDA
|
| 35 |
+
rasterizer, so the upstream `dr.RasterizeGLContext()` calls run headless
|
| 36 |
+
with no OpenGL involved at all.
|
| 37 |
+
- pip-installs `ninja`, `gdown`, `nvdiffrast` (from GitHub,
|
| 38 |
+
`--no-build-isolation` so it compiles against native torch), and a patched
|
| 39 |
+
`chumpy` (py3.12/numpy2 fixes; only needed to unpickle FLAME).
|
| 40 |
+
- clones the upstream repo to `$MAKEUP_EXTRACT_HOME/makeup-extract`
|
| 41 |
+
(default `/workspace/makeup-extract/makeup-extract`).
|
| 42 |
+
- downloads weights:
|
| 43 |
+
- pipeline checkpoints (`coarse_reconstruction.pkl`, `uv_completion.pkl`,
|
| 44 |
+
`makeup_extraction.pkl`) from the paper's official Google Drive folder;
|
| 45 |
+
- FLAME assets `generic_model.pkl` (FLAME2020), `FLAME_masks.pkl`,
|
| 46 |
+
`FLAME_texture.npz` — from the **official** site when
|
| 47 |
+
`FLAME_USERNAME`/`FLAME_PASSWORD` are set, otherwise from the public HF
|
| 48 |
+
mirror `camenduru/show`. Either way the
|
| 49 |
+
[FLAME license](https://flame.is.tue.mpg.de/modellicense.html)
|
| 50 |
+
(non-commercial scientific research) applies — register at
|
| 51 |
+
flame.is.tue.mpg.de;
|
| 52 |
+
- `albedoModel2020_FLAME_albedoPart.npz` from the official
|
| 53 |
+
[AlbedoMM](https://github.com/waps101/AlbedoMM) GitHub release (its own
|
| 54 |
+
non-commercial license applies).
|
| 55 |
+
|
| 56 |
+
## Run
|
| 57 |
+
|
| 58 |
+
```bash
|
| 59 |
+
./makeup_extract_run.sh <photo.jpg> [out_dir]
|
| 60 |
+
# e.g.
|
| 61 |
+
./makeup_extract_run.sh /workspace/face2.jpg
|
| 62 |
+
```
|
| 63 |
+
|
| 64 |
+
Runs all six upstream steps (preprocess → coarse reconstruction → UV
|
| 65 |
+
completion → material refinement → makeup extraction → re-render). Curated
|
| 66 |
+
outputs land in `/workspace/outputs/makeup-extract/<stem>/`; raw per-step
|
| 67 |
+
outputs are cached in `$MAKEUP_EXTRACT_HOME/results/<stem>/` (delete to
|
| 68 |
+
re-run). ~1–2 min per image on an H100 (step 3 is a 500-iteration
|
| 69 |
+
optimization; first run adds nvdiffrast JIT compile + VGG19/resnet18 aux
|
| 70 |
+
downloads, cached under `weights_cache/torch`).
|
| 71 |
+
|
| 72 |
+
## Deviations from upstream
|
| 73 |
+
|
| 74 |
+
- **No upstream file is modified.** All adaptation is done via environment and
|
| 75 |
+
wrapper:
|
| 76 |
+
- `TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1` at runtime (repo predates the
|
| 77 |
+
torch ≥2.6 `torch.load(weights_only=True)` default; checkpoints contain
|
| 78 |
+
non-tensor objects).
|
| 79 |
+
- `TORCH_HOME` redirected to `weights_cache/torch` so auto-downloaded VGG19 /
|
| 80 |
+
resnet18 weights survive pod restarts (`/root` is ephemeral).
|
| 81 |
+
- Non-square inputs are padded to square by the run script before step 0
|
| 82 |
+
(upstream's landmark scaling assumes square input).
|
| 83 |
+
- Dependency versions differ from upstream's `requirements.txt` pins
|
| 84 |
+
(numpy 1.23 → 2.1, kornia 0.6 → 0.8, opencv 4.7 → 5.0, scipy native): the
|
| 85 |
+
code is compatible as-is; no `np.int`-era aliases are used. `chumpy` is
|
| 86 |
+
patched for py3.12/numpy2 (`getargspec` → `getfullargspec`, removed-alias
|
| 87 |
+
import) as in the `deca/` implementation.
|
| 88 |
+
- FLAME assets fall back to a public HF mirror when no FLAME credentials are
|
| 89 |
+
provided (see Install above).
|
| 90 |
+
|
| 91 |
+
## Layout
|
| 92 |
+
|
| 93 |
+
```
|
| 94 |
+
/workspace/makeup-extract/
|
| 95 |
+
makeup_extract_install.sh this installer
|
| 96 |
+
makeup_extract_run.sh single-image runner
|
| 97 |
+
makeup-extract/ upstream clone + checkpoints/ + resources/
|
| 98 |
+
staging/<stem>/ squared input copy
|
| 99 |
+
results/<stem>/ raw per-step outputs (cache)
|
| 100 |
+
weights_cache/torch/ VGG19 / resnet18 hub cache
|
| 101 |
+
/workspace/outputs/makeup-extract/<stem>/ curated outputs
|
| 102 |
+
```
|
makeup-extract/makeup_extract_install.sh
ADDED
|
@@ -0,0 +1,227 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env bash
|
| 2 |
+
# =============================================================================
|
| 3 |
+
# makeup_extract_install.sh — one-shot installer for makeup-extract
|
| 4 |
+
# (Eurographics 2023: "Makeup Extraction of 3D Representation via
|
| 5 |
+
# Illumination-Aware Image Decomposition", Yang et al.)
|
| 6 |
+
# single face photo -> FLAME-based UV materials (albedo / normal / shading)
|
| 7 |
+
# + extracted makeup layers (bare skin, makeup base, alpha, blend)
|
| 8 |
+
# + re-rendered composite.
|
| 9 |
+
#
|
| 10 |
+
# Installs https://github.com/YangXingchao/makeup-extract into the NATIVE
|
| 11 |
+
# python environment (no venv, no conda, no torch reinstall). Tested on:
|
| 12 |
+
# Ubuntu 24.04, python 3.12, torch 2.8.0+cu128, H100
|
| 13 |
+
# (RunPod image plx1029/comfyui-qwen).
|
| 14 |
+
#
|
| 15 |
+
# Idempotent: safe to re-run; finished steps are skipped.
|
| 16 |
+
#
|
| 17 |
+
# Requirements:
|
| 18 |
+
# - torch with CUDA already installed (this script will not install torch)
|
| 19 |
+
# - nvcc + gcc (nvdiffrast CUDA/GL plugins JIT-compile on first use)
|
| 20 |
+
# - NVIDIA EGL driver libs (libEGL_nvidia.so.0) for the headless GL
|
| 21 |
+
# rasterizer context — present on this image
|
| 22 |
+
# - FLAME assets (see below): official downloads need FLAME account creds
|
| 23 |
+
# via env vars FLAME_USERNAME / FLAME_PASSWORD
|
| 24 |
+
# (https://flame.is.tue.mpg.de). Without creds the script falls back to
|
| 25 |
+
# a public Hugging Face mirror (camenduru/show). Either way you must
|
| 26 |
+
# comply with the FLAME license (non-commercial scientific research).
|
| 27 |
+
#
|
| 28 |
+
# Downloads:
|
| 29 |
+
# - pretrained pipeline checkpoints (~1 GB, official Google Drive folder)
|
| 30 |
+
# - FLAME2020 generic_model.pkl (~54 MB), FLAME_masks.pkl (~0.2 MB),
|
| 31 |
+
# FLAME_texture.npz (~1.3 GB)
|
| 32 |
+
# - AlbedoMM albedoModel2020_FLAME_albedoPart.npz (~1.8 GB, official
|
| 33 |
+
# GitHub release, CC BY 4.0 non-commercial per AlbedoMM README)
|
| 34 |
+
#
|
| 35 |
+
# Usage:
|
| 36 |
+
# ./makeup_extract_install.sh
|
| 37 |
+
#
|
| 38 |
+
# Layout (override with MAKEUP_EXTRACT_HOME):
|
| 39 |
+
# $MAKEUP_EXTRACT_HOME/makeup-extract upstream clone + weights
|
| 40 |
+
# $MAKEUP_EXTRACT_HOME/staging per-image input staging
|
| 41 |
+
# $MAKEUP_EXTRACT_HOME/results per-image raw outputs
|
| 42 |
+
# $MAKEUP_EXTRACT_HOME/weights_cache torch hub cache (VGG19, resnet18)
|
| 43 |
+
# =============================================================================
|
| 44 |
+
set -euo pipefail
|
| 45 |
+
|
| 46 |
+
if [ "${1:-}" = "--help" ] || [ "${1:-}" = "-h" ]; then grep '^#' "$0" | sed 's/^# \{0,1\}//'; exit 0; fi
|
| 47 |
+
|
| 48 |
+
M_HOME="${MAKEUP_EXTRACT_HOME:-/workspace/makeup-extract}"
|
| 49 |
+
REPO="$M_HOME/makeup-extract"
|
| 50 |
+
|
| 51 |
+
log() { echo -e "\n\033[1;36m[makeup-extract-install]\033[0m $*"; }
|
| 52 |
+
|
| 53 |
+
mkdir -p "$M_HOME" "$M_HOME/staging" "$M_HOME/results" "$M_HOME/weights_cache"
|
| 54 |
+
cd "$M_HOME"
|
| 55 |
+
|
| 56 |
+
# ---------------------------------------------------------------- sanity checks
|
| 57 |
+
log "checking python/torch/nvcc"
|
| 58 |
+
python3 - <<'EOF'
|
| 59 |
+
import torch
|
| 60 |
+
assert torch.cuda.is_available(), "CUDA torch required"
|
| 61 |
+
print(f" torch {torch.__version__} cuda {torch.version.cuda} ok")
|
| 62 |
+
EOF
|
| 63 |
+
command -v nvcc >/dev/null || echo " WARNING: nvcc not found - nvdiffrast JIT compile will fail at runtime"
|
| 64 |
+
|
| 65 |
+
# ---------------------------------------------------- constraints (protect env)
|
| 66 |
+
# Pin currently-installed shared heavyweights so pip cannot up/downgrade them
|
| 67 |
+
# while installing deps (keeps the ComfyUI env safe).
|
| 68 |
+
log "generating pip constraints from current env"
|
| 69 |
+
python3 - > "$M_HOME/constraints.txt" <<'EOF'
|
| 70 |
+
from importlib.metadata import version, PackageNotFoundError
|
| 71 |
+
for p in ["numpy","torch","torchvision","pillow","scipy","kornia",
|
| 72 |
+
"opencv-python","opencv-python-headless","requests","einops",
|
| 73 |
+
"safetensors","huggingface-hub"]:
|
| 74 |
+
try: print(f"{p}=={version(p)}")
|
| 75 |
+
except PackageNotFoundError: pass
|
| 76 |
+
EOF
|
| 77 |
+
CONS="-c $M_HOME/constraints.txt"
|
| 78 |
+
|
| 79 |
+
# ------------------------------------------------- system GL/EGL dev packages
|
| 80 |
+
# nvdiffrast's GL rasterizer plugin JIT-compiles against EGL/GLES headers.
|
| 81 |
+
# The NVIDIA EGL driver (libEGL_nvidia.so.0) is already on the image; we only
|
| 82 |
+
# add the glvnd dispatcher + headers.
|
| 83 |
+
if ! dpkg -s libglvnd-dev >/dev/null 2>&1; then
|
| 84 |
+
log "installing GL/EGL dev packages (apt)"
|
| 85 |
+
apt-get update -qq
|
| 86 |
+
DEBIAN_FRONTEND=noninteractive apt-get install -y -qq \
|
| 87 |
+
pkg-config libglvnd0 libgl1 libglx0 libegl1 libgles2 \
|
| 88 |
+
libglvnd-dev libgl1-mesa-dev libegl1-mesa-dev libgles2-mesa-dev
|
| 89 |
+
else
|
| 90 |
+
log "GL/EGL dev packages already installed"
|
| 91 |
+
fi
|
| 92 |
+
|
| 93 |
+
# ------------------------------------------------------------------ pip deps
|
| 94 |
+
log "installing python dependencies (native env, constrained)"
|
| 95 |
+
pip install -q $CONS ninja gdown
|
| 96 |
+
|
| 97 |
+
if ! python3 -c "import nvdiffrast" 2>/dev/null; then
|
| 98 |
+
log "installing nvdiffrast from GitHub (compiles CUDA extension against native torch)"
|
| 99 |
+
TORCH_CUDA_ARCH_LIST="$(python3 -c 'import torch; print("%d.%d" % torch.cuda.get_device_capability())')" \
|
| 100 |
+
pip install -q $CONS --no-build-isolation "git+https://github.com/NVlabs/nvdiffrast.git"
|
| 101 |
+
fi
|
| 102 |
+
|
| 103 |
+
# chumpy: only needed to unpickle FLAME. Needs patching for py>=3.11 / numpy>=2.
|
| 104 |
+
if ! python3 -c "import chumpy" 2>/dev/null; then
|
| 105 |
+
log "installing + patching chumpy for py3.12/numpy2"
|
| 106 |
+
pip install -q --no-deps --no-build-isolation chumpy
|
| 107 |
+
CHDIR=$(python3 -c "import importlib.util,os; print(os.path.dirname(importlib.util.find_spec('chumpy').origin))")
|
| 108 |
+
grep -rl "inspect.getargspec" "$CHDIR"/*.py | xargs -r sed -i "s/inspect\.getargspec/inspect.getfullargspec/g"
|
| 109 |
+
sed -i "s/^from numpy import bool, int, float, complex, object, unicode, str, nan, inf$/from numpy import nan, inf/" "$CHDIR/__init__.py"
|
| 110 |
+
python3 -c "import chumpy" || { echo "chumpy patch failed"; exit 1; }
|
| 111 |
+
fi
|
| 112 |
+
|
| 113 |
+
# ------------------------------------------------------------------ repo
|
| 114 |
+
if [ ! -d "$REPO" ]; then
|
| 115 |
+
log "cloning makeup-extract"
|
| 116 |
+
git clone --depth 1 https://github.com/YangXingchao/makeup-extract.git "$REPO"
|
| 117 |
+
fi
|
| 118 |
+
|
| 119 |
+
# -------------------------------------------------------- pipeline checkpoints
|
| 120 |
+
# Official Google Drive folder from the README.
|
| 121 |
+
CKPTS="coarse_reconstruction.pkl uv_completion.pkl makeup_extraction.pkl"
|
| 122 |
+
need_ckpt=0
|
| 123 |
+
for c in $CKPTS; do [ -s "$REPO/checkpoints/$c" ] || need_ckpt=1; done
|
| 124 |
+
if [ "$need_ckpt" = 1 ]; then
|
| 125 |
+
log "downloading pretrained checkpoints (official Google Drive)"
|
| 126 |
+
gdown --folder 1lwkR9JcrbZ7fNylTSJQQEiGnt3s2LQYq -O "$REPO/checkpoints" -q
|
| 127 |
+
fi
|
| 128 |
+
for c in $CKPTS; do
|
| 129 |
+
[ -s "$REPO/checkpoints/$c" ] || { echo "checkpoint $c missing after download"; exit 1; }
|
| 130 |
+
done
|
| 131 |
+
|
| 132 |
+
# ------------------------------------------------------------------ FLAME assets
|
| 133 |
+
RES="$REPO/resources"
|
| 134 |
+
|
| 135 |
+
flame_official() { # $1 = sfile, $2 = out zip
|
| 136 |
+
urle () { local LANG=C i x; for (( i = 0; i < ${#1}; i++ )); do x="${1:i:1}"; [[ "${x}" == [a-zA-Z0-9.~-] ]] && echo -n "${x}" || printf '%%%02X' "'${x}"; done; }
|
| 137 |
+
local U P; U=$(urle "$FLAME_USERNAME"); P=$(urle "$FLAME_PASSWORD")
|
| 138 |
+
wget -q --post-data "username=$U&password=$P" \
|
| 139 |
+
"https://download.is.tue.mpg.de/download.php?domain=flame&sfile=$1&resume=1" \
|
| 140 |
+
-O "$2" --no-check-certificate --continue
|
| 141 |
+
}
|
| 142 |
+
|
| 143 |
+
hf_mirror() { # $1 = path in camenduru/show, $2 = dest file
|
| 144 |
+
python3 - "$1" "$2" <<'EOF'
|
| 145 |
+
import shutil, sys
|
| 146 |
+
from huggingface_hub import hf_hub_download
|
| 147 |
+
src = hf_hub_download("camenduru/show", sys.argv[1])
|
| 148 |
+
shutil.copyfile(src, sys.argv[2])
|
| 149 |
+
EOF
|
| 150 |
+
}
|
| 151 |
+
|
| 152 |
+
HAVE_CREDS=0
|
| 153 |
+
[ -n "${FLAME_USERNAME:-}" ] && [ -n "${FLAME_PASSWORD:-}" ] && HAVE_CREDS=1
|
| 154 |
+
|
| 155 |
+
if [ ! -s "$RES/generic_model.pkl" ]; then
|
| 156 |
+
P3DM_FLAME="/workspace/p3dm/pixel3dmm/src/pixel3dmm/preprocessing/MICA/data/FLAME2020/generic_model.pkl"
|
| 157 |
+
if [ -f "$P3DM_FLAME" ]; then
|
| 158 |
+
log "reusing FLAME 2020 from p3dm install"
|
| 159 |
+
cp "$P3DM_FLAME" "$RES/generic_model.pkl"
|
| 160 |
+
elif [ "$HAVE_CREDS" = 1 ]; then
|
| 161 |
+
log "downloading FLAME 2020 (official, needs FLAME account)"
|
| 162 |
+
flame_official "FLAME2020.zip" "$RES/FLAME2020.zip"
|
| 163 |
+
unzip -o -q "$RES/FLAME2020.zip" -d "$RES/FLAME2020_tmp"
|
| 164 |
+
mv "$RES/FLAME2020_tmp/generic_model.pkl" "$RES/generic_model.pkl"
|
| 165 |
+
rm -rf "$RES/FLAME2020.zip" "$RES/FLAME2020_tmp"
|
| 166 |
+
else
|
| 167 |
+
log "downloading FLAME 2020 generic_model.pkl (HF mirror camenduru/show; set FLAME_USERNAME/FLAME_PASSWORD for official source)"
|
| 168 |
+
hf_mirror "models/models_MICA/FLAME2020/generic_model.pkl" "$RES/generic_model.pkl"
|
| 169 |
+
fi
|
| 170 |
+
fi
|
| 171 |
+
|
| 172 |
+
if [ ! -s "$RES/FLAME_masks.pkl" ]; then
|
| 173 |
+
if [ "$HAVE_CREDS" = 1 ]; then
|
| 174 |
+
log "downloading FLAME vertex masks (official)"
|
| 175 |
+
flame_official "FLAME_masks.zip" "$RES/FLAME_masks.zip"
|
| 176 |
+
unzip -o -q "$RES/FLAME_masks.zip" -d "$RES/FLAME_masks_tmp"
|
| 177 |
+
mv "$RES/FLAME_masks_tmp/FLAME_masks.pkl" "$RES/FLAME_masks.pkl"
|
| 178 |
+
rm -rf "$RES/FLAME_masks.zip" "$RES/FLAME_masks_tmp"
|
| 179 |
+
else
|
| 180 |
+
log "downloading FLAME_masks.pkl (HF mirror)"
|
| 181 |
+
hf_mirror "data/FLAME_masks.pkl" "$RES/FLAME_masks.pkl"
|
| 182 |
+
fi
|
| 183 |
+
fi
|
| 184 |
+
|
| 185 |
+
if [ ! -s "$RES/FLAME_texture.npz" ]; then
|
| 186 |
+
if [ "$HAVE_CREDS" = 1 ]; then
|
| 187 |
+
log "downloading FLAME texture space (official, ~1.3 GB)"
|
| 188 |
+
flame_official "TextureSpace.zip" "$RES/TextureSpace.zip"
|
| 189 |
+
unzip -o -q "$RES/TextureSpace.zip" -d "$RES/TextureSpace_tmp"
|
| 190 |
+
mv "$RES/TextureSpace_tmp/FLAME_texture.npz" "$RES/FLAME_texture.npz"
|
| 191 |
+
rm -rf "$RES/TextureSpace.zip" "$RES/TextureSpace_tmp"
|
| 192 |
+
else
|
| 193 |
+
log "downloading FLAME_texture.npz (HF mirror, ~1.3 GB)"
|
| 194 |
+
hf_mirror "data/flame/FLAME_texture.npz" "$RES/FLAME_texture.npz"
|
| 195 |
+
fi
|
| 196 |
+
fi
|
| 197 |
+
|
| 198 |
+
if [ ! -s "$RES/albedoModel2020_FLAME_albedoPart.npz" ]; then
|
| 199 |
+
log "downloading AlbedoMM albedo model (official GitHub release, ~1.8 GB)"
|
| 200 |
+
wget -q --show-progress --continue \
|
| 201 |
+
"https://github.com/waps101/AlbedoMM/releases/download/v1.0/albedoModel2020_FLAME_albedoPart.npz" \
|
| 202 |
+
-O "$RES/albedoModel2020_FLAME_albedoPart.npz"
|
| 203 |
+
fi
|
| 204 |
+
|
| 205 |
+
# ------------------------------------------------------------------ verify
|
| 206 |
+
log "verifying imports + files"
|
| 207 |
+
REPO_DIR="$REPO" python3 - <<'EOF'
|
| 208 |
+
import os, sys, warnings
|
| 209 |
+
warnings.filterwarnings("ignore")
|
| 210 |
+
import chumpy, gdown, kornia, cv2, nvdiffrast
|
| 211 |
+
repo = os.environ["REPO_DIR"]
|
| 212 |
+
sys.path.insert(0, repo)
|
| 213 |
+
import networks # pulls torchvision resnet50 + icosahedron
|
| 214 |
+
for f in ["checkpoints/coarse_reconstruction.pkl", "checkpoints/uv_completion.pkl",
|
| 215 |
+
"checkpoints/makeup_extraction.pkl",
|
| 216 |
+
"resources/generic_model.pkl", "resources/FLAME_masks.pkl",
|
| 217 |
+
"resources/FLAME_texture.npz",
|
| 218 |
+
"resources/albedoModel2020_FLAME_albedoPart.npz",
|
| 219 |
+
"resources/landmark_embedding.npy", "resources/skin_mask.png",
|
| 220 |
+
"libs/face_parsing/module_weight/79999_iter.pth",
|
| 221 |
+
"libs/face_landmark/checkpoint/mobilenet_224_model_best_gdconv_external.pth.tar"]:
|
| 222 |
+
p = os.path.join(repo, f)
|
| 223 |
+
assert os.path.exists(p) and os.path.getsize(p) > 0, p
|
| 224 |
+
print(" all good")
|
| 225 |
+
EOF
|
| 226 |
+
|
| 227 |
+
log "install complete. run: ./makeup_extract_run.sh <photo.jpg>"
|
makeup-extract/makeup_extract_run.sh
ADDED
|
@@ -0,0 +1,127 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env bash
|
| 2 |
+
# =============================================================================
|
| 3 |
+
# makeup_extract_run.sh — single photo -> 3D makeup decomposition
|
| 4 |
+
#
|
| 5 |
+
# Runs the full 6-step makeup-extract pipeline (Yang et al., EG 2023) on one
|
| 6 |
+
# face image:
|
| 7 |
+
# step 0 landmark crop + face parsing -> aligned_img / mask
|
| 8 |
+
# step 1 coarse FLAME material reconstruction -> UV albedo/normal/shading
|
| 9 |
+
# step 2 UV texture completion (DSD-GAN)
|
| 10 |
+
# step 3 material refinement (500-iter optimization, VGG/TV losses)
|
| 11 |
+
# step 4 makeup extraction (bare skin / base / alpha / blend)
|
| 12 |
+
# step 5 re-render composite with extracted textures
|
| 13 |
+
#
|
| 14 |
+
# Usage:
|
| 15 |
+
# ./makeup_extract_run.sh <photo.jpg> [out_dir]
|
| 16 |
+
#
|
| 17 |
+
# out_dir default: /workspace/outputs/makeup-extract/<image-stem>/
|
| 18 |
+
#
|
| 19 |
+
# Notes:
|
| 20 |
+
# - non-square inputs are padded to square first (upstream assumes square)
|
| 21 |
+
# - raw per-step outputs are cached in $MAKEUP_EXTRACT_HOME/results/<stem>/;
|
| 22 |
+
# delete that folder to force a re-run
|
| 23 |
+
# =============================================================================
|
| 24 |
+
set -euo pipefail
|
| 25 |
+
|
| 26 |
+
if [ $# -lt 1 ] || [ "${1:-}" = "--help" ] || [ "${1:-}" = "-h" ]; then
|
| 27 |
+
grep '^#' "$0" | sed 's/^# \{0,1\}//'; exit 0
|
| 28 |
+
fi
|
| 29 |
+
|
| 30 |
+
M_HOME="${MAKEUP_EXTRACT_HOME:-/workspace/makeup-extract}"
|
| 31 |
+
REPO="$M_HOME/makeup-extract"
|
| 32 |
+
|
| 33 |
+
log() { echo -e "\n\033[1;36m[makeup-extract]\033[0m $*"; }
|
| 34 |
+
|
| 35 |
+
IMG="$(realpath "$1")"
|
| 36 |
+
[ -f "$IMG" ] || { echo "input image not found: $IMG"; exit 1; }
|
| 37 |
+
|
| 38 |
+
for f in checkpoints/coarse_reconstruction.pkl resources/generic_model.pkl \
|
| 39 |
+
resources/FLAME_texture.npz resources/albedoModel2020_FLAME_albedoPart.npz; do
|
| 40 |
+
[ -s "$REPO/$f" ] || { echo "missing $REPO/$f - run ./makeup_extract_install.sh first"; exit 1; }
|
| 41 |
+
done
|
| 42 |
+
|
| 43 |
+
STEM=$(basename "$IMG" | sed 's/\.[^.]*$//' | tr -c 'a-zA-Z0-9' '_' | sed 's/_*$//;s/^_*//' | cut -c1-60)
|
| 44 |
+
STAGE="$M_HOME/staging/$STEM"
|
| 45 |
+
RESULTS="$M_HOME/results/$STEM"
|
| 46 |
+
OUT_DIR="${2:-/workspace/outputs/makeup-extract/$STEM}"
|
| 47 |
+
mkdir -p "$STAGE" "$RESULTS" "$OUT_DIR"
|
| 48 |
+
|
| 49 |
+
export TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1 # repo predates torch 2.6 torch.load change
|
| 50 |
+
export TORCH_HOME="$M_HOME/weights_cache/torch" # VGG19/resnet18 aux weights (keep off ephemeral /root)
|
| 51 |
+
|
| 52 |
+
START=$(date +%s)
|
| 53 |
+
|
| 54 |
+
# ---------------------------------------------------------------- stage input
|
| 55 |
+
# Upstream step_0 assumes a square image; pad to square (centered, black).
|
| 56 |
+
log "staging input (pad to square if needed)"
|
| 57 |
+
IMG_PATH="$IMG" STAGED="$STAGE/input.png" python3 - <<'EOF'
|
| 58 |
+
import os, cv2
|
| 59 |
+
img = cv2.imread(os.environ["IMG_PATH"])
|
| 60 |
+
h, w = img.shape[:2]
|
| 61 |
+
if h != w:
|
| 62 |
+
s = max(h, w)
|
| 63 |
+
top = (s - h) // 2; left = (s - w) // 2
|
| 64 |
+
img = cv2.copyMakeBorder(img, top, s - h - top, left, s - w - left,
|
| 65 |
+
cv2.BORDER_CONSTANT, value=(0, 0, 0))
|
| 66 |
+
print(f" padded {w}x{h} -> {s}x{s}")
|
| 67 |
+
cv2.imwrite(os.environ["STAGED"], img)
|
| 68 |
+
EOF
|
| 69 |
+
|
| 70 |
+
cd "$REPO"
|
| 71 |
+
|
| 72 |
+
# ---------------------------------------------------------------- pipeline
|
| 73 |
+
if [ -s "$RESULTS/render_texture/img_compose_with_bg.png" ]; then
|
| 74 |
+
log "cached results found in $RESULTS (delete to re-run)"
|
| 75 |
+
else
|
| 76 |
+
log "step 0/5: landmark crop + face parsing"
|
| 77 |
+
python3 step_0_preprocess.py -i "$STAGE/input.png" -o "$RESULTS"
|
| 78 |
+
|
| 79 |
+
log "step 1/5: coarse facial material reconstruction"
|
| 80 |
+
python3 step_1_coarse_reconstruction.py \
|
| 81 |
+
--aligned_img_path "$RESULTS/aligned_img.png" \
|
| 82 |
+
--segmented_img_path "$RESULTS/segmented_img.png" \
|
| 83 |
+
-o "$RESULTS/coarse_reconstruction"
|
| 84 |
+
|
| 85 |
+
log "step 2/5: UV completion"
|
| 86 |
+
python3 step_2_uv_completion.py \
|
| 87 |
+
--uv_mm_path "$RESULTS/coarse_reconstruction/uv_rncr.png" \
|
| 88 |
+
--uv_tex_path "$RESULTS/coarse_reconstruction/uv_tex.png" \
|
| 89 |
+
-o "$RESULTS/material_refinement"
|
| 90 |
+
|
| 91 |
+
log "step 3/5: material refinement (500-iter optimization)"
|
| 92 |
+
python3 step_3_material_refinement.py \
|
| 93 |
+
--mm_param_path "$RESULTS/coarse_reconstruction/mm_param.npz" \
|
| 94 |
+
--uv_albe_path "$RESULTS/coarse_reconstruction/uv_albe.png" \
|
| 95 |
+
--uv_norm_path "$RESULTS/coarse_reconstruction/uv_norm.png" \
|
| 96 |
+
--uv_rnsr_path "$RESULTS/coarse_reconstruction/uv_rnsr.png" \
|
| 97 |
+
--completion_tex_path "$RESULTS/material_refinement/completion.png" \
|
| 98 |
+
-o "$RESULTS/material_refinement"
|
| 99 |
+
|
| 100 |
+
log "step 4/5: makeup extraction"
|
| 101 |
+
python3 step_4_makeup_extraction.py \
|
| 102 |
+
--uv_albe_path "$RESULTS/material_refinement/albe_refine.png" \
|
| 103 |
+
-o "$RESULTS/makeup_extraction"
|
| 104 |
+
|
| 105 |
+
log "step 5/5: render composite"
|
| 106 |
+
python3 step_5_render_texture.py \
|
| 107 |
+
--mm_param_path "$RESULTS/coarse_reconstruction/mm_param.npz" \
|
| 108 |
+
--uv_bare_skin_path "$RESULTS/makeup_extraction/bare_skin.png" \
|
| 109 |
+
--uv_make_alpha_path "$RESULTS/makeup_extraction/make_alpha.png" \
|
| 110 |
+
--uv_make_blend_path "$RESULTS/makeup_extraction/make_blend.png" \
|
| 111 |
+
--uv_diffuse_shading_path "$RESULTS/material_refinement/diff_refine.png" \
|
| 112 |
+
--uv_specular_shading_path "$RESULTS/material_refinement/rnsr_refine.png" \
|
| 113 |
+
--input_aligned_image "$RESULTS/aligned_img.png" \
|
| 114 |
+
--aligned_image_mask_path "$RESULTS/mask_img.png" \
|
| 115 |
+
-o "$RESULTS/render_texture"
|
| 116 |
+
fi
|
| 117 |
+
|
| 118 |
+
# ---------------------------------------------------------------- outputs
|
| 119 |
+
log "collecting outputs"
|
| 120 |
+
cp "$RESULTS/aligned_img.png" "$OUT_DIR/"
|
| 121 |
+
cp "$RESULTS/makeup_extraction/"{bare_skin,make_base,make_alpha,make_blend}.png "$OUT_DIR/"
|
| 122 |
+
cp "$RESULTS/material_refinement/"{albe_refine,norm_refine,diff_refine,rnsr_refine,tex_refine,completion}.png "$OUT_DIR/"
|
| 123 |
+
cp "$RESULTS/render_texture/"{img_compose,img_compose_with_bg}.png "$OUT_DIR/"
|
| 124 |
+
|
| 125 |
+
ELAPSED=$(( $(date +%s) - START ))
|
| 126 |
+
log "done in ${ELAPSED}s. outputs in $OUT_DIR"
|
| 127 |
+
ls -la "$OUT_DIR"
|