litert-community/3DDFA-V2-LiteRT

3DDFA_V2 — LiteRT on-device 3D face alignment

3DDFA_V2 (Guo et al., ECCV 2020, MIT) re-authored for LiteRT: fit a 3D morphable face model to a photo. A MobileNetV1 regresses 62 3DMM parameters (pose + 40 shape + 10 expression) on the CompiledModel GPU; the 68 3D face landmarks (and a dense mesh) are reconstructed from the BFM bases host-side.

Verified on a Pixel 8a: fp16 tflite-vs-PyTorch 62-param corr 0.999999, reconstructed landmarks match to 0.02 px; 68 landmarks in well under a second.

Files

file	purpose
tddfa_mb1_fp16.tflite	MobileNetV1 3DMM regressor, crop [1,3,120,120] (BGR, (x−127.5)/128) → 62 params — GPU
tddfa_u_base.bin tddfa_w_shp_base.bin tddfa_w_exp_base.bin	BFM 68-keypoint bases (interleaved x,y,z)
tddfa_param_mean.bin tddfa_param_std.bin	62-param de-normalization

face box → parse_roi → crop 120² (BGR) → [GPU MobileNetV1] → 62 params
  → denorm (·std+mean) → R,offset,α_shp,α_exp
  → verts = u_base + w_shp_base·α_shp + w_exp_base·α_exp   (interleaved [x0,y0,z0,...])
  → R·verts + offset → similar_transform(roi, 120) → 68 landmarks (x,y,z)

Minimal usage (Python)

import numpy as np
from ai_edge_litert.interpreter import Interpreter

# crop the face ROI to 120x120, BGR, NCHW, (x-127.5)/128  -> inp [1,3,120,120]
it = Interpreter(model_path="tddfa_mb1_fp16.tflite"); it.allocate_tensors()
it.set_tensor(it.get_input_details()[0]["index"], inp); it.invoke()
out = it.get_tensor(it.get_output_details()[0]["index"])[0]           # 62 (normalized)

p = out * np.fromfile("tddfa_param_std.bin", np.float32) + np.fromfile("tddfa_param_mean.bin", np.float32)
R = p[:12].reshape(3, 4)[:, :3]; offset = p[:12].reshape(3, 4)[:, 3:4]
u = np.fromfile("tddfa_u_base.bin", np.float32).reshape(-1, 1)
ws = np.fromfile("tddfa_w_shp_base.bin", np.float32).reshape(204, 40)
we = np.fromfile("tddfa_w_exp_base.bin", np.float32).reshape(204, 10)
verts = (u + ws @ p[12:52, None] + we @ p[52:, None]).reshape(3, -1, order="F")
pts = R @ verts + offset                                              # [3,68], map to image via similar_transform

A complete Android sample (face → 68 landmarks) is in google-ai-edge/litert-samples.

Upstream

cleardusk/3DDFA_V2 (MIT). Paper: Towards Fast, Accurate and Stable 3D Dense Face Alignment (ECCV 2020).