Update README.md
Browse filesCC BY-NC-ND 4.0 — Documentation only. Patent Pending.
Copyright (c) 2026 Dmitry Rodin, Nikita Rodin. All rights reserved.
README.md
CHANGED
|
@@ -1,3 +1,90 @@
|
|
| 1 |
-
---
|
| 2 |
-
license: cc-by-nc-nd-4.0
|
| 3 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: cc-by-nc-nd-4.0
|
| 3 |
+
tags:
|
| 4 |
+
- face-animation
|
| 5 |
+
- mobile
|
| 6 |
+
- real-time
|
| 7 |
+
- avatar
|
| 8 |
+
- computer-vision
|
| 9 |
+
- neural-rendering
|
| 10 |
+
- knowledge-distillation
|
| 11 |
+
pipeline_tag: image-to-video
|
| 12 |
+
---
|
| 13 |
+
|
| 14 |
+
# LiveFace
|
| 15 |
+
|
| 16 |
+
**Real-Time Photorealistic Facial Animation on Low-End Mobile Devices**
|
| 17 |
+
|
| 18 |
+
*Patent Pending (USPTO) | [Paper (Zenodo)](https://doi.org/10.5281/zenodo.19477081) | [Website](https://creatora.app)*
|
| 19 |
+
|
| 20 |
+
## What is LiveFace?
|
| 21 |
+
|
| 22 |
+
LiveFace is a patent-pending neural rendering system that turns **a single photo into a photorealistic talking avatar** running at 30 fps on budget mobile devices — fully offline, no cloud required.
|
| 23 |
+
|
| 24 |
+
## Architecture
|
| 25 |
+
|
| 26 |
+
Four compact per-avatar neural decoders + one shared compositor-upscaler:
|
| 27 |
+
|
| 28 |
+
| Module | Parameters | Output | Function |
|
| 29 |
+
|--------|-----------|--------|----------|
|
| 30 |
+
| MouthDecoder | 5-12M | 128x96 RGBA | Lip sync, jaw, emotions |
|
| 31 |
+
| EyeDecoder | 1.3-2M | 192x80 RGBA | Blink, gaze, wink |
|
| 32 |
+
| HairDecoder | 3-5M | 192x192 RGBA | Hair physics, inertia |
|
| 33 |
+
| BodyDecoder | 3-12M | 256x64 RGBA | Breathing, shoulders |
|
| 34 |
+
| Compositor-Upscaler | ~7M (shared) | 360x640 (9:16) | Seam blending, upscale, lighting |
|
| 35 |
+
|
| 36 |
+
**Total: ~20M INT8 parameters | ~19ms per frame on Snapdragon 439**
|
| 37 |
+
|
| 38 |
+
## Key Features
|
| 39 |
+
|
| 40 |
+
- **Photorealistic** — neural rendering, not cartoon or stylized
|
| 41 |
+
- **Real-time** — 30+ fps on budget phones ($100+)
|
| 42 |
+
- **Offline** — fully on-device, no cloud, no internet
|
| 43 |
+
- **One photo** — create avatar from a single selfie
|
| 44 |
+
- **Identity embedding** — 128-dim learnable per-avatar parameter
|
| 45 |
+
- **Dual input** — viseme-based (audio) or landmark-based (MediaPipe)
|
| 46 |
+
- **Portrait 9:16** — optimized for mobile displays
|
| 47 |
+
|
| 48 |
+
## Training
|
| 49 |
+
|
| 50 |
+
Per-avatar decoders are trained via **knowledge distillation**:
|
| 51 |
+
1. Server-side teacher model generates diverse training data from RAVDESS emotional speech videos
|
| 52 |
+
2. Per-frame quality filter (Haar + blur + SSIM) ensures data integrity (~0.6% rejection)
|
| 53 |
+
3. Student decoders learn from teacher-generated pairs with L1 + perceptual loss
|
| 54 |
+
4. Each avatar trains in ~40 minutes on a single GPU
|
| 55 |
+
|
| 56 |
+
## Performance
|
| 57 |
+
|
| 58 |
+
| Device | Compute | Latency | FPS |
|
| 59 |
+
|--------|---------|---------|-----|
|
| 60 |
+
| Snapdragon 439 | ~10 GFLOPS | ~19ms | 30+ |
|
| 61 |
+
| Snapdragon 665 | ~22 GFLOPS | ~12ms | 30+ |
|
| 62 |
+
| Snapdragon 778G | ~65 GFLOPS | ~4ms | 60+ |
|
| 63 |
+
|
| 64 |
+
## Model Weights
|
| 65 |
+
|
| 66 |
+
Model weights are **proprietary** and not distributed in this repository. This page serves as documentation for the LiveFace architecture.
|
| 67 |
+
|
| 68 |
+
For licensing inquiries: **business@creatora.app**
|
| 69 |
+
|
| 70 |
+
## Publications
|
| 71 |
+
|
| 72 |
+
- **Zenodo**: [DOI: 10.5281/zenodo.19477081](https://doi.org/10.5281/zenodo.19477081)
|
| 73 |
+
- **TechRxiv**: Under review
|
| 74 |
+
- **arXiv**: Pending submission (cs.CV)
|
| 75 |
+
|
| 76 |
+
## Authors
|
| 77 |
+
|
| 78 |
+
- **Dmitry Rodin** — Founder & Lead Researcher, Creatora (dmitry.r@creatora.app)
|
| 79 |
+
- **Nikita Rodin** — Texas Tech University (nikita.r@creatora.app)
|
| 80 |
+
|
| 81 |
+
## Citation
|
| 82 |
+
|
| 83 |
+
```bibtex
|
| 84 |
+
@misc{rodin2026liveface,
|
| 85 |
+
title={LiveFace: Real-Time Photorealistic Facial Animation on Low-End Mobile Devices via Compact Per-Avatar Neural Decoders and Universal Compositor-Upscaler},
|
| 86 |
+
author={Dmitry Rodin and Nikita Rodin},
|
| 87 |
+
year={2026},
|
| 88 |
+
doi={10.5281/zenodo.19477081},
|
| 89 |
+
url={https://doi.org/10.5281/zenodo.19477081}
|
| 90 |
+
}
|