---
license: apache-2.0

language:
  - en

tags:
  - talking-head
  - face-animation
  - avatar
  - image-to-video
  - audio-to-video
  - motion-transfer
  - lip-sync
  - face-synthesis
  - video-generation
  - generative-ai
  - multimodal
  - pytorch
  - sad-talker
  - wav2lip
  - rmbg
  - packed-model
---

# PackedAvatar

PackedAvatar is a **self-contained talking-head generation runtime** that bundles the SadTalker-based avatar pipeline into a single `.pt` artifact.

It supports generating animated talking avatars from:

* a single image + audio
* a prebuilt AvatarBank identity
* explicit avatar conditioning bundles
* motion transfer bundles
* reference-video driving
* optional Wav2Lip post-processing

All core runtime assets are packaged inside `PackedAvatar.pt`.

Core model assets are bundled, but a few auxiliary helper weights may still be downloaded on the first run if they are not already cached locally.

---

# What is included

`PackedAvatar.pt` contains:

* SadTalker source code snapshot
* SadTalker checkpoints
* AvatarBank identity system
* Bria RMBG 2.0 background removal assets
* Wav2Lip GAN checkpoint
* BFM / face model assets
* configuration files
* runtime manifests and hashes
* cached avatar metadata

This is a **runtime artifact**, not a training checkpoint.

---

# Repository contents

* `PackedAvatar.pt` — full bundled runtime
* `PackedAvatar.py` — loader + inference engine
* `requirements.txt` — dependencies
* `README.md` — usage guide

---

# Features

* Single-file deployment (`.pt`) for the main runtime
* Full SadTalker pipeline bundled
* AvatarBank identity system
* Image / avatar / motion / video conditioning
* Automatic background removal (Bria RMBG)
* Optional Wav2Lip GAN post-processing
* CPU / CUDA
* Automatic caching and extraction system
* CLI + Python API support

---

# Requirements

* Python 3.10+
* PyTorch
* FFmpeg (for reference-video audio extraction)
* Dependencies listed in `requirements.txt`

GPU is recommended; CPU is supported.

---

# Quick start

## 1) Install dependencies

```bash
pip install -r requirements.txt
```

## 2) Place the bundle

```text
PackedAvatar.pt
```

## 3) Basic generation

```python
from PackedAvatar import PackedAvatar

model = PackedAvatar("PackedAvatar.pt")

video = model.generate(
    source_image="person.jpg",
    driven_audio="speech.wav"
)

print(video)
```

---

# AvatarBank usage

Generate directly from a prebuilt identity:

```python
video = model.generate(
    avatar_id="Rebecca",
    driven_audio="speech.wav"
)
```

No source image is required for this path.

If `avatar_id` is omitted, the runtime selects a default avatar from the packed bank.

---

# Prepacked AvatarBank table

The following avatars are prepacked in the bank.

## Female

| Style | Names                                                                                                                             |
| ----- | --------------------------------------------------------------------------------------------------------------------------------- |
| anime | Alison, Amber, Andrea, Angela, Christine, Cynthia, Heidi, Jennifer, Karla, Kristen, Laura, Nancy, Patricia, Rebecca, Sandra, Tara |
| cyber | Amanda, Brenda, Christina, Janet, Jill, Julie, Lisa, Mallory, Mandy, Martha, Melissa, Michelle, Regina                            |
| drawn | Alyssa, Danielle, Joan, Kaitlyn, Kimberly, Marie, Samantha, Veronica                                                              |
| paint | Alejandra, Barbara, Briana, Brittany, Emily, Jacqueline, Jodi, Mary, Rhonda, Savannah, Tammy, Victoria, Yolanda                   |
| real  | Amy, Ann, Ashley, Colleen, Heather, Holly, Jordan, Kristin, Kristine, Mariah, Pamela, Sara, Sharon                                |

## Male

| Style | Names                                                             |
| ----- | ----------------------------------------------------------------- |
| anime | Brad, Brian, David, Gregory, John, Jose, Lawrence, Robert         |
| cyber | Daniel, Hayden, James, Jeremy, Paul, Ryan, Sean                   |
| drawn | Bobby, George, Gregg, Kevin, Matthew, Ricky, Thomas               |
| paint | Jacob, Justin, Michael, Nicholas, Steven, William, Zachary        |
| real  | Aaron, Andrew, Benjamin, Christopher, Derek, Frank, Jesse, Joseph |

There are **100 avatars total** in the bank.

---

# Default avatar

If no avatar is explicitly selected, the runtime resolves a default in this order:

1. `defaults.default_avatar` from the manifest, if present and valid
2. first real-style male avatar
3. any real-style avatar
4. any male avatar
5. any avatar with embeddings
6. first available avatar entry

---

# Source image mode

```python
video = model.generate(
    source_image="portrait.png",
    driven_audio="speech.wav"
)
```

Pipeline:

```text
image → face detection → crop → 3DMM extraction → animation
```

---

# Background removal (Bria RMBG)

```python
video = model.generate(
    source_image="portrait.png",
    driven_audio="speech.wav",
    remove_background=True
)
```

Pipeline:

```text
image → Bria RMBG → foreground → SadTalker → video
```

---

# Explicit avatar conditioning

`avatar_condition` may be:

* a Python `dict`
* a `.pt` / `.pth` file
* a `.mat` file

When `avatar_condition` is provided, it supersedes `source_image`-driven conditioning.

```python
video = model.generate(
    avatar_condition="my_avatar_condition.pt",
    driven_audio="speech.wav"
)
```

A valid avatar bundle can include fields such as:

* `coeff_3dmm`
* `motion_3dmm`
* `full_3dmm`
* `crop_preview`
* `crop_info`

---

# Motion conditioning

```python
video = model.generate(
    source_image="portrait.png",
    driven_audio="speech.wav",
    motion_condition="motion.pt"
)
```

Supported motion inputs include:

* `motion_3dmm`
* `coeff_3dmm`
* `full_3dmm_seq`
* `full_3dmm`

---

# Reference-video driving

```python
video = model.generate(
    source_image="portrait.png",
    driven_audio="speech.wav",
    use_ref_video=True,
    ref_video="reference.mp4",
    ref_info="pose"
)
```

Supported `ref_info` values:

* `pose`
* `blink`
* `pose+blink`
* `all`

When `ref_info="all"`, the runtime uses the reference video coefficients directly.

---

# Wav2Lip GAN (optional)

```python
video = model.generate(
    source_image="portrait.png",
    driven_audio="speech.wav",
    use_wav2lip=True,
    wav2lip_repo="/path/to/Wav2Lip"
)
```

Post-processes the SadTalker output for improved lip sync.

The bundled checkpoint `checkpoints/wav2lip_gan.pth` is used automatically.

If no runnable Wav2Lip inference code is found, the runtime falls back to the SadTalker video instead of crashing.

---

# Idle mode

Generate with silent audio instead of an input file:

```python
video = model.generate(
    avatar_id="Aaron",
    use_idle_mode=True,
    length_of_audio=4
)
```

---

# Still mode

Reduces head movement:

```python
still_mode=True
```

---

# Expression control

```python
exp_scale=1.2
```

* higher values → more expressive motion
* lower values → more neutral motion

---

# Face render backend

```python
facerender="facevid2vid"
```

---

# Device selection

Automatically chooses:

* CUDA when available
* Apple Silicon MPS on macOS when available
* CPU fallback otherwise

Override:

```python
PackedAvatar(device="cuda")
```

---

# Python API (full example)

```python
from PackedAvatar import PackedAvatar

model = PackedAvatar(
    packed_pt_path="PackedAvatar.pt",
    device="cuda",
    cache_dir="./cache"
)

video = model.generate(
    source_image="speaker.png",
    driven_audio="speech.wav",
    remove_background=True,
    use_wav2lip=True,
    size=512,
    exp_scale=1.2,
    pose_style=1,
    still_mode=False
)

print(video)
```

---

# Preprocessing helpers

The runtime exposes an embedding extraction helper for image or video conditioning:

```python
bundle = model.extract_embeddings(
    input_path="test_image.png",
    crop_or_resize="crop",
    pic_size=256
)
```

Camel-case alias:

```python
bundle = model.ExtractEmbeddings("test_image.png")
```

The returned bundle can be saved and reused as `avatar_condition` or `motion_condition`.

---

# CLI usage

## Basic

```bash
python PackedAvatar.py \
  --source-image person.jpg \
  --driven-audio speech.wav
```

## AvatarBank

```bash
python PackedAvatar.py \
  --avatar-id Rebecca \
  --driven-audio speech.wav
```

## Background removal

```bash
python PackedAvatar.py \
  --source-image portrait.png \
  --driven-audio speech.wav \
  --remove-background
```

## Wav2Lip

```bash
python PackedAvatar.py \
  --source-image portrait.png \
  --driven-audio speech.wav \
  --use-wav2lip \
  --wav2lip-repo /path/to/Wav2Lip
```

## Reference video driving

```bash
python PackedAvatar.py \
  --source-image portrait.png \
  --driven-audio speech.wav \
  --use-ref-video \
  --ref-video reference.mp4 \
  --ref-info pose+blink
```

## Idle mode

```bash
python PackedAvatar.py \
  --avatar-id Aaron \
  --use-idle-mode \
  --length-of-audio 5
```

## Explicit avatar conditioning bundle

```bash
python PackedAvatar.py \
  --avatar-condition avatar_condition.pt \
  --driven-audio speech.wav
```

## Motion conditioning bundle

```bash
python PackedAvatar.py \
  --motion-condition motion_condition.pt \
  --driven-audio speech.wav
```

---

# How it works

PackedAvatar runs a full multimodal pipeline.

## 1. Asset extraction

* extracts SadTalker + checkpoints from `.pt`
* verifies SHA256 hashes
* builds the runtime cache

## 2. Avatar resolution

Priority:

```text
avatar_condition
→ source_image-driven SadTalker path
→ avatar_id / default AvatarBank resolution
```

If `avatar_condition` is provided, it supersedes `source_image` conditioning.

## 3. Preprocessing

* face detection
* cropping
* 3DMM extraction

## 4. Motion generation

* audio → facial coefficients
* or motion transfer injection

## 5. Rendering

* SadTalker / PIRender animation
* frame synthesis

## 6. Optional post-processing

* Wav2Lip GAN lip-sync enhancement
# AvatarBank API

PackedAvatar exposes the packed AvatarBank directly, allowing you to browse, search, inspect, and manage avatars at runtime.

The AvatarBank is loaded automatically when the model is initialized.

```python
from PackedAvatar import PackedAvatar

model = PackedAvatar("PackedAvatar.pt")
```

## List available avatars

```python
avatars = model.list_avatars()

print(avatars)
```

Returns:

```python
[
    "Aaron",
    "Rebecca",
    "Amy",
    ...
]
```

---

## Search by name

Perform a fuzzy search against avatar IDs.

```python
matches = model.fuzzy_search_avatar("rebeca")

print(matches)
```

Example:

```python
["Rebecca"]
```

---

## Query by gender and style

Search the bank using metadata filters.

```python
female_anime = model.query_avatars(
    gender="female",
    style="anime"
)

print(female_anime)
```

Example:

```python
[
    "Alison",
    "Amber",
    "Andrea",
    ...
]
```

Fuzzy matching is supported:

```python
model.query_avatars(
    gender="femal",
    style="anim"
)
```

---

## Retrieve avatar metadata

```python
metadata = model.get_avatar_metadata("Rebecca")

print(metadata)
```

Example:

```python
{
    "gender": "female",
    "style": "anime"
}
```

---

## Retrieve avatar conditioning

Access the full conditioning bundle used internally by PackedAvatar.

```python
avatar = model.get_avatar("Rebecca")
```

Returned fields may include:

```python
{
    "avatar_id": "Rebecca",
    "gender": "female",
    "style": "anime",
    "coeff_3dmm": ...,
    "motion_3dmm": ...,
    "full_3dmm": ...,
    "crop_info": ...,
    "crop_preview": ...
}
```

This bundle can be passed directly as an `avatar_condition`.

```python
video = model.generate(
    avatar_condition=avatar,
    driven_audio="speech.wav"
)
```

---

## Retrieve avatar previews

Preview images stored in the AvatarBank can be accessed directly.

```python
preview = model.get_avatar_preview("Rebecca")

preview.show()
```

Returns a PIL image.

---

## Delete avatars

Remove an avatar from the in-memory runtime.

```python
model.delete_avatar("Rebecca")
```

This only affects the current runtime session.

To persist changes, save the bank.

---

## Save AvatarBank changes

```python
model.save_avatar_bank("ModifiedAvatarBank.pt")
```

This writes the current AvatarBank state to disk.

---

## Check whether an avatar exists

```python
exists = model.avatar_bank.exists("Rebecca")

print(exists)
```

Returns:

```python
True
```

---

## Direct AvatarBank access

The underlying AvatarBank runtime is also exposed.

```python
bank = model.avatar_bank
```

Available methods include:

```python
bank.available_ids()
bank.list_avatars()
bank.exists(...)
bank.query(...)
bank.fuzzy_search_id(...)
bank.get_avatar(...)
bank.get_metadata(...)
bank.get_preview(...)
bank.delete_avatar(...)
bank.save(...)
```

This provides full programmatic access to the packed avatar database.

---

# First run vs later runs

### First run

* extract bundle
* build cache
* initialize models
* download a couple of auxiliary face-analysis weights if they are not already cached locally

### Later runs

* reuse cache
* skip the auxiliary downloads when the files are already present
* faster startup

---

# Performance notes

* GPU is strongly recommended for 512 resolution
* CPU is supported but slower
* Wav2Lip increases runtime cost
* RMBG adds preprocessing overhead

---

# Why PackedAvatar?

Compared to a standard SadTalker setup:

* single `.pt` deployment artifact
* no model downloads for the main runtime
* no external repos required for core use
* built-in AvatarBank system
* built-in background removal
* optional lip-sync enhancement
* fully offline execution after first-run helper caching
* reproducible runtime via bundle hashing

---

# Notes

* This repo is inference-only
* Bundles are treated as trusted artifacts
* Cache is auto-invalidated when the bundle changes
* All runtime dependencies are resolved internally

---

# Credits

Built on top of:

* SadTalker
* FaceVid2Vid / PIRender
* Wav2Lip GAN
* Bria RMBG