File size: 3,342 Bytes
92ea972 35a40bc 92ea972 35a40bc 92ea972 35a40bc 92ea972 b5ca0c5 92ea972 b5ca0c5 92ea972 b5ca0c5 92ea972 35a40bc | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 | ---
license: mit
library_name: pytorch
tags:
- co-speech-gesture-generation
- streaming-generation
- real-time-generation
- motion-generation
- smpl-x
- streamtalk
- arxiv:2608.01643
datasets:
- H-Liu1997/BEAT2
---
# StreamTalk official CFG checkpoints
Official checkpoints for **StreamTalk: Streaming Co-Speech Gesture Generation with Key-Pose Anchoring** (ECCV 2026), a closed-loop method for real-time, long-horizon **co-speech gesture generation**.
- [Paper](https://arxiv.org/abs/2608.01643) · [Hugging Face Paper page](https://huggingface.co/papers/2608.01643)
- [Project page](https://xiangyuezhang.com/StreamTalk/) · [Code](https://github.com/Xiangyue-Zhang/StreamTalk)
- [Generated inference data](https://huggingface.co/datasets/X-Zhang/StreamTalk-Inference-Data)
These are hardware-agnostic FP32 generator state dictionaries for
`DiffusionDITNetPartsFixedExpressions2PostNormInteraction2`. They were retrained
on BEAT2 English with classifier-free condition dropout and selected with an
accelerated H200 implementation of the StreamTalk generation procedure plus
the official EMAGE/PantoMatrix AESK FGD metric.
| File | Intended role | Epoch | CFG | Speaker2 FGD | All FGD |
| --- | --- | ---: | ---: | ---: | ---: |
| `streamtalk_speaker2_combined_e0946_cfg3.pt` | Speaker2 and combined | 946 | 3 | 0.378879 | 0.250287 |
| `streamtalk_speaker_all_e0940_cfg3.pt` | All speakers | 940 | 3 | 0.424477 | 0.217672 |
The combined selection score is
`max(Speaker2 FGD / 0.383, All FGD / 0.293)`.
The checkpoints contain 407 CPU FP32 tensors and 71,167,501 parameters. Both
load strictly into the originally released StreamTalk model schema. H200,
BF16, CUDA Graphs, and four-GPU execution were training/evaluation
infrastructure choices and are not encoded in the checkpoint files.
The reported values are accelerated EMAGE/PantoMatrix AESK FGD measurements,
not paper-exact oracle measurements. A 1e-3 absolute FGD reporting tolerance
was selected for engineering comparisons, but the final CFG=3 values have not
been accepted against a direct B=1/M=1/full-window oracle. BC and DIV have not
been re-evaluated for these retrained checkpoints.
The published inference NPZ bytes were independently re-scored with the
released scorer (metric batch 16): Speaker 2 `0.3788789702354345`; all speakers
`0.21764184426140076`. These differ from the selection-time values by about
`7e-15` and `2.97e-5`, respectively.
Verify downloads before loading:
```bash
python tools/verify_pretrained.py --weights-dir checkpoints/pretrained
```
PyTorch checkpoints use pickle internally. Only load files downloaded from the
linked StreamTalk release and matching the published SHA256 values.
End-to-end inference also requires WavLM Large, the bundled
`SimpleSpeechModel`, SMPL-X neutral, and the BEAT2 retrieval database. Those
runtime assets are independent of the generator checkpoint format.
## Citation
```bibtex
@inproceedings{zhang2026streamtalk,
title={StreamTalk: Streaming Co-Speech Gesture Generation with Key-Pose Anchoring},
author={Zhang, Xiangyue and Li, Jianfang and Zhang, Jiaxu and Yang, Kaixing and Hoi, Steven},
booktitle={European Conference on Computer Vision},
year={2026},
eprint={2608.01643},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2608.01643}
}
```
|