Views
No views yet
motius.pipelines.diffusestylegesture.DiffuseStyleGesturePipeline| Item | Value |
|---|---|
| Task | Speech-to-Gesture |
| Training data | BEAT speakers 2 and 10 |
| Benchmark | BEAT2 English speaker 2 |
| Input | WavLM/acoustic 1133D + aligned FastText 301D + speaker ID |
| Native motion | BEAT 75-joint BVH |
| Motius motion | SMPL-X 55-joint axis-angle + zero expression + translation |
| Motion rate | 30 FPS |
| Window | 150 frames, 30 seed frames, 120-frame stride |
| Diffusion | Official 1,000-step sampler, seed 123456 |
| Checkpoint | model001080000.pt |
| Checkpoint SHA-256 | 2e25d8b257b44e5110a61713476a146e99c34926c763b5dd1c18a6884a857c77 |
| Runtime revision | 9649995cdd0b8ce0d42bd02567b2bd03829a0540 |
| Inference-pinned artifact revision | 8707378e3835ac2d7e17dc977272640190a1daf5 |
| License | MIT model/runtime; CC-BY-SA-3.0 FastText subset |
1bash tools/setup_diffusestylegesture_env.sh
2git clone https://github.com/YoungSeng/DiffuseStyleGesture \
3 ref_repo/DiffuseStyleGesture
4git -C ref_repo/DiffuseStyleGesture checkout \
5 9649995cdd0b8ce0d42bd02567b2bd03829a05400.96376 cosine similarity, while the original checkpoint reproduces the
released features at 0.9999999999 cosine similarity.1gdown 12-cB34qCTvByWT-QtOcZaqwwO21FLSqU \
2 -O outputs/checkpoints/diffusestylegesture/WavLM-Large.ptcrawl-300d-2M.vec.zip. The release
contains the 790 vectors used by the fixed benchmark population as
beat2_speaker2_fasttext_vocab.npz (one source token, berties, is OOV).
The DSG+ checkpoint and source use MIT terms; this compact vector subset
retains FastText's CC-BY-SA-3.0 terms.1python tools/build_diffusestylegesture_beat2_features.py \
2 --runtime-root ref_repo/DiffuseStyleGesture/BEAT-TWH-main \
3 --manifest outputs/evaluation/speech_to_gesture/beat2/_resources/hliu1997_beat2_speaker2_test/manifest.json \
4 --data-root outputs/evaluation/speech_to_gesture/beat2/_resources/hliu1997_beat2_speaker2_test/beat_english_v2.0.0 \
5 --wavlm-checkpoint outputs/checkpoints/diffusestylegesture/WavLM-Large.pt \
6 --fasttext-compact outputs/checkpoints/diffusestylegesture/beat2_speaker2_fasttext_vocab.npz \
7 --out-dir outputs/evaluation/speech_to_gesture/beat2/diffusestylegesture/features
8
9python tools/eval_diffusestylegesture_beat2.py \
10 --runtime-root ref_repo/DiffuseStyleGesture/BEAT-TWH-main \
11 --artifact ZeyuLing/Motius-DiffuseStyleGesture-BEAT \
12 --manifest outputs/evaluation/speech_to_gesture/beat2/_resources/hliu1997_beat2_speaker2_test/manifest.json \
13 --data-root outputs/evaluation/speech_to_gesture/beat2/_resources/hliu1997_beat2_speaker2_test/beat_english_v2.0.0 \
14 --feature-dir outputs/evaluation/speech_to_gesture/beat2/diffusestylegesture/features \
15 --out-dir outputs/evaluation/speech_to_gesture/beat2/diffusestylegesture/predictions| Evaluator | Clips | FGD ↓ | BC ↑ | Diversity | uTMR FID ↓ | uTMR Paired Dist. ↓ |
|---|---|---|---|---|---|---|
| BEAT2 Official | 15 | 65.9364 | 6.3858 | 12.2055 | 0.0930 | 16.0251 |
14.461: that number uses
the GENEA/Talking-With-Hands challenge autoencoder and test population, whereas
this row uses the fixed Motius BEAT2 speaker-2 SMPL-X protocol. The upstream
authors also document that the manual BVH-to-SMPL-X conversion can depress
motion realism on BEAT2, even for reconstruction. Accordingly, this released
row is a verified cross-protocol Motius result rather than a paper-number claim.
The exact GENEA autoencoder checkpoint and Table 1 split were not released, so
the public artifacts are insufficient for an exact 14.461 reproduction.1@inproceedings{yang2023diffusestylegesture,
2 title={DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Models},
3 author={Yang, Sicheng and Wu, Zhiyong and Li, Minglei and Zhang, Zhensong and Hao, Lei and Bao, Weihong and Cheng, Ming and Xiao, Long},
4 booktitle={IJCAI},
5 year={2023}
6}