EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

ImaGGen: Zero-Shot Generation of Co-Speech Semantic Gestures Grounded in Language and Image Input

Hendric Voss, Stefan Kopp

ACM Transactions on Interactive Intelligent Systems · 31.12.2026 · 16 (4) · 1-40

DOI: 10.1145/3845992 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Interactive Intelligent Systems, online erschienen am 12.11.2026. Verfasst von Hendric Voss und 1 weiteren Autor:innen. Die Arbeit nennt 127 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: Deutsche Forschungsgemeinschaft (DFG, German Research Foundation)—CRC 1646 Linguistic Creativity in Communication.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Human communication combines speech with expressive nonverbal cues such as hand gestures that serve manifold communicative functions. Yet, current generative AI-based gesture generation approaches are, for the most part, restricted to simple, repetitive beat gestures that accompany the rhythm of speaking but do not contribute to communicating semantic meaning. This article tackles a core challenge in co-speech gesture synthesis: generating iconic or deictic gestures that are semantically coherent with a verbal utterance. Our basic assumption is that such gestures cannot be derived from language input alone, which inherently lacks the visual meaning that is often carried autonomously by gestures. We therefore introduce a zero-shot system that generates gestures from a given language input and is additionally informed by imagistic input, without manual annotation or human intervention. Our method integrates an image analysis pipeline that extracts key object properties such as shape, symmetry, and alignment, together with a semantic matching module that links these visual details to spoken text. An inverse kinematics engine then synthesizes iconic and deictic gestures and combines them with co-generated natural beat gestures for coherent multimodal communication. A comprehensive user study demonstrates the effectiveness of our approach. In scenarios where speech alone was ambiguous, gestures generated by our system significantly improved participants’ ability to identify object properties, confirming their interpretability and communicative value. While challenges remain in representing complex shapes, our results highlight the importance of context-aware semantic gestures for creating expressive and collaborative virtual agents or avatars, marking a substantial step forward towards efficient and robust, embodied human-agent interaction. More information and example videos are available here: https://hvoss-tech.github.io/ImaGGen/ .

Förderung

  • Deutsche Forschungsgemeinschaft (DFG, German Research Foundation)—CRC 1646 Linguistic Creativity in Communication

Zitieren

APA

Voss, H., & Kopp, S. (2026). ImaGGen: Zero-Shot Generation of Co-Speech Semantic Gestures Grounded in Language and Image Input. ACM Transactions on Interactive Intelligent Systems, 16(4), 1-40. https://doi.org/10.1145/3845992

Vancouver

Voss H, Kopp S. ImaGGen: Zero-Shot Generation of Co-Speech Semantic Gestures Grounded in Language and Image Input. ACM Transactions on Interactive Intelligent Systems. 2026;16(4):1-40. doi:10.1145/3845992

BibTeX
@article{Voss2026,
  author = {Voss, Hendric and Kopp, Stefan},
  title = {ImaGGen: Zero-Shot Generation of Co-Speech Semantic Gestures Grounded in Language and Image Input},
  journal = {ACM Transactions on Interactive Intelligent Systems},
  year = {2026},
  volume = {16},
  number = {4},
  pages = {1--40},
  doi = {10.1145/3845992}
}
Kostenlos starten →