Embodied Creativity
Embodied Creativity - Augmenting Symbolic Narrative Generation with embodied gesture and emotion in a robotic agent
Research in Computational Creativity is rather novel, but shows great advances with the current rise of technology, especially with neural architectures which seem to be able to re-create artworks in all sorts of domains (text generation, image completion). Building generative, computational models might not be enough to actually understand the multitude of processes that play a role in creativity, which is arguably one of the most complex cognitive processes.
One research focus at the Creative Language Systems Group is generative storytelling. In order to advance the field of computational storytelling, my research aims to combine the stance of Embodied Cognition with Computational Creativity in order to disentangle the contributions of agents, bodies and processes in a creative act.
Empirical evaluations suggests that the embodied agent plays a major role in contributing humor to the performance and overall adding greatly to the performance, when compared to non-embodied conditions.
In order to investigate the performance, collaboration and interaction of multiple agents and the synergy that can arise from such a setup, we have developed the Scéalability framework (Mildner, Wicke and Veale, 2020). Scéalability allows to scale the number of interacting agents up or down, whether we need two robots and a speaker or just one speaker narrating the story.
In (Wicke and Veale, 2018a) and (Wicke and Veale, 2018b), we discuss the evolution and implementation of an interactive, embodied storytelling framework that is using a robotic agent to engage in a storytelling process with the audience.
The embodiment draws from insights into gesture research. Furthermore, we explored the concept of image schemas as means to create gestures which are based on notions of embodiment. Insights from this research have been reported in (Wicke and Veale, 2018).
In order to investigate the necessity and effect of an embodied agent, we extend the framework towards a Double Act, which includes a non-embodied agent into the storytelling performance (Wicke, Mildner and Veale, 2019). Such a multi-agent setup allows us to strip apart the contributions of an embodied agent within a creative process.
Spatial Dimensions in Narrative and Performance
A story performed on stage can make use of costumes, light, sound etc. But where those elements are helpful, the spatial movement on the stage seems to be a necessary element of a theatrical performance. The way that the character dynamics change might be reflected in the distance of the actors to one another.
Our multi-agent system can be used to investigate this intuition. Therefore, we test different modes of movement in a storytelling performance. Contrasting complex pantomimic movements of the robots with simple spatial (back and forth) movements in both coherent and incoherent conditions (Wicke, Veale 2019).
A series of empirical studies has shown that the simple spatial movements will be appreciated by an audience as much as the complex pantomimic movements. Both achieve higher performative ratings when presented in a coherent fashion. This indicates that spatial movement can be as important as coherent pantomimic enactment of story actions.
Ideally, the combination of both, spatial schematic movement and pantomimic movements yields the best performance. Results of an empirical study underline this intuition, even though the combined movement approach does not outperform the pantomimic approach (Wicke, Veale 2020).
Gestural Interactivity in Narrative and Performance
So far, we have investigated the performance of the robots and their use of gestures and movements in space. The initial project featured a minimal interactive mode that allowed to craft a story with the robot. Combining the topics of gesture, interactivity and movement we can advance to the next level. The lastest project aims to allow the audience to shape turns of the story using gestures themselves. The robot might enact a marriage proposal, turn and ask the audience for feedback. The user can then give a happy thumbs up or a grimace and a thumbs down. Posture estimation algorithms and facial expression detection will parse the user's feedback and change the story accordingly.