Immersive Machine Learning for Social Attitude Detection in Virtual Reality Narrative Games

Immersive Machine Learning for Social Attitude Detection in Virtual Reality Narrative Games

Immersive Machine Learning for Social Attitude Detection in Virtual Reality Narrative Games

By Georgiana Cristina Dobre, Marco Gillies and Xueni Pan

Abstract

“People can understand how human interaction unfolds and can pinpoint social attitudes such as showing interest or socialengagement with a conversational partner. However, summarizing this with a set of rules is difcult, as our judgment issometimes subtle and subconscious. Hence, it is challenging to program Non-Player Characters (NPCs) to react towards socialsignalsappropriately, which is important for immersive narrative games in Virtual Reality (VR). We collaborated with twogame studios to develop an immersive machine learning (ML) pipeline for detecting social engagement. We collected data from participants-NPC interaction in VR, which was then annotated in the same immersive environment. Game design isa creative process and it is vital to respect designer’s creative vision and judgement. We therefore view annotation as a keypart of the creative process. We trained a reinforcement learning algorithm (PPO) with imitation learning rewards using rawdata (e.g. head position) and socially meaningful derived data (e.g. proxemics); we compared different ML configurations including pre-training and a temporal memory (LSTM). The pre-training and LSTM configuration using derived data performed the best (84% F1-score, 83% accuracy). The models using raw data did not generalize. Overall, this work introducesan immersive ML pipeline for detecting social engagement and demonstrateshow creatives could use ML and VR to expand their ability to design more engaging experiences. Given the pipeline’s results for social engagement detection, we generalize it for detecting human-defined social attitudes.”

Reference

Dobre, G., Gillies, M., & Pan, X. (2022, February). Immersive machine learning for social attitude detection in virtual reality narrative games. Retrieved April 19, 2022, from https://research.gold.ac.uk/id/eprint/31463/

Keywords

Artificial intelligence, expressive body language, gaming, human–computer interaction, virtual agents, virtual reality, research