Connecting through Kinect: A Participatory Approach to Designing a Collaborative Emotion Recognition Game with and for Autistic Individuals

Background: Autistic people face challenges communicating with others in-person which are often reduced online (Gillespie-Lynch et al., 2014). However, evidence that computer-mediated interventions are effective in helping autistic people develop generalizable social-communicative skills remains limited (Whyte et al., 2015). Participatory research wherein autistic people are involved as collaborators in game design/evaluation can help address the disconnect between the potential of computerized autism interventions and limited benefits documented thus far. Parsons and Cobb suggested that effective technologies to support autistic people are developed by aligning Theory, Technologies and Thoughts (autistic perspectives).

Objectives: We aim to document the process of including autistic youth in designing/evaluating a game to help autistic people understand complex emotions and collaborate and to determine if the game is effective at scaffolding collaboration and emotion recognition.

Methods: Our game is designed to support generalizable skills by providing an immersive environment wherein participants can simultaneously engage digitally and in-person while learning how to interpret emotions. Standing near one another, participants complete collaborative emotion matching puzzles using Kinect technology. Animated tutorials teach gameplay. All instructions are nonverbal so that the game is accessible for people with diverse communicative abilities. In the core game, two players use gestures to move puzzle pieces. Each puzzle depicts the outline of a figure in an emotional context. After players construct the body; they must agree on the correct emotion for the face by selecting from three emotions. This collaborative component was adapted from a picture-matching task that was effective at promoting collaboration among minimally verbal autistic children (Holt & Yuill, 2014).

Results: We invite autistic college students in our mentorship program to provide feedback on design and become more involved if interested. Two autistic college students and their mentors provided initial feedback. Later, five autistic students and their mentors were videotaped playing the game and completed structured interviews. Students provided detailed suggestions about how to improve the look, feel and content of the game including: providing demos, making rewards meaningful, improving image quality, and giving the game a relatable storyline. Qualitative coding of game play revealed that players typically looked at the screen rather than each other while playing although they exhibited positive affect and communicated verbally.Therefore, we created a story focused on a protagonist facing a challenge many autistic youth face, bullying, about how friendship (with a friendly dragon) can help one overcome bullying (Figure 1). We added a pause feature to to encourage direct interaction during the collaborative phase and a tutorial demonstrating collaboration. Qualitative coding of five mentees playing the revised game revealed that the pause feature fostered more in-person interaction. After completing usability evaluations, we will examine benefits of repeated engagement with the game for minimally verbal autistic individuals and their siblings.

Conclusions: Our autistic collaborators have provided useful suggestions that have led to extensive improvements in game design. They reminded us to incorporate key design principles (a storyline and an interpretable reward system) that are believed to promote generalization but are often lacking in games designed for autistic people.