Using a Participatory Approach to Iteratively Adapt a Collaborative Emotion Recognition Game for Non-Speaking Autistic Youth

Background: Although autism is partially defined by social difficulties, some autistic people communicate more effectively through computers than in-person (Gillespie-Lynch et al., 2014). Consequently, computer-mediated supports for autistic people are proliferating (Fletcher-Watson, 2014). However, evidence that computer-mediated interventions help autistic people develop generalizable social-communicative skills remains limited (Whyte et al., 2015). Participatory research wherein autistic people collaborate in game design/evaluation may ameliorate the disconnect between the potential of computerized interventions and limited benefits documented thus far.

Objectives: By collaborating with speaking autistic people, we are developing a game to help non-speaking autistic people understand complex emotions and collaborate.

Methods: Connecting through Kinect uses animated tutorials rather than words to be accessible regardless of linguistic ability and promotes generalizable social-communicative skills by providing simultaneous opportunities to engage digitally and in-person. Players solve collaborative emotion matching puzzles using Kinect while standing together. Each puzzle depicts the outline of a figure in an emotional context. After constructing the body, players must agree on the correct emotion for the face by selecting from three emotions. This collaborative component was adapted from a task which promoted collaboration among minimally-verbal children (Holt & Yuill, 2014). In earlier iterations of game development (Sturm et al., 2018), autistic college students spearheaded many improvements including: demos, meaningful rewards, improved image quality, and a relatable storyline (an autistic child overcomes bullying through friendship with a friendly dragon; Figures). After qualitative coding revealed that autistic students initially looked at the screen rather than each other, we added a tutorial depicting collaborative play, an associated pause feature to encourage collaboration, and a face tracking component to automatically quantify collaboration. Autistic college students looked at one another more when playing the revised game. However, some played with mentors who prompted collaboration verbally. After incorporating extensive feedback from 10 autistic college students in our mentorship program, we asked 10 autistic adolescents from a technology camp for feedback.

Results: To evaluate if the game is accessible without spoken instructions, autistic adolescents played the revised game with a player who did not prompt interaction. They exhibited pronounced variability in emotion matching (0-16 errors), utterances (1-47) and looking at one another (0-6). They found the game useful/fun but critiqued motion control inefficiencies and found the collaboration animation confusing, indicating that it should be removed. They highlighted the need for tutorials to let players know that there is a “right answer.” Some felt that “errors” were “glitches” and were surprised to realize that the background was relevant. An adolescent with low language skills kept approaching the Kinect asking “Why can’t I help?” Findings indicate that emotion matching/collaboration are desired learning goals while highlighting areas for improvement.

Conclusions: Our autistic collaborators have spearheaded extensive improvements in game design. They remind us to incorporate key design principles that promote generalization but are often lacking in games designed for autistic people. After incorporating feedback, we will evaluate the game with non-speaking children this spring. The process of collaborating with speaking autistic people to develop computer-mediated interventions for non-speaking youth shows promise.