Towards methods for evaluating and communicating participatory design: A multimodal approach

https://doi.org/10.1016/j.ijhcs.2016.03.004Get rights and content

Highlights

  • Use of Multimodal Analysis to analyze users’ contributions in Participatory Design.

  • Case study with 4 children in Autistic Spectrum Disorder.

  • Multimodal analysis effectively grasped contributions across different resources.

  • Usefulness for guiding design of Serious Games in interdisciplinary teams.

Abstract

Participatory Design (PD) has been proposed as a useful strategy to address pitfalls in the design of serious games for children with special needs. Nonetheless, methodological weaknesses in the analysis of the results of PD workshops may hinder its effectiveness in providing useful and robust design contributions and facilitating communication in multidisciplinary teams. To address this issue we propose the use of multimodal analysis to evaluate participants’ contributions during PD workshops. We present an analysis that was applied in an informant workshop with autistic children, aimed at refining the design of a serious game based on Full-Body Interaction. Results show that multimodal analysis constitutes an effective and coherent method to capture and analyze users’ contributions across a wide range of semiotic resources, thus extending the richness of insights that can be derived from a PD workshop and communicated to the rest of the team. Furthermore, the analysis allowed the identification of fundamental design questions, thus offering a robust empirical ground for supporting dialogue and reflection between multiple stakeholders.

Introduction

Autism is a neurodevelopmental disorder characterized by impairments in social and communication abilities and by restrictive interests and sensory abnormalities (DSM-V). Several therapeutic treatments have been proposed to improve the quality of life of individuals with Autistic Spectrum Disorder (ASD) by facilitating learning of everyday skills (Fuentes-biggi et al., 2006). However, the effectiveness of these treatments is mainly associated with their intensity in terms of weekly hours (Boyd et al., 2014). The time-consuming nature of these treatments led mental health professionals to start exploring the use of serious games to complement traditional treatment methods (Goh et al., 2008). However, gaming interventions showed pitfalls related to difficulties in reaching the therapeutic goals and shortcomings in designing experiences that can be truly engaging and motivating for children with ASD (Goh et al., 2008).

A possible reason behind these pitfalls can be found in the shortage of inclusive design approaches which are capable of integrating the knowledge of mental health experts with the interests of children and the experience of designers. To address this issue several researchers proposed the use of Participatory Design (PD) methods to properly grasp children׳s interests and improve user experience (Benton et al., 2014, Frauenberger et al., 2011, Millen et al., 2010, Piper et al., 2006).

However, PD can represent a demanding experience for children with ASD (Frauenberger et al., 2013). Difficulties in communication may hinder the opportunities for properly capturing and identifying users’ contributions, thus reducing the quality of insights that researchers may obtain from a PD workshop. Nonetheless, children with special needs are often the population that benefits the most from design processes that include them (Frauenberger et al., 2011). Therefore, proper methods to include them in the PD of serious games are needed.

To address this issue we propose combining PD techniques with the use of an analytical framework aimed at offering instruments to deepen into the different dimensions of users’ contributions that can be derived from a PD workshop. Specifically, we propose the use of a framework derived from multimodal analysis (Jewitt, 2013, Kress, 2010) to collect and interpret users’ contributions during PD and hence be able to better transfer them to the rest of the team composed of experts and designers.

Multimodal analysis, by providing concepts and methods which collect and analyze a wide range of resources employed by people to construct meaning (i.e. gestures, gaze, movements, etc.), extends the analysis beyond the intentional communicational prompts, thus overcoming the barriers of communication and expression.

We suggest that multimodal analysis does not only respond to the specific challenges of PD with ASD children, but also represents a useful contribution to research related to defining robust and coherent evaluation methods for PD (Frauenberger et al., 2014). Until now, most research in PD has focused on proposing techniques for eliciting users’ contributions (Druin, 2002, Walsh et al., 2010), while robust approaches for the analysis of PD are scarce. This methodological weakness may generate biases and unbalanced power structures that may be particularly delicate in the complex network of human relations present in the multidisciplinary teams required to design serious games.

Starting from this perspective, a case study will be presented which shows the application of multimodal analysis in a PD workshop with ASD children. The workshop was aimed at refining the design of a serious game based on Full-Body Interaction. Through the description of the proposed approach we will then argue the potential of multimodal analysis as an evaluation tool for PD and as an instrument to guide a design process.

In particular, our study suggests the effectiveness of this approach not only for analyzing participants’ contributions and emergent behaviors from a broad and rigorous perspective, but also as an instrument to guide design refinements and to facilitate communication within the design team and with experts. Within that, we will place special emphasis on discussing its suitability for eliciting crucial design questions and for spotting unsolved design aspects, which are not transparent in other types of analyses.

Thus, we suggest that a multimodal analytical approach, by raising design questions based on empirical data, can offer a fertile ground to support dialogue and reflection between multiple stakeholders (e.g. mental health professionals, designers). In particular, this method provides insights into guiding the discussion on issues related to interaction design, content definition, esthetics and alignment between educational requirements and design choices.

In the following sections we will introduce the theoretical framework of the research and report the procedure and results related to the application of multimodal analysis to PD. Finally, the benefits of the proposed approach for improving iterative design processes will be considered and guidelines will be drawn for future research.

Section snippets

Participatory design for serious games for ASD children

In the last decades a growing attention has been posed on the potential of using serious games to complement traditional treatments for children with ASD (Ben-Sasson et al., 2013, Whalen et al., 2010, Yan, 2011). This approach is based on the interest that children show in digital games and the capacity of games to foster motivation, engagement (Prensky, 2003, Resnick, 2002), and produce behavioral changes (Deterding et al., 2011). Moreover, game-based interventions with children within the

Applying multimodal analysis to Participatory Design

In the present study we describe a PD workshop based on the “children as informants” model (i.e. testing raw prototypes with children in order to get inputs during the design process (Nesset and Large, 2004)). The workshop was aimed at evaluating and improving a serious game based on Full-Body Interaction for ASD children. Specifically, the goals were, on the one hand, to analyze how children interact with and interpret the initial design proposal and, on the other, to include children׳s

Situated and embodied meaning making

The analysis of embodied meaning making was derived from the maps produced to describe children׳s exploration of the space and the movement of their net (Fig. 3). In the following sections we provide a detailed analysis of these different aspects.

Discussion

Our study confirms previous research on the suitability of multimodal analysis for the evaluation of digital technologies (Jewitt, 2013) and widens its application to design processes that are based on PD workshops. Within this context, we suggest that multimodal analysis can offer both direct and indirect benefits to PD.

At a methodological level, multimodal analysis has shown to be suitable to address the challenges related to analyzing PD by systematically attending to the multiple resources

Conclusions

We have presented an exploratory case study aimed at applying multimodal analysis to a PD workshop with autistic children. We have applied the multimodal analysis to enrich the analysis of children׳s contributions and enhance communication of the results of the PD workshop to the rest of the interdisciplinary team. Specifically, the analysis has been structured according to three main concepts derived from socio-semiotics: situated and embodied meaning making, interpreter interests and

Acknowledgments

This project has been funded through the competitive grant RecerCaixa 2013 from the Obra Social la Caixa, Barcelona. We would like to acknowledge the four ASD children that collaborated with us in the PD and their families. We are also very thankful for Sara Price for her review and feedbacks during the paper preparation.

References (71)

  • Benton, L., Vasalou, A., Khaled, R., Johnson, H., Gooch, D., 2014. Diversity for design: a framework for involving...
  • Bertacchini, F., Bilotta, E., Gabriele, L., Vizueta, D., Pantano, P., Tavernise, A., Vena, S., Valenti, A., 2013. An...
  • B.A. Boyd et al.

    Comparative efficacy of LEAP, TEACCH and non-model-specific special education programs for preschoolers with autism spectrum disorders

    J. Autism Dev. Disord.

    (2014)
  • B. Charlton et al.

    Educational games: a technique to accelerate the acquisition of reading skills of children with learning disabilities

    Int. J. Spec. Educ.

    (2005)
  • A. Clark

    Embodiment and the philosophy of mind

    R. Inst. Philos. Suppl.

    (1998)
  • A. Clark

    Listening to and involving young children: a review of research and practice

    Early Child Dev. Care

    (2005)
  • S. Cox

    Intention and meaning in young children׳s drawing

    Int. J. Art Des. Educ.

    (2005)
  • L. Crescenzi et al.

    The role of touch in preschool children׳s learning using iPad versus paper interaction

    Aust. J. Language Lit.

    (2014)
  • M. de Certeau

    The Practice of Everyday Life

    (1984)
  • Derboven, J., Van Mechelen, M., Slegers, K., 2015. Multimodal analysis in participatory design with children: a primary...
  • Deterding, S., Khaled, R., Nacke, L.E., Dixon, D., 2011. Gamification: toward a definition. In: Proceedings of the CHI...
  • A. Druin

    The role of children in the design of new technology

    Behav. Inf. Technol.

    (2002)
  • J. Einarsdottir

    Playschool in pictures: children׳s photographs as a research method

    Early Child Dev. Care

    (2005)
  • R.S. Flewitt et al.

    What are multimodal data and transcription?

  • Frauenberger, C., Good, J., Alcorn, A., Pain, H., 2012. Supporting the design contributions of children with autism...
  • C. Frauenberger et al.

    In pursuit of rigour & accountability in participatory design

    Int. J. Hum.-Comput. Stud.

    (2014)
  • C. Frauenberger et al.

    Designing technology for children with special needs: bridging perspectives through participatory design

    CoDesign

    (2011)
  • J. Fuentes-biggi et al.

    Guía de buena práctica para el tratamiento de los trastornos del espectro autista

    Rev. Neurol.

    (2006)
  • E. Giaccardi et al.

    Embodied narratives : a performative co-design technique

    Proceedings of the Designing Interactive Systems Conference

    (2012)
  • S. Goldin-Meadow

    Learning through gesture

    Wiley Interdiscip. Rev.: Cognit. Sci.

    (2011)
  • C. Goodwin

    Action and embodiment within situated human interaction

    J. Pragmat.

    (2000)
  • O.B. Hansen et al.

    Emotion recognition for mobile devices with a potential use in serious games for autism spectrum disorder

    SGDA

    (2013)
  • F. Happé et al.

    The weak coherence account: detail-focused cognitive style in autism spectrum disorders

    J. Autism Dev. Disord.

    (2006)
  • S. Harding

    Is there a feminist method

  • D. Harper

    Talking about pictures: a case for photo elicitation

    Visual Stud.

    (2002)
  • Cited by (0)

    This paper has been recommended for acceptance by E. Motta

    View full text