Multimodal Digital Humanities

Abstract

Interactive digital technologies with facilities for developing conceptual frameworks and storing multidimensional data at different levels of description have the potential to advance research in semiotics (i.e., the study of sign systems and processes). Such an approach, called multimodal digital semiotics (O’Halloran et al., Text and Talk: Special Edition for Michael Halliday 33(4–5):665–690, 2013), involves a “digital semiosphere” (http://semioticon.com/semiotix/2011/02/multimodal-digital-semiotics/, http://semioticon.com/semiotix/2010/03/multimodal-semiosis-multimodal-semiotics-digital-technologies-and-techniques-for-studying-multimodal-communication/) for building theory and analytical approaches for semiotics research, borrowing from Lotman’s (Sign System Studies 33(1):208, 2005) notion of the semiosphere as “the semiotic space, outside of which semiosis cannot exist.” A digital semiosphere permits the interaction of sign systems in sociocultural processes to be theorized, analyzed, visualized, interpreted and compared, as demonstrated in this chapter. In this case, the underlying basis of the semiotics research is multimodal analytics, “the study of semantic patterns arising from the integration of language, images, and audio resources in multimodal texts” (O’Halloran et al., The Routledge handbook of multimodal analysis, Routledge, London, p. 386, 2014). Multimodal digital semiotics, the digital semiosphere and multimodal analytics move digital humanities (e.g., Berry, Understanding digital humanities, Palgrave, Hampshire, 2012) into the realm of multimodal digital humanities; in this case. “leveraging the potential of the visual and aural media that are part of contemporary life” (Svensson, Digital Humanities Quarterly, 4(1), 2010; see also McPherson, Cinema Journal 48(2):119–123, 2009) for the development of theory, tools and techniques for semiotics research (O’Halloran et al., Visual communication, Gruyter, Berlin, pp. 565–588, 2014; Podlasov and O’Halloran, Critical multimodal studies of popular culture, Routledge, New York, pp. 71–90, 2014).