Frans Flippo
ABSTRACT
Despite the availability of multimodal devices, there are very few commercial multimodal applications available. One reason for this may be the lack of a framework to support development of multimodal applications in reasonable time and with limited resources. This paper describes a multimodal framework enabling rapid development of applications using a variety of modalities and methods for ambiguity resolution, featuring a novel approach to multimodal fusion. An example application is studied that was created using the framework.
- R. A. Bolt. "Put-that-there": Voice and gesture at the graphics interface. Computer Graphics (SIGGRAPH '80 Proceedings), 14(3):262--270, July 1980. Google Scholar
Digital Library
- L. Boves and E. den Os. Multimodal multilingual information services for small mobile terminals (MUST). Technical report, Eurescom, 2002. http://www.eurescom.de/pub/deliverables/documents/P1100-series/P1104/p1104-d1.pdf.Google Scholar
- CAIP. DISCIPLE-mobile computing and collaboration. http://www.caip.rutgers.edu/disciple.Google Scholar
- P. Cohen, M. Johnston, D. McGee, S. Oviatt, J. Pittman, I. Smith, L. Chen, and J. Clow. Quickset: Multimodal interaction for distributed applications. ACM International Multimedia Conference, New York: ACM, pages 31--40, 1997. Google ScholarDigital Library
- M. Fayad and D. C. Schmidt. Object-oriented application frameworks. Communications of the ACM, 40(10):32--38, 1997. Google ScholarDigital Library
- N. Krahnstoever, S. Kettebekov, M. Yeasin, and R. Sharma. A real-time framework for natural multimodal interaction with large screen displays. In Proc. of Fourth Intl. Conference on Multimodal Interfaces (ICMI 2002), Pittsburgh, PA, USA, October 2002. Google ScholarDigital Library
- J. A. Larson and T. V. Raman. W3C multimodal interaction framework. http://www.w3.org/TR/mmi-framework, 2 December 2002. W3C Note.Google Scholar
- M. E. Markiewicz and C. J. Lucena. Object oriented framework development. ACM Crossroads, 2001. Google ScholarDigital Library
- B. Shneiderman. The future of interactive systems and the emergence of direct manipulation. Behaviour and Information Technology, 1(3):237--256, 1982.Google ScholarCross Ref
- The Apache Jakarta Project. Jakarta BSF -- bean scripting framework. http://jakarta.apache.org/bsf.Google Scholar
- D. Toledano, S. Wang, S. Cyphers, and J. Glass. Extending the galaxy communicator architecture for multimodal interaction research. submitted to ACM Trans. on Human-Computer Interaction, Aug 2002.Google Scholar
- J. Vlissides and M. Linton. Unidraw: A framework for building domain-specific graphical editors. ACM Transactions on Information Systems, 8(3):237--268, 1990. Google ScholarDigital Library
Index Terms
- A framework for rapid development of multimodal interfaces
Recommendations
Multimodal slideshow: demonstration of the openinterface interaction development environment
ICMI '08: Proceedings of the 10th international conference on Multimodal interfacesIn this paper, we illustrate the OpenInterface Interaction Development Environment (OIDE) that addresses the design and development of multimodal interfaces. Multimodal interaction software development presents a particular challenge because of the ever ...
User-Centered Modeling for Spoken Language and Multimodal Interfaces
By modeling difficult sources of linguistic variability in spontaneous speech and language, interfaces can be designed that transparently guide human input to match system processing capabilities. Such work is yielding more user-centered and robust ...
A Case Study on the Rapid Development of Natural and Synergistic Multimodal Interfaces for XR Use-Cases
CHI EA '22: Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing SystemsMultimodal Interfaces (MMIs) supporting the synergistic use of natural modalities like speech and gesture have been conceived as promising for spatial or 3D interactions, e.g., in Virtual, Augmented, and Mixed Reality (XR for short). Yet, the currently ...
Comments