Andrea Mattioli
Skip Abstract Section
Skip Supplemental Material SectionAbstract
Automations in the context of smart homes have been adopted more and more frequently; thus, users should be able to control them and create automations most suitable to their needs. Current solutions for this purpose are based on visual apps with conceptual representations of possible automation elements. However, they tend to be static, abstract, and detached from the user's real context. In this paper, we propose a novel solution based on mobile augmented reality, which provides situated, dynamic representations associated with the physical objects available in the current users' context while they are freely moving about. It allows direct interaction with the objects of interest, monitoring nearby objects' automations while moving, and creating new automations or modifying existing ones. It also supports users with recommendations of object and service configurations relevant to complete the editing of the new automations. The paper also reports on a user test, which provided positive feedback.
Supplemental Material
Available for Download
rar
v7mhci195aux.rar (34.2 MB)
Supplemental video
- Ariano, Raffaele, Marco Manca, Fabio Paternò, and Carmen Santoro. 2022. Smartphone-based augmented reality for end-user creation of home automations. Behaviour & Information Technology, published online: 09 Jan 2022, pages 1--17, Taylor and Francis, https://doi.org/10.1080/0144929X.2021.2017482Google Scholar
- Narges Ashtari, Andrea Bunt, Joanna McGrenere, Michael Nebeling, and Parmit K. Chilana. 2020. Creating Augmented and Virtual Reality Applications: Current Practices, Challenges, and Opportunities. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). Association for Computing Machinery, New York, NY, USA, 1--13. https://doi.org/10.1145/3313831.3376722Google ScholarDigital Library
- Bak, Nayeon, Byeong-Mo Chang, and Kwanghoon Choi. "Smart Block: A visual block language and its programming environment for IoT." Journal of Computer Languages 60 (2020): 100999.Google ScholarCross Ref
- Barkhuus, Louise, and Anind Dey. "Is context-aware computing taking control away from the user? Three levels of interactivity examined." In UbiComp 2003: Ubiquitous Computing: 5th International Conference, Seattle, WA, USA, October 12--15, 2003. Proceedings 5, pp. 149--156. Springer Berlin Heidelberg, 2003.Google Scholar
- Brackenbury, Will, Abhimanyu Deora, Jillian Ritchey, Jason Vallee, Weijia He, Guan Wang, Michael L. Littman, and Blase Ur. "How users interpret bugs in trigger-action programming." In Proceedings of the 2019 CHI conference on human factors in computing systems, pp. 1--12. 2019.Google Scholar
- Brich, Julia, Marcel Walch, Michael Rietzler, Michael Weber, and Florian Schaub. "Exploring end user programming needs in home automation." ACM Transactions on Computer-Human Interaction (TOCHI) 24, no. 2 (2017): 1--35.Google ScholarDigital Library
- Castelli, Nico, Corinna Ogonowski, Timo Jakobi, Martin Stein, Gunnar Stevens, and Volker Wulf. "What happened in my home? an end-user development approach for smart home data visualization." In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 853--866. 2017.Google Scholar
- Cena, Federica, Cristina Gena, Claudio Mattutino, Michele Mioli, Barbara Treccani, Fabiana Vernero, and Massimo Zancanaro, Incorporating Personality Traits in User Modeling for EUD, Proceedings 3rd International Workshop on Empowering People in Dealing with Internet of Things Ecosystems, CEUR-WS.org, Vol. 3172, 2022.Google Scholar
- Meghan Clark, Mark W. Newman, and Prabal Dutta. 2022. ARticulate: One-Shot Interactions with Intelligent Assistants in Unfamiliar Smart Spaces Using Augmented Reality. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 6, 1, Article 7, (March 2022), 24 pages. https://doi.org/10.1145/3517235Google ScholarDigital Library
- Fulvio Corno, Luigi De Russis, and Alberto Monge Roffarello. 2019. RecRules: Recommending IF-THEN Rules for End-User Development. ACM Trans. Intell. Syst. Technol. 10, 5, Article 58 (September 2019)Google ScholarDigital Library
- Fulvio Corno, Luigi De Russis, and Alberto Monge Roffarello. 2020. HeyTAP: Bridging the Gaps Between Users' Needs and Technology in IF-THEN Rules via Conversation. In Proceedings of the International Conference on Advanced Visual Interfaces, 1--9. https://doi.org/10.1145/3399715.3399905Google ScholarDigital Library
- Corno, Fulvio, Luigi De Russis, and Alberto Monge Roffarello. "From users' intentions to IF-THEN rules in the Internet of Things." ACM Transactions on Information Systems (TOIS) 39, no. 4 (2021): 1--33.Google ScholarDigital Library
- Joëlle Coutaz, James L. Crowley: A First-Person Experience with End-User Development for Smart Homes. IEEE Pervasive Comput. 15(2): 26--39 (2016)Google ScholarDigital Library
- Deerwester, Scott, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. "Indexing by latent semantic analysis." Journal of the American society for information science 41, no. 6 (1990): 391--407.Google ScholarCross Ref
- Demeure, Alexandre, Sybille Caffiau, Elena Elias, and Camille Roux. "Building and using home automation systems: a field study." In International Symposium on End User Development, pp. 125--140. Springer, Cham, 2015.Google Scholar
- Desolda, Giuseppe, Carmelo Ardito, and Maristella Matera. "Empowering end users to customize their smart environments: model, composition paradigms, and domain-specific tools." ACM Transactions on Computer-Human Interaction (TOCHI) 24, no. 2 (2017): 1--52.Google ScholarDigital Library
- Fogli, Daniela, Matteo Peroni, and Claudia Stefini. "Smart home control through unwitting trigger-action programming." In Proc. 22nd Conf. Distrib. Multimedia Syst.(DMS), pp. 194--201. 2016.Google Scholar
- Fogli, Daniela, Matteo Peroni, and Claudia Stefini. "ImAtHome: Making trigger-action programming easy and fun." Journal of Visual Languages & Computing 42 (2017): 60--75.Google ScholarDigital Library
- Gallo, Simone, and Fabio Paterno. "A Conversational Agent for Creating Flexible Daily Automation." In Proceedings of the 2022 International Conference on Advanced Visual Interfaces, pp. 1--8. 2022.Google Scholar
- Ghiani, Giuseppe, Marco Manca, Fabio Paternò, and Carmen Santoro. "Personalization of context-dependent applications through trigger-action rules." ACM Transactions on Computer-Human Interaction (TOCHI) 24, no. 2 (2017): 1--33.Google ScholarDigital Library
- Gunawardana, Asela, Guy Shani, and Sivan Yogev. "Evaluating recommender systems." In Recommender systems handbook, pp. 547--601. Springer, New York, NY, 2022.Google Scholar
- Haines, Will, Melinda Gervasio, Aaron Spaulding, and Bart Peintner. "Recommendations for end-user development." In Proceedings of the ACM RecSys 2010 Workshop on User-Centric Evaluation of Recommender Systems and Their Interfaces (UCERSTI), pp. 42--49. 2010.Google Scholar
- Valentin Heun, James Hobin, Pattie Maes. 2013. Reality Editor: Programming Smarter Objects. UbiComp (Adjunct Publication) 2013: 307--310J.Google Scholar
- Huang, Justin, and Maya Cakmak. "Supporting mental model accuracy in trigger-action programming." In Proceedings of the 2015 acm international joint conference on pervasive and ubiquitous computing, pp. 215--225. 2015.Google Scholar
- Dongsik Jo and Gerard Jounghyun Kim, ARIoT: Scalable Augmented Reality Framework for Interacting with Internet of Things Appliances Everywhere, IEEE Transactions on Consumer Electronics, Vol. 62, No. 3, August 2016.Google ScholarDigital Library
- Sanghoon Kim and In-Young Ko. 2022. A Conversational Approach for Modifying Service Mashups in IoT Environments. In CHI Conference on Human Factors in Computing Systems (CHI '22), April 29-May 5, 2022, New Orleans, LA, USA. ACM, New York, NY, USA, 16 pages. https://doi.org/10. 1145/3491102.3517655Google ScholarDigital Library
- Amy J. Ko, Brad A. Myers, Mary Beth Rosson, Gregg Rothermel, Mary Shaw, Susan Wiedenbeck, Robin Abraham, Laura Beckwith, Alan Blackwell, Margaret Burnett, Martin Erwig, Chris Scaffidi, Joseph Lawrance, and Henry Lieberman. 2011. The state of the art in end-user software engineering. ACM Computing Surveys 43, 3: 1--44. https://doi.org/10.1145/1922649.1922658Google ScholarDigital Library
- Lau, Jey Han, and Timothy Baldwin. "An empirical evaluation of doc2vec with practical insights into document embedding generation." arXiv preprint arXiv:1607.05368 (2016).Google Scholar
- Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Proceedings of the 31st International Conference on Machine Learning (ICML 2014), pages 1188--1196, Beijing, China.Google Scholar
- Toby Jia-Jun Li, Yuanchun Li, Fanglin Chen, and Brad A. Myers. Programming IoT Devices by Demonstration Using Mobile Apps. In Proceedings of the International, Symposium on End User Development (IS-EUD 2017). Lecture Notes in Computer Science, vol 10303. Springer, Cham. https://doi.org/10.1007/978--3--319--58735--6_1Google Scholar
- Henry Lieberman, Fabio Paternò, Markus Klann, and Volker Wulf. 2006. End-User Development: An Emerging Paradigm. In End User Development, Henry Lieberman, Fabio Paternò and Volker Wulf (eds.). Springer Netherlands, Dordrecht, 1--8. https://doi.org/10.1007/1--4020--5386-X_1Google Scholar
- Mäntyjärvi, Jani, Urpo Tuomela, Ilkka Känsälä, and Jonna Häkkilä. "Context studio: Tool for personalizing context-aware applications in mobile terminals." In Annual Conference of CHISIG, the Computer Human Interaction Special Interest Group of the Ergonomics Society of Australia, OZCHI 2003, pp. 64--73. University of Queensland Press, 2003.Google Scholar
- A. Mattioli, F. Paternò. 2020. A Visual Environment for End-User Creation of IoT Customization Rules with Recommendation Support. Proceedings AVI 2020 Conference, ACM PressGoogle ScholarDigital Library
- A. Mattioli, F. Paternò. 2021. Recommendations for creating trigger-action rules in a block-based environment. Behaviour & Information Technology Journal, Taylor & Francis, Volume 40, 2021 - Issue 10, doi:10.1080/0144929X.2021.1900396Google Scholar
- Knierim, Pascal, Pawel W. Wozniak, Yomna Abdelrahman, and Albrecht Schmidt, Exploring the potential of augmented reality in domestic environments, in: Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services, MobileHCI '19, Association for Computing Machinery, New York, NY, USA, Article 31, 1--12, 2019. doi:10.1145/3338286.3340142.Google ScholarDigital Library
- Rehurek, Radim, and Petr Sojka. "Software framework for topic modelling with large corpora." In Proceedings of the LREC 2010 workshop on new challenges for NLP frameworks. 2010.Google Scholar
- Salovaara, Antti, Andrea Bellucci, Andrea Vianello, and Giulio Jacucci. "Programmable smart home toolkits should better address households' social needs." In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1--14. 2021.Google Scholar
- Ronny Seiger, Maria Gohlke, Uwe Aßmann. 2019. Augmented Reality-Based Process Modelling for the Internet of Things with HoloFlows. BPMDS/EMMSAD@CAiSE 2019: 115--129Google Scholar
- Seiger Ronny; Kühn Romina; Korzetz Mandy & Aßmann Uwe (2021) HoloFlows: modelling of processes for the Internet of Things in mixed reality. Software and Systems Modeling, Available at: https://www.alexandria.unisg.ch/261944/Google Scholar
- Smyth, Barry, and Paul McClave. "Similarity vs. diversity." In International conference on case-based reasoning, pp. 347--361. Springer, Berlin, Heidelberg, 2001.Google Scholar
- Vijay Srinivasan, Christian Koehler, and Hongxia Jin. 2018. RuleSelector: Selecting Conditional Action Rules from User Behavior Patterns. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2Google ScholarDigital Library
- Evropi Stefanidi, Dimitrios Arampatzis, Asterios Leonidis, Maria Korozi, Margherita Antona, George Papagiannakis. 2020. MagiPlay: An Augmented Reality Serious Game Allowing Children to Program Intelligent Environments. In: Gavrilova M., Tan C., Chang J., Thalmann N. (eds) Transactions on Computational Science XXXVII. Lecture Notes in Computer Science, vol 12230. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978--3--662--61983--4_9Google Scholar
- Torres-Ruiz, Miguel, Felix Mata, Roberto Zagal, Giovanni Guzmán, Rolando Quintero, and Marco Moreno-Ibarra. "A recommender system to generate museum itineraries applying augmented reality and social-sensor mining techniques." Virtual Reality 24 (2020): 175--189.Google ScholarDigital Library
- Ur, Blase, Elyse McManus, Melwyn Pak Yong Ho, and Michael L. Littman. "Practical trigger-action programming in the smart home." In Proceedings of the SIGCHI conference on human factors in computing systems, pp. 803--812. 2014.Google Scholar
- Ur, Blase, Melwyn Pak Yong Ho, Stephen Brawner, Jiyun Lee, Sarah Mennicken, Noah Picard, Diane Schulze, and Michael L. Littman. "Trigger-action programming in the wild: An analysis of 200,000 ifttt recipes." In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp. 3227--3231. 2016.Google Scholar
- White, Gary, Christian Cabrera, Andrei Palade, and Siobhán Clarke. "Augmented reality in IoT." In Service-Oriented Computing--ICSOC 2018 Workshops: ADMS, ASOCA, ISYyCC, CloTS, DDBS, and NLS4IoT, Hangzhou, China, November 12--15, 2018, Revised Selected Papers 16, pp. 149--160. Springer International Publishing, 2019.Google Scholar
- Wu, Q., Shen, B., Chen, Y. (2020). Learning to Recommend Trigger-Action Rules for End-User Development. In: Ben Sassi, S., Ducasse, S., Mili, H. (eds) Reuse in Emerging Software Engineering Practices. ICSR 2020. Lecture Notes in Computer Science(), vol 12541. Springer, Cham. https://doi.org/10.1007/978--3-030--64694--3_12Google Scholar
- Xu, Xuhai, Anna Yu, Tanya R. Jonker, Kashyap Todi, Feiyu Lu, Xun Qian, João Marcelo Evangelista Belo et al. "XAIR: A Framework of Explainable AI in Augmented Reality." In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1--30. 2023.Google Scholar
- Yang R., Newman M. W. (2013) Learning from a learning thermostat: lessons for intelligent systems for the home. Proceedings of 2013 ACM conference on Pervasive and ubiquitous computing, 93--102.Google ScholarDigital Library
- Imam Nur Bani Yusuf, Lingxiao Jiang, David Lo, Accurate Generation of Trigger-Action Programs with Domain-Adapted Sequence-to-Sequence Learning, In 30th International Conference on Program Comprehension (ICPC'22), May 16--17, 2022, Virtual Event, USA. ACM, New York, NY, USA, 12 pages. https://doi.org/10.1145/3524610.3527922Google ScholarDigital Library
- Zancanaro, Massimo, Giuseppe Gallitto, Dina Yem, and Barbara Treccani. "Improving Mental Models in IoT End-User Development." HUMAN-CENTRIC COMPUTING AND INFORMATION SCIENCES 12 (2022).Google Scholar
- L. Zhang, W. He, O. Morkved, V. Zhao, M. L. Littman, S. Lu, and B. Ur. 2020. Trace2TAP: Synthesizing Trigger-Action Programs from Traces of Behavior. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 3, Article 104 (September 2020), 26 pages. https://doi.org/10.1145/3411838Google ScholarDigital Library
- Zimmermann, Robert, Daniel Mora, Douglas Cirqueira, Markus Helfert, Marija Bezbradica, Dirk Werth, Wolfgang Jonas Weitzl, René Riedl, and Andreas Auinger. "Enhancing brick-and-mortar store shopping experience with an augmented reality shopping assistant application using personalized recommendations and explainable artificial intelligence." Journal of Research in Interactive Marketing 17, no. 2 (2023): 273--298.Google ScholarCross Ref
Index Terms
- A Mobile Augmented Reality App for Creating, Controlling, Recommending Automations in Smart Homes
Recommendations
Creating Augmented and Virtual Reality Applications: Current Practices, Challenges, and Opportunities
CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing SystemsAugmented Reality (AR) and Virtual Reality (VR) devices are becoming easier to access and use, but the barrier to entry for creating AR/VR applications remains high. Although the recent spike in HCI research on novel AR/VR tools is promising, we lack ...
Haptics in Augmented Reality
ICMCS '99: Proceedings of the IEEE International Conference on Multimedia Computing and Systems - Volume 2An augmented reality system merges synthetic sensory information into a user's perception of a three-dimensional environment. An important performance goal for an augmented reality system is that the user perceives a single seamless environment. In most ...
Augmented reality as perceptual reality
VSMM'06: Proceedings of the 12th international conference on Interactive Technologies and Sociotechnical SystemsAs shown in Paul Milgram et al’s Reality-Virtuality Continuum (1994), Augmented Reality occupies a very unique status in the spectrum of Mixed Reality. Unlike Virtual Reality, which is completely made up of the virtual and has been the most important ...
Comments