RiSH: A robot-integrated smart home for elderly care

https://doi.org/10.1016/j.robot.2017.12.008Get rights and content

Highlights

  • Introduces an overall concept of robot-integrated smart homes (RiSHs) for elderly care.

  • Proposes the hardware structure and software architecture of the RiSH.

  • Implements basic services and applications to demonstrate the operation of the RiSH.

  • Develops sound-based activity monitoring, fall detection and rescue using the robot.

  • Evaluates the RiSH functions through experiments with human subjects in the RiSH testbed.

Abstract

This article presents the development of a robot-integrated smart home (RiSH) which can be used for research in assistive technologies for elderly care. The RiSH integrates a home service robot, a home sensor network, a body sensor network, a mobile device, cloud servers, and remote caregivers. A layered architecture is proposed to guide the design and implementation of the RiSH software. Basic service functions are developed to allow the RiSH to recognize human body activity using an inertial measurement unit (IMU) and the home service robot to perceive the environment through audio signals. Based on these functions, we developed two low-level applications: (1) particle filter-based human localization and tracking using wearable motion sensors and distributed binary sensors; (2) Dynamic Bayesian Network-based human activity recognition using microphones and distributed binary sensors. Both applications extend the robot’s perception beyond its onboard sensors. Utilizing the low-level applications, a high-level application is realized that detects and responds to human falls. We conducted experiments in our RiSH testbed to evaluate auditory perception services, human body activity recognition, human position tracking, sound-based human activity monitoring, and fall detection and rescue. Twelve human subjects were asked to conduct daily activities in the testbed and mimic falls. All data of their movement, body activities, and sound events were collected by the robot. Human trajectories were estimated with a root mean square error of less than 0.2 m. The robot was able to recognize 37 human activities through sound events with an average accuracy of 88% and detect falling sounds with an accuracy of 80% at the frame level. The experiments show the operation of the various components in the RiSH and the capabilities of the home service robot in monitoring and assisting the resident.

Introduction

The elderly population around the world is steadily increasing. The number of people 60 years old and older increased to almost 900 million in 2015 and is forecasted to reach 2 billion by 2050 [1]. Older adults are an important asset to society and need to be cared for in age-friendly physical and social environments. Although services such as adult day care, long term care, and nursing homes can provide the elderly with healthcare, nutritional, social, and other daily living support, the feeling of independence is lost. Elders would prefer to stay in the comfort of their home where they feel more confident than moving to any expensive adult care or healthcare facilities. Hence, if older adults are able to complete self-care activities on their own, it will encourage them to maintain independence and provide them with a sense of accomplishment and ability to enjoy independence longer [2]. The best way to support them is to provide a physical environment that promotes active aging through the use of innovative technologies, such as smart homes and assistive robots.

A smart home environment is defined as a ubiquitous computing application that is able to provide users with context-aware assistive services and home automation. Moreover, smart homes provide comfort, healthcare, and security services to their inhabitants. On the other hand, mobile robots have come into human environments in recent years. They have been used in many places such as factories, offices, hospitals, and homes. For the elderly who live independently in their own residence, home service robots can act as a tool to serve the human, an avatar to represent the caregiver, and a social companion to collaborate and interact with the elderly. In a robot-integrated smart home (RiSH), the home service robots can utilize the smart home sensor networks just like their own sensors, enabling them to better assist the elderly and collaborate with remote caregivers. Therefore, RiSHs would be the perfect use of technology to achieve the goal of caring for the elderly in their own home.

This paper aims to present the overall ideas of a RiSH and introduce a RiSH testbed to support future research in assistive technologies for elderly care. The rest of this paper is organized as follows. Section 2 reviews related works in home service robots, smart homes, and robot-integrated smart homes. Section 3 presents the overall design of the RiSH for elderly care. Section 4 describes the hardware design of the RiSH testbed. Section 5 explains the software architecture of the RiSH. Section 6 presents the implementation of several key basic services. Section 7 details the development of both low-level and high-level applications for the RiSH. Section 8 describes the experiments and gives the results. Section 9 concludes the paper and also discusses the potential future work.

Section snippets

Related works

In recent years there has been much interest in developing robotic technologies for elderly care. Several home service robots or personal robots have been made commercially available, such as Aibo [3], Care-o-Bots [4], and Paro [5]. In academia, researchers have developed many robots for domestic environments, such as Johnny [6], European CompanionAble project’s Hector [7], IRT’s home-assistant robot [8], and Hobbit robot [9]. Those robots were equipped with various functions such as mapping,

Overall concept of RiSHs for elderly care

A smart assistive living environment is expected to provide not only living comfort but also in-home services as human caregivers usually do. Such services include assisting daily activities, providing healthcare, and meeting the need of socialization. Thus, the future RiSHs should provide the elders these services in their own home environments. In this section, we highlight the challenges of elderly care, then propose the overall concept and architecture of RiSHs.

Older adults who live

Hardware design of the RiSH testbed

In this section, we present the hardware design of the RiSH testbed, which is shown in Fig. 2. This testbed consists of a RiSH prototype and an experimental infrastructure. The RiSH is organized according to the overall concept as discussed in the above section. The smart home is connected to the cloud server through the home gateway. The remote caregiver can provide remote daily care services to the elderly as well as collaborate with the home service robot to take care of them. The

Software architecture of the RiSH

In this section, we present the software design of the RiSH. Recently, software architectures of assisted-living home environments have been developed by several groups, such as agent-based architecture [39], logical architecture [40], MAS (Multi-Agent System) architecture [41], and context-aware architecture [42]. However, these architectures are hard to be adopted in the RiSH, mainly because they are designed for systems that are homogeneous and do not provide a comprehensive software

Basic services

This section presents key functions in the service layer, which include auditory perception, coarse human localization, and wearable monitoring services.

Applications for the RiSH

The applications are developed based on the services from the service layer. We first present two low-level applications of monitoring, which are human position tracking and activity monitoring. Then, a high-level application is implemented which focuses on fall detection and rescue.

Experiments and results

We conducted physical experiments to test and evaluate the theoretical framework using the RiSH testbed. The test mainly focuses on the following parts: auditory perception services, human body activity recognition, human position tracking, human activity monitoring, and fall detection and rescue. The experiments show the operation of the components in the RiSH and the capabilities of the home service robot in monitoring and assisting the resident.

To evaluate the performance of the proposed

Conclusions and future works

In this work, we proposed a robot-integrated smart home testbed for elderly care and developed a home service robot platform. A layered architecture was proposed for the development of the software platform. Several key services and three applications of the RiSH were implemented to show the operation of the various components in the RiSH and the system as a whole. The home service robot is equipped with the following auditory perception capabilities: sound localization with an error of less

Acknowledgments

This work was supported by the National Science Foundation (NSF) grant CISE/IIS 1231671/IIS 1427345, the Open Research Project of the State Key Laboratory of Industrial Control Technology ICT170314, Zhejiang University, China, the Basic Public Research Program of Zhejiang Province (No. LGF18F030001) and the Shenzhen Overseas High Level Talent (Peacock Plan) Program (No. KQTD20140630154026047).

Ha Manh Do received his B.Sc. degree in Electronics and Telecommunications from Hanoi University of Technology and Science, Vietnam in 1999. He earned his M.S. degree in 2015 and is currently a Ph.D. candidate in Electrical and Computer Engineering at Oklahoma State University. His research interests include home service robots and smart homes for elderly care, auditory perception, natural language understanding, and deep learning.

References (68)

  • H. Gross, C. Schroeter, S. Mueller, M. Volkhardt, E. Einhorn, A. Bley, C. Martin, T. Langner, M. Merten, Progress in...
  • YamazakiK. et al.

    Home-assistant robot for an aging society

    Proc. IEEE

    (2012)
  • FischingerD. et al.

    Hobbit, a care robot supporting independent living at home: First prototype and lessons learned

    Robot. Auton. Syst.

    (2014)
  • G.D. Abowd, A.F. Bobick, I.A. Essa, E.D. Mynatt, W.A. Rogers, The aware home: A living laboratory for technologies for...
  • Gato-Tech smart house,...
  • MozerM. et al.

    The adaptive house

  • DawadiP. et al.

    Smart home-based longitudinal functional assessment

  • OrpwoodR. et al.

    The gloucester smart house for people with dementiauser-interface aspects

  • D.J. Cook, G.M. Youngblood, E.O. Heierman III, K. Gopalratnam, S. Rao, A. Litvin, F. Khawaja, MavHome: An agent-based...
  • DawadiP.N. et al.

    Automated cognitive health assessment using smart home monitoring of complex tasks

    Syst. Man Cybernet. Syst., IEEE Trans.

    (2013)
  • SuryadevaraN.K. et al.

    Wireless sensor network based home monitoring system for wellness determination of elderly

    IEEE Sens. J.

    (2012)
  • CostaA. et al.

    A caregiver support platform within the scope of an ambient assisted living ecosystem

    Sensors

    (2014)
  • LiuP.R. et al.

    A telemedicine system for remote health and activity monitoring for the elderly

    Telemed. J. e-Health

    (2006)
  • BroxvallM. et al.

    PEIS ecology: Integrating robots into smart environments

  • M. Chen, Y. Ma, S. Ullah, W. Cai, E. Song, ROCHAS: robotics and cloud-assisted healthcare system for empty nester, in:...
  • JohnsonD.O. et al.

    Socially assistive robots: A comprehensive approach to extending independent living

    Int. J. Soc. Robot.

    (2014)
  • TortaE. et al.

    Evaluation of a small socially-assistive humanoid robot in intelligent homes for the care of the elderly

    J. Intell. Robot. Syst.

    (2014)
  • MynattE.D. et al.

    Increasing the opportunities for aging in place

  • LehmannH. et al.

    What should a robot do for you ? - Evaluating the needs of the elderly in the UK

    Interactions

    (2013)
  • Hokuyo laser,...
  • ASUS, Xtion PRO LIVE,...
  • Sony PS3 eye camera,...
  • C. Hacks, e-Health sensor platform for arduino and raspberry Pi...
  • The VN-100 rugged IMU,...
  • Cited by (128)

    • Seeking at-home long-term autonomy of assistive mobile robots through the integration with an IoT-based monitoring system

      2023, Robotics and Autonomous Systems
      Citation Excerpt :

      In our work, the integration with the IoT system of the SAR, enabled us to deploy our system in a real-world setting for a longer period of multiple weeks, in full autonomy and with a deeper interplay between the components. The core functionalities of a system with an architecture similar to the one proposed here, that integrates a service robot, a home sensor network, a body sensor network, a mobile device, cloud servers, and remote caregivers are presented in [40]. Differently from us, results obtained in [40] are achieved in a controlled lab test-bed (similarly, in [39], the same authors present a proof of concept of the system, while we present a fully working system).

    View all citing articles on Scopus

    Ha Manh Do received his B.Sc. degree in Electronics and Telecommunications from Hanoi University of Technology and Science, Vietnam in 1999. He earned his M.S. degree in 2015 and is currently a Ph.D. candidate in Electrical and Computer Engineering at Oklahoma State University. His research interests include home service robots and smart homes for elderly care, auditory perception, natural language understanding, and deep learning.

    Minh Pham received the B.Sc. degree in Computer Science from Hanoi University of Science and Technology, Vietnam in 2007. He is currently pursuing his Ph.D. degree in Electrical and Computer Engineering at Oklahoma State University. His research interests include smart homes and wearable computing.

    Weihua Sheng is the Director of the Laboratory for Advanced Sensing, Computation and Control at Oklahoma State University. He received his Ph.D. degree in Electrical and Computer Engineering from Michigan State University in May 2002. He obtained his M.S and B.S. degrees in Electrical Engineering from Zhejiang University, China in 1997 and 1994, respectively. He is the author of more than 170 peer-reviewed papers in major journals and international conferences. His research interests include mobile robotics, wearable computing, human–robot interaction and intelligent transportation systems. He serves as an associate editor for IEEE Transactions on Automation Science and Engineering.

    Dan Yang received the B.S. and M.S. degrees in biomedical engineering from Northeastern University(NEU), Shenyang, China, in 2002 and 2005, respectively, and the Ph.D. degree in Detection and automatic control engineering in NEU, 2009. From 2009 to 2012, she was a Postdoctoral researcher with the Computer and Science Engineering in NEU. From 2013 July to 2014 July, she was a visiting scholar sponsored by China Scholarship Council, with the School of Electrical and Computer Engineering, Oklahoma State University, Stillwater. She is a lecturer with the Department of Intelligent Perception and Electronics Engineering in NEU. She has been actively involved in research projects with the Fundamental Research Funds for the Central Universities of China, and the National Natural Science Foundation of China. Her current research includes wearable computing, physiological signal detection, and biological electromagnetic systems.

    Meiqin Liu received the B.E. and Ph.D. degrees in control theory and control engineering from Central South University, Changsha, China, in 1994 and 1999, respectively. She was a post-doctoral research fellow with the Huazhong University of Science and Technology, Wuhan, China, from 1999 to 2001. She was a visiting scholar with the University of New Orleans, New Orleans, LA, USA, from 2008 to 2009. She is currently a professor with the College of Electrical Engineering, Zhejiang University, Hangzhou, China. She has authored more than 60 peer reviewed papers, including 33 journal papers. Her current research interests include neural network, robust control, multi-sensor network, and information fusion.

    View full text