Abstract
We propose a three-layered context-aware architecture for monitoring activities of daily life in smart home. This architecture provides for the inclusion of functionalities that range from low-level data collection to high-level context knowledge extraction. We have also devised an upper-level ontology to model the context in which the activities take place. This enables having a common activity-related context representation, on which to infer and share knowledge. Furthermore, we have begun to implement a platform that realizes our architecture and ontology, making use of Microsoft’s Lab of Things (LoT) platform, being the preliminary results on this task also described in the paper.
1. Introduction
Recent advances in sensing, networking, and ambient intelligence technologies have resulted in rapid emergence of smart environments. Among these, smart home (SH) has gained a lot of attention for the provision of enhanced quality of life within the home. One of the motivations for smart home research is the significant worldwide increase of an aging population, since the elderly have specific health issues that have to be considered. This population segment is our main focus, even though much of our work is equally usable for other people who face similar difficulties. If, based on SH, we are able to detect and interpret what this population does in their homes, we could think about enhancing their quality of life, prolonging independent living, and reducing caregivers’ necessary time and healthcare costs in general, without losing the safety that continuous and unobtrusive monitoring provides. Research objectives in this area range from low-level data acquisition by sensors up to high-level data integration and inference of context knowledge. In terms of deploying sensors and devices to collect data in smart home, a major impediment is the big effort required to build a system that enables collecting data from various sources and could be reused afterwards. In order to bring down these barriers, Microsoft’s Lab of Things (LoT) platform facilitates large-scale field studies in homes. LoT is a community-based research platform, which solves the challenges of supporting an extensible range of sensors and devices in home, allows scale and diversity of deployments while running stable and repeatable experiments, and enables community to share not only code but also experiments, data, and participants [1].
Activities of daily life (ADLs) in smart home refer to self-care and domestic tasks performed by the people as a routine. The capacity of performing ADLs without assistance from other people can be considered as a reference for the estimation of the independent living level of the elderly [2]. There are different types of activities, but the common ground to all of them is that they should be recognizable as such by a nontechnician (e.g., “preparing a meal,” “taking a bath,” or “watch television while sitting on the sofa”). ADLs modeling is a challenging task for the development of smart home systems due to their unique characteristics. On one hand, a variety of factors should be taken into consideration to describe an activity, such as location, physical object related to it, and time [3]. These factors may change over time, which makes the ADLs modeling more difficult. On the other hand, different individuals may perform the same activity differently (this can be treated as user’s preference of performing an activity), and the same individual could carry out several activities in sequence or even concurrently rather than performing them separately, which may produce “composite activities” [4]. Thus, an activity may be represented in diverse forms that depend on individuals’ preferences and particularities, leading to various ADLs variants. The models should adequately capture this ADLs diversity. As a solution to this, context plays a core role in our proposed ADLs modeling.
Context is defined as “any information that can be used to characterize the situation of an entity. An entity is a person, place, or object that is considered relevant to the interaction between a user and an application, including the user and applications themselves” in [5]. Zimmermann et al. [6] extend this definition with five categories of context information: individuality context, time context, location context, activity context, and relations context. This enriches the information representation of context. Based on this, the context-aware system uses context to provide relevant information and services adapted to the user’s task. In order to deal with the diverse characteristics of ADLs, the context-aware infrastructure can be used independently of the hardware platform, operating system, and programming language to support a greater range of devices and applications [7]. One challenge of the context-aware infrastructure is to exploit the changing environment with a new class of applications that are aware of the context in which they run and then react to changes in the environment [8].
In this paper, we propose a three-layered context-aware architecture for monitoring elderly ADLs in smart home (see Figure 1). This architecture allows for the inclusion of the necessary functions to represent in a meaningful manner the context in which those ADLs take place and includes the definition of an upper-level ontology to facilitate reasoning and sharing. Besides, we have started to implement a platform, built in part over LoT, to realize and validate the architectural model. The main contributions of our proposal are twofold: (1) the adoption of LoT platform eases the connection of sensors and devices and facilitates the sharing of results and data with the smart home community; and (2) the context ontology is proposed as an approach to model the attachment of the sensor data and context knowledge to the abstraction level representation.
The remainder of the paper is structured as follows. Section 2 summarizes the existing related work. Section 3 describes the three-layered context-aware system architecture, explaining the functionalities provided by each layer. Section 4 includes the characterization of ADLs, the ontology proposed for context representation, and the inference mechanisms based on semantic logic reasoning. Section 5 summarizes the initial experiments implementation and setup. Finally, Section 6 concludes the paper and discusses the future work.
2. Related Work
A number of ADLs monitoring systems in smart home have been developed to satisfy various application requirements. Pulkkinen et al. [9] described a platform for monitoring daily activities by using different types of sensors for elderly people with diabetes at home and helping in detecting if the patients are following the recommended exercise and diet routines. Lara et al. [10] proposed Centinela, a system that continuously monitors five activities (walking, running, sitting, ascending, and descending) by using a single sensing device and a mobile phone. The system consisted of a portable and unobtrusive data collection platform to provide real-time human activity monitoring. Suryadevara et al. [11] developed an intelligent home monitoring system to detect behavioral changes and forecast how the elderly will act. Han et al. [12] proposed a four-layer healthcare framework to predict the risk of depression and diabetes by monitoring long-term disease related activities. All these projects and applications were developed each for specific experiments, without reusability and scalability as the main concern. We built the infrastructure based on LoT platform to bring down this barrier and facilitate research results and data sharing.
In current literature, there exist various ontology-based context models that aim to represent context knowledge in smart environments. Kim and Chung [13] developed an ontology-based healthcare context information model to provide intelligent healthcare services by considering medical references and service environments. The health context data in their study, such as individual data, location data, medical data, and activity data, were obtained by using ubiquitous network and a radio frequency identification system. Lee and Kwon [14] modelled contexts which are generally distributed in smart home environments using several ontology-based models. These ontology-based models included user domain, home domain, social context, and function management ontologies. Based on these context models, a fully personalized healthcare service to a specific user can be provided. Park et al. [15] presented CASS, a context-aware simulation system, to generate the context information associated with virtual sensors and devices in a smart home domain. All these ontologies use specific concepts and statements tailored to concrete application areas. Because of this concreteness, they lack the ability of facilitating knowledge sharing and interoperability among various smart environments applications in an open and extensible manner.
The relevance of activity monitoring for healthcare services in smart home domain has also been explored from the perspective of working with a collection of different information sources. McAvoy et al. [16] proposed an upper-level context ontology for smart environments, which encompassed some important context-related factors such as user, object, location, event, natural element, and temporal characteristics, in a logical and consistent manner. Zhang et al. [17] presented an ontological approach, which can support both distributed context concepts, such as person, event, activity, location, and temporal characteristics, and advanced temporal reasoning capabilities to model ADLs in an assisted living environment. Chen et al. [18] proposed a semantic-enabled agent-based novel approach, which made use of assistive agents for situation comprehension and ADLs recognition, to enhance situation-aware assistive living. Okeyo et al. [19] described in detail an ontological activity modeling, which established relationships between activities and their involved entities, and temporal activity modeling, which defined relationships between the constituent activities of a composite activity. Meditskos et al. [20] proposed an activity patterns ontology to formally represent the relationships that drive the derivation of complex activities in terms of the activity types and temporal relations that need to be satisfied. This activity patterns ontology can serve as a metamodel over domain activity classes by capturing the structural notions of atomic activities as well as complex activities. These works were more focused on activities modeling; however, they did not take user profiles into account during the inference process. User profiles would allow the development of context-aware applications that adjust proactively by taking into account both these preferences and the particular situation. In terms of user profile modeling, Chen et al. [21] created unified explicit ontological activity models to represent specific user ADLs profiles by creating activity instances in terms of the users’ preferences and their specific manner of performing ADLs. The generated user profiles can later be used for realization of personalized ADLs assistance. Skillen et al. [22] created ontological user models for personal information representation and a rule-based mechanism for context-aware service personalization in changing intelligent pervasive environments. In their approach, both static user context, such as name, preference, and current location, and dynamic user context, such as activity and health condition, were considered to provide Help-on-Demand services deployed on a smart phone.
Inspired by this previous work, we build on semantic web technologies to reach context representation and inference related to the detection of ADLs that are carried out in home. The proposed context-aware system tackles the problem of the variability in ADLs modeling by following a holistic approach. We take into account interfacing through daily objects, inferencing mechanisms, sophisticated modeling tools, new semantic approaches, and consideration of modeling user preferences as part of the system.
3. Three-Layered Context-Aware System Architecture
The proposed context-aware system infrastructure aims to collect and manage the context data from heterogeneous sources and then process it to obtain the required context information suitable to target applications and cloud storage. The layered architecture is depicted in Figure 1 (this figure includes some specific terms from Microsoft’s LoT that will be further explained in Section 5 of the paper).
We define the following three layers in the architecture:(i)The context-sensing layer: it is the layer where context is sensed and collected through the heterogeneous sensors and devices embedded in the smart home environment or worn by the user.(ii)The context management layer: it is the layer where the captured data coming from the context-sensing layer is preprocessed, interpreted by the context ontology model, and inferred by the context reasoner.(iii)The application layer: it is the layer which serves as an interface for the users to access the context information provided by the context management layer.
Each layer’s core components together with their detailed functionalities are presented in the following.
3.1. The Context-Sensing Layer
The context-sensing layer consists of both physical hardware (e.g., environmental sensors, accelerometers, and door switches) and network components. This layer is responsible for obtaining raw context information from various sources.
This layer contains two core modules: the device connectivity module and the device functionality module. The device connectivity module deals with the discovery and association of sensors and devices to make them available to the platform. Devices and sensors are in general capable of connecting through multiple paradigms, such as Wi-Fi, Bluetooth, or proprietary protocols. Once discovered, a handle to the device is passed to the device functionality module.
The device functionality module takes the handles provided by the device connectivity module and turns them into high-level APIs that are independent of the low-level details of sensors and devices. It basically constitutes a driver. Then the captured contexts, such as temperature from embedded sensors and door status from door switches, are handled to the context management layer.
3.2. The Context Management Layer
The context management layer works on low-level context-sensing information to provide context abstraction for higher-level context manipulation. It consists of three collaborating components. (1) The data preprocessing component preprocesses the data collected from the context-sensing layer to build a context knowledge base. (2) The context ontology model abstracts ADL-related contexts from heterogeneous sources and converts them to formalized representations in order to facilitate the interoperability and reusability. (3) The context reasoner provides logic reasoning services (e.g., deriving high-level contexts from low-level ones) for ADL-related contexts processing via the reasoning engine. In the following, we describe the functionalities of each of these components:(i)Data preprocessing component: in general, different sensors are needed simultaneously to correctly identify the different activities, and each of them may provide its measurements (i.e., the raw data) in a different format. Raw data by itself is therefore of little (or no) use for the activity detection algorithms. The preprocessing of this data is a must in order to obtain useful and significant information for the application. This preprocessing includes methods related to data cleaning, transformation, segmentation, and reduction, with the objective of converting data into the proper formats. For example, the filtering method is commonly used to eliminate the constant component due to gravity in 3D acceleration data.(ii)Context ontology model: this component provides semantic capabilities by using ontology engineering to model ADL-related context information, such as user, location, interacting objects, and time. The context ontology model aims to provide a homogeneous view over the heterogeneous data in order to enable data sharing, reuse, and multiple applications integration. This context information is stored in the context database. The ADLs ontology we have devised allows the markup of heterogeneous data to generate semantic content, related to both activity modeling and context representation.(iii)Context reasoner: the context reasoner is built upon the context representation and the functionalities of the context ontology model. Its purpose is to extract and define the abstract context information that cannot be directly detected by sensors and devices (e.g., what is the user doing? where is the user located?), in order to provide advanced processing services. This is achieved by using the reasoning engine, supporting the inference of high-level contexts from their corresponding basic contexts, in a process that depends on the availability of collected data as well as the system requirements. We devote Section 4 to the detailed description of the context ontology modeling and reasoning.
3.3. The Application Layer
The application layer makes use of different levels of context information provided by the context-sensing layer and the context management layer. The user interface is placed between the mobile users and the interpreted context information, relieving mobile users from the requirement to directly deal with context management. The ADLs monitoring application provides insight into the elderly’s health status to the caregivers/experts and facilitates the provision of assistance when it is necessary. All collected and processed data can be uploaded to a cloud storage (e.g., Azure) for further research by the research community.
4. Modeling of ADL-Related Context and Reasoning
In this section, we analyze the characterization of ADLs monitored for the elderly, we describe the ontology-based approach chosen for ADLs modeling, and we discuss the context reasoning methods.
4.1. ADLs Analysis
ADLs are used in smart home as an umbrella term encompassing self-care and domestic activities or tasks that the user undertakes routinely. Self-care activities include sleeping, bathing, dressing, eating or taking medicine, and domestic tasks, preparing meals, housekeeping, or watering plants.
The understanding of context information of an activity is critical in activity modeling. Within a smart home environment, there exist many types of contextual information that can be used to characterize an activity, such as interaction with objects, location, and time. The term object refers to the relevant participant in an activity, either a person or a physical object; for example, if the TV set is the object, it can be inferred that we are dealing with the “watching TV” activity. Location is the specific place where an activity occurs; for example, “preparing meals” takes place in the kitchen and “taking a shower” occurs in bathroom. Moreover, the location can help to discard some objects outside the target place. Time is another key characteristic for activity description, since the user may perform an activity in different times but some activities only occur at specific times of the day. For example, the activity “preparing breakfast” usually occurs in the morning. Other contextual information potentially useful for characterizing activities includes human posture, temperature, humidity, and speed. For instance, currently, most of the intelligent activities monitoring systems detect “sleeping” when bed pressure sensors are triggered. However, there is a considerable uncertainty in the “sleeping” recognition defined this way, since the sensors may be triggered by the user not only while sleeping but also while sitting on the bed. Consequently, human posture such as “standing,” “sitting,” and “lying down” may be helpful to recognize the activity more accurately.
According to the complexity and time interval, ADLs could be modelled at multiple levels of granularity. For example, the coarse-grained activity “preparing drinks” can be split into fine-grained activities such as “preparing tea” and “preparing coffee.” Meditskos et al. [23] define the specialization of activities differently. As an example, in their activity pattern, the activity “night sleep” is defined as the overall night sleep activity of a person and the activity “out of bed” is detected when the person gets out of bed. With the addition of context description and activity type interpretation, an “out of bed” activity can be further specialized as a “bed exit” activity, which refers to the “out of bed” activity when it occurs during the “night sleep.” Furthermore, there exist temporal relations between activities. Some composite activities can be performed by a sequence of simple activities, and activities may occur one after the other, concurrently or interleaved with each other. These complex temporal relationships between activities can be modelled based on qualitative temporal knowledge, for example, sequential, concurrent, and interleaved relationships, to represent composite activities. Table 1 shows the detailed activity temporal relationships based on Allen’s temporal relationships definition [24], which positions intervals by comparing their starting and ending points to support qualitative temporal knowledge representation. In terms of the temporal intervals, the activities and their relations can be modelled as sequential activities, concurrent activities, and interleaved activities:(i)Sequential activities: sequential activities take place whenever one activity is performed either before or after another. The before/after and meets/metBy relations are used to associate sequential activities with their temporal intervals. It is worth mentioning that in the case of the before/after relation there is a gap between the two intervals, and the meets/metBy relation could be seen as a specification of the before/after relation when that gap equals zero. For example, the user may take a medicine before performing the activity “sleeping.”(ii)Concurrent activities: concurrent activities occur at the same time; thus they share the same time intervals fully or partially. There are nine relations used to model concurrent activities: overlaps/overlappedBy, starts/startedBy, during/contains, finishes/finishedBy, and equals. The overlaps/overlappedBy relation is used to model activities that have partially shared temporal intervals with different start and end points. The starts/startedBy relation indicates that two intervals have the same start time but one is longer. Conversely, the finishes/finishedBy relation relates to the case of an activity that finishes at the same time as another activity with longer time interval. The during/contains relation shows that the interval of a composite activity encloses the interval of its composing activity. The equals relation indicates that two activities take place during the exact same time interval. Concurrent activities take place either when one user performs two different activities simultaneously or when multiple users perform activities at the same time. For example, one user can perform “watching TV” and “using the computer” at the same time, or two users can perform “drinking tea” together.(iii)Interleaved activities: interleaved activities have time intervals that “preempt” one another, which indicates that an activity may occur within a long and complex activity, with the longer interval containing the shorter one. The during/contains relation is also used to describe the interleaved activities, but, differently from concurrent activities, the activity with the shorter interval is not a component of the other. For example, the user may use the telephone or drink water at intervals that occur during the activity “prepare meals.”
4.2. Design Choices
In consistence with the aforementioned ADLs analysis in smart homes, there are several goals that the proposed approach should aim for. First, the related contextual information should be encoded in the context ontology model to characterize ADLs. Second, different types and granularities of activities, together with their complex temporal relationships, should be supported by the context ontology model. Third, the context ontology model should be flexible enough to accommodate the imperfect inputs and changing activity variations. Fourth, the context ontology model should be interoperable and reusable to achieve a better degree of knowledge sharing within different applications.
To achieve our goals, we have conducted a research on previously existing formalized foundational ontology patterns and propose our ADLs ontology model that combines the following:(i)An upper-level ontology: we chose the DOLCE + DnS Ultralite (DUL) [25] as an upper-level ontology. It is a broadly reused formalized ontology and contains common sense concepts, which enables including interoperability and reusability into our proposed ontology. The DUL ontology provides a set of upper-level concepts to foster interoperability among many middle and lower level ontologies. Moreover, many ontology-based frameworks have been built on top of it or aligned with it.(ii)A domain ontology to characterize activity-related concepts: a domain ontology serves as a core vocabulary for the basic activity contextual information representation. We chose the Simple Event Model (SEM) [26] as the basis of our ADLs domain ontology. The SEM is built on DOLCE + DnS Ultralite ontology. It takes main aspects (e.g., actor, place, and time) of an activity into consideration and allows the developer to integrate external ontologies for a more detailed description of an activity, such as the Friend of a Friend (FOAF) ontology for user modeling, the Sensor and Sensor Network (SSN) ontology for sensor modeling, and the Semantic Web Rule Language (SWRL) temporal ontology for temporal context modeling.(iii)An ontology to describe the composite activities and their temporal relationships: we chose the Event Model F [27] as the basis of activity description model. The participation pattern of F enables formally describing the participation of objects in activities. Mereology pattern considers activities at different abstract levels and enables expressing the relations as composition of activities, and the formalization of Allen’s Time Calculus can be integrated to describe the relative temporal relations among different activities. Moreover, it is aligned with the DOLCE + DnS Ultralite ontology.
4.3. ADLs Ontology Model
The proposed ADLs ontology is constructed by using Protégé ontology editor and represented in Web Ontology Language (OWL). OWL is a semantic web ontology language based on Description Logic (DL). It contains a set of constructors for specifying conceptual structures and relationships (e.g., equivalent, disjoint, and subsumption) as well as supporting inference and reasoning.
Figure 2 shows a fragment of classes, object properties, and data properties in our proposed ontology. The ADLs classes are described by a number of properties; the object properties and data properties are utilized to interlink classes for creating relationships. This ontology enables modeling knowledge at two levels of abstraction. One is obtaining raw data and generating related contextual information from context entities. The other is deducing the high-level activity-concerned information from the raw data and contextual information. The sensors and devices deployed in smart home environments capture the detailed signals of an activity when triggered. These data are received and stored for further processing. Moreover, the ontology enriches these signals by adding semantic metadata to the raw sensor data. Then the semantic contexts as well as sensor data are potentially generated to obtain the high-level context on which to construct smart home applications.
4.4. Context Reasoning
There are mainly two types of context reasoning: ontology reasoning and user-defined rule-based reasoning. Ontology reasoning uses constructs, such as equivalent, disjoint, and subsumption relations, to check class consistency and assert interontology relationships. For example, if the activity sleep occurs in the bedroom, since the bedroom is a part of house and the object property locatedIn exhibits a transitive property, “the activity sleep occurs in the house” can be inferred.
Due to the inability of OWL to express complex rule formations, the Semantic Web Rule Language (SWRL) is used to define rules and add them to the OWL-based ontology. The SWRL-based rules are constructed from an antecedent and a consequent and can be used together with OWL constructors to ensure decidability, soundness, and completeness in reasoning. After the SWRL rules are created, they may be used for inference and reasoning by using a reasoning tool (e.g., Pellet). For example, if activity has an interval and activity has an interval , and the finish time of is before the start time of , then the fact “ is before ” is inferred; moreover, according to the composite activity analysis on Allen’s Time Calculus, it is inferred that activity and activity take place sequentially and activity takes place before activity . This rule is illustrated in the following:
hasValidPeriod (?X, ?pX) ∧ hasValidPeriod (?Y, ?pY) ∧ hasFinish (?pX, ?fX) ∧ hasStart (?pY, ?sY) ∧ before (?fX, ?sY) → before (?pX, ?pY)
5. Initial Implementation and Result
We adopt the LoT research platform, developed by Microsoft, as a base to construct our system infrastructure for monitoring ADLs in smart home. The LoT research platform builds on top of HomeOS [28], which provides standard user interface to configure the Home Hub, enables remote access from web browser, and eases the tasks of managing in-home hardware and writing applications. The LoT platform runs HomeOS on a dedicated computer, the Home Hub, which is utilized to interact with the in-home sensors and devices and host the target applications. This platform also provides a common framework to write and execute applications by utilizing the functionalities of deployed sensors and devices, providing field deployments with a set of capabilities including uploading application data obtained in home to cloud storage, remote monitoring of system health, and remote updating of applications if needed [29].
Regarding the context-sensing layer, in our experiment we use Arduino Uno boards to connect with sensors deployed in smart home. The so-called scout is responsible for locating connected devices and making them available to the platform (serving as the device connectivity module). We have written an Arduino scout to discover the Arduino device on its serial port; once the Arduino device is found, the scout hands over the target device to the device functionality module.
Regarding the device functionality module, the driver implemented is responsible for facilitating the communication between the Arduino device and the Home Hub. The driver exposes the functionalities of the Arduino device by using a service abstraction called role, which contains a set of operations that can be invoked. We have written the Arduino driver to allow for the request and reception of data from the connected sensors.
To perform an initial validation of our approach, we have developed an example device with an Arduino Uno board and an associated LoT application supporting a temperature sensor to measure the temperature and a passive infrared (PIR) sensor to detect motion and presence of humans. The open-source Arduino Software (IDE) is used to write code and upload it to the Arduino board, whose main functions are sending commands from the Home Hub and receiving the data collected from the temperature sensor and PIR sensor. The Arduino scout and driver have been written in C#.
As we can see from Figure 3, when the Arduino scout runs through the DashBoard of the Home Hub, the Arduino device called “HomeOSArduinoDevice_001” can be found and added in “Add Devices” page (Figure 3(a)); then the Arduino driver exposes the functionality of the device in order for the Arduino application to communicate with the device (Figure 3(b)). We also preprocess the analog output of the PIR sensor. When the PIR sensor is triggered, the output is transformed into “presence,” and when the PIR sensor is not triggered, the output is transformed into “absence” (Figure 3(c)). These two states correspond to the presence or absence (respectively) of the user in the area that falls within the PIR sensor range. This contextual information is susceptible to be further used by our ADLs ontology model.
(a)
(b)
(c)
6. Conclusion and Future Work
In this paper, we propose and describe a three-layered context-aware system infrastructure based on Microsoft’s LoT platform for monitoring ADLs in smart home. This infrastructure provides functionalities that range from sensors and devices connection for raw data acquisition to high-level context knowledge extraction and inference in different layers. We have implemented a formal context model based on OWL to represent the ADL-related contextual information. The adoption of a generic upper-level DOLCE + DnS Ultralite ontology facilitates the interoperability and reusability of our context ontology model to achieve a better degree of knowledge sharing among different applications. The initial experiment prototype, even though simple, lays an architectural foundation for our further research.
We are currently focusing on integrating the presented ontology model into our context-aware system and extending the model with inference capacities. The future work includes deploying a set of sensors and devices and testing our application in the Accessible Smart Home of Universidad Politécnica de Madrid.
Competing Interests
The authors declare that they have no competing interests.
Acknowledgments
The authors would like to thank the Telematics for Knowledge and Information Society research group (T>SIC) of Universidad Politécnica de Madrid for its support to this research. The authors also acknowledge the China Scholarship Council that has partially supported the research described in this paper.