Elsevier

Neurocomputing

Volume 126, 27 February 2014, Pages 106-115
Neurocomputing

Activity recognition with android phone using mixture-of-experts co-trained with labeled and unlabeled data

https://doi.org/10.1016/j.neucom.2013.05.044Get rights and content

Abstract

As the number of smartphone users has grown recently, many context-aware services have been studied and launched. Activity recognition becomes one of the important issues for user adaptive services on the mobile phones. Even though many researchers have attempted to recognize a user's activities on a mobile device, it is still difficult to infer human activities from uncertain, incomplete and insufficient mobile sensor data. We present a method to recognize a person's activities from sensors in a mobile phone using mixture-of-experts (ME) model. In order to train the ME model, we have applied global–local co-training (GLCT) algorithm with both labeled and unlabeled data to improve the performance. The GLCT is a variation of co-training that uses a global model and a local model together. To evaluate the usefulness of the proposed method, we have conducted experiments using real datasets collected from Google Android smartphones. This paper is a revised and extended version of a paper that was presented at HAIS 2011.

Introduction

Smart phones, such as Google Android phone and Apple iPhone, have recently incorporated diverse and powerful sensors. The sensors include GPS receivers, microphones, cameras, light sensors, temperature sensors, digital compasses, magnetometers and accelerometers. Because of the small size and the superior computing power of the smartphones, they can become powerful sensing devices that can monitor a user's context in real time. Especially, Android-based smart phones are suitable platforms to collect sensing information because the Android operating system is free, open-source, easy to program, and expected to become a dominant smartphone in the marketplace.

Many context-aware services were introduced and they tried to provide a user with convenience on a mobile phone. For instance, there are social networking applications such as Facebook [1] and MySpace [2]. Foursquare provides a location-based social networking service. It ranks the users by the frequency of visiting a specific location and encourages them to check in the place. Loopt service can recommend some locations for the users, and Whoshere service helps users to share friends' locations. Davis et al. tried to use temporal, spatial, and social contexts to manage multimedia content with a camera phone [3]. Until now, most of the commercial services use only raw data like GPS coordinates.

In the mobile environment, inferring context information becomes an important issue for mobile cooperative services. By capturing more meaningful context in real time, we can develop more adaptive applications to the changing environment and user preferences. Activity recognition technology is one of the core technologies to provide user adaptive services. Many researchers have attempted to infer high-level semantic information from raw data collected in a mobile device. Bellotti et al. developed a Maggiti system which predicted a user's activities (eating, shopping, etc.) and provided suitable services on Windows Mobile platform [4]. Kwapisz et al. studied a method to recognize a user's behavior using cell phone accelerometers [5]. Chen proposed intelligent location-based mobile news service as a kind of location-based service [6]. Other researchers have studied context-aware systems considering efficient energy management. Paek et al. controlled GPS receiver to reduce energy consumption on a smartphone [7]. Ravi et al. predicted battery charge cycle and recommended the charge [8]. Other researchers tried to stop unnecessary functionalities in the context. It is difficult to recognize a user's situation because of uncertainty and incompleteness of context in mobile environment.

Most of the research used various statistical analysis and machine learning techniques such as probabilistic model, fuzzy logic, and case based reasoning. However, it is still difficult to apply them to mobile devices because of two problems. First, although the machine learning techniques are appropriate to deal with vagueness and uncertainty in mobile environment, it is not easy to classify user's activities using only one classifier from incomplete mobile data in a complex environment. The other problem is that many supervised machine learning techniques need hand-labeled data samples for good classification performance. It requires users to annotate the sensor data. In many cases, unlabeled data are significantly easier to come by than labeled ones [9], [10]. These labeled data are fairly expensive to obtain because they require human effort [11]. Therefore, we would like our learning algorithm to be able to take as much advantage of the unlabeled data as possible.

In this paper, we present an activity recognition system using mixture of experts (ME) [12] on Android phone. ME is one of the divide-and-conquer models which are effective solutions to overcome the limitations of mobile environment. The ME is based on several local experts to solve smaller problems and a gating network to combine the solutions from separate experts, each of which is a statistical model for a part of overall problem space. In order to train the ME with labeled and unlabeled data, the global–local co-training (GLCT) method is used. GLCT is one of the variations of co-training [13], and enables us to improve the performance of ME model by minimizing errors which occur due to the unlabeled data. To evaluate the usefulness of the proposed system, we conducted experiments using real datasets collected from Google Android smart phones.

Section snippets

Activity recognition using mobile sensors

There are many attempts to recognize a user's actions and behaviors with mobile sensors. Among all these mobile sensors, the accelerometer is commonly used for activity recognition. All of these Android phones contain tri-axial accelerometers that measure acceleration in all three spatial dimensions. The accelerometers are also capable of detecting the orientation of the device (helped by the fact that they can detect the direction of Earth's gravity), which can provide useful information for

Activity recognition using ME

The proposed activity recognition system consists of three steps: to collect sensor data, preprocess the data and recognize a user's activity. First, after sensor data are collected from sensors on a smartphone, the data are transferred to preprocessing units for extracting features such as mean and standard deviation. The following Eqs. (1), (2), (3) denote the features such as the difference between previous acceleration and current acceleration, average acceleration, and standard deviation

Experiments

This section presents the experiments conducted to evaluate the usefulness of the proposed method on the pattern classification problems. The number of hidden nodes for each neural network is fixed to ten and the learning rate is 0.3. There are two objectives of the experiments. Firstly, it is to prove the usefulness of co-training based on unlabeled data. Secondly, our proposed method shows better performance than semi-supervised self-training as [36].

As shown in Fig. 2, we divide the dataset

Concluding remarks

In this paper, we have proposed an activity recognition system using the mixture-of-experts (ME) model with both labeled and unlabeled data. ME is a variant of divide-and-conquer paradigm, and it is suitable to solve complex problem to recognize a user's activity from uncertain, and incomplete mobile sensor data. The GLCT method is also used to train the ME model. In order to overcome the limitation of the local experts with small amount of training data, we employed the GLCT method to improve

Acknowledgment

This research was supported by the Industrial Strategic Technology Development Program (10044828) funded by the Ministry of Trade, Industry and Energy (MI, Korea), and the ICT R&D Program 2013 funded by the MSIP (Ministry of Science, ICT & Future Planning, Korea).

Young-Seol Lee is a Ph.D. candidate in computer science at Yonsei University. His research interests include Bayesian networks and evolutionary algorithms for context-aware computing and intelligent agents. He received his M.S. degree in computer science from Yonsei University.

References (37)

  • M. Böhmer, G. Bauer, A. Krüger, Exploring the design space of context-aware recommender systems that suggest mobile...
  • J. Paek, J. Kim, R. Govindan, Energy-efficient rate-adaptive GPS-based positioning for smartphones, in: Proceedings of...
  • N. Ravi, J. Scott, L. Han, L. Iftode, Context-aware battery management for mobile phones, in: Proceedings of the Sixth...
  • R.O. Duda et al.

    Pattern Classification and Scene Analysis

    (1973)
  • D. Yarowsky, Unsupervised word sense disambiguation rivaling supervised methods, in: Proceedings of the 33rd Annual...
  • A. Blum, T. Mitchell, Combining labeled and unlabeled data with co-training, in: Proceedings of the Eleventh Annual...
  • R. Jacobs et al.

    Adaptive mixture local experts

    Neural Computation

    (1991)
  • W. Wang et al.

    Analyzing co-training style algorithms

    Lecture Notes in Computer Science

    (2007)
  • Cited by (72)

    • Estimating traffic flow states with smart phone sensor data

      2021, Transportation Research Part C: Emerging Technologies
      Citation Excerpt :

      Liu et al. (2020) mapped GPS trajectory records of the online car-hailing into cells and proposed a fully convolutional model based on semantic segmentation technology to estimate the traffic speed. The accelerometer data has recently been successfully used to detect transportation modes (Lee and Cho, 2014; Vlahogianni and Barmpounakis, 2017; Xu et al., 2011) and study drivers’ aggressive and risky behavior (Predic and Stojanovic, 2015). Choudhury et al. (2017) calculated an initial position from a GPS receiver.

    • CAVIAR: Context-driven Active and Incremental Activity Recognition

      2020, Knowledge-Based Systems
      Citation Excerpt :

      Self-learning methods exploit the starting training set to classify unlabeled data [7], and the most confident predictions are used to update the classifier. Co-learning approaches rely on multiple classifiers trained on different views of the training set, and on a voting mechanism to classify each unlabeled feature vector [21,20]. Based on their confidence, the resulting predictions may be used to update the recognition model of each classifier.

    • A light weight smartphone based human activity recognition system with high accuracy

      2019, Journal of Network and Computer Applications
      Citation Excerpt :

      The other one along with the training partitions were used to test the performance. We performed experiments with time-domain features and classification algorithms used by state-of-the-art human activity recognition systems (Lee and Cho, 2014) (Derawi and Bours, 2013) (Dernbach et al., 2012) (Siirtola et al., 2009). We used following time-domain features: 1) mean, 2) max, 3) min, 4) standard deviation, 5) variance, representing mean, maximum, minimum standard deviation, and variance of activity cycle respectively.

    • Cross-Domain Unseen Activity Recognition Using Transfer Learning

      2022, Proceedings - 2022 IEEE 46th Annual Computers, Software, and Applications Conference, COMPSAC 2022
    View all citing articles on Scopus

    Young-Seol Lee is a Ph.D. candidate in computer science at Yonsei University. His research interests include Bayesian networks and evolutionary algorithms for context-aware computing and intelligent agents. He received his M.S. degree in computer science from Yonsei University.

    Sung-Bae Cho received the B.S. degree in computer science from Yonsei University, Seoul, Korea and the M.S. and Ph.D. degrees in computer science from Korea Advanced Institute of Science and Technology (KAIST), Taejeon, Korea. He was an Invited Researcher of Human Information Processing Research Laboratories at Advanced Telecommunications Research (ATR) Institute, Kyoto, Japan from 1993 to 1995, and a Visiting Scholar at University of New South Wales, Canberra, Australia in 1998. He was also a Visiting Professor at University of British Columbia, Vancouver, Canada from 2005 to 2006. Since 1995, he has been a Professor in the Department of Computer Science, Yonsei University. His research interests include neural networks, pattern recognition, intelligent man–machine interfaces, evolutionary computation, and artificial life. Dr. Cho was awarded outstanding paper prizes from the IEEE Korea Section in 1989 and 1992, and another one from the Korea Information Science Society in 1990. He was also the recipient of the Richard E. Merwin prize from the IEEE Computer Society in 1993. He was listed in Who's Who in Pattern Recognition from the International Association for Pattern Recognition in 1994, and received the best paper awards at International Conference on Soft Computing in 1996 and 1998. Also, he received the best paper award at World Automation Congress in 1998, and listed in Marquis Who's Who in Science and Engineering in 2000 and in Marquis Who's Who in the World in 2001. He is a Senior Member of IEEE and a Member of the Korea Information Science Society, INNS, the IEEE Computational Intelligence Society, and the IEEE Systems, Man, and Cybernetics Society.

    View full text