Scalable and elastic event matching for attribute-based publish/subscribe systems

https://doi.org/10.1016/j.future.2013.09.019Get rights and content

Highlights

  • We propose a scalable event matching service for attribute-based pub/sub model.

  • HPartition adapts to skewed subscriptions and achieves high matching throughput.

  • PDetection adapts to the sudden change of workloads with low latency.

  • We implement a thorough and systematic evaluation of our approach.

Abstract

Due to the sudden change of the arrival live content rate and the skewness of the large-scale subscriptions, the rapid growth of emergency applications presents a new challenge to the current publish/subscribe systems: providing a scalable and elastic event matching service. However, most existing event matching services cannot adapt to the sudden change of the arrival live content rate, and generate a non-uniform distribution of load on the servers because of the skewness of the large-scale subscriptions. To this end, we propose SEMAS, a scalable and elastic event matching service for attribute-based pub/sub systems in the cloud computing environment. SEMAS uses one-hop lookup overlay to reduce the routing latency. Through a hierarchical multi-attribute space partition technique, SEMAS adaptively partitions the skewed subscriptions and maps them into balanced clusters to achieve high matching throughput. The performance-aware detection scheme in SEMAS adaptively adjusts the scale of servers according to the churn of workloads, leading to high performance–price ratio. A prototype system on an OpenStack-based platform demonstrates that SEMAS has a linear increasing matching capacity as the number of servers and the partitioning granularity increase. It is able to elastically adjust the scale of servers and tolerate a large number of server failures with low latency and traffic overhead. Compared with existing cloud based pub/sub systems, SEMAS achieves higher throughput in various workloads.

Introduction

As the real-time requirement of data dissemination becomes increasingly significant in many fields, the emergency applications have received increasing attention, for instance, stock quote distribution, earthquake monitoring  [1], emergency weather alert  [2], smart transportation system  [3], and social networks. Recently, the development of emergency applications demonstrates two trends. One is the sudden change of the arrival live content rate. Take ANSS  [1] as an example, its mission is to provide real-time and accurate seismic information for emergency response personnel. Millions of messages are generated by sensors in a short time when an earthquake happens, while few events are generated if there is no earthquake. The other is the skewness of the large-scale subscriptions. That is, a large number of subscribers demonstrate similar interests. For instance, the dataset  [4] of 297 K users of Facebook shows that the hottest 100 topics together have more than 1.1 million subscribers, while 71% topics have no more than 16 subscribers.

Publish/subscribe (pub/sub) paradigm is a key technology for asynchronous data dissemination that are widely used in the emergency applications. It decouples senders and receivers of the emergency applications in space, time, and synchronization  [5], which enables a pub/sub system to seamlessly expand to massive size. However, traditional pub/sub system faces a number of challenges. Firstly, the system must guarantee real-time event matching capacity when it expands to very large-scale. For instance, Facebook contains billions of users and 684,478 pieces of content are published on average in every minute  [6]. Secondly, the system needs to be elastic to the sudden change of incoming event rate to achieve high performance–price ratio. This is because if a fixed number of servers are deployed in response to the sudden change of incoming event rate, numerous servers are in the idle states delivering few messages in most of the time. Thirdly, the service must be tolerant to the server failures. In emergency applications, a large number of machines and links may be unavailable instantaneously due to hardware errors or operator mistakes, which leads to the loss of events and subscriptions.

Existing pub/sub systems are not adequate to efficiently address all above challenges. In the broker based pub/sub systems  [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], all publishers and subscribers are directly connected to a group of servers, known as brokers. Subscriptions are commonly replicated to all brokers or a part of brokers, so that each broker can match events and forward them to the interested subscribers. However, replicating subscriptions incurs that each event is matched against the same subscriptions for many times, which leads to high matching latency and low scalability when a large number of events and subscriptions arrive. Moreover, it is difficult to provide elastic service in order to deal with the changing workloads. This is because these systems often over-provision brokers to reduce their loads, and do not have financial incentives to reduce the scale of brokers during off-peak hours.

In contrast, a large number of P2P based systems  [17], [18], [19], [20], [21], [22], [23] do not provide dedicated brokers. All nodes are organized into a P2P based overlay  [24], [25], [26], and they act both as publishers and subscribers. All events and subscriptions are forwarded through multi-hop routing. Then the subscriptions falling into the same subspace of the entire content space are organized into a multicast group or stored in a rendezvous node. With the growth of arrival event rate, the multi-hop routing may lead to high latency and traffic overhead. A large body of skewed subscriptions may incur unbalanced load on the multicast groups or rendezvous nodes, which imposes a limit on scalability. Besides that, the P2P based systems are hard to provide elastic service due to the unpredictability of the node behavior.

Recently, cloud computing has become a new infrastructure to develop large-scale distributed applications over the Internet because of its low capital cost, powerful storage and computing capacities  [27], [28], which provides great opportunities to meet the requirements of complex computing  [29] and high seed communication. Cloud server providers scatter geographically many datacenters to support a large number of users around the world. Because of the high bandwidth and reliable links in the cloud environment, Cassandra  [30] proposes to use the consistent hashing  [31] to provide one-hop lookup to organize servers into a scalable overlay. Thus, the main challenges of designing attribute-based pub/subs in the cloud computing environment lie in the partitioning technique and the elastic strategy. Based on the above-mentioned one-hop lookup technique, a number of researches  [32], [33], [34] focus on the cloud computing environment to provide pub/sub services, where the most relevant one to our work is BlueDove  [33]. There are two main components in BlueDove. One is the multi-dimensional subscription space partitioning technique which provides multiple candidate servers for each message. The other is the performance-aware forwarding technique which ensures each message is sent to the least loaded candidate server for matching. However, the partitioning technique in BlueDove may render the system poorly scalable. Firstly, BlueDove divides every single dimension of the content space into multiple clusters, each of which is uniformly mapped to a server. Since each cluster only concerns a specific range of a single dimension, a hot cluster can be easily formed if a large number of users subscribe the same range of one dimension, which severely hurts the matching throughput. Secondly, the performance-aware forwarding scheme of BlueDove enables the system to keep workload balance of servers through dispatching each message to a candidate server with the lowest processing time. However, it intrinsically cannot alleviate the hot spots. When the distribution of subscriptions is flat, each server may be responsible for a number of hot spots, which severely decreases the performance of each server. Besides, although BlueDove provides elastic servers to adapt to a sudden increasing of the workload through detecting system saturation, it does not point out how to adaptively decrease the scale of servers to achieve high performance–price ratio when the event arrival rate decreases.

Motivated by these factors, we present SEMAS, a scalable and elastic event matching service for an attribute-based pub/sub system in the cloud computing environment. Generally speaking, the main novelty of SEMAS lies in a hierarchical multi-attribute space partition technique (called HPartition) and a performance-aware detection technique (called PDetection). The first aims to improve the matching throughput in a scalable manner. HPartition divides the entire content space into multiple hypercubes, each of which concerns a specific range of each dimension. That is, a hot cluster is formed only if a large number of users are interested in the same ranges of all dimensions, which greatly reduces the number of hot clusters. Moreover, HPartition hierarchically divides a hot cluster into multiple small “cold” clusters which are dispatched again to uniform selected servers using consistent hashing. This iterative partition scheme ensures that each cluster has a moderate size which brings low matching latency and high throughput.

The second aims to achieve elastic and reliable event matching. Specifically, PDetection elastically adjusts the scale of servers according to the churn of workloads. To detect the changing of workloads, PDetection adopts a light-weighted gossip based aggregation technique  [35] to periodically check the performance of the servers. When the performance of the worst server is below the minimum threshold or above the maximum threshold, a number of servers are added or removed respectively to keep a high performance–price ratio. To reduce the reconfiguring delay and traffic overhead, PDetection redistributes each cluster through migrating the identifications of subscriptions. By synchronizing the status of all servers, PDetection ensures reliable and continuous event matching service.

To evaluate the performance of SEMAS, we design and implement a prototype on our OpenStack-based testbed. The experimental results show that SEMAS presents linear increasing event matching throughput as the number of servers and the partitioning granularity increase, rapid response to the sudden change of workload, strong reliability to large-scale server failures with low latency and traffic overhead, and higher throughput in various workloads compared with the existing cloud based approaches.

As a summary, the primary contributions are given as follows.

  • We propose a hierarchical multi-attribute space partition scheme to achieve high matching throughput with numerous skewed subscriptions.

  • We propose a performance-aware detection scheme to ensure elastic and reliable matching service with the sudden change of workloads.

  • We implement a thorough and systematic evaluation of SEMAS as well as various matching approaches to evaluate the performance of SEMAS.

The rest of this paper is organized as follows. In Section  2, we outline the related works. In Section  3, we introduce the data model and system architecture of SEMAS. In Section  4, we describe the full SEMAS protocol. We implement SEMAS and analyze its performance in Section  5. Finally, this paper is concluded in Section  6.

Table 1 shows key parameters used in this paper.

Section snippets

Related works

According to the overlay infrastructure, existing works of the attribute-based pub/sub systems can be categorized into the broker based systems, P2P based systems and cloud based systems.

Attribute-based pub/sub model

According to the express power of subscription models, the pub/sub systems are classified into topic-based, attribute-based and content-based systems. In a topic-based model  [45], [46], [47], [48], events are usually matched against subscriptions by a unique identification denoted by a string, called topic. In contrast, a more expressive form is the content-based model  [49], [50], where each subscription can be arbitrary Boolean functions, including disjunction of predicates, conjunction of

SEMAS components

Our objective is to provide scalable and elastic event matching service in the attribute-based pub/sub systems. Although the two-layer overlay (Section  3.2) presents a lot of features that make it suitable for scalable event matching, we still need to provide more details to answer how SEMAS meets the challenge of a sudden change of incoming event rate and the skewness of subscriptions. First, how should we group the skewed subscriptions so that each message only needs to be matched on one

Implementation

We design and implement the prototype of SEMAS on our OpenStack-based  [60] testbed. In order to build the prototype as modular and portable with ease, we use ICE  [61], an object-based middleware, as our fundamental framework. ICE allows users to focus on their application logic, rather than interact with low-level network programming interfaces. Besides the code generated by ICE, we add about 14,000 lines of Java code to implement the prototype. To evaluate the performance of SEMAS, we

Conclusions and future work

This paper introduces SEMAS, a novel scalable and elastic event matching approach for the attribute-based pub/sub systems. SEMAS utilizes an one-hop lookup overlay in the cloud computing environment to reduce the clustering latency. Through a hierarchical multi-attribute space partition technique, SEMAS reaches scalable clustering of subscriptions and matches each event on one cluster. The performance-aware detection technique enables the system to adaptively adjusts the scale of matchers

Acknowledgments

This work was supported by the National Grand Fundamental Research 973 Program of China (Grant No. 2011CB302601), the National Natural Science Foundation of China (Grant No. 61379052), the National High Technology Research and Development 863 Program of China (Grant No. 2013AA01A213), the Natural Science Foundation for Distinguished Young Scholars of Hunan Province (Grant No. S2010J5050), Specialized Research Fund for the Doctoral Program of Higher Education (Grant No. 20124307110015).

Xingkong Ma received the B.S. degree in Computer Science and Technology from the School of Computer of Shandong University, China, in 2007, and received the M.S. in Computer Science and Technology in the School of Computer of National University of Defense Technology, China, in 2009. He is currently a Ph.D. candidate in the School of Computer of National University of Defense Technology. He is a student member of CCF and ACM. His current research interests lie in the areas of data

References (62)

  • Y. Wang et al.

    Research and performance evaluation of data replication technology in distributed storage systems

    Computers and Mathematics with Applications

    (2006)
  • Anss. URL:...
  • Readywarn. URL:...
  • R.K. Ganti et al.

    Greengps: a participatory sensing fuel-efficient maps application

  • M. Gjoka, M. Kurant, C.T. Butts, A. Markopoulou, Walking in Facebook: a case study of unbiased sampling of OSNs, in:...
  • P.T. Eugster et al.

    The many faces of publish/subscribe

    ACM Computing Surveys (CSUR)

    (2003)
  • Datacreatedperminite. URL:...
  • P. Pietzuch, J. Bacon, Hermes: a distributed event-based middleware architecture, in: 22nd International Conference on...
  • F. Cao, J.P. Singh, Efficient event routing in content-based publish/subscribe service network, in: International...
  • G. Banavar, T. Chandra, B. Mukherjee, J. Nagarajarao, R.E. Strom, D.C. Sturman, An efficient multicast protocol for...
  • F. Cao, J.P. Singh, Medym: match-early with dynamic multicast for content-based publish–subscribe networks, 2005, pp....
  • L. Opyrchal, M. Astley, J. Auerbach, G. Banavar, R. Strom, D. Sturman, Exploiting IP multicast in content-based...
  • A. Riabov, Z. Liu, J.L. Wolf, P.S. Yu, L. Zhang, Clustering algorithms for content-based publication–subscription...
  • A. Carzaniga, Architectures for an event notification service scalable to wide-area networks, Ph.D. Thesis, POLITECNICO...
  • Y.-M. Wang et al.

    Summary-based routing for content-based event distribution networks

    Computer Communication Review

    (2004)
  • A. Carzaniga, M.J. Rutherford, A.L. Wolf, A routing scheme for content-based networking, in: IEEE International...
  • W.W. Terpstra, S. Behnel, L. Fiege, A. Zeidler, A.P. Buchmann, A peer-to-peer approach to content-based...
  • I. Aekaterinidis, P. Triantafillou, Pastrystrings: a comprehensive content-based publish/subscribe DHT network, in:...
  • E. Anceaume, M. Gradinariu, A.K. Datta, G. Simon, A. Virgillito, A semantic overlay for self-* peer-to-peer...
  • S. Voulgaris et al.

    Sub-2-sub: self-organizing content-based publish and subscribe for dynamic and large scale collaborative networks, Research Report RR5772, INRIA, Rennes, France

    (2005)
  • C. Zhang, A. Krishnamurthy, R.Y. Wang, J.P. Singh, Combining flexibility and scalability in a peer-to-peer...
  • D.K. Tam et al.

    Building content-based publish/subscribe systems with distributed hash tables

  • A. Gupta, O.D. Sahin, D. Agrawal, A. El Abbadi, Meghdoot: content-based publish/subscribe over P2P networks, in:...
  • W. Li, S. Vuong, Towards a scalable content-based publish/subscribe service over DHT, in: IEEE Global...
  • S. Voulgaris, M. van Steen, Epidemic-style management of semantic overlays for content-based searching, in:...
  • A.I.T. Rowstron, P. Druschel, Pastry: scalable, decentralized object location, and routing for large-scale peer-to-peer...
  • S. Ratnasamy, M. Handley, R.M. Karp, S. Shenker, Application-level multicast using content-addressable networks, 2001,...
  • B. Calder, J. Wang, A. Ogus, N. Nilakantan, A. Skjolsvold, S. McKelvie, Y. Xu, S. Srivastav, J. Wu, H. Simitci, J....
  • X. Lu et al.

    Internet-based virtual computing environment: beyond the data center as a computer

    Future Generation Computer Systems

    (2011)
  • Y. Wang et al.

    A survey of queries over uncertain data

    Knowledge and Information Systems

    (2013)
  • A. Lakshman et al.

    Cassandra: a decentralized structured storage system

    Operating Systems Review

    (2010)
  • Cited by (22)

    • Modular and generic IoT management on the cloud

      2018, Future Generation Computer Systems
      Citation Excerpt :

      The solution is modular, decentralized and reusable [13] thus allows IoT devices to easily attach to the service. We are motivated by the works in publish/subscribe systems in clouds and inter-clouds as in [14–18] ​and [19]. The basic characteristic of the proposed service is the simplicity of use at any time requested by the user.

    • Scalable and elastic total order in content-based publish/subscribe systems

      2015, Computer Networks
      Citation Excerpt :

      As shown in Fig. 1, the framework of SETO contains two layers: the matching layer and the delivery layer. In the matching layer, we use SEMAS [23] to provide scalable and elastic matching service. Note that the objective of our experiments is to evaluate the performance of SETO in the delivery layer.

    • Intelligent big data processing

      2014, Future Generation Computer Systems
    • EdgePub: A Self-Adaptable Distributed MQTT Broker Overlay for the Far-Edge

      2022, 2022 7th International Conference on Fog and Mobile Edge Computing, FMEC 2022
    View all citing articles on Scopus

    Xingkong Ma received the B.S. degree in Computer Science and Technology from the School of Computer of Shandong University, China, in 2007, and received the M.S. in Computer Science and Technology in the School of Computer of National University of Defense Technology, China, in 2009. He is currently a Ph.D. candidate in the School of Computer of National University of Defense Technology. He is a student member of CCF and ACM. His current research interests lie in the areas of data dissemination, publish/subscribe systems, and network computing.

    Yijie Wang received the Ph.D. degree from the National University of Defense Technology, China in 1998. She was a recipient of the National Excellent Doctoral Dissertation (2001), a recipient of Fok Ying Tong Education Foundation Award for Young Teachers (2006) and a recipient of the Natural Science Foundation for Distinguished Young Scholars of Hunan Province (2010). Now she is a Professor in the National Key Laboratory for Parallel and Distributed Processing, National University of Defense Technology. Her research interests include network computing, massive data processing, parallel and distributed processing.

    Qing Qiu received the B.S. degree in Computer Science and Technology from the School of Computer of Shandong University, China, in 2010, and received the M.S. in Computer Science and Technology from the School of Computer of the National University of Defense Technology, China, in 2012. He is a student member of CCF and ACM. His current research interests lie in the areas of data dissemination, publish/subscribe systems, and network computing.

    Weidong Sun received the B.S. degree in computer science and technology from the School of Computer of National University of Defense Technology, China, in 2005, and received the M.S. in computer science and technology in the School of Computer of National University of Defense Technology, China, in 2009. He is currently a Ph.D. candidate in the School of Computer of National University of Defense Technology. He is a student member of CCF and ACM. His current research interests lie in the areas of network computing, massive data processing, cloud computing.

    Xiaoqiang Pei received the B.S. degree in computer science and technology from the School of Computer of National University of Defense Technology, China, in 2009, and received the M.S. in computer science and technology in the School of Computer of National University of Defense Technology, China, in 2011. He is currently a Ph.D. candidate in the School of Computer of National University of Defense Technology. He is a student member of CCF and ACM. His current research interests lie in the areas of network computing, massive data processing, cloud computing.

    View full text