Echo: A peer-to-peer clustering framework for improving communication in DHTs

https://doi.org/10.1016/j.jpdc.2009.06.002Get rights and content

Abstract

One of the main challenges of peer-to-peer (P2P) systems is how to efficiently store and locate the ever-increasing amount of data being shared by participants. Indexing methods have been adapted on top of P2P networks and querying methods have been developed to handle data distribution across different nodes. Among these technologies, Distributed hash tables (DHTs) provide the highest scalable substrates. Although DHTs have many virtues, yet, it is hard to develop certain types of applications. For instance, distributing multimedia content among users who form communities based on their interests or their location is difficult to achieve in plain DHTs.

To support a broader range of applications, we present Echo, a framework that benefits from the recursive structure of DHTs to embed clusters into their structures at almost no cost. While retaining the uniformity, scalability and load balancing of the original designs, Echo improves their scalability by taking advantage of the locality in communication exhibited by communities of users. Since our framework is based on the Cayley graph-theoretic model, it is applicable to a large subset of the most representative DHTs. As an illustrative example, we show how Chord can be transformed into a clustered DHT using our methodology. Furthermore, we give some indicative hints of how six different DHTs–Randomized Chord, Symphony, Kademlia, P-Grid, Tapestry and Pastry–can be transformed into their Echoing versions. Simulations results verify the effectiveness of Echo.

Introduction

Peer-to-peer (P2P)  computing is attracting increasing attention, and many P2P systems have rapidly become basic platforms for users to share information and resources (CPU cycles, memory, storage space and network bandwidth, to name a few) over the Internet. In essence, a “pure” P2P system can be described as a decentralized network system in which all participant computers (also called peers) have symmetric duties and responsibilities. In simplistic terms, all participating nodes act as both clients and servers to one another, yielding a large pool of information sources and computing power. Key features of P2P systems are decentralization, self-organization, dynamism, and fault-tolerance, all of which make the P2P paradigm very attractive for information storage and retrieval.

In terms of network structure, decentralized P2P systems can be classified into two categories, namely unstructured and structured. The first wave of P2P systems implemented unstructured P2P overlays, in which content placement is random and no global structure is maintained (e.g., Gnutella [13]). To search for data, unstructured P2P systems use message flooding to propagate queries. Although such systems are fault-tolerant and resilient to users joining and leaving the network (also known as churn), the search mechanism does not scale [33]. Structured P2P overlays [38], [34], [28], which implement a Distributed Hash Table (DHT) data structure, were proposed to overcome the scaling problems of unstructured systems. DHTs provide the same functionality of a hash table (i.e., the standard Put(key, value) and Get(key) interface), but associating key-value mappings with participating nodes rather than hash buckets. While there are significant implementation differences between DHTs, all achieve efficiency by organizing the participant nodes into a well-defined structure, so that queries can be resolved within O(logN) number of hops, where N is the number of peers in the network. For this reason, the focus of this paper is on structured systems such as Chord [38] and Pastry [34].

Although DHTs have many appealing virtues, yet, it is unclear how to implement certain applications and services using DHTs as a basic building block. For instance, DHTs have poor locality. Although some DHTs such as Pastry make an effort to route queries through nodes with low network latency, the last few hops in any query are essentially random. This is of particular concern for content distribution networks (CDNs), which require the end-to-end latency from the source to each receiver to be low.

One common solution for this problem consists of decomposing the participating nodes of a CDN into clusters, so that the nodes forming an individual cluster experience a significant reduction in network latency. As a representative example, we revisit Coral [12], a decentralized, peer-to-peer web-content distribution network. Coral exploits DNS redirection, together with a sloppy distributed hash table (DSHT) abstraction to redirect each client to a close copy of the requested web page, hosted on some volunteer node.

To attain this goal, Coral uses three levels of clusters and allows a set of nodes to form a cluster only if their average, pairwise round-trip-times are below some threshold, which depends on the cluster level. To be more precise, Coral specifies thresholds of ms, 60 ms, and 20 ms for level-0, -1, and -2 clusters, respectively. The immediate consequence of this is that clients can first query the nodes in higher-level, fast clusters before those in lower-level, slower clusters. This both diminishes the latency of Get and Put operations and increases the chances of returning values stored by nearby nodes. A clarifying example is shown in Fig. 1(a). For a Get on key k, a node first searches in its lowest cluster to take advantage of network proximity. This lookup fails on that level if the node closest to k, node t2, does not store the key. If this happens, Coral continues the query on a higher-level cluster, having already traversed the identifier space up to node t2.

Motivated by the above example, our focus is on clustering techniques for DHTs that improve query efficiency, but maintaining the properties of the clustered versions very close to those in the original DHTs; namely efficiency, scalability, fault-tolerance and load-balancing. In this article, the primary question we address is: “Can clustering improve the query efficiency of structured P2P overlays with minimal disruption?” The answer is affirmative but, as we shall see throughout the paper, an optimal solution is far from being obvious. If sufficient care is not taken, one can easily introduce new pathologies such as node- and link-congestion which arise when homogeneity is broken.

Moreover, developers outside the research community are slow to adopt new solutions, albeit any advantage they may obtain over their predecessors. Since there is a plethora of DHTs in the literature, we believe more reasonable to adapt the standard DHTs than developing a solution based on a new topology.

Although clustering has become a keen point of interest in the last few years, there are still some critical points to be addressed before progress can be made. To clarify their impact, we examine clustering over Chord. The essential question is: How to map the clusters of similar peers onto the one-dimensional Chord identifier space? Since Chord presumes an identifier space of size 2m, there exist many ways to map the clusters onto domain [0,2m), though not all are in consonance with the structural constraints of Chord.

In the case of prefix clustering, the cluster mapping reduces to a simple identifier-assignment rule: all members of a cluster are assigned a common node_id prefix. The consequence is that each cluster is mapped into a continuous subrange of domain [0,2m). An example of this mapping is presented in Fig. 1(b). As shown in the figure, there are three clusters identified by letters A, B and C. While cluster A occupies half of domain [0,2m), clusters B and C are mapped to two equal-length subintervals that together occupy the other half. We note that cluster B is extremely sparse. This will be of special importance as we will see in short.

While presumably practical, prefix clustering conflicts with an important concern, central to many DHT protocols: the uniform distribution of nodes along the identifier space. For example, Chord guarantees its efficiency while nodes are distributed uniformly on the identifier space. Otherwise, the amount of traffic arriving at some nodes could be prohibitively large, leading to a severe performance degradation.

Now we examine the effect that prefix clustering has upon congestion. We use the notion of “congestion-freeness” proposed by Xu et al. in [41]. Before going further, it is necessary to define what we understand for communication load. As in [41], we consider uniform all-to-all communication load. It is defined as follows: for each pair of nodes u and v, uv, there is a unit of traffic flowing from u to v. In our context, this means that node u is looking for an object that node v is responsible for.

Under uniform all-to-all communication load, there are N(N1) routings paths in the network. Assuming that the average path length of the network is h, then the average load on any node should be (N1)h if the network was congestion-free, i.e., if the amount of traffic going through each node was no more than the average.

Now assume the network topology of Fig. 1(b) and let nodes a, b, c, d and e initiate a query targeted to a different node in cluster C. Assuming standard Chord routing [38], observant readers will have noticed that all requests will go through the unique node in cluster B, rapidly overloading it. The reason is that this node takes responsibility for a large interval of the Chord domain and therefore, it attracts a number of in-coming links greater than the average. Consequently, the load imposed on this node will be higher that (N1)h, breaking the uniformity in load. This leads to the conclusion that prefix clustering increases scalability, but it does so by weakening the congestion-free property of Chord.

From the preceding discussion, it is clear that clustering has the potential to improve query throughput. However, there are some issues that must be considered before clustering a DHT. For instance, the load imbalance that might result from clustering itself. The cause is that clustering conflicts with a central principle behind DHT technology: To achieve scalability, de-cluster load across all nodes in the system. For example, in Chord, the expected in-degree of a node (i.e., the number of routing table entries referring to a given node) is O(logN). This property is desirable, as it tends to balance communication load across all computers in the network. However, it requires that peers are uniformly distributed along the identifier space, which limits the possibilities of using clusters.

In this paper, we propose Echo, a framework that combines the homogeneity of load and functionality offered by plain DHTs with the advantages of clustering, which may be diverse, but generally aim to improve query efficiency. In other words, the challenge we face is how to design cluster-based DHTs (CHTs) that obtain the “best” of both worlds, without inheriting the disadvantages of either.

To better understand the contributions of this research, we discuss the aspects Echo considers for coming up with “good” CHTs. To ease exposition, we assume a generic DHT (similar to Chord) that has logarithmic diameter and degree at each node. Then, we consider the following key aspects:

  • Efficiency. Echo should achieve short routing paths with small state per node. At most, a diameter of O(logN) routing hops. In addition, maintenance should be kept minimal, i.e., affecting O(logN) number of computers and mostly no burden on anyone.

  • Scalability. The concept of scalability comprises many aspects. In this work, we focus on numerical scalability. Broadly speaking, we want to show that N can grow without performance degradation in Echo designs.

  • Fault-tolerance. Under normal conditions, the integrity of cluster structures is guaranteed by the maintenance protocol. However, due to unpredictable node and link failures, a peer might suddenly become unreachable. To provide reliable operation, Echo should guarantee that the clustered version of a DHT is also robust against such failures. In addition, it should exhibit a maintenance complexity as tight as possible to the one claimed in the original DHT.

  • Load balancing. A basic principle of P2P paradigm is that load should be well-balanced across all nodes in the network. If random nodes call Put and Get, then the communication load should be equitably distributed among all participating users. This includes preserving the randomness guarantee of consistent hashing and the distributional properties of in-degree, which are very relevant for load balancing in message forwarding.

To the best of our knowledge, Echo is the first work that generically yet smartly augments DHTs to accommodate clusters almost for free. To meet this challenge, Echo makes use of the Cayley graph group-theoretic model [2], an algebraic machinery that has been extensively used to examine the structural properties of interconnection networks (ICNs). There are three powerful reasons for doing this:

  • (1)

    By using the Cayley graph model, we ensure that the developed theory is not tied to a given DHT topology; many important DHT geometries are Cayley graphs like the hypercube(HyperCuP [35]), the torus (CAN [32]), the cube-connected cycles(Cycloid [37]), and the pancake graph (IHOP [31]);

  • (2)

    Every Cayley graph is vertex-symmetric, i.e., a graph in which no node incorporates a more prominent position than the others. This is fundamental for load balancing, since one can expect that all nodes operate in a uniform way and whatever occurs at a node occurs equivalently at all the other nodes. On the search side, this implies that (i) communication load balancing is preserved and (ii) both intra- and inter-cluster routing will not cause hot-spots; and

  • (3)

    The Cayley graph model allows us to abstract our topological problem to a sorting problem, which is tractable programmatically using tools like GAP [7].

Although much of this work focuses on Chord, we will also describe how to adapt other DHTs, including non-deterministic Chord [15], Symphony [26], Kademlia [28] and Pastry [34]. Finally, we will show the potential benefit of clustering as a function of the amount of intra-cluster communication.

The remaining of this paper is structured as follows: In Section 2, we provide a survey of related work. Section 3 gives necessary background information. We describe Echo in Section 4. In Section 5, we briefly point out how to obtain the clustered versions of other six DHTs. In Section 6, we provide another example to show the relevance of the algebraic study of DHTs. In Section 7, we validate our framework and quantify its advantages. We conclude the paper in Section 8.

Section snippets

Related work

Many P2P systems have incorporated clusters into their design as a method for accommodating growth and isolating faults. As an example, consider the following systems. JXTA defines the concept of PeerGroups [14] to decompose the large number of peers into manageable groups. Further, SHARK [3] clusters nodes according to the common interest of users. Also, Considine [8] proposes multiple cluster-based overlays for Chord. The cluster construction in the latter proposal is based on network

Background

In this section, some basic definitions are given, and some notation is introduced.

Echo framework

In this section, we first present an overview of Echo followed by a detailed description of its functionalities.

Adaptability to other DHTs

In this section we demonstrate the usefulness of our methodology, by applying Echo to six families of DHTs that are benchmarks among P2P systems:

Beyond search efficiency: An example of what Cayley graphs can do for DHTs

From the discussion so far, it is clear that the formalism of DHTs in terms of the Cayley graph model provides numerous benefits in routing efficiency. However, it is important to understand that this coupling goes beyond the simple optimization of one particular aspect. The analysis of their Cayley structures provides valuable information over which architects can sustain their decisions. To support this claim, we discuss another example in this section. The goal of this example is to show how

Performance evaluation

To conclude the analysis of Echo, we decided to use a simulator and evaluate Echo-Chord. Since none of the tools currently available exactly fitted our needs, we built our own simulator. We evaluated the following characteristics of Echo-Chord: out-degree distribution, average path length, average query time and load balancing.

The simulation model consisted of N participating peers, grouped in a varying number of clusters nc. Specifically, we set nc to 8, 16 and 32 in all tests, which

Conclusions

We have described Echo, a framework for embedding clusters into DHTs at almost no cost. We have shown how this framework can be applied to plain DHTs that, while yet maintaining homogeneity, can also achieve significant improvements in communication. As a representative example, we have applied our technique to construct a clustered version of Chord. Because our framework is based on the Cayley graph model, it can be made extensible to a wide variety of DHTs, including Chord, Symphony, Pastry,

Marc Sánchez-Artigas is a post-doc researcher in the Department of Computer Engineering and Maths at Universitat Rovira i Virgili, Spain. He obtained his Ph.D. degree in 2009 with European Mention at Universitat Pompeu Fabra, Barcelona. During his Ph.D., he also worked at Ecole Polytechnique Féderale de Lausanne (EPFL) under the supervision of Prof. Karl Aberer. His research interests include peer-to-peer systems, Distributed Hash Tables and Grid computing. He received the Best Paper Award from

References (42)

  • A.-L. Barabasi et al.

    Scale-free characteristics of random networks: The topology of the world-wide web

    Physica A: Statistical Mechanics and its Applications

    (2000)
  • M.C. Heydemann et al.

    On forwarding indices of networks

    Discrete Applied Mathematics

    (1989)
  • Y. Manoussakis et al.

    The forwarding index of directed networks

    Discrete Applied Mathematics

    (1996)
  • K. Aberer et al.

    P-grid: A self-organizing structured p2p system

    SIGMOD Record

    (2003)
  • S.B. Akers et al.

    A group-theoretic model for symmetric interconnection networks

    IEEE Transactions on Computers

    (1989)
  • S. Annapureddy, M.J. Freedman, D. Mazières, Shark: Scaling file servers via cooperative caching, in: NSDI’05:...
  • K. Calvert et al.

    Modeling internet topology

    Communications Magazine, IEEE

    (1997)
  • F.R.K. Chung et al.

    The forwarding index of communication networks

    IEEE Transaction on Information Theory

    (1987)
  • H. Cohen

    A Course in Computational Algebraic Number Theory

    (1993)
  • J. Considine, Cluster-based optimizations for distributed hash tables, Tech. Rep.,...
  • G. Cordasco, A. Sala, 2-chord halved, in: HOT-P2P’05: Proceedings of the Second International Workshop on Hot Topics in...
  • F. Dabek, R. Cox, F. Kaashoek, R. Morris, Vivaldi: A decentralized network coordinate system, in: SIGCOMM’04:...
  • eDonkey, http://www.edonkey2000.com,...
  • M.J. Freedman, E. Freudenthal, D. Mazières, Democratizing content publication with coral, in: NSDI’04: Proceedings of...
  • Gnutella,...
  • L. Gong, Project jxta: A technology overview,...
  • K. Gummadi, R. Gummadi, S. Gribble, S. Ratnasamy, S. Shenker, I. Stoica, The impact of DHT routing geometry on...
  • M.C. Heydemann

    Cayley graphs and interconnection networks

  • H.V. Jagadish, B.C. Ooi, Q.H. Vu, Baton: A balanced tree structure for peer-to-peer networks, in: VLDB’05: Proceedings...
  • D. Karger, E. Lehman, F. Leighton, M. Levine, D. Lewin, R. Panigrahy, Consistent hashing and random trees: Distributed...
  • B. Karp, S. Ratnasamy, S. Rea, S. Shenker, Spurring adoption of dhts with openhash, a public dht service, in: IPTPS’04:...
  • Cited by (0)

    Marc Sánchez-Artigas is a post-doc researcher in the Department of Computer Engineering and Maths at Universitat Rovira i Virgili, Spain. He obtained his Ph.D. degree in 2009 with European Mention at Universitat Pompeu Fabra, Barcelona. During his Ph.D., he also worked at Ecole Polytechnique Féderale de Lausanne (EPFL) under the supervision of Prof. Karl Aberer. His research interests include peer-to-peer systems, Distributed Hash Tables and Grid computing. He received the Best Paper Award from the 32th IEEE Conference of Local Computer Networks (LCN) held in Dublin, 2007. He has participated in several research projects such as IST-POPEYE, PLANET and P2PGRID. Currently, he is serving as a Guest Editor for Computer Networks Journal, Elsevier. Contact him at [email protected].

    Pedro García López is professor at the Computer Engineering and Mathematics Department at the University Rovira i Virgili (Spain). He obtained his Ph.D. in 2003 in the University of Murcia about Collaborative Distributed Systems. During his Ph.D. he also worked in the university of Ghent (Belgium) and GMD-FIT Bonn. His research topics are distributed systems, peer-to-peer systems, software architectures and middleware for collaborative environments. He has published more than 50 papers and participated in several spanish and european research projects. He currently leads in Tarragona the “Architectures and Telematic Services” research group and coordinates the URV team in the project IST-POPEYE (Peer-to-Peer collaborative Working environments over Mobile Ad-Hoc Networks) and P2PGRID (Self-Adjusting Peer-to-Peer and Grid Systems).

    This work was partially funded through project P2PGRID, TIN2007-68050-C03-03, of the Ministry of Education and Science, Spain.

    View full text