Impact analysis of data placement strategies on query efforts in distributed RDF stores

Abstract

In the last years, scalable RDF stores in the cloud have been developed, where graph data is distributed over compute and storage nodes for scaling efforts of query processing and memory needs. One main challenge in these RDF stores is the data placement strategy that can be formalized in terms of graph covers. These graph covers determine whether (a) the triples distribution is well-balanced over all storage nodes (storage balance) (b) different query results may be computed on several compute nodes in parallel (vertical parallelization) and (c) individual query results can be produced only from triples assigned to few – ideally one – storage node (horizontal containment). We analyse the impact of three most commonly used graph cover strategies in these terms and found out that balancing query workload reduces the query execution time more than reducing data transfer over network. To this end, we present our novel benchmark and open source evaluation platform Koral.

Introduction

In the last years, the requirement for RDF stores that can cope with several trillions of triples has emerged. For instance, the number of Schema.org-based facts that are extracted out of the Web have reached the size of three trillions [2]. Another example is the European Bioinformatics Institute (EMBL-EBI) that would like to convert its datasets into RDF resulting in a graph consisting of several trillions of triples. To date no such scalable RDF store exists and the current EBI RDF Platform can handle only 10 billion triples [3].

We pursue the development of a scalable RDF store in the cloud, where graph data is distributed over compute and storage nodes for scaling efforts of query processing and memory needs. The main challenges to be investigated for such development are: (i) strategies for data placement over compute and storage nodes, (ii) strategies for distributed query processing, and (iii) strategies for handling failure of compute and storage nodes. In this paper, we focus on comparing the performance of data placement strategies.

Strategies for data placement may be formalized in terms of graph covers. Each compute and storage node hosts a graph chunk. Each triple is assigned to (at least) one graph chunk and the union of all graph chunks define a (possibly redundant) graph cover. When a query is requested to an RDF store in the cloud, the query is distributed over the different compute and storage nodes. Each node applies the query operators assigned to it on its local data. If the query requires the combination of data from different chunks, the required information has to be transferred between compute nodes.

One graph cover strategy commonly used is the hash cover that assigns triples to compute and storage nodes according to the hash value of, e.  g., their subject (e.  g., used by Virtuoso Clustered Edition [4], YARS2 [[5], [6]], Clustered TDB [7] and Trinity.RDF [8]). In order to reduce the number of transferred intermediate results, hierarchical hash has been proposed as an extension of the hash cover strategy that computes the hash only on IRI prefixes [9]. Another commonly used graph cover strategy is the minimal edge-cut cover that assigns vertices to similarly-sized partitions in a way that the number of edges connecting vertices assigned to different partitions is minimized (e.  g., used by [[10], [11], [12]]). Furthermore, the vertical cover strategy is inspired by relational databases. It partitions the dataset by storing all triples with the same property in one table. Finally, these tables are then distributed among all compute and storage nodes according to the hash on the property. It is used by, e.  g. HadoopRDF [13], Jena-HBase [14] and [15]. In order to reduce the number of transferred intermediate results, [11] proposed to replicate triples at the border of the graph chunks. This idea is also used by systems like VB-Partitioner [16] and D-SPARQ [17].

It is a commonly held belief that query completion is optimized by approaches that emphasize local computation such as minimal edge-cut (cf. [[11], [12], [16]]). The first major contribution of this paper is to challenge this assumption by new experiments. Our results indicate that contrary to commonly held beliefs, query answering with hash covers may outperform query answering with, e.g., minimal edge-cut covers since the load on the different machines is more balanced. Furthermore, when replicating triples on several computers, the high number of duplicate computations may overcome the benefits of the reduced data transfer via network and lead to a worse query performance.

We have performed our experiments with the aim to understand interdependencies of the involved query processing. Thus, we have devised new measures and do not only compare graph cover strategies in terms of query processing time, but in addition we investigate the following dimensions:

• Load time describes the time it takes to create a graph cover. This is an indicator how well a graph cover strategy can scale by horizontal scaling of the cloud.

• Storage balance describes to which extent graph chunks are of similar size. This is an indicator that memory needs can be met with increasing data size by horizontal scaling of the cloud.

• Horizontal containment describes to which extent computation of individual query results is local to one (or few) graph chunk(s). This is an indicator that query processing is (to some extent) robust when the cloud is scaled horizontally.

• Vertical parallelization describes to which extent different query results may be computed in parallel on different compute nodes. This is an indicator that query processing can scale with growing result set sizes by horizontal scaling of the cloud.

Using these measurements, we derive the second important contribution of this paper. We discovered from the analysis of query processing using different graph cover strategies that vertical parallelization (i.e. a well-distributed workload) may be more important than horizontal containment (i.e. minimal data transport) for efficient query processing — even in a commodity network environment (1 GB/s). Furthermore, our analysis revealed that previous experiments like [[11], [16]] and [12] suffered from a setting with highly inefficient methods for data transfer (i.e. based on the Hadoop/HDFS infrastructure) (see [18]).

In order to determine to which extent graph cover strategies lead to efficient query answering, they have to be implemented and evaluated in distributed RDF stores. For instance, [8] and [12] evaluate various RDF stores that use different graph cover strategies, but these evaluations compared the RDF stores as wholes. Thus, their results also reflect the effects of, e.  g., the different indexing strategies and persistence strategies (i.  e., main memory vs. hard disk) used by the different stores. In order to focus on the effects of the different graph cover strategies, other evaluations have used the same system to measure the execution time [16] and [19]. These systems use technologies like Hadoop or HDFS that cause an overhead for data transfer. To avoid the bias of this overhead, the third important contribution of this paper is the flexible open source platform Koral. It executes queries on arbitrary graph covers and transfers the intermediate results within the network.

In short, the contributions of this paper are:

• An explanation why previous evaluations concluded that the amount of data transfer caused by a graph cover strategy is crucial for the query execution effort (Section 6).

• An analysis indicating that (i) hash covers outperform minimal edge-cut covers and vertical covers, (ii) vertical parallelization is more important than horizontal containment and (iii) triple replication reduces query performance due to a high number of duplicate computations (Section 5).

• A benchmark methodology and its implementation that allows for a detailed understanding of the interdependencies of the graph cover strategy and the query processing (Section 4).

This paper is an extended version of the 6 page workshop paper [1].