Incremental k-core decomposition: algorithms and evaluation

Abstract

A k-core of a graph is a maximal connected subgraph in which every vertex is connected to at least k vertices in the subgraph. k-core decomposition is often used in large-scale network analysis, such as community detection, protein function prediction, visualization, and solving NP-hard problems on real networks efficiently, like maximal clique finding. In many real-world applications, networks change over time. As a result, it is essential to develop efficient incremental algorithms for dynamic graph data. In this paper, we propose a suite of incremental k-core decomposition algorithms for dynamic graph data. These algorithms locate a small subgraph that is guaranteed to contain the list of vertices whose maximum k-core values have changed and efficiently process this subgraph to update the k-core decomposition. We present incremental algorithms for both insertion and deletion operations, and propose auxiliary vertex state maintenance techniques that can further accelerate these operations. Our results show a significant reduction in runtime compared to non-incremental alternatives. We illustrate the efficiency of our algorithms on different types of real and synthetic graphs, at varying scales. For a graph of 16 million vertices, we observe relative throughputs reaching a million times, relative to the non-incremental algorithms.

Appendix: Generic RCD maintenance

As stated earlier in Sect. 4.3.1, maintaining RCD values is a non-trivial operation. Yet, it is critical in reducing the scopes of the traversals, potentially bringing down the cost of edge modifications. Overall, efficient mechanisms for maintaining RCD values is needed. Here, we introduce the generic versions of the RCD maintenance algorithms, which update the RCD values of vertices up to the given hop count n. In other words, given the number of hops, n, the proposed algorithms maintain the RCD values for $n, n-1, \ldots , 1$.

multihopPrepareRCDsInsertion (Algorithm 9) is used at the beginning of the multihop traversal-based edge insertion algorithm, explained in Sect. 4.5. It prepares the RCD values before the multihop traversal operation is performed for the inserted edge, $(u_{1}, u_{2})$, and the given hop distance, n. This preparation is needed as the RCD values of the root(s) may have changed due to the updated degrees, and this change may have propagated to RCD values of other vertices. The preparation phase is performed assuming that the K values are intact. Those will be updated during the traversal, and a re-computation of RCDs would be required at the end (multihopRecomputeRCDs procedure).

The preparation starts with determining the root vertices based on their K values. If the K values of the extremities of the inserted edge are not equal, we increment the $RCD (r,h)$ value of root for all $h \le n$. The rationale behind this is that the root vertex gains a new neighbor with a higher K value, and by Definition 7, it increases all RCD values of root by one. Following this increment operation, we check whether the $RCD (r, h)$ has exceeded k, because this implies further changes in $RCD (\cdot ,h+1)$ values of r’s neighbor vertices (by Definition 7). In the preparation phase, $RCD (\cdot ,n)$ of a vertex only changes when $RCD (\cdot ,n-1)$ of a neighbor changes and that is what we are checking for. Remember that $RCD (u,n)$ is the number of u’s neighbors, w, where either $K(u)<K(w)$ or $K(u)=K(w)$ and $RCD (w,n-1) > K(u)$. Throughout the algorithm, we accumulate the vertices whose $RCD (\cdot ,h)$ values just exceed k in the next frontiers set, where h is the hop number. We avoid this accumulation operation if the last hop number h is being processed, since there is no need for further processing in that case. When the hop number h is greater than 1, we process the neighbors (with the same K value) of the vertices in the current frontiers set by incrementing their $RCD (\cdot ,h)$ values. We also perform checks to see whether k is exceeded and accordingly populate the next frontiers set.

If the K values of the extremities of the inserted edge are equal, we do different operations for $h=1$ and $h>1$, where h is the current hop number. For $h=1$, where $RCD (u,1)$ is actually equal to $MCD (u)$, we just increment the $RCD (\cdot ,1)$ values of both extremities of the inserted edge (by Definition 7) and perform checks to see whether k is exceeded and accordingly populate the next frontiers set. If $h>1$, we need to handle the new inserted edge separately. Let us say $u_1$ and $u_2$ are the extremities of the inserted edge. We first check the $RCD (u_1,h-1)$ [and dually $RCD (u_2,h-1)$] is greater than k. If so, we increment the $RCD (u_2,h)$ [and dually $RCD (u_1,h)$] and perform the k value checks to populate the next frontier as needed. After that, we process the neighbors (with the same K value) of the vertices in current frontiers set. One important difference in this step is that we exclude the edge between $u_1$ and $u_2$, because that edge is already handled.

Multihop algorithms are only applicable for the edge insertion operation. For removal, using 1-hop information (MCD values) is necessary and sufficient, as stated in the last paragraph of Sect. 4.3.3. Therefore, going for multihop information does not bring any additional benefit in terms of the running time. However, given that we are interested in sliding window scenarios, where removals happen together with insertions, we need to accommodate the maintenance of RCD values when there is an edge removal. For this purpose, we develop the multihopPrepareRCDsRemoval method. Detailed pseudocode and explanation can be found in “RCD maintenance for edge removal” of appendix.

After the multihop traversal, if the K values of some vertices are incremented, then this will create a cascading effect on RCD values of the vertices around. Efficiently handling the cascades and doing the update operations is again of great importance. Algorithm 10 finds those vertices whose RCD values need to be updated and efficiently updates these RCD values. It has two main parameters: the set of vertices whose K values are updated ($\mathtt{changed}$), and the hop distance until which RCD values are to be updated (n). We start the algorithm by marking the $\mathtt{changed}$ vertices as visited. Throughout the algorithm, we mark the vertices via the visited array to prevent duplicates during the update procedure. In the main for loop (the second one), we process the updates for each hop, in order. At each iteration, we populate the $\mathtt{changed}$ set with the updated vertices and then update the RCD values of the vertices in $\mathtt{changed}$. Cascading effect propagates by a single hop neighborhood at each iteration. In other words, if we assume that a vertex u has its K value updated, and we want 3-hop distance RCD values to be updated; $RCD (1)$, $RCD (2)$ and $RCD (3)$ of u will be updated. Furthermore, $RCD (2)$ and $RCD (3)$ values of some vertices in u’s hop-1 neighborhood will be updated, and $RCD (3)$ values of some vertices in u’s hop-2 neighborhood will be updated.

Pruning the vertices in the neighborhood is critical in making the procedure efficient. For the edge insertion case, given vertex v, we prune the neighborhood vertices by checking whether they are visited previously and whether the K value of the neighbor vertex is either equal to K value of v or equal to K value of v minus 1 (plus 1 for the edge removal case). The reason behind this check is based on Definition 7. $RCD (n)$ of a neighbor vertex may change iff the K values are equal. For the edge insertion case, given that there are also some vertices whose K values are incremented, we need to consider them as well by checking the neighbor vertices with one less K value. Likewise, for the edge removal, we check the neighbor vertices with one more K value, as stated with comments in the pseudocode of Algorithm 10. We accumulate the vertices to be updated in $\mathtt{changed}$ set and update their RCD values for the hop distance at that iteration. computeRCD procedure at the end finds the RCD values for all hop numbers up to h. It basically makes use of Definition 7. In summary, we handle the cascading effect of RCD maintenance efficiently by the aforementioned pruning techniques.

1.1 RCD maintenance for edge removal

Algorithm 11 adjusts the RCD values when there is an edge removal and is very similar to Algorithm 9. One important difference is that, instead of incrementing the RCD values, we need to decrement them whenever necessary. We also check whether the $RCD (\cdot ,h)$ value goes below $k+1$, which implies changes in $RCD (\cdot ,h+1)$ values of neighbor vertices. Another difference between Algorithms 11 and 9 exists when the K values of the removed edge extremities are equal. In this case, we need to remember the RCD values for all hop numbers before the edge removal operation. This enables us to process the hop numbers $h>1$.