Divisive clustering of high dimensional data streams

Abstract

Clustering streaming data is gaining importance as automatic data acquisition technologies are deployed in diverse applications. We propose a fully incremental projected divisive clustering method for high-dimensional data streams that is motivated by high density clustering. The method is capable of identifying clusters in arbitrary subspaces, estimating the number of clusters, and detecting changes in the data distribution which necessitate a revision of the model. The empirical evaluation of the proposed method on numerous real and simulated datasets shows that it is scalable in dimension and number of clusters, is robust to noisy and irrelevant features, and is capable of handling a variety of types of non-stationarity.

Appendix 1: Proofs

Before we can prove Lemma 2, we require the following preliminaries.

The algorithm for computing the dip of a distribution function F constructs a unimodal distribution function G with the following properties: (i) The modal interval of G, [m, M], is equal to the modal interval of the closest unimodal distribution function to F, which we denote by $F^U$, based on the supremum norm; (ii) $\Vert F - G\Vert _\infty = 2\Vert F - F^U\Vert _\infty$; (iii) G is the greatest convex minorant of F on $(-\infty , m]$; (iv) G is the least concave majorant of F on $[M, \infty )$. By construction, the function G is linear between its nodes. A node $n \le m$ of G satisfies $G(n) = \lim \inf _{x \rightarrow n}F(x)$, while a node $n \ge M$ of G satisfies $G(n) = \hbox {limsup}_{x \rightarrow n}F(x)$. If F is the distribution function of a discrete random variable, then G is continuous.

The function $F^U$ can be constructed by finding appropriate values $b<m, B>M$ s.t. $F^U$ is equal to $G+\hbox {Dip}(F)$ on [b, m], equal to $G-\hbox {Dip}(F)$ on [M, B], linearly interpolating between G(m) and G(M) and given any appropriate tails, which we choose to be linearly decreasing/increasing to 0 and 1 respectively.

Before proving Lemma 2, we require the following preliminary result, which relies on the notion of a step linear function.

Definition 3

(Step Linear) A function f is step linear on a non-empty, compact interval $I = [a, b]$, if

$$\begin{aligned} f(x) = \alpha + \beta \left\lfloor (x-a)\frac{n}{b-a}\right\rfloor , \quad \forall x \in I, \end{aligned}$$

for some $\alpha , \beta \in \mathbb {R}$ and $n \in {\mathbb {N}}$.

A step linear function is piecewise constant, and has n equally sized jumps of size $\beta$ spaced equally on I with the final jump ocurring at b. The approximate empirical distribution function $\tilde{F}$ (Sect. 4.2.2) is therefore step linear over the approximating intervals.

Proposition 4

Let f be step linear on an interval $I = [a, b]$, and satisfy $\lim _{x \rightarrow a^-}f(x) = \alpha - \beta$, where $\alpha , \beta$ as in the above definition for f. Let g be liner on I and continuous on a neighbourhood of I. Then

$$\begin{aligned} \sup _{x \in I} \vert f(x) - g(x) \vert\le & {} \max \left\{ \hbox {limsup}_{x \rightarrow a}\vert f(x) - g(x) \vert ,\right. \\&\left. \hbox {limsup}_{x \rightarrow b} \vert f(x) - g(x) \vert \right\} . \end{aligned}$$

Proof

Let $f_m$ and $f^M$ be linear on a neighbourhood of I s.t. they form the closest lower and upper bounding functions of f on I respectively. Since f is step linear, we have,

$$\begin{aligned} \lim _{x \rightarrow a^-}f(x) = f_m(a),&\lim _{x \rightarrow b^-}f(x) = f_m(b),\\ f(a) = f^M(a),&f(b) = f^M(b). \end{aligned}$$

We therefore have, by above and the fact that $g, f_m$, and $f^M$ are linear on I,

$$\begin{aligned} \sup _{x \in I}\vert f(x) - g(x) \vert\le & {} \max \left\{ \sup _{x \in I}\vert f^M(x) - g(x)\vert ,\right. \\&\left. \sup _{x \in I}\vert f_m(x) - g(x) \vert \right\} \\= & {} \max \left\{ \vert f^M(b) - g(b)\vert ,\right. \\&\vert f^M(a) - g(a)\vert , \vert f_m(a) - g(a) \vert ,\\&\left. \vert f_m(b) - g(b)\vert \right\} \\= & {} \max \left\{ \hbox {limsup}_{x \rightarrow a}\vert f(x) - g(x) \vert ,\right. \\&\left. \hbox {limsup}_{x \rightarrow b}\vert f(x) - g(x) \vert \right\} . \end{aligned}$$

$\square$

We are now in a position to prove Lemma 2, which states that the dip of a compactly approximated sample, as described in Sect. 4.2.2, provides a lower bound on the dip of the true sample.

Proof of Lemma 2

Let $I = [a, b]$ be any compact interval and $F_I$ the empirical distribution function of $(\mathcal {X}\cap I^c) \cup \hbox {Unif}(\mathcal {X}, I)$. Assume $\vert \mathcal {X}\cap I \vert >1$, since otherwise $F_I = F_\mathcal {X}$ and we are done. We can assume that the endpoints of I are elements of $\mathcal {X}$ since this defines the same uniform set. $F_\mathcal {X}$ and $F_I$ are therefore equal on $\hbox {Int}(I)^c$. In fact, since $\mathcal {X}$ consists of unique points, $\exists \epsilon > 0$ s.t. $F_I(x) = F_\mathcal {X}(x) \ \forall x \not \in (a+\epsilon , b-\epsilon )$. Define $F^\prime _I$ to be equal to $F_\mathcal {X}^U$ for $x \not \in \hbox {Int}(I)$ and linearly interpolating between $F_X^U(a)$ and $F_X^U(b)$. By construction $F^\prime _I$ is a continuous unimodal distribution function.

We now show $\Vert F_I - F_I^\prime \Vert _\infty \le \Vert F_\mathcal {X}- F_\mathcal {X}^U\Vert _\infty$. To see this, suppose that it is not true, i.e., $\exists x$ s.t. $\vert F_I(x) - F_I^\prime (x)\vert > \sup _y \vert F_\mathcal {X}(y) - F_\mathcal {X}^U(y)\vert$. Clearly $x \in \hbox {Int}(I)$ due to the equalities discussed above and the construction of $F^\prime _I$. Because of the continuity of $F_\mathcal {X}^U$ and $F_I^\prime$ and the equality of $F_\mathcal {X}$ and $F_I$ on $(a, a+\epsilon ) \cup (b-\epsilon , b)$, we have

$$\begin{aligned} \hbox {limsup}_{y \rightarrow a} \vert F_I(y) - F^\prime _I(y) \vert = \hbox {limsup}_{y \rightarrow a}\vert F_\mathcal {X}- F^U_\mathcal {X}(x) \vert \end{aligned}$$

and

$$\begin{aligned} \hbox {limsup}_{y \rightarrow b} \vert F_I(y) - F^\prime _I(y) \vert = \hbox {limsup}_{y \rightarrow b}\vert F_\mathcal {X}- F^U_\mathcal {X}(x) \vert . \end{aligned}$$

But by Proposition 4 one of these left hand sides is at least as large as $\vert F_I(x) - F_I^\prime (x) \vert$, leading to a contradiction.

We have shown that the addition of a single interval cannot increase the dip. We can apply the same logic to the now modified sample $(\mathcal {X}\cap I^c) \cup \hbox {Unif}(\mathcal {X}, I)$, iterating the addition of disjoint intervals to obtain a non-increasing sequence of dips. $\square$

In the above proof, we do not show that $F_I^\prime$ is the closest unimodal distribution function to $F_I$, however its existence necessitates the closest one being at least as close. Now, the sample approximations we employ still contain a full t atoms after t observations, however, they can be stored in ${\mathcal {O}}(k)$ for k intervals. We can easily show that the dip of such a sample approximation can be computed in ${\mathcal {O}}(k)$ time.

Proposition 5

The dip of a sample consisting of k uniform sets with disjoint ranges can be computed in ${\mathcal {O}}(k)$ time.

Proof

We begin by showing that there exists a unimodal distribution function which is linear on the ranges of the uniform sets and which achieves the minimal distance to the empirical distribution function of the sample.

Let F be a continuous unimodal distribution function s.t. $\Vert F - \tilde{F}\Vert _\infty = \hbox {Dip}(\tilde{F})$. Define $F^\prime$ similarly to in the above proof to be the continuous distribution function which is equal to F outside and at the boundaries of the intervals defining the uniform sets and linearly interpolating on them. Using the same logic, we know that $\sup _{x}\vert F^\prime (x) - \tilde{F}(x) \vert \le \sup _x\vert F(x) - \tilde{F}(x)\vert$, hence $\Vert F^\prime - \tilde{F}\Vert _\infty = \hbox {Dip}(\tilde{F})$.

Proposition 4 ensures that points in the interior of the intervals will not be chosen by the dip algorithm as end points of the modal interval of G, nor points at which the difference between the functions is supremal. The possible choices for these locations is therefore ${\mathcal {O}}(k)$, and the algorithm need not evaluate the functions except at the endpoints of the intervals. $\square$

Finally, we provide a proof of Proposition 3.

Proof of Proposition 3

For $s > 1$ we have $\Vert v_s - v_{s-1}\Vert = \Vert v_s\Vert \Vert v_s-v_{s-1}\Vert \ge \vert v_s \cdot (v_s - v_{s-1})\vert = \vert v_sv_{s-1}-1\vert$, since $\Vert v_t\Vert = 1 \ \forall t$. Therefore, since $\{v_t\}_{t=1}^\infty$ is almost surely convergent, and therefore almost surely Cauchy, we have $v_s \cdot v_{s-1} \xrightarrow {a.s.} 1 \Rightarrow \arccos (v_s \cdot v_{s-1}) \xrightarrow {a.s.}0$. Now, we can easily show that,

$$\begin{aligned} \lambda _t \le \gamma ^{t-1}\lambda _1 + (1-\gamma )\sum _{i=1}^{t-2}\gamma ^i\arccos (v_{t-i}\cdot v_{t-i-1}). \end{aligned}$$

Take $\epsilon > 0$ and t large enough that $\gamma ^{t-1}\lambda _1 < \gamma \epsilon$, and $t>k+2$, where $k = \lfloor \log (\epsilon (1-\gamma )/2\pi )/\log (\gamma )-1\rfloor$. Consider,

$$\begin{aligned} \sum _{i=1}^{t-2}\gamma ^i\arccos (v_{t-i}\cdot v_{t-i-1})\le & {} \sum _{i=1}^{k}\arccos (v_{t-i}\cdot v_{t-i-1})\\&+\frac{\pi \gamma ^{k+1}}{1-\gamma }, \end{aligned}$$

and $\frac{\pi \gamma ^{k+1}}{1-\gamma } \le \frac{\epsilon }{2}$. In all,

$$\begin{aligned} \lambda _t > \epsilon \Rightarrow \sum _{i=0}^k\arccos (v_{t-i}\cdot v_{t-i-1}) > \epsilon /2. \end{aligned}$$

Notice that k does not depend on t. With probability 1, for any given $\epsilon >0$ there is a ${\mathcal {T}}$ s.t. $T>{\mathcal {T}}$ implies $\sum _{i=0}^k\arccos (v_{T-i}\cdot v_{T-i-1})\le \epsilon /2$, implying $\lambda _T \le \epsilon$ for all $T > {\mathcal {T}}$, and the result follows. $\square$