EAD: elasticity aware deduplication manager for datacenters with multi-tier storage systems

Abstract

The popularity of Big Data applications places pressures on storage systems to efficiently scale to meet the demand. At the same time, new developments like solid-state drives have changed to traditional storage hierarchy. Cloud storage systems are transitioning towards a hybrid architecture consisting of large amounts of memory, solid-state disks (SSDs), and traditional magnetic hard disks (HD). This paper presents elasticity aware deduplication (EAD), a data deduplication framework designed for multi-tier cloud storage architectures consisting of SSD and HD. EAD dynamically adjusts the deduplication parameters at runtime in order to improve performance. Experimental results indicate that EAD is able to detect more than 98% of all duplicate data, but it only consumes less than 5% of expected memory space. Additionally, EAD saves approximately 74% of overall IO access cost compared to the traditional design.

Appendix: Proof and error analysis of expectation of duplication ratio (EDR)

Real-world workloads in enterprise environments have different I/O behaviors, but their pattern can be regresses to some long-term predicable streams. This can be regarded as a super set of number of $n_s$ homogeneous T-size segments. This stream can be modeled by an average segment (Seg), such that any results which holds on this average segment also holds for the entire stream, i.e., $EDR_{ Cur }(Stream) \approx DR_{ Cur }(Stream)$

$$\begin{aligned} \lim _{ {n_s} \rightarrow \infty }{ DR_{ Cur }(Stream) =DR(Seg) }. \end{aligned}$$

(1)

Therefore, we can reduce the problem from “level 3” to “level 2”. In this section, we demonstrate how EDR(Seg) is close to $DR_(Seg)$ during runtime. We first make the following definitions:

Table 3 Four cases with different sampling degrees and dupSet sizes

Definition 1

dupSet We first define duplicated set (“dupSet”) as the set of same-value chunks in a segment. For example, in Fig. 18 case 2, there are four dupSets, i.e.,

$$\begin{aligned} dupSet_1=\{1,1,\}, dupSet_2=\{2,2,2,2,2,2\},\nonumber \\ dupSet_3=\{3,3,3\}, dupSet_4=\{4,4,4,4,4\}. \end{aligned}$$

(2)

Definition 2

Fully and partial sampled segments For each dupSet $d_i$ among T chunks in a segment, if at least one chunk per dupSet is sampled in ${\mathbb {B}}$, then this segment is considered fully sampled; otherwise it is partially sampled. Note that the worst case of the partially sampled segment is non-samples, which will happen if $\kappa >0$. Based on this definition, we can divide the problem into four cases as shown in Table 3. Fig. 18 also shows examples for each case. Later we prove that Case 1 and 3 are accurate, while Case 2 and 4 are with known estimation errors. Here we also show the probabilities of each cases. Given the size of each segment T, size of each dupSet $d_i$, sample number $\kappa (\ge n)$, and total number of dupSets n, the probability that a segment is fully sampled is:

$$\begin{aligned} P_{ FullSmp }=\frac{ \prod _{ i=1 }^{ n }{ ({ C }_{ 1 }^{ d_{ i } }) } }{ { C }_{ \kappa }^{ T } } =\frac{ ({ C }_{ 1 }^{ d })^{ n } }{ { C }_{ \kappa }^{ T } } =\frac{ d^{ n } }{ { C }_{ \kappa }^{ T } } , \end{aligned}$$

(3)

and the probability that a segment is partial sampled is:

$$\begin{aligned} P_{ PartSmp }=1-\frac{ d^{ n } }{ { C }_{ \kappa }^{ T } } . \end{aligned}$$

(4)

Definition 3

Estimation error To evaluate the estimation accuracy, we define the estimation error $\delta$ which is the distance between EDR and DR:

$$\begin{aligned} \delta =\left| EDR(Seg)-DR(Seg)\right| , \end{aligned}$$

(5)

where $\delta \in [0, +\infty )$, and the less $\delta$ the more accuracy EDR is. When $\delta =0$, the estimation is fully accurate. Based on these assumption and definitions, we now calculate the $\delta$ of each case:

[Case 1] Fully sampled, equal DupSet size

We first calculate the actual $DR_{Cur}$ of a segment:

$$\begin{aligned} DR_(Seg) = 1 - \frac{n}{T}=1-\frac{1}{d}. \end{aligned}$$

(6)

We then investiage the EDR. Since the “fully sampled” case ensures there is at least one sample per dupSet, we can denote $\kappa =n+\varepsilon$, where $\varepsilon$ ($\in [0,T-n])$ is the redundant samples that are duplicated with existing one sample per dupSet. We further let each dupSet $d_i$ being assigned $\kappa _i=1+\varepsilon _i$ samples, where $\varepsilon _i$ is the number of redundant samples of $d_i$, and $\sum _{ i\in {\mathbb {B}}}^{ }{\varepsilon _i}=\varepsilon$. Fig. 18 Case 1 illustrates an example, where $\kappa _3=1+\varepsilon _3=1+1=2$. Therefore, EDR(Seg) can be calculated as:

$$\begin{aligned} EDR(Seg) =1-\frac{ 1 }{ \kappa } \sum _{ i\in { { { \mathbb {B} } } } }^{ }{ \left ( \frac{h_{Smp_i}}{h_{Seg_i}}\right)} =1-\frac{ 1 }{ \kappa } \left(\sum _{ i\in {\mathbb {B}} }^{ }{ \frac{ 1 }{ d_{ i } } } +\sum _{ i\in {\mathbb {B}} }^{ }{ \frac{ \varepsilon _{ i } }{ d_{ i } } } \right). \end{aligned}$$

(7)

Since in Case 1 all dupSets have the same size ($d_i=d$), Eq. 7 can be simplified as:

$$\begin{aligned} lhs=1-\frac{ 1 }{ \kappa } \left(\frac{ 1 }{ d } \cdot n+\frac{ 1 }{ d } \cdot \varepsilon \right) =1-\frac{ 1 }{ \kappa d } (n+\varepsilon ) =1-\frac{ \kappa }{ \kappa d } =1-\frac{ 1 }{ d } . \end{aligned}$$

(8)

δ = 0, which proves that our EDR can reflect the duplication ratio of the accumulated workload with 100% accuracy in Case 1.

[Case 2] Fully sampled, non-equal DupSetSize

Fig. 18

Examples of different sampling scenarios

It is straightforward to get the real $DR_(Seg)$ of one segment as:

$$\begin{aligned} DR(Seg)=1-\frac{ n }{ T } =1-\frac{ 1 }{ \bar{ d } } . \end{aligned}$$

(9)

However, to calculate EDR, we need to divide the “fully sampled” case into two sub cases: (2.1) Exact fully sampled: each dupSet has exactly one sample in ${\mathbb {B}}$, and $\kappa = n$; and (2.2) Redundantly fully sampled: at least one dupSet has more than one samples in ${\mathbb {B}}$, i.e., $\forall \kappa _i \ge 1, i \in [1,n]$ and $\kappa > n$, where $\kappa _i$ is the sample number of $d_i$.

[Case 2.1] Exact fully sampled

Since $\kappa = n$ and $\kappa _i=1$, we have:

$$\begin{aligned} EDR(Seg)=1-\frac{ 1 }{ \kappa } \sum _{ i\in {\mathbb {B} }}^{ }{ \left(\frac{ h_{Smp_i} }{ h_{Seg_i}} \right) } =1-\frac{ 1 }{ n } \left(\sum _{ i=1 }^{ n }{ \frac{ 1 }{ d_{ i } } } \right). \end{aligned}$$

(10)

Based on Eqs. 9 and 10, the estimation error is:

$$\begin{aligned} \delta & =\left| EDR(Seg)-DR(Seg)\right| =\left| \frac{ 1 }{ \bar{ d } } -\frac{ 1 }{ n } \left(\sum _{ i=1 }^{ n }{ \frac{ 1 }{ d_{ i } } } \right)\right| \nonumber \\ & =\left| \frac{ 1 }{ A(D) } -\frac{ 1 }{ H(D) } \right| =\frac{ 1 }{ H(D) }-\frac{ 1 }{ A(D) }. \end{aligned}$$

(11)

In Eq. 11, we use notation A(D) and H(D) to represent the arithmetic mean and harmonic mean of sample set $D=\{d_i|d_i \in Seg\}$ respectively. It is always true that $0 \le A(D) \le H(D)$, so we remove the absolute value sign. Aiming for better performance, we further investigate under what case $\delta$ can be minimized. We conduct several experiments tuning dupSet’s sizes under the “exact fully sampled in non-equal dupSet” case.

Figure 19 shows seven representative $\delta$ curves with different size of one subSet (subSet A). There are three dupSets A,B,C in the segment with size of T. For each curve, we iterate the size of dupSet B. Obviously dupSet C’s size is $(T-|A|-|B|)$. We observe that (1) when the remaining two dupSets (B and C) have exactly same sizes, the $\delta$ will be the lowest of that curve; and (2) when all of three dupSets have the same sizes, the $\delta$ is the global lowest among all curves. That is to say, the more equalikely of size of each dupSet is, the lower estimation error will be.

Fig. 19

$\delta$ of a three-dupSet segment with different dupSet sizes

Fig. 20

$\delta$ of three-dupSet segment with different redundant samples

[Case 2.2] Redundantly fully sampled

All dupSets are sampled and some of them have more than one sample, i.e., $\kappa =n+\varepsilon$. We have:

$$\begin{aligned} EDR(Seg)=1-\frac{ 1 }{ \kappa } \sum _{ i\in { { \mathbb {B} } } }^{ }{ \left(\frac{ h_{ Smp_{ i } } }{ h_{ Seg_{ i } } } \right) } =1-\frac{ 1 }{ n+\varepsilon } \left(\sum _{ i\in {\mathbb {B}} }^{ }{ \frac{ 1+\varepsilon _{ i } }{ d_{ i } } } \right). \end{aligned}$$

(12)

The estimation error is:

$$\begin{aligned} \delta =\left| EDR(Seg)-DR(Seg)\right| =\left| \frac{ 1 }{ \bar{ d } } -\frac{ 1 }{ n+\varepsilon } \left(\sum _{ i\in {\mathbb {B}} }^{ }{ \frac{ 1+\varepsilon _{ i } }{ d_{ i } } } \right)\right| . \end{aligned}$$

(13)

Here we introduce a super set $D_{Sup}$, which is extension of D, i.e., $D_{Sup}=\{d_i|x_j \in {\mathbb {B}}, {x_j}\in d_i\}$. For example, in Fig. 18 Case 2, $D=\{1,2,3,4\}$ and $D_{Sup}=\{1,2,3,3,4\}$. Therefore, we can use the harmonic mean of $D_{Sup}$ to help to represent the second part of Eq. 13. We further use $\Delta (D_{Sup}) = \sum _{ i \in {\mathbb {B}} }^{ }{ (\varepsilon _i \cdot d_i })$, to represent the total size of dupSets that those redundant samples are associated with, where $\varepsilon _i$ is the number of redundant samples from $d_i$. For example, in Fig. 18 Case 2, we have $\Delta (D_{Sup}) = 1 \times 3 =3$. Therefore, Eq. 13 equals to:

$$\begin{aligned} \delta =\left| \frac{ 1 }{ A(D) } -\frac{ 1 }{ H(D_{Sup}) } \right| \end{aligned}$$

$$\begin{aligned} & = \left| \left[\frac{ 1 }{ A(D_{Sup}) } -\frac{ 1 }{ H(D_{Sup}) }\right] +\left[\frac{ 1 }{ A(D) }-\frac{ 1 }{ A(D_{Sup}) } \right]\right| \nonumber \\ & =\left| \left[\frac{ 1 }{ A(D_{Sup}) } -\frac{ 1 }{ H(D_{Sup}) }\right] +\left[\frac{ n }{ T }-\frac{ n+\varepsilon }{ T+\Delta (D_{Sup})}\right] \right| . \end{aligned}$$

(14)

As shown in Eq. 14, the non-equal case estimation error comes from two parts: (1) the difference between arithmetic and harmonic mean (same as Eq. 11); and (2) the number of picked samples which are not necessarily proportional to each corresponding dupSets’ sizes (i.e., dupSets’ weights in the segment). To further investigate $\delta$, Fig. 20 shows the relationship between $\delta$ and different number and distribution of redundant samples in a three-non-equal-dupSet segment example. In this experiment, $|dupSetA|=10\%T$, $|dupSetB|=30$,and $|dupSetC|=60\%T$. We increase the number of samples from each dupSet in different orders. For example, curve with “10, 30, 60” means that firstly each dupSet has one sample, and then we keep adding one more sample of $10\%T$ dupSet until it is fully indexed. Later we repeat it for $30\%T$ and $60\%T$ dupSets until the entire segment is fully indexed. We observe that, if we pick lots of samples from $10\%T$ dupSet at beginning, the error will reach the worst. More the samples from one dupSet, more the weight of that dupSet will be in the final estimation. It will be more accurate if the number of picked samples are proportional to each dupSet, or from a dupSet who shares a relatively big fraction of the segment. Notice that, we also conduct experiments where picking samples with random or some distributions, and consecutively we end up with the same conclusion. Thus, we show part of results to demonstrate the bounds of $\delta$.

[Case 3] Partially sampled, equal DupSetSize

Let $T_S$ be the sampled part of a segment, $n'$ be the number of sampled dupSets, and $\varepsilon '$ be the total redundant samples from those sampled dupSets. We have $n'+\varepsilon '=\kappa$. Then EDR of the segment is:

$$\begin{aligned} EDR(Seg)=1-\frac{ 1 }{ \kappa } \sum _{ i\in { { { \mathbb {B} } } } }^{ }{ \left(\frac{ h_{ Smp_{ i } } }{ h_{ Seg_{ i } } } \right) } \end{aligned}$$

$$\begin{aligned} =1-\frac{ 1 }{ n'+\varepsilon ' } \left(\sum _{ i\in { \mathbb {B}(T_{ S }) } }^{ }{ \frac{ 1 }{ d_{ i } } } +\sum _{ i\in { \mathbb {B}(T_{ S }) } }^{ }{ \frac{ \varepsilon '_{ i } }{ d_{ i } } } \right) \end{aligned}$$

(15)

$$\begin{aligned} =1-\frac{ 1 }{ n'+\varepsilon ' } \left(\frac{ n' }{ d } +\frac{ \varepsilon ' }{ d } \right) =1-\frac{ 1 }{ d } = DR(Seg). \end{aligned}$$

(16)

Therefore, like Case 1, EDR can 100% reflect the DR(Seg) in Case 3.

$$\begin{aligned} \delta =\left| EDR(Seg)-DR(Seg)\right| =0. \end{aligned}$$

(17)

[Case 4] Partially sampled, non-equal DupSetSize

Similar to Case 2, the $\delta$ can be calculated as:

$$\begin{aligned} \delta =\left| EDR(Seg)-DR(Seg) \right| =\left| \frac{ 1 }{ \bar{ d } } -\frac{ 1 }{ n'+\varepsilon ' } \left(\sum _{ i\in { B } }^{ }{ \frac{ 1+\varepsilon '_{ i } }{ d_{ i } } } \right) \right| . \end{aligned}$$

(18)

Also, Eq. 18 can further be divided into two sub cases:

[Case 4.1] Exact partially sampled

To differentiate with fully sampled case, we use the notation ${\mathbb {B(T_S)}}$ to represent the estimation base that has partial samples of all dupSets ($T_S$ of segment T and $T_S$ is a subset of T). Similar to Case 2.1, we let ${ { D } }_{ }^{ T_{ S } }=\{d_i|i \in \mathbb {B(T_S)}\}$ be the sampled dupSet’s set. Since each sampled dupSet has only one sample, so $\varepsilon '=0$, and:

$$\begin{aligned} \delta =\left| \frac{ 1 }{ A(D) } -\frac{ 1 }{ H({ { D } }_{ }^{ T_{ S } }) } \right| \end{aligned}$$

$$\begin{aligned} =\left| \left[\frac{ 1 }{ A({ { D } }_{ }^{ T_{ S } }) } -\frac{ 1 }{ H({ { D } }_{ }^{ T_{ S } }) } \right]+\left[\frac{ 1 }{ A(D) } -\frac{ 1 }{ A({ { D } }_{ }^{ T_{ S } }) } \right] \right| . \end{aligned}$$

(19)

[Case 4.2] Redundantly partially sampled

Similar to Case 2.2, let ${ { D } }_{ Sup }^{ T_{ S } }=\{d_i|x_j \in {\mathbb {B(T_S)}}, {x_j}\in d_i\}$ represent the super set of ${ { D } }_{ }^{ T_{ S } }$. For those sampled dupSets, there exist redundant samples, i.e., $\varepsilon '\ne 0$, as a result we have:

$$\begin{aligned} \delta =\left| \frac{ 1 }{ A(D) } -\frac{ 1 }{ H({ { D } }_{ Sup }^{ T_{ S } }) } \right| \end{aligned}$$

$$\begin{aligned} =\left| \left[\frac{ 1 }{ A({ { D } }_{ Sup }^{ T_{ S } }) } -\frac{ 1 }{ H({ { D } }_{ Sup }^{ T_{ S } }) } \right]+\left[\frac{ 1 }{ A(D) } -\frac{ 1 }{ A({ { D } }_{ Sup }^{ T_{ S } }) } \right] \right| . \end{aligned}$$

(20)

We can see from Eqs. 19 and 20, the estimation error of partially sampled case comes from two source: arithmetic and harmonic means difference, and using partial dupSets to estimate the entire segment. The former has already been explained in Case 2, and the latter is simply based on the coverage of ${\mathbb {B}}(T_S)$ over the entire segment.