A Comparison of Cryptanalytic Tradeoff Algorithms

Abstract

Three time-memory tradeoff algorithms are compared in this paper. Specifically, the classical tradeoff algorithm by Hellman, the distinguished point tradeoff method, and the rainbow table method, in their non-perfect table versions, are treated.

We show that, under parameters and assumptions that are typically considered in theoretic discussions of the tradeoff algorithms, the Hellman and distinguished point tradeoffs perform very close to each other and the rainbow table method performs somewhat better than the other two algorithms. Our method of comparison can easily be applied to other situations, where the conclusions could be different.

The analysis of tradeoff efficiency presented in this paper does not ignore the effects of false alarms and also covers techniques for reducing storage, such as ending point truncations and index tables. Our comparison of algorithms fully takes into account success probabilities and precomputation efforts.

1 Introduction

There are numerous security systems in use today that rely on passwords. Access to much of the content on a network requires one to log in with a password, and many file formats today have security features that restrict access to the file until the correct password is supplied. These systems are usually based on a password hash technique, which operates by storing a one-way function image of the password in the file or on the system. Indeed, storing the password in its raw form within the file to which one wishes to set access control would be meaningless. Authentication of a user is performed by recomputing the one-way function image from a freshly supplied password and comparing the result with the stored password hash.

A time-memory tradeoff algorithm attempts to recover the password from the knowledge of the one-way function image, with the help of a table created through precomputation. The massive precomputation that is required before the actual attack can be mounted is the largest barrier in applying the time-memory tradeoff technique to any specific security system. However, the precomputation cost is roughly proportional to the size of the password space and, since many users do not use strong passwords, the tradeoff attacker is free to choose a manageable set consisting of short or more likely passwords and decide to be satisfied with recovering only those passwords belonging to this set. Then the precomputation requirement does not stand as an impenetrable barrier to the tradeoff attack.

It has long been known that properly salting a password can remove any realistic threats of the time-memory tradeoff attacks. The security system concatenates a randomly generated string (salt) of sufficient length to the user-supplied password before computing the one-way function image. The salt value that was used is stored alongside the computed password hash so that it is available to the system for the one-way function recomputation whenever a user needs to be authenticated. The effective number of passwords is increased by the use of salts and this can increase the precomputation requirement of a tradeoff attack to an unrealistic degree.

Nevertheless, the salting countermeasure is still not being used in many proprietary systems, and some systems are known to be using both the newer salted and the older non-salted versions of the security system simultaneously to remain compatible with older systems. Hence, the time-memory tradeoff technique still remains a powerful tool against these vulnerable password hash systems. Since human-generated passwords will continue to be used for the foreseeable future, one would like to fully understand the powers and limitations of the tradeoff techniques.

There are a large number of tradeoff algorithm variants, and we will restrict ourselves to the three major tradeoff algorithms in this work. The first algorithm we study is the original tradeoff algorithm [14] devised by Hellman. The second algorithm is the distinguished point method, which is attributed to Rivest in [10]. The number of table lookups that are required by a Hellman tradeoff is significantly reduced in this slightly modified method. The final algorithm we consider is the rainbow table method [23], announced by Oechslin. The precomputation table for this method is structurally different from the previous two versions.

Let us briefly mention some of the more notable tradeoff variants or techniques that we are not treating in this work. The first is the perfect table version of the distinguished point method [8]. This is a variant of the distinguished point method where some of the redundancies contained in the precomputed tables are removed and replaced with nonoverlapping data generated through additional precomputation. The more efficient usage of storage leads to better performance during the actual attack, at the expense of higher precomputation cost. The removal of redundancies is facilitated by the distinguished point technique and cannot be done as easily with the classical Hellman algorithm, but the rainbow table method also admits a perfect table version [23] naturally. The perfect table versions of tradeoff algorithms are of interest due to their better efficiency during the attack phase. However, analyzing them at the accuracy level aimed for by the current paper is quite delicate, and is left as a subject of future study.

Another class of tradeoff variants that we do not consider is the multi-target versions of the tradeoff algorithms [2, 5, 6, 13], which are usually referred to as the time-memory-data tradeoffs. The objective of these algorithms is to recover at least one of the many original inputs that were used to create the multiple one-way function images that are supplied as inversion targets. This class of algorithms attracted attention as realistic attacks on stream ciphers, but present-day stream ciphers are designed to withstand these attacks. The most practical application of the tradeoff technique today is with the password hash systems, and we will present the current work with this application in mind.

Even though a considerable portion of this paper is devoted to the performance analyses of the three major tradeoff algorithms, the main motivation for this work was to determine which time-memory tradeoff algorithm is the best. Providing a fair and acceptable answer to this seemingly simple question is the ultimate goal of this paper.

It has been shown [3, 4] that, if we restrict ourselves to a certain class of algorithms, the explicit tradeoff algorithms that are known today already achieve the best tradeoff efficiency one can hope for, at least asymptotically. However, the measure of efficiency considered by this theory is only accurate up to a small multiplicative factor. In practice, experience seems to be a critical factor in deciding which algorithm to use, and researchers have varied opinions on which algorithm performs better.

Comparison of tradeoff algorithms has been a controversial subject. There are claims of superiority of one algorithm over another, but, in many cases, these arguments are either heuristic or based on complexity analyses that are not accurate up to small constant factors. There are at least two obstacles to providing a fair comparison of tradeoff algorithms. The first is that the online time of each algorithm is hard to predict accurately, due to certain events called false alarms. Some answers to this problem may be found in [1, 15] for the Hellman and rainbow cases. The current paper relies heavily on these results. The second obstacle concerns the minimal number of bits required to store each precomputation table entry. In particular, a technique for storage optimization called ending point truncation has not yet been fully analyzed.

There is a naturally occurring measure of how efficiently a tradeoff algorithm balances time against storage in achieving its goal, and the accurate value of this efficiency measure becomes accessible once the first obstacle mentioned above is resolved. As was first noted in [3, 4], the measure of tradeoff efficiency has been expressed in different units for different algorithms. In this work, by extending the approach of [3, 4], we carefully convert the tradeoff efficiency measures for the three algorithms to a common unit so that they may be directly compared. The unification of units is intimately connected to the second obstacle mentioned above. We also carefully treat the time taken for table lookups during our initial transition of units.

The above two obstacles that are due to our lack of accuracy in presenting the tradeoff efficiency figures can be overcome through rigorous algorithm analyses, but there is yet another problem which is related to the precomputation cost. Currently there is no widely accepted way of comparing two algorithms that can achieve different tradeoff performances only after the investment of different precomputation efforts. Due to this difficulty, many comparisons of tradeoff algorithms have focused on the above-mentioned measure of balancing capability and have ignored the cost of precomputation.

In this work, we clear all the obstacles mentioned so far and provide a fair comparison between tradeoff algorithms. More precisely, we present a method to visualize what can be achieved by each algorithm in terms of precomputation cost and tradeoff efficiency. This will be done in a unified way so that the range of choices made possible by each algorithm can directly be compared against each other. A tradeoff implementer can use this information to decide on which algorithm to use and which set of parameters to use with the algorithm. The judgement of which algorithm is more suitable depends on how the user values the precomputation cost and tradeoff efficiency relative to each other, and, in most cases, the judgement cannot be done in an objective manner.

While presenting the above comparison method, we will mainly focus on a certain set of parameters and environmental assumptions that are typically considered during theoretic analyses of tradeoff algorithms. Under the circumstances under focus, the performances of the classical Hellman and the distinguished point methods are shown to be very close to each other. When placed under the additional requirement that the success rates of the tradeoff algorithms must be high, the rainbow table method is shown to outperform the other two algorithms. These comparison conclusions will stand true for any relative valuing of the precomputation cost and tradeoff efficiency, as long as we are working with a typical situation. Comparisons for other situations can easily be done by following through our methods, and the resulting conclusions could be different.

The remainder of this paper is organized as follows. In the next section, we fix the notation and terminologies while reviewing previous results related to this work. Section 3 clarifies the connection between the theory of tradeoff algorithms and the use of the algorithms in attacking password hash systems. In Sects. 4, 5, and 6, we study the distinguished point, Hellman, and rainbow table tradeoff algorithms, in turn. For each algorithm, we present an accurate tradeoff efficiency figure that does not ignore small multiplicative factors and also analyze the applicable storage reduction techniques. These sections overcome the first and second obstacles that were mentioned previously. Comparisons of tradeoff efficiencies under different parameter sets for the same algorithm are made in Sect. 7. Finally, our goal of algorithm comparison is reached in Sect. 8, and the work is summarized in Sect. 9. The experimental data supporting the arguments of this paper are given in Appendix E. We acknowledge that a small part of this work was previously made public through [21].

2 Time-Memory Tradeoff Algorithms

In this section we review the basic theory of time-memory tradeoffs and fix the notation that is used throughout the paper. We introduce previous results that are related to the results here, but make no attempt at providing a complete history or survey of the time-memory tradeoff technique. In particular, the perfect table tradeoffs algorithms are explained, but advancements concerning their analyses or comparisons are not introduced.

Below, after stating some simple technical facts, we describe the three major tradeoff algorithms, and then explain some auxiliary techniques that can enhance their tradeoff efficiency. The descriptions are condensed, and readers that are new to the time-memory tradeoff technique should consult the original papers for more detail.

Throughout this paper, the function will always act on a set  of size N and the k-times iterated composition F∘⋯∘F of F is written as F ^k.

2.1 Technical Preliminaries

Many of the results given in this paper are expected values for random functions. In very rough terms, a random function F is a function that assigns independent and random values to each of its arguments . As briefly discussed in [12, 16, 24], working with a random function is equivalent to choosing a function uniformly at random from the set of all functions of a certain domain and codomain. In other words, any expected value expressed for a random function is an average computed over all functions.

For large positive integers a and b such that a=O(b), we can use the approximation

$$ \biggl(1-\frac{1}{{\textup {\textsf {b}}}} \biggr)^{\textup {\textsf {a}}}\approx e^{-{\textup {\textsf {a}}}/{\textup {\textsf {b}}}}, $$

which is very accurate. For example, when a=b, the error in the approximation is bounded by $\frac{e}{{\textup {\textsf {b}}}}$. This approximation is frequently used in the tradeoff literature without any explanation and is also used very frequently in this paper. Its use can be justified through easy computation, which is explicitly carried out in Appendix A.

The final technical fact we present concerns the image size of a random function. Let be a random function. If is of size m ₀, then the size of is expected to be

$$ m_1 = {\textup {\textsf {N}}}\biggl\{1- \biggl(1 - \frac{1}{{\textup {\textsf {N}}}} \biggr)^{m_0} \biggr\} \approx {\textup {\textsf {N}}}\bigl(1- e^{-m_0/{\textup {\textsf {N}}}} \bigr). $$

(1)

An elementary proof of this statement can be given by treating it as a classical occupancy problem.

More generally, the expected kth iterated image size can be iteratively computed through

$$ m_j = {\textup {\textsf {N}}}\bigl(1- e^{-m_{j-1}/{\textup {\textsf {N}}}} \bigr) \quad (j=1,\dots,k), $$

(2)

starting from . This is stated in [11, 20] to hold asymptotically. The explicit statements given there are only for the case when the input set  is the complete domain , but the case where  is strictly smaller than the complete domain is used in [23] to state the success probability of a non-perfect rainbow table. The relation between (1) and (2) is discussed in detail in Appendix B.

2.2 Overview of the Tradeoff Technique

Let F be fixed to a publicly known one-way function. The goal of any tradeoff algorithm is to recover the input x, when it is given the function image y=F(x). The correct answer x and the inversion target y may occasionally be referred to as the password and password hash, respectively.

Any tradeoff algorithm consists of a precomputation phase and an online phase. The precomputation phase algorithm gathers information about the one-way function F through extensive computation and stores a condensed digest of the gathered information in a precomputation table. When an inversion target y=F(x) is given, the online phase algorithm is executed to recover x from y, using the precomputation table as reference.

To be meaningful as an attack, the size M of the precomputation table must be smaller than N and the online phase algorithm should return the answer in a time T that is shorter than N. Note that N is the size of the complete dictionary and is also the time required for an exhaustive search. A tradeoff algorithm should allow tradeoffs between storage and online time in the sense that online attack time T can be reduced by using a larger storage M and, conversely, smaller M can be used if a longer T is acceptable. Tradeoff algorithms are usually implemented with the intention of running a large number of online phases after a single precomputation phase. This gives one justification for a precomputation effort that is larger than exhaustive search.

Even though every implementation of the tradeoff technique works with a specific one-way function F, analyses of the tradeoff techniques are always done with the assumption that F is a random function.

2.3 Hellman Tradeoff

The first algorithm we explain is the classical tradeoff algorithm of Hellman [14].

2.3.1 Parameter Setup

Certain parameters need to be fixed before the precomputation phase can be started. Positive integers m and t that satisfy the relation mt ²≈N are fixed. This equation is referred to as the matrix stopping rule. Another positive integer ℓ≈t, which will become the number of tables, is also fixed.

In this paper, we let the parameters m and t satisfy mt ²=H _msc N, with a matrix stopping constant H _msc that is neither very large nor too close to zero. Much of the tradeoff literature sets H _msc=1. The conditions we have given to H _msc and ℓ may (inaccurately) be expressed as H _msc=Θ(1) and ℓ=Θ(t), respectively. The parameters are always assumed to be reasonable in the sense that 1≪m,t≪N. The tradeoff algorithms behave somewhat differently when instantiated with extreme parameters.

The reduction functions , one for each k=1,…,ℓ, are fixed. These may be any family of simple bijections that are very easy to compute. When N is a power of 2 and consists of non-negative integers less than N, bit permutations or XOR-ing by constants are practical choices for reduction functions. The colored iterating functions are defined through F _k =R _k ∘F.

2.3.2 Precomputation Phase

In the precomputation phase, the process explained below is repeated ℓ times, once for each 1≤k≤ℓ, to build ℓ tables.

We start by choosing m random starting points . Hellman required each of these starting points to be chosen independently at random, but most researchers today see the starting points as being distinct. For each 1≤i≤m, we initially set $\mathbf {x}_{i,0}^{k} = \mathbf {sp}_{i}^{k}$ and recursively compute $\mathbf {x}_{i,j}^{k} = F_{k}(\mathbf {x}_{i,j-1}^{k})$ for 0<j≤t. The final point reached by each chain of iterative computations is said to be an ending point $\mathbf {ep}_{i}^{k} = \mathbf {x}_{i,t}^{k} = F_{k}^{t}(\mathbf {sp}_{i}^{k})$. The ordered pairs $\{(\mathbf {sp}_{i}^{k}, \mathbf {ep}_{i}^{k})\}^{m}_{i=1}$ are stored as the kth Hellman table, after being sorted with respect to the ending points.

The collection of all points $\{\mathbf {x}_{i,j}^{k}\}_{i,j}$, associated with an iterating function F _k of one color k, is said to be a Hellman matrix of size m×t. One usually visualizes a Hellman matrix as follows:

$$ \begin{array}{ccccccccccc} \mathbf {sp}_1^k = \mathbf {x}_{1,0}^k & \xrightarrow{\ F_k\ } & \mathbf {x}_{1,1}^k & \xrightarrow{\ F_k\ } & \mathbf {x}_{1,2}^k & \xrightarrow{\ F_k\ } & \cdots & \xrightarrow{\ F_k\ } & \mathbf {x}_{1,t-1}^k & \xrightarrow{\ F_k\ } & \mathbf {x}_{1,t}^k = \mathbf {ep}_0^k,\\[2pt] \mathbf {sp}_2^k = \mathbf {x}_{2,0}^k & \xrightarrow{\ F_k\ } & \mathbf {x}_{2,1}^k & \xrightarrow{\ F_k\ } & \mathbf {x}_{2,2}^k & \xrightarrow{\ F_k\ } & \cdots & \xrightarrow{\ F_k\ } & \mathbf {x}_{2,t-1}^k & \xrightarrow{\ F_k\ } & \mathbf {x}_{2,t}^k = \mathbf {ep}_1^k,\\ \vdots &&&&&&&&&& \vdots\\ \mathbf {sp}_m^k = \mathbf {x}_{m,0}^k & \xrightarrow{\ F_k\ } & \mathbf {x}_{m,1}^k & \xrightarrow{\ F_k\ } & \mathbf {x}_{m,2}^k & \xrightarrow{\ F_k\ } & \cdots & \xrightarrow{\ F_k\ } & \mathbf {x}_{m,t-1}^k & \xrightarrow{\ F_k\ } & \mathbf {x}_{m,t}^k = \mathbf {ep}_m^k . \end{array} $$

It consists of m rows and t+1 columns. We number the columns so that the starting point column is the 0th column and the ending point column is the tth column. Each row of a Hellman matrix is a precomputation chain. Any chain of points from  that has been formed by iteratively applying an F _k of the same color k is a Hellman chain.

2.3.3 Online Phase

Once the inversion target y=F(x) is given, the process explained below is repeated for each 1≤k≤ℓ, until the correct answer x is found. Occasionally, the algorithm will report failure in returning the answer after processing all ℓ indices k.

We first compute $\mathbf {y}_{1}^{k} = R_{k}(\mathbf {y}) = F_{k}(\mathbf {x})$ and check if this appears as one of the ending points in the kth Hellman table. The table lookup is repeatedly done for each recursively computed $\mathbf {y}_{j}^{k} = F_{k}(\mathbf {y}_{j-1}^{k})$, until $\mathbf {y}_{t}^{k} = F_{k}^{t}(\mathbf {x})$ has been searched for in the table. The Hellman chain

$$ (\mathbf {x}\xrightarrow{\ F_k\ }\!\! )\ \mathbf {y}_1^k \xrightarrow{\ F_k\ } \mathbf {y}_2^k \xrightarrow{\ F_k\ } \mathbf {y}_3^k \xrightarrow{\ F_k\ } \cdots \xrightarrow{\ F_k\ } \mathbf {y}_j^k $$

that is computed through this process is referred to as the online chain for the kth Hellman table.

Whenever a match $\mathbf {y}_{j}^{k} = \mathbf {ep}_{i}^{k}$ is found, the corresponding starting point $\mathbf {sp}_{i}^{k}$ is retrieved from the kth Hellman table, and the associated precomputation chain is (partially) regenerated to obtain $\mathbf {x}_{\mathrm{tmp}} = \mathbf {x}_{i,t-j}^{k} = F_{k}^{t-j}(\mathbf {sp}_{i}^{k})$. Since

$$ F_k^j(\mathbf {x}_{\mathrm{tmp}}) = F_k^j \bigl(F_k^{t-j}\bigl(\mathbf {sp}_i^k \bigr)\bigr) = \mathbf {ep}_i^k = \mathbf {y}_j^k = F_k^{j-1}(\mathbf {y}_1) = F_k^j( \mathbf {x}), $$

there is a chance that x _tmp=x. This is why the jth iteration of the online phase for a specific table is sometimes referred to as searching for the answer x in the (t−j)th column of the Hellman matrix. If multiple ending points match the current end of the online chain, one must not forget to regenerate all the corresponding precomputation chains.

Even though the existence of x in the (t−j)th column of a Hellman matrix will surely imply the collision of $\mathbf {y}_{j}^{k}$ with an ending point, the converse is not true unless F _k is injective. An ending point collision could be caused by a merge between the online chain and a precomputation chain. Hence, the online phase algorithm must check whether the candidate answer x _tmp is the correct answer x. The candidate is clearly incorrect if F(x _tmp)≠y, but a full verification requires more information than is contained in y; this is explained in more detail in Sect. 3. If the candidate x _tmp is found to be incorrect, the event is referred to as a false alarm, in which case the online phase resumes the iterative computations of the online chain.

2.3.4 Success Probability

The algorithm description for the Hellman tradeoff is complete, and we now give some rough analyses.

The success of inversion is intimately related to how many distinct points are covered by the Hellman matrices. Assume that there are not too many duplicates in an m×t Hellman matrix and consider the addition of one more precomputation chain to this matrix. The existing Hellman matrix and the new chain contain approximately mt and t points, respectively. Since the matrix stopping rule gives mt⋅t≈N, we know from the birthday paradox that there is a high chance that the new chain and the existing Hellman matrix will contain a common element. Hence, the new chain is likely to merge into an existing precomputation chain, and much of the computation that was done to create this additional chain goes to waste. Hence, it makes little sense to continue enlarging a Hellman matrix beyond the m×t bound set through the matrix stopping rule. This is the reason for using multiple small tables, rather than a very large table. The discussion given so far also indicates that there will not be too many duplicates within the matrix until one comes close to the m×t bound.

Let us use |HM| to denote the expected number of distinct nodes contained in a Hellman matrix. The probability of successful inversion after the processing of a single Hellman table is $\frac{|\texttt {HM}|}{{\textup {\textsf {N}}}}$. Hellman [14] provided the lower bound

$$ \frac{|\texttt {HM}|}{{\textup {\textsf {N}}}} \geq \frac{1}{{\textup {\textsf {N}}}} \sum _{i=1}^m \sum_{j=1}^t \biggl(1-\frac {it}{{\textup {\textsf {N}}}} \biggr)^j $$

(3)

and used it to explain the appropriateness of the matrix stopping rule. The arguments given above involving the birthday paradox are from [5, 6], and are not found in [14].

When all ℓ≈t tables are processed, assuming that the reduction functions provide independence between tables, the probability of success becomes

$$ 1- \biggl(1-\frac{|\texttt {HM}|}{{\textup {\textsf {N}}}} \biggr)^\ell \approx 1-\exp \biggl(-\frac{\ell |\texttt {HM}|}{{\textup {\textsf {N}}}} \biggr). $$

(4)

Since the number of duplicates within each Hellman matrix is kept low by the matrix stopping rule, we have |HM|≈mt. Recalling that ℓ≈t and applying the matrix stopping rule, we can state that the probability of the Hellman tradeoff in successfully recovering the correct answer x is approximately $1-\frac{1}{e} \approx 63.2~\%$. This is sufficiently large for the Hellman algorithm to be meaningful as an attack.

Interestingly, the original paper [14] does not explicitly express the success probability (4) of the complete algorithm. It is only stated that the inverse of the right-hand side of (3) should be taken as the approximate number of precomputation tables to be created. However, statements similar to (4) may be found in works as far back as [17, 18].

In [18], the right-hand side of (3) was carefully approximated, so that the bound could be rewritten as

$$ \frac{|\texttt {HM}|}{{\textup {\textsf {N}}}} \geq \frac{mt}{{\textup {\textsf {N}}}} \frac{1}{\textup {\texttt {H}}_{\mathrm {msc}}} \int_0^{ \textup {\texttt {H}}_{\mathrm {msc}}} \frac{1-e^{-x}}{x} \,dx. $$

(5)

The experimental data provided in the work supported the correctness of this bound but also showed that this bound was far from being tight. For example, at H _msc=1, the test data provided was $\frac {|\texttt{HM}|}{{\textup {\textsf {N}}}} = 0.85 \frac{mt}{{\textup {\textsf {N}}}}$, while the right-hand side of (5) was $0.80 \frac{mt}{{\textup {\textsf {N}}}}$.

This discrepancy was resolved in [9, 19], which computed the expected value |HM| itself, rather than its lower bound. This result is copied as Proposition 21 in the main body of the current paper.

The success probability of the Hellman tradeoff was also studied in [27]. However, the inversion problem considered there is different from that considered here. Their analysis is applicable if one wishes to recover any pre-image corresponding to a random image. The inversion problem considered in [27] is neither of the two inversion problems that are discussed here in Sect. 3.4, in that the inversion target is directly chosen without the involvement of an input.

2.3.5 Cost of Resolving Alarms

An upper bound for the number of false alarms per table was given as $\frac{ \textup {\texttt {H}}_{\mathrm {msc}}}{2}$ in [14]. This was combined with the fact that resolving each alarm requires at most t iterations to argue that the side effects of false alarms on the online time complexity were limited.

A much better bound on the effects of false alarms is given in [18] as

$$ (\text{cost of resolving alarms for all tables}) \leq \frac{ \textup {\texttt {H}}_{\mathrm {msc}}}{6} \ell t. $$

(6)

Almost the same content reappears in [15], expressed in the form

$$ \text{(expected cost of resolving alarms per table)} = \frac{ \textup {\texttt {H}}_{\mathrm {msc}}}{6}t. $$

(7)

The proofs given by the two papers for the above two statements are essentially identical.

2.3.6 Tradeoff Curve

We have ℓ≈t tables, each containing m entries, so that the total storage size is M=mℓ≈mt. Disregarding the time taken to treat false alarms, it takes t iterations of the one-way function to process each of the ℓ≈t tables, so the online time complexity is at most T≈tℓ≈t ². Applying the matrix stopping rule to T and M, one can arrive at the tradeoff curve

$$ TM^2 \approx {\textup {\textsf {N}}}^2 $$

(8)

for the Hellman tradeoff.

Conversely, suppose that certain values T and M satisfy the tradeoff curve (8). Then the parameters $t = \sqrt{T}$ and $m = M/\sqrt{T}$ satisfy the matrix stopping rule. When the Hellman tradeoff is implemented with these t, m, and ℓ≈t, it will require storage M and run in online time T.

The tradeoff curve (8) did not appear in the original publication [14]. The above presentation has been adopted from [5, 6].

2.4 DP Tradeoff

The distinguished point method, which we shall refer to simply as the DP tradeoff, is a simple modification of the Hellman tradeoff. The introduction of the DP technique is attributed to Rivest in the book [10], but no corresponding publication can be found. The perfect table version of the DP tradeoff was first studied in [7, 8], followed by some further analyses in [1, 26, 28], but literature analyzing the non-perfect DP tradeoffs, which we deal with in this work, is hard to find.

2.4.1 Parameter Setup

As in the Hellman tradeoff, one fixes positive integers m and t satisfying the matrix stopping rule mt ²≈N. Reduction functions are chosen and colored iterating functions F _k =R _k ∘F are defined as before. Our work will use the notation mt ²=D _msc N with a matrix stopping constant D _msc=Θ(1). As in the Hellman tradeoff, ℓ=Θ(t) will be the number of tables. The parameters are always assumed to be reasonable in the sense that 1≪m,t≪N.

One fixes a property which is satisfied by a random element of  with probability $\frac {1}{t}$. This distinguishing property should be very easy to check. For example, suppose that t and N are powers of 2 and that the set consists of non-negative integers less than N. Then, one usually defines an element of  to be a distinguished point, or a DP, if the first logt bits of its binary representation are zero.

2.4.2 Precomputation Phase

Rather than fixing the length of each precomputation chain to t, the precomputation iterations $\mathbf {x}_{i,j}^{k} = F_{k}(\mathbf {x}_{i,j-1}^{k})$ are continued until the current chain end $\mathbf {x}_{i,j}^{k}$ is found to be a DP. The resulting m precomputation chains will be of varying lengths, but their average length will be t. As in the Hellman tradeoff, the m starting point and ending point pairs are stored as a DP table and ℓ tables are constructed, each corresponding to a different color 1≤k≤ℓ.

Any chain computed through iterative applications of a single F _k that ends at a DP is a DP chain. The collection of all precomputed DP chains associated with one DP table is referred to as a DP matrix, even though the collection can no longer be visualized as a rectangular shaped matrix.

2.4.3 Online Phase

Given the inversion target y=F(x), the online phase of the DP tradeoff proceeds quite similarly to the Hellman tradeoff online phase. However, since only DPs can be found among the ending points, table lookups are done only when the iteratively computed $\mathbf {y}_{j}^{k}$ is found to be a DP. Since no precomputation chain contains a DP in the middle part of the chain, the online chain iterations for any single DP table is terminated at its first DP occurrence.

Resolving alarms is slightly tricky with the DP tradeoffs. Because the length of each precomputation chain is not known, one regenerates the precomputation chain until either $\mathbf {y}_{1}^{k}$ is reached or a DP, which sits at the end of the precomputation chain, is reached. One can store the length of each precomputation chain in the DP table [7, 8] to remove this problem, but this has the side effect of increasing the precomputation table size, and is not considered here. If multiple ending points match the current end of the online chain, all corresponding precomputation chains need to be regenerated.

2.4.4 Preliminary Analysis

The success probability (4) is also valid for the DP tradeoff, when |HM| is replaced with the number of distinct entries in a DP matrix. Since the average length of the precomputed DP chains is t, each DP matrix covers approximately mt points and the previous rough approximation $1-\frac{1}{e}$ for the success rate remains valid for the DP tradeoffs. The online chain is likely to reach a DP in approximately t iterations, so that the number of online iterations is T≈ℓt≈t ², when the efforts made to resolve alarms are ignored. Combining this with the precomputation table size, which is M=ℓm≈mt, we find that the tradeoff curve (8) is also valid for the DP tradeoff.

2.4.5 Chain Length Bound

In practice, a chain may fall into a loop that does not contain a DP and thus may never reach a DP. Hence, any implementation of the DP tradeoff sets a chain length bound [7, 8], which we denote by ${\hat {t}}$, and any chain that fails to reach a DP within this bound, during either the precomputation phase or the online phase, is discarded. The precomputation phase of a DP tradeoff must generate additional chains to fill in the discarded chains.

Even though some of our results are stated in a way that displays its dependence on ${\hat {t}}$, we are mainly interested in the case where ${\hat {t}}$ is sufficiently larger than t. The number of discarded chains is minimized by such a choice, and most of the precomputation is put to good use. Since the precomputation cost is the main barrier to any large-scale implementation of the tradeoff technique, such a choice is natural in practice.

If a chain is generated with the random function, the probability for it to become a DP chain within the chain length bound ${\hat {t}}$ is

$$ 1 - \biggl(1-\frac {1}{t}\biggr)^{\hat {t}}\approx1 - e^{-{\hat {t}}/t}. $$

(9)

This easy statement may be found in [7].

2.5 Rainbow Tradeoff

The rainbow table method was introduced by Oechslin [23]. From this point on, we will refer to the rainbow table method simply as the rainbow tradeoff.

2.5.1 Parameter Setup

One starts with positive integers m and t satisfying the matrix stopping rule mt≈N. Notice that this equation is different from the matrix stopping rules for the previous two algorithms. In this work, we use the notation mt=R _msc N with the matrix stopping constant R _msc=Θ(1). Unlike the previous two algorithms, a small number of tables ℓ=Θ(1) is used with the rainbow tradeoff. The parameters are always assumed to be reasonable in the sense that 1≪m,t≪N. Reduction functions are fixed as before, but these have double indices that are made to run over j=1,…,t and k=1,…,ℓ. The doubly colored iterating functions are defined through $F_{j,k} = R_{j}^{k}\circ F$.

2.5.2 Precomputation Phase

Instead of using a single reduction function for each table, t different reduction functions are sequentially applied to create a precomputation chain of length t. Each precomputation table stores the information from m chains. More explicitly, the ith precomputation chain for the kth rainbow table takes the form

$$ \mathbf {sp}_i^k = \mathbf {x}_{i,0}^k \xrightarrow{\ F_{1,k}\ } \mathbf {x}_{i,1}^k \xrightarrow{\ F_{2,k}\ } \mathbf {x}_{i,2}^k \xrightarrow{\ F_{3,k}\ } \cdots \xrightarrow{\ F_{t-1,k}\ } \mathbf {x}_{1,t-1}^k \xrightarrow{\ F_{t,k}\ } \mathbf {x}_{i,t}^k = \mathbf {ep}_i^k, $$

where 1≤i≤m and 1≤k≤ℓ. Each of these is a rainbow chain.

The complete set of m chains for any fixed k is an m×t rainbow matrix, and the set of pairs $\{(\mathbf {sp}_{i}^{k},\mathbf {ep}_{i}^{k})\}_{i}$ is stored as the kth rainbow table after being sorted on the ending points. Columns of a rainbow matrix are numbered from the 0th, containing the starting points, to the tth, containing the ending points.

2.5.3 Online Phase

Let the inversion target y=F(x) be given for the online phase. For each j=1,…,t and k=1,…,ℓ, we compute the jth online chain for the kth table

$$ (\mathbf {x}\xrightarrow{\ F_{t-j+1,k}\ }\!\! )\ \mathbf {y}_{t-j+1}^{k,j} \xrightarrow{\ F_{t-j+2,k}\ } \mathbf {y}_{t-j+2}^{k,j} \xrightarrow{\ F_{t-j+3,k}\ } \cdots \xrightarrow{\ F_{t-1,k}\ } \mathbf {y}_{t-1}^{k,j} \xrightarrow{\ F_{t,k}\ } \mathbf {y}_{t}^{k,j}, $$

through iterative computation, starting from the point

$$\mathbf {y}_{t-j+1}^{k,j} = R_{t-j+1}^k(\mathbf {y}) = F_{t-j+1,k}(\mathbf {x}). $$

After each chain computation, the chain end $\mathbf {y}_{t}^{k,j}$ is searched for among the ending points of the kth rainbow table. The absence of a collision indicates that the correct answer x does not belong to the (t−j)th column of the rainbow matrix. The appropriate precomputation chain is regenerated whenever a collision is found. Many of these regenerations will lead to the announcement of a false alarm.

The order of incrementing the double indices during the online phase requires clarification. One should take the chain length j-index to be the outer loop and the table number k-index to be the inner loop. In other words, for any index j, one computes the jth online chains for all ℓ tables, before computing any of the (j+1)th online chains. This is referred to as the parallel processing of rainbow tables. The opposite nesting of the loops is called the sequential processing of tables. As was already noted in [23], the parallel approach is more efficient in terms of the expected number of one-way function invocations. Parallel processing of tables is more commonly considered, and this is the approach we assume throughout this work.

2.5.4 Success Probability

In [23], one can find the success probability of a rainbow tradeoff that uses a single table written as

$$ 1 - \prod_{j=0}^{t-1} \biggl(1-\frac{m_j}{{\textup {\textsf {N}}}} \biggr), $$

(10)

where m ₀=m and m _j are recursively computed through (2). However, this was not simplified into a closed form formula there.

While studying the perfect table version of the rainbow tradeoff, the work [1] restricts to the m=N case and gives the approximation

$$ \prod_{j=t-i}^{t-1} \biggl(1-\frac{m_j}{{\textup {\textsf {N}}}} \biggr) \approx\frac{t-i}{t} \frac{t-i+1}{t+1}. $$

(11)

Notice that the range of indices in the left-hand side product is shorter than that appearing in (10). The left-hand side product of i terms expresses the probability for the first i online chain computations for a single table (non-perfect) rainbow tradeoff to fail in returning the correct answer x. This expression is valid for any m, even though the right-hand side approximation is appropriate only for m=N.

After almost repeating the computations done by [1], the work [15] obtains a generalization of (11) that is valid for any m. The result is restated as Lemma 28 in the main body of this paper. Neither (11) nor Lemma 28 were explicitly stated as separate results in the referenced papers, but they can be inferred from parts of their proofs.

2.5.5 Preliminary Analysis

A collision of points from two rainbow chains will result in merging chains only if the collision occurred at a matching color index. When a new rainbow chain is added to an existing m×t rainbow matrix that contains no collisions within each column, the probability of not experiencing a merge can be expressed as $(1-\frac{m}{{\textup {\textsf {N}}}})^{t} \approx e^{-\frac{mt}{{\textup {\textsf {N}}}}}$. Hence, the matrix stopping rule mt≈N is the correct boundary at which collisions among precomputation chains start to become problematic.

Let us assume the use of a single table for the rest of this rough analysis. Ignoring collisions within each rainbow matrix column, the success probability (10) may roughly be approximated as $1-(1-\frac{m}{{\textup {\textsf {N}}}})^{t} \approx1 - e^{-mt/{\textup {\textsf {N}}}} \approx1-\frac{1}{e}$. This is equal to what we saw during the rough analyses for both the Hellman and DP tradeoffs.

Notice that the computations for the jth online chain cannot reuse any of the information computed for previous online chains. Hence, the number of one-way function iterations required for the computation of all online chains is $T = 0 + 1 + \cdots+ (t-1) \approx\frac {t^{2}}{2}$. The storage size for the single rainbow table is M=m. Recalling the matrix stopping rule mt≈N, the tradeoff curve can be written as

$$ TM^2 \approx\frac{1}{2} {\textup {\textsf {N}}}^2. $$

(12)

The above time complexity analysis appears in [23], from which the tradeoff curve directly follows.

2.5.6 Further Analysis

The preliminary analysis given above corresponds to the worst case where the complete table is processed. In practice, the online phase is likely to terminate before computing the tth online chain. On the other hand, the cost of resolving alarms has been ignored. Hence, the rough analysis does not give the true worst-case complexity.

The work [15] provides an accurate analysis of the time complexity for rainbow tradeoffs. The expected number of one-way function iterations required to process a single rainbow table was expressed as an explicit rational function of R _msc times t ². A similar result for the additional number of one-way function iterations required to process alarms was also stated. However, the results were restricted to the single table case. We do not state their results here, but their results are reobtained if we substitute ℓ=1 into (22), appearing in the main body of this paper.

2.6 Perfect Table Tradeoffs

The main objective of introducing the DP technique was to reduce the number of table lookups that occur in the Hellman tradeoff. However, it was soon noticed that DPs allow easy detection of merging chains. During the precomputation phase of a perfect table version of the DP tradeoff [7, 8], one removes chain collisions by keeping only the longest of the merging chains. Chains are additionally generated until m non-merging DP chains have been collected. The resulting perfect DP matrix contains no overlapping points. The online phase of the perfect DP tradeoff is identical to the non-perfect version. The work [7] gives credit to the unpublished work [25] for independently introducing the same algorithm.

Detection of merging chains is also easily done with the rainbow tradeoff. The perfect table version of the rainbow tradeoff [23] stores information for just one chain from each set of merging chains. Unlike the DP case, a perfect rainbow matrix may contain overlapping points if they belong to different columns.

The perfect table version of the Hellman tradeoff refers to the case where the Hellman matrix contains no overlapping points. Some discussions may be found in [1, 27]. However, generating a perfect Hellman table is costly, and its use is not considered to be practical.

Since there are fewer or no overlaps in a perfect table, these provide better coverage of the search space than their corresponding non-perfect versions for the same amount of storage. Hence, perfect tradeoffs are likely to be more efficient than non-perfect tradeoffs. However, this gain in tradeoff efficiency is paid for with the precomputation that was wasted in generating the discarded chains.

The extra precomputation required for the use of perfect tradeoffs may not seem to be of importance. However, the precomputation cost can be critical when implementing tradeoffs at the limit of one’s resources. Consider a large-scale implementation for which the precomputation may take several months on a large cluster of computers. In such a situation, extending the precomputation period by another few months or doubling the number of computers allocated to the precomputation task will not be a viable option, even if it promised a significant advantage in the online tradeoff efficiency.

Even though there are analyses of perfect tradeoffs [1, 7, 8, 15, 23, 28], dealing with them at the accuracy level aimed for by the current paper is considerably more complicated than for the non-perfect tradeoffs. This is especially true with the perfect DP tradeoffs. In view of relative practicality and theoretic accessibility, we deal only with the non-perfect versions of tradeoff algorithms in this work. Inclusion of the perfect tradeoffs into the comparison results obtained in this paper is left as a subject for future study.

2.7 Storage Optimization

The storage size M appearing in the tradeoff curves (8) and (12) refers to the total number of starting point and ending point pairs that need to be stored in the tradeoff tables. In practice, it is important to know the physical size, or the number of bits, required for the table. Each starting point and ending point pair can surely be stored in 2logN bits, but there are techniques that allow more efficient use of storage.

Below, we assume that a suitable method of enumerating the elements of  has been fixed and treat elements of  as logN-bit integers. This enumeration is trivial when  is the set of all bit strings of certain length, but may require a small amount of work when  is given as the set of passwords satisfying certain complicated linguistic structures.

2.7.1 Consecutive Starting Points

The first storage reduction technique we review is the use of starting points that require less storage. The work [6] does this while implementing an attack on a specific system, and [7] mentions this as a well-known trick without giving any reference. A clear understanding of random functions shows that the starting points may be chosen in any manner, as long as it has no relation to the graph structure of the specific one-way function under attack.

A practical method of choosing starting points is to use consecutive integers [1]. The integers 0 through m−1 will work for any (non-perfect) table. Inter-table collisions among the starting points can be removed by concatenating the table index to the consecutive integers [4]. Note that the table index need only be recorded once for each table. However, the effect of joining table numbers is almost nonexistent on even the second columns of the precomputation matrices, so this detail is not very important. In any case, the starting points can be stored in logm bits, rather than logN bits.

The experiment provided by Hellman [14], supporting the arguments concerning the success probability, was executed with starting points set to small numbers, rather than random points. However, it is not clear if this was intended to reduce the storage size.

2.7.2 Taking Advantage of the DP Definition

In the case of DP tradeoffs, any information that can be recovered from the definition of a DP may be removed from the ending point before storage. For example, if a prefix consisting of logt zero bits defines a DP, the logt bits of zeros can be removed from each ending point without any loss of information. This method was actively used in [6] and clearly stated in [28], but seems trivial enough to have been widely known before these works.

2.7.3 Index Table

The work [6] introduces the index table method. This is a degenerate form of a widely known technique called hash tables, which is explained in Appendix D.

To facilitate fast table lookups, the precomputation tables are usually sorted on the ending points before being written to storage. Let us focus on the {(logm)−ε} most significant bits of each ending point in the sorted table, where ε is any small positive integer. Assuming that the ending points are randomly distributed, for each integer $0 \leq i < \frac{m}{2^{\varepsilon}}$, we can expect to find approximately 2^ε consecutive entries in the sorted table that have the {(logm)−ε} bit prefix of the ending point equal to integer i. Hence, one can remove {(logm)−ε} bits from each ending point and replace it with an index table that points to the starting positions for each i value without losing any information. The number of entries contained in the index table is only $\frac{m}{2^{\varepsilon}}$; hence the additional storage required by its introduction can be ignored. An example is illustrated by Fig. 1.

Fig. 1.

Index table technique. (The sorted list on the left-hand side is transformed to the right-hand side list, which contains two less bits per entry.)

In practice, the index table could store the number of entries corresponding to each index value rather than the full physical addresses. With such an approach, since only very small number of bits is required to store each count, even the use of ε=0 could be considered.

2.7.4 Ending Point Truncation

The methods described so far reduce the storage size without losing any information concerning each starting point and ending point pair. However, this is not so with the final storage reduction method we describe, which is to simply truncate a part of the ending point before storage.

The truncation of ending points was done in [6] for a specific tradeoff implementation, where it was simply stated that the number of bits they allocate is sufficient for identification purposes. In [4],¹ under the assumption that m≈N ^1/3, it is claimed that the ending points of a DP table can be compressed to slightly more than $\frac{1}{3}\log {\textup {\textsf {N}}}$ bits. It is also claimed that the ending points for the rainbow tradeoff can be compressed to slightly more than $\frac{2}{3}\log {\textup {\textsf {N}}}$ bits. The paper does not provide any justification for these claims.

During the online phase, when a table lookup is required, the object to be searched for in the table is truncated to the same length and compared with the truncated ending points of the table. The table lookups may now falsely return a match even when a merge between the online chain and a precomputation chain did not occur. Still, since we were already expecting false alarms, no new measure needs to be devised to deal with the new type of false alarms. Aggressive ending point truncation will cause more frequent false alarms; hence the degree of truncation should be carefully controlled.

The word truncation may give the impression that such a method is applicable only when the space  consists of bit strings. On spaces that look different, any surjective map that is pre-image uniform, in the sense that the number of pre-images for each element in the range is identical, can serve as the truncation operation. In practice, password hashes are usually bit strings and one does not apply the reduction function at the end of a chain, so truncations can easily be done.

2.8 Parameter Optimization

Choosing the parameters m, t, and ℓ for a concrete tradeoff implementation is not an easy task.

The work [18] starts with the assumption that the cost, in dollars, of a tradeoff attack implementation is proportional to the storage size and the number of one-way function computations the online phase machine can perform per unit time. This allows one to consider the lowest possible monetary cost of an attack machine that must succeed with a given probability and finish within a preset real-world time. Expressions giving lower and upper bounds for the optimal cost are presented, and parameters t, m, and ℓ that can achieve the optimal cost are also found. The optimal parameters that are stated depend on the relative cost of storage versus one-way function computations at unit speed.

This analysis is one of the few that takes false alarms into account when computing the time complexity of the online phase. However, the analysis relied on the bounds (5) and (6), which are not very tight, and the upper bound for the optimal cost was simply taken to be an approximation for the optimal cost. Also, while defining the optimal cost, the amount of precomputation was fixed to what is required for a single exhaustive search.

The measure of efficiency used in the current work is different from the monetary cost discussed in [18]. Our interest is in how efficient each tradeoff algorithm is in balancing storage against online time. This balancing ability changes with the amount of precomputation that is invested and the required success rate. The optimal monetary cost for implementation can easily be computed whenever this balancing ability is accurately fixed.

In [17], an attempt was made to optimize the success probability of Hellman tradeoff, while keeping both the time and storage complexities constant. The gain in success probability was paid for with larger precomputation.

There are two parts of their argument that introduce inaccuracy into their results. Since they did not have access to a good expression for the time complexity, it was not possible for them to keep the time complexity exactly constant. They had to be satisfied with keeping ℓt, which is an upper bound for the time complexity in the absence of false alarms, constant. The second point was that they lacked knowledge of the exact success probability and had resorted to using its lower bound given by (5).

The general conclusions of [17] may still be correct, but the details, in particular, the explicit optimal parameters and values, will need to be recomputed with the information given in the current paper. A little more light was shed on the attempt by [23], but the discussion there still relied on rough estimates of time complexity and success probability.

2.9 Comparison of Tradeoff Algorithms

Let us attempt a comparison of the three tradeoff algorithms we have explained, based on their tradeoff curves that are already available. Both the Hellman and DP tradeoff curves are given by (8) and the rainbow tradeoff curve is given by (12). Considering the case where the same storage M is given to the three tradeoff algorithms, the tradeoff curves imply that the rainbow tradeoff will require only half the number of one-way function invocations compared to the other two algorithms during the online phase. In addition to giving an argument that is equivalent to what we have just described, the work [23] argues heuristically that the rainbow tradeoff is at an advantage over the DP tradeoff concerning false alarm issues.

The claimed efficiency of the rainbow tradeoff over the DP tradeoff is refuted in [3, 4]² with the observation that the number of physical bits required to store each entry of the tradeoff table has been ignored in [23].

Assume the use of typical parameters m=t=ℓ=N ^1/3 for the DP tradeoff. Recalling the contents of Sect. 2.7, one finds that the starting points for the DP tradeoff can be stored in $\frac{1}{3}\log {\textup {\textsf {N}}}$ bits. It is claimed in [4] that the ending points can first be compressed to slightly more than $\frac {1}{3}\log {\textup {\textsf {N}}}$ bits and then further compressed to a very small number of bits by applying the index table method. Hence each entry of a DP table requires slightly more than $\frac{1}{3}\log {\textup {\textsf {N}}}$ bits to record. In the case of rainbow tradeoffs, one assumes the typical parameters m=N ^2/3, t=N ^1/3, and ℓ=1. Then each starting point requires $\frac{2}{3}\log {\textup {\textsf {N}}}$ bits. The ending point is first compressed to $\frac{2}{3}\log {\textup {\textsf {N}}}$ bits, and then most of this is removed through the index table method.

Accepting the above arguments, we see that each entry of a rainbow table requires twice the number of bits required by an entry of a DP table. When given the same physical amount of storage, the DP tradeoff can store twice as many starting point and ending point pairs. This translates to a gain in online time by a factor of four through the tradeoff curve. In conclusion, the DP tradeoff will run two times faster than the rainbow tradeoff for the same physical amount of storage.

The more recent work [1] once again advocates the rainbow tradeoff and tries to explain that the arguments of [4] that we have explained so far are misleading. They emphasize that the advantage of the rainbow tradeoff claimed in [23] was by a factor of at least two, rather than just two. This is a reasonable point to make, but their ensuing arguments seem to indicate that they were not aware of the ending point truncation method, which was taken into account in [4]. One could interpret this as showing how uninformative [4] was in treating the ending point truncation method.

As we will verify in this work, the claims of [3, 4] were mostly correct, but there are hidden issues that can overturn their conclusion. The first is that the tradeoff curves given by (8) and (12) are not accurate. Both of these correspond to the worst case where the algorithms are executed to the end without the correct answer being found. In fact, this was the point made by [1], although it was used to support only the rainbow tradeoff. One must also note that the effects of false alarms have been ignored by both tradeoff curves so that neither accurately reflects even the worst-case complexity.

The second issue is that the success probabilities of the two algorithms may not be precisely equal at the typical parameters. We have already noted that both algorithms have an approximate success probability of $1-\frac{1}{e}$ at the typical parameters, but this is an extremely rough estimate, and the running time of a tradeoff algorithm is very sensitive to the required success rate. The controversy explained here is discussed in more detail in Sect. 8.4, after we have developed the necessary tools.

The comparison claims of [23] and [3, 4] were made using parameters that require precomputation equal to a single exhaustive search. Recent comparison claims that deal with the perfect tables, which we do not treat in this paper, have the tendency to completely ignore the precomputation cost. Neither approach reflects what can be done in practice. The difficulty of including the precomputation cost into the comparison of tradeoff algorithms seems to have been one reason why perfect tradeoffs have received more focus recently. They certainly appear more attractive, when precomputation is ignored.

2.10 Checkpoint

The checkpoint [1] technique allows for the resolving of alarms without the regeneration of the precomputation chain. This technique is applicable to both Hellman and rainbow tradeoffs. Application to the DP tradeoff is also possible but slightly more complicated due to the variations in chain lengths.

A column of the precomputation matrix is designated as the checkpoint before precomputation. After generation of each precomputation chain, the least significant bit of the chain element that sits at the checkpoint column is appended to the starting point and ending point pair that is to be recorded in the precomputation table. During the online phase, we proceed as usual until an alarm is encountered. At each collision, the online chain is aligned with the colliding precomputation chain at the ending points. If the online chain is long enough, the least significant bits of the two points that belong to the checkpoint column are compared. If the two checkpoint bits do not match, the ending point collision must have resulted from a merge of chains, and the collision is declared a false alarm. If the checkpoint bits do match, the precomputation chain is regenerated as usual to resolve the alarm.

The use of checkpoints filters out some of the efforts spent on precomputation chain regeneration. One can generalize what has been explained to multiple checkpoint columns, consider other methods of extracting a checkpoint bit, or collect more than one bit of information from each checkpoint column.

An analysis of the effects of checkpoints in reducing online time was given by [1] for the perfect rainbow tables. Analyses for Hellman tradeoffs and single table (non-perfect) rainbow tradeoffs were performed in [15]. With a single checkpoint at the optimal position, the Hellman tradeoff online time decreases by 3.17 % at H _msc=1, and the online time of a single table non-perfect rainbow tradeoff decreases by 5.91 % at R _msc=1. The effects of checkpoints are more visible at higher H _msc and R _msc values.

The advantage of checkpoints must be compared with its side effect on the storage size. After the techniques of Sect. 2.7 have been applied, even a single bit difference in table entry size could translate to a meaningful size ratio change. For example, at 50 bits per table entry, if the increase of a single bit per table entry caused by the use of checkpoints was instead allocated to enlarge the number of table entries, the online time would have reduced by $1 - (\frac{50}{51})^{2} = 3.88~\%$. This is better than the above-mentioned 3.17 % reduction effect of checkpoints on the Hellman tradeoff, and the 5.91 % reduction effect on rainbow tradeoff should be interpreted as achieving only approximately 2.0 % extra reduction.

Since the effects of checkpoints are small and selective applications of checkpoints will affect all algorithms in the positive direction, its effect on the final comparison of algorithms will be minimal. On the other hand, consideration of the checkpoint technique would add another layer of complication to our analysis. Hence, the analysis given in the current paper does not consider the use of checkpoints. However, we are not claiming that the use of checkpoints should not be considered in practice.

3 Applying Time-Memory Tradeoff to Password Hashes

One usually states the objective of a tradeoff algorithm as the inversion of a one-way function. A closer look reveals that there are two versions of the inversion problem, and we will explain how one of these corresponds to the applications of the tradeoff technique to password hash systems. Issues concerning the use of random functions in the theoretic analysis of tradeoff algorithms are also discussed in this section.

In this section, we refer to the one-way function image as the password hash and the input as the password.

3.1 Password Hash

Let us briefly explain how the security features of many file formats that rely on passwords for access control work in their very basic form.

The designer of the system chooses and fixes a one-way function H. This one-way function is a part of the file format specification and is usually considered to be public. In fact, the one-way function definition can be extracted from the related software even if it was not originally made public. When the owner of a file following this format wants access control to be applied to the file, the user supplies a password x. An encryption key is derived from the password, and the main content of the file is replaced by its encryption under this key. Then the image y=H(x) of the user password, under the one-way function specified for the file format, is added to the file. Finally, any record of the encryption key and the raw password supplied by the user is destroyed.

Later, when authentication is required for file access, the supporting software asks for a password. The one-way function image H(x′) of the newly supplied password x′ is computed by the software and is compared with the corresponding information y stored within the file. If a perfect match y=H(x′) is found, equality x=x′ is assumed, the main body of the file is decrypted using the key derived from the password x′, and access to the decrypted content is granted. Note that the one-way function image y of the correct password is stored within the file without any protection and is accessible to anyone that has obtained the file.

The user authentication procedure for computer system logins works in much the same way. At the time of initial user registration to the system, the one-way function image of the password supplied by the user is recorded in a file that is stored within the system. In this case, access to the one-way function images may be harder for the attacker to achieve than in the above case, but this information is often sent over the network in the clear to a group of computers, so that each of these computers may allow authenticated logins to a user that has registered at a central server.

3.2 Uniqueness of the Pre-image to a Password Hash

Out of theoretic curiosity, we first ask whether a password hash uniquely determines the password. This should seem obvious in any practical usages of the password hash systems.

Proposition 1

Let be a random function. Given any password , the number of inputs that H maps to the password hash H(x) is expected to be .

Proof

Since H is a random function, we can first assign a randomly chosen value of  to H(x) and then define all the other function values. The probability for any one of the later assignments to strike H(x), which is an explicitly fixed value in , is . Each later assignment is independent of all other assignments, and we can expect the number of later assignments to H(x) to be . □

Readers should not misinterpret the above proposition as giving the pre-image size of a random under a random H. For the random function H, the distribution on  produced by H(x) is the uniform distribution for each fixed , and every is expected to have -many pre-images, rather than . This is not in contradiction with the proposition, as the proposition deals with the distribution on  produced from random inputs by the specific H that has been constructed, and this is different from the uniform distribution on . Those points of  that lie outside , for the specifically constructed H, do not have any chance of appearing.

One can also ask for the pre-image size of a random password hash . Note that this question can only be asked after the random function H has been fully constructed. The corresponding answer will depend on the size of , but, when , this should be close to

Once again, this question is not related to the content of the above proposition. It deals with the uniform distribution on , which is different from the distribution on  given by the fully specified H. Those points with larger pre-image sets will have a larger probability of appearing than those with smaller pre-image sets.

Consider an application of the tradeoff technique to a block cipher whose key length is equal to its block length. In such a case, one is working with , and Proposition 1 states that there will be approximately two keys, on average, that map to a given target ciphertext. This is probably larger than what many would have naively expected. Of course, in practice, one usually assumes the use of a second ciphertext to almost uniquely identify the key. In fact, if one interprets the key to two ciphertexts mapping as a new one-way function, then Proposition 1 claims that the key is almost uniquely determined from the two ciphertexts.

Let us next discuss what Proposition 1 implies for systems that rely on passwords for access control. These systems are usually designed so that the space  of potential hash values is significantly larger than the space  of admissible passwords. A typical password hash would be a bit string of at least 128 bits in length, and the number of alphanumeric passwords consisting of ten characters is only 62¹⁰≈2^59.5. In such a case, Proposition 1 shows that a password hash H(x), produced from a password x, will almost always identify x uniquely.

Furthermore, in practice, the set of all passwords admissible by the security system is not of much importance. Since human-generated passwords are not uniformly distributed within the complete admissible password space, the tradeoff attacker first fixes a manageable subset from the set of all passwords and decides to be satisfied with recovering only those passwords that lie in . The size of this subset is determined by the computational power that the attacker can allocate to the precomputation phase and should preferably cover the passwords that are most likely to be used. In fact, it has been shown [22] that human-memorable passwords can be enumerated efficiently. Under such a setting the password hash set  is immensely larger than the set of passwords  that is being considered, and hence the password hash determines the password uniquely.

For the remainder of this paper, we assume that the target system for the application of the tradeoff technique is such that , implying that the password hash uniquely determines the password.

3.3 The Reduction Function

The tradeoff technique requires the one-way function to be iterated. Since the codomain  of the one-way function is usually larger than the domain , iteration is achieved by utilizing a reduction function . One role of the reduction function is to let a password hash be interpreted as another password. As any theoretic treatment of the tradeoff technique assumes R∘H to be a random function, let us check whether this is appropriate.

Proposition 2

Let be a divisor of , so that is an integer. Let be any fixed function that is pre-image uniform in the sense that it is exactly -to-1. If is a random function, then is a random function.

Proof

In more precise terms, we want to show that the distribution on , produced from the uniform distribution on , through the mapping H↦R∘H, is a uniform distribution.

Let be any specific function. It suffices to show that, after random construction of a function , we will find R∘H=F ₀ with probability . Note that is a partition of  into cells of size . The event F ₀=R∘H will happen if and only if the value assigned as H(x) belongs to the cell R ⁻¹(F ₀(x)), for every . Since the size of R ⁻¹(F ₀(x)) is always , and since the assignment to H(x) is independent and random for every x, the probability of arriving at F ₀=R∘H is

as claimed. □

Every application of the time-memory tradeoff technique to a security system involves a specific one-way function , and there is no strictly logical reason to believe that the specific H will display the properties expected of a random function. Hence we need to discuss if predicting the behavior of an explicit tradeoff implementation with arguments concerning random functions can be justified in practice.

There are two ways to resolve this problem. The first is to appeal to our intuition. When one ignores his knowledge of the inner working of the given specific function, it will seem as if the function is returning independently and randomly generated values to each given input. Hence, viewed from the outside, it looks as if the specific function is a random function in the construction sense. The second argument, which seems slightly more plausible, is that the one-way function used in the security system is in fact a function that has been selected from the pool of all functions. Unless we had chosen the one-way function in an unusual way, any property exhibited by a specific function will be close to the property averaged over all functions. Further discussion related to this second argument may be found in Appendix B.

We have thus partly justified the use of random functions in place of specific one-way functions when analyzing the behavior of time-memory tradeoffs. What we have shown through Proposition 2 is that if we may treat the specific one-way function H as a random function, then the same can be done with the function . Hence, throughout this paper, while analyzing the behavior of time-memory tradeoffs, we shall work with a random function whose domain and codomain coincide.

3.4 Two Versions of the Inversion Problem

Discussions of this subsection should be read with the Hellman tradeoff in mind. However, the content can easily be translated to language that is appropriate for any other tradeoff algorithm.

We have already mentioned that we shall work in the situation where the sets satisfy , so that a password hash almost always determines a unique password. We also know that any analysis of time-memory tradeoff behavior is usually done with a random function , whose image does not uniquely determine the input. In actual implementations, reduction functions are defined and the online phase algorithm works with the colored iterating functions .

The unique password x corresponding to inversion target y=H(x) is obtained through the tradeoff algorithm as follows. The online phase algorithm is given y, and R _k (y)=H _k (x) is passed onto its sub-algorithm that processes the kth table. The best the sub-algorithm can do is return inputs  satisfying H _k (x)=H _k (x). Since this relation is weaker than x=x, the parent algorithm must verify whether the password candidate x is the correct password x by testing the relation H(x)=y.

Let us discuss how often during the online phase such candidate checks need to be performed. Assume that the precomputation algorithm required iterations of H to complete. We will have ε=Θ(1) in practice. For exactly the same reason given in the proof of Proposition 1, the expectation for the number of x appearing in the kth Hellman matrix that maps to H _k (x) under H _k , combined over all k, is upper bounded by ε+1, which is a small number. Hence, the cost of such candidate checks may safely be ignored.

During a tradeoff algorithm analysis, one does not mention anything about H or R, the source of the inversion problem, and simply assumes that the inversion target y=F(x) is given, for some function . Note that in this setting, the password hash y does not uniquely determine the password x. However, the goal of the tradeoff algorithm in this paper will be to find the correct password x that was used to create y, rather than any password x that corresponds to the given y through F(x)=y. The any version may be useful when working to find the pre-image of a cryptographic hash function, but the the version is suitable when looking for the correct password to an access control mechanism. A clear distinction between these two inversion problems was first made in [15].

Since it is logically impossible to distinguish between the many pre-images with only the  information, our analysis will focus on whether x is among the possibly multiple pre-images to y, returned by the tradeoff algorithm. The determination of whether each returned value is the correct password is assumed to be done outside the tradeoff algorithm.

The difference between looking for the pre-image versus any pre-image implies that the tradeoff algorithm will succeed under different circumstances. The the version succeeds if and only if the correct password x had appeared as an input to the one-way function F during the precomputation phase, i.e., if x is among the precomputation matrix entries excluding the ending points. On the other hand, the any version succeeds if and only if the image y=F(x) had appeared as the function output during the precomputation phase, i.e., if y is among the precomputation matrix entries excluding the starting points. The two approaches will show differences in properties such as success probability and online running time.

Let us add a final word of caution—both inversion problems we have discussed require the target y=F(x) to be fixed through a random choice of the input x. One should distinguish this from the case where the inversion target is directly chosen at random from either the image space or the codomain. These variants do not seem to fit any naturally occurring real-world situation.

4 DP Tradeoff

A complexity analysis of the DP tradeoff is given in this section. We present a formula for computing the probability of success for the non-perfect DP algorithm and provide a tradeoff curve which takes the effects of false alarms into account. We also discuss the number of bits required to efficiently store the starting point and ending point pairs.

In this work, to simplify some of our proofs, we assume that the starting points are always chosen among non-DPs. Hence, in a precomputed DP chain, every point preceding the ending point, including the starting point, is a non-DP. A rigorous treatment that allows starting points to be DPs can be performed, but differences between results from such an analysis and those presented in this work will be negligible.

Recall the probability for a random chain to become a DP chain within the chain length bound ${\hat {t}}$, given by (9). Rather than requiring each table to contain exactly m entries, we assume that each precomputation DP matrix is always generated from

$$ m_0 = \frac{m}{1-e^{-{\hat {t}}/t}} $$

(13)

distinct starting points. Then we can expect to collect approximately m chains that terminate at DPs.

All of our tradeoff algorithm analyses are done under the assumption that the one-way function is the random function. In particular, many expectations mentioned hereinafter are to be understood as averages made over the choice of all functions. Most of our arguments will be made over a single table, so we remove the display of dependence on the reduction functions from all the notation.

4.1 Probability of Success

Let us discuss the probability of success for a DP tradeoff under a given set of parameters. We first present a general formula connecting precomputation and probability of success and then show how to compute these for specific parameters. Our first lemma is quite trivial.

Lemma 3

The number of one-way function invocations required in either creating a DP chain or stopping at the ${\hat {t}}$ th iteration without having reached a DP is expected to be

$$ t \bigl(1-e^{-{\hat {t}}/t}\bigr). $$

Proof

It suffices to add the probabilities of having to compute the successive iterations. Since the next iteration is computed if and only if a DP has not yet been reached, the expected one-way function invocation count is

$$ \sum_{i=1}^{{\hat {t}}} { \biggl(1- \frac{1}{t} \biggr)}^{i-1} = t \biggl\{1 - { \biggl(1- \frac{1}{t} \biggr)}^{{\hat {t}}} \biggr\}, $$

which we can approximate to what is stated. □

In the above proof, we have implicitly assumed the one-way function to be a random function and computed the probability for the first i assignments to be non-DPs. A more exact analysis would additionally consider the possibility for the next assignment to produce a previously assigned value. We have not done so because the above result was good enough as an approximation.

Clearly, the success rate of a tradeoff algorithm is intimately connected to the amount of precomputation, so let us present a way to write down the precomputation.

Proposition 4

The precomputation phase of the DP tradeoff is expected to require mtℓ one-way function invocations.

Proof

We know from Lemma 3 that each attempt at a DP chain creation is expected to require $t(1-e^{-{\hat {t}}/t})$ one-way function invocations. Recall that the creation of a single DP table is to start with $m_{0} = \frac{m}{1-e^{-{\hat {t}}/t}}$ chains. Together, these imply that the creation of a single DP table is expected to consume mt one-way function invocations. Hence, the total precomputation requirement may be written as mtℓ. □

This proposition is trivially true when the chain length bound is not set, but what we have shown is that the precomputation cost does not depend on the chain length bound. We define the precomputation coefficient for the DP tradeoff to be $\textup {\texttt {D}}_{\mathrm {pc}}= \frac{mt\ell}{ {\textup {\textsf {N}}}}$, so that the precomputation cost of a DP tradeoff is D _pc N.

The coverage rate D _cr of a DP table is defined to be the expected number of distinct nodes that appear among the DP chains as inputs to the one-way function, divided by mt. Since our starting points are always non-DPs, all of the nodes that are counted will be non-DPs. The mentioned expectation is an average over the choice of one-way functions. In other words, the coverage rate is a certain expected value for the random function. Our next statement reduces the search for success rate to the computation of the coverage rate.

Proposition 5

The success probability of the DP tradeoff is

$$ \textup {\texttt {D}}_{\mathrm {ps}}= 1 - e^{-\textup {\texttt {D}}_{\mathrm{cr}} \textup {\texttt {D}}_{\mathrm {pc}}}. $$

Proof

If we are given y=F(x) as the inversion target, the DP tradeoff will succeed in recovering the correct answer x, if and only if x had appeared as one of the inputs to the one-way function during the creation of the DP table. As was discussed in Sect. 3.4, this is not equivalent to asking for the appearance of y among the output values. The objective of recovering the correct, rather than any inverse, corresponds to finding x among the one-way function inputs.

By definition of the coverage rate, a single DP matrix is expected to contain D _cr mt distinct nodes that were used as inputs to the one-way function. Hence the processing of a single table will fail in returning the correct answer with probability $(1 - \frac{\textup {\texttt {D}}_{\mathrm{cr}} mt}{{\textup {\textsf {N}}}} )$. The success probability of the complete DP tradeoff process is given by

$$ \textup {\texttt {D}}_{\mathrm {ps}}= 1 - \biggl(1 - \frac{\textup {\texttt {D}}_{\mathrm{cr}}mt}{{\textup {\textsf {N}}}} \biggr)^\ell \approx1 - \exp \biggl(-\textup {\texttt {D}}_{\mathrm{cr}}\frac{mt\ell}{ {\textup {\textsf {N}}}} \biggr) = 1 - e^{-\textup {\texttt {D}}_{\mathrm{cr}} \textup {\texttt {D}}_{\mathrm {pc}}}, $$

assuming that the multiple tables are independent. □

We confide that our treatment in the proof of separate tables as being independent does not strictly conform to the assumption of F being a single random function.

This lemma is almost identical to (4), which had already appeared in many works. We wrote out the proof in detail, only because most previous works did not clarify whether the inputs or outputs of the random functions were being counted. In fact, many of them did not even clarify which version of the inversion problem was being considered, as it did not matter for their intended rough analysis.

If the creator of the inversion target y=F(x) chooses x to be a DP, the online phase will definitely fail. The success probability would be very low for such challenges even if the starting points were allowed to be DPs. For our analysis to be applicable, the challenge x needs to be chosen without reference to the structure of the DP tradeoff table. Note that this is not as strong a requirement as asking for the choice of x to be random. In practice, since distinguishing properties are defined with reference to the password hashes rather than the passwords, such challenges do not cause any problem.

For the remainder of this subsection, all chains belonging to the DP matrix will be seen as having been aligned at the starting points, rather than at the ending points, and the starting point column will be referred to as the 0th column.

The above expression for probability of success can only be put to use if we know how to compute the coverage rate. Our computation of the coverage rate will be done in two steps. Of the m ₀ chains generated, only m will be DP chains, but we disregard this in the first step and count the number of new nodes added by each column of the extended matrix. The sum of these values is the total number of all distinct input entries. In the second step, we will count the number of nodes that belonged to chains not ending at DPs and subtract these from the total count.

Let us write m _j for the number of new non-DP nodes added by the jth column. The number m ₀ of distinct starting points, stated by (13), conforms to this notation.

Lemma 6

The number of new non-DP nodes added by each column satisfies the recurrence relation

$$ m_j = {\textup {\textsf {N}}}\biggl\{1-\exp \biggl(-\frac{m_{j-1}}{{\textup {\textsf {N}}}} \biggr) \biggr\} { \biggl(1-\frac{1}{t} \biggr)} \biggl(1-\frac{\sum_{i=0}^{j-1} m_i}{{\textup {\textsf {N}}}(1-1/t)} \biggr). $$

Proof

Suppose a node positioned in the (j−1)th column is old, in the sense that it has appeared in one of the 0th through (j−2)th columns. Application of the random function to this node will not result in a random element of , but a node that had appeared in one of the 1-st through (j−1)th columns. Hence when counting new nodes of the jth column we need only consider the nodes of the jth column that are assigned as images to new nodes of the (j−1)th column. Recalling (1), we write this as the ${\textup {\textsf {N}}}\{ 1-\exp (-\frac{m_{j-1}}{{\textup {\textsf {N}}}} ) \}$ part appearing in the claimed equation.

Of the distinct entries that have appeared in the jth column, that are not automatically old, we want to filter out the DPs. The previous count is made to correspond to the non-DPs by multiplying by a $(1-\frac {1}{t})$ factor.

Still, not all of these non-DPs are new nodes. Those that have appeared in previous columns are removed by multiplying $(1-\frac{\sum_{i} m_{i}}{{\textup {\textsf {N}}}(1-1/t)} )$. Notice that we have ${\textup {\textsf {N}}}(1-\frac {1}{t})$, rather than N, in the denominator, as we are dealing only with non-DPs at this point. □

The next two lemmas are technical computation results. We first turn the recursive formula for m _j into a difference equation concerning a certain sum of m _j .

Lemma 7

Let $\mu_{i} = \frac{m_{i}}{{\textup {\textsf {N}}}(1-1/t)}$ and $\sigma_{j} = \sum_{i=0}^{j-1}\mu_{i}$. Then, σ _j satisfies the recursive formula

$$ \sigma_{j+1} - \sigma_j = \frac{m_0}{{\textup {\textsf {N}}}} - \frac {1}{t}\sigma_j - \frac{1}{2}\sigma_j^2 \quad \text{\textit{with}}\ \sigma_0 = 0, $$

which is accurate up to modulo  $O (\frac{1}{t^{3}} )$.

Proof

It is straightforward to rewrite the recursive formula of Lemma 6 in terms of the notation μ _j :

$$ \mu_j = \biggl\{1 - \exp \biggl(-{ \biggl(1-\frac{1}{t} \biggr)}\mu_{j-1} \biggr) \biggr\} \Biggl(1 - \sum _{i=0}^{j-1}\mu_i \Biggr). $$

This may be rewritten once again as

$$ \exp \biggl(-{ \biggl(1-\frac{1}{t} \biggr)}\mu_{j-1} \biggr) = 1-\frac{\mu_j}{1-\sigma_j} = \frac{1 - \sigma_{j+1}}{1-\sigma_j}. $$

Now, by taking products of both sides over j=1,…,k, we obtain

$$ \exp \biggl(-{ \biggl(1-\frac{1}{t} \biggr)}\sigma_k \biggr) = \frac{1-\sigma_{k+1}}{1-\sigma_1}. $$

We have thus arrived at a relation involving only the σ _k notation.

By expanding the exponential function in its Taylor series, we obtain

$$ \sigma_{k+1} = 1 - (1-\sigma_1) \biggl\{ 1 - { \biggl(1- \frac{1}{t} \biggr)}\sigma_k + \frac{1}{2} { \biggl(1- \frac {1}{t} \biggr)}^2 \sigma_k^2 - \cdots \biggr\}, $$

and we can modify the above into the difference equation

$$ \sigma_{k+1} - \sigma_k = \sigma_1 - \biggl( \sigma_1 + \frac {1}{t}- \frac{\sigma_1}{t} \biggr) \sigma_k - \frac{1}{2} (1-\sigma_1) { \biggl(1-\frac{1}{t} \biggr)}^2 \sigma_k^2 + \cdots. $$

Noting that the left-hand side σ _k+1−σ _k =μ _k is of order $O (\frac{m}{{\textup {\textsf {N}}}} ) = O (\frac{1}{t^{2}} )$, we remove every term on the right-hand side of $O (\frac {1}{t^{3}} )$ order. This may easily be done after noting that σ ₁=μ ₀ is $O (\frac{1}{t^{2}} )$ and that σ _k is $O (\frac{m k}{{\textup {\textsf {N}}}} )$, which is at most $O (\frac {1}{t})$. The simplified equation is now

$$ \sigma_{k+1} - \sigma_k = \mu_0 - \frac {1}{t}\sigma_k - \frac{1}{2}\sigma_k^2 + O \biggl(\frac{1}{t^3} \biggr). $$

It is clear that the initial condition σ ₁=μ ₀ may be replaced by σ ₀=0, under this recursive formula. As a final tweak, we subtract $\frac{m_{0}}{{\textup {\textsf {N}}}(t-1)}$, which is of $O (\frac{1}{t^{3}} )$ order, from the constant term $\mu_{0} = \frac {m_{0}}{{\textup {\textsf {N}}}(1-1/t)} = \frac{m_{0}}{{\textup {\textsf {N}}}} (1+\frac{1}{t-1} )$, to arrive at the claimed formula. □

Now that we have a difference equation, we can obtain σ _k through an application of the Euler method.

Lemma 8

For each non-negative integer k, we have

$$ m_k \approx {\textup {\textsf {N}}}\bigl(\sigma(k+1)-\sigma(k) \bigr) $$

where

$$ \sigma(k) = \frac{\varXi^2 - 1}{t} \frac{\exp (\varXi\frac{k}{t} )-1}{ (\varXi+1)\exp (\varXi\frac{k}{t} )+(\varXi-1)} \quad\text{\textit{with}}\ \varXi= \sqrt{1 + \frac{2 \textup {\texttt {D}}_{\mathrm {msc}}}{1-e^{-{\hat {t}}/t}}}. $$

Proof

Let a function σ:R→R be the unique solution to the differential equation

$$ \frac{d}{dk}\sigma = \frac{m_0}{{\textup {\textsf {N}}}} - \frac {1}{t}\sigma- \frac{1}{2} \sigma^2 \quad\text{and}\quad \sigma(0) = 0. $$

(14)

If one defines the sequence {σ _k } _k≥0 through the corresponding difference equation

$$ \sigma_{k+1} - \sigma_k = \frac{m_0}{{\textup {\textsf {N}}}} - \frac {1}{t}\sigma_k - \frac{1}{2}\sigma_k^2 \quad \text{and}\quad \sigma_0 = 0, $$

(15)

then the Euler method tells us that σ(k), the evaluation of the function σ at the non-negative integer k, may be approximated by the sequence value σ _k . We may turn this the other way around to present approximate values of σ _k through the function evaluations σ(k).

The unique solution to differential equation (14) is

$$ \sigma(k) = \frac{2m_0 t}{{\textup {\textsf {N}}}} \frac{\exp (\sqrt{1+\frac{2m_0 t^2}{{\textup {\textsf {N}}}}}\frac{k}{t} )-1}{ (\sqrt{1+\frac{2m_0 t^2}{{\textup {\textsf {N}}}}}+1 ) \exp (\sqrt{1+\frac{2m_0 t^2}{{\textup {\textsf {N}}}}}\frac{k}{t} ) + (\sqrt{1+\frac{2m_0 t^2}{{\textup {\textsf {N}}}}}-1 )}. $$

The form of σ(k) stated by this lemma is obtained when (13) and mt ²=D _msc N are substituted.

Since the definition of σ _k given by (15) is identical to the approximate recursive relation of Lemma 7, we have

$$ \sigma(k) \approx\sigma_k = \sum_{i=0}^{k-1} \mu_i, \quad\text{where}\ \mu_i = \frac{m_i}{{\textup {\textsf {N}}}(1-1/t)}. $$

This allows us to write

$$ m_k \approx {\textup {\textsf {N}}}{ \biggl(1-\frac{1}{t} \biggr)} \bigl( \sigma(k+1)-\sigma (k) \bigr) \approx {\textup {\textsf {N}}}\bigl(\sigma(k+1)-\sigma(k) \bigr), $$

where the $\frac {1}{t}$ term removal is justifiable, as it is of strictly smaller order. □

This completes the first step of the coverage rate computation. The coverage rate corresponds to the number of distinct non-DP nodes contained in just the DP chains, but the currently computed m _k includes all points contained in even the non-DP chains. We need to account for these nodes belonging to non-DP chain nodes. This is the second step in finding the coverage rate.

Proposition 9

The coverage rate of a single DP table is expected to be

$$ \textup {\texttt {D}}_{\mathrm{cr}}= \frac{2}{e^{{\hat {t}}/t}-1} \int_{0}^{{{\hat {t}}}/{t}} \frac{\exp(\varXi u)-1}{(\varXi+1)\exp(\varXi u)+(\varXi-1)} \exp(u)\, du, $$

where $\varXi= \sqrt{1 + \frac{2 \textup {\texttt {D}}_{\mathrm {msc}}}{1-e^{-{\hat {t}}/t}}}$.

Proof

To count the number of distinct non-DPs belonging to all DP chains, we need to subtract the number of all new points belonging to non-DP chains from $\sum_{i=0}^{{\hat {t}}-1} m_{i}$. Before doing this, we first need to consider whether any of these points may not also appear within a DP chain and take the status of being a new point when the non-DP chain is removed.

It is clear that any new node belonging to a non-DP chain cannot have appeared in a column previous to its position, as the node is supposed to be new. Furthermore, such a node cannot appear within the DP chains in the same column or any future columns, since it would then reach a DP before the chain length bound is exceeded. Hence new nodes belonging to non-DP chains do not appear within any DP chains, and we may safely remove all of these new points without worrying about their possible contribution to coverage by DP chains.

Now, let us count how many points belong to non-DP chains, one column at a time. We start with the 0th column. Among all m ₀ chains, even though we do not know ahead of time which ones they would turn out to be, there will be $m_{0}(1-\frac {1}{t})^{{\hat {t}}}$ chains that do not reach a DP even after ${\hat {t}}$ more iterations. Hence $m_{0} (1-\frac {1}{t})^{{\hat {t}}}$ nodes among the m ₀ nodes belonging to the 0th column need to be removed from the count of new nodes. As for the 1st column, we had focused on m ₁ chains, but $m_{1} (1-\frac {1}{t})^{{\hat {t}}-1}$ nodes among these will not reach a DP before exceeding the chain length bound, and they need to be removed. The general term is now clear.

The coverage rate of a single DP table can thus be stated as

$$ \frac{1}{mt} \sum_{k=0}^{{\hat {t}}-1} m_k \biggl\{1-{ \biggl(1-\frac {1}{t} \biggr)}^{{\hat {t}}-k} \biggr\}. $$

Using Lemma 8, we can approximate this to

Since the coverage rate is of O(1) order and the first term $\frac {\sigma({\hat {t}})}{ \textup {\texttt {D}}_{\mathrm {msc}}}$ is of $O (\frac {1}{t})$ order, we simply discard the first term, and the summation term can be approximated by the integral

$$ \frac{t}{ \textup {\texttt {D}}_{\mathrm {msc}}} e^{-{\hat {t}}/t} \int_{0}^{{{\hat {t}}}/{t}} \sigma(t u) \exp(u)\, du, $$

when $\frac {1}{t}$ is small. The claimed formula follows after substitution of σ(tu), as given by Lemma 8, and some simplifications. □

We state the case where ${\hat {t}}$ is sufficiently large separately for later use.

Proposition 10

The expected coverage rate of a single DP table is approximately

$$ \textup {\texttt {D}}_{\mathrm{cr}}= \frac{2}{\sqrt{1 + 2 \textup {\texttt {D}}_{\mathrm {msc}}}+1}, $$

when the chain length bound  ${\hat {t}}$ is sufficiently large.

Proof

When the chain length bound ${\hat {t}}$ is sufficiently large, almost all of the m ₀≈m chains that are generated will terminate with a DP, and hence the coverage rate may be computed as $\frac{1}{mt} \sum_{i=0}^{{\hat {t}}-1} m_{i}$.

Based on Lemma 8, we may write

$$ \textup {\texttt {D}}_{\mathrm{cr}} \approx \frac{\sum_{i=0}^{{\hat {t}}-1} m_i}{mt} = \frac{{\textup {\textsf {N}}}\sigma({\hat {t}})}{mt} = \frac{2}{1-e^{-{\hat {t}}/t}} \frac{e^{\varXi {\hat {t}}/t} - 1}{(\varXi+1) e^{\varXi {{\hat {t}}}/{t}} + ( \varXi-1)}, $$

where $\varXi= \sqrt{1 + \frac{2 \textup {\texttt {D}}_{\mathrm {msc}}}{1-e^{-{\hat {t}}/t}}}$. When ${\hat {t}}$ is sufficiently larger than t, this is approximate to what is claimed. □

A careful reading of this proof shows that $\frac{{\hat {t}}}{t}$ does not need to be very large for the final approximation to be accurate. A ratio between ${\hat {t}}$ and t of such a not-too-large order is all we assume when we use the expression ${\hat {t}}$ is sufficiently large. We are not referring to the limit ${\hat {t}}\rightarrow\infty$. To the contrary, we wish to have ${\hat {t}}$ and t of somewhat similar order so that the approximation $(1-\frac {1}{t})^{{\hat {t}}}\approx e^{-{\hat {t}}/t}$ remains valid.

4.2 Time-Memory Tradeoff Curve

Our next goal is to summarize the ability of the DP tradeoff algorithm in balancing storage against online time into a single tradeoff equation.

This subsection is easier to follow if one visualizes the chains of the DP matrix as having been aligned at the ending points. The online iterations for the processing of a single DP table are counted starting from the 1st iteration. That is, checking whether y=F(x) is among the DPs in the DP table is referred to as the 1st iteration.

Our first task is to find the probability for merges to occur between DP chains.

Lemma 11

Fix a random function and suppose that we are given a precomputed DP chain of length  $j\leq {\hat {t}}$, generated with F from a random non-DP starting point. If a second chain is generated with F from a random starting point, the probability for it to become a DP chain of length i and merge with the given precomputed chain is

$$ \frac{t}{{\textup {\textsf {N}}}} \biggl\{\exp \biggl(\frac{\min\{i,j\}}{t} \biggr) - 1 \biggr\} \exp \biggl(-\frac{i}{t} \biggr). $$

Proof

Within this proof, let us refer to the event of the second chain becoming a DP chain of length i and merging with the precomputed chain simply as the event.

We first restrict ourselves to the i≤j case and fix the notation for the two chains as follows:

The nodes x ₀, …, x _j−1 are non-DPs and x _j is a DP.

Let us consider all possible scenarios by which the event can occur. If the randomly chosen starting point z ₀ happens to be equal to x _j−i , then the second chain will follow the first chain and the event surely will occur. On the other hand, if z ₀ is either one of the points x ₀, …, x _j−i−1, x _j−i+1, …, x _j−1, or a DP, then the event cannot occur. In the remaining case, i.e., when z ₀ is neither a DP nor any one of the points x ₀, …, x _j−1, then the possibility of the event occurring remains. Furthermore, in this last case, we may freely set F(z ₀) to a randomly chosen point of .

The above argument may now be repeated. If the randomly chosen z ₁=F(z ₀) is equal to x _j−i+1, then the event occurs. If z ₁ is either a DP or one of the points x ₀, …, x _j−i , x _j−i+2, …, x _j−1, then the event cannot occur. And if z ₁ is neither a DP nor one of the points x ₀, …, x _j−1, then the event occurrence is yet undecided and we are free to define z ₂=F(z ₁) to a random point of .

Hence, when i≤j, the probability for the event to occur may be written as

$$ \frac{1}{{\textup {\textsf {N}}}} + \biggl(1-\frac {1}{t}-\frac{j}{{\textup {\textsf {N}}}} \biggr)\frac{1}{{\textup {\textsf {N}}}} + \biggl(1-\frac {1}{t}-\frac{j}{{\textup {\textsf {N}}}} \biggr)^2\frac{1}{{\textup {\textsf {N}}}} + \cdots + \biggl(1-\frac {1}{t}-\frac{j}{{\textup {\textsf {N}}}} \biggr)^i\frac{1}{{\textup {\textsf {N}}}}, $$

which is equal to

$$ \frac{1}{{\textup {\textsf {N}}}} \frac{1- (1-\frac {1}{t}-\frac{j}{{\textup {\textsf {N}}}} )^{i+1}}{1- (1-\frac {1}{t}- \frac{j}{{\textup {\textsf {N}}}} )}. $$

Noting that $\frac{j}{{\textup {\textsf {N}}}} \ll \frac {1}{t}$ and using $(1-\frac {1}{t})^{i+1} \approx (1-\frac {1}{t})^{i} \approx\exp (-\frac {i}{t} )$, we can approximate this as

$$ \frac{t}{{\textup {\textsf {N}}}} \biggl\{1-\exp \biggl(-\frac{i}{t} \biggr) \biggr\}. $$

We can similarly work with the i≥j case. The event can occur only if the beginning random choices z ₀,…,z _i−j−1 are made among non-DPs that are different from x ₀,…,x _j−1. The probability for the event to occur is

$$ \biggl(1-\frac {1}{t}-\frac{j}{{\textup {\textsf {N}}}} \biggr)^{i-j}\frac{1}{{\textup {\textsf {N}}}} + \biggl(1-\frac {1}{t}-\frac{j}{{\textup {\textsf {N}}}} \biggr)^{i-j+1}\frac{1}{{\textup {\textsf {N}}}} + \cdots + \biggl(1-\frac {1}{t}-\frac{j}{{\textup {\textsf {N}}}} \biggr)^i\frac{1}{{\textup {\textsf {N}}}}, $$

which is approximately

$$ \frac{t}{{\textup {\textsf {N}}}} \biggl\{\exp \biggl(-\frac{i-j}{t} \biggr)-\exp \biggl(- \frac{i}{t} \biggr) \biggr\}. $$

The results for the cases i≤j and i≥j can be combined and stated as claimed. □

With the probability of alarms in our hands, we can compute the cost induced by false alarms.

Lemma 12

The number of extra one-way function invocations induced by alarms is expected to be

$$ t \frac{ \textup {\texttt {D}}_{\mathrm {msc}}}{1-e^{-{\hat {t}}/t}} \biggl\{ 2 - 8 e^{-{\hat {t}}/(2t)} + \biggl(5 + 3({ {\hat {t}}}/{t}) - \frac{1}{2}({{\hat {t}}}/{t})^2 \biggr) e^{-{\hat {t}}/t} + e^{-2{\hat {t}}/t} \biggr\}, $$

for each DP table.

Proof

When the chains are generated from m ₀ non-DP starting points as given by (13), one can expect to collect

$$ \frac{m}{1-e^{-{\hat {t}}/t}} { \biggl(1-\frac{1}{t} \biggr)}^{j-1} \frac {1}{t}\approx \frac{\frac{m}{t}}{1-e^{-{\hat {t}}/t}} \exp \biggl(- \frac{j}{t} \biggr) $$

(16)

DP chains of length j.

The probability of collision between the online chain and any one of these DP chains of length j, at the ith iteration of the online phase, is given by Lemma 11. Here, the 1st iteration deals with an online chain of length one, rather than zero, that starts at the unknown correct answer and ends at the inversion target.

The third component is the work required at each collision. If we take advantage of the fact that there is a chain length bound, in most cases, the number of iterations required to deal with a collision between a precomputed chain of length j and an online chain of length i will be $\min\{{\hat {t}}-i+1,j\}$. The only exception is when a pre-image to the inversion target is found, which is rare enough to be ignored.

Multiplying the three components and summing over all possible indices i and j, the expected number of iterations can be expressed as

$$ \sum_{i=1}^{\hat {t}}\sum _{j=1}^{\hat {t}}\frac{\frac{m}{t}}{1-e^{-{\hat {t}}/t}} \exp \biggl(- \frac{j}{t} \biggr)\cdot \frac{t}{{\textup {\textsf {N}}}} \biggl\{\exp \biggl( \frac{\min\{i,j\}}{t} \biggr)-1 \biggr\} \exp \biggl(-\frac{i}{t} \biggr)\cdot \min\{{\hat {t}}-i+1,j\}. $$

Replacing $\frac{i}{t}$ with u and $\frac{j}{t}$ with v, the above can be approximated by the integral

$$ \frac{\frac{mt^2}{{\textup {\textsf {N}}}} t}{1-e^{-{\hat {t}}/t}} \int_0^{{\hat {t}}/t} \int _0^{{\hat {t}}/t} \exp(-u)\exp(-v) \bigl\{\exp \bigl(\min \{u,v\} \bigr)-1 \bigr\} \min \biggl\{\frac{\hat{t}}{t}-u,v \biggr\} \,dv\,du, $$

when $\frac {1}{t}$ is small. The claimed value appears when this definite integral is computed. □

Finally, we write the tradeoff curve for the DP tradeoff in a way that takes into account the extra cost of alarm resolving.

Theorem 13

The time-memory tradeoff curve for the DP tradeoff is TM ²=D _tc N ², where the tradeoff coefficient is

Proof

The ith DP table is processed if and only if all previous tables did not return the correct answer. The probability of such a failure is $(1-\frac{\textup {\texttt {D}}_{\mathrm{cr}}mt}{{\textup {\textsf {N}}}} )^{i-1}$. The time required in processing a single table is the sum of one-way function invocation counts given by Lemma 3 and Lemma 12. Hence the expected total running time of the DP tradeoff may be written as

$$ T = \sum_{i=1}^\ell \biggl(\!1- \frac{\textup {\texttt {D}}_{\mathrm{cr}}mt}{{\textup {\textsf {N}}}} \biggr)^{i-1}\! \biggl\{ \bigl(1-e^{-{\hat {t}}/t} \bigr) + \frac{ \textup {\texttt {D}}_{\mathrm {msc}}}{1-e^{-{\hat {t}}/t}} \biggl( \!2 - \frac{8}{e^{{\hat {t}}/2t}} + \frac{5+\frac{3{\hat {t}}}{t}-\frac{{\hat {t}}^2}{2t^2}}{e^{{\hat {t}}/t}} + \frac{1}{e^{2{\hat {t}}/t}} \biggr) \!\biggr\} t. $$

The summation index i appears only in the first multiplicative factor, and we can easily check that

$$ \sum_{i=1}^\ell \biggl(1-\frac{\textup {\texttt {D}}_{\mathrm{cr}}mt}{{\textup {\textsf {N}}}} \biggr)^{i-1} = \frac{{\textup {\textsf {N}}}}{\textup {\texttt {D}}_{\mathrm{cr}}mt} \biggl \{1- \biggl(1-\frac{\textup {\texttt {D}}_{\mathrm{cr}}mt}{{\textup {\textsf {N}}}} \biggr)^{\ell} \biggr\} = \frac{ \textup {\texttt {D}}_{\mathrm {ps}}}{\textup {\texttt {D}}_{\mathrm{cr}} \textup {\texttt {D}}_{\mathrm {msc}}} t, $$

(17)

where the second equality follows from Proposition 5. The running time can now be rewritten as

$$ T = \frac{ \textup {\texttt {D}}_{\mathrm {ps}}}{\textup {\texttt {D}}_{\mathrm{cr}} \textup {\texttt {D}}_{\mathrm {msc}}} \biggl\{ \bigl(1-e^{-{\hat {t}}/t} \bigr) + \frac{ \textup {\texttt {D}}_{\mathrm {msc}}}{1-e^{-{\hat {t}}/t}} \biggl( 2 - \frac{8}{e^{{\hat {t}}/2t}} + \frac{5+\frac{3{\hat {t}}}{t}-\frac{{\hat {t}}^2}{2t^2}}{e^{{\hat {t}}/t}} + \frac{1}{e^{2{\hat {t}}/t}} \biggr) \biggr\} t^2. $$

(18)

Since the storage is M=mℓ, we have

The claim is reached by observing that

$$ \textup {\texttt {D}}_{\mathrm {pc}}^2 = \frac{(\textup {\texttt {D}}_{\mathrm{cr}} \textup {\texttt {D}}_{\mathrm {pc}})^2}{\textup {\texttt {D}}_{\mathrm{cr}}^2} = \frac{ \{\ln (1-\textup {\texttt {D}}_{\mathrm {ps}}) \}^2}{\textup {\texttt {D}}_{\mathrm{cr}}^2}, $$

where the second equality is again an application of Proposition 5. □

Let us emphasize that the tradeoff coefficient D _tc is an expected value rather than a bound. The tradeoff curve was computed without restricting to the worst case, in which the algorithm fails after processing all tables. The following statement is an immediate consequence of the preceding theorem.

Corollary 14

The time-memory tradeoff curve for the DP tradeoff is TM ²=D _tc N ² with

$$ \textup {\texttt {D}}_{\mathrm {tc}}= \biggl(2+\frac{1}{ \textup {\texttt {D}}_{\mathrm {msc}}} \biggr) \frac{1}{\textup {\texttt {D}}_{\mathrm{cr}}^3} \textup {\texttt {D}}_{\mathrm {ps}}\bigl\{ \ln(1- \textup {\texttt {D}}_{\mathrm {ps}}) \bigr\}^2, $$

when the chain length bound  ${\hat {t}}$ is sufficiently large.

We make the number of table lookups explicit for later use.

Lemma 15

The online processing of the DP tradeoff that uses the parameters m, t, ℓ, and  ${\hat {t}}$ is expected to require $t\frac{\textup {\texttt {D}}_{\mathrm {ps}}}{\textup {\texttt {D}}_{\mathrm{cr}} \textup {\texttt {D}}_{\mathrm {msc}}}$ lookups to the DP tables.

Proof

The ith DP table is processed if and only if all previous tables have failed in returning the correct answer and the processing of each table requires a single table lookup. Hence, the expected total number of table lookups is given by (17), as claimed. □

The dependence of this result on the chain length bound ${\hat {t}}$ is hidden inside the D _cr term.

4.3 Efficient Use of Storage

The storage size M appearing in any tradeoff curve refers to the total number of starting point and ending point pairs that need to be stored in the tradeoff tables. As explained in Sect. 2.7, the number of bits required to store a single starting and ending point pair will be different for each tradeoff algorithm. The focus of this section is on analyzing the ending point truncation technique explained in Sect. 2.7.4 for the DP tradeoffs.

It seems that the intention of the works [4, 6] while using ending point truncation was to keep slightly more than logm bits of each ending point, so that each ending point within a DP table could be identified almost uniquely. However, this would also imply that almost every lookup to the precomputation table will generate a match of truncated points.

Let us start with a rough preliminary analysis of the situation where logm bits are stored for each ending point. The online chain creation during processing of a table requires Θ(t) iterations of the one-way function and will generate a single lookup to the table. The alarm that is almost surely generated by the lookup will require Θ(t) additional one-way function iterations to resolve. Hence, the total cost per table processing remains at Θ(t) even with ending points truncated to logm bits, and the truncation to logm bits seem reasonable. Truncation to smaller than logm bits will result in the return of multiple collisions at the single table lookup and will quickly become problematic.

Although one is guaranteed not to see a radical change in the online time complexity after truncating ending points to logm bits, the above analysis does not provide implementers with the information on how close to logm bits one may venture without experiencing visible side effects to the online time complexity. For now, implementers can only repeatedly tweak and make test runs to decide on the appropriate degree of truncation.

Consider an ending point truncation method for which two random points of , truncated in the specified manner, will have probability $\frac{1}{r}$ of matching with each other. We shall express such a situation as having $\frac{1}{r}$ probability of truncated match. For example, if logt bits from the ending points were truncated with D _msc=1, so that (logm+logt) bits remain, then the truncated matches would occur with probability $\frac {1}{mt}$. When truncating ending point DPs, one should truncate the random-looking part, rather than the distinguished part. Removal of the distinguished part can always be undone, and does not cause any loss of ending point information.

Lemma 16

Assume the use of ending point truncation with the truncated match probability set to  $\frac{1}{r}$. The number of extra one-way function invocations induced by truncation-related alarms is expected to be

$$ t \frac{1 - 2({\hat {t}}/t) e^{-{\hat {t}}/t} - e^{-2{\hat {t}}/t}}{ 1-e^{-{\hat {t}}/t}} \frac{mt}{r}, $$

for each DP table.

Proof

Consider a random function and suppose that the first chain, generated with F and a random non-DP starting point, became a DP chain of length $j\leq\nobreak {\hat {t}}$. Now, suppose a second chain is generated with F from a random non-DP starting point. Let us compute the probability for the second chain to become a DP chain of length i and not merge with the first chain, but have the same truncated ending point as the first chain.

The first i nodes of the second chain must be chosen among non-DPs that are different from the j pre-ending points of the first chain. The ith node chosen, when truncated, needs to agree with the truncated ending point of the first chain. Note that this agreement already requires the final point to be a DP. Thus the probability we aimed to write can be expressed as

$$ \biggl(1-\frac {1}{t}-\frac{j}{{\textup {\textsf {N}}}} \biggr)^i \biggl( \frac{1}{r} - \frac {1}{{\textup {\textsf {N}}}} \biggr) \approx \exp \biggl(- \frac{i}{t} \biggr)\frac{1}{r}. $$

(19)

Now, we can combine the number of DP chains of length j, as given by (16), together with the probability of non-merging truncated collision with such a chain, as given by (19), to write the cost of truncation-related false alarms as

$$ \sum_{i=1}^{\hat {t}}\sum _{j=1}^{\hat {t}}\frac{\frac{m}{t}}{1-e^{-{\hat {t}}/t}} \exp \biggl(- \frac{j}{t} \biggr)\cdot \exp \biggl(-\frac{i}{t} \biggr) \frac{1}{r}\cdot \min\{{\hat {t}}-i+1,j\}. $$

It now suffices to simplify this expression. Replacing $\frac{i}{t}$ with u and $\frac{j}{t}$ with v, the above can be approximated by the definite integral

$$ \frac{mt^2}{1-e^{-{\hat {t}}/t}} \frac{1}{r} \int_0^{{\hat {t}}/t} \int_0^{{\hat {t}}/t} \exp(-u)\exp(-v) \min \biggl\{ \frac{\hat{t}}{t}-u,v \biggr\} \,dv\,du, $$

when $\frac {1}{t}$ is small. We arrive at the claimed value when this is explicitly computed. □

Combining Lemmas 3, 12, and 16, we know that the online processing of a single DP table requires

invocations of the one-way function. When ${\hat {t}}$ is sufficiently large, this simplifies to

$$ t + t 2 \textup {\texttt {D}}_{\mathrm {msc}}+ t \frac{mt}{r}, $$

with each additive term corresponding to the three terms given before. The ratio of the original number of iterations to the number of extra iterations incurred by truncations is

$$ (t+t 2 \textup {\texttt {D}}_{\mathrm {msc}}) : t \frac{mt}{r} = r : \frac{mt}{1+2 \textup {\texttt {D}}_{\mathrm {msc}}}. $$

The choice of $r = \frac{mt}{1+2 \textup {\texttt {D}}_{\mathrm {msc}}}$ will give an implementation whose added cost of truncation-related alarms increases the nontruncated original cost by 100 %. Noting that a truncated match probability of $\frac{1}{r}$ is achieved by leaving logr bits after truncation, we summarize what we have discussed in the following statement.

Proposition 17

Fix a set of parameters for a DP tradeoff such that the chain length bound  ${\hat {t}}$ is sufficiently large. Suppose that the online phase of the DP tradeoff implementation that stores each ending point in full requires T iterations of the one-way function to complete. Then, an implementation that leaves

$$ \log m + \log t - \log(1+2 \textup {\texttt {D}}_{\mathrm {msc}}) \pm\varepsilon $$

bits per ending point after truncation, where ε is a small non-negative integer, requires 2^∓ε T additional iterations of the one-way function to complete.

Let us recall the contents of Sect. 2.7 and summarize how DP table storage can be optimized. Sequential use of starting points allows each starting point to be recorded in approximately logm bits. One can truncate and leave slightly more than logm+logt bits in each ending point and experience minimal side effect on the online running time. The decision on the exact degree of truncation can be made with the help of Proposition 17. Of the remaining approximately logm+logt bits of the ending point, we do not need to store the logt bits that are fixed through the distinguishing property. Furthermore, the index table technique allows us to remove almost logm more bits without any loss of information. In all, logm bits are required to store each starting point, and a very small number of bits are required to store each ending point. We have thus confirmed the claims of [4, 6] theoretically.

Example 18

Consider an extremely large tradeoff implementation with N=2⁷⁵ and assume the typical parameters $m \approx t \approx\ell \approx {\textup {\textsf {N}}}^{\frac{1}{3}} = 2^{25}$. Each starting point requires 25 bits. The DP definition allows removal of 25 bits from each ending point. We assume removal of 23 further bits through the index table method. Let us approximate log(1+2D _msc)≈2. Then, each table entry will require 25+ε bits.

Let T be the number of one-way function iterations required for the online chain creation and the resolving of alarms in the absence of ending point truncations. When ε is changed from 4 to 3, the storage decreases by $\frac{29-28}{29} \approx3.45~\%$ while the iterations increase by 5.88 % from $(1+\frac{1}{2^{4}})T$ to $(1+\frac {1}{2^{3}})T$. This tradeoff is better than the tradeoff achievable through the changes in m, t, and ℓ. However, when similar calculations are made for the change of ε from 3 to 2, one can confirm that the increase in online time is not worth the decrease in storage.

In summary, for the assumed rough range of parameters, it is advisable to allocate approximately 28 bits per table entry and accept the $\frac{9}{8}T$ online time, even though this is visibly different from T.

5 Hellman Tradeoff

In this section, we gather facts about the complexity of the Hellman tradeoff. As in the previous section, the reduction functions are kept hidden during the analysis.

Our first statement is quite trivial.

Proposition 19

The precomputation phase of the Hellman tradeoff requires mtℓ one-way function invocations.

We define the precomputation coefficient for the Hellman tradeoff to be $\textup {\texttt {H}}_{\mathrm {pc}}= \frac{mt\ell}{ {\textup {\textsf {N}}}}$, so that the precomputation cost of a Hellman tradeoff is H _pc N. The next proposition is a restatement of (4).

Proposition 20

The success probability of the Hellman tradeoff is

$$ \textup {\texttt {H}}_{\mathrm {ps}}= 1 - e^{-\textup {\texttt {H}}_{\mathrm{cr}} \textup {\texttt {H}}_{\mathrm {pc}}}. $$

We next state the coverage rate, so that the above expression for probability of success can be put to use. This is a trivial modification of statements from [9, 19].

Proposition 21

The coverage rate of a single Hellman table is expected to be

$$ \textup {\texttt {H}}_{\mathrm{cr}}= \frac{\sqrt{2}}{\sqrt{ \textup {\texttt {H}}_{\mathrm {msc}}}} \frac{e^{\sqrt{2 \textup {\texttt {H}}_{\mathrm {msc}}}}-1}{e^{\sqrt{2 \textup {\texttt {H}}_{\mathrm {msc}}}}+1}. $$

The tradeoff efficiency of the Hellman tradeoff is compactly expressed by the following time-memory tradeoff curve. This result takes into account the cost of resolving alarms, and, unlike (8), which semi-corresponds to an upper bound on the efficiency, expresses the average behavior.

Theorem 22

The time-memory tradeoff curve for the Hellman tradeoff is TM ²=H _tc N ², where the tradeoff coefficient is

$$ \textup {\texttt {H}}_{\mathrm {tc}}= \biggl(\frac{1}{ \textup {\texttt {H}}_{\mathrm {msc}}}+\frac{1}{6} \biggr) \frac{1}{\textup {\texttt {H}}_{\mathrm{cr}}^3} \textup {\texttt {H}}_{\mathrm {ps}}\bigl\{\ln(1- \textup {\texttt {H}}_{\mathrm {ps}}) \bigr\}^2. $$

Proof

The ith Hellman table is processed if and only if all previous tables have failed in returning the correct answer. The probability of such a failure is $(1-\frac{\textup {\texttt {H}}_{\mathrm{cr}}mt}{{\textup {\textsf {N}}}} )^{i-1}$. Recalling the number of one-way function invocations required per Hellman table to resolve false alarms (7), the number of all iterations required per table can be written as $(1+\frac { \textup {\texttt {H}}_{\mathrm {msc}}}{6} )t$. The expected total running time of the Hellman tradeoff may be written as

$$ T = \sum_{i=1}^\ell \biggl(1-\frac{\textup {\texttt {H}}_{\mathrm{cr}}mt}{{\textup {\textsf {N}}}} \biggr)^{i-1} \biggl(1+\frac{ \textup {\texttt {H}}_{\mathrm {msc}}}{6} \biggr) t. $$

(20)

The summation index i appears only in the first multiplicative factor, and we can easily check that

$$ \sum_{i=1}^\ell \biggl(1- \frac{\textup {\texttt {H}}_{\mathrm{cr}}mt}{{\textup {\textsf {N}}}} \biggr)^{i-1} = \frac{{\textup {\textsf {N}}}}{\textup {\texttt {H}}_{\mathrm{cr}}mt} \biggl\{1- \biggl(1-\frac{\textup {\texttt {H}}_{\mathrm{cr}}mt}{{\textup {\textsf {N}}}} \biggr)^{\ell} \biggr\} = \frac{ \textup {\texttt {H}}_{\mathrm {ps}}}{\textup {\texttt {H}}_{\mathrm{cr}} \textup {\texttt {H}}_{\mathrm {msc}}} t, $$

where the final equality follows from Proposition 20. Returning to (20), the execution time can now be written as

$$ T = \biggl(\frac{1}{ \textup {\texttt {H}}_{\mathrm {msc}}}+\frac{1}{6} \biggr) \frac{ \textup {\texttt {H}}_{\mathrm {ps}}}{\textup {\texttt {H}}_{\mathrm{cr}}} t^2. $$

(21)

Since the storage size is M=mℓ, we have

where the final equality again relies on Proposition 20. □

The time T, stated during the above proof as (21), counts the number of one-way function computations, and includes the efforts for resolving alarms. Since the number of table lookups will be smaller, we make this count explicit.

Lemma 23

The online processing of the Hellman tradeoff that uses the parameters m, t, and ℓ is expected to require $t^{2} \frac{ \textup {\texttt {H}}_{\mathrm {ps}}}{\textup {\texttt {H}}_{\mathrm{cr}} \textup {\texttt {H}}_{\mathrm {msc}}}$ lookups to the Hellman tables.

The proof to this lemma is almost identical to that of Lemma 15. The only difference is that the processing of each table requires t lookups, rather than one.

After reading the proof to Theorem 22, one can easily write the expected cost of resolving alarms for the Hellman tradeoff as $\frac{ \textup {\texttt {H}}_{\mathrm {ps}}}{6 \textup {\texttt {H}}_{\mathrm{cr}}} t^{2}$, and by following through the relations

$$ \frac{ \textup {\texttt {H}}_{\mathrm {ps}}}{6 \textup {\texttt {H}}_{\mathrm{cr}}} t^2 = \frac{1-e^{-\textup {\texttt {H}}_{\mathrm{cr}} \textup {\texttt {H}}_{\mathrm {pc}}}}{6 \textup {\texttt {H}}_{\mathrm{cr}}} t^2 \leq\frac{1-(1-\textup {\texttt {H}}_{\mathrm{cr}} \textup {\texttt {H}}_{\mathrm {pc}})}{6 \textup {\texttt {H}}_{\mathrm{cr}}} t^2 = \frac{mt\ell}{6{\textup {\textsf {N}}}} t^2 = \frac{ \textup {\texttt {H}}_{\mathrm {msc}}}{6} t\ell, $$

we can recover the old approximation (6). This shows that the bound (6) is far from being tight, unless H _cr H _pc≪1.

We have so far secured access to the precomputation cost, the success probability, and the tradeoff efficiency of the Hellman tradeoff. It remains to discuss the use of storage. Three of the approaches to storage reduction that were discussed in Sect. 2.7 are applicable to the Hellman tradeoff, and we provide an analysis of the ending point truncation method below.

Let us start with a preliminary analysis. Assume that ending points are truncated so that logm bits are stored for each ending point. Then the table entries are uniquely identifiable, but each table lookup would return one truncated match on average. The cost of resolving alarms becomes $t + (t-1) +\cdots+ 1 \approx\frac{t^{2}}{2}$ per table. This dominates the online chain creation cost of t, so truncation to logm bits is not an acceptable method.

A more exact analysis of ending point truncation is given next. We reuse the concept of truncated match probability, previously defined for the DP tradeoff, with the Hellman tradeoff.

Lemma 24

Assume the use of ending point truncation with the truncated match probability set to  $\frac{1}{r}$. The number of extra one-way function invocations induced by truncation-related alarms is expected to be

$$ t \frac{mt}{2r}, $$

for each Hellman table.

Proof

Fix a random function and suppose that we are given a precomputed chain of length t, generated with F from a random starting point. Now consider a second chain generated with F from a random starting point. The probability for it to produce an alarm related to truncation, i.e., a truncated ending point match without a merge with the first chain, on the ith iteration, is

$$ \biggl(1-\frac{1}{{\textup {\textsf {N}}}} \biggr)^i \biggl(\frac{1}{r} - \frac {1}{{\textup {\textsf {N}}}} \biggr) \approx \biggl(1-\frac{i}{{\textup {\textsf {N}}}} \biggr) \biggl( \frac{1}{r} - \frac{1}{{\textup {\textsf {N}}}} \biggr) \approx\frac{1}{r}. $$

This is because the first i nodes of the second chain must be chosen among nodes that are different from the t pre-ending points of the first chain.

Taking account of all m precomputed chains, the cost induced by the truncation-related alarms can now be written as

$$ \sum_{i=1}^t \frac{m}{r} (t-i+1) \approx \frac{mt^2}{r} \sum_{i=1}^t \biggl(1-\frac{i}{t} \biggr) \frac{1}{t}. $$

When $\frac {1}{t}$ is small, by replacing $\frac{i}{t}$ with u, the above can be approximated with the definite integral

$$ \frac{mt^2}{r} \int_0^{1} (1-u) \,du, $$

which computes to $\frac{mt^{2}}{2r}$, as claimed. □

Combining this with what we saw during the proof of Theorem 22, the total online time required to deal with a single Hellman table can be stated as

$$ t + t \frac{ \textup {\texttt {H}}_{\mathrm {msc}}}{6} + t \frac{mt}{2r}. $$

Arguing as we did in the previous section concerning ending point truncations for the DP tradeoffs, we can come to the following conclusion.

Proposition 25

Fix a set of parameters for the Hellman tradeoff and suppose that its implementation which stores full ending point information requires T iterations of the one-way function to complete the online phase. Then, an implementation that leaves

$$ \log m + \log t - \log \biggl(2+\frac{ \textup {\texttt {H}}_{\mathrm {msc}}}{3} \biggr) \pm\varepsilon $$

bits per ending point after truncation, where ε is a small non-negative integer, requires 2^∓ε T additional iterations of the one-way function to complete.

We can summarize how Hellman table storage can be optimized after recalling the contents of Sect. 2.7. Each starting point requires logm bits. Ending points may be truncated so that slightly more than logm+logt bits remain without experiencing visible side effects on the online running time. The decision on the exact degree of truncation can be made with the help of Proposition 25. Using the index table technique, almost logm additional bits can be removed without any loss of information. In all, logm bits are required for each starting point and slightly more than logt bits are required for each ending point. This is very different from the conclusions for the DP tradeoff.

Example 26

Let us reuse the parameters of Example 18. Assuming that the index table allows removal of 23 bits and accepting the approximation $\log (2+\frac{ \textup {\texttt {H}}_{\mathrm {msc}}}{3} ) \approx1$, each table entry is seen to require 25+26+ε bits.

With T equal to the nontruncated iterations, when ε is changed from 5 to 4, the storage decreases by $\frac{56-55}{56} \approx1.79~\%$, while the iterations increase by $\{(1+\frac {1}{2^{4}})T-(1+\frac{1}{2^{5}})T\}/\{(1+\frac{1}{2^{5}})T\} \approx3.03~\%$. This is an acceptable tradeoff. However, the change of ε from 4 to 3 results in a 1.82 % decrease in storage, which cannot justify the corresponding 5.88 % increase in online time.

In summary, for the assumed rough range of parameters, it is advisable to allocate approximately 55 bits per table entry and accept the $\frac{17}{16}T$ online time, which is slightly higher than T.

6 Rainbow Tradeoff

In this section, we gather facts about the rainbow tradeoff. Recall that multiple rainbow tables are to be processed in parallel. The 1st iteration of a rainbow tradeoff online phase will refer to the ℓ-many searchings of $\mathbf {y}_{t}^{k,1} = F_{t,k}(\mathbf {x})$ among the ending points of the kth rainbow table with the index k running from 1 to ℓ. The jth iteration will require (j−1)⋅ℓ invocations of the one-way function and ℓ lookups to different tables.

Our first claim is a direct consequence of the relation mt=R _msc N that defines the notation R _msc.

Proposition 27

The precomputation phase of the rainbow tradeoff requires R _pc N one-way function invocations, where the precomputation coefficient is R _pc=R _msc ℓ.

The contents of the following lemma for the ℓ=1 case were already used in certain computations of [15], but let us restate it here in a more readily accessible form. The first statement of this lemma is a trivial extension of the past result (10).

Lemma 28

The probability for the first k iterations of the online phase to fail is

$$ \prod_{i=1}^k \biggl(1 - \frac{m_{t-i}}{{\textup {\textsf {N}}}} \biggr)^\ell, $$

where m ₀=m and $\frac{m_{i+1}}{{\textup {\textsf {N}}}} = 1-\exp (-\frac {m_{i}}{{\textup {\textsf {N}}}} )$. This product may be approximated by

$$ \biggl( 1 - \frac{ \textup {\texttt {R}}_{\mathrm {msc}}}{2+ \textup {\texttt {R}}_{\mathrm {msc}}} \frac{k+1}{t} \biggr)^{2\ell}. $$

Proof

The second statement is based on the approximation

$$ \frac{m_i}{{\textup {\textsf {N}}}} \approx\frac{1}{{\textup {\textsf {N}}}/m + i/2}, $$

which appears in [15]. This is a very small generalization of a result from [1], which treated the m=N case. After rewriting this as

$$ 1 - \frac{m_{t-i}}{{\textup {\textsf {N}}}} \approx\frac{2{\textup {\textsf {N}}}+ m(t-i-2)}{2{\textup {\textsf {N}}}+ m(t-i)}, $$

the sequential cancellations within the product become visible, and we arrive at

$$ \prod_{i=1}^k \biggl(1 - \frac{m_{t-i}}{{\textup {\textsf {N}}}} \biggr)^\ell \approx \biggl\{ \frac{2{\textup {\textsf {N}}}+ m(t-k-1)}{2{\textup {\textsf {N}}}+m(t-1)} \frac{2{\textup {\textsf {N}}}+ m(t-k-2)}{2{\textup {\textsf {N}}}+m(t-2)} \biggr\}^\ell \approx \biggl\{ 1 - \frac{ \textup {\texttt {R}}_{\mathrm {msc}}\frac{k+1}{t}}{2+ \textup {\texttt {R}}_{\mathrm {msc}}} \biggr\}^{2\ell}, $$

which is the claimed approximation. □

We can arrive at the next claim by substituting k=t into the above lemma and ignoring an insignificant term.

Proposition 29

The success probability of the rainbow tradeoff is

$$ \textup {\texttt {R}}_{\mathrm {ps}}= 1 - \biggl(\frac{2}{2+ \textup {\texttt {R}}_{\mathrm {msc}}} \biggr)^{2\ell}. $$

The tradeoff efficiency of the rainbow tradeoff is compactly expressed by the following theorem. The average efficiency, rather than the worst-case situation, is expressed by this result, and the effects of false alarms have been taken into account.

Theorem 30

The time-memory tradeoff curve for the rainbow tradeoff is TM ²=R _tc N ², where the tradeoff coefficient is

Proof

Substituting k=i−1 into Lemma 28, we know that the ith iteration is processed with probability $(1-\frac { \textup {\texttt {R}}_{\mathrm {msc}}}{2+ \textup {\texttt {R}}_{\mathrm {msc}}}\frac{i}{t} )^{2\ell}$. The probability of alarm occurrence associated with a single chain in a single rainbow matrix at the ith iteration may be inferred from [15] to be $\frac{i+1}{{\textup {\textsf {N}}}}$. The reasoning behind this second statement is identical to the proof that led to the older results (6) and (7).

Hence, the expected total running time of the rainbow tradeoff, taking into account the cost of resolving alarms associated with all m rows, may be written as

This may be approximated by the definite integral

$$ T = t^2 \ell\int_0^1 u \bigl\{ 1 + \textup {\texttt {R}}_{\mathrm {msc}}(1-u) \bigr\} \biggl(1-\frac{ \textup {\texttt {R}}_{\mathrm {msc}}}{2+ \textup {\texttt {R}}_{\mathrm {msc}}} u \biggr)^{2\ell}\,du, $$

which computes to

(22)

It now suffices to combine this with the storage size M=mℓ and simplify to arrive at the claim. □

The time T appearing in the above tradeoff curve gives the count of one-way function invocations and ignores table lookups.

Lemma 31

The online processing of the rainbow tradeoff is expected to require

$$ t \ell \frac{2+ \textup {\texttt {R}}_{\mathrm {msc}}-2 (\frac{2}{2+ \textup {\texttt {R}}_{\mathrm {msc}}} )^{2\ell}}{(2\ell +1) \textup {\texttt {R}}_{\mathrm {msc}}} $$

lookups to the rainbow tables.

Proof

At the start of the proof of Theorem 30, we saw that the ith iteration is processed with probability $(1-\frac{\textup {\texttt {R}}_{\mathrm {msc}}}{2+ \textup {\texttt {R}}_{\mathrm {msc}}}\frac{i}{t} )^{2\ell}$. Since each iteration requires ℓ table lookups, it suffices to compute

$$ \sum_{i=1}^t \ell \biggl(1- \frac{ \textup {\texttt {R}}_{\mathrm {msc}}}{2+ \textup {\texttt {R}}_{\mathrm {msc}}} \frac{i}{t} \biggr)^{2\ell} \approx t \ell \int_0^1 \biggl(1-\frac{ \textup {\texttt {R}}_{\mathrm {msc}}}{2+ \textup {\texttt {R}}_{\mathrm {msc}}} u \biggr)^{2\ell}\,du, $$

to arrive at the expected number of table lookups. □

We now turn to the issue of efficient storage use. The number of online iterations, which is of Θ(t ² ℓ) order, is much larger than the number of table lookups, given by the above lemma as being of Θ(tℓ) order. This indicates that truncation to slightly more than logm bits, which allows unique identification of table entries, should be reasonable. A more accurate analysis is given below. We reuse the concept of truncated match probability, defined for the DP tradeoffs, also in the rainbow tradeoff case.

Lemma 32

Assume the use of ending point truncation with the truncated match probability set to  $\frac{1}{r}$. The number of additional one-way function invocations induced by alarms related to ending point truncations is expected to be

$$ t^2 \ell \frac{m}{r} \frac{(-2+(2\ell+1) \textup {\texttt {R}}_{\mathrm {msc}})(2+ \textup {\texttt {R}}_{\mathrm {msc}}) + 4 (\frac{2}{2+ \textup {\texttt {R}}_{\mathrm {msc}}} )^{2\ell}}{ (2\ell+1)(2\ell+2) \textup {\texttt {R}}_{\mathrm {msc}}^2}. $$

Proof

For exactly the same reason given in the proof of Lemma 24, the probability for a randomly generated second chain to produce a truncation induced alarm without merging with the first chain is

$$ \biggl(1-\frac{1}{{\textup {\textsf {N}}}} \biggr)^i \biggl(\frac{1}{r} - \frac {1}{{\textup {\textsf {N}}}} \biggr) \approx \biggl(1-\frac{i}{{\textup {\textsf {N}}}} \biggr) \biggl( \frac{1}{r} - \frac{1}{{\textup {\textsf {N}}}} \biggr) \approx\frac{1}{r}. $$

After recalling Lemma 28, the probability for the ith iteration to be processed, and taking into account all the mℓ precomputed chains, the expected online cost can be written as

$$ \sum_{i=1}^t (t-i+1) \frac{m\ell}{r} \biggl(1-\frac{ \textup {\texttt {R}}_{\mathrm {msc}}}{2+ \textup {\texttt {R}}_{\mathrm {msc}}} \frac{i}{t} \biggr)^{2\ell}. $$

Replacing $\frac{i}{t}$ with u, the above can be approximated by the definite integral

$$ \frac{mt^2 \ell}{r} \int_0^{1} (1-u) \biggl(1-\frac{ \textup {\texttt {R}}_{\mathrm {msc}}}{2+ \textup {\texttt {R}}_{\mathrm {msc}}} u \biggr)^{2\ell} \,du, $$

when $\frac {1}{t}$ is small, and the claimed value appears when this is computed. □

After reviewing the arguments concerning ending point truncation made for the DP and Hellman tradeoffs, we can combine (22) and Lemma 32 to write the effects of ending point truncation in terms of the number of bits remaining.

Proposition 33

Fix a set of parameters for the rainbow tradeoff and suppose that its implementation which stores full ending point information is expected to require T iterations of the one-way function for the online phase. Then, an implementation that leaves

bits per ending point after truncation, where ε is a small non-negative integer, requires 2^∓ε T additional iterations of the one-way function to complete.

Referencing Sect. 2.7, let us summarize the number of bits required to store each starting point and ending point pair. Each starting point requires logm bits. Ending points may be truncated so that slightly more than logm bits remain without visible side effects on the online running time. The index table method allows most of the remaining logm bits to be removed from the ending point without any loss of information. In all, logm bits are required for each starting point and only a very small number of bits are required for each ending point. We have thus confirmed the claims of [4, 6].

Example 34

The parameters for a rainbow tradeoff that roughly correspond to those used in Examples 18 and 26 are m=2⁵⁰, t=2²⁵, and ℓ=1. Assume that the index table allows removal of 48 bits. The middle term appearing in the equation of Proposition 33 for the parameters being used is $\log\frac{215}{228} \approx0$. Each table entry will require 50+2+ε bits.

Let T be the number of iterations expected of a nontruncated implementation. When ε is changed from 6 to 5, the storage decreases by $\frac{58-57}{58} \approx1.72~\%$, while the iterations increase by $\{(1+\frac{1}{2^{5}})T-(1+\frac{1}{2^{6}})T\}/\{(1+\frac {1}{2^{6}})T\} \approx1.54~\%$. This is an acceptable tradeoff. However, the change of ε from 5 to 4 results in a 1.75 % decrease in storage, which cannot justify the corresponding 3.03 % increase in online time.

In summary, for the assumed rough range of parameters, it is advisable to allocate approximately 55 bits per table entry and accept the $\frac{33}{32}T$ online time, which is only slightly higher than T.

7 Optimal Tradeoff Parameters

In this section, we find the optimal set of parameters for the three tradeoff algorithms. The notion of optimality in this section ignores the cost of precomputation.

Let us present our initial arguments in terms of the Hellman tradeoff. The balance between time and memory achievable by the Hellman tradeoff is expressed by the tradeoff curve TM ²=H _tc N ². It is clear that the Hellman algorithm at parameters m, t, and ℓ that bring about a smaller tradeoff coefficient H _tc will require less resources to run. In other words, tradeoff coefficient H _tc is a measure of the tradeoff efficiency, with a smaller value representing a more desirable balancing of storage and online time.

The tradeoff coefficient H _tc is fully determined by the parameters m, t, and ℓ. It should first be noticed that a better tradeoff coefficient should always be achievable, if one decides to sacrifice the success probability of finding the correct answer. Hence, any comparison between two Hellman tradeoff coefficients, achievable through two different sets of parameters, should be done under the condition that they produce the same success probability.

Arguments similar to the above may be made for the DP and rainbow tradeoffs. Hence, for each of the three algorithms, we will work to find the smallest tradeoff coefficient achievable under a fixed requirement on the success rate.

The smallest possible tradeoff coefficient value for a tradeoff algorithm is referred to as the tradeoff characteristic in [1], where it is used to compare the perfect version of the rainbow table method against other algorithms. However, we wish for the optimal tradeoff coefficients given in this work to be understood separately for each algorithm. Using it to argue the superiority of one algorithm over another may seem plausible, but is of limited value in practice. Parameters achieving better tradeoff efficiency may require more precomputation, and with large-scale implementations of the tradeoff technique, lowering the precomputation cost may be significantly more valuable than achieving better tradeoff efficiency. Our purpose of locating the optimal tradeoff parameters is so that they may be used in the next section to bound the range of parameters, when making fair comparisons between different algorithms.

7.1 DP Tradeoff

The parameter set that achieves the optimal DP tradeoff efficiency, under a fixed requirement on the probability of success, is given below.

Proposition 35

Let 0<D _ps<1 be any fixed value. The DP tradeoff, under any set of parameters m, t, ℓ, and ${\hat {t}}$, that are subject to the relations

$$ mt^2 = 1.26453 {\textup {\textsf {N}}},\qquad \ell= 1.28007 \bigl\{ -\ln({1- \textup {\texttt {D}}_{\mathrm {ps}}}) \bigr\} t, \quad\text{\textit{and}}\quad {\hat {t}}= 2.59169 t, $$

attains the given value  D _ps as its probability of success, and exhibits a tradeoff performance corresponding to

$$ \textup {\texttt {D}}_{\mathrm {tc}}= 5.49370 \textup {\texttt {D}}_{\mathrm {ps}}\bigl\{\ln(1- \textup {\texttt {D}}_{\mathrm {ps}}) \bigr\}^2, $$

as the four parameters are varied. Under any such choice of parameters, the number of one-way function invocations required for the precomputation phase is

$$ \textup {\texttt {D}}_{\mathrm {pc}}{\textup {\textsf {N}}}= 1.61869 \bigl\{ -\ln({1- \textup {\texttt {D}}_{\mathrm {ps}}}) \bigr\} {\textup {\textsf {N}}}. $$

The three relations restricting the parameter choices give optimal parameters in the sense that no choice of m, t, ℓ, and  ${\hat {t}}$ can lead to a tradeoff coefficient smaller than the above while achieving D _ps as its probability of success.

Proof

The relation of Proposition 5 may equivalently be stated as

$$ \ell = \frac{{\textup {\textsf {N}}}}{\textup {\texttt {D}}_{\mathrm{cr}} mt} \bigl\{-\ln({1- \textup {\texttt {D}}_{\mathrm {ps}}}) \bigr\} = \frac{1}{\textup {\texttt {D}}_{\mathrm{cr}} \textup {\texttt {D}}_{\mathrm {msc}}} \bigl\{ -\ln({1- \textup {\texttt {D}}_{\mathrm {ps}}}) \bigr\} t. $$

(23)

Now, referencing Proposition 9, we know that the DP coverage rate $\textup {\texttt {D}}_{\mathrm{cr}}= \textup {\texttt {D}}_{\mathrm{cr}}[ \textup {\texttt {D}}_{\mathrm {msc}}, {\hat {t}}/t]$ may be treated as a function of the two variables D _msc and $\frac{\hat{t}}{t}$. Hence, given any m, t, ${\hat {t}}$, and D _ps, if we set $\textup {\texttt {D}}_{\mathrm {msc}}= \frac{mt^{2}}{{\textup {\textsf {N}}}}$ and $\textup {\texttt {D}}_{\mathrm{cr}}= \textup {\texttt {D}}_{\mathrm{cr}}[ \textup {\texttt {D}}_{\mathrm {msc}}, {\hat {t}}/t]$, and also fix ℓ through the relation (23), then the DP tradeoff with these parameters will always achieve the success probability of D _ps. We remark that ℓ must be set to an integer, but since the right-hand side of (23) is rather large, the error to the success probability, introduced by taking the nearest integer to the right-hand side value, will be very small.

Keeping in mind that we may freely choose m, t, and ${\hat {t}}$ and still obtain any requested success probability, we now work to minimize the DP tradeoff coefficient D _tc, as given by Theorem 13. We drop from the expression for D _tc any part that depends only on D _ps and consider

$$ \textup {\texttt {D}}_{\mathrm {tmp}}\biggl[ \textup {\texttt {D}}_{\mathrm {msc}}, \frac{\hat{t}}{t} \biggr] = \frac{ (2 \textup {\texttt {D}}_{\mathrm {msc}}+1) - \frac{8 \textup {\texttt {D}}_{\mathrm {msc}}}{e^{{\hat {t}}/2t}} + \frac{(5+\frac{3{\hat {t}}}{t}-\frac{{\hat {t}}^2}{2t^2})\textup {\texttt {D}}_{\mathrm {msc}}-2}{e^{{\hat {t}}/t}} + \frac{ \textup {\texttt {D}}_{\mathrm {msc}}+1}{e^{2{\hat {t}}/t}} }{ (1-e^{-{\hat {t}}/t} ) \textup {\texttt {D}}_{\mathrm{cr}} [ \textup {\texttt {D}}_{\mathrm {msc}}, \frac{\hat{t}}{t} ]^3 \textup {\texttt {D}}_{\mathrm {msc}}}, $$

(24)

which is a function of the two variables D _msc and $\frac{\hat {t}}{t}$. It is clear that, when the probability of success requirement is fixed, minimizing D _tc is equivalent to minimizing $\textup {\texttt {D}}_{\mathrm {tmp}}[\textup {\texttt {D}}_{\mathrm {msc}}, {\hat {t}}/t]$. Note that, even though $\textup {\texttt {D}}_{\mathrm {msc}}= \frac{mt^{2}}{{\textup {\textsf {N}}}}$ and ${\hat {t}}/t$ share the parameter t, since we are free to set m, t, and ${\hat {t}}$ to any value, there are enough degrees of freedom, and we may treat D _msc and ${\hat {t}}/t$ as independent variables when looking for the minimum of $\textup {\texttt {D}}_{\mathrm {tmp}}[ \textup {\texttt {D}}_{\mathrm {msc}}, {\hat {t}}/t]$.

After $\textup {\texttt {D}}_{\mathrm{cr}}[ \textup {\texttt {D}}_{\mathrm {msc}}, {\hat {t}}/t]$, as given by Proposition 9, is substituted into the right-hand side of (24), we can use numerical methods to find its minimum. One discovers that the minimum value of D _tmp=5.49370 is obtained at D _msc=1.25453 and ${\hat {t}}/t = 2.59169$. The claimed relation between ℓ and t follows from (23). The final claim concerning the precomputation cost is obtained by combining Proposition 4 with the first two relations stated by the claim. □

The parameter set that achieves the minimum tradeoff coefficient for the DP tradeoff is visible through Fig. 2. It plots $\textup {\texttt {D}}_{\mathrm {tmp}}= \frac{ \textup {\texttt {D}}_{\mathrm {tc}}}{ \textup {\texttt {D}}_{\mathrm {ps}}\{\ln(1- \textup {\texttt {D}}_{\mathrm {ps}})\}^{2}}$, which is given by (24), as a function of the variables D _msc and ${\hat {t}}/t$.

Fig. 2.

Tradeoff coefficient for DP tradeoff at fixed probability of success ($\textup {\texttt {D}}_{\mathrm {tmp}}= \frac{ \textup {\texttt {D}}_{\mathrm {tc}}}{ \textup {\texttt {D}}_{\mathrm {ps}}\cdot\{\ln(1- \textup {\texttt {D}}_{\mathrm {ps}})\}^{2}}$).

The tradeoff curve reflected by this proposition allows us to say more about the tradeoff than the previously known rough curve (8). Suppose that, for some fixed set of parameters, the success rate of the DP tradeoff is not too small, and suppose that one wishes to increase the success rate, to the extent that the failure rate becomes the square of its current value. Then, for an optimal choice of parameters, the D _ps factor will change little and the {ln(1−D _ps)}² factor will increase by a factor of four. Hence, one must allow an increase in the online time by a factor of four or use twice the current storage. The proposition also shows that one must endure twice the precomputation cost to achieve this aim. Of course, the simplest way of doing this would be to double the number of tables, while keeping all other parameters the same.

While the above result gives the parameters that achieve the optimal tradeoff efficiency, in practical applications, precomputation is very costly and one is more likely to choose a sufficiently large ${\hat {t}}$, so as not to discard any of the precomputed results.

Proposition 36

Let 0<D _ps<1 be any fixed value. When the use of a sufficiently large  ${\hat {t}}$ is assumed, the DP tradeoff, under any set of parameters m, t, and ℓ, that are subject to the relations

$$ mt^2 = 0.562047 {\textup {\textsf {N}}}\quad\text{\textit{and}}\quad \ell= 2.18614 \bigl\{ -\ln({1- \textup {\texttt {D}}_{\mathrm {ps}}}) \bigr\} t, $$

attains the given value  D _ps as its probability of success, and exhibits a tradeoff performance corresponding to

$$ \textup {\texttt {D}}_{\mathrm {tc}}= 7.01057 \textup {\texttt {D}}_{\mathrm {ps}}\bigl\{\ln(1- \textup {\texttt {D}}_{\mathrm {ps}}) \bigr\}^2, $$

as the three parameters are varied. Under any such choice of parameters, the number of one-way function invocations required for the precomputation phase is

$$ \textup {\texttt {D}}_{\mathrm {pc}}{\textup {\textsf {N}}}= 1.22871 \bigl\{ -\ln({1- \textup {\texttt {D}}_{\mathrm {ps}}}) \bigr\} {\textup {\textsf {N}}}. $$

The two relations restricting the parameter choices give optimal parameters in the sense that, when ${\hat {t}}$ is sufficiently large, no choice of m, t, and ℓ can lead to a tradeoff coefficient smaller than the above while achieving  D _ps as its probability of success.

Proof

The proof is almost identical to that of Proposition 35. The only difference is that we rely on Proposition 10 to view D _cr as a function of D _msc and obtain the tradeoff coefficient from Corollary 14, so that

$$ \textup {\texttt {D}}_{\mathrm {tc}}= \biggl(2+\frac{1}{ \textup {\texttt {D}}_{\mathrm {msc}}} \biggr) \biggl( \frac{\sqrt{1 + 2 \textup {\texttt {D}}_{\mathrm {msc}}}+1}{2} \biggr)^3 \textup {\texttt {D}}_{\mathrm {ps}}\bigl\{\ln(1- \textup {\texttt {D}}_{\mathrm {ps}}) \bigr \}^2. $$

(25)

It suffices to minimize

$$ \textup {\texttt {D}}_{\mathrm {tmp}}[ \textup {\texttt {D}}_{\mathrm {msc}}] = \frac{ \textup {\texttt {D}}_{\mathrm {tc}}}{ \textup {\texttt {D}}_{\mathrm {ps}}\{\ln(1- \textup {\texttt {D}}_{\mathrm {ps}})\}^2} = \biggl(2+\frac{1}{ \textup {\texttt {D}}_{\mathrm {msc}}} \biggr) \biggl(\frac{\sqrt{1 + 2 \textup {\texttt {D}}_{\mathrm {msc}}}+1}{2} \biggr)^3, $$

which is a function of the single variable D _msc. □

In comparison to the previous optimal set of parameters that utilizes ${\hat {t}}$ as a free variable, this version shows a less efficient tradeoff, but requires less precomputation. The behavior of the DP tradeoff coefficient with sufficiently large ${\hat {t}}$, under a fixed requirement for success rate, is given as the left-hand graph of Fig. 3. The point of minimum tradeoff coefficient is marked, together with the position corresponding to the more commonly used matrix stopping rule of D _msc=1. The advantage of using a smaller matrix stopping constant than usual is clearly visible.

Fig. 3.

Tradeoff coefficients at fixed probability of success for the DP tradeoff with a sufficiently large ${\hat {t}}$ and the Hellman tradeoff.

7.2 Hellman Tradeoff

We now turn to the Hellman tradeoff. This is very similar to the DP tradeoff case that uses a sufficiently large ${\hat {t}}$.

Proposition 37

Let 0<H _ps<1 be any fixed value. The Hellman tradeoff, under any set of parameters m, t, and ℓ, that are subject to the relations

$$ mt^2 = 2.25433 {\textup {\textsf {N}}}\quad\text{\textit{and}}\quad \ell= 0.598941 \bigl\{ -\ln({1- \textup {\texttt {H}}_{\mathrm {ps}}}) \bigr\} t, $$

attains the given  H _ps as its probability of success, and exhibits a tradeoff performance corresponding to

$$ \textup {\texttt {H}}_{\mathrm {tc}}= 1.50217 \textup {\texttt {H}}_{\mathrm {ps}}\bigl\{\ln(1- \textup {\texttt {H}}_{\mathrm {ps}}) \bigr\}^2, $$

as the three parameters are varied. Under any such choice of parameters, the number of one-way function invocations required for the precomputation phase is

$$ \textup {\texttt {H}}_{\mathrm {pc}}{\textup {\textsf {N}}}= 1.35021 \bigl\{ -\ln({1- \textup {\texttt {H}}_{\mathrm {ps}}}) \bigr\} {\textup {\textsf {N}}}. $$

The two relations restricting the parameter choices give optimal parameters in the sense that no choice of m, t, and ℓ can lead to a tradeoff coefficient smaller than the above while achieving H _ps as its probability of success.

Proof

The proof given here shall be concise, since it is similar to those of Propositions 35 and 36. Based on Proposition 20, we may fix $\ell= \frac {1}{\textup {\texttt {H}}_{\mathrm{cr}} \textup {\texttt {H}}_{\mathrm {msc}}} \{-\ln({1- \textup {\texttt {H}}_{\mathrm {ps}}})\} t$. Reference to Proposition 21 shows that the Hellman coverage rate H _cr=H _cr[H _msc] may be seen as a function of $\textup {\texttt {H}}_{\mathrm {msc}}= \frac {mt^{2}}{{\textup {\textsf {N}}}}$. Hence, given any m, t, and H _ps, we can set ℓ to an appropriate value with which the Hellman tradeoff achieves a success probability of H _ps.

We now work to minimize the Hellman tradeoff coefficient. By combining Theorem 22 and Proposition 21, we obtain

$$ \textup {\texttt {H}}_{\mathrm {tc}}= \biggl(\frac{1}{ \textup {\texttt {H}}_{\mathrm {msc}}}+\frac{1}{6} \biggr) \biggl( \frac{\sqrt{ \textup {\texttt {H}}_{\mathrm {msc}}}}{\sqrt{2}} \frac{e^{\sqrt{2 \textup {\texttt {H}}_{\mathrm {msc}}}}+1}{e^{\sqrt{2 \textup {\texttt {H}}_{\mathrm {msc}}}}-1} \biggr)^3 \textup {\texttt {H}}_{\mathrm {ps}}\bigl\{\ln(1- \textup {\texttt {H}}_{\mathrm {ps}}) \bigr\}^2. $$

(26)

For a fixed success probability, it suffices to minimize the part that depends only on the single variable H _msc.

One can use numeric methods to identify the minimum value $\frac{\textup {\texttt {H}}_{\mathrm {tc}}}{\textup {\texttt {H}}_{\mathrm {ps}}\{\ln(1- \textup {\texttt {H}}_{\mathrm {ps}})\}^{2}} = 1.50217$, which is attained at H _msc=2.25433. The two remaining constants appearing in the proposition may now be obtained through appropriate evaluations. □

The most typical Hellman tradeoff, which is set to use mt ²=N and ℓ=t, attains a success probability of 57.68 % and the tradeoff curve TM ²=0.7797N ², when the cost of resolving alarms is taken into account. In comparison, the choice of mt ²=2.2543N and ℓ=0.5160t, suggested by Proposition 37, gives TM ²=0.6409N ², while achieving the same success rate. This improvement in tradeoff efficiency is visible through the right-hand graph of Fig. 3, where the two dots mark the two parameter choices we have just discussed.

The price paid for this better tradeoff efficiency is the increase in precomputation from N to 1.1630N. Indeed, after combining Propositions 20 and 21 into

$$ \textup {\texttt {H}}_{\mathrm {pc}}= \frac{\sqrt{ \textup {\texttt {H}}_{\mathrm {msc}}}}{\sqrt{2}} \frac{e^{\sqrt{2 \textup {\texttt {H}}_{\mathrm {msc}}}}+1}{e^{\sqrt{2 \textup {\texttt {H}}_{\mathrm {msc}}}}-1} \bigl\{ - \ln({1- \textup {\texttt {H}}_{\mathrm {ps}}}) \bigr\}, $$

(27)

one can check that the precomputation H _pc[H _msc] required under any fixed probability of success is an increasing function of H _msc. Hence, while any point that is situated to the left of the minimal point in Fig. 3 may not be optimal in view of tradeoff efficiency, it corresponds to less precomputation. Depending on the available computational resources, one may choose to lower the precomputation cost rather than increase the tradeoff efficiency. On the other hand, increasing H _msc beyond the minimizing value 2.25433 will have bad effects on both the precomputation and the tradeoff efficiency and should be avoided.

Let us briefly return to the DP tradeoff that uses a sufficiently large ${\hat {t}}$. By combining Propositions 5 and 10, we can write

$$ \textup {\texttt {D}}_{\mathrm {pc}}= \frac{\sqrt{1 + 2 \textup {\texttt {D}}_{\mathrm {msc}}}+1}{2} \bigl\{ -\ln({1- \textup {\texttt {D}}_{\mathrm {ps}}}) \bigr\}, $$

(28)

and, as with the Hellman tradeoff, confirm that D _pc is an increasing function of D _msc. Since we know from Proposition 36 that the best performance is achieved at D _msc=0.562047, the choice of D _msc≤0.562047 may be reasonable in view of the lower precomputation cost, but using D _msc>0.562047 should be avoided. In particular, the use of D _msc=1 cannot be justified.

7.3 Rainbow Tradeoff

The analyses of optimal parameters for the DP and Hellman tradeoffs were very similar. However, the rainbow tradeoff does not allow the same approach, because we have less control over the parameter ℓ. The number of tables ℓ used with the DP and Hellman tradeoffs are quite large, and we had treated ℓ as if it were a continuous variable. In the rainbow tradeoff case, the table count is usually a small integer, and we must keep in mind that it takes only discrete values.

Let us start with a fixed number of tables ℓ. For any given requirement on the success rate, we can rewrite Proposition 29 as

$$ \textup {\texttt {R}}_{\mathrm {msc}}= 2 \bigl\{({1 - \textup {\texttt {R}}_{\mathrm {ps}}})^{-1/2\ell} - 1 \bigr\} $$

(29)

and understand this as a lower bound on R _msc that can be used with ℓ to achieve R _ps. It is clear that increasing R _msc under a fixed ℓ will increase the precomputation cost R _msc ℓ N. One can also work with the tradeoff coefficient R _tc, as provided by Theorem 30, to confirm that increasing R _msc under a fixed ℓ will reduce the tradeoff efficiency. Hence, under any fixed ℓ, the exact value of R _msc, suggested by (29), should be used to achieve the required success rate.

We can now treat R _msc as a function of the success rate requirement R _ps, for any fixed ℓ. After substituting R _msc, as given by (29), into the tradeoff coefficient of Theorem 30, one can rewrite it as

(30)

For each fixed ℓ, this is a function of the single variable R _ps. A plot of this is given as Fig. 4 for table counts ℓ=1, 2, and 3. The right-hand box is a magnified partial view of the left-hand box in logarithmic scale.

Fig. 4.

Tradeoff coefficient of rainbow tradeoff as a function of success rate requirement at small number of tables (ℓ=1: dotted, ℓ=2: dashed, ℓ=3: solid).

Recalling that a smaller tradeoff coefficient implies better tradeoff efficiency, one can clearly read from the figure that the use of ℓ=1 is optimal when the requirement for success rate is very low and that the use of successively higher numbers of tables becomes optimal as the success rate requirement is made more stringent. We have numerically solved for the explicit probabilities at which the transition to the next table count should be made and have recorded this in Table 1.

Table 1. Range of success probability requirements for which each table count ℓ is optimal.

Let us briefly explain the content of the table with examples. Suppose one aims to achieve a success probability of 99.9 % with the rainbow tradeoff. Since 0.999 sits between 0.998775 and 0.999314, it is optimal to use ten tables. If one is requested to set the probability of failure to $\frac{1}{2^{7}}$, we locate −7 between −6.17353 and −7.08171 and conclude that six tables would be optimal. To understand the other three columns of the table, let us focus on the row that sits between ℓ=1 and ℓ=2. The use of a single table with R _msc=1.87905, or the use of two tables at R _msc=0.785335 will both result in an optimal tradeoff coefficient of R _tc=1.48026=2^0.565848 and a success rate of 73.4166 %.

Note that any given success rate requirement R _ps makes a certain number of tables ℓ as optimal, and the ℓ value fixes R _msc through (29). Since the tradeoff coefficient of Theorem 30 is already determined by ℓ and R _msc, and since the relation (29) guarantees the R _ps success rate, any parameter set satisfying the mentioned restriction will be optimal in view of the tradeoff coefficient. Let us gather what we have discussed in a proposition.

Proposition 38

Let 0<R _ps<1 be any given fixed value. Locate the table count ℓ from Table  1 that corresponds to the given  R _ps and compute

$$ \textup {\texttt {R}}_{\mathrm {msc}}= 2 \bigl\{({1 - \textup {\texttt {R}}_{\mathrm {ps}}})^{-1/(2\ell)} - 1 \bigr\}. $$

Then the rainbow tradeoff that uses the located ℓ and any parameters m and t satisfying the relation

$$ mt = \textup {\texttt {R}}_{\mathrm {msc}}{\textup {\textsf {N}}}$$

attains the given value  R _ps as its probability of success. The tradeoff performance corresponding to

can be observed as m and t are varied under the restriction. With any such choice of parameters, the number of one-way function invocations required for the precomputation phase is

$$ \textup {\texttt {R}}_{\mathrm {pc}}{\textup {\textsf {N}}}= \textup {\texttt {R}}_{\mathrm {msc}}\ell {\textup {\textsf {N}}}. $$

The choice of ℓ through Table  1 and the single relation concerning m and t lead to optimal parameters in the sense that no choice of m, t, and ℓ can result in a tradeoff coefficient smaller than the above while achieving  R _ps as its probability of success.

To be strictly logical, one must also consider the possibility that allowing the multiple tables to be of different sizes may lead to better tradeoff coefficients. The case of three tables with the most general table sizes is analyzed in [21], and the conclusion is made that optimal tradeoff performance is achieved at equal sized tables. The method used can probably be extended to larger numbers of tables, but the required computations will be much more complicated than the computations done in this work. Since the examination of the three-table case showed that we are not likely to gain anything from the more general analysis, we chose to work with equal sized tables. However, for the case of perfect rainbow tables, we have reasons to believe that this extra flexibility will bring about better tradeoff performance.

Finally, we want to provide an argument that is analogous to what was discussed at the end of Sect. 7.2. One can check that

$$ \textup {\texttt {R}}_{\mathrm {pc}}= \textup {\texttt {R}}_{\mathrm {msc}}\ell= 2 \ell \bigl\{ ({1 - \textup {\texttt {R}}_{\mathrm {ps}}})^{-1/(2\ell)} - 1 \bigr\} $$

(31)

is a decreasing function of ℓ, for each fixed R _ps. Hence, use of an ℓ count that is larger than what is suggested by Table 1 will decrease the precomputation requirement at the cost of reduced tradeoff efficiency. This may be preferable in some situations. On the other hand, use of an ℓ count that is smaller than the optimal count will have bad effects on both the precomputation cost and tradeoff efficiency, and should be avoided.

8 Comparison of Tradeoff Performances

All the ingredients required for a fair comparison of performances between the tradeoff algorithms are now ready. Any discussion of the DP tradeoff in this section assumes that the chain length bound ${\hat {t}}$ is sufficiently large.

8.1 Conversion of the Tradeoff Coefficients to a Common Unit

It is clear that for any comparison of tradeoff algorithms to be fair, the algorithms must be made to present the same probability of success. One must also consider the precomputation cost required by each algorithm, and this aspect will be considered later on in this section. For now, we focus on the fact that the tradeoff coefficient is a measure of tradeoff efficiency. Let us assume that the DP, Hellman, and rainbow tradeoff algorithms display the respective tradeoff curves

$$ T_\textup {\texttt {D}}M_\textup {\texttt {D}}^2 = \textup {\texttt {D}}_{\mathrm {tc}}{\textup {\textsf {N}}}^2, \qquad T_\textup {\texttt {H}}M_\textup {\texttt {H}}^2 = \textup {\texttt {H}}_{\mathrm {tc}}{\textup {\textsf {N}}}^2, \quad\text{and}\quad T_\textup {\texttt {R}}M_\textup {\texttt {R}}^2 = \textup {\texttt {R}}_{\mathrm {tc}}{\textup {\textsf {N}}}^2, $$

(32)

at the same success rate. We will discuss how to interpret the ratio D _tc:H _tc:R _tc of the tradeoff coefficients as a ratio of tradeoff efficiencies.

8.1.1 Unit for Storage

Let us first consider the storage variable M. For the moment, we will disregard any issues concerning the time unit.

In all three tradeoff algorithms, M represents the number of starting point and ending point pairs that need to be stored, but the actual number of bits required to store each table entry will be different among the tradeoff algorithms. We saw through Propositions 17, 25, and 33 that the number of bits required to store each table entry is as follows for each tradeoff algorithm.

Let us assume from this point on that the ending point truncations for the three algorithms were done in such a way that their effects on the online time are minimal. In particular, we assume that the contents of Corollary 14, Theorems 22 and 30 remain valid after ending point truncation. We further assume that the slightly more bits mentioned above can be ignored.

A fair comparison of tradeoff performances would express storages for the three algorithms in terms of number of bits that are required for the precomputation tables rather than the number of starting point and ending point pairs. Under the two assumptions made, one is led to focus on the ratio

$$ (\log m_\textup {\texttt {D}})^2 \textup {\texttt {D}}_{\mathrm {tc}}: (\log m_\textup {\texttt {H}}+ \log t_\textup {\texttt {H}})^2 \textup {\texttt {H}}_{\mathrm {tc}}: (\log m_\textup {\texttt {R}})^2 \textup {\texttt {R}}_{\mathrm {tc}}, $$

(33)

rather than the raw tradeoff coefficient ratio D _tc:H _tc:R _tc. The bit sizes per entry are multiplied in squares because any change in storage affects the tradeoff efficiency through a square factor.

The implementation environment and tradeoff requirements will place the choice of suitable parameters into a certain range, and it is reasonable to assume that the parameters that would be chosen for each algorithm would be related through

$$ \log t_\textup {\texttt {D}}\approx\log t_\textup {\texttt {H}}\approx \log t_\textup {\texttt {R}}, \qquad \log m_\textup {\texttt {D}}\approx\log m_\textup {\texttt {H}}, \quad \text{and}\quad \log m_\textup {\texttt {R}}\approx\log m_\textup {\texttt {H}}+ \log t_\textup {\texttt {H}}. $$

(34)

Some readers may object that our discussion on the number of bits required for each table entry makes m _D =2m _H more reasonable than m _D =m _H , but this difference by a factor of two is lost in the approximations when they are converted bit sizes, as is done in the expression (34).

Assuming the rough correspondence (34) between parameters, the ratio (33) simplifies to

$$ \biggl(\frac{\log m_\textup {\texttt {D}}}{\log m_\textup {\texttt {R}}} \biggr)^2\textup {\texttt {D}}_{\mathrm {tc}}: \textup {\texttt {H}}_{\mathrm {tc}}: \textup {\texttt {R}}_{\mathrm {tc}}. $$

(35)

When issues concerning time units are ignored, this is the correct ratio to focus on when comparing the tradeoff efficiencies of different algorithms.

8.1.2 Unit for Online Time

Unification of the time unit T is now considered. Issues concerning the storage unit, which we have already discussed, are ignored for the moment.

Recall that the time variable T used in the tradeoff curves counts the number of one-way function iterations and ignores the table lookups. Hence, parameter sets which lead to identical times T _D =T _H =T _R do not guarantee that the simultaneous executions of the three algorithms will finish at the same time. For a fair interpretation of a tradeoff coefficient ratio as a ratio of tradeoff efficiency, the difference in the time units used by the algorithms must be taken into account.

It is reasonable to expect the time taken for a single one-way function iteration by the three algorithms to be quite similar. Let us fix the notation and express this common time length as |Itr|. We also fix the notation |TL-D|, |TL-H|, and |TL-R| for the time required for lookups to the DP, Hellman, and rainbow tables, respectively. Depending on the implementation platform, it is possible to experience |TL-D|≈|TL-H|≪|TL-R|, even when equal sized storages are allocated to the three algorithms, since the DP or Hellman tradeoffs utilize a large number of small tables, whereas the rainbow tradeoff uses a small number of large tables.

Referencing Lemma 15, the real-world time required to process the online phase of a DP tradeoff can be written as $T_{\textup {\texttt {D}}}|\textup {Itr}|+ t_{\textup {\texttt {D}}}\frac{ \textup {\texttt {D}}_{\mathrm {ps}}}{\textup {\texttt {D}}_{\mathrm{cr}} \textup {\texttt {D}}_{\mathrm {msc}}} |\textup {TL-}\textup {\texttt {D}}|$. Since we know from (18) that $T_{\textup {\texttt {D}}}= t_{\textup {\texttt {D}}}^{2} \frac { \textup {\texttt {D}}_{\mathrm {ps}}}{\textup {\texttt {D}}_{\mathrm{cr}} \textup {\texttt {D}}_{\mathrm {msc}}}(1+2 \textup {\texttt {D}}_{\mathrm {msc}})$, the real-world online time for DP tradeoff can be expressed as

$$ \biggl( 1 + \frac{1}{1+2 \textup {\texttt {D}}_{\mathrm {msc}}} \frac{|\textup {TL-}\textup {\texttt {D}}|}{t_\textup {\texttt {D}}|\textup {Itr}|} \biggr) T_\textup {\texttt {D}}|\textup {Itr}|. $$

(36)

Similarly, gathering information from (21) and Lemma 23, the real-world execution time for the Hellman online phase can be written as

$$ \biggl( 1+ \frac{6}{6+ \textup {\texttt {H}}_{\mathrm {msc}}} \frac{|\textup {TL-}\textup {\texttt {H}}|}{|\textup {Itr}|} \biggr) T_\textup {\texttt {H}}|\textup {Itr}|. $$

(37)

The corresponding expression for the rainbow tradeoff, relying on (22) and Lemma 31, is given by

$$ \biggl( 1 + \textup {\texttt {R}}_{\mathrm{tmp}}[\ell, \textup {\texttt {R}}_{\mathrm {ps}}] \frac{|\textup {TL-}\textup {\texttt {R}}|}{t_\textup {\texttt {R}}|\textup {Itr}|} \biggr) T_\textup {\texttt {R}}|\textup {Itr}|, $$

(38)

where

is of Θ(1) order. We have used (29) to remove all occurrences of R _msc in the expression, because our graphs for each fixed R _ps in the later part of this section are drawn using ℓ as a parameter.

The three equations (36), (37), and (38) can be used to easily find the correct way to compare tradeoff coefficients. For example, consider the simplest case where all table lookups are negligible, i.e., when |TL-D|,|TL-H|,|TL-R|,≪|Itr|. Then, all the second terms in the three equations are negligible. Hence, the raw coefficient ratio D _tc:H _tc:R _tc reflects the true tradeoff efficiency ratio of the three algorithms.

Let us next consider the case where |Itr|≪|TL-D|≈|TL-H|≤|TL-R|≪t _D |Itr|≈t _R |Itr|. This might be the situation experienced by a large implementation that requires disk accesses for table lookups. The probable use of large t _D and t _R partially justifies the third inequality. In this case, the second term of (37) dominates all other five terms of the three equations. The Hellman tradeoff clearly cannot compete with the other two algorithms, and the comparison between the DP and rainbow tradeoffs can fairly be done with D _tc:R _tc.

The final example we consider is when |Itr|≈|TL-D|≈|TL-H|≤|TL-R|≪t _D |Itr|≈t _R |Itr|. Then neither of the two terms of (37) dominates the other and neither can be ignored. The appropriate ratio to study when comparing tradeoff algorithms would be

$$ \textup {\texttt {D}}_{\mathrm {tc}}: \biggl(1+\frac{6}{6+ \textup {\texttt {H}}_{\mathrm {msc}}}\frac{|\textup {TL-}\textup {\texttt {H}}|}{|\textup {Itr}|} \biggr) \textup {\texttt {H}}_{\mathrm {tc}}: \textup {\texttt {R}}_{\mathrm {tc}}. $$

(39)

There are many other cases to consider, but the correct way to adjust the tradeoff coefficients so that they reflect the tradeoff efficiency ratio of the tradeoff algorithms can easily be found from (36), (37), and (38).

This ends our discussion on the unit of time, but let us briefly digress and discuss the exceptional situation of |TL-R|≫t _R |Itr| for the rainbow tradeoff. This could happen when the precomputation tables must be reached over the Internet during the online phase. Then, table lookups dominate the online phase, and we can combine T _R =Θ(tℓ), M _R =Θ(mℓ), and ℓ=Θ(1) to conclude that T _R M _R ∝N. At first thought, this might seem to be a much better tradeoff curve than the usual TM ²∝N ² curve.

The counterintuitive conclusion hides the fact that the unit of time T _R is now |TL-R|, rather than |Itr|. Furthermore, unless N is small, the assumption |TL-R|≫t _R |Itr| cannot continue to hold as t _R is increased, so the tradeoff curve will eventually return to the usual TM ²∝N ² after a certain point. The tradeoff curve T _R M _R ∝N remains valid when t _R is moved in the decreasing direction, but having T _R M _R constant is worse than having $T_{\textup {\texttt {R}}}M_{\textup {\texttt {R}}}^{2}$ constant in that direction.

Similar arguments may be made for the DP tradeoff, but lookups to DP tables over a slow network are even less likely to be seen than with the rainbow tradeoffs. Since each individual DP table is rather small, each could be stored on the node that computes the online chain corresponding to that table.

8.1.3 Combined Unit Conversion

The storage unit conversion and the time unit conversion are orthogonal, and the two conversions may simply be multiplied to give modified tradeoff coefficients that are appropriate for comparisons of different tradeoff algorithms. For example, under the reasonable assumption (34), we know that the storage conversion must follow (35). If the one-way function computation and table lookup speeds satisfy |Itr|≈|TL-D|≈|TL-H|≤|TL-R|≪t _D |Itr|≈t _R |Itr|, the time unit conversion must follow (39). Combing the two, we know that comparisons of tradeoff algorithms must focus on

$$ \biggl(\frac{\log m_\textup {\texttt {D}}}{\log m_\textup {\texttt {R}}} \biggr)^2\textup {\texttt {D}}_{\mathrm {tc}}: \biggl(1+ \frac{6}{6+ \textup {\texttt {H}}_{\mathrm {msc}}}\frac{|\textup {TL-}\textup {\texttt {H}}|}{|\textup {Itr}|} \biggr) \textup {\texttt {H}}_{\mathrm {tc}}: \textup {\texttt {R}}_{\mathrm {tc}}, $$

under the stated circumstances.

In our further discussions below, we will mainly restrict ourselves to parameter sets that roughly satisfy

$$ \log m_\textup {\texttt {D}}\approx\log m_\textup {\texttt {H}}\approx \log t_\textup {\texttt {D}}\approx\log t_\textup {\texttt {H}}\approx \log t_\textup {\texttt {R}}\approx\frac{1}{3}\log {\textup {\textsf {N}}}\quad\text{and}\quad \log m_\textup {\texttt {R}}\approx\frac{2}{3}\log {\textup {\textsf {N}}}$$

and mostly assume that the time required for a single table lookup is negligible in comparison to that required for a single one-way function computation. Under these assumptions, the ratio that needs to be studied when comparing tradeoff efficiencies is

$$ \frac{1}{4}\textup {\texttt {D}}_{\mathrm {tc}}: \textup {\texttt {H}}_{\mathrm {tc}}: \textup {\texttt {R}}_{\mathrm {tc}}. $$

(40)

We shall refer to the situation that has just been described as the typical situation, as it often appears during theoretic developments of the tradeoff technique. However, we do not claim this to be typical in practical applications of the tradeoff technique.

We emphasize that our further discussions given below concerning tradeoff performance comparisons will only be valid under the typical situation assumption. If the environment and tradeoff performance requirements make parameter choices such that $\log m_{\textup {\texttt {D}}}\not\approx \log t_{\textup {\texttt {D}}}$ is more appropriate, or if the table lookup delays cannot be ignored, the algorithm comparison conclusions will be different. Still, one will be able to use the information explained in this subsection to easily make the proper adjustments.

Even for the typical situation, the ratio (40) can be made more accurate for each explicit situation. Based on Examples 18, 26, and 34, we can state that

$$ 28^2 \frac{9}{8} \textup {\texttt {D}}_{\mathrm {tc}}: 55^2 \frac{17}{16} \textup {\texttt {H}}_{\mathrm {tc}}: 55^2 \frac{33}{32} \textup {\texttt {R}}_{\mathrm {tc}}= 1.00 \textup {\texttt {D}}_{\mathrm {tc}}: 3.64 \textup {\texttt {H}}_{\mathrm {tc}}: 3.54 \textup {\texttt {R}}_{\mathrm {tc}}, $$

is a more accurate version of (40), for the typical situation with N=2⁷⁵. This new ratio does not ignore the extra one-way function invocations caused by ending point truncations and does not ignore the slightly more bits discussed at the start of Sect. 8.1.1.

8.2 DP Tradeoff Versus Hellman Tradeoff

As discussed in the previous subsection, it suffices to compare $\frac {1}{4} \textup {\texttt {D}}_{\mathrm {tc}}$ against H _tc for a fair comparison between the DP and Hellman tradeoffs. We are assuming the typical situation explained at the end of the previous subsection; any conclusions we make could be different under different circumstances. The precomputation effort is finally considered during tradeoff comparison in this section.

Propositions 36 and 37 show that the optimal tradeoff efficiencies of the two algorithms are given by

$$ \frac{1}{4} \textup {\texttt {D}}_{\mathrm {tc}}= 1.75264 \textup {\texttt {D}}_{\mathrm {ps}}\bigl\{\ln(1- \textup {\texttt {D}}_{\mathrm {ps}}) \bigr\}^2 \quad\text{and}\quad \textup {\texttt {H}}_{\mathrm {tc}}= 1.50217 \textup {\texttt {H}}_{\mathrm {ps}}\bigl\{\ln(1- \textup {\texttt {H}}_{\mathrm {ps}}) \bigr \}^2. $$

One may want to conclude that the Hellman tradeoff, with the smaller tradeoff coefficient, is more efficient, but this is acceptable only when the precomputation cost can be totally ignored. In practice, precomputation cost is the largest barrier to any large-scale deployment of tradeoff algorithms and is hard to ignore.

The precomputation costs required to achieve the above tradeoff efficiencies are

$$ \textup {\texttt {D}}_{\mathrm {pc}}= 1.22871 \bigl\{ -\ln({1- \textup {\texttt {D}}_{\mathrm {ps}}}) \bigr\} \quad\text{and}\quad \textup {\texttt {H}}_{\mathrm {pc}}= 1.35021 \bigl\{ -\ln({1- \textup {\texttt {H}}_{\mathrm {ps}}}) \bigr\}. $$

The precomputation cost of the DP tradeoff is lower and we are faced with the problem of comparing high efficiency at high cost against low efficiency at low cost.

After a moment of thought, one must admit that such a comparison cannot be done in an objective manner. The comparison must reflect how valuable tradeoff efficiency is to the user and how willing one is to invest more time and resources into the precomputation phase. There is no unit with which to express either of these unquantifiable values. Furthermore, one must also question whether it is reasonable to compare the two tradeoffs at parameters giving their respective optimal tradeoff efficiencies. Non-optimal parameters may be preferable under many situations in view of lower precomputation cost.

We can conclude that all we can do is present the range of choices that can be made with each algorithm and allow the users to make their conclusions based on their explicit circumstances. The crucial information that must be displayed to allow easy judgement of which tradeoff is more suitable is the relation between tradeoff efficiency and precomputation cost. This must be done at each fixed requirement for the inversion success rate.

As was previously noted through (28) and (27), when under a fixed probability of success requirement, both D _pc and H _pc are functions of their respective D _msc and H _msc values. The tradeoff coefficients D _tc and H _tc, under a fixed success rate requirement, were similarly expressed as functions of the corresponding D _msc and H _msc values in (25) and (26).

For a comparison of the DP tradeoff against the Hellman tradeoff, it now suffices to present the graphs

$$ \biggl\{\biggl( \textup {\texttt {D}}_{\mathrm {pc}}[ \textup {\texttt {D}}_{\mathrm {msc}}] ,\frac{1}{4} \textup {\texttt {D}}_{\mathrm {tc}}[ \textup {\texttt {D}}_{\mathrm {msc}}] \biggr) \biggm{|} \textup {\texttt {D}}_{\mathrm {msc}}\leq 0.562047 \biggr\} $$

(41)

and

$$ \bigl\{\bigl( \textup {\texttt {H}}_{\mathrm {pc}}[ \textup {\texttt {H}}_{\mathrm {msc}}] , \textup {\texttt {H}}_{\mathrm {tc}}[ \textup {\texttt {H}}_{\mathrm {msc}}]\bigr) \bigm{|}\textup {\texttt {H}}_{\mathrm {msc}}\leq2.25433 \bigr\}, $$

(42)

where the bounds on D _msc and H _msc were placed in accordance with the discussion at the end of Sect. 7.2. These graphs are given in Fig. 5. Since the two graphs are to be compared at identical success rate requirements D _ps=H _ps, we have removed the common parts that depend on the success probability from both of the cases before plotting the graphs. Hence, the graphs do not depend on the success rate and are valid for all success rate requirements. Both graphs extend further upward, but the right ends, corresponding to the optimal tradeoff performances, are clearly marked with dots.

Fig. 5.

The tradeoff coefficients $\frac{1}{4} \textup {\texttt {D}}_{\mathrm {tc}}$ (dotted) and H _tc (dashed) in relation to their respective precomputation cost.

The two graphs are very close to each other. Even though slightly better tradeoff efficiency can be obtained with the Hellman tradeoff at higher precomputation cost, in practice, unless parameters far from the typical m≈t≈N ^1/3 region are to be used, the DP tradeoff will be favored in view of fewer table lookups. For example, if the table lookup time makes $\frac{1}{5} \textup {\texttt {D}}_{\mathrm {tc}}: \textup {\texttt {H}}_{\mathrm {tc}}$ a more appropriate measure of tradeoff performance ratio than the current $\frac{1}{4} \textup {\texttt {D}}_{\mathrm {tc}}: \textup {\texttt {H}}_{\mathrm {tc}}$, the dotted curve for the DP tradeoff would move down and present itself as a more advantageous algorithm.

If table lookup time is absolutely negligible in comparison to the one-way function computation time, there is a short range of parameter sets with which the Hellman tradeoff can slightly outperform the DP tradeoff using the same amount of precomputation. If table lookup time is negligible and precomputation is not to be considered, the Hellman tradeoff can be slightly better.

8.3 Rainbow Tradeoff Versus DP and Hellman Tradeoffs

We now include the rainbow tradeoff into the comparison graphs. As was discussed in Sect. 8.1, we assume the typical situation concerning the approximate range of parameters and table lookup time, and consider comparisons between $\frac{1}{4} \textup {\texttt {D}}_{\mathrm {tc}}$, H _tc, and R _tc to be fair.

In addition to the graphs (41) and (42), we need to plot all possible (R _pc,R _tc) points. We can first check through (31) that R _pc can be seen as a function of the table count ℓ, when the success rate requirement R _ps is fixed. As for the tradeoff coefficient, equation (30) presents it as a function of just ℓ, when R _ps is fixed. Given any requirement on the success rate R _ps, it is now possible to draw the graph

$$ \bigl\{\bigl( \textup {\texttt {R}}_{\mathrm {pc}}[\ell] , \textup {\texttt {R}}_{\mathrm {tc}}[\ell]\bigr) \bigm{|}\ell\geq \text{optimal table count for $ \textup {\texttt {R}}_{\mathrm {ps}}$} \bigr\}, $$

(43)

where the optimal table count can be obtained from Table 1. Note that this is no longer a continuous graph, but a discrete set of points. In the strict sense, the previous graphs for the DP and Hellman tradeoffs were also discrete sets of points, but unless N is very small, the points are extremely close to each other.

Unlike our comparison between DP and Hellman tradeoffs, the parts that depend on R _ps appearing in the expressions (31) and (30) are not identical to those appearing in the corresponding expressions (28), (27), (25), and (26). Hence, separate graphs need to be drawn for each success rate. This is given in Fig. 6 for some success rates.

Fig. 6.

Tradeoff coefficients $\frac{1}{4} \textup {\texttt {D}}_{\mathrm {tc}}$ (dotted), H _tc (dashed), and R _tc (large dots) in relation to their respective precomputation cost at success rates 25 %, 50 %, 75 %, 90 %, 95 %, and 99 % (X-axis: D _pc, H _pc, and R _pc; Y-axis: $\frac{1}{4} \textup {\texttt {D}}_{\mathrm {tc}}$, H _tc, and R _tc).

In all of the graphs, one can see that the curve for the rainbow tradeoff sits closer to the origin than the curves for the DP and Hellman tradeoffs. Note that a graph sitting lower shows better tradeoff efficiency and being positioned more to the left implies lower precomputation cost. In all the cases except for the ones corresponding to 25 % and 50 % success rates, given any position on the curve for either the DP or Hellman tradeoff there is a rainbow tradeoff position that presents better tradeoff performance at a smaller precomputation cost. Use of the rainbow tradeoff is definitely advisable in these cases.

The existence of better rainbow position is also mostly true in the 50 % case. The exception is marked with an ⊗ on the curve for the Hellman tradeoff. This position is very slightly to the left of the optimal rainbow position and hence corresponds to less precomputation than the optimal rainbow position. At the same time, it is positioned lower than the second best rainbow position and hence shows better tradeoff efficiency than this second best position. Hence, there can be no rainbow tradeoff parameter set that can replace the Hellman position marked with an ⊗ without at least very slightly sacrificing either the precomputation cost or the tradeoff efficiency. Still, anybody will agree that this exception is quite unreasonable, and one would normally choose to sacrifice the extremely small amount of either the precomputation cost or the tradeoff performance for a visibly better value of the other factor.

The 25 % case also displays the rainbow tradeoff requiring less precomputation than the other two tradeoffs in achieving equal tradeoff efficiency, but the awkward exceptional position discussed for the 50 % case can be found here as rather large segments. In addition, the best tradeoff efficiency achievable by the rainbow tradeoff falls short of what is reachable by the other two algorithms. Hence there will be situations where the DP or Hellman tradeoff is preferable over the rainbow tradeoff, when required to achieve a 25 % success rate.

The relative advantage of using the rainbow tradeoff is clearly seen to grow with the increase in the success rate requirement. For the 99 % success rate case, it seems almost safe to say that the rainbow tradeoff performs approximately two times better than the other two tradeoff algorithms in any of their reasonable usages.

In conclusion, the use of the rainbow tradeoff is advisable for high success rate requirements, and there may occasionally be low success rate applications with special situations where the other two tradeoffs are preferable. We emphasize once more that this conclusion is only valid under the typical situation assumption explained in Sect. 8.1. For example, if we must work with parameters such that 2logm _D ≈logt _D and 2logm _H ≈logt _H and table lookups are negligible, then comparison of the coefficients $\frac{1}{9} \textup {\texttt {D}}_{\mathrm {tc}}$, H _tc, and R _tc would be appropriate. This would bring the curve for the DP tradeoff lower and we would arrive at a different conclusion.

8.4 Revisit to the Preliminary Tradeoff Comparison

In Sect. 2.9, we recalled how [23] claimed the rainbow tradeoff to be more efficient than the DP tradeoff by a factor of two. We also explained how [3, 4] pointed out that the two algorithms require different numbers of bits to represent each table entry and argued that the DP tradeoff was twice as efficient as the rainbow tradeoff. Since our conclusions of Sect. 8.3 are once again supportive of the rainbow tradeoff, let us explain where in the arguments of [3, 4] the inaccuracies were introduced. Details of the current paper, including the proofs, need to be understood if the computations of this section are to be followed.

According to Propositions 5, 10, and Corollary 14, the DP tradeoff performance at parameters m=t=ℓ=N ^1/3 and a sufficiently large chain length bound is given by

$$ \textup {\texttt {D}}_{\mathrm {ps}}= 51.9~\%, \qquad \textup {\texttt {D}}_{\mathrm {tc}}= 2.13, \qquad \textup {\texttt {D}}_{\mathrm {pc}}= 1. $$

(44)

In comparison, Proposition 29 and Theorem 30 allow us to state that the rainbow tradeoff at the naturally corresponding parameters m=N ^2/3, t=N ^1/3, and ℓ=1 shows the performance

$$ \textup {\texttt {R}}_{\mathrm {ps}}= 55.6~\%, \qquad \textup {\texttt {R}}_{\mathrm {tc}}= 0.422, \qquad \textup {\texttt {R}}_{\mathrm {pc}}= 1. $$

(45)

As claimed in [3, 4] and confirmed in Sect. 8, we must apply an adjustment factor to compensate for the difference in bits required per table entry before comparing these two sets of figures. Comparing $\frac{1}{4} \textup {\texttt {D}}_{\mathrm {tc}}= 0.532$ against R _tc=0.422, we can conclude that, for the same amount of physical storage, the rainbow tradeoff is both faster and succeeds more often than the DP tradeoff. This disagrees with the claim of [3, 4] and does not go against our conclusion, which stated that the rainbow tradeoff is slightly better than the DP tradeoff at low success rates.

The main argument of [3, 4] that the number of bits required to store each entry of the rainbow tradeoff is twice that required for the DP tradeoff was certainly correct. The primary source of their incorrect conclusion is the inaccurate estimations of running time complexities for the two algorithms. The tradeoff coefficient for the DP was estimated at 1, but in reality, it was a much larger D _tc=2.13.

After understanding the details of the proof to Theorem 13, one can compute that, out of the value 2.13, the part that corresponds to the online chain computation is only 0.709. This is smaller than 1, the estimate of [3, 4], but the remaining 1.42, which is due to the resolving of alarms, was much larger. In the case of the rainbow tradeoff, the tradeoff coefficient was estimated at 0.5 by [3, 4], and the actual value R _tc=0.422 was smaller. Details of the proof to Theorem 30 show that, out of the 0.422, the cost of online chain creation corresponds to a mere 0.306 and the cost of resolving alarms corresponds to an even smaller 0.117.

The true online chain creation efforts for the two algorithms being smaller than the initial rough estimates is a consequence of the algorithms terminating prematurely with the discovery of the correct answer, and the upper bounds for the cost of online chain creation given by the preliminary analysis [3, 4] were correct. Since $\frac{1}{4}\times0.709$ is less than 0.306, a comparison of the two algorithms based only on the online chain creation time would have concluded that the DP tradeoff was superior. In fact, the ratio $\frac {0.709/4}{0.306} \approx0.579$ is somewhat in agreement with the performance ratio of two that was claimed by [3, 4], based on their rough upper bounds. However, when the costs of resolving alarms were taken into account, the conclusions were quite the opposite. This is a clear indication that a careful analysis of the cost associated with resolving of alarms was necessary for a fair comparison of tradeoff algorithms.

Let us now discuss how sensitive a role the success rate plays in making algorithm comparisons. Note that the parameters used in [3, 4] achieved success probabilities D _ps=51.9 % and R _ps=55.6 %. According to Proposition 36, the optimal tradeoff performances of the DP tradeoff at the two success rates are

$$ \textup {\texttt {D}}_{\mathrm {ps}}= 51.9~\%, \qquad \textup {\texttt {D}}_{\mathrm {tc}}= 1.95, \qquad \textup {\texttt {D}}_{\mathrm {pc}}= 0.899, $$

(46)

and

$$ \textup {\texttt {D}}_{\mathrm {ps}}= 55.6~\%, \qquad \textup {\texttt {D}}_{\mathrm {tc}}= 2.56, \qquad \textup {\texttt {D}}_{\mathrm {pc}}= 0.996. $$

(47)

The figures of (46) show that the typical parameters m=t=ℓ=N ^1/3 considered in [3, 4] should not be used. We can obtain the success probability of (44) at a better tradeoff efficiency and with a smaller investment in precomputation.

A comparison of (46) and (47) clearly shows that a small difference in success rate can lead to a large difference in the optimal tradeoff coefficient. It can be seen from Proposition 36 that the optimal tradeoff coefficient will become even more sensitive to the success probability as the demand on success rate is increased.

The figures we gave concerning the success rate difference were not as dramatic as those concerning the alarm resolving cost in that no conclusion was overturned. However, since the performances of the different algorithms are close to each other, it is clear that the ability to accurately predict the success probabilities of tradeoff algorithms is critical in comparing the tradeoff algorithms.

9 Conclusion

In this work, we analyzed the running time complexities of the DP, Hellman, and rainbow tradeoffs, and summarized their abilities to balance storage against online time as tradeoff curves that are correct up to small multiplicative factors. These results were used in the later part of this work to compare the performances of tradeoff algorithms against each other. Our comparison is different from previous attempts in that the efforts for precomputation have been taken into account.

Although we did provide explicit statements comparing the three tradeoff algorithms, our conclusions are only true under certain assumptions concerning the tradeoff environment. We emphasize once more that one should not blindly extend our conclusions to other situations. Rather, researchers should see this work as providing the tools and methodology for fair comparisons of tradeoff algorithms and use them to arrive at their own final judgements specific to their circumstances.

One conclusion we can provide about the relative performances of different tradeoff algorithms is that their differences will be small. The practical inconvenience of having to align each entry of the precomputed table at a byte boundary has not been considered in this work, and the performance differences between algorithms can be so small that such obscure issues may be of equal importance in practice. This fact is disappointing to us as authors of the current work, but should be relieving to practitioners of the tradeoff algorithm that are not concerned with small performance differences. Nevertheless, even if one decides to ignore small performance differences, the comparison graphs of the previous section show that a meaningful reduction in precomputation cost can be achieved with only a small sacrifice in tradeoff efficiency, and being able to take advantage of this knowledge will be of practical importance. Furthermore, with extremely large-scale implementations, having accurate access to the small differences will be of significant value.

Complexity analyses of perfect table versions of the tradeoff algorithms at the accuracy level treated in this paper and their inclusion into the tradeoff performance comparison picture remain to be done. Perfect table tradeoffs are expected to display better tradeoff efficiency and are certainly of interest, even though they require larger amounts of precomputation.

Appendices

Appendix A. Technical Approximation

The following lemma shows that the approximation $(1-\frac {1}{{\textup {\textsf {b}}}} )^{{\textup {\textsf {a}}}}\approx e^{-\frac{{\textup {\textsf {a}}}}{{\textup {\textsf {b}}}}}$, which we have used frequently in this work, is very accurate for large integers a and b such that a=O(b).

Lemma 39

For positive integers  a and  b, we have

$$ \biggl|\exp \biggl(-\frac{{\textup {\textsf {a}}}}{{\textup {\textsf {b}}}} \biggr) - \biggl(1-\frac {1}{{\textup {\textsf {b}}}} \biggr)^{\textup {\textsf {a}}}\biggr| < \biggl\{ \frac{1}{2}\frac{{\textup {\textsf {a}}}}{{\textup {\textsf {b}}}^2} + \frac{1}{({\textup {\textsf {a}}}+1)!} \biggl(\frac{{\textup {\textsf {a}}}}{{\textup {\textsf {b}}}} \biggr)^{{\textup {\textsf {a}}}+1} \biggr\} \exp \biggl(\frac{{\textup {\textsf {a}}}}{{\textup {\textsf {b}}}} \biggr). $$

Proof

We start by writing $\exp (-\frac{{\textup {\textsf {a}}}}{{\textup {\textsf {b}}}} )$ in its Taylor series form and fully expanding the term $(1-\frac{1}{{\textup {\textsf {b}}}})^{{\textup {\textsf {a}}}}$.

After noting that the beginning two pairs of terms cancel out, we collect corresponding pairs from the two sequences of terms and bound the above by

$$ \biggl\{ \biggl|\frac{{\textup {\textsf {a}}}^2}{2!} - \binom{{\textup {\textsf {a}}}}{2} \biggr| \frac {1}{{\textup {\textsf {b}}}^2} + \cdots + \biggl|\frac{{\textup {\textsf {a}}}^{\textup {\textsf {a}}}}{{\textup {\textsf {a}}}!} - \binom{{\textup {\textsf {a}}}}{{\textup {\textsf {a}}}} \biggr| \frac{1}{{\textup {\textsf {b}}}^{\textup {\textsf {a}}}} \biggr\} + \biggl\{ \frac{1}{({\textup {\textsf {a}}}+1)!} \biggl( \frac{{\textup {\textsf {a}}}}{{\textup {\textsf {b}}}} \biggr)^{{\textup {\textsf {a}}}+1} + \cdots \biggr\}. $$

(A.1)

It is easy to see that

for every k≥2, where the last inequality can be checked through induction on k. This shows that the terms of (A.1) that appear inside the first set of braces are bounded by

As for the second set of braces from (A.1), it is easy to see that

$$ \frac{1}{({\textup {\textsf {a}}}+1)!} \biggl(\frac{{\textup {\textsf {a}}}}{{\textup {\textsf {b}}}} \biggr)^{{\textup {\textsf {a}}}+1} \exp \biggl(\frac{{\textup {\textsf {a}}}}{{\textup {\textsf {b}}}} \biggr) $$

can serve as its very rough bound. It now suffices to gather the two bounds to arrive at the claim. □

Appendix B. Random Function Arguments

Any analysis of a tradeoff algorithm assumes the one-way function F to be a one-way function, and most results given in this work as equations are certain values expected of a random function. In other words, we have been stating values that had been averaged over the choice of all functions . In this section, we point out that many of the arguments made during these computations are not strictly correct, and we then try to justify heuristically that the existing logical error may safely be ignored.

2.1 B.1 Existence of a Logical Gap

Recall the expected image size of a random function given by (1) and the expected iterated image sizes given by (2). The claim that (1) implies (2) is acceptable in the realm of cryptology. In this subsection, we clarify that there is a small logical gap in such a claim.

Let us rewrite (1) as an explicit self-contained statement which is precisely correct.

Lemma 40

Let be a random function on a finite set of size  N. If is of size m ₀, then the size of is expected to be

$$ m_1 = {\textup {\textsf {N}}}\biggl\{1- \biggl(1 - \frac{1}{{\textup {\textsf {N}}}} \biggr)^{m_0} \biggr\}. $$

The proof of this lemma is quite trivial. It suffices to consider the ratio of points among  that remain untouched throughout the sequential assignments made to elements of  for the random function construction.

We want to emphasize two things about this lemma. The first is that the value claimed by this lemma is the exact expected value and does not involve any approximation. In fact, the largest reason for rewriting the statement here was to remove the approximate expression. The second point we make is that the statement of this lemma does not contain any averaging over input sets. The expected image size claim holds true for every set of size m ₀.

Discussing just the double iteration case will be sufficient for our purposes. Let us define

$$ m_1 = {\textup {\textsf {N}}}\biggl\{1- \biggl(1 - \frac{1}{{\textup {\textsf {N}}}} \biggr)^{m_0} \biggr\} \quad\text{and}\quad m_2 = {\textup {\textsf {N}}}\biggl\{1- \biggl(1 - \frac{1}{{\textup {\textsf {N}}}} \biggr)^{m_1} \biggr\}, $$

(B.1)

for any given m ₀. One might believe that m ₂ is the expected size of , when is a random function and is of size m ₀. Since Lemma 40 contains no approximation, one might expect (B.1) to hold exactly. However, this reasonable prediction is not met, at least in the strict sense, by the explicit example given below.

The set of all functions F:{0,1}→{0,1} can be visualized as follows.

When the input set  is a single point, the image size expectation is clearly 1. This is in agreement with the value $2 \{1- (1-\frac{1}{2} )^{1} \} = 1$, computed according to Lemma 40. When the input set is the complete domain {0,1}, the image size expectation is $E_{F} [ |F(\{0,1\} )| ] = \frac{1}{4}\cdot1 + \frac{1}{4}\cdot2 + \frac {1}{4}\cdot2 + \frac{1}{4}\cdot1 = \frac{3}{2}$, and this is also identical to the value $E_{F} [ |F(\{0,1\})| ] = 2 \{1- (1-\frac{1}{2} )^{2} \} = \frac{3}{2}$, computed according to Lemma 40. We have just verified that Lemma 40, which had already been proved, holds exactly for the case, regardless of the input set size and the choice of the set itself. Now, the four functions F ²=F∘F can be visualized as follows.

When the input set  is taken to be the complete domain, the expected image size of the double iteration is

$$ E_{F} \bigl[ \bigl|F^2\bigl(\{0,1\}\bigr)\bigr| \bigr] = \frac{2}{4}\cdot1 + \frac{2}{4}\cdot2 = \frac{3}{2}. $$

(B.2)

In comparison, the corresponding value computed through (B.1) is

$$ 2 \biggl\{1- \biggl(1-\frac{1}{2} \biggr)^{2\{1-(1-1/2)^2\}} \biggr\} = 2 \biggl\{1- \biggl(1-\frac{1}{2} \biggr)^{3/2}\biggr\} \approx1.293. $$

(B.3)

The two values given above are clearly in disagreement.

A cryptographer would naturally attempt to rectify the current situation by relaxing the strict correlation between the two functions that are being composed. Let and be two independent random functions operating on a finite set of size N. One would like to claim that if is of size m ₀, then the size of is expected to be the m ₂ value given by (B.1). This second version for the doubly iterated image size expectation seems structurally much simpler to analyze than the previous attempt, and one might be tempted to say that the modified claim is a trivial consequence of Lemma 40.

We again turn to the example F,G:{0,1}→{0,1}. The complete set of all possible double iterations can be visualized as follows.

When the input set  is the full domain {0,1}, after separately counting the number of functions with image sizes one and two, the expected image size can be computed as

$$ E_{F,G} \bigl[ \bigl|G\bigl(F\bigl(\{0,1\}\bigr) \bigr) \bigr| \bigr] = \frac{12}{16}\cdot1 + \frac{4}{16}\cdot2 = \frac{5}{4}. $$

(B.4)

Once again, this disagrees with (B.3), which was computed through (B.1).

It is now clear that (2) does not directly follow from (1). The claims to the iterated image sizes are not consequences of the single step image size, at least not without additional arguments. The logical gap persists even when all iterations are allowed to be independent random functions.

2.2 B.2 Narrowing the Logical Gap

The failed attempt (B.1) at giving a doubly iterated image size expectation had substituted the m ₁ value in the place of m ₀ in the single step result Lemma 40. This reuse of average value in the computation of another average value was the source of our problem. In reality, as can be seen in the two counterexamples, inputs to the second step function are not all of m ₁ size, but of varying sizes that only average to m ₁. After this simple observation, we can state that, if is a set of size m ₀ such that the image size is exactly m ₁ for every choice of function F and the image size is exactly m ₂ for every choice of function F and every input set of size m ₁, then m ₂ is the exact expected size of . The assumptions included in this statement cannot be met, but it is reasonable to expect the conclusion to hold approximately, when a slight relaxation is given to the assumptions. We are thus justified in stating that, if for the vast majority of the sets  and functions , the image size is very close to , then the m ₂ of (B.1) will be a good approximation for the doubly iterated image size expectation.

Therefore, we consider the images of a fixed set  under different functions F and discuss how their sizes are distributed around its average. Let us use μ _N,m and σ _N,m to denote the average and standard deviation of the image set size . These are to be computed for a fixed input set of size m and with running over all possible function choices. We already know $\mu_{{\textup {\textsf {N}}},m} \approx {\textup {\textsf {N}}}\{ 1- \exp (-\frac{m}{{\textup {\textsf {N}}}} ) \}$. A proof of the following lemma is given in Appendix C.

Lemma 41

We have $\frac{\sigma_{{\textup {\textsf {N}}},m}}{\mu_{{\textup {\textsf {N}}},m}} < \frac {2}{\sqrt{{\textup {\textsf {N}}}}}$ for all  N and m.

According to Chebyshev’s inequality, at least 99 % of the N ^N image sizes will fall within the range μ _N,m ±10σ _N,m . The above lemma states that this deviation of sizes from the mean is bounded by $\frac{20\mu_{{\textup {\textsf {N}}},m}}{\sqrt {{\textup {\textsf {N}}}}}$. Hence, the distribution or clustering of image sizes around the expected value μ _N,m will tighten, at least in comparison to the expected value, as N is increased.

This observation can be restated in more plain terms as follows. Suppose we take some input set and measure its image size under a single function, chosen at random, and take it to be an estimate of the true average image size. We make it clear that the averaging over multiple measurements made with multiple functions is not being performed here. In such a situation, we can expect each measurement to return a larger number of significant digits as N is increased. Let us briefly work with some explicit numbers. For parameters N=2⁶⁴ and m=2⁵⁰ the average image size can be computed to be μ _N,m ≈1.13×10¹⁶. For the same parameters, the standard deviation is bounded by σ _N,m ≤5.24×10⁵. Chebyshev’s inequality ensures that at least 99 % of the N ^N image sizes will lie in the range μ _N,m ±10σ _N,m , which is 1.13×10¹⁶±5.24×10⁶ in the current situation. For any practical purposes, we can believe that close to 10 significant digits from any single measurement are highly likely to be identical to those of the true expected value.

Let us summarize the discussion of this subsection. For any function acting on a large set that was chosen at random and any input set of size m ₀, the image size of the first iteration will be very close to the m ₁ value given by (B.1). At the second iterated application of the same function, even though the input size was not exactly m ₁, we can expect the output size to be very close to the m ₂ value given by (B.1). Actually, the output size could be different from m ₂ even if the input size was exactly m ₁. In any case, the fact that the standard deviation of the image sizes is very small relative to its expected value implies a tight clustering of image sizes, and allows us to believe that the formula (2) will predict doubly iterated image sizes with accuracy, in the sense that a large number of significant digits are returned. The heuristic arguments of this subsection have added further justification to the already acceptable cryptographic argument that (1) implies (2).

2.3 B.3 Other Reuses of Average Values

The intention of this section was not to test the validity of (2). In fact, although the authors of the current paper are unfit to verify its correctness, a full proof is provided in [11] for at least the case when  is the full domain. What we have done so far in the current section is to first point out that average values have erroneously been reused in the computation of other average values and then argue heuristically that such methods are still acceptable as long as the distribution of values that are being treated is tightly gathered around the average. This reasoning does not have to be restricted to the discussion of iterated image sizes, or even random function arguments.

There are many occasions in this paper where an average value was used during the computation of another average value. It should now be clear that (10), stating the success probability of a single rainbow matrix, is also slightly problematic, but acceptable. The different reduction functions at each rainbow matrix column do not provide independence of the colored iterating functions, and the existing logical gap would not be closed even if different columns were processed with independent random functions. However, the small standard deviation of image sizes justifies (10) as a good approximation.

The success probability (4) of the DP and Hellman tradeoffs, computed from the average number of points in a tradeoff matrix, is another example of average value reuse. We have not checked if the standard deviation of the coverage rate is small, but we know from experience that (4) predicts the correct value accurately, so this should not be a problem. In fact, this situation is less problematic than the iterated image case, because the arguments become strictly correct when independent random functions are used in different tables.

Readers may have noticed that we were more careful in reusing average values in Sect. 4.2. The distribution of chain lengths in a DP matrix can be inferred from (16), and it is clear that the lengths are not at all centered around the average length t. Hence, we were careful to work with the full range of possible chain lengths, rather than treat t as being the typical precomputation or online chain length. In particular, we did not treat the DP matrix as consisting of m chains of identical length t. This cautious handling of chains should not be confused with our free use of the value (16) itself, which is an expected value, in other computations.

Appendix C. Standard Deviation of Image Sizes

The purpose of the section is to provide a proof to Lemma 41 concerning the standard deviation of image sizes. We first prepare a couple of technical lemmas.

Lemma 42

Let be a random function. Fix a subset of size m and let be any two distinct points. The probability for to contain both y ₁ and y ₂ is

$$ \biggl\{ 1 - \biggl(1-\frac{1}{{\textup {\textsf {N}}}} \biggr)^m \biggr \}^2 - \biggl(1-\frac{1}{{\textup {\textsf {N}}}} \biggr)^m \biggl\{ \biggl(1-\frac{1}{{\textup {\textsf {N}}}} \biggr)^m - \biggl(1-\frac{1}{{\textup {\textsf {N}}}-1} \biggr)^m \biggr\}. $$

Proof

The probability under consideration may be computed as follows:

In each additive term, the part $\binom{m}{k} (\frac{1}{{\textup {\textsf {N}}}} )^{k} (1-\frac{1}{{\textup {\textsf {N}}}} )^{m-k}$ gives the probability for exactly k out of the m inputs to map to y ₁. The remaining $\{ 1 - (1-\frac{1}{{\textup {\textsf {N}}}-1} )^{m-k} \}$ part is the probability for at least one of the (m−k) inputs that are known not to have reached y ₁ to map to y ₂. The above sum is equal to the expression

To check this claim, it suffices to expand the first two pairs of braces. This expression can be rewritten in the form stated by this lemma. □

Lemma 43

For positive integers N and m, we have

$$ \biggl(1-\frac{1}{{\textup {\textsf {N}}}} \biggr)^m - \biggl(1-\frac{1}{{\textup {\textsf {N}}}-1} \biggr)^m \geq \frac{m}{{\textup {\textsf {N}}}({\textup {\textsf {N}}}-1)} \biggl(1-\frac{1}{{\textup {\textsf {N}}}-1} \biggr)^{m-1}. $$

Proof

It suffices to check the following sequence of equalities and inequality:

In fact, a similar upper bound is also easy to obtain. □

In the remainder of this section, will be a fixed set of size m. For each , let us define the function by

The dependence of χ _y on was not made explicit in the notation since we will keep fixed for the rest of this section. The size of the image of under any function can be expressed in terms of this indicator function as

Using this observation, one can present

(C.1)

where y′ is any fixed point of , as an alternative way of writing the proof to Lemma 40.

Let us fix the notation

and view this as a random variable defined on the space , which is given the uniform probability distribution. It maps each element F to the positive integer . Equation (C.1) is equivalent to

$$ E[\chi] = {\textup {\textsf {N}}}\biggl\{1 - \biggl(1 - \frac{1}{{\textup {\textsf {N}}}} \biggr)^m \biggr\} $$

(C.2)

and we need to work with the standard deviation

$$ \textup{stdev}(\chi) = \sqrt{E\bigl[\chi^2\bigr] - \bigl(E[\chi] \bigr)^2}. $$

One can easily check that

where $\mathbf {y}_{1}'$ and $\mathbf {y}_{2}'$ are any two distinct points of . The expectation $E [\chi_{\mathbf {y}_{1}'}\chi_{\mathbf {y}_{2}'} ]$ is equal to the probability for both $\mathbf {y}_{1}'$ and $\mathbf {y}_{2}'$ to belong to the image space, and this is the content of Lemma 42. Referring also to (C.2) and Lemma 43, we can compute a bound for the variance as follows:

Here, the second inequality follows from the observation $(1-\frac {1}{{\textup {\textsf {N}}}} )^{m} \geq1 - \frac{m}{{\textup {\textsf {N}}}}$. The final expression allows us to state that $\textup {stdev}(\chi) \leq\frac{{m}}{\sqrt{{\textup {\textsf {N}}}}}$.

On the other hand, from the observation $(1-\frac{1}{{\textup {\textsf {N}}}} )^{m} \leq1 - \frac{m}{{\textup {\textsf {N}}}} + \frac{m(m-1)}{2{\textup {\textsf {N}}}^{2}}$, which holds for every m≤N, we know that

$$ E[\chi] \geq {\textup {\textsf {N}}}\biggl\{ \frac{m}{{\textup {\textsf {N}}}} - \frac{m(m-1)}{2{\textup {\textsf {N}}}^2} \biggr\} > {\textup {\textsf {N}}}\biggl(\frac{m}{{\textup {\textsf {N}}}} - \frac{m}{2{\textup {\textsf {N}}}} \biggr) = \frac{m}{2}. $$

Finally, by combining the two bounds, we can state that

$$ \frac{\textup{stdev}(\chi)}{E[\chi]} < \frac{2}{\sqrt{{\textup {\textsf {N}}}}}. $$

This concludes the proof of Lemma 41.

Appendix D. Note on the Index Tables Method

The index table method can be seen as a special case of a more general and widely known data structure called hash tables. To store m starting point and ending point pairs, one first fixes a hash function that maps elements of  to logm-bit strings. This function need not be a cryptographic hash function, although the same term is used. Instead of sorting the data, each starting point and ending point pair is recorded at the position in the storage addressed by the hash value of the ending point. Collisions of addresses are inevitable, but there are various ways to deal with this problem.

Table lookups to hash tables are performed by first hashing the ending point to be searched for in the table and fetching the data located at the address pointed to by the hash value. Since the address holds logm bits of information, even if almost logm bits from each ending point are removed before storage, we can reliably determine whether or not a match has occurred.

One advantage of the hash table method, other than reducing storage and not requiring any sorting, is that it provides constant time table lookups. In comparison, a lookup to a sorted table requires time that is logarithmic in the table size.

If the hash function is set to return the first {(logm)−ε} bits of its input and buckets to hold approximately 2^ε table entries are placed at the position pointed to by each hash value, then the hash table technique reduces to the index table technique.

Appendix E. Experimental Results

In this section we verify that the main parts of our arguments agree well with the experimental results. Experiments are done to check the validity of our results concerning the coverage rate and the cost of false alarms for the DP tradeoff. Analogous testing for the Hellman and rainbow tradeoffs is not provided, as this testing was done in [15]. We also provide experimental evidence supporting our arguments surrounding the effects of the ending point truncation method.

Since averaging over all functions defined on any reasonably large space is not at all possible, all our tests were conducted with a very small subset of explicitly constructed one-way functions. The one-way function used was always the encryption key to ciphertext mapping, under a fixed plaintext, computed with the block cipher AES-128. Different randomly chosen plaintexts were used to provide multiple one-way functions. The size of the input space was controlled by utilizing only a small number of key bits and padding the remaining key bits with zeros. The output space size was controlled by masking the ciphertext to an appropriate bit length. When working with the DP tradeoff, as discussed at the start of Sect. 4, we constructed $m_{0} = \frac{m}{1-e^{-{\hat {t}}/t}}$ precomputation chains and gathered every resulting DP chain, rather than incrementally generating additional chains until m DP chains were collected.

5.1 E.1 Coverage Rate of DP Tradeoffs

The experimental results supporting Proposition 9, which presents the coverage rate of a DP table, are given in Table 2. The coverage rate was measured by simply storing all DP matrix entries while constructing the DP chains and later counting the number of distinct matrix entries that were used as inputs to the one-way function. Each test result value given in the table is an average over 100 experiments. Different randomly generated plaintexts for AES were used for each of these experiments. All the tests were done on a space of size N=2³⁰. One can check that the test figures are very close to what the theory predicted.

Table 2. Coverage rate of DP tradeoff (N=2³⁰)

5.2 E.2 Cost of Resolving Alarms for the DP Tradeoff

Our next goal is to check the validity of our arguments concerning the time complexity that incorporates the extra cost of false alarms. We could do this with the expression for time complexity stated during the proof of Theorem 13, but such an approach would hide much of the inner workings. Hence, we decided to verify the following lemma, which allows access to much finer detail.

Lemma 44

Consider the DP tradeoff. The expected number of chain collisions at the ith iteration of the online phase is

$$ \frac {1}{t}\frac{ \textup {\texttt {D}}_{\mathrm {msc}}}{1-e^{-{\hat {t}}/t}} \biggl\{ - e^{-{\hat {t}}/t} + e^{-{\hat {t}}/t} \exp \biggl(-\frac{i}{t} \biggr) + \frac{i}{t} \exp \biggl(- \frac{i}{t} \biggr) \biggr\}. $$

Proof

The expected number of chain collisions is the sum over all rows of the DP matrix of the respective probabilities for the ith iteration to sound an alarm in association with that row. After reading the proof of Lemma 12, it should be clear that the sum of probabilities we are looking for is

$$ \sum_{j=1}^{\hat {t}}\frac{\frac{m}{t}}{1-e^{-{\hat {t}}/t}} \exp \biggl(-\frac{j}{t} \biggr)\cdot \frac{t}{{\textup {\textsf {N}}}} \biggl\{\exp \biggl( \frac{\min\{i,j\}}{t} \biggr)-1 \biggr\} \exp \biggl(-\frac{i}{t} \biggr). $$

In integral form, this is approximately

$$ \frac {1}{t}\frac{\frac{mt^2}{{\textup {\textsf {N}}}}}{1-e^{-{\hat {t}}/t}} \exp \biggl(-\frac{i}{t} \biggr) \int _0^{{\hat {t}}/t} \exp(-v) \biggl\{\exp \biggl(\min \biggl \{ \frac{i}{t}, v \biggr\} \biggr) - 1 \biggr\} \, dv, $$

which simplifies to what is claimed. □

This lemma contains the core of our arguments given in the main text concerning the cost of alarms, and its verification through experiments should provide good support for the correctness of our theory.

To test this lemma, we first initialized an array of ${\hat {t}}$ counters to zeros. Next, we fixed a one-way function by randomly choosing a plaintext and constructed a DP table with the fixed function. Then, a random password (= zero-padded encryption key) was generated and the password hash (= masked ciphertext) corresponding to that password was computed. The online chain starting from the password hash was computed until a DP was found or the ${\hat {t}}$th iteration was reached. If the online chain terminated at a DP and it was found to reside within the DP table, the counter corresponding to the current online iteration count was incremented. The online chain generation was repeated multiple times with the same table, but with newly generated random keys. Note that, since we are not using perfect tables, it is possible for the online chain to collide simultaneously with more than one entry of the DP table. Care was taken to increment the counter corresponding to the current iteration count as many times as the number of collisions found. The whole process described after the counter initialization step was repeated multiple times, with each repetition using a newly generated one-way function and a DP table.

The test results for four different parameter sets are presented in Fig. 7. Each of these experiments was performed with 2000 tables and 5000 random online chains per table. In each of the four boxes, the barely visible thin dashed line represents our theory as given by Lemma 44. There are ${\hat {t}}$-many tiny dots in each box, and these represent our experimental results. The height of the ith dot, counting from the left, is the value of the ith iteration counter at the end of the experiment divided by 2000×5000, the total number of chains that were utilized. All the experiment results match our theory very well.

Fig. 7.

Expected number of collisions at each iteration of the DP tradeoff (dots: experiment; dashed line: theory).

5.3 E.3 Ending Point Truncation

Finally, we test the validity of our arguments concerning the ending point truncation method for reducing storage. The straightforward approach would be to simply test Lemma 16, Lemma 24, and Lemma 32, which present the cost of truncation related alarms, but we decided to work with the probability of alarms related to truncations, so as to expose more of our argument details to the tests.

Lemma 45

Consider the DP tradeoff that uses an ending point truncation of $\frac {1}{r}$ truncated match probability. At the ith iteration of the online processing of a single DP table, the number of pseudo-collisions that are due to the ending point truncations, i.e., those that are not associated with any true chain collisions, is expected to be  $\frac{m}{r} \exp(-\frac{i}{t})$. The corresponding value for the Hellman tradeoff is  $\frac{m}{r}$, and that for the rainbow tradeoff is also  $\frac{m}{r}$, if one decides to fully process a single rainbow table without terminating, even when the correct answer is found.

Proof

The proof of Lemma 16 shows that the claimed expected value for the DP tradeoff case can be computed as

$$ \sum_{j=1}^{\hat {t}}\frac{\frac{m}{t}}{1-e^{-{\hat {t}}/t}} \exp \biggl(-\frac{j}{t} \biggr)\cdot \exp \biggl(-\frac{i}{t} \biggr) \frac{1}{r} \approx \frac{m}{1-e^{-{\hat {t}}/t}} \int_0^{{\hat {t}}/t} \exp(-v) \,dv \exp \biggl(-\frac{i}{t} \biggr)\frac{1}{r}, $$

which simplifies to what is claimed. The statement for the Hellman tradeoff case follows immediately from the proof of Lemma 24, and the rainbow tradeoff case can be inferred from the proof of Lemma 32. □

The three claims given by this lemma are at the core of our arguments concerning the ending point truncation method, and experimental verification of these statements should provide confidence as to the validity of our arguments given in the main text.

As in the previous section, we generated random tradeoff tables and tested with random online chains for the occurrence of alarms induced from truncations. We stored the full ending points, together with the truncated ending points, in the precomputation table. The full ending point information was used to distinguish between alarms that were caused by ending point truncations and those that arose from true chain collisions.

The test results are given in Figs. 8, 9, and 10. As before, the thin dashed lines are the graphs claimed in Lemma 45 and the numerous tiny dots represent the experimental data. All the test results are in good agreement with the theory. Each of the two diagrams for the DP tradeoff was obtained by averaging over 2000 tables and 5000 online chains per table. For the Hellman tradeoff we generated 2000 tables and 5000 inversion targets per table. The online chain was computed to the full length t for each inversion target, and a search was made for the truncated match with the table elements after each one-way function iteration. In the rainbow tradeoff case, each diagram is the result of 100 tables with 5000 inversion targets per table. Recall that the kth iteration for the rainbow tradeoff refers to a process that consists of (k−1) invocations of the one-way function and one table lookup. Full t iterations were attempted for each inversion target; hence each inversion target generated t searches to the table for truncated matches.

Fig. 8.

Expected number of collisions, induced by ending point truncation, at each iteration of the DP tradeoff (dots: experiment; dashed line: theory).

Fig. 9.

Expected number of collisions, induced by ending point truncation, at each iteration of the Hellman tradeoff (dots: experiment; dashed line: theory).

Fig. 10.

Expected number of collisions, induced by ending point truncation, at each iteration of the rainbow tradeoff (dots: experiment; dashed line: theory).