Automated black-box boundary value detection

Selection: program derivative

We argue that the critical problem in boundary value analysis is judging what is a boundary and what is not. If we could quantify how close to a boundary a given input is, we could then use a plethora of methods to find many such inputs and keep only the ones closest to a boundary. Together such a set could indicate where the boundaries of the software are or at least indicate areas closer to the actual boundaries.

In a previous work (Feldt & Dobslaw, 2019), we proposed the program derivative for quantifying boundariness. The idea is based on a generalisation of the classic definition of the derivative of a function in (mathematical) analysis. In analysis, the definition is typically expressed in terms of one point, x, and a delta value, h, which together define a second point after summation. The derivative is then the limit as the delta value approaches zero:

${\displaystyle \underset{h\to 0}{lim}\frac{f\left(x+h\right)-f\left(x\right)}{\left(x+h\right)-x}=\underset{h\to 0}{lim}\frac{f\left(x+h\right)-f\left(x\right)}{h}}$

The derivative measures the sensitivity to a change in the function given a change in the input value. A large (absolute value of a) derivative indicates that the function changes a lot, even for a minimal input change. If the function f here instead was the SUT, and the slight change in inputs would cross a boundary, it is reasonable that the output would also change more than if the change did not cross a boundary. We could then use this program derivative to screen for input pairs that are good candidates to cross the boundaries of a program.

The key to generalizing from mathematical functions to programs is to realize that programs typically have many more than one input, and their types can differ greatly from numbers. Also, there can be many outputs, and their types might vary from the types of inputs. Instead of simply using subtraction (“-”) both in the numerator and denominator, we need two distance functions, one for the outputs (d_o) and one for the inputs (d_i). Also, rather than finding the closest input to calculate the derivative of a single input, for our purposes here, we only need to quantify the boundariness of any two individual inputs. We thus define the program difference quotient (PDQ) for program P and inputs³ a and b as Feldt & Dobslaw (2019): ${\displaystyle PD{Q}_{d_o,d_i}\left(a,b\right)=\frac{d_o\left(P\left(a\right),P\left(b\right)\right)}{d_i\left(a,b\right)}\text{, where}P\left(x\right)\text{denotes the output of the program for input}x\text{.}}$

Since the PDQ is parameterized on the input and output distance functions, this defines not a single but a whole family of different measures. A tester can choose distance functions to capture meaningful output differences and/or inputs. In the original program derivative paper, Feldt & Dobslaw (2019), we argued for compression-based distance functions as a good and general choice. However, a downside with these is that they can be relatively costly to calculate, which could be a hindrance when used in a search-based loop, potentially requiring many distance calculations. Also, compression-based distances using mainstream string compressors such as zlib or bzip2 might not work well for short strings, as commonly seen in testing.

In this work, we thus use one of the least costly output distance functions one can think of: strlendist as the difference in length of outputs when they are printed as strings. This distance function works regardless of the type of output involved. A downside is that it is coarse-grained and will not detect smaller differences in outputs of the same length. Still, if a simple and fast distance function can suffice to detect boundaries, this can be a good baseline for further investigation. Also, our framework is flexible and can use multiple distance functions for different purposes in its different components. For example, one could use strlendist during search and exploration while using more fine-grained NCD-based output distance when updating the archive or during summarisation (see Fig. 1).

For the input distance function, this will typically vary depending on the SUT. We can use simple vector functions such as Euclidean distance if inputs can be represented as numbers or a vector of numbers. For more complex input types, one can use string-based distance functions like Normalised Compression Distance (NCD) (Feldt et al., 2008; Feldt et al., 2016; Feldt & Dobslaw, 2019) or even simpler ones like Jaccard or the related Overlap Coefficient distance (Jaccard, 1912).

Exploration: generation of candidate pairs

While the program derivative can help evaluate whether an input pair is a good candidate to detect a boundary, there are many possible such pairs to consider. Thus, we need ways to explore the input space and propose good candidate pairs, i.e., that have high program derivative values. A natural way to approach this is as a search-based software engineering problem (Harman & Jones, 2001; Afzal, Torkar & Feldt, 2009; Feldt, 1998), with the program derivative as the goal/fitness function and a search algorithm that tries to optimize for higher program derivative values.

However, detecting the boundaries is insufficient to find and return one candidate pair. Most software will have multiple and different boundaries in their input domain. Furthermore, boundaries are typically stretched out over (consecutive) sets of inputs. The search and exploration procedure we chose should thus output a set of input pairs that are, ideally, spread out over the input domain (to find multiple boundaries) as well as over each boundary (to help testers identify where it is).

An additional concern when using search-based approaches is the shape of the fitness landscape, i.e., how the fitness value changes over the space being searched (Smith, Husbands & O’Shea, 2002; Yu & Miller, 2006). Many search and optimisation approaches assume, or at least benefit from, a smooth landscape, i.e., small steps in the space lead to only a tiny change in the fitness value. Whether we can assume this to be the case for our problem is unclear. The program derivative might be very high right at the boundary while showing minimal variation when inside the partitions on either side of the boundary. Worst case, this could be a kind of needle-in-a-haystack fitness landscape (Yu & Miller, 2006) where there is little to no indication within a sub-domain to guide a search towards its edges, where the boundary is.

Given these differences compared to how search and optimisation have typically been applied in software testing, we formulate our approach in a generic sense. We can then instantiate it using different search procedures and empirically study which are more or less effective. However, given that we are searching for a pair of two inputs with a significant derivative in the program, we must tailor even the most basic random search baseline to this problem.

We, therefore, formulate AutoBVA detection as a generic framework with customizable components of significance. Two essential abstractions are (1) the global sampling/generation strategy to decide starting points for (2) the local exploration/mutation strategy to modify those starting points in a structured way to detect good, candidate boundary pairs. Algorithm 1 outlines this generic, 2-step automated boundary detection. Given a Software under test SUT, it returns a set of boundary candidate pairs (BC). It has three additional parameters: a way to quantify the boundariness of pairs of inputs (Q), a (global) sampling strategy (SS) to propose starting points for local exploration and a (local) boundary search (BS) strategy. These exploration strategies capture two different types of sampling/search procedures. The global one explores the input space of the SUT as a whole by sampling different input points (line 3). The local strategy will then search from the starting point (line 4) by manipulating it by applying mutations. Each of the potential new candidates (in PBC) is then evaluated and their boundariness is compared to a threshold (line 5) and added to the final set (BC) returned. The bytecount function simply captures the fact that we might not use a fixed threshold but rather can allow more complex updating schemes where the threshold is based on the candidates that have already been found. For example, the threshold could be taken as some percentile (say, 90%) of the boundariness values of the candidate set saved so far. Alternatively, even more, elaborate boundariness testing and candidate set update procedures can be used, such as rather than simply adding to the current set whenever a sufficiently good boundary value is found, we could save a top list of the highest boundariness values found so far. The update would then be generalized so that it can also delete previously added candidate pairs that are no longer promising.

 
_______________________ 
Algorithm 1 Automated Boundary Detection - AutoBD-2step____________________ 
Require: Software under test SUT, Boundariness quantifier Q, Sampling strat- 
     egy SS, Boundary search BS 
Ensure: Boundary candidates BC 
 1:  BC = ∅ 
 2:  while stop criterion not reached do 
 3:     input = SS.sample(SUT,BC) # globally sample a starting point 
  4:     PBC = BS.search(SUT,Q,input) # locally explore and detect potential 
        candidate(s) 
  5:     BC = BC ∪{c|c ∈ PBC ∧ Q.evaluate(c) > threshold(BC)} 
 6:  end while 
 7:  return  BC______________________________________________________________________    

The sampling strategy defines where we start looking, i.e., what input we start our search from. The local search strategy defines how we manipulate that input to identify boundary candidates. For complex data types, such as XML trees or images, more sophisticated generators are required. For numbers, the shelf sampling from basic distributions, such as uniform at random, suffices. We use uniform at-random sampling as a sampling baseline.

To illustrate, consider a SUT that takes a single integer number as input. The sampler returns a random number, say 10. Let us assume that the local search has access to increment and decrement mutations. In the most naive form, the local search simply applies each mutation operator once per argument to obtain a neighbourhood of distance one. The candidate pairs (9, 10) and (10, 11) would be probed in the local search. If the SUT in question were actually the above-mentioned bytecount PBC, it would then return all candidates above a threshold - here, the candidate pair (9, 10) since it has the highest PDQ value. Since this is among the simplest mechanisms for deriving boundary candidates, it informed our formulation of the Local Neighbourhood Search below, which we use as a baseline.

The benefit of creating candidate input pairs in two steps may become particularly obvious when the inputs are of a complex and structured datatype, such as an XML tree. If search was done with a single, global exploration step, it would be relatively unlikely that the two inputs of a candidate pair would have a small distance or that there even is a useful way of generating inputs that lead from one point to the other. By doing this in two steps, we could first sample XML tree instances and then explore their neighbourhoods in the input space by small mutations. This mechanism, however, is not limited to exploring only the very near neighbourhood of a point but may be used to explore the space by a repeated application of mutation operators. This idea informed our second local search strategy. Future work may still look into using global sampling only. Still, the combination of arbitrary inputs to detect boundaries seems non-trivial and might require tailored crossover mechanisms for input types.

For this study, we first benchmark the impact of the global sampling (uniform, bituniform) in a screening study to understand its effect on the results. The study is presented in Appendix A. We found that uniform sampling (often referred to as random sampling) performed substantially worse-in fact, it did not find any boundary candidates in any attempt when another feature was deactivated. Consequently, we simplified the experimental design of the main study to bituniform sampling. Next, we explain the two boundary search algorithms mentioned above: Local Neighbour Sampling and Boundary Crossing Search.

 
________________________________________________________________________________________ 
Algorithm 2 search – Local Neighbour Sampling (LNS)____________________________ 
Require: Software under test SUT, mutation operators mos, Starting Point i 
Ensure: potential boundary candidates PBC 
 1:  PBC = ∅ 
 2:  o = SUT.execute(i) 
  3:  for a ∈ arguments(SUT) do 
 4:     for mo ∈ mos[a] do 
 5:         n = mo.apply(i,a) 
  6:         on = SUT.execute(n) 
  7:         c = 〈i,o,n,on〉 
 8:         PBC = PBC ∪{c} 
 9:     end for 
10:  end for 
11:  return  PBC____________________________________________________________________    

 
________________________________________________________________________________________ 
Algorithm 3 search – Boundary Crossing Search (BCS)____________________________ 
Require: Software under test SUT, Boundariness quantifier Q, mutation op- 
     erators mos, Starting Point i 
Ensure: potential boundary candidates PBC 
 1:  a = rand(arguments(SUT)) # select random argument 
  2:  mo = rand(mos[a]) # select random mutation operator for argument 
  3:  inext = mo.apply(i,a) # mutate input a first time in single dimension 
  4:  o = SUT.execute(i),onext = SUT.execute(inext) # produce outputs 
  5:  cinit = 〈i,o,inext,onext〉 # instantiate initial candidate 
  6:  Δinit = Q.evaluate(cinit) # calculate candidate distance 
  7:  c = 〈i1,o1,i2,o2〉, with i1 obtained by a finite number of chained mutations 
     mo of a over i, and 
  8:                                     i2 = mo.apply(i1,a), and 
  9:                                     o1 = SUT.execute(i1), and 
10:                                     o2 = SUT.execute(i2), and 
11:                                     Δc = Q.evaluate(c), and 
12:                                     Δc > Δinit or cinit == c 
13:  return  {c}____________________________________    

Summarisation: validity-value similarity clustering

The boundary candidate set resulting from AutoBD-2step (Algorithm 1) can be extensive. However, human information processing is limited, and more fundamentally many of the boundary pairs found can be similar to each other and represent the same or a very similar boundary. Take, for instance, the Date example with a great number of different boundary pairs crossing the same behavioural boundary between valid dates and the invalid 0 for the day - 〈(0, m, y), (1, m, y)〉. The summary method(s) we choose should thus not only limit the number of candidates presented, but those candidates also need to be different and represent different groups of behaviours and boundaries over the input space.

Furthermore, the goals for the summary step will likely differ depending on what the boundary candidates are to be used for; comparing a specification to actual boundaries in an implementation is a different use case than adding boundary test cases to increase coverage. Thus we cannot provide one general solution for summarisation, and future work will have to inspect different methods to cluster, select, and prioritize but also visualize the boundary candidates.

In the following, we propose one particular but very general summarisation method. We hope it can show several building blocks needed when creating summarisation methods for specific use cases and act as a general fallback method that can provide value regardless of the use case. This is also the method we use in the experimental evaluation of the article. We consider a general, black-box situation with no specification available. Thus, we only want to use information about the boundary candidates themselves. The general idea is to identify meaningful clusters of similar candidates, then sample only one or a few representative candidates per cluster and present them to the tester.

For instance, Table 2 contains a subset of (nine) candidate pairs found by our method for the bytecount example introduced above. Different features can be considered when grouping and prioritizing. We see that candidates differ at least in the output type, i.e., whether the outputs are valid return values or exceptions, as well as in the actual values of inputs and/or outputs themselves. For example, both outputs for the candidate on row 2 are strings and are considered normal, valid return values. On the other hand, for row 9, both outputs are (Julia) exceptions indicating that a string of length 6 (“kMGTPE”) has been accessed at (illegal) positions 9 and 10, respectively.

We argue that the highest level boundary in boundary value analysis is between valid and invalid output values. Any time there is an exception thrown when the SUT is executed, we consider the output to be invalid; if not, the output is valid.⁸ Since we consider pairs of inputs, two outputs per candidate can be grouped into three, what we call, validity groups:

• VV: two valid outputs in the boundary pair.

• VE: one valid output and one invalid (error) output.

• EE: two invalid (error) outputs.

We use validity as the top-level variation dimension and produce individual summaries for these three groups. Table 2 indicates the validity group in the rightmost column (VV for lines 1–7, VE in line 8, and EE in line 9). We may consider other characteristics within each validity group to categorize candidates further. For example, we could use the type of the two outputs to create additional (sub-)groups. This is logical since it is not clear that comparing the similarity of values of different types is always meaningful. However, for many SUTs and their validity groups, the types might not provide further differentiation as they often are the same.

In the final step, we propose to cluster based on the similarity of features within the identified sub-groups. A type-specific similarity function can be used, or a general, string-based distance function can be used after converting the values to strings. We can create a distance matrix per sub-group after calculating the pair-wise similarities (distances). This can then be used with any clustering method while in the experiments in this article, we used k-means clustering (Likas, Vlassis & Verbeek, 2003). We select one representative (or short) boundary pair from each cluster to present to the tester. The distance matrix can also be used with dimensionality reduction methods to visualize the validity-type sub-group. We thus call our summarisation method validity-value similarity clustering.

The duality of having two sites of a boundary exposes more structure than thinking in test cases as single points with the expected outcomes only. Therefore, the validity groups are not meant to be a sufficient differentiator but a tool to exploit general structure for better separation for most, if not all SUTs.

Setup of selection and exploration step

Before the main experiment, we performed two screening studies to configure (1) the (global) sampling strategy and (2) the clustering of boundary candidates (see Appendices A and B). Table 3 summarises the setup of both investigations. The sampling method is fixed as the best-performing configuration from the screening study and uses both bituniform sampling and CTS for all experiments. The boundariness quantifier is based on the program derivative with bytecount for the outputs, i.e., the length difference between the stringified versions of outputs. Since all input parameters of the SUTs in this study are integer types, numerical distance is used (implicitly) as input distance. In the search, we use an increment (add to integer) and a decrement (subtract from integer) as mutation operators. We repeat each experiment execution 20 times to account for variations caused by the pseudorandom number generator used during the search. A constant and permissive threshold of 0 for adding boundary candidates to BC is applied in this study, i.e., all pairs with any difference in output length are added. Setup specific for each investigation is taken up below.

Setup of summarisation step

To summarise a large set of boundary candidates, we have to decide and extract a set of complementing features able to group similar candidates and set apart those that are dissimilar. Ideally, we want a generic procedure which can give good results for many different types of SUTs. We want to select features so boundary pairs with similar outputs are grouped. The focus of this study has been on detection, while the following reasoning regarding simplicity and feature selection guided the applied summarisation.

We implement the AutoBVA summarisation by validity-value similarity clustering using k-means clustering Hartigan & Wong (1979). We choose k-means clustering because it is one of the simplest, well-studied, and understood clustering algorithms and widely available Xu & Tian (2015). Clustering was done per validity group to avoid mixing pairs that have very different output types, namely: VV (String, String), VE (String, Error), and EE (Error, Error). We span a feature space over the boundary candidates to capture a diverse range of properties and allow for a diversified clustering from which to sample representatives per cluster. We extract features from the output differences between boundary candidates since input distances within the boundary candidates are already factored into the selection of the candidates. Moreover, outputs can easily be compared in their “stringified” version using a generic information theoretical measure Q, typically a string distance function.

Our goal is that the features that span the space shall be generic and capture different aspects of the boundary candidates. We, therefore, introduce two feature types: (1) WD captures the differences within a boundary candidate as the distance between the first and the second output; (2) U is a two-attribute feature that captures the uniqueness of a candidate based on the distance between the first (U₁) and second (U₂) output to the corresponding outputs of all other candidates in the set.¹¹ Considering Q as the distance measure chosen for the outputs, we define U and WD for a boundary candidate j ∈ BC, j = (out_j1, out_j2) as: ${\displaystyle W{D}_j=Q\left(j_1,j_2\right)U_j=\left(U_{j1},U_{j2}\right);\mathrm{ where }{U}_{jk}=\sum _{j^{\prime }\in BC}Q\left(j_k,j_k^{\prime }\right),\mathrm{k\; =\; 1,2}}$

To understand which combination of distance measures (Q) yields better clustering of boundary candidates, we conducted a screening study using three different string distances to measure the distance between the outputs, namely, strlendist, Levensthein (Lev) and Overlap Coefficient of length two.¹² These common metrics cover different aspects of string (dis)similarity, each with its own trade-off. For instance, strlendist is efficient but ignores the characters of both strings, whereas Overlap compares combinations of characters but disregards specific sequences in which those characters show up (e.g., missing complete names or identifiers); lastly, Lev is the least efficient but more sensitive to differences between the strings. Nonetheless, all three measures only consider lexicographic similarity and are not sensitive to semantics, such as synonyms or antonyms.

For simplicity, the screening study was done only on the bytecount SUT. Details of the screening study and examples of features extracted from boundary candidates are presented in Appendix B. Our screening study reveals that strlendist (WD) and Overlap (WD, U) is the best combination of features and distance measures that yield clusters of good fit with high discriminatory power. Choosing those types of models yields more clusters that can be differentiated with high accuracy, hence allowing for a more consistent comparison of cluster coverage between exploration strategies. Moreover, clearer clusters are also useful in practice, allowing AutoBVA to suggest testers with more diverse individual boundary candidates.

Formally, we create a feature Matrix M over boundary candidates of a SUT with each row i representing each attribute from the features over each boundary candidate j, as defined below.

${\displaystyle M=\left[\begin{array}{c}\hfill WD\hfill \\ \hfill U\hfill \end{array}\right],}$

For this experiment, each M has four rows, one per attribute in the chosen features: strlendist (WD), Overlap (WD), Overlap (U₁) and Overlap (U₂). The number of columns (j) varies depending on the number of candidates found per exploration approach and SUT. Since k-means clustering is a heuristic algorithm, we run the clustering 100 times on each SUTs feature matrix M to retain the model of clustering of best fit according to the Silhouette score. To evaluate the coverage of the boundary candidates (RQ2), we choose the clustering discriminating best, i.e., the one resulting in the most clusters based on the top five percentile Silhouette scores.

We improve overall clustering quality by selecting only a diverse subset of boundary candidates for the clustering. For that, we create an initial diversity matrix as of above using 1,000 randomly selected candidates.¹³ We then substitute the least diverse 100 candidates (based on the sum of all normalized diversity readings) with 100 of the remaining candidates. Until no more candidates exist, we repeat this step to receive M for clustering. In the second step, we assign all candidates not part of M to the cluster with the closest cluster centre. A positive side-effect of this procedure is that it is much more memory efficient (limiting matrix sizes roughly to 1,000 candidates × 4 features).

RQ1-Boundary candidate quantities

Table 4 summarises the number of common and unique boundary candidates found by the two search strategies, LNS and BCS of Investigation 1. For each SUT and search strategy, it shows results for the 30-second and the 600-second runs individually. For each time control, the mean and standard deviation, over the 20 repetitions, of the number of potential boundary candidates as well as the number of unique candidates, is listed. For example, we can see that BCS in a 600-second run for the Date SUT finds, on average, 897.4 +/- 82.6 PBCs out of a total of 45456 unique ones (found over all runs and search strategies). And overall, 7,276 of the total 45,456 PBCs were uniquely only found by BCS, i.e., in none of the LNS runs, any of these 7,276 were found.

We see that overall, AutoBVA produces a large number of boundary candidates with either exploration strategy. Except for bytecount, there is also a large increase in the number of candidates found as the execution time increases. While the number of candidates found per second does taper off also for BMI-class, BMI and Julia Date (from 166, 552, and 81 per second for the short runs to 117, 380, and 76 for the long runs, respectively), longer execution clearly has the potential to identify more candidates. Since the 20 times longer execution time for bytecount only finds one additional candidate (58 total versus 57 for 30 s), it might be useful to terminate search and exploration when the rate of new candidates found goes below some threshold.

For the bytecount SUT, only BCS finds a unique set of candidates, meaning that BCS also identified all boundaries identified by LNS. This means that only 14 (58–44) of all candidates found were found by LNS, even after 20 runs of 600 s, and BCS also found all of those. For the other SUTs, LNS clearly finds more candidates and more unique candidates, between 5 and 15 times more, depending on the SUT.

The boundary detection statistics for Investigation 2 are summarised in Table 5. Boundary candidates could be identified with both exploration strategies in roughly 76% of the exported SUTs and 70% of the non-exported SUTs. Even though BCS could cover more SUTs in numbers, the difference is not statistically significant. LNS detects significantly more candidates.

Thus, overall, LNS produces a higher quantity of boundaries. This is expected since it is a random search strategy with minimal local exploration. The effect can likely be explained by two reasons that may also interplay. First, the low algorithmic overhead of the LNS search method enables it to make more calls to the underlying SUT, given a fixed time budget. Second, a proportion of the BCS searches can fail and return no boundary candidates since the input landscape does not regularly lead to changes in output partition through single-dimensional mutations, i.e., to one input. However, the quantity of boundary candidates might not directly translate to finding more diverse and “better” boundary candidates; next, we thus consider RQs 2 and 3 addressing this issue.

RQ2-Robust coverage of diverse behaviours

Using our validity-value similarity clustering summarisation method, we obtained between six and 15 clusters in Investigation 1 across the different SUTs and validity groups (VV, VE and EE) with high clustering quality scores (see Table 6). We see that most clusters were differentiated in the VV group. No EE candidates were identified for BMI and BMI-class because of the lack of detected boundary candidates. We also find that, except for the clustering for BMI in group VE, all attempted clusterings had a high discriminate score.

We summarise the coverage of clusters per SUT, exploration strategy, and execution time in Table 7. For example, we can see that for the Julia Date SUT, after we merged all candidates found by any of the methods in any of the runs and clustered them, we found 11 clusters. In a 30-second run, LNS covered 4.9 +/- 0.3 of them and covered one cluster that was not covered by BCS (in a 30-second run), while in a 600-second run, BCS covered 7.5 +/- 0.8 clusters and covered six that were not covered by LNS (in a 600-second run).

LNS shows consistent but modest cluster coverage growth with increasing running time. In other words, on average, boundary candidates were found using LNS to cover one or two more clusters when increasing the execution time from 30 s to 10 min. In contrast, BCS shows more cluster coverage improvement over time, where five additional clusters were covered when searching for 10 min, both for BMI classification and Julia Date. It shows no such growth for bytecount or BMI. Still, it has also “saturated” for these SUTs already after 30 s, i.e., it has covered the total number of clusters found after 30 s and thus has little to no potential for further improvements.

In many cases, BCS and LNS cover the same clusters, but some exceptions exist. For instance, only BCS found boundaries between the valid-error and error-error partitions for bytecount-clusters 7 and 8 (see ‘Bytecount’). In contrast, considering the 30-second search, only LNS identified candidates in BMI classification that cover the transitions between (underweight, normal) and (normal, overweight)-clusters 8 and 13 (see Section ‘BMI classification’). However, with increased execution time, BCS was the only strategy to find unique clusters (final column of Table 7). Particularly, if we look at Julia Date (10 min), BCS covers six unique clusters-including two clusters with “valid” outputs but unexpectedly long month strings. In RQ3, we further explain and compare these clusters for each SUT and argue their importance.

The comparisons above highlight the trade-off between time and effectiveness of the exploration strategies. Overall, LNS can be more effective in covering clusters in a short execution (BMI-class and Julia Data), but this is not always the case (e.g., bytecount). And with more execution time, BCS generally catches up to LNS (BMI-class) and often surpasses it (bytecount and Julia Date) both on average and in the number of uncovered unique clusters. Clearly, attributes of the SUTs will affect cost-effectiveness, e.g., the number of arguments in the input, (complexity of) specification, or the theoretical number of clusters that could be obtained to capture boundary behaviour.

From Table 7, we also note that the standard deviations are typically low, so the method is overall robust to random variations during the search. Still, we do note that the best method for Julia Date (BCS) only finds 7–8 of the total 11 clusters. This is not so for the other three SUTs, where it tends to find all of the clusters in a 600-second run.

For Investigation 2, Table 8 details the success rates of the boundary candidate summarisation. Summarisation is considered failed for a validity group where no boundary candidates could be found in that group. For instance, for 85% of the SUTs, no VE boundaries were found; thus, no summarisation could be attempted. The separation into good and bad summarisations is entirely based on the Silhouette score, where SS ≥ 0.6 is considered good, otherwise bad. While there are differences between the exported and non-exported SUTs, nothing sticks out, except that the exported functions have a lower overall summarisation success rate, with 61% of the SUTs, compared to the 68% of the non-exported ones. For all Validity Groups, good clusterings far outnumber the bad ones. It should be noted that the total numbers for good clustering in Table 8 counts those SUTs for which good quality clustering could be obtained for at least one validity group - a number smaller than the sum over all high-quality clusterings over all validity groups. Given that candidates could be detected for roughly 70% of the SUTs (see RQ1), and 64% could successfully be summarised, summarisation seems successful in separating the candidates in most cases.

Table 9 looks at the exploration strategies that impact boundary detection by validating how many clusters could be covered per 30 s run (average and standard deviation). The finding capabilities are relatively stable for both strategies, but the simpler LNS detects most candidates for these short runs over the board.

RQ3-Identifying relevant boundaries

While the main focus here will be on Investigation 1, one subsection is devoted to the findings of the broader Investigation 2 and the generalizability of AutoBVA.

None of the SUTs in Investigation 1 have a formal specification to which we can compare the actual behaviour of the implementations as highlighted by the identified boundary candidates. Thus, we cannot judge if any of the identified boundary candidate pairs indicate real faults.

Also, in practice, even if there was a formal specification, it might not be entirely correct or incomplete, i.e., there might be situations/inputs for which it does not fully specify the expected behaviour. Human judgment would then be needed to decide what, if anything, would need to be updated or changed in response to unexpected behaviour uncovered during testing. In industry, it is more common with informal specifications consisting of requirements in natural language, which can further exacerbate these problems. However, a relative benefit of automated black-box boundary exploration with the techniques proposed here is that they can potentially help identify several of these problems in the specification (incompleteness or even incorrectness), the implementation (bugs), and/or both. And even if no issues are identified, our proposed techniques can help strengthen the test suite.

Below, we go through each SUT, in turn, and manually analyse the boundary pairs identified and if they actually did uncover relevant (expected and unexpected) behaviour or even indicate actual faults. We used the clusters identified by the summarisation process (see RQ2 above) as the starting point. For clusters of a size smaller than 50 boundary pairs, we went through all of them. For larger clusters, we randomly sampled pairs, stratified by the total size of the outputs and analysed them, from smaller sub-groups to larger ones until saturation, i.e., looking at least at 50 pairs and going on further until no new, interesting or unexpected behaviour was found. For additional detail, we also calculated program derivative values using the Overlap Coefficient (based on 2-grams) function as output distance and checked all top-10 ranked pairs, per cluster. In the following, we highlight the critical findings per cluster and SUT.

To support our reporting on the manual analysis, we extracted tables with cluster representatives (see Tables 10–13). Unfortunately, some table entries had to be shortened for brevity. The original values and details can be found as part of the reproduction package (https://doi.org/10.5281/zenodo.7677012). Since the answers to RQ1 and RQ2 above indicated that BCS was sometimes more effective (even if not always as efficient as LNS), the tables have a column showing how many of the total candidates per cluster were found by BCS.

Boundary candidates types and tester actions

Table 17 shows five actions a tester can typically take concerning one or a set of similar boundary candidates. The columns to the right show the artefacts likely to be affected in each case: the specification, the implementation, or the test suite.

A Skip action typically means that either the tester does not find the candidate(s) relevant/useful, or it is handled correctly by the implementation, clear enough in the specification, and tested well enough. Even in the latter case, it is crucial that an automated testing method can also identify boundaries of this type; if it frequently misses them, a tester might lose confidence in the tool. An ExtendTests action would happen when the identified behaviour is correct according to the specification but not yet well covered by the test suite. The Debug action occurs when a fault exists in the implementation, i.e., the specification describes the correct behaviour, but the implementation does not comply. The Clarify situation is a consequence of a behaviour that is intended but not explicitly specified. Finally, the Refine action is a consequence of a boundary that is not desired but when the specification is either incorrect or incomplete. In this situation, we must clarify the specification and debug/fix the implementation. For the last actions, the test suite will also typically be extended to ensure the same problem does not occur in the future.¹⁶

One type of boundary candidate we identified in the manual analysis was the under-specified input range, i.e., for inputs that are not expected, which leads to unexpected behaviour. One example, for BMI-class, was the candidate with outputs that move from Underweight to Severely Obese by a change in body weight from 0 to 1 kg for a person with a height of 1 cm. Another example is Julia Date, with a very large year input that led to nonsensical outputs that showed no resemblance to the inputs, likely due to some overflow in the internal representation of date values. This type of boundary would typically lead to either a Debug or a Refine action, depending on whether an update of the specification is meaningful/necessary or not.

Other problems were identified concerning the bytecount SUT. We used this SUT since it is known as the most copied (Java) code snippet on StackOverflow and has several bugs as described in Lundblad (2019): a rounding error, correctness for large values over 64 bits, and the fact that negative inputs are not appropriately handled. As described in our results above, the latter was clearly identified and can be seen as an under-specified problem. It likely leads to a Refine action since it is unclear how to handle negative inputs from the specification. The correctness for large values is still a problem even though the Integer type also includes 128-bit integers and the BigInt type to handle arbitrarily large integers in Julia. However, the problem manifests in the boundary candidates that lead to BoundsErrors since they also try to access larger byte suffix categories after exabytes (“EB”). A Refine or at least a Debug action is probably called for. The boundary candidates capturing the rounding error of bytecount, e.g., kB to MB, were all identified in the searches (cluster 5 in Table 10) and represent wrongly positioned boundaries. While it is unclear if the rounding error can be directly spotted, even given the boundary examples in the cluster, it is contained in all the cluster boundaries. A diligent tester is likely to investigate critical boundaries in depth, and the fact that they are all in the same cluster can help make this checking more likely. This will lead to a Debug action.

Threats to validity

Our analyses involve simple comparisons through descriptive statistics (RQ1 + RQ2) and qualitative analysis of the clusters (RQ3). We mitigate conclusion validity threats in our quantitative comparison by focusing on how the sampling strategies complement each other rather than simply comparing which is better.

By analyzing the uniqueness of candidates and clusters, we identified essential trade-offs. For RQ3, we did not exhaustively inspect all candidates in larger clusters. Hence there is a risk we miss important ones. We mitigate this risk by random sampling and iterative analysis until saturation in conclusions.

We operationalize the constructs of our proposed approach with a limited set of treatments and dependent variables. Consequently, the primary construct validity threats are our choices of (1) configuration (string length for PD, mutation operators, sampling methods, etc.), (2) SUTs with integer inputs, (3) measures of uniqueness, clustering, and coverage/inspection of clusters. However, given the novelty of the approach, we argue it would be very difficult to analyze the overall results if we used more advanced or just more choices for these contextual factors. Future work is needed to understand performance implications better as these factors vary.

We use clustering as an attempt for summarising boundary candidates into a presentable subset for testers.

To a certain degree, the “relevance/importance of a boundary” is subjective. Hence an optimal or perfect summarisation is hardly attainable. It should also be mentioned that in a real-world situation, a single search run with sufficiently many boundary candidates should suffice to do both the clustering and summary extraction - which will be faster and simpler than running many times over and building an overall clustering model over the entire set of found candidates. However, the number of clusters found in each run was smaller than the total number for some SUTs, even for BCS, so the effect of execution time needs further study in the future.

We mitigate internal validity threats by doing pilot studies and testing our instrumentation. The screening studies helped us to identify feasible and consistent strategies for sampling candidates and clustering outputs instead of going for arbitrary choices. Encouraging was also that our method revealed a fault in our implementation of the BMI value function (Table 13)¹⁷ Regarding verifying the search procedure and system itself, we use automated unit tests in Julia to mitigate faults in our implementation (https://docs.julialang.org/en/v1/stdlib/Test/#Basic-Unit-Tests).

Lastly, our results cannot be generalised beyond the scope of our experimental study, i.e., finding boundaries for unit-level SUTs that take integers as input. On the one hand, various aspects in AutoBVA indicate it might be generally applicable, such as the concept of the program derivative and its basis on string distances rather than type-specific distance function. However, more general test data generation algorithms, mutation operators, and distance functions will need to be experimented with to increase the external validity.