Machine learning meets visualization – Experiences and lessons learned

2.1 Visual parameter space analysis for ML

The creation of an ML model often involves setting so-called hyper-parameters such as the number of layers, number of epochs, or the number of dimensions in latent space [31], [65]. To set these parameters, a common approach is to rely on trial and error. Specifically with larger parameter spaces, however, trial and error can become a tedious, unsystematic, and error-prone process; analysts easily forget what exact parameterizations they looked at five minutes ago. A more systematic approach is to instead employ VPSA, i. e., to sample a larger collection of parameter values and visualize the space for the user to explore. If the objectives are well-defined, these steps might also be automated [71]. Yet, often these types of problems are ill-defined and, as such, necessitate a human-in-the-loop [20], [59]. Multiple different objectives might need to be weighted, objectives might not be even clearly characterized yet, and uncertainties and sensitivities might further influence a decision. To mitigate these problems, we have leveraged VPSA, which primarily has been used for classical simulation models in the past but works similarly well, in our experience, for ML models [55].

Figure 1

The VisCoDeR system offers an interactive color legend for DR algorithms and their parameterizations (a), meta-parameterization and control over the projection technique (b), meta map of 1004 DR results with activated proximity visualization (c), and a detailed view of the selected DR result (d) [15], allowing for visual parameter space analysis of DR methods.

As an example, let us assume that we want to find a good dimensionality reduction (DR) model for a given dataset. DR models take multi-dimensional data as input and then output a lower-dimensional projection of the data, either for further usage in the ML pipeline or for the purpose of visualization. In terms of visualization, the output is usually 2D and is typically represented in the form of a scatterplot. Still, before we are able to do that, we need to select among many different DR techniques, such as UMAP [39], t-SNE [63], LLE [48], ISOMAP [62], MDS-based methods [33], PCA [30], etc. [21], and for some of them we also need to set additional parameters. While quantitative error metrics exist, in the end, picking a good visual 2D projection is often still in the eye of the beholder, calling for a human-in-the-loop approach (at least still at the time of writing this article).

Applying VPSA to the space of DR models and parameters, we developed VisCoDeR [15],^[2] as shown in Figure 1. The central idea of VisCoDeR is a 2D overview scatterplot, which is called a meta map, that encodes each instance of a parameterized DR model output as a point in the view (see Figure 1 (c)). The meta map is based on a DR model in turn again (here t-SNE, but others are possible). As such, if two points are close in the view, the respective DR instances are similar to each other (see Figure 1 (d)). The distance is computed based on the visual similarity of two DR instances (that is, 2D scatterplots), imitating the human perception of the DR output. Like many Visual Analytics tools, VisCoDeR also provides linking and brushing functionality between the different views, most importantly between the meta map (outputs) and the respective input parameter space (see Figure 1 (a)).

Using the tool, analysts can now gain insights into the role of each parameter for the different DR models. For instance, t-SNE [63] requires users to set two parameters, namely perplexity (relates to number of nearest neighbors) and epsilon (learning rate) [68]. The question is now how these parameters affect the visual output of t-SNE and how sensitive they are to changes. To that end, we sampled 1,000 different t-SNE parameterizations, created the respective 2D scatterplots, and visualized them in the VisCoDeR meta map. Using linking and brushing between the meta map and input parameter space, we can now smoothly hover over the two parameters, see Figure 1(a). This interaction reveals, that perplexity has a smooth sensitivity, that is, the visual output changes gradually with the change of the perplexity parameter, see the center of Figure 1(c) where the color encoding for perplexity changes from orange to pink. In this area, t-SNE scatterplots with the same or similar perplexity are close in the meta map. On the other hand, we can see that epsilon, which is encoded by the brightness channel, has seemingly little impact on the t-SNE outcome as t-SNE plots with the same epsilon appear all over the place in the meta map.

In this example, VPSA helped us to conduct a sensitivity analysis of t-SNE on a given dataset. Other tasks that are supported are multi-objective optimization, uncertainty analysis, partitioning, outlier detection and fitting [55]. VPSA is by no means restricted to DR models, but works on all sorts of input-output-based models, which is the case for many ML approaches. We, for instance, also used to make the exploration of hyper-parameters of classification models more systematic [28], and also for deep learning models [27].

Figure 2

FDive [17] learns to classify interesting from uninteresting data items via an iterative active learning process. It chooses the best-fitting feature descriptor and distance function to reflect the users’ notion of interestingness as a distance measure. (1) Users select interesting data items labeling a set of query items. (2) The labels determine similarity measures, i. e., feature descriptor and distance function. (3) The system uses the selected distance measure to learn a classification model. The user can explore and refine the model by supplying more labels for uncertain data items neat the decision boundaries. © IEEE 2019.

2.2 Explainable AI with visual interactive learning

Another important use case for leveraging VIS of ML is Active Learning (AL). In AL, only few labels are available, and a user needs to be prompted to provide more and more labels on the way, following a strategy that maximizes the effectiveness of the labeling process by requesting labels for data items with an uncertain classification result. The new labels are used in subsequent steps to improve the underlying model. The model is trained on sparse data. Thus the model needs to be assessed for quality and convey its reasoning, especially when the AL classifier is prompting for a label. As such, there is an intrinsic need for human-computer interaction. While classical Active Learning leverages primarily simple labeling interfaces, VIS allows much more sophisticated ways for users to interact with the models, supporting not just active but also inter-active learning [49].

One way to use VIS in the AL pipeline is to support users in understanding the space of (labeled and unlabeled) instances, with the idea that the visualization can help users to select, implicitly or explicitly, further items for labeling. To that end, we developed a system called FDive [17], which learns to distinguish interesting from uninteresting data through an iteratively improving classifier (see Figure 2). To do so, it uses a set of feature descriptors and distance functions to represent the similarity of data items as a distance measure. This allows for interpretable distances, where analysts can derive which data properties are essential for the classification. First, users can express their preference for a data item by labeling a set of items prompted by the system (see Figure 2 (1)). Second, these labels are used to choose a feature descriptor and distance function combination by their ability to represent the user’s preference through distance relations (see Figure 2 (2)).

Finally, the system applies the selected combination to learn a Self-Organizing Map-based classifier (see Figure 2 (3)). This special type of model can be explored and refined by supplying labels at any position. However, uncertain classifications are highlighted to guide users toward items with uncertain labels. This process can be repeated to improve the classification and assess it. We applied our tool to connectomics, a sub-filed of neurology where scientists try to map out neuronal connections using electron microscopy images to detect neuronal synapses. There are roughly one billion synapses in 1 mm 3 of a brain tissue. Thus, methods of automatically detecting images depicting neuronal synapses are necessary.

With FDive, the analyst can label a small subset of images determined by the system, whether they show neuronal synapses or not (see Figure 2 (1)). The system selects the best separating measure for the labeled images. The analyst can observe the implied data space through an MDS protection of the dataset (see Figure 2 (2)). We found that descriptors focusing on the image texture generally worked the best for this task, since the system immediately converged on a texture-based description of the images. Finally, the analyst can explore the model to determine the classifier’s quality.

The classifier has a nested hierarchy, allowing for a more and more fine-grained classification. Clusters with an uncertain classification are highlighted to guide the analyst towards them. The analyst can supply extra labels for the images in those clusters to improve the classifier. We observed that images containing similar cell structures are clustered, including those depicting a neuronal synapse (see Figure 2 (3)). The analyst repeated that process seven times, iteratively supplying more labels until the analyst considered the model adequate while converging on a specific texture descriptor.

It is an interesting question when to use which type of interface. Simple problems such as image classification with a few classes probably are dealt well with a simple labeling interface that is prompted by the algorithm on demand. Complex and ill-defined domain problems, such as connection classification in neurology, on the other hand, might benefit from close integration of labeling and analytical components. First studies are available that explicitly seek to characterize this space [7], [8] and understand how to visually encode the data in such cases [25].

3.1 Class separability for scatterplots

Following a series of earlier empirical work [11], [53], [57], [58], the first task we were interested in was judging class separability in scatterplots [3], [54]. In Figure 3, (a) and (b) show two separable classes while (c) and (d) show two non-separable classes that are easy for humans to differentiate. The basic idea was then to use ML to train a model that imitates and predicts these human judgments and that rates class separability like humans do. Having such a model would then allow us to automatically spot “interesting” views in large scatterplot matrices [60], [69], guide the selection of DR methods [57], or even search dimensional subspaces [61]—in a way that humans would.

The idea of modeling class separability originated from Sips et al. [60], who proposed hand-crafted measures for that purpose, in a similar vein as the venerable Scagnostics measures [69]. When using these measures for a large collection of 816 scatterplots, we, however found the generalization to other (“unseen”) datasets was poor [58]. As generalizability is a strength of ML, we thus were wondering in how far ML could lead to better models and measures for class separability. With that in mind, we used a carefully collected and cleaned dataset of expert judgments [57] and trained a binary classifier (separable or non-separable) for class separation in scatterplots [54]. Bootstrapping was used to ensure generalizability to other unseen datasets.

Using this approach, we then proposed and automatically evaluated 2002 systematically generated separation measures/models [3]. Using these in the model selection phase of the framework, we indeed found that many of our novel measures substantially outperformed the best state-of-the-art measures. While the best state-of-the-art measure had an accuracy of 82.5 % (bootstrapped AUC-ROC), the best new measure had an average accuracy of 92.7 %, and overall, 58 % of the new measures outperformed the traditional best measure. Of course, our work here is just a starting point. Our proposed model is relatively simple and is still far from nuanced human perception in judging class separability. Still, already with the simple model, we got very good performance, indicating that an ML approach might be a good fit here.

Figure 4

SineStream [12] minimizes the impact of sine illusion effects originating from strong slopes. The arrows highlight parts of the streamgraph, where SineStream [12] represents the thickness of layers more accurately than methodologies by Bryon & Wattenberg [13] and by Bartolomeo & Hu [4]. It improves the readability of streamgraphs by aligning each layer’s orthogonal and vertical orientations. © IEEE 2022.

3.2 Cluster detection in scatterplots

Another widespread and closely-related task in scatterplots is identifying classes in analysis scenarios where the data is unlabeled [11]. Here the analyst has to deal with monochrome scatterplots. The goal is to identify cluster structures, meaning that groups of data items are separated visually, either by an empty area where no data items are located or by differences in density for overlapping or nested clusters. There are multiple approaches to tackle this problem. Firstly, one could apply clustering algorithms, such as DBSCAN [32] or CLIQUE [23] to the scatterplot visualization (i. e., image space), trying to detect well-separated clusters. Secondly, this problem also has been tackled with the idea of classical quality measures, such as the “clumpiness” Scagnostics measure [69]. However, we were wondering in how far such heuristic approaches can capture the human understanding of a cluster, which might include the notion of non-globular clusters, clusters that are only separated visually by a small gap, or clusters that are nested, differing only in density. We hypothesized that a more nuanced approach could be useful to capture the notion of what defines a visual cluster from a human perspective and whether ML could provide a better solution for this problem.

We thus proposed ClustMe [1], an ML-based approach to the idea of “clumpiness”. We built ClustMe based on data collected from a study with 34 participants. The participants judged the cluster patterns in 1000 scatterplots of synthetically generated datasets. We generated the datasets by adapting the parameters of a simple Gaussian Mixture Model with two components. To quantify the number of clusters in a scatterplot, the participants counted the number of clusters they could see.

We then created ClustMe by choosing the model that best predicts the human intuition. We performed another study to evaluate ClustMe, in which 31 study participants ranked 435 pairs of scatterplots of real-world and generated data in terms of perceived cluster patterns. We then compared the performance of ClustMe to four other state-of-the-art clustering measures using this data. We also included the “clumpiness” measure of Scagnostics in this comparison. The results showed that ClustMe, out of all measurements, was most consistent with the human rankings. This work again showed evidence that an ML-based approach can mimic classical quality measures and, as such, can be used to improve VIS.

3.3 Example beyond scatterplots

While scatterplots are a good starting point, practical visualization, of course, offers many more other types of visual encodings. These can also benefit from a similar modeling approach that helps to automatically optimize parameters of visualizations [40].

We, for instance, developed SineStream [12] to push forward the current state-of-the-art for streamgraph visualizations (see Figure 4). Following a similar process as on class separability above, SineStream was based on the main idea of improving readability by minimizing sine illusion effects in streamgraphs. Such effects reflect the tendency of humans to take the orthogonal rather than the vertical distance between two curves as their distance. In SineStream, we minimize this illusion by optimizing the ordering of the different streams. Quantitative experiments and user studies demonstrated that SineStream improved the readability and aesthetics of streamgraphs compared to state-of-the-art methods.

In comparison to the approaches in Sections 3.1 and 3.2, we, however did not use ML in this example, but classical optimization (simulated annealing in this case). In vain, the idea to automatically optimize visual parameters based on human perception is the same, though. An interesting question for future work is, of course, how far ML might be able to provide further improvements for such approaches as well.

4.1 Challenges

Based on our experience working on these topics, we now want to take a step back and reflect on some challenges in combining ML and VIS.

4.2 Limitations

We would like to explicitly remind the reader that these above contributions are only from our own experiences. We would like to encourage further collaborations between the VIS and ML approaches with this article. The presented approaches should not be seen as a comprehensive characterization of all combinations of VIS and ML, but only as some examples. Of course, there are many other ways to combine ML and VIS.

From the approaches presented in this article, we would like to highlight two concrete limitations. First, we see one current limitation in the area of explaining ML models through effective interactive interfaces [15], [17]. Currently, most systems are based on the linking and brushing technique. While this is a strong technique, for effective integration of VIS and ML, we will also need semantic interaction [19], interactive learning [7], and maybe even in devices other than mouse and keyboard [38].

Second, in using ML to improve VIS, we found that there are limitations to modeling human perception. ClustMe [1] and SepMe [3] are two approaches that heavily rely on the data gathered in studies. With these ML techniques, we are steering into the domain of User Modeling. Such a model can only express a general notion for the task at hand since ML techniques can only aggregate the results of a study and thus can differ from an individual’s perception and judgment.