A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges

2.1.1 Automata, Instructions, and Schemas.

Finite-state automata and grammars. A deterministic finite-state automaton is specified by defining a finite set of states, a finite set of allowed input symbols, a transition function (defines all transitions in the automaton), a start state, and a finite set of accepting states. The current state can change in response to an input. The joint current state and input symbol uniquely determine the next state of the automaton.

An intuitive example of an automaton controlling the logic of a turnstile is presented in Figure 1. The set of input symbols is { “Token”, “Push” } and the set of states is { “Unlocked”, “Locked” }. The state diagram in Figure 1 can be used to derive the transition function.

Fig. 1.

View Figure

HDC/VSA-based implementations of finite-state automata were proposed in [307, 425]. Random HVs are assigned to represent states (\(\mathbf {u}\) for “Unlocked”; \(\mathbf {l}\) for “Locked”) and input symbols (\(\mathbf {t}\) for “Token”; \(\mathbf {p}\) for “Push”). These HVs are used to form a compositional HV \(\mathbf {a}\) for the transition function. The transformation is similar to the one used for the directed graphs in Section 3.5.1 in [222]. However, the HV representing the input symbol for the automaton is bound to the edge HV that corresponds to the binding of the HVs for the current and the next state. For instance, going from “Locked” to “Unlocked” upon receiving “Token” is represented as (1) \(\begin{equation} \mathbf {t} \circ \mathbf {l} \circ \rho (\mathbf {u}). \end{equation}\) Given the HVs of all transitions, the transition function \(\mathbf {a}\) is represented as their superposition: (2) \(\begin{equation} \mathbf {a} = \mathbf {p} \circ \mathbf {l} \circ \rho (\mathbf {l}) + \mathbf {t} \circ \mathbf {l} \circ \rho (\mathbf {u}) + \mathbf {p} \circ \mathbf {u} \circ \rho (\mathbf {l}) + \mathbf {t} \circ \mathbf {u} \circ \rho (\mathbf {g}). \end{equation}\)

The next state is obtained by querying \(\mathbf {a}\) with the binding of HVs of the current state and of the input symbol, followed by the inverse permutation of the resultant HV that returns the noisy version of the next state’s HV.¹ For example, if the current state is \(\mathbf {l}\) and \(\mathbf {p}\) is received, then (3) \(\begin{equation} \rho ^{-1}(\mathbf {a} \circ \mathbf {p} \circ \mathbf {l}) = \mathbf {l} + \mathrm{noise}. \end{equation}\) This noisy HV is used as the query for the item memory to obtain the noiseless atomic HV \(\mathbf {l}\).

Transformations of pushdown automata and context-free grammars into HVs have been presented in [12]. A proposal for implementing Turing machines and cellular automata was given in [204].

In [226], the Holographic Reduced Representations (HRR) model was used to represent Fluid Construction Grammars, which is a formalism that allows designing construction grammars and using them for language parsing and production. Another work related to parsing is [392], which presented an HDC/VSA implementation of a general-purpose left-corner parsing with simple grammars. An alternative approach to parsing with HVs using a constraint-based parser has been presented in [21].

A related research direction was initiated by beim Graben and colleagues [11, 12, 13, 421]. It concerns establishing a mathematically rigorous formalism of Tensor Product Representations using the concept of a Fock space [83]. The studies focus largely on use-cases in computational linguistics, semantic processing, and quantum logic. The usage of the Fock space formalism for formulating minimalist grammars was presented in [11]. Syntactic language processing as part of phenomenological modeling was reported in [13]. An accessible and practical entry point to the area can be found in [421].

Controllers, instructions, schemas. In [249], using the Multiply-Add-Permute model (MAP), it was demonstrated how to manually construct a compositional HV that implements a simple behavior strategy of a robot. Sensor inputs as well as actions were represented as atomic HVs. Combinations of sensor inputs as well as combinations of actions were represented as a superposition of the corresponding atomic HVs. HVs of particular sensor input combinations were bound to the corresponding HV of action combinations. The bound HVs of all possible sensor-action rules were superimposed to produce the compositional HV of the robot controller. Unbinding this HV with the current HV of sensor input combinations results in the noisy HV of the proper action combination. This idea was extended further in [297] through a proposed algorithm to “learn” a compositional HV representing a robot’s controller using the sensor-actuator values obtained from successful navigation runs. This mode of robot operation is known as “learning by demonstration.” It was realized as the superposition of the current “controller” HV with the HV corresponding to the binding of the sensor-actuator values—in case the current sensor HV was dissimilar to the ones already present in the “controller” HV. Another work studying robot navigation is [269], which investigated a number of ways to form compositional HVs representing sensory data and explored the integration of the resultant HVs with neural network instead of the “controller.”

In [39], using HRR, instructions were represented as a sequence of rules and rules as a sequence of their antecedent and consequent elements. Multiplicative bindings with position HVs were used to represent the sequence. Antecedents and consequents, in turn, were represented as HVs of their elements using binding and superposition operations. This approach was used as a part of instruction parsing in a cognitive architecture [394]. In [245], a proposal was sketched for a HDC/VSA-based processor, where both data and instructions were represented as HVs.

In [295], HVs for the representation of “schemas” in the form \(\lt\)context, action, result\(\gt\) were used. A more general approach for modeling the behavior of intelligent agents as “functional acts” was considered in [337] for Sparse Binary Distributed Representations (SBDR) (see also Section 3.2.2). It is based on HVs representing triples \(\lt\)current situation, action, resulting situation\(\gt\) (which essentially correspond to “schemas”), with the associated evaluations and costs. Finally, it is worth recalling that, in general, data structures to be represented by HVs do not have to be limited to “schemas.” For example, a recent proposal in [104] suggested that HVs are well suited for forming representations of the JSON format that can include several levels of hierarchy.

Membership query and frequency estimation. Section 3.1.2 in [222] presented the transformation of sets and multisets into HVs. When implemented with the SBDR model, it becomes evident that Bloom Filters [19] are a special case of HDC/VSA [223], which have been used in a myriad of applications involving the membership query. It is beyond the scope of this survey to overview them all; therefore, the interested readers are refereed to a survey in [400]). When implementing the transformation of multisets via the Sparse Block Codes model, a similar connection can be made to count-min sketch [50] that is used commonly for estimating frequency distributions in data streams (some applications are presented in [50]). The use of the HDC/VSA principles for constructing hash tables has been recently considered in [135].

2.1.2 Transmission of Data Structures.

Communications. The main motivation for using HDC/VSA in the communication context is their robustness to noise due to the distributed nature of HVs. Let us consider three similar but varying applications of HDC/VSA.

In [165], it was shown how to use Binary Spatter Codes (BSC) (Section 2.3.6 in [222]) in collective communication for sensing purposes. Multiple devices wirelessly sent their specific HVs representing some of their sensory data (the paper used temperature as a show-case). It was proposed to receive them in a manner that implements the superposition operation. This superposition HV was then analyzed by calculating the \(\text{dist}_{\text{Man}}\) of the normalized superposition and the atomic HVs and comparing with a threshold. For instance, the case that a particular temperature was transmitted could be detected. Another version of analysis allowed checking how many devices had been exposed to a particular temperature. The proposed communication scheme does not require a control mechanism for getting multiple access to the medium. So it can be useful in scenarios where there are multiple devices that have to report their states to some central node. Recently, it has been shown how such over-the-air superposition can be used for on-chip communications to scale up the architecture with multiple transmitters and receivers [126]. This has been done by carefully engineering modulation constellations, and it paves the way for a large number of physically distributed associative memories (as wireless-augmented receivers) to reliably perform the similarity search given a slightly different version of a query HV as their input.

In [211], BSC was used in the context of medium access control protocols for wireless sensor networks. A device was forming a compositional HV representing the device’s sensory data (by the superposition of multiplicative bindings), which was then transmitted to the communication medium. It was assumed that the receiver knew the atomic HVs and, thus, could recover the information represented in the received compositional HV. The application scope of this approach is for scenarios where the communication medium is very harsh so that a high redundancy of HVs is useful for reliably transmitting the data.

In [199], it was proposed to combine forward error correction and modulation using Fourier HRR (Section 2.3.5 in [222]). The scheme represented individual pieces of data by using complex-valued HVs that were then combined into a compositional HV using permutation and superposition operations. Unnormalized complex values of the compositional HV were transmitted to the communication medium. The iterative decoding of the received compositional HV significantly increased the code rate. The application scope of this scheme is robust communication in a low signal-to-noise ratio regime. The scheme at a lower coding rate was compared to the low-density parity check and polar codes in terms of achieved bit error rates, while featuring lower decoding complexity. To improve its signal-to-noise ratio gain, a soft-feedback iterative decoding was proposed [140] to additionally take the estimation’s confidence into account. That improved the signal-to-noise ratio gain by 0.2 dB at a bit-error-rate of \(10^{-4}\). In further works, the scheme has been applied to collision-tolerant narrowband communications [150], massive machine type communications [151], and near-channel classification [140].

Distributed orchestration. Another use-case of HDC/VSA in the context of transmission of data structures is distributed orchestration. The key idea presented in [382, 383] was to use BSC to communicate a workflow in a decentralized manner between the devices involved in the application described by the workflow. Workflows were represented and communicated as compositional HVs constructed using the primitives for representing sequences (Section 3.3 in [222]) and directed acyclic graphs (Section 3.5.2 in [222]). In [384], the approach was implemented in a particular workflow system: Node-RED. In [8], the approach was extended further to take into account the level of trust associated with various elements when selecting services.

2.1.3 String Processing.

In [329], it was proposed to obtain HVs of a word using permutations (cyclic shifts) of HVs of its letters to associate a letter with its position in the word. The conjunction was used to bind together all the obtained letter-in-position HVs. HVs of words formed in such a way were then used to obtain n-grams of sequences of words using the same procedure. The obtained HVs were used to estimate the frequencies of various word sequences in texts to create a model of human text reader interests. Note that here the conjunction result is somewhat similar to all input HVs.

An interesting property of sequence representation using permutations of its element HVs (Section 3.3 in [222]) is that the HV of a shifted sequence can be obtained by the permutation of the sequence HV as a whole [206, 213, 284]. This property was leveraged in [213] for searching the best alignment (shift) of two sequences, i.e., the alignment that provides the maximum number of coinciding symbols. This can be used, e.g., for identifying common substrings. Such a representation, however, does not preserve the similarity of symbols’ HVs in nearby positions, which would be useful for, e.g., spell checking. This can be addressed by, e.g., extending the permutation-based representations as in [215], where the resultant compositional HVs were evaluated on a permuted text, which was successfully reconstructed. An approach to transforming sequences into sparse HVs from [333], which preserves the similarity of symbols at nearby positions and is shift-equivariant, was applied to the spellchecking task.

An algorithm for searching a query string in the base string was proposed in [315] and modified in [204]. It is based on the idea of representing finite-state automata in an HV (see Section 2.1.1). The algorithm represented the base string as a non-deterministic finite-state automaton [328]. The symbols of the base string corresponded to the transitions between the states of the automaton. The automaton in turn was represented as a compositional HV. The automaton was initialized as a superposition of all atomic HVs corresponding to the states. The query substring was presented to the automaton symbol by symbol. If after the presentation of the whole substring the automaton appeared in one of the valid states, then this indicated the presence of the query substring in the base string.

In [200], the MAP model was used for DNA string matching. The key idea was as in [173]: the base DNA string was represented by one or several HVs containing a superposition of all n-grams of predefined size(s). HVs of n-grams were formed by multiplicative binding of appropriately permuted HVs of their symbols (Section 3.3.4 in [222]). A query string was considered present in the base DNA string if the similarity between its HV and the compositional HV(s) of the base DNA string was higher than the predefined threshold. The threshold value determined the balance between true and false positives, similarly to Bloom filters (see Section 3.1.2 in [222]). The approach was evaluated on two databases of DNA strings: escherichia coli and human chromosome 14. The main promise of the approach in [200] is the possibility to accelerate string matching with application-specific integrated circuits due to the simplicity and parallelizability of the HDC/VSA operations.

2.1.4 Factorization.

The resonator networks [88, 197] were proposed as a way to solve factorization problems, which often emerge during the recovery procedure within HDC/VSA (Section 2.2.5 in [222]). It is expected that the resonator networks will be useful for solving various factorization problems, but this requires formulating the problem for the HDC/VSA domain. An initial attempt to decompose synthetic scenes was demonstrated in [87]. A more recent formulation for the integer factorization was presented in [203]. The transformation of numbers into HVs was based on the fractional power encoding [89, 90] (Section 3.2.1 in [222]) combined with log transformation, and the resonator network was used to solve the integer factorization problem. The approach was evaluated on the factorization of semiprimes.

2.2.1 Word Embeddings: Context HVs for Words and Texts.

The key idea for constructing context vectors is usually referred to as the “distributional semantics hypothesis” [130], suggesting that linguistic items with similar distributions have similar meanings. The distributions are calculated as frequencies of item occurrence in particular contexts by using a document corpus. For items-words the contexts could be, e.g., documents, paragraphs, sentences, or sequences of words close to a focus word. For example, the generalized vector space model [423] used documents as contexts for words in information retrieval. In computational linguistics and machine learning, vectors are also known as embeddings.

In principle, context vectors can be obtained in any domain where objects and contexts could be defined. Below, we will only focus on the context vector methods that are commonly attributed to HDC/VSA. They usually transform the frequency distributions into HVs of particular formats, which we will call context HVs.

Historically, the first proposal to form context HVs for words was that of Gallant, e.g., [29, 102, 103, 105]. These studies, however, did not become known widely, but see [134]. In fact, two most influential HDC/VSA-based methods for context HVs are Random Indexing (RI) [181, 366] and Bound Encoding of the Aggregate Language Environment (BEAGLE) [172].

Random Indexing. The RI method [181, 366] was originally proposed in [181] as a simple alternative to Latent Semantic Analysis (LSA) [246]. Instead of the expensive Singular Value Decomposition (SVD) used in LSA for the dimensionality reduction of a word-document matrix, RI uses the multiplication by a random matrix, thereby performing random projection (Section 3.2.3 in [222]). The Random Projection (RP) matrix was ternary (\(\lbrace -1, 0, 1\rbrace\)) and sparse. Each row of the RP matrix is seen as a “random index” (hence RI) assigned to each context (in that case, the document). In implementation, the frequency matrix was not formed explicitly, and the resultant context HVs were formed by scanning the document corpus and adding the document’s random index vector to the context HV of each word in the document. The sign function can be used to obtain binary context HVs. The similarity between unnormalized context HVs was measured by \(\text{sim}_{\text{cos}}\). The synonymy part of TOEFL was used as the benchmark to demonstrate that a performance comparable to LSA could be achieved but at lower computational costs due to the lack of SVD. In [312], it was proposed to speed up LSA by using RP before SVD as a preprocessing step. In [185], similarly to [105, 256], RI was modified to use a narrow context window consisting of only few adjacent words on each side of the focus word.

In [367], permutations were used to represent the order information within the context window. Further extensions of RI included generalization to multidimensional arrays (N-way RI) [368] and inclusion of extra-linguistic features [183]. The RI was also extended to the case of training corpora that include information about relations between words. This model is called Predication-based Semantic Indexing (PSI) [43, 45, 46]. PSI has been mainly used in biomedical informatics for literature-based discovery, such as identification of links between pharmaceutical substances and diseases they treat (see [46] for more details). Later, PSI was extended to the Embedding of Semantic Predications (ESP) model that incorporates some aspects of “neural” word embeddings from [274] and similarity preserving HVs (Section 3.2 in [222]) for representing time periods [416].

It is worth mentioning that the optimization-based method [397] for obtaining similarity preserving HVs from co-occurrence statistics can be contrasted to RI. RI also uses co-occurrence statistics but implicitly (i.e., without constructing the co-occurrence matrix). The difference, however, is that the RI is optimization-free and it forms HVs through a single pass via the training data. Thus, it can be executed online in an incremental manner, while the optimization required by [397] calls for iterative processing, which might be more suitable for offline operations.

Bound Encoding of the Aggregate Language Environment. There is an alternative method for constructing context HVs with HDC/VSA, known as BEAGLE [172]. It was proposed independently of RI and used HRR to form the HVs of n-grams of words within a sentence, providing a representation of the order and context HVs of words. Words were initially represented with random atomic HVs as in HRR. The context HV of a word was obtained from two “parts.” The first part included summing the atomic HVs of the words in the sentence other than the focus word for all corpus sentences. The second part was contributed by the word order HVs, which were formed as the superposition of the word n-gram HVs (with n between 2 and 7). The n-gram HVs were formed with the circular convolution-based binding of the special atomic HV in place of the focus word and the atomic HVs of the other word(s) in the n-gram. The word order in an n-gram was represented recursively, first by binding the HVs of the left and the right word permuted differently and then by binding the resultant HV with the next right word, again using permutations for the “left” and “right” relative positions. The total context HV was the superposition of the HVs for the two parts. The similarity measure was \(\text{sim}_{\text{cos}}\).

Later, [357] presented a modified version of BEAGLE using random permutations. The authors found that their model was both more scalable to large corpora and gave better fits to semantic similarity than the circular convolution-based representation. A comprehensive treatment of methods for constructing context HVs of phrases and sentences as opposed to individual words was presented in [283]. In a similar spirit, in [257] the HRR model was used to construct compositional HVs that were able to discover language regularities resembling syntax and semantics. The comparison of different sentence embeddings (including BEAGLE) in terms of their ability to represent syntactic constructions was provided in [196]. It has been demonstrated that context HVs formed by BEAGLE can account for a variety of semantic category effects such as typicality, priming, and acquisition of semantic and lexical categories. The effect of using different lexical materials to form context HVs with BEAGLE was demonstrated in [170], while the use of negative information for context HVs was assessed in [171]. In [194], BEAGLE was extended to a Hierarchical Holographic Model by augmenting it with additional levels corresponding to higher-order associations, e.g., part-of-speech and syntactic relations. The Hierarchical Holographic Model was used further in [191] to investigate what grammatical information is available in context HVs produced by the model.

Due to high similarity between BEAGLE and RI, both methods were compared against each other in [357, 358]. It was shown that both methods demonstrate similar results on a set of semantic tasks using a Wikipedia corpus for training. The main difference was that RI is much faster as it does not use the circular convolution operation. For placing the methods in the general context of word embeddings, please refer to [419].

[271] aimed to address the representation of similarity, rather than relatedness represented in context HVs by BEAGLE and RI. To do so, for each word the authors represented its most relevant semantic features taken from a knowledge base ConceptNet. The context HV of a word was formed using BSC as a superposition of its semantic feature HVs formed by role-filler bindings. The results from measuring the semantic similarity between pairs of concepts were presented using the SimLex-999 dataset.

Table 2 provides a non-comprehensive summary of the studies that applied the BEAGLE and RI methods to various linguistic tasks. The interested readers are referred to [42], which is a survey describing the applications of RI, PSI, and related methods, including the biomedical domain. The range of applications described in [42] covers word-sense disambiguation, bilingual information extraction, visualization of relations between terms, and document retrieval. It is worth noting that there is a software package called “Semantic vectors” [414, 415, 418] that implements many of the methods mentioned above and provides the main building blocks for designing further modifications of the methods.

2.2.2 Similarity Estimation of Biomedical Signals.

In [220, 221], BSC was applied to biomedical signals: heart rate and respiration. The need for comparing these signals emerged in the scope of a deep breathing test for assessing autonomic function. HDC/VSA was used to analyze cardiorespiratory synchronization by comparing the similarity between heart rate and respiration using feature-based analysis. Feature vectors were extracted from the signals and transformed into HVs by using role-filler bindings (Section 3.1.3 in [222]) and representations of scalars (Section 3.2 in [222]). These HVs were in turn classified into different degrees of cardiorespiratory synchronization/desynchronization. The signals were obtained from the healthy adult controls, patients with cardiac autonomic neuropathy, and patients with myocardial infarction. It was shown that, as expected, the similarity between different HVs were lower for patients with the cardiac autonomic neuropathy and myocardial infarction patients than for the healthy controls.

Another application of BSC was the identification of the ictogenic (i.e., seizure generating) brain regions from intracranial electroencephalography (iEEG) signals [28]. The algorithm first transformed iEEG time series from each electrode into a sequence of symbolic local binary pattern codes, from which a binary HV was obtained for each brain state (e.g., ictal or interictal). It then identified the ictogenic brain regions by measuring the relative distances between the learned HVs from different groups of electrodes. Such the identification was done by one-way ANOVA tests at two levels of spatial resolution, the cerebral hemispheres and lobes.

2.2.3 Similarity Estimation of Images.

In [296], the HDC/VSA two-dimensional (2D) image representations (Section 3.4 in [222]) were applied for an aggregation of local descriptors extracted from images. Local image descriptors were real-valued vectors whose dimensionality was controlled by RP (Section 3.2.3 in [222]). To represent a position inside an interval, the authors concatenated parts of two basis HVs and used several intervals, as in [339, 341] but using MAP. Position HVs for x and y were bound to represent (x,y); see Section 3.4 in [222]. Subsequently, projected local image descriptors were bound with their position HVs, using component-wise multiplication, and the bound HVs were superimposed to represent the whole image. The image HVs obtained with different algorithms for extracting the descriptors could also be aggregated using the superposition operation. When compared to the standard aggregation methods in (mobile robotics) place recognition experiments, HVs of the aggregated descriptors exhibited an average performance better than alternative methods (except the exhaustive pair-wise comparison). A very similar concept was demonstrated in [285] using an image classification task, see also Table 15. One of the proposed ways of forming image HV used the superposition of three binary HVs obtained from three different hashing neural networks. The HVs representing the aggregated descriptors provided a higher classification accuracy. Finally, similarity-preserving shift-equivariant representation of images in HVs using permutations was proposed in [438].

2.3.1 Classification Based on Feature Vectors.

Language identification with the vector of n-gram statistics of letters. In [173], it was shown how to form a compositional HV corresponding to n-gram statistics (see Section 3.3.4 in [222]). The work also introduced a task of identifying a language among 21 European languages. Since then, the task was used in several studies summarized in Table 3.

Classification of texts. Table 4 summarizes the efforts of using HDC/VSA for text classification. The works in this domain dealt with different tasks such as text categorization, news identification, and intent classification. Most of the works [1, 332, 381] used HVs as a way of representing data for conventional machine learning classification algorithms.

Classification of feature vectors extracted from acoustic signals. Classification of various acoustic signals using HVs is provided in Table 5. Tasks were mainly related to speech recognition, e.g., recognition of spoken letters or words.

Fault classification. Studies on applying HDC/VSA to fault classification are limited. We are only aware of two such use-cases (summarized in Table 6) applied to the problems of anomaly detection in a power plant and ball bearings. An earlier work on micro machine-tool acoustic diagnostics was presented in [240].

Automotive data. Table 7³ presents studies where HDC/VSA was used with automotive data, mainly in autonomous driving scenarios.

Behavioral signals. Studies that used behavioral signals are summarized in Table 8. One of the most common applications was activity recognition, but other tasks were considered as well (see the table).

Biomedical data. Currently explored applications of HDC/VSA on biomedical data can be categorized into five types of data: electromyography (EMG) signals, electroencephalography (EEG) signals, cardiotocography (CTG) signals, DNA sequences, and surface-enhanced laser desorption/ionization time-of-flight (SELDI-TOF) mass spectrometry. Most of the works so far have been conducted on EMG and EEG signals. In fact, there is a study [344], which provides an in-depth coverage of applying HDC/VSA to these modalities, so please refer to this article for a detailed overview of the area.

EMG signals. HDC/VSA was applied to EMG signals for the task of hand gesture recognition. This was done for several different transformations of data into HVs and on different datasets. Refer to Table 9 for the summary.

EEG and iEEG signals. EEG and iEEG signals were used with HDC/VSA for human–machine interfaces and epileptic seizure detection. These efforts are overviewed in Table 10. It is worth mentioning systematization efforts in [309, 310], which reported an assessment of several HDC/VSA models and transformations used for epileptic seizure detection. Another recent work [372] provides a tutorial on applying HDC/VSA for iEEG seizure detection.

CTG signals. So far, there is only one work [10] where CTG signals were used. It is summarized in Table 11.

DNA sequences. Table 12 presents two studies that used DNA sequences in classification tasks.

SELDI-TOF mass spectrometry. Table 13 summarizes a study that used SELDI-TOF mass spectrometry for classifying sensitivity of glioma to chemotherapy.

Multi-modal signals. Studies involding multi-modal signals are summarized in Table 14.

2.3.2 Classification of Images or Their Properties.

Table 15 provides an overview of the efforts involving images. Since using raw pixels directly would rarely result in a good performance, HVs were produced either from features extracted from images or using HVs obtained from neural networks (see Section 3.4.3 in [222]), which took images as an input.

2.3.3 Classification of Structured Data.

Classification of structured data can be tricky with conventional machine learning algorithms, since local representations of structured data might not be convenient to use with vector classifiers, especially when the data involve some sorts of hierarchies. HDC/VSA should be well suited for structured data, since they allow representing various structures (including hierarchies) as HVs. To the best of our knowledge, however, the number of such studies is very limited (see Table 16). One such study in [385] used SBDR to predict the properties of chemical compounds and provided the state-of-the-art performance. A more recent example was demonstrated in [258], where 2D molecular structures were transformed into HVs that were used to construct classifiers for drug discovery problems. The approach outperformed the baseline methods on a collection of 30 tasks. Finally, in [302] it was proposed to classify graphs using HDC/VSA. A graph was represented as a superposition of HVs corresponding to vertices and edges. The proposed approach was evaluated on six graph classification datasets; when compared to the baseline approaches, it demonstrated comparable accuracy on four datasets and much shorter training time on all of them.

3.1.1 Holistic Transformations.

Holistic transformations in database processing with HVs. In this section, we consider simple examples of HDC/VSA processing of a small database. This can be treated as analogical reasoning (though analogy researchers might disagree) in the form of query answering using simple analogs (we refer to them as records) without explicitly taking into account the constraints on analogical reasoning mentioned in the following sections.

[176] introduced the transformation of database records into HVs as an alternative to their symbolic representations. Each record is a set of role-filler bindings and is represented by HVs using transformations for role-filler bindings (Section 3.1.3 in [222]) and sets (Section 3.1.2 in [222]). We might be interested in querying the whole base for some property or in processing a pair of records. For example, knowing a filler of one of the roles in one record (the role is not known) can enable one to get the filler of that role in another record. In [176], the records were persons with attributes (i.e., roles) such as “name,” “gender,” and “age” (see Table 17).

Several ways were proposed to use HDC/VSA operations for querying the records by HVs. An example query to the database could be “What is the age of Lee who is a female?” The correct record for this query will be \(\mathbf {LF6}\) and the answer is 66. Depending on the prior knowledge, different cases were considered. All the cases below assume that there is an item memory with the HVs of all the roles, fillers, and database records.

Case 1. We know the role-filler bindings name:Lee, gender:female, and only the role “age” from the third role-filler binding whose filler we want to find. Solution 1. The query is represented as \(\mathbf {name} \circ \mathbf {Lee}\) + \(\mathbf {gender}\circ \mathbf {female}\). The base memory will return \(\mathbf {LF6}\) as the closest match using the similarity of HVs. Unbinding \(\mathbf {age} \oslash \mathbf {LF6}\) results in the noisy version of HV for \(\mathbf {66}\), and the clean-up procedure returns the value associated with the nearest HV in the item memory, i.e., \(\mathbf {66}\), which is the answer.

Case 2. The following HVs are available: record \(\mathbf {PM6}\) and the fillers \(\mathbf {Pat}\), \(\mathbf {male}\), and \(\mathbf {Lee}\), as well as the roles \(\mathbf {female}\) and \(\mathbf {age}\). Solution 2a. First, we find \(\mathbf {name}\) by the clean-up procedure on \(\mathbf {Pat} \oslash \mathbf {PM6}\), and \(\mathbf {gender}\) by the clean-up procedure on \(\mathbf {male} \oslash \mathbf {PM6}\). Then we apply the previous solution (Solution 1). Solution 2b. This solution⁴ uses the correspondences \(\mathbf {Pat} \leftrightarrow \mathbf {Lee}\) and \(\mathbf {male} \leftrightarrow \mathbf {female}\) by forming the transformation HV \(\mathbf {T} = \mathbf {Pat}\circ \mathbf {Lee} + \mathbf {male}\circ \mathbf {female}\). The transformation \(\mathbf {T} \oslash \mathbf {PM6}\) returns approximate \(\mathbf {LF6}\) (see [176] for a detailed explanation), and its clean-up provides exact \(\mathbf {LF6}\). Then, as above, \(\mathbf {age} \oslash \mathbf {LF6}\) after the clean-up procedure returns \(\mathbf {66}\). Note that such a transformation is intended for HDC/VSA models where the binding operation is self-inverse (e.g., BSC).

Case 3. Only \(\mathbf {33}\) and \(\mathbf {PF3}\) are known, and the task is to find an analog of \(\mathbf {33}\) in \(\mathbf {LF6}\), that is, \(\mathbf {66}\). A trivial solution would be to get \(\mathbf {age}\) as the result of the clean-up procedure for \(\mathbf {33} \oslash \mathbf {PF3}\), and then get \(\mathbf {33}\) by \(\mathbf {age} \oslash \mathbf {PF3}\) and the clean-up procedure. But there is a more interesting way, which can be considered as an analogical solution. Solution 3. One step solution is \(\mathbf {LF6} \oslash (\mathbf {33}\oslash \mathbf {PF3})\). This exemplifies a possibility of processing without the intermediate use of a clean-up procedure.

In some HDC/VSA models (e.g., BSC), however, the answer for this solution will be ambiguous, being equally similar to \(\mathbf {33}\) and \(\mathbf {66}\). This is due to the self-inverse property of the binding operation in BSC. Note that both \(\mathbf {LF6}\) and \(\mathbf {PF3}\) include \(\mathbf {gender}\circ \mathbf {female}\) as the part of their records. Unbinding \(\mathbf {33}\) with \(\mathbf {PF3}\) creates \(\mathbf {33} \circ \mathbf {gender}\circ \mathbf {female}\) (since \(\oslash =\circ\)) among other bindings. When unbinding the result with \(\mathbf {LF6}\), the HV \(\mathbf {gender}\circ \mathbf {female}\) will cancel out, thus releasing \(\mathbf {33}\), which will interfere with the correct answer, which is \(\mathbf {66}\). This effect would not appear if the records would had different fillers in their role-filler bindings. For example, if instead of \(\mathbf {LF6}\) we consider \(\mathbf {LM6}\), then \(\mathbf {33} \circ \mathbf {PF3} \circ \mathbf {LM6}\) produces the correct answer.

In models with self-inverse binding, the result of \(\mathbf {PF3} \oslash \mathbf {LM6}\) can be seen as an interpretation of \(\mathbf {PF3}\) in terms of \(\mathbf {LM6}\) or vice versa, because the result of this operation is \(\mathbf {Pat} \circ \mathbf {Lee} + \mathbf {male} \circ \mathbf {female} + \mathbf {33} \circ \mathbf {66} + \mathrm{noise}\) (since \(\oslash =\circ\)). This allows answering queries of the form “which filler in \(\mathbf {PF3}\) plays the same role as (something) in \(\mathbf {LM6}\)?” by unbinding \(\mathbf {PF3} \oslash \mathbf {LM6}\) with the required filler HV, resulting in the noisy answer HV.

Note that Solution 1 resembles the standard processing of records in databases. We first identify the record and then check the value of the role of interest. Solutions 2b and 3 are examples of a different type of computing sometimes called “holistic mapping” [176] or “transformation without decomposition” [323]. We call it “holistic transformation” to not confuse it with the transformation of input data into HVs or with mapping in analogical reasoning. This holistic transformation of HVs is commonly illustrated by an example well known under the name “Dollar of Mexico” [177, 180, 182].

In essence, the “Dollar of Mexico” show-case solves some simple “proportional analogies” of the form A:B :: C:D as (i.e., United States:Mexico :: Dollar:?). These analogies are also known to be solvable by addition and subtraction of the “neural” word embeddings of the corresponding concepts [274, 318]. Following a similar approach, the authors in [316] proposed to improve the results by training shallow neural networks using a dependency path of relations between terms in sentences.

It should be noted that there is no direct analog to this kind of processing by holistic transformation (using geometric properties of the representational space) in the conventional symbol manipulation [323]. The holistic transformation of HVs can be seen as a parallel alternative to the conventional sequential search.

Learning holistic transformations from examples. Learning systematic transformations from examples was investigated in [300, 301] for HRR. Previously, this capability was shown in [323] only for manually constructing the so-called transformation HV. In [300, 301], the transformation HV was obtained from several training pairs of HVs. One of the proposed approaches to obtaining the transformation HV was to use the gradient descent by iterating through all examples until the optimization converged. The experiments demonstrated that the learned transformation HVs were able to generalize to previously unseen compositional structures with novel elements. A high level of systematicity was indicated by the ability of transformation HVs to generalize to novel elements in structures of a complexity higher than the structures provided as training examples. The capability of BSC to learn holistic transformations was also presented in [178, 182]. However, the disadvantage of such holistic transformations is their bidirectionality, which is due to the fact that the unbinding operation in BSC is equivalent to the binding operation. This complication can be resolved by using either the permutation or an additional associative memory as the binding operation as proposed in [76].

The holistic transformation of the kind considered above was used to associate, e.g., sensory and motor/action data via binding HVs. For example, in [214, 216], BSC was applied to form a scene representation in the experiments with honey bees. It was shown to mimic learning in honey bees using the transformation as in [178, 182]. In [77], an HDC/VSA-based approach for learning behaviors, based on observing and associating sensory and motor/action data represented by HVs, was proposed. A potential shortcoming of the approaches to learning holistic transformations presented above is that the objects/relations are assumed to be dissimilar to each other. The learning might not work as expected given that there is some similarity structure between the objects/relations used as training examples. This direction deserves further investigation.

3.1.2 Analogical Reasoning.

We begin with a brief introduction to analogical reasoning by summarizing basic analogical processes as well as some of their properties. Two final sections discuss applications of HDC/VSA to modeling analogical retrieval and mapping.

Basics of analogical reasoning modeling. Modeling analogical reasoning in humans is an important frontier for Artificial Intelligence. Because it allows analogical problem solving as well as even more universal cognitive processes taking the problem structure into account and applying knowledge acquired in different domains. Analogical reasoning theories [114, 115, 117, 118, 227] usually consider and model the following basic processes: description, retrieval (also known as access or search), mapping, and inference.

Any model for analogical reasoning usually works with analogical episodes (or simply analogs). The description process concerns the representation of analogical episodes. The analogical episodes are usually modeled as systems of hierarchical relations (predicates), consisting of elements in the form of entities (objects) and relations of various hierarchical levels. Entities belong to some subject domain (e.g., the sun and planet) and are described by attributes (features, properties), which in essence are relations with a single argument (e.g., mass and temperature). Relations (e.g., attract, more, and cause) define relations between the elements of analogical episodes. Arguments of relations may be objects, attributes, and other relations. It is assumed that a collection of base (source) analogical episodes is stored in a memory.

The retrieval process searches the memory (in models, the base of analogical episodes) to find the closest analog(s) for the given target (input, query) analog. The similarity between episodes is used as a criterion for the search. Once the base analogical episode(s) closest to the target is identified, the mapping process finds the correspondences between the elements of two analogical episodes: the target and the base ones. The inference process concerns the transfer of the knowledge from the base analogical episode(s) to the target analogical episode. This new knowledge obtained about the target may, e.g., provide the solution to the problem specified by the target analogical episode. Because the analogical reasoning is not a deductive mechanism, the candidate inferences are only hypotheses and must be evaluated and checked (see, e.g., [115] and references therein).

Cognitive science has identified quite a few properties demonstrated by subjects when performing analogical reasoning. Two types of similarity that influence processing of analogical episodes are distinguished. Structural similarity (which should not be confused with the structural similarity in HDC/VSA) reflects how the elements of analogs are arranged with respect to each other, that is, in terms of the relations between the elements [74, 117, 118]. Analogs are also matched by the “surface” or “superficial” similarity [85, 114] based on common analogs’ elements or a broader “semantic” similarity [74, 154, 401], based on, e.g., joint membership in a taxonomic category or on similarity of characteristic feature vectors. Experiments based on human assessment of similarities and analogies confirmed that both surface (semantic) and structural similarity are necessary for sound retrieval [85]. The structural similarity in the retrieval process is considered less important than that in the mapping process; however, the models of retrieval that take into account only the surface similarity are considered inadequate. These properties are expected to be demonstrated by the computational models of analogy [116]. In the following sections, we discuss applications of HDC/VSA for analogical reasoning.

Analogical retrieval. It is known that humans retrieve some types of analogs more readily than others. Psychologists identified different types of similarities and ordered them according to the ease of retrieval [85, 114, 363, 413]. The similarity types are summarized in Table 18, relative to the base analogical episode. Simple examples of animal stories (adapted by Plate from [401]) with those similarity types are also presented. All analogical episodes have the same first-order relations (in the example, bite() and flee()). There are also higher-order relations cause(bite, flee) and cause(flee, bite) and attribute relations (dog, human, mouse, cat).

In addition to common first-order relations, the literal similarity (LS) also assumes both the same higher-order relations (in the example, the single relation cause(bite, flee)) and object attributes. The true analogy (AN) has the same higher-order relations, but different object attributes. The surface features (SF) has the same object attributes but different higher-order relations. The first-order only relations (FOR) differ in both higher-order relations and attributes. For the analogical retrieval, it is believed that the retrievability order is expressed as LS \(\ge\) SF \(\gt\) AN \(\ge\) FOR [85, 114, 363, 413].

Researchers studying analogical reasoning proposed a number of heuristics-based models of the analogical retrieval. The most influential of them are still “many are called but few are chosen,” which operate with symbolic structures [85], and analog retrieval by constraint satisfaction using localist neural network structures [401]. The structure of analogical episodes should be taken into account in their similarity estimation. This requires alignment and finding correspondences between elements of the analogical episodes, as in the mapping (Section 3.1.2 below), which is computationally expensive. Moreover, unlike for mapping, where only two analogical episodes are considered, in the retrieval process alignment should be repeated for the target analogical episode and each of the base analogical episodes, making such an implementation of the retrieval prohibitive.

To reduce the computational costs, a two-stage filter and refine (F&R) approach is used in the traditional models of analogical retrieval. At the filtering step, the target analogical episode is compared to all the base analogical episodes using a low-cost similarity of their feature vectors (that only counts the frequency of symbolic names in the analogical episodes, without taking into account the structure). The most similar base analogical episodes are selected as prospective candidates. At the refining step, the candidates are then compared to the target analogical episode by the value of their structural similarity. Computationally expensive mapping algorithms (Section 3.1.2 below) are used for calculating the structural similarity. As the final result, the analogical episodes with the highest structural similarity are returned.

HDC/VSA have been applied to modeling of the analogical retrieval by Plate (see, e.g., [319, 320, 324, 326] and references therein). In HDC/VSA, both the set of structure elements and their arrangements influence their HV similarity, so that similar structures (in this case, analogical episodes) produce similar HVs. Because the HV similarity measures are not computationally expensive, the two-stage F&R approach of the traditional models is not needed. Using HRR, it was shown that the results obtained by a single-stage HV similarity estimation are consistent with both the empirical results in psychological experiments as well as the aforementioned leading traditional models of the analogical retrieval. Note that Plate experimented with analogical episodes different from those tested in the leading models, but they still belong to the proper similarity types, as shown in Table 18. Similar results were also reported in [330] for SBDR using Plate’s episodes.

The study [338] applied SBDR to represent the analogical episodes in the manner of [330]. However, the performance was evaluated using the test bases of the most advanced models of analogical retrieval. The results demonstrated some increase in the recall and a noticeable increase in the precision compared to the leading traditional (two-stage) models. The authors also compared the computational complexity and found that in most of the cases the HDC/VSA approach had advantages over the traditional models.

Analogical mapping. The most influential models of analogical mapping include the Structure Mapping Engine (SME) [79] and its versions and further developments [84], as well as the Analogical Constraint Mapping Engine (ACME) [147]. SME is a symbolic model that uses a local-to-global alignment process to determine correspondences between the elements of analogical episodes. SME’s drawback is a rather poor account of semantic similarity. Also, structure matching in SME is computationally expensive, so it is prohibitively expensive to use SME during retrieval by comparing input to each of (many) analogical episodes in the memory containing all analogical episodes for a structure-sensitive comparison.

ACME is a localist connectionist model that determines analogical mappings using a parallel constraint satisfaction network. Unlike SME, ACME relies not only on the structural information, but also takes into account semantic and pragmatic constraints. ACME is usually even more computationally expensive than SME.

Further models of mapping based on HRR and BSC were proposed that use techniques based on holistic transformations (Section 3.1.1). One of the limitations of these studies is that the approach was not demonstrated to be scalable to large analogical episodes. HRR was also used in another model for the analogical mapping [74] called DRAMA, where the similarity between HVs was used to initialize a localist network involved in the mapping.

In [330], similarity of HVs (formed with SBDR) of the analogical episodes was used for their mapping. However, this technique worked only for the most straightforward mapping cases. In [331], several alternative techniques for mapping with SBDR were proposed (including direct similarity mapping, re-representation by substitution of identical HVs, and parallel traversing of structures using higher-level roles) and some of them were demonstrated on complex analogies. However, the techniques are rather sophisticated and used sequential operations.

In [386], a kind of re-representation of an analog’s element HVs was proposed to allow the analogical mapping of the resultant HVs on the basis of similarity. The re-representation approach included the superposition of two HVs. One of those HVs was obtained as the HV for the episode’s element using the usual representational scheme of the episode, e.g., the HV used for the retrieval (compositional structure representation by HVs, see Section 2.2.4 in [222]). This HV took into account the semantic similarity. The other HV was the superposition of HVs of elements higher-level roles. This took into account the structural similarity. The proposed procedure was tested in several experiments on simple analogical episodes used in previous studies (e.g., in [326, 331], and on rather complex analogical episodes previously used only in the leading state-of-the-art models, e.g., “Water Flow-Heat Flow,” “Solar System-Atom,” and “Schools” [114, 147]. It produced the correct mapping results. The analogical inference was also considered. The computational complexity of the proposed approach was rather low and was largely affected by the dimensionality of HVs.

The problem with the current models of HDC/VSA for analogical mapping is that they lack interaction and competition of consistent alternative mappings. They could probably be improved by using an approach involving the associative memory akin to [109].

Finally, an important aspect of HDC/VSA usage for analogical mapping and reasoning is the compatibility with the existing well-established formats of knowledge representation. This will facilitate the unification of symbolic and subsymbolic approaches for cognitive modeling and Artificial Intelligence. The work in [272] presented a proof of concept of the mapping between Resource Description Framework Schema ontology and HDC/VSA.

Graph isomorphism. The ability to identify some form of “structural isomorphism” is an important component of analogical mapping [79, 326]. The abstract formulation of isomorphism is graph isomorphism. In [109], an interesting scheme was proposed for finding the graph isomorphism with HDC/VSA and associative memory. The scheme used the mechanism proposed in [250]. The paper presented the HDC/VSA-based implementation of the algorithm proposed in [317], which used replicator equations and treated the problem as a maximal-clique-finding problem. In essence, the HDC/VSA-based implementation transformed the continuous optimization problem into a high-dimensional space where all aspects of the problem to be solved were represented as HVs. The simulation study (unfortunately performed on a simple graph) showed that the distributed version of the algorithm using the mechanism from [250] mimics the dynamics of the localist implementation from [317].

3.1.3 Cognitive Modeling.

In this part, we briefly cover known examples of using HDC/VSA for modeling particular cognitive capabilities, such as sequence memorization or problem-solving, for cognitive tasks like the Wason task [71], n-back task [120], Wisconsin card sorting test [174], Raven’s Progressive Matrices [354], or the Tower of Hanoi [393].

HDC/VSA as a component of cognitive models. As argued in [195], HDC/VSA is an important tool for cognitive modeling. In cognitive science, HDC/VSA has been commonly used as a part of computational models replicating experimental data obtained from humans. For example, in [22] HVs were used as the representational scheme of a computational model. The model was tested using categorization studies considering three competing accounts of concepts: “prototype theory,” “exemplar theory,” and “theory theory.” The model was shown to be able to replicate experimental data from categorization studies for each of the accounts. It is also worth mentioning that there are numerous works using context HVs (Section 2.2.1) to form models replicating the results obtained in language-related cognitive studies, see, e.g., [41, 166, 167, 168, 169, 172, 358, 378, 399].

Modeling human working memory with HDC/VSA. A topic that was studied by different research groups working on HDC/VSA is sequence memorization and recall. For example, it was demonstrated in [128] (see Section 3.3 in [222]) that a HDC/VSA-based representation of sequences performed better than localist representations when compared on the standard benchmark for behavioral effects. Some studies [20, 38, 70, 96, 121, 190, 292, 359] demonstrated how the recall of sequences represented in HVs (Section 3.3 in [222]), albeit with slightly different encoding, can reproduce the performance of human subjects on remembering sequences. This is profound as it demonstrates that simple HDC/VSA operations can reproduce basic experimental findings of human memory studies. An alternative model was proposed in [30]. Importantly, this work linked the neuroscience literature to the modeling of memorizing sequences with HDC/VSA.

Raven’s Progressive Matrices. Raven’s Progressive Matrices is a nonverbal test used to measure general human intelligence and abstract reasoning. Some simple geometrical figures are placed in a matrix of panels, and the task is to select the figure for an empty panel from a set of possibilities [253]. Note that the key work related to Raven’s progressive matrices goes back to the 1930s (see [356]). It was widely used to test human fluid intelligence and abstract reasoning since the 1990s [31]. The task was first brought to the scope of HDC/VSA in [354, 355] using HRR and its subsequent implementation in spiking neurons. Later studies [75, 251] demonstrated that other HDC/VSA models (BSC and MAP, respectively) can also be used to create systems capable of solving a limited set of Raven’s Progressive Matrices containing only the progression rule.

In all studies the key ingredient of the solution was the representation of a geometrical panel by its HV (e.g., giving access to the symbolic representation of the panel followed by using role-filler bindings of the shapes and their quantity present in the panel). Subsequently, the HV corresponding to the transformation between the HVs of the adjacent pairs of panels was obtained using the ideas from [301] (see Section 3.1.1). The transformation HV was then used to form a prediction HV for the blank panel in the matrix. The candidate answer with the HV most similar to the prediction HV was then chosen as the answer to the test. All of the previously described studies have two limitations: First, they assume the perception system provides the symbolic representations that support the reasoning for solving Raven’s Progressive Matrices test, and, second, they only support the progression rule. [144] addressed these limitations by positioning VSA/HDC as a common language between a neural network (to solve the perception issue) and a symbolic logical reasoning engine (to support more rules). Specifically, it exploited the superposition of multiplicative bindings in a neural network to describe raw sensory visual objects in a panel and used Fourier Holographic Reduced Representations (FHRR) to efficiently emulate a symbolic logical reasoning with a rich set of rules [144].

The Tower of Hanoi task. The Tower of Hanoi task, which is a simple mathematical puzzle, is another example of a task used to assess problem solving capabilities. The task involves three pegs and a fixed number of disks of different sizes with holes in the middle such that they can be placed on the pegs. Given a starting position, the goal is to move the disks to a target position. Only one disk can be moved at a time and a larger disk cannot be placed on top of a smaller disk.

In [393], an HDC/VSA-based model capable of solving the Tower of Hanoi tasks was presented. The binding and superposition operations were used to form HVs of the current position and a set of goals identified by the model. The model implemented a known algorithm for solving the task given valid starting and target positions. The performance of the model was compared to that of humans regarding of time delays, which were found to be qualitatively similar.

Modeling the visual similarity of words. Modeling human perception of word similarity with HDC/VSA was based on the experimental data obtained for human subjects in [62]. The task was to model the human patterns of delays in priming tasks with the similarity values of sequence HVs obtained from various HDC/VSA models and for various schemes for sequence representation (Section 3.3 in [222]).

In the task of modeling restrictions on the perception of word similarity, four types of restrictions (a total of 20 similarity patterns) were summarized in [128]. In [128], the BSC model was employed to represent symbols in their positions; various string representations with substrings were also used. Symbols in positions with correlated HVs to represent nearby positions were studied in [49] for the BSC, FHRR, and HRR models. Their results demonstrated partial satisfaction of the restrictions. However, substring representations from [51] with HRR, and symbols-at-correlated positions representations from [333] with SBDR using permutations as well as the ones from [335] with FHRR met all the restrictions for certain choices of the similarity measure and values of a scheme’s hyperparameters. In the task of finding correlations between human and model similarity data, [333, 335] demonstrated results were on a par with those of the string kernel similarity measures from [129].

General-purpose rule-based and logic-based inference with HVs. The study in [391] presented a spiking neurons model that was positioned as a general-purpose neural controller. The controller was playing a role analogous to a production system capable of applying inference rules. HDC/VSA and their operations played a key role in the model providing the basis for representing symbols and their relations. The model was demonstrated to perform several tasks: repeating the alphabet, repeating the alphabet starting from a particular letter, and answering simple questions similar to the ones in Section 3.1.1. Another realization of a production system with the Tensor Product Representations model was demonstrated in [65]. Several examples of hierarchical reasoning using the superposition operation on context HVs representing hyponyms to form representations of hypernyms were presented in [230].

In [396], it was demonstrated how HVs can be used to represent a knowledge base with clauses for further performing deductive inference on them. The work widely used negation for logical inference, which was also discussed in [234]. Reasoning with HRR using modus ponens and modus tollens rules was demonstrated in [244]. The works in [127, 376, 377] discussed the usage of VSA/HDC for the realization of context logic language and demonstrated the inference procedures.

3.1.4 Computer Vision and Scene Analysis.

This section summarizes different aspects of using HDC/VSA for processing visual data. This is one of the newest and least explored application areas of HDC/VSA.

Visual analogies. In [426], a simple analogy-making scenario was demonstrated on 2D images of natural objects (e.g., bird, horse, and automobile). This work took an image representing a particular category, e.g., a bird. The HVs of images were obtained through using convolutional neural networks (Section 3.4.3 in [222]) and cellular automata computations (see [428] for the method description). Several (e.g., 50 in [426]) such binary HVs (e.g., for images of different birds) were superimposed together to form the HV of a category, e.g., (4) \(\begin{equation} \mathbf {land}=\mathbf {animal} \circ \mathbf {horse} + \mathbf {vehicle} \circ \mathbf {automobile}, \end{equation}\) (5) \(\begin{equation} \mathbf {air}=\mathbf {animal} \circ \mathbf {bird} + \mathbf {vehicle} \circ \mathbf {airplane}. \end{equation}\) The category HVs were used to form some statements using the HDC/VSA operations. Inspired by the well-known example of “Dollar of Mexico?” (as in the techniques of Section 3.1.1), it was shown that one could perform queries of a similar form as “What is the Automobile of Air?” (\(\mathbf {AoA}\)) but using HVs formed from the 2D images: (6) \(\begin{equation} \mathbf {AoA}= \mathbf {air} \oslash (\mathbf {land} \oslash \mathbf {automobile}) . \end{equation}\) The system demonstrated a high accuracy (98%) of correct analogy-making on previously unseen images of automobiles.

Reasoning on visual scenes and Visual Question Answering. Visual Question Answering is defined as a task where an agent should answer a question about a given visual scene. In [290], a trainable model was presented that used HDC/VSA for this task. The model in [290] differs from the state-of-the-art solutions that usually include a combination of a recurrent neural network (handles questions and provides answers) and a convolutional neural network (handles visual scenes). The model included two parts. The first part transformed a visual scene into an HV describing the scene using a neural network. This part used only one feed-forward neural network, which took a visual scene and returned its HV. The second part of the model defined the item memory of atomic HVs as well as HVs of questions along with their evaluation conditions in terms of cosine similarity thresholds. The neural network was trained to produce HVs associated with a dataset of simple visual scenes (two figures in various combinations of four possible shapes, colors, and positions). The gradient descent used errors from the question answering on the training data, which included five predefined questions. It was shown that the trained network successfully produced HVs that answered questions for new unseen visual scenes. The five considered questions were answered with 100% accuracy. On previously unseen questions the model demonstrated an accuracy in the range of 60–72%.

Similarly to [290], there were attempts in [144, 198, 263] to train neural networks to output HVs representing a structured description of scenes (see also Section 3.4.3 in [222]), which could then be used for computing visual analogies.

Another approach to Visual Question Answering with HDC/VSA was outlined in [231, 233], where a visual scene was first preprocessed to identify objects and construct a scene data structure called the causal matrix (it stored some object attributes including positions). This data structure describing a scene was transformed into an HV that could then be queried using HDC/VSA operations similar to those from [290]. In [202], it was applied to a dataset constructed to facilitate visual navigation in human-centered environments. This approach was further extended from Visual Question Answering to Visual Dialog in [232].

Another application of HDC/VSA for representation and reasoning on visual scenes, similar in its spirit to the Visual Question Answering, was presented in [412]. The approach represented visual scenes in the form of HVs using Fourier HRR. The paper transformed continuous positions in an image to complex-valued HVs such that the spatial relation between positions was preserved in the HVs (see the “fractional power encoding” in Sections 3.2.1 and 3.4.2 in [222]). During the evaluation, handwritten digits and their positions were identified using a neural network with an attention mechanism. Then the identified information was used to create a complex-valued compositional HV describing the scene. Such scene HVs were used to answer relational queries like “which digit is below 2 and to the left of 1?”

A further exploration of this task was presented in [254]. The approach in [412] also demonstrated solving a simple navigation problem in a maze. In [228], navigation tasks in 2D environments were further studied by using HVs as an input to a neural network producing outputs directing the movement. Neural networks trained with HVs demonstrated the best performance among methods considered. There was also a recent attempt to implement this continuous representation with spiking neurons [67].

3.2.1 Semantic Pointer Architecture Unified Network.

The most well-known example of a cognitive architecture using HDC/VSA is called “Spaun” for Semantic Pointer Architecture Unified Network (see its overview in [73] and a detailed description in [72]). Spaun is a large spiking neural network (2.5 million neurons) that uses HRR for data representation and manipulation. In the architecture, HVs play the role of “semantic pointers” [22] aiming to integrate connectionist and symbolic approaches. It has an “arm” for drawing its outputs as well as “eyes” for sensing inputs in the form of 2D images (handwritten or typed characters). Spaun (without modifying the architecture) was demonstrated in eight different cognitive tasks that require different behaviors. The tasks used to demonstrate the capabilities of Spaun were as follows: copy drawing of handwritten digits, handwritten digit recognition, reinforcement learning on a three-armed bandit task, serial working memory, counting of digits, question answering given a list of handwritten digits, rapid variable creation, and fluid syntactic or semantic reasoning. The same principles were used in [52] to represent the WordNet knowledge base with HVs allowing enriching, e.g., Spaun, with a memory storing some prior knowledge.

3.2.2 Associative-Projective Neural Networks.

The cognitive architecture of Associative-Projective Neural Networks (APNNs) that use the SBDR model was proposed and presented in [234, 238, 241, 242, 337]. The goal was to show how to construct a complex hierarchical model of the world that presumably exists in humans’ and higher animals’ brains, as a step toward Artificial Intelligence.

Two hierarchy types were considered: the compositional (part-whole) as well as the categorization or generalization (class-instance or is-a) ones. An example of the compositional hierarchy is letters \(\rightarrow\) words \(\rightarrow\) sentences \(\rightarrow\) paragraphs \(\rightarrow\) text. Another example is the description of knowledge base episodes in terms of their elements of various complexity from attributes to objects to relations to higher-order relations (see Section 3.1.2). An example of the categorization hierarchy is dog \(\rightarrow\) spaniel \(\rightarrow\) spaniel Rover or apple \(\rightarrow\) big red apple \(\rightarrow\) this big red apple in hand.

The proposed world model relies on models of various modalities, including sensory ones (visual, acoustic, tactile, motoric, etc.) and more abstract modalities (linguistics, planning, reasoning, abstract thinking, etc.), that are organized in hierarchies. The models are required for objects of different nature, e.g., events, real objects, feelings, attributes, and so on. Models (their representations) of various modalities can be combined, resulting in multi-modal representations of objects and associating them with the behavioral schemes (reactions to objects or situations), see details in [234, 238, 241, 242, 337].

The APNN architecture is based on HDC/VSA (though it was proposed long before the terms HDC and VSA appeared); in particular, models are represented by SBDR (Section 2.3.8 in [222]). An approach to formation, storage, and modification of hierarchical models was proposed. This is facilitated by the capability to represent in HVs of fixed dimensionality (for items of various complexity and generality) various heterogeneous data types, e.g., numeric data, images, words, sequences, structures (Section 3 in [222]). As usual for HDC/VSA, the model HVs can be constructed on-the-fly (without learning). APNNs have a multi-module, multi-level, and multi-modal design. A module forms, stores, and processes many HVs representing models of objects of a certain modality and of a certain level of compositional hierarchy. A module’s HVs are constructed from HVs obtained from other modules, such as lower-level modules of the same modality, or from modules of other modalities. The lowest level of the compositional hierarchy consists of modules providing a representation grounding (atomic HVs).

For SBDR, a HV is similar to the HVs of its elements of a lower compositional hierarchy level, as well as to the HVs of the higher level, of which the HV is an element. So, using similarity search (in the item memory of levels) it is possible to recover both the lower-level element HVs and compositional HVs of the higher level.

Each module has a long-term memory where it stores its HVs. A Hopfield-like distributed auto-associative memory [98, 99, 125] was suggested as a module memory. It performs the clean-up procedure for noisy or partial HVs by a similarity search. However, its unique property is the formation of the second main hierarchy type, i.e., of generalization (class-instance) hierarchies. It is formed when many similar (correlated) HVs are stored (memorized), based on the idea of Hebb’s cell assemblies including cores (subsets of HV 1-components often occurring together, corresponding to, e.g., typical features of categories and object-prototypes) and fringes (features of specific objects), see [125, 337].

It is acknowledged that a world model comprising domain(s) specific knowledge as well as information about an agent itself is necessary for any intelligent behavior. Such a model allows comprehension of the world by an intelligent agent and assists it in its interactions with the environment, e.g., through predictions of action outcomes.

The main problem with the APNN architecture is that not all its aspects have been modeled. For example, there are the following questions, which do not have exact answers:

• How do we extract objects and their parts of various hierarchical levels?

• How do we determine the hierarchical level an object belongs to?

• How do we work with an object that may belong to different hierarchy levels and modules?

• How do we represent objects invariant of their transformations?

Also, modeling cores and fringes formation in distributed auto-associative memory is still fragmentary as of now. Finally, it worth noting that similar ideas are currently being developed in the context of deep neural networks [63, 119].

3.2.3 Hierarchical Temporal Memory.

An interesting connection between HDC/VSA and a well-known architecture called Hierarchical Temporal Memory (HTM) [132] was presented in [308]. The work showed how HTM can be trained in its usual sequential manner to support basic HDC/VSA operations: binding and superposition of sparse HVs, which are natively used by HTM. Even though permutations were not discussed, it is likely that they also could be implemented, so that HTM could be seen as another HDC/VSA model, which additionally has a learning engine in its core.

3.2.4 Learning Intelligent Distribution Agent.

A version of the well-known symbolic cognitive architecture Learning Intelligent Distribution Agent [97], working with HVs, was presented in [389]. In particular, the Modular Composite Representations model was used [388]. Moreover, memory mechanisms in the proposed architecture were also related to HDC/VSA: An extension of Sparse Distributed Memory [175], known as Integer Sparse Distributed Memory [390], was used. The usage of HVs allowed resolving some of the issues with the original model [97] such as representation capability, flexibility, and scalability.

3.2.5 Memories for Cognitive Architectures.

Memory is one of the key components of any cognitive architecture and for modeling cognitive abilities of humans. There is a plethora of memory models. For example, MINERVA 2 [146] is an influential computational model of long-term memory. However, in its original formulation MINERVA 2 was not very suitable for an implementation in connectionist systems. Nevertheless, it was demonstrated in [193] that the Tensor Product Representations model (Section 2.3.2 in [222]) can be used to formalize MINERVA 2 as a fixed size tensor of order four. Moreover, it was demonstrated that the lateral inhibition mechanism for HDC/VSA [109] and HRR can be used to approximate MINERVA 2 with HVs. HVs allowed compressing the exact formulation of the model, which relies on tensors, into several HVs, thus making the model more computationally tractable at the cost of lossy representation in HVs.

Another example of using HVs (with HRR) for representing concepts is a Holographic Declarative Memory [3, 190, 192] related to BEAGLE [172] (see Section 2.2.1). It was proposed as a declarative and procedural memory in cognitive architectures. It was shown that the memory can account for many effects such as primacy, recency, probability estimation, interference between memories, and others.

In [166, 169], BEAGLE (Section 2.2.1) was extended to store (instead of one context HV per word) episodic memory of the observed data as HVs of all the contexts. This extension was called the instance theory of semantics. Each word was represented by an atomic random HV. A word’s context (a sentence) HV is constructed as a superposition of its word HVs and is stored in the memory. The HV of some query word is constructed as follows. First, the \(\text{sim}_{\text{cos}}\) of the query word HV and each context HV is calculated and raised to a power, producing a vector of “trace activations.” Then, context HVs are weighted by traces and summed to produce the retrieved (“semantic”) HV of the query word.

The study in [53] introduced a “weighted expectancy subtraction” mechanism that formed actual context HV as follows. First, the context HV produced the retrieved HV as explained above. Then, the retrieved HV was weighted and subtracted from the initial context HV. During the retrieval, the weighted HV of the second retrieval iteration was subtracted from the HV of the first retrieval iteration. This allowed flexibly controlling the construction of general versus specific semantic knowledge. The work in [305] proposed CogNGen, a core of a cognitive architecture that combines predictive processing based on neural generative coding and HDC/VSA models of human memory. The CogNGen architecture learns across diverse tasks and models human performance at larger scales.

4.1.1 Context HVs.

When it comes to context HVs, Random Indexing and Bound Encoding of the Aggregate Language Environment have appeared as improvements to Latent Semantic Analysis, e.g., they do not require Singular Value Decomposition and can naturally take order information into account. However, they were largely overshadowed after the introduction of “neural” word embeddings in Word2vec [274] or GloVe [318]. The latter are the result of an iterative process, which takes numerous passes via training data to converge. At the same time, an important fact is that distributional models such as Latent Semantic Analysis can in fact benefit from some techniques used in neural word embeddings [248]. Concerning, e.g., Bound Encoding of the Aggregate Language Environment, as recently demonstrated in [171], the method can benefit from negative information. Nevertheless, the current de facto situation in the natural language processing community is that Bound Encoding of the Aggregate Language Environment and Random Indexing methods are rarely the first candidates when it comes to choosing word embeddings. However, since Bound Encoding of the Aggregate Language Environment has been proposed within cognitive science community, it still plays an important role in modeling cognitive phenomena related to memory and semantics [53, 166]. Also, in contrast to the iterative methods, Random Indexing and Bound Encoding of the Aggregate Language Environment only require a single pass through the training data to form context HVs. In some situations, this could be an advantage, especially since the natural language processing community is becoming increasingly concerned about the computational costs of algorithms [395].

4.1.2 Classification.

While right now the classification with HDC/VSA is flourishing, there are still important aspects that are often not taken into account in these studies.

Formation of HVs. An important aspect of the formation of HVs is the initial extraction of features from raw data such as 2D images or acoustic signals. Usually, directly transforming raw data into HVs does not result in a good performance, so an additional step of extracting meaningful features is required.

Another important aspect is that, when constructing HVs from feature data for classification, in most cases the transformation of data into HVs is somewhat ad hoc. While there likely will not be a straightforward answer to how transforming data into HVs, it is still important to mention several issues.

It is a well-known fact that the advantage of nonlinear transformation is that classes not linearly separable in the original representation, might become linearly separable after a proper nonlinear transformation to a high-dimensional space (often called lift). This allows using not only k-Nearest Neighbor classifiers but also well-developed linear classifiers to solve problems that are not linearly separable. So nonlinearity is an important aspect of transforming data into HVs. All transformations of data into HVs that we are aware of seem to be nonlinear. However, there are no studies that scrutinize and characterize the nonlinearity properties of HVs obtained from the compositional approach. Moreover, most of the studies choose a particular transformation of numeric vector and stick to it. One of the most common choices is randomized “float” coding [224, 339, 344, 417]. There is, however, a recent study [89, 90] that established a promising connection between kernel methods [6, 347] and the fractional power encoding for representing numeric vectors as well as an earlier algorithm for approximating a particular type of kernels (tree kernels) with the HRR model [430]. In our opinion, the transformation of data into HVs is a hyperparameter of the model and using, e.g., cross-validation to choose the most promising transformation will likely be the best strategy when considering a range of different datasets.

Choice of classifier. As we saw in Section 2.3, centroids are probably the most common approach to forming a classifier in HDC/VSA. This is understandable, since centroids have an important advantage in terms of computational costs—they are very easy to compute. However, as pointed out in [184], the result of superposition does not provide generalization in itself, it is just a representation of combinations of HVs of training samples. Practically, it means that the centroids are not the best performing approach when it comes to classification performance. One way to improve the performance is to assign weights when including new samples into centroids [136, 342]. It was also shown that the perceptron learning rule in [158] and loss-based objective in [139] might significantly improve centroids. Earlier work on HV-based classifiers, e.g., [236, 237, 332, 336] also used linear perceptron and Support Vector Machine classifiers with encouraging results. Note that a large-margin perceptron usually trains much faster than a Support Vector Machine for big data, while providing classification quality at the same level as the Support Vector Machine and usually much higher than that of the standard perceptron. Another recent result [64] is that centroids can be easily combined with a known conventional classifier: generalized learning vector quantization [370]. Using an HDC/VSA transformation of data to HV, the authors obtained state-of-the-art classification results on a benchmark [80]. In general, we believe that when inventing new mechanisms of classification with HDC/VSA, it is important to report the results on collections of datasets instead of only a handful of datasets. For example, for feature-based classification, the UCI Machine Learning Repository [66] and subsets thereof (e.g., [80]) are a common choice (examples of HDC/VSA using it are [64, 92, 210]). For univariate temporal signals, the UCR Time Series Archive [57] is a good option that was used, e.g., in [375]. If a reported mechanism targets a more specific application area, then it would be desirable to evaluate it on a relevant collection for that area.

Many other types of classifiers such as k-Nearest Neighbors [187] are also likely to work with HVs as their input. When HVs are generated by a nonlinear transformation (in [187], using an HDC/VSA-guided convolutional neural network feature extractor), k-Nearest Neighbors classifier forms an “explicit memory” in memory-augmented neural networks or Neural Turing Machines [123]. The contents of the explicit memory can be compressed using outer products with randomized labels [208]. As mentioned in the previous section, linear classifiers can be used with nonlinearly transformed HVs. For example, the ridge regression, which is commonly used for randomized neural networks [371], performed well with HVs [361]. However, not all conventional classifiers work well with HVs [1]. That is because some of the algorithms (e.g., decision tree or Naïve Bayes) assume that any component of a vector can be interpreted on its own. It is a reasonable assumption when components of vectors are meaningful features, but in HVs a component does not usually have a meaningful interpretation. In the case of HDC/VSA with sparse representations, special attention should be given to classifiers that benefit from sparsity. Examples of such classifiers are the sparse Support Vector Machine [78] and winnow algorithm [252].

Applications in machine learning beyond classification. There are also efforts to apply HDC/VSA within machine learning outside of classification. Examples of such efforts are using data transformed into HVs for clustering [7, 137], unsupervised learning [279, 306], multi-task learning [32, 33, 34], distributed learning [149, 361], model compression [32, 143, 361, 362], and ensemble learning [25, 409].

It is expected that fractional power encoding [89, 229, 319] (Section 3.2.1 in [222]) is going to be a particularly fruitful method for enabling new applications beyond classification. This expectation is based on two facts. First, fractional power encoding is known to approximate kernels, which allows for an efficient implementation of kernel methods. There are already examples of its use to implement methods for probability density estimation [89, 90], kernel regression [89, 90], Gaussian processes-based mutual information exploration [101], representing probability statements [100], path integration [68], and reinforcement learning [9]. Second, fractional power encoding provides a simple but powerful way for representing numeric data in HVs, which allows numerous applications relying on such data. Some recent examples include simulation and prediction of dynamical systems [406], reasoning on 2D images [87, 254, 412], navigation in 2D environments [228, 412], representation of time series [375], and even modeling in neuroscience [67, 86].

We did not devote separate sections to these efforts as the studies are still scarce, but the interested readers are kindly referred to the above works for the initial investigations on these topics.

4.1.3 Real-world Use-cases and New Application Areas.

Section 2 demonstrated that there have been numerous attempts to apply HDC/VSA in a diverse range of scenarios spanning from communications to analogical reasoning. As we can see from Section 2.3, the most recent uptick in the research activity was applying HDC/VSA to classification tasks. In the near future, we are likely to see them being applied to solving classification tasks in new domains. Examples of such new domains recently appeared in, e.g., [407], where HDC/VSA was applied to branch prediction in processor cores and [380], where HDC/VSA was applied to food profiling.

Concerning the applications of analogical reasoning (Section 3.1.2), the major bottleneck is still the transformation of textual, speech or pictorial descriptions of analogical episodes to directed ordered acyclic graphs that can then be transformed into HVs (Section 3.5.2 in [222]). Note that this problem concerns not only analogical reasoning based on HDC/VSA but all methods that use predicate-based descriptions as inputs.

Nevertheless, there is still a considerable way to go to demonstrate how HDC/VSA-based solutions scale up to real-world problems. We, however, strongly believe that, similarly to the modern reincarnation of connectionist models, eventually research will distill the niches where the advantages of HDC/VSA are self-evident. Currently, one promising niche seems to be the time series classification [375], particularly in-sensor classification of biomedical signals [288] and prosthetic grasping [303]. Furthermore, the exploration of HDC/VSA in novel application domains should be continued. For instance, there were recent applications in communication [140, 150, 151, 199] and in distributed systems [383] (see Section 2.1.2), which were not foreseen by the community. Another recent example is the attempt to apply HDC/VSA to robotics problems [265, 266, 267, 284, 296, 298, 404].

4.2.1 HVs as Input to Neural Networks.

One of the most obvious ways to make an interplay between HDC/VSA and neural networks is by using HVs to represent input to neural networks. This is a rather natural combination of the two, because, in essence, neural networks often work with distributed representations. So, processing information distributed in HVs is not an issue for neural networks. However, since HVs are high dimensional, it is not always possible to use them as the input: The size of neural networks’ input layer should be set to D (e.g., in [259], whereby a fully connected “readout” layer for a task of choice was trained on D-dimensional input HVs) or even to a tensor composed of HVs (e.g., to represent each position in the retina by its HV, without superposition of HVs). Moreover, the local structure of the input signal space may become different from that used, e.g., in convolutional neural networks. This could require very different neural network architectures compared to modern deep neural networks.

There are, nevertheless, scenarios where using HVs with neural networks appeared to be beneficial. First, HVs are useful in situations when the data to be fed to a neural network are high dimensional and sparse. Then HVs can be used to form more compact distributed representations of these data. A typical example of using such high-dimensional and sparse data is n-gram statistics. There are works that studied tradeoffs between the dimensionality of HVs representing n-gram statistics (see Section 3.3.4 in [222]) and the performance of neural networks using these HVs as their input [1, 219]. These works demonstrated that it is possible to achieve the same or very similar classification performance with networks of much smaller size. Moreover, the degradation of the classification performance is gradual with the decreasing size of HVs, so their dimensionality can be used to control the tradeoff between the size of the network and its performance. On top of creating more compact representations, an additional advantage of HVs might lie in making HVs binary as in, e.g., Binary Spatter Codes. This might be leveraged in situations where the whole model is binarized [381].

Also, HVs may be useful when the size of input is not fixed but instead could vary for different inputs. Since neural networks are not flexible in changing their architecture, HDC/VSA can be used to take care of forming fixed size HVs for input of variable size. This mode of a neural network interface has been demonstrated in an automotive context to represent either varying number of intersections being crossed by a vehicle [7] or the dynamically changing environment around a vehicle [276, 280]. Further promising avenues for this mode are graphs and natural language processing, since there is a lot of structure in both, which can potentially be represented in HVs [183, 259]. Some investigations in this direction using Tensor Product Representations were presented in [36].

We foresee that this interface mode might expand the applicability of neural networks, as it allows relieving the pressure of forming the task either with fixed size input or in the form of, e.g., a sequence suitable for recurrent neural networks. However, it may require a replacement of the widely used convolutional layers. Although, there are new results [403] suggesting that fully connected neural networks might be a good architecture even for vision tasks.

4.2.2 The Use of Neural Networks for Producing HVs.

Transforming data into HVs (see Section 3 in [222]) might be a non-trivial task, especially when data are unstructured and of non-symbolic nature as, e.g., in the case of images (Sections 3.4.1 and 3.4.2 in [222]). Also, those transformations are usually not learned. This challenge stimulates the interface between neural networks and HDC/VSA in the other direction, i.e., to transform activations of neural network layer(s) into HVs. For example, as mentioned in Section 3.4.3 in [222], it is very common to use activations of convolutional neural networks to form HVs of images. This is commonly done using standard pre-trained neural networks [285, 296, 427]. Two challenges here are to increase the dimensionality and change the format of the neural network representations to conform with the HV format requirements. The former one is generally addressed by expanding the dimensionality, e.g., by random projection, possibly with a subsequent binarization by thresholding [138, 296]. Some neural networks already produce binary vectors (see [285]), and the transformation into HVs was done by randomly repeating these binary vectors to get the necessary dimensionality. To address the latter one, in [139, 144, 187, 208], the authors guided a convolutional neural network to produce HDC/VSA-conforming vectors with the aid of proper attention, sharpening, and loss functions. The sign of HV components can be used to transform them into bipolar HVs (of the same dimensionality). These approaches train neural networks from scratch (as with meta-learning in [139, 187, 208] or additive loss in [144]) such that the activations of the network resemble quasi-orthogonal HVs for, e.g., images of unrelated classes. In [285, 296], the authors superimposed HVs obtained from several neural networks, which improved the results in applications. Yet another promising avenue is make the processes of classification and reconstruction (i.e., generation) of raw sensory data simultaneously. One particular realization of this idea, called “bridge networks,” was recently presented in [304]. Finally, it is worth mentioning that a neural network does not necessarily need to produce HVs, but it can benefit from the HDC/VSA operations by improving its retrieval performance through superimposing multiple permuted versions of an output vector, as demonstrated in [56].

4.2.3 HDC/VSA for Simplifying Neural Networks.

In [2], it was shown that it is possible to treat the functionality of binarized neural networks with the ideas from high-dimensional geometry. The paper has demonstrated that binarized networks work because of the properties of binary high-dimensional spaces, i.e., the properties used in Binary Spatter Codes [179]. While it is an interesting qualitative result, it did not provide a concrete way to make the two benefit from each other. This is not obvious, since in the standard neural networks all weights are trained via backpropagation, which is rather different from the HDC/VSA principles.

There is, however, a family of randomized neural networks [371] where a part of the network is initialized randomly and stays fixed. There are two versions of such networks: feed-forward (e.g., random vector functional link networks [155] or extreme learning machines [152]) and recurrent (e.g., echo state networks [164] or reservoir computing [255]). The way the randomness is used in these networks can be expressed in terms of HDC/VSA operations for the both feed-forward [210] and recurrent [205] versions. Conventionally, randomized neural networks were used with real-valued representations. However, since it was realized that these networks can be interpreted in terms of HDC/VSA, it appeared natural to use binary/integer variants (as in Binary Spatter Codes and Multiply-Add-Permute) to produce activations of hidden layers of the networks. This opened the avenue for efficient hardware implementations of such randomized neural networks. Yet another connection between HDC/VSA and feed-forward randomized neural networks was demonstrated in [325] where it was shown that HRR’s binding operation can be approximated by such networks. Finally, in [23] it was shown that the address mechanism from the Sparse Distributed Memory [175] approximates the attention mechanism [405] used in modern neural networks.

4.2.4 HDC/VSA for Explaining Neural Networks.

It was discussed in Section 2.4 in [222] that the capacity theory [91] applies to different HDC/VSA models. As mentioned in the previous section, randomized recurrent neural networks, known as reservoir computing/echo state networks, can be formulated using HDC/VSA. Therefore, capacity theory can also be used to explain memory characteristics of reservoir computing. Moreover, using the abstract idea of dissecting a network into mapping and classifier parts [314], it is possible to apply capacity theory for predicting the accuracy of other types of neural networks (such as deep convolutional neural networks) [225].

In [264], it was shown that Tensor Product Representations approximate representations of structures learned by recurrent neural networks.

4.2.5 The Use of HDC/VSA with Spiking Neural Networks.

Another direction of the interplay between HDC/VSA and neural networks is their usage in the context of spiking neural networks (SNN). It is especially important in the context of emerging neuromorphic platforms [58, 273]. The main advantage HDC/VSA can bring into the SNN domain is the ease of transformation to spiking activities, either with rate-based coding for HDC/VSA models with scalar components or phase-to-timing coding for HDC/VSA models with phasor components. The Spaun cognitive architecture overviewed in Section 3.2.1 is one of the first examples where the HRR model was used in the context of SNN. The latest developments [93, 94] use FHRR and HRR to implement associative memory and k-Nearest Neighbor classifiers on SNN. Further, these memories were proposed as building blocks for the realization of a holistic HDC/VSA-based unsupervised learning pipeline on SNN [306]. While in [16] the Sparse Block Codes model was mapped to an SNN circuit. In other related efforts, an event-based dynamic vision sensor [142, 284] or an SNN [291, 436] was used to perform the initial processing of the input signals that were then transformed to HVs to form the prediction model.

These works provide some initial evidence of the expressiveness of HDC/VSA, on the one hand, and compatibility with SNNs, on the other. We therefore foresee that using HDC/VSA as a programming/design abstraction for various cognitive functionalities will soon manifest itself in the emergence of novel SNN-based applications.

4.2.6 “Hybrid” Solutions.

By “hybrid” in this context we refer to solutions that use both neural networks and some elements of HDC/VSA. Currently, a particularly common primitive used in such hybrid solutions is the representation of a set of role-filler bindings or superposition of multiplicative bindings. For example, in [37] the weights of several neural networks were stored jointly by using the superposition operation, which alleviated the problem of “catastrophic forgetting.” In [420], activations of layers of a deep neural network were used as filler HVs. They were bound to the corresponding random role HVs and all role-filler bindings were aggregated in a single superposition HV that in turn was used to successfully detect out-of-distribution data. Similarly, in [296, 299, 398], activations of several neural networks were combined together via HDC/VSA operations. In [296, 299], this idea was used to form a single HV compactly representing the aggregated neural networks-based image descriptor while in [398] outputs of multiple neural networks were fused together to solve classification problems. In [107], the superposition of role-filler bindings was used to simultaneously represent the output of a deep neural network when solving multi-label classification tasks. In [144], the activations of the last layer generate a query HV that resembles the superposition of the visual objects available in a panel, whereby each object is uniquely represented by multiplicative binding of its attributes’ HVs. In addition, a review of hybrid solutions combining Tensor Product Representations and neural networks such as Tensor Product Generation Networks [153] can be found in [387]. Finally, it is worth noting that all these solutions in some way relied on the idea of “computing in superposition” [204] suggesting that HVs can be used to simultaneously manipulate several pieces of information.