Variable Chromosome Genetic Algorithm for Structure Learning in Neural Networks to Imitate Human Brain

Featured Application

Neural Network Structure Learning is expected to overcome difficulty of constructing a neural network structure.

Abstract

This paper proposes the variable chromosome genetic algorithm (VCGA) for structure learning in neural networks. Currently, the structural parameters of neural networks, i.e., number of neurons, coupling relations, number of layers, etc., have mostly been designed on the basis of heuristic knowledge of an artificial intelligence (AI) expert. To overcome this limitation, in this study evolutionary approach (EA) has been utilized to automatically generate the proper artificial neural network (ANN) structures. VCGA has a new genetic operation called a chromosome attachment. By applying the VCGA, the initial ANN structures can be flexibly evolved toward the proper structure. The case study applied to the typical exclusive or (XOR) problem shows the feasibility of our methodology. Our approach is differentiated with others in that it uses a variable chromosome in the genetic algorithm. It makes a neural network structure vary naturally, both constructively and destructively. It has been shown that the XOR problem is successfully optimized using a VCGA with a chromosome attachment to learn the structure of neural networks. Research on the structure learning of more complex problems is the topic of our future research.

1. Introduction

The purpose of artificial intelligence (AI) is to imitate the human intelligence [1]. In order to imitate human intelligence, the research of AI has been developed as an algorithm to figure out a certain issue. Artificial neural network (ANN) is one of these algorithms. In other words, the development of ANN is to imitate the process of the human brain [2]. The ANN is already widely used for an estimation of automotive brake pressure, which has a great potential to achieve a simple design of the braking control system without sensors [3].

The compute unified device architecture (CUDA) is a parallel processing technology that used GPU. Since the introduction of the CUDA, deep learning has been rapidly developed and applied to various fields in the industry. Recently, big tech companies and car makers such as Google, BMW, and Tesla have expanded their interests to research in deep learning [4]. In particular, the convolutional neural network (CNN) is widely implemented to image recognition [5,6] and the recurrent neural network (RNN) is used for various applications such as sentence and voice recognition [7].

As ANNs have been scaled up and improved, they have become much more complex [8]. Even to realistically imitate a human brain, a system must be able to acquire the individual personality models of that individual [9]. Although human engineers are able to tune and optimize parameters through experimentation, neural networks are prone to have complex topologies or hyper-parameters. Previously, the design of neural networks was based on heuristics [10]. Since the structure significantly varies according to the number of neurons or the purpose of the problem, the optimal structure is hard to find.

Currently, employed architectures have mostly been developed by human experts, which is a time-consuming and error-prone process [11]. As deep learning has scaled up to more challenging tasks, the architectures have become difficult to design by hand [8]. For these reasons, various studies have been carried out to generate the network architecture automatically [11,12]. Structure learning is a very useful instrument that is able to automatically find an appropriate ANN architecture [13]. For this reason, the structure learning algorithm is used to generate the topology of ANN in this study.

This study proposed a chromosome attachment as a new genetic operation to implement the structure learning of ANN. It makes the ANN vary both constructively and destructively. Through this operation, a structure learning method in the process of a neural architecture search (NAS) is possible. NAS has three challenges: Designing a network structure, putting components into the network topology, and hyper-parameters tuning [8]. The proposed method in this study presents a challenge through structure learning.

Evolutionary approach (EA) is usually used to automatically design the topology of an ANN topology. There are many neural network architecture search algorithms with an EA. Generally, constructive or destructive algorithms are used to automatically design architectures [14]. However, most EAs only use a constructive algorithm.

A typical constructive algorithm starting with a minimal network is the genetic algorithm (GA). GA has been the most common method used for designing neural network architecture [11,12,13,14,15,16,17,18]. In these studies, GA was used to generate the network structures depending on their purposes. It can be applied to a system having a network structure, such as a Bayesian network or a neural network.

In most research focused on the automatic generation of the neural network structure, they were carried out to control the number of neurons through a genetic algorithm in a network structure that has one hidden layer [13,16,17]. In these studies, the neurons belonging to the hidden layer are increased or decreased according to generations while searching the suitable number of neurons.

NeuroEvolution of Augmenting Topologies (NEAT) is a representative EA methodology. NEAT outperform the best fixed topology methodology on challenging benchmarking reinforcement learning [15]. NEAT is used as a typical evolutionary approach [8]. The constructive algorithm uses GA, starting with a minimal structure of artificial neural network. The initial neural network has a minimal structure, however it becomes gradually more complex to search an optimal architecture with NEAT methodology.

On the other hand, NEAT methodology only can evolve the neural network architecture to a larger structure because it has to start with a minimal structure. If the neural network architecture in the solution area is too large to increase gradually, it is time consuming to search a right answer. Thus, destructive operation is necessary to design a more complex or larger neural network.

Variable chromosome genetic algorithm (VCGA) is applied to this method to implement the structure learning of ANN so as to imitate a human brain. The previous structure learning method of the ANN could only evolve constructively. However, VCGA uses chromosome attachment for a system that is able to evolve the structure of an ANN, not only constructively but also destructively. Destructive method like pruning is necessary. Since the size of the ANN is unknown at the beginning, design method such as pruning is required. Furthermore, the structure learning of a big neural network spends a large amount of time [16,19].

The proposed method is applied to the problem of generating a XOR network. Although this task is a simple problem, it requires growing and pruning hidden neurons [15].

The rest of this paper is organized as follows. Section 2 describes the proposed methodology of applying VCGA to neural network structure learning. Section 3 analyzes the results of simple case study. Finally, conclusions and future works are drawn in Section 4.

2. Methodology

As discussed in Section 2.2, previous methodologies typically start with a minimal structure of neural network in order to become an optimized architecture. In contrast, VCGA with a chromosome attachment is used to generate a neural architecture both constructively and destructively in this study. The chromosome attachment operation gives a destructive approach to GA. It makes possible for neurons and ls to increase or decrease various directions in order to search an optimized architecture on a large solution area. Thus, it is more suitable to search a large and complex architecture of neural network.

2.1. Structure Learning Based on Variable Chromosome Genetic Algorithm

In this study, VCGA is used to search the neural network architecture. Figure 1 provides an overview of methodology used in this paper in the structure learning process. In Phase 1, the chromosome set is generated first. These chromosomes represent neurons and connections. An individual has multiple chromosomes and it composes an ANN structure. In Phase 2, each generated ANN structures are simulated. Simulation results are calculated by fitness function and ANN structures are sorted by result score. In Phase 3, genetic operations are implemented to make offspring: Selection, cross over, mutation, and attachment. These new generated chromosome set make new individuals and an ANN structure on Phase 1. Equation (1) shows a fitness function.

$$Fitness value=\frac{1}{2}score -\frac{1}{2}D\times N\left\{2\times \left(score-0.5\right)\right\}$$

(1)

Score means the accuracy of predicted output value of the ANN structure and it has a real value between 0.0 and 1.0. D means a dependency on the number of chromosomes. The larger D is, the more important the minimum structure of the ANN is. And the smaller D is, the more important the accuracy of the predicted output value is. N means relative number of chromosomes; (number of own chromosomes)/(number of average chromosomes). In this case study, the constant D was set to 0.2.

2.2. Chromosome Type of Artificial Neural Network

In order to encode an ANN into a chromosome, ANN is represented as two cases of chromosome types. A chromosome can be expressed as an ANN topology [20].

Figure 2 shows two types of chromosome type: Neuron information and connection information. The chromosome type of neuron information consists of the unique number of neuron, the number of the input ports, and the number of the output ports as shown in Figure 2a.

The chromosome type of connection information consists of the unique number of target neuron, the number of the port to input, and the unique number of the starting neuron as shown in Figure 2b.

Figure 3 shows examples of chromosomes and ANN topologies. The chromosome type of the neuron N5 is shown in Figure 3b, assuming that there is a neural network of the ANN topology shown in Figure 3a. It means the neuron N5 has two input ports and one output port. The chromosome type of connection information is shown in Figure 3c. It is a connection from N3 to N5 by port 1 of N5.

Chromosomes composed of these two types of chromosome constitute a single individual. The number of chromosomes in an individual means the sum of the number of chromosomes that store neuron information and the number of chromosomes that store connection information.

Figure 3 shows examples of chromosome types and ANN topologies. The chromosome type of the neuron N5 is shown in Figure 3b, assuming that there is a neural network of the ANN topology shown in Figure 3a. It means the neuron N5 has two input ports and one output port. And the connection from N3 to N5 has a chromosome type of the form shown in Figure 3c.

Chromosomes composed of these two types of chromosome types constitute a single entity. The number of chromosomes in an individual means the sum of the number of chromosomes that store neuron information and the number of chromosomes that store link information.

2.3. Application of Genetic Operation

The genetic operations applied in VCGA are mutation, cross-over, and chromosome attachment [21]. Mutation and cross-over can change the contents of chromosomes as shown in Figure 4. It changes the number of input ports or output ports of existing neurons, and target or start the neuron of existing connections. For example, mutation modifies a selected gene of a chromosome randomly as shown in Figure 4a and cross-over exchanges a gene of each parent as shown in Figure 4b.

On the other hand, chromosome attachment adds or discards a neuron or a connection between neurons by changing the number of chromosomes in an individual as shown in Figure 5. The mutation in biology has two types: Gene mutation and chromosome mutation. Well-known mutations (in biology) like mutation (in GA) or cross-over can be treated as a kind of gene mutation which changes the base sequence of DNA [22]. However, chromosome mutation only changes the number of chromosomes. Chromosome attachment is a chromosome mutation [23]. It is also called nondisjunction in biology [24]. It makes a change that varies the number of chromosome offspring. This is a kind of mutation that causes chromosome aberration. However, through this operation both new neurons and connections may appear or existing neurons may disappear. Since there are no restrictions on the layer, theoretically there may also be multi-hidden layers and skip connections. This operation is first applied to the GA and EA through this study.

3. Case Study: XOR Problem

As a case study, the proposed methodology was applied to solve the XOR problem [25]. The XOR problem is the most fundamental form of the problem in multilayer perceptron. Although the correct answer is already a well-known problem, it is very important for the imitation of the human brain. The proposed methodology is compared to NEAT. The simulation automatically found the optimized structure for the XOR without giving any information about the correct answer. If it succeeds in finding the correct answer, it should have five chromosomes of neuron information, six chromosomes of connection information, and 11 chromosomes in total.

Two simulations for verifying a chromosome attachment operation applied on EA of ANN were implemented. The first simulation was start from a minimal structure and was compared to the NEAT methodology. The second simulation started from a large structure. It verified that the chromosome attachment operation was useful for auto-generating large and complex ANN structure.

3.1. Initialization

NEAT began with an initial population of uniform topology. In contrast, the proposed methodology began with a random population. The initial populations generated 200 individuals. The average number of chromosomes in the first generation was about 8 in the first simulation. In the second simulation, the average number of chromosomes in the first generation was about 19 since they have larger structures than the first simulation. The probabilities of mutation, cross-over, and attachment were all 0.2 and the learning rate of the neural network was 0.2. This value was approximate to NEAT [15]. The simulation was implemented for a total of 300 generations in the first simulation and 2000 generations in the second simulation.

3.2. Simulation Result

Figure 6 plots the score and the number of chromosomes of the neural network structure with the highest score for each generation of the first simulation. The score of the highest performing species began from 0.65 and increased to 0.95. If a score of 0.9 or higher was obtained, the generated neural network could correctly obtain the exclusive or (XOR) value. This system found a structure for XOR in 48 generations. This solution structure had 32 chromosomes. As the generation progressed, the number of chromosomes gradually decreased. It took 218 generations to find the optimal form with 11 chromosomes for the first time. Since then, it appeared to oscillate and converge.

Figure 7 plots the results of the second simulation. In this simulation, species of the first generation were fully connected with 5 neurons. The score of the highest performing species began from 0.97 and decreased to 0.95, however, it increased again to 0.97. This system started from a completed ANN structure. However, it decreased the number of chromosomes after starting. Neurons and connections decreased together until the number of chromosomes reached 11.

Table 1. Comparison with NeuroEvolution of Augmenting Topologies (NEAT) in exclusive or (XOR) problem.

Method	Average of Generation Finding Solution	Average Number of Hidden Nodes	Average Number of Disabled Connections	Evolution Directed to Decreasing
NEAT	32	2.35	7.48	impossible
using variable chromosome genetic algorithm (VCGA)	about 40	2	0	possible

shows a comparison between the results of structure learning with NEAT and the proposed method about the XOR problem. The NEAT methodology found a solution faster than the proposed methodology using VCGA. However, NEAT evolved networks with 2.35 hidden neurons and 7.48 disabled connections, while the proposed method’s solutions used 2 hidden neurons and 0 disabled connections. Since NEAT only uses the add operation, it cannot evolve with the destructive approach. On the other hand, the proposed methodology with VCGA allows decreasing neurons or connections with the chromosome attachment. The result is important because it shows that a methodology using VCGA has the potential to be applied to large and complex problems while performing as well as NEAT.

4. Conclusions

In this study, the structure learning methodology for an ANN using a VCGA with a chromosome attachment was proposed. Previous methods including NEAT only use the constructive approach. Thus, it only can design an ANN from a minimal structure. Due to this limit, the previous methodology has difficulty in designing large structures. However, the proposed method enabled the evolutionary algorithm to destructive approach by using the chromosome attachment. As a result, the given initial ANN could be tuned to auto-design the structure after structure learning, whether the initial ANN was a minimum structure or not. It is important to note that the approach could learn a larger ANN.

The proposed methodology evolved the ANNs, starting with a minimal structure and completed structure. It succeeded in finding the right answer to the XOR problem without any information as part of a simple case study that compared it with the previous method. On the other hand, using the proposed methodology, it was possible to generate more complex neural network structures faster because of the bout constructive and destructive approach. It is expected that the cost of using specialists in designing ANN structures will also reduce.

The expansion of searching space to more complex ANN such as image recognition and situation awareness of auto driving system is the topic of our future research.