Learning the Use of Artificial Intelligence in Heterogeneous Catalysis

Introduction

The creation of predictive models in the oil industry is of the utmost importance. For example, models have been developed for oil production (Li and Horne, 2003; Irisarri et al., 2016; Hutahaean et al., 2017) and oil transformation (Farrusseng et al., 2003; Wang et al., 2019). These models, however, have not been entirely successful (Cai et al., 2021). Therefore, they have been constantly evolving and have now included artificial intelligence techniques. Examples of this are the models to find new catalysts (Günay and Yıldırım, 2021; Goldsmith et al., 2018; Lamoureux et al., 2019; Yang et al., 2020), the models to control oil exploitation (Davtyan et al., 2020; Liu et al., 2020; Tsvaki et al., 2020), and the models to analyze oil transformation (Al-Jamimi et al., 2019; Sircar et al., 2021).

In recent decades, models based on artificial intelligence techniques have performed impressive predictions in different knowledge fields. For example, in exact sciences (Sauceda et al., 2021; Hajibabaei and Kim, 2021; Cerioti et al., 2021; Bahlke et al., 2020; Chmiela et al., 2020; Saar et al., 2021; Artrith and Urban, 2016; Ch’ng et al., 2017; Shallue and Vanderburg, 2018; Sadowski et al., 2016), in social sciences (Ng et al., 2020; Lattner and Grachten, 2019), in technology (Bae et al., 2021; Cunneen et al., 2019; Feldt et al., 2018; Huang et al., 2014), and health sciences (Bashyam et al., 2020; Zhou et al., 2019; Themistocleous et al., 2021; Lagree et al., 2021).

The models based on artificial intelligence techniques contrast with the traditional models used for predictions in the oil industry. These techniques learn from the data associated with the process under study and generate relationships between the variables representing the process. It is important to remark that the learning mechanisms in artificial intelligence are generic, as it is the calculus, regardless of the source of the data to which it is applied. For example, the data to feed the learning system can be images of different types of objects (Wang et al., 2021; Torralba et al., 2008), or people´s tastes in movies (Keshava et al., 2020; Lash and Zhao, 2016), or the buyer behavior of shopping (Overgoor et al., 2019). The data may also come from words in a text (Devlin et al., 2019) or be associated with virus detection assays of people with Covid-19 (Zaobi et al., 2021), or come from a set of hydrodesulfurization experiments (Al-Jamimi et al., 2019).

The manuscript presents some applications of artificial intelligence techniques in heterogeneous catalysis. It includes the use of these techniques in the study of the hydrodesulfurization process. First, we present a brief overview of these techniques to give the reader, without experience in these techniques, the essential elements to understand the applications.

The first application shows the use of artificial intelligence techniques to predict the binding energy when a molecule interacts with a solid. In particular, the binding energy of CO or H when adsorbed by metals. The following application describes a learning machine that predicts surface areas and compares them with those obtained with the BET technique. The subsequent application searches for new nanoporous materials of interest in catalysis. The data comes from modeling the surface area of 6,500 zeolite structures.

In the following applications, the data come from experiments in research laboratories or production plants. First, we describe a learning machine that predicts new catalysts based on Ru for ammonia decomposition. In this case, to generate the data, ruthenium is replaced with three different elements (one at a time) to create catalysts that decompose ammonia under different experimental conditions. These experiments produced a data set to train the learning machine that predicts new Ru-based catalysts.

The applications that follow are related to the process of hydrodesulfurization. These kinds of applications were the first to be used in heterogeneous catalysis. For example, in 1996, one of these applications (Berger et al., 1996) predicted the hydrodesulfurization of atmospheric gas oil.

The last described application in the manuscript refers to the control of the hydrodesulfurization process in a production plant. In this case, the control occurs by using an online prediction based on time sequences.

Materials and Methods

A Basic Introduction to Machine Learning

As mentioned above, artificial intelligence techniques generate learning machines using data sets produced during the analysis of a topic of interest. As a result, these machines predict events that could occur in this topic.

Of the learning techniques, two stand out: supervised learning and unsupervised learning. Since everything presented in the paper is related to supervised learning, we describe the essential elements of this learning method for the non-specialist in artificial intelligence. Our presentation of this learning method does not follow any traditional way of presenting it. Instead, we explain the method using an example of heterogeneous catalysis, where the reader is a specialist.

When we develop a new catalyst for hydrodesulfurization, the goal is to bring out a product with low sulfur content. This content (which we describe as the variable Y) is the variable that determines the formulation of the catalyst, as well as the temperature, pressure, amount of hydrogen, and other variables related to the process that will use the catalyst. We mark out these last quantities with the vector X = (X1, X2, X3,…, Xn).

The use of artificial intelligence techniques to develop a new hydrodesulfurization catalyst requires a set of events, called samples, with values (X, Y). They can be obtained by modeling the hydrodesulfurization process or performing experiments in a laboratory or from the variables associated with the process in an industrial plant. Typically, 80% of the samples are used for training the machine, and the rest for testing the machine’s effectiveness to predict new catalysts.

The basic principle of learning is the following. First, we construct the function F(W, X), which estimates the sulfur content and is defined by the parameters W = (W1, W2, W3…). We create a metric to estimate the difference between the prediction of the sulfur content, F(W, X), and their values, Y, in the samples. With the average of these differences, we construct the loss function, J (W, X, Y), whose gradients with respect to each parameter permit updating the parameters. This updating continues until the loss function reaches its minimal value. Figure 1 depicts a general sketch of such a learning machine.

FIGURE 1. Sketch representing a learning machine. F(W, X), fed with the variable X, is represented by the left square. Its output (blue arrow) provides the right square, together with the variable Y, to form the error function, J(W, X, Y), during training . The parameters W are modified in this square to bring the function F(W, X) up to date.

There exist various algorithms to construct a learning machine (Breiman, 2001; Dhillon and Verma, 2020; Scherer et al., 2020; Schütt et al., 2018; Shazeer et al., 2018; Shawe-Taylor and Sun, 2011), from the simplest ones like the Random Forest (Breiman, 2001) which requires only a few samples, to algorithms based on artificial neural networks (Shazeer et al., 2018), which could require a large number samples for training. These last algorithms could contain millions of parameters (Krizhevsky et al., 2017). Going into detail on these algorithms is beyond the purpose of the present manuscript. However, the references for these algorithms are a good start for readers who want to dabble in them.

Results and Discussion

Adsorption Energy Prediction

When a molecule nears a solid, its interaction with the atoms on the solid’s surface determines the adsorption energy. If the solid is a catalyst, this energy determines many catalytic properties. Consequently, the analysis of this energy could help to develop new catalysts.

Although quantum mechanical calculations can provide this energy, they require long calculation times and much computing power. Therefore, the use of only this methodology to search for new catalysts generates a bottleneck.

An alternative to avoid this bottleneck is to generate these energies with the techniques of artificial intelligence. For example, through supervised learning using the Deep Learning technique. In this case, the constructed learning machine correlates the local atomic distribution and the properties of the atoms with the adsorption energy through the function F(W, X).

In this type of learning, it is necessary to define the target variable (the variable Y), which, in the present case, is the adsorption energy. It is also important to propose the variables from the system that determine this energy. For example, these variables can be the atomic properties of the atoms and their local atomic order (Behler, 2011; Back et al., 2019; Gómez-Peralta and Bokhimi, 2021). These variables can also correspond to relationships between the atomic radii (Gómez – Peralta and Bokhimi, 2020).

The function, F(W, X), generated with the Deep Learning technique, contains many parameters with values optimized during the training. Therefore, this learning procedure requires a considerable number of samples obtained from experiments or models. In the present case, each sample (X, Y) contains information related to the local distribution of the atoms and their chemical properties (variable X). In addition, it also includes the adsorption energy generated by quantum mechanical calculations (variable Y).

Thousands of samples (X, Y) are created by building atomic distributions used to perform quantum mechanical computations. The function F(W, X) maps X in Y through the relationship that lies between them without explicit modeling; for one sample, this function outputs a value that compares with the adsorption energy Y generating a difference value. For all samples, the average difference value gives the loss function J (W, X, Y), used to update the parameters W of the function F(W, X). This process repeats until the loss function reaches its minimal value to complete the learning process. It is important to note that the computations include the interaction between the molecule with catalytic and non-catalytic sites.

Of the samples, a percentage, about 80%, is used for training, and the rest to check the ability of the function F(W, X) to predict adsorption energies. This prediction allows finding new catalysts for specific catalytic processes related to the molecule adsorbed on the solid.

As an example of this methodology, (Back et al., 2019) used the Deep Learning technique to predict the adsorption energy (variable Y) of CO or H when they interact, each separately, with various surfaces of pure metals, metal alloys, or intermetallic compounds. Their model is a modification of the previously reported model for the prediction of the physicochemical properties of crystalline compounds (Xie and Grossman, 2018; Chen et al., 2019).

Back et al. identified (variable X) each atom with the group and period to which it belongs, its electronegativity, its covalent radius, its number of valence electrons, its first ionization energy, its electron affinity, the block to which they belong, and its atomic volume. They model the interaction between an atom with its local environment with its associated Voronoi polygon (Blatov, 2004).

The number of samples used by Back et al. for training was 43,237 for each adsorbate on the solid. Each sample had 37 different elements and 96 stoichiometries. In these samples, the number of different spatial groups was 110, while the number of crystal facets was 41.

The learning machine for CO adsorption had 4,938 parameters W, while the machine used for H adsorption utilized 6,738 parameters W.

For 12,000 samples excluded from the training set, the learning machine predicts adsorption energies for CO and H with a Mean Absolute Error (MAE) of only 0.15 eV. This value is lower than that obtained in previous studies by the authors using many alloys as the solid (Tran and Ulissi, 2018).

Surface Area Prediction

In heterogeneous catalysis, nanoporous materials are of particular interest because they have a large surface area. Hence, to help to understand the catalytic properties, this area should be determined accurately. The traditional method to get this area is the Brunauer-Emmett-Teller (BET) method, where for measuring this area, the material adsorbs inert molecules at low temperatures. But, the surface area estimation using this method is not always precise, especially for materials with large surface areas, where some of the adsorbed molecules are no more on a monolayer. Thus, for example, the reported surface areas of metal-organic frameworks (MOFs) are exaggerated (Gómez-Gualdrón et al., 2016; Sinha et al., 2019).

Datar et al. (Datar et al., 2020) developed a machine learning approach for estimating surface area to correct this problem. They combine machine learning with molecular modeling to derive argon isotherms at 87 K for over 300 metal-organic frameworks from the CoRE-MOF database (Chung et al., 2014; Chung et al., 2019). They use a Lennard-Jones potential with a cut radius of 12 Å (Rappe et al., 1992). Finally, they applied the BET theory to these isotherms to determine the surface area of these MOFs by using the SESAMI algorithm (Sinha et al., 2019), focusing primarily on the true monolayer area (Gómez-Gualdrón et al., 2016).

Their machine learning model relates the features that describe the adsorption isotherms and the surface area (variable Y). The isotherm features (variable X) were constructed by dividing the pressure into seven regions on a logarithmic scale. These regions are related linearly to the true monolayer areas. The model parameters were optimized using the Least Absolute Shrinkage and Selection Operator (Tibshirani, 1996). It includes regularization to reduce overfitting. For the training, Datar et al. used 40% for the samples for the training and 60% for the testing. The training evolution was analyzed using the cross-validation method (Zhang, 1993).

The learning machine predicts more precise surface areas than the BET method. For example, for samples with surface areas greater than 3,500 m²/ g, with the learning machine, only 2% of the structures were predicted with surface area errors more significant than 20%, while for the BET method, the latter figure was 54%.

With the trained learning machine, the surface area of 68 MOF structures of the CoRE-MOF database was obtained, modeling the isotherms with the adsorption of N₂ (77 K). The predictions of the surface area using these isotherms with the learning machine trained with the isotherms generated with the adsorption of Ar also gave better results than those obtained by the BET approach. The estimated areas with the learning machine were scaled to a ratio of 1.148 since the area associated with the N₂ molecule is 0.163 nm², while the one associated with the Ar atom is 0.142 nm² (Mikhail and Brunauer, 1975).

Adsorption Isotherm Prediction

Discovering new materials is of the utmost importance from both technological and scientific points of view. That is why, in recent years, the use of artificial intelligence techniques to speed up their search has proliferated (Goldsmith et al., 2018; Gupta et al., 2018; Gómez-Peralta and Bokhimi 2020; Chen et al., 2021; Cho and Lin, 2021; Konno et al., 2021). Of particular interest is the finding of nanoporous materials because they have a large surface area that is attractive in heterogeneous catalysis (Cho and Lin, 2021).

These materials can exist in hundreds of thousands. However, nowadays, selecting those with an attractive surface area for heterogeneous catalysis occurs through the slow procedure of trial and error. As an alternative to this discovery procedure, Cho et al. (Cho and Lin, 2021) developed a learning machine based on convolution neural networks (CNN) that predicts the surface area of such materials.

Their methodology analyzes methane adsorption on zeolites, but it applies to any molecule adsorption in any nanoporous material. In their research, Cho et al. used 6,500 zeolite structures with lattice parameters less than 24 Å, selected from the Predicted Crystallography Database (Pophale et al., 2011).

They calculated the methane adsorption isotherm (variable Y) at 300 K on every zeolite structure for 14 different pressure values between 0.0005 and 200 bar. For that, they employed the Monte Carlo method approach, using a Lennard-Jones type interaction potential (García-Pérez et al., 2007) with a cut-off radius of 12 Å. Their learning machine had the LeNet-5 architecture (LeCun et al., 1998), commonly used to detect objects in two-dimensional images (Qin et al., 2020; Jain et al., 2021).

Each zeolite structure was modeled with a three-dimensional image equivalent to a cubic cell with 24 Å per side. Each of the 24 × 24 × 24 image voxel values (variable X) corresponded to its potential as a methane adsorption active site. Filters of 2 × 2 × 2, 3 × 3 × 3, and 5 × 5 × 5 voxels built the different layers of the convolutional neural network. Of the total samples, 90% were for training and 10% for testing the model. The use of data augmentation techniques (Shorten and Khoshgoftaar, 2019) decreased the loss values, achieving squared means errors of up to 0.015 mol/ kg.

Their learning machine predicts adsorption isotherms in fractions of seconds, which contrasts with the times required by using the Monte Carlo method, where to calculate the adsorption isotherm in a zeolite requires dozens of CPU hours. This difference in time is significant since there are hundreds of thousands of such zeolite structures from which it is interesting to obtain their adsorption isotherms.

These studies show the potential of convolutional neural networks to develop other applications in heterogeneous catalysis. For example, for the capture of carbon dioxide or removing H₂S or SO₂ during sour gas sweetening.

Prediction of Novel Catalysts

The learning machines described above, for their learning, were fed with features defined by the properties of the elements and features obtained with calculations based on the atomic distribution. This methodology allowed generating a considerable number of samples for the training.

In contrast, Williams et al. (Williams et al., 2020) describe a learning machine in which part of the content of the samples comes from experiments, which limited the number of samples to less than 500.

In Machine Learning, when the number of samples is tiny, and the goal is to have a minor error in the predictions, it is convenient to choose an algorithm that learns correctly with this number of samples. One of these algorithms is the Random Forest (Breiman, 2001), which Williams et al. used to create their learning machine based on 100 tree predictors.

Their learning machine looked for new catalysts based on ruthenium for ammonia decomposition. They started with the ruthenium catalyst Ru₄K₁₂, promoted with potassium, and supported on gamma-alumina. Their research sought to predict new catalysts by substituting Ru with 33 different elements (Mg, Ca, Sc, Ti, V, Cr, Mn, Fe, Co, Ni, Cu, Zn, Sr, Y, Zr, Nb, Mo, Rh, Pd, Ag, Cd, In, Sn, Hf, Ta, W, Re, Os, Ir, Pt, Au, Pb, Bi).

To develop their first learning machine, they prepared catalysts by replacing one Ru atom with an M atom (M = Ca, Mn, In). The composition of the catalysts was 3 wt% Ru, 1 wt% M and 12 wt% K. For each catalyst, they measured the ammonia conversion at 250, 300, and 350°C; this conversion is the target variable (variable Y) that supervises the learning. It is to notice that selecting these elements to replace Ru maximizes the difference between the features used in the learning.

The features (variable X) used to characterize the ammonia decomposition process were the catalyst composition, the operating conditions, the synthesis variables, and the atomic properties. Only five of these features were necessary to get accurate predictions, being the reactor temperature the most important. The other four features were related to the electronic structure of the atoms: the number of d-shell valence electrons, the electronegativity, the covalent radius, and the adjusted work function. It is important to remember that in 1985 Falicov et al. proposed that the number of d-shell valence electrons was essential to describe the catalytic activity (Falicov and Somorjai, 1985).

When evaluating the learning machine by replacing Ru with an atom M of the remaining 30 elements, the best ammonia conversions occurred with M = Sr, Mg, Sc, or Y.

These predictions motivated the authors to synthesize different catalysts by substituting Ru with the following 19 elements: Cu, Ni, Cr, W, Hf, Zn, Bi, Pd, Mo, Y, Sc, Sr, Mg, Os, Pt, Au, Nb, Fe, and Rh. When they catalyzed ammonia conversion, the highest conversions corresponded to the substitution of Ru with Sr, Mg, Sc, or Y, which matches the learning machine’s predictions, and demonstrates the usefulness of this machine to predict novel catalysts.

Williams et al. extended their model by generating new sets of catalysts using, for each set, three different atoms to replace Ru, selecting the atoms from the 19 elements mentioned in the previous paragraph, plus Ca, Mg, and In.

With these 22 elements replacing Ru, 60 different sets of three catalysts were built, generating 60 different learning machines. These machines predominantly predict that the catalysts that produce a high ammonia conversion are those in which Ru is replaced by Ca, Sr, Sc, Y, and Mg. These predicted conversions have a 10% error compared with those measured in experiments.

The results show that artificial intelligence techniques and relatively few experiments are sufficient to save both cost and time to discover new catalysts.

Sulfur Content Prediction After Hydrodesulfurization

Since the inception of machine learning techniques, they have been used to predict sulfur content after hydrodesulfurization (HDS). In 1996, Berger et al. (1996) constructed a learning machine based on a Feed-Forward Neural Network with only one hidden layer containing three nodes. Their experiments provided the 25 samples utilized to train and test this learning machine. Eighteen of them were for its training and seven to test it. For all the hydrodesulfurization experiments, they employed a catalyst with 4.2 wt% Co, 16.7 wt% Mo and 0.4 wt% S, with a hydrogen to oil volume ratio of 500.

The target variable (variable Y) was the sulfur content after the hydrodesulfurization process, with values between 10 and 1870 ppm. The features (variable X) that characterized the process were: temperature, with values between 348 and 360°C, pressure, with values between 600 and 1,200 psi, liquid hourly space velocity with values between 0.4 and 6.0 h⁻¹, and the sulfur content of the heavy atmospheric gas oil, with values between 7,500 and 12,000 ppm. Since, in this study, the experimental sulfur output values have little change in the pressure range used, the study shows that, in this case, the pressure does not have a significant effect on the sulfur output predictions.

Despite the tiny number of samples used in training, the sulfur content predictions with the test samples were within 10% of the experimental values.

Another learning machine, developed by Al-Jamimi et al. (Al-Jamimi et al., 2019), predicts the sulfur content after an HDS process. They used a Support Vector Machine (Shawe-Taylor and Sun, 2011) combined with a genetic algorithm (Katoch et al., 2021) as the learning technique.

The ultimate sulfur content in the product (variable Y), with values between 0.0 and 1,258 ppm, is the variable that supervises the learning. The features (variable X) that characterized the HDS process were temperature, with values of 200 300 and 400°C, pressure, with values between 25 and 75 bar; hydrogen dosage, with values between 0.4 and 0.8 g, the initial sulfur concentration, with values between 439 and 1,500 ppm and the type of fuel. The authors generated 34 samples performing HDS experiments, 24 to train the learning machine, and 10 to test its prediction capability. After its training, the machine delivered predictions regarding the product’s sulfur content within 5.5 percent of the experimental results.

In the literature, some publications also report the use of artificial intelligence techniques to examine the evolution of the HDS process in industrial plants over time (Ma et al., 2020) or the evolution of this process in a refinery, with temperature, pressure, and hydrogen dosage (Arce-Medina and Paz-Paredes, 2009; Al-Jamimi et al., 2018). The literature also covers the use of these techniques for predictions in other oil industry processes (Sagheer and Kotb, 2019; Liu et al., 2020).

Conclusions

We describe some applications of artificial intelligence techniques in heterogeneous catalysis. Since they all correspond to supervised learning, first, we give a brief introduction to this kind of learning, emphasizing the variables X and Y that define it. Then, when we use this learning to analyze a problem, the essential point is to find these variables to represent it. Therefore, for each of the described applications, there is a particular emphasis on finding these variables. This emphasis will help extract this information when one is interested in a new application. In the end, the description intends to give readers some clues for using these techniques in their research or the industrial applications related to hydrodesulfurization.

The reported applications describe the building of learning machines that infer adsorption energies, surface areas, adsorption isotherms of nanoporous materials, novel catalysts, and the sulfur content after hydrodesulfurization.

The learning machine that predicts the adsorption energy learns from the interaction of CO or H on surfaces of pure metals, metal alloys, or intermetallic compounds. The one that predicts surface areas learns from the adsorption of argon on 300 different metal-organic frameworks. The machine that forecasts adsorption isotherms learn from the adsorption of methane on 6,500 zeolite structures. The machine that infers novel catalysts based on ruthenium learns from the ammonia decomposition using catalysts generated by the partial substitution of Ru with 22 different atoms. Finally, the machine inferring the sulfur content after hydrodesulfurization learns from the variables used to perform the experiments: temperature, pressure, hydrogen dosage, and initial oil sulfur concentration.

Learning machines can predict adsorption energies with mean absolute errors of 0.15 eV for a diverse chemical space. They predict more precise surface areas of porous materials than the BET technique and can calculate their isotherms much faster than the Monte Carlo method. These machines can also predict new catalysts by learning from the catalytic behavior of materials generated through atomic substitutions. When the machines learn from the variables associated with a hydrodesulfurization process, they can predict the sulfur content in the final product.

Author Contributions

XB. Conceptualization, Methodology, Resources, Writing-original draft, review and editing.

Funding

This work was financial supported by the laboratory, LAREC, from the Instituto de Física, Universidad Nacional Autónoma de México. Mexico.

Conflict of Interest

The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

Mr. Antonio Morales provided technical assistance to the author.

References