A regularized stochastic configuration network based on weighted mean of vectors for regression

SCN

SCN is a novel randomized incremental learner framework with a supervisory mechanism. The universal approximation property of SCN is guaranteed by its innovative inequality constraint. The network structure of SCN is depicted in Fig. 1.

Let $\mathrm{\Gamma }:=\{g_1,g_2,g_3,\dots \}$ be a set of real-valued functions, span( $\mathrm{\Gamma }$) denotes a function space spanned by $\mathrm{\Gamma }$, and $L_2(D)$ represents the space of all Lebesgue measurable functions $f=[f_1,f_2,\cdots ,f_m]:R^d\to R^m$ on a set $D\subset R^d$. $f_{L-1}(x)={\displaystyle \sum _j^{L-1}{\beta }_j}{g}_j(w_j^{T}x+b_j)$ represents the output of a single layer feed-forward network with $L-1$ hidden nodes, where $w_j$ and $b_j$ denote the parameters of the $j$th hidden node, and $g_j(\cdot )$ is the activation function of the $j$th hidden node. The inner product of $\theta =[{\theta }_1,{\theta }_2,\dots ,{\theta }_m]:R^d\to R^m$ and $f$ is:

(1) $$<f,\theta >:={\sum }_{q=1}^m<f_q,{\theta }_q>={\sum }_{q=1}^m{\int }_D{f}_q(x){\theta }_q(x)dx.$$

Given a training dataset $(x_i,t_i{)}_{i=1}^N$, $x_i=[x_{i,1},\dots ,x_{i,d}]\in R^d$ and $t_i=[t_{i,1},\dots ,t_{i,m}]\in R^m$, suppose that an established SCN model contains $L-1$ hidden nodes. The network residual error is:

(2) $$\begin{gathered} E_{L-1}(x)=f(x)-f_{L-1}(x)=[E_{L-1,1}(x),E_{L-1,2}(x),\dots ,E_{L-1,m}(x)]\in R^{N\times m}, \\ {E}_{L-1,1}(x)=[E_{L-1,1}(x_1),E_{L-1,1}(x_2),\dots ,E_{L-1,1}(x_N){]}^T, \\ {E}_{L-1,2}(x)=[E_{L-1,2}(x_1),E_{L-1,2}(x_2),\dots ,E_{L-1,2}(x_N){]}^T, \\ ⋮ \\ {E}_{L-1,m}(x)=[E_{L-1,m}(x_1),E_{L-1,m}(x_2),\dots ,E_{L-1,m}(x_N){]}^T. \end{gathered}$$where $f$ is the given target function, and $f_{L-1}$ represents the output of the network with $L-1$ hidden nodes. The residual error of SCN gradually decreased as the number of hidden neurons increased. If the value of $E_{L-1}$ was larger than the tolerance threshold $\tau$, SCN added a new hidden node into the network until $E_L$ was smaller than $\tau$. The parameters of the added hidden node were assigned randomly under a set of inequalities.

(3) $$<E_{L-1,q},g_L{>}^2\ge b_g^2(1-r-{\mu }_L)||e_{L-1,q}|{|}^2,q=1,2,...,m,$$where $<E_{L-1,q},g_L>$ is the inner product of $E_{L-1,q}$ and $g_L$, $\{{\mu }_L\}$ represents a nonnegative real number sequence with $li{m}_{L\to +\mathrm{\infty }}{\mu }_L=0$, and ${\mu }_L\le (1-r)$. $g$ indicates a non-linear activation function. $\mathrm{\forall }g\in \mathrm{\Gamma }$ (span( $\mathrm{\Gamma }$) is dense in $L_2$ space), $0<\parallel g\parallel <b_g$ ( $b_g\in R^{+}$). $r$ determines the strictness of the inequality constraint, and $0<r<1$. The output weights were evaluated by:

(4) $${\beta }^{\ast }=\arg \underset{\beta }{min}\left||H_L\beta -T\right|{|}_F^2=H_L^{†}T,$$where $H_L^{†}$ represents the Moore-Penrose inverse of $H_L$ and $H_L$ is the output matrix of the hidden layer. Readers may refer to Wang & Li (2017b) for more details on the SCN framework and associated algorithms.

INFO

INFO is a new population-based optimization algorithm that employs updating rule, vector combining, and local search to move the population’s position in D dimensional search domains. Given a population with $N_P$ vectors, $X_{l,j}^g=\{x_{l,1}^g,x_{l,2}^g,...,x_{l,D}^g\},l=1,2,...,N_P$.

• Updating rule

INFO randomly selected three differential vectors ( $a1\ne a2\ne a3$) to calculate the weighted mean of vectors. To increase the diversity of the population, the best, better, and worst solutions were employed to define the MeanRule (mean-based rule).

(5) $$MeanRule=k\times WM{1}_l^g+(1-k)\times WM{2}_l^g.$$

(6) $$\begin{array}{ll}WM{1}_l^g& =\alpha \times \frac{w_1(x_{a1}-x_{a2})+w_2(x_{a1}-x_{a3})+w_3(x_{a2}-x_{a3})}{w_1+w_2+w_3+\epsilon }+\epsilon \times rand,\\ \hfill l& =1,2,\mathrm{...},N_p,\hfill \end{array}$$where

(7) $$w_1=\cos ((f(x_{a1})-f(x_{a2}))+\pi )\times exp(-\frac{f(x_{a1})-f(x_{a2})}{\omega }).$$

(8) $$w_2=\cos ((f(x_{a1})-f(x_{a3}))+\pi )\times exp(-\frac{f(x_{a1})-f(x_{a3})}{\omega }).$$

(9) $$w_3=\cos ((f(x_{a2})-f(x_{a3}))+\pi )\times exp(-\frac{f(x_{a2})-f(x_{a3})}{\omega }).$$

(10) $$\omega =max(f(x_{a1}),f(x_{a2}),f(x_{a3})).$$

(11) $$\begin{array}{ll}WM{2}_l^g& =\alpha \times \frac{w_1(x_{bs}-x_{bt})+w_2(x_{bs}-x_{ws})+w_3(x_{bt}-x_{ws})}{w_1+w_2+w_3+\epsilon }+\epsilon \times rand,\\ \hfill l& =1,2,...,N_P,\hfill \end{array}$$where

(12) $$w_1=\cos ((f(x_{bs})-f(x_{bt}))+\pi )\times exp(-\frac{f(x_{bs})-f(x_{bt})}{\omega }).$$

(13) $$w_2=\cos ((f(x_{bs})-f(x_{ws}))+\pi )\times exp(-\frac{f(x_{bs})-f(x_{ws})}{\omega }).$$

(14) $$w_3=\cos ((f(x_{bt})-f(x_{ws}))+\pi )\times exp(-\frac{f(x_{bt})-f(x_{ws})}{\omega }).$$

(15) $$\omega =f(x_{ws}).$$

The weighted mean of vectors was used to generate two new vectors.

(16) $$\{\begin{array}{c}\{\begin{array}{c}z{1}_l^g=x_l^g+\sigma \times MeanRule+randn\times \frac{(x_{bs}-x_{a1}^g)}{(f(x_{bs})-f(x_{a1}^g)+1)},\hfill \\ z{2}_l^g=x_{bs}+\sigma \times MeanRule+randn\times \frac{(x_{a1}^g-x_{a2}^g)}{(f(x_{a1}^g)-f(x_{a2}^g)+1)},\hfill \end{array}rand<0.5,\hfill \\ \{\begin{array}{c}z{1}_l^g=x_a^g+\sigma \times MeanRule+randn\times \frac{(x_{a2}^g-x_{a3}^g)}{(f(x_{a2}^g)-f(x_{a3}^g)+1)},\hfill \\ z{2}_l^g=x_{bt}+\sigma \times MeanRule+randn\times \frac{(x_{a1}^g-x_{a2}^g)}{(f(x_{a1}^g)-f(x_{a2}^g)+1)},\hfill \end{array}rand\ge 0.5,\hfill \end{array}$$where $f(x)$ was defined as the objective function, three different integers ( $a1,a2,a3$) were randomly chosen from $[1,N_P]$, $z{1}_l^g$ and $z{2}_l^g$ were two new vectors, and $\sigma$ was the scaling rate of a vector.

• Vector combining

The two new vectors $z{1}_l^g$ and $z{2}_l^g$ were combined with vector $x_l^g$ to generate a new vector ${\mu }_l^g$.

(17) $$\{\begin{array}{c}\{\begin{array}{c}{\mu }_l^g=z{1}_l^g+\mu .|z{1}_l^g-z{2}_l^g|,rand<0.5,\\ {\mu }_l^g=z{2}_l^g+\mu .|z{1}_l^g-z{2}_l^g|,rand\ge 0.5,\end{array}rand<0.5,\\ {\mu }_l^g=x_l^g,rand\ge 0.5,\hfill \end{array}$$where ${\mu }_l^g$ was the composite vector of the $g$th generation.

• Local search

The local search operator used the global position ( $x_{best}^g$) and the MeanRule to help INFO convergence to global optima.

(18) $$\{\begin{array}{cc}{\mu }_l^g=x_{bs}+randn\times (MeanRule+randn\times (x_{bs}^g-x_{a1}^g)),& rand<0.5,\\ {\mu }_l^g=x_{rnd}+randn\times (MeanRule+randn\times (v_1\times x_{bs}-v_2\times x_{rnd})),& rand\ge 0.5,\end{array}$$in which

(19) $$x_{rnd}=\varphi \times x_{avg}+(1-\varphi )\times (\varphi \times x_{bt}+(1-\varphi )\times x_{bs}).$$

(20) $$x_{avg}=\frac{(x_a+x_b+x_c)}{3}.$$where $rand$ and $\varphi$ were two random values within [0,1] and (0,1) respectively. The random value $v_1$ and $v_2$ increased the best position’s influence on the vector. INFO updated the best vector ( $x_{best}$) and returned $x_{best}^g$ as the final solution. For more details about the INFO algorithm, refer to Ahmadianfar et al. (2022).

RSCN

Given a training dataset $(x_i,t_i{)}_{i=1}^N$, the objective function of the SCN with $L_2$-norm penalty term could be expressed as:

(21) $$\begin{gathered} min:J=\frac{1}{2}||\beta |{|}^2+\frac{\eta }{2}{\sum }_{i=1}^N{E}_i^2, \\ s.t:h(x_i)\beta =t_i-E_i,\mathrm{\forall }i, \end{gathered}$$where $h(x_i)$ stands for the hidden output for the input $x_i$, $\eta$ is a non-negative real number, and the regularization coefficient $\eta$ balances the residual error $({\displaystyle \sum _{i=1}^N{E}_i^2)}$ and norm of the output weights ( $||\beta |{|}^2$).

SCN added new hidden neuron ${\beta }_L,g_L,$ incrementally leading to $f_L=f_{L-1}+{\beta }_L{g}_L$. After dynamically adjusting the output weights during the training process, the current residual error was added to the $L_2$ regularization term. After adding the Lth new hidden node into an established SCN model with $L-1$ hidden nodes, a new objective function was introduced. (22) $$\begin{array}{ll}f({\beta }_L)& =\frac{1}{2}\left[\begin{array}{c}\beta {\beta }_L\end{array}\right]\left[\begin{array}{c}\beta \\ {\beta }_L\end{array}\right]+\frac{\eta }{2}{\Vert E_L\Vert }^2\\ & =\frac{1}{2}||\beta {\Vert {}^2+\frac{1}{2}||{\beta }_L\Vert }^2+\frac{\eta }{2}||E_{L-1}-{\beta }_L(g_L+\frac{{\Vert g_L\Vert }^2}{\gamma }{E}_{L-1})|{|}^2.\end{array}$$where $\gamma$ is the positive scale factor of feedback residual error. The derivative of function Eq. (22) with respect to ${\beta }_L$ is:

(23) $$\frac{\mathrm{\partial }f({\beta }_L)}{\mathrm{\partial }{\beta }_L}={\beta }_L-\eta ⟨E_{L-1},g_L+\frac{||g_L|{|}^2}{\gamma }{E}_{L-1}⟩+\eta {\beta }_L||g_L+\frac{||g_L|{|}^2}{\gamma }{E}_{L-1}|{|}^2.$$

Letting Eq. (23) be equal to 0, The output weights of the L-th hidden node was obtained by:

(24) $${\beta }_L=\frac{⟨E_{L-1},g_L+\frac{||g_L|{|}^2}{\gamma }{E}_{L-1}⟩}{||g_L+\frac{||g_L|{|}^2}{\gamma }{E}_{L-1}|{|}^2+\frac{1}{\eta }}.$$

Theorem 1. Assume that $span(\mathrm{\Gamma })$ is dense in $L_2$ space. Given $0<r<1$, $0<\eta$, $0<\gamma$, and a nonnegative real number sequence ${\mu }_L$, with ${\mu }_L\ge 1-r$ and ${\lim }_{L\to +\infty }{\mu }_L=0$. $\mathrm{\forall }g\in \mathrm{\Gamma }$, $0<||g||<b_g$ for some $b_g\in {\mathbb{R}}^{+}$. For $L=1,2,...$ and $q=1,2,...,m$, the random basis function $g_L$ is generated by Eq. (25), and the output weights of the Lth hidden neuron are obtained by Eq. (26). Then, the SCN has ${\lim }_{L\to +\infty }\left|\right|f-f_L\left|\right|=0$.

(25) $$\begin{array}{c}\frac{{⟨E_{L-1,q},g_L+\frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}⟩}^2}{{(||g_L+\frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}|{|}^2+\frac{1}{\eta })}^2/(||g_L+\frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}|{|}^2+\frac{2}{\eta })}\ge (1-r-{\mu }_L)||E_{L-1,q}|{|}^2,\\ q=1,2,...,m.\hfill \end{array}$$

(26) $${\beta }_{L,q}=\frac{⟨E_{L-1,q},g_L+\frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}⟩}{||g_L+\frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}|{|}^2+\frac{1}{\eta }},q=1,2,...,m.$$

Proof. First, the monotonically decreasing property of $||E_L||$ will be proved.

(27) $$\begin{array}{rl}& ||E_L|{|}^2-||E_{L-1}|{|}^2=\\ & \sum _{q=1}^m(⟨E_{L-1,q}-{\beta }_L(g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}),E_{L-1,q}-{\beta }_L(g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q})⟩-⟨E_{L-1,q},E_{L-1,q}⟩)}}\\ & =\sum _{q=1}^m(-2⟨E_{L-1,q},{\beta }_L(g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q})⟩+⟨{\beta }_L(g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}),{\beta }_L(g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q})⟩)}}}\\ & =\sum _{q=1}^m(-2{\displaystyle \frac{{⟨E_{L-1,q},{\beta }_L(g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q})⟩}}^2}{||g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}|{|}^2+{\displaystyle \frac{1}{\eta }}}}+{\displaystyle \frac{{⟨E_{L-1,q},g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}⟩}}^2||g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}|{|}^2}}{{(||g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}|{|}^2+{\displaystyle \frac{1}{\eta })}}}^2})}}\\ & =-\sum _{q=1}^m{\displaystyle \frac{(||g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}|{|}^2+{\displaystyle \frac{2}{\eta }){⟨E_{L-1,q},g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}⟩}}^2}}}{{(||g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}|{|}^2+{\displaystyle \frac{1}{\eta })}}}^2}}\\ & =-\sum _{q=1}^m{\displaystyle \frac{{⟨E_{L-1,q},g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}⟩}}^2}{{(||g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}|{|}^2+{\displaystyle \frac{1}{\eta })}}}^2/(||g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}|{|}^2+{\displaystyle \frac{2}{\eta })}}}}\le 0.\end{array}$$

The monotonically decreasing property of $||E_L||$ has been proven. From Eqs. (25)–(27):

(28) $$\begin{array}{rl}& ||E_L|{|}^2-(r+{\mu }_L)||E_{L-1}|{|}^2\\ & =\sum _{q=1}^m(⟨E_{L-1,q}-{\beta }_L(g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}),E_{L-1,q}-{\beta }_L(g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q})⟩-(r+{\mu }_L)⟨E_{L-1,q},E_{L-1,q}⟩)}}\\ & =\sum _{q=1}^m((1-r-{\mu }_L)⟨E_{L-1,q},E_{L-1,q}⟩-2⟨E_{L-1,q},{\beta }_L(g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q})⟩+⟨{\beta }_L(g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}),{\beta }_L(g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q})⟩)}}}\\ & =(1-r-{\mu }_L)||E_{L-1}|{|}^2-\sum _{q=1}^m(-2{\displaystyle \frac{{⟨E_{L-1,q},{\beta }_L(g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q})⟩}}^2}{||g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}|{|}^2+{\displaystyle \frac{1}{\eta }}}}+{\displaystyle \frac{{⟨E_{L-1,q},g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}⟩}}^2||g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}|{|}^2}}{{(||g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}|{|}^2+{\displaystyle \frac{1}{\eta })}}}^2})}}\\ & =(1-r-{\mu }_L)||E_{L-1}|{|}^2-\sum _{q=1}^m{\displaystyle \frac{(||g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}|{|}^2+{\displaystyle \frac{2}{\eta }){⟨E_{L-1,q},g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}⟩}}^2}}}{{(||g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}|{|}^2+{\displaystyle \frac{1}{\eta })}}}^2}}\\ & =(1-r-{\mu }_L)||E_{L-1}|{|}^2-\sum _{q=1}^m{\displaystyle \frac{{⟨E_{L-1,q},g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}⟩}}^2}{{(||g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}|{|}^2+{\displaystyle \frac{1}{\eta })}}}^2/(||g_L+{\displaystyle \frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}|{|}^2+{\displaystyle \frac{2}{\eta })}}}}.\end{array}$$

According to Eq. (25):

(29) $$||E_L|{|}^2-(r+{\mu }_L)||E_{L-1}|{|}^2\le 0.$$

Therefore:

(30) $$||E_L|{|}^2\le r||E_{L-1}|{|}^2+{\mu }_L||E_{L-1}|{|}^2.$$

Theorem 1 has given ${\lim }_{L\to +\infty }{\mu }_L=0$, which means ${\lim }_{L\to +\infty }{\mu }_L\left|\right|E_{L-1}|{|}^2=0$. Based on Eq. (30), ${\lim }_{L\to +\infty }\left|\right|E_L|{|}^2=0$. Therefore, ${\lim }_{L\to +\infty }\left|\right|E_L\left|\right|=0$.

RSCN-INFO algorithm

INFO is a very competitive new optimization algorithm. In this section, INFO is applied to optimize the parameter λ, the contractive factor $r$, the regularization coefficient $\eta$, and the positive scale factor $\gamma$ for RSCN. The widely-used root mean square error (RMSE) is employed as the fitness function.

(32) $$RMSE=\sqrt{\frac{1}{N}\sum _{i=1}^N{[\sum _{j=1}^L{\beta }_j{g}_j(w_j^T{x}_i+b_j)-t_i]}^2}.$$

For convenience’s sake, ${\xi }_{L,q},q=1,2,...,m$ is defined to describe the algorithm. The pseudo-code of RSCN-INFO is summarized in Algorithm 1.

(33) $${\xi }_{L,q}=\frac{{⟨E_{L-1,q},g_L+\frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}⟩}^2}{\zeta }-(1-r-{\mu }_L)||E_{L-1,q}|{|}^2$$where

(34) $$\zeta =\frac{{(||g_L+\frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}|{|}^2+\frac{1}{\eta })}^2}{||g_L+\frac{||g_L|{|}^2}{\gamma }{E}_{L-1,q}|{|}^2+\frac{2}{\eta }}.$$

The calculation complexity

The computational complexity of the INFO algorithm depends on the size of the population $N_P$, the times of iterations $G_{max}$, and the dimensional search domain D. The complexity of INFO is $O(N_P\times G_{max}\times D)$. For SCN, assume that a set of datasets with N inputs $X=\{x_1,x_2,...,x_N\}$, and the maximum number of hidden layer neurons of SCN is $L_{max}$. The main cost of SCN is caused by computing Moore-Penrose pseudo-inverse $H_L^{†}T$. A rough estimate of the computational complexity of $H_L^{†}$ can be expressed as $O(N{L}_{max}^3+N^2{L}_{max}^2+L_{max}^4)$ (Li & Wang, 2017). Note that the cost of $H_L^{†}T$ is calculated by the widely-used singular value decomposition. Hence, the total complexity of RSCN-INFO is $O(N_P\times G_{max}\times D\times (N{L}_{max}^3+N^2{L}_{max}^2+L_{max}^4))$.

Function approximation

Let the real-valued function $f(x)$ be defined as follows (Tyukin & Prokhorov, 2009):

(35) $$y=0.2e^{-{(10x-4)}^2}+0.5e^{-{(80x-40)}^2}+0.3e^{-{(80x-20)}^2},x\in [0,1].$$