1 Introduction

Users of the Internet and other networks share and transmit millions of color images every day; these images are used in different applications such as telemedicine, distance learning, business, and military. Securing digital images is extremely important to prevent image content loss during transmission and hide image information from attackers.

Different techniques such as watermarking (Hosny et al. 2018; Hosny et al. 2019; Hosny et al. 2021a, b), image steganography (Zhou et al. 2019; Kadhim et al. 2020), and image encryption (Naskar et al. 2020) are frequently used for securing digital images. Image encryption technique is based on two main stages: encryption and decryption. In the encryption stage, the input image is converted into an unreadable image using a secret key. In the decryption stage, the contents are retrieved by using the same key. One main advantage of using encryption over other methods is that the image is retrieved without losing information. The RGB color space is commonly used in color image encryption algorithms. Each pixel in the color image consists of three values, one in each channel. Most color image encryption algorithms encrypt each channel independently.

Scientists proposed different techniques of color image encryption. These techniques depend on a chaotic system (Wang et al. 2019a, b; Parvaz and Zarebnia 2018; Wang et al. 2019a, b; Xian et al. 2020; Liu et al. 2019a, b), DNA computing (Jithin and Sankar 2020; Nematzadeh et al. 2020), and compressive sensing (Yao et al. 2019). Also, deep learning approaches are applied in image encryption, (Abro et al. 2020; Ali et al. 2020; Chen et al. 2019), to increase the robustness of 2D/3D image encryption. This technique utilizes a fast and effective CNN denoiser based on the principle of deep learning. In Ding et al. (2020), the authors utilize the Cycle-GAN network as the primary learning network to encrypt and decrypt the medical image. Ding et al. (2021) proposed a new deep learning-based stream cipher generator, DeepKeyGen, designed to generate the private key to encrypt medical images.

Generally, color image encryption techniques involved confusion and diffusion of pixels. Confusion is the process of changing pixels' arrangement without changing the pixel value. This step individually does not give satisfactory results in encryption. For improved security, the confusion step is usually combined with the diffusion step, in which the values of pixels are changed based on specific mathematical operations.

Chaotic systems are divided into two main classes (low and high dimensions). These systems have many valuable characteristics, such as randomness, ergodicity, complex behavior, and sensitivity to control parameters and their initial conditions. Also, the keyspace generated by most of the chaotic systems is very large. Based on their ability to improve encryption algorithms' efficiency, various chaotic systems are utilized in color image encryption (Li et al. 2018; Li et al. 2019a, b; Yang and Liao 2018).

Pak and Huang (2017) show that low-dimensional (LD) chaotic systems contain a combination of one-dimensional chaotic maps. Teng et al. (2018) converted the color image to one bit-level image by combining the three primary channels, Red, Green, and Blue. Then, the combined image was scrambled using a skew tent map. Irani et al. (2019) designed the chaotic coupled-Sine map. This new map is used for scrambling color images. Despite the simple structure of low-dimensional systems, the keyspace is small, achieving a lower security level. Essaid et al. (2019) proposed a new method for encrypting both color and grey images using a secure variant of Hill cipher and an improved 1D chaotic maps (logistic map, sine map, and Chebyshev map). Wu et al. (2015) presented a color image encryption algorithm by combining DNA sequences with multiple improved 1D chaotic maps. Kamal et al. (2021) proposed a new algorithm for encrypting grey and color medical images. This algorithm is based on image blocks and a chaotic logistic map.

The high-dimension (HD) chaotic systems are characterized by having complex structures and multiple parameters. These properties enable HD systems to overcome the weaknesses of LD systems. Liu et al. (2019a, b) encrypted color images using dynamic DNA and the 4-D memristive hyper-chaos. Zhang and Han (2021) proposed a new color image encryption scheme based on dynamic DNA coding, six-dimensional (6D) hyperchaotic system, and image hashing. Kaur et al. (2020) presented minimax differential evolution-based 7D hyperchaotic map that is used for encrypting color images. Sahari and Boukemara (2018) combined two chaotic maps, piecewise and logistic, to design a 3D chaotic map. This 3D map is used in information security applications. Wu et al. (2016) proposed a new color image encryption method by combining the discrete wavelet transform with a 6D hyperchaotic system. Zhou et al. (2018) proposed quantum-based encryption of color images using quantum cross-exchange with a 5D hyperchaotic system.

The fractional-order polynomials and functions show better performance than the corresponding integer-order ones in color image analysis Hosny et al. (2020a), pattern recognition Hosny et al. (2020b), Image-based Diagnosis of COVID-19 Abd Elaziz et al. (2020), Plant disease recognition Kaur et al. (2019), and improved recognition of bacterial species Chen et al. (2018). Generally, the chaotic systems of fractional orders are more complex and more accurate than the integer-orders chaotic systems. Accordingly, recent fractional-orders chaotic systems-based image encryption methods were proposed (Yang et al. 2019; Yang et al. 2020a, b; Yang et al. 2020a, b). These encryption algorithms are limited to encrypting gray-scale images.

Color images contain more information than grey images. So, encrypting color images with high efficiency in considerable time is a great challenge. Some recent color image encryption techniques have shortcomings, such as small keyspace; the key does not depend on the original image, making it weak against differential attacks. Other algorithms cannot resist different kinds of attacks. There is not much previous work that used fractional-order chaotic systems in encrypting color images. These shortages motivate the authors to propose a novel utilization of chaotic systems of fractional orders in the encryption of color images. The main contributions of this paper are:

  1. 1.

    A 4D fractional-order hyperchaotic Chen system is applied to generate the secret key used for scrambling the plain image, where the system's initial conditions are based on the plain image.

  2. 2.

    The diffusion step is based on the Fibonacci Q-matrix.

  3. 3.

    Integration between the 4D fractional-order hyperchaotic Chen system and Fibonacci Q-matrix assures a high level of security and can resist different kinds of attacks.

A new three-stage encryption algorithm for color images was proposed. In this algorithm, the 4D hyperchaotic Chen system of fractional orders is combined with the Fibonacci Q-matrix. Step#1: the input image decomposed into the primary color channels, R, G, & B. Step#2: the confusion and diffusion operations are performed independently for each channel, where the 4D hyperchaotic Chen system fractional-orders used to generate random numbers to permit pixel positions. Step#3: the permitted image is divided into small blocks where each of them is diffused by applying the Fibonacci Q-matrix. Various experiments were performed to demonstrate the efficiency and its ability to resist attacks.

The following subsections are: Sect. 2 includes the preliminaries of the 4D hyperchaotic Chen system of fractional-orders and the Fibonacci Q-matrix. The proposed method for color images is described in Sect. 3. The experiments, results, and discussion are presented in Sect. 4. The conclusion is presented in Sect. 5.

2 Preliminaries

2.1 4D hyperchaotic Chen system of Fractional-order

Li et al. (2005) defined the hyperchaotic Chen system of integer-order, while Hegazi and Matouk (2011) used the principles of fractional calculus to derive the hyperchaotic Chen system of fractional-order.

$$\left\{\begin{array}{ll}{D}^{\alpha }x=a\left(y-x\right)+u\\ {D}^{\alpha }y=\gamma x-xz+cy\\ {D}^{\alpha }z=y-bz\\ {D}^{\alpha }u=yz+du\end{array} \right.$$
(1)

where \(\alpha \) is the fractional-order, \(a,b,c,d,\gamma \) are constants and \(x,y,z,u\) represent the state variable of the system. A chaotic system is hyperchaotic when the number of positive Lyapunov exponents is \(\ge 1\). For \(a=35\), \(b=3\), \(c=12\), \(d=0.3\), \(\gamma =7\), and \(\alpha =0.97,\) the system possesses two positive Lyapunov exponents, which means that it is hyperchaotic of fractional-order.

2.2 Fibonacci Q-matrix

The Fibonacci sequence is defined by:

$${F}_{n}={F}_{n-1}+{F}_{n-2}, n>2$$
(2)

where \({F}_{n}\) is Fibonacci number and \({F}_{1}={F}_{2}=1\). The Fibonacci \(Q\) matrix is a square matrix with size \(2\times 2\) that is given by:

$$Q=\left[\begin{array}{cc}1& 1\\ 1& 0\end{array}\right] $$
(3)

The nth power of the \(Q\) matrix is defined by:

$${Q}^{n}=\left[\begin{array}{cc}{F}_{n+1}& {F}_{n}\\ {F}_{n}& {F}_{n-1}\end{array}\right] $$
(4)

The inverse matrix \({Q}^{-n}\) has the following form:

$${Q}^{-n}=\left[\begin{array}{cc}{F}_{n-1}& {-F}_{n}\\ -{F}_{n}& {F}_{n+1}\end{array}\right] $$
(5)

All symbols used in this section are defined in Table 1.

Table 1 Symbols descriptions
Full size table

3 The proposed encryption algorithm

The input color image of size, \(M\times N\times 3\), decomposed into three channels of the uni-size, \(M\times N\), and then we encrypt each channel independently. Our encryption algorithm depends on three main steps. First: the fractional-order hyperchaotic sequence is generated. Second: the input color image is scrambled, and then diffusion of scrambled image is performed.

The scrambling and diffusion processes were executed twice to get the encrypted image. Figure 1 shows a visualized flowchart for one round. In this figure, the input color image “Lena” is decomposed into three grey images. Then, the fractional sequences are computed based on these images. After that, each grey image is scrambled by the generated sequence and diffused using the Fibonacci \(\mathrm{Q}\) matrix. Finally, the three encrypted images are combined to obtain the encrypted color image. Also, the decryption process is illustrated in this section to retrieve the original image.

Fig. 1
figure 1

Flow chart of one round of color image encryption

Full size image

Figure 2 shows one round of the decryption process where the encrypted image is converted into three grey images. First, the diffusion steps using the Fibonacci \(\mathrm{Q}\) matrix are applied. Then the same sequence (\(S\)) generated in encryption is used in the scrambling step.

Fig. 2
figure 2

Flow chart of one round of color image decryption

Full size image

3.1 Generating a fractional hyperchaotic sequence

Step#1: the initial condition of the Chen hyperchaotic system of fractional-order, as defined in Eq. (1), is image-dependent. The image is converted into vector \(P,\) and then the initial condition is calculated by:

$${\mathrm{x}}_{1}=\frac{\sum_{i=1}^{MN}P(i)+(\mathrm{M}\times \mathrm{N })}{{2}^{23}+(\mathrm{M}\times \mathrm{N}) } $$
(6)
$${\mathrm{x}}_{\mathrm{i}}=\mathrm{mod}\left({\mathrm{x}}_{\mathrm{i}-1}\times {10}^{6},1\right)\mathrm{ i}=\mathrm{2,3},4 $$
(7)

where \({\mathrm{x}}_{1},{\mathrm{x}}_{2},{\mathrm{x}}_{3},{\mathrm{x}}_{4}\) refer to the initial conditions of the fractional hyperchaotic system, \(\mathrm{MN}\) is the length of the image vector \(P\), and the mod is the modulo operation.

Step#2: get the sequence L by iterating the system in (1) \({N}_{0}+MN\)/4 and then discard \({N}_{0}\).

Step#3: sort the sequence L in ascending order and return sorted pixels position in vector \(S\) with size MN.

3.2 Image scrambling

Image scrambling is the process of changing the position of the pixels without changing their value. In our algorithm, the image vector \(P\) scrambled by the sequence \(S\) that is defined by:

$${R}_{i}=\mathrm{P}\left({S}_{i}\right), i=1:MN $$
(8)

where \(R\) is the scrambled vector obtained from changing the pixel's position in \(P\) using the values in the vector \(S\).

3.3 Diffusion

In diffusion, the Fibonacci Q-matrix is used to change image pixels value as follows:

Step#1: reshape scrambled vector \(R\) into the matrix \({R}^{^{\prime}}\) with size \(M\times N\).

Step#2: divide the matrix \({R}^{^{\prime}}\) into sub-blocks of sizes \(2\times 2\).

Step#3: obtain the encrypted image \(E\) by multiplying each sub-block in the scrambled image \({R}^{^{\prime}}\) with the Fibonacci \(\mathrm{Q}\) matrix (\({\mathrm{Q}}^{10}\)) defined in Eq. (4) as follows:

$$\left[\begin{array}{cc}{E}_{i,j}& {E}_{i,j+1}\\ {E}_{i+1,j}& {E}_{i+1,j+1}\end{array}\right]=\left[\begin{array}{cc}{\mathrm{R{^{\prime}}}}_{i,j}& {\mathrm{R{^{\prime}}}}_{i,j+1}\\ {\mathrm{R{^{\prime}}}}_{i+1,j}& {\mathrm{R{^{\prime}}}}_{i+1,j+1}\end{array}\right]\left[\begin{array}{cc}89& 55\\ 55& 34\end{array}\right] mod \,\,256 $$
(9)

where \(i=1: M\), \(j=1:N\) with a unified step 2.

figure a

3.4 Decryption

The inverse process of encryption is decryption. This process aims to retrieve the input image from the encrypted one. In the proposed algorithm, the diffusion Eq. (9) changed:

$$\left[\begin{array}{cc}{\mathrm{R}}_{i,j}& {\mathrm{R}}_{i,j+1}\\ {\mathrm{R}}_{i+1,j}& {\mathrm{R}}_{i+1,j+1}\end{array}\right] =\left[\begin{array}{cc}{E}_{i,j}& {E}_{i,j+1}\\ {E}_{i+1,j}& {E}_{i+1,j+1}\end{array}\right]\left[\begin{array}{cc}34& -55\\ -55& 89\end{array}\right] mod\,\, 256 $$
(10)

where \(i=1: M\), \(j=1:N\) with a unified step 2.

Also, the sequence \(S\) that generated in Sect. 3.1 used to retrieve the original image by replacing Eq. (8) in the scrambling process with the following equation:

$$P\left({S}_{i}\right)={R}_{i} , i=1:MN $$
(11)

4 Experiments and results

Various experiments were conducted to show the efficiency of the proposed method in color image encryption. These experiments are based on computing information entropy and the correlation between adjacent pixels to show the high randomness of the image encrypted using our proposed algorithm.

Also, the efficiency of our algorithm in resisting differential attacks, brute force attacks, noise, and data attacks is presented. In these experiments, the authors used standard color images that are available in SIPI datasets. The experiments were performed using MATLAB (R2015a) with a Laptop computer equipped with Core i5-2430 M 2.4GH CPU and 4 GB RAM. Table 2 shows all test images used for evaluating our algorithm. In Table 2, there is an eight-color test image with two sizes (\(256\times 256\) and \(512\times 512\)). Also, the names of all the test images are stated in Table 2.

Table 2 Standard color images
Full size table

4.1 Information entropy

Image randomness is usually measured using the information entropy, where a successful image encryption method can generate an encrypted image with high randomness. The entropy is computed for each primary color channel, R, G, and B in color images. If the entropy for each channel is near 8, this means the high randomness of the color image. The mathematical formulation for entropy is:

$$H\left(s\right)=-\sum_{i=1}^{k}P({s}_{i}){\mathrm{log}}_{2}P({s}_{i}) $$
(12)

where \(k\) refers to the total number of pixels in the image, \(P({s}_{i})\) refers to the probability of \({s}_{i}\). The entropy is computed for eight standard color images in the three channels, where the three primary channels of these images are encrypted using the proposed method. Obtained values are shown in Table 3. The values of entropy for all color images in all channels are almost identical to the optimum value.

Table 3 Entropy for different color images in three channels
Full size table

In another experiment, the color image of Lena was encrypted by using our method and the recent algorithms (Chai et al. 2019; Li et al. 2019a, b; Rehman et al. 2018; Wu et al. 2018; Zhang et.al 2020; Hosny et al. 2021a, b; Xuejing and Zihui 2020). The values of the entropy are calculated and shown in Table 4. Generally, the average entropy (encrypted image) using our technique outperforms the methods (Rehman et al. 2018; Wu et al. 2018; Zhang et.al 2020) and is very similar to the results of the methods (Chai et al. 2019; Li et al. 2019a, b; Hosny et al. 2021a, b; Xuejing and Zihui 2020). The results indicate that our algorithm can produce an encrypted image with high randomness as all values are near 8.

Table 4 Comparison of entropy of Lena
Full size table

4.2 Correlation analysis

Typically original image has a high correlation between adjacent pixels. A good encryption algorithm should remove this correlation between neighboring pixels. In successfully encrypted images, the correlation between adjacent pixels in the encrypted image should 0. For any two adjacent pixels \(x\) and \(y\), their correlation is:

$${r}_{x,y}=\frac{E(\left(x-E\left(x\right)\right)\left(y-E\left(y\right)\right))}{\sqrt{D\left(x\right)D(y)}} $$
(13)

where

$$E\left(x\right)=\frac{1}{N}\sum_{i=1}^{N}{x}_{i} , D\left(x\right)=\frac{1}{N}\sum_{i=1}^{N}{({x}_{i}-E(x))}^{2} $$
(14)

The correlation in diagonal, horizontal, & vertical is computed for each primary channel. Table 5 shows the obtained results. All correlation values of the different color images are near 0, indicating that our algorithm broke the strong correlation.

Table 5 Correlation coefficients for a different color image in three directions
Full size table

Additional experiments were performed to calculate the correlation coefficients of the novel encryption methods (Chai et al. 2019; Li et al. 2019a, b; Rehman et al. 2018; Wu et al. 2018; Zhang et.al 2020; Hosny et al. 2021a, b; Xuejing and Zihui 2020)., where the obtained results are shown in Table 6. As displayed in Fig. 3, a visualized correlation analysis proves the concept of a strong correlation between adjacent pixels as all pixels clustered in the diagonal direction. However, in Fig. 4, the selected adjacent pixels from the encrypted image occupied the whole space, indicating a weak correlation.

Table 6 Comparison of correlation coefficients for lena
Full size table
Fig. 3.
figure 3

Correlation analysis of Lena in three directions for the three channels

Full size image
Fig. 4.
figure 4

Correlation analysis of encrypted image Lena in three directions for the three channels

Full size image

4.3 Differential attack

The efficiency of image encryption algorithms depends on their high sensitivity to minimal image changes. The algorithm is considered more efficient when a minimal input image change produces a distinguishable encrypted image. Attackers try to trace differences between two images, which are encrypted using the exact encryption method from the input images with only a one-pixel difference. This process enables attackers to find a relation between the input and encrypted image to guess the secret key. This kind of attack is known as a differential attack. The higher efficiency of an algorithm makes it more difficult for attackers to guess image information. The resistance to this attack was measured quantitively using NPCR and UACI:

$$\mathrm{NPCR}=\frac{1}{\mathrm{M}\times \mathrm{N}}\sum_{\mathrm{i}=1}^{\mathrm{N}}\sum_{\mathrm{j}=1}^{\mathrm{M}}\mathrm{D}\left(\mathrm{i},\mathrm{j}\right)\times 100\left(\mathrm{\%}\right) $$
(15)

where NPCR is the number of pixels change rate while NPCR is the unified average Change intensity.

$$\mathrm{D}\left(\mathrm{i},\mathrm{j}\right)=\left\{\begin{array}{c}0, if \,{\mathrm{E}}_{1}\left(\mathrm{i},\mathrm{j}\right)={\mathrm{E}}_{2}\left(\mathrm{i},\mathrm{j}\right),\\ 1, if \,{\mathrm{E}}_{1}\left(\mathrm{i},\mathrm{j}\right)\ne {\mathrm{E}}_{2}\left(\mathrm{i},\mathrm{j}\right),\end{array}\right. $$
(16)
$$\mathrm{UACI}=\frac{1}{\mathrm{M}\times \mathrm{N}}\sum_{\mathrm{i}=1}^{\mathrm{N}}\sum_{\mathrm{j}=1}^{\mathrm{M}}\frac{\left|{\mathrm{E}}_{1}\left(\mathrm{i},\mathrm{j}\right)-{\mathrm{E}}_{2}(\mathrm{i},\mathrm{j})\right|}{255} \times 100\left(\mathrm{\%}\right) $$
(17)

where \({\mathrm{E}}_{1}\) and \({\mathrm{E}}_{2}\) refer to encrypted images from the same input image by changing a single pixel.

In the literature of image encryption, 99.6094 \(\mathrm{\%}\), 33.4635 \(\mathrm{\%}\) are the typical values of NPCR and UACI. An experiment was performed where we randomly changed on pixels in the input image then encrypted both the original input image and the modified one using our algorithm. The NPCR and UACI were computed for both encrypted images. As shown in Table 7, the obtained results show that all NPCR and UACI for the three primary channels are almost equal to the typical values. Also, Table 8 compares the results of our algorithm and the methods (Chai et al. 2019; Li et al. 2019a, b; Rehman et al. 2018; Wu et al. 2018; Zhang et.al 2020; Hosny et al. 2021a, b; Xuejing and Zihui 2020). The average values of NPCR and UACI are ideal. Thus, encrypted images produced using our algorithm are secured against differential attacks.

Table 7 NPCR and UACI for different color images
Full size table
Table 8 NPCR and UACI for Lena using different methods
Full size table

4.4 Histogram analysis

The histogram for each grey level of the color image shows its pixel value distribution. The histogram of both original and encrypted images must be entirely different. Also, the histogram of the original image is usually distributed randomly. However, the histogram of an encrypted image should be uniform. Mathematically, the variances of histograms are calculated by:

$$Var\left(Y\right)=\frac{1}{{n}^{2}}\sum_{i=1}^{n}\sum_{j=1}^{n}\frac{1}{2}{({Y}_{i}-{Y}_{j})}^{2} $$
(18)

The variance of the histogram reflects the histogram uniform distribution for the encrypted image, where both are inversely proportional. Low variance means a uniform histogram. Table 9 shows the variances of histograms for the test color images. The variances of the encrypted images are smaller than the variances of the original images. Figure 5 shows the histogram of the original and encrypted primary channels of the Lena color image. The histograms of the encrypted channels are uniform, while the histograms of the original channels are not uniform.

Table 9 Variances of histograms for both Original and encrypted images
Full size table
Fig. 5
figure 5

Histograms of Lena and its encrypted image using the proposed algorithm. Original image in the first row: a R-channel. b G-channel. c B-channel. The encrypted image in the second row: d R-channel. e G-channel. f B-channel

Full size image

4.5 Data cut and noise attacks

Encrypted images may be exposed to data loss or noise when transmitted over the network. An encryption method is successful when efficiently restoring the encrypted image after noise and data cut attacks. This ability is measured by peak signal to noise ratio (PSNR), which is calculated by:

$$PSNR=10\times {\mathrm{log}}_{10}\left(\frac{{255}^{2}}{MSE}\right) \left(db\right) $$
(19)

The \(\mathrm{MSE}\) is defined by:

$$\mathrm{MSE}=\frac{1}{\mathrm{MN}}\sum_{\mathrm{i}=1}^{\mathrm{M}}\sum_{\mathrm{j}=1}^{\mathrm{N}}{\left|{\mathrm{I}}_{O}\left(\mathrm{i},\mathrm{j}\right)-{I}_{D}(\mathrm{i},\mathrm{j})\right|}^{2} $$
(20)

where \({\mathrm{I}}_{O}\) & \({I}_{D}\) refer to original and encrypted images.

The value of PSNR is directly proportional to image quality, where high values reflect a high similarity between the decrypted and the original image. When the value of PSNR is above 35, it is challenging to distinguish between the original image and the decrypted image. To test the ability of our algorithm in resisting these attacks, we do the following experiments on the encrypted image:

  1. (1)

    Add “salt & pepper” noise (SPN) with density 0.002 and 0.005.

  2. (2)

    Cut attack by cropping with size \(64\times 64\) and\(128\times 128\) at the left corner.

Image contents still recognized that prove the robustness of our algorithm against noise and data cut attacks. Figure 6 shows the decryption results after attacking the encrypted image with different sizes of data cut—the decryption of Lena after adding Salt and Pepper noise with different densities presented in Fig. 7. Also, Table 10 shows the PSNR values for two-color images in the three channels. When we add noise with density 0.002 and 0.005, the average PSNR of the three images' three channels is 29db and 26 dB. While in the case of data cut of size 128, the average PSNR for Lena is 12, and for Baboon is 18. When the data cut is size 64, PSNR average values for Lena and Baboon are 18 and 24, respectively. The experimental results indicate that our algorithm has an excellent performance in restoring encrypted images after noise and data cut attacks.

Fig. 6
figure 6

Data cut attack. a Encrypted Lena with a cut of size \(64\times 64\). b The decryption of a. c Encrypted Lena with a cut of size \(128\times 128\). d Decrypted image of c

Full size image
Fig. 7
figure 7

noise attack. a Noisy encrypted image (0.005, Salt and Pepper noise. b Decrypted image of a, c Noisy encrypted image (0.002, Salt and Pepper noise. d Decrypted image of c

Full size image
Table 10 The PSNR uses different attacks
Full size table

4.6 Keyspace

Attackers try all possible keys in the keyspace of an encryption algorithm to know the correct secret key. If the keyspace is large, it is more difficult for attackers to guess it correctly. This kind of attack is known as a brute force attack. The efficiency of the encryption method depends on having a large keyspace to resist this attack. When the keyspace is larger than \({2}^{100}\) the algorithm can achieve a higher level of security. In our encryption algorithm for color images, the secret key constructed by the four initial conditions of the fractional hyperchaotic system, \(a,b,c,d,\gamma ,\alpha \) and \({N}_{0}\). If we consider the computation precision is \({10}^{15}\), the size of the keyspace is \({N}_{0}\times {10}^{150}\). Therefore, the keyspace of this method is sufficient to resist this attack.

4.7 Key sensitivity

The robustness of the encryption algorithm depends on its sensitivity to the secret key. In other words, minimal changes in the generation process of the secret key result in another key. Therefore, the modified key cannot be utilized in decrypting the encrypted image by the original key. The decrypted image should be a noisy image without any information about the original image. In this experiment, the original key was used in encrypting the image of Lena, and the encrypted image using this key is shown in Fig. 8b. When we decrypt the image in Fig. 8b with changing \({\mathrm{x}}_{4}\) in the initial conditions to \({\mathrm{x}}_{4}+0000000001\) the plain image is not restored as in Fig. 8c. The original secret key will only restore the plain image, as seen in Fig. 8d.

Fig. 8.
figure 8

a Original image. b Encrypted image of a with the original key. c Decrypted image of b with the modified key. d decrypted image of b with the original key

Full size image

5 Conclusion

Novel utilization of hyperchaotic systems of fractional-order in encrypting color images proposed. In this three-stage encryption algorithm, the 4D hyperchaotic Chen system of fractional-order is used in changing the pixel position. The diffused color image is divided into a group of \(2\times 2\) blocks, and then the Fibonacci Q-matrix with \(n=10\) is used in changing the pixels values for each block. The new encryption algorithm showed high sensitivity to any minimal modifications to the secret key and the pixel distribution where an entirely different encrypted image was obtained. The obtained results ensure the success of the proposed color image encryption algorithm to resist all frequent attacks.

The future work will focus on improving the running speed of our proposed algorithm. Also, we will study applying the proposed algorithm in video encryption and super-resolution images. The emerging deep learning-based encryption approach for color images is another future work.