Review
Intrusion detection techniques in cloud environment: A survey

https://doi.org/10.1016/j.jnca.2016.10.015Get rights and content

Abstract

Security is of paramount importance in this new era of on-demand Cloud Computing. Researchers have provided a survey on several intrusion detection techniques for detecting intrusions in the cloud computing environment. Most of them provide a discussion over traditional misuse and anomaly detection techniques. Virtual Machine Introspection (VMI) techniques are very helpful in detecting various stealth attacks targeting user-level and kernel-level processes running in virtual machines (VMs) by placing the analyzing component outside the VM generally at hypervisor. Hypervisor Introspection (HVI) techniques ensure the hypervisor security and prevent a compromised hypervisor to launch further attacks on VMs running over it. Introspection techniques introspect the hypervisor by using hardware-assisted virtualization-enabled technologies. The main focus of our paper is to provide an exhaustive literature survey of various Intrusion Detection techniques proposed for cloud environment with an analysis of their attack detection capability. We propose a threat model and attack taxonomy in cloud environment to elucidate the vulnerabilities in cloud. Our taxonomy of IDS techniques represent the state of the art classification and provides a detailed study of techniques with their distinctive features. We have provided a deep insight into Virtual Machine Introspection (VMI) and Hypervisor Introspection (HVI) based techniques in the survey. Specific research challenges are identified to give future direction to researchers. We hope that our work will enable researchers to launch and dive deep into intrusion detection approaches in a cloud environment.

Introduction

Hacking incidents are increasing day by day as technology evolves. Companies are changing the way they operate. Security issues in such a complex technological environment are posing significant challenges. Attacks are reported by cloud providers and users regularly. For instance, the French research outfit VUPEN Security (Mimiso, 2012) discovered the Virtual Machine Escape attack. The exploit targets a vulnerability that affects the way Intel processors implement error handling in the AMD SYSRET instruction. In Jan 2013, European Network and Information Security Agency (ENISA) reported (Dekker et al., 2013) that Dropbox was attacked by Distributed Denial of Service (DDoS) attacks and suffered a substantial loss of service for more than 15 hours affecting all users across the globe. DDoS botnets attacks also happened against the Amazon Cloud. Security researchers (Dee, 2014) have found the exploit on the Amazon Cloud platform through the ElasticSearch distributed search engine tool. Hackers attacked Amazon EC2 virtual machines using cve-2014–3120 exploit in ElasticSearch ver. 1.1 x. Researchers have also found that many enterprises are still using these vulnerable versions. According to Symantec (2015), 494 vulnerabilities and two zero-day vulnerabilities were disclosed during the month of January in 2015. W32. Ramnit! html was the most common malware that had been blocked. Verizon (2015) reported 55% incidents were insider abuses/attacks this year. In case of web applications attacks, stolen credentials accounted for 50.7%, backdoors were 40.5%, SQL Injection were 19%, brute force were 6.4%, and cross site scripting (XSS) attacks were some 6.3%. Cisco (2015) has reported that malware developers are using web browser add-ons as a medium for distributing malware and unwanted applications. They stated that 56% of all OpenSSL versions were due to older versions leading to OpenSSL attacks.

In the last few years, research has been carried out to tackle such security problems. The importance of well-organized architecture and security roles have become even greater with the popularity of Cloud Computing. Cloud Security Alliance (CSA) (Smith, 2012) provides best practice in cloud security such as security as a service model for cloud environment. Various researchers working in the field of cloud security have proposed intrusion detection systems (IDS) as a defensive approach. An IDS is a security tool that captures and monitors the network traffic and/or system logs, and scans the system/network for suspicious activities. It further alerts the system or cloud administrator about the attacks. There are different types of IDS based on the location where the IDS is deployed, e.g. Host-based IDS, Network-based IDS and Hypervisor-based IDS. Host-based intrusion detection system (HIDS) monitors individual hosts (physical/virtual). It sends alerts to the user if it detects suspicious activities such as modification or deletion of system files, unwanted sequence of system calls or unwanted configuration changes at virtual machine (VM) or in other cloud regions. Network-based intrusion detection system (NIDS) is usually placed at network points such as gateway or routers to check for anomalies in network traffic. Hypervisor based IDS is deployed at the hypervisor (Virtual Machine Monitor (VMM)) or privileged VM and is capable of capturing the state information of all VMs running on top of the hypervisor. It can maintain and enforce different security policies for each VM, based on its requirements.

Different intrusion detection techniques used in a cloud environment include misuse detection, anomaly detection, virtual machine introspection (VMI), hypervisor introspection (HVI) and a combination of hybrid techniques. Misuse detection techniques maintain rules for known attack signatures. These rules can be derived either by using the knowledge based systems which contain database of known attacks signatures or by using machine learning algorithms that are used in the determination of behavioral profiles of the users based on known suspicious activities (Barbara and Jajodia, 2002). Anomaly detection systems detect anomalies based on the expected behavior of the system. Any deviation from the expected behavior is signaled as anomalous (Garcia-Teodoro et al., 2009).

Another well known technique is that of VMI. The basic principle behind the VMI technique is that it performs introspection of programs running in a VM to determine any malicious program change or execution of some abnormal or malicious code (Hebbal et al., 2015). There are different approaches to VM introspection such as guest-OS hook based, VM state access based, kernel debugging based, interrupt based and hypercall authentication based; they bridge the semantic gap in interpreting the low-level information available at a VM to high level semantic state of a VM. There are several open source based VMI tools such as Ether (Dinaburg et al., 2008) and DRAKVUF (Lengyel et al., 2014) that perform introspection of VMs from outside.

VMI techniques leverage VMM technology that was lacking in traditional IDS approaches. Hypervisor or VMM is a software that creates and runs VMs. It can access any of the VM spawned by it. It emulates the physical hardware and prevents direct access to physical hardware. In Xen hypervisor based cloud environment such as Openstack (2015), VMI based IDS can be configured to run at the privileged domain of VMM; In Xen, Dom0 is a privileged domain that starts first and manages the unprivileged domains (untrusted guest domains), DomU. However, if a VMM is compromised, the VMI tool will be under the control of the attackers. HVI based security approach mainly depends on the hardware assistance to perform introspection of hypervisor/host OS kernel states and detect various attacks such as hardware attacks, rootkit attacks and side channel attacks.

In this paper, we address the limitations of existing surveys (Modi et al., 2013; Patel et al., 2010) and provide a detailed study of the detection mechanisms in the IDS. We also give a detailed discussion of the threat model, attacks and deployment approaches of IDS in a cloud environment. Our major contributions of this paper can be summarized as follows:

  • We propose a threat model and attack taxonomy and provide a detailed discussion of the various attacks related to the cloud environment.

  • We provide a classification of IDS deployment approaches for a cloud environment, with an analysis of their advantages and disadvantages.

  • We propose a classification of intrusion detection mechanisms in the cloud environment. The detailed analysis of techniques is intended to provide the readers a coherent view of the security solutions that currently exist.

  • A classification of VMI techniques is proposed and discussed in detail, especially for detecting attacks from VM to hypervisor (VM-VMM) and stealthy rootkit attacks at the VM.

  • A classification of HVI techniques is proposed and discussed especially for detecting attacks from hypervisor to VM (VMM-VM) and hardware attacks at the VMM.

  • Finally, we identify specific research challenges and outline some possible future directions in cloud based IDS.

The rest of the paper is organized as follows: Section 2 gives the background on cloud based IDS and and highlights the difference between traditional IDS and cloud based IDS. Section 3 describes the threat model for a cloud environment and outlines an attack taxonomy. Section 4 describes the different deployment approaches of IDS in a cloud. Section 5 describes the proposed IDS taxonomy and presents a detailed study of intrusion detection techniques for a cloud environment. Section 6 provides observations and summarizes some potential research challenges. Section 7 provides a comparison of our paper with other related surveys. Finally, section 8 concludes the paper.

Section snippets

Evolution of cloud based IDS

Traditional IDS systems have been applied to a cloud environment by several researchers. For example, Roschke et al. (2009) proposed a Snort based IDS architecture named as VM-Integrated IDS to detect anomalies. Modi et al. (2012a) used Snort and machine learning classifiers to detect anomalies in the network traffic between VMs. Alarifi and Wolthusen (2012) used traditional ‘Bag of System Calls’ based approach to detect anomalous sequences present in the user programs during execution. Gupta

Threat model and attacks in a cloud environment

A cloud environment typically consists of three types of servers: Cloud Controller Server (CCS), Cloud Compute Server (CCoS) and Cloud Networking Server (CNS) (Openstack, 2015). All the management related tasks of a cloud are handled at CCS whereas CCoS hosts various virtual machines (VMs). CNS facilitates configuration of network, IP allocation and traffic routing to cloud servers. It also enables VMs to connect to the Internet. There are typically three networks in cloud: tenant network,

Deployment of IDS in cloud

We have classified the deployment approaches of IDS in a cloud environment into five categories: (A) In-Guest agent based approach (B) In-VMM agent based approach (C) Network-Monitor based approach (D) Collaborative agent based approach (E) Distributed approach as listed and compared in Table 1.

In-Guest agent based approach configures and executes the security tools inside a VM and hence it has good visibility of the monitored VM. It can perform deep scanning of packets leaving or entering the

Taxonomy of IDS techniques

In this section, we present the state of the art IDS techniques in a cloud environment as shown in Fig. 3. We have classified techniques of intrusion detection into five types: (i) Misuse Detection (ii) Anomaly Detection (iii) Virtual Machine Introspection (VMI) (iv) Hypervisor Introspection (HVI) and (v) Hybrid techniques. Each technique is further classified based on the detection approach employed (described in detail under each subsection). We begin with the discussion on misuse detection

References (142)

  • Alarifi, S.S., Wolthusen, S.D., 2012. Detecting anomalies in iaas environments through virtual machine host system call...
  • Azab, A.M., Ning, P., Wang, Z., Jiang, X., Zhang, X., Skalsky, N.C., 2010. Hypersentry: enabling stealthy in-context...
  • Barbara, D., Jajodia, S., 2002. Applications of data mining in computer security. Vol.6. Springer Science & Business...
  • D. Baysa et al.

    Structural entropy and metamorphic malware

    J. Comput. Virol. Hacking Tech.

    (2013)
  • Benninger, C., Neville, S.W., Yazır, Y.O., Matthews, C., Coady, Y., 2012. Maitland: Lighter-weight vm introspection to...
  • Ben-Yehuda, M., Day, M.D., Dubitzky, Z., Factor, M., Har’El, N., Gordon, A., Liguori, A., Wasserman, O., Yassour,...
  • M. Bernaschi et al.

    Remus: a security-enhanced operating system

    ACM Trans. Inf. Syst. Secur

    (2002)
  • Bharadwaja, S., Sun, W., Niamat, M., Shen, F., 2011. Collabra: a xen hypervisor based collaborative intrusion detection...
  • Bolte, M., Sievers, M., Birkenheuer, G., Niehörster, O., Brinkmann, A., 2010. Non-intrusive virtualization management...
  • M. Carbone et al.

    Taming virtualization

    IEEE Secur. Priv.

    (2008)
  • Chen, L., Wei, Z., Cui, Z., Chen, M., Pan, H., Bao, Y., 2014. Cmd: classification-based memory deduplication through...
  • Chen, P.M., Noble, B.D., 2001. When virtual is better than real [operating s ystem relocation to virtual machines]. In:...
  • Cisco, 2015. Cisco Annual Security Report....
  • C. Cortes et al.

    Support vector machine

    Mach. Learn.

    (1995)
  • CVE, 2007. CVE Details: The Ultimate Security Vulnerabiltiy Data Source....
  • Dee, J., July 2014. Amazon cloud infested with DDoS...
  • Dekker, M., Liveri, D., Lakka, M., Dec 2013. Cloud security incident reporting, framework for reporting about major...
  • R. Denz et al.

    A survey on securing the virtual cloud

    J. Cloud Comput.: Adv. Syst. Appl.

    (2013)
  • Dinaburg, A., Royal, P., Sharif, M., Lee, W., 2008. Ether: malware analysis via hardware virtualization extensions. In:...
  • Ding, B., He, Y., Wu, Y., Lin, Y., 2013. Hyperverify: a vm-assisted architecture for monitoring hypervisor non-control...
  • P. Ferrie

    Attacks on more virtual machine emulators

    Symantec Technol. Exch.

    (2007)
  • Flores, J.J., Antolino, A., Garcia, J.M., Solorio, F.C., 2012. Hybrid network anomaly detection–learning hmms through...
  • Forrest, S., Hofmeyr, S., Somayaji, A., Longstaff, T., et al., 1996. A sense of self for unix processes. In: IEEE...
  • Garfinkel, S., 1999. Architects of the Information Society: 35 Years of the Laboratory for Computer Science at MIT. Mit...
  • B. Grobauer et al.

    Understanding cloud computing vulnerabilities

    IEEE Secur. Priv.

    (2011)
  • I. Gul et al.

    Distributed cloud intrusion detection model

    Int. J. Adv. Sci. Technol.

    (2011)
  • S. Gupta et al.

    Taxonomy of cloud security

    Int. J. Comput. Sci., Eng. Appl.

    (2013)
  • S. Gupta et al.

    System cum program-wide lightweight malicious program execution detection scheme for cloud

    Inf. Secur. J.: A Glob. Perspect.

    (2014)
  • S. Gupta et al.

    An immediate system call sequence based approach for detecting malicious program executions in cloud environment

    Wirel. Pers. Commun.

    (2015)
  • J.A. Halderman et al.

    Lest we remembercold-boot attacks on encryption keys

    Commun. ACM

    (2009)
  • Heaven, V., 2012. Computer virus collection....
  • Hebbal, Y., Laniepce, S., Menaud, J.-M., 2015. Virtual machine introspection: Techniques and applications.In:...
  • Hecht-Nielsen, R., 1989. Theory of the backpropagation neural network. In: International Joint Conference on Neural...
  • P. Helman et al.

    A statistically based system for prioritizing information exploration under uncertainty

    IEEE Trans. Syst. Man Cybern. Part A: Syst. Hum.

    (1997)
  • Hex-Rays, 2011. IDA Support: Download Center.URL...
  • S.A. Hofmeyr et al.

    Intrusion detection using sequences of system calls

    J. Comput. Secur.

    (1998)
  • Hwang, T., Shin, Y., Son, K., Park, H., 2013. Design of a hypervisor-based rootkit detection method for virtualized...
  • Intel, 2004. Intelligent Platform Management Interface (IPMI). URL...
  • Kallenberg, C., Butterworth, J., Kovah, X., Cornwell, C., 2013. Defeating signed bios enforcement. EkoParty, Buenos...
  • Kang, D.-K., Fuller, D., Honavar, V., 2005. Learning classifiers for misuse andanomaly detection using a bag of system...
  • Cited by (184)

    View all citing articles on Scopus
    View full text