HiFIND: A high-speed flow-level intrusion detection approach with DoS resiliency

Abstract

Global-scale attacks like worms and botnets are increasing in frequency, severity and sophistication, making it critical to detect outbursts at routers/gateways instead of end hosts. In this paper, leveraging data streaming techniques such as the reversible sketch, we design HiFIND, a High-speed Flow-level Intrusion Detection system. In contrast to existing intrusion detection systems, HiFIND: (i) is scalable to flow-level detection on high-speed networks; (ii) is DoS resilient; (iii) can distinguish SYN flooding and various port scans (mostly for worm propagation) for effective mitigation; (iv) enables aggregate detection over multiple routers/gateways; and (v) separates anomalies to limit false positives in detection. Both theoretical analysis and evaluation with several router traces show that HiFIND achieves these properties. To the best of our knowledge, HiFIND is the first online DoS resilient flow-level intrusion detection system for high-speed networks (e.g. OC192), even for the worst-case traffic of 40-byte-packet streams with each packet forming a flow.

Introduction

Traffic anomalies and attacks are commonplace in today’s networks, and identifying them rapidly and accurately is critical for operators of large networks. With the rapid growth of network bandwidth and fast emergence of new attacks, viruses and worms, existing network intrusion detection systems (IDS) [1], [2], [3], [4] are insufficient due to lack of the following features.

Most existing IDSes reside on a single host, only examining application-level [1] and system-level [2] logs, as such detailed information can identify attacks on individual machines. However, today’s fast propagating viruses/worms (e.g., SQL Slammer worm) can infect most of the vulnerable machines in the Internet within 10 min [5] or even less than 30 s with some highly virulent techniques [6], [7]. Thus, it is crucial to identify such outbreaks in their early phases, which can only be achieved by detection at high-speed routers instead of at end hosts [8]. Further, the edge network and even the backbone are suggested as good vantage points for worm containment [9]. Worm detection on those high-speed networks is a crucial prerequisite for containment. However, the existing schemes are not scalable to the link speeds and number of flows for high-speed networks.

Given a high-speed router, e.g., an OC-192 link (10 Gbps), each 40-byte TCP packet only has 32 ns to proceed [10]. For data recording in high-speed network IDSes, it is difficult for software-based approaches to keep up with the link speed. Thus, the recording part of high-speed IDSes has to be hardware implementable, and the following three performance features are strongly desirable: (1) a small amount of memory usage (to be implemented in SRAM); (2) a small number of memory accesses per-packet [11], [12]; and (3) scalability to a large key space size. The last constraint is especially important for the coming decade: IPv6 with its 128 bit IP address is being adopted, especially in Asia. Thus, the system should scale to a key space of $2^{128}$ or $2^{256}$. Meanwhile, other features are strong constraints which exclude many possible designs.

To bypass detection by an IDS, attackers can execute denial-of-service (DoS) attacks, or fool the IDS to raise many false positives so that real attack alerts are ignored. Thus, the attack resiliency of an IDS itself is very important. However, existing IDSes often keep per-flow states for detection, which is vulnerable for DoS attacks.

Accurate attack mitigation usually requires IDSes to pinpoint the attack type and attack flows. To this end, we need to detect intrusions at the flow-level instead of based on the overall traffic [13], [14]. Furthermore, we want to differentiate different types of attacks because mitigation schemes vary with the types of attacks. For example, for SYN flooding, attackers often spoof their IPs, but we can start the SYN defender [15], and/or change the IP address of the given domain name for the victim machines to alleviate the DoS effects. On the other hand, for port scans, we will use an ingress filter to block the traffic from the attackers’ IPs.

Most existing network IDSes assume detection is on a single-router or gateway. However, as multi-homing, load balancing based routing, and policy routing become prevalent, asymmetric routing appears, and ingress and egress traffic go through different routers. Even for a connection between a certain source and destination, the packets may traverse different paths due to per-packet load balancing of routers which use a round-robin method to determine which path each packet takes to the destination [16], [17]. Thus, observation from a single vantage point is often incomplete and affects detection accuracy. Meanwhile, it is very hard to copy all traffic from one-router to other routers/IDSes due to the huge volume of data.

To detect unknown attacks and polymorphic worms, statistics-based instead of signature-based intrusion detections have been adopted widely. However, many network element faults, e.g., congestion/failures, router misconfigurations, and polluted DNS entries, can lead to traffic anomalies which will be detected as attacks.

To meet the requirements above, we propose a new paradigm called DoS resilient High-speed Flow-level INtrusion Detection, HiFIND [18] leveraging recent work on data streaming computation and in particular, sketches [19], [20]. Sketches are a kind of compact data streaming data structure which record traffic for given keys and are capable of reporting heavy traffic keys. Sketches are also linear, meaning we can take linear combinations of multiple sketches. (for details please refer to Section 3). Essentially, we want to detect as many attacks as possible. As the first step towards this ambitious goal, we aim to detect various port scans (port scans are an important way to detect large-scale worm propagation and botnet probing) and TCP SYN flooding. This will serve as an essential building block for the high-speed IDSes. Our goal is to identify and distinguish the port scans and SYN flooding in real-time on high-speed networks, and to obtain the attacks’ key characteristics for mitigation. Note that while each of these attacks seems relatively easy to detect separately, or in an offline setting, it is in fact very hard to detect a mixture of attacks online at flow-level for high-speed networks. To the best of our knowledge, HiFIND is the first DoS resilient high-speed flow-level intrusion detection approach for port scans and TCP SYN flooding for high-speed networks (like 10 s of Gigabit links, e.g., OC192), even for the worst-case traffic of 40-byte-packet streams with each packet forming a flow.

To this end, we leverage and improve sketches, an efficient tool for data streaming computation, to record flow-level traffic as the basis for statistical intrusion detection. Although proposed in [19], [20], sketches have not been applied to building IDSes for the following reasons:

• Sketches can only record certain aggregated metrics for some given keys. For each flow, there are numerous possible keys: source/destination IP addresses, source/destination ports, source/destination prefixes, protocols, etc., and any of these combinations. Since, it is not feasible to try all possible combinations of the metrics, given the threat model, what would be the minimal set of metrics for monitoring?

• Existing sketches are all one-dimensional, i.e., they can only record the values for a specific metric. However, various forms of attacks are often hard to identify with such single dimensional information. For example, both horizontal scans and un-spoofed SYN flooding exhibit a large number of unsuccessful connections aggregated with the (source IP, destination port) pair. However, it is difficult to differentiate these different kinds of attacks unless the distribution of the attacks on the destination IPs are also considered.

In this paper, we address these two challenges and build the HiFIND prototype system to meet the five requirements mentioned before. We make the following contributions:

• We analyze the attributes in TCP/IP headers and select an optimal small set of metrics for flow-level sketch-based traffic monitoring and intrusion detection. Based on that, we build the HiFIND prototype which is DoS resilient and can provide high-speed flow-level intrusion detection online as demonstrated by both analytical and experimental results.

• To analyze the attack root cause for mitigation, we design efficient two-dimensional (2D) sketches to distinguish different types of attacks. Both analytical and empirical results show the effectiveness of the 2D sketches.

• We aggregate the compact sketches from multiple vantage points (e.g., routers) to detect intrusion in the face of asymmetric routing and multi-path routing caused by per-packet load balancing of routers. To the best of our knowledge, HiFIND is the first system that can work in such environments.

• For false positive reduction, we propose several heuristics to separate SYN floodings from network/server congestions and misconfigurations (e.g., polluted DNS entries).

As shown in Fig. 1, HiFIND detection systems can be implemented as black boxes attached to high-speed routers (edge network routers or backbone routers) of ISPs without affecting the normal operation of the routers.

Detection on edge networks is particularly critical, powerful and efficient (without deploying IDSes on all the edge hosts), according to a recent research agenda for large-scale malicious code by DARPA [8].

For evaluation, we first test the router traffic traces collected at Lawrence Berkeley National Labs. We then apply HiFIND for on-site detection at the Northwestern University (NU) edge routers: we record each minute of traffic with reversible sketches on the fly. At the end of each minute, we use the recorded sketches for online detection. In particular, the one day experiment data consist of 239M network flows of 1.8TB total traffic. We validate the SYN flooding and port scans detected, and find the HiFIND system is highly accurate. The 2D sketches successfully separate the SYN flooding from port scans, and the heuristics effectively reduce false positives of SYN flooding. The evaluation demonstrates that HiFIND significantly outperforms existing approaches like Threshold Random Walk (TRW) [21], TRW with approximate caches (TRW-AC) [22], and Change-Point Monitoring (CPM) [13], [14]. Compared with statistical detection on complete flow-level data logs, we have almost the same detection accuracy, but use much less memory.

The HiFIND system runs in real-time, and requires a small number of memory accesses per-packet. With a Pentium Xeon 3.2 GHz machine and normal DRAM memory, we record 239M flows with one reversible sketch in 20.6 s, i.e., 11.6M insertions/second. For the worst-case scenario with all 40-byte packets, this translates to around 3.7 Gbps. Our prototype single FPGA board for reversible sketches can achieve a throughput of over 16 Gbps for all 40-byte-packet streams. For the NU on-site experiments over a total of 1430 min, HiFIND on average uses only 0.34 s to detect intrusions for each minute of traffic, and the standard deviation is 0.64 s.

The organization of this paper is as follows. First, we survey related work in Section 2. In Section 3, we introduce the sketches and reversible sketches as the basis for high-speed network monitoring. We then introduce the HiFIND architecture, discuss the threat model and flow-level detection design in Section 4. The two-dimensional sketches are presented in Section 5, and evaluation methodology and results are in Section 6. Finally, we show the potential limitations of HiFIND in Section 7 and conclude in Section 8.