Intrusion Detection Systems Overview

Firewalls are the modern-day equivalent to dead bolts and security bars. The purpose of a firewall is to prevent unauthorized access. Just as locks can be manipulated, firewalls can also be compromised. Enterprises long ago learned not to rely on locks alone. Someone or something must be present to protect the company’s assets once someone or something has breached the first line of defense. IDSs are the modern-day equivalent to the burglar alarm. IDSs constantly monitor the network to look for suspicious activity and, once discovered, can be configured to notify security personnel of the suspected intrusion. Unlike burglar alarms, which can only send an alert that a breach has been made, an IDS can also be configured to take action to prevent further access, while sending alarms and recording information about the intruder(s).

While the basic function of all IDS systems is to detect intruders, two different types of intrusion-detection systems exist—host-based IDSs and network-based IDSs—and two different methodologies are used to detect intruders. Each can use one of two methods to detect intruders. Host-based IDSs are typically software installed on host computers and are used to analyze all traffic received by the host computer. Network-based intrusion detection uses probes to analyze and monitor all traffic on the target network.

IDS systems can use one of two possible methods to detect intruders. The first method—profile-based IDS—uses profiles created by the security administrator to define normal traffic and activity. Traffic that doesn’t match a configured profile is called an anomaly. Because profile-based detection will alarm once a set threshold of anomalies is exceeded, profile-based detection is also referred to as anomaly detection. The second method of detection is called signature-based detection. Signature-based detection systems have a preconfigured set of signatures that are compared with network traffic to detect an attack or intrusion. Just as virus scanners use signatures to recognize viruses, IDSs use signatures to recognize common attacks or exploits used by hackers.

While numerous IDS vendors and methods exist, all IDSs can be described and evaluated by examining the type of IDS, host, or network, plus the methods used to trigger an alarm, profile, or signature. IDS systems that use a combination of host and network, or trigger on both signatures and profiles, are called hybrid systems. Hybrid systems attempt to combine the strengths of each type and detection method, while eliminating the weaknesses.

Host- and Network-Based IDSs

IDS systems can be installed on a host, on the network, or on a combination of both. Each monitoring location has its own benefits and drawbacks. Network monitoring requires the installation of probes on the network that’s to be monitored, while host-based monitoring requires that software be installed on all host systems to be monitored. Both types of IDS are configured to send alarms and events to a central location.

Host-based and network-based IDSs send alerts and alarms to a centralized location where security administrators can view them. CIDS sensors can be managed using Cisco Secure Policy Manager (CSPM). Additionally, sensors can be configured using the built in Device Manager application included with the sensor software. Alarms and events can be collected and viewed using Cisco’s Event Viewer. These applications are discussed in more detail in Chapter 25.

Host-Based Detection

By installing a software agent on all hosts, host-based IDSs monitor all system activities, log files, and network traffic received. Host-based systems might also monitor the OS, system calls, audit logs, and error messages on the host system. While network probes can detect an attack, only host-based systems can determine whether the attack was successful. Additionally, host-based systems can record what the attacker performed on the compromised host.

Not all attacks are performed over the network. By gaining physical access to a computer system, an intruder can attack a system or data without generating any network traffic. Host-based systems can detect attacks that are out-of-band or performed from the console, but a savvy intruder with knowledge of the IDSs can quickly disable any detection software once physical access is accomplished. Neither host nor network-based IDS is a replacement for proper physical security.

Note?

Out-of-band is a term used to describe any traffic or access that doesn’t traverse the public or monitored network. Common out-of-band access methods are a private network, a console connection, or an RS232 serial connection between two or more hosts.

Another advantage for host-based monitoring is its capability to detect stealth attacks. Some network-based IDS systems can be fooled by playing with the fragmentation or TTL of the packets. For a host to process traffic, it must be received and reassembled, and, therefore, it’s subject to monitoring by the host-based IDS. Packet fragmentation and Time to Live (TTL) manipulation are discussed in the section “Network-Based IDS.”

Host-Based Drawbacks

Host-based IDSs have four key disadvantages or weaknesses:

Manageability
Micro view of network attacks
Compromised hosts
Operating systems limitations

Depending on host-based IDSs requires the IDS software be installed on all hosts. This can become an administrative nightmare as version control, software maintenance, and software configuration become a time-consuming and complicated task.

Because host-based systems only analyze traffic received by the host, they can’t detect common reconnaissance attacks performed against the host or a range of hosts. Host- based IDSs won’t detect ping sweeps or port scans performed across a range of hosts.

When a host-based IDS detects an attack, it sends a notification to the management station or the sensor responsible for collecting and recording alarms. If the host is compromised, it might be possible for the intruder to disable the IDS software or that host’s network connection. If the IDS software or the network connection are disabled, the host won’t generate an alert.

IDS software must be installed on each system on the network to provide complete coverage of the network. In heterogeneous environments, this can become a problem because the IDS software must support so many different OSs. Before deciding on an IDS system, you should ensure the IDS software will run on each OS present on the network.

Network-Based IDSs

Network-based IDSs uses probes or sensors installed throughout the network. These probes sniff the network looking for traffic that matches a defined profile or signature. Sensors receive and analyze traffic in real time. Once a traffic pattern or signature is recognized, the probe sends an alert to the management platform and can be configured to take action to prevent any additional access.

Note?

Sniffing the network is a term used to describe a network interface that receives all network traffic. Typically, a network interface only processes packets that are specifically addressed to the host or the broadcast address. By placing the interface in Promiscuous mode, the interface can receive, process, and analyze all traffic traversing the network, regardless of the Layer 2 or 3 address. Legitimate tools used by network engineers to view all network traffic are commonly called network sniffers.

Network-based detection has the advantage of seeing intrusions from a network perspective. While host-based detection can’t detect a ping sweep or a port scan across multiple hosts, network-based IDSs can easily detect such reconnaissance attacks. Network-based sensors generate an alert when these reconnaissance attacks are discovered.

Network-based IDSs don’t depend on the resources of a host machine and, therefore, won’t use valuable network and CPU cycles on mission-critical servers. Additionally, because there’s no software to install, there are no issues with OS compatibility. With network-based IDSs, the IDS software runs only on the director and the sensors.

Sensors have an interface used for monitoring the network (monitoring interface) and a second interface used for command and control (command and control interface). Sensors can’t send network traffic on the monitoring interface (MI) and are, thereby, invisible to anyone unfamiliar with the security features of the network. The command and control interface (CCI) can send and receive management traffic, and this is the interface used to communicate with the management or director computer(s).

To ensure proper security, the network-based IDS should have a separate and highly secure management network. The director platforms and sensors will communicate with one another via this secured management network. Each probe will use this network to send alarms to the director platform, and the director platform will use this management network to configure and update each network sensor.

Network-Based Drawbacks

Network-based intrusion detection has four basic weaknesses:

Bandwidth
Packet fragmentation and reassembly
TTL manipulation
Encryption

One of the more difficult weaknesses to compensate for is the bandwidth limitation of network-based intrusion detection. Network probes must receive all network traffic, reassemble that traffic, and analyze the traffic. As network speeds increase, so must the capabilities of the intrusion detection probes. The solution is to ensure the network is designed properly to allow the strategic placement of multiple probes. As the network grows, more probes can be added to ensure proper coverage and security.

One way hackers attempt to conceal their activity from network-based IDS is to fragment their packets. Each protocol has a maximum packet size. If data to be sent across the network is larger than the maximum packet size, the packet must be fragmented. Fragmentation is simply the process of breaking up data into smaller pieces. These pieces must be put back together in the correct order for proper analyzing. Some OSs reassemble packets first to last, while others reassemble packets last to first. The order of reassembly isn’t an issue as long as no overlap occurs. If a fragmentation overlap occurs, the sensor must know the correct reassembly process. Many hackers attempt to prevent detection by sending overlapping fragmented packets. A sensor won’t detect intrusion activity if the sensor is unable to reassemble the packets correctly.

Note?

Maximum packet size, also known as maximum transmission unit (MTU), is dependant on the type of network used. Ethernet typically has a MTU of 1,500 bytes, while token ring, for example, can have an MTU of 8,000 bytes or more.

Manipulating the TTL field on multiple packets is another technique used to fool network-based sensors. All TCP/IP packets have a TTL field, which specifies how long a packet should be considered valid on the network. The initial value of the TTL field can range from 1 to 255. Each time a packet passes through a router, the TTL is decremented by 1. When the TTL value reaches zero, the packet is discarded and is no longer forwarded. Figure 23-2 shows how an attacker might use a TTL attack to try and fool a network-based IDS. As you can see in Figure 23-2, the attacker can send packets with a lowered TTL that will reach the sensor, but not the host. Additional packets are sent that reach both the sensor and the host. TTL manipulation is ineffective if the probe and the host are located on the same local area network (LAN).

Figure 23-2: Manipulating network-based sensors using TTL

Using a TTL attack, an intruder can send hundreds of packets to the target network with a lowered TTL. Packets with the lowered TTL reach the sensor, but won’t be forwarded to the host. When the packets reach the router, the TTL is decremented and the packet is discarded. At the same time, the intruder could then send actual attack traffic with a normal TTL that will reach both the sensor and the host. In essence, the attacker is attempting to confuse the sensor by hiding malicious traffic within a stream of valid or nonsuspicious traffic. Because an attacker must have a detailed view of the network beforehand, this isn’t a simple attack to use successfully. This type of attack can only be used against network-based IDSs.

The third weakness of network-based IDSs is created through the use of encryption. To work properly, an IDS must read the contents of packets that cross the network. If the data held within the packet is encrypted, the sensor has no way of viewing the contents. If your network uses VPNs, IPSec, or any other form of encryption, then it’s important to place sensors outside the encrypted tunnels. Outside the encrypted tunnel, packets are no longer encrypted and, therefore, can be read by the sensor.

Hybrid IDSs

Hybrid IDSs attempt to combine the benefits of each type of IDS, while eliminating the weaknesses. In a hybrid system, both the sensors and the hosts report to a centralized management or director platform.

Presenting information gathered from both network-based sensors and host-based software can be a challenge for hybrid IDS vendors. While more information is usually better, too much information from too many sources can be difficult to manage and understand.

While hybrid systems attempt to combine the benefits of both network and host- based systems, they also can incorporate both signature-based and anomaly-triggering mechanisms. Triggering mechanisms, discussed in the next section, describe how an IDS system detects intruders.

IDS Triggers

The purpose of an IDS system is to alert the appropriate personnel once an intrusion is detected. Burglar alarm systems trigger an alarm based on a motion detector, a broken window, or an opened door. IDS also have two types of triggering mechanisms.

IDSs are merely packet sniffers with the capability to do some basic analyzing. These systems don’t inherently know the difference between normal traffic and malicious traffic. For the IDS to recognize malicious traffic or activity, you must first “teach” the IDS what constitutes an attack. The IDS then compares actual network traffic or computer activity with what has been defined as malicious and, if a match is made, an alarm is triggered.

Not all systems use the same method of triggering an alarm and each type of triggering system has its own strengths and weaknesses. To choose the appropriate IDS system for your environment, you must first understand each type of triggering system, as well as the benefits and drawbacks of each. Modern IDSs use two types of triggering mechanisms:

Anomaly Detection (Profile based)
Misuse Detection (Signature based)

STUDY TIP?
The terms “anomaly detection” and “profile based,” and the terms “misuse detection” and “signature based” are used interchangeably.

Misuse Detection

Misuse detection is commonly referred to as signature-based detection. Misuse detection requires the use of signature files that identify intrusive activity. The signature files used in misuse detection are analogous to signature files commonly used in virus-scanning software to identify viruses on computer systems.

A signature file is a set of rules used to identify common intrusive activity. The research of highly skilled engineers discovered attacks, patterns, and methods to write signature files to identify them. As more attack methods and exploits are discovered, the IDS vendor will provide updates to signature files, just as virus-scanning vendors provide updates to their own software. Once the signature files are updated, the IDS system will begin analyzing all activity searching for a match. If activity or traffic is found that matches the signature, an alarm is triggered. IDS systems typically come with a database of signatures for common attacks and exploits.

Signature-Based Benefits

Signature files are created based on known attack methods and activity, so if a match is made, a high probability exists that an attack is underway. Misuse detection, unlike anomaly detection, will have fewer false positive reports because matches are based on a known intrusive activity, not just unusual traffic. Signature-based detection doesn’t monitor traffic patterns or look for anomalies. Instead, it monitors activity simply looking for a match to any configured signature.

Because misuse detection relies on signatures—not traffic patterns—the IDS system can be configured and can begin protecting the network immediately. The signatures contained in the signature database contain the known intrusive activity and a description of the signature. Each signature in the database can be viewed, enabled, or disabled. Different levels of alarms, as well as different preventative actions, can be configured for individual signatures, giving security administrators granular control of their IDS systems.

Misuse detection is easier to understand and configure than anomaly-based systems. Signature files can be viewed so administrators can understand what actions must be matched for an alarm to be generated. Security administrators can enable signatures, and then perform a test on the network and view the resulting alarm that’s generated. Because misuse detection is easier to understand, implement, and test, administrators have a higher degree of control and more confidence in their IDSs.

Misuse Detection Drawbacks

While there are many benefits to misuse detection triggers, some drawbacks exist to this form of intrusion detection. Misuse detection is simpler to configure and understand, but this simplicity comes at a cost of lost functionality and administrative overhead. Misuse detection has the following drawbacks:

Inability to detect new or unknown attacks
Inability to detect variations of known attacks
Signature database administration
Sensors must maintain state information

Inability to Detect New Attacks

Misuse detection accomplishes its mission by comparing computer network activity to known intrusive activity defined in the signature database. If an attack is instigated that doesn’t match a known intrusive activity, the sensors typically won’t generate and alert. The IDS system using misuse detection must be aware of the activity of an attack before it can identify that attack. New attacks that haven’t previously been used or discovered normally won’t be detected by a misuse detection IDS. Signature files are created to be as flexible as possible and, in some cases, a previously unknown attack will be detected by the IDS. Even though an exact match might not occur, the IDS could detect a previously unknown attack that uses a similar method of attack or intrusion activity. IDS systems must be updated with the latest signatures to be effective. Even if an IDS system has been updated with the latest signature database, it’s possible that new types of attacks will won’t generate an alert.

Inability to Detect Attack Variations

Intruders also have access to the signature files and IDS systems used by security administrators. Because this information is available to everyone, hackers can use this information to test and alter their attack. By altering the attack in some minor way, an intruder might be able to perform an intrusion without being detected (false positive). Signature files are static—they don’t adapt as some anomaly-based systems do. If an attack doesn’t match a signature file, the sensors won’t generate an alarm. Because the signature files are included with the IDS systems, intruders are aware of what will and what won’t generate an alarm. Armed with this knowledge, an intruder can customize their attacks to defeat the IDS.

Signature File Administration

The responsibility of the security administrator is to ensure the database file is current. The security administrator must also configure the probes with the signatures they want the probes to use, as well as the severity level of each matched signature. Keeping the signature database current with constant updates and applying those updates to all sensors can be a difficult and time-consuming task.

State Information

Just like firewalls, sensors must maintain state data. Sensors simply match activity or traffic to preconfigured signatures. In some cases, the amount of data to match a signature could be spread across multiple packets and a variable amount of time. Additionally, hackers might fragment their packets before sending them across the network in an attempt to prevent the packets from being analyzed. Sensors must record this information and recompile it to match it against any signatures. The maximum amount of time a probe must record the state—from the first packet until a match is made—is called the event horizon.

The event horizon can range from minutes for some signatures to days or weeks for others. For example, a security administrator might want to be alerted if anyone performs a port scan against their network, but might not want to be notified if only one or two ports are scanned. Some patient hackers might only scan two ports every four hours for a month. Within a couple of weeks, this hacker could have found all the services available on the network, which means it’s important for the sensor to remember what ports have been scanned and by whom. But how long should the sensor remember this information? The amount of time the sensor is configured to keep state information for a given signature is called the event horizon for that signature. Some signature files, such as those that detect reconnaissance attacks, have an event horizon that spans weeks, while other attacks have an event horizon that spans the time the user is logged into the network. The event horizon is a variable contained in the signature files.

Sensors have a limited amount of storage available. Most sensors keep this state information recorded in memory for fast retrieval, but the storage space is limited. Hackers might attempt to disable your IDS systems by sending them so much information that the sensor(s) run out of resources and can no longer record state data. Once the sensor has reached its memory limits, it no longer analyzes any additional information. This would be an example of a DoS attack against the IDS itself.

Anomaly Detection

Anomaly- or profile-based triggering analyzes computer activity and network traffic looking for anomalies. If an anomaly is found, an alarm is triggered. An anomaly is any deviation or departure from the normal or common order, form, or rule. Because this type of detection is looking for any activity or traffic that isn’t normal, the security administrator must first define what is normal activity or traffic. Security administrators can define normal activity by creating user group profiles.

A user group profile represents a baseline of normal computer activity and network traffic for a given user group. User groups are defined by the security engineer and can be used to represent users or computers with common job functions, or users and computers within the same departments. Typically, user groups should be divided according to the activities and network resources each group uses. A web server farm could have its own profile based on web traffic, while mail servers could have another profile based on SMTP. You wouldn’t expect telnet traffic destined for your web servers or SSH traffic destined for your mail servers. For these reasons, you should have different profiles for each type of service offered on your network.

Various techniques are used for building user profiles and some IDS can be configured to build their own profiles. The typical methods used to build user group profiles are statistical sampling, rule-based, and neural networks. Each profile is used as a definition for normal user and network activity. If a user deviates too far from their defined profile, the IDS system will generate an alert.

Building Profiles Using Statistical Sampling

With statistical sampling, alarms are generated based on deviations from a normal state. Normal state is defined by sampling normal activity and traffic for a given period of time. Normalcy is based on the average or median of all activity or traffic. Deviations are measured by calculating the standard deviations.

Standard deviations are simply the amount of activity or data that matches, or doesn’t match, a sample, and they measure the deviation from a normal state. For example, if 60 percent of all your data falls within one standard deviation and 35 percent of all your traffic falls within two standard deviations, then only 5 percent of your data falls within three or more deviations. Using this method, the IDS system can detect how abnormal specific activity or traffic is.

Building Profiles Using the Rule-Based Approach

Rule-based profile building is accomplished by defining rules to define normal user behavior. You must create rules that define normal user activity, and these are created by sampling computer and network activity for a given amount of time. Once the data set has been collected, rules can be created to define normal activity. The rules are models representing normal computer and network activity. Any traffic that doesn’t match the rules is considered abnormal and generates alarms.

Building Profiles Using Neural Networks

Just as a psychologist can use inkblots to discover how you relate information in your mind, neural networks can use matrix(s) to relate normal activity on your network or computer systems. Neural networks are built or trained by presenting the IDS system with large amounts of data and rules about data relationships. Neural networks attempt to use artificial intelligence to build matrixes based on the given information. Relationships between these data inputs are used to build a matrix modeled after the biological neurons, such as those found in the human brain. Once the neural network is established, it can be used as a model or definition of normal activity. Any activity that doesn’t map correctly to the matrix or neural network is considered abnormal and generates an alarm.

Anomaly Detection Benefits

Using anomaly detection as the triggering mechanism has many benefits. With anomaly-based detection, the intruder never knows what might or might not generate an alarm, because he or she doesn’t have access to the profiles used to detect an attack. User group profiles are much like a dynamic signature database that changes as your network changes. With signature-based detection, the intruder can test on their own IDS system what will generate an alert. Signature files are provided with a purchased IDS system, so a hacker could use their own IDS system to perform testing. Once the hacker understands what will generate an alert, the attacker can then customize his or her attack methodology and tools to defeat the IDS. Because anomaly detection doesn’t use a preconfigured signature database, intruders can’t be sure what activity will generate an alert.

Anomaly detection can quickly detect an internal attack using a compromised user account. If a user account belonging to an administrative assistant is being used to perform system administration, the IDS system using anomaly detection will generate an alarm as long as that account isn’t normally used for system administration.

The biggest advantage to anomaly- or profile-based detection is it isn’t based on a set of preconfigured signatures or known attacks. Profiles can be dynamic and can use artificial intelligence to determine what normal activity is. Because profile-based detection isn’t based on known signatures, it’s better suited to detect previously unknown or unpublished attacks as long as the attack deviates from normal activity (profile). Profile-based detection can be used to detect new attack methods, which signature-based detection won’t detect.

Anomaly-Based Drawbacks

While many benefits exist to using anomaly- or profile-based detection, many drawbacks also exist with this method of intrusion detection. Many of the drawbacks of anomaly detection have to do with the creation of user group profiles, as well as the quality of these profiles. Drawbacks with anomaly detection include the following:

High initial prep time
No protection during initial training time
Constant update of profiles as users’ habits change
Defining normal behavior can be difficult
False positives, false negatives
Hard to understand

Difficulties with User Group Profiles

Anomaly-based detection relies on the use of user group profiles. The IDS is only as good as the profiles being used to define what normal activity is. Profiles are a baseline of normal activity, created by sampling network traffic and activity over a set period of time. While creating the user profiles, it’s vital no intrusive activity occurs on the network and all systems are free of backdoors or Trojan horses. If intrusive activity occurs on the network during the initial training time, the intrusive activity will be included in the profile and, therefore, the activity will seen as normal activity.

The initial training time should consist of enough data to truly represent normal activity and traffic. The training time could range from days to weeks or even months. Defining normal activity can be a daunting task. What normal activity is in one month could or could not be normal the next month. Users aren’t compelled to use the same applications and perform the same functions without deviation. Defining normal activity is even more challenging in environments where users’ jobs or responsibilities change often. As users’ habits change, the profiles describing normal activity for those users must also change. Additionally, while the system is being “trained,” the IDS provides no protection, so it’s vital no intrusive activity occurs during this training period.

Creating user profiles can be difficult for advanced users or diverse groups of users. If a user group contains a vast amount of users that all perform different functions, then it’s difficult to differentiate normal activity from intrusive activity. System administrators, network engineers, and Unix administrators all generate activity that wouldn’t be permissible for other types of users. For this reason, segregating different users according to resources and applications each group uses is important.

Some systems can be configured to update the profile constantly, based on traffic and activity as it’s being measured. Statistical sampling, discussed in the previous section, constantly monitors the network and uses the data collected to update the profile. The benefit is this: the profile is always kept current with user activity changes, however, a hacker can use this feature to manipulate the IDS. A hacker could slowly begin performing intrusive activity over a long period of time. Starting with small amounts of activity, and slowly increasing the amount of traffic and activity, the hacker can train the IDS to ignore the intrusive activity. The IDS system will slowly begin to consider the intrusive activity as normal, which will result in false negatives.

False Reporting

A false negative is a situation when intrusive activity is on the network or systems, yet the intrusive activity goes undetected by the IDS system. If the activity is considered normal, then an alert won’t be generated. Anomaly detection is only as good as the profile used to detect intrusive activity. Signature-based detection systems tend to have more false negatives than anomaly-based systems because they aren’t suited to discovering new methods of attack.

A false positive occurs when the IDS system generates an alarm for activity that isn’t considered intrusive. Car alarms, for example, commonly report false positives. IDS systems should be continually tuned to strike a balance between false negatives and false positives. Too many false positives and the IDS system will soon be ignored, much like car alarms are today. Even worse, too many false negatives could result in a great deal of damage. Anomaly-based systems tend to have more false positives because they’re looking for anything out of the ordinary.

Difficult to Understand

The last major drawback to anomaly-based detection is its complexity. Statistical sampling, rule-based, and neural networks are all profile- building strategies that are hard to explain and understand. Signature-based detection is much simpler to understand: if a given activity matches a signature, then an alarm is sent, along with a notification of which signature was matched. Anomaly detection requires a more in-depth understanding and it’s harder to discover why the system generated an alert. Because of its complexity, many security administrators have a difficult time understanding the system and are uncomfortable with the IDS. This lack of understanding might also cause lack of confidence in their IDS.