Threat Intelligence and SIEM (Part 1) — Reactive Security

Foreword

This blog post is the first part in a series about reactive versus proactive security with security information and event management (SIEM) and threat intelligence (TI). In this post we will present an overview of reactive SIEM, what it does, how it works, and its limitations. Please note this post will cover a significant amount of background information and details, but will not cover how to deploy and configure SIEM, what vendor to choose, or how to implement use cases.

SIEM Overview

Security information and event management (SIEM) is a solution that provides a bird’s eye view of an IT infrastructure. It fulfills two main objectives: (1) detecting in (near) real-time security incidents, and (2) efficiently managing logs. These objectives were respectively called security event management (SEM) and security information management (SIM), but nowadays these functions have been merged into a single capability known as SIEM. From a high-level point of view, a SIEM collects information (e.g., logs, events, flows) from various devices on a network, correlates and analyzes the data to detect incidents and abnormal patterns of activity, and, finally, stores the information for later use (reporting, behavior profiling, etc.). When successfully deployed and configured, a SIEM helps organizations:

-Discover internal/external threats. -Monitor (privileged) user activity and access to resources. -Provide compliance reporting. -Support incident response.

Typical Architecture

As previously mentioned, the SIEM will gather logs and events from a heterogeneous collection of data sources which can be grouped into four categories:

-Network devices (routers, switches, etc.) -Security devices (IDP/IPS, firewalls, etc.) -Servers (Web, mail, etc.) -Applications

For each device, a collector will be used to gather and normalize its information before forwarding the logs to the central engine — the heart of the SIEM where correlations and analyses take place. Finally, the logs will be stored in a database for a certain amount of time depending on the organization’s retention policy. The typical architecture described above can be depicted as follows:

Figure 1: Typical Environment: Devices send their logs directly to the SIEM where collectors, central engine, and database are located in a single appliance. In a distributed environment, the collectors and database can be physically located at a different place.

Depending on the SIEM vendor, the term “collector” can be changed to “agent” or “connector.” Some vendors also offer “smart connectors” which automatically detect the type of device they are connected to, simply by recognizing the logs they receive. To remain as generic as possible, we will keep the term “collector.”

The Real Value of Logs, Events, and Flows

A SIEM system can make use of diverse information types. The primary type is log data, usually meant for several purposes such as debugging, system administration, and security audits. The most commonly used standard for logging is Syslog.¹ Depending on the device, the standard can change. For instance in the case of Web servers, most of them will use the Common Log Format² or other proprietary formats.

Another type of information that can be retrieved by a SIEM is events. Events are usually produced by security devices or controls such as IDPS or Identity and Access Management (IAM) systems. It can be, for example, input validation failures (e.g., invalid parameter names/value, protocol violations) or application errors and systems events (e.g., runtime errors, connectivity problems, performance issues, etc.). Events can be correlated together with other information to provide higher intelligence into log management.

Compared to an ordinary logger, a SIEM system can use various conditions to check whether certain events are matching a rule, and depending on the latter, an alarm can be triggered. For example, let us consider a port scan: when a firewall receives a single packet on port 20, it can send a log “connection attempt to ftp service,” nothing dramatic. But when it receives packets with the destination port set from 20 to 100 in less than two seconds, all these events sent to the SIEM can match the rule “port scan,” which will trigger an alert to the security team: some malicious activities are undergoing!

We can list four categories of conditions:

-Event Based: An IDS reports a signature X targeted at host Y and vulnerability scanner knows that Y is vulnerable. It triggers an alert. -Rule Based: If X + Y + Z then do A, or If X repeats more than three times in interval Y then do Z.

-Anomaly Based: If the traffic on port X exceeds the standard deviation of historic traffic patterns then trigger an alert (e.g., new worm, bot communicating with C&C). -Risk Based: If attack type is destructive (e.g., buffer overflow versus SYN scan), and target is a critical asset (production server versus workstation), then trigger an alert.

Some event standards have been proposed to improve interoperability and simplify integration of devices. For example, ArcSight came up with the Common Event Format³ and IBM proposed Log Event Extended Format and Splunk Common Information Model. Finally the last type of data a SIEM can use is traffic flow,⁴ providing a better overview of the network activity. The two main standards are NetFlow (RFC 3954) and IPFIX (RFC 5101 and 5102). The problem with these two formats is that they do not provide information above the Layer 4 of the OSI model. To remedy this issue, some vendors propose application-aware flows which help detect threats through the analysis of the packet content, using Deep Packet Inspection (e.g., IBM’s QFlow). The central engine will correlate all of the gathered information by using diverse algorithms and data-mining techniques. These techniques will identify suspicious patterns and behaviors, and provide great help for intrusion detection and auditing.

Limitations

Deploying a SIEM solution can be quite complex and expensive: the price of appliances, the time for configuration and tuning, and the expertise required for daily use/maintenance can discourage customers. After purchase and deployment, the recurrent question is “now what do we do?” and enterprises tend to answer by using a “monitor-and-respond strategy.⁵” By using the SIEM in a signature-based defense approach, the security team (or the security operation center [SOC] team) will monitor activities and regularly update the security devices with signatures of known threats.

Figure 2: Reacting upon incident flow chart.

Upon detection, the team will investigate the alert, escalate it to the Incident Response (IR) team for remediation if they cannot resolve the issue directly themselves (e.g., stop the attack if still ongoing, re-image compromised systems, etc.), and finally report to the board. This overall process can take quite some time, and “time is money,” especially when it comes to security. Moreover, since the signature-based approach only protects from known threats, the anomaly-based approach, which focuses on detecting abnormal behaviors, should in theory help detect unknown threats, but practically it significantly increases the false-positive rate. The direct consequence is the time required by the security team to investigate them, increasing the chance of missing true-positives. In a survey of the Ponemon Institute, they discovered that on average a company will have up to 17,000 alerts per weeks, but only 700 will be investigated!

A recent Ponemon Institute study⁶ found that companies spend $1.27 million annually on average by wasting time “responding to inaccurate and erroneous intelligence.” In addition, the rise of targeted attacks, so-called “advanced persistent threats” (APTs), makes it clear that this traditional reactive security posture is no longer sufficient. With shrinking security budgets,⁷

companies need to find new efficient ways to defend themselves. The question we have to answer is: How do we get ahead of threats? It is time to become proactive.

In the next post we’ll define threat intelligence and see how to leverage its possibilities with SIEM.