Disrupting the Disruptors: How to Threat Hunt Like a Pro

Editor’s Note: The following blog post is a summary of an RFUN 2017 customer presentation featuring Ismael Valenzuela from McAfee.

As the saying goes, the best defense is a good offense. When it comes to cybersecurity, that means shifting from merely responding to intrusions and attacks to actively searching out threats and destroying them. Having the capacity and know-how to make this stance shift is a key element of a mature information security operations center (SOC), says Ismael Valenzuela, who recently gave a presentation on threat hunting at RFUN 2017.

Valenzuela has worked in cybersecurity for decades and has been a member of the Foundstone team at McAfee for six years, performing incident response in the United States, Europe, and the Middle East. He is also a SANS-certified instructor who has taught classes on continuous monitoring, forensics, and security operations for the past seven years.

During his presentation, Valenzuela talked extensively about the difference between incident response and threat hunting, focusing on the qualities that a SOC needs to effectively hunt threats and some of the challenges they face, as well as what he called the three big “knows” that every SOC should focus on: knowing your enemy, knowing your network, and knowing your tools. He concluded his talk with a look at how automation, artificial intelligence, and machine learning are impacting the field, arguing that they are ultimately just new tools that can supplement, but never replace, a team of experienced humans.

Threat Hunting Takes Maturity

First, how are threat hunting and incident response different? Any incident response typically follows the same broad pattern: get an alert of some unusual activity on your network, look for the source of the alert, and then take preventative action, if necessary. Threat hunting also means looking for problems and taking preventative action, but the process starts without an alert, which Valenzuela notes comprises the main difference between the two. It requires a shift in thinking — incident response, by definition, is always reactive, whereas threat hunting is proactive. The hunter becomes the hunted. For this reason, threat hunting requires a certain degree of maturity in a SOC.

“Maturity” in this context is somewhat of an ineffable concept, going beyond simply having a certain number of employees, a robust set of protocols, or access to the latest and greatest tools and programs. Though a mature SOC will have protocols to follow and extensive documentation defining how to look for threats, Valenzuela emphasizes that threat hunting is actually hurt by a lack of flexibility and an excessive adherence to protocol. He compares threat hunting to looking for treasure without a map, saying, “It’s as much an art as it is a science. The ability to think outside the box, to anticipate an attacker’s move, and have strong intuition based on experience are crucial.” There are no shortcuts to developing that intuition — just accumulated experience.

Having an experienced SOC doesn’t just mean that your team is made up of people who have dealt with a lot of attacks, either — truly valuable experience comes from reflection on those attacks and a refinement of systems and processes.

Growing Pains

What does a growing degree of maturity look like in the cybersecurity capacities of an organization? Valenzuela provides a “maturity model,” showing in more detail when organizations can think about moving from a reactive to a more proactive stance. An emphasis is placed here on the basics: threat hunting is not a replacement for a simpler level of cybersecurity, but an addition to fundamentals, like keeping applications and operating systems up to date, continuously monitoring your network, and so on.

Valenzuela notes that some organizations make the mistake of jumping too quickly into hunting for threats. In his experience, their line of thinking goes something like this: “Okay, we know we’ve already been compromised. So why bother focus on detecting bad stuff and reacting to it — let’s go hunting for bad stuff first!” In reality, this is dropping their security stance by ignoring the fundamentals.

So, step one is to cover the basics — even the most immature organization should take some preventative steps. The next step is to look for indicators of compromise (IOCs), or unusual activity of any kind. According to a threat-hunting survey conducted by McAfee, the top IOCs typically relied upon by threat hunters are, in order:

• Network traffic denied by firewalls or IPs
• Unusual DNS requests
• Signs of DDoS activity and geographic irregularities
• Suspicious registry or system file changes
• Large numbers of requests for the same file
• Login red flags, like brute-force attacks
• Unusual activity within privileged accounts
• HTML response size
• Domain
• URL
• File name
• File hash
• Mismatched port-application traffic
• Unexpected patching of systems
• Unusual north-south or east-west network traffic

And certainly, these are all great places to start, Valenzuela says. But one big problem is that many of these IOCs, like IP addresses, domains, and hashes, are provided externally, meaning that it can be nearly painless for the attacker to change them. This is not a reliable way of identifying threats.

This is just one of the many challenges to threat hunting, according to Valenzuela. Other challenges include inexperience and bias; a lack of formal training programs, which often leads to a desire to have a “playbook” of actions to take, reducing those essential qualities of flexibility and intuition born of experience to a series of rote processes; the sheer volume of data to contend with; spending too much time on data collection and not enough on analysis; and the adoption of new tools as a bandage for a lack of or inefficient processes. Dealing with these challenges takes an awareness of the three “knows” mentioned before: knowing the enemy, knowing your network, and knowing the tools at your disposal.

The 3 Knows

Knowing the enemy means recognizing that behind every attack is a person, or a group of people, with a distinct motivation that underlies their methodology. As noted above, attackers can change their IOCs very quickly, meaning that just because someone has seen an indicator of an attack before, doesn’t mean the next attack will resemble it. Mature threat hunting takes an understanding of the tactics, techniques, and procedures (TTPs) of an attacker. Again, there is a great deal of intuition required here in picking the right hypothesis and asking the right questions to provide context and think critically about the data you gather. For example, is this attack financially motivated? Is it coming from a competitor?

Knowing the network means threat hunters must be familiar with normal patterns within their network before they will be able to effectively spot abnormal ones. Create a baseline. Attackers often demonstrate greater familiarity with an organization’s network than the organization itself, often because inexperienced organizations will not look at their network from the outside. “You need to know what normal looks like before starting hunting,” Valenzuela says, “but even if you start hunting in a blind way, you’re going to find lots of false positives, things that behave like malware, lots of end-user applications, and lots of craziness going on there that will behave like malware.”

Knowing your tools ultimately depends upon knowing the tasks and challenges that your organization is uniquely faced with. Having tools to automate and streamline certain processes, like gathering and analyzing data and identifying malware, is essential to both incident response and threat hunting, but it is not enough to simply have the “best” or “most advanced” commercially available applications, because as Valenzuela emphasizes, “There’s no such thing as a perfect tool.”

AI and the Future of Cyber Threat Hunting

One such category of tools is the growing field of artificial intelligence and machine learning. To those who believe such tools represent the future of cybersecurity, Valenzuela offers a cautionary reminder: “Don’t forget that it’s ultimately a human who’s trying to attack you. The same technology is available to both sides, and just as quickly as new models become more effective at threat detection, malicious actors grow more capable at confusing those models. A whole subfield of artificial intelligence, called adversarial machine learning or adversarial AI, is devoted to this.”

That doesn’t mean to never rely on artificial intelligence, of course. Valenzuela instead advocates for human-machine teaming. Intelligent machines are good at quickly automating repetitive tasks, solving complex math problems, and recognizing and classifying patterns; conversely, humans have passion, creativity, and intuition, and are able to analyze context to judge motivation, determine root causes, and think strategically. Each is far more effective in complement than alone.

Conclusion

In summary, Valenzuela believes that mature cyber threat hunting is a lot like playing a game of Clue. Was it Colonel Mustard, with the candlestick, in the kitchen? Identifying an attacker and understanding their TTPs requires formulating the right hypothesis and asking the right questions. Investigation playbooks should be about learning how to think critically, not simply following a scripted set of procedures and actions.

Start with the basics. Build a strong security foundation that enables effective incident response. Know your enemy, your network, and your tools. Then, identify hunting cases that can benefit from both human judgment and intuition, and machine speed and pattern recognition. Allow experts to ask the right questions, and revise your techniques. Improve iteratively. Collaborate with others in the same field, sharing your experience and techniques, and learning from theirs.

Oh, and one last word of advice from Valenzuela: Choose the right day of the week to start hunting on, because if you look for something, chances are you’re going to find it. “Hunting on a Friday, it’s a bad idea,” he says, “especially if you don’t want to spend the rest of your weekend in incident response mode.” And before the holidays is even worse — “Especially if you value your family.”