Going Deep and Dark: Mining Threat Intelligence From the Hidden Web

Everybody has heard of the dark web. But for most, it’s a nebulous concept at best.

Even the name makes it sound more like a plot from a spy novel than a real thing. It conjures up images of mob bosses arguing over territory at a back room card table.

The truth, of course, is altogether different. It’s much less fantastical, and much more commercial.

Recorded Future and Infosecurity Magazine recently co-hosted a webinar to demystify the dark web, and explore the ways legitimate security officers can navigate it in search of valuable threat intelligence. The webinar featured three expert speakers: Andrei Barysevich, director of advanced collection at Recorded Future, Chris Pace, technology advocate at Recorded Future, and Konrad Smelkovs, senior manager at KPMG cyber defense service.

The Internet’s Back Room Card Table

So, what exactly is the dark web?

To kick off the webinar, Andrei Barysevich tackled the elephant in the room. Very few people understand how the dark web really operates, so here’s the analogy he gave:

"Imagine a nondescript entrance to a bar in a dark alley. A place you won't find in the phonebook. If you know the secret knock and password, they'll let you in. Otherwise, good luck next time. The same concept applies to dark web communities, which you won’t find through Google or any other search engine. Only current members can tell you how to find a particular community."

He went on to explain that many dark web markets can only be accessed using the Tor browser, and in order for your membership to be accepted, you may be required to pay a fee. In many cases this fee is nominal, but for more exclusive dark web communities it could stretch to thousands of dollars, and you may also need several current members to vouch for you.

Much has been made of the scale of the dark web, with many news outlets reporting that Google only indexes somewhere between five and 20 percent of websites. But these statistics are misleading, as the overwhelming majority of non-indexed pages are perfectly legitimate, but hidden from search engines behind paywalls or within databases.

Instead, it’s best to think in terms of purely illegal activity. In this context the dark web consists of a few hundred (maybe more) illicit communities, where members buy and sell everything from custom-made tools for cyber crime to stolen data, drugs, weapons, and more.

Another common misconception about the dark web is that only members of powerful crime syndicates use it to conduct business. In reality, the average dark web “vendor” is a small-time hacker, often called a “script kiddie,” who only engages in illicit activities on a part-time basis. In fact, many members of dark web communities barely qualify as criminals at all, looking merely to purchase compromised accounts from popular services (e.g., Spotify), or discounted software.

But while the contents of dark web markets may be unavailable elsewhere, their appearance would be strikingly familiar for the majority of people. Many of the most significant dark web markets utilize a simple, bulletin board style interface that looks very much like Craigslist.

By contrast, the image above is taken from the popular HANSA market, which plays host to a wide variety of low-level cyber criminals, drug dealers, and fraudsters. This particular market is free to join, and doesn’t require any existing members to vouch for new applicants. As a result, membership is fairly high by dark web standards, likely falling in the tens or even hundreds of thousands, making the more automated eBay-style layout necessary.

Of course, as we’ve already mentioned, not all dark web markets are so promiscuous. The most secretive communities are highly selective about who they’ll grant membership, and often have no more than a couple hundred members.

What’s in It for Me?

Now that we’ve covered the basics of dark web communities, it’s time to answer a very important question. Given that it takes time and resources to gain access to these communities, is it really worth it? And what exactly is so great about threat intelligence from the dark web?

One of the most valuable applications of dark web research is the identification of compromised assets or information. Healthcare organizations, for instance, might identify stolen patient records. Financial institutions can analyze stolen payment information to identify common points of purchase and minimize fraudulent transactions.

In a recent example, a multinational software company managed to prevent the sale of highly sensitive source code. After identifying the illegal sale through dark web research, analysts were able to engage with local law enforcement and the organization in question to identify the threat actor (who turned out to be an employee of the company) and prevent the sale from happening.

In another case, a government resource was penetrated by an Eastern European cyber criminal, who attempted to sell the vulnerability he had identified to unfriendly state actors. Once again, the sale was identified through dark web research, law enforcement agencies were alerted, and the sale was prevented.

And the thing to note is, none of this would have been possible without dark web research. In most cases, the intelligence that can be sourced from the dark web isn’t available anywhere else. Sure, some of the tools and tactics shared through lower-level communities may find their way onto the open internet, but truly high-level threat actors are rarely so careless.

If your organization is breached, and sensitive information is stolen, there’s a strong chance it will show up on the dark web before you know an attack has taken place.

Sometimes, by analyzing threat actor communications in dark web communities, you can even identify attacks before they happen, giving you the opportunity to close vulnerabilities or strengthen technical controls preemptively.

That’s the benefit of dark web research. It may require more time and resources to obtain than intelligence from the open web, but the payoff is often dramatically higher.

Of course, if you want these benefits, and you’re looking to source the intelligence yourself, direct access to the dark web is essential.

But valuable dark web threat intelligence doesn’t come without a few risks.

Not All Fun and Games

One of the most important questions to ask when collecting and analyzing threat intelligence is how it will benefit the organization. And when KPMG’s Konrad Smelkovs spoke during the webinar, this was the very first thing he covered:

"The press, blogs, and social media are full of interesting tidbits lifted from the dark web. That’s great that some enterprising hacker is selling a little bit of leaked data, but how does it affect me? What’s the value?"

Without a clear plan for how threat intelligence will be used, and a toolset that can automatically prioritize threats and discount false positives, threat intelligence can tie up resources without contributing to your security profile in a meaningful way.

However, when sourcing threat intelligence from the dark web, this issue runs far deeper. It takes a lot of time and energy to infiltrate the more secretive dark web markets, but there’s more at stake than just wasted resources. Once membership has been granted, an inexperienced researcher can easily wind up putting his or her organization at more risk by acting suspiciously.

“In a recent case, a poorly trained researcher openly inquired about a penetration test solution for a newly adopted infrastructure, which was tested at a limited number of corporate locations.” Andrei Barysevich explained. “Within a couple of hours there was a huge surge in malicious activity targeting the same network.”

It may be tempting to ask straightforward questions, but criminal communities aren’t fond of intruders, and it’s easy for them to spot a researcher who hasn’t taken the time to learn how they operate and communicate. Andrei’s advice to would-be dark web researchers was simple:

"I would be very cautious about revealing information or asking questions in underground communities. Listen, read, but do not engage criminal actors."

A lot of valuable intelligence can be gleaned without ever directly engaging hostile actors. Unless you have the time and inclination to get into dark web intelligence for the long haul, it’s probably best to leave the real investigative work to the professionals.

Striking a Balance

At this point, you might be thinking dark web intelligence is more trouble than it’s worth. After all, as Konrad Smelkovs pointed out during the webinar, every organization needs a level of situational awareness, but not every organization needs a team of expert security researchers.

So, how can you strike this balance? According to Chris Pace, the answer is to start out small and gradually build up until you have what you need.

“Begin with some form of dark web monitoring. Find a way to monitor for things like incident detection, mentions of your brand, and mentions of technology that you use,” he explained.

“There are technologies, and ours is one, that will enable you to put a sensor out there to find out what exists in those underground forums. At the moment you may not have the capacity, the investment, or the will to invest heavily in very specific and targeted threat intelligence gathering. Consider monitoring and alerting as a starting point.”

And the advantage of starting this way isn't purely financial. By using an advanced threat intelligence solution such as Recorded Future, you can identify potential threats or breaches without ever accessing or interacting with underground communities. Naturally, this dramatically reduces the level of risk involved with your threat intelligence collection process.

Of course, if you do want to move to the next level, gathering unique and actionable threat intelligence will require experienced analysts who know the communities, know the actors, and have the skillset to identify what’s real and what isn’t.

Ultimately, it’s about determining what your organization needs from threat intelligence. If you’re a financial institution or a defense contractor, then yes, you likely do need a fully functional threat intelligence capability. That means building an expert team of security analysts and researchers, and arming them with the tools they need to quickly gather, sort, prioritize, and analyze threat intelligence from both open and dark web sources.

On the other hand, if your organization is in a less targeted industry, or you don't have the budget for this level of investment, it's possible to start much smaller. Threat intelligence solutions such as Recorded Future dramatically reduce the analyst time required to triage potential threats, and can completely remove the need for direct access to underground communities in many cases.

Either way, the value of threat intelligence from the dark web is undeniable. If you’re serious about developing a strong threat intelligence capability, you should strongly consider expanding your sources beyond the open internet.

Learn More

For more information, view the webinar in its entirety. If you'd like to learn more about how Recorded Future can help your organization plan for future threats, contact us for a demo today.