Threat Analyst Insights: Defining Success for Regular Reporting

Even if a company does not have a dedicated threat intelligence team, chances are that a CISO or other senior leadership receives some kind of regular reporting related to cyber threats. Sometimes this is the responsibility of internal writers, and sometimes these reports are collations of reports created by third-party vendors. At Recorded Future, for example, we generate weekly, monthly, and quarterly reports for customers in many different verticals for a variety of different business needs.

Otto von Bismarck is famously misquoted as having said, “Laws are like sausages. It’s better not to see them being made.” Unlike laws and sausages, however, it’s a very good idea to see how regular reports are made, since transparency here is key to getting value out of such reports in the first place. In fact, creating a regular report is similar to setting up an effective threat intelligence program, in that both depend on effectively defining a number of items — metrics quantifying the data in the report, the scope and depth of the report, and its intended audience. Thoroughly understanding each of these items can sometimes radically change a report from how it was originally envisioned.

1. Metrics

Metrics are simply a means of measurement. The metrics that are used in a report can differ between organizations, but having metrics and clearly understanding what they are is an inescapable part of good intelligence. A regular report can measure individual items (such as number of disclosed vulnerabilities), qualitative impact (such as the severity of a cyberattack), or the quality of one’s own data (such as more or less confidence in a source), and usually, these reports will measure all of these and more.

It is important for people who write or create regular reports to step back and address what their products are even meant to contribute — how are the final readers using the report? If a creator can’t tie every part of a report to specific intelligence or business goals, then regardless of whether individual reports are valuable, the process will be a failure. For example, a finance company that receives quarterly reports around cyber threats should decide whether they will be used to primarily inform patch prioritization, security training, business development, or technology upgrades — or all of these together, reinforcing the role of financial services cybersecurity.

2. Scope

Defining scope is not just about deciding what details need to be present in a report. Setting out what a report will include also helps the recipient honestly examine how much time it takes to consume a product. There may be many significant events occurring in the past week, month, or year, but if they can’t all be understood and applied, they should not be included in the scope of that report. The simplest expression of this idea is, “Does this report have all the analysis — and only the analysis — that I need?”

Some of our customers receive regular reporting that covers a concentric-circle model of threats — customer-specific, industry-specific, and general. Others have a narrowly defined set of queries or topics for which they want information. These differences represent thinking about scope correctly beforehand on a number of levels, including time frame, topic, and sourcing.

3. Depth

Depth is similar to scope, but while scope can generally be thought of in horizontal terms, depth is best thought of vertically and is more of a concern for the creator than the consumer. Once a scope is defined, the amount of detail for each part of a report must be considered as well. This might not always be as obvious as page count, either — I’ve worked on weekly stories that took half a day’s worth of research to produce two short paragraphs of final copy. When considering the depth of research or detail, a good specific question is, “How many minutes can I spend writing this?”

Defining depth helps with secondary questions as well. A vulnerability patching schedule can benefit from better defining the detail in a vulnerability report — the creator should decide beforehand whether this report would involve, for example, all recently disclosed vulnerabilities in a tech stack, just those vulnerabilities with a high CVSS score, or just those actively exploited in the wild.

4. Audience

When talking about audience, it is easy to think strictly about binaries like “strategic” versus “tactical” or “C-suite” versus “analysts.” While these are useful binaries, there are other ways of thinking about audience. Most importantly, is the reader able to take meaningful action based on indicators or recommendations, or would that need to go to others? If it needs to go to others, is the report going to the right people in the first place? Regular reports will probably not succeed if they are sent to individuals without the time, skill, or authority to do anything with them.

Conclusion

The four items above are by no means the only ones that could or should be defined for a regular report, but they are the most basic. With all of these factors well defined, a report is much more likely to bring value to final readers. Additionally, definitions alleviate pressure on analysts, who might otherwise be blamed for not including an item or covering it at a certain depth. When an analyst does not need to constantly wonder about issues of metrics, scope, depth, or audience, they are given the freedom to produce truly meaningful reports.