Gamification of Incident Response in the Cyber Range

This is Recorded Future, inside threat intelligence for cybersecurity.

Dave Bittner:

Hello everyone, and welcome to episode 148 of the Recorded Future podcast. I’m Dave Bittner from the CyberWire.

When it comes to incident response — just like in sports — you’ve got to practice like you play. In warfare, they say, “No plan survives contact with the enemy,” and heavyweight boxing champion Mike Tyson had his own version: “Everyone has a plan until they get punched in the face.”

The point is, until you are actually in the heat of a high-pressure situation, it’s highly unlikely that you’ll be able to predict how you and the members of your team will react.

Our guest this week is Christopher Crummey. He’s executive director of the X-Force Command Centers at IBM Security. Christopher and his team create highly realistic simulations of cybersecurity incidents to help organizations evaluate how they’ll respond when the heat is on and the pressure is high. Stay with us.

Christopher Crummey:

This is my 28th year at IBM. I was a school teacher right out of college and then have been working for IBM ever since. And was one of these things where I was very much customer facing and found this particular job at the cyber range that combined my history, my background, my passion, and was a really great fit. It’s interesting to see the team that I’ve put together there, where it is people that have music backgrounds, people that have theater backgrounds, people that have cyber backgrounds all bringing together these great and unique skills.

Dave Bittner:

Yeah. That fascinates me because that is something that I hear more and more particularly as we have this workforce shortage. That folks like you are reaching out and finding people from all different disciplines and walks of life and finding their place within cybersecurity.

Christopher Crummey:

Exactly. That ability to communicate the key business points along with the technical points actually requires many different skills, storytelling skills, the ability to communicate. One of the best practices we talk about in the range is something called a BLUF, a bottom line up front and being able to do that is a key feature.

Dave Bittner:

So 28 years at IBM means that you have seen a lot of changes in IBM itself. The company has gone through some evolutions over that time.

Christopher Crummey:

Absolutely, we’ve seen … When I first started DOS was king and 1-2-3 for DOS was the big item back then. So yeah, it’s gone through some amazing changes and that is a fascinating part of IBM, is its ability to adapt to the market as quickly as it does sometimes.

Dave Bittner:

So let’s dig in and talk about the work that you’re doing with the cyber ranges. For folks who aren’t familiar with what that’s all about, can you describe it for us?

Christopher Crummey:

Yeah. So we provide a highly immersive gamified tabletop, for lack of a better word, in the cyber range. So it’s for certain skills that walk in. It’s hands on technical training, for a CEO and his or her staff. It would be a full business response, but it’s 100% gamified. So the way I describe it is, cyber best practices meets a game of clue, meets a Disney roller coaster ride. We’ve got one in Cambridge, Massachusetts. We’ve got a dedicated cyber truck that runs through Europe with 20 seats on it with a rolling data center that’s traveling through Europe. I was just in Vienna a week or two ago with the truck.

And then we also bring this to customers in a command on-site model, if you will, where we bring iPads and other things to customers at a hotel or at their location. So we’ve put about 6,000 customers in the last three plus years through these experiences. So it allows you to really get a feel for where your gaps are, where your strengths are, and then what that action plan is going to be once you get back to the office.

Dave Bittner:

Well, let’s walk through it together. What is a typical day like for this? If I send my team to experience a day at the range, what can we expect?

Christopher Crummey:

So basically once you get to … Let’s start with Cambridge, if you will. You get to Cambridge, you get seated and then we start with this idea of one of the best practices. So that’s one of the things you realize is, we’ve had some of the most mature customers in the world come through the range. We take and re-talk about and re-educate some of their best practices on a daily basis. So one of the first things we do is how are you organized in the room? One of them is this concept of a fusion center. So, we introduce a cyber incident in something we call left and right of boom, where boom is where you are literally on the defense and left of boom is where all the technical stuff happens. The phishing email and the malware and the credentials and all this type of stuff.

But there’s the right of boom where you have to talk to the press and write a holding statement and maybe engage with law enforcement. Or deal specifically with regulators or deal with an investigation by the department of financial services or something of that nature. Because we do focus on different industries. So starting with that concept, you’re going to be organized to be able to respond to this in a full business response. That is probably one of the biggest mistakes that we see customers make is that they think a cyber response is a technical response. Reality is a full business response. So someone’s going to play the role of legal, someone’s going to play the role of cyber, someone’s going to play the role of risk and fraud. Someone’s going to play the role of communications and PR, someone’s going to play the role of HR, et cetera, et cetera.

Now we’re going to unfold real incidents that have happened and now we’re going to see how you respond to them. How do your runbooks stand up to this? Most importantly, how does your culture stand up to this? In a lot of ways we feel like we are in a best practices laboratory where we look at, how do they handle it from a cyber perspective, how do they handle it from a leadership perspective, how do they handle it from a crisis perspective? I’ll turn to a person in the middle of the situation and I go, “Are you in a crisis?” He or she will say, “Yes.” And I turn to the person to the right and they go, “No.” I’m like, “All right. Maybe this is one of your action points for later is to have a common language as you go through this.”

So everything we do in the range is based on true stories and we reverse engineered them. So maybe one part of it is coming from a recent breach. Maybe another part’s coming from a recent ransomware moment. Maybe another is coming from a headline somewhere and you’re going to live through all of those things in real time, under pressure, not having the right information and you really figure out where you are. Does your runbook stand up to this? Do you even know who to call? What is the definition of a crisis? When do you go under privileged and confidential at this moment? This is where the psychology comes in, this is where the human part really shows its face and so that’s what makes it fascinating, is how much of this is the human side.

Dave Bittner:

As you’re going through this exercise, does it happen that the people … It becomes real for them, that at some point it’s as if they’re not play acting anymore. You can see that the actual experience is becoming visceral.

Christopher Crummey:

1,000%, and we have a statement that we say that you train like you fight and fight like you train, because you’ll find out exactly where you are when you’re under pressure. I mean tabletops are great and they’re important. But what we want to do is, we really want you to be under pressure. People are literally shaking, they are sweating, they are arguing, they’re swearing like truck drivers. That’s what I mean by, I feel like we’ve put on the lab coat and just see the psychology of this and where the human biases completely take control of the moment. The visceral physical reaction. I warn people how the stress hormone of cortisol is going to change how they react today and it does. Cortisol has an 18 minute half-life in your brain.

I’m warning the leadership that your team, when they are put under crisis, the cortisol stress hormone will make them make very bad decisions. They don’t see it until we point it out at the end of the day and then the biases come into play. There’s two of them that really stick out. One, is this concept of confirmation bias. So the second they … Let’s say there’s a data breach or at least they assume that there’s a data breach. So the other golden rule in the range is validate, verify, and somehow they get notified that let’s say there’s a data breach happening and they automatically assume they lost it. They automatically assume it’s their data and we might reveal at the end that it was their ecosystem that lost it. This data is eight years old, so it’s not new. So they just go, “Oh my God, we totally assumed it was.”

I’m like, “Exactly.” The bad guys to a certain degree are counting on that. And then we also see this concept of job bias, where in the middle of a crisis someone feels like, “Oh my God, I have to do my job. I’m going to now jump in and derail this incident response because of my job.” Because they’ve never tested this before. They’ve never felt like a … So to a certain degree they panic and feel like they have to jump in. So it is a … David, it is fascinating, part best practices, part psychology, part leadership, part crisis all woven together when you put someone under pressure.

Dave Bittner:

How important do you suppose it is to get them on site with you? To get them out of their natural environment?

Christopher Crummey:

I think it’s really, really important and we don’t really allow remote participants. Because of the immersion and the gamification and other aspects that gets them into the game, that gets them into the experience. So that’s really, really important. We have fake stock and Twitter that runs in the room and decisions that you make in the room affect the stock and the Twitter. We want to figure out when do other parts of the business kick in. If the stock drops more than six to eight percent, does an entirely different team have to be involved now? Do other things kick in? One of the other things we talk a lot about is all the things that have to stop happening when you’re in the middle of a crisis. All the things that are in your runbook, but all the things you need to stop doing while you’re in the situation.

So those things are really, really important to feel like you’re there and in the experience. It’s one of those things where you can see in the beginning, they’re not sure what’s going on, but wow, do they really … They really get into the game. I remember a funny moment where, I mentioned, keep your eye on the stock and the Twitter over here. And about 30 minutes later I look over and this guy has written down every minute change the stock went through. So I didn’t want him to go that far, but he was in. He was in the game. So, I appreciated him doing that.

Dave Bittner:

But it must be fascinating also to be able to track what different groups value. Like you say, there’s people that are going to be keeping their eye on the stock price. There are people who are going to be keeping their eye on the social media. They’re people who are going to be keeping their eye on the data itself. And what an interesting window into how different people prioritize things within an organization.

Christopher Crummey:

It’s so true. Again, that’s where the job bias kicks in to a certain degree. One of the major best practices that we talk about is this concept of a North star and this concept of a commander’s intent. So for a leader to lead, they need to give everyone in the room what is the North star? Is the North star responding to the stock? Is the North star responding to the regulators? Is the North star responding to the customers? What is the North star? If you remember the Maersk situation a couple of summers ago, their CEO released my favorite commander’s intent I’ve ever heard. If you remember, they got hit by the NotPetya really, really hard. But he issued a communication that said, “Do what’s right for the customer. Do not wait for headquarters. We’ll accept the cost.”

I thought that was one of the best leadership moments in a crisis I’ve ever heard because it empowers the employees, they know exactly what success looks like. In the middle of a crisis, there’s a really good chance I’m not talking to my boss for the next 12 hours, so I know what to do and I know what that means. The North star was the customer. So you hit it right on the head. The job bias says, “I’m in legal, this is the direction we need to head in. I’m in communications, this is the direction.” So the leadership needs to give the North star. And then the commander’s intent is that module or that communication that gives you the success.

Dave Bittner:

Do you have any insights on the difference between what you would categorize as mature organizations versus ones that may be at an earlier part in their journey with the types of things that bubble up, the types of things they discover about themselves?

Christopher Crummey:

Yeah, and I would say 1,000%, it’s culture. 1,000%, it’s the culture of the company. I mean if you look at how strongly Equifax has come back from their situation, it is very much about how they’re organized. It’s very much about their culture. It’s how people report into Equifax. It is financial incentives on how you handle security.

So for example, I think mature customers address the human side stronger than anyone else. What I mean by the human side is, for example, they will train their employees at home first. Meaning, rather than coming in and having cyber training at work, they’re like, “Listen, let me make you better at home as a father or a mom or a sibling and give you awareness about your WiFi at home and your passwords and two-factor authentication.” All that comes into work. So if you really want to make your culture stronger, start at home.

And then the other thing is they’ll bring in pocket runbooks. They’ll bring in wallet runbooks and the mature customers always sharpen the saw after an incident is over. They spend a lot of time reviewing what worked and what didn’t work. These are the things that we see a lot in mature customers and especially how they relate to business continuity and how they relate to crisis. That cyber is at the same level as a hurricane or an active shooter or other things. Cyber is a brand new language and a brand new set of steps that you need the employees to go through. So those are some of the mature things that we’ve seen. Sometimes we get customers that are like, “I’m sorry, I don’t even … Can we back up? What is a runbook? I’m sorry, can we just go back that far?” So, it varies on size and industry and maturity, but I would say the culture is the number one indicator of maturity.

Dave Bittner:

I think about some of the CEOs that I’ve known who are busy running their companies and running at high speed all the time. I can imagine them saying to their team, listen, you guys go and you guys play for a day. I’ve got stuff to do. How do you overcome that buy in? How do you get that buy in from all levels of the company to say this is time well spent?

Christopher Crummey:

This is a great question and we’ve actually done it on Saturdays, because that was the only time the CEO and his or her team could actually do this. They realize that this is not a tabletop, that this is a fully interactive situation. We also go and not only do we play the role of the defenders, but we also play the role of the hackers. So we have a whole team that will educate the executives on why they are 12 times more likely to be a part of a cyber incident than any other part of the corporation. We’ve got a team of folks that will do OSINT research on the CEO, with permission, of course. They are floored as to what we can find out and how we would fake craft a phishing email and the information we can find out by the fact that your daughter said happy birthday uncle. So now I know the maiden name and it just continues and continues and continues as you know.

That is the aha moment, right? Where the email that you get afterwards was, thank you. We’ve gotten more funding and more head count for next year because now I’ve got alignment of the executive staff. They understand this better because you personalized it to the corporation and you personalized it to them. It’s the currency in the range and when we do the blue and the red experiences is aha moments. And when someone has that aha moment of how creative the bad guys are and how patient the bad guys are and the mistakes that others have made. That is when you start driving that culture and driving an understanding of why they need to invest in both the human and the tools and the teams and other aspects. That to me is one of the reasons why it’s been so successful, is the fact that it does address those cultural situations through aha moments.

Dave Bittner:

What are some of the main lessons that you’ve learned about incident response by running this range? The experiences that you’ve had with all these people, these thousands of people who’ve come through. What are the take homes for folks who are building their own incident response programs?

Christopher Crummey:

Don’t let the range be the first time you’ve ever met face to face. Sometimes it’s not logical because you’re global in a lot of ways. We find and we’re pretty spot on on this. We could tell probably in the first 30 minutes if someone was a prior law enforcement, military, or a first responder, they do the best in the range. They do the best. Because they train to crisis, they train to go towards the boom. There are moments where we purposely drive chaos in the room with actors and phone calls and stock is dropping and the … We literally drive crisis to the point where it’s very difficult, extremely difficult.

People will push the phone … I mean we laugh at it sometimes, but they’ll push the phone to the person next to them because they don’t want to answer. So this is good to get them to feel, to get a taste of what a crisis feels and looks like. So they’re better prepared for when it really, really does happen. Because that’s the important part. But we can pick up on previous experiences relatively quickly.

Dave Bittner:

Yeah. How interesting it is that we seem to be at long last really bringing into focus this human side of things and this highly technical world that we’re in, in cybersecurity. That how much of the balance hangs on the human side of things.

Christopher Crummey:

Go back to that mature customer thing for a second. We talk about something called frictionless security where the mature customers are like, “Okay. We have an anti-USB policy here.” That’s one of the things we show customers, the executives, is how dangerous a USB stick is and a USB Ninja and all these other things. They’re like, “Oh wow, that’s what we’re up against.” But for example, if you take away the USB stick for companies but don’t give them an alternative to share files, you actually make yourself more vulnerable. We’ve seen customers go to a 15 character password and provide password managers for everyone, because human beings are not designed to manage and remember 15 character passwords.

So help your employees do the right thing by making it frictionless and giving them an understanding of why you need to change your passwords. And why the bad guys are taking advantage of the fact that you reuse the gym password for your Netflix password, for your bank password. They’re going to leverage that to their advantage. Once they understand how and why they get those aha moments and their cyber awareness does come up. But the frictionless part I think is key not only for the culture but to make it easy for your employees to do the right thing.

Dave Bittner:

Our thanks to Christopher Crummey from the X-Force Command Centers at IBM security for joining us.

Don’t forget to sign up for the Recorded Future Cyber Daily email, where every day you’ll receive the top results for trending technical indicators that are crossing the web, cyber news, targeted industries, threat actors, exploited vulnerabilities, malware, suspicious IP addresses, and much more. You can find that at recordedfuture.com/intel.

We hope you’ve enjoyed the show and that you’ll subscribe and help spread the word among your colleagues and online. The Recorded Future podcast production team includes Coordinating Producer Monica Todros, Executive Producer Greg Barrette. The show is produced by the CyberWire, with Editor John Petrik, Executive Producer Peter Kilpe, and I’m Dave Bittner.

Thanks for listening.