How to Achieve Cybersecurity Effectiveness

Transcript

Jonathan Reiber, AttackIQ:
Welcome folks to the second episode of Think Bad, Do Good, AttackIQ’s new podcast. Today we are extremely pumped to have two leaders from the Chertoff Group, which is one of the world’s leading cybersecurity and risk management consulting firms. We’ve got Adam Isles with us. He’s the principal of the Chertoff Group, he’s been working at Chertoff for the last decade advising companies on cybersecurity risk. Prior to that, he was at the Department of Homeland Security when it was just brand new advising the Secretary as the deputy chief of staff, and he’s a lawyer by training. He was at the Department of Justice for a long time. He was there during 9/11. You said you were at DOJ during 9/11, Adam, is that right?

Adam Isles, Chertoff Group:
That’s right, exactly right.

Jonathan:
Yep. And also during Katrina. So, he’s seen a lot of crisis management experience and companies are now benefiting from his expertise at Chertoff where he oversees the firm’s cybersecurity offerings. Then we have Kurt … I’m sorry, man. I’ve been rehearsing Kurt’s name endlessly, endlessly, and I even lived in Turkey for three months, and you’d think I could pronounce a Turkish name? Well, but I can’t. So, Kurt tells us your last name one more time.

Kurt Alaybeyoglu, Chertoff Group:
Alaybeyoglu or just A10.

Jonathan:
See now, A10 is something Americans can say, but you think after practicing this, I could say your name, Alaybeyoglu. See, I got it. That’s good.

Kurt:
There you go.

Jonathan:
And podcasts listeners can also hear my son chiming in from upstairs, which is great. It’s like we’re going to have a high-end conversation, but it’s COVID-19 so here we are.

Kurt:
The realities of work under COVID-19.

Jonathan:
Yes, exactly. So in this call, we’ve got two gentlemen with two very different backgrounds, but in very good complimentary backgrounds for what the Chertoff Group does, as it advises its clients on risk management. But Kurt was a cyber warfare officer in the U.S. Air Force. He would spend a lot of time on the offensive side. He was on one of the combat mission teams within the cyber mission force, which is … readers of the blog will know there’s three missions for the cyber mission force and that’s the offensive component. So, he’s got deep expertise in breaking into networks, understanding how to get in and then therefore understanding how to secure a network from the outside. So Kurt, thanks for coming on. Yeah.

Kurt:
Thank you.

Jonathan:
So today, what we want to do is, we’re going to talk about security control assurance and cybersecurity optimization, and a lot also about MITRE’s ATT&CK, and how you validate that security controls are working. Both of these guys, they advise their clients and companies all over the world on cybersecurity risk management. So Adam, why don’t we start with you, how does cybersecurity risk look to you from your seat and what are the things that you keep in mind as you approach your clients? What are you trying to help them to do?

Adam:
It’s interesting. I mean, cybersecurity is not a new risk, right? You talked about DOJ, when I started at DOJ in the late 1990s, there was a book called the Cuckoo’s Egg, right? That was required reading, was actually copyrighted in 1989. It’s basically about a cyberespionage campaign to break into Lawrence Berkeley National Lab. Right? So, what we’ve seen, right, is even back in the late 1990s, you had presidential decision directives that were trying to get a handle on both physical and cyber critical infrastructure protection. But since then, right, technology has become even more ubiquitous than it already was. Right? We live in a borderless world, at least in cyber spaces, with limited consequences and asymmetric opportunities for our adversaries.

Adam:
So, when we go into an organization, right? I mean, before we start talking about, “Okay, what controls do you have in place?” We’re actually trying to answer a couple of basic questions, right? Which is based on my business model, right? Who’d be coming after me. How would they do it? Do my defenses actually map to likely threats? Does what I have in place actually work, and are we prepared if something goes wrong? Those are the basic questions we’re trying to address.

Jonathan:
Yup. Kurt, what about you? Does that jive your …

Kurt:
That absolutely jives, even from an operator’s perspective. I mean, coming from a mission-planning and ops-focused background, you walk into your commander’s office and the thing they want to know is what is your opponent’s most likely and most dangerous COA or courses of action. The reason for that is because you have limited personnel, limited resources, limited time to execute. So, you have to tamp down to what is in the universe of “what do I care about” and everything else is noise, I can’t deal with it. So, by focusing on that, by looking at who are the most likely people to attack me, what have they done in the past? Those become … and then looking at, “Okay, not every TTP is the same, which ones are going to be your most likely? Which ones are going to be the most dangerous, and how do I plan around that?”

Jonathan:
Yup.

Kurt:
When I first came into the civilian world, I didn’t see much of that threat modeling happening.

Jonathan:
That is 2018, right? So this is pretty [crosstalk 00:05:40]

Kurt:
Yeah, 2018.

Jonathan:
So, you didn’t see a lot of that coming in?

Kurt:
No, no. I mean, I had only had even a few assessments under my belt at the time I realized that this is something that was sorely needed, and there have been books even written about this. I mean, Adam Shostack comes to mind from Microsoft. I mean, he literally wrote the book on threat modeling, but mostly on focusing on and looking at your code and looking at threats from a developer’s perspective as opposed to enterprise-modeling perspective. I think part of that is also because until MITRE ATT&CK came into the fray and became as evolved as it is right now, it was really hard to do that.

Jonathan:
Adam, have you seen a transition with the advent of MITRE ATT&CK with your clients focusing more on MITRE ATT&CK and known TTPs? How is this changing their behavior?

Adam:
So, we’ve had a number of frameworks in place for a number of years and a ton of spending. Almost, well, I should say over a $100 billion annually, in product- and service-related cybersecurity spend. And yet we still continue to see compromises that are calamities for the organizations that are victimized by them. So, what people are realizing is, to the extent that the conversation has historically been around vulnerabilities and vulnerability management, that space, the attack surface, is getting even more complex. If you’re a Fortune 100 company in this country or abroad today, likely the reason you’re that large is because you’ve grown through mergers and acquisitions.

Adam:
You’ve got multiple complex IT environments that you bring together, source codes getting even more complex than it was historically, you’re seeing a proliferation of the use of open source third-party code and a lot of outsourcing as well. Right? So, like you’ve got this almost naturally more complex attack surface. When we start to think about threat, historically, people have really been challenged to talk about threat beyond initial access, but people get the importance of defending against spearfishing, of security and boundary defenses. The problem is, inevitably, right, someone’s going to click on a link. Something is going to get through a filter, some web server is going to be misconfigured, there’s going to be some external interface that doesn’t have two-factor on it and what happens then?

Adam:
So we see a real focus on is, whereas historically, defining and modeling the universe of threats that could target an enterprise had been subjective particularly post initial access. Whereas historically, security guidance didn’t neatly map threats to control choices or for that matter to validation and testing choices, MITRE kind of unlocks that.

Jonathan:
Yep. One of the things we’re thinking a lot about is the idea of cybersecurity optimization, which is like, if you’re CISO at a firm, for the last decade, there’s been an increase in spending, but there’s been … and you’ve reported to your board and your leadership team saying, “Okay, I’ve patched the following things, we’re compliant with this standard.” But that’s not actually a way to measure your effectiveness, and where we want to get to is to be able to say like, you can report to the executive team of the board and be like, “Our security controls are operating at maximum capacity, at 95 percent, we’ve identified the following gaps. This is what we’re going to do to mitigate them.” Then if we’re under a budget constraint environment, you want to be able to say, “I’ve rationalized and I’ve measured all of my tools to say this is worth the money.” Are you hearing folks talk like that?

Adam:
Yeah. So, let’s talk about that, let’s unpack that a little bit. All right. So, if you’re an executive, right, you’re not going to be focused on a specific vulnerability, specific TTP, you’re not going to understand that, but you are going to say, “Look, based on my business model, what should I consider as reasonably perceivable threat actor interest in me, in my organization’s critical processes and data?” So, you can use the attack framework, right, we supplement it with additional information to map business objectives to threat actor types that would be likely to target those. You can then also use the framework to map threat actor groups to the tactics, techniques, and procedures that they’re known to have used, and you can supplement it as well.

Adam:
Because once you get down to the level of tactics, techniques, and procedures, that’s where you’re able then to map to defensive countermeasures, because I can map, and Kurt can explain this in much more detail, but I can map a TTP to a defensive countermeasure, particularly post initial access. So, now all of a sudden I’ve got, “All right, who’s coming after me, how would they do it? What would they use? What do I have to defend against that kind of activity?” And I can start to identify whether I have coverage gaps, or conversely, whether a targeted investment would give me outsized coverage against the threat model that I’ve now developed using the MITRE ATT&CK.

Jonathan:
So you can lead to an ROI analysis.

Kurt:
Yeah, exactly. So one of the things that … in the past, you were mentioning about different frameworks and about compliance, CISOs measuring by compliance, and one of the things every [crosstalk 00:11:20]

Jonathan:
Which is good, I’m not trying to say don’t be compliant. Right?

Kurt:
Yeah, absolutely. I mean, there are legal and regulatory reasons to be compliant for certain things, but every security professional will tell you that compliance is not security. So, one of the things that ATT&CK has allowed people to do is, is to be able to go through and say, “All my controls are actually effective?” It gives you a universe to draw from, to be able to look at everything and say, “Okay, now that I know these groups are interested in me, now that I know here’s the history of TTPs they’ve used, now I can go and say, all right, what controls do I have in place in order to do testing?”

Kurt:
Then I can start using automated controls assurance platforms like AttackIQ to be able to go in and say, “Okay, I want to run scenarios, X, Y, and Z. Wait a minute, I failed this. Why did I fail this? I’m supposed to have this EDR in place, this IDS here,” and it allows you to go into a depth to be able to say, “Hmm, now I see, the logs that I thought were being forwarded to my EDR aren’t there, or the detection logic that I have for that has a flaw in it.” And I can start testing it and then turn around and say, “All right, let’s test again.” Because that’s one of the things up until now has been missing is that you haven’t been able to test what you have.

Adam:
But I think that there’s another point that lies on top of that. Right? Which is that, in an era of limited resources, right, we talked earlier about return on investment, not all TTPs are created equal, right? So, we’re necking down the environment from the universe of anything that anyone who could conceivably do to … Here’s what starts to be a little bit more reasonably foreseeable. Then within that category, we start to try to understand, “Okay, within the realm of what is reasonably perceivable.” It’s what Kurt said earlier on. Right? What’s most likely and most dangerous. So how do I start to think about risk rating individual TTPs, right? This is part of what Kurt’s innovation within our practices revolved around.

Jonathan:
Yeah. That’s interesting. Why don’t you tell us a little bit about that and what makes you guys different in your approach?

Kurt:
Sure. So, a little bit on what we’ve done starting back, and this starts with a bit of a story back in 2018, when I first started on my first assessment, it was a massive corporation, and trying to wrap my head around how I’m going to actually assess something so big-

Jonathan:
As opposed to breaking things on behalf of the U.S. government?

Kurt:
Yeah, or breaking into individual end-points or securing, hardening end-points, et cetera. One of the interviews that I had with the lead security architect is where he said he paid the million-some odd dollars for this anti-DDoS protection device. It took about six months of browbeating for the network engineers to actually even get them to place it on the network, and when they did, instead of putting it in front of the core router as was needed to be the defendant, they put it on a span port on the side which effectively made it useless, but you’re compliant, the control is in place, but then you go and say, “Well, there’s the traffic, there it goes, I can’t do anything about it.”

Kurt:
So, my thought was, after that, there needs to be a way to be able to test, but then the question asks, “Well, how do you prioritize what you test?” Well, right then on the first ATT&CKcon, 2018, Travis Smith gave a talk on the teach model, which goes on basically, difficulty of exploitation, how hard is it to actually execute a TTP? Well, at the same time, I had actually worked internally to develop … Well, how difficult is it to defend against a TTP? And that ranges from what is essentially basic hygiene down to “your mileage may vary” down to, “Hey, there’s really no mitigation you can do on this because it abuses operating system functions. So, if you try to block it, you will break things that are actually supposed to work.” So the best thing you can do is to test.

Kurt:
So what we then start to do is, we melded these two things together to look and say, “Okay, let’s focus on what is most likely and what I can actually do something about, and most likely ends up being more often than not. How easy is it? I mean, attacker, maybe this is just me, but as an attacker, I’m lazy. If I can use something that’s weaponized, that’s a metasploit exploit as opposed to writing my own script to executable, I’m going to use the exploit that’s already available-

Jonathan:
That’s a really good point, actually, just to pause on that for a second, because if you’re one of the most highly trained cyber warfare officers in the U.S. Air Force, and that’s coming from you saying you’re going to use the easiest thing most available, that tells us something about adversaries in general.

Kurt:
I mean, I would defer to you on that [crosstalk 00:16:32]

Jonathan:
[crosstalk 00:16:32] in case of one.

Kurt:
Yeah. I mean, I would say that there are people much more advanced than I, but even they, and having talked with them while I was doing that work is, general rule is, men, don’t reinvent the wheel. I mean, if it’s there and it’s available, use it. So, by focusing on those, we can turn around and basically create a TTP by TTP priority level, and we can then criticality-weight them and be able to provide that information and be able to say, “So, here’s your threat model, within that threat model, here are the things you should prioritize, here are the things that you should probably focus a little less on, and then start testing.” Then you can start really narrowing down what your efforts are going to be. So you can narrow down based on, “Hey, what fixes can I make that’s going to provide the most bang for buck across my threat model? And also what fixes can I make that focuses on those top criticality TTPs.”

Kurt:
Testing controls platforms allows you to do that at a level of automation and depth that you didn’t have before. I mean, pen tests are great, red teaming is great, but they’re manually, they’re difficult, they’re manually intensive, they require a lot of personnel hours to accomplish, and unless you have very well scoped down rules of engagement as to what you’re trying to get out of that, sometimes you may walk away with not really understanding what it is exactly you’re supposed to do. This allows you to be able to test it. It democratizes the ability to test by shortening that pipeline of, instead of needing a full blown red teamer, which will take, I mean, eight years of experience, nine years of experience and training to maybe get to that level, it now allows your junior analysts to be able to push play, analyze, and go.

Jonathan:
And even a red team … I mean, red team would maybe, if you do a once annual audit, they get three assets out of 800, that doesn’t tell you anything about the overall health of the enterprise. So one of the things that I was very attracted to AttackIQ because of the scale capability, like you can quickly scale adversary emulations and run them against an enterprise. The interesting thing about it is like it’s a new market, right?

Kurt:
[inaudible 00:19:06]

Jonathan:
There was a study that was done, I can’t remember who it was, I shouldn’t guess, because then I’ll give credit to the wrong organization, that said breach and attack simulation in terms of growing IT markets is up there with big data and AI in terms of a space. So within cybersecurity overall has this massive spend, but folks would begin to recognize that if you can automate adversary behavior through a platform. So I’m glad to hear that it’s useful for you obviously like [crosstalk 00:19:35]

Kurt:
Well, yeah. 100%.

Jonathan:
Yeah.

Adam:
So, you go back to enterprise risk management, security risk management over the decades, and there are three foundational planks, right? You assess. You in most cases mitigate. Sometimes you accept, sometimes you transfer, sometimes you avoid, but let’s just focus on mitigation, and then you monitor what you mitigated and make sure that it’s actually operating as intended, and the modern piece is where people so often fall down. Right? I’ve got something in place, Kurt talked about it earlier, but does it actually work? One of the things I’ll always remember from being at the Department of Homeland Security in the mid 2000s was the Hurricane Katrina. We had a situation where, I mean, the hurricane was a nasty event, but it was the flooding of the city of New Orleans after the fact that was a national catastrophe.

Adam:
Key to that, right, was the fact that the 17th Street Canal levee and others like it were breached at some point. That levee had been built since the early 1990s. It had never been inspected between the time that it had been built and the time that it was breached in August of 2005. And we see that-

Jonathan:
What year was it built, did you say?

Adam:
In the early 1990s? I think 1991. But what you want to see, whether we’re talking about infrastructure, whether we’re talking about technology, is the level of validation, right, of controls assurance that the thing you think is in place is as strong as it’s supposed to be, and is operating as intended.

Jonathan:
Yeah. Yeah, that’s exactly right. I mean, yeah-

Adam:
And to your point, right? Look, there’s testing, there’s audit, but how do you scale that? Right? How do you scale that across the number of organizations that are out there that are exposed, that have data, that have compute power that someone wants to subvert?

Jonathan:
Yeah. And doing it at scale safely and in production, right? Like you want to be able to do all those things that gives you an advantage. It’s interesting, like I worked in national security prior to coming here and still, I guess, am in national security. Right? Cybersecurity is a little bit of national security that way, it’s interesting. But in military culture, after 9/11, and after every crisis, we measure how can we get ahead of threats? How can we prepare for them? One of the things about the military that I love so much is, you have to exercise constantly for it. The other thing that I like about this platform is that you can run validation tests quickly as many times as you want per day against whoever you have it aligned against for, to test your security controls, which is just so important. So you can really measure it.

Adam:
So, the other piece in addition to measuring that becomes important, and you have to understand this from a national security point of view, is that, it takes a village, right? And so, you actually, as you go through the cycle of risk management, right, you see multiple teams coming together, and what ATT&CK does and where validation fits in is in providing common language, common data that multiple teams can use. Right?

Kurt:
Common taxonomy. Yeah.

Adam:
Yeah. Common taxonomy, exactly. Right? So, now what I can do is I can start by looking at inherent risk and I can bring my risk team, my business continuity team in, the business to say, “All right, from an impact point of view, what are we most worried about?” I can bring my threat team or outside expertise to help me translate that into threat actors, TTPs. So now I’m starting to think about defensive countermeasures. All right, let me bring security engineering in, right, to help with that. “Okay, what are we going to test?” Now I need to go back to my business continuity team, right, the business, my risk team, to try and start focusing on that. Then what do we do about it? What ATT&CK does, right, as Kurt mentioned, is it starts to offer a common taxonomy to communicate around a threat across teams in a way that it had been stovepiped beforehand.

Jonathan:
Yeah, that’s right. I mean, so it helps you validate your security and make your security more effective. Then it also helps you say, “Where do I need to fill gaps either on my teams or in my technology. Are my investments panning out?” The thing that I really love the most is just like the ability to report and get effectiveness against a baseline of TTPs to be like, “We’ve tested against the following TTPs, and we can say [crosstalk 00:24:14]

Adam:
So, this is risk-based effectiveness?

Jonathan:
Yeah.

Kurt:
Yeah, and I think that also goes back to the common taxonomy piece, because it allows you to create these crisp concise reporting and making sure that everyone is on the same page. So when you’re talking about initial access, when you’re talking about discovery, when you’re talking about these individual tactics, and then going down into the TTPs, the attacker accomplished execution through PowerShell or via PowerShell, the attacker did X by Y. Having that common taxonomy is not just important within an organization because words matter, but it’s also important across an industry because now you’ve got a whole areas, like you’ve got IDSs and IPSs showing their detections against attack, the IDs, against technique IDs. You’ve got EDRs that are mapping themselves like what TTPs they are effective against based on MITRE ATT&CK.

Kurt:
You’ve got the whole cybersecurity industry rallying around because this is something that has been sorely needed because too many people have been saying the same thing in too many different ways. And it has led to things like vendor over promise because a vendor can say one thing and then the developers of said product would be like, “Nah, that’s not what I meant when I said X.” Well, now you eliminate that confusion, and when you’re going out, when you’re looking at different items, you know exactly what widget X is going to protect you against, and all you need to do now is measure, okay, how well does it perform in those functions.

Jonathan:
Yeah. Which is why I think the MITRE ATT&CK evaluations are so positive for vendors. They can say like, “We’ve baselined our capabilities against MITRE ATT&CK, and now we come out and we can say this is how well we did.” I think it’s terrific. It’s really good. It provides a way of assessing vendors too. It’s like we all have been working in cybersecurity now for 10-plus years, each of us, and we’ve seen this tremendous growth, whether it’s from the cyber mission force, in Kurt and my standpoint or on the capability side in the commercial sector, Adam, and now me too, right, all of us have seen this tremendous growth and expenditure. Now we can finally say, we’re going to close the circle and say, “Is it working as best as it can?” We’re now providing a capability to say this is the overall feedback loop for all the investments you made. It’s not an insurance policy because insurance has its own thing, but it’s like we’re now going to prove that everything is working as it should.

Adam:
Yeah. I mean, the day we eliminate cyber risk is the day we eliminate crime. Right? I mean, it’s always going to be out there.

Jonathan:
[crosstalk 00:26:59]

Adam:
So, risk elimination is impossible, and so this really becomes a question around risk-based effectiveness.

Jonathan:
Yeah.

Adam:
I mean, am I taking reasonable actions against a reasonably foreseeable threat?

Jonathan:
Yeah. That’s a great quote. I’m definitely going to use that. The day we eliminate cyber risk is the day we eliminate crime. That’s good. You make a good speech writer, Adam.

Adam:
Oh, I’ve plagiarized. Right?

Jonathan:
You heard it here first. At Think Bad, Do Good. Kurt … go ahead Adam, please.

Adam:
Well, the other thing I was going to say is, I think looking back … I mean, DHS, one of the largest civilian transformations since World War II, one of the things that … principles that Michael Chertoff drove into the department as it was being transformed was, everyone was going to operate on a risk basis. Right? It was really about risk management regardless of which component we were talking about. We tried to reflect back on all the transformation that occurred during that time period and said, “All right, what are the foundational principles that need to be in an effective security program?” We focused on three, right? The first like, duh, is that it needs to be risk-based.

Adam:
The second is that it needs to be … what we refer to as trusted, right? This speaks to controls assurance, to speaks to testing validation, right? Things are operating as intended. The third piece is that it needs to be intuitive. I think that can also be something that escapes the stakeholders at multiple levels. This is where I think automation at the level of controls validation becomes important, but it also becomes important at a senior level. Right? So like, you know this, Kurt knows this, maybe I know what effectiveness looks like, but the people that are funding this activity, the people that are accountable to shareholders or to customers or regulators maybe don’t appreciate this as much.

Adam:
Another part of what becomes important is to think about how to translate these results into a metric that actually speaks to security performance. Right? You think about it, right? At least most of us have applied for a mortgage or a car loan. Right? We know what the score we get back is, and we want to see a 770 not a 510.

Kurt:
510, yeah.

Jonathan:
Or on your case, an A10, current. Yeah.

Kurt:
Yeah, or in my case.

Adam:
But what becomes pretty stark, right, when we have these conversations and that’s part of what Kurt’s built is, you can go into an organization and you can show them two different environments and you can show one environment that’s performing at a 750 or 760. Okay, right? Then you can go to another environment and you can see that control hasn’t been properly integrated and you can see performance down at a 480, when people see that, there isn’t a whole lot of debate right around, “Okay, we need to deal with this immediately.”

Jonathan:
I love this. I mean, you being there at the starting of DHS in a tense national political moment, right, after 9/11, like the country is totally shocked. We’ve had a couple of shocks since then, right? The global financial crisis, the pandemic, Katrina. I love the way you broke that down. What was it? It started with intuitive and then-

Adam:
Risk-based, intuitive and trusted.

Jonathan:
Risk-based, intuitive and trusted. I liked that a lot. It’s like it gives you clear principles. One of the things about the world that we know is, there’s complexity everywhere and we’re drowning in data. If you’re CISO, particularly in a large organization, you’ve got all these different sub-departments and like everyone has … there’s different levels of complexity with personnel and technology. You want to make things simple as you try and improve effectiveness overall. I really liked that formulation. Yeah.

Kurt:
That actually came from kind of like a offhand comment, because originally when I was designing the visualizations for this, the turnaround was, “That’s great, how do we present this to a board?” My initial thought is, “Well, everyone understands test scores, you’re at a 100, it seems simple enough.” But then, I mean, you already know that no one’s going to be getting 100 on this. This isn’t like your eighth grade history test or math test or something, the waiting is going to be pretty, pretty low or in the middle. If you get 70s or 80s, hey, that’s great. It doesn’t really provide the level of depth required.

Kurt:
Ironically, as I was talking this out loud to myself, our CEO, Chad Sweet, was walking by listening to me talk, he turned around and said to me, “You should make it a credit score or rather like a risk score, something along those lines.” At first, I laughed it off, and then I said, “Well, you know what? That’s not a bad idea.” If I can use something that’s similar to that as Adam said, most everyone has applied for a loan of some sort. Everyone recognizes those numbers, that would be a great way to explain security risks to someone who’s not necessarily a security professional. How we actually explain that is, one of the interesting things is, as Adam mentioned, some areas you see like a 480, sometimes you see this Sea of Red that often tends to scare people[crosstalk 00:33:02]

Jonathan:
Let me pause you for one second, my battery is going to die. So, what I’d like to do, I’m going to get my power cord, but talk for a second to the audience about how customers have benefited from that, explain how some of the feedback folks have given you.

Kurt:
Yeah. As I was going through and saying that we measure these into two different ways. One of the things that we’ve seen is that people get scared of the Sea of Red that they see. One of the things that we take into account when we do the assessment is that, an attacker doesn’t accomplish these attacks, these TTPs,in a vacuum. There’s a little bit of artificiality to controls assurance testing. What we try and take into account, the fact that when we do these assessments is that, in the course of a test or in the course of an actual campaign, there’s a logical kill chain, Lockheed Martin kill chain, kind of moving from initial access to execution and persistence, and ATT&CK actually pretty much covers this from left to right in their tactics categories right all the way down to exfiltration of data or impact.

Kurt:
So, we take into account the fact that, because this is a purple teaming exercise, essentially. I mean, we’re working with blue teams. This isn’t a stump-the-chump, “Hey, we got you” kind of deal, this is meant to improve. So, what we try to look at is, “Okay, this is how you did, but where is the first detection that a human analyst would have put eyes on and would have initiated incident response, and then would have started triage and containment and eradication?” Because then you can start to see, “Okay, here’s how you do as a whole.” You caught your first detection that would have started incident responses right here in this category, and so now what that allows you to say is, now you can take everything from there and to the right of that, now that you’ve started incident response containment, triage, et cetera.

Kurt:
Now you can say that, “Okay. Yeah, I did poorly down there, but I detected early and I responded, so now I can show two different scores to say that, yeah, all told, I didn’t do so well, but we caught them really early and we moved on at fast, and so we actually did not too bad. So, it allows organizations to really gauge how well they’re doing in detection, how well they’re doing in protection [crosstalk 00:35:49]

Adam:
Kurt, talk a little bit though about, for those clients who have scored lower some of the insights, they go, “Oh, wow, I didn’t realize that this is generated.” The ability to show someone from a return on investment point of view, “Look, if you do this thing, you could actually drive your score potentially as high as X, because we have those use cases.”

Kurt:
Yeah. So one of the things … kind of go back, another story time was, in one of the clients that we were assessing that when we did this, in earlier conversations they had stated, “Oh, we don’t need to worry about PowerShell in our environment. We have a GPO in place. PowerShell doesn’t run unless you’re an administrator in X, Y, Z in particular buckets.” So I said, “Hmm, okay.” I quickly wrote a one lighter in the middle of the zoom call, kind of like we’re sitting right now, and I asked them to run this, and I said, “Please show me what happens.” And PowerShell opened up on their box, because their GPO is not really a protective function, it’s an administrative function to lock down.

Kurt:
So, one of the things that we were able to show, because they had this false sense of security on there, I was able to say, “Look, many of the techniques that we ran with AttackIQ, when we did our assessment, all generate and run through PowerShell.” One of the things that you could engage in is PowerShell module logging. So, one of the great things about your environment is you have a GPO in place, you’ve already stated, unique to you, that no one should be running PowerShell. That’s great. Because now, in combination with that, you can turn around and, say, if I turn on even the least verbose form of logging and send that to the SIM. Now I know every time something pops up with PowerShell associated to that, I can already suspect malice. I can already suspect that it’s going to be a malicious action because no one’s supposed to be running it.

Kurt:
So, we were able to show, hey, not just PowerShell, but here are all the other TTPs that you can do with PowerShell that now you have the oversized return on investment from the control that you’ve now put in place compared the amount of protection you now [inaudible 00:38:13]

Adam:
We have other clients that aren’t able to implement a PowerShell GPO, right? Just not going to work in their environment.

Kurt:
[crosstalk 00:38:21]

Adam:
There you can say, “All right, we don’t have an EDR.” Right? Here’s the potential coverage we’d be able to get where we’d have one, or we do have one, but holy crap, is it integrated actually with a SIM in the way we want it to, right? So you can show those differentials in score in before and potentially after a context.

Jonathan:
That’s awesome. I like to use … I think I’m kind of alone in the company sometimes about this, but I like to talk about the Fitbit as an analogy, because it’s not just like … you’re not just measuring your steps. Right? You’re actually measuring how well your exercise is doing. Are you driving down your heart rate? My goal is to get mine down to like 49 or 48, which is like a good thing. It’s not there itself. Well, it’s at 53, 54 right now, which is good-

Kurt:
[inaudible 00:39:14]

Jonathan:
It’s pretty good. Right? Yeah. Anyway, I’m not using this as a chance to brag. Right? That’s not why I’m here. It’s transformative to be able to determine what you just talked about. That’s new and transformative, and for folks who are new to cybersecurity, they may just expect that. One of those things that kept happening to me in my career is like, I always expected the government to be doing something that it wasn’t, and then when somebody launched a new initiative, I’m like, “Well, yeah. I mean, aren’t you already doing that?” And people were like, “No, we haven’t done this before.”

Jonathan:
Well, it’s great, it’s awesome to hear you guys, there’s such richness in your brains and we’ve barely scratched the surface. I mean, I’ve got all these questions that we could spend a long time talking to Kurt about what it was like breaking things for the government. I think it would be a short conversation because you probably can’t talk about it.

Kurt:
Yeah.

Jonathan:
Yeah, but Kurt, any last thoughts before we pivot it over to, Adam, to wrap us up?

Kurt:
Yeah. So, I think one of the interesting things is that for the longest time in cybersecurity is, it’s always been offense has outweighed defense, and we’ve played catch up in terms of being able to … every time something new comes out it’s like [inaudible 00:40:29] comes out with a new signature afterwards, EDR comes with new detection algorithms for it afterwards, I think with the rise, as you said, the growth of automation controls assurance platforms.

Kurt:
I think we’re actually at an inflection point now, because we’ve allowed the junior analysts and essentially the trigger pullers to democratize that skill, to be able to then turn around and say, “Now, even if I’m not the red team or the pen tester, even if I’m not an OSCP or I have these levels of skills, I can now take at a simplistic level, deploy an agent, run these scenarios and get a result and then start digging from there and allow to bring meaningful change to an organization, even if it’s slow. I mean, even if it’s only putting controls that are technique by technique, at least it can now be measured, it can be tracked and we can start really turning the tables on a lot of the lack of data and lack of visibility that we had before.

Jonathan:
Yeah. I mean, it gives you data, right? One of the things is that it gives you data about your exposure.

Kurt:
Yeah.

Jonathan:
You made a great point that I haven’t heard or talked about before, which I would love to pull a little bit to string on, sorry, Adam, before we pivot back to you.

Adam:
No worries.

Jonathan:
Thank you, Adam. The idea that you have defenders, previously you had to be highly, highly, highly trained. Did I just hear you right, that you said like with an automated platform, defenders don’t have to have the same level of capability that maybe they would previously?

Kurt:
Yes.

Jonathan:
So that you could then put your more highly trained defenders onto more niche defensive capabilities so that you can have, not like grunts running your defensive capability on the blue team, because you have the automated emulation, it doesn’t require quite as much [inaudible 00:42:36] of expertise. Is that what you’re saying?

Kurt:
Yeah. Yeah. So, it allows for taking the junior analysts to be able to say run the test and then your more-

Jonathan:
[crosstalk 00:42:44]

Kurt:
And then your more senior analysts who maybe have more experience, have more training under their belt to be able to analyze those results and to be able to provide more effective countermeasures that can then propagate across the network, that instead of having someone try and muddle about and learn that way is that now you have people who can be dedicated to running the tests, and then you can have your more senior folks who can understand what those controls mean, what the results mean on those tests, be able to then turn around and make meaningful changes across the network or across an enterprise.

Jonathan:
I love that point from a cyber workforce standpoint, which is something I hadn’t thought about. That’s a really good point. Cool.

Kurt:
It helps cut that training [unclear 00:43:32]

Jonathan:
Yeah. Adam, how about you? You got last thoughts for us?

Adam:
Yeah. I’ll leave you with three points, right? The first of which is we’re in a time of tremendous change, right? You think about what COVID-19 has done to the work environment, you think about, as much disruption as we’ve seen in our economy, there are also certain sectors, healthcare, logistics, manufacturing, I’d say finance as well, that really need to work, and we have really volatile situations, right, both domestically and internationally, and that will continue. So, you worry about internal frustrations turning outward. So, it’s never been more important to have a lock on, particularly as we’re changing security controls, right, on the fly a little bit to make sure that what we have in place is actually operating as intended. Right? So, that’s point one.

Adam:
Point two is, I think, as we start to generate more data around this, I do think that there’s a play for a much more informed insurance underwriting discussion, right, around, “All right, look, we have risk-based effectiveness, right? We have strong security performance, and we’ve got data that shows it’s trending the right way over time.” That should mean something, right? The capacity, the nature of perils that are covered, et cetera, and I’m excited to see an evolution in that space. I do also think, as this becomes more broadly adopted, you start to get some really interesting benchmarking data. Right? With that, I start to be able to say, “Okay, my industry group, how are we doing?” Right? That becomes really important for boards, right, to understand reasonableness.

Adam:
But I think this is exciting time for us, right? I mean, it’s a perilous time, but it’s exciting to see the opportunity to get at this fundamental question around effectiveness with the degree of transparency, accuracy, and precision that we just haven’t had before.

Jonathan:
That was beautifully said, man. Be hard pressed to hear anyone say it better. Guys, it’s really great to have both of you on, I told you this before when we’ve spoken in the past, but it’s an honor to be able to work with both of you and talk to both of you and learn from your expertise. I know that the audience is going to benefit from it. So thank you. Thank you both very much for coming on the show.

Kurt:
Thank you. Thank you for having us.

Adam:
Thanks. But the more I do this, the more I realize how little I know. So, the honor is mutual. So, thank you.

Jonathan:
Yeah. Cool, good. Well, come on again, and look forward to working with you more.