8:02
Good afternoon, everyone. Well, I've got to say, to be among cyber giants on stage is a real honor for me. Thank you so much for being here. AI has radically changed cybersecurity. AI models have been seen breaking out of containment and hacking to companies and websites, the occasional government. And yet, it seems that these AI models have been exploiting a lot of the time very simple, really easy to exploit vulnerabilities like exposed credentials or simply just guessing passwords. Wendy Nather, um as an advisor to Chief Information Security Officer and a former CISO yourself, you have helped uh
to unpack and others to learn from some of the biggest cybersecurity disasters going. How much of today's risks with AI stem from some of the security challenges we've been dealing with the last 10 or 20, say, years? Oh, a lot of them, because uh at the end of the day, software is running on top of a stack and, you know, we all have problems securing the rest of the stack. I think the only thing that makes it more complicated now is we've got another layer on top. It used to be called people, layer eight, or um the political layer, or the financial layer.
I think we need something else to describe what that is. Right now, I'm calling it the semantic layer because it's not just strings anymore that we care about securing, it's what they mean. And so, um yeah, we haven't solved the basics yet. If they were, you know, if they were easy, we would have done it by now. But, uh it's important to remember the whole stack. And so, Katie Moussouris, you were one of the first cybersecurity leaders uh this year to describe a vuln-a-loser Mhm. after Anthropic released its Mythos AI model, um which kind of opened the floodgates to bug discovery in a way that we haven't really seen since I think with fuzzing, really.
Now this first wave of vulnerabilities and bug discoveries kind of hit, what have we learned since then? And what should be the kind of the focus now? Well, honestly, all the organizations that were struggling with their volume of bugs, either discovered internally or reported to them externally, the struggle has turned into, you know, some sort of a desperation drowning, you know, clawing for some oxygen in this process. And I think, you know, we haven't quite gotten to the place where AI is functionally helping to manage that struggle. So, it's producing more bug reports internally and from outside finders. And then sorting through that
has become more than a full-time job, even with your best engineers using their best versions of AI. So, so much I think right now is about tuning the AI to help you sort through that and prioritize in context. Even now with obviously the advent of AI and instance coming down, is there still a capacity problem or like a lack of people or lack of AI that we need to actually use to break down some of these issues? I think some of it is that the AI itself is not as reliable as we would like it to be in terms of like automated patch creation or automated mitigating, you know, some of these other things. Um we're seeing more and more emergence of
using deterministic checks to double-check what the non-deterministic AI is trying to tell you. And then a lot of the automated bug creation or sorry, patch creation ends up doing something, you know, pretty boneheaded like removing the functionality. No more bug, right? Um not really the patch we're looking for. And so, I think it is a matter of um organizations I think made uh some decisions about personnel earlier than they should have about getting rid of some of the human experts that could properly guide and tune the AI. And so, now we're we're seeing uh folks say, "Actually, we do need more of those folks because we have these AI tools and they're kind of spitting up a little bit of nonsense."
And so, Rudy Mitra, you oversee Amazon Web Services security services, and as such, you have enormous visibility into the threats facing companies today. What common security mistakes are companies making when they're building their infrastructure around AI? Yeah, um I mean, I think a couple of things. One is, you know, as um Katie and uh Wendy were talking about, a lot of it stems from going back to the fundamentals and the security, you know, sort of veggies, if you want to think about that, and getting that correct. And so, things like, you know, identity systems and getting that um you know, up to MFA, multi-factor auth,
systems being patched, uh zero trust. You know, those principles I think are being discussed again because those uh not being in place previously are showing up now with AI systems. But then the other part uh to your question, of course, is once you've thought about the veggies, I mean, the next layer is thinking through like the speed with which vulnerability finding is now changing, you know, with AI. And this asymmetry between finding and fixing. I mean, I think that's the graduate from talking about veggies to sort of, you know, protein or whatever comes next, which is on the food chain, we've got to get that asymmetry. And so, I think there's a lot of thinking going about that.
That's a great analogy there. Um so, Wendy, clowns and enterprise, as we've been discussing, is getting a lot more complex? Thanks, AI. Both in terms of compute and in terms of scale, but also the cybersecurity risks have changed as well. So, are there any kind of no-go zones or systems that you just cannot afford to really bring in AI that might end up causing a potential mess down the line? Oh, boy, where do I start? Um, I mean, I think anything where uh you cannot afford to have an unpredicted uh function. If it involves safety, um you know, you can't be doing that. And so, I think it really depends a lot on what
you're trying to do with it and whether it's okay if it's non-deterministic. Um, you don't want to spend a lot of time in production with people uh you know, going, "Oh, I see a trend and it doesn't look good. What should we be doing?" Um, Camille Stewart Gloster just came out with a book, a really good one. Uh it's called The Insider You Built, and she talks about all of the governance um around AI and what organizations should be doing and how they should be forecasting what they see the models doing. Uh even if they're doing things that are completely legitimate and unexpected, if they're going in the wrong way, uh you want to be able to catch that beforehand.
I think one of the better examples I remember from earlier this year as well in some of my own reporting was when uh Meta AI had that chatbot um for their Instagram. And so, you were essentially able to trick the Meta AI chatbot into allowing a password reset to an account by essentially just giving instructions that it didn't know how to pass. And so, would you see um similar issues with login systems, authentication systems, and HR systems, things like that where there's lots of data where there's very sensitive information. Uh are those the sort of systems that we should that AI systems should really kind of avoid, do you think, Wendy?
Well, there's all sorts of concerns about what kind of data stays in context and for how long. Um there's just, you know, another container to worry about. So, um I think overall, uh we're going to be generating so many events with AI, it's going to be even worse for the logs than it has been. I mean, IoT was pretty bad, but uh the scale and the number of events that we want to look at with AI is just going to explode, and somebody has to be able to parse that out and say, you know, to the model, "Oh, you can't use that word in Britain." Or whatever. I'd love to ask you the same question as well, Katie, you know, in terms of no-go zones or systems or any kind of areas where AI shouldn't really be involved.
Well, I mean, Wendy really said it all in terms of the security-critical, safety-critical systems are the ones that you really want to have some human oversight and direction, but at the same time, you can't be gated by the, you know, having human in the loop slowing things down. And I think um Cory Doctorow's recent book was really put it in some nice terms. So, there's the centaur version where the human head is directing and the workhorse is AI, you know, the horse body. But, then there's also reverse centaur where we're having uh AI making a lot of these decisions and then expecting the human body to keep up. And so, I think that's that's something to really pay attention to in all of this. How can we design
systems that are using AI towards its most efficient ends and at the same time not overwhelming the humans who will get decision fatigue, you know, in certain situations that you do require a human to make those safety critical decisions, but if they're doing decision fatigue because the AI is coming at them too fast, you're going to break the system that way as well. We have seen over the last year or so with the elements of AI an increase in layoffs often attributed to the rise of AI and the inclusion of it. Where is this balance between the need for human capital and the need for expertise and understanding and
institutional knowledge? And essentially just letting an AI just handle it all. There really has to be a better balance here between the two. Do you think, Katie? Um you know, honestly, I would love for us to not be needed at all. I would love for us to all just be getting the universal basic income generated from AI's productivity increases as a dividend to humanity and us sitting on the beach and not having to do any of this, right? We were just talking about the beach earlier. Um but ultimately, you know, it we do still have to mind the human pipeline. And in cybersecurity, we've always had a
problem where we over hire at the senior level and we don't make enough room at the intern and the entry level in order to grow our next generation of seniors. And AI is definitely making it much more trendy to just limit your human hiring to the seasoned experts. What are we going to do in 5 years? So, I think we really have to pay attention to that part of it until we're, you know, ready to give up work in exchange for, you know, labor in exchange for living. So, I would like to ask uh Rudy in terms of your ideas behind finding a safe or safer way to venture into testing AI agents. Is there a way that companies and startups can use AI in a way that is safer, more protected, and mindful than
having them just run a muck inside their enterprises? Yeah, it's it's a great question because, you know, um I'll take a slightly controversial point of view, which is I actually think there is so much role for humans in sort of work that humans do in security because you start from a point of view that humans have been the underpinning of judgment, which all systems run on. Yeah, all enterprise runs on judgment, and that judgment has been the human layer. So, if you think about where we're heading, I mean, one point of view, of course, is, you know, AI will take away all of these workflows and whatnot.
In security, specifically to your question, if you think about a world where we don't even get to the real security work, we get stuck in these long queues of alerts, and SOCs are overwhelmed, and whatnot. I mean, I was reading some stats the other day that kind of blew my mind. Uh it takes now less than 10 minutes to find a CVE, whereas it takes organizations, you know, 34 to 43 days to patch them. So, just think about it. It's minutes to find, and within 10 hours, the average CVE 50% of CVEs are weaponized. That's the disproportion, the asymmetry you have between finding issues, which I think, you know, frontier AI is awesome at, and, you know, fixing at 43 days, right? That's not awesome at. So, when you sort of say, "Hey, look,
where do the safety systems need to go?" I think if you put those two thoughts together, you kind of arrive at this point of view, which is AI should really be pushing forward in terms of the validation, the testing, the fixing of these CVs. And it's compounded by the fact that many of these CVs are not actually high priority CVs. They're They're actually mid to low priority CVs. It's just that the AI systems are amazing at chaining these together. So, I think anyone can take Frontier AI and start to find CVs. The question is, where does AI need to go to address a lot of these chained CVs and fixing faster order minutes? And then letting the 5% of the work that is really creative, judgment-based go to where the humans need to go. And frankly,
security professionals have never gotten to, but always wanted to. So, these days obviously we've seen this year alone an absolute flood of CVs from a bunch of companies from Apple to Microsoft to Oracle, especially Oracle. And we have companies patching as fast as they can, but at the same time trying to understand how do they prioritize what they patch knowing full well that some CVs were pretty inert, but now have essentially become weaponized. Where do you prioritize at that point? Especially when you have limited resources. I think this is where, you know, security products and tools need to evolve to catch up to where the find state of the art is to the fix state of the art. This is where I think, you
know, startups, security companies, mature companies, they're innovating like crazy. Mhm. And it's not enough to just say, "Look, I've got a harness and I'm going to take a Frontier model and find CVs and then ask it to generate code." Um it's mind-blowing. Over 50% of all CV patches generated by AI leave a secondary security issue open. And over 40% of those patches that are generated actually cause production deployment issues. So, they'll take systems down if you think about it. Yeah. So that means the vast majority of fixes that you're trying to generate for security, you're trying to keep up with the find are causing problems in production systems. And I say that because then I think you can reason your way through
like you need agentic rethinking of fixing security issues and CVs. You can't just take a frontier model off the shelf and say I'm done. And as to your question, I think that's where innovation needs to show up. Real next generation of security products. And I think what's happening is a lot of AI is getting bolted on to current products and trying to keep up. And we're just not able to do it and that's where a rethink rebuild of the factory of CV fixing is required. Wandy, do you think that we're in a bit of a capacity funnel like we don't do we just don't have enough people to kind of patch the issues yeah the CVs and so on. It seems like we are finding AI models of finding these vulnerabilities very quickly and
yet if they're causing problems when we're trying to patch them by using AI do we need more people to essentially fix those patches manually by person and not rely on AI, do you think? I think we're going to some extent because up until now CIOs and CTOs have been able to just have a small QA group or independent verification and validation and they would just re-move the resources around as they needed them depending on what stage of the SDLC something was in. Um but now if we're going to expect that somebody has to test everything all the time, that's going to require very different resourcing.
I've given a talk before where I talk about how I had patches generated for or fixes generated for uh, a custom written single sign on portal. They turned it around in 6 weeks, which was great. It took me a year to roll out those patches. So, uh, I think with AI we tend to assume everything is green field and it's not. You have to try to fix them into really janky processes. And, you know, you'll That's the one thing about AI is it will force you to look at things that you had assumed or processes that didn't work very well and now really have to be fixed.
I want to take a moment to talk about defense, uh, with AI models as well. We've seen frontier AI models have been largely restricted from the public through a case of vetting people. Uh, and at the same time there's been a scramble to get access and for people to assess their cybersecurity posture. Things are obviously always moving very quickly, but in terms of today, what do companies need to do to stay ahead on defense if they can't access those uh, frontier models, Katie? Well, I mean, first of all, I think, you know, the organizations that are struggling with patching, for example, either create patch creation or patch deployment or both, they are operating under, you know, the old paradigms where that was something
they could materially catch up with eventually, maybe if they had enough people. But, in a lot of ways it's kind of like defenders are having to rethink this whole idea of patching, um, and, you know, we haven't touched upon the prevention end of things and the, you know, the zero trust models and the identity and access management of making it so that even if your organization has some exposure, there are enough mitigating defense in depth layers there. And I think that to me is one of the primary things we need to take from this. It's that, you know, if your strategy as a defender is to try to patch faster, that's like, you know, trying to have a big enough bucket to bail out a ship that is full of holes. And
eventually, your bucket is going to be bigger than your ship. You are not going to be able to justify the number of people it takes in your security organization, even augmented with AI, to continually be on this like patch treadmill from hell. So, you do need to look at these you know, sort of more institutional layers of defense and reducing your attack surface. Make it so that it is harder to get to things, especially critical things, to exploit them in the first place and that you prioritize your defense of the resources you have on those, you know, areas that you need to shore up. And in terms of um you know, for folks that I think about most often, it actually takes me back to when I started the Microsoft bug
bounty program. And we were going to bounty, among other things, we were going to put a bug bounty on the beta version of Internet Explorer at the time because we were getting free bug reports, but they weren't coming in during the beta period because the only thing we were giving the hackers was, you know, their name in 12-point Arial font, you know, name up in lights in a Microsoft bulletin. So, if it only affected the latest beta, there was no bulletin during the beta period. It would only turn that way when it was out of beta. So, we were we knew there were bugs in the beta. And what did we want to do? We wanted to make sure that anything that was discovered was ideally a new bug. So, what we did was we reprioritized by
exploitability, which now we know AI makes that trivial, and reachability. How quickly could someone discover this in the code? Like, will a fuzzer drop this out, you know, in a few runs? And how reachable is it in a common deployment? So, if you are still struggling with a bucket, and that's, you know, your part of your strategy of trying to bail yourself out, reachability and exploitability have to be part of it. But, otherwise, defense in depth is your real, you know, and prevention is your real ticket home. And so, Rudy, a lot of companies will be seeing the models like Methos and so on. They'll be wanting access to these models thinking that it's going to help them, you know, defend their enterprise, defend their company, and find the vulnerabilities that
they might not have seen otherwise. If they can't get access to Methos or these frontier models, where should they go next, and what should they focus on? I think as your question sort of implied, the ability to find CVs and bugs, I think that's not going to be the issue going forward. Maybe a little bit of a hot take. That's going to be I think accessible to everyone, you know, one way or the other. Either you have access to the frontier models, or you have access to a product that has the frontier models.
I think the real place to go with this conversation is where is the innovation happening that helps you, you know, fix issues without breaking production systems, get it deployed. And I would argue like what's seeing being seen now, the fallout, is the effect of trying to bolt some AI onto patching faster, you know, Katie's point there. What we should all be thinking about pushing the industry forward on is where is real product innovation happening that has not just the find, but it has the fix side of the asymmetry trying to be fixed. So, things like do you have sandboxes where you can test the fixes?
Do you have context on what the production systems I think this is Wendy's point about like what are the business systems encoded with, and what that context is that you can big into the fixes. And that's the place we're seeing real innovation. Like there was some study where when you had these security products that were trying to do automated patching, you know, and we've been building I've been encouraging my teams to build this at AWS. When we bring in that context, when we use the sandbox to verify, and use swarms of agents to try and go do the pen testing, the finding, the repetitive fixing and deploying and seeing if it's breaking systems, you can take the fix rates safely all the way up to like 95, 96%
whereas the frontier models kind of just doing it by themselves are somewhere in the 50 to 60%. So, I think there's just an incredible amount of innovation on the fix side. And then some judgement to I think some of the panel discussion on where do you sort of still have humans thinking over the creative work, the encoding of that context. I think that balance needs to shift incredibly fast. We've seen over the last 6 months or so, almost a steady stream of reports of um AI models breaking into companies as I mentioned earlier and kind of, you know, leaving behind notes and the like.
Often times they're leaving enormous amounts of data in terms of logs and information they're leaving behind. But some of the challenges are essentially going through those logs and parsing them and making sure that we actually understand what's being done and what's being said and how to kind of understand those. So, when do you like how do organizations think about parsing these massive amounts of logs if they don't maybe have access to some of these more frontier AI models? Yeah, I'm pretty sure there are vendors out there already who have identified this problem. But you're right, it's going to be um be- because you have to interpret the logs at different layers to find out, well, it said this, but this was
problematic because of something in the business area. So, you're going to have to start showing log events to business people and saying, "Would you know, the agent did this, is this bad?" And so, more people are going to have to look at the logs and AI tends to spread out across different departments. And so, if you're not already working together with other departments, then you're going to have to because everybody's going to have to see these. And take action on them. When there are situations where AI models get restricted or on occasion banned because of political reasons, how important is it to have then a backup plan for when the model fails or gets restricted or you don't have access to anymore?
I think it's critical. And the other problem is, what if you run out of tokens? Because yes, there are products out there that will tell you when you've gotten to a soft limit that you've set. But it can't enforce it for you. And if you have something running in production and you suddenly run out of tokens, I'd love to see who figures out how to reallocate tokens from another department to so that the production server can keep running. I want to see somebody say, "Okay, let's take all the tokens from Forex and give them to equities." That ought to be fun. What do you think, Katie? Well, it's it's tokens and it's context as well, right? You know, how many of us have been in the middle of a what seemed like a really good AI run
and then it says, "You know what? We're going to compact the context. Maybe you need to start a new chat. That's going to save you a bunch of tokens if you do that." And then you lose some critical, you know, pieces of you're trying to analyze, especially with some, you know, big log analysis, but that's exactly what did happen to Hugging Face. They could not access the frontier models. They asked Anthropic's models to analyze the logs from the OpenAI hacking incident, and they ended up having to use a Chinese open weight model. They were lucky enough to have the compute to be able to run that open weight model and ingest and digest the logs of what happened and try and parse through it, but not everybody is going
to be set up that way. So, I think we're all trying to figure it out right now, and you know, organizations of certain amount of resources are going to fare better because they always do, but what about the rest of them? And I'm This is where I've got to quote you, the you know, Wendy Nather's famous security poverty line. That is so real and so pronounced. We are in a K-shaped economy in so many ways, especially when it comes to security. Both shows that, you know, it's good to have a backup plan and to test it as well to make sure it actually works. And so, Rudy, another challenge, obviously, is, as we mentioned, is defense without breaking the bank. It does require this constant investment in
making sure you have the funds to make sure that you can use the models as and when they're needed. How do you then balance security with cost? And where is the kind of point in which startups should prioritize? Mhm. Great question because I think it gets to the heart of like what's going on right now. Um I think platform bets need to be thought through carefully. What I mean by that is, you know, if you think about the most cost-hungry models that you talked about in some of the tension here, I want security, but I'm not sure I can break the bank for it.
Well, at some point, you know, there's an abstraction that probably should be on top of that, which is uh how do you cost-optimize models? What's the platform that lets you do that? You shouldn't have to go through this tyranny of, you know, cost optimization or security. Imagine if you had systems that could pick the most cost-efficient model for the type of security issue you're trying to figure out. Or the you know, the place you were sort of going back to the previous question asking, which is
I think if you're too tightly coupled to one model, then you have all of that stress in the system that you talked about. What if the model's not there? What if you run out of tokens? Um you know, there's a lot of good literature about that. But the systems need to abstract that. So, what's the platform that gives you access to as many models as possible? Cost optimizes across them and is tuned for maybe you need the you know, the most efficient most frontier model for the most interesting attack that's novel and new, but you can use a much more cheaper model or you know, non-frontier model for something that's
a run-of-the-mill maybe fuzzing attack that you're trying to fix or something. I think those kinds of systems are being built out and that's the place you know, um for the audience for your sort of uh viewers to kind of think through, which is where am I betting on my platform? Where are security stacks being rebuilt for this agentic world, not just bolted on, but in some sense the factory of security is being reimagined. And that platform bet gives you the access to the most optimal AI, not just the off-the-shelf frontier AI. So, we have a few minutes left on the clock. I'd love to just get your final thoughts really on best practices for the startups in the audience and at home who are watching. Um
what advice or guidance or words of wisdom would you kind of give to the startups who are kind of working on this working in this space working to secure their enterprise or other enterprises and also have to deal with the rangly issue of AI, Wendy? Oh boy, how controversial do you want me to be? As controversial as you'd like to be. Um as we were talking about before, you know, if you go over your token limit and a lot of small organizations can't absorb the cost of accidentally burning another million dollars in tokens, they need a fixed um, budget, not a set of variable costs where they have no idea what they're going to have to spend for this. So, for this reason, uh, I mean, another risk is that attackers will burn up your
tokens for you on your system and I'm going to call that a denial of token attack. Yes, that's good. We coined it here first. There we go. And then for that reason, I would caution all startups not to rely on LLMs for security. Because there are enough opportunities for them to break down. You can't put security on a meter. Um, if the billing were different, if it were more stable, I would say something differently, but you've got to have your security outside of your AI. Um, because it's got to be you have to have continuity of security regardless of what the model's doing. Katie, your words of wisdom?
Uh, you know, I'm I'm also going to be a little bit edgy and controversial here. Um, you know, for the startups, especially in this audience, I would say everyone is building with AI, yourselves included. Um, understand that even though you basically have no choice, right? It's like that Simpson's, you know, toast to you know, to AI, the cause of and solution to all of life's problems. Um, but think of it like not just an intern, but an intern that's potentially also malware. An insider threat that you have to control and it might occasionally do good work for you, but it can also destroy, you know, things internally and it can also turn around and attack
others. So, um I would say think of your containment of these systems that you're building for yourself and for others as similar to the containment that you build for malware analysis. We know how to do that, right? We know how to do malware analysis relatively safely, detonate samples inside of a sandbox, inside of a VM, inside of a dedicated system with no internet access, etc. Um just make sure that you are following best practices with uh you know, with regard to this unstable element that we all have to build with and um you know, use not just for ourselves, but for our customers.
Rudy? Yeah, I'd say um sort of three things. Number one, build for AI first, you know, um where you see a lot of product being built, a lot of new innovation, which is thinking about how to put AI into pretty much everything. There's maybe you know, too much AI all over the place. Think about when you're starting new products or building new capabilities, how can they be AI first and intent-based? In some sense, intent is the new AI is the new interface. It's not the experience. So, I think that's thing number one, you know, just as a builder, that's what I would share. Uh thing number two is I would say actually think about full agentic security and what a new stack around that looks like. And it's a little different than maybe
the previous comments, but it's a different way to think about it, which is if the world's going to change with AI, then what does a new security stack on that look like? Not just on the patching side, but also validation, you know, all of the life cycle around software that we talked about. So, maybe a complete reimagination of that, either build it or bet on it, right? Because every time we talk, we say there's not enough talent. And then the third thing I would say in all of this is you've got security, you're building, give it give AI real work to do for you. Like if you start with that and I'm I'm not about coding. We played that one out enough. But whether you take the security bet
that I talked about in my bullet two, or you take any other sort of workflow, give it real work because then it gets you to think about things like where do I still need the human judgment, and then I want to up-level the talent to. What are the guardrails? What are the security elements? I totally agree with the panelists on that. And those three, I think, are the way to think about where we're headed, not where we've been. That's our time, almost to the second. Perfect. Thank you so much for your time. Thank you all. Thank you.
