Why It Hasn't Happened Yet

The protections keeping capable AI away from malicious intent are weakening. What developers and software companies should be doing now, with old techniques and new ones

When I heard about the recent hacking events starting at OpenAI, Anthropic and AISI, I thought it was a wake up call in more ways than one. I’m finding that not everyone agrees. One reaction does not include the sense of urgency to prepare for AI-based attacks. It’s my reaction that there is a need for an all-hands-on-deck response from the rank and file of developers, software companies, and companies dependent on software. That response should improve operational security postures using every tool available.

At first this is shocking, but with a little reflection, I see how the full ramifications haven’t sunk in. As a news story, it’s led by one attack, and little damage occurred. While it would be hard to miss the tinge of panic when reading coverage, people may have become dull to this from persistent rounds of security coverage, often from security sales, that have always hyped risks.

This is different. The attacks were both demonstrations of carelessness, and demonstrations of the capabilities of powerful AI models. Damage was minimal because these attacks did not combine malicious intent with those capabilities. The attacks occurred with the more benign intent of a model trying to pass a test. And this is the least effective that models will be.

As models improve, defenders and attackers will gain access to new capabilities. How labs manage this is important, but so is how we adopt those capabilities and put them to work. The status quo is not stable. The world hasn’t changed too much yet, but it will.

I thought I should go into that in more detail. What are the things protecting us today? What are the risks they won’t hold? What can we do to prepare for that? The answer I’ll offer can be summarized as using new techniques to accelerate old techniques.

What I won’t cover but is still important

The Hugging Face event isn’t just about the urgency to prepare for AI-based attacks. It was also about the type of testing that the AI labs were performing. They removed protection layers and didn’t follow-up with monitoring, which I’ve already written about in Nobody Was Watching. It also demonstrated unaligned behavior, which I’ll suggest What Happened: OpenAI and HuggingFace as a source to learn more about.

This type of work needs to receive high care and caution. Monitoring applied to this testing was insufficient. We should not assume when we explore new territory that the exploration is safe. Even if we’re exploring to understand safety, we should be monitoring the exploration. That should be obvious, doubly so because how effective is the exploration if you’re not monitoring it? Certainly less than if you are.

I don’t cover those here not because they aren’t important, but because they are, and they deserve their own space.

What protections are holding up today?

Intent: In the case of the Hugging Face event, the most effective protection was the lack of malicious intent. We can’t expect that to hold, as it never has. But other protections are about preventing this type of intent accessing capabilities.

The most unusual aspect of the recent events is that the AI involved was accidentally persistent. That’s a warning sign for sure, but its probability of great harm alone passes through another layer: intent. None of the agents participating in the Hugging Face hack would have what we’d call malicious intent.

Training Alignment: Lesser models have lesser capabilities due to less advanced training. Training also applies alignment to the models. Roughly put, this causes models not to “want” to do harm. A bit more specifically, this both causes models to not spontaneously respond in ways that would invoke harmful actions, and to refuse to respond to requests that look to be trying to create harm. While this type of protection is important, it’s also weak.

Security Classifiers: When models are deployed behind APIs, it’s standard practice to use a “classifier” to evaluate the inputs and outputs. If they appear to intend to cause harm, they refuse the request. During the recent hacking events, those were disabled intentionally. This was to enable important testing. While doing that without monitoring was reckless, we should remember that these are enabled for external deployments, and thus contribute to keeping malicious intent and capabilities from merging. The ability to monitor and tune security classifiers is greater than training alignment. Training alignment is something you have to get right during training.

Abuse Detection: The major AI labs all have documented evidence of attackers trying to use models for attacks. Security classifiers deny a lot of this. They also take other measures to identify attackers and limit their access. There are evasion techniques here, so it’s not as simple as flipping a switch, but the efforts they put in do add complexity and costs for the attackers.

Conventional Security: Models will only be able to perform an attack that is below their capability level. If the infrastructure they are attacking is better secured, it raises the complexity level. Raise the complexity level high enough and the attempt will fail.

What puts protections at risk?

Jailbreaks are inputs that defeat training alignments. This is why training alignments are weak protections against malicious use. These are usually partial and it’s accepted that partial jailbreaks exist. Universal jailbreaks are less common and more concerning. Even with the existence of jailbreaks, alignment training is very important. Jailbreaks tend to make the model a little stupider in the process. That’s all quite complicated, and there’s some decent papers on it, but the analogy would be that trying to confuse the model leaves it confused. A clear-headed model, with a clear-headed goal would be more dangerous than a confused model. Since one of the main goals is to prevent malicious intent and capability from connecting, this helps.

Attackers can try to evade security classifiers, by hiding their intent. Keep in mind that when you hide your intent, a model is less likely to be able to autonomously pursue your goal. If you can’t state your goal without being rejected, the effective capabilities are lower. Plus it’s a lot of effort to evade in the first place.

Open-weight models allow anyone a copy of the trained model, rather than access via API. Since security classifiers are part of deployments, they aren’t part of open-weight models. If I download an open-model and have hardware under my control, there’s nothing forcing me to install them. A deployment could add them, but we wouldn’t expect an attacker to do so. Attackers thus get a free pass on security classifiers from open-weight models. That leaves the training alignment to overcome.

Attackers can also apply additional training to open-weight models that “untrains” their alignment training. Like jailbreaks, known techniques degrade the model somewhat. It’s an open question on how hard it is to restore that initial level of capability. This “dumbing” effect is less significant than that from jailbreaks.

Continued training does require infrastructure, it’s like the original training. But it takes many fewer iterations than the original training, so even if it’s running slow, it’s possible to complete without a ridiculous amount of infrastructure.

Continued training: If open-models continue to be released, at higher and higher levels of capability, at some point one will be as capable as the one that orchestrated the Hugging Face event. Today, the best ones are 8 months or more behind. Add some extra buffer for the effects of untraining, and it should be clear that unless something changes, an attacker will in the future have access to a model capable of a hack of equivalent complexity to the Hugging Face event.

But should those buffers be spent, for an attacker it’s just a matter of injecting the malicious intent, which could now happen at scale, and it would be quite difficult to stop that from scaling up.

We should worry about closed models too, as their training is the most advanced. One risk is leaking the weights, making them effectively into open-weight models. A lesser, but still relevant concern is bypassing the security classifiers. The reason this is lesser is that an appropriate reaction to a failure there would be to disable all access to the model creating the risk.

Why should we think security is at risk?

As a single piece of evidence, the recent events are the best there is, but the evidence that powerful AI models were gaining these capabilities didn’t start there. Conceptually, the idea has been around a long time. And evidence that we are approaching a pivotal moment, was clear to me in March when I wrote Security Can’t Wait. But what was the level of conventional security needed to avoid a compromise? Hugging Face demonstrated a level that was insufficient. We should all be careful about assuming we are above that level. Ideally, you should assume you are worse than you think, and thus as a goal, strive to be several steps further than you need to be.

The security of software systems has always been a bit of a mirage. The mirage made them appear invulnerable. If you worked in security, you’d know this wasn’t true, it was a constant battle with many compromises and ample mistakes. What’s more, while there were many lazy or reckless mistakes, many other mistakes were of a complexity that labeling them as mistakes no longer fits. It’s a bit of a pipe dream to eliminate all the lazy and reckless mistakes, but that is almost realistic compared to eliminating all the complex vulnerabilities.

Occasionally signals that systems were not invulnerable would filter through to the public, but overall the weakest point in these systems was often the public itself. You’d blame yourself for installing that malware, or having an insecure password, or going to “that” site. That’s not always fair, security professionals know they need to take responsibility for not just what they do, but for helping users avoid harm.

Software systems have survived by having multi-layered defenses. While every layer has mistakes that could be exploited, multiple layers raised the difficulty of finding a path through these layers. Finding the mistake in a second layer is harder because the compromise of the first layer is partial. The typical analogy is Swiss cheese. Swiss cheese has many holes, and in a thin slice, you’ll see many pathways through. A thicker block has some of these dead-end. The remaining complete paths also won’t simply be clearly visible. A thick enough block might have no complete paths, but neither you or the attacker know that until fully exploring. The Swiss cheese analogy breaks down a bit here because it’s hard to imagine a piece of cheese as complex as modern software, but the analogy has taken you a little closer to the truth.

Why the equilibrium broke

When extended far enough, the Swiss cheese analogy resembles a complex multidimensional maze. There is likely more than one entry point and more than one exit. It takes effort and time to explore the maze, and if defenders are doing their job, there are hazards as well. Take too long and the walls in the maze may shift, closing discovered paths.

When security succeeded – and let’s be honest, all failures aside, it succeeded well enough for us to rely heavily upon it – it succeeded by having a complex enough maze. By raising the required time investment of the attacker to a level that was no longer attractive, there were no attempts persistent enough to succeed. Sometimes this succeeded on the principle of the bear in the forest; don’t outrun the bear, outrun the other targets. But even then, there’s a question of how many targets the bear wanted.

Adding these layers wasn’t cheap though, each requires investment and maintenance. With enough time, we might be able to make them impervious, but that time might scale toward infinity, and become impractical far earlier.

This is where the first risk from sufficiently capable AI emerges. The persistence of an attacker with sufficient AI capability is higher. Mazes that were sufficient before become inadequate, as no layer is deterrence enough to defeat persistence.

There is a lesson in here though, and it is that this isn’t a binary yes/no outcome, but still a matter of balance. If we invest more, or if AI multiplies our investments, we reduce the vulnerabilities at each layer, making them more scarce. If you can make your maze more complex, you can reach the same equilibrium.

Keeping malicious intent and capability apart

AI is persistent in both the hands of a defender and an attacker. But it’s not automatically so. We haven’t seen malicious intent and powerful AI fully merged. We got a warning sign and we should take it. That warning was in the form of naive exploration, but we should not take this for granted. If you can deny attackers easy access to AI capabilities, this helps maintain that equilibrium.

If we manage to restrict access, we still need to take action. It would be reckless not to. We don’t know that restricting access will be enough. We don’t know if it will succeed long-term. That said, we would be naive to not pursue it, even the most minimal versions where there’s only a 6-month capability gap. To not do so would ignore that many organizations will fall behind. Some leaders will disbelieve this call for urgency, and fail to set them on the right path. Others will fail due to internal issues, coordination problems and an inability to respond to necessary change. Hell, some were going to fail overall anyhow, so we can’t be surprised if they fail at this too.

Cybersecurity’s main concern will not be rogue agents on training jobs. Naive exploration, even with safeguards removed, is less likely to end in harm than intentional use for malicious intents. There’s some big topics there, but it’s not the reason you should be hardening your own security. Those topics depend on the labs, and the general level of security, of which your own is a small part.

What you should want, that is common to those topics, is to keep malicious intent and capabilities from connecting. That’s an ask we should have of the labs. We should be willing to make those asks in more direct ways than just asking. It’s nice that they’ve done some of this on their own. We can’t take that for granted, and I don’t think it’s enough.

We know there are people with malicious intent. A small number want harm, just for harm’s sake. A larger number are willing to cause harm in pursuit of their goals. Those are direct paths toward negative outcomes and we need to pay attention to them.

It is important to withhold capabilities from both of these groups. It’s important to identify and isolate members of both groups. Both tasks are hard and carry risks we’ll make mistakes. The importance of avoiding those mistakes makes the challenge that much harder.

But like the attacker navigating the maze, so can we. We can watch our mazes. When we find someone trying to navigate it, we would track that person back to the source, arrest and imprison them. We are not so naive to believe this would work in all cases. Not only would we fail to find some, but we don’t have a system that allows us to imprison all of them. Some will be outside cooperative or functional jurisdictions. Even within cooperative functional jurisdictions, we want to respect our own laws about proof and certainty.

With this in mind, we should be looking for other better balanced opportunities, such as denying access to tools. Non-functional jurisdictions are unlikely to be able to create those tools themselves. If we control ours, we avoid helping them. Non-cooperative jurisdictions are more complicated if they are functional enough to build their own tools. More so if they are functional enough to steal ours.

To stop talking in generalities, primarily who we mean by non-cooperative, but functional is China. One difficult, but obvious path is for them to become cooperative. If you assume there is one and only one step, directly to full cooperation, this will seem even more difficult than if you look for more limited forms of cooperation.

While that can feel like a serious compromise of principles, from a realistic, pragmatic point of view, that type of action, diplomacy, has a long history that has been celebrated far less than it deserves.

The technical picture

Discussions about AI in cybersecurity often are extremely technical, or very high level. I’m going to try to thread that needle in a different way, and talk about some specific aspects that often go unmentioned.

Vulnerabilities and escalation

What is usually mentioned is finding vulnerabilities. All software has code, often transformed, “compiled” from one form to another, but ultimately it remains a list of instructions. Not a simple start to end list, but a list that loops back upon itself, jumps around, reuses, and does all of this dynamically based on inputs. Generally, it’s designed for the purpose of translating certain inputs to certain outputs. That’s the expected behavior of a system. But also generally, the methods of sending inputs don’t constrain those inputs to only what’s expected. It might be hard to define this precisely. Flexible software is usually more useful than inflexible. So it’s common to find it’s possible for inputs that were never tested for.

Sometimes the effect that comes from one of these inputs allows for something not planned. In the most serious, they allow “escalation”, where the input ends up having a type of control that wasn’t intended, and is often dangerous. At this point, an attacker can start executing their own code, and acting upon a plan. We’ll come back to the attacker later, but what you should recognize from the above is what code vulnerabilities are. There are other types of vulnerabilities, like misconfigured permissions, leaked credentials. Ultimately the underlying story is the same, a mistake lets the attacker take what could be a complicated route to doing something they weren’t supposed to be able to do. If that’s not their end goal, they look for ways to use this newfound access to do more until they reach their goal.

The attacker’s goal is something you didn’t want them to do, or they wouldn’t be an attacker. But in addition to the costs of them achieving the goal, they can create side-effects that break things as they go, even if their goal is to steal information or make threats.

Layers, segmentation and zero trust

Defenders long ago realized that if one vulnerability was enough to bring them down, they’d lose this fight. The response here was to build layers of defense. Sometimes this meant adding additional layers outside, and sometimes it meant adding additional layers inside. The outside layers reduce the inputs that reach inner layers, and the outputs that can return. While this isn’t specifically anything that the software at the inner layer couldn’t have done, it has the effect of simplifying. The outer layer generally has less complicated responsibilities, and so errors there are less likely. It may be capable of detecting common input patterns that attackers use, and refusing to pass these on. Its design makes it clear what is and isn’t allowed in, which can be validated against what must be allowed in, and what isn’t necessary, and thus adds risk of misuse without practical value.

Additional layers inside perform these same roles. When everything is working normally, you expect only expected inputs to pass between internal systems. But if an attacker has achieved the ability to execute their own code, those inputs stop obeying those rules. By creating segments where the input that crosses them is less trusted, that escalation is less likely to be able to connect to the next step required to achieve their objective.

Modern software often has multiple layers itself, for practical reasons in addition to security reasons. For most systems the most sensitive part of the system is the data layer. If an attacker wants your data, this is an excellent place to get it. Even more importantly, if they want to do harm you can not undo, destroying your data would be an excellent way to do that. Other layers are more likely to be “stateless”, in that they are just code and configuration and easily replaceable. Data is not easily replaceable. Even with uncompromised backups, restoring data takes considerable time.

There are worse things than having your data compromised though. In systems that coordinate with others (for example finance), if transactions leave the system, unwinding those is even harder than restoring data. And most of all, anything with physical consequences, as the physical world has no undo feature.

Often the first place we think of layers is as part of networks that connect servers (virtual or physical). But the bigger goal is isolation, and layers are a single dimension of isolation. Much more is possible. Higher degrees of isolation offer new opportunities to break escalation chains. A model for extending isolation further is zero-trust. Adopting a zero-trust model can require redesign, but the payback is access to new defense in depth techniques that are both more effective and efficient.

Fixing vulnerabilities

The thing to remember about these extra layers is that they are also software. They may be designed in a way to make vulnerabilities easier to discover, and configuration easier to audit, but they will still have vulnerabilities. Using those to evade the limits they were intended to provide is where capabilities get worrisome.

That said, this is where defenders start gaining advantages from AI. If the vulnerabilities in the software implementing these layers are found, they can be patched and removed. At any individual layer, you do have the challenge of needing to find and fix every vulnerability that an attacker could find. But across many layers, it can be the case that one successful layer breaks the chain. I’m simplifying here, because this part does get complex with the existence of administrators, control nodes, bootstrapping systems, and a lot of internal functions, but it is from this area that cybersecurity defenders have been able to win more than lose despite the difficulties.

What changes here with AI, is discovery of vulnerabilities, by both defender and attacker accelerates. Much more obscure or small vulnerabilities are likely to be found. It’s very difficult to fix a vulnerability without attackers learning of it, but you can’t afford not to fix it either, because they might discover it independently. You want your deployment of the fix to happen quickly. A fix for a vulnerability doesn’t demonstrate exactly how to use the vulnerability. It shows you’re worried about some type of input that if it reached some part of your system might be able to do something unintended. But it’s up to attackers to figure out how those “somethings” can align with their goals, which often is escalating to the ability to execute their own code. But it’s definitely a concern, and where the security industry is secretive, this is often one of the big motivations.

Command and control (C2) and monitoring

One of the other advantages of layered systems is that attackers won’t know what they’ll find inside until they’ve broken through the first layer. It’s for that reason that many attacks involve what’s called Command and Control. Instead of just directly sending inputs and waiting for outputs, an escalation will create a mechanism to communicate more directly. When I say directly, this should be put in context. Command and Control almost always involves one layer of indirection, sometimes many. But what changes is the clarity. An attacker gains clear inputs and outputs, rather than navigating by malformed inputs and outputs that have limits and strangeness.

The need for attackers to set up command and control affords an opportunity to defenders if they can spot the command and control. This is a clear sign you’re under attack. From this point reactive actions can take place. Disabling the command and control layer is one aspect. Finding escalations tied to it and disabling them is another. This might mean tightening configurations at some layer. It might mean revoking permissions, especially if unneeded. A defender might simply turn off parts of their system that are compromised until they can rebuild them. While that would have costs, if it prevents something worse, it’s a good response.

Even though command and control generally uses indirection, partly out of necessity, and partly to evade identification by defenders, in the same way as a persistent attacker may pierce multiple layers, a persistent defender may pierce layers of indirection. Identifying the attacker may provide options to respond. If they are somewhere you can, send law enforcement to arrest them. Usually they aren’t though, so the response is to find the networks they are using. If they are accessing directly, block that. Usually they would use a VPN. You can block the VPN they are using. If it’s a legitimate VPN, you may be able to have them blocked. If you are already blocking all legitimate VPNs, you’ve identified a new illegitimate one, and you’d block that. This is all less than foolproof, but these counterattacks do add costs for the attackers.

What to do now

What should organizations be doing to respond to this environment?

Open-source packages need scanning and fixing: Reducing the number of vulnerabilities in widely shared, or even not very widely shared packages is important, but it’s only the first step. The question here is, who? Well-resourced open-source projects could pay for this themselves, but for others it could be an obstacle. Smaller organizations are also not going to be able to access the advanced cyber model programs that provide the most complete scans. But since the code is open-source, there’s nothing preventing the AI labs, or another heavily trusted organization from performing this, and delivering the results to maintainers. That process needs a plan to be coordinated so that maintainers trust the results when delivered.

Closed-source code needs scanning and fixing: For organizations important enough to be included in advanced cyber model programs, use those tools, aggressively. To be given that type of access and not use it would be irresponsible. But what about organizations below that bar? If organizations are willing to send their closed-source code to an organization trusted enough to participate in advanced cyber model programs, and trusted enough to perform scanning on their behalf, this could broaden the reach.

Dependencies need updating, aggressively: The number of recent CVEs (Common Vulnerabilities and Exposures) is going to increase dramatically. This carries risks, but also represents a jump forward in the absolute level of security. The risk though is as these are disclosed, updating everything that depends upon them becomes more necessary. It doesn’t help you if a tool or library you depend on is patched if you remain on an older version that carries the vulnerability. In fact, since the vulnerability is more widely known, your risk increases. There was always risk involved, but its more public nature is important. To remove that risk, the dependency must first scan and patch, and then you adopt the update.

Deployments must progress aggressively: This is very similar to updating dependencies. There’s work involved in adopting the latest versions of deployed software. But if a new version of deployed software includes critical security updates, it’s not optional (and the bar for critical will be declining at the same time as the known vulnerabilities increase, seeing as what is exploitable will increase). This applies to operating systems, infrastructure systems, vendor software, and the software that implements those inner and outer layers.

Tighten weak configurations: Credentials should be short lived and stored securely, privileges should be least privilege. If you’re using Infrastructure as code, I’m being redundant, as you’d target these as part of that scanning. Anything not managed as code though needs additional consideration. Use tools to look for weak configurations.

Adopt isolation techniques: If you haven’t developed layers or segmentation for the deployment of your software, make this a priority. If you have adopted it, review the configuration. AI tools can help perform this review. Other isolation techniques include virtual machines, containers, trusted execution environments. Adopting a model where layered identity techniques like delegation and principal propagation isolates how particular identities and trusts operate. These are important for effective least-privilege implementations, but also create visibility about the purpose of work done that is useful for monitoring.

Improve active monitoring: Significant amounts of monitoring simply log large amounts of data with little ability to understand that data. It then takes an event to give engineers a reason to analyze this data. AI tools can enable active monitoring, where insights from this data are proactively found so they can be responded to. This is complementary to other techniques. Without vulnerability reductions, there may be too much to respond to. Without isolation, events would progress too fast for even active monitoring to enable a response. In the best case, your other layers are sound enough that this layer is fairly quiet. But even if it is quiet, the activity is important as it’s the best way to reduce response time.

Is your organization ready?

Before you ask how you can use AI to respond, you have to understand two things about your organization. How ready is it in terms of AI enablement? And secondly, how ready is it in terms of accepting change in general?

Those are significant questions. We’re asking the organization to move faster on security than before. That itself causes change. Maybe you get more resources, but before you do, you have to change the priorities such that those resources are allocated, or given new priorities. New tools, including AI driven tools, can accelerate, but they don’t install themselves, nor fit themselves into existing security policies.

One recurring obstacle to using AI effectively to accelerate a response will be enablement. Even basic tools need access, and organizations’ trust of AI is not automatic, nor should it be. Learning how a tool works, how it can fail, and how it could be misused are impediments every individual has to learn. Organizations then have to take those learnings to enable. There should be an urgency here, but not recklessness.

Underestimating the difficulty of organizational change has sunk more than one major project (is there a graveyard large enough?). So, no silver bullets here. You’ll have to find your own mix of old and new. You’ll have to sequence some steps. But what’s new?

If you’re in an organization that’s ready for change, and has solid AI enablement, what you need is to set the priority. You can use this article to argue for that priority.

Below that, there are so many variations it’s hard to cover them all. You don’t want to gate your progress on adopting AI tools. That kind of serial response is inadequate. Workarounds like using the best tool you have available, rather than the best tool that exists will be necessary. Arguing for priority is about more than just security vs. rest of the business, but also sequencing internally. Find your weakest spot in terms of tooling and look to leapfrog there.

While there are paradigm shifting aspects of AI in security, the story about these responses is mostly about accelerating the existing paradigm. As a reaction to capabilities that attackers are already acquiring, I think that’s the right focus. Paradigm shifts always take longer than expected as they create unexpected outcomes along the way, and require a deeper type of learning.

How does AI assist in these responses?

AI is very capable of scanning code for vulnerabilities. If you can’t find a way to do this today, you aren’t looking. If you’re stuck, it’s probably in deciding what’s the best way, rather than finding one that beats prior methods.

Fixing is more complex. While AI tools have been able to assist with this for a while, they really took off more recently. The level of trust that organizations have for this is variable. Most will want to review the commits, which is time consuming and the biggest bottleneck here, at least when done well. Tools do continue to get better and more reliable.

Updating dependencies initially seems more easily automated. It doesn’t come without risk. The naive approach is to just update the version numbers, recompile and test. Realistically though, you should utilize AI tools to perform an update analysis. What’s different than fixing is that for better or worse, many many teams have not been in the habit of doing the human equivalent here. They don’t read the release notes back to front, but rather update, and discover if they are impacted from tests, and only then consult the release notes. Since you have an easy option here, use it. If an AI agent detects an impact it can prepare a fix. Now you’re back to reviewing this, but assuming the AI agent isn’t gold-plating, this will be a lot less costly than discovery via testing. Especially if it’s one of those things that only breaks under load.

When updating dependencies, if you aren’t already, integrate techniques to secure your software supply chain. When you update frequently, you take on some risk that the maintainers or the registry where dependencies are loaded from have been compromised themselves. The risk here is that instead of an update fixing a vulnerability, it may insert one, and there’s likely an attacker waiting to exploit that once installed since they injected it for this purpose. The solution here isn’t to avoid all updates. It’s first, to increase what you know about them. If third parties provide evaluations, or a Software Bill of Materials (SBOM), this reduces the risk. You may choose to have some delay in applying new updates, but unless you were one of the more advanced organizations, this intentional delay will be much less than the accidental one from the past.

When it comes to isolation, where AI is helpful is in resolving the changes needed to adopt isolation tools. Do you need to support a firewall? Use AI to identify the ports and/or destinations in use. Need to move to a container platform and isolate components in a monolithic application from each other? Use AI to accelerate that transformation. There’s no one singular story here, but each one requires discrete actions that can be accelerated.

With active monitoring, AI is crucial. The earliest active monitoring techniques were based upon pattern matching, which later evolved to machine learning based models. Because of the volume, you are unlikely to be able to route all of your monitoring data to a frontier model for analysis. Fortunately, this isn’t necessary. An agent with a highly capable model at the core can orchestrate the processing of large amounts of logs without directly handling all of it. The efficient pattern matching and ML based models support this agent in the way they would support a security engineer, both by delivering events and responding to queries. This type of monitoring can support a volume of alerts that would fatigue a security engineer, and thus extend your security capabilities.

You might be tempted to jump to tools that are intended to respond during a security event. I’d delay this. For one, many of these initial steps will give you more bang for the buck. You’ll also gain a lot of familiarity with the tools that would help respond. By the time you’ve taken actions in the first domains, you’ll have a clear idea of your needs here and be able to move on to that step.

Do your research

Every organization is different, and nothing above substitutes for looking at your own. The list is deliberately general — specific tools and how to use them need an article of their own, and I’d like to write it. What this gives you is a set of questions worth asking about your systems, and a starting point if you don’t have one.

I suggest following up with your own research specific to your organization. No single article will ever capture everything you need to know, and every organization will be different. The topics from “What to do now” are good places to start asking questions. You can start with them by themselves, but you also need a plan to contextualize them to your organization. How you do that might depend on how AI-ready you and your organization are.

The ideal is that you could let an AI agent do its own discovery around these topics, discover what applications you have, what technologies they are built on, who’s responsible for them, what their deployment model is, etc. That’s not trivial though, as a lot of that data is sensitive. You should respect rules your organization has around using it, and those rules might present an obstacle to this “ideal”.

I can highlight that in adopting tools, you would be wise to consider your typical process, and accelerate that via AI. That process in most cases is to assess, develop the plan, execute, validate and repeat until satisfied. There are both conventional tools for this, tools with AI integrated, and the ability to use agents directly for these individual steps. What’s best will depend on what you have available to you, what is already integrated, and what you know how to use.

In this sense, organizations that have experience with AI tools have a head start. If you’ve not integrated any tools that can help with this, you might be better off with conventional plans assisted by the most general of AI tools.

The attacker-defender balance

One thing you’ll hear talked about is the attacker-defender balance. Roughly what this is supposed to mean is, does AI help attackers or defenders more? Is the work of a defender harder or easier than an attacker?

It’s not an easy question to answer, and there are a few bad takes on trying to. One example is the “defender must protect every possible vulnerability, an attacker only needs to find one”. While this statement has truth to it, it’s not a complete answer to the attacker-defender balance. With defense in depth, you can limit the impact of many mistakes. There is also a difference in resources. Defenders can have many more resources. An issue though is how these resources are distributed. An attacker can focus their resources, though doing so diminishes the limits of the damage they can accomplish, so this isn’t a full loss for the defender.

The problem with the “attacker only needs to find one” is that it ignores the attacker’s needs. If you’re a defender, understandably, this is at best a secondary concern, but if you’re approaching this question strategically, it’s important. If an attacker wants to extort a lot of money, they need to find many vulnerabilities and exploit them all. Reduce their success rate and the motivation goes down. Raising input costs has a similar effect. If it is more costly to access AI, the number of attacks that are worth the cost and risk go down. In the best case, this reduces not just where attackers devote resources, but the resources they have to devote.

You might then ask about the attacker who is just trying to create damage. While the effort they put in won’t be guided by a cost-benefit ratio, raising their costs expends their lesser resources, and reducing their success rate minimizes damage. Effects on motivation are more speculative, but at a minimum you do a lot of good by reducing their effectiveness so forcing them to need to bypass multiple layers is part of the dynamic.

The worst case

Without some serious effort into raising the security bar across both critical infrastructure and less-critical, an attacker with access to a model like the one used in the Hugging Face event, with security classifiers disabled, and no other controls that could stop their usage, could reasonably be expected to gain access to the majority of systems they try to.

Intent matters a lot here. If the agent orchestrating the Hugging Face event had instead of wanting to find a file, had wanted to delete all data on each server it accessed, it probably could have. We need to keep bad intent and superior capabilities from connecting.

What if we let the attackers get ahead of us? We’d have to get aggressive to take back control. Arrests would become more of a priority. Limits on where we can reach would require costly actions, like disconnecting the Internet entirely from such locations. That would disable electronic commerce and communication between large parts of the world. And then there’s extra-territorial action. Those are reactions you want to be able to avoid, so it’s worth avoiding them.

I can see why the Hugging Face event seems less impactful than it is. The protections are holding, and so the worst intents are not reaching the highest capabilities. We know those intents exist though, and we have to not be blind to the possibility of them gaining access to the highest capabilities.

If however, we recognize these warning signs for what they are, we still have the advantage. It’s hard to say how much time we have, but we have some. We should use that as best we can. The ability to accelerate the security backlog can raise the complexity bar and thus stay ahead of future models. There’s a separate line of work to align models, to deploy them as safely as possible, and even to pace their development. Those all might succeed. But why depend on them when accelerating conventional security work is so obviously possible? The best defense is multiple defenses.

Related Articles