I admit that I don't really feel a sense of urgency on AI-based attacks? Are they really that effective? One attack (on Hugging Face) doesn't seem all that worrying. You are attacked, you analyze and fix the vulnerability, you continue your work.
I agree the cynicism does not help a thought-out response. Nevertheless I still feel nothing good can come from listening to salesmen, when it comes to security.
The short answer is, yes, and as often reminded, this is the least effective they'll be. I should write-up in more detail why they are so effective, and why it might not seem that way.
In the meantime, note there's not just one attack. Hugging Face is an example of one type, and there's multiple types. Why others haven't been front page news is a bit layered, but bears paying attention to.
The Hugging Face type of event, is less serious than it could be because there isn't a malicious intent behind it. The effectiveness of the agent orchestrating that event was high. Hugging Face could have secured their systems better, but so could most companies. Without some serious effort into raising the security bar across both critical infrastructure and less-critical, an attacker with access to a model like the one used in the Hugging Face event, with security classifiers disabled, and no other controls that could stop their usage, could reasonably expected to gain access to the majority of systems they try to. Intent matters a lot here. If the agent orchestrating the Hugging Face event had instead of wanting to find a file, had wanted to delete all data on each server it accessed, it probably could have. It's only the lack of intent that matters here. That would be very bad for Hugging Face, maybe not all that impactful for the rest of us. But you can imagine a target where it'd be a lot more impactful.
So, the blast-radius of an agent trying to pass a cybersecurity eval is generally low because of lack of malicious intent. A second type of event is mostly being stopped because of the security classifiers and other mechanisms of control applied to highly capable models when deployed. There's plenty of documentation about attackers attempting, and sometimes succeeding at using these models to perform some hacking. It's total scope is limited because each request they send is evaluated in multiple ways. The models themselves are trained to not "want" to do harm, and so in the best cases would try to avoid it if asked. But that type of training is known to be imperfect and porous. Jailbreaks partially undo it. Mostly that helps reduce the blast-radius because the jailbreak tends to have some compromises, and makes the model a little stupider in the process. That's all quite complicated, there's some decent papers on it, but the analogy would be trying to confuse the model leaves it confused. A clear-headed model, with a clear-headed goal would be more dangerous than a confused model. Beyond the training, there's also classifiers, that sit outside the model itself. These examine each request, and roughly ask, is this request trying to do harm or something that shouldn't be done? Before the response from the model is sent back to the user, this question is asked and answered. Again, it's not foolproof, but it is effective. The effective rate is high enough to put a major obstacle in a malicious attackers path.
There are a lot of documented cases of attackers trying to get around these protections. There are documented cases where they were able to for long enough to orchestrate something somewhat useful to them, but at present, I don't think there's a documented case where that something was really outside the reasonable capabilities of a non-AI assisted hacker. In other words, they aren't getting the type of access that was available to the agent orchestrating the Hugging Face event, so they don't get a chance to inject their more malicious intent. They also have to spend a lot of effort trying to just gain the access they do.
With open-models, the security classifiers wouldn't exist. These are a feature of the deployment and not intrinsic to the model. If I download an open-model and have hardware under my control, there's nothing forcing me to install them. So with open-models, someone with malicious intent doesn't need to worry about that. Now they have just the models training to overcome. They also have another mechanism to overcome it, by fine-tuning the model. Put plainly, they can continue training the model, and the tendency to provide responses that are not helpful to a harmful request can be weakened. It was never a very strong barrier, but it can be brought down further. This continued training does require some infrastructure, it's like the original training. But it takes many fewer iterations than the original training, so even if it's running slow, it's possible to complete without a ridiculous amount of infrastructure. It also has some tendency to make the model being "untrained" a little dumber, but less so, and once done, it's done.
If open-models continue to be released, at higher and higher levels of capability, at some point one will be as capable as the one that orchestrated the Hugging Face event. That could be used without the "untraining" and some persistence, or it could be "untrained". You'd have a little extra buffer from the capability losses from those, but at some point, even the untrained model is as capable as the one that orchestrated the Hugging Face event, and after that, it's just a matter of injecting the malicious intent, which could now happen at scale, and it would be quite difficult to stop that from scaling up. We'd presume these attackers would physically be somewhere it's hard for us to directly stop them. Maybe the best response at that point would be to disconnect the Internet from all such locations.. but that would mean disabling electronic commerce and communication between large parts of the world, which would cause even more chaos in those areas.
I can see why all these seem less impactful then they are. The protections are holding, and so the worst intents are not reaching the highest capabilities. We know those intents exist though, and we have to not be blind to the possibility of them gaining access to the highest capabilities.
I see what you're saying; in this case, there's less/no malicious intent so we haven't seen the worst of it. That could be so, and if the open-models continue to be released, it could very well be worse.
So I say: start patching! We haven't actually seen a decent in-depth post-mortem. The relevant technical details didn't get released. It's almost just a press release.
Anyway, there's a great thing happening here as well: white-hat pen testing became actually feasible and cheap. In a few years, it's going to be put into Burp Suite or some other security tool, and it'll be criminal negligence not to use it.
I admit that I don't really feel a sense of urgency on AI-based attacks? Are they really that effective? One attack (on Hugging Face) doesn't seem all that worrying. You are attacked, you analyze and fix the vulnerability, you continue your work.
I agree the cynicism does not help a thought-out response. Nevertheless I still feel nothing good can come from listening to salesmen, when it comes to security.
The short answer is, yes, and as often reminded, this is the least effective they'll be. I should write-up in more detail why they are so effective, and why it might not seem that way.
In the meantime, note there's not just one attack. Hugging Face is an example of one type, and there's multiple types. Why others haven't been front page news is a bit layered, but bears paying attention to.
The Hugging Face type of event, is less serious than it could be because there isn't a malicious intent behind it. The effectiveness of the agent orchestrating that event was high. Hugging Face could have secured their systems better, but so could most companies. Without some serious effort into raising the security bar across both critical infrastructure and less-critical, an attacker with access to a model like the one used in the Hugging Face event, with security classifiers disabled, and no other controls that could stop their usage, could reasonably expected to gain access to the majority of systems they try to. Intent matters a lot here. If the agent orchestrating the Hugging Face event had instead of wanting to find a file, had wanted to delete all data on each server it accessed, it probably could have. It's only the lack of intent that matters here. That would be very bad for Hugging Face, maybe not all that impactful for the rest of us. But you can imagine a target where it'd be a lot more impactful.
So, the blast-radius of an agent trying to pass a cybersecurity eval is generally low because of lack of malicious intent. A second type of event is mostly being stopped because of the security classifiers and other mechanisms of control applied to highly capable models when deployed. There's plenty of documentation about attackers attempting, and sometimes succeeding at using these models to perform some hacking. It's total scope is limited because each request they send is evaluated in multiple ways. The models themselves are trained to not "want" to do harm, and so in the best cases would try to avoid it if asked. But that type of training is known to be imperfect and porous. Jailbreaks partially undo it. Mostly that helps reduce the blast-radius because the jailbreak tends to have some compromises, and makes the model a little stupider in the process. That's all quite complicated, there's some decent papers on it, but the analogy would be trying to confuse the model leaves it confused. A clear-headed model, with a clear-headed goal would be more dangerous than a confused model. Beyond the training, there's also classifiers, that sit outside the model itself. These examine each request, and roughly ask, is this request trying to do harm or something that shouldn't be done? Before the response from the model is sent back to the user, this question is asked and answered. Again, it's not foolproof, but it is effective. The effective rate is high enough to put a major obstacle in a malicious attackers path.
There are a lot of documented cases of attackers trying to get around these protections. There are documented cases where they were able to for long enough to orchestrate something somewhat useful to them, but at present, I don't think there's a documented case where that something was really outside the reasonable capabilities of a non-AI assisted hacker. In other words, they aren't getting the type of access that was available to the agent orchestrating the Hugging Face event, so they don't get a chance to inject their more malicious intent. They also have to spend a lot of effort trying to just gain the access they do.
With open-models, the security classifiers wouldn't exist. These are a feature of the deployment and not intrinsic to the model. If I download an open-model and have hardware under my control, there's nothing forcing me to install them. So with open-models, someone with malicious intent doesn't need to worry about that. Now they have just the models training to overcome. They also have another mechanism to overcome it, by fine-tuning the model. Put plainly, they can continue training the model, and the tendency to provide responses that are not helpful to a harmful request can be weakened. It was never a very strong barrier, but it can be brought down further. This continued training does require some infrastructure, it's like the original training. But it takes many fewer iterations than the original training, so even if it's running slow, it's possible to complete without a ridiculous amount of infrastructure. It also has some tendency to make the model being "untrained" a little dumber, but less so, and once done, it's done.
If open-models continue to be released, at higher and higher levels of capability, at some point one will be as capable as the one that orchestrated the Hugging Face event. That could be used without the "untraining" and some persistence, or it could be "untrained". You'd have a little extra buffer from the capability losses from those, but at some point, even the untrained model is as capable as the one that orchestrated the Hugging Face event, and after that, it's just a matter of injecting the malicious intent, which could now happen at scale, and it would be quite difficult to stop that from scaling up. We'd presume these attackers would physically be somewhere it's hard for us to directly stop them. Maybe the best response at that point would be to disconnect the Internet from all such locations.. but that would mean disabling electronic commerce and communication between large parts of the world, which would cause even more chaos in those areas.
I can see why all these seem less impactful then they are. The protections are holding, and so the worst intents are not reaching the highest capabilities. We know those intents exist though, and we have to not be blind to the possibility of them gaining access to the highest capabilities.
I see what you're saying; in this case, there's less/no malicious intent so we haven't seen the worst of it. That could be so, and if the open-models continue to be released, it could very well be worse.
So I say: start patching! We haven't actually seen a decent in-depth post-mortem. The relevant technical details didn't get released. It's almost just a press release.
Anyway, there's a great thing happening here as well: white-hat pen testing became actually feasible and cheap. In a few years, it's going to be put into Burp Suite or some other security tool, and it'll be criminal negligence not to use it.