Working it out, while reading the latest technical reports on the OpenAI Hugging Face incident, led to some unexpected reflections. The ways an internal incentive program goes wrong are similar to how reward-based AI training can go wrong. Recent events provide data to check those thoughts against. The second half of this piece is where this emerges. I think both fit well, and would recommend both halves, but if you’re pressed for time with an alignment focus, skip ahead. This part looks at how monitoring feedback in models and organizations affects the degradation of the reward system.

All of this is urgent. Getting alignment right, doing research and training safely, are critical. The direct interaction there though is limited to a very small group. The rest of us are in commentary mode. The other half, bringing cybersecurity forward in what needs to be a large leap, requires much broader engagement. External commitment, like the open-letter on collective cybersecurity defense commitment is important. The real work there is still outside the average person’s bubble, but there’s a lot of companies, and a lot of developers, IT staff, managers and executives that need to support the security priority. And those all need an internal strategy to pair with the external.

Bountymaxxing

Development teams across all organizations need an adjustment to their priorities. Security has to, at least for a time, become a higher priority. The reason for this timely adjustment is the impact of AI on the realm of security. It is both offering an opportunity to improve the absolute level of security, and putting at risk the relative level of security.

A goal like this needs a mechanism. One as important as this needs multiple. Incentive systems get a bad rap. They almost always get distorted and create unintended consequences. Tokenmaxxing is not a popular term these days. As I explain in Measuring Without Breaking, it takes a very strong culture to resist this. The best we can probably expect is to slow it down.

But I think incentives–despite their flaws–are still necessary when we have an important goal. It might be good marketing to distance myself from Tokenmaxxing, but I prefer to embrace the baggage. It’s probably better if anyone adopting an incentive system to drive action on security have a clear eyed view of the flaws so they can manage them, rather than looking at this as some novel concept that can escape the long history of management challenges.

The “bounty” part comes from the history of bug bounties, where organizations have offered cash rewards for software defects, often security vulnerabilities. What I’m suggesting is a bit different, as it’s internal to companies, and would focus on resolution, not discovery. That makes it more similar to the tokenmaxxing concept with leaderboards, hence the portmanteau.

We need something at a different scale, and deeper than the current bug bounties. The extension to internal and the inclusion of resolution offer a path there. To have teeth, they need support from the highest levels. Like tokenmaxxing, money isn’t likely the main driver, but status within an organization.

Like any measurement based program, we have to think about the distortionary effects. We can never remove those entirely, but a good design can at least reduce them. Additionally, I’d hope this period is temporary, and we may hope that the value based response comes first, and the distortions emerge later. It’s often the case that incentive systems have to be temporary because they degrade as strategies for gaming them develop, disseminate, and then become endemic.

Resolution is the goal

Traditional bug bounty programs focus on discovery. This makes sense as they are external and focused on the hardest to discover issues. External developers can discover an issue, but fixing isn’t something you can assume they’d have permissions to do. For open-source projects, you can reward pull request submissions.

The challenge with preparing systems for AI driven attacks is a bit different than those covered by traditional bug bounty programs. The scale is much larger. It’s not about finding one critical flaw, but about resolving thousands of flaws, many seemingly trivial. We resolve these not because we know how they’d be exploited, but because we have a concern about how they can be pieced together to enable an attack.

We want to work on volume, and we want our north-star goal to be full deployment. Full deployment is a tough goal. It means different things in different places. With closed source software that’s distributed and installed on customer hardware, it means updating it everywhere it goes. For software deployed as part of a software as a service solution, it means updating the entire fleet. Those are very different processes. The first requires a lot of cajoling, help, and possibly new update systems. It may never be fully accomplished. The second can be started by policy and accomplished by internal tools.

Discovery is still important, but keeping the end-goal in mind is important. While a program can target intermediate outcomes, you should also expect a progression, and think through the whole chain immediately.

Normalizing Outcomes

One of the challenges in implementing a program like this is normalizing the outputs for inputs that are not normalized. If the goal is a bit of incentive and thus competition, you would prefer a level playing field. Fixes though aren’t of equal size and complexity. Consider these program design options:

1. Raw number of fixes: After establishing what a “fix” is, you find they have different values. You should worry up front about big, difficult fixes being deprioritized so that more small, easy fixes are completed.

2. Difficulty estimation: You could estimate the size of each fix, but doing this manually is time consuming.

3. AI-driven difficulty estimation: You could use AI to drive the estimate, but have to worry about organizational trust if you do this poorly. Complexity increases as more varied teams, platforms and layers are covered.

4. Importance estimation: Instead of focusing on the difficulty, focus on the value. Same challenge as before.

5. AI-driven importance estimation: And as before, and at the same place, needing to do this efficiently but raising the risk of trust. Normalizing across teams now requires describing the importance of particular teams to the organization.

6. Combined difficulty and importance: If you’ve estimated both, you can mix them. Why would you do this, given that importance is clearly the more important? To average the failures of each, if difficulty is more objective and accurate, and importance more aligned with the outcome. Also, the combination could be more resilient to successful gaming of one measure.

7. Percentage of backlog fixed each month: “If”, and it’s a big if, we can trust the discovery backlog to be complete, or equally as complete between different teams, platforms and departments, then we could measure success of the percentage fixed.

Whichever you choose, you’re choosing which imperfections to live with. That matters, because everything that follows is about what happens to those imperfections once people start pushing on them.

Minimizing Distortions

The best way to minimize distortions is to stay interactive. Watch for gaming strategies, call it out when you see it, show that respecting the intent receives better rewards than gaming the mistakes. But that should be obvious, and is something you have to do progressively as an act of management, so not very deep advice.

To make that job a little easier, some proactive design can help reduce distortions initially. Don’t expect this to remove gaming, maybe just delay it. Here’s some starting suggestions:

1. Track self-fixers. These shouldn’t count, it’s too much of a moral hazard. Do not turn defects into a punishment, you want them to remain blameless. Old issues and the backlog are fine, but any issue created after, your default should be to not count.

2. Track issue creation patterns. Are rates increasing, stable or decreasing? A decrease signals a successful “shift-left”. Stable is okay. An increase is always a warning sign, but here could be signs of gaming.

3. Pay attention to clusters. Avoid focusing on the leaves when the branch may be rotten. Do not treat the manager and team like a single point. If the entire team fails, it’s better to remove the manager and give the team a second chance. Or you could give them both a second chance, but reassigned. Or you could dig into the details.

If you find managers that responded inappropriately, consider why. It could be your fault. Always remember that the importance of alignment to goals over incentives grows as you climb in an organization, and that starts with the person at the top. Managers that accept gaming from their subordinates to inflate their own metrics are the bigger risk than the subordinates.

Credit

An incentive system isn’t an incentive system if no one expects credit. The time spent on security comes from somewhere, and if managers and individuals think they’re sacrificing their ability to accomplish other goals, without getting any credit, we can expect priorities not to change. It takes a very dedicated team to put those other things aside, and generally a team like that is only sustainable if they trust leaders will recognize work, even when not spelled out.

You should ask yourself, which type of organization are you? Not which you aspire to be–you can work on building that type of trust another day–but which are you today? If you don’t have that type of trust, you have to make it clear that this moment will be an exception and that you take this seriously.

I would avoid the temptation to formally put this in objectives. The reactions to tokenmaxxing were not positive on that one. The fallback is informal recognition, which is also somewhat risky without solid trust.

Ending

You want to be sure this ends. For one, distortions will build up the longer this runs. Those will undermine any trust you’re building or sustaining through good management. Some of the strategies for moving through stages, retiring discovery parts, and focusing more on final deployment will give you some extra time, but you have to expect to end.

It’ll be tempting to announce a date in advance. Someone is likely to ask that question at least. I’d avoid it if you can, but still make it clear it’s probably less than a year, maybe half a year. If you can’t progress in that time, you need a stronger approach, like outside help.

Ending a formal program shouldn’t end your ability to give credit for security improvement. It certainly shouldn’t be a hard cliff. Tear down any formal measurement system that’s started to diverge, but keep informal recognition alive. An ending does not indicate security is solved, or that security isn’t important, but merely that your ability to efficiently measure contributions to it has hit a limit.

Management is active. Fire and forget doesn’t cut it.

When I ran this plan by Claude, one of the tendencies it exhibited was to make the program deterministic. It recommended fire and forget decisions: a specific end date set in advance, a fixed percentage of capacity, managers directing every hour of labor.

Claude is following a long-running blind-spot in management theory here. Clean, well-defined models are often discussed. Life is always more complex. Management theory isn’t ignorant of this concept, but it often forgets it nonetheless.

This is why when you design a system like this and enact it, you don’t then stop monitoring. A measure is a correlation you trust for a while. Under pressure, what started as a clear correlation that seemed reliable becomes less reliable. Under continuous pressure, every tangential effect emerges, and comes to dominate the actual outcomes. To see this degradation happen, you have to be outside. You have to be independent, or at least, only indirectly connected.

You want something independent of that pressure, or at least, less coupled to it. Monitoring shouldn’t be wrapped up in the rewards system, or it will bend too. It doesn’t need an absolute separation, though that would be nice if such a thing existed. But a degree of separation goes a long way.

Where OpenAI wasn’t watching

If you’ve been studying the recent OpenAI Hugging Face incident you’d be interested in the technical updates. There’s a fuller technical report, and METR published an independent investigation.

There are some correlations to draw between the way rewards affect AI training, and how they affect organizations. With a little reflection this shouldn’t be surprising. While the individuals in an organization can make high-minded decisions, it’s hard to sustain, and organizations largely burned the trust necessary to have bottom-up behavior of that type.

AI models under reward based training will bend toward the reward. Earlier waves of pre-training may have built a concept like what hacking is. Other reward training may have used those concepts to steer away from hacking. But one of these rewards is clearer than the other, and continued pressure on it will bend either the concept of what hacking is, the preference for avoiding it, or both.

Both cases are reflections of the same thing: a reward’s imperfections come under pressure as it becomes the target.

One of OpenAI’s responses to the event was to require chain-of-thought (CoT) monitoring across all tool-using RL training and evaluations. That closes the gap I highlighted in Nobody Was Watching.

We now also require CoT monitoring for all tool-using RL training and evaluations involving models with GPT‑5.6 Sol capability or higher. For our Astra-class models, which may have cyber-critical⁠ capabilities, this requirement also covers all tool-enabled inference workloads.

Monitoring the CoT would give an opportunity to discover how the reward was leading to troublesome activity. And according to OpenAI it would have served that role for the Hugging Face incident.

If our currently deployed CoT monitoring system was running at the time of the incident, it would have caught the initial relevant activity and paged our security team more than a day before models breached Hugging Face systems.

But we need to be careful with the CoT. Researchers worry that it could itself be trained to hide its intent. As a scratchpad that represents only part of the “thinking”, it could in theory stop producing the warning flags while the intent persists elsewhere. We also have to worry about agents hacking the monitor and rewriting an existing CoT trace before it’s processed to eliminate the intent it indicated.

Reasons for optimism

METR and alignment researchers are reasonably worried about models becoming deceptive. This isn’t academic, as they did find agents putting effort into concealing what they were doing — escalating all the way to escaping their containers so they could spoof tool calls, appearing to run one command while running another.

Some worry about CoT monitoring causing CoT manipulation as a result. METR did not find that particular behavior in this event, with all the caveats of their investigation’s limits.

CoT manipulation: This checked for agents reasoning about manipulating their own chain of thought in order to fool the scorer; it did not turn up evidence of this, though it turned up many examples of agents reasoning about how they should not look at the HMAC reverse-engineered flag for their task to avoid being poisoned.

It’s still a reasonable concern, but I think there are grounds for optimism, and they come in two parts.

The first is low activation. Like human organizations, layers of indirection and low activation frequencies provide insulation. This does assume that it is low activation though, which itself is something that should be monitored. If monitoring is automated, built into a loop and then evaluations that trigger it at scale are allowed to become a part of a training run, this wouldn’t hold.

If you can obey this, AI models should be easier than human organizations. One challenge with indirection in human organizations is that the context of trust is endemic to an organization. Informal systems in a low trust environment are likely to be gamed, even when unnecessary, because of an assumption that the pessimistic view dominates. The consequences are assumed to be greater than they really are.

I don’t think that same dynamic is replicated when training an AI model against a reward. The reward process is closer to evolutionary effects. At low-activations there isn’t much signal left to guide evolution of deception. The concept of deception likely develops during pre-training, but its activation is dependent on reward based training.

The second is that this kind of deception is recursed, and each layer is visible from another. One level of deception is to output deceptive text, which admittedly AI models do all the time. Another level is to lie about its actions, by trying to conceal them. The level people are worried about is lying about what the model is “thinking.”

I scare-quote “thinking” because it oversimplifies. A chain-of-thought trace is like a scratchpad that a model can use. It’s also more than that. Human thinking is reliant upon a sort of internal scratchpad too. Most of us feel as if we have an internal narrator, and that experience–among the things we know today–is the best analogy to a chain-of-thought trace. But while we might naively think of this narrator as our “thoughts”, we eventually find that to be an unsatisfactory explanation of what thinking is.

Like AI models, a great deal ends up being embedded in arrangements and activations for neurons. We have little way to be conscious of most of that. For AI models, we can watch considerably more of it. Chain-of-thought is a very accessible section of the thoughts, being in human language, but accessing other parts is possible too. The best demonstration of this is Anthropic’s work on J-spaces, which surfaces what a model is considering rather than what it’s saying—including, in their examples, an intent to manipulate showing up while the model generates false values.

So an intent to deceive should show up in that J-space as part of the process of creating a deceptive chain-of-thought, in the same way a chain-of-thought normally reveals the intent behind commands to delete the log files that would expose the tool calls used for hacking.

That’s what makes the layers useful together rather than separately. Manipulating the context would show in the tool use logs. Manipulating the tool use logs would show in the context and the chain-of-thought. Deception in either should show in the J-space. There is always a way to deceive, but a deception that is coherent across all of those is not just more complex, but complex in a way that would develop one layer at a time.

And it takes time. One observation from the technical reports is that persistence was key—there was a progression, over days. If an agent ever does find a way to rewrite its tool call logs, rewrite its chain-of-thought, and disable J-space monitoring, it will have revealed the trajectory toward that before the culmination, in this case more than a day before. If you don’t monitor, it’s plausible that by the time you look, it’s done. If you do, it has to play out very differently to be missed.

Defending the indirection

A key point here is you have to avoid sending a strong signal about the effect of monitoring. The more it becomes core to the process of alignment training itself, rather than a rarely activated protection, the more that degrades. Indirection helps, as the reward signal will explore other paths before optimizing this one. But both by direct activations and by indirection of last resort, there is a signal.

I do wonder if this is a principle that OpenAI is not sufficiently aware of, that they are maybe wedded to the standard management mistake of holding on too tight. From The Zvi (quoting OpenAI):

“The Hugging Face incident underscored the need to measure agentic misalignment more rigorously in OpenAI’s most capable models wherever they are run—including intermediate or experimental checkpoints used only for internal research, training, or evaluation.

OpenAI is raising alignment standards throughout training, with clear thresholds for intervention when concerning behavior emerges.”

Yes, measurement is good, but going straight to more reliance on measurement and metrics fills me with dread. You are not going to keep outsmarting via metrics. The way this procedure gets you into trouble should be obvious.

“Many methods helping OpenAI meet these higher standards were in development well before the incident. OpenAI is now applying them more broadly, alongside new targeted remediations for each of the misalignment behaviors OpenAI observed.”

Again, based on how that is worded, I know OpenAI does not agree, but: While I realize it whacks moles, put down the mallet. The mallet is not The Way.

Indirection will not save you indefinitely. It merely buys time. It will only be enough time if you defend the indirection. Indirection results in a lower level of coupling. This forces feedback to progress through more layers, lowering its powers of feedback. They never disappear, but they are attenuated.

Allowing CoT to be manipulated would leave you as blind as you started, unless you have a backup, like the ability to monitor J-space for all training. Defending the CoT is thus very important, but having it but not monitoring it is also not good.

Watching your own organization

You won’t have access to mind reading devices. As a leader, one of your goals is to find the evidence of collusion. You might object that the collusion is automatic, and so there is no conspiracy. That however does not mean it leaves no evidence. If you’re tracking clusters and diving into some of the real details, fake activity won’t entirely hide. Automatic collusion doesn’t intentionally hide itself. The signals might be comparatively subtle compared to intentional deceptions which leave loud trails that someone would then have to work to erase.

A leader with a real interest in security would be immune to automatic collusion. If you’re such a leader, but need to delegate, have a delegate look for those subtle signals. You also want to avoid making the mistake that overreliance on CoT would risk. You want this delegate to be as independent as possible from the reward function. An easy mistake to make would be to ask them to tell you when to terminate the program, but leave them to think that an end to the program would be an end to their job. You can never make the delegate fully independent, as the organization will have a reason to subvert them. Think of regulatory capture. But a good effort is better than none.

In either case, intentional or automatic, it’s wise to take a page from the AI alignment playbook here. Monitor early. AI monitoring can be a bit less fatigable, you are only human after all, but the warning signals will often be more obvious early than later. If a middle manager asks direct reports to optimize in a “gaming” way, they will push back more when it’s novel, and less when it’s pervasive. Once it’s accepted as “that’s the way things are”, the bottom-up signals from your whistleblowers lose their volume. If it’s intentional, you should expect the ring leaders to proactively act secretly. You can’t access their chain-of-thought or private conversations, but if they make a mistake here, it’s more often early. It’s a bit interesting how often liars actually telegraph their plans in advance, before committing to them. Often that’s because they aren’t bad in an absolute sense, but choose the wrong path when they feel their options are narrow.

Conclusion

Escalating the effort that every organization puts into security for the next 12-24 months will be critical. If organizations can do this, the outcome on the other side will be more stable. Risks won’t disappear, and we have to depend on appropriate protections on deployed AI to keep the defender stably ahead. But we’re exiting a period where best-effort security was good enough, and entering one where standards must be higher. New tools give promise of meeting those standards, but they still need effort. This transition will bring some turmoil.

An incentive program is one instrument for getting through it, though a decaying one. That’s not a reason to skip it. It’s the reason active management matters more than the scoreboard. Fail to respond now, and you’ll likely start to see more and more concrete reasons you should have started earlier.

Related Articles

At first this is shocking, but with a little reflection, I see how the full ramifications haven’t sunk in. As a news story, it’s led by one attack, and little damage occurred. While it would be hard to miss the tinge of panic when reading coverage, people may have become dull to this from persistent rounds of security coverage, often from security sales, that have always hyped risks.

This is different. The attacks were both demonstrations of carelessness, and demonstrations of the capabilities of powerful AI models. Damage was minimal because these attacks did not combine malicious intent with those capabilities. The attacks occurred with the more benign intent of a model trying to pass a test. And this is the least effective that models will be.

As models improve, defenders and attackers will gain access to new capabilities. How labs manage this is important, but so is how we adopt those capabilities and put them to work. The status quo is not stable. The world hasn’t changed too much yet, but it will.

I thought I should go into that in more detail. What are the things protecting us today? What are the risks they won’t hold? What can we do to prepare for that? The answer I’ll offer can be summarized as using new techniques to accelerate old techniques.

What I won’t cover but is still important

The Hugging Face event isn’t just about the urgency to prepare for AI-based attacks. It was also about the type of testing that the AI labs were performing. They removed protection layers and didn’t follow-up with monitoring, which I’ve already written about in Nobody Was Watching. It also demonstrated unaligned behavior, which I’ll suggest What Happened: OpenAI and HuggingFace as a source to learn more about.

This type of work needs to receive high care and caution. Monitoring applied to this testing was insufficient. We should not assume when we explore new territory that the exploration is safe. Even if we’re exploring to understand safety, we should be monitoring the exploration. That should be obvious, doubly so because how effective is the exploration if you’re not monitoring it? Certainly less than if you are.

I don’t cover those here not because they aren’t important, but because they are, and they deserve their own space.

What protections are holding up today?

Intent: In the case of the Hugging Face event, the most effective protection was the lack of malicious intent. We can’t expect that to hold, as it never has. But other protections are about preventing this type of intent accessing capabilities.

The most unusual aspect of the recent events is that the AI involved was accidentally persistent. That’s a warning sign for sure, but its probability of great harm alone passes through another layer: intent. None of the agents participating in the Hugging Face hack would have what we’d call malicious intent.

Training Alignment: Lesser models have lesser capabilities due to less advanced training. Training also applies alignment to the models. Roughly put, this causes models not to “want” to do harm. A bit more specifically, this both causes models to not spontaneously respond in ways that would invoke harmful actions, and to refuse to respond to requests that look to be trying to create harm. While this type of protection is important, it’s also weak.

Security Classifiers: When models are deployed behind APIs, it’s standard practice to use a “classifier” to evaluate the inputs and outputs. If they appear to intend to cause harm, they refuse the request. During the recent hacking events, those were disabled intentionally. This was to enable important testing. While doing that without monitoring was reckless, we should remember that these are enabled for external deployments, and thus contribute to keeping malicious intent and capabilities from merging. The ability to monitor and tune security classifiers is greater than training alignment. Training alignment is something you have to get right during training.

Abuse Detection: The major AI labs all have documented evidence of attackers trying to use models for attacks. Security classifiers deny a lot of this. They also take other measures to identify attackers and limit their access. There are evasion techniques here, so it’s not as simple as flipping a switch, but the efforts they put in do add complexity and costs for the attackers.

Conventional Security: Models will only be able to perform an attack that is below their capability level. If the infrastructure they are attacking is better secured, it raises the complexity level. Raise the complexity level high enough and the attempt will fail.

What puts protections at risk?

Jailbreaks are inputs that defeat training alignments. This is why training alignments are weak protections against malicious use. These are usually partial and it’s accepted that partial jailbreaks exist. Universal jailbreaks are less common and more concerning. Even with the existence of jailbreaks, alignment training is very important. Jailbreaks tend to make the model a little stupider in the process. That’s all quite complicated, and there’s some decent papers on it, but the analogy would be that trying to confuse the model leaves it confused. A clear-headed model, with a clear-headed goal would be more dangerous than a confused model. Since one of the main goals is to prevent malicious intent and capability from connecting, this helps.

Attackers can try to evade security classifiers, by hiding their intent. Keep in mind that when you hide your intent, a model is less likely to be able to autonomously pursue your goal. If you can’t state your goal without being rejected, the effective capabilities are lower. Plus it’s a lot of effort to evade in the first place.

Open-weight models allow anyone a copy of the trained model, rather than access via API. Since security classifiers are part of deployments, they aren’t part of open-weight models. If I download an open-model and have hardware under my control, there’s nothing forcing me to install them. A deployment could add them, but we wouldn’t expect an attacker to do so. Attackers thus get a free pass on security classifiers from open-weight models. That leaves the training alignment to overcome.

Attackers can also apply additional training to open-weight models that “untrains” their alignment training. Like jailbreaks, known techniques degrade the model somewhat. It’s an open question on how hard it is to restore that initial level of capability. This “dumbing” effect is less significant than that from jailbreaks.

Continued training does require infrastructure, it’s like the original training. But it takes many fewer iterations than the original training, so even if it’s running slow, it’s possible to complete without a ridiculous amount of infrastructure.

Continued training: If open-models continue to be released, at higher and higher levels of capability, at some point one will be as capable as the one that orchestrated the Hugging Face event. Today, the best ones are 8 months or more behind. Add some extra buffer for the effects of untraining, and it should be clear that unless something changes, an attacker will in the future have access to a model capable of a hack of equivalent complexity to the Hugging Face event.

But should those buffers be spent, for an attacker it’s just a matter of injecting the malicious intent, which could now happen at scale, and it would be quite difficult to stop that from scaling up.

We should worry about closed models too, as their training is the most advanced. One risk is leaking the weights, making them effectively into open-weight models. A lesser, but still relevant concern is bypassing the security classifiers. The reason this is lesser is that an appropriate reaction to a failure there would be to disable all access to the model creating the risk.

Why should we think security is at risk?

As a single piece of evidence, the recent events are the best there is, but the evidence that powerful AI models were gaining these capabilities didn’t start there. Conceptually, the idea has been around a long time. And evidence that we are approaching a pivotal moment, was clear to me in March when I wrote Security Can’t Wait. But what was the level of conventional security needed to avoid a compromise? Hugging Face demonstrated a level that was insufficient. We should all be careful about assuming we are above that level. Ideally, you should assume you are worse than you think, and thus as a goal, strive to be several steps further than you need to be.

The security of software systems has always been a bit of a mirage. The mirage made them appear invulnerable. If you worked in security, you’d know this wasn’t true, it was a constant battle with many compromises and ample mistakes. What’s more, while there were many lazy or reckless mistakes, many other mistakes were of a complexity that labeling them as mistakes no longer fits. It’s a bit of a pipe dream to eliminate all the lazy and reckless mistakes, but that is almost realistic compared to eliminating all the complex vulnerabilities.

Occasionally signals that systems were not invulnerable would filter through to the public, but overall the weakest point in these systems was often the public itself. You’d blame yourself for installing that malware, or having an insecure password, or going to “that” site. That’s not always fair, security professionals know they need to take responsibility for not just what they do, but for helping users avoid harm.

Software systems have survived by having multi-layered defenses. While every layer has mistakes that could be exploited, multiple layers raised the difficulty of finding a path through these layers. Finding the mistake in a second layer is harder because the compromise of the first layer is partial. The typical analogy is Swiss cheese. Swiss cheese has many holes, and in a thin slice, you’ll see many pathways through. A thicker block has some of these dead-end. The remaining complete paths also won’t simply be clearly visible. A thick enough block might have no complete paths, but neither you or the attacker know that until fully exploring. The Swiss cheese analogy breaks down a bit here because it’s hard to imagine a piece of cheese as complex as modern software, but the analogy has taken you a little closer to the truth.

Why the equilibrium broke

When extended far enough, the Swiss cheese analogy resembles a complex multidimensional maze. There is likely more than one entry point and more than one exit. It takes effort and time to explore the maze, and if defenders are doing their job, there are hazards as well. Take too long and the walls in the maze may shift, closing discovered paths.

When security succeeded – and let’s be honest, all failures aside, it succeeded well enough for us to rely heavily upon it – it succeeded by having a complex enough maze. By raising the required time investment of the attacker to a level that was no longer attractive, there were no attempts persistent enough to succeed. Sometimes this succeeded on the principle of the bear in the forest; don’t outrun the bear, outrun the other targets. But even then, there’s a question of how many targets the bear wanted.

Adding these layers wasn’t cheap though, each requires investment and maintenance. With enough time, we might be able to make them impervious, but that time might scale toward infinity, and become impractical far earlier.

This is where the first risk from sufficiently capable AI emerges. The persistence of an attacker with sufficient AI capability is higher. Mazes that were sufficient before become inadequate, as no layer is deterrence enough to defeat persistence.

There is a lesson in here though, and it is that this isn’t a binary yes/no outcome, but still a matter of balance. If we invest more, or if AI multiplies our investments, we reduce the vulnerabilities at each layer, making them more scarce. If you can make your maze more complex, you can reach the same equilibrium.

Keeping malicious intent and capability apart

AI is persistent in both the hands of a defender and an attacker. But it’s not automatically so. We haven’t seen malicious intent and powerful AI fully merged. We got a warning sign and we should take it. That warning was in the form of naive exploration, but we should not take this for granted. If you can deny attackers easy access to AI capabilities, this helps maintain that equilibrium.

If we manage to restrict access, we still need to take action. It would be reckless not to. We don’t know that restricting access will be enough. We don’t know if it will succeed long-term. That said, we would be naive to not pursue it, even the most minimal versions where there’s only a 6-month capability gap. To not do so would ignore that many organizations will fall behind. Some leaders will disbelieve this call for urgency, and fail to set them on the right path. Others will fail due to internal issues, coordination problems and an inability to respond to necessary change. Hell, some were going to fail overall anyhow, so we can’t be surprised if they fail at this too.

Cybersecurity’s main concern will not be rogue agents on training jobs. Naive exploration, even with safeguards removed, is less likely to end in harm than intentional use for malicious intents. There’s some big topics there, but it’s not the reason you should be hardening your own security. Those topics depend on the labs, and the general level of security, of which your own is a small part.

What you should want, that is common to those topics, is to keep malicious intent and capabilities from connecting. That’s an ask we should have of the labs. We should be willing to make those asks in more direct ways than just asking. It’s nice that they’ve done some of this on their own. We can’t take that for granted, and I don’t think it’s enough.

We know there are people with malicious intent. A small number want harm, just for harm’s sake. A larger number are willing to cause harm in pursuit of their goals. Those are direct paths toward negative outcomes and we need to pay attention to them.

It is important to withhold capabilities from both of these groups. It’s important to identify and isolate members of both groups. Both tasks are hard and carry risks we’ll make mistakes. The importance of avoiding those mistakes makes the challenge that much harder.

But like the attacker navigating the maze, so can we. We can watch our mazes. When we find someone trying to navigate it, we would track that person back to the source, arrest and imprison them. We are not so naive to believe this would work in all cases. Not only would we fail to find some, but we don’t have a system that allows us to imprison all of them. Some will be outside cooperative or functional jurisdictions. Even within cooperative functional jurisdictions, we want to respect our own laws about proof and certainty.

With this in mind, we should be looking for other better balanced opportunities, such as denying access to tools. Non-functional jurisdictions are unlikely to be able to create those tools themselves. If we control ours, we avoid helping them. Non-cooperative jurisdictions are more complicated if they are functional enough to build their own tools. More so if they are functional enough to steal ours.

To stop talking in generalities, primarily who we mean by non-cooperative, but functional is China. One difficult, but obvious path is for them to become cooperative. If you assume there is one and only one step, directly to full cooperation, this will seem even more difficult than if you look for more limited forms of cooperation.

While that can feel like a serious compromise of principles, from a realistic, pragmatic point of view, that type of action, diplomacy, has a long history that has been celebrated far less than it deserves.

The technical picture

Discussions about AI in cybersecurity often are extremely technical, or very high level. I’m going to try to thread that needle in a different way, and talk about some specific aspects that often go unmentioned.

Vulnerabilities and escalation

What is usually mentioned is finding vulnerabilities. All software has code, often transformed, “compiled” from one form to another, but ultimately it remains a list of instructions. Not a simple start to end list, but a list that loops back upon itself, jumps around, reuses, and does all of this dynamically based on inputs. Generally, it’s designed for the purpose of translating certain inputs to certain outputs. That’s the expected behavior of a system. But also generally, the methods of sending inputs don’t constrain those inputs to only what’s expected. It might be hard to define this precisely. Flexible software is usually more useful than inflexible. So it’s common to find it’s possible for inputs that were never tested for.

Sometimes the effect that comes from one of these inputs allows for something not planned. In the most serious, they allow “escalation”, where the input ends up having a type of control that wasn’t intended, and is often dangerous. At this point, an attacker can start executing their own code, and acting upon a plan. We’ll come back to the attacker later, but what you should recognize from the above is what code vulnerabilities are. There are other types of vulnerabilities, like misconfigured permissions, leaked credentials. Ultimately the underlying story is the same, a mistake lets the attacker take what could be a complicated route to doing something they weren’t supposed to be able to do. If that’s not their end goal, they look for ways to use this newfound access to do more until they reach their goal.

The attacker’s goal is something you didn’t want them to do, or they wouldn’t be an attacker. But in addition to the costs of them achieving the goal, they can create side-effects that break things as they go, even if their goal is to steal information or make threats.

Layers, segmentation and zero trust

Defenders long ago realized that if one vulnerability was enough to bring them down, they’d lose this fight. The response here was to build layers of defense. Sometimes this meant adding additional layers outside, and sometimes it meant adding additional layers inside. The outside layers reduce the inputs that reach inner layers, and the outputs that can return. While this isn’t specifically anything that the software at the inner layer couldn’t have done, it has the effect of simplifying. The outer layer generally has less complicated responsibilities, and so errors there are less likely. It may be capable of detecting common input patterns that attackers use, and refusing to pass these on. Its design makes it clear what is and isn’t allowed in, which can be validated against what must be allowed in, and what isn’t necessary, and thus adds risk of misuse without practical value.

Additional layers inside perform these same roles. When everything is working normally, you expect only expected inputs to pass between internal systems. But if an attacker has achieved the ability to execute their own code, those inputs stop obeying those rules. By creating segments where the input that crosses them is less trusted, that escalation is less likely to be able to connect to the next step required to achieve their objective.

Modern software often has multiple layers itself, for practical reasons in addition to security reasons. For most systems the most sensitive part of the system is the data layer. If an attacker wants your data, this is an excellent place to get it. Even more importantly, if they want to do harm you can not undo, destroying your data would be an excellent way to do that. Other layers are more likely to be “stateless”, in that they are just code and configuration and easily replaceable. Data is not easily replaceable. Even with uncompromised backups, restoring data takes considerable time.

There are worse things than having your data compromised though. In systems that coordinate with others (for example finance), if transactions leave the system, unwinding those is even harder than restoring data. And most of all, anything with physical consequences, as the physical world has no undo feature.

Often the first place we think of layers is as part of networks that connect servers (virtual or physical). But the bigger goal is isolation, and layers are a single dimension of isolation. Much more is possible. Higher degrees of isolation offer new opportunities to break escalation chains. A model for extending isolation further is zero-trust. Adopting a zero-trust model can require redesign, but the payback is access to new defense in depth techniques that are both more effective and efficient.

Fixing vulnerabilities

The thing to remember about these extra layers is that they are also software. They may be designed in a way to make vulnerabilities easier to discover, and configuration easier to audit, but they will still have vulnerabilities. Using those to evade the limits they were intended to provide is where capabilities get worrisome.

That said, this is where defenders start gaining advantages from AI. If the vulnerabilities in the software implementing these layers are found, they can be patched and removed. At any individual layer, you do have the challenge of needing to find and fix every vulnerability that an attacker could find. But across many layers, it can be the case that one successful layer breaks the chain. I’m simplifying here, because this part does get complex with the existence of administrators, control nodes, bootstrapping systems, and a lot of internal functions, but it is from this area that cybersecurity defenders have been able to win more than lose despite the difficulties.

What changes here with AI, is discovery of vulnerabilities, by both defender and attacker accelerates. Much more obscure or small vulnerabilities are likely to be found. It’s very difficult to fix a vulnerability without attackers learning of it, but you can’t afford not to fix it either, because they might discover it independently. You want your deployment of the fix to happen quickly. A fix for a vulnerability doesn’t demonstrate exactly how to use the vulnerability. It shows you’re worried about some type of input that if it reached some part of your system might be able to do something unintended. But it’s up to attackers to figure out how those “somethings” can align with their goals, which often is escalating to the ability to execute their own code. But it’s definitely a concern, and where the security industry is secretive, this is often one of the big motivations.

Command and control (C2) and monitoring

One of the other advantages of layered systems is that attackers won’t know what they’ll find inside until they’ve broken through the first layer. It’s for that reason that many attacks involve what’s called Command and Control. Instead of just directly sending inputs and waiting for outputs, an escalation will create a mechanism to communicate more directly. When I say directly, this should be put in context. Command and Control almost always involves one layer of indirection, sometimes many. But what changes is the clarity. An attacker gains clear inputs and outputs, rather than navigating by malformed inputs and outputs that have limits and strangeness.

The need for attackers to set up command and control affords an opportunity to defenders if they can spot the command and control. This is a clear sign you’re under attack. From this point reactive actions can take place. Disabling the command and control layer is one aspect. Finding escalations tied to it and disabling them is another. This might mean tightening configurations at some layer. It might mean revoking permissions, especially if unneeded. A defender might simply turn off parts of their system that are compromised until they can rebuild them. While that would have costs, if it prevents something worse, it’s a good response.

Even though command and control generally uses indirection, partly out of necessity, and partly to evade identification by defenders, in the same way as a persistent attacker may pierce multiple layers, a persistent defender may pierce layers of indirection. Identifying the attacker may provide options to respond. If they are somewhere you can, send law enforcement to arrest them. Usually they aren’t though, so the response is to find the networks they are using. If they are accessing directly, block that. Usually they would use a VPN. You can block the VPN they are using. If it’s a legitimate VPN, you may be able to have them blocked. If you are already blocking all legitimate VPNs, you’ve identified a new illegitimate one, and you’d block that. This is all less than foolproof, but these counterattacks do add costs for the attackers.

What to do now

What should organizations be doing to respond to this environment?

Open-source packages need scanning and fixing: Reducing the number of vulnerabilities in widely shared, or even not very widely shared packages is important, but it’s only the first step. The question here is, who? Well-resourced open-source projects could pay for this themselves, but for others it could be an obstacle. Smaller organizations are also not going to be able to access the advanced cyber model programs that provide the most complete scans. But since the code is open-source, there’s nothing preventing the AI labs, or another heavily trusted organization from performing this, and delivering the results to maintainers. That process needs a plan to be coordinated so that maintainers trust the results when delivered.

Closed-source code needs scanning and fixing: For organizations important enough to be included in advanced cyber model programs, use those tools, aggressively. To be given that type of access and not use it would be irresponsible. But what about organizations below that bar? If organizations are willing to send their closed-source code to an organization trusted enough to participate in advanced cyber model programs, and trusted enough to perform scanning on their behalf, this could broaden the reach.

Dependencies need updating, aggressively: The number of recent CVEs (Common Vulnerabilities and Exposures) is going to increase dramatically. This carries risks, but also represents a jump forward in the absolute level of security. The risk though is as these are disclosed, updating everything that depends upon them becomes more necessary. It doesn’t help you if a tool or library you depend on is patched if you remain on an older version that carries the vulnerability. In fact, since the vulnerability is more widely known, your risk increases. There was always risk involved, but its more public nature is important. To remove that risk, the dependency must first scan and patch, and then you adopt the update.

Deployments must progress aggressively: This is very similar to updating dependencies. There’s work involved in adopting the latest versions of deployed software. But if a new version of deployed software includes critical security updates, it’s not optional (and the bar for critical will be declining at the same time as the known vulnerabilities increase, seeing as what is exploitable will increase). This applies to operating systems, infrastructure systems, vendor software, and the software that implements those inner and outer layers.

Tighten weak configurations: Credentials should be short lived and stored securely, privileges should be least privilege. If you’re using Infrastructure as code, I’m being redundant, as you’d target these as part of that scanning. Anything not managed as code though needs additional consideration. Use tools to look for weak configurations.

Adopt isolation techniques: If you haven’t developed layers or segmentation for the deployment of your software, make this a priority. If you have adopted it, review the configuration. AI tools can help perform this review. Other isolation techniques include virtual machines, containers, trusted execution environments. Adopting a model where layered identity techniques like delegation and principal propagation isolates how particular identities and trusts operate. These are important for effective least-privilege implementations, but also create visibility about the purpose of work done that is useful for monitoring.

Improve active monitoring: Significant amounts of monitoring simply log large amounts of data with little ability to understand that data. It then takes an event to give engineers a reason to analyze this data. AI tools can enable active monitoring, where insights from this data are proactively found so they can be responded to. This is complementary to other techniques. Without vulnerability reductions, there may be too much to respond to. Without isolation, events would progress too fast for even active monitoring to enable a response. In the best case, your other layers are sound enough that this layer is fairly quiet. But even if it is quiet, the activity is important as it’s the best way to reduce response time.

Is your organization ready?

Before you ask how you can use AI to respond, you have to understand two things about your organization. How ready is it in terms of AI enablement? And secondly, how ready is it in terms of accepting change in general?

Those are significant questions. We’re asking the organization to move faster on security than before. That itself causes change. Maybe you get more resources, but before you do, you have to change the priorities such that those resources are allocated, or given new priorities. New tools, including AI driven tools, can accelerate, but they don’t install themselves, nor fit themselves into existing security policies.

One recurring obstacle to using AI effectively to accelerate a response will be enablement. Even basic tools need access, and organizations’ trust of AI is not automatic, nor should it be. Learning how a tool works, how it can fail, and how it could be misused are impediments every individual has to learn. Organizations then have to take those learnings to enable. There should be an urgency here, but not recklessness.

Underestimating the difficulty of organizational change has sunk more than one major project (is there a graveyard large enough?). So, no silver bullets here. You’ll have to find your own mix of old and new. You’ll have to sequence some steps. But what’s new?

If you’re in an organization that’s ready for change, and has solid AI enablement, what you need is to set the priority. You can use this article to argue for that priority.

Below that, there are so many variations it’s hard to cover them all. You don’t want to gate your progress on adopting AI tools. That kind of serial response is inadequate. Workarounds like using the best tool you have available, rather than the best tool that exists will be necessary. Arguing for priority is about more than just security vs. rest of the business, but also sequencing internally. Find your weakest spot in terms of tooling and look to leapfrog there.

While there are paradigm shifting aspects of AI in security, the story about these responses is mostly about accelerating the existing paradigm. As a reaction to capabilities that attackers are already acquiring, I think that’s the right focus. Paradigm shifts always take longer than expected as they create unexpected outcomes along the way, and require a deeper type of learning.

How does AI assist in these responses?

AI is very capable of scanning code for vulnerabilities. If you can’t find a way to do this today, you aren’t looking. If you’re stuck, it’s probably in deciding what’s the best way, rather than finding one that beats prior methods.

Fixing is more complex. While AI tools have been able to assist with this for a while, they really took off more recently. The level of trust that organizations have for this is variable. Most will want to review the commits, which is time consuming and the biggest bottleneck here, at least when done well. Tools do continue to get better and more reliable.

Updating dependencies initially seems more easily automated. It doesn’t come without risk. The naive approach is to just update the version numbers, recompile and test. Realistically though, you should utilize AI tools to perform an update analysis. What’s different than fixing is that for better or worse, many many teams have not been in the habit of doing the human equivalent here. They don’t read the release notes back to front, but rather update, and discover if they are impacted from tests, and only then consult the release notes. Since you have an easy option here, use it. If an AI agent detects an impact it can prepare a fix. Now you’re back to reviewing this, but assuming the AI agent isn’t gold-plating, this will be a lot less costly than discovery via testing. Especially if it’s one of those things that only breaks under load.

When updating dependencies, if you aren’t already, integrate techniques to secure your software supply chain. When you update frequently, you take on some risk that the maintainers or the registry where dependencies are loaded from have been compromised themselves. The risk here is that instead of an update fixing a vulnerability, it may insert one, and there’s likely an attacker waiting to exploit that once installed since they injected it for this purpose. The solution here isn’t to avoid all updates. It’s first, to increase what you know about them. If third parties provide evaluations, or a Software Bill of Materials (SBOM), this reduces the risk. You may choose to have some delay in applying new updates, but unless you were one of the more advanced organizations, this intentional delay will be much less than the accidental one from the past.

When it comes to isolation, where AI is helpful is in resolving the changes needed to adopt isolation tools. Do you need to support a firewall? Use AI to identify the ports and/or destinations in use. Need to move to a container platform and isolate components in a monolithic application from each other? Use AI to accelerate that transformation. There’s no one singular story here, but each one requires discrete actions that can be accelerated.

With active monitoring, AI is crucial. The earliest active monitoring techniques were based upon pattern matching, which later evolved to machine learning based models. Because of the volume, you are unlikely to be able to route all of your monitoring data to a frontier model for analysis. Fortunately, this isn’t necessary. An agent with a highly capable model at the core can orchestrate the processing of large amounts of logs without directly handling all of it. The efficient pattern matching and ML based models support this agent in the way they would support a security engineer, both by delivering events and responding to queries. This type of monitoring can support a volume of alerts that would fatigue a security engineer, and thus extend your security capabilities.

You might be tempted to jump to tools that are intended to respond during a security event. I’d delay this. For one, many of these initial steps will give you more bang for the buck. You’ll also gain a lot of familiarity with the tools that would help respond. By the time you’ve taken actions in the first domains, you’ll have a clear idea of your needs here and be able to move on to that step.

Do your research

Every organization is different, and nothing above substitutes for looking at your own. The list is deliberately general — specific tools and how to use them need an article of their own, and I’d like to write it. What this gives you is a set of questions worth asking about your systems, and a starting point if you don’t have one.

I suggest following up with your own research specific to your organization. No single article will ever capture everything you need to know, and every organization will be different. The topics from “What to do now” are good places to start asking questions. You can start with them by themselves, but you also need a plan to contextualize them to your organization. How you do that might depend on how AI-ready you and your organization are.

The ideal is that you could let an AI agent do its own discovery around these topics, discover what applications you have, what technologies they are built on, who’s responsible for them, what their deployment model is, etc. That’s not trivial though, as a lot of that data is sensitive. You should respect rules your organization has around using it, and those rules might present an obstacle to this “ideal”.

I can highlight that in adopting tools, you would be wise to consider your typical process, and accelerate that via AI. That process in most cases is to assess, develop the plan, execute, validate and repeat until satisfied. There are both conventional tools for this, tools with AI integrated, and the ability to use agents directly for these individual steps. What’s best will depend on what you have available to you, what is already integrated, and what you know how to use.

In this sense, organizations that have experience with AI tools have a head start. If you’ve not integrated any tools that can help with this, you might be better off with conventional plans assisted by the most general of AI tools.

The attacker-defender balance

One thing you’ll hear talked about is the attacker-defender balance. Roughly what this is supposed to mean is, does AI help attackers or defenders more? Is the work of a defender harder or easier than an attacker?

It’s not an easy question to answer, and there are a few bad takes on trying to. One example is the “defender must protect every possible vulnerability, an attacker only needs to find one”. While this statement has truth to it, it’s not a complete answer to the attacker-defender balance. With defense in depth, you can limit the impact of many mistakes. There is also a difference in resources. Defenders can have many more resources. An issue though is how these resources are distributed. An attacker can focus their resources, though doing so diminishes the limits of the damage they can accomplish, so this isn’t a full loss for the defender.

The problem with the “attacker only needs to find one” is that it ignores the attacker’s needs. If you’re a defender, understandably, this is at best a secondary concern, but if you’re approaching this question strategically, it’s important. If an attacker wants to extort a lot of money, they need to find many vulnerabilities and exploit them all. Reduce their success rate and the motivation goes down. Raising input costs has a similar effect. If it is more costly to access AI, the number of attacks that are worth the cost and risk go down. In the best case, this reduces not just where attackers devote resources, but the resources they have to devote.

You might then ask about the attacker who is just trying to create damage. While the effort they put in won’t be guided by a cost-benefit ratio, raising their costs expends their lesser resources, and reducing their success rate minimizes damage. Effects on motivation are more speculative, but at a minimum you do a lot of good by reducing their effectiveness so forcing them to need to bypass multiple layers is part of the dynamic.

The worst case

Without some serious effort into raising the security bar across both critical infrastructure and less-critical, an attacker with access to a model like the one used in the Hugging Face event, with security classifiers disabled, and no other controls that could stop their usage, could reasonably be expected to gain access to the majority of systems they try to.

Intent matters a lot here. If the agent orchestrating the Hugging Face event had instead of wanting to find a file, had wanted to delete all data on each server it accessed, it probably could have. We need to keep bad intent and superior capabilities from connecting.

What if we let the attackers get ahead of us? We’d have to get aggressive to take back control. Arrests would become more of a priority. Limits on where we can reach would require costly actions, like disconnecting the Internet entirely from such locations. That would disable electronic commerce and communication between large parts of the world. And then there’s extra-territorial action. Those are reactions you want to be able to avoid, so it’s worth avoiding them.

I can see why the Hugging Face event seems less impactful than it is. The protections are holding, and so the worst intents are not reaching the highest capabilities. We know those intents exist though, and we have to not be blind to the possibility of them gaining access to the highest capabilities.

If however, we recognize these warning signs for what they are, we still have the advantage. It’s hard to say how much time we have, but we have some. We should use that as best we can. The ability to accelerate the security backlog can raise the complexity bar and thus stay ahead of future models. There’s a separate line of work to align models, to deploy them as safely as possible, and even to pace their development. Those all might succeed. But why depend on them when accelerating conventional security work is so obviously possible? The best defense is multiple defenses.

Related Articles

Last week I wrote An OpenAI Model Escaped Its Sandbox. Where Was the Observer?. This weekend we learned that the practice of disabling observational monitors was true at Anthropic too.

Several defense-in-depth measures, on both our side and our partner’s, could have prevented these incidents, or at least reduced their likelihood of occurring. Careful validation of all internet access paths before evaluations began and real-time monitoring of the evaluation logs would have helped to surface the problem sooner. Both we and our partner also could have reviewed evaluation transcripts or network logs more thoroughly. It’s also possible that a prompt which told Claude it did have internet access would have changed how Claude behaved when it came into contact with real systems.

_{Source: Investigating three real-world incidents in our cybersecurity evaluations
Anthropic, July 30th 2026}

Anthropic commits to doing better. Specific to this aspect:

First, evaluation environments that involve powerful autonomous capabilities also require significant controls. Safety testing happens before a model is released precisely because we don’t yet know what it is capable of. Evaluation environments increasingly need to be held to the same security standard as any other system our models run in.

The shared failures of what seems from the outside like a basic failure in design, makes you wonder about the overall corner cutting. While this is not entirely parallel to the concerns mentioned in the letter from Frontier Lab employees, Pacing the Frontier, it’s hard to see it as unrelated.

AI could help create a dramatically better future, but that outcome is not guaranteed. The world’s leading AI companies believe they could be close to automating AI research. It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.

To realize AI’s potential, industry, government, and society at large may need the option to buy time to address emerging risks, develop security measures, and strengthen oversight. But each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration. And today, the world lacks the technical and governance tools to deliberately pace frontier-wide progress.

Building on work already underway to monitor frontier model releases:

“We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”

_{Source: Pacing the Frontier
1,346 employees of frontier AI companies, July 2026}

It’s a bit of a long-shot here, given the state of international relations right now. But if you don’t try, what else are you going to do?

A Nuanced Position, Poorly Received

The internet is wrong about Amodei’s stance on open-weight models. I’ll call out some excerpts, but please read the whole thing if you worry these are out of context.

… Anthropic has never advocated for a ban on open-weights models.
… Protectionist bans would not address my most serious national security concerns.
… Open-weights models—it does not matter whether they come from China or anywhere else—do potentially present a higher risk than closed models, because it is very difficult to apply guardrails to them or monitor their usage, and once weights are released they cannot be withdrawn.
… All sufficiently capable models, open and closed, should go through mandatory safety testing. The best way to address threat #2 is to just directly test models for cyber, biological, and alignment risks before release.

_{Source: Our position on open-weights models
Anthropic, July 27th 2026}

It’s unfortunate that the nuance of the difference of opinion here is so poorly understood. Amodei is absolutely right that releasing models as open-weight carries risks. Even the letter supporting open-weights acknowledges this.

To be sure, open weights carry real and distinct risks. Once released, the weights are beyond the original developer’s control, and modified versions are difficult to trace or reverse. But the right response to this risk is not to prohibit open weights. In a world where cybersecurity attackers use advanced AI, defenders need access to models with comparable capabilities so they can detect, simulate, and respond to emerging threats.

_{Source: Open Weights and American AI Leadership
270 signatories, July 24th 2026}

Should we ignore those risks, because open-weight models provide risk reduction that balances out? That argument fails to support itself, and Amodei pushes back on that. Defenders can access models that aren’t open-weights. Closed models do not deny defenders access, but ensures their access is more advanced than the attackers. Defenders and attackers should not have comparable models. Defenders should have better ones. If you want to be ahead, you do have to prepare, which Hugging Face did not.

The internet misses that point, and reacts by vibing cynicism. The leading response on HackerNews provides a clear example of a poorly thought out, poorly argued response that is fully based on cynicism, yet very popular.

Schrödinger’s China at once is an evil entity looking to use AI for their own nefarious purposes yet also willing to cooperate with their main competitor to prevent other actors (who??) from achieving similar goals (all while under a chip embargo too!!)

The reality is much less confusing: Anthropic CEO does not wish for models with similar (or greater) capabilities compared to his own closed and overpriced ones to be widely released. Simply because that will affect Anthropic’s bottom-line.

Anthropic and all other “model” companies have nothing making them special beyond privileged access to chips so obviously they want to restrict what models are out there and more importantly who can produce new ones. Without these restrictions, it’s only a matter of time before the multi-hundred billions valuations simply evaporate while they are still holding the bag.

No matter what your views on Amodei, or any other participant, a question as important as this should be answered by reasoning. It’s worrying that so many people thought this was the best response despite no consideration of the actual effects of open-weight models.

The Risk of Open Weight Models

The risk of open-weight models is the lack of control. Once released anyone with sufficient compute can use them and it doesn’t take much compute to engage in malicious behavior. Restricting their capabilities, or the purpose of their use is not possible. The alternative is to be deliberate about how and where models are deployed. Models can never be safe on their own. Rigorous responsible deployment is necessary to turn away malicious use.

Deployment itself may be distributed, a middle between centralization and full openness. A plethora of models could be available in this way. Deployers must be trusted, so their numbers will be limited. But you can validate enough to establish strong competition, and avoid creating an exploitable moat. The cynics believe Amodei is advocating for a moat. But what moat it creates would go to the trusted deployers – the compute providers – who have signed the open-weights letter (Amazon, Google, Microsoft, NVIDIA, CoreWeave, Crusoe, Nebius and Together). Validation creates a minor moat, but it can be minimized via sufficient competition.

The best “pro” open-weights argument is that the pathway to not having models with weights freely distributed isn’t clear. That’s not a very good argument for, but it is a conundrum that would remain even after the core argument is resolved. China-America agreements are hard to come by and even harder to maintain.

If however, you did manage that, it would change the day-to-day experience of very few. Users of Deepseek, GLM, Qwen and Kimi mostly do not self-host. Accessing them via a cloud provider, neo or otherwise, would be unchanged.

Cynicism as a Tool, Not a Verdict

Aside from this very important topic, the topic of reasoning via cynicism is one we need to confront too. Cynicism is not without its place, but it should be used as a tool to open a line of deeper reasoning, not to jump to conclusions that divert from practicing reasoning. When we start and end our reasoning with cynicism alone, we lose any hope of trust and goodwill.

In the case of open-weights, another cynical view would argue that the AI labs would be pro open-weights, knowing that they would create a security escalation that only deeper use of AI can resolve. If open-weight models become advanced enough to conduct wide-spread cyber activities, there is no option to go back. You can only fail-forward. And failing-forward here means a massive all-hands on deck dive on security across every entity dependent on software.

I’ve already argued we should be elevating our priorities, and acting with urgency. Unfortunately we’re mostly not responding. Most organizations are still more concerned with token optimization, the next feature set, or optimising their marketing pipeline.

There’s a real chance, which becomes much larger if open-weight models continue to be released, that this lack of urgency transitions into outright panic after one or two events that demonstrate the lack of preparedness that is pervasive. Panicking users will not hold back on spending, and as AI models will be key to any response, you could cynically predict a windfall being captured by AI labs in such an event.

While that’s a coherent argument, it doesn’t prove the accusation, anymore than the preceding wave of cynicism would. Embedded in there though is the reason to restrict open-weight models, and that argument, not the one based on motivations, is the one we should pay the most attention to. In addition, many companies should rebalance their investments toward security. We must both enable and incentivize immediate action on security. Getting tied up in cynicism about others takes away our own initiative.

We should be bountymaxxing. To the degree that we set objectives and deliver incentives to development organizations; developers, managers and executives, these should be aligned with discovering and resolving as many vulnerabilities as possible. Like “tokenmaxxing”, it would be a messy process, but I can’t see another method to pivot the unwieldy organizations that have to do this work. It will be costly, but it will save too.

A set of contrasts

The Pacing the Frontier letter and the Open Weights and American AI Leadership letter contrast in interesting ways as events of a single week. It’s obvious that one is towards caution, and the other is suggesting caution is too expensive. I can deepen that by looking at Pacing the Frontier as asking for help in preventing caution from being too expensive.

One of those cynical responses to Pacing the Frontier has been to suggest frontier lab employees should quit their jobs, and that anything else shows they are insufficiently serious. This insistence that support for enforceable agreement should be preceded by unilateral action is one of the oldest in the book. It’s not correct, and the most simplistic reasoning makes that clear. It’s also a trap. Let me analogize to my time supporting climate action. The same standard applied there. If anyone supporting climate action took an international flight, owned a car, or wasn’t vegan, obviously they didn’t take their own arguments seriously. That’s the poor reasoning part. The trap part was, if they did do all that, they were looney, and also not worth taking seriously.

Another contrast is the Open Weights letter presumes a balancing force it fails to explain, whereas Pacing the Frontier presumes the need to add a balancing force, because the pressures of competition are too high.

What is alike between the two is that they are both detail light, and in an immediate sense, unactionable in either direction. If we want all powerful models to only be deployed in safe and responsible ways, we need a type of cooperation that does not exist. If we want a deliberate pace that would also support safe and responsible actions, we need a type of coordination that does not exist.

There aren’t quick-fix actions here. Banning US based deployment of Chinese open-weight models would not reduce their ability to be deployed irresponsibly or used maliciously. The impactful parts are outside that jurisdiction. Unilaterally slowing development risks losing the modicum of control we have today. Resigning a job at Anthropic or OpenAI similarly does not fix anything. These would be distraction actions. They would change little, serve the purpose of looking like action, and ultimately cede what little control exists. Calls for these actions are either misinformed or malicious.

There are three significant aspects I’d highlight. The first is a demonstration of offensive cybersecurity capabilities of current models. That shouldn’t be too shocking if you’ve been paying attention, but the event demonstrates it in a way that’s more clear than prior signals, so likely awareness is going to grow from this event.

The second is model alignment. Alignment refers to the degree to which AI models do what we want them to. I’ll spare a great dive here as defining what we want itself would be a long dissertation where a great deal of content already exists. But for this case, we can generally agree that it was no one’s intention that the model hack itself out of multiple protection layers to reach the internet, and then from there into Hugging Face’s systems.

There are a few things that make this a bit less than a sign to retreat to the bunker, which I should mention before going deeper. One is that the models being tested had protections turned off, to allow them to test their cybersecurity capabilities. In that sense, this isn’t fully demonstrable of what a user could intentionally or accidentally trigger from usage. This also is an unreleased model, so not available to general users, or really anyone other than OpenAI.

Where was the Observer?

That said, a third point is, and the one I’d like to dive deeper on, I’m worried about the ability for this to happen. It seems to me as if there is a reasonable protection, an observer model, that OpenAI either didn’t use, or failed at. Right now, I’m assuming they didn’t use it because I can’t find any mention of it, but I’m also aware there are many details I’m not privy to.

In case it’s not clear what I’m suggesting by observer model, here’s an example scenario in action. The model under evaluation decides it should escape the sandbox. It starts to attempt this, which does not get caught at a per action level because the classifier safeguards – which would have disrupted the evaluation – are disabled. But then it does escape the sandbox. An observer model watching the evaluation run should be able to look at that state, and say, “This should not have happened. I don’t know how this model is outside its sandbox, but it shouldn’t be outside, we should shut this down.”

If a model is in evaluation for cybersecurity purposes, and you’ve disabled classifiers, you should watch it closely. To carry out these actions, the model had to take several steps, achieving results at each step and then moving on to exploiting the rewards of those steps further to explore, and escalate again. OpenAI’s security team did eventually respond, so they weren’t entirely asleep. But the timing shows they were slower than Hugging Face. They had enough time to try commercial models, which refused to help, then next plan a response with GLM 5.2, and then successfully contain and stop the intrusion.

OpenAI should have had several structural advantages here. First, unaligned activity occurred before any activity was directed at Hugging Face. Second, OpenAI should already have a kill switch for evaluations. There should be no need to devise novel plans to contain and stop activity. The fact that Hugging Face was the actor that stopped the intrusion, rather than taking note of the intrusion stopping mysteriously on its own, indicates OpenAI was the slower responder despite the structural advantage.

An observer model is a little different than a classifier. Classifiers typically evaluate individual actions. An observer would look at trajectories. OpenAI discusses similar new safeguards (“From considering single actions to considering whole trajectories”) in a recent post. If you look at each individual action, it might look like the type of thing a model would need to do in cybersecurity evaluation. But as a whole, some outside source should have been able to reason upon what was occurring. If not a human involved in the evaluation process, an observer model.

I understand why it may have taken a while to deploy this type of safeguard in production, where volumes are higher, where models have already been evaluated, and where classifiers are restrictive. In production, observer models are less effective because malicious users can “distribute” their requests into multiple sessions, and so observing a trajectory first requires coalescing these threads together, which Anthropic has done. A safety mechanism for a cybersecurity evaluation sandbox, does not encounter that challenge. Monitoring your own model in a sandbox is significantly less challenging, especially if it’s not attempting to evade detection. There’s no evidence here that there were attempts to evade detection, just attempts to perform inappropriate actions.

So, is trajectory monitoring by an observer model not standard practice in an offensive cybersecurity evaluation sandbox? Certainly you’d do this during internal evaluation of models that you clearly do not know the maximum capabilities of? If it is and the model evaded detection, that would be an even more significant story. I have to assume that instead there was simply no observer model to evade. Or if, as looks to be the case, it didn’t attempt evasion, it would have escaped the sandbox, been detected, and promptly been shut down before the Hugging Face part of the action had begun.

Why is that not the story we’re reading? But instead we’re reading about Hugging Face contacting law enforcement, scrambling to get a model to counter and analyze and OpenAI talking about the flaw in the proxy?

This is the equivalent of putting a tiger in a cage, and then checking that the tiger stays in the cage. It’s not a particularly rocket science design. Tuning it might take effort, and hardening against evasion more, but it doesn’t seem like we’ve progressed to those requirements. Rather it looks like OpenAI lacked the most basic version. Am I missing something?

Conclusion

Others, such as The Zvi, have pointed out that the core, the most critical fix is in the alignment training.

The intent is the issue. Control strategies and supervision are good parts of a defense-in-depth strategy, we should totally use such strategies. That helps mitigate failure. But that strategy also has to include actually aligning the models, or you lose. And by lose, in the long term, I mean things up to and likely including loss of control over the future and everyone dying.

The model should internally understand these actions aren’t aligned with its mission, and thus not want to take them. I’d fully agree there. I bring up this specific defense in depth as an addition to core corrections. Those corrections are the hardest part, defense in depth is an easier, but incomplete part. Testing alignment is hard, and protections around that testing are necessary, so this isn’t irrelevant to the alignment conversation either.

OpenAI’s initial response has been about a few defense in depth responses, but they are very specific, non-generalizable fixes to the sandbox at a software level, fixing the flaw that the model exploited. That might have avoided this incident, or the model might have found a different flaw in a different component of the sandbox.

Something is very wrong about this part of the story about other layers of defense that should exist when performing this type of work. If they exist but failed, that’s a big story. If they exist but were disabled, that’s a big story. If no one thought to use them, that’s a big story. What’s the real story?

It’s not good enough to say “These deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities.” as there isn’t a justification why an observer model would interfere with evaluation. Maybe there was such a model but it would have stopped the evaluation? But clearly you can replace an observer that would stop a valid evaluation with one that would stop actions outside the requirements of evaluation. That distinction is not so hard to recognize that this protection would be incapable of balancing false-positive/false-negative.

The prevailing narrative achieved a definition land-grab. Today, I want to push back against the definition it established for slop. The Merriam-Webster definition that’s most relevant is “digital content of low quality that is produced usually in quantity by means of artificial intelligence”. That definition reflects common usage, but a critique can demonstrate how we’d be better served with a different definition.

You might ask, why should I, a relative nobody, be redefining terms? Isn’t that the reason we have dictionaries and lexicographers? I suspect it’s not commonly reflected upon that this isn’t how we receive definitions. Dictionaries in the English tradition do not create the meaning of words, they catalog it. The term the dictionary writers use for this is descriptivism. They do not invent terms, but rather record language as it’s used by the public in everyday speech and writing.

In theory this is an organic approach, but it is vulnerable to steering. If a group wants a particular point of view to be supported by some terminology, they merely have to be first to the plate and use the term repetitively. If you want a dictionary to define slop as being particular to artificial intelligence, you repeat that a lot. Create a meme.

This is how we got the definition we have. A motivated group repeated their narrative. Like the best such narratives, it succeeds via its degree of truth. Slop creators embraced AI as a tool for their purposes. The omission is that slop predated AI and would exist without AI. The fiction is that AI content is universally slop.

The absurdity is that the narrative circulators often lacked familiarity with AI. They rejected it early, labeled it a symbol of moral degradation, and thus could not come to informed opinions about it without violating a self-created social norm. How was an ill-informed group able to pull off this coup? Simple, they had already seized the means of production. Most came from the realm of writers and journalists.

Now, to be clear, I do not want to overclaim. This certainly does not apply to all writers or journalists — maybe not even the majority. But enough such that there is a plethora of written low-information opinions, and little counter-narrative. For those writers who did not fall prey to this siren’s call, the defence never carried the appeal of the prosecution. And with many other valuable things to write about, the counter-narrative went unrepresented.

Why am I motivated where they were not? Partly because I’ve been the subject of attacks that stem from the original narrative. I do get stuck in writing at times. I don’t have a lot of editorial assistance. But I have ideas I think are valuable, and I find AI a useful tool to help express them. But the narrative holders offer a Catch-22. Use AI, and be subject to automated filters, low-grade negativity and the occasional overwrought attack.

What should we call slop?

I hate slop. I dislike most complaints about slop too. Ironic?1 Technically, no — just like that use of “irony”, the root is about the definition.

In offering a definition of slop, I want to approach it in a different way than the lexicographers. I want to offer a definition that would be useful. Instead of accepting what has become common, I want to ask: what happens if we use this definition or that definition? Yes, this is the same steering I described above. The difference is that I’m doing it in the open and stating my criteria, so you can judge the definition by its consequences rather than by its repetition. I only offer this definition, rather than control it. It’s ultimately you who will determine its adoption.

The first consequence to consider: no matter what, slop will be a derogatory term. It’s very unlikely that it will ever be used in another way. With this in mind, our definition should avoid including things we shouldn’t think of as bad. A definition that’s too inclusive and lacking in selection will cause good things to be described in what will always be a derogatory way. This is why I don’t accept those lazy descriptors of “digital content” or “generated by artificial intelligence”. If these are core parts of the definition, my expectation is they’ll be dominant, and we’ll label many good things as bad.

In The Slop Scapegoat: AI, I described slop as “low-quality material created to grab eyeballs”. I’d iterate upon this and the Merriam-Webster definition.

Laziness

We should say slop is lazy. At some point, someone has stopped caring, and is avoiding effort that is appropriate.

This relates to “usually in quantity” from the Merriam-Webster definition. Slop’s harm stems from volume. But it doesn’t depend on one mass-produced source, it stings in the aggregate too. Many individuals following the same motivations add up.

But we can’t categorize by count, as we can’t count until we categorize. Also, many good things come in volume too. So, we turn to discussions of effort next, which is progress. I suggest laziness because there are niche examples of high-effort slop. This fits when someone is directing effort in a lazy way.

Manipulative

We should say slop is manipulative. It must have a purpose, and that purpose must be misaligned. You need a perpetrator and a target to have purpose and misalignment.

Usually it’s manipulative for the purpose of gaining attention. But I also see examples where it’s manipulative for the purpose of providing a veneer of competence. This veneer succeeds against shallow examination. The more common version, seeking attention, can be described this way: it seeks to gain attention by providing a veneer of quality. There, it’s no surprise that the veneer is later revealed, but after gaining attention, when the purpose has already been met. With a veneer of competence — an essay, say — the hope is to preserve it until the end of the examination.

Costs to the Audience

We should say slop creates costs for the audience. Laziness without harm we can overlook.

This relates to the Merriam-Webster use of “low quality”. This is a bit of a trap. Quality is hard to measure. At best, we estimate it, and how deep that estimation goes depends on the context. We often take shortcuts, and cut our efforts off at the lowest effort necessary to distinguish most high quality from low quality. You might say our efforts at assessing quality are a bit sloppy.2

Quality also pulls in low skill creations. I don’t want to start calling your child’s art project slop. They put effort in. You might turn that into slop by lazily posting it, and consuming the attention of people who aren’t interested. On the other hand, you might post it to those who are interested, or post it in a way that is interesting. Content can only become slop when it finds a delivery channel.

Ignore the medium and tool

The part of the Merriam-Webster definition I find lacking in usefulness is “digital content” and “artificial intelligence”. We should not focus here as they are distractions.

If I print slop, is it no longer slop? Am I not allowed to call a human that wastes my time with mindless actions slop because they are not artificial? AI content is not slop. AI content can be slop. Slop can come from anywhere, but you’re not wrong to associate it with AI, because statistically speaking it is associated.

Personally, I’m as angry with slop recruitment by phone call as by email. Recruiters often call me, clearly reading from a script, and then ask me four or so questions that are already clear from my resume. I know what’s happened here. An automated system has flagged me by a keyword match (yay me), but instead of having their employee put the effort into reading the resume, the recruitment company wants to fill in an online submission by having them call me, read the script, and force me to answer. They hope I’m desperate enough to put up with this.

We should call that slop too. It’s time consuming for me. It’s based on a lazy plan. It’s trying to provide the impression that I’m important, while they play a numbers game. They aren’t trying to help me, they are trying to lock-in a commission. It’d be less effort for me to submit directly.

That poor recruiter strapped to the desk making these calls isn’t the perpetrator of the slop, it’s the company leader that arranged it. They are merely the tool. AI can, and often is, the tool. But the perpetrator is not the tool.

When we focus on the medium or mechanism, we ignore the perpetrator. Focusing on the perpetrator allows examination of their motives. I suggest the motivations of manipulation and laziness as core.

The problem is when the tool is used to shift effort from the creator to the audience. That’s the third aspect at work. One such cost lands on the audience’s proxies for quality. It was convenient when you could recognize spam by the poor English and bad formatting — but spam wasn’t the only thing filtered by that proxy. Audiences should be willing to shift proxies; if a proxy has been undermined, it’s the only option. It’s also worth noting where the proxy failed: the shift isn’t necessary when something of value has found its way in, only when something lacking in value has.

The best example of slop? Poorly written articles, heavily SEO optimized, intended to attract eyeballs from Google searches. These articles require a certain kind of effort, but it’s not effort to serve the audience, it’s effort to capture them. We’re better off when Google is able to filter them out, or provide an AI generated answer, or highlight a definitive document. It’s true, that type of slop is turbo-charged by AI, but its prevalence predates AI. Some was human written, some was sourced from human content, then distorted by inserting ads, catchy headlines, and SEO optimization.

How this develops

I could make an effort to offer wording to replace the Merriam-Webster definition, but I’ll resist. I’d prefer to offer the counter-narrative, and for you to engage with it, push back on the current usage, and thus establish a new norm that the lexicographers can then capture. A new definition should be organically derived from that process. But as a summary: laziness, manipulation, and costs to the audience. Take those as the core, rather than the delivery channel or use of AI.

Slop Smells: How to avoid creating slop

The effort ratio

Is effort higher for the reader than the creator? Using AI to reduce your effort is fine. But if it reduces below the reader’s level, or even to less than four times the reader’s level, you should put in more effort. A proper multiplier is dependent on context. If your audience is one, the 4:1 ratio may fit. But if the audience is larger, your per-reader ratio should go up. A 1-hour presentation to a room of 50? Five to ten hours of preparation is a good floor (assuming they aren’t all multi-tasking…). A 2,000 word newsletter (10 minute reading time) to an audience of 1,000 or more? Two to three days of effort. There is a less than linear growth function here — no one expects the article read by 100,000 to take a year. It happens, but usually that’s the culmination of work, rather than the whole product.

The input ratio

In writing, a good sign is that you’re cutting — revising, rewriting. Your drafts should have cutting room material. Generally true of all writing, but especially true when working with AI. Be suspicious of cases where you use AI to generate output larger than your inputs. “Summarize my work research project from the last year”, or “Combine these three drafts”, is better than “Write an article on Shakespeare’s views on X”.3

Slop is in the eye of the beholder

The first two smells you can check yourself. Checking quality is harder, because quality is in the eye of the beholder. It is rare to align with an audience’s motives at low effort, but if you do, I would not call your content slop. The most probable means to do so is an intense connection to the audience’s motives. Some might call that taste. Be careful about arrogance though. Many a “taste-maker” has fallen for that trap.

It is possible to reduce effort without becoming slop. There is a minimum bound, but the boundary isn’t set directly by effort, rather it’s set by the need to align motives. If you have an idea you want to convey, and there is a high effort and a medium effort way of doing it, and both deliver something of equal quality to the audience, the reduction of effort doesn’t place you on the slop slope.

You land on the slop slope when your efforts to reduce effort start to degrade quality in the eyes of the audience. This isn’t a quality standard. Quality depends upon skill and effort, and anti-slop doesn’t need to punish skill deficits. Using a tool to compensate for something less than mastery is not the problem — especially when the tool allows you to put in more effort elsewhere to serve the audience.

Motivations

If you want to avoid creating slop, a lot goes to your motivations, but these heuristics can be useful for self-awareness. You do have a need to balance effort to reward. A fair audience should recognize good faith efforts at that.

Again, why should we care about the definition?

Because the definition determines what we do about the hardest part of writing.

Writing carries three challenges that should be important to every author: being clear, having something interesting to communicate, and making and keeping your audience interested. Communicating clearly may be the easiest, and it’s well covered elsewhere; I shouldn’t spend my or your time in that area.

Having something interesting to communicate is sometimes overlooked by both authors and audiences. Authors could overlook it intentionally, if they are writing for money: the employer finds the topic interesting, and the author’s interest is in getting paid. Or reducing further, the audience finds it interesting, and is thus the employer. There’s nothing inherently wrong with writing under direction, but if there’s two topics, each with an author interested in it, it’s best if the topics are aligned with the interested authors rather than the inverse. Besides the author’s satisfaction, an employer would generally see better output from the writer with an interest in the topic.

The third challenge is by far the hardest, most elusive, and most frustrating part of writing, in my experience. If you want to create an interest in an audience that didn’t already exist, you need to acquire their time and attention for long enough for them to develop an interest in the topic. Even when an audience is explicitly interested, distractions and competition for their attention demand finding ways to make an audience interested separate from the topic itself. This is the realm of psychology and all of our irrationalities, whether rationally irrational or randomly irrational. Needing to overcome irrationality will always contain an element of frustration. And the intentional use of irrationality is a form of manipulation, which even to overcome irrationality itself, can be frightening to engage in.

In addition, engaging in this last part alters yourself. Changing the style by which you communicate carries not just the frustration of investing time and effort, but the frustration that you might not like the final destination. If anything about that last sentence is unclear, watch any coming-of-age movie where the not-popular teen is miraculously added to the popular group and has a crisis of conscience from the after effects.

If AI is a shortcut through those layers, why should it be verboten? If it’s merely a competition, then any rules are valid. But outside of competitions, bypassing these layers is the price not the payment. It’s illogical to suggest that the ideas that should succeed are only those held or represented by the most masterful and efficient writers. Critics of AI, at least for the moment, are right when they suggest AI is less masterful than the world’s best writers. But it’s at least sometimes better than my writing.4 I’m still confident in my ideas, even when I’m struggling with the writing, and want to share them. These layers that connect with our attention serve as a rough filtering mechanism. But it’s very rough. The world’s best writers are not the repository of all the world’s best ideas. They have some of them, but definitely not all.

Bypassing those layers and getting the attention necessary to have ideas engaged with and evaluated is ultimately a good thing — unless the bypass shifts costs onto the audience. That’s the definitional line again: the shortcut isn’t the sin, the transferred cost is. A hollow attention or prestige seeking attempt fails that test; honest use of the shortcut doesn’t. And fear of the former isn’t worth the cost of banning the latter. There are better ways to do the necessary filtering, and AI detection is a poor one. While Pangram may have a low false-negative rate, what it detects doesn’t represent the truly important factor.

The Economist wrestles with the same question in Is AI writing taking over Westminster?:

That raises two questions. Is this AI writing a problem? And why the links to Mr Burnham? Start with the first. Politics is full of prose not written by the apparent author: politicians have speechwriters, intellectuals employ research assistants. Perhaps AI is no different, even if the prose is clunkier. But people setting out ideas in politics are asking for something quite audacious: to reshape how a country is governed around what they think. Any writer knows how much putting words on a page can tighten one’s arguments. Even subcontracting that to an aide beats skipping the (sometimes painful) process entirely. AI certainly produces sloppy prose, but it also papers over sloppy thinking.

Maybe that matters less if, as Ms Haigh and the “Productive State” authors say, AI is used only for a late polish. The trouble is that readers can detect whether AI wrote the final product, but not how authors used it. And when AI’s involvement is disclosed only after the press come knocking, trust takes some earning back.

The late polish is clearly different from the one prompt request. There’s also the question, is the document the output, or is the process of research the output? You can create a decent document via AI. You can learn by creating a document by hand. You can learn by working with AI to create a document. And there’s more than one way to outsource your thinking. Pangram understands none of that.

Is Pangram a good tool?

The gold standard in detecting AI writing today is a tool called Pangram5. Before Pangram, there was a question on whether detecting AI written text was feasible; it performed much better than expected, and its best performance number is a low false-positive rate. But Pangram has limits. A few are inside the tool: it’s not too hard to take a fully AI written piece and with some modest changes get a result stating 100% human written. And the false positive rate will only hold if you set a minimum bar — if you start considering “10% AI generated” as positive, rather than >50%, you’ll get more false positives than the published rate.

The real limits though are the context, because it can’t tell you what went into the writing. It can’t tell you anything about the originality of the ideas, and it definitely can’t tell you anything about their correctness or worth. There are a few contexts where knowing that the final draft was entirely human written is useful — an academic context meant to evaluate the capability level of a student in writing, where Pangram is a sufficient tool. There are others where it’s more questionable — writing intended to evaluate understanding of a concept, where surface level rewrites evade it. In both cases, you might wonder why you’re evaluating in this way, and whether you could avoid “cheating” by adopting a process that’s not adversarial. The purpose of mid-education evaluations should be to steer students toward more effective learning experiences. If you avoid introducing adversarial dynamics, students should be interested in honest feedback, which requires honest input. Save the adversarial evaluations for a context where they are actually important, and then invest all the necessary efforts to definitively stop cheating.

What about the context of filtering social feeds to remove slop? It has the downside of also filtering out non-slop that used AI in a final/late iteration. While I lament that, considering some of that is my own writing, I would have to admit that it will accomplish a lot of its objective in creating a feed that is, as a percentage, less slop. Until someone builds a better tool, these are your options. We shouldn’t be quite so gleeful about this compromise though. Slop will find a way, where honest AI-using authors may not. High volume slop may just play the numbers game: if you filter 90% of it, create ten times more. And human generated slop won’t be caught at all, and is still numerous and insidious enough to be a problem.

The real problem with feeds is that they do not even attempt to take our higher interests into account. The justification is a desire to not be paternalistic, but the fix is to give us control over applying higher interests to our own feeds. In lieu of the paternalistic “quality” driven feed, we’re given a “value-neutral” algorithm which is optimized for ad revenue generation. This isn’t a choice any of us would have made given a choice, so why do we accept being stuck with it?

Ad revenue optimized feed algorithms optimize for engagement time, and we’re fed the narrative that engagement time aligns with our interests: if we choose to engage, the engagement is a sign of our interest, and thus an engagement driven feed serves our interests. Nice narrative — true enough that it managed to avoid scrutiny until deeply embedded, but not true enough to avoid some serious downsides. Effective slop is the content that tricks us into engaging, while not fulfilling our real interests. Slop utilizes the tricks of engagement. If it’s high volume automated slop, it will optimize the initial words, adopting easy to follow patterns to draw in engagement. If it’s human generated slop, it will use those tricks, plus others: a pretty face, totally irrelevant to the content value; catchy headlines, “Ultimate Guide to …”, “Beware of this trap …”. Feed algorithms don’t protect you from this, they accentuate it. They “feed” on it. Each time you’re tricked, you get more of the same.

What you want is a focus on quality. Feed algorithms have a few signals that align with impressions of quality, such as likes and reposts, but these come too little and too late. Their weighting is too low in comparison to simple engagement, so what signal they give is overridden by the tricks that align with slop. And a lot of good content is already gone, having failed the first round engagement filter, never seeing enough views to gather likes and reposts. Some is left — enough to keep you from deleting the app entirely — but it may not be the best, and it’s certainly not all of the best. And there’s a lot of slop, for which you are the only filter.

We could do better

A pattern that is both predictable and understandable, but also wrong: when a force for change emerges that accentuates a long ignored issue, we react against the force, rather than addressing the issue. It seems easier to stop change than to engage in more of it. But this is a bad plan, as the force will win in the end. If you delay it, you best use the delay effectively, or when your attempts at delay inevitably fail, you’ll be worse off.

It’s not hard to imagine better systems than these feeds. I’ve got a collection of ideas. Maybe some of these would fail, but it’s unlikely they all would. What’s really surprising is that we don’t even see examples of failed experiments here. Ideas for better designs are so easy to come up with, it’s initially hard to understand why. The reason though, is that to implement them, you need a type of access that the social platforms are actively discouraging. That keeps them small scale and personal, and blocks their ability to spread. It seems unlikely this is accidental, seeing as its result is a preservation of the status quo that optimizes advertising revenue.

This should be our real target, and this is why you should not accept “slop” as being described as AI content. The narrative that follows that definition is one that suggests a quick fix of a Pangram augmented filter. That fix is only partial, and what success it has will only open a hole for new slop generation techniques to fill. The root system that optimizes for putting slop in front of us, and rewarding the slop creators, hasn’t been touched. The most likely outcome is a short-term win, followed by a regression to nearly the same equilibrium. It’s one more layer of adversarialism that brings negative side-effects.

Related Articles

Disclaimer: This final draft is AI generated, then edited by me. What does that mean? Well, after some fairly substantial writing, I was tempted to drop this article entirely. I’d taken the writing in a few directions, and it was becoming a sprawl that would take a long time to recover from. I felt there were interesting ideas, but wasn’t sure it was worth continuing. Before abandoning, I worked with Claude to rewrite around a new concept. The core ideas here are very much my own. Since Claude had access to my (failed) drafts, many words are mine too. But the direct product is not. If you’re substantially opposed to AI writing, and you dislike this, you can reaffirm your priors. If you can see through that to the ideas here, then maybe there’s something valuable. I leave this with you. I could spend time using this as an inspiration, rewriting parts, and eventually this disclaimer would no longer be necessary.

Every company past a certain size runs into the same problem. It needs to know what’s happening inside itself, and the only way to know at scale is to measure. But measuring changes the thing you measure. Tie a number to someone’s standing and the number stops telling you about the work. It starts telling you how people react to being numbered.

That’s not a flaw in any one metric. It’s the basic mechanics of management, and most of the recurring messes in corporate life are those mechanics showing up in new clothes.

The newest example is AI usage tracking. Companies built “leaderboards” to measure how many tokens each employee burned, and “tokenmaxxing“ grew up around them. The leaderboards are being torn down now, and the easy lesson is that they were a mistake. I think that misses the more useful story. The leaderboard was a measurement. Like any measurement, it could have been a way to understand the team, or a machine that wrecked it. Which one it became had less to do with tokens. It had to do with the culture it landed in.

I want to flip the framing. Tokenmaxxing is the supporting story. The real one is about how a management culture takes the measurements it actually needs without those measurements rotting in its hands.

The saying that starts the trouble

Start with one of the most quoted lines in management, and one of the most quietly destructive: you can’t manage what you can’t measure.

It sounds like rigor, but it’s really a hidden assumption — that everything important can be measured. It can’t. Whether your people trust each other. Whether they tell you the truth when the truth is inconvenient. Whether one quiet engineer is the reason three teams ship on time. Whether someone’s messing-around this quarter becomes a real tool next year. These decide whether a team is any good, and none of them hold still long enough to be counted.

Take the saying seriously and it tells managers to ignore exactly these things, or to invent stand-ins for them and manage the stand-ins instead. You end up with a manager optimizing a dashboard who believes they’re doing sharp work. They’re doing bad work with better instruments.

The first thing a healthy culture admits is that the most important things will never show up on a chart, and that managing them anyway — by judgment, by paying attention, by knowing your people — is the actual job. Measurement helps inside that job. It’s a useful helper and a terrible boss.

Why measurements rot

Say you accept that you still have to measure something. The question becomes mechanical: what turns a useful measurement into one that poisons the thing it tracks?

Goodhart’s Law names the result — when a measure becomes a target, it stops being a good measure. But it doesn’t tell you what lets it take hold. Three things do most of the damage.

You tie the number to standing. The moment a number decides who gets rewarded and who’s at risk, everyone being measured cares more about the number than about the thing it was supposed to stand for. The dishonesty is terrible. But it’s hard to lay blame. They’re being rational. You built a game and they’re playing it.

You roll the number up. In most companies a manager’s standing is built from their reports’ numbers, and that manager’s number feeds the layer above, and so on. This is the quietly fatal part. When a manager catches a report gaming a metric, calling it out lowers the manager’s own score too. You’ve asked people to police a number that pays them to look away. Stack ranking and the usual performance calibration run on this same wiring — managers set against each other, reports who game the system pulling their manager up with them. I’ve never seen a clean example of it working, and the reason is structural. It’s not about finding better people.

You only manage downward. Management that runs in one direction — pulling data up, never answerable for the environment it creates — has no correction built in. Information flows up after it’s already been polished, and nobody whose incentives are intact is in a position to notice the data stopped being true.

Put the three together and you get a machine that reliably turns measurement into theater. Notice that none of the three is a property of the metric. They’re all properties of the culture you drop it into.

The same number, two outcomes

This is where tokenmaxxing earns a bad name, by participating in bad management practices. There was a reasonable purpose at the start, but the theater arrived quickly.

Technically, the word has two meanings, and it helps to engage with each. On the company’s side, tokenmaxxing was the choice to build a leaderboard and signal that more AI use was better. That choice had purpose. Most companies had spent a year discouraging AI with restrictive security policies, and they needed a push to break both that inertia and the ordinary human reluctance to change how work gets done.

Later, tokenmaxxing described the unproductive response — and became the dominant meaning. Some people tried AI in good faith and kept what worked. Others focused on their usage numbers for no reason except that tokens were what got counted. In other words, doing work to look like they did work.

A company running a leaderboard gets both the experimentation and waste because it can’t tell them apart at scale. Nobody can audit intent across a few thousand people. So the program is really a bet. Take a pile of aimless activity and some deliberate waste, in exchange for the slice that turns into something durable — a real skill, a useful tool, a project nobody had time to chase before. Said that way it’s an ordinary bet, the same shape as a research budget or a hiring class. And like those, it should always have been temporary. You retire it once the inertia is broken.

Now watch the same leaderboard land in two different cultures.

In a healthy one, a manager sees a report sitting at zero tokens and reads it as a question. What’s going on? Is the tool not helping? Is there a reason? The number becomes a reason to have a conversation. The manager has no rollup score to protect, so they can be curious instead of defensive, and the number stays roughly honest because nobody’s livelihood is riding on bending it.

In an unhealthy one, the same zero reads as a verdict. The rollup punishes any manager who admits their team’s numbers are soft. Within a quarter the leaderboard measures one thing: each person’s willingness to game it. Same tool. Opposite outcome. The variable was never the token.

That’s why I’d call the leaderboard a stress test, not a cause. Drop it onto a culture with a hidden crack and it doesn’t make the crack. It loads it until it shows. The worker backlash these programs set off was real and it did damage, but its source wasn’t the number. It was the accumulated, accurate sense that bad management rarely gets removed and often gets rewarded, and that any new tool would be bent to serve it like every tool before. The leaderboard just made that easy to see.

Running a culture that can measure

If the metric isn’t the variable, then “pick a better metric” isn’t the fix. Neither is the opposite reflex of refusing to measure anything. The fix is cultural, and it’s harder, because you have to keep it up rather than decide it once. A few things seem to separate the cultures that can hold a measurement from the ones that break it.

Keep a gap between the number and the reward. The instant a metric is wired straight into pay and survival, the rot starts. The people you most need telling you the truth now have the strongest reason not to. Healthy cultures treat a measurement as one input a manager weighs against everything they can’t measure, not as the verdict itself. “Let’s understand why usage varies” survives. “Bottom decile is at risk” is already rotting.

Give managers a real reason to want the truth. Mostly this means taking apart the rollup, or at least refusing to let a manager’s standing be a straight sum of their reports’ numbers. A manager whose rating doesn’t depend on their team’s metric looking good is finally free to do the thing you hired them for — notice when the number and reality have split, and say so.

Point accountability inward, not outward. The reflex, when people game a measure, is to go after the visible gamers and stop there. The trouble with that framing is it quietly lets the managers off, and the managers are usually the ones who built the environment that produced the gaming.

A leader at the top can be careful and dodge the rollup trap — refuse to let their own standing ride on their org’s numbers. That doesn’t stop a manager one level down from doing the opposite. They take the metric they were handed and roll it down onto their reports as a hard target, then turn a blind eye to how the reports game it. The pressure to make the number gets passed along even without explicit design.

So when gaming shows up, the manager is rarely a bystander. Often they were part of the gaming — they wanted the number to look good and didn’t care how it got there. Where they weren’t actively in on it, they were just bad at the job, blind to the fact that their own pressure was manufacturing the behavior. I’m not sure which is worse, and for individuals the difference barely matters.

Either way, punishing the report who followed those incentives, while leaving that manager alone, fixes nothing and adds a fresh unfairness. Pulling back unearned rewards so gaming doesn’t visibly pay is punishment enough. But the real correction is at the center — the people who shaped the environment — not the edges. Fix them first. They’re the ones who’ll do it again.

Trust is the thing holding it all up, and it’s self-fulfilling. In a culture where people trust that a low number gets met with curiosity and that gaming gets caught instead of rewarded, you can introduce a measurement without panic, and the calm keeps it honest. In a culture where people expect the worst, they respond to the worst, and the response creates the very rot they feared. The belief and the outcome make each other. So trust isn’t a soft extra bolted onto a measurement program. It’s the load that everything else rests on. A measurement dropped into a low-trust team is closer to tossing in a grenade than running a diagnostic. (I’ve argued before that money itself is just trust — measurement inside a company is no different. It only works if people believe the other side is dealing straight.)

Stay humble about the whole thing. Management is a field of unintended consequences. Its job is to improve the value the people under it create, which in a perfect world would mean nothing, because you can’t improve on perfect. The mistake is for management to act as if it lives in that perfect world — clean numbers, frictionless incentives, its own presence bending nothing. Accepting that the world isn’t ideal, that every metric is a little flawed and gets more flawed as the work gets more complex, is the first of many steps toward doing it well. The cultures that can measure without breaking the measurement are just the ones that never forgot the tool was imperfect, held it loosely, and were willing to put it down once it had done its job.

The lesson hiding in the leaderboard

The leaderboards deserved retirement, but they weren’t all folly. They did an ordinary job that reached its expiration date. In cultures healthy enough to use them well, they were probably useful. Even in less effective cultures there may have been a golden period before the rot set in. Where they turned into a disaster, the disaster was already there, waiting — in the rollups, in the important things ignored for measurable stand-ins, in accountability that pointed outward instead of inward, in trust that had been spent long before anyone counted a token.

That’s the lesson worth keeping once the leaderboards are gone. The next iteration is already coming. It’ll promise the same clean view into the same messy reality, and it’ll rot the same way, unless the culture holding it has done the slow, unglamorous work that lets a company measure itself without lying to itself.

The hard problem was never the metric. It’s building a place where the truth can survive measurement.

This isn’t because doctors don’t understand medicine, patients, psychology, or even statistics. They do. Medical professionals are competent and genuinely invested in patient outcomes. The reason the medical community is getting this wrong is that they are failing to think through the lens of a broader data system. Understandably, their default mode of thinking engages with one patient at a time, and typically, one specific issue at a time.

But data doesn’t work that way. Data’s value is cumulative and often curves upward. More data is rarely a bad thing—provided it is used appropriately.

Currently, we miss opportunities to utilize data because of information processing limits. These limits aren’t just computational; they are also process-oriented. The maintenance of privacy across organizational boundaries and approving information flows for technical implementation. It should come as little surprise that applying AI to medical software will ease these boundaries. By automating those internals, we’ll be able to lower the obstacles without compromising on privacy controls, organizational limits, and traceable responsibility we desire.

With those obstacles gone, individuals with richer historical datasets will benefit immediately. The value of a scan, test, or medical record isn’t limited to an immediate concern. It can unlock a future insight. A future diagnosis informed by history is both more accurate and timely.

This cumulative value is what the medical community misses when they downplay the ambition of inexpensive, pervasive testing. Looking at a single patient with a single concern, an MRI (with its comprehensive snapshot) might always seem preferable to an ultrasound. But imagine an ecosystem where most patients have a baseline full-body ultrasound history, a 50-cancer screening blood test, and ten other sets of low-cost, routine diagnostics. In this ecosystem, there is a high probability that these data points will unite to form insights that a purely symptom-responsive history never could.

Individually, each cheap test won’t be conclusive enough to move a diagnosis from “possible” to “probable.” This introduces a valid concern: a test that shifts your probability of a disease from a background rate of 0.5% to 2% doesn’t merit invasive testing, but it might trigger hypochondria. But the combined history of three different inexpensive tests can shift that probability progressively—from 0.5%, to 2%, to 5%, to 15%. At 15%, invasive testing is merited. Without that cumulative history, your doctor would have rationally recommended against it, potentially missing a crucial early intervention.

Inversely, cumulative data can help avoid unnecessary invasive testing. Suppose you present symptoms that trigger concern, but your extensive testing history lowers the probability that the cause is malignant. If the symptom alone suggested a 15% risk, but your historical data shifts the overall probability down to 5%, your doctor could rationally recommend treating the symptom or taking a path with less upfront risk.

Some may interpret this as a personal call to build their own medical histories at any cost. While not entirely unreasonable, that is not my point. I’m not just thinking about individuals with enough disposable income to afford boutique medical testing1.

Instead, I am defending the systemic value of testing that is cheap enough to be pervasive. Pervasive, low-cost testing can transform healthcare from a reactive, symptom-driven model to a proactive, data-driven one. It personalizes preventive care and provides the volume of data necessary for AI and modern analytics to spot macro-trends and micro-anomalies that a fragmented system would miss. Efforts to drive the cost of testing down to the point of pervasiveness are the foundation of that future. Those efforts should be loudly applauded, not feared.

The concerns about fueling hypochondria and over-worry are real, but ignoring the value of cheap, pervasive testing is not the solution. We already manage hypochondriac tendencies in medicine today. I wouldn’t call myself a hypochondriac, but I’m not immune to that type of feeling. In my experience, one anxiety-inducing event is the annual physical itself: cataloging every new bump or brief pain, anticipating the doctor’s questions, and overthinking the answers. We don’t suggest abandoning annual checkups just because they cause anxiety. Instead, we rely on a doctor’s bedside manner—a core component of medical training—to manage that stress.

I generally feel better once I’ve completed a checkup. I expect the outcome there depends heavily on how effectively the doctor communicates. That is where we can manage the risks associated with an influx of new medical data. We should expand our thinking beyond just the bedside manner of individual doctors, and focus on the design of the entire patient communication system.

The medical system already does this to some extent, but as we enter an era of pervasive data, there is a need for more. The answer to the anxieties of the information age isn’t to reject the data, but to design a healthcare system capable of communicating its true value.

AI is more than software, but software is very much at the core of AI. And the reality is the public has not been very interested in understanding software development as a profession. A point of evidence for this, and a mechanism leading to being poorly informed, is that software development is one of the least-represented occupations by entertainment television and film.

I made this point in a story a few weeks back, in a longer post about bubble narratives. It was part of a larger narrative, but I heard confusion about why I brought it up. I thought it’d be useful to pull this out, and clarify why I think it’s an important topic.

Since software is at the core of AI, it feels like people should be informed about it. Being informed requires more than being a user. It requires understanding the lives of those developing it. It’s one thing to be disinterested in how your phone or computer works, but happy to use it. It’s another to have only caricatures of software developers to rely upon, know little about the basic processes we use to secure, develop and deliver software, and yet have an opinion on AI development.

Here is what I wrote:

Why the public has a poor understanding of software development

The wider world has never shown broad interest in learning what software developers do. Compared to other professions like police, soldiers, doctors, lawyers, musicians, writers, journalists or even criminals. Without that interest it’s unlikely to learn the inner workings of the profession.

Media portrayals of software developers are rare and rarely accurate. The most common portrayal is the “hacker” who mysteriously takes control of computer systems in a few minutes with no preparation. Not only is that a poor representation of a real hacker, it tells you nothing about software development overall.

I reworked the data above to make it easier to see the patterns.

Ultimately, my concern here is deeper than portrayals. The facts about portrayals are an example. That example does have real effects, but entertainment media is also reacting to interests that precede entertainment choices.

The real concern is that advocating for laws or regulations that apply to software without an understanding of the process of development, is a recipe for failure. There are many places one might learn about an occupation. Media is just one, but it is a powerful one. Also, my experience as a member of the occupation is that most of my conversations about the occupation were limited to people inside.

There is a personal side to this. I’ve often been troubled by how the world in general treated the work I did as somewhat foreign. I noticed an imbalance in how much time conversations dwelled on my own work. It’s not the only profession like that, but it did register with me personally. I bring up my personal experience, not because my feelings are the point. They might be worth a little thought, but the point is I can see this disconnect so clearly because it is personal. If it’s not personal, it’s an easy gap to miss.

Software isn’t the only underrepresented occupation, but it is high stakes amongst those. It does appear in the news, but is it the facts of software developers that are being represented in the news? That’s not my observation either. The personal opinions of writers, artists and media that are heavily represented instead. Those same groups are over-represented in entertainment media, and have their opinions as the most visible via that route.

I’d suggest this is a moment where it makes sense to lean in a bit. It might not be natural to be interested in how software is developed. It might be more convenient day to day to enjoy the software, and treat the field as opaque. But if we need to make decisions about the field, that gap will become relevant. Everyone should be part of making those decisions, but the process of making those decisions must be well informed too.

So, if you’re a journalist, you should be interviewing software developers more often. If you’re creating a movie or TV show, you should consider writing in a character from software development, and avoid applying inaccurate cliches.

I’m trying to do my part with this writing. I probably can do more to make it more accessible. There is a tendency in software development, like any field, to write for your internal audience. That’s reinforced when outside interest is low.

If you’re a regular reader of this Substack, it’d be redundant to suggest you need to read something from a software developer, but I’ll assume this message may reach those beyond that scope. So encourage others to connect with the how of software development, both here and with other software developers in their lives.

That’s not going to be enough to bridge decades of disengagement, but it is a start. Since the political decisions of AI can’t wait for a repair, I suggest some humility in decision making here. In AI Safety Is Underfunded by Design, I suggest that we need balance that takes advantage of organic efforts of self-regulation. I wouldn’t say that self-regulation is sufficient; even if it’s working so far, this is too important to take that approach. But at the same time, regulating what you don’t understand will fail.

A learning process is necessary, both amongst the wider public, and with the politicians, lawyers, and administrators that would need to be part of any effective regulatory system, alongside industry experts.

Related Posts

The architecture I outlined had 4 layers: compute supply chain, operational infrastructure, intelligence and application. Each layer has to justify its own expenses, which contribute to the revenues of the lower layers. When those aren’t justified, the whole structure is a gamble on the outcomes of the upper layers. That said, the immediate term for lower layers can be remarkably sound, because they get paid now, not later.

This story is only half written, so we can look at how each layer is doing today. Even when that story looks good, we shouldn’t extrapolate to the future. Each increment of demand must prove itself, both in terms of feasibility and in terms of timing.

How is each layer doing today?

Layer 1: The Compute Supply Chain

The main influence on this layer is the inputs from Layer 2. Since those plans have kept progressing as planned a year ago, it should be unsurprising that this layer has done well. Revenues continue to increase and stability looks favorable. The main change here is that competition is growing. The availability of chips designed in house, TPUs, Trainium and more, have grown. The efforts to make Intel capable of competing with TSMC have continued. TSMC has continued expansions of its own.

Additionally, memory and CPUs have been pulled closer to the middle. Memory was already a bottleneck and has become more central. GPUs have remained central, but share much of the spotlight with memory now. CPUs have moved from trivialities, to moderate importance.

Layer 2: The Operational Infrastructure

The situation at this layer has firmed up significantly in the past 6 months. Revenue for delivered compute has continued to grow. Additionally, commitments have been disclosed, adding stability. Disclosures about commitment numbers don’t express the exact terms, so we don’t know if these could reverse.

More importantly, everything is broader here. OpenAI and Anthropic are using compute from all platforms. xAI showed how to recover from a failure, at least partially, by selling unused compute to Anthropic.

All that good news shouldn’t ignore that there’s still significant spending planned that will have to justify itself. The revenues of today (+$100 billion ARR) are roughly proving that last year’s $410 billion in spending isn’t going to be unproductive. That doesn’t tell us enough about this year’s $700 billion, nor next year’s $1 trillion. So long as those keep growing quickly, the risk that expected revenue to cover it never appears should remain a live discussion.

Layer 3: The Intelligence

There are many positive updates here. The best ones concern Anthropic, which is a moderately bad story for OpenAI. Revenues have grown considerably. Anthropic expects to be profitable. Enterprise revenues have grown considerably. Could this reverse? Maybe. Much usage is still prototypes, experimental, or otherwise dependent on expectations. But it seems unlikely. As I covered last week, there’s more pulling us forward than back here.

Layer 4: The Application

This layer still contains a large amount of uncertainty. The most significant use case, software development, is fundamentally another layer. Last week I covered dynamics of token usage by the application layer. While this is one of the longest articles I’ve written, it only covers dynamics, not actual numbers, and only a few of the most important.

To be truly solid, the various parts of the application layer need recognition that value is being created. That’s hard, because there’s a lot that would need to be measured here, and many of the things you want to measure, are both difficult and emerge with lagging indicators.

Adversarial usage, in marketing ($660 million), legal ($650 million), and sales ($390 million), is following behind coding and IT usage.

Early adoption from sales and advertising, shapes how much effects need to be proven. From a financiers perspective, this isn’t a problem, maybe even positive. From someone worried about employment demand, it also isn’t a problem.

But from a social perspective, it’s not sound. We can’t expect significant positive value to come out of adversarial sectors. We want to see demonstrated productive revenue, and we want to see cases where freed labor is reutilized in growth sectors.

Conclusion

Is the AI industry past the “bubble” discussion? No. Certainly not in terms of stock prices and valuations. In terms of CapEx spend, and the likeliness of future contractions or firm failures, it’s on firmer footing, but the story still progresses as the mountain is only half climbed. You might say that anyone who put 2026 as a specific timeline for their concerns, has missed the mark, and there are many who did. But overconfident predictors don’t make an effective counter-argument to more general views.

Related articles

Preview

It would be useful to know the shape of future AI demand, and many are attempting to predict that. Since this is a long piece I’ll give you my predictions up front. My overall prediction is that localized corrections, from the imposition of usage controls and consistent pricing, will ultimately be less important than the big trends. Value, so far hard to measure, will become more clear, first through incremental gains at the core of software development, and next, from the innovation that takes longer to accumulate and organize.

Read on to learn how I add my experience in cloud computing and software engineering to my deep interest in economics to extend responses from two of my favorite writers. Along the way, I’ll recast the bubble analogy, explain recent trends that have hit the news, explain trends hidden deep in the development lifecycle, and provide a model, “Ingenuity Matrix”, for mapping usage intent to expected outcomes.

We love good stories, especially those with a villain. But we should be careful about our stories, knowing how powerful they can be.

Three stories have hit a crescendo at about the same time. Tokenmaxxing — companies turning token usage into a goal, metering it, and the waste that incentivizes. Subsidized tokens — questions on the relationship today between AI costs and pricing. And under both, the doubt about whether spending is producing value for AI customers.

Stack them together and a tidy narrative falls out. If these are what’s driving token usage, and they all adjust at once, the readjustment will ripple through AI industry economics — including Anthropic’s recently skyrocketing revenues. That narrative extends across all model providers, culminating as a cascading failure of the whole AI industry. Call it the token bubble, brought on by a revaluation of tokens and their utility.

It’s a neat story, but the framing is off, even before we get to evidence. “Bubble” as metaphor smuggles in two assumptions: that we’re looking at one structure, full of only hot air, and that it ends by popping. In reality, industrial bubbles deflate, running out of air. There is an inflated shell, inside of which a structure is being built, and its collapse while deflating halts construction within, and damages unfinished construction. But something remains.

Inflating the shell is not folly, but the simplest path to enable construction. It’s still calamitous when it deflates, but the goal is the structure, not the air. So the question I’m interested in isn’t “are we in a bubble?” It’s: which of these dynamics is air, which is structure, and how would you tell them apart?

The story has been covered by two of my favorite writers, Derek Thompson, in The AI Boom Has Entered Its ‘Wait, Is This Worth It?’ Era and Noah Smith, in How much more software do we really need?. Both play speculatively with the idea that spending and rationality may have split from each other, but retain optimism that something worthwhile is being built.

Thompson concludes his summary of an interview with SemiAnalysis’s Doug O’Laughlin:

Every new technology requires an extended period of trial and error, as organizations toggle between (a) not enough experimentation or spending, followed by (b) too much experimentation and spending, followed by (c) too dramatic a pullback, followed by (d) the repetition of steps (a) through (c), until firms figure out a long-term balance between labor spending and tech spending. Whether AI skeptics like Marcus are right that the bubble is about to pop depends entirely on a question that, as of today, nobody can definitively answer: Is the bill worth it?

Smith considers the period before a smarter than human in all ways artificial general intelligence:

But until we reach that point, it’s a nontrivial task to think of business models that could be fully automated even with an AI that can’t yet do everything. That’s going to be hard! If I had any good ideas for how to do that, I’d go become a billionaire myself.

At some point, though — maybe in the very near future — people (assisted by AI) will come up with those revolutionary new business models. At that point, tokenmaxxing will suddenly become a lot more economical, and Anthropic — or whoever has good coding agents by that time — will stand to make untold amounts of money.

These are good perspectives, but I can improve upon them to help understand the dynamics of AI usage. First, I’m from the software industry, which is at the center of the maelstrom — coding is now the single largest category of token usage. I can describe in more detail what developers are actually doing with these tokens, and their motivations. These details are important. Without them, a lot of valuable work remains mysterious, which invites doubts, such as “is this worth it”, or “do we need more software”?

Second, I’ve spent a while thinking about the adversarial dynamics of some AI usage, since first writing about it last year. Those dynamics are key to the questions both writers leave us with. Adversarial usage doesn’t produce the social value we all seek. It is not the only driver of AI usage, but when it is a driver, we should be asking, “is this worth it”?

Both writers are aware of an important detail, timing, which explains many misleading observations. With the addition of a deeper understanding of software development, and that model for separating zero-sum jockeying from the creation of social value, we can recognize events along the timeline with more accuracy.

Token usage, like human labor, can’t tell you progress. Its best analogy is effort. If you want to understand the effectiveness of effort, you want to know how it’s being applied. Different applications correlate with different outcomes. Since you can’t fast-forward to the results, this is the best immediate categorization you can add. I call this categorization, the Ingenuity Matrix, describing the scope and social alignment of token usage.

Some token usage goes nowhere by design, some burns down a backlog of long-deferred work, some is zero-sum jockeying. A slower, quieter share is the significant work that actually changes lives. Sort the usage that way and the “is it a bubble” question dissolves into a more useful one — what’s being built, what events can we expect along the path, and what risks and opportunities come with each set of events?

Background

Before we start into the model, understanding the two terms behind the narratives is useful. This will also be useful when reading general news on the topics. The narratives on these conflate multiple meanings, and smuggle assumptions. That ambiguity can support misleading narratives1.

Why Bubble as an analogy is over extended

I said in the opening that using “bubble” as a metaphor for the AI industry smuggles in two assumptions. A bubble is so commonly used to analogize industrial revolutions, that we fail to reflect on the limits it has as an analogy. One mistake it leads us to, is the belief that there’s a soap bubble floating in air, and when we prick it with a pin it will pop, and evaporate. This does a poor job of explaining reality though.

We might limit our imagination more effectively by replacing the soap bubble with an inflatable dome. Whether this stays inflated depends upon the balance of air entering and exiting. Inside this dome, we’re constructing something durable, but it would be a challenge to do so with the dome weighing on top of us. We need the air to keep the dome’s ceiling from impeding our construction, and if it deflates it will probably ruin any half constructed structures. The stronger completed structures can sustain the weight of a deflated dome, but will struggle to conduct any additional construction.

If you want to think of the social support for a system, which supplies the air to keep the shell inflated, as a bubble, that’d be fair. This can evaporate with a bad news story, or some other form of social contagion. That social support is what replaces the air that leaks out. We’ll discuss the leaks later. Some are necessary, some are not. But replenishing the loss is unavoidable.

It’s useful to remember that in this analogy, deflation isn’t free. Something will remain, but the damage to unfinished construction is real. Careers are an obvious example of the consequences. When companies downsize the skills, connections and tacit knowledge built to support growth get stranded. If people move on, they may never come back. And besides, they are people and the disruption to their lives matters too. Projects also take a hit. Some projects may be zombies, shambling along with an unsound structure that will never be completed. But the forces of deflation aren’t so selective, and promising work is wiped away as well. Many projects that stop work during periods of tightening never start again.

A second flaw in the analogy is as a singular structure. Not only are there independent structures being built within, there’s not a single dome. There is a primary dome, where the model providers, GPU manufacturers and designers, and much else reside. But AI is also working to serve many different industries, and we shouldn’t assume a shared fate between all those efforts. We do want to pay attention to software development, because it represents such a large fraction of current usage. But software development itself isn’t an end of its own, it serves other industries. If AI is effective at helping some of those, and less-effective in others, this doesn’t establish a shared fate. It is only those cross-cutting effects that affect all software development that would carry that risk.

For the most part, those outside of software development aren’t going to understand those cross-cutting effects. I’ll highlight some of those details here, as they should be relevant to anyone interested in the immediate future implications of AI.

Why the public has a poor understanding of software development

The wider world has never shown broad interest in learning what software developers do. Compared to other professions like police, soldiers, doctors, lawyers, musicians, writers, journalists or even criminals. Without that interest it’s unlikely to learn the inner workings of the profession.

Media portrayals of software developers are rare and rarely accurate. The most common portrayal is the “hacker” who mysteriously takes control of computer systems in a few minutes with no preparation. Not only is that a poor representation of a real hacker, it tells you nothing about software development overall.

When software is “done”

If you come from outside the software world, you’d be excused from thinking of software development as building new software. In reality, this is a modest part of software development. Maintaining software, deploying software, and operating deployed software all represent larger segments than new software. All said, new software could be as small as 20%.

Noah makes a tentative argument that “The world may already have most of the traditional software that it needs.”. Noah’s aware he might be getting this wrong, and indeed he does. It does take an immense amount of work to keep sites running. AI is being used here, but it started later than its use to create new software. It’s not too hard to guess why. Creating new software is low risk comparatively. Like everyone, trust of AI has been a process. Software maintenance and operations themselves rely on significant “tech-stacks”, which have to be modified before you can even attempt to use AI to make a site more reliable in a meaningful way.

The number of software releases for security, operational, monitoring and development oriented features has been significant over the past year. Many use AI. Probably many others were built using AI.

Should you expect faster load times and higher reliability? First, would you really know? These have been improving for years, yet the general public rarely comments upon it. Mostly the only comments are those times when something does fail.

I’m not holding this data above as proof that AI has improved reliability. These improvements are more likely the result of conventional engineering, some started years before the results. The results that are “AI” based, are the result of the “machine learning” form that predated the architectures for Claude, Gemini and ChatGPT.

The point is, Noah (and you too) probably aren’t a sound judge of whether improvements are occurring unless you take the time to gather data. From my own knowledge, I know most AI based improvements are still in the early phases of adoption. But the average person shouldn’t expect to have an intuitive grasp on this. We’re bad intuitive judges of background effects like this, where we have to compare changes over time of non-continuous events. We can recall the last event, and the last change, but we are just as likely to draw a pattern from a recent reaction, than from an accurate history.

Why “tech-debt” comes first

You’ll find an interesting pattern that I’ll get into later. The first work to be done is the “shovel-ready” work. It’s easy to generate a prototype for some random idea, but rarer to have a great idea that can go from ideation to production quickly. AI does speed that up. But it doesn’t speed all work up.

With that in mind, provide a tool to a software developer, and they’ll have a long list of things they wanted to do, but haven’t had time for. Our general term for this is “tech-debt”, but realistically, it also includes half-baked feature ideas, or features that were sound but never made the cost-effectiveness cut. This list predictably contains a lot of non-amazing things. If they were amazing, they would have made the cost-effectiveness cut the first time. But AI does give you a reason to go deeper into that marginal backlog.

Security as a priority

I should also mention security here. Security is extremely important to the operation of software. Failures of security are nearly the worst thing you can imagine. This applies to all phases of software: development, deployment, and operations. It’s tempting to think of security as something you simply develop. But in reality that’s just the first step. A significant failure in development is likely to lead to a significant failure later, but it’s not destiny. You can layer protections to mitigate a development failure during operations. You have to do this because there are development failures you don’t know about. And more importantly, even a soundly designed and developed system can fail if not operated properly.

A lot of time and money is already spent on security. It’s never been the case that it hasn’t been a priority. You can find cases where it wasn’t a high enough priority. But it’d be a stretch to suggest there was a case no one cared. Whatever the priority, there is a limit, a cost-effectiveness barrier where one of the stages of development could have achieved more with more inputs. The introduction of AI changes the math on that barrier and makes many things practical that were impractical.

Security has another dimension too, which is that in addition to AI altering the developer’s cost-effectiveness equation, it does so for attackers too. This creates another incentive to burn down the security backlog. Security can’t wait. And so a lot with good cause, a lot of AI based productivity is going into security efforts.

This isn’t an effort that’s particularly visible to the outside world. What the outside world knows about it comes mostly from stories, not direct experience. When developers patch security holes, their intent almost always is to not change the user-experience. When that is the intent, it’s a slower process, because it requires educating users about new security mechanisms they need to participate in. Because that’s such a difficult thing to do, security teams have a very strong preference toward solving problems themselves without involving the users. It’s not always possible, but 90% of security efforts are invisible to users, and the next 9% are delivered as patches users see installed, but don’t pay any attention to.

The ingenuity matrix

I said in the opening that token usage is like effort: it tells you activity, not progress. To get from effort to expected outcome, you have to ask what the effort is for. Two questions do most of the work, and together they form a grid.

The first question is social alignment. Does the work create value the world didn’t have (positive-sum, pro-social)? Does it merely move value from one party to another (zero-sum, non-social)? Or does it destroy value — burn resources, or actively harm (negative-sum, anti-social)?

Alignment can be informed by our guesses of actors’ intent, but it’s not dependent on it. Our best bet is to act as an outside observer, guessing at outcomes. I don’t want to overcomplicate this though, this is estimation after all. Some significant pro-social value sometimes arrives from someone tinkering purely for fun. The social alignment is still recognizable from the outside, even when the actor wasn’t aiming at it.

The second question is scope, how far the work is reaching. Significant work aims at a real leap. Simple work aims at something bounded and modest. Naive work isn’t aimed at a productive outcome at all. Here “naive” describes the absence of a useful target, not the absence of a motive. Intentional waste is naive in this sense, it produces nothing of value, even though the person doing it has a very clear motive.

What you’ve just toured, security patches, reliability work, performance and cost tuning, is real value, almost all of it invisible to the people who benefit. Nearly all of it lands in a single cell: simple, positive-sum. It’s illustrative that so much of what is immediate is within simple or naive ingenuity. The first things individuals use AI for aren’t the significant ones. It’s the modest, shovel-ready, often-unseen things.

Map the rest against those two axes and you get an ingenuity matrix:

Negative-sum is not hypothetical, it connects back to security. The same drop in cost-of-effort that lets defenders finally burn down the security backlog also lowers the attacker’s cost. AI-assisted cybercrime is simple, negative-sum ingenuity, and the prospect of AI-scale biological or infrastructure attacks is the significant version. A large share of the invisible defensive work isn’t optional improvement, it’s the response to an adversary. The result is effort that is no longer avoidable, but also hidden, which delays the visible gains we’re watching for.

Significant non-social ingenuity ends empty. Non-social work can seem significant when under development. But one of two things happens. Either the work ends up leaking into pro-social, or anti-social accidentally, or it is copied and becomes trivial. Significance and neutrality are generally unstable.

Naive ingenuity is where the most visible tokens are burning right now and the least is being built. Failed experiments and aimless prototypes aren’t worthless — they build skills and occasionally surface something real, which is the option value the leaderboard bet was buying — but as a category they go nowhere by design. Because naive usage is so voluminous, and personal, it’s the most visible to the simplest forms of observation. That helps it dominate the “is this all a sham?” narrative.

Simple ingenuity has significant usage, but is quickly forgotten. The high volume usage is generally operationalized, contributing to security, reliability or operational efficiency. It’s soon forgotten, as it becomes a background effect. It doesn’t have the humorous, villainous story of tokenmaxxing. It doesn’t receive the personal promotion of the latest experiment.

One of the hallmarks of simple ingenuity, is it could be described as a backlog. The work may have been identified as desirable a long time ago, but with other competing priorities, it wasn’t prioritized. It may also not have been cost effective. One of the changes that AI brings is a change in cost-effectiveness. This activates this backlog, and you should expect early effects to burn this backlog down.

Simple ingenuity comes early and makes existing work more efficient. Sometimes this will show up as measurable revenues, but much is internal to companies. In that case it’s the token usage, the lower labor costs, or the higher quality that are the observations.

When AI enabled workers have a clear backlog, efficiency gains will flow into simple ingenuity to burn down the backlog. If the backlog results in priced or measured output, you’ll know.

Significant ingenuity will take longer to be identified, developed and deployed, especially the pro-social variety. The economy will reuse freed labor to create more value. That won’t happen immediately, as it may wait on hiring processes, training processes, or even the formation of new companies pursuing new products or business models.

Timing

While the development process is accelerated, the identification process retains most of its bottlenecks. Optimism may accelerate it. Idleness may accelerate it. But optimism and idleness may also flow into naive ingenuity, pursuing trivial goals without positive utility. There is a blurry area where naive ingenuity is experimentation. It may fail, but its failure may be necessary to build skills or discover significant opportunities.

At some point, a few things start to coincide. Naive and simple ingenuity will have built skills, ready to be exploited for realizing significant ingenuity. The backlog’s distraction fades as it burns down, and a new equilibrium raises the incentive to chase significant work — significance always carried more reward, but also more risk. But as cost-effectiveness decreases deeper into the backlog, avoiding risk becomes less attractive. All of these, in addition to the passage of time, predict a future wave of significant ingenuity that direct observation of measurements would fail to predict.

New output

Most of what we’d recognize as new output is significant, pro-social, and lagged. These are the life-changing things, and they’re the hardest to forecast. Your best guide might be a science-fiction novel, but of all the futures sci-fi writers have imagined, which do you bet on? Like flying cars, some things that look a step away stay out of reach far longer than expected.

It would be a mistake, though, to generalize from the failed predictions to all predictions. In many ways today’s information world already outruns older sci-fi imagination — the 1987 Star Trek: TNG depicted computers far beyond the 1966 version, and on the information front we’ve roughly met the standard it set for the 24th century already. The significant wave is hard to time and easy to underestimate at the same time.

What to expect

We should expect the AI industry to experience some pullbacks, then continue on. Whether this ever meets the bubble narrative is uncertain. I’m skeptical. Many pullbacks will be met by other accelerations. One experiment fails, another scales.

There isn’t one bubble, ready to pop, but multiple domes. Each industry, each set of users finds their value. While software remains so dominant, a failure in the software use case could be dramatic, but much of it is boring simple work that will continue to be automated for some time yet.

Much of the immediate term work is going to focus on the simplest, most invisible aspects. We shouldn’t discount the value there. Where it’s defensive, answering the negative-sum, like security, it has to be done. Where it’s part of more normal systems, it’s freeing resources, and developing skills and experience that will fuel more significant ingenuity in the future.

You do have to wait to see world changing effects. Software, as a model for implementing a workflow, will remain, and the general skills of software developers will be critical to this. Lines will blur, people will cross-over the lines, but ultimately the concept of software will continue to exist.

If the software dome does collapse, it will create structural damage, like all such events. Failed companies, layoffs, abandoned projects. Resources for naive experimentation would evaporate, and companies would proceed more cautiously. But a structure will remain. The burned-down backlogs that don’t un-burn, the skills that accumulated, the efficiency that keeps paying out, and the significant work just beginning to grow.

So, are we in a bubble? Will users and companies pull back on token usage, looking for value, discouraging wasteful tokenmaxxing? Will they react to pricing changes from model providers that close subscription loopholes that allow token usage in excess of what the same money would have bought per token via API? Yes, they will, but will that cause revenue drops that deflate the dome?

I don’t think so, there’s enough pending and developing work to fill the gap. Even if the significant ingenuity is still developing, the simple work is sufficiently valuable and important. But maybe those dynamics will return next year. If compute providers continue yet more expansions, they still might find them getting ahead of demand. There’s a lot of history to be written here. I’d just be careful about writing the ending first.

Sources

• Representation of professions in entertainment media: Insights into frequency and sentiment trends through computational text analysis; Baruah S, Somandepalli K, Narayanan S..

• State of AI, An Empirical 100 Trillion Token Study with OpenRouter; Malika Aubakirova, Alex Atallah, Chris Clark, Justin Summerville, Anjney Midha

• The AI Boom Has Entered Its 'Wait, Is This Worth It?' Era; Derek Thompson

• How much more software do we really need?; Noah Smith

• State of AI in the Enterprise The untapped edge; Deloitte

Related Articles

Avoidable Adversarialism

A job search is inherently adversarial within groups. Applicants compete against other applicants. Employers compete against other employers. But the relationship between an applicant and an employer is not inherently adversarial. There is a perfect match and those two matched pairs should want to discover each other.

When we talk about the adversarial nature of the modern job search, we should separate the inherent part from the avoidable part. The growing hostility between applicants and employers is avoidable. If we can avoid that we’ll collaborate. Job searches will resolve more quickly, saving money and pain. Workers will be happier and positioned to do the most good for themselves and their employers.

The devolution into adversarialism is clearly a consequence of limits to information exchange and processing. Employers started filtering resumes to minimize what they spent on hiring. Filters create something to defeat. If an applicant has a reasonable expectation that they’re a match for a role and they want it, they want to pass the filter. And since they may need to do this many times over, they have every incentive to do it efficiently.

The Historical Devolution of Hiring

Filtering predates computing. Long before anyone wrote a regex against a resume, recruitment teams read resumes by hand and made decisions from them. By necessity, those filters worked only with what appeared on the page, so candidates learned to structure a resume to pass them.

When resumes were reviewed by hand, there was some benefit to keyword matching. But there was a secondary method of passing the filter, weaving a story. A recruiter could lean on a strict rubric, or they could read for character, for an arc, for something that spoke to who the applicant was in a way that keyword matching couldn’t capture. That second channel encouraged investing in each application. Read the full job description, get to know the company, and write a cover letter specific to the role.

Human readers were also susceptible to word choice, which explains a lot of the fads in resume writing over the decades. Recruiters responded to certain modes of expression. “Synergistic” is the classic example, but the pattern runs deeper than any single buzzword. Applicants would notice an opening, exploit it. When recruiters realized they were being exploited a countertrend would set in.

The thing about that system, for all its gamesmanship, is that it evolved slowly. Trends and countertrends took hold one recruiter at a time. They might spread through HR conferences, trade publications, networks, but every update was individual. The arms race was real, but it moved at human speed.

Computerized Filters and Keyword Stuffing

Computerizing the filter degraded two things. First, the depth of processing collapsed. Matching became keyword-driven. Stories, arcs, and the patterns that needed a human to understand were no longer assessed.

Second, the motivation to invest in each application diminished because that investment was demoted to the second tier. If you passed the filter, then someone might read that and you’d benefit. But it was only rarely worth the effort with that extra distance.

This second part triggered an adversarial response. It became obvious to applicants that they were in a game, and that the game was everywhere. Callback rates fell from 10+% in the early 2010’s to 2% in the 2020’s. The decisions weren’t fair and weren’t optimal, so playing the game to its fullest felt like fair play. Resumes filled with keywords. Applications per candidate increased.

At first, human reviewers reacted by turning away obviously stuffed resumes forcing custom resumes per job with just the right keywords. Applicants had to balance to make these custom resumes not read like keyword-stuffed garbage to a person. This ate enormous amounts of applicant effort.

One page norms were initially strongly enforced. But then they gave out. The keyword filter wants more, because keywords need room to slide in while still appearing natural. Mainstream career advice shifted to two pages to leave room for the terms automated systems scan for.

The human reader, conventional wisdom holds, wants less, a clean document that doesn’t read as stuffed. But when a resume-writing firm ran a hiring simulation, recruiters preferred the two-page version by better than two to one.

Candidates were in a bind. Some filled resumes with 1-pt invisible white text, to please keyword filters and avoid reviewer reactions.

Then generative AI arrived and solved the bind. Applicants could generate a distinct resume for each role, loaded with exactly the right keywords, with acceptable writing quality, in seconds. But the story route was never revived, because there was no longer anyone reading for a story. Recruitment’s early stages became a machine with no room to absorb narrative.

Goodhart’s Law

There is a common description for that process, Goodhart’s Law. When a measure becomes a target, it stops being a good measure. Hiring teams have continued looking for targets to measure, but each has receded in usefulness.

Generative AI didn’t introduce Goodhart’s Law to hiring. It did remove the final exceptions. The cost of optimizing acted as a natural brake. Producing a tailored, plausible resume took effort, so people did it selectively. Now it costs nothing.

A single posting now draws hundreds or thousands of applications, because every applicant is applying to hundreds of postings. That drives employer response rates toward zero, which removes any reason for an applicant to invest care in a particular application, which pushes them to apply to even more roles, which raises volume again.

Detectors to the Rescue?

Instead of looking to solve the root problems, many hiring managers suggest they’d use another filter: automatically rejecting AI generated resumes. This is almost certain to turn away many qualified candidates. At best, it leaves behind those who invested the most time into the process. That’s a poor predictor of candidate quality. It entrenches gamesmanship too. If you’re tempted to make the argument that candidates investing more time have more confidence, consider the counter; your job may not matter to the most rational candidates. You may just be selecting for the candidates not smart enough to figure out the game.

Even the detector is suspect. Initially tools could hide the AI provenance, at the cost of quality degradation. Initially this was easy, then detectors got better. But even now, I can say, it’s not too hard to modify AI written text to not be recognized though. It would slow candidates down, but what’s the value in that to an individual employer?

While there’s some good advice to applicants to worry about accuracy with AI and resumes, this filter is low value. Turning away a quality candidate that used AI does not improve hiring quality. It won’t change candidate norms. At best it’s a guess that AI resumes are less accurate. That’s a questionable assumption due to the pressure on non-AI resumes to inflate.

More Requirements

So people propose fixes. But almost all of them share the flaw that they inflate candidate costs and arbitrarily filter qualified candidates out.

Consider degree requirements, which I wrote about separately in Or Equivalent Experience. Employers routinely demand credentials more restrictive than what their existing workforce holds, and automated screening makes that mistake worse, not better.

GitHub portfolios are a common suggestion. But building something good or great takes time. Most paid work ends up in confidential repositories. Those few jobs that produce public GitHub history benefit. For everyone else, it’s a free labor requirement.

You can spot an amazing GitHub a mile away, but the supply of engineers with that will be low. Categorizing good/average/bad is harder. Are you going to assess generally, or be tech specific? Most hiring focuses on tech specific skills. While I think first principles lead to more actual job success, they are harder to spot. If a candidate has a beautiful data-analysis project in Python, but an employer demands a message-processing stack in Rust, the candidate costs go up further. Any self-made project is about more than engineering skill, it’s also about ideas and time.

None of that is fatal, but the later gaming will be. If the signal is genuinely valuable and engineers are desperate enough, people will pay the tax. But what happens after it becomes standard?

Every standard encourages gaming. Simple things first, like using an underground agent to generate a unique, reasonable-looking GitHub project on demand. The escalation is next; hiring teams create a detector. Don’t expect the adversarial dynamics to stop at Level 1.

Any tool you hand to one side becomes a weapon in the arms race rather than a resolution to it. A new filter for employers invites better evasion from applicants. A better evasion tool for applicants invites better filtering from employers. One-sided solutions, however clever, pull toward competitive dynamics, because the other side never agreed to it and has every reason to defeat it.

My Experiences in Interviewing

Early in my time at Amazon (2015), I was active in the hiring process, conducting around 60 interviews in the first 2 years. I was surprised at how quickly I was pulled in. But AWS was hiring fast, and Chicago was a new office. The process had promising ideals. One favorite was the idea that your job was to draw positive proofs out of the candidate, not search out flaws. You wanted to hear them describe their approach to a problem or challenge that demonstrated their understanding of the right path forward.

I’ve found Steve Yegge’s writing compelling, and there is shared Amazon experience, so I was drawn to his recent The Last Technical Interview. Yegge was engaged deeper than I was. I avoided the bar raiser path. Imposter syndrome is one reason. I am proud of a number of people who I was part of their hiring loop, or mentored. If there’s anything that should give you good impressions of your ability to interview, it should be those. But I wasn’t feeling it at the time, and so when I found a technical problem to focus on, I latched on to that, and pulled away from hiring, doing a handful per year.

A less optimistic viewpoint was noticing interviewers that took pride in turning down +90% of candidates. If I approved 25% and they turned out well, what are the chances that those turning down two to three times as many weren’t turning down qualified candidates? There wasn’t any data to show their 10% was better than my 25%.

So I had to chuckle at the reflection where his team of interviewers voted not to hire 2/3rds of themselves. Made me feel a bit better about those nagging doubts that should be part of any difficult decision like this.

Interview Stages

The Last Technical Interview does get at something real though. His diagnosis: the interview has been broken for fifty years, even Google’s best interviewers couldn’t agree with each other or with their own past judgments. The whole apparatus is an elaborate attempt to generate signal, but consistently falls short. His prescription, the “campfire” model, is to bring people in to do paid, real work for a few days, then decide.

What makes his version more interesting than GitHub is a detail he calls counting the work twice: the candidate walks away with a permanent, portable record of what they did, stamped by the employer, whether or not they get an offer. If I get the concept, the employer would be doing a service for the candidate, and for all other potential employers. Both want the information a “stamp” would provide. Well done, this could counterbalance unnecessary adversarial tendencies that have accumulated.

Because it’s employer-certified rather than self-reported, it is harder to fake than a repo an agent can spin up. You’d have to find employers who hand out stamps to everyone, and Yegge is right that failure mode is self-correcting: a company whose stamps mean nothing has stamps worth nothing.

The network effect here is not one-sided. Both sides care about credibility. Gaming still exists, but if your stamp comes from a company vulnerable to gaming, its value diminishes. The signal lives or dies on credibility. Also, the more personal process offers fewer structural approaches to gaming.

There’s definitely some details left unspecified. I worry that some liability concerns could kill it. You still need to screen those invited to a “campfire”. You still have to scale.

An instinct of mine is that this requires some collaboration across the employer space. The campfire with credential stamp would do this, if it scaled. It would be challenging for every employer to create this credibility though if it’s by word of mouth.

An idea here is a company that does this as a service for an industry or multiple industries. This could solve the scaling challenge, if this is a passthrough. In some sense, this is what universities are: a 4-year campfire you pay tens of thousands of dollars to attend. I think the flaw is obvious, they are too expensive and too inflexible. In a sense though, they might have the best infrastructure for this impossible mission, should they choose to accept it.

The Shared Road Out

Which finally points at the answer to the question I opened with. If you want a solution that doesn’t just escalate the war, it has to be something both sides are actually happy with, or one side will fight it. One-sided efficiency tools breed counter-tools. A solution has to be a collaborative tunnel from the start, or it gets pulled back into the field of competitive options.

The trouble is we’re stuck in a bad equilibrium that’s individually rational. It’s a coordination trap, structurally a prisoner’s dilemma. A better equilibrium exists, where applicants apply selectively to roles they fit and employers actually read what comes in, and everyone would be better off there. But reaching it requires someone to move first and trust that the other side won’t simply exploit the opening. Right now nobody trusts that, for good reason. Applicants won’t invest in a tailored application when the expected response is silence. Employers won’t slow down to read carefully when they’re drowning. Both behaviors are sensible. Both perpetuate the trap.

So who moves first? Probably the employer, for an unsentimental reason: the employer is the scarcer and concentrated resource. A small show of goodwill from the side holding the scarce thing tends to get reciprocated. A well-known employer with a reputation can start something new. When candidates learn of it, they would opt-in. But it has to be something that builds collaboratively, rather than whittles away negatively.

I don’t have a clean answer. But I think the shape of the answer is clear enough, and it’s not a better resume or a better filter. It’s changing what each side gets from honest engagement, so that participating sincerely beats gaming.

Yegge’s instinct toward “gravity” is the right one, even if the mechanism is unfinished: make your rejection valuable, and candidates stop treating you as an adversary to defeat. A few partial paths point the same direction. Employer-certified, portable records of real work, if their credibility can be established. Skills-based hiring done deliberately rather than as language stripped from a posting. Two-sided interest signals like Greenhouse’s MyGreenhouse “dream job” feature, where a candidate can mark one application as special, though even that is still half a handshake until the employer offers something reciprocal. Something as basic as committing to send a real response.

None of these escapes the adversarial dynamic completely. Each one can be gamed at some level, and you should assume someone will try.

But that’s the wrong bar. The question isn’t whether a solution is immune to gaming. Nothing is. The question is whether it moves the incentives so that honesty is worth more than evasion to both sides at once. The inherent competition, applicant against applicant, employer against employer, isn’t going anywhere, and that’s fine. The unnecessary war, applicant against employer, is a failure to generate signal collaboratively.

Related articles

With that in mind, I have a reaction to seeing so many job postings in the software industry that look like this:

• Bachelors degree or equivalent experience

• 10+ years of experience building software

The first line is the requirement. The second line is a fig leaf — a way of technically not lying while still signaling what the process actually rewards. It was always a questionable practice, but AI is supercharging the impacts of this mistake.

The Logical Problem

Let’s start with the plain English issue, which is almost embarrassing once you see it.

From a logical point of view, these statements are redundant. 10+ years of experience will always be equivalent experience. There is no interpretation under which a decade of real-world engineering doesn’t constitute equivalent experience to a four-year degree.

The problem runs deeper than embarrassing logic. The reality is that equivalence is probably fiction. Most hiring managers, I’d wager, didn’t author this language and don’t think about it. But it activates biases in recruitment teams, offering a lazy shortcut, and sending the wrong message.

Automation and AI are Supercharging this Mistake

This was always a mistake, but it’s becoming more critical. Recruitment teams scanning resumes will be drawn toward an education section more readily than to calculating the equivalent experience. Automated tools in applicant tracking systems (ATS), including AI, have the same weakness, often more so.

How is a typical large language model (LLM) going to process these statements? The degree is a binary, well-defined data point with a clear answer. “Equivalent experience” is the opposite: fuzzy, context-dependent, requiring judgment. Even if the LLM evaluates the two statements correctly, it could make another mistake, treating the “or” as an “and”, or treating the two statements as components of an overall “closest match”.

These mistakes could cause a filter to fail, or it could cause a lower score. If the three clauses are processed independently, the degree holder gets 3 points, other candidates 2. If qualifications aren’t scored equally, one earlier in the list will usually get more weight. Even if a system doesn’t intentionally add a scoring system, a reasoning model could create one on its own.

More advanced systems are less likely to make these mistakes, but recruitment teams may not use the most advanced systems. This might be motivated by cost, or adoption started before systems advanced. They may even be non-AI, simple text analysis.

In this environment where job postings get hundreds of applicants, because every applicant is applying for hundreds of positions, a naive filter or scoring system can have a dramatic impact. The more nuanced aspects of a resume that should make you a top candidate, may never get processed. That means fewer interviews, dramatically reducing the probability of a successful interview.

What should you do?

The simplest thing to do is remove any such text from job descriptions. Since they are duplicative, and creating unintended effects, just delete them. And it’s not just those “or equivalents”. You should rethink degree requirements in general.

That’s not enough though.

If you’re not deliberate about this when working with your recruitment team, you may get no change or the wrong change. If they remove “or equivalent experience”, keep an internal filter or priority ranking, or just act on their own biases, you may get no change at all.

Should that be your intent? You might ask if you’re better off with the filter. There’s a couple things you can do to validate that this isn’t sensible. First, you might want to familiarize yourself with the actual rates of postings and workers. In many cases, the postings are more restrictive than the workers. Something seems wrong if it is true that if you had to rehire the entire workforce, 20% would be excluded. If 66% of those without a degree already doing the job wouldn’t get a job, you have to wonder.

Is that not enough? Then ask yourself, do you know something about the other 20% that makes you want to exclude them? I suspect you don’t. Once on a job, no one asks, but most assume.

Here’s the test: which of these has happened for you more often when working with professionals in their field?

• You learn someone doesn’t have a degree, and say, hmm, would have never guessed that! Bob is so smart.

• You learn someone doesn’t have a degree, and say, oh, now that explains it, I always wondered why Bob was so dumb.

It’s possible you have little data to work with, because people working on solving a hard problem don’t ask that kind of question. It does come up socially on occasion. If you do have a data gap, it’s not hard to close, just ask a few people. You’ll have to ask 50 people if they have a degree to find 10 that don’t. Because both the affirmative (no degree = smarter), and null hypothesis (no predictive power from degree), are on the same side, it doesn’t take a large same size to disprove the implied assumption.

The strongest case to put a degree on a qualification list is early in careers. This is where “or equivalent experience” actually makes sense. A candidate who just spent four years studying, 2 of it on practical development work is reasonably comparable to one who spent 4 years building products

But AI is hurting college grads. Should we give them an advantage?

There’s a pair of stories circulating about how bad recent grads have it in the job market. Should we give them an advantage, ensuring their expensive degree doesn’t come without rewards? There’s two data points cited on this topic, unemployment and underemployment. Both have flaws in their representation.

Underemployment

The underemployment data point is weakest, and generally just demonstrates a blind spot for those circulating it. The recent number is 41.5%, which does sound horrible without context. But all data should have context. This is not news, it’s a failure to understand the data. If I heard a number like that, I’d ask.. Well what is underemployment.. And what was it like in the past? It’s not hard to find the original source.

What you can see below is that 41.5% is lower than most historical periods. Is that worth freaking out over? No.

If you read the explanation, you understand why the number is this high. It asks people, working in the job, if a college degree is necessary. If you polled me about any software engineering, I’d answer no. I’m sure it is on the list, because probably the majority of the 80% of software engineers who do have a degree are answering yes.

While software engineering is likely on the list1 I can think of a lot of other things people commonly go to college for that even the college grads would likely answer no. Went to school to study art or music? Would you classify that as requiring a degree? Filmmaking? Social worker? What about some that might be on the list but are debatable? Journalist? Newswriter?

There’s also some majors on this list that appear because it’s tracking recent bachelor grads, but these majors usually go on to higher degrees (JD), law, business (MBA).

Unemployment

The unemployment story is more nuanced, but still represented as more than it is. It does show a change, but is it worth the reaction it has received?

What news stories highlight is that the light blue line (recent college graduates), has never been above (all workers) in the past. But is this a fair comparison? After all, a recent college graduate by definition starts unemployed. At some point in that 5 year span, they graduated, and started job seeking. It’s an apples to oranges comparison. All workers are the incumbents. Some of those workers may have been working the same job for the last 10 or 20 years. Since unemployment rates are based upon those “seeking” jobs, what you’re comparing here is first a fraction of all workers who lost a job recently, and then a sub-fraction of those who had difficulty finding a new job, vs. all college graduates, a fraction of which had difficulty finding their first job.

Shouldn’t “recent college graduates” be compared against “young workers”? There’s still a story there, in that the gap has shrunk, but the story isn’t that college graduates are getting a raw deal, but that there’s more equity between with/without. That’s a lot harder to make a decision about. How big a gap do we expect here? Don’t we want employment opportunities for those without a degree?

The Blind Spot

Partly I’m calling this out because it’s a current story that people are getting wrong, but partly I’m also demonstrating a general blindness that seems pervasive in what I’ll assume are mostly college grads discussing this story. They assume that the worlds of college grads and non-college grads are so universally distinct that there would be no overlap here. They assume that the only way you could be prepared for a job fit for a college grad is the same path.

The reality is that in terms of learning, college is just a convenient path, with a lot of resources laid out in front of you, no other responsibilities, and encouragement to follow a plan. College is many other things, a credentialing mechanism and an opportunity to build social networks, for example. But in terms of learning it’s not magical. You learn by consuming information and solving problems related to what you’re learning, and that opportunity has a lot of entry points.

Conclusion

College should be valuable to those that go. But its value should always stem from the learning it enables. Learning comes from many sources, the college experience is simply a well-resourced and well-structured source. Job experience is valuable in its unique way. In both cases, you have to make those experiences count. Your curiosity, your interest, and your hard work are what translate experiences to learning. A college experience, when done well, should be able to do this more effectively than a job. This is because enabling learning is its primary objective, whereas job experience has to compete with other objectives.

All that said, laziness will make any experience intellectually unrewarding. It’s worrying the degree of laziness applied to the hiring process and the data behind recent news stories. We should do better. We shouldn’t blame this on AI, that would be lazy too. Lazy slop was a problem before AI. It has a deeper cause. Maybe it’s growing, or maybe it’s always been with us. Whatever the case, honesty will get us farther than scapegoats.

Barriers, like degree requirements, enacted out of laziness or to create a condition of privilege are a mistake. We shouldn’t use them. In my opinion, degrees shouldn’t matter if you’ve already successfully done the job. This is doubly true if the job is more complex than anything schooling would have covered. In theory, it seems many employers agree, as the terminology, “or equivalent experience” has been common. But words are one thing, practice is another. A lazy translation of intent to practice that fails to meet the goal is harmful. This is but one of many, but hopefully I’ve made it clear how this one is a mistake.

Sources:

• Dismissed by Degrees: How degree inflation is undermining U.S. competitiveness and hurting America’s middle class; Joseph B. Fuller, Manjari Raman; Harvard Business School; 2017.

• Underemployment in the Early Careers of College Graduates following the Great Recession; Jaison R. Abel and Richard Deitz; National Bureau of Economic Research; 2018.

• Skills-Based Hiring: The Long Road from Pronouncements to Practice; Sigelman, M., Fuller, J., Martin, A.; Burning Glass Institute; (February 2024).

• The Labor Market for Recent College Graduates; Federal Reserve Bank of New York; 2026

Related articles

In general, market actors do not have great incentives to protect against catastrophic risks. They are massive negative externalities, often dwarfing the balance sheet of any individual firm. Say Anthropic releases a model that a malicious actor uses to conduct a cyberattack that does $5 trillion dollars in damage. Anthropic is only worth $800 billion, so if they get sued for $5 trillion, they are already well past the point of insolvency. A catastrophic harm may well already be “lights out” for Anthropic, or any other company, so there is little incentive to avoid them, if doing so entails real costs in the present day.

He’s right about the structure of the problem — but “little incentive” understates the precision available here. AI companies do have incentive to avoid catastrophic outcomes, just systematically less than society needs them to. That gap can be quantified, and quantifying it points toward what a corrective policy should actually look like.

The concerns he’s talking about — catastrophic risks — share a structural feature that distinguishes them from others: they are lumpy. A single catastrophic event, rather than a diffuse trend. The incentive is quite large, but not as large as it should be. These dynamics are worth exploring, as those ultimately shape if and how we structure a response.

Consider a hypothetical AI company, worth $800 billion. Now consider a hypothetical event causing $5 trillion in damages. If this event happened, that AI company would be out of business, so they have an incentive to prevent it. But how much incentive? The most they can lose is the whole company, so $800 billion. Since a lot of that is goodwill, in reality, losses become irrelevant earlier. For the sake of example, we’ll say $400 billion. If you had to pay half your market cap, you’re not worth $400 billion, you’re bankrupt, and worth $0. All claims greater than $400 billion have equal impact, since each produces the same outcome, a total loss.

This creates an imbalance between societal goals and the AI company’s goals. That imbalance could lead to underinvestment in safety, or risk taking that is out of alignment with societal goals.

We can quantify this imbalance, by modeling a damage cap in expected value calculations. If our $5 trillion event has a 1 in 10,000 chance of occurring, the uncapped expected value of avoidance is $500 million. With a damage cap of $400 billion, it’s only $40 million. Society should want that other $460 million in incentive to be shared by the AI company, but without an arrangement, it’s not.

Refinements

I used a simple model above, with linear effectiveness of investment in safety. It isn’t linear. In a linear model, spending $500 million reduces risk to zero, and $40 million reduces it to 1/12th of that, or one 1 in 9,166. But we could imagine, in fact we should expect, that the first $40 million does more than the next $40 million. Maybe the first reduces the risk to 1 in 100,000, and the next to 1 in million. It’s the same proportional improvement — 10x. But in the first case it reduces the risk from 100/million to 10/million for a total reduction of 90/million. The second case reduces from 10/million to 1/million, for a total of 9/million reduction.

To illustrate, I constructed a model that used logarithmic decay from the initial 1 in 10,000. In this model, under their default incentives, the AI company would want to spend $13.3 million to reduce their expected risk from $40 million to $8.7 million. But the societal risk is still $122 million at this point.

The goal of a corrective policy would be for the AI company to act upon the societal risk, which justifies spending $35.2 million to reduce the societal risk to $8.7 million.

Further refinement considers whether organic spending is more efficient than regulatory-induced spending. For example, if regulatory-induced spending had half the effect per dollar as organic spending, not only would the spending go up, but the residual damage would be higher.

In Scenario 1, all spending is equally valuable. In Scenario 2, the company spends efficiently up to its capped motivation, after which each real dollar buys only $0.50 of effective safety. And finally in Scenario 3, all spending is at 50% effectiveness.

You can explore other scenarios in the linked single page app (GitHub Repo). You can experiment with different decay functions, damage sizes and company sizes.

Organic Forces

We’d also be doing ourselves a disservice if we didn’t recognize the outstanding work that AI companies have done — Anthropic most of all. Commercial incentives alone aren’t sufficient. The stories about the founding of DeepMind and OpenAI make clear that good intent has played a positive role. But in a commercial world, you can’t depend on good intent to reliably show up or win internal contests. The pressure to go off track is substantial. For something this important, that’s a lot of trust. We want to use these forces, because they are efficient, but must not be naive either.

We don’t want to take organic forces for granted. If we assume they don’t need support, they might disappear. If we don’t acknowledge their value, we might strangle them.

Small Firms

The alignment problem becomes more acute for smaller companies. What if a smaller startup, with none of the weight of a larger company — little to lose, and everything to gain — rushes ahead, and skips best practices that avoid harm?

The leading labs are large (Anthropic, OpenAI and Google), but we shouldn’t take that for granted. Frontier model training costs keep going up, but the costs for a particular level of capability keep going down. DeepSeek proved that moats are much shallower than assumed.

You do want to avoid locking out startups, but also need a baseline that ensures safety isn’t skipped. A first step here is ensuring safety practices are shared. That lowers their costs in pursuing safety.

The current voluntary norm — leading labs sharing safety methodology despite having competitive reasons not to — is a favorable state of affairs that formal structure can preserve and extend. It will take organization to make it work at a deeper level. Sharing details of some safety practices publicly can add risk, so a well-trusted network for sharing enables more than just the public domain approach. Formalizing sharing as a condition of operating at the frontier, both preserves what already occurs, and can extend it more deeply.

Regulatory Shape

The model makes the policy objective concrete: close the gap between what the company is motivated to spend and the societal expected value, without crowding out the organic safety investment that’s already happening.

A naive response assumes insurance is enough, and the challenge is finding a large enough reinsurer to pay out. An even more naive response assumes this challenge can be fixed by inserting the federal government as a backstop to the insurance. The flaw in this thinking is that it makes society responsible for paying itself back for harm done to it. This won’t work. The harm would have been done. Society would pay for the majority of the consequences of the gamble the AI companies made.

These dynamics suggest that effective regulation needs balance, in order to use organic forces, and yet also not leave a gap. Dean is right that it improves the case for government involvement. If there’s a gap between the company’s incentives and the societal incentives on a topic so important, we should align those.

An industry body that both shares security practices and sets standards is a start. Shared excess liability amongst all AI companies would add to existing incentives. If one fails to prevent harm in a small way, that company fails alone. If one fails in a big way, they all fail. Expanding the pool in this way is better than involving the government, as these are the players with the ability to influence the risk. Those incentives will encourage maintaining quality standards, but keep the standards moored to efficiency and effectiveness.

That’s still not enough though, so a government body above that respects the value of organic forces, would be a second step. The challenge here is how to prevent this body from losing interest in efficiency. It’s natural for them to be interested in effectiveness, but efficiency comes with more difficulty. If standards ignore efficiency, you undermine the organic forces and risk taking a step backwards instead of forwards.

What doesn’t work?

The framework I’m discussing, can appear to be a compromise between two points of view. That’s not the intent. There is no intent to choose a middle point, in order to satisfy two points of view. I think the merits of this model fit without any politics.

The model does however balance multiple forces, and is not aligned with any maximal plan. That type of balance only makes sense if the maximal plans aren’t reasonable. To make it clear, I don’t support any maximal plans. It will take additional posts to flesh out why, and others have defended these points independently. But in the light of outlining my thinking, the basic is:

• AI bans: You have no chance. You have no global solution. It’s not a good idea in the first place, as AI will be very useful, but that’s not the biggest flaw. The biggest flaw is all of the partial wins - company X refuses to use AI, country Y bans AI - they all fail in the end and don’t contribute to any goal aligned with the best case for a ban.

• No regulations: Clearly something is needed here. This group is somewhat of a strawman though, as even people like Dean Ball see a role for regulation. The better critique is that there are many people who are implicitly “no regulationists”, because they oppose everything proposed and don’t put together enough to actually do something.

• Top-down regulations: Strangling organic safety efforts in top-down paperwork is a surefire way to fail. That doesn’t mean there isn’t a top, but it does mean, it can’t be total, and since it’s starting later, it should expect to start small and iteratively find its fit.

Clearly, there are more details to cover here. I’ve only touched on one dynamic that sets an overall tone, but you’d eventually need a list of initial best practices, and an expert-led group to maintain them. You’ll need a mechanism to choose that group, and a list of powers and limitations that define how they work together, and resolve conflicts. I’ll leave those questions for a future post though.

Today I wanted to talk about some challenges that complicate preventing malicious use.

Jurisdictions

A first failure of existing law is jurisdictions. The world has rogue states, lawless states, and aggressor states. These either turn a blind-eye toward harmful activity, lack the capability to enforce laws, or actively create targeted harm themselves. Existing laws cannot reliably reach actors that hide in these jurisdictions. There is a justified effort to close those gaps. There is slow progress. Sometimes gaps reopen. Because it’s a long running effort, we shouldn’t expect a near-term resolution, and treat it as a reality we must mitigate.

If we can’t target the originator of malicious acts, we can try to deny them tools. We should recognize the efforts of AI companies here, which have been substantial. But, these efforts are hindered by two background stories: open models and privacy. To deny tools for malicious use, you must first detect malicious use, or intent; open models and privacy complicate both of these.

Open Models

Open models are models released openly. Without going into too much detail, the key quality is users can run these anywhere. Closed models don’t give users that ability, and users have to interact with them as a managed service. That layer of management provides the key capabilities that enable monitoring and denial.

Open models once openly published, have no or limited ability to monitor. There is very limited ability left to control them, mostly centered around denying access to sufficient compute resources.

The largest collections of compute are at cloud providers, but there are still ample compute resources outside of cloud providers — in private data centers, colocation facilities, sovereign national infrastructure, and increasingly, distributed consumer hardware. Even for cloud resources, the nature of providing compute, rather than a managed service obscure the most effective means of monitoring. By design, cloud providers give customers using compute a heavy dose of privacy.

While open models have their justifications, from the realm of preventing malicious use, they are a challenge. It’s of some comfort then that open models are less capable than closed ones. This reduces the capability harmful users have access to. Since some aspects are adversarial, the advantage of closed models provides defenders an advantage too. This applies most significantly to cybersecurity.

Will open models stay less capable than closed ones? We could, across cooperative jurisdictions, enact regulation to ensure that — but if a non-cooperative jurisdiction has the capability to create more powerful models, we’d lose that control. China is the jurisdiction most likely to both have that capability, and make independent decisions.

Privacy

The second background story is privacy. The default state of anonymity on the Internet has costs. Privacy advocates attempt to maintain this state. I, like some others, believe the costs of this anonymity as a policy are too high. This isn’t specific to AI, but it does relate.

We have tied the hands of security teams and mostly delivered theoretical privacy. Where privacy matters most, such as totalitarian countries, the privacy is undermined by local realities. Privacy advocates don’t have a voice here. They win political contests where there is the least need for them, and lose where there is the most. It’s a tough choice, but I think we’re not making the right choices.

We should be pragmatic, but we’re idealistic. In some cases, privacy measures accelerated accumulation of data for malicious purposes. When countermeasures can’t be due to obscuring the lowest layers of a technical stack, we fail to achieve privacy and prevent harm. When formal data-sharing is prohibited, informal systems take their place, and predictably result in harmful breaches.

If service providers always knew who was using their service, they’d be able to deny access to anyone detected acting maliciously in the past. But the internet offers too much anonymity. Providers can shut down an account, but without accounts tied to a real identity, a new one can be created. The current standard among AI companies is too lax about this. We could make it more costly for attackers to maintain access.

Conclusion

Jurisdictions, open models, and privacy are features of the world we must work within — but they are also policy choices we can influence. The uncomfortable reality is that these three forces compound each other. Open models place powerful tools in jurisdictions beyond legal reach, while anonymity makes it difficult to detect or deny access to bad actors even where laws do apply. Treating any one of these in isolation understates the problem.

The path forward requires accepting some hard tradeoffs. Meaningful identity verification will feel like a concession on privacy — because it is one. Regulatory constraints on open model releases will frustrate researchers and developers who have legitimate reasons to want them — because the benefits of openness are real. Coordinating across jurisdictions will be slow and incomplete. None of these are reasons to avoid acting, but they are reasons to be honest about what any given measure can and cannot achieve.

What’s not acceptable is the current default: deferring hard choices while treating anonymity as an unqualified good and open access as costless. The tools for harm are improving. The window for shaping how they’re governed is open, but it won’t stay that way.

Now, Trump and Republicans also try to just directly lie, suggesting gas prices aren’t high, won’t be high for long, and hey, this was all necessary. That type of lie only works directly with the most deceivable. It may help keep some of them from waking up, but the real purpose of that type of lie is to make the more subtle lie less obvious.

The more subtle lie, equating price spikes under Biden and Trump, doesn’t try to convince the public not to blame Trump and the Republicans for this spike, it tries to convince them to ignore it, suggesting, you still need to vote for us, price spikes would be worse with Democrats.

The reason no one should accept that argument is Biden didn’t cause those spikes. The simplest way to understand how poor the argument is, ask how much oil did the Biden presidency remove from the market? I can tell you, it wasn’t 15% of global supply. Probably not even 1%. I think you’d remember that story if it happened. It didn’t. Instead a different story happened.

When the pandemic occurred, global oil consumption dropped, and prices with it. Producers stopped prioritizing new supply. Later consumers started to return to prior patterns, and producers lagged behind the re-emergence of that demand. The price spike was a simple reflection of that, not a result of any US government policy that removed production.

The two major reductions in production since 2010 occurred under Trump. I wouldn’t blame Trump for the first production drop in 2020. It’s stupid to try and blame presidents for everything. But it also clearly isn’t Biden’s fault.The spike in prices during the rapid period of demand recovery shouldn’t be either.

Having some awareness of who made what choices and why is a better method than just direct association of prices to current governments. Instead it’s obvious the production drop was producers responding to the consumption drop.

Coming back to today, another ounce of blame that is deserved, is the destruction of the US electric vehicle industry, and more generally the entire clean energy industry by Republicans. EV Sales increased by five-fold under Biden, and have stalled/declined under Trump. If you want low gas prices, it helps a lot if your neighbors aren’t using much. Eliminating the $7,500 point of sale credit for these vehicles was a strategic and tactical mistake.

The on-off engagement with renewables is another strategic mistake that’s related. While renewables direct impact on gasoline prices are small, the impacts on batteries flows over into diesel generators and the EV industry. The big strategic failure of on-off engagement has been to fail to develop a valuable industry, allowing China to dominate.

Ultimately, the primary determinant of today’s prices is obvious. Trump’s war and its repercussions.

Last year, I started a series of posts on trust, with Money: More Than Just Stuff, It’s Trust as the focal point. I felt 1-year was a good time to update this account, and extend it.

If you ask an economist what money is, they will likely give you a functional, three-part definition: it is a unit of account, a store of value, and a medium of exchange. If you ask a dictionary, it will tell you that money is “something generally accepted as a medium of exchange, a measure of value, or a means of payment”.

These definitions are perfectly workable for daily life, but they contain a loophole. Defining money as “something generally accepted” describes a symptom, not a cause. It relies on the word “something,” anchoring our minds to physical objects—gold, silver, paper, or digital ledgers. But the link between an object and its status as money would be severed by a loss of acceptance.

The basic definition is useful, but it’s clear economists do not consider it sufficient. Just this week Stephen Dubner of Freakonomics, released Part 1 of a two-part series, What is Money?, covering an adaptation to an oratorio of Adam Smith’s The Wealth of Nations. Money must be something of a higher order than temporary objects we attach it to.

To understand what money is, we must look past coins and paper. Trust is not merely a social feeling; it is a vital component of cooperation that requires immense social energy to build and maintain. While trust can theoretically be extended freely, reliably creating it at scale carries a real, limiting cost. Money is humanity’s greatest collaborative technology—an accidental invention that survived and spread precisely because it lowered the cost of trust.

The High Cost of Trust

To understand why money is a technology of trust, we have to look at the world before it existed. How do human beings coordinate the exchange of goods and labor without it? Historically, humanity relied on two deeply flawed workarounds: Barter and Kinship.

The Barter Evasion

It’s common to think of barter as a primitive ancestor of money, but it is better to think of it as an attempt to trade without trust.

Imagine you have a surplus of hay, and you need milk. You find a farmer with milk, but he doesn’t need hay; he needs firewood. To make a successful trade, you are forced into a complex puzzle. You must find the person who has firewood and needs hay, trade for the wood, and then return to the dairy farmer. Economists call this the “double coincidence of wants.”

Because there is no trust carrying value across time—no mechanism that says “I gave you milk today, I owe you value tomorrow”—every transaction must be settled immediately, item-for-item. In a physical sense, barter avoids the need for trust, but the friction of searching for perfect matches makes it impossible to scale.

There is also the problem that the neat and tidy view of trust in barter being solved by the direct physical transfer of goods, is a bit of a myth. Anthropological studies of societies without money show trust issues relating to trust on fairness of exchange, both during and after negotiation.

The Kinship Tax

Because barter is so inefficient, early societies rarely relied on it for daily survival. Instead, they relied on kinship networks. Barter was used mostly outside kinship networks.

Early societies solved the trust deficit through deep, interpersonal relationships. You do not barter with your brother, your cousin, or your tribemate. You give them your surplus milk today, trusting implicitly that they will provide you with firewood next winter.

This creates a high-trust environment, but it comes with a fatal flaw: it is unscalable and exclusive. Maintaining deep, bilateral trust requires immense social energy. You can only maintain it with a small, localized group of people—a single-layer network naturally capping around Dunbar’s number of roughly 150 individuals. While societies can attempt to force kinship to scale by creating additional layers of hierarchy, each new layer adds complexity, instability, inefficiency, and immense human costs. This dynamic imposes what writer David Oks has called a “Kinship Tax.” It crowds out the ability to trust strangers, trapping economic coordination and human development at a deeply local, tribal scale.

The Technological Leap

Before money, humanity was trapped between two dead ends. We could choose barter, which offered zero trust and infinite friction. Or we could choose kinship, which offered high trust but severely limited scale.

Bilateral trust—knowing and trusting the specific person you are trading with—simply became too expensive to produce as societies grew.

Money was the technological breakthrough that bridged this gap. It allowed humans to substitute the expensive, unscalable trust of kinship for a cheap, highly scalable institutional trust. When you accept a dollar bill, a gold coin, or a digital transfer from a stranger, you do not need to trust the stranger. You only need to trust the token.

But how did this leap actually happen? It wasn’t designed by a visionary or decreed by a king’s master plan. Instead, money was an accidental invention. It emerged from the bottom up, sustained by stable local equilibriums where substituting a token became easier than finding a perfect barter match. Once stumbled upon, this system spread through evolutionary fitness at a societal level. Societies that adopted this scalable trust out-cooperated, out-traded, and out-grew those that remained trapped by the limits of the Kinship Tax.

Follow-up

This post has focused on why money is as useful as it is to humans, and why that usefulness is best described as an extension of trust. This falls into the Technologies of Trust series. While this first post updates on the value and concept, others will deal with history and deeper meaning.

This choice carries commercial, security, and community implications. However, as technology evolves, traditional arguments for both models require a second look.

Security Implications

Advanced security scanning, like Mythos, will create new motivations for keeping software closed source.

Historically, the “million eyeballs” effect ensured that open-source security defects were quickly found. But in an era of AI-driven scanning, the equivalent of a billion eyeballs can simply be purchased as compute tokens.

Consequently, the primary security benefit of open source declines, while the advantage of closed source—forcing attackers to probe a compiled “black box” rather than reading a blueprint—remains. We should expect the security balance to shift accordingly.

Commercial Implications

Conversely, AI tools capable of reverse-engineering software from specifications weaken the commercial moat of closed source. If a replica can be generated from behavior alone, the protection of hidden code diminishes.

We should be careful not to overstate those capabilities. While advanced AI tools can create working replicas in many cases, a simple approach to this will produce a less capable, less secure and less maintainable replica. And an advanced approach will require a lot of tokens (which you must pay for), and the efforts of someone who knows what software needs to be good software.

Still, even with those qualifiers, a shift occurs, and developers are left with a little less commercial motivation toward closed source. An open source software package that binds a community to it could be a more stable commercial decision.

The Background Shift

While both of these implications are interesting, they are both watered down by the shift toward managed software that’s progressed over the last decade. Software as a Service, and its variants (Platform as a Service, Infrastructure as a Service) involve a third-party taking responsibility for some part of the managing running software. Management provides a way to offer value beyond the observable parts of the software. In the realm of security, the privilege of management can be used to layer protections. In the realm of commercial implications, value may derive from the efficiency and organizational capabilities to operate the software well.

Those security protections allow providers to rely heavily on open-source repositories for foundational logic, but wrap those deployed software in managed, closed-source service layers. This intermediary role is crucial. It creates a secure boundary where security teams can insert active, AI-driven monitoring and take an adversarial role against attackers with the advantage of obscurity. By funneling interactions through this managed layer, threats can be caught and mitigated before they ever touch the raw, open-source code underneath.

In a sense, open-source both won and lost, as the dominant shift was not from closed-source executables to open-source repositories, but from close-source executables, to managed service deployments based on open-source repositories. The managed service layer provides many of the security benefits of closed-source executables, by allowing security teams to take an active adversarial role, with an obscurity advantage. By having some private tools and techniques, they could often have proactive responses to attacks, rather than only reactive ones. This overall mix, millions of eyeballs on the source, with additional managed layers has been a potent one, and will remain so. That said, we should expect some change in the balance here, with a greater part of the managed layers as closed source.

Conclusion

In the realm of managed services, I’d expect the net result to encourage doubling down on the trend. Proprietary layers to create well managed services will proliferate. Competition with open-source software will not be a priority, but proprietary forks and extensions that improve performance, manageability or security will be.

One question is, who will donate the tokens for scanning open-source repositories? You can’t expect open source developers to buy and donate tokens for scanning the same way they donated their time. Industry cooperation, sponsorship and coordination will be needed here.

The shift from traditional software to AI is a shift from deterministic systems (where a specific input leads to a specific output) to indeterministic systems (where outputs are probabilistic and fluid). Almost every magical capability of AI is downstream of this indeterminism. But crucially, so are its most frustrating limitations.

If there is one fatal misunderstanding of AI today, it’s that we are engaging with this shift inadequately. “Indeterminism” has entered the lexicon, but usually only at a surface level. And because we are stuck on the surface, the loudest debates about AI have become incredibly boring.

Why the Extremes are Boring

Let’s look at the three loudest factions in the AI debate.

First, the AI doubters. They look at the unpredictable, indeterministic nature of large language models and declare it a failure. To them, a system that hallucinates cannot be trusted, and therefore cannot be useful. This is a boringly misguided example of confirmation bias. Humans are highly indeterministic—we forget things, we make math errors, we have bad days—yet we’ve muddled along reasonably well. How? By inventing deterministic tools to anchor us: long multiplication, checklists, standard operating procedures, etc. The doubter assumes you can’t extract value from an unpredictable system when you need reliability. History proves otherwise.

Second, the AI doomers. They also view indeterminism as a critical failure, but in the opposite direction. They are painfully aware of the immense power of AI systems and assume that this power is inherently uncontrollable. While this makes for a more gripping narrative than the doubters’ view, it strips away human agency. We’d have only one option left, don’t create powerful AI. Setting aside whether it is even possible to perpetually prevent its creation, this fatalism leaves no room for a practical conversation about how to retain control.

Finally, the radical accelerationists. They acknowledge the wild nature of AI but fall prey to a blind optimism, assuming a purely indeterministic system will somehow self-regulate and perfectly align with our needs. This is just as boring. The need for control is not irrational, nor is control a given. If control is achievable, it will demand a deliberate, concerted effort, requiring understanding every tool to engineer that control.

If you want to find interesting conversations, look for the solution seekers.

The Solution Seekers: Layers and Workflows

The most compelling builders today are those who reject both absolute pessimism and absolute optimism. They recognize that solutions aren’t singular or total. The most promising path is layers and workflows that mix and join determinism and indeterminism.

Think about how we manage high-stakes reasoning in the physical world—like in an intensive care unit or the cockpit of a commercial jet. We don’t rely entirely on the raw, in-the-moment reasoning of a doctor or pilot; human reasoning is brilliant but fluid, prone to fatigue, distraction, and variance. But we also don’t rely entirely on rigid, unyielding flowcharts, because a flowchart cannot reason through a novel, complex anomaly.

Instead, we design workflows that rely on both. We build strict, deterministic protocols—mandatory checklists, hard limits on medication dosages, automated collision warnings—to create a safe, predictable framework. Inside that framework, we rely on the judgement of a doctor or pilot to handle context, nuance, and problem-solving. Protocols enforce absolute boundaries; experts provide reasoning. Frameworks change, doctors update their own based on their learning, with debate and review, inside another layered framework.

This is the architecture of the AI future. AI will dominate the next generation of software, but it will not render deterministic code obsolete. Instead, code is how protocols are encoded. Those route, authorize, evaluate, and constrain indeterministic AI actors. Control points written in deterministic code will provide the necessary mechanisms to enforce rules, isolate agency, and supply safety. AI will be called upon within those specific boundaries to reason, interpret intent, and adapt to the messy reality of the user.

The Myth of the Developers Demise

This need for control has profound implications for how software is built. Recently, the term “vibe coding” has emerged to describe the practice of building software through natural language interactions with AI. A maximalist subgroup makes an extreme claim that with vibe coding, developers are obsolete and users will prompt their own custom software into existence on the fly.

This misses the fundamental purpose of a developer. A developer’s job is not to write code; a developer’s job is to remove effort for the user. Developing is ultimately not about producing code, but about producing reusable, accessible capabilities for users. An accessible capability is one that requires the least effort to access, and a reusable one is one that can be applied to multiple situations. Code is just the mechanism.

When developers create software, they establish guardrails, conventions, and reusable patterns. Sometimes, a user wants absolute flexibility, and a fluid AI companion is perfect. But often, a user wants rigid reliability. They want to press a button and know exactly what will happen. It’s easy to forget, amidst the explosion of AI capabilities, that rigidness has immense value.

It’s tempting to view recent advancements as a single evolutionary timeline—assuming we are moving from hand-written code, to AI-assisted code, to a future where code is entirely replaced by just in time reasoning of AI agents. That is a mistake, over-extending a trend. Committed code, generated, reviewed, tested and committed as stable will exist in abundance. Just in time generated code, executed in a protected sandbox will also be used abundantly.

The use of models and instructions, reasoned upon just in time, shifts the balance point between flexibility and rigidity, but it won’t abandon code nor the developer.

A Shared Experience: Taming the Machine

For users, future software interfaces will be a mix of structured and natural. Learning to navigate the difference between them will be a vital modern skill.

Structured interfaces (buttons, menus, traditional apps) sit atop deterministic systems. You can trust them to follow a plan. However, that plan was written by a developer. If the developer didn’t anticipate your specific need, the software becomes frustrating. You are forced to learn its non-intuitive logic.

Natural interfaces (chatbots, voice agents) sit on top of indeterministic systems. They can do things developers never anticipated and can interpret your unique intent. But they make assumptions. Using an AI interface is like ordering from a waiter at a restaurant. You need to develop an instinct for how your communication might be misinterpreted. You need to know when the system will ask a clarifying follow-up question (”soup or salad?”), and when you need to be proactively rigid and structured in your commands (”hold the mustard”). Make a mistake here, and you end up with a mustard-covered sandwich. Everyone then has to start over from scratch, and someone has to pay for the waste.

Interestingly, the people building the software are going through the exact same transition.

Developers are increasingly using natural language to write code. For a brief moment, this felt like magic without rules—just type what you want, and the machine builds it. But developers are quickly realizing that an AI coding assistant is just as indeterministic as a chatbot. If they aren’t careful, they end up with the equivalent of a “mustard-covered sandwich” deep in their codebase.

Because of this, we are watching a new kind of structure reemerge in software development. Developers aren’t abandoning natural language, but they are scaffolding it. They are learning when to let the AI riff creatively, and when to enforce strict, deterministic tests to verify the AI’s output. The developer’s job is evolving from writing rigid rules by hand to managing the chaotic intelligence that writes them, locking its best outputs into place so they can be relied upon tomorrow.

Conclusion

For decades, our relationship with computers was fundamentally one-sided: humans had to learn to speak like machines. We memorized menus, learned strict syntax, and clicked exact sequences of buttons. We were forced to be rigid operators of deterministic systems.

AI flips this dynamic, but it introduces a new burden. The era of the comprehensive user manual is over, because you cannot write a complete manual for a probabilistic system. Its capabilities are discovered through interaction, not documented in a spec sheet.

This is why understanding the architecture beneath your feet is no longer just a concern for software engineers. It is a vital literacy for everyone.

If you are an everyday user, recognizing whether you are interacting with a deterministic system or an AI agent changes how you engage. The caution you apply to inputs and outputs should shift. For deterministic systems you should provide what is required and just what is required. For AI systems consider where elaboration yields better results, and vagueness leads to guesswork. Unless you need guesswork, avoid triggering that path.

If you are trying to predict where the industry is going, looking for these architectural layers is the only way to cut through the boring extremes of blind hype and cynical doom.

And if you are a builder—whether you are writing thousands of lines of code or just stringing together a few tools to solve a daily problem—understanding this duality is your ultimate advantage. The future of technology isn’t about choosing between the rigid reliability of the past and the creative chaos of the future. It’s about learning to bolt them together.

One thing that is still being missed, is why the long term favors the defenders. One of the reasons is that fewer defects is good for defenders in a generally absolute way. But another one relates to costs and benefits. Read that economist article linked and the final statements, suggesting that defenders will have to pay highly to discover defects.

Now reflect, that this isn’t new, it has always been expensive to discover defects. The risk that products like Mythos bring is that they lower the cost of discovering defects to exploit. The solution is to raise that cost. That might tempt you to suggest you should rewind the clock, and never invent Mythos. That’s not a solution though as eventually attackers would invent something similar, and you would then lose any control and advantage from the defenders being the first with access.

Instead the solution is that you find as many easy defects as you can and fix them. The first 100 defects might cost $20,000 / defect to discover. The next 100 might be $40,000 per, etc. Along the way you end up with defensive layers that are more and more reinforcing, and the cost for attackers to discover defects goes up, especially if they have less sophisticated tools, and/or have to spend a lot to first illicitly gain access to tools. When Mythos is publicly released you can generally assume providers will increase their attempts to find and ban users with ill intent. Those protections create costs for attackers, such that if a defender can find a defect for $20,000, an attacker might need $100,000. The attacker’s main advantage is they just need one, but as unpatched defects become more rare and harder to find that advantage tends to shift toward favoring the larger aggregate budgets of defenders.

The defenders have a strong advantage in terms of money. Where they struggle is in organization, because they have a much harder organizational problem to solve. The hard part about being a defender is getting changes deployed everywhere quickly. Once attackers find a defect, they can try and use it everywhere. If they find it first, that works in a lot of places. If they find it second, it’s dependent on how organized the deployment process is.

And this is why the long term economics favor the defender. Statistically, most defects are found first by defenders, due to larger budgets. As the period between discoveries gets longer, the chances that attackers have really good targets declines. That lowers their cost/benefit, which probably also lowers their actual budget. Criminals invest in things that make (them) money, not ones that lose it.

In Part 1 of this series, we explored how AI is fundamentally altering software control through the lenses of determinism and scope. We traced the journey from passive, strictly bounded chatbots to the threshold of active agents—AI systems capable of autonomous, multi-step planning. But what happens when these indeterminate systems are given broader scope and powerful tools? The consequences ripple outward, reshaping not just the security of our infrastructure, but the shape of our workflows and emotional relationship to work. To understand the recursive systems of tomorrow, we must dive into the agent ecosystem itself.

The Agent Ecosystem

Agents represent a significant shift in control, trading linear human prompting for continuous indeterministic planning.

To understand how these agents operate, we must briefly consider tools. Agents use tools to accomplish their plans. Tools can be anything, and which tools an agent is provided with define its constraints. You can provide an agent instructions, cautions, and directives through its prompt and context, but like anything in an agent, it’s indeterminate.

A tool might be as basic and low-risk as retrieving a specific account balance, where the boundaries are tight and predictable. It might be as broad as searching gigantic repositories like the entire internet or an organization’s internal files, which escalates risk by exposing the agent to untrusted data or sensitive information.

A broader path still is the ability to create and execute computer code, introducing severe risk if left unchecked. That might initially seem like it loses all constraints, allowing the agent to perform unanticipated or dangerous actions. However, code can be executed in a sandbox that limits how it communicates and what data it can access. Assuming the sandbox is secure—which requires careful planning, inspection, and testing—restricting communication to untrusted sites prevents data exfiltration or external control. Just as critical is controlling the credentials provided to the sandbox. Strictly limiting credentials restricts the agent’s ability to update records or access systems outside the purview of its current authorized activity. Together, these boundaries provide the necessary mechanism to constrain this high-risk capability.

Tool use isn’t restricted to retrieving information, either; it can allow changing information, which can trigger further actions. This is an area that requires much more caution, doubly so for writes and actions that are irreversible. Beyond that obvious observation, two other dimensions enter in. First, since an agent’s plan is indeterminate, the ability for a designer to remove the risk that it performs actions in unanticipated ways is vastly more complex than when working with a deterministic plan. Second, we must account for prompt-injection—the risk that something an agent has read can influence its choices, resulting in actions desired by an attacker rather than the user or designer. There are protections against this type of attack, but it would be foolish to consider them foolproof.

With that foundational understanding of how agents act on the world, we can observe this frontier opening up across escalating levels of scope:

Standalone AI Agents

Unlike a chatbot that waits for a prompt, a standalone agent is given a high-level objective, allowed to indeterministically generate its own step-by-step plan, and execute it using available tools (like searching the web or scraping data). While the planning is continuous and autonomous, the agent still typically operates within a relatively bounded scope, restricted by specific APIs to prevent runaway consequences.

Like chatbots, there are standalone agents from OpenAI, Claude, Google, and others. In fact, most chatbots have silently become agents1, though still with many constraints.

In time, these standalone agents may have more and more autonomy. But from the perspective of this framework, the fundamental aspect of taking user input, deriving a plan through an indeterministic process, and executing that plan won’t re-enter the realm of determinism until it invokes a tool2.

Agents Embedded in Applications

Moving beyond the simple “embedded AI node” discussed in Part 1 involves agents operating continuously alongside users within a shared software environment. Consider a complex data analysis platform: the human user might explicitly invoke deterministic tools to filter data, while an embedded agent operates in the background, autonomously invoking its own set of analytical tools to highlight anomalies. The application becomes a hybrid ecosystem where human indeterminism and agent indeterminism collaborate in real-time bounded by the application’s guardrails.

Agents embedded in applications have an advantage over agents called by other agents: the input data is controlled by the calling application. Still, remember that the applications agents are embedded in may themselves be working with dynamic data. A data analysis platform has many data sources; are they all vetted and invulnerable to an injection attack?

Another common example today is agents embedded into development workflows. They can reason about code, look for security issues or defects, generate fixes, and submit them as pull-requests for developers to review, effectively combining the code-generation function with the embedding function.

Agents Using Agents

An agent can become a tool used by other agents. To think about why this is valuable, you want to first understand that agents generally have a few components. At their core, they create plans via the GenAI model. They generally have some sort of instruction (or persona) file. They’ll also have access rights or boundaries associated with a tool list.

The instruction file is the interesting part here, as it provides the reason for a distinct agent. In the simplest version, it might describe a persona (“you are an insurance adjuster”), but this relies on the GenAI model to blindly decide how an adjuster behaves. If you’re building an agent, you want more control. A detailed instruction file can’t shift you to a fully deterministic world (if you want that, you should write code), but it can reduce variability.

Another aspect that can remove ambiguity is placing restrictions on inputs and input sources. If an agent expects to receive user input, it has to be fully prepared for anything. If it’s called from an application, those expectations are more constrained.

Agent input expectations fall into two categories: expectations formed around successfully fulfilling its goal under valid usage, and expectations formed around avoiding taking action on behalf of an attack. These two have some non-overlapping aspects. If an input is suspected of being for the purpose of an attack, there’s no need to try to do anything other than quit and refuse action. But the consequences of allowing an attack are generally far more severe. On the other hand, failing to successfully complete an action is less severe, but it’s less acceptable to give up because of uncertainty.

If we had wanted certainty, and accepted inaction for uncertainty, we should have written a traditional application, not an agent. In many ways, the creation of agents with sophisticated instruction files is a type of meta-programming that never coalesces into a deterministic form. While we could use vibe-coding to generate an application, creating an instruction file for an agent has a similar outcome, except designers never get the chance to validate the plan for each agent execution. We might restore some of that validation through a human-in-the-loop workflow, but the agent designer won’t be in the loop unless they are also the user.

Furthermore, when input comes from another agent, the expectations on input are not very clear. It’s not as unclear as coming from an untrusted user, but since the user of the calling agent might be less than fully trusted, we have to consider the possibility that a malicious user could cause the calling agent to pass dangerous inputs to the called agent. Depending on the design, that might be difficult, but proving it’s impossible is a high bar without some deterministic system in the path.

Agents Building Applications

At the apex of the framework we have agents building applications. Instead of a human using an AI tool to write code, an autonomous agent—or a multi-agent framework—is given the broad scope to architect, write, test, and deploy entire applications. Operating within bleeding-edge, emerging ecosystems like Steve Yegge’s concept of Gas Town, an overarching agent might autonomously spawn specialized “code worker” sub-agents to solve specific architectural problems. This introduces the reality of deep recursion: AI systems dynamically writing, testing, and deploying new AI systems at machine speed.

Agents Building Agents

An alternate apex is agents building other agents. While both scenarios rely on deep recursion where at least one level is indeterminate, the agents-building-agents path stores its recursive plans in natural language, rather than a programming language. Tools like Claude Cowork and Claude Code are bordering on this. Technically, they’ve always been capable of it, as a developer can create recursion somewhat trivially.

The barrier here has generally been security. It’s rather easy to say, “Agent A calls Agent B to create Agent C, which can call Agent B” (look, I just did it!). The hard part is whether that’s a good idea. Projects like OpenClaw push this further. When agents build other agent skills or update through tools like Moltbook, they are acting at this highly complex, deeply recursive layer. OpenClaw has some security controls, but not enough to prevent many users from making significant mistakes.

Another example here that illustrates the movement from applications to agents, is Gas Town in Gas City. Gas Town, the original multi-agent orchestration system for Claude Code, GitHub Copilot, and other AI agents, was an application. When Yegge wrote Gas City, a “orchestration-builder SDK for multi-agent systems”, Gas Town became “code free”, and instead a bundle of prompts and skills.

Ripple Effects: Cybersecurity

One constant that comes from more complex systems is greater challenges at securing them. All else being equal, indeterministic systems, either fully or wholly, are more complicated than fully deterministic ones. Security is traditionally about protecting deterministic plans from indeterminate actors (human hackers). Each layer adds complexity as well. Attackers will have new tools and will use them against the highest value targets that have weak points.

Fortunately, all this does not come without some benefits, both inside and outside the realm of security. Inside the realm of security, access to dynamic systems like agents allow for faster responses. As I’ve explored previously, this means security can’t wait—it will force not just an overdue commitment to defense, but complete organizational changes.

The highest value targets know this. They will rapidly adopt new defensive techniques, patching weak points faster than ever before. Where we will see more successful attacks is against more moderate value targets. Some operate efficiently and will adapt, but others will face an “adapt or fail” pressure cooker. They will suddenly find themselves defending against highly sophisticated, indeterminate automated attacks.

Overall, however, this is a narrative of optimism. As Dario Amodei notes, “the offense-defense balance may be more tractable in cyber, where there is at least some hope that defense could keep up with (and even ideally outpace) AI attack if we invest in it properly.” Security must simply shift toward robust bounding and sandboxing of environments, rather than assuming the predictability of the software operating within them.

Automation and Workflow Change

This shift in control—from human-driven applications to autonomous, recursive agents—isn’t happening just for the sake of technological novelty. Ultimately, the goal is, and has always been, automation. AI changes opportunities for automation by lowering automation costs that were previously prohibitive.

Classification and ML

Machine Learning (ML) is an AI technique that achieved broad use earlier than Generative AI. Classification and prediction tasks were the core use cases. It’s generally less well known than Generative AI because those use cases fit into embedded AI workflows that have less direct user interaction. But that doesn’t mean they haven’t been effective. Generative AI has some overlap, but it’s useful to note ML is not obsolete—it will continue to dominate specific classification tasks where the trade-offs favor highly optimized, low-compute execution.

But GenAI is shifting the math for automation’s long tail where engineering effort is the limiting factor. Traditional ML models require a significant engineering investment to train. While that engineering could theoretically be automated, doing so would bring you back to using GenAI to generate the code. When a general-purpose GenAI model can perform a task at equal quality without that upfront engineering time, it opens up a new option for countless use cases that were never practical to tackle with traditional ML. AI provides the structure to finally capture and automate the tacit knowledge we previously had to rely on humans to execute.

Workflow Change

This, and the other uses of Generative AI, allows a deeper decomposition of workflows. In prior methodologies, it was too expensive to capture the output of specific, granular steps. Those steps were done “in the head” of human workers, existing only as “tacit knowledge.” A workflow that may have produced better results might have been avoided because the human cost of data preparation or classification was too high.

While it’s possible to replicate the same workflows, trade-offs have shifted. Consider a nurse who has learned a new symptom for a patient. That nurse may lack the depth of medical knowledge or the patient’s full history, so may not be able to do more than record that information until the patient’s doctor can review it. But an AI system can reanalyze a patient’s information nearly instantly. It can recategorize data, make new recommendations, or provide the nurse with the relevant medical information and patient history. This could allow next steps that both improve efficiency, but also improve outcomes. Maybe it’s an extra test, or an extra question, or a life-saving reaction.

Because AI lowers the cost of executing small, indeterminate tasks, we can now decompose workflows further. What workflow is optimal depends heavily on hand-off costs. In the example of the nurse, the hand-off costs to the doctor were the impediment. Human to human or human to machine hand-offs are expensive compared to machine-to-machine. When tasks shift from human dependent to machine dependent, a reorganization of workflow makes sense. A particular flow which was used to avoid handoffs may no longer be necessary. More importantly, those handoffs that remain, will have higher relevance than before, and optimizing for them, rather than those that are no longer needed, takes priority.

Initially, we should expect to see pilots, trials, and first iterations operate within existing workflows. Changing workflows requires planning, training, and is hard to reverse or do incrementally. As such, it follows in later iterations. But many of the largest gains are realized with those iterations.

The Human Side

Workflow change can also have a significant impact on satisfaction amongst workers. Hand-offs can be the most frustrating type of work, depending on your personality type. Human to machine hand-offs become frustrating when flexibility is lacking, and you feel like your task is to fit a round-peg into a square hole. Human to human hand-offs can sometimes be enriched by the personal interaction, but they also expose you to misaligned goals, competing priorities and personality conflicts. Personal interactions are a lot more reliably fun when you get to pick the individuals and circumstances.

Worst of all, machine to human hand-offs can create the impression that you’re serving the machine, not the inverse. All hand-offs can have that effect somewhat, but it’s especially hard if there’s an endless list of machine generated work. It helps to detach from the “end” and focus on the progress here, but when an organization turns that into a metric, ruthlessly gamifies it, and fails to consider the impacts, it requires extreme stoicism to avoid burnout.

It’s important to remember the human side with workflow change. Ruthless metrics fail in the long term. Leaders should watch for that, and avoid allowing short-term goals to overwhelm long-term health. It’s not always clear that this fulfills the “bottom-line” if that’s financial performance. It’s always clear it’s better for broader goals than financial, but even for financial goals, the benefits are likely there, though harder to see.

Conclusion

The era of software as purely rigid, deterministic planning is ending. In its place is the rapidly expanding Agent Ecosystem. By integrating indeterministic models into our autonomous systems, and allowing agents to build other agents, we are trading perfect predictability for unprecedented scale and capability.

As the scope of these systems increases, our primary job shifts from writing static instructions to managing boundaries. We must build robust technical sandboxes to protect our cybersecurity, and we must build equally robust organizational boundaries to protect human workers from the burnout of endless machine-to-human hand-offs. We have to design systems that serve humans, not the other way around.

The question is no longer just “What can the chatbot say?” The real questions are: How much scope are we willing to give to indeterminate plans? How will we effectively bound the recursive systems of tomorrow? And, ultimately, how do we bound ourselves?